OAI-SearchBot vs GPTBot

Three separate jobs

These are policy decisions with different consequences, and they are worth deciding separately rather than with a single blanket rule.

• Search and discovery. Retrieval that supports answers with links. Blocking it affects whether you can appear.
• User-triggered retrieval. A fetch made because a person asked about a specific page in a conversation.
• Model training. Collection of content used to train models. Blocking it does not affect whether you appear in answers.

The common mistake

A publisher decides not to contribute to model training, adds a broad disallow, and later finds the site is absent from answers that link out. The training decision was implemented, and so was an unintended visibility decision.

The inverse also happens: a site allows everything in order to appear in answers, without ever making an explicit decision about training.

How to check what you are actually doing

• Read your live robots.txt rather than the version in the repository. Hosts and platforms sometimes serve a different file.
• Check each user agent by name rather than relying on a wildcard rule.
• Confirm the provider's current agent names in their documentation before editing. Names change.
• Verify that access is what you intended after deploying, not before.

What this page does not establish

• Allowing a crawler permits retrieval. It does not promise inclusion, ranking, or that any answer will select your page as a source.
• Agent names and the surfaces they serve are set by the provider and change without notice. Always confirm against current provider documentation.

Sources

• OpenAI crawler and bot documentation
• GetSounding measurement methodology

Run a free audit to see this measurement format applied to your own domain, or read the full measurement method.