Published by Gist GEO · About our content
Check it yourself
- In your browser, go to your website’s address followed by
/robots.txt. - Look for the names in the table below.
- Under any of those names, a line reading
Disallow: /means “keep out of everything.”
Also look for a rule under User-agent: *. That applies to every crawler without its own rule. Disallow: / there shuts everyone out.
| If this name is blocked | This is what you lose |
|---|---|
OAI-SearchBot | Appearing in ChatGPT’s search answers |
Claude-SearchBot | Visibility in Claude’s search results |
PerplexityBot | Appearing in Perplexity’s results |
Googlebot | Google Search, including AI Overviews and AI Mode |
GPTBot, ClaudeBot, Google-Extended | Nothing in search. These control model training |
The usual mistake
Somebody wanted to keep the company’s content out of AI training, so they blocked everything with “AI” in the name. Training and search are separate switches. Block the search crawlers and you leave the answers.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /# AI search may read the site
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Whether to refuse training is a business decision, and plenty of companies do. The second version keeps that choice and stays visible in AI search.
Check your firewall too
robots.txt is not the only door. If your site sits behind a service such as Cloudflare, it has its own AI crawler settings, and they can override what your robots.txt says.
Cloudflare changed its settings on September 15, 2026. Its Block option for AI training now also applies to the crawlers from Google, Bing and Apple, which takes a site out of ordinary search as well. The setting that refuses training and keeps search is called Disallow AI Training. If someone at your company switched on an AI block in the past, ask them to look at what it is set to now.
Who can do this
Whoever manages your website. Editing robots.txt takes about ten minutes, and the change needs roughly a day to be picked up.
One caution: these crawler names and rules come from each company’s own documentation, and they do change. Check the linked pages below before you edit anything.
Sources
- Overview of OpenAI crawlers. OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler?. Anthropic, Apr 7, 2026
- Perplexity crawlers. Perplexity
- Google’s common crawlers (updated July 14, 2026). Google
- Changes to AI crawler controls. Cloudflare, Sep 15, 2026
Questions marketers ask
If I block GPTBot, do I disappear from ChatGPT?
Not from its search answers. OpenAI says GPTBot is for training and OAI-SearchBot is for search, and that each setting works independently.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended controls whether your content is used to train its Gemini models, and that it has no effect on Google Search. AI Overviews and AI Mode are built on the Search index.
How long do changes take?
About a day. OpenAI and Perplexity both say a robots.txt change can take around 24 hours to register.