Published by Gist GEO · About our content
Three different jobs
People say “AI crawler” as if it were one thing. It is three.
| Job | What it does | If you block it |
|---|---|---|
| Training | Collects pages that may be used to train future AI models | Your content is left out of training |
| Search | Builds the index an assistant searches when it writes an answer | You may not appear in that assistant’s answers |
| Live fetch | Reads one page because a person just asked about it | The assistant can’t read that page for that person |
These are separate switches. You can refuse training and still be found.
The main ones, by name
| Company | Training | Search | Live fetch |
|---|---|---|---|
| OpenAI (ChatGPT) | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic (Claude) | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | Not used for training, per Perplexity | PerplexityBot | Perplexity-User |
Google-Extended | Googlebot |
Two notes on Google. Google-Extended is a setting, not a crawler: it tells Google whether your content may be used to train its Gemini models. And Googlebot is the ordinary search crawler. Google’s AI Overviews and AI Mode are built on the same index as its search results.
These names come from each company’s own documentation, and they change. Check the sources below before relying on them.
Why a marketer should care
If the search crawler can’t get in, the assistant has nothing of yours to quote. It will describe you from other people’s pages instead, or leave you out.
It takes two minutes to find out whether that is happening: Can AI read my website?
Sources
- Overview of OpenAI crawlers. OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler?. Anthropic, Apr 7, 2026
- Perplexity crawlers. Perplexity
- Google’s common crawlers (updated July 14, 2026). Google
Questions marketers ask
Can I block training and still appear in AI answers?
Yes. The training and search crawlers have different names and are controlled separately. Can AI read my website? shows how.
Do AI crawlers follow robots.txt?
The companies say their training and search crawlers do. The live-fetch agents are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.
How do I know which crawlers visit my site?
Your server logs or hosting dashboard list visits by name. Whoever runs your website can pull a report.