AI bots fall into four groups: training crawlers, search and citation crawlers, user fetchers and undeclared scrapers. Blocking a training crawler costs you future model knowledge. Blocking a citation crawler removes you from live AI answers. This page lists every major bot and what blocking it costs.
Key Takeaways
- Separate training from citation. Blocking GPTBot and blocking OAI-SearchBot have very different consequences.
- Google-Extended and Applebot-Extended are opt out tokens for model training. Blocking them has no effect on search visibility.
- User fetchers follow different rules. Each vendor decides whether robots.txt applies when a person asks an assistant to read a page.
- Only 21 of 442 B2B SaaS companies in my crawler study block any AI crawler at all.
AI crawler: An automated bot operated by an AI company that visits websites, either to collect training data or to fetch pages so an assistant can answer and cite a question.
What Kinds of AI Bots Visit Your Site?
Four kinds, and each one calls for a different decision. Most robots.txt files treat them as one group, which is where the mistakes come from.
Blocking each group costs you something different.
The rule worth remembering: separate training from citation. Blocking a training crawler slowly reduces what future models know about you, and it is a legitimate choice.
Blocking a search or citation crawler removes you from live AI answers right away. For most B2B companies, that works against you.
Which AI Crawlers Exist, and What Does Blocking Each One Cost?
| User agent | Operator | Type | Respects robots.txt | If you block it |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | Yes | Left out of future ChatGPT training. No effect on live citations. |
| OAI-SearchBot | OpenAI | Search and citation | Yes | You drop out of ChatGPT search citations. |
| ChatGPT-User | OpenAI | User fetcher | Docs say it may not apply | Fires when a person asks ChatGPT to read a page. See the fetcher section. |
| ClaudeBot | Anthropic | Training | Yes | Left out of Claude training. No effect on live citations. |
| Claude-SearchBot | Anthropic | Search and citation | Yes | You drop out of Claude’s answer index. |
| Claude-User | Anthropic | User fetcher | Yes | Claude cannot fetch your page when asked. |
| PerplexityBot | Perplexity | Search and citation | Yes, per its docs | You drop out of Perplexity’s index and citations. |
| Perplexity-User | Perplexity | User fetcher | Disputed | See the fetcher section. |
| Googlebot | Search, feeds AI Overviews | Yes | You disappear from Google Search and AI Overviews. Do not block. | |
| Google-Extended | Opt out token | Directive only | Opts content out of Gemini training. No effect on Search or AI Overviews. | |
| Google-CloudVertexBot | Enterprise retrieval | Yes | Affects grounding in enterprise Vertex AI builds. | |
| Bingbot | Microsoft | Search, feeds Copilot | Yes | You disappear from Bing and Copilot answers. Do not block. |
| Applebot | Apple | Search | Yes | You drop out of Siri and Spotlight suggestions. |
| Applebot-Extended | Apple | Opt out token | Directive only | Opts out of Apple Intelligence training. No effect on Siri or Spotlight. |
| Amazonbot | Amazon | Training and search | Yes | You drop out of Alexa answers and Amazon model training. |
| Meta-ExternalAgent | Meta | Training | Yes | Left out of Meta AI training. |
| Meta-ExternalFetcher | Meta | User fetcher | Varies | Fetches pages for Meta AI assistant features. |
| CCBot | Common Crawl | Data collector | Yes | You leave the open dataset many models train on. |
| DuckAssistBot | DuckDuckGo | Search and assistant | Yes | You drop out of DuckDuckGo’s AI answers. |
| YouBot | You.com | Search and citation | Yes | You drop out of You.com’s answer engine. |
| MistralAI-User | Mistral | User fetcher | Yes | Le Chat cannot fetch your page when asked. |
| cohere-ai | Cohere | Training | Undocumented | Left out of Cohere training. No official documentation. |
| Diffbot | Diffbot | Data collector | Varies | Removes you from Diffbot’s commercial knowledge graph. |
| Bytespider | ByteDance | Undeclared scraper | Reported not to comply | Little downside for most B2B. The most blocked bot in my study. |
A CSV version of this table is available, and I keep it in sync with this page.
Does Blocking Google-Extended Affect AI Overviews?
No. Google-Extended and Applebot-Extended are the most misunderstood rows in this table. Both are opt out tokens, which tell an operator to keep already crawled content out of model training.
Blocking Google-Extended does nothing to your Google Search rankings or your presence in AI Overviews. Those depend on Googlebot.
Companies that block Google-Extended to stay out of AI Overviews have made a change that does not do what they expect.
Do User Fetchers Respect robots.txt?
It depends on the vendor. Automated crawlers decide on their own to visit your site, and robots.txt governs them. User fetchers run because a person asked an assistant to read a specific page.
User fetchers run when a person asks. Each vendor decides whether robots.txt applies.
Anthropic honors robots.txt for Claude-User. OpenAI’s documentation says it may not apply to ChatGPT-User. When Cloudflare tested this in August 2025, it saw ChatGPT-User fetch robots.txt and respect it.
The Perplexity dispute
In August 2025, Cloudflare published evidence that Perplexity used undeclared crawlers to reach pages that had blocked its declared bots. Cloudflare said this involved rotating user agents and network providers, and removed Perplexity from its verified bot list.
Perplexity publicly disputed the findings and said the traffic was not its own. I have included both positions because both are on the record.
The practical takeaway holds either way: robots.txt only works when a bot chooses to honor it.
Which robots.txt Setup Should You Use?
Allow everything (right for most B2B companies)
User-agent: *
Allow: /
Block training, allow citation
This is the setup I found Canva using, and it was the most considered one in my sample. It keeps you in AI answers while opting out of model training.
User-agent: GPTBot
Disallow: /User-agent: ClaudeBot
Disallow: /User-agent: Google-Extended
Disallow: /User-agent: CCBot
Disallow: /User-agent: Meta-ExternalAgent
Disallow: /User-agent: Bytespider
Disallow: /User-agent: OAI-SearchBot
Allow: /User-agent: Claude-SearchBot
Allow: /User-agent: PerplexityBot
Allow: /
Block every declared AI bot
For a B2B software company, this is usually a mistake. Undeclared crawlers ignore it anyway, so the main effect is removing yourself from the answers your buyers read.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
Disallow: /
One warning for all three setups. A Disallow: / rule under User-agent: * blocks every crawler that follows the rules, Googlebot included. Five of the 442 sites in my study had exactly that.
What Should You Do Next?
- Read your robots.txt at yourdomain.com/robots.txt and compare it against the table above.
- Decide on training bots on purpose. Either allow them or use the block training, allow citation setup.
- Confirm the search bots can reach you. OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- Check your subdomains. Your docs site and help center each have their own robots.txt.
- Bookmark this page. I update it every quarter as new bots appear.
What This Directory Does Not Cover
- Whether a bot actually obeys your rules. This page lists each operator’s stated policy. Server logs are the only way to confirm what a bot really does on your site.
- Every scraper on the internet. It covers bots operated by AI companies and major data collectors, and leaves out the long tail of anonymous scrapers.
How I Maintain This Directory
I review this directory every quarter and update the date each time, even when nothing changes. A stale date makes a reference page useless faster than a missing row.
If a bot changes its user agent, its policy or its documentation, or if I have something wrong, email hello@growfusely.com and I will verify and correct it.
Sources: operator documentation from OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Meta, Microsoft and Common Crawl, Cloudflare’s August 2025 report, and my own analysis of 442 B2B SaaS robots.txt files.
How to Cite This Directory
Dholakiya, P. (2026). The AI Crawler Directory. Growfusely. https://growfusely.com/blog/ai-crawler-directory/
Why Partner with Growfusely for AI Search Visibility?
Most AI visibility advice starts with content. I start with access, because a brand AI engines cannot read will not be recommended, however good its content is.
- A crawler and access audit covering robots.txt, rendering and every user agent in this directory.
- A deliberate training versus citation policy, set as a business decision.
- Citation measurement across ChatGPT and Perplexity, so access changes are judged on whether you appear in answers.
- Original research behind every recommendation, including my study of 442 B2B SaaS sites.
If you want a second pair of eyes on what your site allows and blocks, schedule a call.
FAQs
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content to train OpenAI’s models. OAI-SearchBot indexes pages so ChatGPT can answer and cite questions. Blocking GPTBot affects training. Blocking OAI-SearchBot keeps you out of ChatGPT search answers.
Does blocking Google-Extended affect AI Overviews?
No. Google-Extended is an opt out token for Gemini training. AI Overviews come from Google Search, which depends on Googlebot.
Can robots.txt block user fetchers like ChatGPT-User?
Not reliably, and the answer differs by vendor. Anthropic honors robots.txt for Claude-User. OpenAI says it may not apply to ChatGPT-User. Perplexity describes Perplexity-User as an agent acting for a person.
Should a B2B SaaS company block AI crawlers?
Blocking training crawlers is a legitimate choice. Blocking search and citation crawlers removes you from AI answers, which is rarely the intent. Only 21 of 442 companies I studied block anything at all.
How often do AI crawlers change?
Operators add bots and change policies several times a year. I review this directory every quarter.
Read Next
- I checked 442 B2B SaaS sites for AI crawler blocking. Who blocks AI bots, and who blocks the wrong ones.
- Entity drift. Why AI engines hedge when your descriptions disagree.
- llms.txt adoption in B2B SaaS. What 510 companies publish for AI systems.
Pratik Dholakiya is the Founder of Growfusely, a SaaS SEO and AI Search Visibility agency.
