Only 21 of 442 B2B SaaS companies block any AI crawler in robots.txt, and most of those target scrapers. Thirteen block a training bot. Exactly one, Figma, blocks the search bots that put a brand in AI answers.
Key Takeaways
- 21 of 442 companies (4.8%) name and block at least one AI crawler.
- The most blocked crawler is Bytespider, operated by TikTok’s parent, at 13 sites. OpenAI’s GPTBot is blocked by 5.
- 13 companies block a training bot by name. One blocks a search bot by name.
- Canva is the clearest deliberate setup: it blocks ten training and scraping bots and explicitly allows Perplexity’s bots.
- 53 companies explicitly invite OAI-SearchBot and PerplexityBot in by name.
- Five companies carry a blanket robots.txt rule that blocks every crawler, Google included.
AI crawler: An automated bot operated by an AI company that visits websites. Some collect content to train models. Others fetch pages in real time so an assistant can answer a question and cite the source.
Why Did I Run This Study?
For two years, the prediction was that companies would wall off their content from AI. Publishers blocked GPTBot in large numbers, and most people assumed everyone else would follow.
For a B2B software company, that would be self defeating. You want AI engines to read your site, because that is how you end up in the answer when a buyer asks which tools to shortlist.
So I checked whether it happened. I read the robots.txt file of 442 B2B SaaS companies and looked for 18 AI crawler user agents.
How Many B2B SaaS Companies Block AI Crawlers?
Very few. 21 of 442 companies, or 4.8%, block at least one AI crawler by name. The other 421 leave every crawler I checked free to read their site.
Each square is one of 442 B2B SaaS sites. Coral squares block at least one AI crawler.
That is the right call for this category. A publisher earns money when people read its pages, so an AI answer that replaces the visit is a loss.
A software company earns money when buyers choose it. An AI answer that recommends it is a gain. The incentives run in opposite directions, and B2B SaaS has mostly noticed.
Which AI Crawlers Get Blocked Most?
TikTok’s crawler, Bytespider, is blocked by 13 sites. Common Crawl’s CCBot follows at 9 and Diffbot at 7. OpenAI’s GPTBot is blocked by 5.
Number of the 442 sites naming and blocking each crawler in robots.txt.
The pattern points at crawlers that companies associate with bulk data collection. The bots behind the assistants buyers actually use are blocked far less often.
Do Companies Block Training Bots or Citation Bots?
Mostly training bots. Thirteen companies block a training bot by name, and only one blocks a citation bot. Those two groups do very different jobs, so the difference matters.
Sites naming and blocking at least one bot of each type, as a share of 442.
| Training bots | Search and citation bots | |
|---|---|---|
| Examples | GPTBot, ClaudeBot, CCBot, Google-Extended | OAI-SearchBot, PerplexityBot, Claude-SearchBot |
| What they do | Collect content to train future models | Fetch pages so an engine can answer and cite a question |
| What blocking costs you | Future models know less about you | You drop out of live AI answers |
| Sites blocking at least one | 13 | 1 |
Canva shows what the deliberate version looks like. Its robots.txt blocks ten training and scraping bots, including GPTBot, ClaudeBot, CCBot and Bytespider. It also explicitly allows PerplexityBot and Perplexity-User.
In plain terms, Canva’s position is: you may cite us, and you may not train on us. ZoomInfo does something similar, blocking seven data collectors while leaving every search bot alone.
There is a tradeoff worth knowing, though. Many AI answers come from what a model learned in training, not from a live search. I explain that split in how AI search visibility works.
A model that never trained on your content is less likely to know you exist. So blocking training bots protects your content, and it also narrows how well future models understand your brand.
Which Company Blocks the Bots That Cite It?
Figma is the only company in my sample that blocks citation bots by name. Its robots.txt blocks GPTBot, ClaudeBot, CCBot, cohere-ai and Google-Extended, which is a fair training stance.
It also blocks OAI-SearchBot, PerplexityBot and ChatGPT-User. Those are the bots that fetch pages in real time so an engine can answer and cite the source.
OpenAI’s documentation says sites that block OAI-SearchBot will not appear in ChatGPT search results. For a product as widely recommended as Figma, that gives up a lot of visibility.
I am not assuming it was an accident. A brand as well known as Figma may decide it does not need AI answers to be found. Most B2B SaaS companies are not in that position.
Five sites block every crawler
Separately, five companies serve a blanket Disallow rule that applies to all crawlers. That blocks Googlebot as much as GPTBot.
This usually points to a rule left behind after a migration, not a decision about AI. If your marketing site has one, AI visibility is not your first problem.
Who Explicitly Invites AI Answer Engines In?
53 of 442 companies name OAI-SearchBot and PerplexityBot in robots.txt and explicitly allow them. They include Apollo, Chargebee, Checkout.com, Clari, CyberArk, GoCardless, HiBob, LangChain and LastPass.
Looker, Melio, Netlify, Okta, PostHog, Retool, RingCentral, Supermetrics and Zapier are on the list too.
An explicit allow does nothing a default allow does not. What it tells you is that someone looked at the file and made a decision about AI. That is a good sign the company is paying attention.
Do Companies Publish an llms.txt and Still Block AI?
Yes. Thirteen companies publish an llms.txt file and also block an AI crawler. They include GitHub, Notion, Calendly, Loom, Postman, Amplitude, Razorpay and ZoomInfo.
Most of those blocks target scrapers like Bytespider, so the conflict is milder than it sounds. It is still worth an audit. I cover llms.txt adoption in my llms.txt study.
How Do You Check Your Own robots.txt in Two Minutes?
- Open yourdomain.com/robots.txt in a browser. It is a plain text file and anyone can read it.
- Search the page for these six names: GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Claude-SearchBot and Google-Extended.
- Look for a line that reads Disallow: / and check which User-agent group it sits under.
You will land in one of three places:
- No AI bot is named. That is where most B2B SaaS companies are. Nothing is blocked, though nobody has made a decision either.
- A training bot is named and disallowed. That is a reasonable choice, as long as someone made it on purpose.
- A search bot is disallowed, or Disallow: / sits under User-agent: \. Fix this week. The first keeps you out of AI answers. The second keeps you out of Google too.
Check your help center, docs site and any subdomains as well. Each has its own robots.txt, and those are where forgotten rules tend to live.
What Should You Do About It?
- Keep the search bots reachable. OAI-SearchBot, PerplexityBot and Claude-SearchBot are how you appear in live AI answers.
- Write one directive per bot. GPTBot and OAI-SearchBot are separate bots, and so are ClaudeBot and Claude-SearchBot. A rule for one does nothing to the other.
- Decide on training bots deliberately. Blocking GPTBot or ClaudeBot is legitimate. Canva’s setup shows what a considered version looks like.
- Remove any blanket Disallow rule from your marketing site. It usually arrives from a staging environment during a migration.
- Recheck every quarter. Rules get added during migrations, and new AI bots appear several times a year.
A setup that follows the Canva pattern looks like this:
User-agent: GPTBot
Disallow: /User-agent: OAI-SearchBot
Allow: /User-agent: PerplexityBot
Allow: /
For the full list of AI bots and what blocking each one costs you, see the AI Crawler Directory.
What This Study Does Not Tell You
- It measures stated intent, not bot behavior. robots.txt is a request. Some crawlers ignore it.
- It covers established companies. The sample skews toward well known B2B SaaS brands, so treat it as a view of market leaders.
- It is a snapshot. robots.txt files change, and these results reflect September 2026.
How I Ran This Study
I requested /robots.txt from 442 reachable B2B SaaS sites in September 2026 and checked 18 AI crawler user agents.
A site counts as blocking a bot only where robots.txt names that user agent and disallows it. Sites caught by a blanket rule are reported separately. Bots are grouped by each operator’s stated purpose.
How to Cite This Study
Pratik Dholakiya, (2026). AI Crawler Blocking in B2B SaaS: An Analysis of 442 robots.txt Files. Growfusely. https://growfusely.com/blog/ai-crawler-blocking-b2b-saas/
You are welcome to use the charts with a link back to this page. If you want the underlying data for your own analysis, get in touch.
Why Partner with Growfusely for AI Search Visibility?
Most AI visibility advice starts with content. I start with access, because a brand AI engines cannot read will not be recommended, however good its content is.
- A crawler and access audit covering robots.txt, rendering and every AI user agent that matters.
- A deliberate training versus citation policy, set as a business decision instead of inherited from a template.
- Citation measurement across ChatGPT and Perplexity, so access changes are judged on whether you appear in answers.
- Original research behind every recommendation, including this study and my analysis of 1,739 AI citations.
If you want a second pair of eyes on what your site allows and blocks, schedule a call.
FAQs
Should B2B SaaS Companies Block GPTBot?
Usually not. Blocking GPTBot stops OpenAI from training future models on your content, which narrows how well those models understand your brand. It is a legitimate choice, and five companies in my sample made it. Make it on purpose.
What is the Difference Between GPTBot and OAI-SearchBot?
GPTBot collects content to train OpenAI’s models. OAI-SearchBot indexes pages so ChatGPT can answer and cite questions. OpenAI says sites that block OAI-SearchBot will not appear in ChatGPT search results.
Does Blocking AI Crawlers hurt Google Rankings?
Blocking AI bots like GPTBot or Google-Extended does not affect Googlebot. A blanket Disallow rule does, and five sites in my sample have one.
Can Robots.txt Block ChatGPT-User or Perplexity-User?
Not reliably. Both fetch pages when a person asks, and both operators say robots.txt may not apply to them the way it applies to automated crawlers.
Why is TikTok’s Crawler the Most Blocked?
Bytespider has a reputation for aggressive crawling. Companies appear to treat it as a data collector, not as an assistant their buyers use. Common Crawl and Diffbot, second and third, fit the same pattern.
Read Next
- How AI search visibility works. Why training and retrieval are different paths into an answer.
Pratik Dholakiya is the Founder of Growfusely, a SaaS SEO and AI Search Visibility agency.
