Do websites actually block AI crawlers?
Most of the time AI bots are neither blocked nor admitted deliberately — robots.txt simply does not name them. What happens then depends on the general rules: sometimes everything passes, sometimes everything is blocked along with the rest. Either way it is a decision nobody actually made.
Ask whether a site blocks AI bots and the answer is almost always "no". Open the file and the honest answer is usually "we don't know" — the bots are not mentioned at all.
The numbers draw on roughly 500 audits run on our earlier platform and on what we see today. The exports from that work are gone, so these are ranges rather than precise statistics.
What the file usually contains
Three typical shapes:
No bot named at all. A User-agent: * block and a few Disallow lines from
the template. AI bots then follow the general rule — that is, they pass, unless
something stops them.
A CMS default. WordPress or Shopify defaults that close admin paths and nothing else. The AI question does not exist in them.
A broad block with no reason. Less common, but it happens: someone added a strict rule once and nobody remembers why. Useful bots are usually blocked in there alongside the unwanted traffic.
Why "not blocking" is not "admitting"
robots.txt works by name. A block applies only to the agent you name, and so
does permission. There are more names today than most people expect, and they
serve different purposes:
- GPTBot — OpenAI's training crawler;
- OAI-SearchBot — OpenAI's search bot, the one serving answers with links;
- ChatGPT-User — a fetch triggered by a user inside a conversation;
- PerplexityBot, ClaudeBot — the equivalents from other vendors;
- Google-Extended — a separate Google signal that does not change ordinary indexing.
The practical consequence: you can refuse training and still allow search. Those are two different names, and that is the only way to say "do not quote me without a link, but do show me with one".
The most common mistake
Blocking the search bot while thinking about training. The site then disappears from answers that carry links — the part that actually brings visitors — and the training question is still not fully settled.
The second common mistake is a robots.txt that blocks what you want shown:
pricing, FAQs, the guide. Those pages are exactly the citable material.
A five-minute check
Open yourdomain.com/robots.txt and look for the names above. If none of them
appear, the decision has not been made yet — and that is the most common case.
Then decide deliberately. Whether blocking is worth it depends on whether the content is sold separately: should you block AI crawlers.
A free scan does the same check for you, and also tells you whether a bot
that gets past robots.txt receives any text at all.
Related articles
- Should I block AI crawlers: when it is worth it
- GEO in 2026: does AI find your site
- The state of AI search: where we stand
- sitemap.xml and robots.txt: what they do
How this enters the score is described under what we check.
Sources
What this text is based on — primary sources, not retellings:
Frequently asked
Is a `robots.txt` block binding on a bot?
Not technically — it is a convention, not protection. Major vendors publish their bot names and state that they honour the rules; whoever ignores them also hides the name. Real protection lives at the server level, not in this file.
If I block GPTBot, do I disappear from ChatGPT?
Not immediately and not entirely. GPTBot collects training data, while answers with links are served by OAI-SearchBot. To stay in the answers, that one has to be allowed in.
Will blocking `Google-Extended` hurt normal indexing?
No. It is a separate signal aimed at generative products; ordinary Google Search is driven by `Googlebot`, which it does not touch.
What if I do not know who wrote my `robots.txt`?
Read the whole thing — it is usually short — and keep only the lines you can explain. Inherited rules nobody understands do more harm than good.
Check yours
A free scan shows which of the things described here your site already does, and which it does not.