Should I block AI crawlers?
The answer depends on one thing: whether your revenue comes from the visit. For a publisher whose business is advertising on the page, blocking can be the right call. For a service or product company it means disappearing from the answers that could have recommended it.
The question is usually asked as a moral one; it is an economic one. What differs is not your view of AI but where, in your case, the money is made.
When blocking makes sense
When the content is the product: a news site, a recipe site, course material. A model that answers instead of you removes exactly the visit you are paid for.
In that case the training bots (GPTBot, CCBot, Google-Extended) get
blocked while the search ones are often left in, because they carry a link.
When blocking hurts
When the content is a route to the product: services, SaaS, a shop, a local business. Here a mention in an answer is a free recommendation, and blocking means you are simply not among the options.
One practical detail many miss: with ChatGPT-User blocked, a link you send a
client will not open in their conversation. That is no longer visibility,
it is customer service.
Three different bots
GPTBot — training. OAI-SearchBot — search with links. ChatGPT-User — a
page opened at one person's request.
A single Disallow: / for all three is three separate decisions made by
accident.
What to do instead of blocking
If the worry is about content use, there is a softer path: the
Content-Signal line in robots.txt lets you say separately that content may
be cited but not trained on.
It is a declaration, not a gate — a dishonest collector ignores it. But honest ones respect it, and blocking everything "just in case" costs more.
A free scan shows which bots actually get in — live, rather than by
reading what robots.txt says.
Frequently asked
Does blocking protect content?
Partly. Honest crawlers obey, dishonest ones do not. `robots.txt` is a request, not a lock.
Does blocking Google-Extended affect search?
No. It is a separate control for Gemini training; ordinary Google search and indexing are untouched.
What is Content-Signal?
A `robots.txt` line introduced in 2025 that states separately what is allowed: `search`, `ai-input` (real-time answers) and `ai-train` (training). Each can be set to `yes` or `no`.
Can a CDN block bots without my knowledge?
Yes, and it is common. Cloudflare has a separate AI crawler setting that can be on without your decision — then `robots.txt` says "welcome" while the server answers 403.
Check yours
A free scan shows which of the things described here your site already does, and which it does not.