Your robots.txt might be turning away your best customers
The file that tells crawlers what they may read was written for search engines. AI assistants are a different set of names, and most sites never added them.
robots.txt has done the same job since 1994: it sits at the root of your site and tells
automated visitors which parts they’re allowed to read. For twenty years the only visitors that
mattered were search engine crawlers, and most sites set it once and forgot it.
Then a new set of names showed up.
The names that matter now
These are the agents that read the web on behalf of AI assistants:
| Agent | Belongs to | Reads for |
|---|---|---|
GPTBot |
OpenAI | Training and ChatGPT browsing |
OAI-SearchBot |
OpenAI | ChatGPT search results |
ClaudeBot |
Anthropic | Claude’s web access |
PerplexityBot |
Perplexity | Perplexity answers |
Google-Extended |
Gemini and AI Overviews | |
Applebot-Extended |
Apple | Apple Intelligence |
If your robots.txt doesn’t mention them, what happens next depends entirely on how it’s written
— and the common patterns are not in your favour.
The three ways sites get this wrong
Blocking everything but Google. A surprising number of sites do this, usually copied from an old tutorial:
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /That reads as: Google may read everything, nobody else may read anything. Every AI assistant falls into that second rule. This is the worst case and it’s more common than you’d expect.
A platform default you never saw. Several hosted site builders and WordPress security plugins ship rules that block unrecognised agents. You didn’t write it, you may not know it’s there, and it’s doing exactly what it says.
No file at all. Missing robots.txt usually means “everything is allowed”, which sounds fine.
But it also means no Sitemap: line, so a crawler that arrives has no map of your site and has to
discover pages by following links — and it’ll miss the ones nothing links to.
What a good one looks like
For a small business that wants to be found, be explicit rather than relying on defaults:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://yoursite.com/sitemap.xmlNaming each agent explicitly costs nothing and removes any ambiguity about what the catch-all rule means for them.
“But I don’t want my content used for training”
That’s a legitimate position, and it’s a real trade-off rather than a technicality.
Worth separating, though: some of these agents fetch pages at the moment a user asks a question, to answer it. Blocking those doesn’t protect your content from training — it just means the assistant can’t cite you when a customer is actively looking for what you sell.
If that distinction matters to you, allow the retrieval agents (OAI-SearchBot, PerplexityBot)
and make your own call on the training ones. Just make it a decision rather than an accident.
Check yours in ten seconds
Type your domain followed by /robots.txt into a browser. If you see Disallow: / under a
User-agent: * rule, that’s your answer.
Or run the free scan — it checks this along with seventeen other things, and tells you which of them are actually costing you.