Your robots.txt asks the crawlers that answer questions to stay away

What we observed

Your robots.txt disallows one or more of the crawlers whose job is to fetch pages so an AI assistant can cite them: OAI-SearchBot, Claude-SearchBot, PerplexityBot, bingbot.

Why it matters

There are two kinds of AI crawler and they are easy to confuse, because vendors give them similar names.

Training crawlers — GPTBot, ClaudeBot, CCBot, Amazonbot — collect pages to train AI models. Blocking them is a defensible choice about how your work is used, and it does not affect whether you appear in answers.

Answering crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — fetch a page because someone asked a question it might answer. Blocking these is how a site vanishes from AI answers entirely.

Most sites that block an answering crawler did not mean to. The names sit next to each other in blocklists copied from blog posts, and the difference between them is rarely explained.

How to fix it

Open your robots.txt and look for the answering crawlers by name. A block looks like:

User-agent: OAI-SearchBot
Disallow: /

To allow it again, remove that whole block (the User-agent line and the rule under it), or replace the rule with an empty Disallow:, which means “nothing is disallowed”:

User-agent: OAI-SearchBot
Disallow:

Note that those two differ by a single character. Disallow: / blocks everything. Disallow: blocks nothing. They are opposites.

If you want training blocked but answering allowed — an increasingly common choice — name each crawler separately:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow:

Where to edit depends on your site. If your SEO plugin manages robots.txt, edit it there. If a real robots.txt file exists on your server, that file is the only thing that counts.

How to check it worked

curl -s https://example.com/robots.txt

Look for the answering crawlers and confirm each one is either absent or followed by an empty Disallow:. Remember that robots.txt is a request, not a lock — this changes what you are asking crawlers to do, not what your server lets them do.