Your server turns AI crawlers away

What we observed

We sent the same request to your site several times, changing only the User-Agent header — the name each visitor gives for itself. A browser received the page. One or more AI crawlers received something different: a different status code (the server’s short reply, such as 200 for OK or 403 for refused), or a page of a very different size. Googlebot was served normally. That’s what tells us the block is aimed at AI crawlers, not at crawlers in general.

Nothing about your content is involved. The request never got far enough for content to matter.

Why it matters

A crawler that cannot fetch your pages cannot cite them. This is the most complete form of invisibility there is: not ranking badly, not being outranked — simply absent, because the door was shut before anyone looked inside.

It is also the form site owners are least likely to know about, because nothing in WordPress reports it and nothing in your analytics shows a request that was refused.

How to fix it

Work down this list in order. The first one that matches is usually the answer.

1. A security plugin. Wordfence, Solid Security, All In One WP Security, Sucuri and others all come with blocklists, and several include AI crawlers in a “bad bot” preset. Look for a blocking or firewall section and search it for GPTBot, ClaudeBot or AI. If the plugin’s own log shows blocked visits from those names, you have found it.

2. Your host. Managed WordPress hosts sometimes slow down (rate-limit) or refuse AI crawlers on their own servers, before the request ever reaches WordPress, so no plugin can see or change it. The telltale sign is a 429 or 403 response that shows up for AI crawlers and that nothing in your own settings explains. Ask your host directly: “Do you rate-limit or block AI crawler user agents such as GPTBot and ClaudeBot at the platform level? Can it be disabled for my account?” Most will turn it off if you ask.

3. A CDN or WAF. These are services that sit in front of your site and filter traffic before it reaches you — Cloudflare is the most common. If you use Cloudflare, check Security → Bots and any custom WAF rules. Cloudflare’s managed bot rules and its AI crawler settings both affect these crawlers, and they are set in the Cloudflare dashboard, not in WordPress.

4. Server configuration. On Apache, look in .htaccess for BrowserMatchNoCase, SetEnvIf User-Agent or RewriteCond %{HTTP_USER_AGENT} lines naming a crawler. On nginx, these rules live in the server’s configuration files, and usually only your host can change them.

How to check it worked

From a terminal, send the same request twice, once as a browser and once as GPTBot:

SITE="https://example.com"
curl -s -o /dev/null -w "browser:  %{http_code}  %{size_download} bytes\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36" "$SITE"
curl -s -o /dev/null -w "GPTBot:   %{http_code}  %{size_download} bytes\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" "$SITE"

Both lines should show the same status and roughly the same byte count. Check the byte count, not just the status. Some firewalls answer 200 (OK) but send a short challenge page instead of your page. That looks like success, but it isn’t.

One thing to be careful about

Do not fix this by allowing everything. Blocking training crawlers while allowing the ones that fetch pages to answer questions is a legitimate and increasingly common choice — they are different crawlers with different names. If the block was deliberate, the fix is to narrow it, not remove it.