Your server treats all crawlers differently from browsers
It is not AI-specific — Googlebot is affected too, which points at a different cause and a different fix.
What we observed
Crawlers received a different response from your server than a browser did — and Googlebot was affected as well. That is what separates this from an AI-specific block, and it changes what you should go looking for.
Why it matters
If Googlebot is being treated differently from a browser, your problem is considerably bigger than AI visibility. The same mechanism is standing between your site and Google Search.
That said, don’t panic yet: Google may still be indexing you fine. Some protections check a visitor’s IP address to confirm it really is Googlebot, and refuse anything that only claims to be Googlebot. Our check can only claim, so it gets refused. That behaviour is correct, and it produces exactly this finding.
How to fix it
First, find out whether Google is actually affected. Open Google Search Console and use URL Inspection → Test Live URL on a page. If Google fetches it successfully, your protection is verifying Googlebot properly and the real issue is narrower: everything that is not Googlebot is being refused, including every AI crawler.
Then look at what is doing the refusing:
- Bot protection at a CDN. Cloudflare’s Bot Fight Mode and similar features challenge or block anything that looks automated. AI crawlers can’t pass JavaScript challenges (the “checking your browser” pages), so for them a challenge is the same as a block.
- A security plugin’s “block bad bots” setting, which often uses a broad pattern rather than a list of names.
- Host-level rate limiting, where your host limits how many requests it accepts. It shows up as
429when the site is busy, so it comes and goes.
The fix is usually to narrow the rule rather than remove it. Allow the crawlers you want by
name — OAI-SearchBot, Claude-SearchBot, PerplexityBot, bingbot, Googlebot — while
keeping protection against everything else.
How to check it worked
Re-run the check. You want Googlebot and the AI crawlers all to get the same response as the normal browser we test for comparison. If the AI crawlers match but Googlebot still doesn’t, your protection is checking Googlebot’s IP address, which is fine and expected.