We cannot tell you whether your security plugin turns real crawlers away

What we observed

We saw two things that do not fit together:

1. Your security plugin has rules that name AI crawlers, or block them as part of a category.
2. Every AI crawler we imitated — including the ones those rules name — was served a normal page.

The likely explanation is not that the rules are broken. It is that our test cannot trigger them. The self-test sends its requests from your own web server. Security plugins almost always trust that server’s own IP address, so they don’t block the requests WordPress sends to itself (for scheduled tasks and the REST API) and break the site.

So a crawler arriving from the open internet may be treated completely differently from the way our request was treated. We know the rules exist. We can’t see whether they actually block anything.

Why it matters

This is a real blind spot and we would rather name it than paper over it. Every tool that tests your site from inside your site has this limitation, whether or not it says so.

It matters most because the failure it hides is silent. A security plugin blocking GPTBot and ClaudeBot as “bad bots” produces no error, no notice, and no entry anywhere you would think to look. The site simply stops appearing in AI answers, months later, for no visible reason.

How to fix it

There is nothing here to fix until you know whether the rules actually block anything. Test from outside your server:

curl -s -A "GPTBot/1.4" -o /dev/null -w "%{http_code}\n" https://example.com/
curl -s -A "OAI-SearchBot/1.4" -o /dev/null -w "%{http_code}\n" https://example.com/
curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" -o /dev/null -w "%{http_code}\n" https://example.com/

Run these from a laptop on a normal internet connection — not from the server, and not from a network your security plugin is set to trust. If the crawlers get a 403, 429, or a challenge page (a “prove you are human” screen) while the browser gets a 200, the rules are blocking real traffic.

If they are, decide deliberately which crawlers you want blocked:

• Training crawlers (GPTBot, ClaudeBot, CCBot, Amazonbot) — blocking these is a legitimate choice about how your work is used, and does not affect whether you appear in AI answers.
• Answering crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, bingbot) — blocking these is how a site disappears from AI answers. This is almost never intentional.

Then add the answering crawlers to your security plugin’s allowlist, the list of visitors it always lets through (in Wordfence, Firewall → Blocking and the allowlisted-services list; other plugins have an equivalent). Most of these crawlers publish official IP address ranges. Allowing those addresses is safer than trusting the User-Agent header, because anyone can send any name in that header.

How to check it worked

Re-run the curl commands above from outside. A 200 for the answering crawlers means they are getting through. This finding will stay in the plugin either way. It describes a limit on what we can see from your server, not a problem we can watch you fix.