The crawler self-test could not tell you what the internet sees
The check ran, but from a place where its answer would not mean anything. Here is why, and what to do instead.
What we observed
The self-test works by making real HTTP requests to your own site, once as a browser and once as each AI crawler, and comparing what comes back. That only tells you something if the request takes the same path a crawler on the internet would take.
On this site it did not, and we would rather say so than hand you a green tick that means nothing. One of these applied:
- Your site has no public address. Your domain name points to
127.0.0.1, a private address such as192.168.x.x, or ends in a development suffix like.localor.test. A request to it never leaves the machine. This is normal and expected on LocalWP, MAMP, Docker, or a staging server behind a VPN. - The request seemed to stay inside your server. Your domain is public, but the response showed no sign of having passed through your CDN or firewall. Servers often answer requests to their own domain directly, without going out to the internet, so the request never reached the outer layer (such as Cloudflare) that would have blocked or allowed a real crawler.
- We could not confirm the request passed through your CDN or firewall. None of the headers
those services add (such as
cf-ray,x-served-byorx-cache) came back, so we cannot say whether Cloudflare, Sucuri, a WAF or a host-level proxy saw the request.
Why it matters
Most tools in this category will happily report “GPTBot: allowed” from a request that never left the server. That result isn’t exactly wrong, but it’s empty — it describes a conversation your server had with itself.
The parts of this plugin that do not depend on outbound requests are still valid. Your
robots.txt analysis, the attribution of who wrote it, the .htaccess scan and the server-side
rendering check all read your actual configuration and are unaffected by this.
How to fix it
If this is a local or staging site, nothing is broken. Run the check again once the site is live on a public domain. The self-test is the only part of the plugin that needs the public internet, so a missing result here is expected.
If this is a live site, it’s worth finding out why the request stayed inside your server. It means your server sends requests for your own domain straight back to itself instead of out through your CDN:
- Check for a
/etc/hostsentry or a DNS override that points your domain to127.0.0.1or the server’s own IP address. Many control panels add one. - If you are behind Cloudflare or a similar proxy, requests your server sends to its own domain usually don’t go out through Cloudflare. This is by design and not something you need to change.
Either way, you can verify from outside with a request that genuinely comes from the internet:
curl -s -A "GPTBot/1.4" -o /dev/null -w "%{http_code}\n" https://example.com/
curl -s -A "Mozilla/5.0" -o /dev/null -w "%{http_code}\n" https://example.com/Run those from a laptop on a different network, or from any online HTTP request tool. Two different status codes means something is treating the crawler differently. The same code both times means it is not — at least for that page.
How to check it worked
Re-run the check. When the self-test can produce a trustworthy result, this finding disappears and a result for each crawler appears in its place.