What your site serves is not what WordPress generates

What we observed

We asked WordPress what robots.txt it would generate, then fetched the one your site actually serves, the way a crawler would. They do not match. Something sits between the two and is altering or replacing it.

Why it matters

Your robots settings no longer match what your site actually tells crawlers. You can edit them, save them, see them — and the internet still receives something else. So anything you conclude from your settings screen may be wrong, including whether you are blocking an AI crawler.

How to fix it

Start by seeing both versions. Your plugin’s robots screen shows what WordPress intends. This will show what is served:

curl -s https://example.com/robots.txt

The difference usually points at the cause.

A page cache. Caching plugins and CDNs cache /robots.txt like any other URL, so an old copy can keep being served after a change. Clear all caches, including at your CDN, and check again. If they now match, that was it.

A CDN rule. Cloudflare can change robots.txt or serve its own version, and its managed robots.txt features add lines to it. Check Rules and any Workers routes that match /robots.txt.

A file that appeared later. If a real robots.txt file was added after WordPress started generating one, the file wins from that moment. Look in the same folder as wp-config.php.

Your host. Some managed hosts add their own robots rules on their servers, before requests reach WordPress. If nothing above explains it, ask them directly.

How to check it worked

Compare the two again after clearing caches. They should match, apart from spacing and blank lines. If they still differ and you can’t find the cause, the served version is the one that counts. Treat it as the real one, and edit it wherever it is actually coming from.