robots.txt for AI crawlers
Explicit rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, plus a Sitemap reference.
Why it matters: A catch-all Disallow turns every AI assistant away. Many sites have one and nobody knows.
What we fix
Every item, and what it's actually for
No black box. This is the complete list of what we change, what each one does, and why it costs you if it's missing. If you'd rather hand it to your own developer, take it —the checklist version is here.
Essential
These cost the most points and block the most. Several are ten-minute fixes.
Valid, complete sitemap
A sitemap.xml listing every page worth reading, referenced from robots.txt.
Why it matters: Without it a crawler only finds what your navigation links to, and misses the rest.
Real 404s
Missing pages return HTTP 404 instead of 200 with an empty body.
Why it matters: Soft 404s get indexed as real content, then quoted back to a customer as if they were.
One canonical host
www and non-www resolve to a single host via 301 redirect.
Why it matters: Two copies of every page means neither accumulates authority and agents see duplicates.
HTTPS end to end
Valid certificate, http redirects to https, no mixed content.
Why it matters: Agents deprioritise or refuse sites they cannot fetch securely.
Facts as readable text
Hours, phone, address, service area and services present as text, not only in images or widgets.
Why it matters: A crawler reads none of your graphics. If your hours are a JPEG, your hours do not exist.
Recommended
Where most sites lose their remaining ground. The structured data item is the highest-leverage thing on this page.
LocalBusiness structured data
JSON-LD stating name, address, telephone, opening hours, area served and price range.
Why it matters: Turns facts an assistant would have to infer into facts it can state with confidence.
llms.txt summary
A plain-text description of the business at /llms.txt.
Why it matters: Removes the guesswork of parsing a marketing homepage for what you actually do.
Heading structure
Exactly one h1 per page, describing that page rather than your logo.
Why it matters: Multiple or empty h1s leave a crawler with no clear topic for the page.
Titles and descriptions
Factual, specific page titles and meta descriptions instead of slogans.
Why it matters: "Home" tells a machine nothing. "Roof replacement in Raleigh, NC" tells it everything.
Content feed
RSS, Atom or JSON Feed declared in the head.
Why it matters: Lets agents see what changed without re-crawling your whole site.
Clean text negotiation
Serving readable text when an agent requests text/markdown or plain text.
Why it matters: Agents that ask for text and get raw HTML often just move on.
Link headers
Link: response headers pointing at your feed, sitemap and canonical URL.
Why it matters: Discovery without having to parse the page first.
Meaningful alt text and link text
Alt text that describes the image; link text that names the destination.
Why it matters: "Click here" is invisible to a machine. So is "image1.jpg".
Which of these are you failing?
The free scan checks all of them and tells you in about twenty seconds.