robots.txt for AI Crawlers: A Practical Guide
What robots.txt can and cannot do for AI crawlers, worked policies you can copy, and how WordPress serves the file virtually.
robots.txt is a thirty-year-old convention that was standardised as RFC 9309 in 2022, and it is now doing a job it was never designed for: expressing a site's position on AI training, AI search, and user-triggered retrieval. It does that job partially. This guide covers the mechanics the RFC actually specifies, the vendor statements about which agents obey it, four worked policies you can copy, and the WordPress-specific behaviour that catches people out.
What the standard actually requires
RFC 9309 is short and worth reading directly. The parts that matter operationally:
Location and format. The file lives at /robots.txt at the top-level path of the authority, in UTF-8, served as text/plain. It is per scheme, host and port — https://example.com/robots.txt does not govern https://shop.example.com/. ESTABLISHED STANDARD.
Group matching. User-agent matching is case-insensitive. If more than one group matches a crawler, the matching groups' rules must be combined into one group. If no group names the crawler specifically, it must obey the * group if one is present. That last point is the source of most real-world confusion: a crawler that has its own named group ignores the * group entirely. If you write a specific User-agent: GPTBot block, whatever you put in * no longer applies to GPTBot.
Rule precedence. The most specific match wins, where "most specific" means the most octets in the path pattern. Where an allow and a disallow rule are equivalent, the allow should be used. So Disallow: /docs/ plus Allow: /docs/public/ behaves as you would hope.
Percent-encoding. Percent-encoded ASCII octets must be decoded before comparison unless the character is reserved. Non-ASCII paths need encoding to compare reliably.
Size limit. Parsers must handle at least 500 KiB. Anything past that may be discarded, which matters if a plugin has been appending rules unattended for years.
Caching. Crawlers should not use a cached copy for more than 24 hours, unless the file is unreachable. Do not expect a change to take effect instantly.
Status codes. This is the part that silently breaks sites:
| Response | Required crawler behaviour |
|---|---|
| 2xx | Parse and follow the rules |
| 3xx | Follow up to five consecutive redirects, then assume unavailable |
| 4xx | The file is unavailable; the crawler may access any resource on the server |
| 5xx | The file is unreachable; the crawler must assume complete disallow |
A 500 on /robots.txt therefore means "crawl nothing". A misconfigured maintenance mode, a database error, or an origin hiccup that returns 5xx on that one path can take a site out of crawling entirely, and nothing in your analytics will tell you. This is the single check most worth automating, and it is the first thing the AI crawler policy inspector looks at.
What robots.txt cannot do
Four limits, each with a first-party source. Getting these wrong produces policies that do the opposite of what the owner intended.
It is voluntary. Google's own robots.txt introduction states that instructions in robots.txt files cannot enforce crawler behaviour — well-behaved crawlers follow the rules, and not all crawlers do. If content must not be read, the control is authentication, not a text file. OFFICIAL PROVIDER GUIDANCE.
Blocking is not deindexing. Google states robots.txt "is not a mechanism for keeping a web page out of Google". A URL disallowed in robots.txt can still appear in results if other sites link to it; Google simply cannot see the content, so the result has no description. The correct tool for keeping a page out of an index is noindex (which requires the page to be crawlable so the directive can be read) or password protection. OFFICIAL PROVIDER GUIDANCE. This is the most common self-inflicted wound in crawler policy: people disallow a page and add noindex, so the crawler never reads the noindex.
It does not reliably govern user-triggered fetches. OpenAI's bot documentation states that ChatGPT-User is used for user-initiated actions, and that because these actions are initiated by a user, robots.txt rules may not apply. Perplexity is blunter still: its documentation says Perplexity-User generally ignores robots.txt rules, because a user triggered the fetch. Anthropic documents Claude-User as the agent that retrieves content when a user asks for it, alongside ClaudeBot for training and Claude-SearchBot for search indexing. The distinction is real and it is the vendors' own framing: a bulk crawl is one thing, a person pasting your URL into a chat window is another. OFFICIAL PROVIDER GUIDANCE.
It does not distinguish purposes on its own. robots.txt has no vocabulary for "you may read this but not train on it". The only reason training and search can be separated at all is that vendors have chosen to run separate named agents — GPTBot versus OAI-SearchBot, ClaudeBot versus Claude-SearchBot, PerplexityBot (which Perplexity states is not used to crawl content for foundation models) versus Perplexity-User, Google-Extended as a control token distinct from Googlebot. That separation is a vendor courtesy, not a property of the protocol, and it can change. OFFICIAL PROVIDER GUIDANCE for the current agent list; ESTABLISHED STANDARD for the protocol's silence on purpose.
Worked policies
Each policy below is complete and copy-pasteable. Verify agent names against the vendors' current documentation before deploying — this list reflects what those pages said on the date in this article's frontmatter, and vendors add agents without notice.
Policy A — Allow AI search, block AI training
The most common position for a business that wants to be findable in ChatGPT and Claude but does not want its archive used as training data.
# AI search crawlers: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training crawlers: blocked
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everything else
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlTwo things to notice. The explicit Allow: / groups are not strictly necessary — an agent with no rules is unrestricted — but they are self-documenting, and more importantly they prevent those agents from falling back to a restrictive * group you might add later. And Google-Extended sits in the training list because Google documents it as governing Gemini training and Gemini/Vertex grounding, while stating it does not affect Search inclusion or ranking.
Policy B — Block AI comprehensively, keep conventional search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlBe honest with yourself about what this buys. The user-triggered agents are included for completeness, but OpenAI's documentation says robots.txt rules may not apply to them. Blocking OAI-SearchBot means, per OpenAI, that your site will not be shown in ChatGPT search answers — that is a deliberate visibility trade, not a neutral safety measure. And a blocked page can still be described from third-party sources; you have removed your own text from the answer, not removed yourself from the conversation.
Policy C — Allow everything (with hygiene)
The default position, and a defensible one. Nothing is blocked except the admin area.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/wp-sitemap.xmlThe two search-results disallows are ordinary crawl-budget hygiene: internal search result pages are near-infinite and add nothing to an index.
Policy D — Selective, per-directory
Block training crawlers from a members' area and a document archive while leaving marketing content open.
User-agent: GPTBot
Disallow: /members/
Disallow: /research/reports/
Allow: /research/summaries/
User-agent: ClaudeBot
Disallow: /members/
Disallow: /research/reports/
Allow: /research/summaries/
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlAllow: /research/summaries/ is longer than Disallow: /research/reports/ only for URLs under summaries/, so RFC 9309's most-octets rule resolves it correctly. Keep an eye on that when nesting paths — write the rules and then test a few concrete URLs against them rather than trusting the shape.
WordPress specifics
WordPress serves robots.txt virtually by default. There is no file on disk. do_robots() in wp-includes/functions.php builds the output at request time, and the base content is only:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpCore's sitemap subsystem hooks the robots_txt filter to append the Sitemap: line, and only when the site is public. Two consequences follow.
A physical file wins. If robots.txt exists in the web root, the web server serves it and WordPress never runs. Migrations and staging clones love to leave one behind, sometimes containing Disallow: /. Check the filesystem, not just the URL.
The virtual route depends on rewrite handling. The output is triggered from template-loader.php via the do_robots action. On nginx configurations that do not route /robots.txt through index.php, the request can 404. A 4xx is "unavailable, crawl anything" under RFC 9309, so this fails open rather than closed — but it also means your carefully written policy is not being served at all, and you would never know from a browser tab that shows a plausible-looking 404 page. Fetching your own /robots.txt and checking the status code, not just the body, is a real check. It is one of the things SiteRank flags as critical.
Search engine visibility settings do not write Disallow: /. Since WordPress 5.3, discouraging search engines no longer emits a blanket disallow; it uses the robots meta tag instead. If you see Disallow: / on a WordPress site, something else put it there.
Do not edit robots.txt through a plugin and a file at the same time. If Yoast, Rank Math or AIOSEO is managing the robots_txt filter, a second manager produces duplicated or conflicting groups. SiteRank detects the existing owner and proposes a policy for you to approve rather than writing over it.
Verifying, not assuming
Three checks that catch nearly everything:
- Status code.
curl -I https://example.com/robots.txt— anything other than 200 is a finding, and 5xx is an emergency. - Named-group shadowing. For each agent you care about, confirm whether it has its own group. If it does, the
*group is irrelevant to it. - Agent-name accuracy. A typo is silent.
GPT-Bot,Claudebot-1.0andPerplexity Botall match nothing. Compare against the vendor pages, and re-check periodically because the rosters change.
User-agent verification is a separate problem: a header is a claim, not an identity. Vendors publish IP ranges — Anthropic at claude.com/crawling/bots.json, for instance — and reverse-DNS or IP verification is the only way to know whether a request calling itself GPTBot really was. Treat unverified agent strings in your access logs as untrusted input.
Key takeaways
- RFC 9309 is the authority on mechanics: named groups override
*, most-octets wins, 4xx means crawl freely, 5xx means crawl nothing. - robots.txt is voluntary, is not a deindexing tool, and per OpenAI's own documentation may not govern user-triggered fetches.
- Training and search can be separated only because vendors run separate named agents. That is a vendor choice, not a protocol feature.
- Blocking
OAI-SearchBotcosts you ChatGPT search visibility, by OpenAI's own statement. Decide that deliberately. - WordPress's robots.txt is virtual, minimal, and easily broken by a stray file or an nginx rule that 404s the path. Check the status code.
Official sources & further reading
- RFC 9309: Robots Exclusion Protocol — IETF
- Introduction to robots.txt — Google Search Central
- Google crawlers and fetchers (user agents) — Google Search Central
- Overview of OpenAI crawlers — OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
- PerplexityBot and Perplexity-User crawler documentation — Perplexity
Related reading
Frequently asked questions
I disallowed a page and it still appears in Google. What went wrong?
Disallowing is not deindexing, and Google says so directly: robots.txt is not a mechanism for keeping a page out of the index. A disallowed URL can still be listed when other sites link to it, simply without a description, because the crawler was never permitted to read the page itself. If you also added a noindex tag, the disallow is what prevents the crawler from ever seeing it. To remove a page from an index, leave it crawlable and use noindex, or put it behind a password.
How long does a change to robots.txt take to have an effect?
Not instantly, and not on a schedule you control. RFC 9309 says crawlers should not use a cached copy for more than 24 hours, which sets an expectation rather than a guarantee, and operators cache for their own reasons on top of that. Judge a change over days rather than over the same afternoon's logs. Removing a restriction obliges nobody to come back promptly once they notice.
Do I need a robots.txt at all?
Not strictly. A 4xx response means the file is unavailable and a crawler may access any resource, which is much the same practical outcome as an empty allow-everything file — and on WordPress you have one regardless, because core serves it virtually. What you lose without one is the Sitemap: line and any ability to state a per-agent position. The argument for having a file is that it is where crawler policy is expressed; the argument against a complicated one is that every rule is something that can be wrong.
Could my CDN or a caching layer be serving a different robots.txt from the one WordPress builds?
Yes, and it is a common surprise. A physical file in the web root is served by the web server and WordPress never runs; an edge cache can keep serving an old copy after you change the source; and an nginx configuration that does not route /robots.txt through index.php can return a 404 for a policy you believe is live. Fetch the URL from outside your own network and check the status code as well as the body, rather than trusting the screen that composed it.
Does an Allow rule make a crawler fetch my pages?
No. Allow removes a restriction; it does not schedule a visit, and it says nothing about what happens to a page after it is fetched. Crawling, retrieval and citation are separate steps, and a robots.txt rule governs only the first. It is a statement about permission, not a request for attention.