"Should I block AI bots?" is the wrong shape of question, because there is no single AI bot. Every major provider now runs several agents with different jobs, documents them separately, and states different consequences for disallowing each one. SiteRank AI reads your robots.txt, resolves the effective rule for each documented agent, and puts the provider's own stated consequence next to it.
Four jobs, not one switch
The registry sorts agents by what the fetch is for, because that is the axis every provider's own documentation uses.
Search and retrieval crawlers build the index a provider's answer product queries at question time. Googlebot and bingbot are the conventional case; OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot and Perplexity's PerplexityBot are the AI-answer equivalents. Disallowing one removes you from that product's answers. OpenAI states directly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Perplexity's documentation recommends allowing PerplexityBot for the same reason and states it is not used for model training.
Training crawlers collect content that may be used to train foundation models. GPTBot, ClaudeBot and Google-Extended are the documented examples. Disallowing one is a licensing decision about future models, not a visibility decision about today's answers.
User-triggered fetchers retrieve a specific URL because a person asked for it in the product. ChatGPT-User, Claude-User and Perplexity-User are in this group, and this is where the rules diverge. OpenAI states that because ChatGPT-User actions are user-initiated, robots.txt rules may not apply. Perplexity states that Perplexity-User generally ignores robots.txt, since the request originates from an actual user. Anthropic documents Claude-User as one of three bots that respect standard robots.txt exclusions. Those are three different positions, and a single blanket rule for "AI bots" erases all three.
Ad and product validators are narrower. OpenAI's OAI-AdsBot visits only pages submitted as ads, and OpenAI states the data it collects is not used to train foundation models.
Every provider that documents multiple agents also documents that the settings are independent. OpenAI's example is explicit: allow OAI-SearchBot to appear in search results while disallowing GPTBot. Evidence tier for every claim in this section: OFFICIAL PROVIDER GUIDANCE, verified against the linked pages on 9 September 2026.
What the inspector reports
The inspector fetches your robots.txt through the plugin's own guarded HTTP layer and parses it against RFC 9309 precedence rules — longest matching path wins, most specific user-agent group applies, and a group matched by name overrides the wildcard group.
For each agent in the registry it reports: the effective rule your file produces, which group produced it, the agent's documented purpose, its category, and the consequence the provider publishes for disallowing it. It reports the Sitemap: directives the file declares and whether your own rules would let a crawler fetch each one — a sitemap advertised in robots.txt and disallowed by the same file is a common self-inflicted error.
It also reports the file's HTTP status, which catches a failure mode WordPress sites hit regularly: WordPress serves a virtual robots.txt when no physical file exists, and a server rewrite can leave that virtual response returning 404. A robots.txt that answers 404 is not a permissive file — under RFC 9309 it means no rules were retrieved, and crawler behaviour in that state is operator-specific. The inspector flags it as critical.
What it will not do is score you. Allowing a training crawler is a defensible business choice and so is blocking one, so a missing Allow is never reported as a defect and the product never pre-selects "allow all" or "block all".
Proposals are diffs, never writes
Where you do want to change policy, SiteRank AI generates a proposed robots.txt grouped by intent — the AI-search agents together, the training agents together — and shows it as a diff against your current file.
Nothing is written until you act. The proposal is stored, reviewable, editable, and reversible. Your existing rules and Sitemap: lines are carried through rather than replaced, because dropping them is the most common way a robots.txt edit does real damage.
If another plugin already owns your robots.txt output — Yoast, Rank Math, AIOSEO and others all can — the inspector says so and reports where the change belongs instead of fighting for the filter. See WordPress SEO for how ownership detection works.
Verify bots by IP, not by user-agent
A user-agent string is a header, and anyone can send one. A hit in your access log is not evidence that a provider visited you, and AI crawler impersonation is common enough that operators warn about it in their own documentation.
Verification is by published IP list. Anthropic publishes a combined list at claude.com/crawling/bots.json. Perplexity publishes perplexitybot.json and perplexity-user.json and recommends combining user-agent matching with IP verification when writing firewall rules. Google and Bing support reverse DNS verification.
One warning worth repeating: do not use IP blocking as an opt-out mechanism. Anthropic states that blanket IP blocking will not reliably prevent crawling, because it can interfere with the crawler reading your robots.txt in the first place. Blocking the address you use to communicate your preference is self-defeating.
Two mechanical points the inspector surfaces because they catch people out: rules are per host, so example.com/robots.txt does not govern docs.example.com and Anthropic explicitly asks you to add the rule to every subdomain you want to opt out from; and robots.txt is advisory. RFC 9309 describes a voluntary mechanism. If you need enforcement, that is authentication, a paywall, or an edge block — not a text file.
Crawled is not retrieved is not cited is not recommended
This is the distinction the whole feature is built around, so it is worth stating without hedging.
Allowing every AI crawler produces exactly one thing: eligibility. Access is a precondition for retrieval and nothing more. Retrieval is a precondition for citation and nothing more. Citation is not endorsement, and none of the four steps is documented end to end by any provider.
So crawler policy is not an optimisation, cannot be a score component, and is not presented as one here. Google's own guidance on its AI features is that there are no additional requirements and no special optimisations needed beyond normal SEO. If you want to know whether any of this reaches an actual answer, that is a separate measurement, and it lives in citation monitoring.
Official sources & further reading
- Overview of OpenAI crawlers — OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
- Perplexity crawlers — Perplexity
- RFC 9309: Robots Exclusion Protocol — IETF
Related reading
Frequently asked questions
I disallowed GPTBot. Have I removed myself from ChatGPT's answers?
No — those are separate agents with separate settings, and OpenAI documents them that way. GPTBot collects content that may be used to train foundation models; the agent behind search answers is OAI-SearchBot, and OpenAI's own example shows allowing one while disallowing the other. Disallowing a training crawler is a licensing decision about future models, not a decision about today's answers, and the inspector shows the effective rule for each agent separately so the two are never confused.
My robots.txt returns 404 and I never deleted a file. What happened?
WordPress serves a virtual robots.txt when no physical file exists, and a server rewrite can leave that virtual response answering 404 instead of 200. This is common enough that the inspector reports the HTTP status explicitly and flags a 404 as critical. It matters because a 404 is not a permissive file: under RFC 9309 it means no rules were retrieved at all, and what a given operator does in that state is up to them. Fix the server rule so the file answers 200 before you spend any time on its contents.
If a user-triggered fetcher ignores robots.txt, how do I actually stop it?
You mostly cannot, with robots.txt, and the providers say so — OpenAI states that ChatGPT-User actions are user-initiated so robots.txt rules may not apply, and Perplexity states that Perplexity-User generally ignores it. Anthropic takes a different position and documents Claude-User as respecting standard exclusions. Where a fetcher does not honour the file, the mechanism you need is enforcement rather than advice: authentication, a paywall, or a block at the edge. Do not reach for IP blocking as an opt-out, because it can stop a crawler reading the robots.txt that carries your preference in the first place.
Do I have to repeat the rules on every subdomain?
Yes. Rules are per host, so example.com/robots.txt does not govern docs.example.com — Anthropic asks explicitly that the rule be added to every subdomain you want to opt out from, and the same mechanics apply to every other agent. The inspector reads the file for the host it is pointed at, so a site spread across several subdomains needs each one checked on its own.
If I allow every AI crawler, will I be cited more often?
There is no documented basis for saying so. Allowing access produces eligibility and nothing else: access is a precondition for retrieval, retrieval is a precondition for citation, and no provider documents the whole chain. That is why crawler policy is never scored here and never presented as an optimisation — a missing Allow is a business choice, not a defect. Whether any of it reaches a real answer is a separate observation, made through provider APIs under recorded conditions.