Skip to content
siterank.info
Sign in Start free

AI crawlers

Anthropic Crawlers: ClaudeBot, Claude-SearchBot and Claude-User

Anthropic runs three bots and states why IP blocking is the wrong opt-out. Here is each one's purpose and the documented cost of disabling it.

By ankitkumarvig@gmail.com Published September 9, 2026 Last reviewed September 9, 2026 Sources verified September 9, 2026 7 min read

Anthropic documents three web robots, and unusually among the large providers it publishes a "what happens when you disable it" statement for each one, plus an explicit warning against the opt-out method many site owners reach for first. Everything below comes from Anthropic's own help-centre article on crawling, read on 9 September 2026, and from the IP list it links.

The three bots

Bot Anthropic's stated use What Anthropic says disabling it does
ClaudeBot "Helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training" "Signals that the site's future materials should be excluded from our AI model training datasets"
Claude-SearchBot "Navigates the web to improve search result quality for users"; analyses content to improve relevance and accuracy of search responses Prevents indexing for search optimisation, "which may reduce your site's visibility and accuracy in user search results"
Claude-User Supports Claude users; when individuals ask Claude questions, it may access websites with this agent Prevents retrieval of your content in response to a user query, "which may reduce your site's visibility for user-directed web search"

Evidence tier for this table: OFFICIAL PROVIDER GUIDANCE.

Anthropic frames the three-way split as deliberate: it "uses different robots to enable website owner transparency and choice". Read the second column carefully, because the three consequences are not the same kind of thing.

ClaudeBot is a licensing decision. The stated effect is about future materials and training datasets. It is forward-looking by design — a disallow is not a removal request for models already trained — and Anthropic makes no claim that disabling it affects whether Claude can find or cite you today.

Claude-SearchBot is a visibility decision. This is the search crawler. Disabling it means your content is not in the index that Claude's search draws on. Anthropic hedges appropriately with "may reduce", which is more honest than a guarantee in either direction.

Claude-User is the interesting one. It is a user-triggered fetcher, the same functional category as ChatGPT-User and Perplexity-User — but Anthropic documents it as one of three bots governed by robots.txt, and states that disabling it prevents the system retrieving your content in response to a user query. Perplexity says the opposite about its equivalent agent, stating that its user fetcher generally ignores robots.txt. That divergence is precisely why a single blanket "AI bot" rule is a bad instrument: the same category behaves differently at different companies.

What Anthropic commits to

The article sets out four principles, all of which are worth holding Anthropic to and worth quoting when someone claims AI crawlers are lawless:

  • Data collection should be transparent — the named bots above are the agents it uses.
  • Crawling should not be intrusive or disruptive — Anthropic says it is thoughtful about how quickly it crawls the same domains and respects Crawl-delay where appropriate.
  • Its bots respect "do not crawl" signals by honouring industry-standard directives in robots.txt.
  • Its bots respect anti-circumvention technologies — it states it will not attempt to bypass CAPTCHAs.

These are policy commitments, not technical guarantees, which is the nature of robots.txt generally: RFC 9309 describes an advisory protocol. Only authentication and edge blocking enforce anything.

Writing the rules

To block a bot from an entire site, Anthropic gives this form, to be placed in the robots.txt file in your top-level directory:

User-agent: ClaudeBot
Disallow: /

Substitute Claude-SearchBot or Claude-User for the other agents. A policy that declines training use while remaining fully available to Claude's search and to user-directed retrieval looks like this:

# Decline use for model training
User-agent: ClaudeBot
Disallow: /

# Remain findable and retrievable in Claude
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Sitemap: https://example.com/sitemap_index.xml

Anthropic also supports the non-standard Crawl-delay extension for throttling rather than blocking, which is the right tool when the problem is server load rather than policy:

User-agent: ClaudeBot
Crawl-delay: 1

Crawl-delay is not part of RFC 9309 and is not honoured by every operator — Google, for instance, does not support it. Use it only where the operator documents support, as Anthropic does here.

Subdomains are separate

Anthropic states it explicitly: do this "for every subdomain that you wish to opt out from". robots.txt is scoped to a host. A rule at example.com/robots.txt has no effect on docs.example.com, shop.example.com, or a subdomain your marketing team spun up on a third-party platform last year. If you are opting out for licensing reasons and you miss a subdomain, you have not opted out.

Anthropic's advice on robots.txt versus IP blocking

This is the most practically useful paragraph on the page, and it contradicts a common instinct.

Anthropic states that opting out requires modifying robots.txt, and that alternate methods such as blocking the IP addresses its bots operate from "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."

The mechanism is worth being clear about. Your robots.txt is how you communicate the preference. If the crawler cannot reach that file, it cannot read the preference — it just sees a network failure, which is not the same signal as a Disallow. Blocking the address you are trying to talk to is self-defeating. Rate limiting or WAF rules that quietly 403 AI user agents have the same effect, which is one reason to check the edge before assuming your robots.txt is doing what it says.

The published IP list exists for the opposite purpose: verification, not blocking. Anthropic publishes its crawler addresses at https://claude.com/crawling/bots.json and states that a crawler whose source IP is on that list is coming from Anthropic. Use it to confirm that a ClaudeBot string in your logs is genuine, because a user-agent header is trivially forged and impersonation is common. A hit in your access log is not evidence that Anthropic visited you; a hit from a listed address is.

If a bot appears to be misbehaving, Anthropic asks that you contact it directly, and — a detail that saves a round trip — that you write from an email address on the domain you are reporting about, since it is otherwise hard to verify the report.

What not to assume

  • Do not assume a disallow deletes anything. ClaudeBot's documented effect is on future materials entering training datasets. It is not a retroactive removal mechanism, and no provider offers one through robots.txt.
  • Do not assume Claude-User behaves like other user-triggered fetchers. Anthropic documents it as controllable. That is company-specific and could change; re-read the page before relying on it.
  • Do not assume allowing Claude-SearchBot produces citations. It makes you eligible to be indexed. Anthropic documents no mechanism by which a page is selected and named in an answer, and neither does any other provider. Crawled is not retrieved; retrieved is not cited.
  • Do not assume robots.txt is enforcement. It is a request that Anthropic states it honours. That is a policy, not a lock.
  • Do not read "may reduce your site's visibility" as a measured figure. It is a directional statement about a mechanism, not an observation. If you want a number, it has to be measured against a fixed prompt set over time — see our methodology.

Evidence classification

Statement Tier
Each bot's purpose and stated disable consequence OFFICIAL PROVIDER GUIDANCE
Anthropic's bots respect robots.txt and anti-circumvention measures OFFICIAL PROVIDER GUIDANCE
Crawl-delay supported by Anthropic OFFICIAL PROVIDER GUIDANCE (non-standard extension)
IP blocking is an unreliable opt-out because it prevents robots.txt being read OFFICIAL PROVIDER GUIDANCE
robots.txt is advisory, per-host, and not enforcement ESTABLISHED STANDARD (RFC 9309)
Allowing Claude-SearchBot increases citations in Claude Not supported at any tier

Key takeaways

  • Three bots: ClaudeBot for training, Claude-SearchBot for search indexing, Claude-User for user-directed retrieval. Each is controlled independently in robots.txt.
  • Anthropic documents Claude-User as robots.txt-controllable, unlike Perplexity's equivalent agent. Do not generalise user-triggered fetcher behaviour across companies.
  • Disallowing ClaudeBot affects future training data only. It is a licensing choice, not a visibility one, and it is not a defect.
  • Apply rules to every subdomain you want covered.
  • Use claude.com/crawling/bots.json to verify crawler identity. Do not use IP blocking as an opt-out — Anthropic says it can prevent your robots.txt being read at all.

Official sources & further reading