Skip to content
siterank.info
Sign in Start free

AI crawlers

AI Crawlers Explained: Search, Training and User-Triggered Bots

Not every AI bot does the same job. Here are the documented categories, the agents each company publishes, and what blocking one actually costs you.

By ankitkumarvig@gmail.com Published September 9, 2026 Last reviewed September 9, 2026 Sources verified September 9, 2026 12 min read

"Block AI bots" is one instruction, but there is no single AI bot to block. Every major provider now runs several agents with different jobs, publishes them separately, and states different consequences for disallowing each one. A site that adds a blanket rule usually blocks the agent that would have cited it and leaves the one collecting training data untouched, or the other way round. This article sets out the categories the providers themselves document, lists the agents each company currently publishes, and explains what each disallow decision does and does not affect.

Three jobs, not one

The useful distinction is not "AI versus search". It is what the fetch is for, because that is the axis every provider's documentation is organised on.

Search and retrieval crawlers build an index that the provider's answer product queries at question time. They behave like a conventional search crawler: scheduled, breadth-first, robots.txt-respecting. OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot and Perplexity's PerplexityBot are all in this group. Disallowing one removes you from that provider's answers, which is the same trade you make when you disallow Googlebot.

Training crawlers collect content that may become part of a model's training corpus. They are not connected to whether you appear in that provider's answer product today. OpenAI's GPTBot, Anthropic's ClaudeBot, Google's Google-Extended and Apple's Applebot-Extended are the documented examples. Disallowing one is a licensing decision about future models, not a visibility decision.

User-triggered fetchers retrieve a specific URL because a person asked for it in the product — pasted a link, asked a question that needs a page, added a source to a notebook. ChatGPT-User, Claude-User, Perplexity-User and Google's family of fetchers are in this group. This is the category where the rules change: because a human initiated the request, several providers state their fetcher generally ignores robots.txt.

There is a fourth, narrower group worth knowing about: product and safety validators such as OpenAI's OAI-AdsBot, which only visits pages submitted as ads and whose data OpenAI states is not used to train foundation models.

Why the categories are load-bearing

Every provider that documents multiple agents also documents that the settings are independent. OpenAI states each setting is independent of the others and gives the worked case explicitly: allow OAI-SearchBot to appear in search results while disallowing GPTBot to signal that content should not be used for training. Anthropic documents three bots precisely to "enable website owner transparency and choice". Perplexity says each setting works independently.

So the mental model of a single AI on/off switch is wrong in a way that costs money. Disallow: / under GPTBot says nothing at all about whether ChatGPT's search feature can surface your pages, and Disallow: / under OAI-SearchBot removes you from those answers while leaving training crawls governed by a separate rule.

One more asymmetry: OpenAI notes that if a site allows both OAI-SearchBot and GPTBot, it may use the results of a single crawl for both purposes rather than crawling twice. Allowing both is not two crawl budgets.

The agents, by company

Verified against each company's own documentation on 9 September 2026. Bot rosters change; re-check the linked page before writing a rule. Evidence tier for every row in this table: OFFICIAL PROVIDER GUIDANCE.

Token Company Job Documented robots.txt behaviour Source
OAI-SearchBot OpenAI Surfaces sites in ChatGPT's search features Respects it; opted-out sites are not shown in ChatGPT search answers OpenAI crawlers
GPTBot OpenAI Crawls content that may be used in training foundation models Respects it; disallow signals no training use OpenAI crawlers
OAI-AdsBot OpenAI Validates the safety of pages submitted as ChatGPT ads Visits submitted ad pages only; not used for training OpenAI crawlers
ChatGPT-User OpenAI Fetches pages for user actions in ChatGPT and Custom GPTs User-initiated, so "robots.txt rules may not apply" OpenAI crawlers
Claude-SearchBot Anthropic Improves search result quality inside Claude Respects it; disallow prevents indexing for search Anthropic crawlers
ClaudeBot Anthropic Collects web content that may contribute to model training Respects it; disallow excludes future material from training sets Anthropic crawlers
Claude-User Anthropic Accesses sites when a Claude user's question needs them Documented as controllable via robots.txt Anthropic crawlers
PerplexityBot Perplexity Surfaces and links sites in Perplexity results; not used for foundation model training Respects it; allow it to appear in results Perplexity crawlers
Perplexity-User Perplexity Visits a page when a user's question needs it "Since a user requested the fetch, this fetcher generally ignores robots.txt rules" Perplexity crawlers
Googlebot Google Search crawling and indexing Respects it; blocking removes you from Google Search Google's common crawlers
Google-Extended Google Controls use of content for training and grounding Gemini models in Gemini Apps and the Vertex AI API Respects it; does not affect Google Search inclusion and is not a ranking signal Google's common crawlers
GoogleOther Google Generic crawler used by various Google product teams Respects it Google's common crawlers
Google-CloudVertexBot Google Crawls sites on a Vertex AI customer's instruction for agent building Respects it Google's common crawlers
Google-GeminiNotebook, Google-Agent, Google-Read-Aloud, Google-Pinpoint, Google-CWS, GoogleMessages, GoogleProducer, FeedFetcher-Google, Google-Site-Verification Google User-triggered fetches inside specific products Google states that because the fetch was requested by a user, these fetchers generally ignore robots.txt Google user-triggered fetchers
Applebot Apple Powers Spotlight, Siri and Safari search; crawled data may also help train Apple foundation models Respects it; falls back to Googlebot rules if no Applebot rules exist About Applebot
Applebot-Extended Apple Opt-out for using Applebot-crawled data to train Apple's general-purpose foundation models Respects it; pages that disallow it can still appear in Apple search results About Applebot
meta-externalagent Meta Crawls for training foundation AI models and indexing content Respects it Meta web crawlers
meta-webindexer Meta Improves Meta AI search result quality Respects it Meta web crawlers
meta-externalfetcher Meta Fetches individual links at a user's request "May bypass robots.txt rules" Meta web crawlers
meta-externalads Meta Crawls for advertising and business products Respects it Meta web crawlers
facebookexternalhit Meta Fetches shared links for previews May bypass robots.txt for security or integrity checks Meta web crawlers
Amazonbot Amazon Improves Amazon products; may be used to train Amazon AI models Respects robots.txt and honours noarchive, noindex and none meta tags; caches robots.txt for up to 30 days Amazonbot
CCBot Common Crawl Builds the open Common Crawl corpus, widely used as a training input by third parties Respects it; documented disallow example given CCBot

Microsoft's bingbot belongs in any complete picture — it crawls for Bing and Bing results ground Copilot answers — but Bing's crawler help page would not render usable content when fetched on 9 September 2026, so nothing about Bing's current roster or its AI-specific meta directives is asserted here. Treat any second-hand claim about Bing's NOARCHIVE/NOCACHE behaviour as unverified until you have read the guidelines page yourself.

The user-triggered category deserves its own decision

This is where most site owners are surprised. Google, Perplexity and Meta all state in writing that user-triggered fetchers generally ignore robots.txt, and OpenAI says robots.txt rules "may not apply" to ChatGPT-User. The reasoning is consistent across providers: the request is a person acting on your public page, more like a browser than a crawler.

Two consequences follow.

First, a Disallow rule for these tokens is a request, not a control. Anthropic is the exception worth noting — it documents Claude-User as one of the three bots controllable through robots.txt, and states that disabling it prevents content being retrieved in response to a user query. That is a meaningfully different position from Perplexity's, and it is the kind of difference that dissolves if you write one blanket rule for "AI bots".

Second, if you genuinely need to stop user-triggered fetches, robots.txt is the wrong layer. Only authentication, a paywall, or an edge block enforces anything. Everything in robots.txt is voluntary compliance by the operator. That is not a criticism of the protocol — RFC 9309 describes an advisory mechanism and always has — but it is the difference between a policy and a control.

Writing rules that say what you mean

Group by intent, not by company. A policy that allows AI search products to cite you while declining training use looks like this:

# Allow AI search and answer products to find and cite the site
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Decline use of content for foundation model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap_index.xml

The inverse policy — allow training, decline search indexing — is legitimate and rarer; it is what a site that wants long-term presence in model weights but not in live answer products would write. There is no correct default here. Access is a licensing and bandwidth decision, not an optimisation, and no honest audit can score it.

Three mechanical points that catch people out:

  • Rules are per host, including subdomains. Anthropic states this explicitly: apply the rule to every subdomain you want to opt out from. example.com/robots.txt does not govern docs.example.com.
  • Changes are not instant. OpenAI and Amazon both document roughly a 24-hour lag before their systems reflect a robots.txt change. Amazon caches robots.txt for up to 30 days for some purposes.
  • Never drop existing rules or Sitemap: lines when you rewrite the file. This is the single most common way a robots.txt edit causes damage.

Anthropic also documents support for the non-standard Crawl-delay extension, which is not part of RFC 9309 and is not honoured by every operator:

User-agent: ClaudeBot
Crawl-delay: 1

Use it only where the operator documents support. Asserting a crawl-delay, wildcard or Allow behaviour that a specific operator has not documented is guesswork dressed as configuration.

Verifying that a bot is what it claims

A user-agent string is a header. Anyone can send it, and impersonation of AI crawlers is common enough that Common Crawl warns about it on its own documentation page. A hit in your logs is not evidence that a provider visited you.

Verification is by IP. OpenAI publishes a JSON file per agent (searchbot.json, gptbot.json, adsbot.json, chatgpt-user.json under openai.com), Anthropic publishes one combined list at claude.com/crawling/bots.json, and Perplexity publishes perplexitybot.json and perplexity-user.json under perplexity.com. Google and Bing support reverse DNS verification. Perplexity's own guidance for firewall rules is to combine user-agent matching with IP verification rather than trusting either alone.

Do not reach for IP blocking as an opt-out mechanism. Anthropic states directly that blocking its IP addresses may not work correctly or guarantee an opt-out, because it stops the crawler reading your robots.txt in the first place. Blocking the address you use to communicate your preferences is self-defeating.

What none of this buys you

Allowing every AI crawler does not produce citations. The chain is crawled → retrieved → cited → recommended, and each arrow is a separate, mostly undocumented step. Access is a precondition for the first arrow and nothing more. Google says it plainly for its own products: there are no additional requirements to appear in AI Overviews or AI Mode, and no special optimisations are necessary.

The corollary is that crawler policy is not an SEO score component and cannot be presented as one. SiteRank AI inspects your robots.txt against a registry of these agents, reports the effective rule for each operator with the documented consequence beside it, and proposes a policy as a diff you approve. It never pre-selects "allow all" or "block all", and it never reports a missing Allow as a defect. Blocking a training crawler is a defensible business choice; so is allowing one.

Evidence classification

Claim or practice Tier
Each operator's documented agents, purposes and stated consequences OFFICIAL PROVIDER GUIDANCE
robots.txt syntax and precedence ESTABLISHED STANDARD (RFC 9309)
User-agent strings are spoofable; verify by IP ESTABLISHED STANDARD
Allowing a search crawler is a precondition for being cited by that product OFFICIAL PROVIDER GUIDANCE
Allowing a search crawler produces citations Not supported at any tier
Blocking training crawlers affects search or answer visibility Contradicted by Google and Apple documentation

Key takeaways

  • AI crawlers do three documented jobs — search indexing, training collection, and user-triggered fetching — and every provider controls them independently.
  • Blocking a training crawler is a licensing decision. Blocking a search crawler is a visibility decision. They are not interchangeable.
  • User-triggered fetchers generally ignore robots.txt by documented policy at Google, Perplexity and Meta; Anthropic documents Claude-User as controllable.
  • Verify crawlers by published IP list, never by user-agent string.
  • Re-read the provider pages before writing a rule. Rosters change, and asserting a token from memory is how invalid rules get shipped.

Official sources & further reading

Frequently asked questions

Does a Disallow rule actually stop a crawler fetching my pages?

Only if the operator chooses to honour it. The Robots Exclusion Protocol is advisory by design, so a rule is a stated preference that well-behaved operators follow and others ignore. Every consequence in the table above is one the provider has documented for its own agents; none of it is enforced by your server. If content genuinely must not be fetched, the control is authentication, a paywall or an edge block, and nothing less.

How do I check whether a request calling itself GPTBot is genuine?

Verify by IP address, never by the header. OpenAI, Anthropic and Perplexity each publish lists of the addresses their agents use, and Google and Bing support reverse DNS verification; the user-agent string on its own is a claim that anyone can send. Perplexity's own firewall guidance is to match the user agent and verify the address rather than trusting either alone. Until that check has been done, treat the agent name in your logs as untrusted input.

Will a CDN or WAF rule override what my robots.txt says?

Yes, and usually in the direction people do not expect. An edge rule blocks at the network layer regardless of robots.txt, so a site can publish Allow: / for a search agent and still refuse every one of its requests at the edge, with nothing in the robots.txt to explain the absence. Address blocking also carries the self-defeating property Anthropic states directly: a crawler that cannot reach your site cannot read your robots.txt, so it never sees the preferences you wrote. Where the edge and the file disagree, the edge wins.

Do these rules need repeating on every subdomain?

Yes. Rules are per host, so example.com/robots.txt says nothing about docs.example.com or shop.example.com — Anthropic makes the point explicitly for its own agents, and it follows from the protocol rather than from any one provider's policy. Sites commonly opt out on the marketing domain and leave a documentation or support subdomain wide open without noticing. List your hosts and check each one separately.

If I allow every AI crawler, will I get cited?

No, and no provider documents a mechanism by which access would produce a citation. Access is a precondition for the first step of crawled → retrieved → cited, and the later steps are decided by systems that are largely undescribed. Google states there are no additional requirements or special optimisations for appearing in its AI features beyond ordinary search practice. Allowing a crawler removes an obstacle; it does not create an outcome — see crawled is not cited.