Why Being Crawled Does Not Mean Being Cited by an LLM

A server log line showing OAI-SearchBot fetching your pricing page is a real event, and it proves exactly one thing: a crawler fetched your pricing page. It does not mean a model read it, used it, linked to it, or told anyone to buy from you. Between a fetch and a recommendation sit four separate transitions, each of which can fail, and each of which produces a different kind of evidence — or none at all.

This article walks the chain end to end: crawled → retrieved → used → cited → recommended. For each step it states what has to be true for the step to happen, what you can actually observe, and what you cannot.

The chain at a glance

Every arrow in that chain is one-directional and lossy. Crawling is necessary for retrieval by a search-grounded system and sufficient for nothing. Citation is observable; use is not. The step everyone wants to measure — "did our content shape the answer?" — is precisely the one no provider exposes.

Step 1: Crawled

Crawling is the only step in the chain that happens on your infrastructure, which is why it is the only one you can measure without an API.

The AI crawler population splits into three jobs, and the providers document the split themselves.

Three things follow, and they set up everything after.

Blocking the search-side crawler caps the chain at zero. OpenAI states plainly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. That is the one directional guarantee in the whole system, and it runs the wrong way for anyone hoping robots.txt is a growth lever: it can remove you, it cannot add you.

Allowing it guarantees nothing. No provider documents a path from "crawlable" to "cited". Access is a gate, not a pump.

The training crawler and the search crawler are independent. OpenAI's documentation treats them as separate settings. Blocking GPTBot while allowing OAI-SearchBot is a coherent position — out of training data, in the search index — and the should you block AI training crawlers? question turns on values and licensing rather than on visibility arithmetic.

What you can observe here is genuinely good evidence, provided you verify it. A user-agent string is self-reported and trivially forged; verification means reverse DNS or matching against the provider's published IP ranges. An unverified log line is a claim, not an observation.

What you cannot observe: why the crawler came, whether the fetched content was retained, and whether any of it was ever loaded into a context window.

Step 2: Retrieved

Retrieval is the step where a specific question causes a system to assemble a candidate set of documents. In a retrieval-augmented answer, the model does not consult the web at large; it consults whatever the retrieval layer handed it, which is typically a small number of documents or passages selected against the query.

For your page to be in that set, several things must hold simultaneously:

• The page must be in whatever index the system searches, which is not the same index as Google's for most providers and is not published for any of them.
• The retrieval query must match it. Note that the query is often not the user's question. Google describes a "query fan-out" technique for AI Mode, issuing multiple related searches to build a response — so the string that has to match your page may be one you never anticipated.
• The page must survive selection against every other candidate, at passage granularity in most modern stacks.
• The retrieval budget must reach it. OpenAI's web search tool caps search context at a fixed token budget and exposes a search_context_size parameter; there is a ceiling on how much retrieved text reaches the model, whatever the model's own context window.

This step is not observable from outside. No provider publishes what was in the candidate set. The closest thing to evidence is a user-triggered fetch in your logs — ChatGPT-User, Claude-User or Perplexity-User arriving seconds before an answer — which tells you the system went and got your page for someone's live question. That is a strong hint about retrieval for that request, and it is not available for the far more common case where an answer is assembled from an existing index without a fresh fetch.

The practical implication is that everything conventionally described as GEO work — clear entity naming, self-contained passages, accurate headings, structural machine-readability — is an attempt to improve the odds at this step. That is a reasonable theory of the mechanism. It is not a measurement, and Google states that there are no additional requirements or special optimisations for appearing in its AI features beyond ordinary SEO, and specifically that no new machine-readable files, AI text files, or special structured data are required.

Step 3: Used

"Used" means the retrieved content actually influenced the tokens that came out. This is the step people mean when they say "the AI is reading my content", and it is the step with the least evidence attached to it.

The reason is architectural. A model receives retrieved passages as context and generates an answer conditioned on all of it plus everything in its weights. There is no export of which passage produced which clause. Attribution inside the generation is an open research problem, not a product feature, and no major provider exposes anything of the kind.

Two failure modes make this concrete.

Retrieved but ignored. A page can be in the candidate set and contribute nothing, because a better source answered the question. OpenAI's own API makes this visible in aggregate: the response's sources field lists every URL consulted, and the documentation notes it is broader than the inline citations, which show only the most relevant references. Consulted is not cited, and cited is not the same as decisive.

Used but invisible. A model can absorb a fact from your page during training, reproduce it years later with no retrieval and no link, and there is no trace anywhere. This is the reason a mention and a citation are recorded as separate measurements with separate denominators: a mention with no citation suggests parametric knowledge; a citation with no mention means your page supplied a fact while your brand went unnamed.

What you can observe here: nothing. Any tool claiming to report "how much your content influenced the answer" is reporting an inference. The honest version of that metric does not exist yet.

Step 4: Cited

Citation is where measurement becomes possible again, because the provider returns it as structured data.

OpenAI's web search tool returns two distinct things in the response: web_search_call items recording the actions taken (search, open_page, find_in_page), and message annotations of type url_citation carrying the URL, the title and the character positions of the cited span within the answer text. The API also requires that inline citations be made clearly visible and clickable in any interface that displays the output.

That gives a site owner a genuine, reproducible observation: for this prompt, this model, this run, with web search enabled, was our domain among the returned citations? It is binary per response, and repeated sampling turns it into a domain citation rate with a confidence interval. That is what LLM visibility monitoring measures, and it is the only rigorous foothold in the entire chain.

Four cautions keep it honest.

1. Do not count sources as citations. They are different fields with different meanings. A tool that counts consulted URLs and labels the result "citations" is measuring retrieval and mislabelling it.
2. Resolve redirects and link wrappers before matching a host, or your own citations will be undercounted.
3. Domain filters are an instrument, not a result. The API accepts up to 100 allowed or blocked domains. Constraining a query to your own domain and reporting that you were cited is circular.
4. An API observation is not the consumer product. A different system prompt, orchestration layer, retrieval stack and model version sit behind ChatGPT the app. Anything measured through the API describes the API.

There is a matching limit on Google's side: clicks from AI Overviews and AI Mode are reported inside the "Web" search type in the Search Console Performance report, together with everything else, rather than as a separate breakout. Nobody has a first-party report isolating AI feature performance, because Google does not publish one.

Step 5: Recommended

The last step is the one that matters commercially, and it is the softest. Being cited as the source of a definition is not the same as being named as the option someone should choose. A single answer can cite three domains for facts and recommend a fourth product entirely.

What can be observed: whether your entity is named, and — with a human reading the response, or a carefully validated classifier — the sentiment and framing of the naming. "X is a good fit for small teams" and "X is often listed but users report Y is stronger" are both mentions. Collapsing them into one binary throws away the part a business actually cares about.

What cannot be observed: why the model framed you that way, and whether it would frame you the same way for a real user with account history, memory and a different orchestration layer around the model.

The correct posture is to record framing as a separate, clearly-labelled human judgement over stored raw responses, kept apart from the mechanical counts. Never present it as a score. It is qualitative evidence, and it is more useful when it is honest about being qualitative.

What not to assume

• That a log line proves an answer. Crawling and answering are separated by an unknown interval and an unobservable selection process.
• That allowing every AI crawler increases citations. Access removes a blocker. No provider documents it as a positive input.
• That llms.txt closes the gap. No provider documents consuming llms.txt, and the file's presence is not ingestion. EMERGING PRACTICE, no higher — see what is llms.txt?.
• That schema produces citations. Structured data helps machines parse a page. Google does not guarantee even a rich result from valid markup, let alone an AI citation.
• That an absent mention means absent visibility. It means this prompt, this model, this run, this locale, this moment. One query is not a ranking.
• That a rise in citations was caused by your change. Model updates, index refreshes and competitor publishing all move the number, usually at once.

Evidence classification

Key takeaways

• Five steps, not one: crawled, retrieved, used, cited, recommended. Each fails independently.
• Crawling is observable in your logs, and worth verifying rather than trusting the user-agent string.
• Retrieval and use are not observable from outside. Any tool reporting them is inferring.
• Citation is observable through provider APIs as structured annotations, and is the only rigorous measurement in the chain.
• Consulted is not cited; cited is not used; cited is not recommended.
• Access is a precondition. It sets a ceiling, never a floor.

Official sources & further reading

• Overview of OpenAI crawlers — OpenAI
• Web search tool — OpenAI
• Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
• Perplexity bots — Perplexity
• Google crawlers and fetchers — Google Search Central
• AI features and your website — Google Search Central
• Introduction to robots.txt — Google Search Central

Related reading

• AI crawlers explained
• How LLM visibility monitoring actually works
• Should you block AI training crawlers?
• What is llms.txt?
• AI crawler policy for WordPress