How LLM Visibility Monitoring Actually Works

An AI answer engine does not publish a result list you can scrape, and it does not return the same answer twice. That makes visibility in generative answers a sampling problem rather than a scraping problem, and it changes what a monitoring tool can honestly claim. This article describes the mechanics: what an observation is, how a prompt set is designed, what provider APIs actually return, and where the boundary sits between a real measurement and a number someone invented.

The thing being measured

Conventional rank tracking works because a search engine result page is a deterministic, ordered list. Ask twice, get the same ten links in the same order, subject to personalisation and location. Position is a property of the page, so "position 4" is a fact about the world.

A generative answer has none of those properties. It is prose. It is assembled at answer time from a retrieval step whose results vary, and it is produced by a sampling process that varies again. There is no list, so there is no position. What there is, and all there is, is a set of binary facts about a single response:

• Was the target entity named in the answer text?
• Did the answer carry citations at all?
• Was the target domain among the cited sources?
• Which competitors were named alongside it?

Each of those questions has a yes/no answer for one response. Repeat the response enough times under recorded conditions and the proportion of yeses becomes a rate with a confidence interval. That is the entire measurement model. Everything a credible AI visibility tool reports is a rate over repeated binary observations, or it is fabricated.

The observation is the unit of record

The atom of the system is one prompt sent once to one model under recorded conditions, producing one stored response. Without the recorded conditions the response is unusable, because you cannot compare it to anything or reproduce it later.

SiteRank AI stores all of these against every observation, and the methodology page records which formula version produced each stored row. The practical consequence is that any reported percentage can be expanded into the specific responses that produced it. A number that cannot be expanded that way is not evidence.

Controlled prompts

A prompt set is the questionnaire. It is fixed in advance, approved by the site owner, and changed deliberately rather than casually, because every change resets comparability.

Good prompt sets are built from the questions a buyer would actually type, not from keywords. "Best project management tool for a small agency" is a prompt. "project management software" is a keyword, and pasting a keyword into a chat model measures very little, because nobody asks a question that way.

Three categories are worth separating, because they answer different business questions.

Unbranded prompts are the honest test. Branded prompts nearly always produce a mention, because the brand is in the question, and reporting a headline mention rate driven by branded prompts is self-deception. Keep the categories separate in reporting, and be explicit about which prompts drove which number. The related question of who belongs in the comparison is covered in choosing competitors for LLM visibility monitoring.

Prompts are versioned. A prompt monitoring series that silently changed its wording in March is two series drawn as one line.

Mentions and citations are different measurements

A mention is the target entity appearing in the generated text. A citation is the target domain appearing in the response's attribution data. They are independent, and conflating them is the most common error in this field.

A model can name your company from parametric knowledge with no retrieval and no link. That is a mention with no citation, and it suggests the model has absorbed something about you, but not that your website was read today. Conversely, an answer can cite your page as a source without naming your organisation in the prose: the page supplied a fact, the brand went unmentioned.

Both matter. They are recorded separately, carry separate denominators, and are never averaged into a single "visibility score". The mechanics of each rate are worked through in how to measure AI mention rate and citation rate.

What the provider APIs actually give you

This is where the measurement becomes concrete, because it is the API response — not a screenshot, not a scrape — that supplies the evidence.

OpenAI's web search tool documentation describes a response with two relevant parts: a web search call item recording what the model did (search, open_page, find_in_page), and a message item whose annotations include url_citation objects carrying the URL, the title and the location of the citation within the text. Separately, a sources field lists every URL consulted, which the documentation notes is often a larger set than the URLs actually cited. OpenAI also states that inline citations must be made clearly visible and clickable in any interface that displays the output. The tool is exposed on the Responses API for current models, with a legacy path on Chat Completions, and its parameters include search_context_size, an approximate user_location, and domain filters accepting up to 100 allowed or blocked domains. Search context is capped at 128k tokens regardless of the model's own context window.

Two things follow from the distinction between sources and citations, and both are worth stating plainly.

1. Consulted is not cited. A domain can be retrieved and read without appearing in the attribution. A tool that reports "cited" while counting the sources list is measuring retrieval and calling it citation.
2. Domain filters are a research instrument, not a result. Constraining a query to a set of domains and then reporting that those domains were cited is circular.

Perplexity and Anthropic document their web-facing behaviour on the crawler side. Perplexity describes PerplexityBot as designed to surface and link websites in search results while respecting robots.txt, and Perplexity-User as fetching pages on a user's explicit request, noting that it generally ignores robots.txt because a person asked for the fetch. Anthropic documents ClaudeBot as collecting web content for its generative models, Claude-SearchBot as improving search result quality, and Claude-User as supporting user-initiated access, and states that it honours industry-standard directives in robots.txt. OpenAI's crawler documentation draws the same three-way split: GPTBot for training, OAI-SearchBot for the ChatGPT search index, and ChatGPT-User for user-triggered fetches, with the note that robots.txt rules may not apply to the last of those because the visit was requested by a person.

That taxonomy matters to monitoring for exactly one reason: whether a crawler can reach your pages sets an upper bound on whether a grounded answer can cite them, and nothing more. Access is a precondition, not a result. Crawled is not cited develops the point.

Sampling, repetition and volatility

One response is an anecdote. The same prompt sent to the same model an hour later can name a different set of companies, because retrieval shifts and generation samples.

So the same prompt is sent repeatedly under identical conditions, and the disagreement between samples is measured rather than hidden. SiteRank AI reports that disagreement as volatility: the proportion of samples that disagree with the majority answer for the same prompt under an identical condition key — prompt id, prompt version, provider, returned model, web-search flag and locale. It ranges from 0 to 0.5, and with fewer than two samples it reports that it is not measurable rather than reporting zero.

Volatility is a first-class number, not an error bar tucked into a footnote. A mention rate of 40% with near-zero volatility means the model reliably includes you in four prompts out of ten. The same 40% with volatility near its maximum means the answer is close to a coin flip on every prompt, and next week's chart movement will be noise. Those two situations call for different responses, and a tool that reports only the 40% cannot tell them apart.

Every rate is also published with a 95% Wilson score interval computed at the actual sample size. At n=1 that interval spans nearly the whole range, which is the honest answer to "what does one API call prove?"

Provider APIs are not the consumer product

An API observation is evidence about the API. It is not evidence about what a person sees in a chat app, and this is the most important caveat in the discipline.

SiteRank AI observes through provider APIs deliberately, and says so on every chart. The alternative — driving the consumer product with browser automation — produces numbers that look more like "what a user sees" while being unrecordable, unreproducible, and generally contrary to the provider's terms of service. The trade-off is examined in API-based AI visibility tracking vs ChatGPT.

There is a matching limitation on the search-engine side. Google states that clicks from AI Overviews and AI Mode are reported in Search Console's Performance report within the "Web" search type, together with everything else, rather than as a separate breakout. No first-party report isolates AI feature performance, which means nobody has one.

Why "ChatGPT rank #3" is misleading

The phrase fails on four counts, each independently fatal.

1. There is no ordered list. Rank presupposes positions. A paragraph of prose that names three companies does not assign them ranks; the order of mention is a property of sentence construction, not an evaluation.
2. There is no stable population. Rank is relative to a set of competitors. In a generative answer the set is whatever the model chose to mention this time, and it changes between samples.
3. It hides the sample size. "Rank #3" from one API call and "rank #3" from two hundred are presented identically, though only one of them is a measurement.
4. It implies a mechanism nobody has documented. No provider publishes an ordering function for entities named in generated answers. Asserting a rank asserts knowledge of a system whose behaviour is not disclosed.

The defensible statement takes this form: "Across 40 valid observations of prompt set A on model M with web search enabled, between 2026-08-01 and 2026-08-31, the site was mentioned in 22.5% of responses (95% CI 12.3%–37.5%)." Longer, less satisfying, true.

Failed calls are not evidence of absence

An API call that times out, gets rate-limited, or returns an auth error says nothing about whether the model would have mentioned the site. Counting it as a non-mention biases every rate downward, and the bias grows precisely when the provider is having a bad day.

SiteRank AI therefore admits only observations with an ok status into the numerator or the denominator of any rate, and reports the excluded count alongside every metric. If 45 calls were attempted and 5 failed, the denominator is 40 and the interface says so. A tool whose denominator equals the number of calls attempted is either mishandling failures or hiding them.

What this can and cannot support

Causal attribution needs the hardest guard. Model updates, index refreshes, competitor publishing and sampling noise all move these numbers, usually at once. The measurement supports association plus a volatility figure indicating how much of the movement could be noise. It does not support "we did X and it worked". A change in the returned model or the prompt version inside a series breaks comparability outright, and the chart should show a break rather than a continuous line drawn through a changed instrument.

Key takeaways

• The unit is one recorded observation, not a ranking. Every reported rate must decompose into the observations behind it.
• Mention and citation are separate measurements with separate denominators. Never merge them into one score.
• Provider APIs return citation annotations plus a broader list of consulted sources. Consulted is not cited.
• Repeat prompts under identical recorded conditions, and report volatility and a confidence interval, or the number means nothing.
• API observations describe the API, not the consumer chat product.
• Failed calls are excluded from both numerator and denominator, and the exclusion is reported.

Official sources & further reading

• Web search tool — OpenAI
• Overview of OpenAI crawlers — OpenAI
• Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
• Perplexity crawlers — Perplexity
• AI features and your website — Google Search Central

Related reading

• How to measure AI mention rate and citation rate
• What is share of voice in AI search?
• Choosing competitors for LLM visibility monitoring
• How AI search citations change over time
• AI visibility monitoring for WordPress