How API-Based AI Visibility Tracking Differs From ChatGPT.com
An API observation is not a consumer chat answer. Here are the seven layers that differ, and why we label the measurement instead of claiming a ranking.
Every AI visibility tool has the same problem and most of them solve it by not mentioning it. The thing customers want to know is "what does ChatGPT say about us". The thing that can be measured reproducibly, at reasonable cost, with the site owner's own credentials and without violating anyone's terms, is a provider API response under recorded conditions. Those are not the same measurement, and the gap between them is not a rounding error. This article sets out where the two diverge, what that means for how a number should be labelled, and why we do not close the gap by scraping the consumer product.
The short version
We report a sentence of this shape:
Observed through an OpenAI web-grounded API response, model
<exact id as returned>, web search enabled, locale<x>, 2026-09-09, 5 samples.
We do not report:
Your site ranks #3 in ChatGPT.
The first is an observation under stated conditions. The second is a claim about a system we did not query, expressed in a unit — rank — that a generative answer does not have.
Seven layers that differ
1. The retrieval stack
A grounded answer is assembled from documents retrieved at question time, and the index behind the consumer product is not documented to be the index behind the API. OpenAI publishes OAI-SearchBot as the crawler whose job is to surface websites in ChatGPT's search results — which tells you ChatGPT search has a crawl and index pipeline of its own. Whether the API's web search tool queries that same index, a subset of it, or something else, is not stated on the pages we can read. We therefore treat them as different until documented otherwise, which is the conservative direction.
2. Orchestration
Between a question and an answer sits query rewriting, tool selection, multi-step retrieval, re-ranking and post-processing. Google describes its AI features using "query fan-out" — issuing multiple searches across subtopics to build a response — so the queries actually run against an index are not the query the user typed. Consumer products and API endpoints run different orchestration, and the consumer one changes without release notes. Nothing you can send through an API reproduces it.
3. Personalisation and memory
A logged-in chat user brings account context, prior conversation, stored memory, custom instructions and product settings. An API call brings whatever you put in the request. Two people asking the same question in the same product on the same day can get different answers for reasons that have nothing to do with your site. There is no API parameter that reconstructs a stranger's chat history, and if there were, using it would be worse rather than better.
4. Model version
The consumer product routes to whichever model the provider has decided serves that user, on that plan, at that moment. An API call uses the model you name — and even then, the id you send may not be the id you get. OpenAI's models overview lists an alias field for at least one current model pointing a short name at a longer one, but the page does not explain alias and snapshot semantics in general, so we make no assertion about them. What we do is record the exact model id string the API returns on every call, and mark a discontinuity in any series where it changes.
5. Which tools are enabled
Grounded and ungrounded answers are different systems. In the API, web search is a tool you switch on: OpenAI documents { "type": "web_search" } for Responses API integrations, and only some models support it. The consumer product decides for itself when to search, when to answer from parameters, and when to run other tools. An observation taken with search forced on and one taken from a product that chose not to search are not comparable, so the grounding flag is part of our condition key rather than a footnote.
6. Region and locale
OpenAI's web search tool accepts user_location with free-text city and region, a two-letter ISO country, and an IANA timezone. Consumer products infer location by their own means. Both matter for the answer, but they are set through different mechanisms and cannot be assumed equal. We record what we sent; we cannot record what a consumer session would have inferred.
7. Response shape and citation reporting
The API returns citations as structured data: a url_citation annotation carrying the URL, title and location of each cited source, plus a sources field listing every URL retrieved during the search, which is a superset of the inline citations. That structure is exactly what makes reproducible extraction possible. A consumer interface renders citations for humans, and how many appear is a presentation decision. OpenAI also notes that search_context_size "does not set an exact token count or guarantee a specific number of sources or citations" — so even within the API, the count of returned sources is not a stable quantity to build a metric on without a denominator.
What that adds up to
| Layer | API | Consumer product | Can we reconcile? |
|---|---|---|---|
| Index / retrieval | Provider's API-side retrieval | Product's own search pipeline | No — not documented as the same |
| Orchestration | What the endpoint does | Product-specific, undocumented, changes silently | No |
| Personalisation | Only what we send | Account, history, memory, instructions | No |
| Model | Named by us; returned id recorded | Routed by the provider | Partially — we record ours |
| Tools | Explicit flags | Chosen by the product | No |
| Region | user_location we set |
Inferred by the product | Partially |
| Citations | Structured annotations plus a sources list |
Rendered for a reader | No |
Four "no"s in a seven-row table is the honest summary. This is not a calibration problem waiting for a clever correction factor. There is no function from an API observation to a consumer answer.
Why we still measure it
An imperfect instrument is not a useless one, provided you say what it measures. What an API series does give you, reliably:
- Reproducibility. Fixed conditions, recorded provenance, and raw snapshots stored so that a changed extractor can recompute history.
- Comparability over time within itself. Held conditions plus discontinuity marking on model change give a series you can read — see how AI search citations change over time.
- Comparability across entities. Share of voice between you and a user-defined competitor set, measured under identical conditions in the same run, is a fair comparison even though the absolute level is instrument-specific.
- Directional signal on retrievability. If your domain is never in the
sourceslist for prompts squarely about your subject, something upstream is wrong — indexability, structure, entity clarity — and that is actionable regardless of what the consumer product would have shown. - Cost control and consent. Bring your own key, only the prompt text leaves the site, and every call is metered against a budget you set.
The right way to hold this: the API series is a tracer, not a census. It tells you about the direction and stability of your retrievability under a fixed set of conditions. It does not tell you what a given person saw in an app.
Why we do not scrape the consumer product
The obvious shortcut is to drive ChatGPT.com in a headless browser and read the answers. We do not, for four reasons in descending order of how much they should matter to you.
It is against the provider's terms. Respecting them is a standing rule for us, not a case-by-case judgement, and a measurement product whose data collection is a terms violation is a liability its customers inherit.
It would not measure what it claims to either. A scraped session is one account, in one region, with one set of settings, one memory state and whatever model the router gave it. It is not "what ChatGPT says"; it is what one synthetic user saw once. Substituting it for an API observation trades a labelled instrument for an unlabelled one and calls the result realism.
It is not reproducible. Interfaces change, sessions expire, anti-automation measures escalate. A time series built on scraping breaks in ways you cannot distinguish from real change — which is the specific failure mode this whole discipline exists to avoid.
It contaminates the measurement. Automated sessions are not ordinary usage, and treating a bot-driven account's answers as representative of human sessions is an unexamined assumption doing a lot of work.
Where a provider offers no API path to its consumer experience, the correct output is "we cannot measure this", not a proxy presented as equivalent. Saying so costs a feature and buys the only thing a measurement product actually sells.
How this shows up in the product
Nothing in a SiteRank AI visibility report says "ranking", and nothing says "ChatGPT". Every observation carries provider, exact returned model id, parameters, grounding flag, locale, timestamp, prompt id and version, sample index, run id, extracted citations and a bounded raw snapshot. Every metric returns its numerator, denominator, sample size, a 95% Wilson interval and a plain-language definition string — a bare percentage is never returned. Failed and blocked calls are excluded from both numerator and denominator and the excluded count is reported, because a timeout is not evidence that a model omitted your site.
The word "position" appears only where a response is genuinely ordered — an explicit ranked list, or an ordered citation array. Inferring rank from where your name falls in a paragraph is fake precision, and we do not do it. The definitions and version constants are in the measurement methodology.
What not to assume
- That a competitor's tool measures the consumer product because its marketing says ChatGPT. Ask what the observation record contains. If it has no model id, no sample size and no denominator, it is not a measurement.
- That "no mention" means no visibility. It means no mention, in that sample, under those conditions, for that prompt.
- That being cited is being recommended. A citation says the system used the page as a source. Crawled ≠ retrieved ≠ cited ≠ recommended.
- That the gap is closing. It may narrow or widen; both are the provider's decision, not a trend you can extrapolate.
Evidence classification
| Claim | Tier |
|---|---|
API web search returns url_citation annotations and a sources list; only some models support the tool |
OFFICIAL PROVIDER GUIDANCE (OpenAI) |
search_context_size does not guarantee a number of sources or citations |
OFFICIAL PROVIDER GUIDANCE (OpenAI) |
user_location accepts city, region, ISO country and IANA timezone |
OFFICIAL PROVIDER GUIDANCE (OpenAI) |
OAI-SearchBot exists to surface sites in ChatGPT's search results |
OFFICIAL PROVIDER GUIDANCE (OpenAI) |
| AI features may issue multiple searches per query | OFFICIAL PROVIDER GUIDANCE (Google) |
| The API index and the ChatGPT search index are the same | Not documented — we assume they are not |
| An API observation can be converted into a consumer-product result | Not supported at any tier |
| A generative answer has a "rank" or "position" | Not supported; the unit does not exist for unordered prose |
Key takeaways
- Seven layers differ between an API call and a consumer chat session, and four of them cannot be reconciled at all.
- The measurable thing is an observation under stated conditions. Label it that way, every time, including in the UI.
- API observations are still worth taking: they are reproducible, comparable within themselves, and fair across a competitor set measured in the same run.
- Scraping the consumer product breaks provider terms, is not reproducible, and would not measure what it claims to.
- Where no API path to the consumer experience exists, say it cannot be measured rather than shipping a proxy.
Official sources & further reading
- Web search (API tools guide) — OpenAI
- Models overview — OpenAI
- Overview of OpenAI crawlers — OpenAI
- AI features and your website — Google Search Central
Related reading
Frequently asked questions
Why not just type the question into ChatGPT myself?
Do that — as a qualitative check, never as the measurement. What you see is one account, in one region, with your memory, your custom instructions, whatever model the router served you and whatever the product decided about searching, which makes it one person's session on one occasion rather than a repeatable observation. It has no denominator, no recorded conditions and no way for a colleague to reproduce it, so two people can disagree about "what ChatGPT says" and both be right. Read it for texture — how you are described, what you are compared against — and keep the series on the instrument that records its own conditions.
What does an API observation fail to capture about what a real chat user sees?
The four layers in the table above that cannot be reconciled: the retrieval stack behind the product, its orchestration, the personalisation a signed-in person carries, and how citations are selected and rendered for a reader. Model routing and inferred location are only partially reconcilable on top of that. Which is why the right way to hold an API series is as a tracer rather than a census — it describes the direction and stability of your retrievability under conditions you set, and it does not describe what any individual saw in an app.
If the two cannot be reconciled, why pay to measure the API at all?
Because a labelled instrument still answers questions an unlabelled one cannot. It is reproducible, it is comparable with itself over time once discontinuities are marked, and it is fair across a competitor set measured in the same run under identical conditions. It also gives a directional read on retrievability: if your domain never appears in the sources list for prompts squarely about your own subject, something upstream is wrong and worth fixing regardless of what a consumer session would have shown. How those numbers are constructed is set out in how to measure AI mention rate and citation rate.
Would more samples, or more providers, close the gap?
No, and it is worth being clear why. More samples narrow the confidence interval around the quantity the API instrument measures; they tell you nothing about the layers the API never touches, so a tighter interval on the wrong quantity is still the wrong quantity. Adding providers adds instruments, each with its own conditions and its own gap — useful for coverage, but averaging them together produces a number with no definition behind it.
What do I tell a stakeholder who wants a single number for ChatGPT?
Give them the observation instead of the number they asked for: for this prompt set, our domain was cited in n of N valid observations through the OpenAI API with web search enabled, on a recorded model id, with a 95% interval this wide and this many samples excluded as errors. Then say the part that is genuinely unknown — how often the site appears in the consumer product is not measurable from outside, by us or by anyone selling a dashboard that claims otherwise. A stakeholder who understands that distinction can act on the number they do have; one who has been handed a figure nobody can define cannot.