How AI Search Citations Change Over Time
Why the same prompt cites different sources next week, what actually moved, and how to tell a real change from sampling noise.
Ask a web-grounded model the same question twice and you will often get two different sets of cited sources. Ask it next month and you may get a third. None of that requires anything to have happened to your site. A generative answer is the output of a stochastic model reading a changing index through an orchestration layer that the provider revises without notice, and every one of those three components moves on its own schedule. If you intend to report a trend from that system, you first have to be able to tell which layer moved — and, more often than anyone likes, to say that nothing detectable moved at all.
Five things that change, on five different clocks
The model. Providers ship new models and retire old ones, and an id that reads like a stable name may not be. OpenAI's models overview lists an alias field for at least one current model, pointing a short name at a fuller one; the page does not set out the general semantics of aliases and dated snapshots, so we do not assert them. What we do instead is record the model id the API actually returns on every call and compare it to the previous call. That is an observation, not an inference.
The retrieval index. A grounded answer is assembled from documents fetched or looked up at question time. That index is refreshed on the provider's schedule, and the provider does not publish it. A page you published on Monday may be retrievable on Wednesday or in three weeks, and the two are indistinguishable from outside.
The orchestration. Between your prompt and the answer sits query rewriting, tool selection, ranking and post-processing. Google describes AI features issuing "query fan-out" — multiple searches across subtopics to build a response — which means the queries actually run against the index are not the query you typed. OpenAI's web search tool exposes a search_context_size control and notes it "does not set an exact token count or guarantee a specific number of sources or citations". Orchestration changes are invisible from the outside and are never announced.
The web. Competitors publish, pages move, sites change their crawler policy. Your relative position in a candidate set can move without you touching anything.
Sampling. Even with all four of the above held constant, the model is stochastic. Two identical calls can produce different answers, with different sources cited.
The consequence is direct: a difference between two observations is not evidence of anything until you have ruled out the fifth cause, and you cannot attribute it to a specific one of the first four without provider-side information you do not have.
One observation is not a trend
The most common failure in AI visibility reporting is charting n=1. A single call returns a binary — you were mentioned or you were not, your domain was cited or it was not — and a binary drawn as a point on a time axis looks exactly like a trend even though it carries almost no information.
The honest way to express this is an uncertainty interval computed at the actual sample size. At n=1 a Wilson interval for a proportion spans nearly the entire range, which is the correct answer and is why we display it rather than a bare 0% or 100%. Two runs a week apart, each n=1, that differ from each other are perfectly consistent with a completely unchanged system.
| What you have | What you can honestly say |
|---|---|
| 1 observation | "On this date, under these conditions, the answer did / did not cite the domain." |
| n samples, one run | A rate with an interval, and the volatility across those samples. |
| Two runs, same conditions, adequate n | A delta, stated as association, with the intervals shown and overlap noted. |
| Two runs, different model id | Nothing comparable. Mark a discontinuity. |
| Two runs, different prompt text | Nothing comparable. It is a different instrument. |
That last row is worth dwelling on. The prompt is the instrument. Editing a prompt to make it "better" mid-series breaks the series in the same way that recalibrating a thermometer mid-experiment breaks a temperature log. In our data model an edit creates a new prompt version with a new id, and charts do not join across versions.
Volatility as a first-class number
Most tools treat run-to-run variation as an error bar, or hide it. It is more useful as a reported metric, because it tells the reader how much any single observation from that prompt can be trusted — and a high-volatility prompt's trend line is close to meaningless no matter how many points it has.
SiteRank AI defines volatility as the proportion of samples that disagree with the majority outcome for the same prompt under identical conditions. Identical conditions is a precise thing: the condition key is prompt id, prompt version, provider, the model id the API returned, the web-search/grounding flag, and locale.
The prompt id is part of that key deliberately. Pooling several different prompts that happen to share a model would measure between-prompt variation — a different quantity that happens to produce a plausible-looking number. That substitution is the kind of error that survives review because nobody checks what the denominator was.
The metric ranges from 0 to 0.5. Zero means every sample agreed. A half means the samples split evenly, which for a binary outcome is the maximum possible disagreement — there is no such thing as more than half the samples disagreeing with the majority. With fewer than two samples it reports measurable: false rather than 0, because one sample cannot disagree with itself and reporting perfect stability would be a lie by construction.
What volatility is for
- Choosing a sample size. Pick n from the volatility you have observed for that prompt, not from a round number. A prompt that always answers the same way needs fewer samples than one that splits.
- Choosing a schedule. A stable prompt does not need daily sampling; a volatile one is not made trustworthy by it.
- Reading a delta. A five-point move on a prompt with volatility near 0.4 is noise. The same move on a prompt with volatility near 0.02 is worth investigating.
- Deciding whether to chart at all. Some prompts should be shown as a distribution, not a line.
Discontinuities: when the instrument changes
Holding conditions constant across a time series is the whole basis of comparison, and the one condition you cannot hold is the one the provider controls. When the returned model id changes, the series is broken. Continuing the line across that point produces a chart that attributes a provider's model release to your content strategy.
Our approach is mechanical rather than judgemental. Every observation stores the exact model id string as returned by the API and the prompt version. A routine scans each series for changes in either field and marks a break at that index; the chart renders a gap and a labelled marker instead of a connecting segment, and comparisons refuse to span it. The same treatment applies to a prompt version change.
This is deliberately conservative. It means a provider model rollout resets your baseline, and there is no way around that which does not involve pretending two different instruments are one. The alternative — a smooth line through a model change — is the single most misleading artefact in this category of tooling.
Two related rules fall out of the same principle:
- A partial run is never compared against a complete one. If rate limiting or a budget cap cut a run short, the run is partial and is labelled as such.
- Failed calls are excluded from both numerator and denominator, and the excluded count is reported. A timeout is not evidence that the model omitted your site. Counting errors as non-mentions biases every rate downward, and the bias grows exactly when the provider is having a bad day.
What a change does not tell you
Suppose domain citation rate for a prompt cluster moved from 0.21 to 0.34 between two runs, with adequate samples and no model change. What can you say?
You can say the two runs differ, report both intervals, and note whether they overlap. You can note what changed on your side in the interval. You cannot say your content change caused it. Model updates, index refreshes, competitor publishing and sampling noise all move this number, and you have no instrument that separates them. Association, never causation — and where the change does not exceed the sampling noise, the correct report is "no detectable change" rather than a slope drawn through two points.
There is also a ceiling on what any of this measures. An API observation is not what a person sees in a consumer chat product; the retrieval stack, orchestration, personalisation, memory and enabled tools all differ. That gap is large enough to deserve its own treatment — see how API-based AI visibility tracking differs from ChatGPT.com. And a citation is not a recommendation: being listed as a source says the system used your page, not that it endorsed you.
Designing a series you can actually read
- Fix the conditions and record all of them. Provider, returned model id, parameters, grounding flag, locale, region, timestamp, prompt id and version, sample index, run id. An observation without complete provenance is unusable and should not be stored.
- Store the raw response. Extractors and metric definitions change. Without snapshots, history cannot be recomputed, and you will be asked to recompute it.
- Sample more than once per run, always. A single sample measures nothing about a stochastic system.
- Report the denominator with every rate. A percentage without its denominator is not a measurement.
- Schedule by observed volatility and cost, not by habit. Every call spends the site owner's own API credit.
- Expect resets. Provider model changes will break your baselines periodically. That is a property of the subject, not a defect in the tool.
The metric definitions, the condition key, the Wilson interval and the discontinuity rule are all written down and versioned in the measurement methodology, because a formula change has to bump a version — stored rows carry the version that produced them, and a silent formula change would corrupt every historical comparison.
Evidence classification
| Claim or practice | Tier |
|---|---|
| Web-grounded answers return citations as structured annotations, and the number of sources is not guaranteed | OFFICIAL PROVIDER GUIDANCE (OpenAI web search tool) |
| AI features may issue multiple searches per query ("query fan-out") | OFFICIAL PROVIDER GUIDANCE (Google) |
| Generative model output is non-deterministic across identical calls | ESTABLISHED STANDARD — a property of sampling from a distribution |
| A proportion needs an uncertainty interval at the actual n | ESTABLISHED STANDARD — statistics |
| Volatility as proportion disagreeing with the majority under a fixed condition key | Our definition, published and versioned — not a provider metric |
| A model id change breaks comparability of a series | HYPOTHESIS in the strict sense — no provider documents the magnitude — but the conservative treatment is the only defensible one |
| A rise in citation rate was caused by a site change | Not supported at any tier |
| An API observation reflects the consumer product | Not supported at any tier |
Key takeaways
- Five independent things move: the model, the index, the orchestration, the web, and the sampler. A difference between observations names none of them.
- One observation is a fact about one call. It is not a trend, and its interval at n=1 spans nearly everything.
- Volatility — the proportion of samples disagreeing with the majority under identical conditions — belongs on the report, not in an error bar.
- The condition key must include the prompt id, or you measure between-prompt variation by accident.
- When the returned model id or the prompt version changes, mark a discontinuity. Never draw a line through a changed instrument.
- Report association and show volatility beside any delta. Never claim causation.
Official sources & further reading
- Web search (API tools guide) — OpenAI
- Models overview — OpenAI
- AI features and your website — Google Search Central