Answers
Why prompt changes break your AI visibility comparison
When the prompt set shifts between runs, the comparison stops being readable. Here is how a prompt log and an intersection recalculation keep AI visibility numbers traceable.
If the prompt set changes between one run and the next, the comparison stops being readable. The two measurements no longer describe the same set of questions: the questions you put to ChatGPT and to Google in one run are not the same ones as in the run before, so the difference you see may come from the changed perimeter rather than from a change in visibility.
Take an illustrative example, not a product feature. Suppose you noted ten questions put to ChatGPT and to Google, then reran the measurement with twelve prompts and saw the figure rise. Without knowing which prompts came in and which went out, that jump cannot be interpreted.
Three causes of a discontinuity
A discontinuity between two runs has, in practice, three distinct origins. Separate them before you read any number.
- Prompts added or removed. The set changes size: the average or aggregate moves because the base of the calculation changed, not because the engines answered differently.
- Prompts reworded. The question is the same in substance but the text is different. The system treats it as a new prompt; the reader sees the same intent. This is the case that causes the most confusion.
- Missing measurements. When a visibility figure is unavailable, RankAI shows it as N/A, never as zero. Treating N/A as zero creates an artificial discontinuity that looks like a decline.
Recalculating on the intersection with a filled-in log
RankAI's working proposal, not a universal standard, is to keep a prompt log with a stable identifier and recalculate the comparison only on the intersection. Here are the steps, each with a verifiable criterion.
- Assign a stable identifier to every prompt before the run. Verifiable criterion: two consecutive runs must carry the same identifier for the same question, and the count of identifiers must be reconstructable afterwards.
- Classify every prompt as unchanged, reworded or new in the log. Verifiable criterion: every reworded prompt has an explicit reference to the prompt it came from; a prompt without such a reference is classified as new.
- Calculate the comparison only on the intersection of unchanged identifiers. Verifiable criterion: the number of prompts in the intersection is stated next to the value, and unchanged, reworded and new prompts add up to the total set.
- Isolate reworded prompts in a separate reading. Verifiable criterion: reworded prompts do not enter the intersection, because the text changed and the answer is not guaranteed to be equivalent; they appear in a distinct view, not in the main comparison.
What stays outside the comparison
After the recalculation on the intersection, part of the reading stays outside the comparison and must be declared as such: new prompts, reworded prompts and missing measurements. A reworded prompt does not enter the intersection because the different text does not let you attribute the difference to the engine rather than to the wording; it remains readable only as a separate observation. Missing measurements stay N/A and must not be summed or averaged as if they were zero. This way the comparable part is explicit and the non-comparable part is not hidden inside an aggregate number.
What often gets lost is that the readability of a comparison depends not on the quality of the engine being queried but on how traceable the perimeter is. A filled-in log turns a prompt change from an invisible event into a documented one: you know how many questions stayed identical, how many were reworded and how many are new. Our proposal is to treat the intersection as the only basis for the main comparison and to keep everything else in a declared side reading. This avoids two symmetrical errors: attributing to the engine a jump that comes from the question set, or hiding an N/A inside an average. The discipline of the log costs little and makes every later run reconstructable, even at a distance in time and by someone who did not fill in the original log.
A limit you can check
The method described works only if the log is filled in before the run. If identifiers are assigned afterwards, the intersection cannot be verified and the comparison goes back to being an untraceable reading. In that case the only defensible statement is that the two runs are not comparable.
For cases where the perimeter cannot be reconstructed, and for the distinction between what RankAI shows and what it does not measure, see the detailed page.