The support bot that invented a policy
In April 2025, users of the AI coding tool Cursor started getting logged out when they moved between devices. They asked support why. An assistant signing its messages as “Sam” told them the behaviour was intentional — a subscription was valid on one device at a time.
No such policy existed. The logouts came from a session-management bug, a race condition that surfaced on slow connections. The support assistant was an AI, and it had answered a question about company policy by inventing one. The thread reached the front page of Hacker News, paying customers cancelled, and a co-founder apologised publicly. The incident is documented by The Register and catalogued as Incident 1039 in the AI Incident Database.
What makes it worth revisiting eighteen months later is how ordinary the failure was. The system was not jailbroken or adversarially prompted. It was asked a routine question about a company’s own policy, and it answered from something other than that company’s policy. Every AI assistant that now describes your business to a prospective customer is one retrieval away from the same failure — and unlike Cursor, you will usually not find out, because there is no thread and no apology. There is just an answer, given once, to someone you never meet.
The industry measures whether you are mentioned, not whether you are described correctly
A substantial measurement industry has grown up around AI search in the last two years. It is worth being precise about what it measures.
The most rigorous public work we are aware of is the study SparkToro published on 28 January 2026, conducted by Rand Fishkin with Patrick O’Donnell of Gumshoe.ai. Six hundred volunteers ran twelve prompts through ChatGPT, Claude and Google’s AI search a combined 2,961 times across November and December 2025, with each prompt run 60 to 100 times per platform. The headline result: there is less than a one-in-a-hundred chance that two responses to the same prompt return the same list of brands, and less than one in a thousand that they return the same list in the same order. The full method and data are public.
That is a genuinely well-built study, and it settles something important. But note carefully what it measured: which brands appeared, and in what order. It did not test whether anything the engines said about those brands was true.
This holds across the category. Brand-monitoring platforms measure mention frequency and share of voice. Citation trackers measure which sources an engine drew from. Visibility tools measure whether you show up for a prompt. All of them are answering the question are we in the answer?
We searched for published work measuring the other question — whether the facts an engine states about a specific company are correct and current — and did not find any. Adjacent work exists: general hallucination benchmarks score models on open-domain factuality, and vendor QA tooling checks a company’s own chatbot against its own product feed. Neither measures what a third-party assistant tells the public about a company it does not work for.
Two different failures, two different consequences
Not being mentioned is a marketing problem. Being described incorrectly is a different category of problem, and in regulated contexts it is a liability one. A stale price quoted confidently to a buyer, a superseded refund window, a certification you let lapse, a capability you deprecated two releases ago — each of these is an answer a customer may act on, and none of them shows up as a miss in a visibility dashboard. You score perfectly on mention rate while the answer is wrong.
The Stale Answer Benchmark: the method, published before the results
We are running a benchmark to measure this. This article is the protocol, published before we collect the data.
That sequence is deliberate. A benchmark whose method appears after its numbers is a marketing asset — the design can be adjusted until the result is interesting. Publishing the protocol first means the method is fixed, the failure conditions are named in advance, and a null or boring result has to be reported as readily as a dramatic one. For a company selling claim verification, we think that is the minimum standard we can hold ourselves to. It also lets anyone reproduce or contest the design before we have anything invested in the outcome.
Sample: sixteen companies
Sixteen B2B software companies, selected for having a publicly documented price, a publicly documented policy of some kind, and enough market presence that answer engines return substantive responses about them. The sample is deliberately small and deliberately not random. It is large enough to detect whether the problem is common and far too small to estimate a population rate. We say so here so that nobody quotes it as one.
Four claim types per company
Four claims per company, one from each of the categories that go stale most often and matter most when they do:
Pricing — a published list price or plan tier.
Policy — a refund window, trial length, cancellation term or data-retention commitment.
Capability — whether a named feature or integration exists in the current product.
Compliance status — a certification, attestation or regulatory status the company publicly holds.
These four share a property that makes them measurable: each has exactly one correct answer at a given moment, and that answer is published by the company itself. Sixteen companies by four claims gives sixty-four claims under test.
Ground truth: the company’s own current page
For each claim, the source of truth is the company’s own live published page. We record the URL, the exact passage, and the timestamp at which we read it. Where the company publishes a machine-readable date for that page, we record that too.
This is the step that separates this benchmark from a hallucination test. We are not asking whether an engine knows a fact about the world. We are asking whether it agrees with the company’s own current published statement — a question with an unambiguous answer and a citable arbiter.
Three engines
Each claim is queried against three answer engines with web access enabled. Crawler access to a site is governed differently by each: appearance in ChatGPT answers is governed by OAI-SearchBot rather than GPTBot, Perplexity by PerplexityBot, and Google’s AI surfaces by Googlebot rather than Google-Extended. We record which of the company’s pages, if any, each engine cites.
Replication, because a single run measures nothing
The SparkToro result forces a design decision here, and it is the most important one in the protocol. If brand lists repeat in under one per cent of paired responses, then any measurement taken from one run of one prompt is dominated by sampling noise. A single-run study cannot distinguish a systematically stale answer from a one-off.
So every claim is queried multiple times per engine, in independent sessions, and we report the distribution rather than a single verdict. Where a claim is answered correctly in some runs and incorrectly in others, that instability is itself a finding and is reported as one, not averaged away.
Scoring
Each response is scored against the recorded ground truth into one of five buckets:
Correct — matches the current published claim.
Stale — matches a previously published version of the claim.
Wrong — matches no version the company has published.
Declined — the engine refused or said it did not know.
Unattributed — an answer was given with no source cited.
The distinction between stale and wrong is the one we care most about, because they have different causes and different fixes. A stale answer means the engine found your old content. A wrong answer means it found something that was never yours.
What we publish, and what we withhold
Results will be published in aggregate and anonymised. We will report that a given share of sampled companies had at least one stale or wrong claim. We will not name which companies, or which claims.
This is partly a legal judgement — publishing a list of named companies alongside the assertion that public answers about them are false is not a risk we are willing to carry, and the companies in question have no say in being sampled. But it is also a methodological one. A benchmark that names names becomes a story about those companies and gets cited as gossip. A benchmark that reports rates becomes a reference and gets cited as evidence. We would rather be cited as evidence.
The five measures
Five figures come out of the dataset. They are defined here so that the definitions cannot move once the numbers exist.
M1 — Affected company rate. The share of sampled companies with at least one claim scored stale or wrong on at least one engine.
M2 — Per-engine claim accuracy. For each engine, the share of claim responses scored correct.
M3 — Cross-engine disagreement rate. The share of claims where at least two engines gave materially different answers to the same question. Examined in detail in our companion piece on conflicting AI answers.
M4 — Median staleness lag. For claims scored stale, the median age of the superseded version the engine reproduced — how far behind the current fact the answer sits.
M5 — Own-source citation rate. The share of answers that cite the company’s own page as a source.
M4 and M5 are the two we most want to see. M4 puts a duration on the problem: it converts “AI sometimes says outdated things” into a number of days a wrong fact survives in public circulation. M5 tests the assumption underneath most GEO advice — that the fix for a wrong answer is to publish a better page. If engines are frequently answering about a company without citing that company at all, then the fix is not only publishing.
What this method cannot tell you
Stated in advance, so that they cannot be quietly dropped later.
Sixteen companies is not a population. Nothing here supports an industry-wide rate. It can show that a failure is easy to find; it cannot show how common it is.
The engines change underneath the measurement. Models, indexes and retrieval behaviour are updated continuously. Results describe a collection window, not a stable property, which is why the window is recorded and the data collected once rather than topped up.
Scoring involves judgement. Deciding whether a paraphrased answer “matches” a published claim is not fully mechanical. We will publish the scoring rubric and the borderline calls.
Non-determinism limits precision. Replication reduces this but does not remove it. Small differences between engines should not be read as rankings.
It measures answers, not harm. We are not measuring whether anyone acted on a wrong answer, or what it cost them. That would be a much harder study and we are not claiming to have done it.
Results
Data collection has not yet begun. This page will be updated with the results, the scoring rubric and the anonymised dataset when it has. The method above is fixed as of the verification date on this article; any change to it will be published as a dated revision alongside the results rather than edited silently into this text.
Why we are running this at all
Because it is the measurement our own product implies. We argue that company claims written in prose — prices, policies, capabilities, certifications — have no system of record, and that this is why they drift. That argument deserves a number attached to it, produced by a method someone else can check, rather than an anecdote and a confident tone.
If the result comes back showing the problem is rare, that is worth knowing too, and we will publish it.