How to Test a GEO Tool: A Replication Protocol

Here is an experiment that takes four minutes and invalidates most of what you will be shown in a GEO demo.

Open ChatGPT. Ask “what are the best project management tools for small agencies?” Note the brands and their order. Open a new chat and ask the identical question. Note it again. Repeat twice more.

You will usually get four different lists.

That is the measurement environment every GEO tool operates in. A visibility number produced from one run of one prompt on one day is a single draw from a distribution nobody has characterised. It is not wrong. It is far less precise than a dashboard percentage looks.

This is a protocol for finding out how much of a tool’s output is signal. It starts with the thing almost every explanation of this category gets wrong.

The nondeterminism is not temperature

The standard explanation is that models sample from a probability distribution, so identical inputs produce non-identical outputs — set the temperature to zero and you get repeatability. That explanation is wrong, and how it is wrong decides what a protocol can and cannot fix.

In published inference-engineering work on batch invariance, a single prompt on Qwen3-235B was completed 1,000 times at temperature 0 — greedy decoding, no sampling at all. Those 1,000 completions produced 80 unique outputs. All of them were identical for the first 102 tokens. Then they diverged.

The mechanism is batch non-invariance. Your request is served in a batch alongside other people’s, and the composition of that batch changes the order in which floating-point reductions are performed inside the inference kernels. Reorder a sum of floats and the result is not bit-identical. Shift one logit in its last decimal places and, wherever two candidate tokens are near-tied, the argmax flips. Every token after that point belongs to a different answer.

Two consequences follow, and both are structural rather than commercial.

Removing this variance requires control of the inference stack — the batching, the kernels, the serving configuration. No tool in this category has that. They query somebody else’s API or scrape somebody else’s web interface. The noise floor is irreducible for every vendor on the market simultaneously. It is an architectural fact, not a vendor shortcoming.

And the variance is not evenly spread. A brand that is a strong, well-established category association appears in nearly every generation. A brand near the boundary of the model’s category knowledge appears in some and not others. Most brands buying a GEO tool sit near that boundary — which means most brands are measuring the noisiest part of the distribution, then plotting it daily.

The only available defence is sampling. Sampling is the axis on which this category publishes least.

What the standards body says

In August 2026 the IAB published Measuring Visibility in the AI Era (36 pages, Project Eidos). It is the strongest third-party authority currently available, and it is blunt: “More than 20 providers now sell AI visibility measurement tools… There is no common definition of a ‘mention,’ no standard for what constitutes a citation.” Consequently, “two providers measuring the same brand in the same category can produce materially different results”, and “A brand may receive conflicting share of voice figures from two different tools, with no way to determine which is more accurate” — which, it argues, “creates a race to the bottom on claims rather than a competition on quality.”

Two lines are directly operational. It treats “fewer than 50 queries per measurement program as exploratory rather than directional”, and it asks providers to “disclose responses per query” — the single number this article is built around.

Read its limits too. It “does not establish certification or evaluate individual providers”, and it concedes that “Specific quantitative thresholds for acceptable variability remain an open working group question.” The PDF carries no reference list, so at least two of its own figures are unattributed. Apply the standard to the standard.

What the tools actually sample

Sampling depth, as published by each vendor, read at source on 1–2 September 2026. It is the column missing from every feature comparison, including most accounts of what these tools measure.

Eight of the nine either take one draw per cycle or decline to say. Only Evertune claims multi-sampling, and it states its own figure two ways on one page. Its headline is unit-ambiguous too: “100,000 prompts tracked across 11 AI models”. If that counts sampled responses rather than distinct questions, it implies roughly 90–100 unique prompts. That matches the vendor’s own worked example: 100 unique prompts, each asked 100 times. The vendor does not disambiguate and we could not resolve it.

Set that table beside the IAB’s position — “Single-response measurement is not measurement… any metric derived from one response per query reflects a sample of one” — and sampling depth stops being a footnote in a feature comparison. It is the comparison.

How many answers a conclusion needs

This part is plain binomial arithmetic. It is ours, not any vendor’s, and you can check it in a spreadsheet.

Now price a typical trial. Fifty prompts, three engines, one run each, is 150 answers. That is respectable by category standards, and still not enough to separate 20% from 30% with confidence.

Count answers, not prompts. Prompts are what you buy; answers are what you can conclude from.

How many runs per prompt: an open question, not a setting

There is no settled number, and the two most rigorous 2026 preprints disagree about how to spend a measurement budget.

Schulte, Bleeker and Kaufmann (arXiv:2604.07585) want more runs of each prompt, for brand monitoring and for source coverage. Zatuchin (arXiv:2607.13304) decomposes a large response set and recommends the opposite — that budget go to more languages and more models instead of repeats. Their figures, side by side:

Disclosure on the first paper: the lead author works in the AI-visibility industry, so “buy more runs” is directionally self-serving. We could not confirm the specific employer, so weigh the affiliation rather than discount the paper.

Caveat the second properly: the outcome measured there is sentiment polarity, not mention rate, so the ICC does not transfer one-to-one.

The IAB declines to set a threshold at all.

Nobody knows the right sampling design yet. That is a more defensible position than “the tools are wrong”, and more useful, because the one point both papers agree on carries the argument on its own: a single run per prompt is uninformative about stability, whatever else it is good for.

The protocol

Seven steps. Run it on a tool you already pay for, or on a trial — several tools can be tried without a sales call. Budget about a month of elapsed time and roughly four hours of actual work.

1. Freeze a prompt set, size it, then split it

Write at least 50 prompts in real buyer language. Below 50, the IAB calls the exercise exploratory rather than directional. Sixty is a workable set:

• 40 target prompts — the ones you care about.

• 10 control prompts — questions in your category where you have no reasonable chance of appearing.

• 10 anchor prompts — questions with a dominant, unambiguous answer (“what is the largest cloud provider”).

Controls and anchors make the test interpretable. If your anchor prompts move, the instrument is moving, not the market.

2. Make the vendor state sampling depth — and three other things

Ask how many times each prompt is sampled per engine per cycle. The IAB now asks providers to disclose exactly this, and eight of the nine do not publish it. If the answer is one, every daily figure has an unknown error bar. If they cannot answer, that is itself the finding.

Then ask three questions almost nobody asks:

• Logged-in or logged-out collection? OpenAI began showing labelled sponsored placements to logged-in US Free and Go users on 9 February 2026. It opened a self-serve ChatGPT ads manager on 5 May 2026. Login state now decides whether ads are in the sampled surface at all.

• Are sponsored placements separated from organic mentions? No vendor in this set documents that it does. If a paid block renders inside a scraped answer, a “brand mention” count may be counting media.

• How much history is retained, and how many competitors can you track? Competitor count is the meter every headline share-of-voice chart depends on, and several vendors publish no number for it.

3. Run a same-day duplicate

Track the same prompt list twice on the same day — a second project, a second workspace, or a manual re-run. Two measurements of an unchanged world.

Whatever difference you see is your noise floor. Every movement smaller than it, for the rest of your subscription, is uninterpretable.

4. Do nothing for two weeks

No content changes, no schema, no PR, no new pages. This is the hardest step and the one that makes the rest work.

You are measuring how much the number moves when you do nothing. If it drifts five points during a fortnight of inactivity, a five-point gain after a campaign is not evidence of anything.

5. Change exactly one thing

One change, dated, on one surface. Publish one substantive page, or correct one factual claim across your site. Not five things.

Then wait another two weeks. Crawl-to-answer latency is real and varies by engine; a same-week reading measures indexing speed, not visibility.

6. Read the answers, not just the scores

Export the raw answer text for the target prompts, before and after. Mark every factual assertion about your company and check it against what is currently true.

The interesting result is usually not in the visibility column. An engine can move you up three positions while quoting a price you retired last year — the accuracy gap that visibility metrics are not built to surface. And where a tool does check claims, it checks them against a document set you maintain.

Profound is the only vendor that documents the failure mode that follows, and it is a false positive rather than a miss: “a claim about an old pricing tier might be marked accurate if the Knowledge Base doesn’t include the updated pricing page.”

7. Write down what would have falsified you

Before reading the after-state, write the result that would mean the change failed. Then read.

This single step is the difference between a test and a story.

What variance is acceptable

No published cross-vendor standard exists — the IAB says so explicitly — so use your own instrument.

That last row catches vendor-side changes — a prompt-list migration, an engine mix change, a scoring update — that arrive looking exactly like a market event. Ownership changes belong in the same bucket: Semrush completed its acquisition by Adobe on 28 April 2026, and Scrunch has been a Sitecore company since 3 June 2026. Roadmaps change hands with companies.

Audit the vendor’s numbers before you audit your visibility

There is a cheap test that takes an hour and predicts a lot: check whether a vendor’s own published figures agree with each other. Eight of the nine have at least one place where they do not; six are shown here. The ninth, Brandlight, publishes too little to contradict itself.

Otterly’s annual prices are published as per-month figures, against higher monthly prices:

The 23 June 2026 snapshot shows the same annual prices with no badge.

None of this is deceit. Pricing pages get restructured, docs lag releases, corpora get recounted. But a vendor’s disagreement with itself about a number it controls completely tells you how much weight to place on a number it controls not at all.

Two mechanical lessons from the same pass. Read at source, in a browser, and record the state the page shipped in. Peec’s pricing is absent from its server HTML by design — an 885KB fetch of /pricing returns zero price tokens, only skeleton divs, because prices render after a client-side geo-IP chain. curl and automated fetchers see nothing, since they execute no JavaScript. That is how one vendor’s entire price list becomes invisible to every automated comparison pipeline, and much of why third-party pages label Peec’s month-to-month EUR prices as annual ones. Toggle defaults matter for the same reason: Profound’s “Billed yearly” switch ships off, while Peec’s page loads with its annual toggle on.

And never multiply. Profound, Athena and Otterly publish annual prices as per-month figures with no yearly total anywhere. Any annual figure you quote is your own arithmetic — and it can disagree with the vendor’s own page furniture. Athena’s shipped code carries a “2 months free” badge implying $2,950 a year. Athena’s own annual Starter figure is $245/month — itself a string present only in the site’s JS bundle, never in the served HTML. Multiplied by 12, that figure is $2,940. See what the nine actually cost for the full picture.

How to read the statistics in this category

GEO marketing runs on circulating numbers whose provenance dissolves on contact. Two examples do most of the work.

The “21% agreement” figure. It traces to Lead Rescue, a $29/month tracker vendor. The design was 50 buyer-intent prompts, one run per prompt, 150 answers. The page carries no date, no model versions and no author beyond a first name. The most useful sentence is the vendor’s own concession, on the same page: “This experiment was a single run per prompt with no temperature control, so it captures each engine’s taste, not its stability.” That is an accurate account of its own limits, and the failure is downstream — the figure now circulates stripped of it. Fifty queries is exactly the IAB’s exploratory floor.

It is also routinely merged with a different number: Parse’s 21.2%, the share of cited domains common to two repeat ChatGPT answers. That figure rests on 693,509 answers, with at least 12 repeats per prompt and a published dataset. Two constructs, near-identical numbers. An AI search summariser merged them unprompted during our own research for this article. Treat any GEO statistic that reaches you through an AI summary as unverified until you have loaded the originating page.

The “40% visibility uplift”. This one is real and peer-reviewed — Aggarwal et al., arXiv:2311.09735, KDD 2024 — and routinely misapplied. GEO-bench simulates a two-stage engine: Google’s top five sources are fed to GPT-3.5-turbo. The 40% therefore measures re-ranking among five sources that were already retrieved. It does not measure getting retrieved, and it does not measure traffic. Martinez’s 45-study survey (arXiv:2607.14035) puts the boundary precisely: the gains are “conditional on a source already being present in a fixed context; they establish neither organic discoverability nor durable traffic effects.”

Three questions dissolve most of the rest:

1. What was n, in answers, and was it replicated?

2. Who paid for it, and do they sell the remedy?

3. Is there a method section, or only a number?

Apply them to every statistic in a GEO deck, including ours.

What this protocol cannot settle

Honesty about limits, since that is the whole argument here.

• It cannot remove batch non-invariance. Nothing available to a buyer can. The protocol measures the noise floor; it does not lower it.

• It cannot compare two vendors’ absolute numbers. Different prompt populations and different definitions of “mentioned” make cross-tool comparison meaningless. Compare each tool to its own history only.

• It cannot compare them on price per prompt either. The units differ: one vendor’s cheapest prompts run on a single engine while another’s run on three, one meters checks (prompt × platform × location), one meters AI responses with a 5× engine multiplier, and at least one appears to meter custom prompts, industry prompts, personas and page audits as four separate allowances. The figure also moves with the billing basis you compute it on.

• It cannot establish causation from one change. It can establish that a movement exceeded your noise floor, which is a much weaker and much more defensible claim.

• It cannot tell you what engines will do next quarter. Model updates reset the distribution, and no protocol survives a version bump.

• It does not check whether the answers are true. That requires a maintained record of what is currently correct about your company, which no visibility tool contains.

The one-line version

Before you trust a GEO number, measure what it does when nothing happens. Most of the movement in this category is the instrument — and the part that is not, nobody has yet agreed how to sample.

How this was verified

This is the 2 September 2026 revision, and it changes the article’s central claim rather than patching it.

What changed. The first version attributed answer variance to sampling temperature. That is not the mechanism. The figures now in the opening section come from published inference-engineering research on batch invariance surfaced in this verification pass: 1,000 completions of one prompt on Qwen3-235B at temperature 0. They produced 80 unique outputs. They were identical for 102 tokens before divergence. No link is given because we have not re-read the original at source; the finding is reported as the research pass recorded it, and the mechanism, not the specific model, is the point.

New third-party authority. The IAB’s Measuring Visibility in the AI Era (August 2026, Project Eidos) is quoted verbatim throughout and now anchors the article in place of a vendor-versus-vendor framing. Its own stated limits are quoted alongside it.

Sampling depth was read at source for all nine tools on 1–2 September 2026. Evertune’s “100 times per model” is quoted from its own site along with the “up to 100x” wording on the same page, so the inconsistency is reported rather than resolved. Profound’s one-response-per-prompt-per-engine-per-day figure is our arithmetic on its published caps, labelled as such, and is not a vendor statement. “Not published” means we found no figure on any vendor page or document we could read — not that none exists behind an enterprise login.

Sample-size arithmetic is ordinary binomial calculation performed here, not sourced from any vendor or paper:

• ±10 points at 72 answers

• ±5 at ~300

• ~294 per period to detect 20%→30%

Check it.

The runs-per-prompt section reports a disagreement rather than a recommendation, because that is what the literature contains. The lead author’s industry affiliation on arXiv:2604.07585 is disclosed because it is material; the specific employer is not named because we could not confirm it, and an unnamed accusation would be worse than none. Zatuchin’s ICC figures are flagged as measured on sentiment polarity, not mention rate.

Statistics. The 21% cross-engine figure is no longer presented as a finding. It is presented with its n, its single-run design and its publisher’s own concession, which is the useful part. Its collision with Parse’s unrelated 21.2% is stated explicitly. The 40% GEO-bench figure has been added with its scope boundary attached.

Prices and provenance. No price in this article is filled from a third-party aggregator. Brandlight’s entry pricing is reported as not published because brandlight.ai/pricing returned HTTP 404 on 2 September 2026. No annual total is attributed to any vendor; where annual arithmetic appears, it is labelled as arithmetic.

Still unverified. Sampling depth for six of the nine. Whether Evertune’s “100,000 prompts” counts questions or responses. Whether any vendor separates sponsored from organic placements. Scrunch’s tier contents and the ChatGPT advertising material come from summarised reads and should be re-checked at source before anyone relies on them.

No first-party benchmark data appears in this article. The protocol is a method, not a result — if you run it, the numbers are yours, and we would rather you published them than cited ours. No head-to-head study exists in which one party ran an identical prompt set through several commercial tools and published the discrepancies. Two subscriptions and a fortnight would fix that. The protocol is Knowledge Company's, published in full so anyone can replicate it without asking us for a number to quote.

Sources: IAB, Measuring Visibility in the AI Era · Schulte, Bleeker & Kaufmann, arXiv:2604.07585 · Zatuchin, arXiv:2607.13304 · Aggarwal et al., GEO, arXiv:2311.09735 · Martinez, arXiv:2607.14035 · Lead Rescue · Evertune · Peec AI pricing · Profound pricing · Athena credit calculator · Otterly pricing · Ahrefs AI Visibility Index · Scrunch pricing · Brandlight · Testing ads in ChatGPT · Adobe completes Semrush acquisition