How to A/B Test GEO: Proving Your AI Visibility Work Actually Worked

Summary

You cannot split traffic in an AI answer, so the usual A/B machinery does not apply to GEO. What works instead is a matched-pair design: pick two comparable groups of prompts, change the sources feeding one of them, hold the other constant, and compare the difference in inclusion rate over time rather than the raw before-and-after number. This article covers why AI answers resist normal testing, how to structure a test that survives non-determinism, how long to run it, and how to read the result without mistaking noise for a win.

The Direct Answer: How to A/B Test a GEO Change

The sequence is short enough to state up front, and the reasoning behind each step follows below.

• Pick one hypothesis, phrased so it can be wrong. Not “improve our AI visibility” but “adding a structured pricing comparison to our alternatives page will raise our inclusion rate on pricing-intent prompts.”
• Split your prompt set into a treatment group (the prompts the change should affect) and a holdout group (comparable prompts it should not affect). Twelve to twenty prompts per group is a workable minimum.
• Run a baseline for three weeks, sampling every prompt on every engine on a fixed schedule. You are measuring the natural volatility of each group, not just its average.
• Ship exactly one change, and only to the sources feeding the treatment group. Two changes at once make the result uninterpretable.
• Keep measuring for four to six weeks. Retrieval indexes and model behaviour do not update the day you publish.
• Compute the difference-in-differences: (treatment after − treatment before) − (holdout after − holdout before). That number is your effect.
• Decide in advance what size of effect would make you roll the change out more widely, and hold yourself to it after the fact.

Why AI Answers Break Classic A/B Testing

A conventional web test randomises at the user level. Half your visitors see variant A, half see B, and because assignment is random, everything else averages out. None of that infrastructure exists in AI search. You cannot show ChatGPT one version of your page and Gemini another. There is one version of the web, and every engine reads the same one.

Three properties make this harder than it first looks. The first is non-determinism: the same prompt, sent twice a minute apart, can return a different shortlist. Sampling temperature, retrieval variation and personalisation all contribute. A single observation is close to worthless as evidence.

The second is lag. When you publish or earn a new source, it has to be crawled, indexed and then actually retrieved for a given query before it can influence an answer. Depending on the engine and the domain, that is anywhere from a few days to well over a month. Teams routinely declare a change ineffective at day ten, when nothing had propagated yet.

The third is that the ground moves underneath you. Model versions ship, retrieval stacks change, competitors publish their own content. If your inclusion rate rises eight points during your test window, some of that may be your change and some may be a model update that lifted everyone in the category. Without a holdout you have no way to separate the two — and the temptation to claim the whole eight points is considerable.

Designing the Test: Matched Pairs and a Holdout

Since you cannot randomise users, randomise prompts instead. The unit of assignment becomes the prompt, not the visitor. Build a list of prompts you care about, then split it into two groups that are as alike as possible on the dimensions that predict volatility.

Match on intent type first: commercial comparison prompts behave very differently from definitional ones. Then match on your current baseline inclusion rate — a prompt where you appear 80 percent of the time has far less headroom than one where you appear 10 percent, so stacking all your low performers into the treatment group guarantees a flattering result that means nothing. Then match on engine coverage and on language, if you track more than one.

The practical method is to rank your prompts by baseline inclusion rate, pair them off from the top (1st with 2nd, 3rd with 4th, and so on), and assign one member of each pair to treatment at random. You end up with two groups whose distributions are close to identical by construction.

The holdout group matters more than people expect, and the instinct to skip it is strong — it feels like leaving value on the table. It is not. Without it you are measuring your change plus every other thing that happened in those six weeks, and you will not be able to tell them apart.

Establish a Baseline Before You Touch Anything

The baseline period has one job: telling you how much your metrics move when nothing is happening. That number sets the bar your effect has to clear.

Sample every prompt on every engine at a fixed cadence — daily is ideal, three times a week is acceptable — for at least three weeks. Record four things per run: whether you were named, your position in the list, the description attached to you, and which competitors appeared alongside you. Then compute the week-to-week standard deviation of inclusion rate for each group. If your inclusion rate swings by six points week to week with no intervention, a five-point improvement after your change is not a result. It is Tuesday.

This is also where most GEO measurement quietly falls apart, and the reason is mundane. Sampling sixteen prompts across six engines daily is roughly a hundred queries a day, every day, logged consistently, for two months. Done by hand it survives about ten days.

Reading the Results Without Fooling Yourself

Your headline number is the difference-in-differences. Suppose the treatment group went from 34 percent inclusion to 47 percent, a gain of 13 points. The holdout went from 31 percent to 36 percent, a gain of 5 points. Your estimated effect is 8 points, not 13. The other 5 points happened to everyone.

Compare that 8 against the volatility you measured in the baseline. A rough working rule: the effect should exceed roughly twice the week-to-week standard deviation of the treatment group before you treat it as real. With small prompt sets you are not going to reach conventional statistical significance, and pretending otherwise is worse than acknowledging it. Treat the result as directional evidence, and let a repeat test on a different prompt group confirm it.

Look at the engines separately as well as together. It is common for a change to move Perplexity and Copilot sharply while barely touching ChatGPT, because their retrieval behaviour differs. An averaged number hides that, and the per-engine breakdown is often the more actionable finding.

Finally, check the cited sources, not just the score. If your inclusion rate rose and the pages being cited are the ones you changed, the causal story holds together. If it rose and the citations point somewhere entirely unrelated, something else caused it and you have learned less than you think.

A Worked Example: Rewriting a Comparison Page

A B2B SaaS team suspects their alternatives page is too vague for models to lift from. The hypothesis: replacing prose with an explicit, fair comparison table covering pricing, integrations and ideal use case will raise inclusion on comparison-intent prompts.

They select 32 comparison prompts, pair them by baseline inclusion rate, and split 16 into treatment and 16 into holdout. Three weeks of baseline shows treatment averaging 29 percent inclusion with a week-to-week standard deviation of 4 points, and holdout at 31 percent with a similar spread. They rewrite only the pages that serve the treatment prompts, change nothing else, and wait five weeks.

Treatment finishes at 41 percent, holdout at 33 percent. Difference-in-differences: 12 minus 2, so a 10-point effect against a volatility bar of about 8. Directionally real, and the per-engine view shows most of the gain in Perplexity and Gemini with ChatGPT roughly flat. The citation log confirms the rewritten pages are the ones now being pulled. That is a result worth rolling out — and worth re-testing on the holdout prompts next quarter to confirm it replicates.

Where GEO Tests Usually Go Wrong

The failure modes repeat across teams.

• Changing several things at once. A page rewrite plus a new review push plus a Reddit thread produces a number you cannot attribute to anything. Sequence them.
• No holdout. Every model update and competitor move gets counted as your win, and eventually as your loss.
• Calling it too early. Four weeks post-change is the minimum; anything shorter mostly measures crawl lag.
• Prompt sets that are too small. Under about ten prompts per group, a single volatile prompt can swing the whole result.
• Unbalanced groups. Putting your worst-performing prompts in treatment manufactures a gain from regression to the mean alone.
• Measuring inclusion only. A change can leave inclusion flat while significantly improving how you are described or which competitors appear beside you — both of which affect whether the mention converts.
• Moving the goalposts. Decide what counts as success before you see the data, and write it down.

None of this makes GEO testing as clean as a landing-page experiment, and it never will be. But the alternative — shipping changes and checking a prompt by hand a fortnight later — is not measurement at all. It is a vibe with a spreadsheet attached. A matched-pair test with a holdout is more work up front and gives you something you can actually act on: a defensible estimate of whether the thing you did is worth doing again.