The GEO Directory · 18 November 2026 · Published
Your AI visibility score went up. Did anything improve?
Before you buy more tracking or celebrate a higher score, check whether you measured the same thing twice.
Same four mentions.
A much bigger percentage.
10 of 12 complete
4 observed mentions
6 of 12 complete
4 observed mentions
A founder opens a weekly report. AI visibility has risen from 40% to 67%. The tempting next move is to credit the new content, renew the tool and do more of what worked.
But four mentions out of ten completed answers is 40%. Four out of six is about 67%. A dashboard can rise without recording one additional mention.
This does not make AI visibility tracking useless. It makes the question behind the percentage important: which answers, collected under which conditions, are in this number? A score is useful when it supports a decision. It is not useful when a collection problem looks like a marketing win.
01 / Define the outcome
Four signals, four different jobs
Start by naming what you want to improve. These signals can move in different directions, and a single blended score hides why.
A practical measurement vocabulary, not a universal vendor standard
Brand mention
- What you observed
- The answer names your brand or a verified alias.
- What it does not prove
- That the answer recommends you. A warning also mentions you.
- Useful next action
- Read how the brand is described.
Recommendation
- What you observed
- The answer presents your product as a suitable option for the stated need.
- What it does not prove
- That your site is linked, or a buyer followed the advice.
- Useful next action
- Check fit, qualifications and factual accuracy.
Link to your site
- What you observed
- A returned source or citation links to a URL on your domain.
- What it does not prove
- That the answer recommends your product. Your article might support a point about a competitor.
- Useful next action
- Inspect which page and claim the link supports.
Referral visit
- What you observed
- Analytics records a visit attributed to an AI source.
- What it does not prove
- That you know the exact prompt, every prior touchpoint, or incremental revenue.
- Useful next action
- Check landing pages and downstream actions.
OpenAI documents that web-search responses may contain citations, and cautions that sources can be incomplete or wrong. A link needs checking; it is not an endorsement certificate. [2]
A report about unaided discovery also needs to keep branded questions separate. Asking an assistant whether your product is good gives it your name. That can be a useful reputation check, but it is not the same test as asking which product to buy without naming yours.
02 / Work the numbers
The false win you can reproduce
Imagine a small, fixed panel of twelve questions on one named product surface, with one planned check per question in each period. This is deliberately small enough to inspect, not a recommended sample size for estimating a market.
In Period A, ten checks return usable answers. Four mention the brand. In Period B, only six return usable answers, including the same four mentioning it. Four questions that previously returned non-mentions are now missing. Two checks are missing in both periods.
The percentage increase is real arithmetic. The claim that the brand improved does not follow from it. In this example, excluding the newly missing checks explains the increase. We do not know what those checks would have returned.
Now compare only the six questions completed in both periods. The result is four mentions out of six in each: about 67% versus 67%. That is a helpful diagnostic, but it still leaves half of the planned panel out.
You can also state how much the missing answers leave unresolved. In A, the twelve planned checks could contain between four and six mentions, depending on the two missing outcomes: 33% to 50%. In B, the range is four to ten: 33% to 83%. These are missing-outcome bounds for this invented panel, not statistical confidence intervals and not estimates of all AI answers.
The ranges overlap. More importantly, the later measurement is less complete. The responsible conclusion is not that performance fell or rose. It is that this collection cannot establish improvement.
03 / Check comparability
Freeze the measurement, not your business
Before the next report, write a short measurement contract. It can fit in a spreadsheet header:
Decision: whether our new comparison page is appearing in a defined set of buyer answers.
Panel: stable question IDs and exact wording; buyer intent, language and market; branded and unbranded groups kept separate.
Collection: named engine and product surface, API or consumer interface, visible search settings, dates, time zone, repeat rule and retry policy.
Scoring: mention, suitable recommendation and owned-domain link recorded separately; unknown and failed checks retained.
Changes: version any question, market, collection or classification change before joining periods into a trend.
The surface matters. An API answer and a consumer chat answer are not automatically the same observation. OpenAI says location and relevant saved memories can affect search-query rewriting. Google says AI Mode and AI Overviews may use different models and techniques, so their answers and links can differ. Record the conditions you can observe and label the ones you cannot. [2] [1]
Keep a stable core panel for trend checks and a separate discovery panel for new buyer questions. Your market can change without making every week a new, incomparable test. Report the two panels separately until you have a comparable baseline.
Repeat collection according to a rule set before seeing results. Do not rerun only the answers you dislike. Extra runs reveal variation, but closely related questions and repeated model outputs are not automatically independent observations. A handful of repeats cannot justify a precise market-wide confidence claim.
04 / Inspect the evidence
Ask for an export before another subscription
You do not need a second dashboard to discover whether the first can explain its own number. Ask for answer-level evidence for the report you already have.
Recalculate one headline metric
Request stable question IDs, exact prompts, period, engine and surface, market, timestamp, completion status, full answer, returned links and scoring labels. Preserve attempts and retries separately, but use the predeclared rule to decide which attempt enters the score. A retry is not a new buyer.
Read the failures, not only the winners
Separate an answer that omits your brand from a timeout, a truncated response, a refusal and missing citation metadata. Missing metadata does not prove that the original answer had no citations. A zero is a finding; an unknown is an evidence gap.
Recode before looking at the label
Read a small, varied set of raw answers without the dashboard verdict. Include one apparent win, one omission and one ambiguous case. Check entity aliases, negative mentions and whether a recommendation actually fits the buyer. This is a spot-check for errors, not validation of the whole dataset.
Make the tool explain a disagreement
A different score may come from different questions, collection, entity matching, weighting or missingness. Find the layer where the disagreement starts. Do not pick the higher score because it is more encouraging.
These principles are not new inventions. AIBL's proof-of-method guide and Llamat's checklist already make detailed cases for inspecting evidence, failures and collection methods. The worksheet here makes one failure mode visible row by row, so you can reproduce it rather than take another checklist on trust. Both publishers sell tools; apply the same questions to them. [5] [6]
KEEP THE EVIDENCE CONNECTED
Same buyer + market
Or explicit missing
Link ≠ visit
05 / Decide what to do
Fix the measurement, then choose the intervention
What the evidence supports next
Completion changed
- First check
- Failures by topic, engine and market; retry policy; collection changes.
- Decision boundary
- Repair collection before crediting content or changing strategy.
Comparable panel improved
- First check
- Raw answers, recommendation quality, variation and any model or classification change.
- Decision boundary
- Report a bounded improvement in the tested panel, not market dominance or causality.
Links rose, visits did not
- First check
- The cited URLs, supported claims, referral identification and landing pages.
- Decision boundary
- Do not call citations traffic, or assume the panel represents all users.
Visits rose, conversions did not
- First check
- Tracking, landing-page fit, signup friction and the definition of a qualified lead.
- Decision boundary
- Do not buy more visibility to solve an unexamined conversion problem.
Different first-party reports also answer different questions. Bing's AI Performance documentation describes citations across supported Microsoft and partner surfaces; it says grounding queries are sampled and citations do not indicate placement. Google now documents a dedicated Generative AI performance report for AI Overviews and AI Mode impressions, also included in overall Web performance; the report may not appear when a site has insufficient impressions. Neither report is a universal census of every assistant conversation. [3] [11]
Referral analytics is another layer. Plausible, for example, describes its AI channel as visits and on-site outcomes, rather than mentions in a sampled prompt panel. That makes it useful for a different decision, not inherently a better measure of the same thing. Attribution is limited to what the implementation can identify and record. [4]
Do not divide analytics visits by mentions in your test panel to invent an AI click-through rate. The visits and sampled answers are not the same population. Likewise, a better matched-panel score after a content edit is an association, not proof that your edit caused the change.
06 / Report it honestly
A useful update can be four sentences
We observed four brand mentions in each period. Collection completion fell from ten of twelve checks to six of twelve, so mention coverage among completed answers rose from 40% to 67%. On the six checks completed in both periods, coverage was unchanged at four of six. We cannot claim improvement from this report; we will investigate the missing checks and keep referral visits and conversions in a separate report.
That is less exciting than a large green arrow. It is more useful to the person deciding where the next month of effort goes.
When choosing a tool, pay for coverage and workflow you can explain: the buyer questions you care about, the surfaces and markets you need, usable raw evidence, visible failure handling and an export you can recalculate. Engine count and a polished scorecard are not substitutes.
The buying rule is simple: if a vendor cannot show what changed underneath its score, do not use that score to justify more spend. Keep the tool for narrower tasks if it genuinely helps, but narrow the claim with it.
Method and sources
What this article did, and did not test
This is a proposed audit method with a fully invented teaching dataset. No AI visibility platform was performance-tested, no customer results are claimed, and no live prompts were run for the example. Twelve rows make the arithmetic inspectable; they are not a recommended research sample size.
The linked r/SEO discussion already gives an invented denominator-change example and proposes separate mention/completion/failure counts, comparison of common answered questions, failures by topic, distinct mention/citation/recommendation fields and dated raw answers. Those ideas are prior work, not a new method introduced here. This article develops a different twelve-row teaching example, explicit missing-outcome bounds, a CSV and a reporting template to make the problem reproducible. [7]
This article recommends no paid product and makes no ranking claim. Product documentation and reporting details linked in this article were read on 18 November 2026; they can change.
- Google Search Central: AI features and your website
Requirements, search-surface differences and Search Console reporting.
- OpenAI: Searching the web with ChatGPT
Search, citation links, location and memory context.
- Bing: Introducing AI Performance
Publisher citation metrics, supported surfaces and sampled grounding queries.
- Plausible: AI traffic analytics
Referral visits and conversions versus sampled answer visibility. Analytics vendor, not an independent accuracy benchmark.
- AIBL: How to evaluate AI visibility tools
Prior work on frozen trials, failures, evidence exports and comparable denominators. Tool vendor.
- Llamat: AI visibility tool evaluation checklist
Prior work on collection methods, repetition and evidence. Tool vendor.
- r/SEO: Separating improvement from sample change
Prior practitioner example of denominator drift, separate completion and mention counts, common answered questions, failures by topic and distinct outcome fields. Not a prevalence study.
- Obi: product overview and FAQ
Vendor description of software-startup buyer questions and approval-gated website drafts.
- AirOps: content engineering teams
Vendor description of content operations, governance and workflow roles.
- Profound: enterprise capabilities
Vendor description of enterprise controls including SSO and role-based access.
- Google Search Console: Generative AI performance report
Current dedicated AI Overviews and AI Mode impression reporting, availability threshold and relationship to Web reporting.
The GEO Directory. All percentages in the worked example are calculated from invented records, not market observations.
Invented data · 12 planned checks per period
The complete audit worksheet
Every number in the article can be rebuilt from these rows. Missing means the outcome is unknown, not that the brand was absent.
Full ledger
One row per planned check
View and copy the invented CSV
question_id,period_a_status,period_a_mention,period_b_status,period_b_mention Q01,complete,yes,complete,yes Q02,complete,yes,complete,yes Q03,complete,yes,complete,yes Q04,complete,yes,complete,yes Q05,complete,no,complete,no Q06,complete,no,complete,no Q07,complete,no,missing,unknown Q08,complete,no,missing,unknown Q09,complete,no,missing,unknown Q10,complete,no,missing,unknown Q11,missing,unknown,missing,unknown Q12,missing,unknown,missing,unknown
No live measurements, brand names or vendor results are in this file. One check per question per period, on one hypothetical surface.
Recalculate
Counts before percentages
Same twelve planned question IDs in both periods
Period A
- Planned
- 12
- Completed
- 10
- Missing
- 2
- Mentions observed
- 4
- Confirmed non-mentions
- 6
- Completion
- 10/12 = 83.3%
- Mention rate among completed
- 4/10 = 40.0%
- Common completed subset
- Q01-Q06: 4/6 = 66.7%
- Possible mentions in planned panel
- 4 to 6 of 12 = 33.3%-50.0%
Period B
- Planned
- 12
- Completed
- 6
- Missing
- 6
- Mentions observed
- 4
- Confirmed non-mentions
- 2
- Completion
- 6/12 = 50.0%
- Mention rate among completed
- 4/6 = 66.7%
- Common completed subset
- Q01-Q06: 4/6 = 66.7%
- Possible mentions in planned panel
- 4 to 10 of 12 = 33.3%-83.3%
Completion = completed checks / planned checks.
Mention rate among completed = answers mentioning the brand / completed usable answers. This definition applies only to this example; check the vendor's actual metric.
Missing-outcome bounds = observed mentions / planned checks, up to (observed mentions + missing outcomes) / planned checks. This asks what the fixed panel could contain if missing outcomes were all non-mentions or all mentions. It does not infer the missing answers or account for model variation.
Use with your own data
Keep an evidence trail
Extend the CSV with prompt text and version, intent, brand aliases, engine and surface, market, language, timestamp and time zone, attempt ID, status, retry relationship, raw-answer reference, recommendation label, owned citation URL, reviewer and scoring-rule version. Use an explicit unknown value when evidence is missing. Do not fill blanks with zeros.
Keep a record for every planned check. Decide how retries enter the score before collecting. If you change the panel or collection method, mark the break and report comparable subsets alongside completion, not instead of it.
Use this to audit arithmetic and missingness. It does not validate a vendor, establish causal impact or replace a measurement plan suited to your decision.