How we measure AI visibility: engines, samples, metrics and limits

RedClaw Labs measures AI visibility by asking a fixed set of buyer questions to ChatGPT, Gemini and Perplexity through their official APIs, repeating each question several times, and recording whether a brand is named, where it sits in the list, and which sites the answer cites.

This page is the source for that method. Every article we publish with measured numbers links back here instead of restating the method, so if you want to check our arithmetic, argue with it, or run the same thing yourself, this is the page to read. The commercial version of the pipeline is described on AI visibility tracking.

Which engines we query, and how

We use vendor APIs only. We do not drive the consumer chat apps, and we do not route through multi-model gateways, because gateways strip the citation metadata that makes half of these metrics possible.

Engine	Endpoint	Model pinned on 2026-09-20	Sources the API returns
ChatGPT	OpenAI Responses API with web search	gpt-5-mini	Search sources it consulted, plus URL citations attached to the answer text
Gemini	generateContent v1beta with Google Search grounding	gemini-2.5-flash	Grounding chunks with title and URL, after redirect resolution
Perplexity	Agent API, sonar preset	sonar	search_results entries with URL and title

All three adapters were verified against the live APIs on 2026-09-20. Model identifiers live in one configuration block rather than inline, because vendors retire models. A model change is a change of method, not a detail, so it goes in the version log at the bottom of this page.

The consulted / cited distinction matters and only ChatGPT reports both. An answer can read your page and still not cite it. We count those separately, as "consulted but not cited".

Where the questions come from

A measurement is only as meaningful as its question set, so the rules are fixed in code rather than left to judgement.

Questions are generated per topic as persona plus job plus constraint, in the market's own language. The generator is told to keep at least 80% of questions in recommendation or comparison intent, to allow at most one pricing question per topic, and never to write a specific company name. Its output then has to pass a gate before anything is asked of an engine:

Gate check	Rejects the set when
Buyer intent	Fewer than 70% of questions are recommend, compare or vendor_check
Definitions	More than one definition question in the set
Brand leakage	Any question other than a vendor_check contains the brand name or an alias
Length	Any question under 8 or over 200 characters
Near duplicates	Any pair whose character-set Jaccard similarity reaches 0.85

The generator gets three attempts in total, with the specific failure reasons from the gate fed back into each retry. If the third attempt still fails, the offending questions are dropped rather than the whole set.

The brand-leakage rule is the important one. If the question names you, the answer will name you, and the mention rate becomes a measurement of the question rather than of the market. vendor_check questions are the deliberate exception: they name the brand on purpose to see what the engine says about it, so they are excluded from the overall and per-engine rates and reported only on their own row.

For paid monitoring, the question set is agreed with the client before the first run, and questions carry an activation date so the set can grow without silently breaking the trend line.

Sample size and frequency

	Paid monitoring	Free check
Engines	ChatGPT, Gemini, Perplexity	ChatGPT only
Questions	The agreed set	3, generated for the brand
Repeats per question per engine	3	1
Frequency	Weekly	Once per account per month
API calls	Questions × engines × 3	7 (one generation, three answers, three extractions)

Three repeats is a cost decision, not a statistical one, and we say so plainly. It is enough to show that an answer varies between runs. It is not enough to call a small week-over-week move real. The confidence interval exists so nobody has to guess which is which.

A question only becomes trend-comparable once it has been live for at least four weekly runs, which the pipeline implements as an activation date 21 days or more before the current run date. Before that, the question appears in the report but not in the trend.

Metric definitions, as implemented

The denominator for every rate is valid answers: answers that neither errored nor refused. Refusals are counted and shown separately, so a low mention rate caused by an engine declining to recommend anyone does not get quietly folded into the same number as a low mention rate caused by absence.

Metric	How it is computed	Note
Mention rate	Valid answers containing the brand or an alias ÷ valid answers	Reported with a 95% Wilson interval
Average position	Mean of the 1-based index of the first list item naming the brand, over answers where a list position exists	null when no answer put the brand in a list
Citation rate	Valid answers citing at least one owned or satellite domain ÷ valid answers	Answer-level, not URL-level
Citation share	Our cited URLs ÷ all cited URLs across valid answers	URL-level
Share of voice	Answers mentioning us ÷ (answers mentioning us + total competitor mentions)	Asymmetric; see the limits below
Competitor mentions	Sum, over valid answers, of the accepted competitors named in each answer	Only competitors on the confirmed list
Cited sources	Domains appearing in citations, ranked by count, each classified as owned, satellite, competitor or third party	Top 50

Two details in the matching decide whether these numbers mean anything:

Link targets are stripped before matching. The answer text is cleaned of markdown link targets and bare URLs first, so a domain sitting inside a link does not count as the brand being named in prose.

Aliases are counted by maximum, not by sum. Latin aliases match on word boundaries, so "Otter" does not match "Otterly". CJK aliases cannot use word boundaries, so they match directly. When several aliases overlap on the same span, we take the largest single count rather than adding them up, which prevents one mention from being counted three times because you listed three spellings of your name.

Citation URLs are normalised before counting: utm_* parameters are removed so the same page counts once, and Google Maps entity links are dropped, because those are how an engine marks a business name rather than a source it read.

Why the mention rate comes with a range

Mention rate is a proportion from a small number of trials, and at small n the familiar textbook interval behaves badly. The Wilson score interval, introduced by E. B. Wilson in 1927, fixes the two failures that matter here: the normal-approximation interval "narrows to zero width (falsely implying certainty)" as the proportion approaches 0 or 1, and it can produce bounds outside 0 and 1 entirely. Wilson's interval "can be safely employed with small samples and skewed observations" (Wikipedia, Binomial proportion confidence interval, retrieved 2026-09-20).

We compute it at 95%, z = 1.96, in the algebraically equivalent centre-and-half-width form:

centre = (p + z² / 2n) / (1 + z² / n)
half   = z · √( p(1 − p)/n + z² / 4n² ) / (1 + z² / n)

What that looks like in practice, for the same observed 67%:

Mentions / valid answers	Point estimate	Wilson 95%
2 / 3	66.7%	20.8% to 93.9%
6 / 9	66.7%	35.4% to 87.9%
24 / 36	66.7%	50.3% to 79.8%
3 / 3	100%	43.8% to 100%
0 / 3	0%	0% to 56.2%

Read the bottom two rows before you read anything else in a report. Three mentions out of three is compatible with a true rate as low as 44%, and zero out of three is compatible with a true rate as high as 56%. This is why a single free check is a smoke test rather than a baseline, and why we publish intervals instead of round numbers.

The practical rule we use on our own reports: if this week's interval overlaps last week's, the change is not yet evidence of anything.

Known limits

We would rather list these than have someone else find them.

Personalisation is out of scope. We query APIs with no signed-in user, no chat history and no consumer-app context. The result describes what the engine says to an anonymous caller, which is not identical to what it says in any individual's session.

Google AI Overviews and AI Mode are not in our measurement scope. The monitored surfaces are ChatGPT, Gemini and Perplexity. We do not report on AI Overviews, and a number from us should never be read as covering it.

German and Japanese are partly verified. On 2026-09-20 we ran the pipeline end to end in both languages on ChatGPT: question generation produced idiomatic buyer questions with no brand leakage, brand matching hit 3 of 3, and competitor extraction returned sensible names. Gemini and Perplexity answers in those two languages have not been verified yet, because the free check path only runs ChatGPT. Long-running verified languages are Traditional Chinese and English.

Small samples give wide intervals. Nothing in the method removes this. A short question set measured weekly cannot detect a small change, and we will say a result is inconclusive rather than dress it up.

List position misses table-formatted answers. Position detection reads numbered lists, bullets, sub-headings and bold leads. When an engine answers with a comparison table, the brand is recorded as mentioned but contributes no position. Average position is therefore computed over a subset of the answers that mentioned you.

Share of voice is asymmetric. Our side counts one per answer; the competitor side sums every listed competitor in that answer. An answer naming you and five rivals scores 1/6, not 1/2. The figure is comparable across weeks for a fixed competitor list, and not comparable across brands with lists of different lengths.

Alias lists are a human input. A short or generic brand name produces false positives, and a missing spelling produces false negatives. We confirm the alias list with the client and treat a mention rate that never moves off zero as a matching bug until proven otherwise.

Model versions drift. The answers come from third-party models that vendors update and retire. A pinned model changing is a break in the series, and it is logged here rather than smoothed over.

Questions

How many times do you ask each question? Three times per question per engine on paid monitoring, once per question on the free check. AI answers vary between runs, so one answer is an anecdote.

Why do you use official APIs instead of the ChatGPT app? Because the API can be asked the same way every week, and because it returns the source URLs behind an answer. Numbers scraped from a consumer interface are not stable enough for us to put a trend line on.

Do you measure Google AI Overviews? No. AI Overviews and Google's AI Mode are not in our measurement scope. If that surface is what you need measured, we are not the right supplier for it today.

Why is the mention rate shown as a range? Because the point estimate on its own is misleading at these sample sizes. The Wilson 95% interval tells you how much of the number is signal and how much is the sample being small.

What counts as a mention? The brand name or a confirmed alias appearing in the answer prose, after link targets and bare URLs have been stripped out. A domain appearing only inside a link is recorded as a citation, not as a mention.

Why doesn't the question contain my brand name? Because a question that names you produces an answer that names you, every time. The only exception is the vendor_check question type, which names the brand deliberately and is kept out of the headline rates.

How long before a trend means anything? A question needs at least four weekly runs before it enters the trend. Before that, treat each week as a standalone measurement and compare the intervals, not the point estimates.

Version log

Date	Change
2026-09-20	First published. Engine adapters verified against the live ChatGPT, Gemini and Perplexity APIs. German and Japanese verified on ChatGPT for question generation, brand matching and competitor extraction; Gemini and Perplexity in those languages still unverified.

If something here is wrong or unclear, we would rather hear it than not. The service this method backs is described on AI visibility tracking, including what it does and does not cover.