Portrait of Dmytro Nasyrov wearing a dark suit and light blue shirt against a dark background.

Production Architecture with Dmytro Nasyrov

Archives
Software development
About Dmytro Nasyrov
Log in
Subscribe
November 6, 2026

AI Visibility: Keep Failed Checks Out of Citation Rates

Four observation outcomes and explicit denominators separate collection failures, overview presence and target citations in a hypothetical 100-check example.

A citation percentage can fall because a website lost source inclusion. It can also fall because the collector stopped returning usable results. Those situations require different owners, different evidence and different next steps. Combining them in one visibility score makes a report harder to act on precisely when something changes.

Start with the observation behind the percentage. Did the result load? Was an AI Overview present? Did its source links include the target domain? A failed collection cannot answer the last two questions. Recording both as negative turns a monitoring problem into an apparent search-performance problem.

This article develops the measurement distinction in TopClick's AI Overview tracking guide, published November 1, 2026 by Dmytro Nasyrov. Its worked example is hypothetical: 100 scheduled checks, 20 failed collections, 80 valid observations, 40 results containing an overview and 12 citing the target domain. These are teaching numbers, not measurements of Google results or a client campaign.

The practical deliverable is a proposed observation ledger and a reporting decision: publish each rate with its population, retain failed checks in the collection-quality account and investigate coverage before interpreting movement as a visibility change. No live monitoring run or supplier acceptance exercise was performed for this article.

Separate four outcomes before calculating a rate

Treat one scheduled query, in one specified context, at one intended observation time as a check. The check may have several collection attempts, but its intended place in the sample stays identifiable. This distinction becomes important when requests time out or an operator repeats a check manually.

A minimal classification has four mutually exclusive terminal outcomes for the purposes of this example. Unknown and pending work must remain visible until classified; they should not silently enter one of the completed categories.

Outcome Collection usable? Overview present? Target domain cited?
Collection failed No Unknown Unknown
Valid result without overview Yes No Not applicable
Overview without target citation Yes Yes No
Overview with target citation Yes Yes Yes

The second row is especially easy to misclassify. A properly loaded search result with no overview is a valid observation of feature absence. A timeout before the result can be inspected establishes no such absence. Both can leave a citation cell empty, which is why an empty cell is an inadequate data model.

The final two rows describe inclusion within an observed feature. They support a conditional question: when an overview appeared in a usable result, how often did it cite the target? They do not establish how frequently the feature appeared across the panel, how many people saw the answer or how many visits followed.

A collector should state what makes a result usable. Receiving an HTTP response alone may be insufficient if rendering or source extraction remains incomplete. Conversely, the absence of an overview should not become an error merely because an extraction routine expected a panel. Retain the evidence needed to review that classification.

Give each of four percentages its own denominator

Use explicit symbols in the report specification. Let S represent scheduled checks, F failed collections, V valid observations, A valid observations containing an overview and C observations whose overview cites the target domain. For the simplified completed sample, S equals F plus V, while C is no greater than A and A is no greater than V.

These are counting relationships within this proposed model. A real pipeline may also have pending, canceled or excluded checks. If those states exist, extend the reconciliation rather than forcing them into failure or validity to make an equation balance.

Metric Formula Question answered
Collection failure rate F / S What share of scheduled checks failed collection?
Overview trigger rate A / V How often did usable results contain the feature?
Conditional citation rate C / A When the feature appeared, how often was the target cited?
Citation coverage C / V What share of usable results contained a target citation?

Write the formula beside the label, especially when exchanging exports between tools. The term citation rate does not guarantee that two products use the same population. A metric name without a denominator leaves a critical part of the measurement contract unspecified.

Do not display a numerical zero for a ratio with an empty denominator. If there are no valid observations, the trigger rate and citation coverage are unavailable. If no valid result contains an overview, the conditional citation rate is unavailable. Reporting those cases as zero claims a completed negative measurement where the defined conditional population does not exist.

Store counts alongside percentages. Twelve positive observations out of forty eligible overviews carries different information from three hundred positives out of one thousand, although both produce the same percentage. Neither result automatically represents the audience or all searches in a market.

Work through the source example

In TopClick's hypothetical example, 20 failures leave 80 valid observations from 100 scheduled checks. Forty valid results contain an overview. Twelve of those overviews cite the target domain. The four outcomes therefore contain 20 failures, 40 valid results without an overview, 28 overviews without a target citation and 12 citation-positive observations.

The failure rate is 20/100, or 20%. The trigger rate is 40/80, or 50%. Conditional citation rate is 12/40, or 30%. Citation coverage is 12/80, or 15%. Each calculation describes a different part of the same sample.

Dividing twelve citations by all one hundred scheduled checks produces 12%. That can be retained as a separately named operational yield if there is a useful business reason to track it. It cannot replace citation coverage across valid observations without changing the question. Its denominator includes twenty outcomes for which feature presence and citation status are unknown.

The relationship between trigger rate and conditional citation rate also helps explain the result. For this consistently defined sample, 50% multiplied by 30% equals 15% citation coverage. This is arithmetic on the same eligible checks, not a causal account of how Google selects sources.

Before using the identity in an actual report, verify that the component metrics share the same panel, period, market and counting unit. A trigger rate based on unique keywords cannot be multiplied meaningfully by a citation rate based on repeated daily observations and presented as the same measure.

Preserve the scheduled check and every collection attempt

Give the scheduled check a stable identity. Record its panel revision, query text, requested location, query language, device, search surface and planned observation window. Store actual collection time separately. An intended morning observation collected much later needs a visible timing qualification.

Keep collection attempts below that identity. A first timeout followed by a successful retry should preserve both attempt records. The reporting rule then selects the check's accepted observation according to a declared policy. It should not turn one intended check into two independent opportunities to obtain a favorable citation.

For example, a proposed record CH-17 may have attempts T1 and T2. T1 timed out with citation status unknown. T2 completed later with a saved result and a target citation. Retain the failure in attempt-level reliability reporting while showing how CH-17 contributes to the check-level sample under the chosen replacement rule.

That example is a proposed design, not a description of an implemented TopClick collector. It exposes a decision a team or supplier must make: whether a late retry replaces an unresolved check, becomes a separately scheduled observation or stays outside the comparison window. Each option changes interpretation and needs a stable policy.

Never overwrite the original timestamp or failed evidence reference simply because the retry succeeded. Historical collection quality and the timing of the accepted observation remain relevant to anyone investigating a later change. A clean final percentage should not erase the path by which its rows became eligible.

Make the observation ledger reviewable

The source guide describes thirteen worksheet fields covering query context, status, citation evidence and panel revision. For this article, add explicit check identity, attempt identity and classification policy version when designing a pipeline. These additions are proposed controls, rather than claims that the source worksheet already implements them.

A useful review starts from the published metric, follows its included check IDs and opens the stored evidence for a selected observation. The reviewer should be able to see why collection was accepted, whether the feature was present and which source URL established the citation. A screenshot and an extracted URL list answer complementary questions.

An example decision record can stay compact:

Field Proposed entry for CH-17
Check context Panel P3, one query, declared market and device
Initial attempt T1, timeout, overview and citation unknown
Accepted attempt T2, later timestamp, evidence E17 retained
Classification Valid overview with target citation, policy K2
Reporting disposition Include once if the declared time-window rule permits
Unresolved question Does the late collection remain comparable to the baseline?

The unresolved question prevents the example from granting itself acceptance. A complete-looking row is not sufficient when its timing falls outside the declared observation contract. Assign an owner to resolve that question before using the record in a period comparison.

Evidence retention also needs a practical access rule. A report that depends on an expired screenshot link or a source record unavailable to the receiving team cannot support later investigation. Agree who can retrieve the evidence, how long it remains available and what happens when a supplier relationship ends.

Excluding failures does not remove missing-data bias

Removing failed checks from citation denominators prevents a known category error. It does not prove that the remaining observations resemble the missing ones. If a collector fails disproportionately on one device, market or query class, its surviving sample can change composition even when the scheduled panel stays fixed.

Consider a separate fictional diagnostic scenario. One collection period contains usable results across two declared market segments. In the next period, collection fails mostly in one segment. The combined citation percentage may rise because the surviving segment has higher inclusion, even if neither segment improved on its own comparable observations.

The appropriate response is to expose coverage by segment and inspect the failure pattern. Do not label the combined rise an optimization result. A weighted estimate would require an explicit sampling and missingness argument; silently assigning the missing rows the behavior of successful rows supplies neither.

TopClick's source guide states the limit directly:

If failures cluster around particular devices or query types, simply excluding them does not make the remaining sample representative.

TopClick, AI Overview Tracking: How to Monitor Citations and Visibility.

Show scheduled, failed and valid counts next to segment-level rates. Where the missingness prevents a useful comparison, report that limitation and assign collection recovery before recommending content changes. An honest unavailable comparison is more actionable than a precise percentage with an undisclosed population shift.

Compare the same panel before interpreting movement

Version the query panel whenever membership changes. A newly added product category, country or group of branded searches can change the combined result without any change to an existing page. Preserve entry dates and reasons so the reporting population can be reconstructed.

When expanding a panel, retain a comparison for the unchanged subset as well as the larger current sample. Label both populations. The larger panel can support new decisions, while the matched subset helps readers understand whether an apparent change depends on the added queries.

Keep branded and unbranded intent visible. A panel dominated by questions explicitly naming the company describes a different task from a panel asking which suppliers meet a category requirement. Both may matter commercially, but changing their proportions should not silently redefine the headline metric.

Search context belongs in that comparison contract too. Google Search AI Overviews, AI Mode and a standalone assistant response are different surfaces. A result collected from another surface should not fill an unresolved AI Overview observation. Likewise, a different device or location deserves a recorded distinction rather than an implied match.

Choose a repeatable collection and review cadence that the team can maintain. The article does not prescribe a universal number of prompts, a statistical confidence claim or a Google-required frequency. Such choices need the actual decision, sample design and operating constraints, which are not available in this hypothetical case.

Separate a domain citation from a brand mention

A source link to the target domain and a brand name in answer text are different observable events. An overview may cite a technical page without naming its publisher. It may name a company while citing a third-party comparison. A report should preserve those distinctions instead of treating either event as interchangeable visibility credit.

Define how the target domain set is matched. Decide whether subdomains count, retain the original cited URL and record any normalized form separately. If a source link redirects, distinguish the cited address from its eventual destination. A migration can otherwise appear to move citations between domains merely because the matching rule changed.

For the per-check inclusion measures in this article, several links from the same target domain still create one citation-positive observation. Keep a separate link count when useful. Counting every source link as another positive check can push the numerator away from the population its denominator describes.

Preserve the classifier version and any corrections. If a later review changes one row from target absent to target present, record the reason and evidence rather than quietly editing the historical total. A corrected report should identify which classification changed and which previously published number it replaces.

This discipline also helps resolve supplier disagreements. Before arguing about a percentage, compare the target-domain rules, saved answers, collection times and counting unit. A disagreement in those definitions may explain the difference without establishing that either system observed the same result incorrectly.

Keep observation, exposure and business outcome separate

The rates here describe a selected set of recorded checks. They do not count people who saw a result. They do not count clicks, visits, qualified leads or revenue. A citation can be valuable while those other quantities remain unmeasured, but the reporting language must preserve the distinction.

Use analytics and platform reporting as separate evidence streams with their own definitions. This article does not reproduce a current Search Console feature inventory or claim that a standard organic traffic segment isolates AI Overview clicks. Check the applicable primary documentation before adding those claims to a specific implementation brief.

A practical reporting meeting can ask three connected questions: what did the observation panel record, what exposure evidence is available and what happened after visitors reached the site? A change in one stream suggests an investigation across the others; it does not establish the causal connection by itself.

For the same reason, resist converting a higher conditional citation rate directly into a performance promise. The rate may have changed alongside feature trigger frequency, query mix, collection coverage or unrelated site changes. A commercial conclusion needs evidence beyond the arithmetic of an observation sample.

Turn the report into an owned next action

Start the weekly review with collection coverage. If failed checks rose, name the collector owner and the affected contexts. Decide which observations can be recovered under the declared time rule and which comparisons must remain qualified. Do this before opening a content rewrite task based on the headline score.

When valid results show fewer overviews, investigate feature appearance within the same panel. When overviews still appear but target citations disappear repeatedly, inspect the retained answers and newly cited URLs. The action should follow the observed boundary: collector recovery, measurement correction or a source-page investigation.

A useful handover names the observation, evidence, proposed action, responsible owner and next review date. It also names what would count as completion. Repairing a collector is complete under its collection acceptance criteria; it does not prove that a page gained citations. Updating a page is a delivered change; a later observation is still needed to assess inclusion.

That separation is the relevant connection to TopClick's AI visibility monitoring scope: define the panel and reporting context, retain evidence and convert observations into reviewable work. The proposed CH-17 ledger here is an editorial design for that decision, not a client implementation or a measured service result.

Before accepting the next report, pick one displayed percentage and trace it back to the eligible checks. Confirm that failures remain visible, valid absences remain valid and the target citation has retrievable evidence. If that chain cannot be followed, resolve the measurement contract before treating movement as an instruction to change the website.

An acceptance exercise for the reporting contract

Before relying on a new dashboard, propose a small set of deliberately distinguishable records and ask the supplier to explain their treatment. Include a failed request, a usable result without an overview, an overview citing another domain and an overview citing the target. The expected classifications should reproduce the four outcomes before any percentage is displayed.

Add a duplicate delivery of a successful result. The check-level numerator should follow the agreed identity policy instead of rising simply because the same evidence arrived twice. Then add a successful late retry after a timeout. Ask which observation window receives it, whether it replaces the unresolved check and where the initial attempt failure remains visible.

An empty sample deserves its own example. A panel with no valid results should display unavailable visibility rates alongside its collection counts. A panel with valid results but no overviews can support a zero trigger rate and zero citation coverage, while its conditional citation rate remains unavailable. These outcomes distinguish feature absence from an undefined conditional population.

Finally, change the target-domain mapping or panel revision. The reporting system should reveal the changed definition and retain the earlier evidence. A comparison that crosses that boundary should be labeled accordingly, or rebuilt on an explicitly comparable subset. The acceptance question is whether another reviewer can reproduce the inclusion decisions from the retained records.

These are proposed exercises, not tests executed against Google, a monitoring vendor or a TopClick production service. Their purpose is to make the purchasing and handover conversation concrete. The receiving team can identify the required behavior, assign the observation and collection owners and request the actual results before accepting the reporting workflow.

Decide when the comparison must wait

Define a hold decision for reporting, rather than forcing every scheduled review to produce a directional conclusion. The owner can still deliver the observation counts and explain what is missing. Holding an interpretation does not mean hiding the collected data or waiting indefinitely for a perfect sample.

A proposed review card should name the affected panel revision, the missing contexts, the metric whose interpretation is restricted and the evidence needed to lift the restriction. It can also name a smaller, clearly labeled subset that remains usable. Avoid presenting that subset as the original full panel once its eligibility has changed.

The release condition should be specific enough for another person to inspect. Recovering an inaccessible evidence reference, resolving a duplicate classification or establishing the disposition of late observations are different tasks. Each needs the responsible owner and a recorded outcome. A new reporting date alone does not resolve the underlying question.

Where the team accepts a qualified comparison, preserve the qualification with the exported result. A limitation in a meeting note can disappear when a percentage is copied into a management slide. Put the panel, counts, denominator and qualification beside the number so the next reader receives the same measurement boundary.

This gives the reporting owner a practical choice: release a reproducible, qualified comparison, release counts without a directional claim or hold the interpretation pending named evidence. None of those choices requires inventing what the failed checks would have shown.

More insights to read

  • AI Overview Tracking: How to Monitor Citations and Visibility
  • What is Brand Radar and how to use it
  • SEO audit vs retainer
  • SEO, GEO and AEO: defining the scope of work
  • AI features and your website

About the author

Portrait of Dmytro Nasyrov wearing a dark suit and light blue shirt against a dark background.

Dmytro Nasyrov. Photo supplied by the author.

Written by Dmytro Nasyrov PhD, software architect with 24 years of production experience. Dmytro is the CEO of TopClick. His writing covers SEO, AI search and the evidence behind measurement decisions.

Don't miss what's next. Subscribe to Production Architecture with Dmytro Nasyrov:
Older → Feature Flags: Give Temporary Flags an Expiry Decision
Software development
LinkedIn
Twitter
YouTube
Powered by Buttondown, the easiest way to start and grow your newsletter.