The Dashboard Was Reading the Wrong Artifact

We ran an audit across a network of sites and it came back clean. Then we changed one thing about how the audit worked — what it read — and the same network came back with a gap 3,226 links wide.

Nothing about the sites changed between those two runs. Only the artifact being measured.

This is a short post about a specific and correctable error that I now believe is close to universal in automated quality work, and especially in anything checking machine-generated output at volume.

What happened

The audit read source files. That is the natural thing to do: the source is where the content lives, it is structured, it is easy to parse, and it is the thing your team edits. Every field the audit cared about was present and correct in the source.

The site is built from those source files by a generator. Somewhere between the two, some of that data was never rendered.

Concretely, three failures had been sitting in production, invisible:

• Related links that no template read. Across two sites, 1,277 internal links were declared in source, correctly formatted, pointing at real pages. The validation schema accepted the field, so nothing complained. No template consumed it. Every one of those links existed only in the repository.
• Structured data that was never fed. The markup block for article schema was present in the layout on multiple sites. It was never given data. On one site that covered 447 posts. On another, the parent site of the group, there was no article schema at all.
• Anchor text that was machine-shaped. On a third site, 1,318 internal links rendered with the raw URL slug as their visible text — verification-that-cannot-fail where a human would have written the title. Present. Rendered. Counted by any link audit. Useless as a relevance signal, and visibly wrong to a reader.

Every one of these passes a source-level check. Not one of them survives contact with the built page.

Why this is the default failure, not an unusual one

The reason this is so common is that measuring the source is easier at every stage, and the difference only shows up in cases nobody constructed a test for.

Source is structured; built output is a soup of markup. Source is available before you build; built output requires a build. Source is what the team edits, so it is what the team thinks of as the system. And crucially, when source and output agree — which is almost always — the source-level check is correct, so it accrues a long history of being right.

It is right until the day the two diverge, which is exactly the day you needed a check.

The generalised version:

Measure the artifact the consumer receives, not the one the producer authored.

For a website, that is the rendered HTML, not the content file. For an API, the serialised response, not the internal object. For a document pipeline, the exported PDF, not the template. For a data pipeline, the table the analyst queries, not the transformation that wrote it. For a model-serving system, the response after every filter and post-processor, not the raw completion.

Each of those pairs is usually identical. Each is a place I have personally seen them not be.

The AI-specific version of this problem

This matters more than it used to, and here is the mechanism.

When output is generated by a model at volume, nobody reads it all. That is the point — it is the reason for the volume. What replaces reading is a layer of automated checks, and the entire quality position of the system rests on those checks being right.

So the checks become the product. And a check that reads the wrong artifact does not merely fail to catch things: it actively manufactures confidence. Everyone downstream reasonably assumes verification happened. It did. It verified the wrong object, and reported green.

An absent check leaves you appropriately nervous. A green check that measured the wrong thing spends confidence you never earned, on decisions taken by people who had no way to know.

This is the same failure I have written about with quality gates that cannot go red. The shape recurs because it comes from the same instinct: build the check where the data is convenient rather than where the truth is.

What we changed

Three things, each cheap.

The audit now builds the site and reads the output. This is slower. It is the entire fix. Everything else in this post follows from it.

The measurement is expressed in terms of the artifact. Not “221 posts declare related links” but “606 anchors rendered across 219 pages, 0 dead, 0 missing a trailing slash”. The first sentence was true throughout the period when the answer was zero. The second could not have been.

The check has a floor it can fail against. We set a minimum of three incoming references and three outgoing links per post, and made falling below it a failure rather than an observation. A measurement with no threshold is a number on a dashboard; a threshold is what turns it into a control. It failed immediately, on real pages, which is how you know it works.

The correction to the baseline was the uncomfortable part. The previous number was not slightly wrong. It could not see 3,226 links, because it was looking at a layer where they did not exist. Every report built on it had been confidently describing a different system than the one that was live.

The question worth asking about your own dashboards

Pick the metric your team trusts most — the one that gets quoted in a status update without anyone asking how it is derived.

Then ask: at what point in the pipeline is it measured, and how many transformations sit between there and what the customer receives?

If the answer is “several”, you have an untested assumption that all of them are lossless. That assumption may well be correct. It is worth ten minutes to find out, because the failure mode is not a wrong number on a dashboard. It is a right number, about the wrong thing, believed by everyone.

Frequently asked questions

What does it mean to measure the built artifact? Running the build or serialisation your production system performs, and then measuring the output of that process rather than the inputs to it. For a website that means the rendered HTML; for an API, the actual response body.

Why not just check both source and output? You can, and for fast feedback during development a source check is useful. The point is that only the output check can be authoritative, so it must be the one that gates release.

Is this not just integration testing? It is closely related. The distinction worth drawing is that integration tests usually verify behaviour for a handful of constructed cases, whereas this is a census over everything you actually ship — which is what catches the 447-page and 1,318-link failures.

How do you stop a check from silently measuring the wrong thing? Give it a case it must fail. If the check cannot be made to go red by a deliberately broken input, it is not verifying anything, and no green result from it means anything.

Does this slow the pipeline down? Yes, because you have to build before you can measure. In our experience that cost is trivial next to the cost of shipping months of work that was never rendered.