Verification That Cannot Fail Is Not Verification

Last week I rebuilt the artwork pipeline behind a five-site publishing estate: roughly 2,200 articles, each needing a hero image and a social card. Unglamorous work, largely automated, the sort of task you delegate and forget.

What made it worth writing about is that almost every mistake I made was caught late — and always for the same reason. Something had already told me it worked.

This matters more now than it did three years ago. When a team writes code by hand, the bottleneck is production, and verification is a tax you pay at the end. When code is generated quickly — by an agent, a template, a batch script — production stops being the constraint and verification becomes the entire job. The volume of plausible-looking output rises, and your capacity to check it does not.

Under those conditions, a check that always passes is not neutral. It is a liability. No check leaves you appropriately uncertain; you proceed carefully. A check that cannot fail sells you confidence you have not earned, and you spend that confidence downstream on decisions that assume the ground is solid.

Five examples, all from one week.

The write succeeded. The file was in the wrong place.

A batch script resolved its output directory from an environment variable with a fallback chain, and returned the first candidate that existed on disk.

With the variable unset, an empty path in most languages normalises to “the current directory” — which always exists. So the first candidate always won, and the run wrote 369 images into a directory tree that no other part of the system would ever read.

It reported 369 written, zero failed, and every word of that was true. The writes succeeded. The files were valid. They were simply nowhere useful.

No amount of error handling catches this, because nothing errored. The run asked did the write succeed? The question that mattered was did the write land where the rest of the system will look? Those are different questions, and only the second one has business meaning.

The fix is not defensive coding. It is to confirm a location by finding something that must already be inside it, rather than accepting that the path resolves.

The artefact was correct. The label was wrong.

The image renderer ended with a single instruction to save as PNG, regardless of what the destination filename said. So 369 social cards were written as PNG data inside files named .jpg.

Locally, nothing was wrong in any observable way. The files are valid PNGs. Every browser, viewer and preview pane inspects the leading bytes and renders them correctly. The build passed. The pages looked right. A reviewer opening any of those files would have seen exactly the intended image.

The mismatch exists in one place only: the CDN derives the Content-Type header from the file extension. Every card was therefore served to social media scrapers as a JPEG carrying PNG data — and being fetched by scrapers is the sole reason those files exist at all.

This is the failure mode I find most worth internalising, because it is invisible to the obvious test. The artefact was right. The metadata about the artefact was wrong. Opening the file is precisely the operation that works.

A footnote with its own lesson. My first fix was to re-encode everything as genuine JPEG. I measured before committing: total size rose from 18.0MB to 23.0MB, and the peak per-pixel deviation from the original reached 81 of 255 — concentrated on letterform edges, because lossy compression rings against hard boundaries and these cards are nothing but flat colour and crisp type. I was about to accept a visible quality cost to correct a metadata error. The correct fix was to rename the file.

Measure the remedy, not only the defect.

The specification and the artefact disagreed, so review became meaningless

Before replacing 2,032 live images, I commissioned an adversarial design review: four independent reviewers, each given one brand’s written specification, a random sample of ten rendered cards, and an explicit instruction to fail work rather than pass it.

Three of the four reported that the specification did not describe the images in front of them.

One raised a blocking defect — the cards used a serif typeface where the spec mandated a sans. There was no defect. The renderer uses a serif because the live site’s own stylesheet does. The specification predated a branding correction and had never been updated.

So a capable reviewer spent their effort adjudicating a stale document instead of examining pixels, and produced a confident, carefully argued, completely wrong verdict.

This is the structural problem worth naming. When your specification and your artefact disagree, every possible review outcome becomes uninterpretable. A pass might mean the work is good, or that the spec is lenient. A failure might mean the work is broken, or that the spec is obsolete. The verdict carries no information either way, and you cannot tell which case you are in without redoing the review yourself.

The remedy was to stop maintaining the specifications by hand and generate them from the code that draws the images, measuring every contrast ratio rather than restating a figure somebody typed once. A document that cannot drift cannot mislead.

Building that generator produced two more instances of the same disease. My first version measured a colour against a background it never actually sits on, and printed “BANNED” for a value that measures 8.12:1 where it is really used — a generated falsehood, which is strictly worse than the stale prose it replaced. My second version embedded source line numbers, which went stale the instant anyone edited an unrelated part of the file. A document that raises a false alarm on every commit gets ignored, which is how the original specifications died in the first place.

The deployment check was measuring a cache

I pushed 406 replaced images, then polled the live site comparing served bytes against my local file. Ten minutes later, unchanged. I concluded the deployment had not run. This estate has history there — a fix once sat pushed but undeployed for days without anyone noticing.

I was wrong, and my own test was the reason.

Those assets are served with a one-year immutable cache directive. The edge will keep serving the previous bytes regardless of what deploys behind it. My poll could not have detected success under any circumstances. It was measuring the cache and reporting on the deployment.

Testing instead with a URL that had never existed before — and therefore could not be cached — showed the deployment had completed within minutes.

The finding underneath is the more valuable one. immutable is a promise that the content at a URL will never change. Replacing images in place breaks that promise deliberately. The visual refresh is therefore invisible to precisely the returning visitors it was intended to impress, and no amount of rebuilding will fix it.

Worth noting why this never affected the social cards: those were written to paths where no file previously existed, so nothing was cached and the first request populated the edge correctly. The distinction between adding and replacing turns out to carry a caching consequence, not merely a data-loss one. I had reasoned carefully about the second and not at all about the first.

The count was right. The conclusion was wrong.

One reviewer found a card labelled “Serverless Architecture” sitting on an article about something else entirely. A fair catch. So I wrote a check to find every instance across all 2,032 posts.

My first version searched the title and the article body. It returned zero matches. Of course it did — search enough technical prose for a generic technical term and you will always find it somewhere.

My second version searched titles only. It returned 204. At that point I was one step from implementing the reviewer’s proposed remedy across the corpus.

Then I read the 204. They were labels like “ASO” on an article titled App Store Optimization, and “Cybersecurity” on one about password management. Correct category labels, deliberately broader than the headline, working exactly as designed. The genuine defect was perhaps ten articles with poor metadata.

Had I shipped the proposed fix, I would have broken 204 correct cards to repair ten. The reviewer was right about the defect and wrong about the remedy — and the only way to tell was to stop counting and start reading.

That distinction generalises badly in practice and is worth stating plainly: adding more review does not catch this class of error. The review found it. The failure was in the leap from a finding to a fix.

What this changes

Not more testing. Different questions.

Verify the claim, not the operation. “The write succeeded” and “the artefact is where the system expects it” are separate claims, and only the second one has consequences. Most instrumentation reports the first because it is trivially available.

Derive contracts; never transcribe them. Anything maintained in two places is maintained in zero. If a specification can disagree with an implementation, eventually it will, and every review conducted in between is wasted.

Ask what a passing result looks like when the system is broken. If the answer is “identical”, you have not built a check. You have built a ritual that produces reassurance on a fixed schedule.

Read the exceptions before automating the rule. A count tells you how large something is. It never tells you whether it is real. The gap between those two facts is where confident, expensive, systematic mistakes live.

None of this is novel, and I suspect most engineering leaders would endorse all four in a meeting. All four are easy to skip when a green result is already sitting in front of you — and every one of these cost me hours precisely because the green result arrived first and I believed it.

The most expensive event in engineering is not a failure. It is a success you trusted.