Measuring Before You Commit a Quarter

A report told us 294 pages needed rewriting. Each was appearing in search results and receiving no clicks — apparently demand we were failing to capture.

294 rewrites is a quarter of work. It was costed, sequenced and nearly started.

Measured properly, the real number was six.

Where the other 288 went

Three filters, applied in about an hour.

61% were our own indexation checks. When you want to know whether a page is indexed, you search for it with a site: operator. That search records an impression against the page. Do it across a few hundred pages while auditing a migration, and you generate hundreds of impressions with zero clicks — because nobody was looking for anything, somebody was checking something.

These are indistinguishable from real demand in the report. They are the same rows, in the same table, with the same shape.

Most of the remainder were ranking far below the first page. A page appearing at position 31 receives impressions and effectively no clicks, and that is not a content problem. Nobody is rejecting the title; nobody is seeing it. Rewriting a page at position 31 to improve its click-through rate is optimising a variable that has no effect at that position.

What survived was six pages: genuinely on page one, genuinely receiving impressions from real searches, genuinely not being clicked. Those are the ones where a title or description change might actually do something.

★ Insight ───────────────────────────────────── The failure was not that the report was wrong. Every row in it was accurate — those pages did receive impressions and no clicks. The report answered its question correctly and the question was not the one being asked. “Which pages get impressions and no clicks” is a different question from “where is there unmet demand”, and the gap between them was 288 pages of work. Most bad prioritisation comes from a correct answer to an adjacent question. ─────────────────────────────────────────────────

The general shape

This pattern recurs across every kind of business analysis, and it has a consistent structure.

A report produces a list. The list is long, which feels like thoroughness.

The list is treated as a backlog. Length becomes scope. Somebody estimates it.

Nobody asks what is in it that should not be. Because the report is accurate, and accuracy is mistaken for relevance.

Examples we have seen in other businesses, all the same shape:

• A list of “at-risk customers” that is mostly dormant test accounts.
• A list of “unprofitable products” that is mostly items bundled with profitable ones.
• A list of “underperforming pages” that is mostly pages with no commercial purpose.
• A list of “security findings” that is mostly informational notices on systems with no exposure.

In each case the report is correct and the backlog is fiction.

The hour that pays for itself

Three questions, before committing resources to any generated list. None require new tooling.

1. Take a random sample of ten and inspect each individually. Not the top ten — a random ten. Ask whether each genuinely belongs. If more than two do not, the list has a population problem and the total is meaningless until you understand it.

2. Ask what could have produced this row other than the thing you think. Our impressions could have come from real searchers or from us. The report cannot distinguish them, and knowing that a second source exists is most of the work.

3. Ask whether acting on this row could plausibly change anything. A page at position 31 cannot be improved by a better title. An at-risk customer who is a test account cannot be retained. If the intervention cannot work on that row, the row is not a work item regardless of whether it is accurate.

The first question is the highest-value and the least done, because inspecting ten individual items feels like a slower version of reading the summary. It is the only step that surfaces a population error.

Why lists are believed

Worth understanding, because the fix is partly cultural.

A number carries authority that a judgement does not. “294 pages need attention” survives a meeting in a way “I think our titles could be better” does not, so there is a quiet incentive to produce the number and none to interrogate it.

Length reads as rigour. A short list looks like the analysis was shallow. A long one looks thorough, when frequently the opposite is true — a long list is often one that has not been filtered.

Nobody is rewarded for shrinking scope. Coming back with six items after being asked to assess the situation feels like under-delivering, even when it is the more valuable answer by a wide margin.

That last one is worth addressing directly in how work is commissioned. The output of an analysis should be permitted to be “smaller than expected”, and reducing a 294-item backlog to six should be recognised as the result it is.

How to ask for a list you can trust

Most of this is fixable at the point the analysis is commissioned, by asking for two things alongside the output.

Ask for the exclusions. “What did you filter out, and why?” A list produced with no exclusions has almost certainly not been filtered, and the analyst will usually be able to tell you immediately what should have been removed — they simply were not asked.

Ask for the ten. A random sample of ten rows, inspected individually, with a note on each. This is twenty minutes of work and it converts a summary into something verifiable. If the analyst cannot produce it, the list was generated rather than reviewed.

Both requests are easy to make and neither implies distrust. They are the analytical equivalent of asking a builder how they arrived at a quote — normal, expected, and the absence of an answer is itself informative.

The number that matters

We did the six. It took an afternoon.

The quarter that was nearly spent on the other 288 went instead to work that changed something — which is the actual return on that hour of measurement, and it does not appear in any report either.

The habit worth keeping: before committing more than a week to a list somebody generated, spend an hour trying to disqualify its rows. You will either gain confidence in the plan or save most of the quarter, and both are worth the hour.

---

Measurement, prioritisation and the question of what your reports are actually counting is part of the advisory work we do through Ganda Tech Services.