Your Analytics Are Lying to You, and Nothing Will Tell You

In the space of one month, three separate controls failed on systems I am responsible for. Different vendors, different technologies, no connection between them.

They failed identically.

In each case the configuration remained present, well-formed and visibly correct. In each case the effect it was supposed to produce went to zero. And in each case, every dashboard, console and audit tool continued reporting normally, because nothing about the situation was an error.

That is a class of failure, not three incidents, and it is worth naming because the defence against it is different from the defence against outages.

The three

A Google Analytics tag that was not on the page. A GA4 property existed, provisioned and named. A third-party SEO platform reported the site as connected to Google Analytics. The site’s own Content Security Policy explicitly allowed the tag manager domain in script-src and the Analytics collection domains in connect-src — someone had done the careful, unglamorous work of opening exactly the traffic the tag would need.

Three independent signals, all consistent, all pointing the same way. The tag was absent from the page.

Each signal was evidence that somebody had intended to install analytics. Not one was evidence that analytics was installed.

An email open-rate filter that matched nothing. Gmail routes tracking images through a proxy, and Google Workspace administrators keep an image-URL allowlist for entries that should bypass it. Among other things, that allowlist is what lets a CRM tell your own team’s opens apart from your prospects’.

Two changes landed independently: the vendor began building tracking URLs with a variable prefix, and Google Workspace stopped matching more specific subdomain entries. Neither broke anything alone. Together, an allowlist entry written as a subdomain stopped describing anything that existed.

The entry was not deleted. It did not turn red. It remained perfectly well-formed — it simply no longer matched. The open rates kept arriving, uninterrupted, looking exactly like the numbers everyone was used to.

A redirect rule that never fired. A rule to send one hostname to another sat in a site’s configuration for months. It deployed successfully every time. The platform matched on a different part of the request than the rule assumed, so it had never once executed.

Nothing errored. The deploys were green. The rule was right there in the file.

What makes this its own category

An outage announces itself. Something returns a 500, a dashboard goes red, a customer calls. The system’s failure and the system’s reporting of that failure are the same event, and the organisation’s entire incident apparatus is built around that coupling.

These three decouple it. The control’s configuration and the control’s effect are different things, and only one of them is observable from where anyone is looking.

Worse, the observable half is the reassuring one. The GA4 property exists. The allowlist entry is present. The redirect rule is in the config. Anyone who checks will find exactly what they expected to find, and will correctly report that everything is in order.

This is not carelessness. In all three cases someone competent had done the work, and the artefact of that work was still sitting there, still valid. What had gone was the effect, and nothing in any of these stacks has a surface on which “this is still configured but no longer does anything” appears as a fault.

Why it is getting more common, not less

Two structural reasons, both worth planning around.

Configuration is increasingly a contract with someone else’s system. An allowlist entry is a promise about how Google Workspace will match strings. A CSP is a promise about domains a vendor will call. A redirect rule is a promise about how a platform parses a request. You control the text of the promise; the other party controls the interpretation — and can change it, correctly and with notice, without your text becoming invalid.

Everything is layered enough that adjacent answers are available. Ask “is analytics connected?” and three systems will answer honestly about the account association, the property, and the network policy. None of them fetched the page. The more sophisticated your tooling, the more confidently it can answer a question one step to the left of the one you meant.

The check that finds all three

None of these were found by better monitoring. All three were found the same way, and it is cheap.

Compare siblings.

The analytics failure surfaced by fetching the live HTML of all five sites in the group and grepping each for a measurement ID. Four returned one. The fifth returned nothing. The check took under a minute.

What makes that powerful is not the grep. It is that the comparison needs no specification. I was not measuring the site against a documented standard — I did not have one, and writing one is exactly the task that never gets done. I was measuring it against four siblings configured by the same people for the same purpose. One site behaving differently from four others is a finding you cannot argue with, and it requires no documentation to interpret.

If you run more than one of anything — sites, properties, tenants, environments, mail domains — this is available to you today:

1. Fetch or query all of them the way a real consumer would.
2. Extract the one thing that should be present in each.
3. Look at the odd one out.

The odd one out is the finding. You do not need to know what correct looks like; you only need them to disagree.

What I would put in place

Three things, in order of cost.

Verify the effect, not the configuration. For every control that matters, write down what observable outcome proves it is working, and check that. Not “is the tag configured” but “does the page a visitor receives contain the tag”. Not “is the allowlist entry present” but “are internal opens actually excluded from this month’s numbers”. The rewrite is small and it is the whole game.

Run a sibling comparison on a schedule. Quarterly is enough. It costs minutes and it needs no standard to exist first.

Treat vendor change notices as configuration risk. The open-rate failure was announced by the vendor. The notice was accurate and arrived in advance. It went to an inbox, and nothing connected it to the allowlist entry it invalidated. Someone has to own the question “what of ours does this break?”, and it is not a task that assigns itself.

The uncomfortable part

If a control can stop working without producing an error, then the length of time it has been broken is unbounded by anything except how often somebody looks.

I do not know how long the analytics tag was missing. There is no signal to reconstruct it from, because the absence of data looks identical to the absence of traffic. That is not a gap in the investigation; it is a property of this class of failure. The evidence you would need is the thing that was not collected.

Which leaves one honest position: for the controls that matter to your decisions, the interval between checks is your worst-case exposure. Choose it deliberately rather than discovering it later.

---

Frequently asked questions

How do I know if my analytics tag is actually installed? Fetch the live page and search the returned HTML for your measurement ID. Not the plugin settings, not the property console, not an audit tool’s connection report — the page a visitor receives.

Why did three separate tools say analytics was working? Because each answered a real question adjacent to the one that mattered: does a property exist, is an account associated, is the vendor’s domain permitted. None of them fetched the page and looked for the tag.

What is a sibling comparison? Querying every instance you run the same way and looking for the one that differs. It works without a documented specification, which is why it gets done when specification-based checks do not.

Are inflated email open rates a real problem? They are when internal opens are counted as prospect engagement, because the contamination is not random — it concentrates on the messages your own team reviews most, which are usually the ones you are evaluating.

How often should these checks run? Quarterly is sufficient for most controls. The important framing: for anything that fails silently, the interval between checks is your worst-case exposure window.