The Ledger That Disagreed With the Disk

The most expensive data problems in a business are not the ones that throw errors. They are the ones where two systems each hold a version of the same fact, both versions look reasonable, and nothing anywhere is responsible for noticing that they stopped matching.

Here is the smallest possible example, from our own operation, because a small one shows the mechanism without the noise of a disaster.

Our video pipeline keeps a tracking record of what has been produced. In late September three entries in that record were marked as not yet rendered. On the disk, in the ordinary output directory, sat three finished files — complete, correct, written within minutes of each other. The record and the reality had disagreed for nineteen days. No warning fired. No check failed. Every status report in the system was green, and every one of them was accurate about the thing it looked at.

Nobody lost money over three files. The reason it is worth writing about is that the mechanism does not care about scale, and the version of it that runs in a stockroom or an accounts ledger is the same shape.

Two writers, one fact

The structural cause is always the same, and it is worth naming precisely because recognising it is most of the fix.

One process finishes the work. A different process updates the record.

Each does its own job correctly. Neither is responsible for the relationship between them. When the first succeeds and the second does not — because it ran earlier, or ran on a different machine, or was interrupted, or was never called on that particular path — nothing is in an error state. The work exists. The record exists. They simply describe different worlds, and no component in the system has the job of comparing them.

You can find this arrangement in almost any business that has added software over time rather than all at once:

• Sales and inventory. The order system knows an item was sold. The warehouse system knows what is on the shelf. A stock figure is correct in each and wrong for the business.
• Invoicing and banking. The accounting package marks an invoice paid when it is marked paid. The bank knows when money moved. The gap between them is where both chased and uncollected revenue lives.
• CRM and delivery. Sales marks a project won. Operations has no record of it until someone forwards an email.
• Payroll and rostering. Hours worked in one system, hours paid from another.

In each case the two copies were introduced for good reasons, by people solving a real problem, and neither copy is wrong on its own terms.

Silence reads as agreement, and it should not

The part that does the actual damage is not the drift. Drift between independently written copies is a certainty given enough time; it is not a risk to be managed down to zero, it is a property of the arrangement.

The damage comes from how the silence is read.

Nothing errored, so everyone assumes the two sides match. But no comparison ran. A green light and an unasked question look identical from a distance, and every dashboard in the business is built to show you the first while being structurally incapable of showing you the second. A report cannot tell you it is missing a check that does not exist.

This is why “we would have noticed” is rarely true. You notice a system that fails. You do not notice a question nobody asked.

In our case the question was only asked because somebody wanted a count of what had been published and pulled it from both places. That is not a control. That is a coincidence with a person attached to it.

★ Insight ───────────────────────────────────── The instinct after a drift incident is to add an alert. But you cannot alert on a disagreement nobody compares — the alert has to be attached to a comparison that runs, and writing that comparison is the whole job. The alert is the cheap part and it is where most remediation stops, which is why the same incident recurs in a different pair of systems six months later. ─────────────────────────────────────────────────

Why “who owns the reconcile” is the useful question

Ask a business who compares two copies of the same fact and you get one of three answers.

A named person or process, on a stated schedule. Rare, and always the result of something having gone wrong before.

“It gets picked up.” This means somebody notices eventually, usually the person closest to the consequence, usually after the decision that needed the number has already been made. It is not a control, it is a dependency on somebody being curious on the right day.

Nobody. Which is the honest version of the second answer, and the one we would have had to give about our own tracking record.

The distinction matters because the second answer feels like coverage. There is a person, they are competent, they do find things. What there is not is a defined point at which the comparison happens whether or not anybody is curious, and a defined thing that happens when the two sides disagree.

The exercise, and it takes minutes

This needs no software and no project.

1. Name one fact your business stores in two systems. Not a category — a fact. A stock level for a specific product line. The paid status of an invoice. A customer’s current address. A project’s stage.
2. Write down which system each copy lives in, and what writes to it. Usually a different thing writes to each, which is the point.
3. Answer: who compares them, and how often? If the answer is “it gets picked up,” you have the arrangement described above.
4. Answer: what happens when they disagree? If there is no defined answer, the disagreement will be resolved by whoever finds it, in whatever way seems reasonable at the time, and no record of the discrepancy will survive.
5. Do it by hand once, this week, for the fact that would hurt most. Not to fix anything. To find out whether the two copies currently agree, which is a thing you almost certainly do not know.

That fifth step is the one worth insisting on. Most businesses can reason their way to the first four and will still tell you they are probably fine. Running the comparison once, by hand, converts an assumption into a number.

What a reconciliation control actually contains

Most businesses that decide to “add a reconciliation” produce a report and stop. A report is the output. The control is four things, and the report is the least important of them.

A defined comparison. Which two copies, of which fact, compared on what key. Vague versions of this (“we check the stock figures”) collapse under the first edge case, because two systems rarely hold the fact at the same grain — one has units on hand, the other has units available after allocations, and nobody wrote down which one the comparison is about.

A schedule that does not depend on attention. Daily, weekly, whatever suits the rate of drift. The requirement is that it happens when nobody is thinking about it, because the whole failure mode is that nobody was.

A defined response to a disagreement. Which copy wins, who is told, and what happens to the record of the discrepancy. If this is undefined, the first disagreement is resolved by whoever finds it, in whatever way looks reasonable at the time, and nothing survives to tell you it happened twice.

A record that the comparison ran. Not that it passed — that it ran. This is the one that gets left out, and it is the one that distinguishes “the copies agree” from “nobody looked.” A control with no evidence of execution is indistinguishable, six months later, from a control that quietly stopped running.

None of the four requires software. All of them can be a person with a calendar reminder and a shared document, and for one fact in a small business that is often the correct implementation. What matters is that all four exist, because three of them is a habit rather than a control.

The argument for one system, restated honestly

This is the real case for consolidating onto a single platform, and it has almost nothing to do with licence costs or feature comparisons.

A single database cannot disagree with itself.

That is the whole argument. When the stock level is one row that sales, the warehouse and the accounts all read, there is no reconciliation to own, because there is no second copy to reconcile against. The integration work, the nightly sync, the person who checks, the alert when the sync fails — all of that is cost incurred to simulate a property you get for free from not having two copies.

It is not always the right trade. Consolidation has its own costs, and plenty of businesses are correctly running several systems because no single one does all of what they need. But the comparison should be made with the reconciliation burden counted on the multi-system side of the ledger, and it usually is not, because that burden is spread across people’s judgement rather than sitting in an invoice.

For us the fix was smaller than a migration: the tracking record is now written by the same step that produces the file, so there is one writer rather than two, and a reconciliation pass runs over the pair regardless of whether anyone asks. That is a narrower fix than “adopt one system”, and it is the right size for three files.

The question it came from is not narrow at all. Which fact does your business store in two places, and when did anyone last check that the two copies still agree?

---

Consolidating systems of record, and building the reconciliation controls for the ones that stay separate, is work we do through Ganda Tech Services — with platform implementation through Cosmos Web Tech and infrastructure through Cloud Geeks.