Re-Vendoring Loses in Both Directions

Most organisations have at least one shared component that exists in two places: a canonical version somewhere central, and a copy inside a project that needed it.

Everyone knows the copy will drift. The mental model of that drift is almost always wrong, and the wrongness is expensive.

The model everybody has

The assumption is that drift runs one way. Canonical moves forward; the copy sits still; eventually you notice the gap and catch up.

That model makes the fix obvious: copy the newer one over the older one. It is a one-line mental operation and it feels like housekeeping.

We had exactly that situation. A component that signs off on whether a piece of work is allowed to publish existed centrally and in our pipeline, and our copy was seventeen days behind. An audit noticed and blocked the pipeline, correctly — running a component that is missing an upstream fix is worse than running none, because the missing fix is invisible and the pass is not.

So we copied canonical over ours.

What that cost

A cross-implementation parity suite — the same inputs run through both our implementations, asserting they agree — had been passing 23 of 23. After the copy it ran 21, then degraded to a state where one side could not even report a version.

The reason is simple in hindsight. Our copy had gained things canonical did not have: a function defining which checks are required for each kind of work, and an integration with a per-project configuration. Canonical had gained the signing mechanism we were missing.

Both had moved. Each had something the other lacked. A copy in either direction was a downgrade, and the direction we happened to choose determined which capability we lost.

The correct fix was a merge: canonical’s signing spliced into our implementation, keeping our interface. After that, parity ran 23 passed, 0 failed — including the case that actually matters, where a record signed by one implementation verifies under the other.

★ Insight ───────────────────────────────────── The one-directional model persists because of what each side is visible from. From the central team’s seat, the copy is behind — that is the only relationship they can see. From the project’s seat, canonical lacks the local features — also the only relationship visible from there. Both views are accurate and both are half the picture, and whichever team performs the upgrade applies its own half as though it were the whole. ─────────────────────────────────────────────────

Why this is a governance question rather than a build question

The temptation is to file this under developer discipline. It is not, for three reasons.

The failure is silent and delayed. Nothing errors when you overwrite a local improvement. The code compiles, the tests that exist still pass, and the capability simply stops being there. It surfaces weeks later as a behaviour nobody can explain, at which point the overwrite is no longer a candidate explanation because it was recorded as an upgrade.

It is most likely during remediation. The moment you are most likely to re-vendor carelessly is the moment an audit or an incident has told you that you are behind. That is when the pressure is on, the fix looks obvious, and nobody wants to hear that the upgrade needs a design conversation.

It scales with how many copies you have. Every vendored copy is an independent line of divergence. A component copied into six projects has six sets of local improvements, and an upgrade campaign that treats them as six instances of “behind” will silently discard all of them.

The check that made it visible

The only reason we caught this was a parity test — a suite exercising both implementations against the same inputs and asserting agreement.

Without it, the overwrite would have looked like a clean upgrade and shipped. The local capabilities would have disappeared, nothing would have failed, and the loss would have been discovered later as a mystery.

That is the transferable control: when the same logic exists in more than one place, have something that compares them. Not a test of either one in isolation — those both pass. Something that runs the same input through both and asserts the same output.

If a comparison is impractical, the weaker fallback is an inventory: a written list, per copy, of what that copy has that the source does not. It is worse than a test because it depends on people maintaining it. It is much better than nothing, because it makes local divergence a declared fact rather than something only discoverable by reading the diff.

The three strategies, and what each actually costs

Most organisations pick one of these by accident. It is worth picking on purpose, because the costs land in different places.

Vendor and freeze. Copy it in, never update. Cheapest today. The cost arrives as a security advisory against a version you are four years behind, at which point the upgrade is not an upgrade but a migration, and every local change has to be rediscovered by reading the diff.

Vendor and merge. Copy it in, accept that you will make local changes, and merge upstream deliberately each time. This is what we do, and the honest cost is that every upgrade is a small design conversation rather than a command. The benefit is that nothing is ever silently lost, because merging forces somebody to look at both sides.

Depend, do not vendor. Take it as a package and never modify it. Cleanest by a distance — upgrades are genuinely trivial and drift cannot happen. It only works if you never need a local change, and the moment you need one, teams typically vendor it anyway and tell nobody.

The failure we had is specific to the second strategy being run as though it were the third: local changes were made, and upgrades were performed as if none had been.

The organisational tell

There is a question that surfaces this quickly in any team, and it takes about a minute.

“Which of our shared components have we modified locally?”

If the answer is “none, we take them as-is”, ask a follow-up: has anyone ever needed a change and got it merged upstream? If upstream is slow or the component belongs to another team, the pressure to patch locally is enormous, and a local patch nobody declared is exactly the improvement an upgrade will discard.

If the answer is “several, but I would have to look”, that is the honest and common case, and it is the one that needs an inventory.

If nobody can name a single shared component at all, that is not an absence of the problem — it is an absence of visibility, and the copies are still there.

Four questions before any upgrade of a shared component

1. What does the local copy have that the source does not? If nobody can answer, do not proceed — find out. The answer is rarely nothing.
2. Is there a test that would notice if a capability disappeared? If the only tests are of the source’s behaviour, they will pass after the overwrite.
3. Is this a merge or a replacement? Say which, out loud, before starting. Most upgrades are recorded as replacements and should have been merges.
4. Who else has a copy? An upgrade campaign that finds one divergence has almost certainly not found the only one.

What a merge actually involved

For anyone facing the same decision, the concrete shape of the fix, because “merge rather than replace” is easy to say and vague in practice.

Identify what each side gained, separately. Not a diff — a list of capabilities. Ours was short: canonical had gained a signing mechanism; we had gained a required-checks function and a configuration integration. Three items total, and the list took longer to produce than the code change did.

Decide which side’s interface survives. This is the decision that determines how much downstream work follows. We kept ours, because two other pipelines were already calling it and changing the signature would have stranded them. That meant canonical’s new code had to be adapted to our shape rather than the reverse.

Splice the smaller change into the larger context. The signing was self-contained; the local capabilities were woven through. Moving the self-contained thing was far less risky than re-applying three integrated changes onto a fresh copy.

Prove it with the test that failed. Parity had dropped to 21 of 23. The merge was not finished when it compiled — it was finished when parity ran 23 of 23 again, including the cross-implementation case where a record signed by one side verifies under the other.

That last point is the one worth carrying. The test that caught the problem is the test that defines done. If the only thing confirming your merge is that nothing errored, you have repeated the original mistake with more steps.

The broader pattern

This is the same shape as several other failures we have written about this month, and the family resemblance is worth naming: two things that are supposed to agree, no mechanism that compares them, and a silent divergence discovered late.

A vendored library and its source. A record and the thing it records. A gate and the thing it gates. In every case the individual components are correct, nothing errors, and the gap grows quietly until something downstream finally disagrees.

The general remedy is always the same and is always slightly boring: make the comparison an artefact that runs, rather than an assumption people hold.

---

Managing shared components, vendored dependencies and the divergence between them is part of the engineering governance work we do through Ganda Tech Services and Cloud Geeks.