We Measured Our Own Internal Linking. The Engine We Had Could Not Fix It.

We publish across five websites. As of this week the corpus is 2,246 posts, which is a number that sounds like an asset and was, for a long time, mostly inventory.

The measurement that changed how we thought about it: 725 of those posts — 32 per cent — have no internal link anywhere on the estate pointing at them. No path in from a related article, no path in from a service page, nothing. They exist, they are in the sitemap, and the only way to reach one is to already know it is there.

That figure is after a month of work. It was considerably worse when we started.

What follows is what the work actually looked like, including the part where the first engine we built did not work and had to be thrown away.

The measurement came before the strategy, and it was wrong twice

The first version of the link-graph tool reported zero orphans. This was encouraging and completely false.

It had been developed on one machine and it resolved repository paths in a way that only worked on that machine. Everywhere else it found no content, counted nothing, and reported a clean graph. A tool that returns “zero problems” when it cannot see the input is worse than no tool, because it produces confidence rather than silence.

We fixed the path resolution on 28 August. The second measurement was the one that was uncomfortable.

There is a general lesson in that sequence and it is not about paths. A measurement tool’s failure mode should be loud. If it cannot find the corpus, it should refuse to run. Ours returned a number, and the number happened to be the number you would most like to see.

The diagnosis: architecture, not authority

The instinct when content is not performing is to reach for authority — more backlinks, more promotion, more distribution. We had been assuming some version of that.

The external data said otherwise. When we finally pulled competitive numbers rather than guessing, the pattern was unambiguous: on the site with the weakest results, four pages out of 490 had any external backlink at all, and over 98 per cent of the inbound links pointed at the homepage. Meanwhile a sister site with the same publishing cadence, the same tooling and the same team had 572 pages with backlinks.

Two sites, one pipeline, wildly different outcomes. That is not an authority gap you can promote your way out of. That is a structural difference in how the content was connected.

The note I wrote at the time was blunt: this is a link-graph failure, not a sitemap failure. Everything was being published. Nothing was being connected.

For anyone running a content operation at scale, this is the distinction worth internalising. Publishing volume and link architecture are separate investments, and volume without architecture compounds the problem — every new orphan dilutes the crawl budget of the pages you already have.

The first engine, and why it stalled

The first implementation did the obvious thing. For each post, search the corpus for other posts that would be good link targets, score the candidates, pick the best few, insert the links.

It worked on small batches. On the full corpus it stalled, repeatedly, in a way that was maddening to debug because it never crashed. It just slowed to the point where a run would not finish.

We did the things you do. We bounded the pathological single calls. We cached the per-source safety mask so the expensive computation was not repeated. Each change helped and none of them fixed it, because the problem was not in any of the code we were optimising.

The problem was the shape of the question. For each of N posts, search N posts is quadratic, and at N = 2,246 with a scoring function of any sophistication, quadratic is the whole story. No amount of caching changes the exponent.

This is a failure mode I have seen enough times to name it. When a system is slow and every targeted optimisation produces a modest improvement that does not add up to enough, the optimisations are usually correct and the architecture is usually wrong. The tell is that the improvements are real but insufficient — genuine gains that never compound into a working system.

The rewrite: plan the graph, do not search it

The replacement went in on 9 September. It was about ten thousand lines, and the commit message was the design: plan internal links instead of searching for them.

Three passes, each linear:

Pass one builds a profile of every post once. The distinctive phrases, the commercial terms it contains, everything the planner will need. Every document is read exactly once and never re-read.

Pass two plans the entire graph in memory. An inverted index from phrase to candidate targets, degree caps so no page becomes a hub by accident, anchor-diversity limits so the same phrase never points at more than a few destinations. It produces a complete plan — every link, its source, its anchor, its target — before a single file is touched.

Pass three applies the plan. It reads each file once, inserts the links assigned to it, and writes it back.

The full run takes about six seconds.

The speed is the least interesting part of this. The important part is that pass two is inspectable. The plan is a file. Before anything is written, you can read what the engine intends to do, count the links per page, check the anchor distribution, and see which targets are over-subscribed. The searching version made decisions inside a loop where nobody could see them. The planning version makes them all at once, in the open, and lets you refuse.

That is the real argument for planning over searching, and it generalises well beyond internal links. An algorithm that produces a plan you can audit is a fundamentally different tool from one that produces a result you must trust.

Three defects the plan made visible

Because the plan was inspectable, three problems surfaced that the old engine would have committed silently.

Links were landing on lines that quoted money. Our sites carry pricing claims, and those claims are protected by a gate that checks every stated price against a canonical ladder. An inserted link inside a paragraph that quotes a price broke that gate. The first fix masked the sentence containing the price, passed its own test, and left the gate red — because the gate reasons about lines, and in MDX a paragraph is a line. The rule that worked masks the line. Three hundred and fifteen links had been placed on money lines; after the rule, none.

The planner addressed posts by filename while the site addressed them by slug. Astro builds a route from a frontmatter slug: when the author sets one. Thirty-one posts across four sites set one. Every planned link to those posts pointed at a URL that did not exist — and because most were cross-brand, no single repository’s own audit could see them. A sister-site URL is external to that repository, so it gets skipped. They were only found by resolving each applied link against the destination repository’s build output.

Sixteen links shipped to pages that were never there. That is the count from the run above, all from the same slug defect, all repaired by re-deriving the correct URL and checking that the new target existed and the old one did not before making each change.

None of these were visible in the source. The markdown was well-formed, the build succeeded, and the links rendered. They were wrong about the world rather than wrong about syntax, which is the category of defect that survives every check that reads the file rather than resolving what it points at.

Where it landed

The estate now carries 14,090 internal links. Of those, 1,607 point at a commercial page — a service page, a location page, something with an enquiry form rather than a byline — across 96 commercial targets. 6,542 of them cross from one brand to another, which is the connective tissue between five sites that were previously five silos.

Orphans went from the great majority of the corpus to 725. That is still 32 per cent and it is not good enough, but it is a tractable number attached to a specific list of URLs, which it was not in August.

What I would take to another team

Instrument the thing you are about to invest in, and make the instrument fail loudly. We nearly built a content strategy on a tool that reported zero because it could not find the files.

Separate volume from architecture in your planning and your budget. They are different investments with different failure modes, and volume without architecture actively compounds.

When targeted optimisations are real but never sufficient, stop optimising. That pattern is the signature of a wrong question, and the next optimisation will also be real and also insufficient.

Prefer algorithms that emit an auditable plan. The six-second runtime was a side effect. The ability to read the decision before it is executed is the actual product.

Verify against the artifact, and across the boundary. The link that pointed nowhere was well-formed in the source, correct in the build, and broken only when resolved against a different repository. Anything that spans systems needs a check that spans systems, because each system’s own audit will report itself clean.

---

Ash Ganda is CTO and founder of Ganda Tech Services, which runs technology, content and engineering operations for a group of Australian brands.