Green CI, 381 Passing Tests, and a Product That Never Published a Post

Three signals said the work was done. The main branch was green. A suite of 381 tests passed. A browser screenshot showed a post sitting on the calendar exactly where it should be.

Then a 50-agent adversarial audit — 997 tool calls across shipped code, both delivery plans, the tracker, production readiness and the test suite itself — confirmed 23 of 25 findings: 7 critical, 12 high, 4 medium.

The two that mattered were not edge cases. Content created through the External API was never handed to the publisher. And no production code path anywhere marked a post as published.

What was genuinely delivered

The audit is worth taking seriously precisely because most of the roadmap held up.

P0 and P1 — Postgres, CI/CD, authentication, GDPR handling — are real and working. P2’s multi-tenant data model is genuinely shipped, not claimed. The JWT path, the one a user hits through the WordPress plugin or the Chrome extension, reaches the publisher and posts really do go out to the connected networks.

So this is not a story about a product that does nothing. It is a story about the distance between the part that was verified by someone actually using it and the part that was verified only by a machine agreeing with itself.

The two findings that outrank everything else

The External API accepts posts it never publishes. The function that forwards a post to the publisher has exactly two call sites, both inside the JWT controller. The external create path writes a database row, returns 202 Accepted, and stops. There is no scheduler scanning for due posts either. The 202 is not a lie anyone told — it is a lie the architecture tells on everybody’s behalf.

Nothing in production ever marks a post published. The only line that writes status='published' sits inside a NODE_ENV==='test' branch. Production never executes it. The test suite asserts against it.

Read that pair together, because that is where the real lesson is.

The suite was not testing the product. It was testing a branch that exists only for the suite.

Every downstream number inherits that. Dashboard counts, the calendar status pill, the API’s own documented status contract — all permanently wrong, all reported confidently, none of them caught by a green build. A test that exercises code the customer never runs is not a weak test. It is a false one.

Now, take the security finding only

Every request to the external API runs a cost-12 bcrypt comparison on the main event loop — including requests carrying unknown keys, including requests to paths that do not exist. The rate limiter that was supposed to stand in front of this was mounted after authentication, so it never ran at all.

One host at a few requests per second is enough to stall login, health checks and the dashboard for everyone. No credentials required. It just goes on to show that ordering in a middleware stack is not a stylistic preference — it is the whole control.

Three of the findings were mine, from that session

This is the part I would rather leave out, which is exactly why it belongs in.

A default I added to the test setup masked a production branch from CI. A response-contract test I wrote passes vacuously, because an empty array satisfies the schema it validates against. A route mount I placed sits ahead of the counters it was supposed to be measured by.

Generator ≠ approver — including when the generator is me. The discipline is not a comment on anyone’s competence. It is a structural fact: whoever wrote the thing is the worst available judge of whether it works, because they are optimistic by construction and they are grading their own intent rather than their output.

What this changes about a roadmap

The tracker said P7 shipped and P8 completed. In production, P7 returns 404 because its feature flag was never enabled, and P8’s two hostnames have no DNS records at all — the directory is never installed, typechecked, tested or deployed, because the deploy step only syncs the backend.

Neither claim was dishonest. Both were made against a definition of “done” that meant the code exists and the tests pass.

Despite that, the fix is not more process. It is a narrower definition. A phase is not delivered when its code merges. It is delivered when a request from outside the building produces the outcome the phase promised, and something other than the author says so.

Three rounds of adversarial review on this codebase found 15, then 16, then 23 real defects — each round run against a state the previous round had already blessed. The number going up is not a failure of the method. It is the method working, on a surface that was never as verified as its signals suggested.

Where this goes next

The queue reordered itself around evidence rather than around plan sequence.

Close the publish loop first — move dispatch into the shared service so the API-key and JWT paths cannot diverge again, and delete the test-only publish branch so the suite is forced to exercise the real one against a stubbed publisher. Then the rate limiter, in front of authentication where it belongs. Then status reconciliation, rescoped: one correlation id cannot describe a post that fans out to several networks, so per-target rows rather than a single column. Then correct the tracker, and only then turn the flags on.

The hardening phase that was next on the roadmap is still right. It is just not yet.

What a build needs is not a greener signal. What it needs is a judge with the standing to say no — and the honesty to point it at your own work first.

Frequently asked questions

What is an adversarial audit? A review whose explicit goal is to falsify the claim that the work is done, rather than confirm it. The reviewer is rewarded for finding defects, not for agreeing. Run at scale it means many independent agents each attacking a different surface — code, plans, tracker claims, production config, and the test suite itself.

Why did 381 passing tests not catch a broken publish path? Because a test can only fail on code it actually runs. The publish-status assertion targeted a branch gated behind NODE_ENV==='test', so the suite verified a code path that exists only during testing. Coverage thresholds did not catch it either — the floor excluded 2,762 lines of the real publishing path.

Does this mean AI-assisted development produces worse code? No. It produces code faster than the default verification habits can keep up with. The failure here was not authorship, it was that green CI, a passing suite and a screenshot were each treated as evidence of working software when none of them tested the path a customer takes.

What is the single cheapest fix? Delete the test-only branch. Once the suite cannot assert against a fake, it is forced to exercise the real path, and the defect surfaces on the next run instead of in production.

Related

• Adversarial Verification & Fail-Closed Gates: The Mechanics Behind 509 Scheduled, 135 Held, 0 Bad Publishes
• The Self-Correcting Content Pipeline: What 14 Days of Performance-Weighted Auto-Generation Actually Look Like