---
title: "100% coverage, and it still did not run"
description: "A feature was completely dead in production while 178 tests were green and coverage held at 100%. Tests measure lines that ran, not the path that ships."
pubDate: 2026-09-29
author: tusgino
tags: [ai-engineering, architecture, infrastructure, testing]
series:
id: tu-llm-den-harness
title: "From LLM to Harness"
order: 11
cover: /images/covers/coverage-100-ma-van-khong-chay.svg
coverAlt: "One point splitting into two paths: the upper through five ticked boxes, the lower broken toward an empty box"
lang: en
faq:
- q: "How can anything be wrong at 100% coverage?"
a: "It is the path, not the lines. Coverage measures which lines ran, not whether they ran through the path the product actually uses."
- q: "Why did CI let a green suite through?"
a: "Because the only end-to-end test that touched the real path was gated behind an API key, so CI without a key skipped it."
- q: "How do you know a test actually guards anything?"
a: "Reintroduce the old bug and watch the test go red. A test that has never failed has not proven anything."
- q: "Should I copy the per-file 100% coverage gate?"
a: "No, not unless you have the people for it. It is expensive, and it still does not replace a gate that runs the real load path."
---
A feature was completely dead in production while 178 tests were green and coverage held at 100%. Tests measure lines that ran, not the path that ships.
## A true story, with numbers
I saved this story for the end of the series, because it is the thing I want you to remember.
The harness I read keeps a post-mortem directory. The first entry tells this story: the ACP server crashed the instant a real editor connected. Not partially broken — completely dead. The first two RPCs an editor calls both returned an error.
Verbatim:
The bridge was completely non-functional in production despite 178 green unit tests and 100% line coverage.
**178 green tests. 100% coverage. The feature did not work for a single second.**
Reading that the first time, I assumed something was rigged — fake tests, or a coverage number measuring the wrong thing. It was not. Everything was real. Real tests, really executed, really green.
The problem sat somewhere else, and the documentation says it plainly:
Every test mounted the plugin through a path that did not exercise how it actually loads or how its services actually resolve.
Every test mounted the plugin through a **different path** from the one it actually loads through.
## Two paths, one name
Here is the mechanism, and it is worth reading slowly.
The application loads a plugin through a real loader. That loader calls an export-handling function before using it. The in-memory test harness does not. It builds the plugin object by hand, and pre-fills the field the real loader would have filled:
```ts title="how the test harness mounts, versus how the product mounts"
ctx.plugin({ name, inject, apply })
// `inject` is supplied by hand here.
// The export handler is called only by the loader, never by ctx.plugin.
```
The production bug lived in the **loader's** export handling. The test harness walked around exactly that point. So it never touched the bug, even though it ran every line of code around it.
The documentation adds one detail that made me stop. Even calling `ctx.plugin` with a namespace import would not have caught it, because the path is still different.
And the second bug was the same shape. A service lookup crossed a shadow boundary, while the test harness mounted everything flat on one root context and never crossed that boundary at all.
Two independent bugs. One shared reason for slipping through.
## The one e2e, and the lock on it
By now you are asking: was there no end-to-end test?
There were two, and neither saved them.
The first sent an initialization call and checked that stdout was clean. That sounds reasonable. But that call **never reaches the factory** where the bug lived. It takes a different branch.
The second one did reach the right place. And it was **gated behind an API key** — so CI, which has no key, skipped it.
And when it ran locally, it "passed". The documentation explains why, and this is the detail that gave me a chill:
The only test that drove session/new/session/load was key-gated, so CI (no key) skipped it — and locally it "passed" only because a stale built lib/ (with the old code) happened to satisfy module resolution.
A leftover, stale `lib/` directory. It satisfied module resolution. So the test ran against **old code**, and old code does not have the bug.
See the trap: the test was green because it was testing a different version from the one about to ship.
## The line worth framing
Then the documentation says the sentence I want on a wall:
100% line coverage was satisfied the whole time. Coverage proves lines ran; it says nothing about whether the feature works the way it ships.
Coverage proves lines **ran**. It says nothing about whether the feature works **the way it ships**.
The whole time, that 100% was accurate. Nothing was rigged. The number simply answers a different question from the one you thought it answered.
```diagram
type: architecture
title: Two parallel plugin load paths
desc: The test path manually injects fields creating a green illusion; the real ship path fails at runtime
nodes:
- id: testpath
label: Test path (ctx.plugin)
sub: Manually filled inject field
- id: shippath
label: Real shipping path
sub: Loader handles default exports
kind: security
focal: true
- id: green
label: 178 green tests (100%)
sub: False sense of safety
- id: dead
label: Production crash
sub: Missing inject at runtime
focal: true
edges:
- [testpath, green]
- [shippath, dead]
```
## What they fixed
This part is what to copy, not the part that tells the story.
They did not add more tests for the buggy function. They fixed **the test's path**:
- A new e2e drives `session/new` over **real stdio**, booting the whole profile as a subprocess through the real loader.
- That test needs **no API key**, so CI runs it instead of skipping it.
- And they point the subprocess at the repo-root config, so module resolution no longer depends on the working directory — which means the test runs **source**, not a build that may be stale.
Then the final step, and this is the one I want you to remember:
This fails loudly on Bug #1 with no API key. Verified it fails when export default apply is restored.
They **reintroduced the old bug** and confirmed the new test goes red.
A test that has never failed has proven nothing. It is just a green line of code.
## The price of that discipline
That harness keeps discipline at a level I have not seen elsewhere.
The coverage gate is **100% per file** — not 100% across the repo. A well-covered large file cannot subsidise a small file nobody tests. The comment in the config file puts it plainly: *"100% or it doesn't merge."*
And they keep the decisions they rejected. Counted at the exact commit I read:
| Note kind | Count |
|---|---|
| Implemented | 522 |
| Proposed | 40 |
| **Rejected** | **14** |
| Archived | 642 |
| **Total** | **1,218** |
Fourteen rejected decisions were still written down and kept. Add 4 post-mortems and 69 verification scripts.
That is a level of discipline a small repo cannot afford. And here is where I have to be straight with you: **you should not copy it.**
If you are one person writing a blog and building a small harness, per-file 100% is unaffordable. What is worth copying is not the number. It is the **direction of the gate**: a gate must run the path the product runs.
## Tying the series together
The previous ten posts were about mechanism. This one is about the place where a correct mechanism can still be useless.
Remember the first post: the model is a function that remembers nothing. Everything it knows comes from the context you build. If the context-building path has a bug your tests never touch, the model will answer wrongly and sound confident.
Remember the fourth: the loop works until it does not, and there are four ways it breaks. A gate that only walks the success branch will never see the other three.
And here is what connects them. Every piece in this series — the log, the tool pipeline, context cutting, permission — has a **product path** and a **test path**. Those two look so alike that you assume they are one.
They are not one. And the gap between them is where the bugs live.
A green suite says the code you wrote ran the way you described it. It does not say that code is loaded the way the product loads it, or called the way the product calls it. To know the second thing, you have to run that path.
## Do this now
For every test, ask: does it build the object by hand, or does the real system build it? If by hand, it is not testing the load path.
A test that needs an API key is a test CI will skip. If it is the only one touching the real path, you have no gate at all.
So it does not depend on the working directory. Otherwise a test may run against a stale build and go green for the wrong reason.
After fixing, restore the old bug and watch the test go red. A test that has never failed has proven nothing.
## The rule
This whole series is about building the mechanism correctly. The last post is about proving that mechanism runs — and that is the harder job.
Green tests are a necessary condition. They have never been a sufficient one. A test that has never gone red is a test you have not verified.
- 178 green tests and 100% coverage still could not save a feature that was completely dead in production.
- Every test mounted the plugin by hand, walking around the exact point where the real loader handled exports.
- The only e2e touching the real path was gated behind an API key; CI skipped it and local runs went green on a stale build.
- The correct fix is to fix the test's path, then reintroduce the bug to prove the test goes red.
- The per-file 100% gate and 1,218 decision notes are impressive discipline — and unaffordable if you work alone.