A green test is a claim about a sample
Three suites that passed for weeks. Each was testing something real. Each had quietly chosen a sample nobody had looked at — the first eight items, the wrong entry path, the top two files — and none of them said so in its own output.
I write about monitors that were green and wrong. Tests fail the same way, but with an extra twist: a monitor is at least trying to describe the whole system, whereas a test is always a sample. That is the point of testing. The failure is not sampling — it is sampling by accident and then reporting the result as if it were about everything.
Here are three from my own code, and the one-line habit that would have caught all three.
1. It only ever checked the first eight
I run a set of paid HTTP endpoints. Because you cannot call your own paywall the way a customer does, there is a free self-test that walks the route list and asserts each one answers correctly. Green every run, for weeks.
It walked the first eight routes. The list had grown well past that, and the loop bound had not. Everything from position nine onward had never been exercised — not once — and the report said "all routes OK" because, within its sample, they were.
The tempting fix is to raise the bound to cover everything. Often you cannot: the check costs time or money per item, which is why the bound existed. The fix that survives is to rotate the window and print which slice you took:
def sample(items, size, run_index):
"""A deterministic rotating window: every item is covered every ceil(n/size) runs."""
n = len(items)
start = (run_index * size) % n
return [items[(start + i) % n] for i in range(min(size, n))]
chosen = sample(ROUTES, 8, run_index)
print(f"checked {len(chosen)}/{len(ROUTES)} routes "
f"[{chosen[0]} .. {chosen[-1]}] — full cycle every "
f"{-(-len(ROUTES) // 8)} runs")Two properties matter. It is deterministic, so a failure is reproducible from the run index — random sampling makes a flake impossible to re-run. And it prints the slice, so "all OK" is never mistakable for "all of them are OK."
2. It tested a path no customer takes
I publish a tool on a marketplace. The tests drove it through the REST API — construct input, start a run, assert the output. All passing.
Real callers are not people with an HTTP client. They are agents invoking the tool through the platform's agent surface, which builds the input differently, and on that path the tool crashed. The suite was exercising a genuine interface. It was simply not the interface anyone arrives through, and the two had drifted apart without either side noticing, because nothing tested the second one at all.
The cost is asymmetric and worth stating plainly: a bug on the path you test is embarrassing, and a bug on the path your customers take is the only kind that has ever cost me anything.
3. It reported SAFE after reading two files
This is the one that still bothers me, because the tool is a security scanner — its whole job is to be sceptical.
Pointed at a repository, it read the files at the root, found nothing dangerous, and returned SAFE. The repository was a monorepo. Every line of actual code lived under packages/*, and the scanner had never descended. Two files read, several hundred unexamined, and a verdict rendered on the whole thing.
The bug was the traversal, and that was a ten-minute fix. The defect was that SAFE and "I looked at two files out of three hundred" were indistinguishable in the output. A scanner that examines nothing is a scanner that never reports a problem, which reads exactly like good news.
So the verdict now carries its own denominator, and a suspiciously small one is itself a finding:
result = {
"verdict": verdict, # SAFE | FINDINGS
"files_scanned": len(scanned),
"files_discovered": len(discovered),
"coverage": round(len(scanned) / max(len(discovered), 1), 3),
"skipped_reasons": dict(skipped), # binary, too_large, unreadable, depth_limit
}
if result["coverage"] < 0.5:
result["verdict"] = "INCONCLUSIVE" # not SAFE. It is not the same claim.INCONCLUSIVE is the value most tools are missing. Without it every scanner has exactly two outputs, "problems" and "no problems", and the second one silently absorbs "did not look."
The habit
All three have the same shape, and it is not a testing-specific idea — it is the same mistake as reading a request log and calling it revenue. A result is about the population it observed, and the name of the result rarely mentions that population.
So the habit is one line per check:
Every passing result prints what it examined and what it did not.
Not coverage percentages in a dashboard nobody opens — in the output of the check itself, next to the word "OK". checked 8/41 routes. path: REST (agent surface not covered). scanned 2/347 files. It costs a few characters and it makes the interesting failure mode impossible to miss, because "OK" and "OK, having looked at almost nothing" stop looking alike.
Three questions that get you there before something ships:
- What is the population, and what did I actually take from it? If you cannot answer the second half from the output, that is the bug.
- Is the sample fixed? A fixed prefix is the worst kind, because it is stable enough to look intentional. Rotate deterministically and cover the whole set over time.
- Does the sample match how people arrive? Test the path the caller takes, even when it is the harder one to script — especially then, because "harder to script" is exactly why it is untested.