Coverage Is Not Testing

Somewhere right now a team is staring at a production incident in a module with 92% test coverage, wondering how a bug got through a suite that green. The answer is almost always the same: the tests executed the code and checked nothing about it. The structural metrics from part 4 told us how the code is organised; this part asks whether the tests guarding it would notice if it broke — and coverage, it turns out, cannot answer that question at all.

What the percentage actually says

“Coverage” is three different measurements wearing one name, and each one asserts something weaker than people assume.

Line coverage says: this executable line ran at least once during the test run. Branch coverage says: each conditional took both its outcomes at least once. Condition coverage says: each boolean sub-expression inside those conditionals evaluated both true and false. They form a ladder — each rung strictly harder to climb than the last — and most dashboards report only the bottom rung.

Here is a function small enough to see the whole ladder at once:

public decimal ShippingFor(Order order)
{
    if (order.Total >= 50m || order.HasPromoCode)
        return 0m;

    return order.Weight * 1.20m;
}

Two tests — one order at £60, one cheap heavy order — give you 100% line coverage and 100% branch coverage: both statements ran, the conditional went both ways. But HasPromoCode has never been true in any test. Condition coverage is the only rung of the ladder that notices, and hardly anyone measures it. A promo-code regression ships with the dashboard glowing green.

Now the deeper problem. Suppose the tests look like this:

[Fact]
public void Shipping_calculates_without_error()
{
    var calculator = new ShippingCalculator();
    calculator.ShippingFor(_expensiveOrder);
    calculator.ShippingFor(_cheapHeavyOrder);
    // no assertions
}

Every line ran. Every branch ran. Coverage is perfect, and this test would pass if the function returned the order total, a negative number, or the current year. Coverage counts the lines your tests visit; it says nothing about what they check. Even the strictest rung of the ladder only measures execution, and execution is the cheapest thing a test does.

When the number becomes the goal

This is where the pattern from part 1 bites hardest, because coverage is the single most target-able metric in the whole catalogue. Make 80% a gate that blocks merges and reviewers will start seeing tests exactly like the one above — code exercised, nothing asserted, gate satisfied. The team is not being dishonest; they are responding rationally to an incentive that rewards execution and cannot see verification.

The 80% line-coverage figure you meet everywhere is a widely used convention, not a research finding — a number the industry settled on because it is achievable and round, not because a study showed defects fall off a cliff at 80%. Treat it as a social norm with a dress code, not a law with evidence behind it.

My own gate is deliberately narrower and deliberately softer: branch coverage below 70% on the changed files in a pull request produces a warning, not a block. Both choices are load-bearing. Changed files, because a whole-repository target punishes today's contributor for a decade of legacy they did not write, and the only coverage you can actually influence in a review is the coverage of the code in front of you. A warning, because a hard block is an invitation to game — the moment the gate stands between an engineer and Friday, the empty test appears. A warning keeps the conversation human: the reviewer sees it, asks "why is this branch untested?", and sometimes the honest answer is “because it is a logging fallback and not worth a test”, which a blocking gate could never accommodate. That is my practice, built from being on both sides of gamed gates; it is not an industry standard and I do not present it as one.

The honesty check

If coverage cannot tell you whether tests verify anything, what can? Mutation testing — the most underused metric in this whole series.

The mechanism is pleasingly brutal. A mutation tool takes your code and introduces small deliberate bugs, one at a time: + becomes -, >= becomes >, a conditional is negated, a return value is replaced. Each altered version is a mutant. The tool runs your test suite against every mutant. If a test fails, the mutant is killed — your suite noticed the bug. If the whole suite passes, the mutant survived — your suite just certified broken code as correct.

mutation score = killed mutants / total mutants × 100

The score measures the thing coverage pretends to measure: would this suite notice a bug? And the two numbers can diverge completely — a suite of assertion-free tests like the one above has 100% coverage and a mutation score of roughly zero, because no mutant can make a test fail when no test checks anything.

Run a mutation tool against the shipping function and the survivors tell you precisely where the suite is soft. Mutate >= to > and the behaviour changes only for an order at exactly £50 — if no test pins that boundary, the mutant survives, and the survivor report points at the exact operator and the exact missing test. It is the difference between “this file is 92% covered” and “your suite does not know what happens at the free-shipping threshold”.

In .NET I use Stryker.NET; the same Stryker family covers JavaScript and TypeScript, and equivalents exist in most ecosystems. The numbers you will hear around these tools — 60% as a floor for modules you care about, 80% as the mark of a strong suite — are conventions from the Stryker community and its default thresholds, not findings from a study. They are sensible defaults from practitioners who run this at scale, and that is exactly how I hold them: a starting point for the argument, not the argument's end.

Why you cannot mutate everything

The honest limitation: mutation testing is computationally expensive in a way coverage is not. Every mutant means running the test suite again, and a moderately sized codebase generates thousands of mutants. Running it across the whole repository on every pull request is not a realistic ambition for most teams.

So it becomes a scheduling decision rather than a gating one. My cadence: mutation testing runs quarterly, and only on the critical paths — the billing logic, the authorisation checks, the calculation engines, the modules where a silently wrong answer costs real money or trust. Coverage runs on every PR because it is cheap; mutation runs on a calendar because it is expensive; each covers for the other's weakness. Coverage tells you where tests are absent. Mutation tells you where tests are lying.

There is an AI-era reason this pairing matters more now than it did five years ago. An increasing share of the tests in a modern codebase were generated rather than hand-written, and in my experience generated tests are unusually prone to exactly the failure this post is about — plausible structure, thorough execution, shallow or absent assertions. The instinct to double-check machine-written code is widespread: Qodo's State of AI Code Quality 2025 report, surveying 609 developers, found that even among the group reporting the fewest hallucination problems, 75% still hesitate to merge AI-generated code without manual checks. That hesitation is a correct instinct without a number attached. Mutation testing is how you attach the number — it interrogates a suite you did not write line-by-line and reports whether it would actually catch a bug, which is precisely the question you cannot answer by reading generated tests quickly.

The checklist I actually run

• Line coverage: collected everywhere, gated nowhere. It is a map of where tests are absent, nothing more.
• Branch coverage on changed files: warn under 70% in the PR — start a conversation, never block one.
• Condition coverage: worth switching on for modules dense with compound conditionals; ignore it elsewhere.
• Mutation score: quarterly, critical paths only; investigate every surviving mutant in code that moves money or grants access.
• Any suite that gains coverage without gaining assertions gets reviewed as a defect, not celebrated as progress.

Coverage and mutation between them tell you whether the safety net under a file would hold. The next question is which files are going to fall — and for that you stop reading the code and start reading its history. Next, the files that keep changing.