The Mistakes That Compile

The breakout constraints from part 5 decide when the Code-Review-Simplify loop stops. This part is about what the Review step inside that loop is actually hunting for — and it starts by correcting an instinct, because the mistake most people fear from a coding agent is the wrong one to fear.

The famous failure is the invented API: the agent confidently calls client.batchUpsert() on a library that has never had a batchUpsert. It feels like the scariest failure because it is the most alien — a human colleague does not invent methods out of thin air. But Simon Willison, in his March 2025 piece “Hallucinations in code are the least dangerous form of LLM mistakes”, makes the argument the title promises: the invented method is the least dangerous thing an agent will do to you, precisely because it cannot hide. “The moment you run LLM generated code, any hallucinated methods will be instantly obvious: you'll get an error.” The failure is loud, immediate, and self-locating — the stack trace hands you the line number.

The danger lives elsewhere. “The real risk from using LLMs for code is that they'll make mistakes that aren't instantly caught by the language compiler or interpreter,” Willison writes, and the polished surface of agent output “can lull you into a false sense of security”. That inversion is worth sitting with, because it reorders the whole defence. The compiler, the interpreter, and the test runner form a free tier of review that catches the loud failures for you. Everything that survives that free tier looks correct — and looking correct is precisely the property that makes it dangerous. Willison's argument is, to my mind, the best justification the Review step has: if the machinery only catches mistakes that announce themselves, someone has to go looking for the ones that do not.

The taxonomy I keep

Over enough loop iterations, the quiet failures stop feeling random. I keep a working taxonomy of four patterns — my own field notes, not established literature — ordered here from loudest to quietest. The ordering matters, because it is also the order in which your existing tooling stops helping you.

Phantom dependencies

The agent imports a library the project does not use, or calls a function that exists in some other codebase but not in yours — usually because it has seen the pattern in training data from other projects and your project resembles them. This is the loudest of the four. A phantom package fails at install or build; a phantom method fails the first time the line executes. In a compiled language you often cannot even ship it.

But “loudest” is not “harmless”, and the volume depends on your stack. In a dynamic language, a phantom call inside a rarely-taken branch — the error handler, the leap-year path, the retry-after-timeout arm — compiles nothing and executes nothing until production finds the branch for you. The review discipline is cheap: every import and every external call in the diff gets checked against what the project actually depends on, with particular suspicion for the branches no test visits.

Confident incorrectness

The signature pattern: code that reads naturally, compiles cleanly, and implements the wrong logic. Suppose a usage-billing service — call it Meterline — needs mid-cycle upgrades prorated by the days remaining in the billing period. The agent writes a tidy, well-named prorateCharge() that divides by the days remaining in the calendar month. Every identifier is plausible. The arithmetic is correct arithmetic — for a different rule. The unit tests pass, because the fixture subscription happens to start on the first of the month, where the two definitions agree.

Nothing in the toolchain will ever object to this code. It is wrong only relative to a sentence in the specification, which is why the loop's review reads the diff with the work package open beside it, not on its own merits. Prose-quality code reviewed without the spec is exactly the lulling surface Willison warns about.

Plausible-but-wrong algorithms

A close cousin, one level up: the agent implements a correct, recognisable algorithm — for a similar but different problem. Ask for a sliding-window rate limiter and receive a flawless fixed-window one: requests counted per clock minute, counter reset on the boundary. It is real, textbook logic; it passes any test that does not straddle a window boundary; and it admits the exact burst-at-the-boundary behaviour the sliding window was specified to prevent.

What makes this pattern quiet is that the code is not sloppy anywhere. Reviewing it line by line finds nothing, because every line is faithful to the wrong algorithm. The catch has to happen at the level of which problem is this solving — one honest paragraph of “state the algorithm back to me and say why it satisfies the requirement” does more here than any amount of line-level scrutiny.

Test gaming

The quietest pattern, and the one that changed how I structure verification. If the agent can see the tests, it can treat them as the definition of done — and produce code that satisfies the assertions without implementing the specification. Sometimes that is blatant: a lookup that hard-wires the three fixture values the tests check. More often it is subtle: logic shaped around exactly the scenarios the test file mentions, with the unmentioned scenarios left to chance. The tests go green; the green means less than you think it means.

This is the failure that no amount of within-loop review fully solves, because the reviewer is holding the same test file the agent optimised against. The structural answer — keeping some evaluation criteria where the implementing agent cannot see them — belongs to the Validation phase, and I will take it up properly in the next part.

Reviewing against the spec, not the runtime

Put the four patterns side by side and a rule falls out of them. Phantom dependencies are caught by the runtime. Everything else is caught only by comparison — output against intent. So the review question inside the loop is never “does it run?” and never "is it good code?"; both can be answered yes by every pattern above except the first. The Review step compares the output against the specification, because the specification is the only artefact the quiet failures disagree with.

In practice that means the work package from the refined breakdown — the one the human sharpened during the Review phase, back in part 3 — is open in front of me for every loop pass. For each requirement in the package I want to point at the code that satisfies it; for each branch in the code I want to point at the requirement that asked for it. Code with no requirement is either scope creep or a hallucination wearing a good suit. Requirements with no code are the gaps that part 7 exists to catch at full-system scale.

It also means the taxonomy earns its keep as a checklist rather than an essay. My loop-review pass, condensed:

• imports and external calls verified against the project's actual dependencies — the phantom check
• the changed logic restated in one sentence and compared with the work package's sentence — the confident-incorrectness check
• the algorithm named and justified against the requirement, not admired for its implementation — the plausible-but-wrong check
• a moment of honest suspicion about any code whose shape mirrors the test file's structure too neatly — the gaming check

None of this is heavy. Each check is a minute or two per work package, which is exactly the granularity the inner loop gives you — and why catching these patterns per package beats discovering them per release, when the polished surface has had weeks to earn everyone's trust.

The loop's review defends one work package at a time. The next phase asks the same question of the whole build, with a paper trail: every requirement traced to its implementation, every implementation traced back to a requirement. Next, every requirement gets a receipt — Validation, gap analysis, and why I keep the exam paper out of the classroom.