Every Requirement Gets a Receipt

Part 6 ended on an admission: one of the four failure patterns — test gaming — cannot be fully caught inside the Execution loop, because the reviewer is holding the same test file the agent optimised against. The loop defends one work package at a time, with the tools the loop has. Validation is the phase that asks the same fidelity question of the whole build, with better tools, and it answers exactly two questions: did we build what we said we would build, and how efficiently did we build it? The first question gets a paper trail. The second gets a telemetry read. Both feed forward, which is what stops this phase being an audit and makes it part of the engine.

To keep the series' ledger straight before the mechanics: requirements traceability and gap analysis are old, credited disciplines, and I will name their owners as we go. What is mine is the decision to make them PREVC's fourth phase — measured against the Review output specifically — and the two feedback loops the phase drives. Creator voice for the arrangement; credit for the parts.

The matrix that refuses to average

The core instrument is the requirements traceability matrix. Perforce defines it plainly: “A requirements traceability matrix is a tool or document that maps the relationship between requirements and other artifacts like tests, issues, and source code.” In PREVC the two things being mapped are precise: the specification that left Review — the refined breakdown with its constraints and intent — and the implementation that left Execution. One row per requirement; each row points at the code that satisfies it, the test that exercises it, and the verification that confirms it.

The matrix is worth its ceremony because of what falls out when a row will not complete. Four findings, each with a different meaning:

• A requirement with no implementation — a gap. The work package was missed, or silently abandoned mid-loop.
• Implementation with no requirement — scope creep, or a hallucination wearing a good suit. Code that answers a question nobody asked.
• A requirement partially implemented — the incomplete package, usually the one whose loop exited on a time budget.
• A requirement implemented differently than specified — drift, the quietest of the four, and the one the polished surface hides best.

Run Rosterly, this series' invented rota service, through the matrix after its eligibility-rules cycle and the exercise stops being abstract. Requirement 2.1, the qualification check: implemented, tested, receipt issued. Requirement 2.3 — eligibility evaluated against the post-swap rota, the line the human added at the gate in part 3 — implemented, because the loop caught its violation in part 4; receipt issued, with a note about how nearly it was missed. But the matrix also finds a GET /swaps/eligible-staff endpoint no requirement mentions: the agent added it because it seemed useful. Perhaps it is. That is a decision for a human, and until a human makes it, unrequested code is unpriced liability. The matrix's whole value is that it refuses to summarise — eight requirements “mostly done” is not a status, it is eight rows, each either receipted or not.

Gap analysis, credited

The comparison discipline has a name in software quality assurance. Qodo defines it as directly as anyone: “Gap analysis, also known as gap test, identifies disparities between what is delivered and what is required in software testing.” The standard framing compares a present state against a target state; PREVC just makes both states unusually concrete. The target state is the Review output — not the original request, not anybody's memory of the plan, but the specific document a human refined and released through the gate. The present state is the Execution output. The gap is the work remaining, enumerated rather than felt.

That choice of baseline is doing quiet work. Measuring against the Review output means Validation inherits everything Review encoded — the sharpened definitions, the added requirements, the intent boundaries — and it means a defect in that document propagates, which is exactly the argument part 3 made for spending your best hour at that gate. Validation cannot catch an error in its own measuring stick. It can only tell you, precisely, whether the build matches the stick.

Rigour priced to risk

Full traceability sounds heavy, and unpriced it would be. The discipline comes from domains that had no choice: DO-178C, the certification standard for airborne software from RTCA and EUROCAE, and ISO 26262, the automotive functional-safety standard, both mandate requirements traceability — when software failure means physical harm, every requirement carries a receipt, no exceptions, and regulators check the stapling.

Most of us are not writing avionics, and the lesson to take is not the ceiling but the dial: rigour proportional to risk. A production payment API gets the full matrix, every row completed. An internal reporting tool gets spot-checks — trace the five requirements whose failure would actually hurt, sample the rest. The matrix is a form you can fill to any depth; deciding the depth deliberately, per system, is the practice. What I do not accept is depth zero, because depth zero is the single heroic pass this workflow exists to retire — “it seems done” standing in for “it demonstrably matches the spec”.

The exam paper stays out of the classroom

Now the debt from part 6. Test gaming — the agent shaping code around the assertions it can see — survives loop-level review because both the agent and the reviewer are looking at the same test file. The structural fix is to make sure some of the evaluation criteria are criteria the implementing agent has never seen.

The cleanest industrial example I know comes from StrongDM's Software Factory, described in Simon Willison's write-up: behavioural scenarios stored outside the codebase — “similar to a ‘holdout’ set in model training”, as the write-up puts it — so the agents building the product never train against their own exam. Their evaluation runs go further still, exercising the software against a Digital Twin Universe: behavioural clones of Okta, Jira, Slack, Google Docs and kin, so that “works” means works against realistic neighbours rather than against fixtures.

PREVC borrows the principle at whatever scale you have. The Review output itself already functions as a mild holdout — the agent implements package by package, while Validation compares against the whole specification, intent notes and all. But the stronger habit is keeping a slice of acceptance scenarios physically outside the repository the agent works in: for Rosterly, a handful of swap-eligibility scenarios in a separate store, run only at Validation. When those pass, the green means what green should mean. When they fail while the in-repo tests pass — that divergence is the test-gaming detector, and finding it here costs one phase, not one production incident.

The loop tells on itself

The second Validation question — how efficiently did we build it — reads the telemetry the Execution loop left behind. Part 5 promised this moment: every work package recorded its loop count and which breakout constraint fired, and Validation is where that record gets read. The questions are simple. Which packages needed the most iterations? Where did hallucinations cluster? Which packages needed the most simplification — where did the agent over-build most stubbornly? What was the distribution of exits: quality thresholds (convergence) or iteration caps (thrash)?

None of this changes the current release. All of it prices the process. A category of package that consistently needs four loops is not an Execution problem — it is Planning and Review producing breakdowns that category defeats, and that finding belongs to the next cycle, not this one.

Two arrows out

Which is why Validation ends with two feedback loops rather than a report — both of them my extensions to the borrowed skeleton, and the reason I keep insisting this phase is not an audit.

V→E, gap correction. Every gap the matrix found sends its specific work package back into the Code-Review-Simplify loop. Surgical, not wholesale: Rosterly's unrequested endpoint gets a human decision and either a requirement or a deletion; nothing that holds a receipt gets touched. Re-executing everything because something failed is how validation phases make themselves hated.

V→C, efficiency data. The telemetry read — loop counts, hallucination hotspots, simplification frequency, constraint-fire distribution — travels forward to Confirmation, where it becomes process improvement rather than trivia.

The phase's concrete outputs, then: a traceability assessment, a gap list with severities and recommended corrections, an execution-efficiency report, and a go/no-go decision. When the answer is go, the cycle's technical story is over — and its most skipped, most quietly valuable phase begins. Next, Confirmation closes the circle: the retrospective with named suspects, and the release step that writes the next cycle's requirements before it starts.