What Two Hundred Copies Teach You

Eleven parts ago, this series opened with a claim: infrastructure-as-code libraries fail in a predictable way. Having now walked the whole estate — the five-file chassis, the state design, two generations of pipeline, the glue modules, the naming contract, the Logic App quartets, the escape hatches, the SQL arc — it is time to keep the promise the word honest has been making all along. This is the retrospective: what a couple hundred copies of a good idea actually do to that idea, told as anonymized patterns from an estate I worked on, because every one of these will happen to yours.

Scaffolding metastasizes

The estate's core mechanic was copy-the-template: sixteen golden starters, each new service a copy, fill in the locals. It made stack one fast, stack ten trivial — and it made every template defect a population. TEMPLATE_USER: markers still sitting in production stacks. A typo'd tag value faithfully reproduced across every environment of every copy. Never-filled placeholder pvars files. The half-finished Private DNS template from the module catalog, whose gaps every copy inherited and every team re-debugged alone.

The specimen I keep returning to: a golden template whose prod pvars file set the wrong variable name — so every faithful copy would, on its first production run, resolve an empty agent pool. The failure was deferred, distributed and identical everywhere; the fix was one line, two hundred times.

The lesson is not “don't scaffold”. It is that copying is a release mechanism with no recall. A registry module can ship a fix in a version bump; a template cannot. So: lint templates harder than production code, smoke-plan every environment of a template before it ships, and keep the copied surface as small as possible — every line in a template is a line you can never patch centrally again.

Pinning without governance curates a museum

Everything in the estate was pinned — Terraform CLI per stack, module tags per stack — which is correct, and which quietly produced: CLI versions spanning 0.13.6 to 1.3.5 in one repository, siblings of a single logical application planning under different language semantics, and one module consumed at seven different release tags simultaneously. Which behaviour does your-module have in production? All seven.

Pinning answers “what changed?” — nothing without a diff. But pins without a fleet-wide bump cadence mean every stack ages at its own rate, and the estate becomes a museum where every exhibit runs a different decade. The fix is boring and organizational: pin everything, then schedule the herd migration — a quarterly window where a bot (Renovate-style) raises the bumps and teams merge them. The alternative is upgrading seven years of drift at once, under duress, when a CVE forces you.

Interfaces are forever; so are their typos

Three exhibits from earlier parts belong in the closing argument together. The module whose folder name carries a misspelling into dozens of source strings (Part 5). The twin modules one hyphen apart, both alive because nobody can prove which callers depend on which divergence. The state key still bearing a service's previous name years after the rename (Part 7).

One rule generates all three: an interface becomes permanent the moment it has a second caller — and in a copy-based estate, the second caller arrives the same afternoon. Rename ruthlessly in week one; after that, treat any rename as a migration with an inventory and a checklist, or accept the typo as load-bearing vocabulary.

Secrets in locals

The workspace-map idiom of Part 2 makes it delightfully easy to write any value six times, once per environment. Including, it turns out, API keys. The same convenience that hardcodes a shared plan name hardcodes a credential — reviewed, merged, and then immortal in git history. The estate also carried committed certificate files, and plaintext passwords in commented-out code — a reminder that comments are code as far as a secret scanner is concerned, and that “we deleted it in the next commit” removes nothing.

This is the strongest argument in the estate for two cheap controls: values-by-reference only (locals hold Key Vault references, never materials) and a pre-commit secret scanner plus a one-time history scan. Neither requires anyone's permission to adopt. Relatedly: half-populated environments — prod keys left as "" by the template — are the same failure at lower stakes; strict lookups turn both from a production surprise into a plan error.

The ladder is real, and you can stand on two rungs

The estate's most hopeful fact: the manual re-plan console and the gated plan-artifact pipeline of Part 4 belonged to the same team, a few years apart. Nobody rewrote the world; the riskiest stacks got the mature pipeline first and the rest kept working. Maturity in a large estate is not a migration, it is a gradient — the failure mode is not being on a low rung, it is having no plan to climb.

If I ran this estate today

The concrete list, in the order I would actually do it — feedback first, ceremony later:

1. PR-triggered fmt/validate/plan on every stack. Changes no governance posture; shortens every feedback loop in the estate. Days of work.
2. Secret scanning — pre-commit and history. Days, and possibly the highest-value item on the list.
3. tflint + checkov on the templates, then the estate. The templates first — that is where defects breed.
4. Plan-artifact promotion everywhere, not just SQL. The pattern is proven in-house; make Gen 2 the only generation.
5. Scheduled drift detection — nightly plan -detailed-exitcode fanned across stacks, failures to a dashboard. A thousand small states are a thousand quiet places for drift to live.
6. Automated module-tag and CLI bumps on a cadence, closing the museum.
7. OIDC workload-identity federation to replace standing managed identities on pet Windows agents — with the Part 9 escape hatches slimmed (azapi where possible) so hosted agents become viable at all.
8. Terraform 1.x everywhere, with a state-migration runbook per stack — last, because everything above makes it safer.

Notice what is absent: no rewrite, no re-platforming, no “move it all to a new tool”. The estate's bones — thin roots, per-env state isolation, glue modules, naming discipline — held up. Everything on the list is feedback and hygiene around a design that worked.

The closing thought

Two hundred copies do not primarily teach you about Terraform. They teach you that whatever you build becomes a population, and populations obey different laws than individuals: defects propagate, conventions drift into dialects, interfaces fossilize, and the cost of any fix multiplies by the census. The craft, then, is not writing the perfect stack — it is choosing carefully which fifty lines get copied two hundred times, and building the feedback loops that let a population evolve instead of just accumulate.

That is the series. If you are starting an estate of your own: begin with Part 1's five files, wire the feedback loops from this part before the copies multiply, and be honest in your own retrospective when the time comes. It will come around stack fifty.