What a Homegrown Discovery Layer Owes You

Seven parts ago this series promised honest trade-offs, and along the way it has confessed piecemeal: a cache bug here, a certificate bypass there, an exception type that lies about its cause. This final part collects the whole ledger in one place — because the real lesson of building your own discovery layer isn't any single flaw, it's learning which category each flaw belongs to. Some are debts: unambiguously wrong, payable tomorrow, cheap. Some are trade-offs: costs consciously exchanged for something worth more at the time. And some are graduation signals: evidence you've outgrown the pattern itself. Confusing the categories is how teams either rewrite things that only needed a patch, or patch things that needed a funeral.

The debts — wrong, and cheap to fix

The certificate bypass. DangerousAcceptAnyServerCertificateValidator in the original; a lambda returning true in the rewrite. Every endpoint this library called was an *.azurewebsites.net host with a platform-managed certificate that would have validated perfectly — the bypass defended against a dev-environment problem that no longer existed and disabled TLS authentication for the entire estate's east-west traffic in exchange. It costs one deleted line and a regression test in each environment. This is the item I'd fix before finishing the coffee.

The inverted cache condition. Part 2's confession: GetSecret caches only empty values, so every real secret is re-fetched from the vault on every call, forever — and Part 7 showed it surviving the rewrite verbatim. One flipped !, one unit test (resolve twice, assert one fetch) to keep it flipped.

The thread-unsafe singleton cache. Plain Dictionary mutated from concurrent invocations inside a singleton. ConcurrentDictionary.GetOrAdd is a mechanical substitution that also deletes the check-then-add race.

The lying exception. “No endpoint matched your abbreviation” throws ArgumentNullException (Part 4), and substring routing means an ambiguous name binds silently to registration order. A domain exception that prints ListOfServices, plus a startup guard that no friendly name contains another: an afternoon.

Payloads in error logs. Part 3 logs full request bodies on failure. Wonderful in integration, a retention and privacy liability compounding quietly in production. Log a correlation id and a payload hash; the transaction store already holds the body under governance.

Notice what the debts have in common: none is architectural. Every one is a local fix a competent reviewer would demand — which means every one is really a process finding. This library shipped with no tests and, evidently, review that didn't look at the unglamorous lines. The pattern was sound; the safety net was missing.

The trade-offs — costs I'd sign for again (mostly)

No liveness, no health. A registry record says where a service should be, never whether it's up. A real registry heartbeats; this one can advertise a corpse indefinitely. Signed willingly: on Azure PaaS the platform owns instance health, the estate had monitoring where it mattered, and the alternative was operating a liveness system for a problem that occurred approximately never.

Process-lifetime caching with no invalidation. Key rotation lands only when consumers recycle (Part 2). The producer side rotates keys on infrastructure applies; nothing pushes the news. Signed, with more hesitation: Function Apps recycle often, rotations were rare and human-watched. But this is the trade-off closest to expiring — a TTL on cache entries (even a blunt thirty minutes) buys rotation-tolerance for twenty lines, and I'd add it before the next rotation, not after.

The naming convention as the only contract. <service>-<env>, {"URL","key","type"} — no schema version, no compiler, no registry service enforcing agreement between the Terraform that writes and the .NET that reads. Renames fail silently; a too-new type value throws in consumers. Signed: this is the pattern. The mitigation isn't machinery, it's the version-skew discipline from Part 2 and treating record-shape changes as estate-wide changes. The producer-side article reached the same verdict from the other direction.

Newtonsoft JObject in public signatures; two serializers in one package. Period-correct choices that aged into coupling. Not worth a breaking change on their own; worth fixing in any future generation.

A tiny bespoke library instead of a product. The whole thing is a few hundred lines a .NET developer reads in an afternoon, with zero new infrastructure to run, patch, or pay for. That was the entire thesis, and production years validated it. The cost — everything in this article is your problem, with no vendor roadmap coming to fix it — is the honest price of the smallness.

The graduation signals

The pattern was right for its estate. Here is what would tell me an estate has outgrown it — each signal pointing at a different successor, because “graduate” doesn't mean “adopt a mesh”, it means pay for exactly the layer that started hurting:

• Rotation cadence outruns process recycles — keys rotating on a schedule, consumers holding stale ones. You want push-based configuration: Azure App Configuration with sentinel-key refresh, keeping Key Vault references underneath. The record shape survives; the transport for change is what you're buying.
• Routing policy outgrows the record — the moment you want per-consumer authorization, throttling, response caching, or to move a backend without re-registering it, the discovery layer is trying to become a gateway. Put API Management in front and let records point at one stable façade; the registry then only ever changes when the gateway moves, which is never.
• The calling-convention switch grows a third and fourth arm — more runtimes, gRPC, event-driven invocation. That's Dapr's service-invocation building block: naming, mTLS, retries and observability as a sidecar, on Container Apps or AKS.
• You actually need instances — the day the estate runs replicated self-hosted workloads where which instance matters, you have the problem meshes solve, and none of the patterns in this series apply. That day, on this estate, never came.
• And the quiet one: resilience policy wants to be uniform. When every team is hand-rolling retries around InvokePostApiAsync, centralizing Polly/Microsoft.Extensions.Resilience pipelines in the shared client is the cheapest graduation on this list — it's ten lines in Part 5's AddHttpClient call, and it should have been in the v1 rewrite.

What it owes you

If you build one of these — and below a few hundred services on PaaS, I'll still argue you should consider it before renting a control plane — the layer owes your estate four things this series has been circling the whole time. Legibility: any engineer can read the entire mechanism in a sitting; the moment your discovery layer needs its own on-call rotation, it has failed this test. Honesty about its edges: no liveness, no push invalidation, convention-enforced naming — written down, taught, and re-decided on purpose once a year rather than re-discovered in incidents. A safety net proportional to its blast radius: this library sat under every service in the estate and had zero tests; the bugs that survived — the inverted cache, the bypass — were exactly the ones tests and review catch and architecture doesn't. And a successor story: the graduation signals above, agreed before you need them, so the debate happens over criteria instead of over an outage.

The estate got the first two and paid, in small installments, for skipping the third. The fourth is this article.

That's the series. A registry made of secrets (the producer half), a resolver made of three environment variables and a dictionary, an HTTP layer that hides two calling conventions behind one enum, a convention that turns class names into dependencies, a NuGet pipeline that made it a platform, a logger that eats its own cooking, and a rewrite that shows what pain-scoped refactoring fixes and what it faithfully preserves. No mesh. It held. Knowing why it held — and exactly where it wouldn't have — is the part worth taking with you.