Everything Observed, Nothing Survivable

An estate with Jaeger, Prometheus, Grafana, Seq, Consul and Vault, and no circuit breaker, is a shape I have walked into more than once. It is not incompetence. Observability is additive — you install a container, add a package, and the dashboards fill up with something. Resilience is subtractive: every timeout, bulkhead and breaker is a decision to fail earlier than you otherwise would, and nobody ever got applause for a request that was rejected on purpose.

Part 14 closed the estate lens. This is the retrospective — what fourteen parts of source-reading actually taught, and where I would and would not repeat any of it.

The instrumentation is genuinely good

Let me be specific, because this half of the ledger is real work.

Serilog with structured CorrelationId enrichment in all eight units, console, file, Seq and ELK sinks configurable per environment, excludePaths covering /, /ping and /metrics, and a twelve-entry excludeProperties scrub list replicated identically everywhere. Jaeger with the RabbitMQ plugin, so traces cross the bus rather than stopping at the publish. Prometheus on /metrics for every unit, with a correctly built RabbitMQ image exposing the Prometheus plugin on 15692.

And the piece that ties it together, which I rate above anything else in the estate: one correlation identifier spans HTTP and AMQP. CorrelationIdFactory holds it in an AsyncLocal; LogContextMiddleware pushes it into Serilog's LogContext; CustomProxyHttpClientFactory stamps x-correlation-id on every proxied request; MessagingMiddleware attaches it to the AMQP message. That is the hard part of distributed tracing and most systems never finish it.

Two more things earn their place. The { code, reason } error envelope, derived automatically from exception type names — AdNotFoundException becomes ad_not_found via Underscore().Replace("_exception", ""), memoised in a ConcurrentDictionary — is the same shape across HTTP responses, the AdActionRejected bus message and the gRPC ActionRejected proto. One error contract, three transports, no manual mapping. That is a deliberate estate-wide convention and it is the best piece of API design here.

And the rejected-event mechanism itself. IExceptionToMessageMapper turns a domain exception thrown by a consumer into a published event, which the Pusher then streams to the originating user's browser. That is how you close the loop on a 202 Accepted — an asynchronous failure travelling back to the person who caused it over a completely different transport. Most systems that hand out 202s never build it. Trill built it, and then the correlating id was missing from the event so the saga could not use it (part 12) and the fan-out was a point-to-point queue so only one browser got it (part 9). The idea is better than the wiring, which is the estate's recurring sentence.

The survivability column is empty

grep -r 'Polly\|CircuitBreaker\|WaitAndRetry\|Bulkhead' across all eleven repositories: zero hits. The only retry configuration anywhere is rabbitMq.retries: 3, retryInterval: 2 — six seconds of broker connection tolerance — and Convey's internal httpClient.retries: 3.

grep AddHealthChecks\|MapHealthChecks: zero. No healthcheck: on any container in either compose file. No depends_on anywhere.

grep -r 'x-dead-letter': zero. No poison-message parking, anywhere.

grep prefetch: zero. No consumer concurrency limit, no flow control.

Prometheus sets evaluation_interval: 5s and declares no rule_files:; there is no Alertmanager in either compose file. Grafana runs with no dashboards, no datasource provisioning and no volume, so it starts empty every time. sampler: "const" with no param means 100 per cent trace sampling — fine for a demo, a cost incident at scale, and worth naming as the classic default that ships to production.

The concrete consequence: Ads calls Users and Stories over HTTP with no timeout, no breaker and no bulkhead, so a slow Users service becomes a thread-pool problem in Ads and there is nothing to stop it propagating. There is no failure isolation anywhere in this estate.

Twelve test projects, zero test files

This is the finding that explains most of the others. There are twelve test projects across the microservice estate — unit and integration pairs for five services, an integration project for the gateway, a unit project for the Web. Every one has xUnit, Microsoft.NET.Test.Sdk and coverlet.collector wired.

Not one of them contains a single .cs file. I checked all twelve.

And scripts/test.sh, byte-identical in nine repositories, is:

#!/bin/bash
dotnet test

which exits 0. The estate ships a green test command that asserts nothing. Every defect in this series — the discarded route values, the character-count Content-Length, the point-to-point fan-out, the unreachable compensation, the dropped Role field — is the kind of thing one test would have caught, and there was never anywhere for that test to live.

Nor could much of it be run. Every Dockerfile pins SDK 3.1 against net5.0 projects, so no part of this estate has built as a container since April 2021. Nine dockerize.sh scripts branch on $TRAVIS_BRANCH for a CI system that does not exist here. The only paths that have ever worked are the host-process ones.

The secrets lesson, stated as a pattern

The Users service ships a JWT signing certificate in certs/, and the private half is git-tracked along with the public one. The passphrase sits in plaintext in appsettings.json beside the path to the file. The public certificate is byte-identical in all eight service repositories, used for signature validation.

It is a self-signed CN=localhost certificate in a public teaching repository, so the practical risk is nil and I will not reproduce anything from it. The pattern is what matters, and it is exactly what leaks in estates where the risk is not nil:

• A private key and its passphrase committed together means the passphrase provides no protection at all. Anyone with the repository has both halves.
• A certificate valid for twelve months is a rotation story, and there is no rotation story here. This one expired on 2021-03-01 — six weeks before the estate's final code commit. Nobody noticed, because nobody was running it by then.
• Eight byte-identical copies of a trust anchor means rotating it is an eight-repository, hand-edited change with no tooling and no way to verify completeness.

Nothing here would have looked different if the key had mattered. That is the whole lesson: the shape of the mistake is identical whether the certificate protects a demo or a payroll system, and the only variable is who finds it.

The same paragraph covers the default credentials — RabbitMQ guest/guest, a Seq API key and a Vault dev root token, all committed and all consistent across nine appsettings.json files. Fine for a demo; worth naming as the default-credential pattern, because the day one of those files is copied into something real is the day the default becomes production.

When not to build it this way

Nine deployable units for a Twitter clone with five domain concepts is more distribution than the domain asks for, and the estate demonstrates the bill precisely.

Do not distribute before you have the tooling to distribute. Roughly sixty files across this estate differ from their siblings only by a namespace: MessageBroker.cs in five repositories with a two-line diff between two of them, CorrelationIdFactory.cs in eight, LogContextMiddleware.cs in eight, ContractAttribute.cs in four, build.sh and test.sh byte-identical in nine, the trust anchor in eight, both .proto files in two apiece. There is no private feed, no submodule, no shared project, no template. The distribution mechanism for shared code in this estate is the clipboard, and every one of those copies is a future divergence nobody will be told about.

Do not split a domain across repositories without a contract mechanism. Part 13 found two drifted contracts out of fourteen. That is a good hit rate for a human and an inadequate one for an interface.

Do not adopt a saga until the events carry correlating identifiers. The single design decision that killed Trill's compensation apparatus was AdActionRejected(string reason, string code) — a rejection that cannot say what it rejected.

And do not let configuration stand in for capability. Twenty-one components in this estate are implied by settings or package references and absent from source. Redis is configured in seven units, AddRedis() is called in six, and exactly one — Timeline — opens a connection. The other five carry a startup dependency on infrastructure they never use.

The version of Trill I would actually build for this domain is one process with module boundaries, one database with schema separation, and the messaging kept in-process behind the same IMessageBroker interface — so that the day a module genuinely needs its own deployment cadence, extracting it is a transport change rather than an architecture change.

Which is not a hypothetical, and it is not my idea. The same author built exactly that, three months later, as a deliberate second implementation of the same finished product.

What I would keep

Not everything here is a caution. Four things I would lift wholesale:

1. The Extensions.AddCore() / UseCore() convention. One fluent chain, one file per service, and you know the unit's entire capability set in thirty seconds. It is the single best legibility decision in the estate.
2. The { code, reason } envelope derived from exception type names, shared across every transport. Almost free, and it makes clients writable.
3. The correlation identifier spanning HTTP and AMQP. Do this before you install Jaeger, not after.
4. The rejected-event path back to the user. If you hand out 202 Accepted, you owe the caller a way to learn that the work failed. Build the return channel, and put a correlating id on it.

The one-line summary of fifteen parts: Trill is an estate that is easy to observe and impossible to operate, and the gap between those two is exactly the set of decisions that only get made when something is running in front of users. Everything the author could build from a design instinct is here and mostly right. Everything that comes from an incident is missing.

There is one more reading to do, and it is the most interesting artefact in the whole estate: the same product, by the same author, rebuilt three months later as a modular monolith and maintained in lockstep with this one. Two implementations of a finished system is a comparison you almost never get. It starts at the same app, built twice.