The Logger That Discovers Itself

Every distributed estate eventually grows a second logging system. The first one — Application Insights, in our case — answers the operator's questions: exceptions, latencies, dependency maps. The second answers the business's questions: what exactly did we send to the pricing system for transaction 4711, and what came back? Those payloads need to live somewhere cheap, durable, and organized by transaction rather than by timestamp — which is to say, blob storage.

The estate's answer was a dedicated logging function: POST it a JSON document and a transaction id, and it writes the document to the right place in blob storage. Every service in every tier logs through it. Which raises the question this article exists for: how do all those services find the logger?

They look it up in the registry, of course. The logger is a service like any other — one Key Vault record, {"URL","key","type"}, name plus environment. And the component that does that lookup ships inside the discovery library itself.

LogToBlob

namespace Platform.ServiceDiscovery;

public interface ILogToBlob
{
    Task<HttpResponseMessage> LogDetails(string transactionId, JObject logDetails);
}

public class LogToBlob : ILogToBlob
{
    public async Task<HttpResponseMessage> LogDetails(
        string transactionId, JObject logDetails)
    {
        string loggerServiceName = Environment.GetEnvironmentVariable("SystemBlobStorage");
        string blobMethodName    = Environment.GetEnvironmentVariable("SaveLogToBlobMethod");
        if (string.IsNullOrWhiteSpace(loggerServiceName))
            throw new Exception("Logging subsystem name is missing from configuration.");
        if (string.IsNullOrWhiteSpace(blobMethodName))
            throw new Exception("Logging method name is missing from configuration.");

        var queryParams = new Dictionary<string, string>
        {
            { "transaction_id", transactionId }
        };

        var kvSettings = new KeyVaultSetting();
        Instance logger = kvSettings.GetSubSystemInstance(loggerServiceName);

        // POST the log document to the resolved logging function,
        // x-functions-key from the record, transaction_id on the query string
        // ... (the HTTP mechanics of Part 3) ...
    }
}

Two configuration values, and note what they are: not a URL and not a key, but a logical service name and a method name. SystemBlobStorage says which registry entry is the logger; SaveLogToBlobMethod says which operation on it to call. The actual location and credential come from the registry at runtime, same as every other dependency. Point dev and prod at different logger deployments by changing nothing — the environment suffix from Part 2 does it.

Making even the method name configuration rather than a constant looks like over-engineering until you've lived through renaming an HTTP route on a function that a hundred deployed packages call by hard-coded name. The pinned-version NuGet distribution from Part 5 means a constant baked into the library propagates at upgrade speed — months — while an app setting propagates at deploy speed. For the one route the library itself dials, that difference justified the setting.

The thread through everything

The transaction_id query parameter is the piece that made this logger worth having. Every business transaction entering the estate got an id at the edge, and every service passed it along. When a process-tier orchestrator called four subsystems via the proxy, each hop logged its request and response documents under the same id — so the blob container accumulated, per transaction, the complete conversational record: what was asked, what each system answered, in order.

Support lived on this. “Why did this quote come out wrong” stopped being a log-trawling expedition across five Application Insights instances and became: fetch the transaction's folder, read the payloads. The Error envelope from Part 4 — code, message, additional_message, uniform across subsystems — landed in the same documents, so a failure's shape was recognizable no matter which tier produced it. None of this is sophisticated. All of it is the difference between observable and explainable.

The posture on failure is the one place this component is strict where the rest of the library is lenient:

HttpResponseMessage response = await client.SendAsync(request);
if (!response.IsSuccessStatusCode)
{
    throw new Exception("Unable to log to blob: "
        + response.Content.ReadAsStringAsync().Result);
}

The transport layer in Part 3 logs failures and returns the response, leaving judgment to the caller. LogToBlob throws. That's a real policy decision hiding in an exception: it declares the audit record a required effect of the transaction, not a best-effort nicety — if the estate cannot write the record, the operation fails loudly rather than proceeding unaudited. For payloads that exist to answer compliance-shaped questions, I still think loud is correct. But notice the flip side: the logging function is now availability-coupled to every transaction in the estate, a quiet single point of failure that no architecture diagram showed. (A degraded path — fall back to Application Insights and continue — is what I'd design today, and the .Result inside that error path is a sync-over-async blocking call the rewrite cleaned up.)

Bootstraps and other snakes' tails

A discovery library that contains a component that uses discovery invites two suspicious questions, and both deserve straight answers.

Is there a circularity? No — but only because of an asymmetry worth making explicit. The logging function itself writes to blob storage using its own storage binding; it does not log through LogToBlob. The recursion terminates at the first hop because the logger's own audit trail is ordinary platform telemetry, not transaction documents. If the logger had consumed the library's logging path, a logging failure would have tried to log its failure through the failing logger. Nothing in the code prevents that configuration; only the deployment convention does. Self-referential systems always have one of these — a base case enforced by discipline rather than structure — and the honest move is to know where yours is.

What must exist before what? The registry vault and its record for the logger must exist before any service can audit anything — which they do, because the Terraform half writes registry records as part of deploying each service, logger included. The bootstrap order is: vault, then logger (whose deployment registers it), then everything else. The estate's provisioning pipelines encoded that order implicitly. Nobody ever wrote it down, and the day someone stood up a fresh environment from scratch, the implicitness was the bug — services deployed before the logger's record existed threw on their first transaction. Fail-fast worked as designed; the sequencing knowledge lived in one person's head.

One implementation wart to name, because it undercuts a principle this series has been building: LogToBlob constructs new KeyVaultSetting() directly instead of receiving IKeyVaultSetting from DI. That means it bypasses the singleton — its resolutions populate a private, per-call cache rather than the shared one, so every LogDetails call re-resolves the logger's record from the vault (construction, then one fetch, then the object is discarded). Correct behaviour, needless chatter, and untestable to boot — you can't fake the vault under it. It's the one class in the library that doesn't drink its own DI champagne; the v1 generation fixed exactly this by injecting the HTTP service and vault client properly.

Which brings the series to that second generation. The library you've seen across five parts — synchronous vault reads, out parameters, a fresh HttpClient per call, certificate validation waved through — was rewritten top to bottom while dozens of consumers depended on the original. How it shipped both generations in one package without breaking anyone is next: Rewriting a Live Library in a v1 Namespace.