One Assignment Freezes Every Trace

Part 8 quoted the async dispatcher's loop and left one line unexamined. Here it is again, with everything else stripped away:

await foreach (var envelope in _messageChannel.Reader.ReadAllAsync(stoppingToken))
{
    try
    {
        _contextAccessor.Context ??= envelope.MessageContext.Context;
        await _moduleClient.PublishAsync(envelope.Message, stoppingToken);
    }
    // ...
}

??= — assign if null. In a loop that runs once per message, for the lifetime of the process.

Every log line produced by every handler downstream of that dispatcher carries a correlation id, a trace id, a request id and a user id, and after the first message they are all wrong. Not missing — wrong, in the most convincing possible way, because they are real values from a real request that really happened.

Where the identity lives

ContextAccessor is the holder:

public sealed class ContextAccessor
{
    private static readonly AsyncLocal<ContextHolder> Holder = new();

    public IContext Context
    {
        get => Holder.Value?.Context;
        set
        {
            var holder = Holder.Value;
            if (holder != null)
            {
                holder.Context = null;
            }

            if (value != null)
            {
                Holder.Value = new ContextHolder {Context = value};
            }
        }
    }

    private class ContextHolder
    {
        public IContext Context;
    }
}

This is the IHttpContextAccessor pattern, copied faithfully: a static AsyncLocal<T> holding a mutable box, with the setter nulling the old box's field before installing a new one. The indirection exists so that a value set deeper in an async call tree can be cleared by an ancestor, which is what makes the ASP.NET Core version safe across request boundaries.

IContext is then resolved through a transient factory:

services.AddSingleton<ContextAccessor>();
services.AddTransient(sp => sp.GetRequiredService<ContextAccessor>().Context);

That is a clever piece of composition and it is worth admiring for a second. The ambient identity crosses DI scope boundaries for free, which matters enormously in this framework, because — as part 14 will show — every dispatcher is a singleton that creates its own scope per operation. A command handler resolved from CommandDispatcher's private scope still sees the HTTP request's identity, without anyone threading a parameter through five layers. The HTTP middleware sets it once:

app.Use((ctx, next) =>
{
    ctx.RequestServices.GetRequiredService<ContextAccessor>().Context = new Context(ctx);;
    return next();
});

(The double semicolon is in the source. It compiles.)

Why ??= behaves differently here than it looks

The instinct on reading x ??= y inside a loop is “it assigns on the first iteration and skips afterwards, because x is now non-null”. That is exactly what happens — and the reason it stays non-null is the part that needs the async execution model to explain.

AsyncLocal<T> is not thread-local. Its value flows with the execution context, which is captured at each await and restored on the continuation. Critically, an assignment to an AsyncLocal inside an async method persists forward through that method's own continuations: the method's execution context is copy-on-write, so a write creates a new context that every subsequent await in that same method inherits.

ExecuteAsync is one async method that runs for the entire process lifetime. The first envelope assigns Holder.Value. That assignment lives in the execution context that flows into the second iteration, and the third, and the ten-thousandth. _contextAccessor.Context is never null again, so ??= never assigns again, and every subsequent message is published under the first envelope's Context.

Meanwhile the correct value was right there. MessageEnvelope is a record with two fields:

internal record MessageEnvelope(IMessage Message, IMessageContext MessageContext);

The envelope was built by AsyncMessageDispatcher, carried through the channel, and is in scope on the line above. The dispatcher reads envelope.MessageContext.Context and then only conditionally uses it, and passes envelope.Message — not the envelope — onward to ModuleClient, which then re-fetches the context from an IMemoryCache keyed on the message object. The envelope you built, carried across the boundary, and then routed around.

What it looks like in production

Say three users hit the API in the same second. Alice completes her customer profile, Bob verifies his, Carol starts a deposit. Three commands, three commits, three events into the channel. The dispatcher wakes up.

Alice's CustomerCompleted is first. Context is null, so it is assigned Alice's — her correlation id, her trace id, her user id. The receivers in Payments and Wallets handle it and log correctly.

Bob's CustomerVerified is second. Context is not null, so nothing is assigned. LoggingEventHandlerDecorator reads _context.RequestId, _context.TraceId and _context.Identity?.Id from the accessor and writes:

Handling an event: customer_verified (customers) [Request ID: <Alice's>,
Message ID: <Bob's>, Correlation ID: <Alice's>, Trace ID: '<Alice's>',
User ID: '<Alice's>]...

The MessageId is Bob's, because that comes from the message-context side table via IMemoryCache rather than from the accessor. Everything else is Alice's. If you filter your logs by Alice's correlation id to investigate her complaint, you get her request plus every message the process handled afterwards — Bob's, Carol's, and everyone else's for as long as the pod lives.

That is worse than having no correlation at all, because a missing field is obviously missing and a wrong field is evidence. Anyone tracing an incident through those logs is being told, structurally and confidently, that Alice's request caused Carol's deposit.

The fix, and the better fix

The minimal change is one character: = instead of ??=. That gives each message its own context for the duration of its publish, and because AsyncLocal writes flow forward, the next iteration overwrites cleanly.

The better change is to stop routing around the envelope. ModuleClient.PublishAsync already re-registers the context against the translated instance:

if (message is IMessage messageData)
{
    var messageContext = _messageContextProvider.Get(messageData);
    _messageContextRegistry.Set((IMessage)receiverMessage, messageContext);
}

It fetches the context from the cache using the message as the key, having been handed only the message. Passing the MessageEnvelope through instead of envelope.Message would remove the cache lookup, remove the dependence on a one-minute sliding expiration, and make the identity flow explicit rather than ambient. The envelope was designed for exactly this and is discarded at the one hop where it would earn its keep.

There is a scoping argument for keeping the AsyncLocal too: the decorators resolve IContext from DI, not from the envelope, so the ambient value has to be correct regardless. Fine — set it per message with =, and keep the envelope as the source of truth that the assignment reads from.

Why nothing catches it

Ask what test would have found this, and the answer is uncomfortable.

A unit test of AsyncDispatcherJob would have to publish two messages with different contexts and assert that the second handler observes the second context. That is a test somebody writes only if they already suspect the bug — nobody writes “assert that the second message has its own correlation id” prophylactically.

An integration test would not find it either, because the shared framework's test projects contain no tests at all. Inflow.Shared.Tests is marked <IsTestProject>false</IsTestProject> and exports five helpers plus a fake broker; Inflow.Shared.Tests.EndToEnd is a single abstract base class. There is not one [Fact] covering the broker, the dispatchers or the context accessor. And the module-level integration tests that do exist run one request at a time, which is exactly the shape that cannot see this: with one message, ??= and = behave identically.

That is the general property of the defect and the reason it is worth writing about. It only manifests on the second message, so every test that publishes one message passes, and every test that publishes many is a test somebody wrote for a different reason. The class of bugs that require sequencing to reveal — first-call caching, ??= in loops, lazily-initialised statics, HttpClient handler reuse — are all invisible to the test shape most people default to.

The cheap defence is not a test. It is making the assignment unconditional, so that there is no first-call special case to get wrong. = has no second-message behaviour distinct from its first-message behaviour, which means one test covers both.

The wider hole this sits in

While we are here, one more thing about correlation in this estate. The middleware that mints correlation ids is:

public static IApplicationBuilder UseCorrelationId(this IApplicationBuilder app)
    => app.Use((ctx, next) =>
    {
        ctx.Items.Add(CorrelationIdKey, Guid.NewGuid());
        return next();
    });

A fresh Guid per request. No inbound header is read. There is no X-Correlation-ID lookup, no W3C traceparent handling, no ActivitySource, no OpenTelemetry package reference, and no health endpoint anywhere in the repository. A correlation id cannot enter this system from outside; it can only be born here. That is entirely consistent for a self-contained monolith demo, and entirely incompatible with the microservices future the rest of the framework is visibly shaped for.

The rule I take from this one: an ambient value that is written once and read everywhere needs its write to be as loud as its reads. ??= is a quiet write. In a loop that lives as long as the process, quiet is the same as wrong.

Next, the outbox that commits somewhere else — a subsystem built to make delivery survive a crash, defeated by a single CreateScope().