One Correlation ID Across Three Transports

The first production incident in any newly distributed system follows a script. Something went wrong for one user, the evidence is spread across five services' logs, and someone asks the innocent question: “can we see everything that happened for that request?” If the answer is no, the next four hours are spent grepping timestamps and guessing. Correlation IDs exist so that question has a one-query answer: mint an identifier at the front door, carry it through every hop, log it everywhere.

Carry it through every hop is the hard part, because every transport carries context differently — and forgetting one hop doesn't fail loudly. It just leaves a gap you discover mid-incident. FeedR, the open-source DevMentors sample this series dissects, is a perfect specimen for studying this: it implements correlation across three transports (HTTP, Redis pub/sub, Pulsar), gets two of them right in instructive ways, and silently loses the thread at the third. Episode 10 of their video series added this feature; reading its implementation end-to-end is the cheapest observability education I can offer you.

Hop 1: born at the gateway

The ID originates in the YARP gateway (part 2), as a request transform applied to every proxied call:

transforms.AddRequestTransform(transform =>
{
    var correlationId = Guid.NewGuid().ToString("N");
    transform.ProxyRequest.Headers.AddCorrelationId(correlationId);
    return ValueTask.CompletedTask;
});

AddCorrelationId is a FeedR.Shared extension that stamps a correlation-id header on the outgoing proxy request. The gateway is the correct birthplace — it's the one component every external request crosses, so every request enters the interior mesh already labelled.

As noted in part 2, this transform always mints a fresh ID, discarding anything the caller sent. Defensible (don't trust external input as your primary trace key) but it forecloses a useful capability: a client that retries can't tag its retries as related, and a mobile app can't hand you the ID it showed the user in an error dialog. The pattern I use in real gateways: accept an inbound ID into a separate field (client-correlation-id), always mint your own authoritative one, log both.

Hop 2: HTTP middleware and the Items dictionary

Every FeedR service calls app.UseCorrelationId(), a four-line middleware from FeedR.Shared.Observability:

public static IApplicationBuilder UseCorrelationId(this IApplicationBuilder app)
    => app.Use(async (ctx, next) =>
    {
        if (!ctx.Request.Headers.TryGetValue("correlation-id", out var correlationId))
        {
            correlationId = Guid.NewGuid().ToString("N");
        }

        ctx.Items["correlation-id"] = correlationId.ToString();
        await next();
    });

Read from the header, fall back to minting (so a service called directly, bypassing the gateway, still has an ID), then park the value in HttpContext.Items — the per-request scratch dictionary that lives exactly as long as the request. A matching GetCorrelationId() extension reads it back anywhere an HttpContext is reachable.

Items is the right container and the wrong final destination. Right, because it's request-scoped by construction — no async-local subtleties, no leakage between requests. Wrong as an endpoint, because nothing in the sample's logging pipeline reads it: FeedR's log lines interpolate the correlation ID manually in the two or three places an author remembered to. The production-grade move is one more middleware line — using (logger.BeginScope(new { CorrelationId = correlationId })) around next() — so every log statement in the request's execution, in every class, carries the ID without any author remembering anything. Scopes are the difference between “correlation is available” and "correlation is ambient", and ambient is the only version that survives contact with a team.

Hop 3: the gap

Now follow the data. The pricing tick is born in the quotes service — inside PricingBackgroundService, in a generator loop that no HTTP request is driving. It's published to Redis as a bare serialized CurrencyPair(Symbol, Value, Timestamp). Look at that record's fields again. No correlation ID. The IStreamPublisher interface (part 4) has no envelope, no headers parameter, no metadata — a topic and a payload is all it can say.

The aggregator receives the tick, and when its handler decides to place an order (part 7), the Pulsar publisher tries to recover the trace context:

var correlationId = _contextAccessor.HttpContext?.GetCorrelationId()
                    ?? Guid.NewGuid().ToString("N");

IHttpContextAccessor in a background service's call path returns null — there is no ambient HTTP request; the tick arrived over Redis. So the fallback fires and mints a brand-new GUID. The code runs, the logs fill with plausible-looking IDs, and the chain is severed: the ID the gateway stamped on POST /pricing/start, the ID in the quotes service's request log, and the ID on the OrderPlaced event are three unrelated values. Ask “show me everything that happened because of this start request” and the answer stops at the Redis hop.

This is my favorite bug-that-isn't-a-bug in the whole repository, for three reasons. First, it's invisible in every happy-path test — correlation gaps only surface when a human tries to follow a thread. Second, it's not a coding error; it's an architecture gap: the transient tier's contract simply has no slot for context, so no amount of careful call-site code can carry it. Third, the fix forces a genuine design decision rather than a patch:

• Put context in the payload — add a CorrelationId field to every streamed record. Works, pollutes every domain type with plumbing, and every new record can forget it.
• Envelope the transient tier too — make IStreamPublisher wrap payloads the way MessageEnvelope<T> wraps messages (part 7). The symmetric, honest fix; costs a breaking change to the seam and a few bytes per tick.
• Question the premise — for tick data, per-request correlation may be the wrong question. A tick isn't caused by a request; it's caused by a generator that a request once started. Arguably the right lineage is causationId (the start command) plus per-tick identity, which is a different feature than correlation.

That third bullet is the deep lesson: correlation IDs model request-shaped causality, and streaming systems aren't always request-shaped. The gap in FeedR isn't sloppiness — it's the exact point where the request/response mental model stops fitting the architecture, rendered in code.

Hop 4: metadata done right

Where a durable envelope does exist, FeedR does the textbook thing. The Pulsar publisher puts the correlation ID in message properties — transport metadata, outside the payload:

var metadata = new MessageMetadata
{
    ["custom_id"] = Guid.NewGuid().ToString("N"),
    ["producer"] = _producerName,
    ["correlationId"] = correlationId,
};
await producer.Send(metadata, payload);

And the subscriber surfaces it into the typed envelope the handler receives:

handler(new MessageEnvelope<T>(payload, message.Properties["correlationId"]));

The notifier then logs the order with its correlation ID — the last hop displays the thread it was handed. Payload stays pure domain data; context rides where infrastructure can see it (dead-letter tooling, topic browsers, and tracing systems can read properties without knowing your types). If the aggregator had a real ID to give, everything from here on would just work. The plumbing is correct; it's fed a broken value.

What W3C Trace Context would change

FeedR hand-rolls all of this, which is exactly right for a sample whose job is to show you the moving parts. In 2026, production .NET gives you most of it for free, and it's worth mapping the hand-rolled version onto the standard one:

• ASP.NET Core and HttpClient already propagate W3C traceparent headers automatically through Activity — the HTTP hops in FeedR would correlate with zero code today.
• The hard hops are the same hard hops: no standard propagates context through Redis pub/sub for you. OpenTelemetry's answer is explicit Propagator.Inject/Extract calls around your serialization — which is FeedR's envelope decision wearing standard clothes. The architecture question doesn't disappear; it just gets a spec-shaped API.
• traceparent carries a hierarchy (trace ID → span ID → parent) rather than a single flat GUID, which is precisely the correlationId + causationId split the tick-lineage discussion wanted.
• The interesting corner: both quotes' Program.cs and the gateway transform actually reference Activity.Current in the source — vestigially, values captured and unused. The episode flirted with the standard mechanism and shipped the manual one, and honestly, for teaching purposes I think that was the better call. You can see a header named correlation-id; Activity ambient magic is invisible until you know what to look for.

My rule for real systems: use OTel/Activity as the mechanism, but only after making every hop's propagation explicit and testable — an integration test that publishes through the seam and asserts the context arrives. If FeedR had that one test, the Redis gap would have been a red bar instead of a surprise.

The audit exercise

Here's the takeaway shaped as homework, because this one pays for itself: list every transport in your system — HTTP, queues, pub/sub, scheduled jobs, database-polling workers, webhooks. For each, answer two questions: where does trace context ride? and what happens when the hop starts from no ambient context? Every “um” in your answers is a gap like FeedR's aggregator — currently invisible, guaranteed to matter during exactly the incident where you can least afford it. FeedR compresses that whole audit into six services you can read in an afternoon, gap included. That the gap ships in the sample is not a flaw in the curriculum. It is the curriculum.

Next, part 9 turns to testing — specifically the problem that makes most teams give up on testing event-driven code: how do you assert that a message was published when there's nothing to await? FeedR's answer involves WebApplicationFactory, a TaskCompletionSource, and a test that quietly rides real Redis.