The AMQP Property the Broker Vetoed

You add caller identity to your published messages, because the consumers keep asking who asked for this and the answer keeps being “an HTTP request, four services ago.” The change is one line — the protocol has a field for exactly this, it is called user-id, and it takes a string. You deploy it. Everything works in the demo. Then somebody logs in.

Part 5 took apart the promise a 202 makes on a use: rabbitmq route. This part follows one feature on that route through its entire life, which lasted four days and left a mark on the gateway that is still there.

Three commits, one evening, and a Sunday

The whole arc is legible in git and the timestamps matter, so here they are:

Eighty-one minutes from shipped to disabled, and four days from disabled to gone.

The addition was three small edits. A parameter on the publishing interface, a property assignment in the client, and one line in the handler:

-            IDictionary<string, object> headers = null)
+            IDictionary<string, object> headers = null, string userId = null)
...
-            properties.Headers = new Dictionary<string, object>();
+            properties.Headers = headers ?? new Dictionary<string, object>();
+            properties.UserId = userId;

— git show 9b08077 -- extensions/Ntrada.Extensions.RabbitMq/Clients/RabbitMqClient.cs

And in the handler, var userId = executionData.UserId; passed straight through to Send. It is exactly the change anybody would make. IBasicProperties.UserId is right there in the client library, sitting next to MessageId and CorrelationId, which the code was already setting. Three properties, one API, one obvious pattern.

Why the broker refused

user-id is not a free-text header. It is one of the very few AMQP basic properties that RabbitMQ validates on the server side: the broker requires the user-id on a published message to match, exactly, the username the publishing connection authenticated as. Publish with a mismatched user-id and the broker closes the channel with a PRECONDITION_FAILED error.

The gateway holds one long-lived connection, authenticated as whatever account is in the rabbitmq: block of the YAML — in every shipped configuration, the broker's well-known default development account. And ExecutionData.UserId is:

UserId = context.Request.HttpContext.User?.Identity?.Name,

— src\Ntrada\Requests\RequestProcessor.cs:58

The authenticated subject of the caller's JWT. Two different things wearing the same word. user-id in AMQP is an assertion about the connection; UserId in the gateway is a fact about the caller. The protocol field exists so that a consumer can trust the publisher's identity, which is precisely why the broker enforces it and precisely why it can never carry an end user through a shared connection.

Notice the shape of the failure, because it explains the four days. When no caller is authenticated, User?.Identity?.Name is null, properties.UserId is null, and an absent user-id is perfectly legal — the broker validates it only when present. So the feature works flawlessly for every unauthenticated request. The estate's own sample authenticates nothing at all: auth.global is false, no route sets auth: true, and the .rest collection beside the sample config sends six requests without a single Authorization header. Every test this repository can perform on that feature passes. It only breaks in a deployment where somebody presents a token, which is to say it only breaks in production.

There is a nastier second-order detail, and it follows directly from part 5. The channel is created and disposed inside Send, around a single BasicPublish that waits for nothing. A channel-level PRECONDITION_FAILED arrives asynchronously; whether it surfaces as an exception on the disposal path or is simply swallowed as the channel goes away is a timing question. Either way the message is discarded by the broker, and the caller may still receive a 202. The one route in the gateway that publishes messages had a failure mode where the only party who learns the message was rejected is the broker's own log.

The bug that rode in with it

The same commit carried a second defect that is more transferable than the first, and it was found faster.

properties.Headers = headers ?? new Dictionary<string, object>();

headers is _options.Headers — a property of the singleton RabbitMqOptions registered in RabbitMqExtension.Add. So when an operator configured extensions: rabbitmq: headers:, the per-message properties object took a reference to the gateway's own configuration dictionary. Four lines later:

properties.Headers.Add(_messageContextHeader, JsonConvert.SerializeObject(context));

That Add mutates the shared configuration object. Request one succeeds and permanently injects a message_context key into the singleton options. Request two calls Add with the same key on a dictionary that already contains it, gets ArgumentException: An item with the same key has already been added, and every request from then until process restart fails.

A works-in-the-demo, dies-on-request-two bug, produced by a ?? that reads like an optimisation. It only fires when headers: is configured, because otherwise headers is null and the ?? allocates a fresh dictionary — so the shipped sample, which configures no headers, would never have shown it.

794cd39, seventy-four minutes later, is the right fix: allocate a fresh dictionary, copy into it, and guard the copy.

-            properties.Headers = headers ?? new Dictionary<string, object>();
+            properties.Headers = new Dictionary<string, object>();
...
             if (headers is {})
             {
                 foreach (var (key, value) in headers)
                 {
+                    if (string.IsNullOrWhiteSpace(key) || value is null)
+                    {
+                        continue;
+                    }
                     properties.Headers.TryAdd(key, value);

— git show 794cd39

The added null-and-whitespace guard is defending against a real thing: a YAML key written with no value binds to null, and the config binder produces it happily. Small, correct, complete.

Shared mutable configuration aliased into per-request state is the most portable bug in this whole estate. It has nothing to do with RabbitMQ. It is what happens whenever a singleton options object exposes a mutable collection and something downstream treats it as its own.

What was left behind

Seven minutes after the headers fix, 101b398 deleted var userId = executionData.UserId; and dropped the argument from the call — the minimal change that stops the bleeding while leaving the API intact. Four days later, on the Sunday, e62db79 removed the parameter from IRabbitMqClient, removed properties.UserId = userId; from the client, and the feature was gone.

Deleting it was the correct call, and deleting it quickly was the right way to make it. Four days from “this does not work” to “it is not in the codebase” is fast for a solo maintainer, and leaving a half-wired identity feature in place — a parameter nothing passes, a property nothing sets — would have been considerably worse than the absence.

The residue is what makes this worth a whole article. After e62db79, the gateway propagates no caller identity into a published message at all. ExecutionData.UserId and ExecutionData.Claims are still populated on every request, and they are still handed to IContextBuilder.Build(executionData) — which is the seam designed to carry exactly this. The only implementation registered is NullContextBuilder:

internal sealed class NullContextBuilder : IContextBuilder
{
    public object Build(ExecutionData executionData) => null;
}

— extensions\Ntrada.Extensions.RabbitMq\Contexts\NullContextBuilder.cs

So the message_context header, which the shipped configuration enables by default, carries the literal string {} on every message. And the supported way to replace it was itself deleted: before ea4f297 (2019-09-22), RabbitMqExtension.Add honoured a context.custom: true flag that told the extension not to register a default, leaving the host free to supply its own builder. That commit replaced the whole block with an unconditional registration of the null implementation and dropped custom: from the options class. The extension point survived; the documented way to use it did not.

The durable lesson is about where identity is allowed to travel. A gateway is the only component in a topology that knows who the caller is, and every hop after it is working from whatever the gateway chose to record. Pick a channel the downstream can read and the infrastructure will not adjudicate — a custom header, an envelope, a claim set serialised into the message body. user-id fails that test on both counts: the broker owns its meaning, and its meaning is not the one you wanted.

Next, a rename that unbound a signing key — and the ten minutes that separated the last edit of the README from the commit that invalidated it.