The Retry Path Nobody Took

Part five walked the live retry path: when a handler throws an infrastructure exception, a Polly policy wrapped around it waits and tries again, in-process, up to three times. That is the retry the estate actually uses. This part is about the retry it doesn't use — a second, complete implementation sitting directly below the first in the same file, wired to a real RabbitMQ retry exchange, and called by nothing. It is my favourite kind of source-reading find, because the author left the reason for abandoning it in a two-line comment, and that comment is worth more than most design documents.

Two methods, one caller

BusSubscriber contains two private handlers. The one that runs, TryHandleAsync, uses Polly for in-process retry — part five. The one that does not run is TryHandleWithRequeuingAsync:

// RabbitMQ retry that will publish a message to the retry queue.
// Keep in mind that it might get processed by the other services using the same routing key and wildcards.
private async Task<Acknowledgement> TryHandleWithRequeuingAsync<TMessage>(TMessage message,
    CorrelationContext correlationContext, Func<Task> handle,
    Func<TMessage, DShopException, IRejectedEvent> onError = null)
{
    try
    {
        await handle();
        return new Ack();
    }
    catch (Exception exception)
    {
        if (exception is DShopException dShopException && onError != null)
        {
            var rejectedEvent = onError(message, dShopException);
            await _busClient.PublishAsync(rejectedEvent, ctx => ctx.UseMessageContext(correlationContext));
            return new Ack();
        }

        if (correlationContext.Retries >= _retries)
        {
            await _busClient.PublishAsync(RejectedEvent.For(message.GetType().Name),
                ctx => ctx.UseMessageContext(correlationContext));
            throw new Exception($"...after {correlationContext.Retries} retries.", exception);
        }

        return Retry.In(TimeSpan.FromSeconds(_retryInterval));
    }
}

This is not a stub. It has the same two-lane taxonomy as its live sibling — domain failures publish a rejected event and Ack, infrastructure failures retry — but it retries through the broker instead of in memory. Look at the exit on the last line: Retry.In(...) is a RawRabbit acknowledgement that hands the message to the retry infrastructure — the orders.retry exchange and the orders.retry_for_..._in_2000_ms queue whose names we decoded back in part two. And it counts attempts using correlationContext.Retries, a field maintained by a dedicated pipe middleware:

private class RetryStagedMiddleware : StagedMiddleware
{
    public override string StageMarker => RawRabbit.Pipe.StageMarker.MessageDeserialized;

    public override async Task InvokeAsync(IPipeContext context, CancellationToken token)
    {
        var retry = context.GetRetryInformation();
        if (context.GetMessageContext() is CorrelationContext message)
            message.Retries = retry.NumberOfRetries;
        await Next.InvokeAsync(context, token);
    }
}

So the full picture is: a retry exchange named by convention, a retry queue whose delay is baked into its name, a pipe middleware that reads the broker's retry count into the correlation context, and a handler that uses that count to decide when to give up. An entire broker-side retry subsystem, present and consistent. And SubscribeCommand and SubscribeEvent both call TryHandleAsync. Nothing calls TryHandleWithRequeuingAsync. The subsystem is dead on arrival — every piece built, none of it reachable.

The comment is the design document

Why build all of it and then not use it? The author told us, in the two comments that bracket the decision. On the dead method:

“Keep in mind that it might get processed by the other services using the same routing key and wildcards.”

And on the live one:

“Internal retry for services that subscribe to the multiple events of the same types. It does not interfere with the routing keys and wildcards.”

Read those together and the whole story falls out, and it reaches back to part two. Because routing keys are derived mechanically from type names and services subscribe with wildcards, a message re-published to a shared retry exchange could be picked up by a different service than the one that failed it. Broker-side retry, in this topology, is not safely targeted — the retry exchange has no idea which consumer needs the redelivery, and the wildcard bindings mean the wrong one might grab it. So the author reached for in-process Polly instead, which retries inside the exact consumer that failed and cannot leak the message sideways. The topology chosen in part two made the elegant retry mechanism unsafe, and the fallback is the one that runs.

That is a real distributed-systems lesson, not a code-quality nitpick: convention-derived routing with wildcards buys you effortless pub-sub and takes away your ability to address a single consumer for a redelivery. The two are the same coin.

What broker retry needs that this topology denied it

It is worth naming what the dead path would have needed to be safe, because it is the whole argument for how the mature framework later re-enabled it. Broker-side retry is only sound when a redelivered message returns to the exact consumer that failed it — which means each consumer needs its own retry queue, bound so that nothing else can pick the message up. In this estate the retry exchange is named per namespace (orders.retry), and consumers bind with wildcards, so a redelivered orders.* message is fair game for any service listening on that pattern. There is no per-consumer isolation, so there is no safe address to send the retry back to. The in-process Polly loop sidesteps the whole problem by never letting the message leave the failing consumer at all.

There is a second, quieter tax. The RetryStagedMiddleware above runs on every message, reading RawRabbit's retry count into correlationContext.Retries — faithfully maintaining a number that only TryHandleWithRequeuingAsync ever reads, and TryHandleWithRequeuingAsync never runs. So every message in the estate pays for a counter that feeds a dead method. It is harmless and it is pure overhead, the kind of thing that accumulates when a capability is abandoned in pieces rather than removed whole.

Should dead code stay?

The maintainer's question is whether TryHandleWithRequeuingAsync and its middleware should have been deleted. Orthodox hygiene says yes — dead code rots, confuses readers, and shows up in coverage reports as a permanent red mark. But I would argue this is the rare case where leaving it was the better call, because of the comment. The abandoned method is a signpost: it documents that broker-retry was considered, implemented, and rejected for a specific, correct reason. Delete the code and you delete the evidence; the next engineer re-derives “why don't we just use RabbitMQ's retry exchange?” from scratch and maybe gets it wrong. This is the same move I have made in my own changelogs — leave the observation where the next person will trip over it rather than in a document nobody opens. The tax is a RetryStagedMiddleware that faithfully maintains a Retries counter no live code reads, and a retry exchange the naming conventions cheerfully name and nothing ever fills.

When the authors rebuilt this as Convey, broker-side retry came back — but with the message brokers redesigned so redelivery targets the right consumer, resolving the exact tension this comment describes. The dead method here is the bug report that fix was written against.

Two parts from now the absences become the theme outright. But the retry subsystem is the first: a whole capability, present in name and shape, that the estate does not actually have. Next, the full absence audit — the outbox, the inbox, the dead-letter consumer — every gap that later became a Convey package.