Idempotency Aspired, Not Enforced

An at-least-once message bus makes one promise and withholds another. It promises the message will be delivered — and reserves the right to deliver it more than once. Every reconnect, every un-acked redelivery, every in-process retry is a chance for a handler to see the same message twice. The bus's half of the bargain is delivery; your half is making the handler safe to run twice. DShop's framework keeps its half and quietly assumes yours. This part reads what that assumption costs when the handler doing arithmetic on stock never got the memo.

The setup: competing consumers, at least once

Every service subscribes to its events on a queue named for the service, so two instances of the same service compete for the same queue — competing consumers, which is the correct way to scale a consumer horizontally. On top of that, the framework wraps every handler in an in-process retry. When a handler throws, the bus subscriber re-runs it with a Polly policy — three attempts, two seconds apart by default:

var retryPolicy = Policy
    .Handle<Exception>()
    .WaitAndRetryAsync(_retries, i => TimeSpan.FromSeconds(_retryInterval));

Between horizontal scaling and this retry, a given message can reach a handler more than once through two entirely ordinary paths: the broker redelivers it after a consumer drops, or the handler half-finishes, throws, and Polly runs it again from the top. Neither is exotic. Both are the normal weather of a message bus. (The mechanism, and a second, dead retry path the framework wrote and never wired, are the companion series' The Retry Path Nobody Took.)

So the estate has arranged for at-least-once delivery on purpose. The question is whether the handlers are ready for it.

Most handlers survive by luck of shape

Read the event handlers across the services and most of them are, in fact, safe to run twice — not by design, but because their shape happens to be idempotent. Customers' ProductCreatedHandler does an AddAsync; ProductUpdatedHandler does an UpdateAsync. Run either twice and the second run overwrites the replica with the same values it already holds. An upsert is naturally idempotent: the end state does not depend on how many times you applied it. A handler that sets state to a value is safe to redeliver; a handler that changes state by an amount is not — and that is the whole fault line.

Storage's handlers fall on the safe side too, because — as Part 5 showed — they re-fetch the authoritative record and overwrite. Re-fetch-and-set is redelivery-proof by construction. The estate is accidentally mostly-idempotent, and you could read a dozen handlers and conclude it had thought about the problem.

Then you reach the one handler that does arithmetic.

The handler that decrements

Reserving stock is ReserveProductsHandler, and it changes quantity by an amount:

public async Task HandleAsync(ReserveProducts command, ICorrelationContext context)
{
    foreach ((Guid productId, int quantity) in command.Products)
    {
        var product = await _productsRepository.GetAsync(productId);
        if (product == null) continue;

        product.SetQuantity(product.Quantity - quantity);   // read, subtract, write
        await _productsRepository.UpdateAsync(product);
    }
    await _busPublisher.PublishAsync(new ProductsReserved(command.OrderId, command.Products), context);
}

Read the decrement: get the product, set its quantity to current − quantity, save. There is no idempotency key, no “have I already reserved for this order?” check, no record that this ReserveProducts was handled. Deliver this command twice and the stock is decremented twice; a two-unit order removes four units from inventory, and nothing anywhere notices. The ProductsReserved it publishes is idempotent-ish downstream, but the local side-effect — the arithmetic on quantity — is applied once per delivery, not once per order.

Now layer the retry on top, because it makes the loop worse than simple redelivery. The Polly policy re-runs the entire method on any exception. Suppose the loop reserves product A, reserves product B, then throws saving product C (a transient Mongo blip). Polly re-runs from the top: A is decremented again, B is decremented again, C is retried. A partial failure mid-loop, retried, double-decrements every product that already succeeded — the retry that was meant to add reliability instead multiplies the side-effect, because the unit of retry (the whole handler) is larger than the unit of work (one product). The reservation saga's compensation, ReleaseProducts, has the identical shape and the identical flaw in reverse: redeliver a release and stock is credited twice.

Aspired, not enforced

Here is the fair framing, because the estate is not careless — it is incomplete in a specific, instructive way. The standard recipe for exactly-once effects over an at-least-once bus is well known: at-least-once delivery plus idempotent handlers, usually backed by an idempotent receiver — an inbox table or a seen-message-id set that drops duplicates before the handler runs. DShop built the first half. It has competing consumers, redelivery, and retry — all the machinery that produces duplicates. It has no inbox, no message-id dedup, no idempotency key on any command; the framework's absence audit (What the Framework Doesn't Have) lists the idempotent receiver among the things it never grew. So idempotency is aspired — the architecture assumes it — and enforced nowhere. It is left to each handler author, and most of them accidentally got it right by writing upserts, and the one who wrote a decrement got it wrong, and nothing in the estate could tell the difference.

To be fair to the design, teaching frameworks routinely stop here; “make your handlers idempotent” is a legitimate thing to leave to the consumer, and Convey — the successor — later added the messaging-outbox and inbox pieces that make it enforceable rather than aspirational. But the lesson DShop teaches by leaving the gap open is the sharper one: an at-least-once bus does not have idempotent handlers; it has handlers that are idempotent by accident until one of them isn't.

The rule of thumb

• If a handler changes state by an amount, it needs an idempotency key. Set-to-a-value handlers survive redelivery for free; add/subtract handlers do not, and stock, balances, and counters are exactly the add/subtract kind.
• Make the unit of retry equal the unit of work. A retry that re-runs a whole loop over already-applied side-effects is worse than no retry. Either make each iteration idempotent, or retry one item at a time.
• At-least-once is a decision you must finish. Turning on redelivery and retry is choosing duplicates; if you don't also add a dedup gate, you have chosen duplicates without choosing to handle them.

Next: CQRS Across a Service Boundary — the service that subscribes to almost every event in the estate, serves read-only queries, and holds nothing you can't rebuild.