The Outbox That Keeps Your Events Honest

Here is the oldest bug in event-driven architecture, written as two lines of perfectly reasonable code:

await _repository.AddAsync(order);        // write 1: the database
await _publisher.PublishAsync(@event);    // write 2: the broker

Two writes, two systems, no transaction spanning them. Crash between the lines and you have an order no other service will ever hear about. Reverse the lines and crash, and you have announced an order that does not exist. I have met both versions in production; the second is worse, because downstream services acted on the phantom.

This is the dual-write problem, and there is essentially one honest solution short of distributed transactions: write the event into the same database, in the same transaction, as the state change — then let something else deliver it later. That is the transactional outbox, and Convey.MessageBrokers.Outbox (from DevMentors' open-source toolkit; I read and run this code, I didn't write it) is one of the cleanest small implementations you can study, because it fits the whole pattern — outbox, background dispatcher, and inbox deduplication — into a package you can read in an hour.

The API is one branch

Convey's own sample Orders service shows the intended usage, verbatim from CreateOrderHandler:

// samples/Conveyor.Services.Orders/Commands/Handlers/CreateOrderHandler.cs
await _repository.AddAsync(order);
var @event = new OrderCreated(order.Id);
if (_outbox.Enabled)
{
    await _outbox.SendAsync(@event, spanContext: spanContext);
    return;
}
await _publisher.PublishAsync(@event, spanContext: spanContext);

IMessageOutbox.SendAsync has the same shape as IBusPublisher.PublishAsync — message, optional ids, correlation context, headers — but instead of touching RabbitMQ it serializes the event and stores an outbox row. The handler returns with the order and its announcement durably in MongoDB. The if (_outbox.Enabled) branch means the same handler works with the feature toggled either way, which is exactly how you want to roll a pattern like this out: config first, faith later.

Enabling it is the usual chassis move:

.AddRabbitMq()
.AddMessageOutbox(o => o.AddMongo())

with an outbox section — enabled, type (whether dispatch reads from memory or straight from the store), expiry, intervalMilliseconds, and collection names for the two tables this pattern needs. Two, because the package implements both halves of the reliability story.

The dispatcher: a timer with a blind spot

Stored events have to reach the broker eventually. OutboxProcessor is a hosted service with a plain System.Threading.Timer:

// Processors/OutboxProcessor.cs
_timer = new Timer(SendOutboxMessages, null, TimeSpan.Zero, _interval);
...
private void SendOutboxMessages(object state)
{
    _ = SendOutboxMessagesAsync();   // fire-and-forget
}

Each tick loads unsent rows, publishes them via the regular IBusPublisher (span context and correlation headers included, so traces survive the detour — part 21), and marks them processed. On publish failure the row simply stays unsent and the next tick tries again. Delivery latency is therefore your configured interval in the worst case; at the sample's 2000 ms, nobody notices.

But look at that discarded task. The timer callback launches the async work and does not await or guard it. If a tick takes longer than the interval — a broker hiccup, a big backlog after downtime — the next tick starts a second, concurrent sweep over the same unsent rows. Both sweeps publish, both mark processed, and your consumers receive duplicates in a burst precisely when the system is already unhappy. A one-field re-entrancy guard would prevent it; the package doesn't have one. It also offers Sequential and Parallel processing modes for ordering within a sweep — but cross-sweep overlap defeats ordering promises anyway.

I want to be fair: duplicates here are a degradation, not a correctness bug, because the whole pattern is at-least-once by design. Which is why the other half of the package exists.

The inbox: idempotency as infrastructure

At-least-once delivery means every consumer must tolerate replays — the broker redelivers after an unacked crash, the outbox double-sweeps, an operator replays a dead-letter queue (part 10). You can chase idempotency handler by handler, or you can do what Convey does: record processed message ids and skip repeats.

On the consuming side, the outbox package hooks message handling so that each incoming message id is checked against an inbox collection; already-seen ids are acknowledged without invoking the handler, and newly processed ids are recorded. This works because the publisher (part 9) stamps every message with a MessageId — uniform stamping paying rent again. The expiry option bounds the inbox's growth: entries older than the window are purged, on the theory that a redelivery arriving days later is a replay you want to notice.

Note the precise guarantee: this is exactly-once processing, not exactly-once delivery. The message still travels more than once; the inbox makes the second arrival a no-op. And the guarantee is only as strong as the id discipline — a producer that regenerates ids on retry (or an operator who republishes with fresh ids) walks straight past it.

Mongo and EF: two stores, two levels of finish

The storage adapters are separate packages, and the difference between them is instructive.

Outbox.Mongo is the finished one. It can wrap the aggregate write and the outbox insert in a MongoDB session transaction (with a DisableTransactions opt-out for standalone servers that don't support them), and it creates TTL indexes so processed rows and expired inbox entries clean themselves up — the database does the janitorial work.

Outbox.EntityFramework stores outbox and inbox rows through your DbContext — which means atomicity comes for free with SaveChanges when you share the context — but there is no cleanup path: the tables grow until you notice, usually via a DBA with questions. There is also an in-memory type for local development, which is exactly as durable as it sounds.

The EF gap points at the one contract the package cannot enforce for you: the outbox is only transactional if the aggregate write and the outbox write actually share a transaction. Use the Mongo repository from one session and the outbox from another, and you have rebuilt the dual-write problem one layer down, with better-looking code. When I audit an outbox adoption, that is the first thing I check.

What I'd take, what I'd add

The pattern earns its keep the first time a deploy interrupts a handler mid-flight and nothing is lost. Convey's rendition gets the essentials right: same-database staging, broker-agnostic dispatch through the normal publisher, trace context preserved, inbox dedup as a partner feature rather than an afterthought, TTL-based hygiene on Mongo.

What I would add before trusting it at serious volume: a re-entrancy guard on the processor tick, cleanup for the EF store, and a metric on outbox depth — because a growing unsent count is the earliest signal that your broker and your database have started disagreeing about reality, and the entire point of this pattern is that they never do so permanently.

Next part: the persistence layer those outbox rows live in — Convey's Mongo repository, and the thinnest Redis integration you will ever see.