The Outbox With Transactions Turned Off

Part 10 ended on a cliff: every replica in the estate is only as trustworthy as the events that feed it, and every one of those events passes through Convey's outbox. This part opens the drawer. The outbox pattern exists to solve exactly one problem — the dual write, where a service must change its own state and announce the change, and a crash between the two leaves the system lying to itself. Pacco enables the pattern in seven services. Then, in the same twelve lines of configuration, every single one of them switches the solution off.

Here is the outbox section from Pacco.Services.Orders/src/Pacco.Services.Orders.Api/appsettings.json; the other six are the same block with the same flag:

"outbox": {
  "enabled": true,
  "type": "sequential",
  "expiry": 3600,
  "intervalMilliseconds": 2000,
  "inboxCollection": "inbox",
  "outboxCollection": "outbox",
  "disableTransactions": true
}

Orders, Parcels, Availability, Customers, Deliveries, Vehicles, Identity — seven appsettings files, seven "disableTransactions": true. The only bus-speaking services without the flag are Operations and OrderMaker, and they escape it by configuring no outbox at all. There is no service in the estate running the outbox with its transactions on.

What the pattern promised

Follow a publish through Orders to see what the outbox is for. Handlers never touch the bus directly; they call the service's MessageBroker, and its PublishAsync (in Pacco.Services.Orders.Infrastructure/Services/MessageBroker.cs) forks on one condition:

var messageId = Guid.NewGuid().ToString("N");
if (_outbox.Enabled)
{
    await _outbox.SendAsync(@event, originatedMessageId, messageId, correlationId,
        spanContext, correlationContext, headers);
    continue;
}

await _busPublisher.PublishAsync(@event, messageId, correlationId, spanContext,
    correlationContext, headers);

With the outbox enabled, an integration event is not published at all — it is saved, as a serialised document in the service's own Mongo database, in the same store as the aggregate it describes. A background OutboxProcessor wakes every two seconds (intervalMilliseconds), reads the unsent rows, publishes them to RabbitMQ, and marks them processed. The point of this indirection is atomicity: because the business write and the event write hit the same database, they can share a transaction. Either the order changed and the event is queued, or neither happened. That is the entire value proposition — Guaranteed Delivery bought by making two writes into one.

disableTransactions: true deletes the word “one” from that sentence.

The method that reads its own configuration

The consuming side makes the loss precise. Every command and event handler in these services is wrapped by an OutboxCommandHandlerDecorator/OutboxEventHandlerDecorator pair, which routes the handler through Convey's MongoMessageOutbox.HandleAsync. Here is that method from the framework's netcore3.1 branch — the 0.4.* line Pacco floats on — trimmed to its spine:

if (await _inboxRepository.ExistsAsync(m => m.Id == messageId))
{
    return; // already processed
}

IClientSessionHandle session = null;
if (_transactionsEnabled)
{
    session = await _sessionFactory.CreateAsync();
    session.StartTransaction();
}

try
{
    await handler();                       // your write + your outbox insert
    await _inboxRepository.AddAsync(new InboxMessage
    {
        Id = messageId,
        ProcessedAt = DateTime.UtcNow
    });

    if (session is {})
    {
        await session.CommitTransactionAsync();
    }
}
catch
{
    if (session is {})
    {
        await session.AbortTransactionAsync();
    }
    throw;
}

Read it slowly, because the flag changes every line's meaning.

• The inbox check first — Convey ships an Idempotent Receiver. The AMQP message id of every consumed message is recorded in the inbox collection, and a redelivery with a known id is skipped before your handler runs. This is real, and it works.
• The session that never starts — with disableTransactions: true, _transactionsEnabled is false and session stays null. Everything after that point still executes, but the is {} guards turn the commit and the abort into no-ops. The transactional outbox degrades, silently and by configuration, into a sequenced outbox: the same writes, in the same order, with nothing binding them together.
• Three writes, no rope — inside handler() sits the aggregate write (a ReplaceOneAsync) and, via MessageBroker, the outbox insert; after it comes the inbox insert. Three separate Mongo operations where the pattern's diagram shows one atomic box.

Three crash windows open up, in ascending severity:

1. After the outbox insert, before the inbox insert. The broker redelivers; the inbox has no record; the handler runs again. This is the window the pattern expects — at-least-once delivery — and it is survivable if handlers tolerate duplicates. Hold that thought.
2. After the aggregate write, before the outbox insert. The order changed state and no event will ever say so. Orders' own document says Approved; the outbox is empty; Parcels, Deliveries and the rest carry on believing the old fact forever. The exact ghost the pattern exists to exorcise, back in the house.
3. Both windows at once, at scale. Because nothing in the estate monitors the outbox or reconciles projections, neither failure announces itself. The system does not go down; it quietly forks into services that remember different histories — the part 8 disease, this time by infrastructure rather than by a domain bug.

To be fair, the compose file explains it

Before treating the flag as negligence, look at how the estate runs. The umbrella repo's mongo-rabbit-redis.yml starts Mongo as a single mongo container — one mongod, no --replSet, no init script. MongoDB multi-document transactions require a replica set; on a standalone server, StartTransaction throws the moment a session is used. With transactions enabled, every handler in every service would fail on its first message. disableTransactions: true is not a considered consistency stance — it is the price of a demo estate that starts with docker-compose up, paid in the currency of the very guarantee the outbox advertises. The honest reading is a teaching estate showing the pattern's shape while its default environment cannot honour the pattern's promise. The unforgivable version would be copying this appsettings block into production against a replica set that could have done it properly — which is precisely what copy-paste estates do, and why a flag this consequential deserves a comment it does not have.

Belt present, suspenders thrown away

Now return to crash window 1. At-least-once delivery is fine if duplicates are harmless, and Convey's inbox absorbs most of them. But the inbox key is the broker's message id — it can only recognise a redelivery of the same message. A crash before the inbox write, a republished event after an outbox recovery, the same fact arriving twice with two ids: all of these reach the handler. So the last line of defence is handler idempotency, and this estate's handlers are anti-idempotent. From part 10, Orders' CustomerCreatedHandler:

if (await _customerRepository.ExistsAsync(@event.CustomerId))
{
    throw new CustomerAlreadyAddedException(@event.CustomerId);
}

Parcels' twin throws CustomerAlreadyExistsException; Customers' SignedUpHandler throws CustomerAlreadyCreatedException when Identity's SignedUp event comes round again. Three services, one convention: the expected outcome of redelivery is an exception. A duplicate CustomerCreated — a fact the consumer already knows, the easiest no-op in distributed systems — is nacked, retried by RabbitMQ, and nacked again, poisoning its way through the retry budget.

The detail that elevates this from bug to finding is in Orders' MessageToLogTemplateMapper, where CustomerCreated carries a dedicated error template: "Customer with id: {CustomerId} was already added." Someone anticipated the duplicate clearly enough to write its log message — and still shaped the handler to treat it as a failure. The estate knows redelivery happens; it just refuses to forgive it.

Put the two halves together and the configuration reads like a paradox. The outbox — the mechanism that creates duplicate deliveries by design — has its atomicity off, so duplicates and ghosts are both more likely. The inbox — the mechanism that absorbs duplicates — is on, but it is the shallow layer, and the deep layer beneath it throws. Guaranteed Delivery in name; at-least-once-if-lucky in fact; exactly-once assumed by the code that can least afford to assume it. My rule of thumb from operating this pattern elsewhere: an outbox without a transaction is a buffer, not a guarantee, and an inbox without idempotent handlers is a bloom filter, not a shield. You need all four corners — atomic write, drain, dedup, tolerant handler — before you get to say “guaranteed”.

None of this is exercised by a single automated test anywhere in the thirteen repos — the estate's CI story, in which nine pipelines run dotnet test against zero test projects, is the test pyramid CI never runs in the companion series.

The checklist the flag implies

If you run Convey's outbox — or any outbox — this estate's configuration is a list of what to verify, inverted:

• Prove the transaction is real. On Mongo, that means a replica set in every environment, including local compose; a flag like disableTransactions should fail loudly anywhere but dev.
• Treat redelivery as normal input. Existence checks that throw should no-op (or update); reserve exceptions for facts that contradict, not facts that repeat.
• Watch the drawer. An unsent-outbox metric and an age alarm are twenty lines; without them, window 2 is invisible until a customer notices.
• Test the crash windows. Kill the process between writes in an integration test harness once, and you will never again believe a diagram over a flag.

The outbox is where commands end their journey; the next part walks back to where they begin — and finds every command in the estate arriving through two front doors at once, HTTP and AMQP, with queries confined to one. The CQRS split is usually an application-layer diagram; in Pacco you can read it off the wire: CQRS at the transport layer.