In-Process Is Not Synchronous

The most over-claimed benefit of a modular monolith is that it makes eventual consistency go away. One process, one heap, one call stack — surely if Users publishes UserCreated and Stories handles it, that happens before the HTTP response goes out.

The Trill monolith's own end-to-end test says otherwise, in one line (tests/Trill.Tests.EndToEnd/StoryScenarioTests.cs:27):

private static Task WaitForEventsAsync() => Task.Delay(2000);

Part 10 counted what consolidation removed operationally. This part is about what it didn't remove, which is the part people forget.

The path a published event takes

InMemoryMessageBroker.PublishAsync is where the decision happens (Messaging/Brokers/InMemoryMessageBroker.cs:62-73):

foreach (var message in messages)
{
    var name = message.GetType().Name.Underscore();
    _logger.LogInformation($"Publishing a message: '{name}' with ID: '{message.Id:N}'...");
    if (_messagingOptions.UseBackgroundDispatcher)
    {
        await _asyncMessageDispatcher.PublishAsync(message);
        continue;
    }

    await _moduleClient.PublishAsync(message);
}

Two paths, one boolean. The second is genuinely synchronous — ModuleClient.PublishAsync awaits every registered receiver's handler before returning, so a command handler that publishes an event does not complete until every module has processed it.

The first is not, and it is the one that ships. appsettings.json:47 sets "useBackgroundDispatcher": true.

AsyncMessageDispatcher.PublishAsync is two lines:

public async Task PublishAsync<T>(T message) where T : class, IMessage
    => await _channel.Writer.WriteAsync(message);

MessageChannel is barely a class:

private readonly Channel<IMessage> _messages = Channel.CreateUnbounded<IMessage>();

And BackgroundDispatcher — a single BackgroundService — drains it:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    _logger.LogInformation("Running the background event dispatcher...");
    await foreach (var @event in _channel.Reader.ReadAllAsync(stoppingToken))
    {
        try
        {
            await _moduleClient.PublishAsync(@event);
        }
        catch (Exception exception)
        {
            _logger.LogError(exception, exception.Message);
        }
    }

    _logger.LogInformation("Finished running the background event dispatcher.");
}

• On an unbounded channel WriteAsync completes immediately, so the publishing handler returns as soon as the message is queued. The HTTP request finishes; the fan-out has not started.
• One reader, sequential dispatch. There is exactly one BackgroundDispatcher, and it awaits each PublishAsync before reading the next message. That buys global event ordering as a side effect of being one process — a property distributed systems pay dearly for.
• The catch logs and drops. A handler that throws produces one log line at Error and the message is gone. No retry, no dead-letter, no poison counter. This is at-most-once delivery, chosen implicitly by the shape of a try/catch.
• The last line never runs. On shutdown ReadAllAsync honours the token by throwing, which exits past the final log statement. “Finished running the background event dispatcher.” is unreachable in the normal stop path — a two-line tell that nobody watched this during a graceful shutdown.

The test that proves it

StoryScenarioTests is the payoff test of the whole repository — the one that demonstrates two modules cooperating in one process:

[Fact]
public async Task given_new_user_account_story_should_be_sent()
{
    var userId = Guid.NewGuid();
    await Client.SignUpAsync(userId);
    await WaitForEventsAsync();
    var auth = await Client.SignInAsync(userId);
    var storyId = await Client.SendStoryAsync(auth.UserId);
    var storyDto = await Client.GetStoryAsync(storyId);
    storyDto.ShouldNotBeNull();
    storyDto.Id.ShouldBe(storyId);
}

The WaitForEventsAsync() on line three is not defensive padding. It is load-bearing. POST /users-module/sign-up returns 201 as soon as the user document is written; the UserCreated event it publishes is sitting in a channel. Stories' UserCreatedHandler is what creates the local User projection that StoryAuthorPolicy checks before allowing a story. Send the story too early and it is rejected because, as far as Stories is concerned, the author does not exist.

The one test that proves the architecture works is a sleep-based race, and that is simultaneously the strongest and the weakest artefact in the repository. Strongest, because it is honest: the authors could have set useBackgroundDispatcher: false in the test environment and made the whole problem disappear, and they did not. Weakest, because two seconds is a guess, and a guess that is too small on a loaded CI agent and too large on every green run.

The fix is not exotic. Give the dispatcher an observable idle signal — a TaskCompletionSource that completes when the channel drains, or a test-only IMessageChannel decorator that counts in-flight messages — and replace the delay with an await. Fifteen lines, and the flakiest test in the repository becomes deterministic.

What you gained and what you did not

State it as a table, because this is the row people get wrong:

You gained ordering and you lost the network. You did not gain synchrony and you did not gain durability. Every eventual-consistency bug that exists in the distributed build — read-your-own-writes, a projection that lags, a handler that fails after the caller has gone — exists here too, at a shorter timescale and with less machinery to observe it.

The durability half deserves emphasis because there is nothing to soften it. Channel.CreateUnbounded has no back-pressure, so a slow consumer grows the heap without limit, and it has no persistence, so whatever is queued at shutdown is lost. The outbox exists to fix exactly this — MongoOutbox, OutboxProcessor, roughly 300 lines across the messaging folder, fully wired — and it is "enabled": false in every configuration in the repository (appsettings.json:52-56). The mechanism that would make the queue survivable ships in the off position and is exercised by no test.

Same primitive, opposite outcome

There is a lovely pairing across the two builds here, and it is worth reading together. Trill.Pusher — the microservices push service — also builds its distribution on System.Threading.Channels, and it gets it wrong: a single Channel<T> is used for pub/sub fan-out to N connected browsers, so each story reaches exactly one of them, at random. The class name is even plural.

The monolith uses the same primitive correctly, because it uses it for the right pattern. A Channel<T> is a work queue, not a topic. One writer set, one reader, and the fan-out happens inside the loop via _moduleClient.PublishAsync(@event), which is where fan-out belongs. Same authors, same primitive, opposite outcome, and the difference is entirely whether the channel is being asked to be a queue or a topic. If you take one reusable lesson out of this series, that is a strong candidate.

The conference-sample monolith built the same seven files — channel, dispatcher, background job, in-memory broker — with the same drop-on-error semantics and the same empty IMessage marker. This repository's version is the more complete one: IMessage at least carries an id and a correlation id, an inbox decorator exists for idempotency, and an outbox exists for durability. It just has both of them switched off.

The one module that does it properly

There is a second asynchrony in this system, and it is worth separating because only one module implements it. Stories runs the full two-tier pattern — domain events inside the module, integration events across it (RateStoryHandler.cs:52-56):

var domainEvents = rating.Events.ToArray();
await _domainEventDispatcher.DispatchAsync(domainEvents);

var integrationEvents = _eventMapper.Map(domainEvents).ToArray();
await _messageBroker.PublishAsync(integrationEvents);

Domain events are dispatched synchronously and in-module; only their mapped integration counterparts go on the channel. IEventMapper is the translation boundary, and it is a switch expression with a _ => null arm that InMemoryMessageBroker filters out (:41) — so an unmapped domain event silently produces nothing across the boundary.

Every other module publishes integration events straight out of a command handler. Six modules, one two-tier implementation: the pattern is demonstrated, not adopted. That matters here because the two-tier split is the only thing that lets you reason about which of your events are in-order-and-immediate and which are queued-and-eventual. Where it is absent, they are the same call and only the message type tells you which world you are in.

What I would change

Three things, none of them large, and all three would make the architecture's real properties visible rather than implied.

Say the delivery semantics out loud. One comment above the catch in BackgroundDispatcher declaring at-most-once on purpose would stop every future reader from assuming otherwise. Right now the semantics are an emergent property of a try/catch.

Bound the channel. Channel.CreateBounded<IMessage>(capacity) with a wait-on-full policy is a one-line change that forces the interesting question — block the request or shed the event? — which unbounded lets you defer until a production incident asks it for you.

Make the test deterministic, because a two-second sleep in the flagship test teaches every reader that this is how you test in-process eventing, and it is not.

None of that changes the architecture. It changes how much of the architecture you have to infer.

Next, the finding that neither column enjoys: the rewrite kept every bug.