Durable Queues on an Ephemeral Disk

durable: true is the flag people reach for when they want to stop worrying about message loss. It is also the flag that produces the most confident wrong answers in incident reviews, because durability at the broker is a promise about one specific failure — broker restart — and it is only as good as the storage underneath it. Set the flag, mount nothing, and you have paid the write cost of durability for none of the benefit.

Part 3 catalogued the components Trill's configuration promises and its code never delivers. This one is different in kind: the code does deliver it, correctly, and a different file takes it away.

The broker side is right

Every appsettings.json in the estate configures RabbitMQ identically. Here is the Pusher's, and the API Gateway's differs only in the two names:

"rabbitMq": {
  "connectionName": "pusher",
  "retries": 3,
  "retryInterval": 2,
  "conventionsCasing": "snakeCase",
  "exchange": {
    "declare": true,
    "durable": true,
    "autoDelete": false,
    "type": "topic",
    "name": "pusher"
  },
  "queue": {
    "declare": true,
    "durable": true,
    "exclusive": false,
    "autoDelete": false,
    "template": "pusher/{{exchange}}.{{message}}"
  },
  "context": { "enabled": true, "header": "message_context" },
  "spanContextHeader": "span_context"
}

Read it slowly, because the messaging conventions here are the estate's one genuinely enforced standard.

• type: topic, one exchange per bounded context. stories, users, ads, plus a per-application exchange for the gateway, the Pusher and the Saga. That is a clean Message Bus with a Publish-Subscribe Channel per domain.
• "template": "<service>/{{exchange}}.{{message}}" produces queue names like pusher/stories.story_sent. Queue-per-consumer-per-message-type is a textbook Durable Subscriber, and scaling replicas turns each queue into Competing Consumers for free. It is the right default and it is applied consistently across all eight units.
• durable: true and autoDelete: false on both exchange and queue means the topology survives a broker restart and queues persist when the last consumer disconnects.
• conventionsCasing: "snakeCase" turns the class name StorySent into the routing key story_sent on both sides. The wire contract is therefore the class name plus its JSON shape — a decision with consequences that part 13 traces in detail.
• retries: 3, retryInterval: 2 is six seconds of connection tolerance at startup, which matters more than it looks because of what comes next.

Nothing about that block is wrong. If you were reviewing this file in isolation you would sign it off.

The container side cancels it

Trill\compose\infrastructure.yml brings up ten containers on a trill-network. This is the RabbitMQ service, in full:

  rabbitmq:
    build: ./rabbitmq
    container_name: rabbitmq
    restart: unless-stopped
    networks:
      - trill
    ports:
      - 5672:5672
      - 15672:15672
      - 15692:15692
    # volumes: 
    #   - rabbitmq:/var/lib/rabbitmq

Lines 88–89 are commented out. So is Consul's volume (12–13), Grafana's (35–36), Prometheus's (75–76) and Seq's (112–113). At the bottom of the file, the volumes: block declares mongo and redis and comments out the other five. Two of seven stateful containers keep their state.

Every exchange and queue in this estate is declared durable and hosted on a container filesystem that is discarded on docker-compose down. Undelivered messages, every log line ever shipped to Seq, every Consul registration, every Grafana dashboard you might have built — gone, without an error, because nothing about the teardown looks like data loss.

I want to be scrupulously fair about the severity here, because this is a sample estate and the stakes are zero. Nobody is losing customer orders. And commenting out volumes during development is a deliberate and common choice: it is how you get a clean broker on every run instead of fighting stale queue bindings from a schema you changed an hour ago. The problem is not the decision; it is that the decision is invisible from where its consequences are configured. The durable: true flag is in eight appsettings.json files, and the reason it does nothing is one commented line in a file none of those services can see.

That is the pattern worth carrying out of here: a reliability guarantee is a property of the whole stack, and it is always cancelled at the layer that nobody reviews. Durable queues on ephemeral disks; fsync on a battery-less RAID controller; a retry policy in front of a non-idempotent handler; RequiredDuringSchedulingIgnoredDuringExecution on a single-node cluster. Same shape every time.

The two volumes that survived, and why

It is worth asking which two containers kept their storage, because the answer is not arbitrary. mongo and redis have volumes; Consul, Grafana, Prometheus, RabbitMQ and Seq do not.

Those two are the ones holding domain state. Mongo is where stories, users, ads and the outbox collections live; Redis is Timeline's actual data store, sorted sets of story ids per user. Lose either and the demo is empty on the next run, and you notice immediately. Lose the other five and nothing visible changes — the broker redeclares its topology on connection because every appsettings.json sets "declare": true on both exchange and queue, Consul re-registers, Prometheus starts scraping again, and Seq is empty in a way that looks exactly like “nothing has happened yet”.

Every container whose loss is silent lost its volume; every container whose loss is loud kept one. That is not a decision anybody made; it is what happens when volumes are added reactively, in response to the one symptom that was annoying enough to fix. And it is precisely why the durability question is worth asking deliberately rather than symptomatically — the state you most want to survive a restart is usually the state whose absence you would not notice for a week.

The Mongo service carries a second commented block, at lines 57–59, which would have set a root username and password. With it commented out, Mongo runs with authentication disabled and every service's connection string is a bare mongodb://mongo:27017. Correct for a demo; worth naming, because that connection string is exactly what gets copied.

No health checks, no ordering, six seconds of grace

Two more properties of the compose files compound it.

There is no healthcheck: on any container in either compose file, and grep AddHealthChecks\|MapHealthChecks across all eleven repositories returns nothing. Convey's /ping endpoint is the only liveness signal in the estate, and — as part 3 noted — it is unmapped on the one service whose configuration asks Consul to poll it.

There is also no depends_on anywhere, in either compose file, for any service. Ten infrastructure containers start simultaneously; the nine application processes start whenever their runner starts them. A service that reaches RabbitMQ before the broker has finished booting has exactly retries: 3, retryInterval: 2 to work with — six seconds. Since restart: unless-stopped is set on the infrastructure containers but the applications are host processes under PM2 or Tye, a broker that takes seven seconds to become available produces nine application crashes and no automatic recovery.

The two absences point the same way. Without health checks there is nothing to order against; without ordering there is nothing for a health check to gate. Each makes the other look unnecessary.

The one thing wired end to end

Against all of that, compose/rabbitmq/ deserves its paragraph:

FROM rabbitmq:3-management
COPY ./plugins /etc/rabbitmq/enabled_plugins

with plugins containing exactly:

[rabbitmq_management,rabbitmq_prometheus].

That is an Erlang term, and the terminating full stop is mandatory — omit it and the plugin file silently fails to parse. People get this wrong constantly. Here it is right, port 15692 is exposed on the container, and prometheus.yml has a matching rabbitmq scrape job targeting rabbitmq:15692. This is the only infrastructure component in the estate that is correct at every layer: image, plugin file, exposed port, scrape target. It is a small thing, and it is exactly what the rest of the compose stack would look like if anyone had run it end to end once.

What the wire never carries

The topology tells you what the estate reaches for. The absences tell you what it never needed to. grep -r 'deadLetter\|dead_letter\|x-dead-letter' across all eleven repositories returns zero hits: no x-dead-letter-exchange argument on any queue declaration, no dead-letter queue in the broker image, no poison-message handling anywhere. A consumer that throws repeatedly re-queues or drops depending on Convey's default, with no operator-visible parking bay.

grep prefetch and grep -i qos return zero as well. There is no consumer concurrency limit and no flow control on any queue in the estate — which pairs badly with the unbounded in-process channel part 10 will get to, since neither layer applies backpressure to the other.

And there is no message TTL: no x-message-ttl argument, no per-message expiration. Combined with the ephemeral volume, that gives Trill's messages exactly two possible lifetimes — forever, or until the next docker-compose down. There is no in-between, which is another way of saying there is no retention policy at all.

To be fair once more: a Dead Letter Channel and a TTL policy are the sort of thing you add when you have an operator, and this estate has never had one. Naming them as absences is not a demand that a teaching repository ship them; it is a checklist for the reader who is about to lift these appsettings.json files wholesale into something that does.

Next, the component that publishes onto all this carefully declared topology without knowing the name of a single message it sends: a gateway that publishes what it cannot name.