The Streaming Seam: Redis Pub/Sub Behind an Interface

Every distributed system has a seam where “my process” ends and “the wire” begins, and how you cut that seam decides how testable, swappable, and honest your architecture is. In FeedR — the open-source DevMentors sample this series walks through — the seam is two interfaces so small they fit in a tweet, and the story of what sits behind them is the most instructive thing in the whole repository. Including one place where the name on the door doesn't match what's inside.

Two methods, total

Here is the entire streaming abstraction from FeedR.Shared:

public interface IStreamPublisher
{
    Task PublishAsync<T>(string topic, T data) where T : class;
}

public interface IStreamSubscriber
{
    Task SubscribeAsync<T>(string topic, Action<T> handler) where T : class;
}

A topic name, a payload, a handler. No message IDs, no acknowledgements, no offsets, no headers. That minimalism is a choice with consequences we'll get to — but first, appreciate what it enables. The quotes service publishes ticks with PublishAsync("pricing", currencyPair) and has no idea Redis exists. The aggregator subscribes with SubscribeAsync<CurrencyPair>("pricing", handler) and has no idea who publishes. Every service in the system programs against these two methods, and the transport is a registration detail.

The no-op default and the DI last-wins trick

The part I show junior engineers first: FeedR.Shared ships a default implementation that does nothing.

internal sealed class DefaultStreamPublisher : IStreamPublisher
{
    public Task PublishAsync<T>(string topic, T data) where T : class
        => Task.CompletedTask;
}

And two registration extensions:

services.AddStreaming();        // registers the no-op pair
services.AddRedisStreaming();   // registers the Redis pair

Every FeedR service calls both, in that order. This works because of a .NET DI behavior that's underused as a design tool: when multiple registrations exist for the same service type, resolving a single instance gives you the last one registered. AddRedisStreaming() after AddStreaming() means Redis wins. Comment out one line in Program.cs and the service still builds, still runs, still handles HTTP — it just streams into the void.

Is relying on registration order fragile? Somewhat — AddStreaming() after AddRedisStreaming() would silently disable your transport, which is a fun outage to diagnose. The more explicit version of this idea is TryAddSingleton (defaults yield to anything already registered) or an options callback choosing the transport by name. But the underlying pattern — cross-cutting abstractions ship with inert defaults so the system degrades to “runs but doesn't distribute” rather than “doesn't start” — is one I've adopted wholesale. It makes unit tests trivially quiet and lets you boot a service on a machine with no infrastructure at all.

What's actually behind the seam

The Redis implementation is startlingly small. Publishing:

internal sealed class RedisStreamPublisher : IStreamPublisher
{
    private readonly ISerializer _serializer;
    private readonly ISubscriber _subscriber;

    public RedisStreamPublisher(IConnectionMultiplexer multiplexer, ISerializer serializer)
    {
        _serializer = serializer;
        _subscriber = multiplexer.GetSubscriber();
    }

    public Task PublishAsync<T>(string topic, T data) where T : class
        => _subscriber.PublishAsync(topic, _serializer.Serialize(data));
}

Subscribing is the mirror image: _subscriber.SubscribeAsync(topic, ...), deserialize, invoke the handler. The IConnectionMultiplexer is registered once as a singleton — the correct lifetime for StackExchange.Redis, which multiplexes everything over a shared connection — from a one-key options section (redis:connectionString, which flips from localhost to redis in the docker environment).

Now the sign on the door. These classes live in FeedR.Shared.Redis.Streaming, the extension is AddRedisStreaming, the interfaces say stream — but GetSubscriber().PublishAsync is Redis pub/sub, not Redis Streams. Different Redis feature, radically different guarantees:

• Pub/sub is a broadcast in the moment. A message goes to every currently connected subscriber and then ceases to exist. No storage, no replay, no acknowledgement, no consumer groups. Restart the aggregator and every tick published during the restart is simply gone. Delivery is at-most-once.
• Redis Streams (XADD/XREADGROUP) is an append-only log. Messages persist, consumers track positions, consumer groups share work, unacknowledged entries can be reclaimed. Delivery is at-least-once and offline consumers catch up.

For FeedR's tier-one traffic this is the right call, not a shortcut: a missed price tick is superseded by the next one in under a second, and paying for persistence on data with a one-second half-life is waste. But the naming matters more than it seems. I have watched a team consume an abstraction called “streaming”, assume replay semantics, and design a downstream reconciliation process around messages that were never stored. The seam hides the transport; it must not hide the guarantees. If I owned this library I'd either rename the namespace to Redis.PubSub or — better — put the guarantee in the interface name: ITransientStreamPublisher tells a consumer everything they need to fear.

The contract nobody wrote down

What travels over the wire? JSON, produced by a serializer that is itself a service:

private static readonly JsonSerializerOptions Options = new()
{
    PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
    PropertyNameCaseInsensitive = true,
    Converters = { new JsonStringEnumConverter(JsonNamingPolicy.CamelCase) }
};

Registering ISerializer in DI instead of sprinkling JsonSerializer.Serialize calls around is quiet discipline: one place defines the wire format, and publisher and subscriber cannot drift apart on naming policy.

But notice what doesn't exist: a shared contract assembly. The quotes service defines CurrencyPair(string Symbol, decimal Value, long Timestamp) — and the aggregator defines its own, structurally identical record in its own namespace. The schema agreement is by convention: matching property names, camelCased in transit, case-insensitively bound on arrival. Rename a property on one side and nothing fails at compile time; the field just arrives as default on the other side at runtime.

Duplicating DTOs per service is actually defensible — shared contract packages create the coupling and versioning misery microservices exist to avoid, and I've written before about preferring convention over shared binaries. But convention-based contracts demand a compensating control: consumer-driven contract tests, schema registry, or at minimum integration tests that push a real payload through a real serializer (part 9 shows FeedR doing a version of exactly that). The sample has the duplication without the compensating control, which is the tell that this is a demo — the pattern is production-shaped, the safety net isn't there.

The subscriber's hidden thread story

One more subtlety worth pulling out, because it bites people in real systems. The aggregator's consuming side looks like this (from PricingStreamBackgroundService):

await _subscriber.SubscribeAsync<CurrencyPair>("pricing", currencyPair =>
{
    _logger.LogInformation("...");
    _ = _pricingHandler.HandleAsync(currencyPair);
});

Two things are happening. First, the handler signature is Action<T> — synchronous — so the async HandleAsync gets discarded with _ =, another fire-and-forget whose exceptions vanish (the same trade discussed in part 3, less deliberate this time; an Func<T, Task> handler signature would have made the async story explicit). Second, StackExchange.Redis executes pub/sub handlers sequentially per channel by default, on its own internal reader infrastructure. A slow handler doesn't just delay one message — it backs up the channel. The sample's handler is fast, so the trap stays shut. Load it up with I/O and you'd want ConnectionMultiplexer configured for concurrent dispatch or an internal Channel<T> hop: receive fast, enqueue, process at your own pace — the part 3 pattern again, now defending a network boundary instead of an HTTP one.

The scorecard

What this seam gets right: two-method surface, inert defaults, singleton multiplexer, serializer-as-a-service, transport chosen in one line of composition root. What to add before trusting it with your system: truthful naming for the delivery guarantee, an async handler signature, error handling on both sides of the wire, and a compensating control for the convention-based contracts.

And when a feed comes along whose messages must not evaporate? FeedR has an answer for that too — a second, parallel seam with envelopes, acknowledgements, and a durable broker behind it. That's the Pulsar story, and it's part 7. Before that, part 5 turns to the other real-time edge of the quotes service: pushing those same ticks to clients over gRPC server streaming, including the one piece of code in this repository I most want to rewrite.