Fan-Out on Write, One Round Trip at a Time

Timeline delivery has two textbook answers and both are wrong on their own. Fan-out on read means every feed load joins the follow graph against the post store, which collapses at the first popular account. Fan-out on write means every post is copied into every follower's list, which makes an account with a million followers a million writes and a rating update a million updates. The interesting systems do neither purely. Part 10 found a format-specifier bug in Trill's Timeline service; this part is about the design that bug is sitting inside, which is the best thing in the estate.

The hybrid

Trill.Services.Timeline.Core/Events/External/Handlers/StorySentHandler.cs, the second half of HandleAsync:

await _storage.AddStoryAsync(story);
_logger.LogInformation($"Added a story with ID: '{@event.StoryId}'.");
var followers = await _storage.GetAllFollowersAsync(@event.Author.Id);
if (!followers.Any())
{
    _logger.LogInformation($"No followers of author: '{@event.Author.Id}' have been found to add a story: '{@event.StoryId}'.");
    return;
}

foreach (var follower in followers)
{
    await _storage.AddStoryToTimelineAsync(follower, story);
}

And the two storage methods it calls:

public async Task AddStoryAsync(Story story)
{
    await _database.StringSetAsync(GetStoryKey(story.Id), JsonSerializer.Serialize(story));
}

public async Task AddStoryToTimelineAsync(Guid userId, Story story)
{
    var score = story.Visibility.From.ToUnixTimeMilliseconds();
    var entry = new SortedSetEntry(story.Id.ToString("N"), score);
    await _database.SortedSetAddAsync(GetTimelineKey(userId), new[] {entry});
}

The story body is stored exactly once, as a single JSON string key, and what gets fanned out is a sorted-set member carrying only the identifier. That is the whole idea, and it buys three properties at once.

• Bounded write amplification. Fan-out costs one ZADD of a short member per follower, not a copy of the payload. A 200-character story with a five-tag array and an embedded author is written once regardless of audience size.
• Cheap mutation. SetStoryTotalRatingAsync updates the single stories:{id} key. A rating change on a story with 50,000 recipients touches one key, not 50,000. Pure fan-out on write cannot do that without a second index.
• Free windowing. The sorted set's score is the story's visibility start in unix milliseconds, so SortedSetRangeByScoreAsync(key, minScore, maxScore) performs the seven-day window as a range scan on an already-sorted structure. No filtering, no sorting, no secondary index.

The read completes the shape:

var storyIds = await _database.SortedSetRangeByScoreAsync(GetTimelineKey(userId), minScore, maxScore);
var storyKeys = storyIds.Select(x => (RedisKey) GetStoryKey((long)x)).ToArray();
var storyEntries = await _database.StringGetAsync(storyKeys);

One ZRANGEBYSCORE, then one variadic MGET for the whole page. Two round trips to render a timeline, whatever its length. If it were not for the (long)x cast that part 10 dismantles, this would be a small, correct, genuinely good piece of infrastructure — 104 lines that most people would reach for a queue and a projection database to achieve.

The loop

Now the execution. foreach (var follower in followers) await _storage.AddStoryToTimelineAsync(follower, story); is one await per iteration, and each await is a network round trip. There is no IBatch, no CreateTransaction, no Task.WhenAll, no chunking, and no CancellationToken on the handler at all.

At a 0.5 ms round trip on a local network, an author with 50,000 followers occupies a RabbitMQ consumer for twenty-five seconds while producing 50,000 sequential single-command requests. Redis itself is not the bottleneck — it will absorb far more than that — the latency is. StackExchange.Redis exists precisely to avoid this, and the fix is two lines:

public async Task AddStoriesToTimelinesAsync(IReadOnlyCollection<Guid> userIds, Story story)
{
    var score = story.Visibility.From.ToUnixTimeMilliseconds();
    var batch = _database.CreateBatch();
    var tasks = userIds
        .Select(id => batch.SortedSetAddAsync(GetTimelineKey(id), story.Id, score))
        .ToArray();
    batch.Execute();
    await Task.WhenAll(tasks);
}

One pipelined flush instead of N round trips, and the same fan-out semantics. Chunk it at a few thousand per batch and the memory profile stays flat too.

The follower fetch has the same shape of problem one level up. GetAllFollowersAsync is a bare SMEMBERS:

var followers = await _database.SetMembersAsync(GetFollowersKey(userId));

return followers.Select(x => Guid.Parse(x)).ToArray();

Unbounded, unpaged, and materialised into a Guid[] before the loop starts — so the peak memory of a fan-out is proportional to the author's follower count. SSCAN with a cursor, feeding batches into the pipelined write above, turns an O(followers) allocation into a constant one.

The rules the design encodes, including the ones it did not mean to

Reading the storage layer as a product specification is the other reward of a file this small. Five behaviours are encoded here and none of them is written down anywhere else in the estate.

The window is seven days and it is not configurable. minScore = GetScore(from ?? DateTime.UtcNow.AddDays(-7)). IStorage.GetTimelineAsync takes optional from and to parameters, and the only caller — the inline endpoint in Api/Startup.cs — never passes either and parses no query string. The two parameters are dead, and the API's window is a hard-coded literal.

The score is Visibility.From, not CreatedAt. A story whose visibility begins tomorrow gets a score above maxScore = now, so it is excluded until its start date arrives. That is correct for scheduled content and it has a consequence part 13 picks up: Trill's ad-creation form forces the start date into the future, so every ad is invisible in every timeline on the day it is booked.

Visibility is then filtered a second time, after the fetch:

var story = JsonSerializer.Deserialize<Story>(entry);
if (story.Visibility.From <= now && story.Visibility.To >= now)
{
    stories.Add(story);
}

The score handles the start; this handles the end. It works, but it means expired stories are fetched over the wire and discarded, and it means the sorted set never shrinks.

Paging is a shape, not a behaviour. The return is:

return new Paged<Story>
{
    Items = stories,
    CurrentPage = 1,
    TotalPages = 1,
    TotalResults = stories.Count,
    ResultsPerPage = stories.Count
};

Four constants. The envelope matches the Paged<T> contract that Ads, Stories and Users also return, and means nothing. A client that renders a pager from these fields renders one page, always.

The author is not in their own timeline. StorySentHandler fans out to GetAllFollowersAsync(@event.Author.Id) and nothing else, and a user is not a member of their own follower set. You cannot see what you posted.

What is missing around the edges

The delivery guarantees are worth naming precisely, because they are close to right by accident.

• No idempotency. Infrastructure/Extensions.cs:61-62 registers only the two logging decorators. Ads and Analytics register logging and outbox; Timeline is the one service in the estate with no inbox deduplication. A redelivered StorySent re-runs the entire fan-out. It survives — StringSetAsync is an overwrite and ZADD with the same member and the same score is a no-op — but it survives because the score is deterministic, not because anything decided it should.
• SetStoryTotalRatingAsync is read-modify-write with no transaction. It fetches the story JSON, sets one integer, re-serialises and writes it back. Two concurrent StoryRated events lose one update. WATCH/MULTI or a small Lua script would make it atomic; better still, keep the total in its own key so the two writers never touch the same value.
• An out-of-order StoryRated is dropped silently. If a rating arrives before the story it rates, SetStoryTotalRatingAsync finds no entry and returns. No retry, no dead letter, no counter, no watermark. Separate exchanges and independent queues make this reachable.
• No backfill and no cleanup. FollowAsync adds to a set and seeds nothing, so a new follower sees none of the author's history. UnfollowAsync removes the set member and leaves every already-fanned-out story in place, so you keep seeing an unfollowed account for up to seven days. And nothing is ever evicted: AddStoryAsync sets no TTL, no sorted set is ever trimmed, and the seven-day window is enforced only at read time. Redis is holding every story ever published, in RAM, forever, behind a query that will only ever return a week of them.

One last detail that is small and completely disabling. Both follow handlers log this:

_logger.LogInformation($"User with ID: '{@event.FollowerId}' followed '{@event.FollowerId}'.");

FollowerId twice. FolloweeId is never logged, in either handler. The single observability signal for the entire follow graph tells you who did something and not to whom — which, in a service whose whole job is the follow graph, is the one field that matters.

And the project file claims capabilities the service does not have: Convey.MessageBrokers.Outbox and Convey.MessageBrokers.Outbox.Mongo are referenced by a service with no Mongo and no AddMessageOutbox call, while AddCommandHandlers(), AddQueryHandlers() and both in-memory dispatchers are registered in a repository containing zero commands, zero queries and zero handlers of either kind. Reading this estate's configuration and package lists as a description of what exists is the fastest way to be wrong about it.

The honest summary: the design is a distributed-systems answer and the implementation is a script. Storing the body once and fanning out identifiers is the decision that takes experience; batching the writes is the decision that takes ten minutes. Trill made the hard one and skipped the easy one, which is a much better failure than the reverse.

Next, the service that consumes the same events and turns them into the only numbers the product ever reports: trending means all-time and frozen.