A Load Balancer With Amnesia

Part ten left one thing unexplained: when the client-side discovery strategy asks “who is products?” and Consul answers with three instances, how does it choose one — and how does it avoid sending every request to the same box? The answer is ConsulServicesRegistry, a small class that implements a thoughtful “pick an instance you haven't used recently” policy, backed by a dictionary that remembers which instances it has handed out. It is a good design undone by one attribute on its registration, and the failure is a perfect little lesson in how dependency-injection lifetimes silently decide whether your stateful class has state at all.

The design: round-robin by memory

The registry keeps a per-service set of instance ids it has already returned, and prefers instances not in that set:

public class ConsulServicesRegistry : IConsulServicesRegistry
{
    private readonly Random _random = new Random();
    private readonly IConsulClient _client;
    private readonly IDictionary<string, ISet<string>> _usedServices
        = new Dictionary<string, ISet<string>>();

    public async Task<AgentService> GetAsync(string name)
    {
        var services = GetServices((await _client.Agent.Services()).Response, name);
        if (!services.Any()) return null;

        if (!_usedServices.ContainsKey(name))
            _usedServices[name] = new HashSet<string>();
        else if (services.Count == _usedServices[name].Count)
            _usedServices[name].Clear();          // used them all — start over

        return GetService(services, name);
    }

    private AgentService GetService(IList<AgentService> services, string name)
    {
        var unused = services.Where(s => !_usedServices[name].Contains(s.ID)).ToList();
        var service = unused.Any()
            ? unused[_random.Next(0, unused.Count)]   // random among the not-yet-used
            : services.First();
        _usedServices[name].Add(service.ID);
        return service;
    }
}

Read the intent and it is genuinely reasonable: pick a random instance you have not used yet; once you have cycled through all of them, clear the tally and start a fresh round. It is randomised round-robin with a memory — a legitimate client-side load-balancing policy that spreads calls across instances instead of hammering services.First(). The whole thing hinges on _usedServices surviving between calls to GetAsync. If that dictionary persists, the policy works. If it resets, every call sees an empty tally, every call takes the “random among all instances” branch, and the careful round-robin collapses into “pick one at random every time” — which still balances, roughly, but throws away the anti-repetition guarantee the class was written to provide.

So: does _usedServices persist?

The registration that erases the memory

Back in AddConsul:

services.AddTransient<IConsulServicesRegistry, ConsulServicesRegistry>();
services.AddTransient<ConsulServiceDiscoveryMessageHandler>();

AddTransient means a brand-new ConsulServicesRegistry — with a brand-new, empty _usedServices — is created every time one is resolved. A class whose entire job is to remember things across calls is registered with the one lifetime that guarantees it remembers nothing. Resolve it fresh per request and the memory is stillborn; the round-robin never gets past its first step.

There is a rescue, and it is worth understanding because it is accidental. The registry is not resolved directly per request — it is injected into ConsulServiceDiscoveryMessageHandler, which is registered as an HttpClient message handler. And IHttpClientFactory caches the handler chain, by default for two minutes, reusing the same handler instance (and therefore the same injected registry, and therefore the same _usedServices) across every request that client makes in that window. So the memory does survive — not because the DI lifetime intends it to, but because HttpClientFactory's handler pooling holds the transient handler alive for a couple of minutes. The class works by luck of a caching layer two abstractions away.

The lifetime that makes the class correct is one nobody declared. Change the handler-lifetime setting, or resolve the registry anywhere outside the message-handler path, and the round-robin silently degrades — no error, no log, just a load-balancing policy quietly not balancing the way it reads.

And when it does remember, it races

Here is the twist that makes this more than a lifetime footnote. The only time _usedServices persists is when the message handler is shared — and a shared message handler is used by concurrent requests. ConsulServicesRegistry mutates a plain Dictionary and HashSet with no lock, and Random is likewise not thread-safe. So in exactly the scenario where the memory works (one cached handler serving a burst of parallel calls), multiple threads call GetAsync on the same instance at once, reading and writing _usedServices and _random without synchronisation. Concurrent HashSet mutation can corrupt the set; concurrent Random.Next can wedge the generator into returning zeros. The class only has a memory when it is shared, and when it is shared it is a data race. The two properties that were supposed to make it correct — persistence and reuse — are the same property that makes it unsafe.

The fix and the lesson

The correction is small: register the registry as a singleton (or scoped to match the handler) and make its state thread-safe — a ConcurrentDictionary, a lock around the read-modify-write, and a thread-safe random (or Random.Shared on modern .NET). Then the memory is intentional and the concurrency is handled, and the lifetime is no longer doing the round-robin a favour by accident.

The durable takeaway is a rule I apply every time I register a service: a class with fields that must survive between calls is making a lifetime claim, and AddTransient denies it. When you write private readonly Dictionary as instance state, you have declared “I need to live longer than one call” — and the registration has to agree, out loud, or the state is a lie. Here the state was rescued by an unrelated cache and punished by the concurrency that rescue implies. It is the clearest example in the estate of DI lifetime being a correctness decision, not a performance tuning knob.

Next, another class that reaches past the type system to get its way — the reflection that writes a fresh Guid straight into an immutable command's backing field, defeating the immutability the command was designed to have.