Sticky Sessions, Wrong Cache

State between requests is where scale-out goes to die. A single-instance web app can keep session in memory and cache in a dictionary and never notice the difference; the same code behind three instances behaves like a person with three moods, and which mood you get depends on which instance the load balancer picked. The attendance platform ran on three App Service instances and carried a set of state decisions that each made sense alone and contradicted each other in combination — plus one cache expiry with a calendar bug. After part nine's read paths, this is the state that lives between requests.

Two caches, both configured

The composition root wired up two distributed caches in the same ConfigureServices:

services.AddDistributedMemoryCache();          // backs Session - in-process
services.AddSession(o =>
{
    o.IdleTimeout = TimeSpan.FromSeconds(1000); // "short timeout for easy testing"
    o.Cookie.Name = "Portal.Session";
});

services.AddStackExchangeRedisCache(o =>        // backs feature caches - out-of-process
{
    o.Configuration = _vault["RedisConnection"];
    o.InstanceName = "attendance";
});

The name AddDistributedMemoryCache is one of ASP.NET Core's genuine traps. It sounds distributed. It implements IDistributedCache, the distributed-cache interface. It is not distributed — it is an in-process, single-instance dictionary that satisfies the interface so that code written against IDistributedCache runs without a real backing store during development. It is a stand-in for a distributed cache, not one.

So the platform had two caches with opposite topologies. Session sat in AddDistributedMemoryCache — in-process, per-instance, invisible across the fleet. The feature caches (birthdays, org-hierarchy fragments) sat in real Redis — shared across all three instances. One kind of state was local and one was global, and nothing in the code signalled which was which. AddDistributedMemoryCache is the most misleadingly named method in ASP.NET Core: it is the non-distributed cache that implements the distributed interface.

The affinity contradiction

In-memory session behind multiple instances only works if every request from a given user lands on the same instance — sticky sessions, which App Service provides through an ARR affinity cookie. The platform's web.config, however, contained:

<httpProtocol>
  <customHeaders>
    <add name="Arr-Disable-Session-Affinity" value="true" />
  </customHeaders>
</httpProtocol>

Affinity was deliberately switched off. That is a sound instinct in general — affinity undermines even load distribution and masks statelessness bugs, so a well-designed stateless app should disable it. But this app was not stateless: its session lived in per-instance memory. Turn affinity off, and each request round-robins across three instances, each holding a different in-memory session store. A user's session data existed on whichever instance created it and nowhere else, so roughly two out of three requests saw an empty session. Symptoms: intermittent logouts, form state vanishing between steps, “it forgot what I selected” — the classic non-deterministic scale-out bug that never reproduces because it depends on which instance answered.

The two settings were each correct in isolation and lethal together. Disabling affinity is right for a stateless app; in-memory session is fine on a single instance. The combination — the specific pairing this app shipped — is the one that breaks. The fix is to make session actually distributed: point it at the Redis that was already provisioned and already connected, by registering the Redis cache as session's backing store instead of the memory stand-in. The infrastructure was there; one line of wiring pointed session at the wrong one.

A thousand seconds of “easy testing”

The session idle timeout was TimeSpan.FromSeconds(1000), carrying the comment “short timeout for easy testing”. A developer once wanted sessions to expire quickly so they could exercise the timeout path without waiting, set it to a round-ish small number, and — as with the “1000-second easy testing” values that litter every codebase — it shipped. In production, sixteen minutes and forty seconds of idle logs a user out mid-task. On an approval screen where a manager reviews a month of attendance one employee at a time, sixteen minutes is normal working pace, so users were silently timed out while actively working, then bounced to login on their next click.

The lesson is procedural, not technical: a magic number with a “for testing” comment is a to-do list item that shipped as a feature. These constants are where development-time convenience leaks into production behaviour, and the comment is the tell — it documents that the value was chosen for the developer's clock, not the user's. Grep your config for round numbers and the word “testing”.

The expiry that throws every December

My favourite, because it is a genuine time bomb with a known fuse date. A feature cache set its Redis entries to expire at the start of the following month, computed like this:

var now = DateTime.Now;
var expiresAt = new DateTime(now.Year, now.Month + 1, 1);   // first of next month
cache.Set(key, value, new DistributedCacheEntryOptions { AbsoluteExpiration = expiresAt });

Read the arithmetic. now.Month + 1 gives the next month number. In January it is 2, in November it is 12 — fine. In December, now.Month is 12, so now.Month + 1 is 13, and new DateTime(year, 13, 1) throws ArgumentOutOfRangeException: there is no thirteenth month. Every December, this cache-set path threw for the entire month. Whatever feature it backed either fell through to its uncached slow path or, depending on the surrounding catch, failed outright — once a year, seasonally, mysteriously, for thirty-one days.

The correct expression is DateTime.Now.AddMonths(1), which understands calendars — it rolls December into next January and even handles the day-of-month edge cases that manual construction gets wrong. The bug is a tiny one, but it is the shape of bug worth internalising: manual date component arithmetic works in every month you test in and fails in the one you don't. Whoever wrote it tested in some month that was not December, saw it work, and shipped a defect with a one-month-a-year duty cycle. Date maths belongs to the date library, always — a lesson the companion series' timezone article hammers from the batch-job side.

What made this one especially slippery is that “cache set fails” is a quiet failure by design. Caching is meant to be transparent — a cache miss falls through to the source of truth and the user sees correct data, just slower. So a Set that throws every December did not produce wrong answers; it produced a feature that was mysteriously sluggish for a month, every year, with no error the user could report beyond “it feels slow lately”. If the surrounding code caught and swallowed the exception (and in this estate, it usually did — see the swallow-to-default habit throughout this series), then the only symptom was latency, and latency has a thousand causes. A bug that manifests solely as seasonal slowness, with the exception eaten before it reaches a log, is nearly unfindable by observation; you find it by reading the date arithmetic and noticing month thirteen, which nobody does until they already suspect the line. The defence is upstream of the bug: never let a cache-set failure be silent, because a silent cache is a cache you cannot debug.

The ledger

State between requests, in ascending order of how hard it is to catch:

1. The 1000-second timeout — visible, annoying, easy to find once someone complains.
2. Two caches, one misleadingly named — invisible until you ask “is this shared?” and discover half your state isn't.
3. In-memory session with affinity off — non-deterministic, blamed on “flaky network”, reproduces one request in three.
4. The December expiry — dormant eleven months, then a calendar-driven outage nobody connects to a date-arithmetic line written in some sunny month long ago.

The through-line: none of these threw at the developer. They all threw at the user, in production, under conditions the laptop never recreated — three instances, real months, real idle time. That is the signature of state bugs, and the reason state deserves the paranoia the happy path never earns.

Speaking of state machines: the platform let employees start a virtual work-from-home timer and stop it from a link in an email — and that one feature contains the estate's single deliberate authorisation check, the one place a URL parameter is compared instead of overwritten. Next.