Every Archive Needs an Expiry Date

Part 1 ended with a dare: if you have ever written IsDeleted = true and wondered who is ever going to clean that up — there's a timestamp with your name on it. This is that post. Because an archive without retention isn't a solution to the soft-delete graveyard, it's the graveyard with better paperwork: rows accumulating forever, just in a different table. The fix is to make every archive row answer, at birth, the question nobody asks until the compliance audit: when do you leave?

Retention is decided at write time

The policy lives on the attribute — TtlDays = 365 on Order, 730 on Product, and the default -1 meaning keep forever. What matters is where the policy becomes data. The interceptor stamps it in the same breath as the archive itself:

if (attr.TtlDays >= 0)
    SetProp(archiveEntry, "ExpiresAt", archivedAt.AddDays(attr.TtlDays));

That one line is a bigger design decision than it looks. The alternative — computing expiry at cleanup time as ArchivedAt + TtlDays — keeps the policy in code, which means changing TtlDays retroactively re-sentences every existing row. Stamping ExpiresAt at archive time freezes the policy that was in force when the deletion happened: rows archived under the 365-day regime keep their 365-day expiry even after you tighten the attribute to 90. For anything compliance-adjacent, that's the correct default — retention promises are made at collection time — and if you ever do need to re-sentence old rows, an UPDATE on a plain timestamp column is honest, auditable SQL.

Two edge values fall out of the >= 0 comparison and both are intentional. -1 (or any negative) writes ExpiresAt = null — the row is immortal, and part 4 showed it doesn't even pay for the index (HasIndex(x => x.ExpiresAt) is only configured when the type has a TTL). And TtlDays = 0 stamps ExpiresAt = archivedAt — expired on arrival, deleted on the next cleanup pass. That sounds like a footgun until you need it: “archive for the snapshot, retain nothing” is a real request in data-minimization regimes, and TtlDays = 0 is exactly that contract.

The reaper

TtlCleanupService is the second BackgroundService in the library, and the slower one — the default interval is 24 hours against the snapshot service's 10 seconds, because expiry has day-level granularity and nobody needs a millisecond-accurate reaper. Like its sibling, the loop is packaging and the public method is the product:

public async Task RunCleanupAsync(CancellationToken ct = default)
{
    using var scope = _scopeFactory.CreateScope();
    var db = (DbContext)scope.ServiceProvider.GetRequiredService<IArchiveDbContext>();

    var expirableTypes = ArchiveRegistry.AllArchivableTypes()
        .Where(t => ArchiveRegistry.GetAttribute(t).TtlDays >= 0)
        .ToList();

    if (expirableTypes.Count == 0) return;

Then, per expirable type, the generic gymnastics: the service knows the archive type only as a Type, so it builds Set<TArchive>() reflectively and leans on IQueryable<T>'s covariance to query through the shared interface:

var setMethod = typeof(DbContext).GetMethod(nameof(DbContext.Set), Type.EmptyTypes)!
                                 .MakeGenericMethod(archiveType);
var dbSet     = setMethod.Invoke(db, null)!;

var queryable = (IQueryable<IArchiveEntry>)dbSet;
var expired   = await queryable
    .Where(a => a.ExpiresAt != null && a.ExpiresAt < now)
    .OrderBy(a => a.ExpiresAt)
    .Take(_options.BatchSizePerType)
    .ToListAsync(ct);

db.RemoveRange(expired.Cast<object>());

The shape of that query is the operational safety story. Take(BatchSizePerType) — default 500 — bounds every pass: no DELETE of three years' backlog in one transaction, no lock storm on the archive table the morning after you first enable TTL on an old estate. OrderBy(ExpiresAt) makes the batch deterministic and oldest-first, so a backlog drains in expiry order across successive runs rather than thrashing randomly. And now is captured once outside the loop, so every table in a pass is judged against the same instant. If a pass leaves survivors, the next pass (or the next Lambda invocation — keep reading) takes the next 500. Steady-state, the daily crop is small; the batching exists for the pathological days.

A caveat for horizontally scaled hosts: the hosted service runs in every replica, and nothing coordinates them. Two instances reaping the same expired rows mostly resolves benignly — the loser's DELETE affects zero rows and EF surfaces it as a concurrency exception caught by the loop's outer handler — but it's noise you don't need. If you run more than one replica, either elect a single instance to enable EnableTtlCleanup, or better, turn the flag off everywhere and give the job to an external scheduler, which is where the serverless section below is headed anyway.

One thing the reaper deliberately isn't: soft. These are hard DELETEs — no IsArchiveDeleted flag, no archive of the archive. Retention means gone, and turtles-all-the-way-down archiving is how you fail the audit you built all this for.

The rows the reaper doesn't visit

Honesty section. The cleanup loop iterates archive types only. Snapshot job rows — including big Complete payloads — have no ExpiresAt and are never cleaned. When an expired archive row is deleted, the FK from part 4 (ON DELETE SET NULL) means any snapshot it pointed at survives with dangling-pointer dignity: the tree document outlives every row that fed it. Whether that's a bug or a feature depends on your retention lawyer — arguably the snapshot is the long-term record and the archive rows are working data — but today the library doesn't let you choose: archive rows expire, snapshots persist until you write the DELETE yourself. The obvious extension is a SnapshotTtlDays knob stamping ExpiresAt on the job ticket and a second loop in the reaper; it isn't written yet, and until it is, snapshot retention is your operational chore. Budget for it: at one snapshot per deleted aggregate, it's slow-growing, but “slow-growing and unbounded” is exactly the shape of problem this post exists to kill.

Also worth restating from part 3: RemoveRange on 500 tracked instances is fine at these batch sizes, but this loop is the one place in the library where EF Core's ExecuteDeleteAsync is the right tool — set-based, no materialization, no tracking — and would drop the loop's memory profile to nothing. It's not used because the archive rows are loaded through the interface-typed detour above, and rewriting that as a generic ExecuteDelete is tangled with the same codegen future part 5 confessed to.

Switching off the loops

Both background services carry the same escape hatch, and the options make it a first-class hosting decision rather than a hack:

builder.Services.AddArchiveKit(options =>
{
    // On Lambda: set both to false and trigger the services via EventBridge instead
    options.EnableSnapshotService = false;
    options.EnableTtlCleanup      = false;
});

With the flags off, nothing polls — but SnapshotService.ProcessBatchAsync and TtlCleanupService.RunCleanupAsync are still ordinary public methods on classes you can resolve or construct. An EventBridge rule firing a Lambda nightly, a Kubernetes CronJob, a Hangfire recurring job, a console app under Task Scheduler: each one is four lines wrapping RunCleanupAsync. The archive write path needs none of this — the interceptor rides inside your normal request-scoped SaveChangesAsync — so the serverless deployment story is honestly better than the always-on one: pay for a container only when there's reaping to do.

That closes the write side, the storage, the reassembly, and the retention. What's left is the part your future self actually experiences: standing in front of the archive tables with a question, and getting an answer back out.

Next up, and last: querying the graveyard — typed columns, jsonb operators, the 202-polling snapshot endpoint, and the restore path that doesn't exist yet.