Rebuilding Objects That No Longer Exist

There is a moment in this library that still feels faintly illegal to me: a background service assembles a complete parent-child JSON document for an order whose row was physically deleted minutes ago. No primary table is queried — there's nothing there to query. The reconstruction runs entirely on archive rows, joined by the correlation id from part 6, using the ScalarPayload copies from part 4. This post walks the machinery: the job ticket, the state machine, the assembly, and the limitation I owe you before you find it yourself.

A ticket, not a task

Part 2 showed the interceptor's restraint: when Snapshot = true on the root, it writes a row, not a result — Status = Pending, Payload = null, correlation id attached — in the same atomic batch as the delete. That makes the snapshot table an outbox in all but name: the request for expensive work commits with the transaction that made the work necessary, so no crash window can lose it. The job carries its own lifecycle:

public enum SnapshotStatus
{
    NotRequested = 0,  // Snapshot=false — no snapshot work queued
    Pending      = 1,  // ticket created — service hasn't run yet
    Complete     = 2,  // Payload is populated
    Failed       = 3   // FailureReason is populated
}

plus RetryCount, CompletedAt, and FailureReason — everything an operator needs to answer “where is my snapshot?” with a query instead of a log dive. The Status column is indexed (part 4), so the pending-jobs poll and your “anything stuck in Failed?” dashboard query both stay cheap as the table grows.

The loop and its escape hatch

SnapshotService is a plain BackgroundService polling on an interval (default 10 seconds), and its entire poll body is a try/catch around one public method:

/// <summary>
/// Processes one batch of pending snapshot jobs.
/// Call this directly from a Lambda handler instead of running the hosted service.
/// </summary>
public async Task ProcessBatchAsync(CancellationToken ct = default)

That method being public is a design position, not an accident: the poll loop is packaging, the batch is the product. Hosting styles come and go — Kubernetes wants a hosted service, Lambda wants a handler, a cron container wants a console call — so the work is shaped as “process one batch, then return,” and every host wraps it however it likes. Part 8's TTL service repeats the pattern.

Inside a batch, the service fans out over the registry, skipping types that never asked for snapshots, and pulls pending jobs oldest-first through the IArchiveDbContext abstraction:

foreach (var entityType in ArchiveRegistry.AllArchivableTypes())
{
    var attr = ArchiveRegistry.GetAttribute(entityType);
    if (!attr.Snapshot) continue;

    var snapshotType = ArchiveRegistry.GetSnapshotType(entityType);
    var archiveType  = ArchiveRegistry.GetArchiveType(entityType);

    var pendingJobs = await db.GetPendingSnapshots(snapshotType, _options.BatchSize, ct);
    ...
}

IArchiveDbContext is worth a pause. The service needs to query your tables in your context without referencing your assembly, so the library defines a three-method interface — GetPendingSnapshots, GetArchivesByCorrelation, SaveChangesAsync — and you implement it on your DbContext (the sample does it with EF's untyped Set(Type) plus a covariant cast). It's the dependency-inversion move that keeps ArchiveKit.EFCore ignorant of AppDbContext while still doing real work inside it.

Assembly from evidence

The reconstruction itself is almost embarrassingly simple, which is the point — all the hard work was done at archive time:

var group = await db.GetArchivesByCorrelation(archiveType, correlationId, ct);

var root     = group.FirstOrDefault(a => a.IsRoot) ?? group.First();
var children = group.Where(a => !a.IsRoot).ToList();

var rootScalars = DeserializeScalars(root.ScalarPayload);

var childrenByType = children
    .GroupBy(a => a.GetType().Name.Replace("Archive", ""))
    .ToDictionary(
        g => g.Key,
        g => g.Select(a => DeserializeScalars(a.ScalarPayload)).ToList());

var tree = new
{
    root     = rootScalars,
    children = childrenByType,
    meta     = new { correlationId, archivedAt = root.ArchivedAt, entityCount = group.Count }
};

job.Payload     = JsonSerializer.Serialize(tree, _jsonOptions);
job.Status      = SnapshotStatus.Complete;
job.CompletedAt = DateTime.UtcNow;

Fetch the correlation group, crown the root (with a graceful fallback if IsRoot is missing), bucket the rest by type name, and inflate every ScalarPayload back into a dictionary. The output is a self-describing document — root, children keyed by entity name, and a meta block — stored as jsonb in the job row. Note what makes this deletion-proof: the payloads were captured from the change tracker before the delete committed, so the service is assembling from evidence, not from live data. The primary tables could be truncated and this code wouldn't notice.

Failure gets a state, not a log line

The error handling is the part I'd defend hardest in review:

catch (Exception ex)
{
    job.RetryCount++;

    if (job.RetryCount >= _options.MaxRetries)
    {
        MarkFailed(job, ex.Message);   // Status=Failed, FailureReason, CompletedAt
    }
    // else: leave as Pending — retried next poll cycle
}

A failed attempt increments the counter and leaves the job Pending, so retry costs no scheduler, no queue, no delay bookkeeping — the next poll simply picks it up again, which also means the retry interval is your poll interval. After MaxRetries (default 3) the job parks in Failed with the reason attached: terminal, visible, queryable. And one edge is pre-decided rather than left to the exception handler — a correlation id with zero archive rows fails immediately, no retries, because absent rows won't materialize on attempt two. Retrying the unretryable is how queues fill with zombies.

The honest cost of this minimalism: retries are same-interval rather than backed off, ex.Message without stack trace is thin forensics for the terminal state, and a poisoned job burns three attempts of real work before parking. All true, all acceptable at the intended scale — snapshot assembly touches a handful of rows, not external systems, so the classic reasons for exponential backoff barely apply.

The gap: one table's worth of children

Now the limitation. Look again at the fetch: GetArchivesByCorrelation(archiveType, correlationId) — one archive type, and the service passes the root entity's. So for an order-with-lines deletion, the query hits order_archive only. The OrderLineArchive rows sharing the correlation id live in order_line_archive, and the current assembly never reads them. The children bucket, in practice, holds only non-root rows from the root's own table (say, a second order archived in the same save) — not the lines. Part 1 described snapshots as capturing the parent and its lines; the schema fully supports that, the correlation id spans both tables, but the service as written stops one table short.

The fix is small and shaped like the loop that already exists — iterate every registered archive type, query each for the correlation id, merge:

var group = new List<IArchiveEntry>();
foreach (var t in ArchiveRegistry.AllArchivableTypes())
    group.AddRange(await db.GetArchivesByCorrelation(
        ArchiveRegistry.GetArchiveType(t), correlationId, ct));

at the cost of one query per registered type per job (fine at small registries; a UNION-style batched fetch is the eventual answer). It's unwritten for the usual unglamorous reason — the single-table version proved the pipeline end to end — but it's the first line of the library's changelog-in-waiting, and until it lands, treat snapshots as root-plus-same-table documents.

Snapshots complete the write side of the story: every deletion now leaves a typed archive row, a jsonb payload, and — for the entities that want it — a reassembled tree. All of it accumulating forever, which is its own quiet failure mode.

Next up: TTL policies, the cleanup service, and giving every archive an expiry date — the timestamp part 1 promised you.