The Watermark and the Walk

Parts 2 and 3 built the machine: attributes declared, pipeline compiled, registry populated. Nothing has migrated yet. VersionKit does its work at the last possible moment — when a payload is actually read — and this part is about that moment: the watermark that decides where the walk starts, the walk itself, the public API around it, and the traps System.Text.Json lays for any code that treats JSON as dictionaries.

Reading the stamp

Every payload VersionKit has ever written carries "__version": n. When a payload comes back in, the first job is reading that stamp — and the code is more interesting than it looks:

var version = payload.TryGetValue("__version", out var v) && v is JsonElement je
    ? je.GetInt32()
    : 1;

Two decisions are packed in there. The obvious one: a missing watermark means v1. Payloads written before you adopted VersionKit have no stamp; treating them as v1 means adoption requires no backfill, no touch of existing data — you declare history and old payloads just start walking.

The subtler one: the check only recognises JsonElement. When System.Text.Json deserialises into Dictionary<string, object?>, every value — numbers included — arrives as a JsonElement, so payloads that came from JSON parse correctly. But a dictionary assembled in code, with a plain int watermark, falls through to v1. The EF Core integration does exactly that (it stamps row["__version"] = registry.SchemaVersion with an int before migrating), which means those payloads get walked from the bottom regardless of their true version.

Why is that not a disaster? Because of Part 3's idempotency rule: every operation is conditional, so re-running v1→v2 against a v2 payload renames nothing (the old key is gone), defaults nothing (the key exists), removes nothing that matters. The walk wastes a few dictionary lookups and arrives at the right answer. I would rather the type check were broader — v is int i ? i : ... is on the list — but the system was designed so that getting the starting version wrong in the downward direction is safe. That property has saved me more than once, and it is the reason I treat operation idempotency as a hard invariant rather than a style preference.

The walk

The migrator's core loop was previewed in Part 1; here it is in full context:

public Dictionary<string, object?> MigrateTo(
    Dictionary<string, object?> payload, int targetVersion)
{
    if (targetVersion < 1 || targetVersion > CurrentVersion)
        throw new ArgumentOutOfRangeException(nameof(targetVersion), ...);

    var version = /* watermark read, as above */;
    if (version >= targetVersion) return payload;

    var current = payload;
    for (int from = version; from < targetVersion; from++)
    {
        int to = from + 1;

        if (_handWritten.TryGetValue((from, to), out var hw))
        {
            current = hw.Migrate(current);   // replaces the whole step — Part 5
            continue;
        }

        var next = new Dictionary<string, object?>(current);
        var step = _steps.FirstOrDefault(s => s.FromVersion == from && s.ToVersion == to);
        if (step is not null)
            foreach (var op in step.Operations)
                op.Apply(next);

        next["__version"] = to;
        current = next;
    }
    return current;
}

Details that matter in production:

• Each step works on a copy. new Dictionary<>(current) per step means a hand-written migrator that throws mid-walk cannot leave you holding a half-mutated payload; the last completed step's dictionary is intact.
• The watermark is stamped every step, not once at the end. If you stop early — deliberately or by exception — the payload honestly says where it got to.
• There is no downward path. A payload already at or above the target returns unchanged. VersionKit migrates history up; it never rewrites current data to look old. If you need “show me this as v2”, you take an old payload and stop the walk early — you never demote a new one.

The runner: the API you actually call

VersionedMigrator works in dictionaries; applications work in JSON strings and typed entities. MigrationRunner is the bridge, injected as a singleton:

var result = runner.Migrate<Order>(storedJson);

// result.Entity       — a fully-typed, current-version Order
// result.FromVersion  — the version the payload was stored at
// result.ToVersion    — the version it was migrated to
// result.WasMigrated  — false if it was already current

The result record is small but deliberate. WasMigrated is the hook for observability — the sample API surfaces it directly, and in a real system it is the number you chart to watch lazy migration debt shrink as the backfill (Part 6) does its work. FromVersion tells you what your storage actually contains, which is otherwise invisible once reads always return the current shape.

Inside, the runner resolves the migrator from the registry, handles the case where T is not versioned at all (straight deserialisation, WasMigrated = false — so you can run every read through the runner without special-casing), runs the walk, and then does the final typed hop:

private static T Deserialise<T>(Dictionary<string, object?> payload) where T : class
{
    // round-trip through JSON to leverage System.Text.Json's deserialiser
    var json = JsonSerializer.Serialize(payload, _jsonOpts);
    return JsonSerializer.Deserialize<T>(json, _jsonOpts) ?? throw ...;
}

Yes: serialize the dictionary back to JSON, then deserialise into T. It looks wasteful, and it is — one extra serialisation per migrated read. The alternative is hand-rolling a dictionary-to-object mapper that understands nested objects, collections, nullable value types, custom converters, case-insensitivity… which is to say, reimplementing System.Text.Json badly. The round trip buys total fidelity with the serializer's behaviour (the options used are PropertyNameCaseInsensitive = true, ignore-nulls-when-writing) at the cost of microseconds on a path that already did I/O. I measured the temptation and kept the round trip.

The JsonElement tax

Part 1 promised the System.Text.Json gotchas; here is the bill. The moment you deserialise JSON into Dictionary<string, object?>, nothing in that dictionary is a CLR primitive. "Total": 1500.00 is not a double; it is a JsonElement with ValueKind = Number. "IsExpedited": "true" is a JsonElement string. Any code that pattern-matches on raw is string silently fails.

Every VersionKit extension point that touches raw values has to cope. The sample's computed-default provider is the canonical example:

public object? GetDefault(IReadOnlyDictionary<string, object?> payload)
{
    if (!payload.TryGetValue("Total", out var raw)) return false;

    var total = raw switch
    {
        decimal d                            => d,
        double  d                            => (decimal)d,
        JsonElement je
            when je.TryGetDecimal(out var d) => d,
        _                                    => 0m
    };
    return total >= HighValueThreshold;
}

The decimal/double arms are there because payloads do not only come from JSON — the EF path (Part 8) builds dictionaries from DbDataReader, where Total really is a decimal. A provider that handles both sources is a provider you never have to think about again. The same defensive shape appears in the type-migrator adapter (Part 5), which coerces JsonElement values into TFrom before invoking your conversion.

If you write your own providers or hand-written migrators, this is the rule to internalise: values are JsonElement when the payload came from JSON, CLR types when it came from a database row, and your code should accept either. It is the least elegant part of building on System.Text.Json, and also non-negotiable — the alternative serializers with looser object models bring their own, worse, surprises.

Reading history on purpose

Because the walk takes a target, it can stop early:

// how did this order look under the v2 schema?
var result = runner.MigrateTo<Order>(storedJson, targetVersion: 2);

I added this for testing — it lets you assert each step in isolation, walking a v1 fixture to v2 and checking exactly the v2 shape. But it earns its keep in anger for audit questions. “What did the system say this order was, at the time?” is usually answered with archaeology: restore a backup, check out a two-year-old tag, squint. With versioned payloads it is a parameter: take the stored v1 payload, walk it to the version that was current at the date in question, and you are looking at the entity as that era's code saw it. The sample API exposes it as GET /migrate/order/{id}/version/{v} and it is the demo that consistently lands with compliance-minded audiences.

One caveat for honesty: the target-version read migrates stored old data up to an intermediate version. Deserialising the result into the current T (which is what the generic API does) means fields that did not exist at the target version simply come back defaulted. For true point-in-time fidelity you inspect the returned payload dictionary or keep era-specific DTOs; the library gives you the correctly-shaped data, and what type to pour it into is your call.

Lazy migration now works end to end: stamp, walk, deserialise, result. But two of the walk's participants have only been waved at — the hand-written migrator that hijacks a step, and the type migrator doing conversions inside one. They are the escape hatches, and they deserve their own part.

Next: When Attributes Aren't Enough — hand-written migrators, type migrators, and computed default providers, up close.