How Pactify Decides You Broke the Contract

Every test framework has a judgment engine at its core, and most of us never read it. We trust that “assert equals” means what we think it means. With contract testing that trust is riskier, because the judgment engine is the contract's enforcement — if the matcher is lenient where you believe it's strict, your green pipeline is testifying to something nobody checked.

So this post reads the judgment engine of Pactify — the open-source .NET contract testing library from DevMentors this series dissects — line by line. It's one class, HttpInteractionVerifier, about 140 lines. Reading it changed how I configure the library, and turned up two honest bugs I'd never have found from the documentation. This is the post where the "read your tools' source" thesis pays out.

Step zero: rebuilding the request

Each interaction replay starts by turning the recorded method string back into an HttpClient call:

// src/Pactify/Verifiers/HttpInteractionVerifier.cs
private Func<string, Task<HttpResponseMessage>> GetHttpMethod(string method)
{
    switch (method)
    {
        case "GET":    return path => _httpClient.GetAsync(path);
        case "POST":   return path => _httpClient.PostAsync(path, null);
        case "PUT":    return path => _httpClient.PutAsync(path, null);
        case "DELETE": return path => _httpClient.DeleteAsync(path);
        default: throw new PactifyException(ErrorMessages.UnknownHttpMethodDefined);
    }
}

Two boundaries of the whole tool are visible in ten lines. Only four verbs exist — record a PATCH interaction and verification throws. And look at the POST and PUT cases: the content argument is null. Requests are replayed without bodies — necessarily, since part 2 showed the request builder can't record one. A “when I POST this reservation payload” contract is not expressible: the request will arrive at your endpoint empty, your model binder will bind nulls or reject it, and the contract degenerates to “the route exists.” Pactify's home turf is read APIs and body-less commands; know that before you promise your architecture review board otherwise.

Also worth noticing: no request headers are sent either. If your provider routes on Accept or requires an API key middleware, the replayed request won't carry them — vs Pact proper, where request headers are recorded and replayed.

Status, then headers

The response arrives and three checks run in sequence, accumulating into an error list (never throwing early — part 5's “collect everything” philosophy continues down here). Status is one line: recorded 200 versus actual, mismatch appends an error. Headers are more interesting:

foreach (var header in definition.Response.Headers)
{
    var (name, value) = response.Headers
        .Concat(response.Content.Headers)
        .Where(h => h.Key == header.Key)
        .Select(h => (Name: h.Key, Value: h.Value.FirstOrDefault()))
        .FirstOrDefault();
    ...
}

The Concat is a genuinely thoughtful touch that anyone who's fought HttpClient will recognize: .NET splits response headers across two collections, and Content-Type — the header every pact records — lives on response.Content.Headers, not response.Headers. Miss that and every verification fails mysteriously. Pactify gets it right.

But look at the comparison: h.Key == header.Key. Ordinal, case-sensitive. Here's the part that surprised me: the IgnoreCasing option — which the consumer set believing it made matching case-insensitive — is not consulted for header names. It only applies to body property names, as we'll see below. Record "content-type" in lowercase (Pactify's own README example pact does exactly this!) and verification reports the header missing, regardless of options. HTTP header names are case-insensitive by RFC; the matcher is stricter than the protocol it's testing. The rule I derived: record header names in the exact canonical casing ASP.NET Core emits, and treat IgnoreCasing as a body-only switch no matter what its name implies.

Direction matters too: the loop iterates recorded headers and probes the real response. Extra headers in the real response are fine — the contract is a floor, not a ceiling. That's the correct semantics for consumer-driven contracts, and it holds for the body as well.

The body walk: one level deep

The heart of the engine. Both bodies are dictionaries by now — the expected body came out of the pact's ExpandoObjectConverter (part 3), the actual body via JsonConvert.DeserializeObject<ExpandoObject>(json):

foreach (var pair in (IDictionary<string, object>)expectedBody)
{
    var stringComparision = options.IgnoreCasing
        ? StringComparison.InvariantCultureIgnoreCase
        : StringComparison.InvariantCulture;

    var providedProperty = providedBody
        .FirstOrDefault(p => p.Key.Equals(pair.Key, stringComparision)).Value;

    if (providedProperty is null)
    {
        errors.Add(...);                        // property missing
    }
    else
    {
        if (options.IgnoreContractValues)
        {
            continue;                           // shape-only mode: done
        }
        var propertyHasExpectedValue = providedProperty.Equals(pair.Value);
        ...
    }
}

Read what it does — and doesn't do:

It walks the top level only. No recursion. If your recorded body has a nested dimensions object, the matcher checks that a property named dimensions exists, and in value-checking mode compares the two nested objects with… .Equals. Both are ExpandoObjects, which don't override Equals — that's reference equality between two objects deserialized from different JSON documents. It can never be true. A nested object in your contract, under IgnoreContractValues = false, is an unconditional failure. Under IgnoreContractValues = true it's a presence check only — nothing inside it is ever inspected, at any setting.

Value equality is CLR equality between Json.NET's guesses. Numbers deserialize as long or double; your recorded decimal 12 and the API's actual 12 can disagree at the type level (12L.Equals(12.0) is false) even when the JSON is semantically identical. Value mode is reliable for strings, booleans and like-typed integers, and treacherous past that.

null means "missing." The presence check is providedProperty is null — a property that's genuinely present with a JSON null value is indistinguishable from an absent one, and fails as missing. Given NullValueHandling.Include faithfully records nulls on the consumer side, there's a mild irony: the library preserves nulls in the contract but can't verify them.

Assemble the pieces and the design intent is clear: Pactify is a structural contract checker. Its native gear is IgnoreContractValues = true — “every property my consumer reads still exists, at the casing I expect, with a 200 and the right content type.” That check is exactly the regression that renamed field from part 1's outage, and Pactify catches it perfectly, in-memory, every build. The value-checking mode reads like an aspiration the matching engine doesn't yet have the machinery (matchers, recursion, type coercion) to honour. Where Pact proper lets you say “a string shaped like this regex,” Pactify lets you say “this key exists.” Configure it for what it is and it will not lie to you.

The two bugs in the error path

Now the part you'd only ever find in the source. Here's the error helper and one of its call sites:

private static string GetErrorMessage(string message, params object[] messageParams)
    => string.Format(message, messageParams);

// header value mismatch:
var message = GetErrorMessage(ErrorMessages.IncorrectReposnseHeaderValue,
    header.Key, value);

And the format string it's filling (typo in the constant name is as shipped):

public static readonly string IncorrectReposnseHeaderValue =
    "Expected response header {0} to have value {1}, but was {2}";

Three placeholders. Two arguments. string.Format with a {2} and only two args doesn't produce a truncated message — it throws FormatException. The body-value mismatch path has the identical flaw ("Expected response body property {0} to have value {1}, but was {2}" fed only the expected and actual values — the property name was never passed at all). So the two most informative failure messages in the library — which value was wrong — are unreachable: any run that would produce them dies with FormatException: Index (zero based) must be greater than or equal to zero... instead of a diagnostic.

I hit this in anger before I understood it: a provider deliberately broken to test the pipeline produced not “property price expected 12 but was 13” but a bare FormatException from deep inside the verifier, which I spent an embarrassing while attributing to my own Startup. Only reading ErrorMessages.cs — typos, argument lists and all — resolved it.

Why does a shipped library have crashes on two assertion paths? Because of everything above: in the library's natural mode (IgnoreContractValues = true, exact-cased headers, no header-value assertions beyond Content-Type which rarely drifts) these branches never execute. The happy paths and the presence-check failures — the ones the README demos exercise — are solid. The untravelled branches went untested. There's no snark in this observation: it's a small young OSS project (these are precisely the “good first PR” fixes — one-line argument additions, plus a test), and exactly the same class of dormant bug lives in the untravelled branches of code I maintain too. The lesson isn't “Pactify bad.” It's that your test tooling's failure paths are load-bearing and nobody exercises them — for any tool whose green checkmark you stake deploys on, the hour spent reading its judgment engine is the highest-yield code reading you'll do.

What I actually configure now

My post-source-reading defaults: IgnoreContractValues = true always; record only properties the consumer truly reads (anonymous objects, flat where possible — nested objects are presence-checks at best); header expectations limited to Content-Type in ASP.NET Core's canonical casing; GET and DELETE interactions only, since POST/PUT replay bodiless; and one deliberately-broken provider run in every new setup, so the team sees what failure output actually looks like before trusting the green.

Next post we leave the core package for Pactify's other NuGet: Pactify.AspNetCore, which answers a different question entirely — what if the provider could serve its contract classes over HTTP, discovered by attribute?