Service-to-Service Trust: mTLS in the Chassis

User authentication answers “who is this person?” — part 17's JWTs. But most traffic inside a microservices estate has no person in it: it is Orders calling Pricing, a scheduler calling everyone, a batch job with a connection string. The question becomes "is this machine allowed to talk to me?", and the strongest widely deployed answer is mutual TLS — the caller proves possession of a certificate's private key, the callee checks the certificate against policy.

Convey — DevMentors' open-source toolkit; I read and run it, I didn't build it — ships this in Convey.WebApi.Security, with a small crypto-helper sibling in Convey.Security. It is the one security-critical package in the toolkit, its samples actually exercise it (unlike the JWT packages), and it contains two of the most instructive mis-defaults in the entire codebase. That combination is exactly why it deserves a careful read.

The shape: a middleware and an ACL

Wiring is the usual chassis pair — AddCertificateAuthentication() binding the security section, UseCertificateAuthentication() in the pipeline:

"security": {
  "certificate": {
    "enabled": true,
    "header": "Certificate",
    "allowedDomains": [ "internal.example.com" ],
    "allowSubdomains": true,
    "acl": {
      "orders-service": {
        "validIssuer": "internal-ca",
        "validThumbprint": "<thumbprint>",
        "permissions": [ "pricing:read" ]
      }
    }
  }
}

CertificateMiddleware then runs every request through a sequence: extract the client certificate, verify its chain, look its subject up in the ACL, check issuer/thumbprint/serial against the entry, and finally delegate fine-grained checks to an ICertificatePermissionValidator. Failures split correctly: no certificate or a bad chain is 401 (you are nobody), a valid certificate that fails the ACL is 403 (I know who you are; no). Getting that distinction right matters more than it looks — it is the difference between “renew your cert” and “call security” when you are reading logs at 2am.

The ACL keys on certificate subject, with CN= normalization, and allowSubdomains + allowedDomains pre-expand into a subject set at startup. Identity is the subject; trust is the issuer/thumbprint pins; authorization is the permissions list. Three concerns, one config block — dense, but the right three concerns.

The forwarded-certificate trick

The middleware's most production-shaped feature is how it gets the certificate. In the textbook mTLS diagram, TLS terminates at your Kestrel and HttpContext.Connection.ClientCertificate is populated. In real estates, TLS usually terminates earlier — a gateway, an ingress, a layer-7 proxy — and the connection your service sees carries no client cert at all. The standard workaround is for the proxy to pass the certificate along in a header, and Convey wires ASP.NET Core's certificate forwarding to decode it:

// Convey.WebApi.Security/Extensions.cs
builder.Services.AddCertificateForwarding(c =>
{
    c.CertificateHeader = options.Certificate.GetHeaderName();
    c.HeaderConverter = headerValue =>
        string.IsNullOrWhiteSpace(headerValue)
            ? null
            : new X509Certificate2(StringToByteArray(headerValue));
});

The header value is hex-decoded into the raw certificate bytes and rehydrated into an X509Certificate2. Which immediately raises the question every forwarded-credential design must answer: what stops a caller from just… sending that header themselves? The certificate in the header is only the public part — the proxy already did the proof-of-possession handshake — so the entire scheme's security now rests on one invariant: the header must only be settable by your proxy. Strip it at the edge, accept it from the proxy's address only, or the middleware is authenticating a copy-pasteable string. The package cannot enforce that for you; it is a deployment property. Write it in the runbook.

(Two housekeeping notes for source readers: the rehydrated X509Certificate2 is never disposed — a handle leak under sustained load — and chain verification honors a skipRevocationCheck flag, defaulting to online revocation checking, with each chain element properly disposed and chain-status failures logged.)

Two defaults that look like security

Here is where the package becomes a teaching text.

The permission validator that always says yes. The last step of the pipeline — the fine-grained permissions check — is delegated to ICertificatePermissionValidator, and the default implementation returns true. Unconditionally. The hook exists so you can plug in real logic, and defaulting a hook to permissive is a defensible library choice — but the configuration surface tells a different story: you can populate permissions: ["pricing:read"] in the ACL, watch requests flow, and reasonably believe those strings are being enforced. They are checked by nothing. A default that renders visible configuration inert is a trap regardless of intent. If a knob does nothing until you write code, the knob should not accept values silently.

The host allow-list that trusts the caller. allowedHosts lets listed hosts skip certificate checks entirely — a pragmatic escape hatch for health probes and legacy callers. But the middleware matches it against the request's Host or its x-forwarded-for header — a value the caller controls. Any client that sends x-forwarded-for: <allowed-host> walks past mTLS entirely. Behind a trusted proxy that overwrites the header this is survivable; exposed directly, it is a bypass with documentation. The same header-trust mistake appears in the Prometheus endpoint guard (part 20) — it is a pattern to grep for in any codebase, including yours: any security decision keyed on a client-suppliable header is a decision the client makes.

Neither of these is exotic. That is the point. They are the two most common ways security middleware fails review: the default that doesn't enforce, and the trust decision made on forgeable input.

The crypto drawer

Convey.Security, the sibling package, registers IEncryptor, IHasher and ISigner — AES encryption, SHA-256 hashing, RSA signing with an X509Certificate2, the natural partner to a PKI story where Vault issues the certs (part 16). The signer is fine for its purpose. The encryptor I would not ship: the key is Encoding.UTF8.GetBytes(yourKey) with no key-derivation function (a passphrase either breaks or is the key material), the mode is unauthenticated CBC (malleable ciphertext, no integrity), and two of the byte[] overloads are outright buggy — one writes the array's type name into the stream, one wires the decrypt CryptoStream in the wrong direction. The string-based paths that the toolkit itself uses work; the rest is a reminder that crypto helpers are the one part of a chassis where “small and readable” is not a virtue that substitutes for review. In 2026, IDataProtector or AES-GCM via AesGcm is the boring correct answer.

The checklist this package writes

Machine identity via certificates remains the right idea, whether you hand-roll it like this or let a mesh do it with SPIFFE ids. What this source adds to the checklist: 401 versus 403 must mean different things; forwarded credentials are only as trustworthy as the invariant that only the proxy can set them; permissive defaults behind configurable-looking knobs are traps; client-controlled headers must never gate security; and the crypto drawer needs an owner with opinions.

Convey got the shape of all of this right in a few hundred lines — which is exactly why its two mis-defaults are so valuable. They are the mistakes competent code makes, preserved in the open where we can learn from them cheaply.

Next: back to the pipeline's bloodstream — the logger you configure once, and the correlation context that makes forty services read like one log.