Load Order Decides What Your Traces See

Your gateway's error rate in Jaeger is zero. It has been zero all week, which felt reassuring until somebody pulled the application logs and found a wall of unhandled exceptions being converted to 400s at the edge. The traces are all there — one span per request, correct timings, no gaps — and every single one is unflagged. Nothing is broken. Nothing is misconfigured. Two assemblies loaded in an order that happens not to suit you.

Part 3 established that Ntrada finds its extensions by reflecting over already-loaded assemblies. This part follows what that scan produces into the middleware pipeline, because the sequence it hands over is the sequence your requests run through.

One line of sorting

Order is an int? on IExtensionOptions, and it is consumed in exactly one place:

_extensions = new HashSet<IEnabledExtension>(extensions.OrderBy(e => e.Options.Order));

— src\Ntrada\Extensions\ExtensionProvider.cs:45

Four things are true about that line, and three of them are problems.

null sorts first. Comparer<int?>.Default ranks null below every value, so an extension with no order: runs before one with order: 1. If you want something last you must number everything, or number the one you care about and accept that every unnumbered extension precedes it. That is the opposite of the intuition most people bring to an optional priority field, where “unspecified” usually means “I do not care, put me anywhere convenient.”

The ordering is thrown away at the type level and preserved by accident. OrderBy returns an ordered sequence; wrapping it in new HashSet<IEnabledExtension>(...) discards every guarantee. ISet<T> promises nothing whatsoever about enumeration order. .NET's HashSet<T> happens to enumerate in insertion order when no elements have been removed, so this works today, on this runtime, for this usage — and both consumers, AddExtensions and UseExtensions, depend on it. It is correctness by implementation detail, in the one place where correctness is ordering.

When Order is null everywhere, the tiebreak is the reflection scan. OrderBy is documented as stable, so equal keys keep source order — and source order here is AppDomain.CurrentDomain.GetAssemblies().SelectMany(s => s.GetTypes()). That is assembly-load order crossed with metadata-token order within each assembly. The middleware pipeline of an un-ordered Ntrada gateway is sequenced by the order the CLR happened to load its extension assemblies in.

No shipped configuration sets order: anywhere. Not samples\Ntrada.Samples.Api\ntrada.yml, not src\Ntrada.Host\ntrada.yml, not the README's 124-line advanced block, not one of the six per-extension .yml fragments. The key is real, bound, consumed and completely undemonstrated.

The fourth fact is the only place an operator can see any of this:

extension.Extension.Use(app, optionsProvider);
var orderMessage = extension.Options.Order.HasValue
    ? $" [order: {extension.Options.Order}]"
    : string.Empty;
logger.LogInformation($"Enabled extension: '{extension.Extension.Name}' " +
                      $"({extension.Extension.Description}){orderMessage}");

— src\Ntrada\NtradaExtensions.cs:281-286

Read the shape of that log line. It is emitted after Use has already run, so the pipeline is built before you are told about it. And when Order is null the message says nothing about ordering at all — so the default configuration produces a startup log that lists your extensions in their effective order while giving you no signal that the order was arbitrary. The log tells you the sequence and hides the fact that it was not chosen.

Two middlewares that disagree about exceptions

Now the consequence, and it is specific. Two of the six extensions register middleware, and they treat exceptions in exactly incompatible ways.

The error handler catches and does not rethrow:

public async Task InvokeAsync(HttpContext context, RequestDelegate next)
{
    try
    {
        await next(context);
    }
    catch (Exception exception)
    {
        _logger.LogError(exception, exception.Message);
        await HandleErrorAsync(context, exception);
    }
}

— extensions\Ntrada.Extensions.CustomErrors\ErrorHandlerMiddleware.cs:21-32

The Jaeger middleware tags the span and does rethrow:

span.Log($"Processing HTTP {method}: {context.Request.Path}");
try
{
    await _next(context);
}
catch (Exception ex)
{
    span.SetTag(Tags.Error, true);
    span.Log(ex.Message);
    throw;
}
finally
{
    scope?.Dispose();
}

— extensions\Ntrada.Extensions.Tracing\JaegerHttpMiddleware.cs:28-44

Both are correct in isolation. ErrorHandlerMiddleware is a terminal handler and terminal handlers swallow; JaegerHttpMiddleware is an observer and observers rethrow. The whole question is which one is outermost, and in ASP.NET Core the first-registered middleware is the outermost.

• CustomErrors registers first. Pipeline is ErrorHandler → Jaeger → endpoint. An exception propagates up through Jaeger's catch, gets Tags.Error, is rethrown at line 39, and CustomErrors converts it to a 400. Traces show errors.
• Tracing registers first. Pipeline is Jaeger → ErrorHandler → endpoint. ErrorHandlerMiddleware catches the exception and returns a 400 without rethrowing. Jaeger's catch never executes. Every production failure becomes an untagged, apparently-successful span.

Which one you get is decided by the order the CLR loaded two assemblies in. Two extensions, no declared dependency between them, no vocabulary in the model for declaring one, and their composition determines whether your error rate is visible in your tracing system at all.

The failure mode is not “tracing is broken.” It is "tracing is working, and reporting success." That is strictly worse, because a broken observability tool gets fixed and a lying one gets trusted.

The same trap, one layer out

CORS has the mirror-image version of the problem. CorsExtension.Use is a single app.UseCors("CorsPolicy"). If ErrorHandlerMiddleware sits outside the CORS middleware, the 400 it produces is written on a response that never passes through UseCors, so it carries no Access-Control-Allow-Origin header. A browser then sees the gateway's error not as a 400 with a JSON body, but as an opaque CORS failure — the response is discarded before any JavaScript can read the errors array the gateway went to the trouble of writing.

Three extensions whose position in an ASP.NET Core pipeline materially changes observable behaviour, and one optional integer to arrange them with.

One order for two different jobs

There is a structural limitation underneath all of this that is worth naming separately, because it is the part a plugin author feels first.

GetAll() sorts once and memoises. Both AddExtensions, which runs inside ConfigureServices, and UseExtensions, which runs inside Configure, iterate the same cached set. So the same number governs service registration order and middleware pipeline order. There is no way to say “register my services before everyone else's so they can be decorated, but put my middleware innermost.” Those are genuinely different concerns with genuinely different correct answers, and the model collapses them into one int?.

What the fix costs, and what it was fair to expect

The fix an operator can apply today is two YAML lines:

extensions:
  customErrors:
    order: 1
    includeExceptionMessage: true
  tracing:
    order: 2
    serviceName: ntrada

That is it. Register the error handler outermost, the tracer inside it, and errors reach the span before they are converted. The mechanism works; it has always worked. What is missing is any statement, anywhere, that the choice exists — order: appears in no README, no sample, and no extension fragment in the repository.

Judged fairly against 2019, having an ordering knob at all put this model ahead of several plugin systems of the period, which simply loaded in scan order and let you find out. The int? is also the right type — it distinguishes “I asked for position 0” from “I have no opinion”, which a plain int cannot. The design instinct was sound.

The defect is that the model puts ordering entirely in the operator's hands and then never tells the operator that ordering exists. An extension author knows their middleware must wrap or be wrapped; that knowledge is available at the moment the extension is written and is thrown away because IExtension has nowhere to put it. Two members — a DependsOn and a Position — would have moved this decision from a YAML file nobody edits to the code that actually knows the answer.

There is a rule of thumb in here worth keeping: if the ordering of two components changes what your monitoring reports, the ordering is not configuration — it is a contract, and contracts belong in code.

Next, a status code that promises more than the code behind it delivers.