The Test Pyramid CI Never Runs

Part 4 closed on a needle: nothing in the estate would ever catch the missing indexes, because almost nothing runs. This part substantiates that, and the finding is stranger than “a sample with thin tests”. Pacco's CI and Pacco's tests are both present — twelve Travis pipelines, a genuinely well-built test pyramid — and they are almost perfectly disjoint. The repositories that run tests have none, and the repository with tests doesn't run them.

Twelve pipelines, one missing line

Every repository ships the same four scripts (build.sh, test.sh, dockerize.sh, start.sh) and a .travis.yml that calls them. test.sh is two lines:

#!/bin/bash
dotnet test

Here is the script: section of Deliveries' Travis file, which is also Customers', Identity's, Vehicles', Operations', OrderMaker's, Pricing's, Orders', Parcels', and both gateways':

script:
  - ./scripts/build.sh
  - ./scripts/test.sh

And here is Availability's — the only one in the estate that differs:

script:
  - ./scripts/build.sh

One line, deleted in exactly one repo. To see why that deletion is the most honest line of CI configuration in the estate, put it next to the test inventory:

Nine repositories run dotnet test against solutions containing zero test projects. dotnet test with nothing to test exits zero; the badge goes green; the badge has been green since 2020. Two repositories run it against one contract test each. And the one repository with a real pyramid — four unit tests, one integration test, five end-to-end tests, one performance test — is the one whose pipeline never executes them.

The pyramid that exists

Availability's tests deserve a proper look before we ask why CI skips them, because they are not filler — they are the estate's house style for testing, fully worked out in one repo. The unit layer reads like this:

// Pacco.Services.Availability.Tests.Unit/Core/Entities/CreateResourceTests.cs
public class CreateResourceTests
{
    private Resource Act(AggregateId id, IEnumerable<string> tags) => Resource.Create(id, tags);

    [Fact]
    public void given_valid_id_and_tags_resource_should_be_created()
    {
        var id = new AggregateId();
        var tags = new[] {"tag"};

        var resource = Act(id, tags);

        resource.ShouldNotBeNull();
        resource.Events.Count().ShouldBe(1);
        resource.Events.Single().ShouldBeOfType<ResourceCreated>();
    }
}

Note the idiom: an Act helper as the first line of the class, snake_case scenario names, Shouldly assertions — and in the larger test classes, a #region Arrange at the bottom of the file, inverting the usual AAA layout so the scenario reads before the plumbing. The assertions are semantic gold, too: they pin down that Resource.Create must raise exactly one ResourceCreated, that empty tags must throw, that the reserve handler must call both UpdateAsync and the event processor — the intended contracts of the domain, written down nowhere else.

The integration layer is the estate's most instructive test. It publishes a command to real RabbitMQ, then waits for the resulting event and the resulting document:

// Pacco.Services.Availability.Tests.Integration/Async/AddResourceTests.cs
[Fact]
public async Task add_resource_command_should_add_document_with_given_id_to_database()
{
    var command = new AddResource(Guid.NewGuid(), new[] {"tag"});

    var tcs = _rabbitMqFixture
        .SubscribeAndGet<ResourceAdded, ResourceDocument>(Exchange,
            _mongoDbFixture.GetAsync, command.ResourceId);

    await Act(command);   // publish AddResource to the "availability" exchange

    var document = await tcs.Task;
    document.Id.ShouldBe(command.ResourceId);
    document.Tags.ShouldBe(command.Tags);
}

Bus in, bus out, database asserted, one test — the whole command pipeline (subscription, handler, repository, event mapping, publish) verified in a single round trip. The fixture behind it keeps a small honest ledger of its own: the consumer callback deserialises the received message and then never uses it, the class implements the Dispose(bool) pattern without declaring IDisposable, and the RabbitMQ connection is never closed. And there is a subtler completeness problem underneath: handlers across the estate are internal sealed, so testing any service's internals requires an InternalsVisibleTo grant — a convention only Availability completed, and even there imperfectly, since one of its EnableInternalsTesting.cs files grants visibility to Pacco.Services.Availability.UnitTests, a project that does not exist (the real one is Tests.Unit; a second, correct attribute elsewhere saves it). Testing the other nine services would begin with plumbing work each repo would have to copy — the consumer tax again, now levied on tests. The end-to-end layer drives HTTP through a WebApplicationFactory subclass that forces a tests environment, asserting the 201, the Location: resources/{id} header, and the document. And the performance layer is an NBomber scenario inside an xUnit [Fact], firing GETs at a hardcoded http://localhost:5001/resources for three seconds and asserting at least 100 requests per second — a smoke SLA you run by hand, wearing a unit test's clothes.

Why the one honest repo went dark

Look again at what those tests need: the integration test wants a live RabbitMQ on localhost; the integration, e2e, and Pact fixtures want a live Mongo; NBomber wants the actual service already running on port 5001. There are no Testcontainers here — this is 2020, netcoreapp3.1, and the fixtures connect to localhost and hope. Travis, as configured in these repos, provisions none of that. Run dotnet test on Availability's pipeline and the pyramid fails not because the code is wrong but because the world is missing.

So someone made the honest edit: they removed the test step from the one pipeline where it would fail meaningfully, and left it in the nine pipelines where it succeeds meaninglessly. I want to be fair about the shape of the mistake — no one decided “let's not test”; each repo made a locally sensible move, and the estate-level absurdity emerged from twelve locally sensible moves that no single pipeline can see — the polyrepo trade-off from part 2, wearing CI clothes. But the emergent result stands: a green badge on eleven of twelve Pacco repos certifies that the test step ran, not that any test exists. My rule of thumb since reading this estate: any CI step that cannot fail is worse than no step, because it manufactures confidence at zero cost.

The two repos where dotnet test finds real work — Orders and Parcels — are only a partial exception. Each contains exactly one test: a Pactify consumer test that writes a contract file, and a provider test that reads it back — through the file path ../../../../../../pacts, six directories up, outside the repository, in a folder that only exists if you cloned the whole organisation side by side, with the provider additionally requiring a seeded local Mongo. Whether that pair can honestly pass on a clean CI runner is its own story, and it is the next part's.

The untested middle

One more row of the ledger, and it is the sharpest. Customers and Pricing have no tests at all — no unit project, no fixture, nothing — while their Travis files dutifully run dotnet test. These are not incidental services. Customers owns the state machine that gates the whole estate (a customer must be Valid before anything can be reserved) and the VIP policy that flips accounts at twenty completed orders. Pricing owns the only genuine arithmetic in the system, the loyalty discount ladder — the code where an off-by-one changes what people pay. The services with the most business math are precisely the untested ones, and the pattern generalises beyond this estate: infrastructure-heavy services attract tests because their seams are visible, while calculator services look “too simple to test” right up until the discount ladder quietly charges full price. What is verified anywhere is worth stating precisely: one aggregate's domain events, one command's bus round trip, one HTTP surface, one paged query, and one GET contract between two services. Everything else — every saga step, every rejected-event path, every consistency mechanism this series and its siblings have examined, including the outbox the domain series showed running with transactions off — ships on faith.

Next, the two tests that do run get their own part: pacts through a folder six levels up — consumer-driven contracts with no broker, a Huge Weapon fixture, and shape-only verification, as the estate's entire answer to “how do ten services stay honest with each other”.