A Failing Test Should Explain Itself

When a test fails in an environment you can't easily reproduce, resist the urge to fix it. Change the test so the failure explains itself, and only then go looking for the cause. A red build that says nothing more than "expected success, got an error" is not a diagnosis. It is an invitation to guess, and every guess costs a full pipeline run.

I was reminded of this recently on a small side project. A smoke test that booted the web app and checked that its front-end bootstrap loaded passed happily on my Windows machine and failed on the Linux CI runner. The assertion reported a status code and nothing else. A status code is the least interesting thing a server says when something goes wrong. The interesting part is the body, which usually contains the actual exception, and the test was throwing it away.

Spend the first change on visibility

The temptation at that point is to start theorising. Perhaps it's file casing, because Linux cares and Windows doesn't. Perhaps it's a path separator. Perhaps a package behaves differently. Each theory is plausible, and each one costs a commit, a push, a wait for the runner and a fresh disappointment. I've done that loop enough times to know it rarely converges quickly, because plausible theories are cheap and the evidence is expensive.

So the first change was not a fix. The test now included the response body in its failure message. That single line of change turned "it failed" into "it failed, and here is what the server said". It also taught me something about the test itself: it had been written to answer "did it work?" and never "if not, what happened?". Those are different questions, and the second one is the one you need at the worst possible time, when you are looking at a log from a machine you can't sit in front of.

The next step followed the same logic. The server was catching its own exception and returning a bland error page, which is sensible behaviour in production and unhelpful in a test. Exposing server exceptions in the testing environment gave the runner something real to report. Only after those two changes did the actual cause appear, and it was almost embarrassingly small: the test host wasn't configured to serve the static assets the application assumed would be there. A configuration gap in the test harness, not a defect in the product.

Three small changes, and the last one was the only one that fixed anything. It would be easy to look at that and see two wasted steps. I see the opposite. Without the first two, the third would have been a guess that happened to land, and I'd have learned nothing about how to find the next one.

The same principle applies to the pipeline

There was a smaller version of the same problem in the pipeline itself. Test artefact uploads were hitting a storage quota, and that failure was being reported in a way that obscured whether the tests had passed. A build that fails because it couldn't save its own receipts tells you nothing about the thing you wanted to know. I changed it so the upload failure could no longer mask the test result. Reporting problems should be reported, but they shouldn't be allowed to talk over the answer.

Running pull request checks on a self-hosted Linux runner, with the database test container on that runner's network, was the third piece. That's more of an environment decision than a diagnostic one, but it served the same end. If production runs on Linux, the place that tells you the truth about it should be Linux too. A pipeline that quietly differs from where the software actually lives will eventually tell you something comfortable and wrong.

What it costs, and why it's worth it

None of this is free. Putting a response body into a failure message means thinking about what might end up in build logs, and anything sensitive has no business there. In a system that handles personal or health data I'd want that output scrubbed or restricted to a test environment with synthetic data. The same goes for exposing server exceptions: that is a testing-only switch, and it needs to stay one. Better diagnostics are not a licence to leak.

There's also a commercial angle I think we undersell. Every blind retry of a pipeline is a few minutes of waiting and a context switch for whoever is watching it. On a team, that multiplies. A failure message that carries its own explanation pays for itself the first time somebody who didn't write the test has to read it, which is nearly always a different person from the author, or the author six months later.

The habit I've come to prefer is this. When a test fails and the message doesn't tell you why, treat the message as the bug. Fix that first, on its own, even if it feels like a detour. If the failure is reproducible locally, you may never need the improved message. If it isn't, you will be glad of it, and so will the next person who meets a similar red build with no idea where to start.

A test that can only say "no" has done half its job. The other half is helping the person who has to act on the answer, and that is the half most of us forget until it's the middle of a long afternoon and the runner is on its fourth attempt.