Widen The Net Before You Trust The Silence

When you set up diagnostics for an intermittent fault, don't shape the capture around the error you can see. Capture the whole class of failure, then prove the capture works before you accept an empty result. A net built around your hypothesis can only ever catch your hypothesis, and silence from it tells you almost nothing.

I watched this play out recently while a couple of engineers chased intermittent gateway timeouts on a staging web server. Users saw a 504. So the sensible first move was to switch on failed-request tracing and write a rule that captured 504 responses. They reproduced the problem, the client plainly showed a 504, and the trace folder stayed empty.

That moment is where a lot of investigations quietly go wrong. An empty log invites a conclusion. Maybe the problem isn't on this server. Maybe it's the network. Maybe it's a different tier entirely. Each of those is a perfectly plausible story, and each one sends you off to a different part of the system on the strength of a capture that may simply not have been looking in the right place.

The status code the user sees is a symptom

The first correction was to widen the rule from one status code to the whole 500 to 599 range. That's a better default for a simple reason: a 504 is a statement about what some layer reported, not necessarily about where the fault started. Anything in the 5xx range means something broke badly enough that the server noticed. Capturing the range costs very little and removes a guess you didn't need to make.

It still wasn't enough, and the reason is the interesting part. Before trusting the setup, they did something I wish more people did. They deliberately broke something on the server, in a way that should produce a failure, and watched what happened. The server returned a 400. The browser, meanwhile, still presented what looked like a 504-style failure.

So the code in front of the user was being produced somewhere above the real fault. A layer in the front-end path was taking an underlying error and turning it into a timeout-shaped one. A rule written for 504s would never have fired, no matter how many times they reproduced the problem, because the server never said 504 in the first place. The rule was fine. The assumption underneath it was wrong.

They widened the capture again, down to include the 4xx range, and from then on the trace had something to say.

Prove the net can catch something

The step I'd keep even if I forgot everything else is the proof that the capture itself works. Tracing is one of those facilities that fails by saying nothing. If it can't write to disk because of a permissions problem, or the rule never matches, or the provider isn't enabled, you don't get an error. You get an empty folder, which looks exactly like a system that has nothing to report.

The cheap fix is to temporarily add a rule that should match something boringly common, such as successful responses, and confirm files appear. That rule is noisy and has to be removed afterwards, which is the sort of thing worth writing down so it doesn't linger. But for a few minutes it converts "the log is empty" from an ambiguous observation into evidence. If the happy-path rule writes files and the failure rule doesn't, you've learned something real about the system rather than about your tooling.

I think of it as the same instinct as testing a smoke alarm before you rely on it. Pressing the button feels slightly silly when nothing is on fire. It's much sillier to discover the battery was flat after the fire.

Why we narrow the net in the first place

It's worth being fair to the original instinct, because narrowing is rarely laziness. Broad captures are noisy. On a busy server a wide rule produces volume, and volume costs disk, attention and sometimes performance. Engineers under pressure reach for a precise filter because they want a clean signal, and a specific symptom feels like it deserves a specific trap.

The trouble is that precision is only useful once you know what you're looking for. During diagnosis you don't. That's the entire problem. A narrow filter is a prediction about the cause, and predictions made before you have evidence are exactly the thing you shouldn't be hard-wiring into your instruments. Start wide, accept the noise for a bounded time, and narrow once the evidence tells you where to look. The cost of the extra volume is temporary and the cost of chasing the wrong layer for a day is not.

There's also a human cost to getting this wrong that doesn't show up in any log. Every hour spent investigating the wrong tier is an hour someone else is waiting, and a person waiting on a fix tends to hear "we couldn't find anything" as "we're not taking it seriously". Teams lose trust in a diagnosis process when it produces confident dead ends.

The surprise that wasn't part of the plan

One more detail from the session is a useful reminder. During the same exercise they hit an unrelated 404, and it turned out to be caused by someone else making a change on the shared environment while the test was running. Nobody had planned for that, and it would have been easy to fold it into the story as part of the original fault.

Because the capture was wide enough to record it, they could see it was a separate event and set it aside. A narrow net wouldn't have shown it at all, or worse, would have shown just enough to be misleading. Wide captures don't only help you find the thing you're looking for. They help you identify the things you aren't looking for, so you can stop mistaking them for it.

What I'd do differently next time

If I were handing this to someone as a habit rather than a story, it would be small. Capture the class of failure, not the code you suspect. Break something on purpose and watch which code actually comes out. Add a rule that must match and confirm it does, then remove it. Treat any empty result as unverified until the capture has shown it can see something.

None of it is clever. That's rather the point. The expensive mistakes in diagnosis rarely come from missing some sophisticated technique. They come from trusting an instrument we never checked, and then reading its silence as an answer. A system that says nothing hasn't told you it's healthy. It has only told you that, so far, you haven't asked it the right question.