A Status Code Is A Symptom Not A Diagnosis

When an intermittent failure turns up everywhere, resist the urge to treat its error code as one problem. A gateway timeout is the name of a symptom: something upstream didn't answer in time. Dozens of different faults can produce exactly that sentence. If you group your failures by the code and then hunt for the cause, you are quietly assuming the thing you are trying to find out, which is that there is only one cause.

I've been living with a version of this over the last couple of weeks. A production system started returning timeouts for some users. It was first reported from one part of the country, and for a while that was the whole story: one region, one problem, find what is special about that region. Then reports arrived from another state. Then the error tracker, which records every failure whether or not anyone bothers to complain, showed the same timeout scattered across a good many users and places, some of whom hadn't noticed or hadn't thought it worth mentioning. One person looking at it found it in their own history from days earlier and admitted they hadn't even noticed at the time.

The first scope is the reporter's scope

The footprint of a bug report is the footprint of the people who reported it. That sounds obvious written down, yet in the moment it behaves like data about the system. "Only one region" felt like a clue, and a seductive one, because it points at something concrete: a network path, a front door, an address range. Some of those suspicions may even be right. That region does have its own entry point, so it is a place where this kind of fault could be easier to see or worse to experience.

But the reports were never a map of where the failure happens. They were a map of who noticed, who had a reason to write in, and who happened to be doing something that day that made a timeout painful. A clinician halfway through a task complains. A background refresh that fails quietly does not. Before you build a theory on geography, find a record of failures that doesn't depend on anyone's willingness to tell you about them. The error tracker showed a far wider picture within minutes of someone sharing a screen, and it changed the conversation from "what's wrong with that region" to "what is common to all of this", which is a much better question.

One code, at least two stories

The second trap is subtler. Once the error tracker was open, everything under the one timeout heading looked like a single population. It was, by construction: the tool had grouped by the error. That grouping is a convenience, not a finding.

At the same time, the same timeout was being seen in a test environment, mostly from automated runs, and almost never through normal human use. In production, a couple of weeks earlier, someone had been able to trigger it every single time for a window, and then it simply stopped. Nobody could point to a change that explained it, and the logs from that period were already gone.

It is very tempting to join those together. Same code, same kind of endpoint, so surely one root cause and one fix. A colleague said, plainly, that the error signature being the same didn't mean the cause was, and that the test environment and production should be kept apart until evidence tied them together. I agree. A server that is overloaded can time out. A path that is blocked can time out. A dependency that is slow to answer can time out. A background process eating a machine's resources can make any of the above worse. If you treat these as one bug, a fix for one of them will look like a partial failure of the fix, and you will spend a week arguing with yourself about why it "only half worked".

Separate investigations are cheaper than a merged one that never converges. They also let you notice when two stories turn out to be one after all, which is a satisfying discovery in its own right and a trustworthy one, because it was earned rather than assumed.

Disappearing is not fixing

The window where the fault reproduced every time and then went away deserves its own warning. A problem that vanishes without a known change hasn't been solved. It has gone quiet, and quiet is not a state anyone should be comfortable with when real people use the system for important work. I'd rather admit that the earlier episode is unexplained than let it slide into the "fixed" column and quietly raise the confidence of every later theory. The honest position is that something changed, nobody knows what, and it could change back.

That has a practical consequence. When something is reproducible, even briefly, capture everything you can while it lasts: the exact request, the path it took, the timing, the state of the machines involved. A fault that happens on demand is the rarest and most valuable condition in debugging, and it is easy to spend it on making the problem go away rather than on understanding it. That is a natural trade under pressure, since the people waiting on you care about the symptom ending. It still leaves you with a fault that will return with no evidence to meet it.

What I'd do differently, and what I still don't know

Here is the habit I'm trying to build. Treat the error code as the start of a question, then split the population before you form a theory. Split by environment, by kind of client, by entry path, by time of day, by what the user was doing. Keep a separate list of what you actually know against what a convenient grouping has implied. Be suspicious of any scope that comes from complaints alone, and go and find the version of the record that doesn't depend on a person choosing to speak.

And be willing to say what is still open. I don't yet know whether the failures I'm looking at share one cause or several. I have a short list of suspects and a better idea of which evidence would tell them apart. That is a less satisfying thing to put in a status update than "root cause identified", but it is true, and the people relying on the system are better served by a truthful "not yet, here's how we'll find out" than by a confident story that unravels on Monday.

There is also a human side. People reporting a fault are trying to help, and the geography or the timing they give you is genuinely useful. It just belongs in the evidence pile, not at the top of it. Thank them, keep their detail, and then go and check it against something that doesn't care who was watching.