You Don't Know Your Limit Until You've Broken It On Purpose

If you have never pushed your system until it fails, you do not know where its limit is. You have a belief about it, and beliefs about capacity are usually wrong in ways you would not have guessed. Run a deliberate load test against a non-production copy, find the point where it starts to fail, and treat that number as the beginning of the conversation rather than the end of it.

The reason I'm saying this now is a recent investigation that started the way these things usually do: with a complaint that was vague, intermittent and geographically awkward. Logins were occasionally slow, and sometimes they ended in a gateway timeout. It was happening for people in one place and nobody else. The most recent change to the system touched nothing near authentication, so it was a poor suspect. That left the team doing what teams do when a problem will not show itself, which is to form theories. The favourite was the network, partly because the network is always a plausible villain and partly because nobody could disprove it.

A theory you cannot test is a comfortable thing to hold. It lets everyone keep working while feeling busy. What broke the stalemate was an admission that nobody had ever stress tested the web tier. Not once. The system had been running happily for a long time, and it had simply never been asked how many people it could serve at once before it started dropping them.

Reproducing the symptom is most of the work

So we asked. A load tool, simulating concurrent logins against a staging server, did in an afternoon what days of speculation had not. At a thousand simulated users, well over a third of requests failed. That was unsurprising and not very informative, because a thousand users is a number you pick to prove a point. The useful part was stepping down. A hundred users still failed at a similar rate. Seventy-five did a bit better. Sixty did about the same. Fifty was clean on the first run.

That ladder is a bisection of capacity, and I've come to prefer it to any single big test. You are looking for the knee in the curve, the point where behaviour changes character. Once you have even a rough idea of where it sits, "intermittent timeouts for some people" becomes "this misbehaves somewhere around a few dozen concurrent sessions". That is something an engineer can reason about, argue with and attack.

It also moved the conversation off the network theory without anyone having to win an argument. The failure reproduced on demand, inside a single environment, with no long-distance link anywhere in the picture.

The resource you're watching is rarely the resource that's limiting you

The detail I found most instructive was what the machine's own gauges said while it was failing. At one point the error rate went up while CPU went down. Timeouts showed up when the processor was relaxing, not when it was straining. That is the signature of something waiting rather than working, which points at a limit on connections or concurrency rather than raw compute.

This is why the instinct to "just add more CPU" so often disappoints. Capacity is not one number. It is the smallest of several, and the smallest is frequently one nobody put on a dashboard because nobody thought it would matter. You only discover which one it is by loading the system until something gives, then looking at what was actually exhausted.

Treat the numbers as direction, not truth

I'd be misleading you if I made this sound cleaner than it was. Staging is not production, so the exact thresholds do not transfer. The production counters showed real traffic comfortably below the level that had broken staging, which means either the environments differ in some important way or something else is also in play. And the test itself can lie. One rerun at the "clean" level came back with a fifth of requests failing, and the cause turned out to be a leftover loop setting in the test plan, not the server. Had we believed that run, we would have drawn the wrong line.

So the honest conclusion at the end was modest: heavy concurrent load reproduces the problem, and the next question is what counts as heavy and where the real ceiling sits in production. That is unresolved, and I think it's better to say so than to announce a root cause because it makes a tidier story. A measured "we don't know yet" is more useful to the next person than a confident answer that has to be walked back.

Break things politely

Two practical notes, because deliberately failing a system has side effects. A test that hammers logins from a single address looks, to anyone watching the network, remarkably like an attack, and it exposed something we had not been thinking about: nothing throttled or blocked thousands of login attempts arriving from one place. That is a finding in its own right, and a slightly sobering one for a product that holds sensitive data. So tell the people who watch the perimeter before you start, and do it somewhere you can afford to break.

The other note is about people. The most honest remark in the room was, roughly, that we had found it, just not the way we would have chosen. There was relief at having a direction and a little unease that this had never been done. I think that's the right tone. A system that has run well for years has not been proven safe; it has been lucky or lightly used. Saying so without blame is part of how a team gets better at asking the question earlier.

If you take one thing from this, make it cheap to ask. Pick a quiet afternoon, point a load tool at a copy of the system, and find out where it falls over before a customer does it for you.