A Permission Change Can Switch Off Your Evidence

Be most careful when you tighten the permission that carries your own diagnostics. If a workload writes its logs through an identity, and you narrow that identity too far, the failure arrives with the lights off: the service may be perfectly healthy, the logs simply stop, and the thing you would normally read to find out what went wrong is the thing you just broke. Least privilege is the right instinct. It deserves a verification plan that does not depend on the signal it might remove.

I reviewed a change this week that made the point cleanly. A container platform runs workloads under an execution identity, the one used to pull images and ship logs before the application's own code has a say. That identity had been given the platform's stock managed policy, which is generous in the way vendor defaults usually are. It could write to every log group in the account and pull from every image repository. Nobody had decided that. It was simply what the quick-start attached, and it had been quietly true for as long as the service had existed.

The change replaced it with a small inline policy: pull from the one repository the images actually live in, and write to the service's own log group, nothing else. One action, the one that fetches a registry token, cannot be scoped to a resource at all, so it stays broad by necessity. The author said so plainly instead of pretending the policy was cleaner than it was, which I value more than a tidy diff. It was a good change. I still wanted to slow down.

The evidence runs through the thing you are changing

Most permission tightening has an obvious canary. Remove a database role's ability to delete rows and a failing delete screams. Here the canary is the log stream. If the new policy has the wrong resource path, or a typo in one action, the symptom is not an error in your dashboard. It is an absence. Logs that used to appear do not, and an absence is the hardest kind of failure to notice, because nothing is there to be noticed.

I have written before about how a quiet system is not the same as a healthy one, and this is the same trap from a different side. An empty log group looks identical to a calm service. If you are watching for trouble in the logs, you will see nothing, conclude nothing is wrong, and carry on, while an error that would normally have been printed is being thrown away at the door.

What a plan can and cannot tell you

The reasonable pre-merge checks all passed. The configuration validated, the format was clean, the new input guards had tests that failed when the guard was removed. I asked for the stronger claim, that image pulls and log delivery still worked, and the honest answer was that no merge request pipeline could produce it. The role that applies infrastructure trusts only the default branch, so a plan on a feature branch stops before it can even assume it. A plan would not have proved delivery either way. It shows what will change, not whether the platform accepts the new shape at runtime.

This is where review conversations often go slightly wrong. A reviewer lists acceptance criteria that cannot be verified from the diff, and an author replies that they cannot be verified from a diff, and both are right and nothing has moved. What moved things forward was the author agreeing that the evidence has to come after the apply, and naming it up front: the most recent event time in each log group the task definitions write to, and the credential report confirming that nothing is using a long-lived key to write logs behind the new policy's back. Both can be checked in minutes. Neither can be checked before merge. Writing them into the change turns a vague worry into a short list someone can tick off.

Swaps have a gap

There is a second, smaller lesson in the mechanics. The identity's address does not change, so no running service is replaced, which is lovely. But the apply detaches the old policy and creates the new one in parallel, and permission changes take a few seconds to propagate. For that window, a batch of log writes can be refused. Seconds of dropped logs is a cheap price for most services. It is worth knowing about, though, because it is exactly the sort of gap that shows up as a puzzling hole in a timeline three weeks later when someone is chasing an unrelated incident.

You can avoid it by creating the narrow grant first and removing the broad one second, but that costs a two-step rollout, and for a low-risk internal service I would not insist on it. For something where those first seconds of logging matter, I would. The decision is about how much you value the evidence from that window, which is a commercial and operational judgement, not a security one.

What I take from it

Tightening access is good for the people who trust you with their data, and I would rather defend a narrow policy than explain a broad one. Defaults that nobody chose are a liability that grows quietly, and replacing one is worth doing. But the quality of the change is not just the policy. It is whether you can tell, afterwards, that the system still does what it did.

So before applying a narrower grant, I now ask a small question. If I get this wrong, how will I find out? If the answer runs through the permission I am editing, the plan is incomplete. Pick a signal that lives somewhere else, name it in the change, and look at it after the apply. That is a few minutes of work, and it is the difference between a hardening that is believed and one that is verified.