The standard recipe for getting an AI to write safe infrastructure code is to put a checker in the loop. The model writes some Terraform, a scanner grades it against a published security benchmark, the failures go back to the model, it tries again. Repeat until it passes or the budget runs out. Every published result on this approach says the same thing: compliance goes up with each round.

Two researchers at Colorado State went back through thousands of these repair runs and looked at something nobody had measured: not whether the failing checks got fixed, but whether the passing ones stayed passing. In about one run in seven, a security control that passed at one step was failing at the next. Tighten the definition to count only the cases with no measurement ambiguity whatsoever and it falls to 3.3%, which the authors adopt as their own headline rather than the larger number they were entitled to keep.

Either figure is invisible in the way these loops normally report themselves. The score they publish is the best result achieved so far, which by construction can only go up.

"What the data does support robustly is the qualitative claim: iterative IaC repair sometimes degrades a previously-passing security check, the effect is not negligible, and cumulative-best reporting hides it entirely."

The mechanism matters more than the rate, because it isn't really about Terraform. Nearly four in five of the breakages happened when the model, asked to fix one finding, rewrote whole blocks of code instead of editing the offending line. The amount of code changed turned out to be the strongest available warning sign that something had broken. The loop gives the model no reason to be careful, because it never mentions what was already working:

"Just as security went unexamined during Mozilla's bug fixes, current feedback loops never tell the model which checks its previous iteration was already passing."

Three things follow for anyone running agents against a scorecard. Any loop reported on a best-so-far basis is describing achievements, not state. That holds for test pass rates, coverage and lint scores as much as for Terraform, so ask a vendor for the per-attempt trajectory rather than the headline. Diff size is a free control: if an agent rewrote a file to clear one finding, that's the change that needs a human, whatever the scanner now says. And iterations aren't free. Improvement here flattened by the third attempt, after which the loop was mostly buying churn.

"Tools should not blindly iterate to convergence."

The finding will get over-read, though. This is two models from late 2024, AWS Terraform only, one run per setting, with a static scan standing in for real security. The 3.3% isn't a rate to quote for anyone else's stack. What travels is the shape: an agent that rewrites rather than edits will break things nobody was watching. About a third of those breakages repaired themselves a step later, which is a good reason not to halt on the first one, and a poor reason to relax about the rest.

Sources

1. Agyekum, B., Santos, F., "Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair" — https://arxiv.org/abs/2608.13404