Silent Failures
The most expensive bug is not always the one that crashes. Crashes are rude, but at least they are honest: they interrupt you, they make noise, and they force somebody to look at whatever broke. The nastier bug is the one that fails and still reports success. Those are the ones I have lost the most time to, and almost none of them looked like failures in the moment they happened. Instead they looked like green builds, completed jobs, and finished reports, right up until I went looking for something and found it had quietly not been there for weeks.
A quick note before I get into it: I have spent the last few weeks writing about the big-picture stuff, AI governance, the ethics of pulling the ladder up behind us, the philosophical weather of all this, and I got a bit tired of it. So this week I wanted to come back down to the workshop floor and write about something concrete I actually deal with, hands on the tools. Silent failures are exactly that kind of problem.
This piece is mostly about that silent half, but it is worth saying up front that the silent failure has a louder twin, and they are two sides of one coin. At one extreme a system tells you nothing when something breaks. At the other it tells you everything, all the time, until the noise is so constant that you tune the whole channel out. They look like opposites, but they fail you the same way: you end up with a signal you do not act on. One you cannot see; the other you have trained yourself to ignore. The cure for both turns out to be the same discipline, and I will come back to it.