I never said what I was counting
Hello,
Last issue I tested 27 machine identity cases. The lab has grown since: it now has 67 cases, and 10 of my guesses were wrong. This issue is about the last batch, 19 cases I had designed and never run.
Same method as before. I wrote the guesses, hashed the file and committed it before any code existed. Everything is made-up data, and I wrote both the guesses and the code, so a match proves little. The misses are the useful part.
Where I was wrong
Five of the 19 cases had a wrong guess. Two were the same mistake: I never said what I was counting.
I guessed 4 alerts in one case. The model alerts on each of the 5 creation attempts, refused or not. Alerting only on the ones that succeed would give 1.
I guessed a connector would deliver 0 events without controls. It delivered 14 of 20 and dropped 6 whether or not a heartbeat existed. The heartbeat adds the alarm and the replay, not the delivery.
I guessed a drill would find 2 faults. It found 1. A routing check catches one fault before the drill runs, and a second holder makes the other harmless.
Controls that changed nothing
In 10 of the 19 cases, at least one control could be switched off alone without changing anything the case counts. Sometimes another control already did the job: the cloud policy refused the 3 console accounts, so my weekly sweep had nothing left to find. Sometimes the control is a log, a review or an approval, and a model that counts access outcomes cannot see those. They still matter for evidence. My test just cannot measure them.
Every row has a status
The catalogue has 193 rows, and each now has a final status. Of the 147 technical rows, 60 were run in this lab, 35 were tested in an earlier lab, 6 were partly tested and 46 were not run. Each of those 46 has a reason, and 31 of them need a real product or environment. I would rather show a gap than let it look like coverage.
Read more
The write-up: roshantrivedi.co.in/writing/non-human-identity-lab-round-19/
Round 19 results: roshantrivedi.co.in/nhi-lab/round-19/
All 193 rows: roshantrivedi.co.in/nhi-lab/register/
One question for you
When you write down what you expect from a control, do you also write what is being counted?
Reply and tell me. If you can think of a case I should add, I will add it and keep the original on the page.
Roshan