The Weekly Cybers #28
Some thoughts on the CrowdStrike outage, another busy week for the eSafety Commissioner, a few AI stories (of course), and more.
Welcome
Timing is everything. Just as I was finishing off last week’s newsletter, writing that it was “a relatively quiet week”, the whole CrowdStrike thing happened. Now, a week later, what’s left to say?
I’ve decided to simply link to some of the better explanations, and throw in a handful of personal observations.
Apart from that, the eSafety Commissioner has been in the news again, there’s a bunch of AI stories, and more.
So about that whole CrowdStrike thing...
OK, you already know the main narrative of what’s been billed as the largest IT outage in history.
At 04:09 UTC last Friday, CrowdStrike, a cybersecurity firm that in their words provides real-time indicators of attack, hyper-accurate detection, and automated protection, pushed out a dodgy threat detection update, which crashed Windows computers.
Some 8.5 million computers were affected. That sounds a lot, but there’s a couple billion computers running Windows in its various forms.
Problem one, those computers belonged to big organisations which believe they need something like CrowdStrike’s hardcore real-time defences: airlines, airports, banks, media, emergency services, healthcare, governments. We notice when they go down.
Problem two, the bad update needed to be removed from each affected computer before it could be rebooted. Unless an organisation’s network is set up to allow this — something that’s both complicated and expensive — that usually means physically going to every computer, rebooting it in safe mode, and doing the thing. That takes time.
(As an aside, five stars to senior systems engineer Rob Woltz who figured out a way to speed things up using a barcode scanner.)
Despite the whining of PR experts, I reckon CrowdStrike did a pretty good job. Dear PR experts, it’s not up to CrowdStrike to tell Wayne from Castle Hill when Liquorland’s checkouts will be working again. That’s Liquorland’s problem.
If you look at CrowdStrike’s own incident reporting — and I’ve confirmed this — they identified the problem, killed the bad update, and issued how-to-fix instructions in under 90 minutes.
After that it’s up to CrowdStrike’s customers to have their own processes for accessing, fixing, and rebooting every computer. Obviously that was easier for some and much, much harder for others. Lessons will be learned.
Now you can argue that the bad update should never have gone out. Sure. But “should” is one of those tricky words. I should do more exercise. Politicians should not be corrupt. Hand me that hammer, this should do the trick.
CrowdStrike has promised to do more testing of updates before sending them out. The executive summary (PDF) of their most recent Preliminary Post Incident Review (PIR) says they will now implement a “refined deployment strategy”:
Adopt a staggered deployment strategy, starting with a canary deployment to a small subset of systems before a further staged rollout. Enhance monitoring of sensor and system performance during the staggered content deployment to identify and mitigate issues promptly. Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed. Provide notifications of content updates and timing. [Link added.]
All of which means that until now they were just blasting out the updates to everyone all at once, at whatever random times. As I say, lessons will be learned.
But now the key question for every affected organisation is how they’ll improve their own processes so that they can recover from any similar problem in the future.
After all, it might not be a friend causing an accidental mass crash. It might be, you know, The Hackers™. And in that scenario, there won’t be a stressed but helpful customer support engineer advising you how to fix things.
Was it fair to call it a “Microsoft outage”? Is the whole industry broken?
I understand why many news headlines referred to it as a “Microsoft outage”. Ordinary folks have heard of Microsoft. They haven’t heard of CrowdStrike. Until now. Gotta connect with the audience.
But it was CrowdStrike’s fault, right? It was their bad update that crashed the computers? Yes. And yes.
But it’s not quite that clear-cut.
The software in question, the CrowdStrike Falcon Sensor which is inserted deep into the host computer’s operating system to watch for signs of malicious activity, has to run as a kernel driver.
Without going into the details, that means it has access to everything. It’s something you need to be able to trust. And yet when loaded with a bad update, this particular kernel driver crashed the whole system.
As tech PR and writer Ed Zitron wrote, Microsoft really screwed up here, and it reflects what he sees as a systemic problem in the industry.
What we're seeing today isn’t just a major fuckup, but the first of what will be many systematic [sic] failures — some small, some potentially larger — that are the natural byproduct of the growth-at-all-costs ecosystem where any attempt to save money by outsourcing major systems is one that simply must be taken to please the shareholder.
I’ll leave you to read it for yourself. Zitron makes a number of technical assertions which I’m not equipped to check. But it’s an interesting essay nonetheless.
Is the cost of the CrowdStrike outage overblown?
Cloud and supply chain insurer Parametrix reckons that the outage could cost Fortune 500 companies US$5.4 billion. But don’t get out your violins. As of 28 March the Fortune 500 represents US$18.8 trillion in revenues. That’s a revenue loss of 0.029%, or about $20 off an average Australian’s annual salary. Blink and you’d miss it.
Mind you, CrowdStrike’s share price dropped around 25% in two days of trading. That would’ve hurt.
At the time of writing, CrowdStrike says 97% of affected sensors are back online.
Disclosure: I have previously attended media briefings where CrowdStrike provided lunch.
eSafety Commissioner seeks “systemic” powers
It was another action-packed week for Australia’s eSafety Commission, Julie Inman Grant.
On Wednesday she issued more legal notices to big tech companies, threatening fines of $782,500 per day if they don’t report every six months on their progress in tackling child abuse material.
The targets were Apple, Google, Meta, and Microsoft, and the platforms Discord, Skype, Snap, and WhatsApp.
Inman Grant said Meta “admitted it did not always share information between its services when an account is banned for child abuse”, and that “eight different Google services, including YouTube, are not blocking links to websites that are known to contain child abuse material” — because the major players share lists of such sites, particularly INTERPOL’s International Child Sexual Exploitation database.
The commissioner also told a senate committee she’s after “technical regulation powers” to “de-platform” certain types of online apps in “systemic” ways — for example by tacking the app developers, the app stores, and other platforms on which they rely.
As iTnews wrote:
In her immediate sights are so-called “de-clothing” or “nudification” apps that she said were being used “to create synthetic child sexual abuse material”; as well as “tracking and monitoring apps” which were used as tools of coercive control over partners.
One of the biggest problems, she says, is failing to tackle “the mass creation of fake and imposter accounts”.
“They’re allowing these predators and criminal organisations to literally colonise their platforms and target people on the platforms,” she said.
Julie Inman Grant firmly in conservative media crosshairs
While partisan politics is generally outside the remit of this humble newsletter, it’s worth noting what appears to be a coordinated conservative call for the eSafety Commissioner to be sacked.
Sky News Australia described Inman Grant as “Australia’s controversial eSafety boss” after she described Donald Trump as a fake news “superspreader” who “savagely abuses foes online” in a recent speech. Their story quoted at length — and this may come as a shock — a spokesperson from the Institute of Public Affairs (IPA).
Far-right group blog Spectator Australia went one further with their headline, Peter Dutton must take a stand against the eSafety Commission ($), saying the speech revealed “what many believe to be anti-conservative political bias”.
Both outlets have previously criticised Inman Grant for her actions against Elon Musk’s X on freedom-of-speech grounds.
Also in the news
- Defence is hiring more than 500 new IT staff as it moves away from using so many contractors.
- Queensland’s Liberal-National Party posted a deepfake video on TikTok showing Labor premier Steven Miles dancing. It was labelled as AI, but it’s still triggered calls for tougher political advertising laws.
- At The Conversation, Griffith University’s Susan Grantham asks: When it comes to political advertising, is AI ever OK?.
- “The last remaining chance of anyone being prosecuted over the illegal robodebt fake debt creation scheme appears to have withered on the vine,” writes Julian Bajkowski at The Mandarin. Australian Federal Police dropped an investigation into allegations that a witness deliberately provided false evidence to the royal commission.
- The new Administrative Review Tribunal (ART) starts 14 October, replacing the Administrative Appeals Tribunal (AAT).
- The Australian Communications and Media Authority (ACMA) has formally warned ten telcos for breaching the new Financial Hardship Industry Standard designed to protect customers struggling to pay their internet and phone bills.
Data breaches and hacks of note
- The BlackSuit ransomware group claims to have stolen data from Reward Hospitality.
- Police have arrested six Australians for allegedly sending 318 million scam messages as part of what is known as a SIM box scam.
- South Australia’s Wattle Range Council has been hit with a LockBit ransomware attack. They’re on the Limestone Coast.
Elsewhere
- On Monday Meta released a new large language model (LLM) called Llama, an acronym for “Large Language Model Meta AI”, as free open source software. WIRED notes that this will stir up debate over the dangers of releasing AI without guardrails ($).
- Guardian Australia set up some blank user accounts on Facebook and Instagram’s. “Three months later, without any input, they were riddled with sexist and misogynistic content.”
- The Australian College of Nursing has developed a new Position Statement on Artificial Intelligence (PDF).
- The US Department of Homeland Security has a denial-of-service robot to raid smart homes. “NEO carries an onboard computer and antenna array that will allow officers the ability to create a ‘denial-of-service’ event to disable ‘Internet of Things’ devices that could potentially cause harm while entry is made.”
- “Australia’s telcos are giving away free phones and gift vouchers, in last-minute bid to shift customers off 3G,” reports ABC News.
- Attention tech policy nerds, friend of the pod ANU’s Tech Policy Design Centre has launched its inaugural in-person, catered, four-day course, Foundations of Tech Policy. One batch in August, one in September, 25 places each.
- The Australasian Society for Computers & Law is hosting a consultation roundtable on Australia’s Cyber Security Strategy under the Chatham House Rule on Friday 2 August. Warning: It’s being held via Teams.
Inquiries of note
- The Australian Competition and Consumer Commission (ACCC) will examine potential competition issues relating to generative AI as part of its five-year Digital Platform Services Inquiry, among other issues. They’ve issued an issues paper, and submissions close 23 August.
What’s next?
Parliament is currently on its long winter break until Monday 12 August.
Any questions or comments? Just reply to this email. Cheers.
The Weekly Cybers is a personal look at what the Australian government has been saying and doing in the digital and cyber realms, on various adjacent topics, and whatever else interests me, Stilgherrian, published every Friday afternoon (nearly).
If I’ve missed anything, or if there’s any specific items you’d like me to follow, please let me know.
If you find this newsletter useful, please consider throwing a tip into the tip jar.
This is not specifically a cyber *security* newsletter. For that that I recommend Risky Biz News and Cyber Daily, among others.