Friday, July 19th 2024
![Microsoft](https://tpucdn.com/images/news/microsoft-v1723816949774.png)
Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World
A faulty software update to enterprise computers by cybersecurity firm CrowdStrike has taken millions of computers offline, most of which are in a commercial or enterprise environment, or are Azure deployments. CrowdStrike provides periodic software and security updates to commercial PCs, enterprise PCs, and cloud instances, with a high degree of automation. The latest update reportedly breaks the Windows bootloader, causing bluescreens of death (BSODs), and if configured, invokes Windows Recovery. Enterprises tend to bulletproof the bootloaders of their client machines, and disable generic Windows Recovery tools from Microsoft, which means businesses around the world are left with large numbers of machines that will each take manual fixing. The so-called "Windows CrowdStrike BSOD deluge" has hit critical businesses such as banks, airlines, supermarket chains, and TV broadcasters. Meanwhile, sysadmins on Reddit are wishing each other a happy weekend.
Source:
The Verge
234 Comments on Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World
What I get from the video is that CrowdStrike created a device driver that can dynamically load updated modules from a specified directory. This effectively creates an engine that runs untrusted and unapproved code in kernel land. If that doesn't scare the shit out of you, I really don't know what will.
What makes it even more scary is that CrowdStrike did not include any input validation into their code thus why this whole fiasco happened. They failed to check for the most basic of issues, a file full of null data. OOPS!
Honestly, I'd be very damned surprised that CrowdStrike survives this whole mess.
then imagine NOT testing those updates before they went live
Then imagine why your company no longer exists
(this post may contain traces of sarcasm)
That doesn't fly at kernel level, especially when something is installed as a BOOT level driver. Those could be anything for interfacing with hardware like CPUs, GPUs and even some accelerators like PCI-E/SAS storage. There tends to be a lot of these but I usually reconfigure them to behave a bit differently on my systems before and after "first" boot.
Usually you'll see something flagged differently in ErrorControl than these two examples. Something like Critical - Log error & fail boot. In that situation when it fails, you'd get CrowdStrike'd hard.
......
I was here on day 0 and this thread was lit up with 4 pages by the time I got in. One whole ass page per hour and this isn't even a security forum.
You can bet every single one of those sites went wildly spinning themselves into orbit over this one.
This had so much reach that even the solar observer YouTuber guys had to chime in about it:
"If it was the sun, trust me, I would tell you."
Identify faulty driver(s) located in the one suspicious subdir where all boot critical drivers are located on the system, delete and reboot.
Simple as.
I didn't want to hammer that message home because one, I'm not a CrowdStrike customer and like most people here have identified nobody that is a customer for this LITERAL WHO. This isn't a security risk to anyone here and judging from half the threads it looks like I'm one of maybe three people reading fully equipped to deal with such a crisis in the first place. That part on its own is WILD. If deleting the null drivers wasn't enough, you'd have to go thumbing for some CrowdStrike service and hard delete that too.
Do you....You guys like fishing around in remote mounted hives for boot level drivers under CurrentControlSet and taking those risks?
Again, I'm equipped to deal with it and even I don't like doing it. This is exactly how we end up with the kinds of trust issues that lead to developing these emergency skills in the first place.
Anyway you're not going to like this but CrowdStrike will survive this flub and that's pretty much the basis for why the software even exists. What does that mean? I'll get to it. They have enough customers, obviously. Would I want CrowdStrike software running on any of my systems? Maybe if I had some highly targeted (lol no) mission critical (double LOL) VM or baremetal that's susceptible to Day 0 AI driven attacks or some absolutely insane pre-historic malware like Blaster that gets into every networked Windows box faster than a ninja sex party. Unlike those customers with ~8 million bricked machines, I don't subscribe to the kind of philosophy that permits these types of problems to reconstitute. I avoid updates on personal snowflake servers. I don't even like rebooting the server.
The software exists on the idea of rapid response to emerging threats, which is kind of along the lines of antivirus.
The problem started with one squirrely update that didn't ship correctly and people quickly applied it because they trust the vendor like that.
The fix was shipped out just over half an hour later but 8 million boxes rebooted before they could receive it.
Those 8 million boxes went offline and didn't need the protection anymore, which is a fail for production but NOT a fail for security.
It inconvenienced a bunch of IT pros and devops with a surprise recovery key audit to perform a fix because a lot of those systems had BitLocker/encryption and other complications involved.
So what I want to know is how many of those CrowdStrike systems that didn't go down, are still out in the wild and how often do they reboot after updates?
That might be something to check out. Honestly this right here should be the majority response. It won't happen because those subscribers have a completely separate philosophy and an entire other universe of problems to go with it. It might shake a few of them out of it though. Enough of these guys need to start asking some deep questions like "is this worth it?"
That sounds like a good policy. Great. So, if this guy has that kind of staged updates how did his company get hit by this whole damn mess? Oh yeah... CrowdStrike delivered the faulty update as an update that would be pushed out regardless of what update stage you have a particular system in. It didn't matter if you had a system in the stage three update ring, it too got the update. YIKES!!!
Hmmm...
I strongly suspect the EU would have been perfectly satisfied with an equivalent to eBPF.
1) A list of clients that use CS
2) The methods that the software uses
3) A way to push an infected content file that gets run thru the kernel driver
Someone allready is selling a COMPLETE list of CS users as well.
I thought that really was a no go unless the systems didn't work in the first place.
www.tomshardware.com/software/linux/crowdstrike-issues-go-beyond-windows-companys-security-software-has-reportedly-been-causing-kernel-panics-since-at-least-april
Also, fun fact:
Part of his story might as well have been in Chinese as I had no idea what he was talking about :eek:, but for some of us here it will make sense.