Friday, July 19th 2024
Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World
A faulty software update to enterprise computers by cybersecurity firm CrowdStrike has taken millions of computers offline, most of which are in a commercial or enterprise environment, or are Azure deployments. CrowdStrike provides periodic software and security updates to commercial PCs, enterprise PCs, and cloud instances, with a high degree of automation. The latest update reportedly breaks the Windows bootloader, causing bluescreens of death (BSODs), and if configured, invokes Windows Recovery. Enterprises tend to bulletproof the bootloaders of their client machines, and disable generic Windows Recovery tools from Microsoft, which means businesses around the world are left with large numbers of machines that will each take manual fixing. The so-called "Windows CrowdStrike BSOD deluge" has hit critical businesses such as banks, airlines, supermarket chains, and TV broadcasters. Meanwhile, sysadmins on Reddit are wishing each other a happy weekend.
Source:
The Verge
234 Comments on Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World
You'd have to boot into single user mode, too.
The real difference of cause is that Linux and FreeBSD users generally only run kernel modules that came with the kernel, not some closed-source third-party garbage. Except for the NVidia drivers. Errr... FreeBSD has dtrace, Linux has eBPF. But we can't know whether they would be sufficient for Crowdstrike. They have a Linux version, I bet they use a kernel module, too.
All tech fails at some point
A security guard was placing two shopping carts "fixed together with something" blocking the access to these checkouts, which is why i asked if it was related to this global issue, to which he said yes.
what?
Had great fun helping end-users try to get their machines back online, and then explain why they then couldn't access any company services.
It mostly came back online quite quickly, but our AD was still having problems yesterday evening, causing problems for user authentification, which is used across most of our sites and services...so the sites and services were up, but people couldn't log into them.
Our Bitlocker key server wasn't available for most of yesterday morning, but came back up pretty quickly thankfully.
We are expecting a few things to still be down on monday, as there aren't very many people available to go to the still down critical machines manually.
Just want to put my 2c that this shouldn't have happened.
Any deployment should be tested nefore release, and even when released, to one or two "test" customers who get high support and low prices for their help with testing, and potential risks.
Hats off to all those sysadmins that have to spend their whole weekend, and more, getting these systems back up manually.
The reason why this happened is because the CrowdStrike agent is a boot level driver. This means that it gets loaded pretty much before most of anything else, except when you boot in Safe Mode. Then, only absolutely necessary drivers are loaded. You also need Safe Mode to be able to delete the offending file, since in a regular session (when the PC wouldn't crash) the file would be in use and thus locked.
I must admit, when I read about the fix I couldn't believe my eyes. A file with the .sys extension is usually a driver. This means actual executable code. Usually anti-malware and HIPS applications work with some form of pattern file. CrowdStrike really does distribute its "signature" updates as executable code. And therein lies the problem. I don't know how many of you know about coding and pointers in particular, but here goes: CrowdStrike tried to call some code in that update (C-00000291*.sys). The problem was, the file CrowdStrike had pushed contained zeros. Now, when you try to call or dereference a pointer of 0 (nullptr), that just won't fly. Usually, to get around potential nullptrs you make a check for it before trying to use the pointer. You can also use try/catch statements. Apparently, someone at CrowdStrike didn't think this was necessary. And... BOOOM!
At the company I work, we also got hit pretty hard by this issue. While our company is actually on the smaller size, the corporation that owns us uses CrowdStrike. A lot of us are tech-savvy, being developers. Still we weren't able to help ourselves because these days you're not allowed to have admin permissions on your workstation. Our consultants are issued laptops, which, because they're used both on- and off-site, are BitLocker-encrypted. That's not necessarily a problem, because each consultant has their key. What they don't have is the recovery key, which for some reason is needed when you actually manage to get into Repair Mode. We had to have our sys admin take a break from his vacation to help get us up and running again. Many systems are still down, because there only was time to bring the most important ones back on-line.
And yes, this could just as easily have hit *nix and macOS. But the majority of businesses out there use Windows. Like it or not.