Friday, July 19th 2024

Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

by
btarunr
 Discuss (148 Comments)
A faulty software update to enterprise computers by cybersecurity firm CrowdStrike has taken millions of computers offline, most of which are in a commercial or enterprise environment, or are Azure deployments. CrowdStrike provides periodic software and security updates to commercial PCs, enterprise PCs, and cloud instances, with a high degree of automation. The latest update reportedly breaks the Windows bootloader, causing bluescreens of death (BSODs), and if configured, invokes Windows Recovery. Enterprises tend to bulletproof the bootloaders of their client machines, and disable generic Windows Recovery tools from Microsoft, which means businesses around the world are left with large numbers of machines that will each take manual fixing. The so-called "Windows CrowdStrike BSOD deluge" has hit critical businesses such as banks, airlines, supermarket chains, and TV broadcasters. Meanwhile, sysadmins on Reddit are wishing each other a happy weekend.
Source: The Verge

Related News

Add your own comment

148 Comments on Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

#76
mab1376
Onasi@mab1376
This is fair, though I fear that would require them to essentially overhaul the entire kernel at this point, Vista/NT6-style, which isn’t in the cards anytime soon, I assume.
Doubtfully anytime soon, but Apple made this switch in 2020 with the release of Catalina, so it's not entirely unfathomable.
Posted on Reply
#77
Wirko
mab1376TBF if Microsoft offered user-mode APIs into kernel events, it wouldn't be necessary to install a kernel driver.
Is that implemented in any OS? I don't see how it could be done with sufficient functionality but without compromising security.
Posted on Reply
#78
mab1376
WirkoIs that implemented in any OS? I don't see how it could be done with sufficient functionality but without compromising security.
It's implemented in OSX as of Catalina and can be done in Linux by enabling additional flags when compiling the kernel.
Posted on Reply
#80
Onasi
mab1376Doubtfully anytime soon, but Apple made this switch in 2020 with the release of Catalina, so it's not entirely unfathomable.
They were driven to it by their switch to their own silicon, to be fair. Wasn’t just out of the goodness of their hearts.
WirkoIs that implemented in any OS? I don't see how it could be done with sufficient functionality but without compromising security.
Kinda. You can compile Linux to allow such functionality and it was the reasoning for “system extensions” in OSX Catalina and onwards.
Oh, I was sniped on this, lol.
Posted on Reply
#81
micropage7
faulty win update need a new win update over the previous win update
Posted on Reply
#82
mab1376
OnasiThey were driven to it by their switch to their own silicon, to be fair. Wasn’t just out of the goodness of their hearts.
it is preferable regardless due to this exact issue, it lowers the risk of 3rd party suppliers.
Posted on Reply
#83
forman313
WirkoAnother problem is PCs that won't boot.

Although ... isn't there a thing called Intel Management Engine, which system admins can use to access disks and everything on a PC even if it's turned off or unable to boot?
Yes there is.

For AMD there is DASH and AMC (AMD Management Console) supported by the PRO processors. You get remote access with KVM, USB ISO boot redirection, power control +++. As long as the device is connected to AC/DC or in modern standby and has a connection to the internet, you get access. Even when its powered of.

In this case, you could just use the boot redirect and fire up your favorite rescue ISO, like UBCD, Hirens etc. Just need to make sure you have the tools necessary for open/disable Bitfokker.


I used to be a sysadmin, and I have never missed doing it. But today I miss it even less.
Posted on Reply
#84
DaemonForce
JcRabbitJudging by the chaos out there, this is what the Y2K bug could have been, but wasn't (because we made sure on time that it would not turn into anything like this - and thus it became a non event).
This bug was never actually fixed. Developers each have their own flavor of it too. The only thing that really happened was a shifting of goalposts.
Posted on Reply
#85
bug
mab1376TBF if Microsoft offered user-mode APIs into kernel events, it wouldn't be necessary to install a kernel driver.
Calls from user space into the kernel space incur some overhead. It's hard to enable that without taxing the performance.
Posted on Reply
#86
Neo_Morpheus
forman313Yes there is.

For AMD there is DASH and AMC (AMD Management Console) supported by the PRO processors. You get remote access with KVM, USB ISO boot redirection, power control +++. As long as the device is connected to AC/DC or in modern standby and has a connection to the internet, you get access. Even when its powered of.
The funny thing is many places forbid using this.

One such example was a multibillion media company I worked which refused to allow the usage due to be “too chatty”

That was InfoSec and network team official response by the way.
Posted on Reply
#87
mab1376
bugCalls from user space into the kernel space incur some overhead. It's hard to enable that without taxing the performance.
Good point, but it would most likely be nominal except on extremely busy servers.
Posted on Reply
#88
ErikG
Shou MikoI already heard this didn't work for everyone including the registery fix.

I checked serveral Windows 10 installations at work incl. the one I use at work and I haven't found anything and a lot of my customers are running Windows 11 so hope they are more safe than Windows 10 users.


Correct.
Working for me, 50+ machines unlocked.
Posted on Reply
#89
Aquinus
Resident Wat-man
AssimilatorOh good, the Linux idiots have arrived to shit on things they have zero understanding of.
I don't know, man. What I do know is that I have an engineer that currently can't do his job because his laptop is bricked because of this. All of my engineers with Macs (including myself,) are fine. So while I don't understand exactly what CrowdStrike did, I do understand its side-effects. Same thing with our servers. Our product in particular is mostly on a flavor of Linux and is not impacted by any of this. The parts of the business with Windows servers on the other hand are actively war rooming to fix all of this.

So say what you will, but this is a huge issue for businesses that use CrowdStrike with Windows machines.
Posted on Reply
#90
mab1376
AquinusI don't know, man. What I do know is that I have an engineer that currently can't do his job because his laptop is bricked because of this. All of my engineers with Macs (including myself,) are fine. So while I don't understand exactly what CrowdStrike did, I do understand its side-effects. Same thing with our servers. Our product in particular is mostly on a flavor of Linux and is not impacted by any of this. The parts of the business with Windows servers on the other hand are actively war rooming to fix all of this.

So say what you will, but this is a huge issue for businesses that use CrowdStrike with Windows machines.
Mac uses an API to collect kernel events, so the kernel driver required on Windows doesn't exist on Mac. Linux has user mode and kernel mode sensors available depending on the kernel, but I don't think kernel mode Linux hosts were affected as I suspect the bug was only introduced into the Windows code base at CrowdStrike.

As mentioned above, user mode APIs for kernel events compared to a kernel driver does have a performance impact.
Posted on Reply
#91
Makaveli
Thank god we don't use cloudstrike in our environment. I woke up to my teams chat going banana's and had to tell everyone to calm down we don't use this product.
Posted on Reply
#92
Assimilator
AquinusI don't know, man. What I do know is that I have an engineer that currently can't do his job because his laptop is bricked because of this. All of my engineers with Macs (including myself,) are fine. So while I don't understand exactly what CrowdStrike did, I do understand its side-effects. Same thing with our servers. Our product in particular is mostly on a flavor of Linux and is not impacted by any of this. The parts of the business with Windows servers on the other hand are actively war rooming to fix all of this.

So say what you will, but this is a huge issue for businesses that use CrowdStrike with Windows machines.
And, again, for the 100th time, none of that is the fault of Microsoft or Windows. Crowdstrike shipped a broken update and Crowdstrike bricked those machines.

Yes, you could argue it's ultimately Microsoft's fault for not building a sufficiently isolated kernel, but that's very much ignoring the forest for the trees in this case.
Posted on Reply
#93
Aquinus
Resident Wat-man
AssimilatorAnd, again, for the 100th time, none of that is the fault of Microsoft or Windows. Crowdstrike shipped a broken update and Crowdstrike bricked those machines.

Yes, you could argue it's ultimately Microsoft's fault for not building a sufficiently isolated kernel, but that's very much ignoring the forest for the trees in this case.
Sure, but the lay person doesn't care about the nuances of how CrowdStrike uses Windows' APIs. That's what I'm trying to get at. I know that it's CrowdStrike's fault, but regardless, the outcome is the same. My engineer still can't do his job and there are servers needing to be recovered, but my Mac users are just fine.
Posted on Reply
#94
P4-630
AssimilatorAnd, again, for the 100th time, none of that is the fault of Microsoft or Windows. Crowdstrike shipped a broken update and Crowdstrike bricked those machines.
And again
P4-630due to CrowdStrike Falcon-package failure affecting Windows.
Posted on Reply
#95
bug
mab1376Mac uses an API to collect kernel events, so the kernel driver required on Windows doesn't exist on Mac. Linux has user mode and kernel mode sensors available depending on the kernel, but I don't think kernel mode Linux hosts were affected as I suspect the bug was only introduced into the Windows code base at CrowdStrike.

As mentioned above, user mode APIs for kernel events compared to a kernel driver does have a performance impact.
The elephant in the room being: if it affects so many systems, how the hell did it go undetected all the way to prod? Though it could be a case of "we tested one thing and released another".
Posted on Reply
#96
Makaveli
mab1376Exactly the boat I'm in... I'm the infosec manager so I'm just the one documenting the wreckage.
Seriously I fought back against Bitlocker encryption on all machines for this reason at my last place because I told them recovery after an event like this is a major pain in the ass. I believe I saw a mcafee update in the past brick one of my workstation. but for me I thought ahead had and images and other thing done to recover my own machine. Now of course encryption is important so they did it anyways but none of my other co-workers took the extra steps I did encase of a disaster.
Posted on Reply
#97
P4-630
The outage was caused by a major outage 'after a software update' at the cybersecurity company CrowdStrike.
The company now says it has found a solution to the problem.
“The problem has been identified, isolated and a solution implemented,” Crowdstrike CEO said.
Posted on Reply
#98
mab1376
bugThe elephant in the room being: if it affects so many systems, how the hell did it go undetected all the way prod? Though it could be a case of "we tested one thing and released another".
that question is exactly why their stock is tanking.
MakaveliSeriously I fought back against Bitlocker encryption on all machines for this reason at my last place because I told them recovery after an event like this is a major pain in the ass. I believe I saw a mcafee update in the past brick one of my workstation. but for me I thought ahead had and images and other thing done to recover my own machine. Now of course encryption is important so they did it anyways but none of my other co-workers took the extra steps I did encase of a disaster.
BitLocker is required for our ISO27001 certification if a machine has sensitive info on it, which most do in my environment.
Posted on Reply
#100
Makaveli
mab1376that question is exactly why their stock is tanking.


BitLocker is required for our ISO27001 certification if a machine has sensitive info on it, which most do in my environment.
This I know as we had alot of machines with sensitive info so it was happening regardless of what I said.

Most companies these days bitlocker is on by default.
Posted on Reply
Add your own comment
Jul 19th, 2024 13:16 EDT change timezone

Latest GPU Drivers

New Forum Posts

Popular Reviews

Controversial News Posts