Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

[Onasi]

@mab1376
This is fair, though I fear that would require them to essentially overhaul the entire kernel at this point, Vista/NT6-style, which isn’t in the cards anytime soon, I assume.

 

[mab1376]

@mab1376
This is fair, though I fear that would require them to essentially overhaul the entire kernel at this point, Vista/NT6-style, which isn’t in the cards anytime soon, I assume.

Doubtfully anytime soon, but Apple made this switch in 2020 with the release of Catalina, so it's not entirely unfathomable.

 

[Wirko]

TBF if Microsoft offered user-mode APIs into kernel events, it wouldn't be necessary to install a kernel driver.

Is that implemented in any OS? I don't see how it could be done with sufficient functionality but without compromising security.

 

[mab1376]

Is that implemented in any OS? I don't see how it could be done with sufficient functionality but without compromising security.

It's implemented in OSX as of Catalina and can be done in Linux by enabling additional flags when compiling the kernel.

 

[Assimilator]

If you mean me

I do not.

 

[Onasi]

Doubtfully anytime soon, but Apple made this switch in 2020 with the release of Catalina, so it's not entirely unfathomable.

They were driven to it by their switch to their own silicon, to be fair. Wasn’t just out of the goodness of their hearts.

Is that implemented in any OS? I don't see how it could be done with sufficient functionality but without compromising security.

Kinda. You can compile Linux to allow such functionality and it was the reasoning for “system extensions” in OSX Catalina and onwards.
Oh, I was sniped on this, lol.

 

[micropage7]

faulty win update need a new win update over the previous win update

 

[mab1376]

They were driven to it by their switch to their own silicon, to be fair. Wasn’t just out of the goodness of their hearts.

it is preferable regardless due to this exact issue, it lowers the risk of 3rd party suppliers.

 

[forman313]

Another problem is PCs that won't boot.

Although ... isn't there a thing called Intel Management Engine, which system admins can use to access disks and everything on a PC even if it's turned off or unable to boot?

Yes there is.

For AMD there is DASH and AMC (AMD Management Console) supported by the PRO processors. You get remote access with KVM, USB ISO boot redirection, power control +++. As long as the device is connected to AC/DC or in modern standby and has a connection to the internet, you get access. Even when its powered of.

In this case, you could just use the boot redirect and fire up your favorite rescue ISO, like UBCD, Hirens etc. Just need to make sure you have the tools necessary for open/disable Bitfokker.


I used to be a sysadmin, and I have never missed doing it. But today I miss it even less.

 

[DaemonForce]

Judging by the chaos out there, this is what the Y2K bug could have been, but wasn't (because we made sure on time that it would not turn into anything like this - and thus it became a non event).

This bug was never actually fixed. Developers each have their own flavor of it too. The only thing that really happened was a shifting of goalposts.

 

[bug]

TBF if Microsoft offered user-mode APIs into kernel events, it wouldn't be necessary to install a kernel driver.

Calls from user space into the kernel space incur some overhead. It's hard to enable that without taxing the performance.

 

[Neo_Morpheus]

Yes there is.

For AMD there is DASH and AMC (AMD Management Console) supported by the PRO processors. You get remote access with KVM, USB ISO boot redirection, power control +++. As long as the device is connected to AC/DC or in modern standby and has a connection to the internet, you get access. Even when its powered of.

The funny thing is many places forbid using this.

One such example was a multibillion media company I worked which refused to allow the usage due to be “too chatty”

That was InfoSec and network team official response by the way.

 

[mab1376]

Calls from user space into the kernel space incur some overhead. It's hard to enable that without taxing the performance.

Good point, but it would most likely be nominal except on extremely busy servers.

 

[ErikG]

I already heard this didn't work for everyone including the registery fix.

I checked serveral Windows 10 installations at work incl. the one I use at work and I haven't found anything and a lot of my customers are running Windows 11 so hope they are more safe than Windows 10 users.


Correct.

Working for me, 50+ machines unlocked.

 

[Aquinus]

Oh good, the Linux idiots have arrived to shit on things they have zero understanding of.

I don't know, man. What I do know is that I have an engineer that currently can't do his job because his laptop is bricked because of this. All of my engineers with Macs (including myself,) are fine. So while I don't understand exactly what CrowdStrike did, I do understand its side-effects. Same thing with our servers. Our product in particular is mostly on a flavor of Linux and is not impacted by any of this. The parts of the business with Windows servers on the other hand are actively war rooming to fix all of this.

So say what you will, but this is a huge issue for businesses that use CrowdStrike with Windows machines.

 

[mab1376]

I don't know, man. What I do know is that I have an engineer that currently can't do his job because his laptop is bricked because of this. All of my engineers with Macs (including myself,) are fine. So while I don't understand exactly what CrowdStrike did, I do understand its side-effects. Same thing with our servers. Our product in particular is mostly on a flavor of Linux and is not impacted by any of this. The parts of the business with Windows servers on the other hand are actively war rooming to fix all of this.

So say what you will, but this is a huge issue for businesses that use CrowdStrike with Windows machines.

Mac uses an API to collect kernel events, so the kernel driver required on Windows doesn't exist on Mac. Linux has user mode and kernel mode sensors available depending on the kernel, but I don't think kernel mode Linux hosts were affected as I suspect the bug was only introduced into the Windows code base at CrowdStrike.

As mentioned above, user mode APIs for kernel events compared to a kernel driver does have a performance impact.

 

[Makaveli]

Thank god we don't use cloudstrike in our environment. I woke up to my teams chat going banana's and had to tell everyone to calm down we don't use this product.

 

[Assimilator]

I don't know, man. What I do know is that I have an engineer that currently can't do his job because his laptop is bricked because of this. All of my engineers with Macs (including myself,) are fine. So while I don't understand exactly what CrowdStrike did, I do understand its side-effects. Same thing with our servers. Our product in particular is mostly on a flavor of Linux and is not impacted by any of this. The parts of the business with Windows servers on the other hand are actively war rooming to fix all of this.

So say what you will, but this is a huge issue for businesses that use CrowdStrike with Windows machines.

And, again, for the 100th time, none of that is the fault of Microsoft or Windows. Crowdstrike shipped a broken update and Crowdstrike bricked those machines.

Yes, you could argue it's ultimately Microsoft's fault for not building a sufficiently isolated kernel, but that's very much ignoring the forest for the trees in this case.

 

[Aquinus]

And, again, for the 100th time, none of that is the fault of Microsoft or Windows. Crowdstrike shipped a broken update and Crowdstrike bricked those machines.

Yes, you could argue it's ultimately Microsoft's fault for not building a sufficiently isolated kernel, but that's very much ignoring the forest for the trees in this case.

Sure, but the lay person doesn't care about the nuances of how CrowdStrike uses Windows' APIs. That's what I'm trying to get at. I know that it's CrowdStrike's fault, but regardless, the outcome is the same. My engineer still can't do his job and there are servers needing to be recovered, but my Mac users are just fine.

 

[P4-630]

And, again, for the 100th time, none of that is the fault of Microsoft or Windows. Crowdstrike shipped a broken update and Crowdstrike bricked those machines.

And again

due to CrowdStrike Falcon-package failure affecting Windows.

 

[bug]

Mac uses an API to collect kernel events, so the kernel driver required on Windows doesn't exist on Mac. Linux has user mode and kernel mode sensors available depending on the kernel, but I don't think kernel mode Linux hosts were affected as I suspect the bug was only introduced into the Windows code base at CrowdStrike.

As mentioned above, user mode APIs for kernel events compared to a kernel driver does have a performance impact.

The elephant in the room being: if it affects so many systems, how the hell did it go undetected all the way to prod? Though it could be a case of "we tested one thing and released another".

 

[Makaveli]

Exactly the boat I'm in... I'm the infosec manager so I'm just the one documenting the wreckage.

Seriously I fought back against Bitlocker encryption on all machines for this reason at my last place because I told them recovery after an event like this is a major pain in the ass. I believe I saw a mcafee update in the past brick one of my workstation. but for me I thought ahead had and images and other thing done to recover my own machine. Now of course encryption is important so they did it anyways but none of my other co-workers took the extra steps I did encase of a disaster.

 

[P4-630]

The outage was caused by a major outage 'after a software update' at the cybersecurity company CrowdStrike.
The company now says it has found a solution to the problem.
“The problem has been identified, isolated and a solution implemented,” Crowdstrike CEO said.

 

[mab1376]

The elephant in the room being: if it affects so many systems, how the hell did it go undetected all the way prod? Though it could be a case of "we tested one thing and released another".

that question is exactly why their stock is tanking.

Seriously I fought back against Bitlocker encryption on all machines for this reason at my last place because I told them recovery after an event like this is a major pain in the ass. I believe I saw a mcafee update in the past brick one of my workstation. but for me I thought ahead had and images and other thing done to recover my own machine. Now of course encryption is important so they did it anyways but none of my other co-workers took the extra steps I did encase of a disaster.

BitLocker is required for our ISO27001 certification if a machine has sensitive info on it, which most do in my environment.

 

[TheLostSwede]

It's not affecting everyone...