Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

[btarunr]

A faulty software update to enterprise computers by cybersecurity firm CrowdStrike has taken millions of computers offline, most of which are in a commercial or enterprise environment, or are Azure deployments. CrowdStrike provides periodic software and security updates to commercial PCs, enterprise PCs, and cloud instances, with a high degree of automation. The latest update reportedly breaks the Windows bootloader, causing bluescreens of death (BSODs), and if configured, invokes Windows Recovery. Enterprises tend to bulletproof the bootloaders of their client machines, and disable generic Windows Recovery tools from Microsoft, which means businesses around the world are left with large numbers of machines that will each take manual fixing. The so-called "Windows CrowdStrike BSOD deluge" has hit critical businesses such as banks, airlines, supermarket chains, and TV broadcasters. Meanwhile, sysadmins on Reddit are wishing each other a happy weekend.



View at TechPowerUp Main Site | Source

 

[Anoniem]

Well, happy weekend other sys admins out there Glad we don’t use CrowdStrike services/solutions.

 

[wolf]

Complete meltdown in Perth WA. Banks offline, grocery stores offline, a bloke couldn't get his zinger box from KFC after he knocked off. Triple J (radio) playing back to back bangers from a USB drive.

Feels like just about the only place not impacted was my workplace

 

[N3utro]

Good update management in a medium to large scale enterprise is to have multiple rings of deployment: the IT lab, the IT department users, non critical departments, critical ones. You never deploy windows updates directly, you manage your rings with WSUS. Bad management from these companies.

 

[Assimilator]

This doesn't just affect sysadmins, it affects anyone who uses a third party that uses Crowdstrike... ASK ME HOW I KNOW.

Microsoft's 365 platform is also having a bit of a wobble due to a seemingly unrelated issue with a configuration SNAFU in Azure.

Good update management in a medium to large scale enterprise is to have multiple rings of deployment: the IT lab, the IT department users, non critical departments, critical ones. You never deploy windows updates directly, you manage your rings with WSUS. Bad management from these companies.

The whole point of providers like Crowdstrike is that part of the service fee is for them to do that verification, so that you don't have to. Without such providers, small companies with minimal IT departments couldn't exist.

 

[N3utro]

This doesn't just affect sysadmins, it affects anyone who uses a third party that uses Crowdstrike... ASK ME HOW I KNOW.

Microsoft's 365 platform is also having a bit of a wobble due to a seemingly unrelated issue with a configuration SNAFU in Azure.


The whole point of providers like Crowdstrike is that part of the service fee is for them to do that verification, so that you don't have to. Without such providers, small companies with minimal IT departments couldn't exist.

"banks, airlines, supermarket chains," these are not small companies

 

[cerulliber]

Monday hirings QA at Crowdstrike and get 10x ROI. share prices increased. problem fixed, just don't be greed.

 

[Crackong]

Windows Update breaks things.

First time? . jpg

 

[WonkoTheSaneUK]

Pour one out for sysadmins, who have just learned that the fix is to log into each affected PC one at a time and delete the single bad file from each one.
It's going to be a loooooooooooooooooooooooooooooooooooooooong day for those in bigger organizations!

 

[Chaitanya]

MS needs a good screwing over and Windows needs to be offline like good old Win 7(and older).

 

[AnarchoPrimitiv]

Good update management in a medium to large scale enterprise is to have multiple rings of deployment: the IT lab, the IT department users, non critical departments, critical ones. You never deploy windows updates directly, you manage your rings with WSUS. Bad management from these companies.

Wouldn't be surprised if they fired a bunch of the people responsible for that to increase quarterly earnings.

 

[Onasi]

Windows Update breaks things.

MS needs a good screwing over and Windows needs to be offline like good old Win 7(and older).

I love to bag on MS just as the next guy, but it isn’t about them this time around.

 

[JcRabbit]

Judging by the chaos out there, this is what the Y2K bug could have been, but wasn't (because we made sure on time that it would not turn into anything like this - and thus it became a non event).

 

[ErikG]

Solution:

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally.

 

[Dazz023]

Windows Update breaks things.

First time? . jpg

Just clickbait headline to attract anti-MS crowd.

 

[the54thvoid]

Just clickbait headline to attract anti-MS crowd.

Not really - the header implicitly says,

Faulty Windows Update from CrowdStrike

It's blaming Crowdstrike.

 

[Assimilator]

"banks, airlines, supermarket chains," these are not small companies

But they are capitalist companies, and capitalism is all about maximising profits, and that means buying as little equipment as possible and hiring as few people as possible.

I love to bag on MS just as the next guy, but it isn’t about them this time around.

You're expecting the anti-Microsoft crowd to be capable of basic reading comprehension...

Not really - the header implicitly says,


It's blaming Crowdstrike.

It still implies it's somehow to do with Windows Update, though. A better headline would be "Windows security vendor releases faulty product update, hits banks and airlines around the world".

 

[Shou Miko]

Solution:

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally.

I already heard this didn't work for everyone including the registery fix.

I checked serveral Windows 10 installations at work incl. the one I use at work and I haven't found anything and a lot of my customers are running Windows 11 so hope they are more safe than Windows 10 users.

It's blaming Crowdstrike.

Correct.

 

[Kn0xxPT]

and then MS wants people move to Cloud Based systems .....
... lets ruin everyones day because of a "online bug" ....

 

[Chomiq]

But they are capitalist companies, and capitalism is all about maximising profits, and that means buying as little equipment as possible and hiring as few people as possible.


You're expecting the anti-Microsoft crowd to be capable of basic reading comprehension...


It still implies it's somehow to do with Windows Update, though. A better headline would be "Windows security vendor releases faulty product update, hits banks and airlines around the world".

Yeah, pretty much every news headline has MS logo in it but yeah, this is a CrowdStrike issue.

 

[DeathtoGnomes]

Someone at CrowdStrike is getting bent over the desk.

I bet IT people will be creating better recovery drives/discs after this blunder. I wonder if they heard of a thing called, QA.

"this wouldnt have happened if AI did all the work"

 

[katzi]

Windows Update breaks things.

First time? . jpg

Imagine blaming microsoft/windows update, for a 3rd party security software bricking windows.

 

[Onasi]

and then MS wants people move to Cloud Based systems .....
... lets ruin everyones day because of a "online bug" ....

*sigh* It’s not MS. It’s not about cloud based OS. It’s about a separate, non-MS affiliated cybersecurity firm pushing out a scuffed update for their endpoint enterprise solution that corrupted Windows boot-loader for their clients. It literally doesn’t affect anyone on a consumer level personally.

 

[efikkan]

Running Windows on critical systems, especially connected to the Internet and receiving automatic updates causing a global outage of numerous services, who saw this one coming? (trigger warning: sarcasm)

Having client PCs go offline may not be surprising, but seeing banks, traders, airlines, media companies etc., having their central services being offline from an update, that's just ridiculous. Come on guys, it's not 1995 any more, this level of incompetence isn't excusable. If you're making billions you can afford having properly trained staff and a properly managed tech "stack" with whatever appropriate failovers, backups, recovery images/procedures, etc. is needed to ensure reliability and security.

And yes, Microsoft certainly deserves blame for how easily their systems break, and for how tedious it is to roll back.

Thanks to CrowdStrike for accidentally revealing which companies who doesn't know how to handle their tech.

 

[Daven]

Is it just me or do others think critical IT and society infrastructure services need to switch from Windows to Linux?

I don’t want this to be last thing I see before I die.