Friday, July 19th 2024

Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

A faulty software update to enterprise computers by cybersecurity firm CrowdStrike has taken millions of computers offline, most of which are in a commercial or enterprise environment, or are Azure deployments. CrowdStrike provides periodic software and security updates to commercial PCs, enterprise PCs, and cloud instances, with a high degree of automation. The latest update reportedly breaks the Windows bootloader, causing bluescreens of death (BSODs), and if configured, invokes Windows Recovery. Enterprises tend to bulletproof the bootloaders of their client machines, and disable generic Windows Recovery tools from Microsoft, which means businesses around the world are left with large numbers of machines that will each take manual fixing. The so-called "Windows CrowdStrike BSOD deluge" has hit critical businesses such as banks, airlines, supermarket chains, and TV broadcasters. Meanwhile, sysadmins on Reddit are wishing each other a happy weekend.
Source: The Verge
Add your own comment

234 Comments on Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

#126
Super Firm Tofu
R-T-BSomeone absolutely has, albeit maybe not directly. We've been impacted here by logmein services being down.
Not my personal corpo provided device, but the most I can say is currently in the environment there's around 3000 devices still requiring a fix, a non-zero amount being windows servers.
Posted on Reply
#128
R-T-B
Onasi@R-T-B
Come on, man, I specifically said “smaller CONSUMER install base”. I was talking in this context and this context alone. I am well aware of Linux server market share.
But then you went on to imply that makes it a useless target, which is patently false.
Posted on Reply
#129
ZoneDymo
OnasiMS isn’t the ones who contract this firm, no. Where did you even infer it?
well from the title "faulty windows update"
I dont think we would call it a windows update if the update isnt for the OS which would be MS's responsibility
Posted on Reply
#131
Solaris17
Super Dainty Moderator
ZoneDymowell from the title "faulty windows update"
I dont think we would call it a windows update if the update isnt for the OS which would be MS's responsibility
The title is wrong, I wouldn't put much stake in it. This is and is only a crowdstrike issue; they even admitted it.

If you really want to blame someone, try your management that under funded the IT dept so much that didnt have the budget to roll this out to testing before it hit mass.

For the rest, please keep wack conspiracy theories away from the thread.
Posted on Reply
#132
Onasi
R-T-BBut then you went on to imply that makes it a useless target, which is patently false.
For security threats targeting consumer PCs (and, I suppose, end-point enterprise)? Yes, it is. Again, that was the context. Same context that “just use Linux bruh” advocates operate on and I was talking about them. Servers and datacenters are a completely different kettle of fish and are definitely not what the majority of threats target. The most serious ones, sure, but not the most numerous. That was my implication. I was talking quantitative, not qualitative. I guess we fundamentally misunderstood each other.
Posted on Reply
#133
thesmokingman
It's kind of insane that Crowdstrike used so many windows shit boxes instead of nix. :kookoo:
Posted on Reply
#134
Makaveli
CrowdStrike's market cap plunges $12.5 billion in wake of global outage

heads gonna roll

Posted on Reply
#135
64K
MakaveliCrowdStrike's market cap plunges $12.5 billion in wake of global outage

heads gonna roll

Maybe but don't put too much into investors getting panicky. They drink way too much coffee and don't sleep very well.
Posted on Reply
#136
thesmokingman
MakaveliCrowdStrike's market cap plunges $12.5 billion in wake of global outage

heads gonna roll

I'm surprised it's only down 8%-9% atm, wtf should be triple that at least.
Posted on Reply
#137
Solaris17
Super Dainty Moderator
64KMaybe but don't put too much into investors getting panicky. They drink way too much coffee and don't sleep very well.
Agree. Even in this thread some are up and arms but given post history barely know what is involved.

This could have happened to anyone, and while it is big it also unfairly paints crowdstrikes position.

With Kaspersky getting banned everyone must shift to another provider...thats millions of end points.

There are a ton of other super big players.

Carbon black
Emsisoft
MS Defender EPS
ESET
Cylance
Sophos

All are massive massive players with EDR used in a biz setting.
Posted on Reply
#138
Tomorrow
OnasiANY OS can be bricked by such a thing.
No it cant.
ChomiqTheir first mistake was rolling update to Production on Friday.
And now they and countless others have their weekends ruined. I hope they at least learn from their mistake.
Easy RhinoRight, it is EDR. But does Windows HAVE to install this at the kernel level to be as effective as say OSX or Linux installing it in userspace? That is an OS design decision, isn't it?
Yes it is. And this is why it's also Windows's fault that a bad 3rd party update can bring down the whole OS.
LycanwolfenAhhhh Remember when all Computers had the Tap the F8 and you had the menu to boot into safe mode. Then Microsoft removed this feature on all computers since windows 8 and 10 and 11. Remember that good ole last known good configurations.

I have it enabled all all our machines on the network just in case something like this happens. I think ahead.
bcdedit /set {default} bootmenupolicy legacy

It works on all systems even ones with secure boot. It does not affect the boot processs any it's just there for emergencies when you need it most.

All the people in my life that called I just said tap F8 and wait for menu and then goto Last know Good Config. All working fine now.

Cheers all
In theory the system itself should detect and offer the Recovery after three failed boot attempts but in practice it does not always work. I too have F8 manually enabled. When booting from working windows then holding Shift while selecting restart or entering REAGENTC /boottore to terminal will force it to boot directly into Recovery. The problems start when Windows is unable to boot and automatic boot detection fails.

The most catastrophic case i had was with one Win10 machine. 22H2 update to it screwed up partition tables so bad that none of the fixes worked and attaching this disk to any other Windows compatible PC caused either BSOD or not even making past POST like it was on the original machine. Even doing a byte per byte clone to another empty disk produced the same issue when that cloned disk was attached. I've never seen anything like that before because initially i thought it was just a bad disk (SSD). Imagine trying to fix a machine that hangs in the POST. It was a nightmare. I ended up putting Win11 on it and manually migrating the data off the faulty disk.
mab1376TBF if Microsoft offered user-mode APIs into kernel events, it wouldn't be necessary to install a kernel driver.
Indeed. Im not usually in favor of locking down more but in some cases i have to begrudgingly admit it does have it's benefits.
Onasi@mab1376
This is fair, though I fear that would require them to essentially overhaul the entire kernel at this point, Vista/NT6-style, which isn’t in the cards anytime soon, I assume.
Are talking about the same company here? Microsoft - the company who cant even re-add some of the most requested features to Taskbar code that existed in prior Windows versions. The same company who took ages to add tabs to File Explorer. At this point i think it's better if they leave the kernel as is because looking at their track record i would trust them near that code with a ten feet pole.
OnasiIt’s the same logic as with the “Linux is totally more secure, guys” arguments. While yeah, there are some advantages to the way it handles security, the main reason it’s “more secure” is just the sheer fact that nobody cares there is a significantly smaller consumer install base. If Linux someday magically becomes the lead desktop OS we can expect the same scale of security issues and threats as Windows.
R-T-BLolwut. Linux is way more used in serverland and probably has an overall larger and even more importantly, more valuable installbase than windows.
Onasi@R-T-B
Come on, man, I specifically said “smaller CONSUMER install base”. I was talking in this context and this context alone. I am well aware of Linux server market share.
So because most Linux machines are servers they dont have to deal with security issues? I find that hard to believe.
Most Windows related security issues are caused by users, not the OS itself.
Also something having a small market share does not mean it's more secure - for example running Windows XP on an internet connected machine.
Posted on Reply
#139
Solaris17
Super Dainty Moderator
TomorrowSo because most Linux machines are servers they dont have to deal with security issues? I find that hard to believe.
Fun fact, crowdstrike has a linux agent, and if you are doing anything regulatory you need these things installed, even on linux.
Posted on Reply
#140
Steevo
One of my vendors got this and I stood watching their IT guy sweating trying to figure out what was going on. Glad I haven't turned the keys over for any of my side gigs. If I could get Starlink to assign IP addresses or actually give a static IP my life would be perfect.
Posted on Reply
#141
Onasi
TomorrowNo it cant.
Good talk.
TomorrowAre talking about the same company here? Microsoft - the company who cant even re-add some of the most requested features to Taskbar code that existed in prior Windows versions. The same company who took ages to add tabs to File Explorer. At this point i think it's better if they leave the kernel as is because looking at their track record i would trust them near that code with a ten feet pole.
As @Assimilator would say, I hate to quote myself, but:
OnasiWhy are we acting like MS engineers (and I do mean engineers, not people who shove marketing driven shit on top of a good core) are incompetent mole-people who fail at basic tasks?
TomorrowSo because most Linux machines are servers they dont have to deal with security issues? I find that hard to believe.
Never said that. The character of the threats is markedly different though and so is the quantity.
TomorrowAlso something having a small market share does not mean it's more secure - for example running Windows XP on an internet connected machine.
Are we drawing parallels between Linux desktop usage and using an outdated, unsupported OS? You realize this is silly, right,

Anyway, this is my last post on the issue since I feel the thread was derailed by my participation on this topic a bit. My apologies to @Solaris17 and @the54thvoid.
Posted on Reply
#142
thesmokingman
Josh on CNBS was always saying how you need Crowdstrike in your life er I mean portfolio, lmao.
Posted on Reply
#143
remixedcat
BTW I had the same sorta thing happen when webroot anti-virus decided to roll out an update that caused a blue screen and there was a messed up DLL file just like this and I had to boot into recovery mode to delete that file (I'll add it in an edit or reply later for the exact file name cuz I can't remember it off the top of my head.)

THESE COMPANIES NEED MORE QA!!!!
Posted on Reply
#144
Steevo
thesmokingmanJosh on CNBS was always saying how you need Crowdstrike in your life er I mean portfolio, lmao.
Using a single company to manage all the computers without oversight and testing for critical companies is just asking for problems.

A warranty is only as good as the company making it. IT is only as strong as its weakest link.

If you are in IT and don't have a master machine/password/configuration spreadsheet/flowchart that is saved to a removable drive in a safe or printed out you are setting the company up for failure, if you know it all and die or something happens the next person gets screwed.
Posted on Reply
#145
Vayra86
Once again support and proof the cloud is both blessing and curse. Redundancy must be introduced locally. A nice reality check vs Microsofts recent cloud only push.

Also yet another reason to run LTSC. Or have a mirror of your environment capable of running on Linux.
remixedcatBTW I had the same sorta thing happen when webroot anti-virus decided to roll out an update that caused a blue screen and there was a messed up DLL file just like this and I had to boot into recovery mode to delete that file (I'll add it in an edit or reply later for the exact file name cuz I can't remember it off the top of my head.)

THESE COMPANIES NEED MORE QA!!!
No amount of QA will prevent the risk of stacking so many interdependent infra and services on top of each other. Fact of life: if you depend on many others, you are vulnerable. Mitigate the risk sure. Prevention? Forget it.
Posted on Reply
#146
A Computer Guy
This is kind of similar thing is what happened with Asus routers I think last year or the year before that. MacAfee pushed an update crippling Asus routers regardless if you enabled that feature or not requiring a reflash to fix it assuming it didn't crash in the process.
Posted on Reply
#147
Gooigi's Ex
Just want throw my voice out there that I work from home and my job got bent as well
Posted on Reply
#148
Vayra86
Gooigi's ExJust want throw my voice out there that I work from home and my job got bent as well
Nice day off then eh. Great weather for a nice stroll in the woods I say :) Thats honestly what I do if IT infra goes down. Companies want to be stupid its their problem Im not going to sit there hitting F5 for their idiocy.

Im also the consultant telling them they need redundancy and tight risk mgmt. ;)

And since 2022 especially in the EU with Russia in the east I just cant fathom why we havent taken more measures to mitigate risk to online services. Asleep at the wheel. Its completely irresponsible; you just have to have an offline method of running the biz on hand.
Posted on Reply
#149
windwhirl
thesmokingmanIt's kind of insane that Crowdstrike used so many windows shit boxes instead of nix. :kookoo:
It's people using CrowdStrike on Windows, not CrowdStrike using Windows.

Also, I heard some Linux systems are getting hit because they also have a Linux version of whatever CrowdStrike software there's on Windows that's getting hit with the bug.

Semi-related everything seems to be working fine in my country. Probably because not many or no one uses CrowdStrike (too expensive and probably no reps in the country)
Posted on Reply
Add your own comment
Nov 21st, 2024 07:00 EST change timezone

New Forum Posts

Popular Reviews

Controversial News Posts