Friday, July 19th 2024

Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

by
btarunr
 Discuss (234 Comments)
A faulty software update to enterprise computers by cybersecurity firm CrowdStrike has taken millions of computers offline, most of which are in a commercial or enterprise environment, or are Azure deployments. CrowdStrike provides periodic software and security updates to commercial PCs, enterprise PCs, and cloud instances, with a high degree of automation. The latest update reportedly breaks the Windows bootloader, causing bluescreens of death (BSODs), and if configured, invokes Windows Recovery. Enterprises tend to bulletproof the bootloaders of their client machines, and disable generic Windows Recovery tools from Microsoft, which means businesses around the world are left with large numbers of machines that will each take manual fixing. The so-called "Windows CrowdStrike BSOD deluge" has hit critical businesses such as banks, airlines, supermarket chains, and TV broadcasters. Meanwhile, sysadmins on Reddit are wishing each other a happy weekend.
Source: The Verge

Related News

Add your own comment

234 Comments on Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

#151
Vya Domus
DavenIs it just me or do others think critical IT and society infrastructure services need to switch from Windows to Linux?
Anyone that used Linux for any extend period of time knows you can absolutely destroy a Linux installation in an irrecoverable fashion 10X as easily as you can on Windows.
Posted on Reply
#152
Makaveli
Vya DomusAnyone that used Linux for any extend period of time knows you can absolutely destroy a Linux installation in an irrecoverable fashion 10X as easily as you can on Windows.
windows systems are easy to recover as long as encryption isn't in the way.

Experience doing it on multiple different versions of windows over the decades.
Posted on Reply
#153
mab1376
Makaveliwindows systems are easy to recover as long as encryption isn't in the way.

Experience doing it on multiple different versions of windows over the decades.
its easy with bitlocker too, just an extra step to retrieve the key.
Posted on Reply
#154
phints
RIP CrowdStrike, you fucked us all. Now if only Linux got their shit together the world would have switched to it 20 years ago when I begged and begged.

Microsoft should fork their OS just like they did with NT in the early 2000s. Windows 11 remains for 3-5 years as mainstream, meanwhile on the other branch it becomes a Linux distro. Then eventually Windows 12 or whatever takes over as full POSIX compliant Linux distro... one can dream.
Posted on Reply
#155
remixedcat
Vayra86Once again support and proof the cloud is both blessing and curse. Redundancy must be introduced locally. A nice reality check vs Microsofts recent cloud only push.

Also yet another reason to run LTSC. Or have a mirror of your environment capable of running on Linux.


No amount of QA will prevent the risk of stacking so many interdependent infra and services on top of each other. Fact of life: if you depend on many others, you are vulnerable. Mitigate the risk sure. Prevention? Forget it.
how about having a system set in place that prevents a release till several QA engineers have signed on it and verified that it works?
Posted on Reply
#156
Vayra86
remixedcathow about having a system set in place that prevents a release till several QA engineers have signed on it and verified that it works?
Code is never fool proof in a world moving as fast as it is today. There is always a time constraint
Posted on Reply
#157
windwhirl
phintsMicrosoft should fork their OS just like they did with NT in the early 2000s. Windows 11 remains for 3-5 years as mainstream, meanwhile on the other branch it becomes a Linux distro.
That's not how it works, that's probably very dumb business-wise, it's not gonna ever happen.
Posted on Reply
#158
tfp
Was in the middle of a release last night when this hit. Good fun.
Posted on Reply
#159
A Computer Guy
remixedcathow about having a system set in place that prevents a release till several QA engineers have signed on it and verified that it works?
There is this concept called staging, typically it starts by rolling out updates to a small group to ensure there are no major problems before rolling it out progressively to larger groups. This mitigates risks and/or being completely overwhelmed by support calls should some some major blocker be found.
Posted on Reply
#160
R-T-B
thesmokingmanIt's kind of insane that Crowdstrike used so many windows shit boxes instead of nix. :kookoo:
It's the other way around bro. Crowdstrike is a product for windows boxes, and is installed on them.
Posted on Reply
#161
HTC
So ... i just went to the hypermarket ... and it was affected by this CrowdStrike problem ...

Thing i found weird is that only the SELF SERVICE payment area was affected: non self service WAS NOT affected.
Posted on Reply
#162
R-T-B
mechtechStill works

Actually banks are amongst those having issues. I'd carry cash for a bit.
Posted on Reply
#163
Easy Rhino
Linux Advocate
Vayra86Code is never fool proof in a world moving as fast as it is today. There is always a time constraint
That's the real problem. The world is moving way too fast. Very little time to think and plan out changes to systems. Very few people turn off their devices for the weekend and disconnect from it all and leave time to think. Corporations all pushing everyone to GO GO GO and then this inevitably happens. People have lost perspective.
Posted on Reply
#164
R-T-B
Solaris17Fun fact, crowdstrike has a linux agent, and if you are doing anything regulatory you need these things installed, even on linux.
Or not. I fought tooth and nail to avoid it. And I did. Might not be possible everywhere but at least at my lowly records storage role it was possible. I just have to jump through a longer list of OTHER compliance proofs, but worth it to avoid headaches like this.
Posted on Reply
#165
kondamin
Imagine the headlines if it was still kaspersky doing the majority of the worlds security
Posted on Reply
#166
Solaris17
Super Dainty Moderator
R-T-BOr not. I fought tooth and nail to avoid it. And I did. Might not be possible everywhere but at least at my lowly records storage role it was possible. I just have to jump through a longer list of OTHER compliance proofs, but worth it to avoid headaches like this.
Thats rough; with any luck I never get to deal with your company or any other company that fights to work around security practices. There are so many out there already.
Posted on Reply
#167
remixedcat
A Computer GuyThere is this concept called staging, typically it starts by rolling out updates to a small group to ensure there are no major problems before rolling it out progressively to larger groups. This mitigates risks and/or being completely overwhelmed by support calls should some some major blocker be found.
yep and apperantly they didn't do this!!!
Posted on Reply
#168
windwhirl
kondaminImagine the headlines if it was still kaspersky doing the majority of the worlds security
That's completely irrelevant since Kaspersky had nothing to do with this incident.
Posted on Reply
#169
R-T-B
Solaris17Thats rough; with any luck I never get to deal with your company or any other company that fights to work around security practices. There are so many out there already.
A company doing security for you is not a mandatory security practice, or even a good one. It would just be lazy at my level and globally it leads to what we are witnessing.

Note I am completely functioning through this event and have a perfect security track record to date. I don't get lax just because I don't sign off to someone else. I think you should know that by now.

My sole job in my org is security of the county records. The buck stops here and newsflash, this makes me care.

Frankly, I'm thinking you are either trolling, or playing the contrarian leveling that comment at me.
Posted on Reply
#170
trparky
TomorrowAnd this is why it's also Windows's fault that a bad 3rd party update can bring down the whole OS.
The kernel isn't like other kinds of programs where if something goes wrong, we can just restart the program; it doesn't work like that at the kernel-level. If something goes wrong there, the system can be left in an unknown state where you can't know if there's any data corruption so it's best to halt the system and (in the case of Windows) show a BSOD or (in Linux) show a kernel panic.

Now, could Microsoft code in some kind of way to check to see if a driver has failed to load and if it repeatedly fails to load producing a BSOD, then it's dropped out after X number of times? Sure, that's probably possible and probably something that they should include in some kind of future update.

There's a whole Wikipedia article about the incident... 2024 CrowdStrike incident - Wikipedia
Cybersecurity consultant Troy Hunt characterised the incident as the "largest IT outage in history", comparing the impact with the Y2K bug by saying: "This is basically what we were all worried about with Y2K, except it's actually happened this time"
Posted on Reply
#171
damric
Las Vegas late last night early this morning:

Posted on Reply
#172
DaemonForce
remixedcatTHESE COMPANIES NEED MORE QA!!!!
There it is. That's the lesson.
Vya DomusAnyone that used Linux for any extend period of time knows you can absolutely destroy a Linux installation in an irrecoverable fashion 10X as easily as you can on Windows.
This is something I worry about whenever using a linux system for any extended period. Like, I get that it takes a lot of irresponsible behavior to break a linux system but it could also be something like adjusting something at boot and suddenly vmlinuz doko...Do people just image their entire linux partitions or something? How is linux recovery at a granular level? Does it even exist? I'm pretty sure it does, somehow but I've never been put into the situation where it happens. I can restore any Windows box with boot issues by running a simple bcdedit and can go deeper with removing/replacing system files but I don't have any management tools in place to do that to linux.
kondaminImagine the headlines if it was still kaspersky doing the majority of the worlds security
Every news outlet would turn plaid and explode. Amazing.
windwhirlThat's completely irrelevant since Kaspersky had nothing to do with this incident.
How would you feel if you didn't eat breakfast today? :rolleyes:
Posted on Reply
#173
Darmok N Jalad
DMVs in my bi-state area have been down all day. The local news is calling it a “Microsoft outage,” showing signs on business doors, calling it the same thing. It doesn’t help that this problem creates a Microsoft Blue Screen, so it’s a messaging disaster for Microsoft, even if they aren’t the problem.
Posted on Reply
#174
Assimilator
Vayra86Once again support and proof the cloud is both blessing and curse. Redundancy must be introduced locally. A nice reality check vs Microsofts recent cloud only push.
People running in Azure were completely unaffected...
Posted on Reply
#175
Vayra86
AssimilatorPeople running in Azure were completely unaffected...
Oh is that why gov sites all over the globe are down? O365 included? You might want to double check your info. Im not getting mine from a news site. Even despite MS redundancy and maximum reliability policies those went down simply because MS lost four data locations in the US. Closer to my workspace we lost Azure devops.

Additionally we arent out of the woods yet even with the Crowdstrike update rolled back; contrary to what news outlets say now.
Posted on Reply
Add your own comment
Nov 21st, 2024 06:43 EST change timezone

Latest GPU Drivers

New Forum Posts

Popular Reviews

Controversial News Posts