Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

[Tek-Check]

Wow! What a catastrophe...

 

[Vya Domus]

Is it just me or do others think critical IT and society infrastructure services need to switch from Windows to Linux?

Anyone that used Linux for any extend period of time knows you can absolutely destroy a Linux installation in an irrecoverable fashion 10X as easily as you can on Windows.

 

[Makaveli]

Anyone that used Linux for any extend period of time knows you can absolutely destroy a Linux installation in an irrecoverable fashion 10X as easily as you can on Windows.

windows systems are easy to recover as long as encryption isn't in the way.

Experience doing it on multiple different versions of windows over the decades.

 

[mab1376]

windows systems are easy to recover as long as encryption isn't in the way.

Experience doing it on multiple different versions of windows over the decades.

its easy with bitlocker too, just an extra step to retrieve the key.

 

[phints]

RIP CrowdStrike, you fucked us all. Now if only Linux got their shit together the world would have switched to it 20 years ago when I begged and begged.

Microsoft should fork their OS just like they did with NT in the early 2000s. Windows 11 remains for 3-5 years as mainstream, meanwhile on the other branch it becomes a Linux distro. Then eventually Windows 12 or whatever takes over as full POSIX compliant Linux distro... one can dream.

 

[remixedcat]

Once again support and proof the cloud is both blessing and curse. Redundancy must be introduced locally. A nice reality check vs Microsofts recent cloud only push.

Also yet another reason to run LTSC. Or have a mirror of your environment capable of running on Linux.


No amount of QA will prevent the risk of stacking so many interdependent infra and services on top of each other. Fact of life: if you depend on many others, you are vulnerable. Mitigate the risk sure. Prevention? Forget it.

how about having a system set in place that prevents a release till several QA engineers have signed on it and verified that it works?

 

[Vayra86]

how about having a system set in place that prevents a release till several QA engineers have signed on it and verified that it works?

Code is never fool proof in a world moving as fast as it is today. There is always a time constraint

 

[windwhirl]

Microsoft should fork their OS just like they did with NT in the early 2000s. Windows 11 remains for 3-5 years as mainstream, meanwhile on the other branch it becomes a Linux distro.

That's not how it works, that's probably very dumb business-wise, it's not gonna ever happen.

 

[tfp]

Was in the middle of a release last night when this hit. Good fun.

 

[A Computer Guy]

how about having a system set in place that prevents a release till several QA engineers have signed on it and verified that it works?

There is this concept called staging, typically it starts by rolling out updates to a small group to ensure there are no major problems before rolling it out progressively to larger groups. This mitigates risks and/or being completely overwhelmed by support calls should some some major blocker be found.

 

[R-T-B]

It's kind of insane that Crowdstrike used so many windows shit boxes instead of nix.

It's the other way around bro. Crowdstrike is a product for windows boxes, and is installed on them.

 

[HTC]

So ... i just went to the hypermarket ... and it was affected by this CrowdStrike problem ...

Thing i found weird is that only the SELF SERVICE payment area was affected: non self service WAS NOT affected.

 

[R-T-B]

Still works

View attachment 355703

Actually banks are amongst those having issues. I'd carry cash for a bit.

 

[Easy Rhino]

Code is never fool proof in a world moving as fast as it is today. There is always a time constraint

That's the real problem. The world is moving way too fast. Very little time to think and plan out changes to systems. Very few people turn off their devices for the weekend and disconnect from it all and leave time to think. Corporations all pushing everyone to GO GO GO and then this inevitably happens. People have lost perspective.

 

[R-T-B]

Fun fact, crowdstrike has a linux agent, and if you are doing anything regulatory you need these things installed, even on linux.

Or not. I fought tooth and nail to avoid it. And I did. Might not be possible everywhere but at least at my lowly records storage role it was possible. I just have to jump through a longer list of OTHER compliance proofs, but worth it to avoid headaches like this.

 

[kondamin]

Imagine the headlines if it was still kaspersky doing the majority of the worlds security

 

[Solaris17]

Or not. I fought tooth and nail to avoid it. And I did. Might not be possible everywhere but at least at my lowly records storage role it was possible. I just have to jump through a longer list of OTHER compliance proofs, but worth it to avoid headaches like this.

Thats rough; with any luck I never get to deal with your company or any other company that fights to work around security practices. There are so many out there already.

 

[remixedcat]

There is this concept called staging, typically it starts by rolling out updates to a small group to ensure there are no major problems before rolling it out progressively to larger groups. This mitigates risks and/or being completely overwhelmed by support calls should some some major blocker be found.

yep and apperantly they didn't do this!!!

 

[windwhirl]

Imagine the headlines if it was still kaspersky doing the majority of the worlds security

That's completely irrelevant since Kaspersky had nothing to do with this incident.

 

[R-T-B]

Thats rough; with any luck I never get to deal with your company or any other company that fights to work around security practices. There are so many out there already.

A company doing security for you is not a mandatory security practice, or even a good one. It would just be lazy at my level and globally it leads to what we are witnessing.

Note I am completely functioning through this event and have a perfect security track record to date. I don't get lax just because I don't sign off to someone else. I think you should know that by now.

My sole job in my org is security of the county records. The buck stops here and newsflash, this makes me care.

Frankly, I'm thinking you are either trolling, or playing the contrarian leveling that comment at me.

 

[trparky]

And this is why it's also Windows's fault that a bad 3rd party update can bring down the whole OS.

The kernel isn't like other kinds of programs where if something goes wrong, we can just restart the program; it doesn't work like that at the kernel-level. If something goes wrong there, the system can be left in an unknown state where you can't know if there's any data corruption so it's best to halt the system and (in the case of Windows) show a BSOD or (in Linux) show a kernel panic.

Now, could Microsoft code in some kind of way to check to see if a driver has failed to load and if it repeatedly fails to load producing a BSOD, then it's dropped out after X number of times? Sure, that's probably possible and probably something that they should include in some kind of future update.

There's a whole Wikipedia article about the incident... 2024 CrowdStrike incident - Wikipedia

Cybersecurity consultant Troy Hunt characterised the incident as the "largest IT outage in history", comparing the impact with the Y2K bug by saying: "This is basically what we were all worried about with Y2K, except it's actually happened this time"

 

[damric]

Las Vegas late last night early this morning:

 

[DaemonForce]

THESE COMPANIES NEED MORE QA!!!!

There it is. That's the lesson.

Anyone that used Linux for any extend period of time knows you can absolutely destroy a Linux installation in an irrecoverable fashion 10X as easily as you can on Windows.

This is something I worry about whenever using a linux system for any extended period. Like, I get that it takes a lot of irresponsible behavior to break a linux system but it could also be something like adjusting something at boot and suddenly vmlinuz doko...Do people just image their entire linux partitions or something? How is linux recovery at a granular level? Does it even exist? I'm pretty sure it does, somehow but I've never been put into the situation where it happens. I can restore any Windows box with boot issues by running a simple bcdedit and can go deeper with removing/replacing system files but I don't have any management tools in place to do that to linux.

Imagine the headlines if it was still kaspersky doing the majority of the worlds security

Every news outlet would turn plaid and explode. Amazing.

That's completely irrelevant since Kaspersky had nothing to do with this incident.

How would you feel if you didn't eat breakfast today?

 

[Darmok N Jalad]

DMVs in my bi-state area have been down all day. The local news is calling it a “Microsoft outage,” showing signs on business doors, calling it the same thing. It doesn’t help that this problem creates a Microsoft Blue Screen, so it’s a messaging disaster for Microsoft, even if they aren’t the problem.

 

[Assimilator]

Once again support and proof the cloud is both blessing and curse. Redundancy must be introduced locally. A nice reality check vs Microsofts recent cloud only push.

People running in Azure were completely unaffected...