Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

[mab1376]

Saw an announcement that NYC mayor was going to do a press conference about the Microsoft outage.

Such headlines are everywhere.

I dont know if CS will survive after this.

I think they will, but they will certainly lose a sizeable chunk of their customers. Their stock hasn't tanked nearly as much as I expected, but California hasn't fully woken up yet lol.

Someone absolutely has, albeit not directly. We've been impacted here by logmein services being down.

I was affected and teams are actively working on repairs.

 

[Super Firm Tofu]

Someone absolutely has, albeit maybe not directly. We've been impacted here by logmein services being down.

Not my personal corpo provided device, but the most I can say is currently in the environment there's around 3000 devices still requiring a fix, a non-zero amount being windows servers.

 

[HTC]

Someone absolutely has, albeit maybe not directly. We've been impacted here by logmein services being down.

I'm on sick leave so dunno if the MES we use @ work is in any way affected by this or not:

https://www.sap.com/products/scm/execution-mes/what-is-mes.html

 

[R-T-B]

@R-T-B
Come on, man, I specifically said “smaller CONSUMER install base”. I was talking in this context and this context alone. I am well aware of Linux server market share.

But then you went on to imply that makes it a useless target, which is patently false.

 

[ZoneDymo]

MS isn’t the ones who contract this firm, no. Where did you even infer it?

well from the title "faulty windows update"
I dont think we would call it a windows update if the update isnt for the OS which would be MS's responsibility

 

[mechtech]

Still works

 

[Solaris17]

well from the title "faulty windows update"
I dont think we would call it a windows update if the update isnt for the OS which would be MS's responsibility

The title is wrong, I wouldn't put much stake in it. This is and is only a crowdstrike issue; they even admitted it.

If you really want to blame someone, try your management that under funded the IT dept so much that didnt have the budget to roll this out to testing before it hit mass.

For the rest, please keep wack conspiracy theories away from the thread.

 

[Onasi]

But then you went on to imply that makes it a useless target, which is patently false.

For security threats targeting consumer PCs (and, I suppose, end-point enterprise)? Yes, it is. Again, that was the context. Same context that “just use Linux bruh” advocates operate on and I was talking about them. Servers and datacenters are a completely different kettle of fish and are definitely not what the majority of threats target. The most serious ones, sure, but not the most numerous. That was my implication. I was talking quantitative, not qualitative. I guess we fundamentally misunderstood each other.

 

[thesmokingman]

It's kind of insane that Crowdstrike used so many windows shit boxes instead of nix.

 

[Makaveli]

CrowdStrike's market cap plunges $12.5 billion in wake of global outage

heads gonna roll

 

[64K]

CrowdStrike's market cap plunges $12.5 billion in wake of global outage

heads gonna roll

Maybe but don't put too much into investors getting panicky. They drink way too much coffee and don't sleep very well.

 

[thesmokingman]

CrowdStrike's market cap plunges $12.5 billion in wake of global outage

heads gonna roll

I'm surprised it's only down 8%-9% atm, wtf should be triple that at least.

 

[Solaris17]

Maybe but don't put too much into investors getting panicky. They drink way too much coffee and don't sleep very well.

Agree. Even in this thread some are up and arms but given post history barely know what is involved.

This could have happened to anyone, and while it is big it also unfairly paints crowdstrikes position.

With Kaspersky getting banned everyone must shift to another provider...thats millions of end points.

There are a ton of other super big players.

Carbon black
Emsisoft
MS Defender EPS
ESET
Cylance
Sophos

All are massive massive players with EDR used in a biz setting.

 

[Tomorrow]

ANY OS can be bricked by such a thing.

No it cant.

Their first mistake was rolling update to Production on Friday.

And now they and countless others have their weekends ruined. I hope they at least learn from their mistake.

Right, it is EDR. But does Windows HAVE to install this at the kernel level to be as effective as say OSX or Linux installing it in userspace? That is an OS design decision, isn't it?

Yes it is. And this is why it's also Windows's fault that a bad 3rd party update can bring down the whole OS.

Ahhhh Remember when all Computers had the Tap the F8 and you had the menu to boot into safe mode. Then Microsoft removed this feature on all computers since windows 8 and 10 and 11. Remember that good ole last known good configurations.

I have it enabled all all our machines on the network just in case something like this happens. I think ahead.
bcdedit /set {default} bootmenupolicy legacy

It works on all systems even ones with secure boot. It does not affect the boot processs any it's just there for emergencies when you need it most.

All the people in my life that called I just said tap F8 and wait for menu and then goto Last know Good Config. All working fine now.

Cheers all

In theory the system itself should detect and offer the Recovery after three failed boot attempts but in practice it does not always work. I too have F8 manually enabled. When booting from working windows then holding Shift while selecting restart or entering REAGENTC /boottore to terminal will force it to boot directly into Recovery. The problems start when Windows is unable to boot and automatic boot detection fails.

The most catastrophic case i had was with one Win10 machine. 22H2 update to it screwed up partition tables so bad that none of the fixes worked and attaching this disk to any other Windows compatible PC caused either BSOD or not even making past POST like it was on the original machine. Even doing a byte per byte clone to another empty disk produced the same issue when that cloned disk was attached. I've never seen anything like that before because initially i thought it was just a bad disk (SSD). Imagine trying to fix a machine that hangs in the POST. It was a nightmare. I ended up putting Win11 on it and manually migrating the data off the faulty disk.

TBF if Microsoft offered user-mode APIs into kernel events, it wouldn't be necessary to install a kernel driver.

Indeed. Im not usually in favor of locking down more but in some cases i have to begrudgingly admit it does have it's benefits.

@mab1376
This is fair, though I fear that would require them to essentially overhaul the entire kernel at this point, Vista/NT6-style, which isn’t in the cards anytime soon, I assume.

Are talking about the same company here? Microsoft - the company who cant even re-add some of the most requested features to Taskbar code that existed in prior Windows versions. The same company who took ages to add tabs to File Explorer. At this point i think it's better if they leave the kernel as is because looking at their track record i would trust them near that code with a ten feet pole.

It’s the same logic as with the “Linux is totally more secure, guys” arguments. While yeah, there are some advantages to the way it handles security, the main reason it’s “more secure” is just the sheer fact that nobody cares there is a significantly smaller consumer install base. If Linux someday magically becomes the lead desktop OS we can expect the same scale of security issues and threats as Windows.

Lolwut. Linux is way more used in serverland and probably has an overall larger and even more importantly, more valuable installbase than windows.

@R-T-B
Come on, man, I specifically said “smaller CONSUMER install base”. I was talking in this context and this context alone. I am well aware of Linux server market share.

So because most Linux machines are servers they dont have to deal with security issues? I find that hard to believe.
Most Windows related security issues are caused by users, not the OS itself.
Also something having a small market share does not mean it's more secure - for example running Windows XP on an internet connected machine.

 

[Solaris17]

So because most Linux machines are servers they dont have to deal with security issues? I find that hard to believe.

Fun fact, crowdstrike has a linux agent, and if you are doing anything regulatory you need these things installed, even on linux.

 

[Steevo]

One of my vendors got this and I stood watching their IT guy sweating trying to figure out what was going on. Glad I haven't turned the keys over for any of my side gigs. If I could get Starlink to assign IP addresses or actually give a static IP my life would be perfect.

 

[Onasi]

No it cant.

Good talk.

Are talking about the same company here? Microsoft - the company who cant even re-add some of the most requested features to Taskbar code that existed in prior Windows versions. The same company who took ages to add tabs to File Explorer. At this point i think it's better if they leave the kernel as is because looking at their track record i would trust them near that code with a ten feet pole.

As @Assimilator would say, I hate to quote myself, but:

Why are we acting like MS engineers (and I do mean engineers, not people who shove marketing driven shit on top of a good core) are incompetent mole-people who fail at basic tasks?

So because most Linux machines are servers they dont have to deal with security issues? I find that hard to believe.

Never said that. The character of the threats is markedly different though and so is the quantity.

Also something having a small market share does not mean it's more secure - for example running Windows XP on an internet connected machine.

Are we drawing parallels between Linux desktop usage and using an outdated, unsupported OS? You realize this is silly, right,

Anyway, this is my last post on the issue since I feel the thread was derailed by my participation on this topic a bit. My apologies to @Solaris17 and @the54thvoid.

 

[thesmokingman]

Josh on CNBS was always saying how you need Crowdstrike in your life er I mean portfolio, lmao.

 

[remixedcat]

BTW I had the same sorta thing happen when webroot anti-virus decided to roll out an update that caused a blue screen and there was a messed up DLL file just like this and I had to boot into recovery mode to delete that file (I'll add it in an edit or reply later for the exact file name cuz I can't remember it off the top of my head.)

THESE COMPANIES NEED MORE QA!!!!

 

[Steevo]

Josh on CNBS was always saying how you need Crowdstrike in your life er I mean portfolio, lmao.

Using a single company to manage all the computers without oversight and testing for critical companies is just asking for problems.

A warranty is only as good as the company making it. IT is only as strong as its weakest link.

If you are in IT and don't have a master machine/password/configuration spreadsheet/flowchart that is saved to a removable drive in a safe or printed out you are setting the company up for failure, if you know it all and die or something happens the next person gets screwed.

 

[Vayra86]

Once again support and proof the cloud is both blessing and curse. Redundancy must be introduced locally. A nice reality check vs Microsofts recent cloud only push.

Also yet another reason to run LTSC. Or have a mirror of your environment capable of running on Linux.

BTW I had the same sorta thing happen when webroot anti-virus decided to roll out an update that caused a blue screen and there was a messed up DLL file just like this and I had to boot into recovery mode to delete that file (I'll add it in an edit or reply later for the exact file name cuz I can't remember it off the top of my head.)

THESE COMPANIES NEED MORE QA!!!

No amount of QA will prevent the risk of stacking so many interdependent infra and services on top of each other. Fact of life: if you depend on many others, you are vulnerable. Mitigate the risk sure. Prevention? Forget it.

 

[A Computer Guy]

This is kind of similar thing is what happened with Asus routers I think last year or the year before that. MacAfee pushed an update crippling Asus routers regardless if you enabled that feature or not requiring a reflash to fix it assuming it didn't crash in the process.

 

[Gooigi's Ex]

Just want throw my voice out there that I work from home and my job got bent as well

 

[Vayra86]

Just want throw my voice out there that I work from home and my job got bent as well

Nice day off then eh. Great weather for a nice stroll in the woods I say Thats honestly what I do if IT infra goes down. Companies want to be stupid its their problem Im not going to sit there hitting F5 for their idiocy.

Im also the consultant telling them they need redundancy and tight risk mgmt.

And since 2022 especially in the EU with Russia in the east I just cant fathom why we havent taken more measures to mitigate risk to online services. Asleep at the wheel. Its completely irresponsible; you just have to have an offline method of running the biz on hand.

 

[windwhirl]

It's kind of insane that Crowdstrike used so many windows shit boxes instead of nix.

It's people using CrowdStrike on Windows, not CrowdStrike using Windows.

Also, I heard some Linux systems are getting hit because they also have a Linux version of whatever CrowdStrike software there's on Windows that's getting hit with the bug.

Semi-related everything seems to be working fine in my country. Probably because not many or no one uses CrowdStrike (too expensive and probably no reps in the country)