Faulty Windows Update from CrowdStrike Hits Banks and Airlines Around the World

[Makaveli]

that question is exactly why their stock is tanking.


BitLocker is required for our ISO27001 certification if a machine has sensitive info on it, which most do in my environment.

This I know as we had alot of machines with sensitive info so it was happening regardless of what I said.

Most companies these days bitlocker is on by default.

 

[Assimilator]

The elephant in the room being: if it affects so many systems, how the hell did it go undetected all the way prod? Though it could be a case of "we tested one thing and released another".

MS has had similar SNAFUs with some of their updates. The underlying problem is that there are so very many Windows systems, with so very many discrete hardware and software combinations, that there is almost certainly going to be at least one of those systems negatively affected by an update. And the more intrusive the update, the more likely the fallout.

Testing can only get you so far because no matter how thoroughly you do it, you will never be able to replicate every single possible machine configuration that an update may be installed on. So it's very much a question of "good enough". Except that, in some cases, it isn't - and this is one of them.

This, BTW, is another thing that the "Windows bad" horde fails to understand. The possible combinations of Linux/Mac machines and software is so much smaller and so much less complicated, precisely due to a far smaller install base. For Linux there is also the advantage that most machines are servers, so bad patches there manifest as "third party is down" to the outside world and thus no news articles; for Apple it's the fact that they only have to support machines that are less than 4 years old (M1 launched November 2020), so it would be really hard to screw up.

 

[64K]

MS has had similar SNAFUs with some of their updates. The underlying problem is that there are so very many Windows systems, with so very many discrete hardware and software combinations, that there is almost certainly going to be at least one of those systems negatively affected by an update. And the more intrusive the update, the more likely the fallout.

Testing can only get you so far because no matter how thoroughly you do it, you will never be able to replicate every single possible machine configuration that an update may be installed on. So it's very much a question of "good enough". Except that, in some cases, it isn't - and this is one of them.

This, BTW, is another thing that the "Windows bad" horde fails to understand. The possible combinations of Linux/Mac machines and software is so much smaller and so much less complicated, precisely due to a far smaller install base. For Linux there is also the advantage that most machines are servers, so bad patches there manifest as "third party is down" to the outside world and thus no news articles; for Apple it's the fact that they only have to support machines that are less than 4 years old (M1 launched November 2020), so it would be really hard to screw up.

Sure, that is rational but how about when a Windows update just resets your personal settings for no reason or changes your login preference to a MS email account password when you don't even have a MS email account? That's not based on dealing with the complexities of numerous hardware and software combinations. That's just not bothering to test an update properly before dumping it on customers.

 

[Halo3Addict]

Can you not see from the comments that the title is, at minimum, confusing and misleading, and at maximum, clickbait. You should change it..

 

[bug]

MS has had similar SNAFUs with some of their updates. The underlying problem is that there are so very many Windows systems, with so very many discrete hardware and software combinations, that there is almost certainly going to be at least one of those systems negatively affected by an update. And the more intrusive the update, the more likely the fallout.

Testing can only get you so far because no matter how thoroughly you do it, you will never be able to replicate every single possible machine configuration that an update may be installed on. So it's very much a question of "good enough". Except that, in some cases, it isn't - and this is one of them.

This, BTW, is another thing that the "Windows bad" horde fails to understand. The possible combinations of Linux/Mac machines and software is so much smaller and so much less complicated, precisely due to a far smaller install base. For Linux there is also the advantage that most machines are servers, so bad patches there manifest as "third party is down" to the outside world and thus no news articles; for Apple it's the fact that they only have to support machines that are less than 4 years old (M1 launched November 2020), so it would be really hard to screw up.

I disagree. What you say applies to Microsoft SNAFUs (and yes, there is an argument regarding the many PC configurations in this case).
Crowdstrike's SNAFU seems to be affecting way more than an odd 1-2% which would be explained away by particular configurations.

 

[DaemonForce]

Microsoft releases an operating system and there's a bare minimum spec (that guys like me always manage to skirt) to install and run that OS, then there is the recommended spec for the experience to NOT be absolute hell.

How many guys do you think run Win10 on a single core system with 2GB ram?
How about a dual or quad core with 4GB?

That first one flat out doesn't exist anywhere outside of WinPE mode because it's insufferable.
That 2nd one is my Surface 3 tablet, which has never been a fun experience at any point in history.
The majority of Win10 users are probably running some 4/6/8 core chip with a bare minimum of 8GB, some GPU that is 3-16GB and gigabit ethernet.
If you think about it, that really narrows down the product stack and gives you a good idea of who and what you would expect to be running this or that.
Make some assumptions.

 

[Onasi]

This, BTW, is another thing that the "Windows bad" horde fails to understand. The possible combinations of Linux/Mac machines and software is so much smaller and so much less complicated, precisely due to a far smaller install base. For Linux there is also the advantage that most machines are servers, so bad patches there manifest as "third party is down" to the outside world and thus no news articles; for Apple it's the fact that they only have to support machines that are less than 4 years old (M1 launched November 2020), so it would be really hard to screw up.

It’s the same logic as with the “Linux is totally more secure, guys” arguments. While yeah, there are some advantages to the way it handles security, the main reason it’s “more secure” is just the sheer fact that nobody cares there is a significantly smaller consumer install base. If Linux someday magically becomes the lead desktop OS we can expect the same scale of security issues and threats as Windows.

 

[Assimilator]

I disagree. What you say applies to Microsoft SNAFUs (and yes, there is an argument regarding the many PC configurations in this case).
Crowdstrike's SNAFU seems to be affecting way more than an odd 1-2% which would be explained away by particular configurations.

That's fair - I guess as a software dev who has lived in Windows land for two decades, and worked pretty closely with various IT/devops and vendors as a result, I have a pretty good appreciation for how difficult and painful it is to support machines from both sides of the fence. So probably I'm being a little lenient on Crowdstrike here - but that's also because we don't have enough information at this time to know what % of their install base has been affected by this.

If Linux someday magically becomes the lead desktop OS we can expect the same scale of security issues and threats as Windows.

I would honestly love for that day to come, because most of the "Windows bad" people would suddenly shut the fuck up forever. Imagine an internet where discussions about Windows aren't guaranteed to attract at least one post telling you that everything would be fine if you just used Linux! The dream...

 

[chrcoluk]

This crisis is showing up all the companies that have inadequate rollout processes in place.

 

[Easo]

Suggestion to TP - when your headline starts with Windows Update then geniuses will read it as Microsoft's fault, not whatever comes after.
Suggestion to others - the hate boner for Microsoft is ridiculous here, as well as lack of understanding how things work.

 

[R-T-B]

So you feel MS is in no way to blame?

No.

arnt they the ones who have a contract with this firm?

No.

is it not up to MS to check and verify this crap before letting it through?

Again, no.

This is a third party people. There's a lot of issues to discuss and blame to dole out but none of them involve MS this time around.

 

[R-T-B]

Please keep the misinformation about this at home. If you can't factually verify it beyond gut feels and youtube vids, its misinformation.

 

[Mindweaver]

Has anyone in this thread been affected or know someone?

Crowdstrikes workaround

Workaround Steps for individual hosts:

1. Boot Windows into Safe Mode or the Windows Recovery Environment.
2. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally.

 

[R-T-B]

It’s the same logic as with the “Linux is totally more secure, guys” arguments. While yeah, there are some advantages to the way it handles security, the main reason it’s “more secure” is just the sheer fact that nobody cares there is a significantly smaller consumer install base. If Linux someday magically becomes the lead desktop OS we can expect the same scale of security issues and threats as Windows.

Lolwut. Linux is way more used in serverland and probably has an overall larger and even more importantly, more valuable installbase than windows.

 

[HTC]

Has anyone in this thread been affected or know someone?

That i'm aware of, no.

 

[mab1376]

Has anyone in this thread been affected or know someone?

Crowdstrikes workaround

Workaround Steps for individual hosts:

1. Boot Windows into Safe Mode or the Windows Recovery Environment.
2. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally.

yes, this worked on my workstation this morning, had to get my Bitlocker key from my admin team.

 

[Mindweaver]

I would just load into Windows Recovery then select Terminal. Then copy and paste below.

del %WINDIR%\System32\drivers\CrowdStrike\C-00000291*.sys

 

[Onasi]

@R-T-B
Come on, man, I specifically said “smaller CONSUMER install base”. I was talking in this context and this context alone. I am well aware of Linux server market share.

 

[Neo_Morpheus]

Has anyone in this thread been affected or know someone?

Crowdstrikes workaround

Workaround Steps for individual hosts:

1. Boot Windows into Safe Mode or the Windows Recovery Environment.
2. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally.

As mentioned before, it's nice, but it must be done by the user and in a bitlocker enabled machine in an AD environment, the recovery key is in a Windows server that might also be unaccessible.

Not to mention, a pain in the ass to grab and enter manually in each system.

I think it will be quicker but more painful due to possible data loss, to simply re image.

 

[Konceptz]

Since when did Crowdstrike deploy Windows updates? The headline needs to be changed, a Faulty Crowdstrike update, is breaking Windows machines.

 

[64K]

Suggestion to TP - when your headline starts with Windows Update then geniuses will read it as Microsoft's fault, not whatever comes after.
Suggestion to others - the hate boner for Microsoft is ridiculous here, as well as lack of understanding how things work.

That's where the train jumps the track and crashes. Of course this particular instance isn't MS fault but you have the MS apologists coming in and starting the name calling and making wild claims that MS shouldn't be held accountable when they are at fault (once again not saying this time they are) because of the complexities involved in taking care of the hundreds of millions of customers that they have made and are still making a damn fortune from. Bottom line, can we really claim that MS has a proper QA dept without busting out laughing?

 

[Mindweaver]

As mentioned before, it's nice, but it must be done by the user and in a bitlocker enabled machine in an AD environment, the recovery key is in a Windows server that might also be unaccessible.

Not to mention, a pain in the ass to grab and enter manually in each system.

I think it will be quicker but more painful due to possible data loss, to simply re image.

I use AzureAD daily I am aware of how the recovery keys work and agree it will be a pain having to manually key in the keys. A quick solution is having a Zebra usb scanner. Then you can load the recovery keys in a spread sheet then set the keys to a free 3of9. It won't be an easy task but do-able.

 

[Klemc]

Microsoft will become "rich again", bc it's not their fault but thecrowdstrike's will be courted and have to pay (aworld level refund), right ?

 

[Neo_Morpheus]

Saw an announcement that NYC mayor was going to do a press conference about the Microsoft outage.

Such headlines are everywhere.

I dont know if CS will survive after this.

I use AzureAD daily I am aware of how the recovery keys work and agree it will be a pain having to manually key in the keys. A quick solution is having a Zebra usb scanner. Then you can load the recovery keys in a spread sheet then set the keys to a free 3of9. It won't be an easy task but do-able.

Interesting option.

On another note, Poor remote users without local admin privileges.

 

[R-T-B]

That i'm aware of, no.

Someone absolutely has, albeit maybe not directly. We've been impacted here by logmein services being down.