Friday, July 14th 2017
Windows 10 Process-Termination Bug Slows Down Mighty 24-Core System to a Crawl
So, you work for Google. Awesome, right? Yeah. You know what else is awesome? Your 24-Core, 48-thread Intel build system with 64 GBs of ram and a nice SSD. Life is good man. So, you've done your code work for the day on Chrome, because that's what you do, remember? (Yeah, that's right, it's awesome). Before you go off to collect your google-check, you click "compile" and expect a speedy result from your wicked fast system.
Only you don't get it... Instead, your system comes grinding to a lurching halt, and mouse movement becomes difficult. Fighting against what appears to be an impending system crash, you hit your trusty "CTRL-ALT-DELETE" and bring up task manager... to find only 50% CPU/RAM utilization. Why then, was everything stopping?
If you would throw up your arms and walk out of the office, this is why you don't work for Google. For Google programmer Bruce Dawson, there was only one logical way to handle this: "So I did what I always do - I grabbed an ETW trace and analyzed it. The result was the discovery of a serious process-destruction performance bug in Windows 10."
This is an excerpt from a long, detailed blog post by Bruce titled "24-core CPU and I can't move my mouse" on his Wordpress blog randomascii. In it, he details a serious new bug that is only present in Windows 10 (not other versions). Process destruction appears to be serialized.
What does that mean, exactly? It means when a process "dies" or closes, it must go through a single thread to handle this. In this critical part of the OS which every process must eventually partake in, Windows 10 is actually single threaded.
To be fair, this is not a normal issue an end user would encounter. But developers often spawn lots of processes and close them just as often. They use high-end multi-core CPUs to speed this along. Bruce notes that in his case, his 24-core CPU only made things worse, as it actually caused the build process to spawn more build processes, and thus, even more had to close. And because they all go through the same single threaded queue, the OS grinds to a halt during this operation, and performance peak is never realized.
As for whether this is a big bug if you aren't a developer: Well that's up for debate. Certainly not directly, I'd wager, but as a former user of OS/2 and witness to Microsoft's campaign against it back in the day, I can't help but be reminded of Microsoft FUD surrounding OS/2's SIQ issue that persisted even years after it had been fixed. Does this not feel somewhat like sweet, sweet karma for MS from my perspective? Maybe, but honestly, that doesn't help anyone.
Hopefully a fix will be out soon, and unlike the OS/2 days, the memory of this bug will be short lived.
Source:
randomascii Wordpress Blog
Only you don't get it... Instead, your system comes grinding to a lurching halt, and mouse movement becomes difficult. Fighting against what appears to be an impending system crash, you hit your trusty "CTRL-ALT-DELETE" and bring up task manager... to find only 50% CPU/RAM utilization. Why then, was everything stopping?
If you would throw up your arms and walk out of the office, this is why you don't work for Google. For Google programmer Bruce Dawson, there was only one logical way to handle this: "So I did what I always do - I grabbed an ETW trace and analyzed it. The result was the discovery of a serious process-destruction performance bug in Windows 10."
What does that mean, exactly? It means when a process "dies" or closes, it must go through a single thread to handle this. In this critical part of the OS which every process must eventually partake in, Windows 10 is actually single threaded.
To be fair, this is not a normal issue an end user would encounter. But developers often spawn lots of processes and close them just as often. They use high-end multi-core CPUs to speed this along. Bruce notes that in his case, his 24-core CPU only made things worse, as it actually caused the build process to spawn more build processes, and thus, even more had to close. And because they all go through the same single threaded queue, the OS grinds to a halt during this operation, and performance peak is never realized.
As for whether this is a big bug if you aren't a developer: Well that's up for debate. Certainly not directly, I'd wager, but as a former user of OS/2 and witness to Microsoft's campaign against it back in the day, I can't help but be reminded of Microsoft FUD surrounding OS/2's SIQ issue that persisted even years after it had been fixed. Does this not feel somewhat like sweet, sweet karma for MS from my perspective? Maybe, but honestly, that doesn't help anyone.
Hopefully a fix will be out soon, and unlike the OS/2 days, the memory of this bug will be short lived.
107 Comments on Windows 10 Process-Termination Bug Slows Down Mighty 24-Core System to a Crawl
Also you would think they should be using Linux for their work.
Or I woke up grumpy because my internet is only now back after comcast had me netless for almost a week. Take your pick. ;)Yes. It's not even broken in 7.
Plus...use Win7....Win 10 blows on so many level...another shocker.
I have 124 proccesses and 2308 threads......
I have seen 200 processes on my PC in the past. Main PC down so I am on server waiting for RMA of MB so server doesnt have all my extra programs installed.
Depending on why the locks are being used, it might not be possible to remove them...or cause a larger performance problem elsewhere as a queue backs up.
Garbage collection/process closure is likely reflected by the amount of memory the process uses. He says he's using less than half of 64 GiB of RAM. If his compiler software is using, say 20-30 GiB of RAM and it doesn't properly scrub itself as it goes, it's very possible that the operating system has to find 20-30 GiB of memory it is using and reallocate it to free.
They claim to have reported it to Microsoft. Author should have waited for a response before going to the internet. Wouldn't be surprised if Microsoft points out a flaw in the program Google needs to fix and doesn't update Windows at all.
Note: a single lock can cause a cascade of higher level threads to wait. Example, if you have 100 threads that read a single locked variable in another thread, those 100 threads are halted until the lock clears. Application is still multithreaded; it just doesn't act like it.
gomacc.exe could easily be locking a variable that Windows needs access to in order to close the process.