Windows 10 Process-Termination Bug Slows Down Mighty 24-Core System to a Crawl

[trparky]

I dont like 7 either

Please don't tell me you like Windows XP.

 

[Athlonite]

Please don't tell me you like Windows XP.

HaHa LOL he probably still uses DOS 6.22 and Windows 3.11

 

[springs113]

Maybe I just failed to see the sarcasm from a guy with an Intel CPU.

Or I woke up grumpy because my internet is only now back after comcast had me netless for almost a week. Take your pick.



Yes. It's not even broken in 7.

Ouch i feel your pain. I have crapcast too, they have some lazy shards working for them, it's like playing Russian roulette to find a decent rep that's knowledgeable and also one that is coherent at the same time. Charge that one to the house i guess. Him having that Intel CPU is his counter for not being labeled a fanboy 'ya know ehehe.

 

[RejZoR]

That's what happens when your browser shits gazillion processes for every little shit in a browser...

 

[dj-electric]

Windows sucks... The only reason i use this shitty OS is to game.

 

[newtekie1]

The thing that bugs me most about his entire Blog post is he never mentions what version of Windows 10 he uses.

Is he using the Creators Update? Is he still on the original Windows 10 release? Has this already been fixed? We don't know, because he didn't tell us what build of Windows 10 he is on.

And I'd expect a developer to know about the different builds of Windows 10...

 

[OneMoar]

anybody running a outdated version of windows: deserves to have there pc soaked in gasoline lit on fire and then thrown off a tall building

this is a edge case bug thats only a issue when you have 40 fucking cores and need to terminate 30 processes
and I bet you windows server doesn't have the issue

 

[FordGT90Concept]

Only time there's mass process termination in a server environment is before shutting down.

 

[OneMoar]

Only time there's mass process termination in a server environment is before shutting down.

which is why i said I bet windows server doesn't have the issue

this is just a case of somebody using a desktop-os on a server and then wondering why stuff is broke

 

[rtwjunkie]

anybody running a outdated version of windows: deserves to have there pc soaked in gasoline lit on fire and then thrown off a tall building

I'm gonna say what everyone else who see that and is thinking but is afraid to say:
WTF is WRONG with you?

 

[OneMoar]

I'm gonna say what everyone else who see that and is thinking but is afraid to say:
WTF is WRONG with you?

_EVERYTHING_
but thats neither here nor there if you need a example of the damage a outdated os can do Cough windows 7 and wannacry cough cough

 

[trparky]

I wrote a quick and dirty console program in .NET that spawns 1000 child processes that simply sleeps and does nothing (an infinite loop) and then proceeds to kill said child processes. Getting Windows to respond to just about anything while it was killing the child processes was difficult at best. This is a fully updated version of Windows 10, Build 15063.483. So yeah, this seems like an issue at the Windows kernel level.

 

[FordGT90Concept]

Try one process with a 1000 threads. You know, like any sane program would be. Bet there isn't a problem, yet the amount of work that can be accomplished is more or less the same.

 

[OneMoar]

I wrote a quick and dirty console program in .NET that spawns 1000 child processes that simply sleeps and does nothing (an infinite loop) and then proceeds to kill said child processes. Getting Windows to respond to just about anything while it was killing the child processes was difficult at best. This is a fully updated version of Windows 10, Build 15063.483. So yeah, this seems like an issue at the Windows kernel level.

try it on windows server

 

[trparky]

I don't have a Windows Server laying around. If you have one I can give you the source code for the console app I wrote.

 

[Ferrum Master]

I wrote a quick and dirty console program in .NET that spawns 1000 child processes that simply sleeps and does nothing (an infinite loop) and then proceeds to kill said child processes. Getting Windows to respond to just about anything while it was killing the child processes was difficult at best. This is a fully updated version of Windows 10, Build 15063.483. So yeah, this seems like an issue at the Windows kernel level.

Try to send it, I'll test on recent insiders.

 

[FordGT90Concept]

I don't have a Windows Server laying around. If you have one I can give you the source code for the console app I wrote.

Hit me with it. I'll see about modifying it to multithreaded too.


I think what it's going to boil down to is that it's like having 1000 files on a file system versus having 1000 files archived into one library. It isn't abundantly obvious but those 1000 files on the file system have 4 MiB of overhead and are generally slow to access because the HDD/SDD has to read from the file system to find out where the data is then move to the file and read it. If they're all in one library, you're moving a pointer about one single file that the file system already looked up. Access time is much faster because the overhead is exponentially less.

Processes are the same way. The process has to keep record of all of its memory use. Threads can quickly request and release memory because there isn't much in the way of overhead (e.g. security/NXbit/etc.). Processes need to not only get permission to execute, they have to load all of their modules (10-30 is not uncommon). All of that needs to be unloaded from the memory while the process winds down.

 

[trparky]

Here it is, it's quick and dirty.

 

[FordGT90Concept]

Windows 10:

Windows Server 2012 R2:

 

[trparky]

I updated the code to included a threaded example where it creates 1000 child threads as part of the main executable process. The system does not suffer the same performance penalty that the system endures when killing 1000 child processes. Killing threads is not nearly as tasking on the system as killing whole processes.

 

[FordGT90Concept]

Here's my own threads program on the same Windows 10 machine above using Thread.Abort():

Here's the same program but instead using _Stop = true (peaceful termination):

using System;
using System.Threading;
namespace LotsOThreads
{
    class Program
    {
        private static Thread[] _Threads = new Thread[1000];
        private static volatile bool _Stop = false;
        static void Main(string[] args)
        {
            Console.WriteLine("Start threads...");
            for (int i = 0; i < _Threads.Length; i++)
            {
                _Threads[i] = new Thread(Worker);
                _Threads[i].Start();
            }
            Console.WriteLine("Waiting a second...");
            Thread.Sleep(1000);
            Console.WriteLine("Killing threads...");
            //for (int i = 0; i < _Threads.Length; i++)  // painful
            //     _Threads[i].Abort();
            _Stop = true;  // peaceful
            Console.WriteLine("Done.");
        }
        private static void Worker()
        {
            while (!_Stop)
                Thread.Sleep(1000);
        }
    }
}

As I said, whomever wrote gomacc.exe is an idiot.

Like one big file is better than lots of small files, one big process with lots of threads is better than lots of processes with one thread. The former in both cases creates a lot of hidden overhead.


To recap: left is bad (spam processes), right is good (spam threads)...

 

[trparky]

Killing a thread is more "violent" than a graceful exit which is what we would like to simulate since killing a process is what causes this issue to begin with.

 

[FordGT90Concept]

Actually, the best way to simulate thread killing is Process.Kill.

Process.GetCurrentProcess().Kill() = instant close, no hitch
Environment.Exit(0) = instant close, no hitch
_Stop = true = moderate close, no hitch
Thread.Abort() = slow close, no hitch

 

[OneMoar]

has nothing todo with memory anyway the problem is that termination is serial not parallel
has nothing todo with memory-management or disk i/o
from the original post:

Well, what do you know. Process creation is CPU bound, as it should be. Process shutdown, however, is CPU bound at the beginning and the end, but there is a long period in the middle (about a second) where it is serialized – using just one of the eight hyperthreads on the system, as 1,000 processes fight over a single lock inside of NtGdiCloseProcess. This is a serious problem. This period represents a time when programs will hang and mouse movements will hitch – and sometimes this serialized period is several seconds longer.

Threads can't be killed though, only aborted. Aborted threads are still gracefully unwound by the parent process.

the fuck they can't
thats exactly the issue thread killing is getting stuck in a lock as NtGdiCloseProcess can only work though one thread at a time and its hanging

the process needs to cooperate yes but you can most certainly force-term a thread

 

[FordGT90Concept]

has nothing todo with memory anyway the problem is that termination is serial not parallel
has nothing todo with memory-management or disk i/o
from the original post:

Well, what do you know. Process creation is CPU bound, as it should be. Process shutdown, however, is CPU bound at the beginning and the end, but there is a long period in the middle (about a second) where it is serialized – using just one of the eight hyperthreads on the system, as 1,000 processes fight over a single lock inside of NtGdiCloseProcess. This is a serious problem. This period represents a time when programs will hang and mouse movements will hitch – and sometimes this serialized period is several seconds longer.

It's memory. CPU usage is all over the place, memory is a straight ramp. The hitch/hang occurs when freeing all the memory that was just consumed by 1000 processes.


Hmm, out of curiousity, I'm going to try killing a single processes that uses over 10 GiB of memory...

Edit: No perceived hitch:

That is one process consuming over 12 GiB of RAM (probably some page file too), switched over to the Proccesses tab, end task, then switched back to performance and watched it unwind while moving the mouse in a circle. I never saw it stop.