# trying the near impossible... most powerful GOD BOX in a miniITX case



## venturi (Feb 16, 2019)

*I wasn't even sure where to post it as  I have some questions that don't fall into any particular category.*


*I decided to ask the question: what is the most powerful build I can do and keep it tiny, and run of a normal household socket.*

*This was my first list, and I stuck to it:*

*Asus c621e sage
Dual Xeon 8180M
384GB of ram
4x Titan V including Titan V CEO ed 32GB
960pro nvme (OS)
10x SSD RAID (Apps and Backup 8x 2x )
1600W PSU (digital) noiseless
MS 2016 Data Center, Ubuntu
TT miniITX case*


*reverse engineering the design to fit in a small enough case proved a bit challenging:*

*The case had to be deconstructed and rebuilt better, reinforced and be given new hard points.
Wires and connectors had to be remade by hand to do the low profile turns.
Cable management had to be ZEN like.
It would have to run cool all by itself and be rock stable.*

*so after a week of getting parts in and building the toy (after the daytime job as to be able to afford this little hobby)*

*well, here is the mostly complete and fully functional final result:*






*It looks harmless enough, but it was a lot of work,*





*The drives had to be RAIDED and stacked in their own cooling tower*





*The 4x Titan V required decent power that had to be shared with the dual 8180M xeons*

*those required a little order and management*









*while I was building it, with all its guts (wires) hanging out, I ran a quick cinebench to see how it would do*

score: *8857*

*

*

*temps seem reasonable under light load, temps rise about 10-15 'F for cpus under heavy load (up to about 40-41'C)
video cards can get as hot under load as 68'C*

*

*




*so it runs fast and stable, games look amazing and are smooth at 4k+, not to mention the little bit of work I use it for (deep learning, clinical decision support, isotopes, medical imaging, Lara Croft, etc)*


*SO I have some questions that answers seem to exist NOWHERE on the net.*

*Questions from bios*

*1. what is "Asus Engine Booster"? (no, its not the asus OC tune setting)*

*2. I like using the latest bios, but the prior bios without the microcode mitigation is a faster machine, is there a way to edit out a microcode mitigation in the bios and keep the latest one?*

*Question on Video cards.*

*1. I like using the lowest card on the stack as primary. This is so because it has the most amount of cool air and I don't hit any thermal limits (I do run mgpu, and I am currently trying to get auto diff sli to work). The question is: how do I get the bios and post screen to come up if the board want to make any card but this one a primary card?  I do have a work around for now: I havea DP cable in there and when windows loads it runs fine, I also have an HDMI cable in the 4th card for bios and post so with one monitor I can have both bios, post , and run state. I would like to avoid having to use the HDMI cable -seems not elegant.*







*I really need to tidy those wires up....*


*Anyhow, any help on the questions would be greatly appreciated.*

*Grazie*


*V*


----------



## Flyordie (Feb 16, 2019)

Just here to admire the hardware.  lol.  Not gonna lie, its a damn awesome build. :-o


----------



## venturi (Feb 17, 2019)

Thank you, hopefully I'll get enough time off from work to actually play on it


----------



## lexluthermiester (Feb 17, 2019)

So love that case. Why is it on the floor though? I'd move the speakers to the sides of the display and put the case on the left side of the desktop to be displayed with pride! Not to mention keeping away from dust that collects on the floor.


----------



## Mescalamba (Feb 17, 2019)

This would belong to XtremeSystems for others to admire, if it was still alive.

Quite insane machine.

Would probably try to watercool it with external big radiator. But ofc it would be bigger. Awesome nonetheless. And insanely expensive..


----------



## Mussels (Feb 17, 2019)

And i thought i had a powerful ITX rig :/

How do the titans handle being so closely stacked together heat-wise?


----------



## lexluthermiester (Feb 17, 2019)

Mussels said:


> How do the titans handle being so closely stacked together heat-wise?


While I can't speak for the OP, being in an open-air case like it is likely gives plenty of room to breath and thus heat may not be a problem. That's my hypothesis.


----------



## venturi (Feb 17, 2019)

Mussels said:


> And i thought i had a powerful ITX rig :/
> 
> How do the titans handle being so closely stacked together heat-wise?



The titans, in the room at 21-22C ambient, run idle at about 29'C.  Under load they can get as high as 68'C (on occasion even 71'C)

I removed the back plate to the titans and this allowed more room and heat dissipation. The back plate has a plastic layer under the metal plate and the whole thing is more of an insulator -so I got rid of it. Cards are very happy. So I don't really experience any thermal throttling, thermal throttling (t would occurs between 85'-91'C)

I have attached the temps on medium -light load







lexluthermiester said:


> So love that case. Why is it on the floor though? I'd move the speakers to the sides of the display and put the case on the left side of the desktop to be displayed with pride! Not to mention keeping away from dust that collects on the floor.



I was going to move it to the top of the desk, just getting organized now. For its size, the server can fit easily on either side of the monitor.


----------



## steen (Feb 17, 2019)

First, congrats on the build. Does the c621e bios allow PCIEx16 n slot boot priority? I assume not. Examine the bios & see if there is a hidden option. I'm not suggesting to flash an edit, but send a request through to Asus tech support. Re: microcode edit - yes you can. I wouldn't. It is possible (although I haven't looked at the latest Xeon bioses) that they're encrypted. You'll need the USB flash back feature, etc. You also need to consider the version of microcode in the latest BIOS as well as that bootstrapped by the OSes you're running. You may not achieve much unless you disable OS mitigations (which I don't recommend). Titan V CEO do very well on some data sets, much better than my lowly Quadro.


----------



## venturi (Feb 17, 2019)

steen said:


> First, congrats on the build. Does the c621e bios allow PCIEx16 n slot boot priority? I assume not. Examine the bios & see if there is a hidden option. I'm not suggesting to flash an edit, but send a request through to Asus tech support. Re: microcode edit - yes you can. I wouldn't. It is possible (although I haven't looked at the latest Xeon bioses) that they're encrypted. You'll need the USB flash back feature, etc. You also need to consider the version of microcode in the latest BIOS as well as that bootstrapped by the OSes you're running. You may not achieve much unless you disable OS mitigations (which I don't recommend). Titan V CEO do very well on some data sets, much better than my lowly Quadro.



I  did not see a priority or sequence setting in the bios for the pcie slots.omly optometrist enable ir dissable.
Disabling prom but for all except the one I want yielded lots of beeping and no post, so I reverted back.

Unfortunately the microcode mitigation’s hurt performance, and I’d rather not have them if they are a negative on perf. I disabled the os mitigation’s by adding ‘3’ to the dword in the feature section in reg/meme I’m HKLM and that helped substantially on raid performance and video performance


----------



## FordGT90Concept (Feb 17, 2019)

venturi said:


> 1. what is "Asus Engine Booster"? (no, its not the asus OC tune setting)


Might be ASUS branding for Intel Turbo Boost.  Was it enabled by default?  If yes, I assume they are one in the same.



venturi said:


> 2. I like using the latest bios, but the prior bios without the microcode mitigation is a faster machine, is there a way to edit out a microcode mitigation in the bios and keep the latest one?


You could find the most recent BIOS for the board that don't have the specter/meltdown fixes.  The board is fairly old so most of the non-CPU microcode fixes should have been implemented in earlier BIOS revisions.  The board is roughly a year old.  The BIOS may have had the specter/meltdown fixes baked in from the first revision.  I'd just leave well enough alone.



venturi said:


> 1. I like using the lowest card on the stack as primary. This is so because it has the most amount of cool air and I don't hit any thermal limits (I do run mgpu, and I am currently trying to get auto diff sli to work). The question is: how do I get the bios and post screen to come up if the board want to make any card but this one a primary card?  I do have a work around for now: I havea DP cable in there and when windows loads it runs fine, I also have an HDMI cable in the 4th card for bios and post so with one monitor I can have both bios, post , and run state. I would like to avoid having to use the HDMI cable -seems not elegant.


When a computer boots, it enumerates displays and connected monitors.  The first card with a connected display gets flagged as primary until the driver loads.  If all of your displays are plugged into the bottom card, theoretically it should always be considered the primary card.  The other cards should be in an idle power state unless SLI give them something to do.

Alternatively, the BIOS may have an option for selecting the first GPU to boot.  Usually that choice only enumerates IGP (integrated) and PEG (PCI Express Graphics).  Maybe your board lists all of the PCIE slots (PEG1, PEG2, PEG3, etc.) so you can tell it which specific card to load.


----------



## FreedomEclipse (Feb 17, 2019)

yeah.

If you ever just want to get rid of this thing for free. Im willing to pay for shipping. Its the least i could do for a good *good friend *of mine*. *


----------



## venturi (Feb 17, 2019)

FordGT90Concept said:


> Might be ASUS branding for Intel Turbo Boost.  Was it enabled by default?  If yes, I assume they are one in the same.
> 
> 
> When a computer boots, it enumerates displays and connected monitors.  The first card with a connected display gets flagged as primary until the driver loads.  If all of your displays are plugged into the bottom card, theoretically it should always be considered the primary card.  The other cards should be in an idle power state unless SLI give them something to do.
> ...




The intel turbo boost is in another section of the bios. This asus engine booster feature is disabled by default. If it’s enabled it gives a choice of level 1, level 2, and level 3.

It’s not the oc level either, that is in another section as well.



Asus made the video boot of the bios dependent on the amount of cards that are plugged into the board, alas, plugging a card in the card like,  does not yield video during post/ boot

The bios, unless it’s wording I don’t know, does not have a pcie/card selector. Hence my reaching out.


I called asus tech support, unfortunately I spent the first hour trying to prove to tech support that asus actually made the board and that it was their product... it went downhill from there...



FreedomEclipse said:


> yeah.
> 
> If you ever just want to get rid of this thing for free. Im willing to pay for shipping. Its the least i could do for a good *good friend *of mine*. *



Ok, but there might be a slight handling charge to go with the shipping cost....


----------



## FordGT90Concept (Feb 17, 2019)

venturi said:


> The intel turbo boost is in another section of the bios. This asus engine booster feature is disabled by default. If it’s enabled it gives a choice of level 1, level 2, and level 3.
> 
> It’s not the oc level either, that is in another section as well.


I looked through all of the documentation I could find including manual and BIOS updates and there are no references to it.



venturi said:


> Asus made the video boot of the bios dependent on the amount of cards that are plugged into the board, alas, plugging a card in the card like,  does not yield video during post/ boot
> 
> The bios, unless it’s wording I don’t know, does not have a pcie/card selector. Hence my reaching out.


That's pretty stupid.  Your work around is probably the best you're going to get then, sadly.



venturi said:


> I called asus tech support, unfortunately I spent the first hour trying to prove to tech support that asus actually made the board and that it was their product... it went downhill from there...


That's terrible.  Maybe got that response because it's technically a workstation/business board.  Might be a separate department or tier 2 support that handles them.


----------



## blobster21 (Feb 17, 2019)

Congratulation for this godbox.

How much did you spend for the hardware ?


----------



## XXL_AI (Feb 17, 2019)

blobster21 said:


> Congratulation for this godbox.
> 
> How much did you spend for the hardware ?


looking at the specs, it must be around tesla model 3 level.


----------



## venturi (Feb 17, 2019)

blobster21 said:


> Congratulation for this godbox.
> 
> How much did you spend for the hardware ?



Cost: pound of flesh and a pint of blood

Over the years, I've paid for this hobby.

over the course of 20 years, I've built god boxes along the way. At least, in their own time they were consider unique.

19 years ago, I was the first to water-cool a dual cpu rig, chipset and video card..  I used a motorcycle oil cooler for the radiator.

I also was the first to overclock a duallie.

15 years ago I was one of the first to QUAD Sli

I was the first to overclock/watercool video cads from voodoo, specter, stb etc and the obsidian x-24 dual gpu.

I was the first to overclock the 3dlabs willdcat card and 17 years ago had a VR headset working with gyros off. The wildcat card was the first integrated dual GPU.

A few years ago I was the one that implemented 4x quad SLI on pascal as soon as nvidia tried to disable the 3 and 4 way sli on pascal - I have it fully functional and with nice proportional scaling.



here are some of the pictures from those years:




yes, even the chipset on this duallie















yes, that was really first qua slI




dual PSU, cable management, quad sli and all 8 scsi drives are raided, circa mid 2000s

I took 6 years off as I worked on macs, windows vista made me leave the MS world




















P5




P3




P1 - latest




ending with where things are right now

I only build mutlti cpu/ multi gpu / with Raid  rigs



well, that was 20 years, enjoy


----------



## notb (Feb 17, 2019)

venturi said:


> I was going to move it to the top of the desk, just getting organized now. For its size, the server can fit easily on either side of the monitor.


Any noise and heat concerns?
It's basically a server in an open case.


----------



## Vario (Feb 17, 2019)

My budget, it would be near impossible.
With your budget, it is possible.  
I am estimating this machine cost around $40,000.


----------



## blobster21 (Feb 17, 2019)

XXL_AI said:


> looking at the specs, it must be around tesla model 3 level.



Yes i think you're spot on.


----------



## venturi (Feb 17, 2019)

notb said:


> Any noise and heat concerns?
> It's basically a server in an open case.



Runs silent, runs cool, no heat or noise issues.  please see earlier pic with temps.

Even the power supply fan seldom comes on, and that is only when the unit is maxed with all 4 gpus going.
Under load, the gpus do get louder, but its not bad. I have not heard the gpu blower fans max-up yet.

I also made a lot of sections on TIM, example, every gpu was dismantled and reassembled with better TIM and thermal pads that were much better than factory.

I made a lot of effort in selecting fans, fan blade, rpms, and positioning for cool and quiet.


----------



## Vario (Feb 17, 2019)

What heatsinks and fans are you running on those 28 core monsters?  Looks like Noiseblockers?  I wonder if the CPU would cool better on other heatsink orientations.  Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan.  Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.


----------



## venturi (Feb 17, 2019)

Vario said:


> What heat-sinks and fans are you running on those 28 core monsters?  Looks like Noiseblockers?  I wonder if the CPU would cool better on other heat-sink orientations.  Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan.  Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.




Orientation is limited. The fans are in pull mode, not push. The air is pulled up and out the top. There wasn't enough room for the fans in push mode. However, I tested it with one less video card in place in push mode and found that pull mode is 1-2 'C cooler (in PULL mode). So the orientation and pull mode ended up being the most efficient.

My primary card is on the bottom, when all 4 gpus are going full the pull mode does not acquire a waft of warm air, instead, all 4 blower gpus push warm ait out the back pulling cool air in. The result is that the chipset and surrounding area are receiving cooler air. Again, tested both ways. But I will be exploring alternative alignments.  In push mode (down) the 4th card was being warmed up by the 2 cpu coolers and it was fighting the natural rise of heat.

The fans are noise blocker, 1800rpm, PWM.  The heatsinks are Noctua NH-D9 DX-3647 4U

my preferred cpu cooler/fan orientation is push mode from right to left - but the market is sparce and the noctua 92mm is only SILENT configuration i can find. There are Supermicro/Dynatron coolers that will fit but sound like a hair drier compared to the noctuas

please see temp chart under medium load in a 73'F room:


----------



## notb (Feb 17, 2019)

venturi said:


> Runs silent, runs cool, no heat or noise issues.  please see earlier pic with temps.


Misunderstanding. I didn't mean temperature of the components. I really meant heat. 
At full load you'll be near that 1600W the PSU delivers. Essentially, it's like a small heater, isn't it?

As for the noise: Titans are one of the loudest cards around. I had the "pleasure" of working with them lately. Just a single card in a Dell tower and I had to use headphones to concentrate on math. But good luck. ;-)


Vario said:


> I wonder if the CPU would cool better on other heatsink orientations.  Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan.  Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.


It's more efficient to push air into the heatsink than pull from. I bet that's what he's doing.
Similarly, it's better to get air through CPU first and onto GPUs. GPUs are usually operating far from thermal limits and they can take it.
That's also how servers work (disks -> CPUs -> GPUs).


----------



## XXL_AI (Feb 18, 2019)

Titan V at full load can consume up to 300W when it peaks, 4 of them so it is 1200W for the GPU, each CPU is rated for 205W but we know intel always lies so count it as 250W, 500W for CPU, I didn't even count the ram sticks and ssds here and it is already reached to 1700W peak, you have 1600W, you'll have PSU issues if you want to run deep learning/machine learning applications at the long run. 

but, since you are running windows and not using all the capability of the machine for running deep learning and only have presumably a single 4K screen you will never going to utilize all the gpus with gaming, even if you push all the limits of the hardware you have you'll stay under 1600W but not much, it'd better to have dual PSU or something at the realm of 2000W at least.

there are tons of tweaks on the nvidia hardware, so if we take those into the consideration, when you want to train a dataset you can halve the gpu power and save up to 700W from GPUs, since dataset training uses barely any cpu, we can add 100W, and another 200W for ram and storage, in the minimum consumption you'll again hit ~1000W it is a small heater and you'll need active airflow on the motherboard which I don't see on your photos. servergrade motherboards rely on active airflow and horizontal usage, you'll damage $30000 build no more than 2 years with active use.

if the room is 73F, how the hell SSD's running at 68F?


----------



## xkm1948 (Feb 18, 2019)

384gb ram

384gb ram

384gb ram
384gb ram

384gb ram

384gb ram
384gb ram

384gb ram

384gb ram
384gb ram

384gb ram

384gb ram
384gb ram

384gb ram

384gb ram

OH MY F**ING GOD


----------



## venturi (Feb 18, 2019)

XXL_AI said:


> Titan V at full load can consume up to 300W when it peaks, 4 of them so it is 1200W for the GPU, each CPU is rated for 205W but we know intel always lies so count it as 250W, 500W for CPU, I didn't even count the ram sticks and ssds here and it is already reached to 1700W peak, you have 1600W, you'll have PSU issues if you want to run deep learning/machine learning applications at the long run.
> 
> but, since you are running windows and not using all the capability of the machine for running deep learning and only have presumably a single 4K screen you will never going to utilize all the gpus with gaming, even if you push all the limits of the hardware you have you'll stay under 1600W but not much, it'd better to have dual PSU or something at the realm of 2000W at least.
> 
> ...




It seems to be working and measurements seem cool. The ssd a 68'F. Well, not all components are equal to ambient. A glass of water in a 90'F room isn't necessarily 90'F itself. Maybe drive activity had been low for a while...heck I don't know. I use AIDA for temperature checking. When I touch the SSD stack it feels cool to the touch so the temps are in line. (I just touched it again)


CPU heat sinks are at best lukewarm to the touch. So I'm pretty sure its ok. Bios temp reporting is close to aida reporting as well. So everything seems lined up from two separate tools and touching it.
Seem to be running cool and happy.


I am doing some deep learning in clinical decision support on it and MLP with diagnostic imaging. So far it got the cards warm, but the fans only spun up a little on the gpus. I also use it for 3d reconstruction and so far that seems to run ok.

Thank you, I will look closely in the future and keep an eye out for the issue you enumerated/



xkm1948 said:


> 384gb ram
> 
> 384gb ram
> 
> ...




my prior build had 1TB of ram, I feel like I'm taking a step backwards....


oh, and I love your signature!


----------



## notb (Feb 18, 2019)

venturi said:


> A glass of water in a 90'F room isn't necessarily 90'F itself.


Because you just took it out from the fridge?


> CPU heat sinks are at best lukewarm to the touch. So I'm pretty sure its ok. Bios temp reporting is close to aida reporting as well. So everything seems lined up from two separate tools and touching it.
> Seem to be running cool and happy.


You could build a small house for the price of this server. I'm sure you can afford a basic pyrometer - they're like $20-40 (and $100-200 for models used in labs).


----------



## Wavetrex (Feb 18, 2019)

Absolute total jaw drop.
Needing a vacuum cleaner to suck all the pieces that were originally my jaw.

@venturi You sir have won the #1 Platinum prize for the most ub3r-1337 g33k in the world.


----------



## MrGenius (Feb 18, 2019)

Not impressed.....................................….j/k! 

Obligatory "Yeah...but will it run Crysis?"


----------



## XXL_AI (Feb 18, 2019)

venturi said:


> It seems to be working and measurements seem cool. The ssd a 68'F. Well, not all components are equal to ambient. A glass of water in a 90'F room isn't necessarily 90'F itself. Maybe drive activity had been low for a while...heck I don't know. I use AIDA for temperature checking. When I touch the SSD stack it feels cool to the touch so the temps are in line. (I just touched it again)
> 
> 
> CPU heat sinks are at best lukewarm to the touch. So I'm pretty sure its ok. Bios temp reporting is close to aida reporting as well. So everything seems lined up from two separate tools and touching it.
> ...



basic physics.
https://van.physics.illinois.edu/qa...t=why-do-metals-feel-cold-or-hot-to-the-touch


----------



## lexluthermiester (Feb 18, 2019)

venturi said:


> my prior build had 1TB of ram, I feel like I'm taking a step backwards....


Good Grief! I have 24GB of ram in my current system(which has only once come close to maxing out) and can put up to 96GB in it. I have no idea what I would even do with 384GB of ram, let alone 1TB.



MrGenius said:


> Obligatory "Yeah...but will it run Crysis?"


Likely several instances of it given the CPU/RAM/GPU resources.


----------



## venturi (Feb 18, 2019)

I apologize for not knowing the physics. Maybe near the floor the air is slightly cooler than higher up in the room. Maybe the unit was still cool from our house night time temps, we like it cold.

either way, the ssd stack is cool to the touch. I think aida is relatively accurate in reporting what the various sensors have. Regardless, the unit seems to be running no more than lukewarm. 



lexluthermiester said:


> Good Grief! I have 24GB of ram in my current system(which has only once come close to maxing out) and can put up to 96GB in it. I have no idea what I would even do with 384GB of ram, let alone 1TB.



have a medical application / db than when I run it on my PC and run a client to talk to it - I've seen usage as high at 210GB of ram. I honestly have not seen more than that from a single application.  I have ran multiple simultaneous VM's and that has encroached into high ram usage.

Example: host an EHR with 4 oracle DB, 10-15 client instances and then run a test for a software /DB update while running an end of day billing reports system - -> ram almost used up.



yes, I do on occasion take my work home with me. Its my personal rig, but work always spills over to home.


----------



## Wavetrex (Feb 18, 2019)

venturi said:


> have a medical application / db


Ah, that explains how you can afford this stuff.
Pharma is BIG.


----------



## FreedomEclipse (Feb 18, 2019)

Great -- so we have our own PharmaBro in the house.


----------



## 15th Warlock (Feb 18, 2019)

So much awesome in this thread! I tip my hat to you sir!

I love that case!


----------



## Durvelle27 (Feb 18, 2019)

Man I need a rig like this in my life.


----------



## notb (Feb 18, 2019)

lexluthermiester said:


> Good Grief! I have 24GB of ram in my current system(which has only once come close to maxing out) and can put up to 96GB in it. I have no idea what I would even do with 384GB of ram, let alone 1TB.


In-memory databases will suck as much as they can. Nothing special really. And if your RAM is much bigger than data on hard drives, they'll make additional indexes or rearranged copies to speed up queries.
With a traditional DB, where data is pulled from the drives, all operations are still done in RAM (and you can have a huge cache).


venturi said:


> either way, the ssd stack is cool to the touch. I think aida is relatively accurate in reporting what the various sensors have. Regardless, the unit seems to be running no more than lukewarm.


Yeah, buy a pyrometer, seriously. It'll change your life. 
There are many cheap brands on Amazon that I don't know.
Fluke is a good mid-range option known globally: $100, 1.5K accuracy.
https://www.amazon.com/Fluke-MAX-Thermometer-Contact-Degree-x/dp/B008EW837S/


----------



## Mussels (Feb 18, 2019)

FreedomEclipse said:


> Great -- so we have our own PharmaBro in the house.



we finally found out what big pharmas secret agenda is... honestly im okay with this.


----------



## FreedomEclipse (Feb 18, 2019)

Mussels said:


> we finally found out what big pharmas secret agenda is... honestly im okay with this.



Imagine how quick he could climb the F@H or WCG TPU charts if he put that machine into action


----------



## Vario (Feb 18, 2019)

The Antec Spot Cool would be helpful if you ever decide you need active cooling anywhere on the motherboard in the future.


----------



## phill (Feb 18, 2019)

There's nothing quite like a bit of overkill, I love it 



FreedomEclipse said:


> Imagine how quick he could climb the F@H or WCG TPU charts if he put that machine into action



I'd just cry at the power consumption   We could always ask @venturi to join our lovely WCG and FAH team


----------



## thebluebumblebee (Feb 18, 2019)

"uATX case.  Nah, I think I'll stuff an SSI EEB board in there."


----------



## venturi (Feb 18, 2019)

thebluebumblebee said:


> "uATX case.  Nah, I think I'll stuff an SSI EEB board in there."



..well... it wasn't easy...   and all KY in the world still required some modding


----------



## thebluebumblebee (Feb 18, 2019)

Ah, I see your problem.  KY is a water based lubricant.  You needed to use a silicone grease.  Would have made things much less painful.


----------



## lexluthermiester (Feb 18, 2019)

notb said:


> In-memory databases will suck as much as they can. Nothing special really. And if your RAM is much bigger than data on hard drives, they'll make additional indexes or rearranged copies to speed up queries.
> With a traditional DB, where data is pulled from the drives, all operations are still done in RAM (and you can have a huge cache).


Excellent points, and those useage models are easy to understand. There can't be too many home users out there with an active/running database though.



venturi said:


> ..well... it wasn't easy... and all KY in the world still required some modding





thebluebumblebee said:


> Ah, I see your problem. KY is a water based lubricant. You needed to use a silicone grease. Would have made things much less painful.


I think they were saying that tongue-in-cheek..


----------



## thebluebumblebee (Feb 18, 2019)

lexluthermiester said:


> I think they were saying that tongue-in-cheek..


And so was I.


----------



## notb (Feb 18, 2019)

lexluthermiester said:


> Excellent points, and those useage models are easy to understand. There can't be too many home users out there with an active/running database though.


Well, that depends. If you're a pro gamer, a surgeon or a journalist, than you're most likely not running databases at home.
If you're a data analyst or a programmer, you most likely do. 

I have around 200 "friends" on Facebook - 50 of which I met before university (random), 50 from studies and the rest since I started working.
I'd say 40-50 of 200 have "hobbies" that utilize DBs. By comparison, just 10-20 are regular PC gamers (at least that's how many admits ;-)). By "regular" I mean once a week.

And the possible use of huge RAM doesn't end on databases (or IT stuff in general).
Big renders and video editing will consume a lot as well. Which means you don't really have to know how RAM works (or even how much you have in your OEM-built workstation) to have huge needs.

Will I ever be able to afford 384GB RAM for my home PC? No, quite unlikely. Not unless we get an affordable alternative (like Optane).
Could I use it for my hobbies? Yeah, I can imagine that (even for what I was doing this weekend).


----------



## lexluthermiester (Feb 18, 2019)

notb said:


> Big renders and video editing will consume a lot as well.


I do video editing regularly and can tell you that unless the video work is 2160p+ ultra high bitrate it does not need anywhere near 384GB of ram, let alone 1TB.


----------



## notb (Feb 18, 2019)

lexluthermiester said:


> I do video editing regularly and can tell you that unless the video work is 2160p+ ultra high bitrate it does not need anywhere near 384GB of ram, let alone 1TB.


"unless the video work is 2160p"
Hello 2014?
Yeah, 4K can be dealt with using around 64GB RAM. You'll need double that for comfortable 8K editing.
And while you're at it, why don't you check the bitrate of raw 4K footage, e.g. ARRIRAW?


----------



## thebluebumblebee (Feb 18, 2019)

@lexluthermiester , @notb , remember that this is a 2P system.  That 384 GB of RAM is divided between the two processors so, effectively, they each _only_ have 192 GB.  oops


----------



## lexluthermiester (Feb 18, 2019)

thebluebumblebee said:


> @lexluthermiester , @notb , remember that this is a 2P system.  That 384 GB of RAM is divided between the two processors so, effectively, they each _only_ have 192 GB.


While this is true, both sets of ram are unified in a single address space and each CPU can address the other CPU's ram.


----------



## venturi (Feb 18, 2019)

The pool is fully available to applications from either thread system, unless it’s Numa and the config specifies otherwise

So there really is 384 that can be addressed regardless of cpu bucket

Also there is cache in die, snoop, qpi etc 

So the pool is available beyond the bounds of one cpu or the other


Example

4x gpu, 2 gpus are laned to each processor, but an app that addresses all 4 gpus can use them across the cpu lanes and memory.
Otherwise you’d have two separate PCs


----------



## lexluthermiester (Feb 18, 2019)

notb said:


> "unless the video work is 2160p"
> Hello 2014?
> Yeah, 4K can be dealt with using around 64GB RAM. You'll need double that for comfortable 8K editing.
> And while you're at it, why don't you check the bitrate of raw 4K footage, e.g. ARRIRAW?


I have 24GB in my editing system and it barely tops 11GB with 1080p@3072kbps. I've thought about going up to 48GB but just don't see the need.


----------



## notb (Feb 19, 2019)

thebluebumblebee said:


> @lexluthermiester , @notb , remember that this is a 2P system.  That 384 GB of RAM is divided between the two processors so, effectively, they each _only_ have 192 GB.


That's a very surprising theory from a senior member. ;-)


lexluthermiester said:


> I have 24GB in my editing system and it barely tops 11GB with 1080p@3072kbps. I've thought about going up to 48GB but just don't see the need.


3072 kbps? A typing error?
3072 kbps is orders of magnitude less than even some consumer cameras can put out. 
If you're building a workstation for high-end video editing, 64GB is the minimum for 4K. And 4K is the minimal resolution used today, to be honest. Other than that, it's the matter of workflow.

With the multi-core setups we can build today, you're really encouraged to run few sessions parallelly. A lot of software won't utilize more than 3-4 cores. As shown by Puget, there's really not that much gain in Premiere Pro when moving from 8 to 16 cores (maybe 20%). So you run 2+ sessions and the RAM needs go up.
I work mostly in R and it's the same story. It's a lot cheaper to just get more RAM and run multiple sessions than to waste time on forcing the code to use more resources. But hey - you're a fan of our new many-core reality. You should know this. ;-)


----------



## Jetster (Feb 19, 2019)

I love the way the SSDs are mounted. I would want that case except for the dust


----------



## notb (Feb 19, 2019)

Jetster said:


> I love the way the SSDs are mounted. I would want that case except for the dust


Surprisingly, dust is not a big problem with open cases with large airflow (as in this case). It kind of vacuums itself.

I'd be more worried about mechanical damage and fluids.
I try not to eat near the PC, but I almost always have something to drink. Yesterday I sneezed few seconds after a sip of a shake and spent the next 10 minutes cleaning the keyboard and the screen.
So yeah... had I kept an open-case PC on my desk, this could have been the most expensive sneezing in my life.
I guess it could have been much worse... since sneezing can actually kill you while driving.


----------



## Jetster (Feb 19, 2019)

I use to not have any dust. Then my daughter moved home with her two cats


----------



## Berfs1 (Feb 19, 2019)

How the hell did u get 4 Titan v ceos, let me clarify, 1 is difficult enough, but 4?!?! HOWWWWW


----------



## XXL_AI (Feb 19, 2019)

notb said:


> That's a very surprising theory from a senior member. ;-)
> 
> 3072 kbps? A typing error?
> 3072 kbps is orders of magnitude less than even some consumer cameras can put out.
> ...



1080@24p 3072kbps is more than enough for hevc encoding or VP9.

you don't need 64GB ram if you have access to pascal, volta or turing gpus, extract the raw footage as avi (which in size can take up to a TB) then compress it with gstreamer/ffmpeg or already available tools from nvidia. most of the video encoders, including extremely overpriced and underperforming adobe suite doesn't really support gpu video decoding & encoding.


----------



## Splinterdog (Feb 19, 2019)

It's both a labour of love and a work of art.


----------



## venturi (Feb 19, 2019)

Berfs1 said:


> How the hell did u get 4 Titan v ceos, let me clarify, 1 is difficult enough, but 4?!?! HOWWWWW



I wish!

It’s 1 Titan V 32gb as primary and 3x Titan v regular
As cards 2,3,4

Beyond playing games, the rig gets used on occasion for work, and since there are 36 hours in a day, I also went back to school and I’m doing my dissertation on deep learning for clinical decision support / diagnostic imaging (still deciding on CNN  & algorithms).





4 Titan v ceo would have meant I had 20% of all existing ceo cards in the world, there are only 20



Jetster said:


> I love the way the SSDs are mounted. I would want that case except for the dust



I really don’t have a dust issue. A can of air on occasion does the trick

(we don't have pets, rugs, cloth furniture...just three kids (4 if you include me      )


----------



## notb (Feb 19, 2019)

XXL_AI said:


> 1080@24p 3072kbps is more than enough for hevc encoding or VP9.
> 
> you don't need 64GB ram if you have access to pascal, volta or turing gpus, extract the raw footage as avi (which in size can take up to a TB) then compress it with gstreamer/ffmpeg or already available tools from nvidia. most of the video encoders, including extremely overpriced and underperforming adobe suite doesn't really support gpu video decoding & encoding.


Video editing is not encoding. Encoding is literally just saving the result.

As for the RAM needs:
When you edit a photo/video file, data in RAM is uncompressed and usually takes a lot more space than the RAW file does on the disk. 
RAW contains all the data read from the sensor, which is 1 value per pixel. It's flat and with minimal sufficient bit depth. Some compression is also possible.
After you read a RAW into RAM, it is interpolated to RGB (3 values per pixel) and has a larger bit depth.
It's a huge difference. For most RAW formats in use today, photos will take 3-8 times more space once read into RAM. Video bitrate will increase as well.

And of course this is not limited to multimedia. Most file formats use some compression (encoding).
For example: popular databases will compress data to ~50% on average. But if you have specific data and a specific DB engine, you may see much higher gains.
Yesterday a friend at work asked me for help with a query - the error message said there wasn't enough RAM, which looked weird since the tables were 10GB total. It turned out one of them had over 1000:1 compression ratio, so yeah... 256GB of RAM wasn't enough.

So don't be shocked by the amounts of RAM in workstations and servers. Just because they have 384 GB of RAM doesn't mean they won't choke on a 16GB input. 

But this is really OT, so let's end it here. If you want to discuss video/photo editing, we can make a separate thread. I'll be happy to help.


----------



## venturi (Feb 20, 2019)

So, a small update

my original question was:

*what is "Asus Engine Booster"? (no, its not the Asus OC tune setting)* in the bios


I called Asus Tech Support again, and had to start a ticket (got a case number) just to find out what Asus Engine Booster (and the sub-settings) does. It has been bumped / escalated twice but no one knows yet.

 I just got an email that my case has been *closed* and to please fill out  customer satisfaction survey.

Seriously?

They closed the case?  The only response I got was that it was being escalated.

Customer satisfaction survey? How about a flaming bag of poop?


----------



## notb (Feb 20, 2019)

venturi said:


> So, a small update
> 
> my original question was:
> 
> ...


I don't have your mobo (or any ASUS one, for that matter), so I can't even check whether you've spelled that correctly.
Thing is... google hasn't really heard about "Asus Engine Booster" either.
So there's a good chance it does nothing. Do you have a beta BIOS or something like that? Where did you get your mobo from?

Maybe "closed ticket" basically means that somewhere in Taiwan an e-mail was sent: "hey Peter, remember that feature we tested in December, but cancelled in the end? We forgot to remove it from the BIOS menu. Drop it in the next update."
Although, it's quite unlikely that the guy is "a Peter"...

The sad part here is that C621E SAGE is a very niche board. They don't sell many of them and, despite the "premium" segment and high price tag, it may not get as much Quality Control/Testing love as the mainstream models.
There's like 38 mln people in my country and there are literally 3 of these available in online stores here.


----------



## lexluthermiester (Feb 20, 2019)

notb said:


> 3072 kbps? A typing error?


Not at all. That is actually above the specs really needed by H264/H265 for good picture production. Most movies converted from Bluray that have solid picture fidelity only require 2048kbps. So 50% more is over-kill.


----------



## Mussels (Feb 20, 2019)

I've been watching netflix, and the netflix made TV shows are only 3Mb/s at 1080p - because they're using the new codecs (HEVC/H265) decrease the bandwidth needed.

Good luck dealing with asus tech support... its so rare to find good tech support for PC hardware, other than warranty replacements


----------



## venturi (Feb 20, 2019)

notb said:


> I don't have your mobo (or any ASUS one, for that matter), so I can't even check whether you've spelled that correctly.
> Thing is... google hasn't really heard about "Asus Engine Booster" either.
> So there's a good chance it does nothing. Do you have a beta BIOS or something like that? Where did you get your mobo from?
> 
> ...




Well actually it’s a mainstream bios, and it does a lot of great things that other duallie bioses/boards don’t:  so-vor, virtual disk host/ , OC, voltage control, mgpu setup, etc

The booster does use more power (which I don’t mind) and performance increases but since it’s an OC board I wanted to know where and how it’s applying more juice

The board has also an OC menu that has its own voltage controls, so the new tab is for something more: with as much money as thing costs I’d like to know what and how the feature works


----------



## notb (Feb 20, 2019)

venturi said:


> Well actually it’s a mainstream bios, and it does a lot of great things that other duallie bioses/boards don’t:  so-vor, virtual disk host/ , OC, voltage control, mgpu setup, etc


I've meant that I haven't seen an ASUS BIOS for a while (since P5G41 ). And when I ask google about "asus engine booster" it returns 4 results - half pointing to your forum posts... Hence the theory, that maybe you're the only person in the world that can see it. ;-)


----------



## thebluebumblebee (Feb 21, 2019)

You'd think they'd have better customer support for those who buy a nearly $600 motherboard.

JayzTwoCents just got the ASUS ROG Dominus W3175X, ($1,800) which seems to be a sister board to the one you have.  I think you should contact him and ask him about ABE.  

```
https://www.youtube.com/watch?time_continue=1&v=FaskdjZQvdk
```


----------



## venturi (Feb 21, 2019)

thebluebumblebee said:


> You'd think they'd have better customer support for those who buy a nearly $600 motherboard.
> 
> JayzTwoCents just got the ASUS ROG Dominus W3175X, ($1,800) which seems to be a sister board to the one you have.  I think you should contact him and ask him about ABE.
> 
> ...



That is a great idea, basically that is just the single board version of the c621e.  Awesome suggestion, I will do that


----------



## notb (Feb 21, 2019)

thebluebumblebee said:


> You'd think they'd have better customer support for those who buy a nearly $600 motherboard.


They should and they likely do. I doubt these halo products are profitable anyway.

It's just a matter of priorities.
There's a problem in mainstream H/Q370 mobo - millions of PCs may malfunction => huge problem for the company. Red flags, crisis meetings and everything.
There's a problem with a halo mobo like this one... well, hopefully they won't forget about the 73 people that bought it, but you shouldn't expect much...


----------



## Berfs1 (Feb 21, 2019)

venturi said:


> I wish!
> 
> It’s 1 Titan V 32gb as primary and 3x Titan v regular
> As cards 2,3,4
> ...


Oh man I was hyped! Oh also, if you don’t mind, could you DM me in Instagram or twitter with stock specs of the Titan v ceo? Also, do you know if it is able to SLI with other regular Titan Vs? Super cool build!




notb said:


> I've meant that I haven't seen an ASUS BIOS for a while (since P5G41 ). And when I ask google about "asus engine booster" it returns 4 results - half pointing to your forum posts... Hence the theory, that maybe you're the only person in the world that can see it. ;-)


it might be a auto overclocking feature related to either the cpu’s VRMs (as in, overclocking the VRM controller’s for cleaner power) or the GPU (yes, motherboard features like this do exist)


----------



## theonek (Feb 21, 2019)

A very expensive hobby! Even NASA don't have such pc's!


----------



## PerfectWave (Feb 21, 2019)

amazing work man!


----------



## notb (Feb 21, 2019)

Berfs1 said:


> it might be a auto overclocking feature related to either the cpu’s VRMs (as in, overclocking the VRM controller’s for cleaner power) or the GPU (yes, motherboard features like this do exist)


Well, we could think all day what it *might be*. But it seems OP was the first on the public web to ask about it. It's not even mentioned in the manual (at least not the English version).
So yeah... this looks like a test feature that shouldn't make it to the production BIOS. And what it does? Who knows?


----------



## Berfs1 (Feb 28, 2019)

notb said:


> Well, we could think all day what it *might be*. But it seems OP was the first on the public web to ask about it. It's not even mentioned in the manual (at least not the English version).
> So yeah... this looks like a test feature that shouldn't make it to the production BIOS. And what it does? Who knows? Maybe it sends your data to North Korea.


Pretty sure that would be illegal if it was sending data to NK...


----------



## notb (Feb 28, 2019)

Berfs1 said:


> Pretty sure that would be illegal if it was sending data to NK...


Of course. Just like backdoors in disks would be illegal. And in routers. And phones.
It's good nothing like that happens, ever.


----------



## robot zombie (Mar 1, 2019)

Geez dude. Just doing some quick estimating, that machine has a value of $46000... going by a combination of MSRP's and current market. How far off am I ?


----------



## Vario (Mar 1, 2019)

robot zombie said:


> Geez dude. Just doing some quick estimating, that machine has a value of $46000... going by a combination of MSRP's and current market. How far off am I ?


My guess was in that ballpark as well.


----------



## robot zombie (Mar 1, 2019)

Vario said:


> My guess was in that ballpark as well.


Gotta be... I mean those CPU's alone could run $20k or more for the pair, unless there's something I'm missing.. And then those Titans are, what? $3.5k a pop? And then you have the coolers, that SSD array, top-tier, high wattage PSU, fancy itx enclosure, all of that RAM. I just realized I didn't even factor in the mobo... it's pure madness. When I was doing the math I went conservative on the price of the RAM/SSD's. All of that stuff is gonna add up quick no matter how you look at it...


----------



## venturi (Mar 1, 2019)

I'm not sure about the price either...

A lot of it was from my prior build, and then paid made the differences, then the prior before that, etc.
I scored the 8180M pair for a fraction of the cost as they are QS samples (not ES), and the Titan V CEo 32GB was given to me for free (but I did own 4 titan V's before that 32gb version was given to me.  The case was the cheapest part, $77.90 from amazon - thermaltake P1 miniITX.

...and a jar of KY to fit all that in a miniITX case.


----------



## Vario (Mar 1, 2019)

The Titan V CEO is a true collector's piece.


----------



## MrGenius (Mar 1, 2019)

venturi said:


> ...the Titan V CEo 32GB was given to me for free (but I did own 4 titan V's before that 32gb version was given to me for free...


Nothing is ever free. 

Must be nice to not have to pay for it in cash money though.


----------



## lemonadesoda (Mar 2, 2019)

Respect for that build venturi. Nice work.


----------



## venturi (Mar 8, 2019)

New Cinebench R20 score

is *18708* a good score?







happy to report that ray tracing is alive and well in Metro Exodus with my Titan Vs


----------



## Mussels (Mar 8, 2019)

We've done it lads

We found the PC that can play crysis


----------



## ArbitraryAffection (Mar 8, 2019)

I'm literally speechless. I just... I... yeah.

Wow.



Mussels said:


> We've done it lads
> 
> We found the PC that can play crysis


^ This.



venturi said:


> New Cinebench R20 score
> 
> is *18708* a good score?



is 18708 a good score
is higher than the highest included result in cinebench r20
yes.

XD


----------



## lexluthermiester (Mar 8, 2019)

venturi said:


> New Cinebench R20 score
> 
> is *18708* a good score?
> 
> ...


Spendy CPU. And *DAMN*! I wonder if that's a world record?


----------



## ArbitraryAffection (Mar 8, 2019)

lexluthermiester said:


> Spendy CPU. And *DAMN*! I wonder if that's a world record?


Would R20 actually even run on, say, a "system" with 8 sockets with 8180's in each?. Is that even possible to run one program on such a setup? cuz this is probably THE, or if not, very close to THE highest single motherboard result i think

Edit: i wonder what dual EPYC 7601's would get.


----------



## venturi (Mar 8, 2019)

ArbitraryAffection said:


> Would R20 actually even run on, say, a "system" with 8 sockets with 8180's in each?. Is that even possible to run one program on such a setup? cuz this is probably THE, or if not, very close to THE highest single motherboard result i think



 ...in a miniITX case


----------



## lexluthermiester (Mar 8, 2019)

ArbitraryAffection said:


> Would R20 actually even run on, say, a "system" with 8 sockets each with 8180's in each?. Is that even possible to run one program on such a setup?


Oh yes. Programs that are SMP aware will generally use any CPU/Core available to the system. In the case of Cinebench, it is specifically written to use all CPU's/cores Windows will allow. 8x56cores? I'd love to see THAT score!


----------



## phanbuey (Mar 8, 2019)

We are reeeeeaaaaaally stretching the definition of 'itx' here.

This is an itx case:





The core P1 ITX is a bit larger than volume wise than most compact atx cases.

It's an amazing beast performance / size wise.  But definitely got clickbaited on the itx front .


----------



## venturi (Mar 8, 2019)

phanbuey said:


> We are reeeeeaaaaaally stretching the definition of 'itx' here.
> 
> This is an itx case:
> View attachment 118276
> ...




Thermaltake sells it as a "minITX" case. well yes, of course its not the smallest example of a miniITX case - and I can't change the laws of physics....

(at the same time - my whole case is almost the same size as the motherboard itself, that wasn't easy)


----------



## notb (Mar 8, 2019)

ArbitraryAffection said:


> Would R20 actually even run on, say, a "system" with 8 sockets with 8180's in each?. Is that even possible to run one program on such a setup? cuz this is probably THE, or if not, very close to THE highest single motherboard result i think


You can run almost any program on a multicore platform. Windows sees all cores available.

Rendering in particular can be split into any number of jobs - as long as each process has enough pixels to make it effective ;-). Some engines support distributed rendering (I've used V-Ray).


----------



## phanbuey (Mar 8, 2019)

It is an amazing rig. 

I can imagine that room heating up noticeably with that under full load lol.


----------



## venturi (Mar 8, 2019)

the rig runs cool and silent, but I'm sure the room is getting the blast


----------



## venturi (Mar 10, 2019)

here is the prototype SSD caddy I made for the unit and the lattice for the video cards


----------



## Anymal (Mar 19, 2019)

Why the hell do you want to put it in the smallest case possible?


----------



## venturi (Mar 20, 2019)

Anymal said:


> Why the hell do you want to put it in the smallest case possible?



I like small, quiet, efficient and running cool temps.  I did not see any reason to build a behemoth when I could make it small enough to fit on a desktop. I find efficiency elegant.
Form follows function.


....and all that power in such a small dense space.


Well, the real answer is that this build was based on what I like, not real reason beyond my taste.


----------



## venturi (Apr 4, 2019)

as an example of the array speed on this build. Here is the end report of a full backup showing the sustained write speed.


----------



## theonek (Apr 4, 2019)

How did you achieve this transfer with sata drives only? I saw they are in raid but the sata controllers limit is around 1500MB....


----------



## venturi (Apr 4, 2019)

theonek said:


> How did you achieve this transfer with sata drives only? I saw they are in raid but the sata controllers limit is around 1500MB....


I don't know anything about a sata  controller limit other than on the board the unit is 32gbs hardwired. So 25gbs is within spec.  if it its just reading then it gets up past the 31.5gbs on occasion. This is consistent vs burst; this is on three plus minutes of constant write activity.

maybe the 1500 is an old sata controller limit. Lastly I did back up the whole C drive in 3 minutes, 20 -30 sec was the post backup verification. So, in raw numbers alone the transfer had to actually occur / the rate seems accurate.

on the read and write speeds: I was just sharing the speed of the array stack. Reads alone achieve 30gbs, then again it is an all ssd array using all samsung pro ssds


----------



## theonek (Apr 4, 2019)

well these 32gbs are wired to M2 slot on the mobo for an M2 NVME ssd, sata's are different thing that's why I wonder how you exceed it's limit, there is something more here, maybe it's a server board and has more lanes for it's build-in intel sata controller. Don't think you are using add-on raid card on any slots...


----------



## Deleted member 24505 (Apr 4, 2019)

Yikes that is all.


----------



## venturi (Apr 4, 2019)

theonek said:


> well these 32gbs are wired to M2 slot on the mobo for an M2 NVME ssd, sata's are different thing that's why I wonder how you exceed it's limit, there is something more here, maybe it's a server board and has more lanes for it's build-in intel sata controller. Don't think you are using add-on raid card on any slots...



I'm not using any add on cards. Plus, with 4 Titan Vs on the board, there really aren't any slots available


----------



## venturi (Apr 19, 2019)

Small update
I moved to LRDIMM 2933mhz
So the unit has
768GB of ram (Ecc)

Each LRDIMM is 64GB


----------



## Mussels (Apr 19, 2019)

Just tell us what we need to do, to get you as a sugar daddy

This build is pure madness


----------



## lexluthermiester (Apr 19, 2019)

Mussels said:


> This build is pure madness


Still waiting to hear what the point of it is.


----------



## 64K (Apr 19, 2019)

lexluthermiester said:


> Still waiting to hear what the point of it is.



He said that it was for some gaming but also deep learning, clinical decision support, isotopes, medical imaging. I guess some of those uses require expensive hardware though it's not something I'm familiar with.


----------



## lexluthermiester (Apr 19, 2019)

64K said:


> He said that it was for some gaming but also deep learning, clinical decision support, isotopes, medical imaging. I guess some of those uses require expensive hardware though it's not something I'm familiar with.


I understood that but what is being used is just insane overkill for the tasks being asked of it. Not criticizing of course, just curious whether this rig will be put to practical use or if it's being built for giggles.


----------



## jaggerwild (Apr 19, 2019)

I love how you put it on the floor, then again could eat of those floors....


----------



## venturi (Apr 19, 2019)

lexluthermiester said:


> Still waiting to hear what the point of it is.



Well, on a neural network data training (ML) session I exceeded the 384GB I had so I lifted the ceiling.  I now actually can use most of those resources.


----------



## NdMk2o1o (Apr 19, 2019)

Just for shits and giggles I'm adding your components to my basket on Amazon and even though I haven't added any drives, had to settle for 3x titans and 192GB RAM it's already close to 30k, I expect with the increased RAM, 10x SSD's, Titan V CEO edition etc etc it would be well over £50k  words cant explain how crazy this teeny weeny "computer" is, I guess kudos to you if you can afford something like this, it's friggin insane


----------



## TheMadDutchDude (Apr 19, 2019)

What is this even used for...?!


----------



## thebluebumblebee (Apr 19, 2019)

TheMadDutchDude said:


> What is this even used for...?!


Commander Keen.


----------



## lexluthermiester (Apr 20, 2019)

venturi said:


> Well, on a neural network data training (ML) session I exceeded the 384GB I had so I lifted the ceiling.  I now actually can use most of those resources.


Ok, cool. So an actual purpose, not just for giggles. Nice. TBF, just for giggles is cool too, was just curious.


----------



## storm-chaser (Apr 20, 2019)

Well done. Looks insane! Keep up the good work...


----------



## venturi (Apr 29, 2019)

Ram fan conundrum:

A. I like silent and those fans sound like turbines
B. They won’t fit due to proximity of video card slot, I tried a few

I did buy pure copper heatspreaders and 10 packs of the pad minus 8 thermal grizzly.

I don’t like using the included two sided tape because if you reseat it or try to take it off Theresa good chance you’ll just rip chips right off the ram pcb.

And, the tape is not very thermally conductive.

The minus 8 Pad 1mm and very heat conductive and electrically inert. And isn’t sticky.

At 1mm it’s two thick tours on the heat spreaders, so i’m going to run it through a lasagna press and squish it down to .5mm (and doubling the area it would cover. Then it should be a solution.

LRDIMM best heat spreader design I can come up with :


So here is the process I went through to use pure copper heatsinks, no adhesive, and thermal ceramic clay.

6 of these ceramic clay and wax paper:



lasagna press to reduce 1 mm TIM to .5 mm and gain area.







assembly line



in place:







Dimm Temps
medium low use / idle temps / checking mail / being on this forum temps:

about 42 C average across 12 LRDIMMs


and this concludes the experiment. some observations:

under load the LRDIMMS still get warm / hot but 13C less than before.

after a workout, the dimm temps stay higher longer due to the amount of mass / heat retention added by the ceramic clay and the copper sandwich.

seems to be an improvement overall.


----------



## lexluthermiester (Apr 29, 2019)

venturi said:


> Ram fan conundrum:
> 
> A. I like silent and those fans sound like turbines
> B. They won’t fit due to proximity of video card slot, I tried a few
> ...


Almost a work of art. Definitely a work of excellence!  

As this sentence is being typed out from a 7 year old Dell T3500 that still provides good performance, I wonder how long it will be until that machine will no longer be useful...


----------



## thebluebumblebee (Apr 29, 2019)

The CPU coolers changed?  Do I see some NH-U14S DX-3647?


----------



## venturi (Apr 29, 2019)

thebluebumblebee said:


> The CPU coolers changed?  Do I see some NH-U14S DX-3647?




yes, but not simple. Out of the box the dx-3647 encroach into the video card slot by 1.7mm


some augmented reality later, I was able to make them fit.....


it was hell

here were some picks after surgery (but before ram heat spreaders)


----------



## thebluebumblebee (Apr 29, 2019)

venturi said:


> Out of the box the dx-3647 encroach into the video card slot by 1.7mm


Who's fault is that, Asus or Noctua?  I'd think they, especially Noctua, would want to hear about that.


----------



## Vayra86 (Apr 29, 2019)

venturi said:


> I'm not using any add on cards. Plus, with 4 Titan Vs on the board, there really aren't any slots available



4 Titan V's...
768GB RAM

You wot mate. Can I borrow it? I'll return it I promise



xkm1948 said:


> 384gb ram
> 
> 384gb ram
> 
> ...



No. Its time for another nerdgasm already.

Thoroughly enjoying this thread and your build. Balls to the wall getting redefined in front of our eyes, much like.


----------



## lexluthermiester (Apr 30, 2019)

Vayra86 said:


> You wot mate. Can I borrow it? I'll return it I promise


You are completely bloody bonkers.  LOL! Yeah, THAT'LL happen..


----------



## MrGenius (Apr 30, 2019)

Major fail on the using pads instead of the tape provided with those heat spreaders. That tape is probably the best possible solution. And there's not a snowball's chance in hell those pads perform better(they're mediocre pads to begin with, Fujipoly are WAY better). I've got those same heat spreaders on about a dozen sticks of RAM. Some of which have been removed and reused. Your guess about it ripping ICs off is a bad one. Not even close. For reuse I've replaced the provided tape with the 0.11mm BCP thermal tape(stuff on the blue rolls) you can get all over the place. It's a tiny bit thicker than the stock tape. But it's also a bit wider, and can be cut to length to fit the spreaders better. Resulting in more coverage on the ICs in some instances as well(particularly DDR 1).

I should also mention the fitment issue with those though. Using any kind of tape with dual-sided DIMMs doesn't fit correctly. You need to cut the tabs off the tops to get both spreaders to mount closer to each other. Or you won't be able to get full contact on the RAM ICs. But you'd have figured that out if you'd tried using the tape.


----------



## lexluthermiester (Apr 30, 2019)

MrGenius said:


> Major fail on the using pads instead of the tape provided with those heat spreaders. That tape is probably the best possible solution. And there's not a snowball's chance in hell those pads perform better(they're mediocre pads to begin with, Fujipoly are WAY better). I've got those same heat spreaders on about a dozen sticks of RAM. Some of which have been removed and reused. Your guess about it ripping ICs off is a bad one. Not even close. For reuse I've replaced the provided tape with the 0.11mm BCP thermal tape(stuff on the blue rolls) you can get all over the place. It's a tiny bit thicker than the stock tape. But it's also a bit wider, and can be cut to length to fit the spreaders better. Resulting in more coverage on the ICs in some instances as well(particularly DDR 1).
> 
> I should also mention the fitment issue with those though. Using any kind of tape with dual-sided DIMMs doesn't fit correctly. You need to cut the tabs off the tops to get both spreaders to mount closer to each other. Or you won't be able to get full contact on the RAM ICs. But you'd have figured that out if you'd tried using the tape.


That seems a bit harsh. I have to disagree with you. The spreaders @venturi is using do very well for RAM modules that tend to get hot.


----------



## venturi (Apr 30, 2019)

Mr genius,
The pad 8material has higher thermal conductance than the tape.
But I also want to be able to remove the heatspreaders later without damaging the dimms, the tape is way too dangerous for these expensive dimms.
I’ve seen tape rip chips off the pcb.
Lastly, I appreciate your opinion, but I’ve tested my way up to this process, it was not arrived by accident,

The clay also goes down between  crevices making full contact and limiting the thermos effect.

Again, thank you for your opinion.

I was able to drop the temps 13C, so I am content


----------



## akee99 (May 18, 2019)

very nice Pc   omg


----------



## venturi (Aug 9, 2019)

I made some optimizations. Here is my Cinebench R20 score. Not bad for a small cute little build.

*20143*


----------



## Mussels (Aug 9, 2019)

This isnt the Godbox anymore, needs 2x64 core threadrippers


----------



## lexluthermiester (Aug 9, 2019)

venturi said:


> I made some optimizations. Here is my Cinebench R20 score. Not bad for a small cute little build.
> 
> *20143*
> 
> View attachment 128795


Good Grief mate! Very nice!


----------



## venturi (Aug 9, 2019)

Mussels said:


> This isnt the Godbox anymore, needs 2x64 core threadrippers




If it was on core count alone, then even that 2x64 you refer to would be outdone by any rack box with 4 or 8 cpu sockets

The intent was the most workstation with Audio, 4 way sli or 4 gpu, raid, be very quiet, and run off a single household power outlet

I’m not aware of any 2x64 motherboards that can do that


----------



## Vario (Aug 9, 2019)

venturi said:


> I made some optimizations. Here is my Cinebench R20 score. Not bad for a small cute little build.


It's an impressive build, and the case might be limited to ITX only. but the P1 case is anything but small at 422 x 332 x 380 mm (16.6 x 13.1 x 15 inch)  and 20 lb.  For comparison, my full ATX case is 19x15x8 inch.


----------



## venturi (Aug 9, 2019)

Vario said:


> It's an impressive build, and the case might be limited to ITX only. but the P1 case is anything but small at 422 x 332 x 380 mm (16.6 x 13.1 x 15 inch)  and 20 lb.  For comparison, my full ATX case is 18x15x8 inch.



Very true. The intent was to build off thermaltake’s only making the P1 a miniITX case. With some will power, a motherboard almost the size of the whole case did fit on it (and 2 cpu, 768gb of ram, 4x Titan V, 10x ssd raid, and power, and very quiet)


----------



## Cybrnook2002 (Aug 9, 2019)

venturi said:


> *I wasn't even sure where to post it as  I have some questions that don't fall into any particular category.
> 
> 2. I like using the latest bios, but the prior bios without the microcode mitigation is a faster machine, is there a way to edit out a microcode mitigation in the bios and keep the latest one?*


I scanned the pages, and did not see mention of it. Look into a tool called "UBU" : https://www.win-raid.com/t154f16-Tool-Guide-News-quot-UEFI-BIOS-Updater-quot-UBU.html
Q/A Thread- https://www.win-raid.com/t4531f16-Discussion-UBU-Tool-related-Questions-Reports-and-Suggestions.html

I use it normally to go the other way, on my X99 systems, I use it to keep my MC up to date, since no vendors still supports X99. The tool gives you the ability to slice out and replace MC from your UEFI BIOS.

One word of caution, M$ also deploys MC updates through windows update. So, what WAS happening for me on some of my X99 boards, is that myself (and users around the world) lost the ability to OC our chips. The reason was that newer MC was deployed via M$ update, yet our boards were still on outdated MC so there was a conflict. Just something to keep in mind.


----------



## venturi (Aug 9, 2019)

Cybrnook2002 said:


> I scanned the pages, and did not see mention of it. Look into a tool called "UBU" : https://www.win-raid.com/t154f16-Tool-Guide-News-quot-UEFI-BIOS-Updater-quot-UBU.html
> Q/A Thread- https://www.win-raid.com/t4531f16-Discussion-UBU-Tool-related-Questions-Reports-and-Suggestions.html
> 
> I use it normally to go the other way, on my X99 systems, I use it to keep my MC up to date, since no vendors still supports X99. The tool gives you the ability to slice out and replace MC from your UEFI BIOS.
> ...



I did find where if you’re running MS server - you can using MS provided registry entries set 2 specific variable to not make use of any mitigations, MS provided it as many folks on server were taking a tremendous performance hit.

So in MS server 2019 you can disable / opt out of any mitigations

2nd
I have found a way in this board bios to bypass microcode mitigation in a performance setting


There are several tools around the net to test if you’re protected against the speculative and read ahead mitigations
The tools report when the bios has them off and when the OS has them off. In my case both are off now


----------



## Mussels (Aug 9, 2019)

venturi said:


> If it was on core count alone, then even that 2x64 you refer to would be outdone by any rack box with 4 or 8 cpu sockets
> 
> The intent was the most workstation with Audio, 4 way sli or 4 gpu, raid, be very quiet, and run off a single household power outlet
> 
> I’m not aware of any 2x64 motherboards that can do that


 just a joke, but i do look forward to whatever monster you make after this


----------



## Wavetrex (Aug 9, 2019)

I somehow doubt the new Threadrippers will go to 64 cores.

However, 2x EPYC 7702 (128 cores) would kick this build's nuts so painfully...


----------



## Mussels (Aug 9, 2019)

until now i didnt realise epyc and threadripper were different things

learn something new every day


----------



## venturi (Aug 9, 2019)

Mussels said:


> just a joke, but i do look forward to whatever monster you make after this




Me too.
That day, I go to turn off the pc after doing some emails and a voice says: “Wait....please don’t!”


----------



## Vya Domus (Aug 9, 2019)

Mussels said:


> until now i didnt realise epyc and threadripper were different things



They sort of aren't actually, the physical dies and packages are the same. The only differences come from how they configure them and how the socket is wired, so these differences are pretty much imposed and not necessary.

There is no reason to believe Zen 2 TRs wouldn't go up to 64 cores as well.


----------



## Wavetrex (Aug 9, 2019)

Vya Domus said:


> They sort of aren't actually, the physical dies and packages are the same. The only differences come from how they configure them and how the socket is wired


Having half the memory channels and half the PCI-e lanes is quite a difference in wiring, don't you think ?
Basically, a large amounts of pins on the socket are not used on TR but are used on Epyc.



Vya Domus said:


> There is no reason to believe Zen 2 TRs wouldn't go up to 64 cores as well.


Market segregation.

The 28c Xeon W-3175X vs 32c Threadripper 2990WX was a dick-measuring contest between Intel and AMD for the "HEDT" platform.
Edit: Corrected error in Xeon model number.

With this generation, AMD has won and Intel doesn't have anything faster.
Why would AMD offer a 64-core TR and cut into their EPYC sales when they can stop at 48-core and not cut into their high-margin sales?


----------



## Vya Domus (Aug 9, 2019)

Wavetrex said:


> Having half the memory channels and half the PCI-e lanes is quite a difference in wiring, don't you think ?
> Basically, a large amounts of pins on the socket are not used on TR but are used on Epyc.



I just said the main difference is the _wiring. _I didn't denied that.



Wavetrex said:


> Market segregation.
> 
> The 28c Xeon W-3275 vs Threadripper 2990WX was a dick-measuring contest between Intel and AMD for the "HEDT" platform.
> 
> ...



People said the same thing about how a 32 core TR would cut into EPYC sales. But really it didn't and it wont happen this time either, HEDT is a spec of dust compared to the actual server market.


----------



## lexluthermiester (Aug 9, 2019)

Wavetrex said:


> I somehow doubt the new Threadrippers will go to 64 cores.


There supposed to and AMD has been delivering what they promise.


----------



## Deleted member 178884 (Aug 10, 2019)

Wavetrex said:


> However, 2x EPYC 7702 (128 cores) would kick this build's nuts so painfully...


Not if you take a look at the use case.


----------

