# GOD BOX 2.0



## venturi (Nov 1, 2022)

The specs:
*2x 4090 RTX Founders Edition
2x 8280L (56/112 cores), Asus c621 Sage Dual socket motherboard
1.5 TB ram. DDR4 ECC LRDIMMs
1600W digital power supply
(Data drive) 4x VROC Raid 0 Micron 9300 Max (12.8TB each / 51.2TB array) VROC Premium key
(OS Drive) Sabrent Rocket 4 Plus (8TB). 4x Samsung 860 Pro 4TB each (16TB array)
Asus PA32UCG-K monitor, TT SFF case, MS Data Center 2022 & Ubuntu*

The build:
CPU run at about 41C under load, and RAM hovers at 55C under load.
NVME drives run 28C idle and 49C at the end of a full drive copy.

I'm using 1.5TB of ram, LRDIMMS
upgraded to 2x intel Xeon 8280L processors, the L have higher memory management
Dual 4090 RTX FE
The motherboard is revision 2.1x
Bios 9904 with Resizable Bar
C drive is a Sabrent TLC Rocket 4 plus 8TB
D drive is 4 Micron 9300 Max 12.8TB each (51.2 TB volume) using the U.2 connectors - Using VROC premium key
I have built in (hidden in the case) backup of 4x Samsung 860pro 4TB, as well as a network backup to a NAS in the house
Using VROC premium key, direct raid control and paths - please see pics of the WRITE speed of the internal array

Running MS Data Center 2022 as the OS and Ubuntu

To make it all work and have that in place it took a complete redesign just to have it look similar to what I had before but a lot of base design was changed.
The dual 4090 FE RTXs sit several centimeters above the board compared to where cards normally rest. This required changing all the mounting points and supports for the cards. The video cards needed a custom bracket to hold them stable as they are several centimeters above the board.
This allowed for 4x u.2 connections for the micron 9300 nvme drives in raid to fit to the board 4x u.2 plugs

The C drive now has a large pure copper heatsink on it as well, under load it only reaches 40C, idle is 26C.

The 4090 FE RTXs also operate about 29C in idle. Under load they really don't go past 51C (so far)
ever, most game play hover around 41-47C, application DL/ML may run the cards at max around 51C.

The decrease in temperature is from the additional air channel between the video cards (3.3Cm) and the motherboard. Other temp decreases were inn the CPU, Ram and chipset.

Machine runs very quietly.
Most fans ( including the GPUs) turn off at stable temps (auto settings from motherboard control other fans. CPUs at idle run about 23C (73F) in a 21C house (69-70). Under load they may reach as high as 44C, but not often.
To accommodate the 12.8 TB each 9300 nvme drives, I had to install two cooling fans internally to the design so as to have a quiet airflow over the 4 drives. (see pics) This keeps the 51.2 TB nvme drives drives at around 39-41C under load and 33C at idle. The fans are quiet and invisible to the outside but I will also include a picture of that design.

There is a picture of a Macrium backup report showing the READ speed of the C drive (Sabrent rocket 4 plus) to WRITE the backup to the NVME raid (Micron 9300 4x) at *50.6* Gb/s

PC runs very silently, nice and cool, on average components under load are basically at body temp (feels organic when described as such).
I hope this answers core questions here is a benchmark post to show some of the capability -still first on leaderboard

https://gravitymark.tellusim.com/leaderboard/

and

https://gravitymark.tellusim.com/leaderboard/?size=4k


----------



## caroline! (Nov 2, 2022)

Woo I gotta get me one of those 

Except for the case, I don't like dust.


----------



## dgianstefani (Nov 2, 2022)

Gotta go fast


----------



## Dr. Dro (Nov 2, 2022)

I wish I had a job... 

I approve of the cable management... and everything


----------



## R-T-B (Nov 2, 2022)

Very nice rig.  I'm sure it both works and plays very hard.  Looks sweet too...  sure beats my closet blower mess lol.



dgianstefani said:


> Gotta go fast


If you use this rig to exclusively play Sonic games, I'm pretty sure that's hardware abuse.


----------



## venturi (Nov 2, 2022)

R-T-B said:


> Very nice rig.  I'm sure it both works and plays very hard.  Looks sweet too...  sure beats my closet blower mess lol.
> 
> 
> If you use this rig to exclusively play Sonic games, I'm pretty sure that's hardware abuse.


I do play some games but I also use it for school


To answer what I do with all these builds:

In my dissertation I am authoring algorithms to "clean up" radiology images.



So, in a nutshell, if a 60 minute exam acquires 100% of image data, then taking less data shortens the exam time in a scanner to 30 minutes (generic example). Less data mean anomalies and visual drops below the standard of care (SOC). The reason why entities use shortened exams is so that they can see more patients (slots) for the $$million they spent on a scanner. (Yes, that's not right) - and this is due to the US healthcare model referred to as fee-for-service (FFS)

So vendors sell FDA approved DL software to compensate for the loss of data so images can be returned to SOC and have higher utilization - shorter exam times. The software (I loathe the term Ai, -- loathe ) can also enhance images beyond SOC (about 120% depending on protocol). There isn't any higher reimbursement rate for higher def imaging.


However,

a happier medium, is to run about 60-80% of the original scan time protocol and data and get 115-155% of what would have been a 100% SOC image set through better DL CNN reconstruction (involving k-space and signal to noise ratio enhancements SNR) Hopefully this will enhance patient care by delivering higher diagnostic quality images and aiding the eventual outcome - and that's my goal.

This is not unique to me, vendors have been on this bandwagon for 3 years. I am but a student trying a few different things.

Unfortunately I am not very smart and this has been quite a learning curve for me. The industry brings new technology out and I have to recalibrate what I do. There are so many advancements and drivers in the industry. Constantly changing.

I wanted to included examples of my work (all anonymized and generic) but these are strange times and opted not to show a pre and post reconstruction bank of 1300 slices / enhanced image set in this forum - just to show what one can do with video cards.

I love video games, but its pretty impressive what can be done with GPUs -beyond gaming.

I hope that helps

If you are really interested in this, may I suggest looking at what nVidia is doing in this space:

https://developer.nvidia.com/indust...Clara?,delivery and accelerate drug discovery.


----------



## ShiBDiB (Nov 2, 2022)

Do you live in a clean room?

I don't know why anyone would use an open case like this. All I can think about is how dusty it would be, how my dog would lick my ram, and how my kid would stick his finger in a fan.


----------



## R-T-B (Nov 2, 2022)

ShiBDiB said:


> Do you live in a clean room?
> 
> I don't know why anyone would use an open case like this. All I can think about is how dusty it would be, how my dog would lick my ram, and how my kid would stick his finger in a fan.


He probably lives in a cleaner room than yours.

(and mine, no shame)


----------



## dgianstefani (Nov 2, 2022)

venturi said:


> I do play some games but I also use it for school
> 
> 
> To answer what I do with all these builds:
> ...


Similar to my summer work at the data science building in Swansea medical school, we use deep learning to predict health outcomes from medical history and scans, improving access to appropriate care, by using the SAIL anonymised medical databank and IBM software.  The objective is to use DL techniques and algorithms to do 80% of the work, and analysts clean up the outputs.


----------



## venturi (Nov 2, 2022)

ShiBDiB said:


> Do you live in a clean room?
> 
> I don't know why anyone would use an open case like this. All I can think about is how dusty it would be, how my dog would lick my ram, and how my kid would stick his finger in a fan.


Well... I try to keep it clean.

but its also the only way I could have a dense build that runs cool and quiet.

The former rig's dual 3090 Ti have found a home in my wife's pc (of similar design), also an SFF P1 case





it is an obvious mini-me of mine:


----------



## ShiBDiB (Nov 2, 2022)

R-T-B said:


> He probably lives in a cleaner room than yours.
> 
> (and mine, no shame)



I have carpets and a dog. Dust is unavoidable lol.


----------



## R-T-B (Nov 2, 2022)

ShiBDiB said:


> I have carpets and a dog. Dust is unavoidable lol.


Same here.


----------



## ir_cow (Nov 2, 2022)

Two threads, same topic? https://www.techpowerup.com/forums/threads/dual-4090-build.300555/


----------



## caroline! (Nov 2, 2022)

Dr. Dro said:


> I wish I had a job...


The job:






R-T-B said:


> He probably lives in a cleaner room than yours.
> 
> (and mine, no shame)


I clean the top of my case with a cloth and it's covered in dust again after 5 minutes. F.


----------



## venturi (Nov 2, 2022)

ir_cow said:


> Two threads, same topic? https://www.techpowerup.com/forums/threads/dual-4090-build.300555/


sorry, started in video but decided it belonged in builds/mods like the prior build, it should be in mods/builds

Ashes of the Singularity never ran this fast:


----------



## izy (Nov 2, 2022)

Nice one.


----------



## MrBeasterer (Nov 2, 2022)

Looks like you have connections and knowledge in the related fields.

What are the current industry pain points and bottlenecks for speed to correct diagnosis, and let's say new drugs and treatments including crispr-delivered?

Is it GPU or is it researchers to make the algorithms? What can be done better with bioinformatics and computational power + machine learning?


----------



## Space Lynx (Nov 2, 2022)

caroline! said:


> it's covered in dust again after 5 minutes. F.



I use a HEPA filter vaccuum cleaner to sweep my room. I even sweep the ceiling with the hose attachment. it takes like an extra 3 minutes? 

I then use wet cloth to wipe down surface areas.

and i make sure i wash my curtains every 3-5 months, and my blankets every month. dust is really not an issue for me.

i also run a 99.97% hepa air purifier in my room, that helps a lot, cause it runs 24/7.

i agree it can be annoying though.



venturi said:


> sorry, started in video but decided it belonged in builds/mods like the prior build, it should be in mods/builds
> 
> Ashes of the Singularity never ran this fast:
> View attachment 268232



i really want to see your points per day in folding at home.  

fold for the TPU team!


----------



## xtreemchaos (Nov 2, 2022)

by Jupiter can i join your church that thing deserves worship   well done bud.


----------



## Lei (Nov 2, 2022)

Ladies and gentlemen, there are more transistors in this GOD BOX than entire neurons in your humanly brain


----------



## MrBeasterer (Nov 2, 2022)

Yeah but they are limited by software he he he he.  So my brain still wins


Lei said:


> Ladies and gentlemen, there are more transistors in this GOD BOX than entire neurons in your humanly brain


----------



## Lei (Nov 2, 2022)

R-T-B said:


> He probably lives in a cleaner room than yours.
> 
> (and mine, no shame)


You frog live in a lagoon  Mine however, I think kangaroos and buffalos feel quite at home if there were in my room.








MrBeasterer said:


> Yeah but they are limited by software he he he he.  So my brain still wins


Can you look at 1000 brain tumor images and learn how they look like and then tell me from 5 different tumors which one is which? No you can't....


----------



## qubit (Nov 2, 2022)

Since SLI is no longer supported, what do you use the second GPU for?

Nice build.


----------



## dgianstefani (Nov 2, 2022)

qubit said:


> Since SLI is no longer supported, what do you use the second GPU for?
> 
> Nice build.


Hint, it's not for gaming.

You can pool GPU resources in professional software.


----------



## venturi (Nov 2, 2022)

qubit said:


> Since SLI is no longer supported, what do you use the second GPU for?
> 
> Nice build.


While I can play some games as dual GPU, and I can use my custom running painful auto diff sli workaround, generally speaking it’s not worth it, however even with ONLY one 4090 (sigh) its no slouch.
It also does help having 112 threads and 50+Gb/s drive speed, and OS customizations to optimize ram, nodes, etc. So in general, it just "feels" fast.

the rig is used for deep learning and my dissertation

in deep learning I use 100% of both cards and still need more power, so the answer is that

this is best on a use case scenario


----------



## venturi (Nov 2, 2022)

So I received the modDIY power cables for my 4090s. 

configured with 600w full power each, the corner / bend of the cable happens easily just from gravity. Cables (without sleeving) feel (compared to sleeving) light as hair and just as flexible. No stress on the 4090 sockets.  

I was going to get the right angle adapter from CableMod, but now that the cables are here from nodDIY I no longer see the need for the right angle adapter from Cablemod.

Here are some pics,  seriously, cables just fall into place like a small pony tail. Made cable management easier in a SFF case:


----------



## Chicken Patty (Nov 2, 2022)

I have the cable for my 3090Ti from them and it's fantastic. Good choice.


----------



## AnotherReader (Nov 2, 2022)

Amazing build! After the CPUs, the RAM is the most expensive part. Why did you opt for so much RAM? Are you using the CPUs for Training as well? How hot do the 4090s get when you run something like Blender or Folding at home on them? Do you intend to replace your CPUs with a single upcoming EPYC? If you don't mind sharing, what is your day job?


----------



## qubit (Nov 2, 2022)

Well, it's great that they can be used together in other applications.

@W1zzard any chance you could write an article showing what two 4090s in this kludged "SLI" can do in games?  

Just a few benchies for this I believe would be sufficient.


----------



## venturi (Nov 2, 2022)

AnotherReader said:


> Amazing build! After the CPUs, the RAM is the most expensive part. Why did you opt for so much RAM? Are you using the CPUs for Training as well? How hot do the 4090s get when you run something like Blender or Folding at home on them? Do you intend to replace your CPUs with a single upcoming EPYC? If you don't mind sharing, what is your day job?


it d does get used for gaming, but that's the smallest part of its function, I use iit for deep learning and part of my dissertation, and believe it or not dual 4090s, 1.5TB, and 112 threads gets used up almost immediately on some algorithms.

I explored the epyc motherboards and unfortunately, so farm there aren't any suitable choices, so far.

.my day job?.... if it was only that simple....
Full time doctoral student, two full time jobs in healthcare, owner nuclear pharmacy, owner IT company, part time physicist, DoD contractor, etc.

lucky me, there are 36 hours in a day....


----------



## claes (Nov 2, 2022)

Ngl, the thing I dislike most about TPU is how often people post without reading the thread. @venturi posted what he does with two 490’s a number of times, and people are still asking as if the information isn’t in front of them. Even have people speculating that dual-GPUs can be used liked dual-CPUs when the OP clearly shows a dual-CPU setup…

Congrats on the build @venturi, I’m still impressed with the open chassis results, let alone your work hours, cleaning regimen, and having a family to boot. Hope your dissertation comes together as you hope


----------



## Lei (Nov 3, 2022)

@venturi 
I wonder which part of Nvidia architecture is utilised while deep learning. Is it RT cores, tensor cores or CUDA cores. And why doesn't Nvidia make cards that are designed for AI only? Using all the die size for machine learning.... 

Why wouldn't you buy A6000 for example? 

If you had 2xA6000 cards, you'd be having 96gb vram. Since you have 2x3090ti and 2x4090 sitting there. That would use a lot less power and is way thinner. 

Would you please tell me how faster each epoch gets trained on 4090 comparing to 3090ti?

Either get A6000 or watercool your 4090. In your box, one is blowing hot air into the other. Perhaps you're throttling at least 10% of performance due to heat even if you bear the noise and dust. Liquid cooling those stuff would cost less than 10% of the money you got them for.


----------



## venturi (Nov 3, 2022)

Lei said:


> @venturi
> I wonder which part of Nvidia architecture is utilised while deep learning. Is it RT cores, tensor cores or CUDA cores. And why doesn't Nvidia make cards that are designed for AI only? Using all the die size for machine learning....
> 
> Why wouldn't you buy A6000 for example?
> ...


Well

there a lot of questions in there

so, my goal is to keep the build compact and quiet and able to run on a single household socket.

so while tempting to put in 4 Mai water cooled cards, the power (and space for the coolers) would lose the finesse

secondly, I would like the PC to do more than just my dissertation and just be normal PC in terms of turn it on do email, word docs, excel, remote in to work, surf the net and play games.

third, when I need more horsepower I group the wifes’s PC into a node and while not as efficient, I get it done - that unit is just 10-12 feet away (see pic) with a similar set up

As for the 6000 series cards, - I have one of those builds at work (6 cards in one rack unit) and the 6000 for the money, isn’t justifiable an expense

Last, I need a fast on board data set, so that is why I have the 50Tb array and the 50 GB/s write speed, all in the same PC

yes, the a 6000 is a good choice, but price point vs performance was not there, I pay for all my hardware.


----------



## phill (Nov 18, 2022)

I suddenly feel like I'm not quite as overkill as I thought I was...

Amazing setups and thread, a lot of love and appreciation here from me!!  Outstanding!!


----------



## P4-630 (Nov 18, 2022)

What brand/model of desk chairs are those?


----------



## caroline! (Nov 18, 2022)

Sort of looks like my room, I like the white walls and the light fixture. Keep it simple.

One thing I'd never have though is a glass desk, one punch when playing Touhou and my whole setup is gone. Plus I work on my desk so there's always tools, circuit boards, wires and parts on top of it. I'm messy.


----------



## venturi (Nov 18, 2022)

Hi and thank you

those are “iron horse 12 hour chairs”. They are closer to a car seat than a chair 

the desks glass is tempered 2cm glass,  also know as skyscraper glass, I doubt it would break under “normal” conditions


----------



## Lei (Nov 19, 2022)

venturi said:


> Hi and thank you
> 
> those are “iron horse 12 hour chairs”. They are closer to a car seat than a chair
> 
> the desks glass is tempered 2cm glass,  also know as skyscraper glass, I doubt it would break under “normal” conditions


Show us some pics of radiology images before and after cleaning up.

Here I auto annotated some orchid flowers:


----------



## venturi (Nov 19, 2022)

Wrong quote resending


----------



## P4-630 (Nov 19, 2022)

venturi said:


> Nice taxonomy, I assume it is for the training set or its post validation?
> I assume its at pixel level, colour algorithm?
> 
> As to return your query, here is an example of anonymized pre image set with multiple issues (motion, ringing, shadows, echo, Gibson etc) and acquired at 55% of a normal exam time length (resulting in less data) in a 3T MRI.
> ...




Guess you quoted the wrong person....


----------



## venturi (Nov 19, 2022)

I did... thanks for catching it


----------



## Wirko (Nov 19, 2022)

That's a great demonstration, thanks.

My untrained eye actually sees some blurring, not sharpening, in the processed image. Also, the lone white dot in the left-middle of the brain is completely gone. Is there a possibility that the processing actually remove an important detail? Do doctors need access to unprocessed (or less processed) images too in order to see the full picture?


----------



## venturi (Nov 19, 2022)

Wirko said:


> That's a great demonstration, thanks.
> 
> My untrained eye actually sees some blurring, not sharpening, in the processed image. Also, the lone white dot in the left-middle of the brain is completely gone. Is there a possibility that the processing actually remove an important detail? Do doctors need access to unprocessed (or less processed) images too in order to see the full picture?



A lot of the visible data in the original image is composting from echo shadow overlap moments that enhance on overlap. The image is showing more structures in the postprocess rendering
there is a second step of sharpening, window and level contrast that is performed automatically or by the radiologist, -the POST image has not undergone that process yet (raw), I was just giving an example of the post process. Additionally, the original image is 9% to bright /contrast haze. the post process image may seem dark, but it would normally be viewed on a PACS workstation in a dark room


----------

