# Video of the Day: 23 GeForce GTX 295 Video Cards Installed in a Single Server Rig



## malware (Apr 7, 2009)

Wanna see what crazy is? How about 23 NVIDIA GeForce GTX 295 video cards installed in a single rig. Yep, that's not a typo. Named Atlas Folder, this wholesale pack of G200b GPUs (dubbed: folding server farm) is like an industrial grinder for Folding@Home. Watch the cool YouTube video here and feel free to leave a comment. I bet even NVIDIA engineers dream for such a monstrous distributed computing station.





*View at TechPowerUp Main Site*


----------



## Castiel (Apr 7, 2009)

That is just crazy! Nice find man!


----------



## nicholasblock (Apr 7, 2009)

What a waste!! I don't see any point in that at all


----------



## W1zzard (Apr 7, 2009)

warm in the winter


----------



## ThomasDM (Apr 7, 2009)

It's a Folding@Home farm.

http://atlasfolding.com/?p=448


----------



## tcorbyn (Apr 7, 2009)

WOW.... so would that be like somthing used by OnLive servers?


----------



## shagg (Apr 7, 2009)

Im sure it can run Crysis at 50fps with a resolution of 1920x1800


----------



## newtekie1 (Apr 7, 2009)

Ah, so not all in one machine, but all in one rack, and only 17 actually in the rack...cool I guess.  I wish I could afford to do something like that.


----------



## Disparia (Apr 7, 2009)

Yeah, not as exciting after watching the video...


----------



## Salsoolo (Apr 7, 2009)

fake!
the video is for a folding farm. the link posted by ThomasDM


----------



## BigBruser13 (Apr 7, 2009)

*gtx 390 coming in May*

wow now what is this guy going to do when the quad gtx 390 comes out.
see the pictures of it in the link. You will trip out. I can't believe Techpowerup has not mentioned it at all.


----------



## [I.R.A]_FBi (Apr 7, 2009)

^ its a joke april fools

http://www.tech-forums.net/pc/f78/nvidia-geforce-gtx-390-spy-pics-204159/


----------



## btarunr (Apr 7, 2009)

All running on MSI AMD 790FX motherboards (no SLI). It's far from being the "ultimate gaming hardware", as only one of those 23 cards will be able to render a single 3D application at a time. Should've named it "ultimate folding hardware".


----------



## thebluebumblebee (Apr 7, 2009)

OLD news:
http://forums.techpowerup.com/showpost.php?p=1217159&postcount=4280


----------



## Disparia (Apr 7, 2009)

Unless someone does a real 16-gpu single rig, I'm not going to be impressed. Money = more PPD,  but it doesn't always get you e-peen points!


----------



## PCpraiser100 (Apr 7, 2009)

Great find! That means if I owned this rig I would play Crysis @ max settings while folding!


----------



## goneoffthedeepend (Apr 7, 2009)

The video link is also sort of incorrect. The video is his, but someone lifted and put their name on it. Atlas Folder posted here http://www.maximumpc.com/article/ne...23_geforce_gtx_295_videocards_a_single_server about that in the comments.


----------



## newconroer (Apr 7, 2009)

The only thing exciting about that, or potentially exciting, is if one of the pretty blue lights suddenly went red or dead.


----------



## wolf2009 (Apr 7, 2009)

goneoffthedeepend said:


> The video link is also sort of incorrect. The video is his, but someone lifted and put their name on it. Atlas Folder posted here http://www.maximumpc.com/article/ne...23_geforce_gtx_295_videocards_a_single_server about that in the comments.



*READ THIS GUYS* ! People over at OCN were saying the same thing


----------



## btarunr (Apr 7, 2009)

Story Updated. Thanks for the observations.


----------



## a_ump (Apr 7, 2009)

yea, some hardcore PPD for sure, that's a 1-man team lol. but a waste imo


----------



## lemonadesoda (Apr 7, 2009)

I|m sure that this guy has had a lot of fun building 6 machines into a 19 inch rack.  But 6x 4 GTXs could be improved using a PCIe expansion bay.  Since this is a folding farm and not GPU SLI, he doesnt need x16 slots, so an extension bay would be quite practical. He could use x4 for the cards, and easily drive 8 or 16 cards from ONE PC.


----------



## h3llb3nd4 (Apr 7, 2009)

I want one if I was Bill


----------



## warup89 (Apr 7, 2009)

they stole my PC name >=[ "Atlas"


----------



## soldier242 (Apr 7, 2009)

man thats a lot of crunching power


----------



## h3llb3nd4 (Apr 7, 2009)

correction...*folding* power


----------



## Atlas Folder (Apr 7, 2009)

lemonadesoda said:


> I|m sure that this guy has had a lot of fun building 6 machines into a 19 inch rack.  But 6x 4 GTXs could be improved using a PCIe expansion bay.  Since this is a folding farm and not GPU SLI, he doesnt need x16 slots, so an extension bay would be quite practical. He could use x4 for the cards, and easily drive 8 or 16 cards from ONE PC.


That is not correct.  The nVidia driver only supports 8 GPUs.  I have spent a lot of time researching this issue and at the current time it's not possible to use PCIe extenders or motherboards that have more than 4 PCIe slots for this reason.  It is also not possible to run VMs on a machine with PCIe backplanes because there is no VM software yet that allows virtualization of the GPUs for CUDA.  Xen is working on it though.

You can see a discussion here: http://foldingforum.org/viewtopic.php?f=38&t=9340&start=0

and here: http://foldingforum.org/viewtopic.php?f=38&t=9011

Jason


----------



## ghost101 (Apr 7, 2009)

What ppd do you get? Also whats the power usage like?


----------



## n-ster (Apr 7, 2009)

It's Atlas that has it... Look here for details (ppd etc)


----------



## mtosev (Apr 7, 2009)

HARDCORE


----------



## sol.fides (Apr 7, 2009)

The earth simulator had 41 teraflops of supercomputing power. it was #1 in the world starting in 2002 till 2004. It cost about 72 million dollars. congratulations on building an equivalent for under $30,000.


----------



## mmaakk (Apr 7, 2009)

@ Atlas Folder

On daily basis, how much time do you spend controlling all your Clients? At the moment I have only 6 Clients and sometimes I get more than one down while I'm at work.


----------



## n-ster (Apr 7, 2009)

I believe he isn't a TPU member... unfortunately


----------



## Atlas Folder (Apr 7, 2009)

ghost101 said:
			
		

> What ppd do you get? Also whats the power usage like?



My 24 hour average is ~267,000 PPD.  You can see that here.

The power consumption of a single shelf is ~1,125 Watts.  I have 5 shelves and more misc equipment.  Power consumption is probably around 7,000 Watts continuous.  I haven't added it all up lately.



			
				sol.fides said:
			
		

> The earth simulator had 41 teraflops of supercomputing power. it was #1 in the world starting in 2002 till 2004. It cost about 72 million dollars. congratulations on building an equivalent for under $30,000.



Well thanks, but there is an important difference.  True supercomputers use 64-bit floating point math (double-precision), Folding@Home makes do with 32-bit floating point math (single-precision).  So yes, building supercomputers is getting cheaper fast, but there's still no free (cheap) lunch.  If you need double-precision you still must pony up the cash.



			
				mmaakk said:
			
		

> On daily basis, how much time do you spend controlling all your Clients?


Not much...  I can check for dead EUE clients anywhere from my iPhone by going here, it's updated every 60 seconds.  If I need to make a correction all of the machines are controllable through LogMeIn and I have LogMeIn Ignition on my iPhone so I can log into the servers from my iPhone and tweak things as necessary.

I made a post a couple of weeks ago about how I control and implement all of the clients.  I think it's kinda slick even if I do say so myself, but necessity is the mother of invention.  Controlling all the clients individually got old very, very fast.  I had to think of a better way.  You can see my post about that here.

Thanks for the interest guys,

Jason


----------



## lemonadesoda (Apr 8, 2009)

Thanks for coming to TPU and sharing your experience Jason. Great build(s), nice rack. I can see that perhaps the multiple PC route is easier than the extension bay option.

Did I understand you are driver-limited. Therefore max 8 GPUs per PC, so you would need a minimum of 3 independent PCs to run your 24 cards? Or does the x2 thing knock that down to just 4 cards (8 GPU dies)?

PS. Write to nVidia, show them your stuff, get them to fix the drivers, and sponsor your rack.


----------



## Haytch (Apr 8, 2009)

The Atlas Folder, very nice folding setup. 
I wonder what it will actually be crunching.


----------



## Supreme0verlord (Apr 8, 2009)

Wow that is INSANE! I wish I had the money to do something like this.


----------



## DaMulta (Apr 8, 2009)

He was short a few cards on the bottom and it really kills the look of the system



Just kidding, and that's one hell of a folding rig you have!


----------



## HammerON (Apr 8, 2009)

Welcome to TPU Atlas 
It is neat to see all that awsome hardware (especially the GPUs) working hard to find a cure 
For those that want to learn more about his cause, check out the link to his website at bottom of his sig.


----------



## zCexVe (Apr 8, 2009)

Its for a great cause.I wish his idea becomes true.


----------



## Hayder_Master (Apr 8, 2009)

i think Folding@Home complete all research soon


----------



## laszlo (Apr 8, 2009)

hayder.master said:


> i think Folding@Home complete all research soon



i didn't hear any results like a cure or something and since 2000....

i'm skeptikal about this f@h do you know exactly what data is processed ?

Stanford say is protein folding and other molecular dynamics but is really this?


----------



## The Witcher (Apr 8, 2009)

Finally, someone who is folding not for showing off....


----------



## mdm-adph (Apr 8, 2009)

laszlo said:


> I made a post a couple of weeks ago about how I control and implement all of the clients.  I think it's kinda slick even if I do say so myself, but necessity is the mother of invention.  Controlling all the clients individually got old very, very fast.  I had to think of a better way.  You can see my post about that here.
> 
> Thanks for the interest guys,
> 
> Jason



Welcome -- I think it's really cool what you're doing to help out your Dad.  Anytime anyone criticizes the amount of money it all costs, just remind them about that.

That being said, looks like I've found another admin who names a group of server units after characters in _Atlas Shrugged_.  



laszlo said:


> i didn't hear any results like a cure or something and since 2000....
> 
> i'm skeptikal about this f@h do you know exactly what data is processed ?
> 
> Stanford say is protein folding and other molecular dynamics but is really this?



Ah, critics.  

Cures take a long, long time to develop.


----------



## newtekie1 (Apr 8, 2009)

*@Atlas Folder*: Are you also running SMP clients on the CPUs, or are you just doing pure GPU folding?


----------



## newbielives (Apr 8, 2009)

Why cure cancer when you can play Crysis over 60fps


----------



## FordGT90Concept (Apr 8, 2009)

I was expecting a custom, 23 PCI Express slot motherboard board loaded with GPUs (like a Tesla system).  Needless to say, I'm disappointed.  Anyway, that's quite a blatant waste of money and electricity, in my opinion.


----------



## h3llb3nd4 (Apr 8, 2009)

what cpu are you using Atlas??
I wonder if it is i7s...
I think you can also run WCG while folding...


----------



## newtekie1 (Apr 8, 2009)

h3llb3nd4 said:


> what cpu are you using Atlas??
> I wonder if it is i7s...
> I think you can also run WCG while folding...



The boards he seems to be using looks like the MSI K9A2 Platinum, an AM2 board.  So I would guess he is running some kind of Phenom.


----------



## h3llb3nd4 (Apr 8, 2009)

damn! ... I am still confused about how you link up all those pcs, and how do you get them all onto one monitor...?


----------



## newtekie1 (Apr 8, 2009)

h3llb3nd4 said:


> damn! ... I am still confused about how you link up all those pcs, and how do you get them all onto one monitor...?



They aren't really linked, they don't need to be really.  They are all on the same network, that is about it.

As for connecting them all the to the same monitor, I'm sure the rack has a KVM, which allows multiple machines to be connected to the same Keyboard, Monitor, and Mouse.  Although, once set-up the machines should be able to be run headless, with remote access via LogMeIn or Remote Desktop.


----------



## h3llb3nd4 (Apr 8, 2009)

ahh now I understand...


----------



## mdm-adph (Apr 8, 2009)

FordGT90Concept said:


> I was expecting a custom, 23 PCI Express slot motherboard board loaded with GPUs (like a Tesla system).  Needless to say, I'm disappointed.  Anyway, that's quite a blatant waste of money and electricity, in my opinion.



Please go to the guy's site and read his story before you call it a waste.


----------



## FordGT90Concept (Apr 8, 2009)

There's better ways to hunt down a cure to Huntington's Disease than protein folding simulations through Stanford University.  $30,000 towards the development of nanomachines, for example, would be far more beneficial and practical use of the money.


----------



## newtekie1 (Apr 8, 2009)

FordGT90Concept said:


> There's better ways to hunt down a cure to Huntington's Disease than protein folding simulations through Stanford University.  $30,000 towards the development of nanomachines, for example, would be far more beneficial and practical use of the money.



Actually, $30,000 would likely not provide any useful funding towards real research.  Most of these programs have yearly budgets in the millions, $30,000 would hardly make a difference.  For relatively small amounts of money like $30,000, a Folding farm is actually a pretty good way to put it to use.


----------



## FordGT90Concept (Apr 8, 2009)

Almost all research, aside from pharmaceuticals, is funded by grants and public donations.  $30k is enough to hirer a research for 1/3 to 1/4 of a year.  90 to 120 days which amounts to about 720 to 960 hours of labor is actually rather significant.  That's 720 hours closer to a permanent solution, assuming the researcher hired is a good one.  And hey, you'd be indirectly employing someone for a long period of time.

Even if it is spent on hardware, most research servers are around $100k to $150k.  $30k would put a significant dent in that allowing them to run algorithms on site.


Just remember that most (>99%) research computing is still performed on supercomputers.  Never underestimate the usefulness of owning resources you are dependant upon.


----------



## Atlas Folder (Apr 8, 2009)

To answer some of the questions I've seen:

I'm using AMD Phenom Quad-Core 9550 processors in MSI K9A2 Platinum motherboards.  OS is XP Pro and RAM is 2GB per machine.  There are 5 machines like that currently for 20 cards.  The head or "server" of the rack that houses the data has one card.  I have two cards in my home computer.  23 GTX295 total.  All of the machines run Quad-Core SMP also.  As a matter of fact you can view the rack's FahMon here.

The machines run fine on 2GB, the most I've ever seen one using is 1.35GB.

Jason


----------



## h3llb3nd4 (Apr 9, 2009)

a dead GPU??


----------



## theeldest (Apr 9, 2009)

laszlo said:


> i didn't hear any results like a cure or something and since 2000....
> 
> i'm skeptikal about this f@h do you know exactly what data is processed ?
> 
> Stanford say is protein folding and other molecular dynamics but is really this?



It's actually processing the entire internet to find the best nudie pics.


----------



## h3llb3nd4 (Apr 9, 2009)

perverts wish that their pc could do that...


----------



## theeldest (Apr 9, 2009)

Hey Jason,

I appreciate the Ayn Rand references.


----------



## Atlas Folder (Apr 9, 2009)

Sure, GPUs die.  Folding@Home is a pretty demanding application and I'm running them 24/7.

I went and checked the total current draw from the rack at the moment through my APC metered PDU (networked power strip).  46.4 Amps continuous across the whole machine.

46.6 * 120V = 5,592W

..and who is the Ann Rand person?  

Jason


----------



## mmaakk (Apr 10, 2009)

theeldest said:


> It's actually processing the entire internet to find the best nudie pics.



Yes and if you want a share of the results, WILL HAVE TO FOLD WITH US.


----------

