# University of Antwerp Makes 4k € Supercomputer with Four GeForce 9800 GX2 Cards



## malware (May 31, 2008)

Have you ever thought that gaming parts like NVIDIA's GeForce 9800 GTX video cards for instance, can be used for building a supercomputer. Maybe no, but researchers at the University of Antwerp in Belgium have proven that it's possible to build one. Using four NVIDIA GeForce 9800 GX2 graphics cards, AMD Phenom 9850 processor, 4x2GB Corsair Twinx DDR2 PC6400 memory and MSI K9A2 Platinum motherboard, FASTRA costs less than 4000EUR to build and thanks to NVIDIA's CUDA technology and delivers roughly the same performance as a supercomputer cluster consisting of hundreds of PCs. This new system is used by the ASTRA research group, part of the Vision Lab of the University of Antwerp, to develop new computational methods for tomography. Tomography is a technique used in medical scanners to create three-dimensional images of the internal organs of patients, based on a large number of X-ray photos that are acquired over a range of angles. ASTRA develops new reconstruction techniques that lead to better reconstruction quality than classical methods. You can read more about the FASTRA GPU SuperPC project over here.



 



*View at TechPowerUp Main Site*


----------



## tkpenalty (May 31, 2008)

Hmm... Octa G92s... thats almost two Teraflops isnt it? Supercomputer in a full tower microcomputer. Kewl. Then again GPUs have WAY more number crunching power (FP calc speed) over normal microprocessors...

Thats pretty cheap for a supercomputer.


----------



## Conti027 (May 31, 2008)

looks like the cards would over heat


----------



## Weer (May 31, 2008)

I didn't know Octa-SLi could work.


----------



## tkpenalty (May 31, 2008)

Weer said:


> I didn't know Octa-SLi could work.



With CUDA it will allow it. I Don't think that they are running in SLi, they are just plugged in and working as processors.


----------



## Basard (May 31, 2008)

they should set it up to balance the budget.


----------



## Gam'ster (May 31, 2008)

Thats some nice gear someone there has an eye for good tech lian-li case, nvidia gpus, but i expected an intel proc but rock on AMD 
And its being used for medical research ( a damed good thing ) not some useless crap well never see or hear of again.

Gam


----------



## Weer (May 31, 2008)

tkpenalty said:


> With CUDA it will allow it. I Don't think that they are running in SLi, they are just plugged in and working as processors.



How could it work in full power in CUDA if it's not running in SLi? It will only use the card connected to the monitor(s).


----------



## Mussels (May 31, 2008)

Weer said:


> How could it work in full power in CUDA if it's not running in SLi? It will only use the card connected to the monitor(s).



faulty logic there. SLI allows two images onto one display, but the work they're doing isnt even 3D, nor does it need to be displayed. They're using them as number crunchers through CUDA, not as video cards.


----------



## tkpenalty (May 31, 2008)

Mussels said:


> faulty logic there. SLI allows two images onto one display, but the work they're doing isnt even 3D, nor does it need to be displayed. They're using them as number crunchers through CUDA, not as video cards.



I thought he would have the sense... u finished off what I was saying.

Yes. With CUDA... 9800GX2 become the equivelent of stacks of CPUs. The thing that is awesome about this is the fact that 4000 euro is MUCH cheaper than a supercomputer equivalent -COUGH COUGH NASA'S SUPERCOMPUTER COUGH-


----------



## Deleted member 3 (May 31, 2008)

Weer said:


> I didn't know Octa-SLi could work.



Multithreading using multiple GPU's doesn't require SLI. At work we utilize multiple GPU's without the need of SLI/Crossfire as well. In fact, we can't use either of those.
In theory it could even run on a ATI+NV card.


----------



## Ripper3 (May 31, 2008)

Conti027 said:


> looks like the cards would over heat



The GX2 sucks in air from top bottom and sides, and exhaust it out the back, so it should be alright.


I wonder how well this machine would run if it was running on either dual Opterons or dual Xeons. But if it's able to do what would usually take hours in seconds, I'm sure they don't need any more power that it has now.

Actually, just goes to show the raw, unused power some people have in their machines. Of course, I don't think anyone on these forums has four GX2s in one machine, but the 2900s and 3800s do well at F@H already.


----------



## Disparia (May 31, 2008)

Naw, just build another one. 

Could use multi-socket boards, PCIe expansion systems, etc, but the cost would start to climb considerably compared to a cluster of these boxes.


----------



## Ripper3 (May 31, 2008)

Just remembered that Nvidia has this: http://www.nvidia.com/page/quadroplex.html
I'm sure it's a more expensive solution, and probably not as versatile as four GX2s.

Yeah, jsut checked, definitely cheaper to buy four GX2s and a system to put them in.
One Quadro Plex Model IV costs $10750 :O and it's in backorder.
States it can be used for more than jsut rendering, but for the price, and the potential waiting time, I'd just build a system with four GX2s.
Oh, and it only has two GPUs.


----------



## Random Murderer (May 31, 2008)

tkpenalty said:


> With CUDA it will allow it. I Don't think that they are running in SLi, they are just plugged in and working as processors.



that's how i see it as well, instead of actually using the cards for graphics rendering, they're using each g92 core for floating point calculations, just like f@h does with ati cards(and nvidia cards soon)


----------



## Grimskull (May 31, 2008)

wonder what it would be like to play solitaire on??? bet its just only meh!!


----------



## Cold Storm (May 31, 2008)

Random Murderer said:


> that's how i see it as well, instead of actually using the cards for graphics rendering, they're using each g92 core for floating point calculations, just like f@h does with ati cards(and nvidia cards soon)



Thats the same thing I get out of it as well... and Can't wait for Nvidia cards to work with F@H!!!


----------



## Mussels (May 31, 2008)

Cold Storm said:


> Thats the same thing I get out of it as well... and Can't wait for Nvidia cards to work with F@H!!!



i agree. G92 cores have a lot of processing power, whilst running cool (on anything NOT the reference cooler  )


----------



## Cold Storm (May 31, 2008)

Mussels said:


> i agree. G92 cores have a lot of processing power, whilst running cool (on anything NOT the reference cooler  )



That is true. only thing is that my gx2 is only cooled by that right now! till I find a cooler for it!


----------



## WarEagleAU (May 31, 2008)

Awesome for the Phenom proc.


----------



## tkpenalty (May 31, 2008)

Mussels said:


> i agree. G92 cores have a lot of processing power, whilst running cool (on anything NOT the reference cooler  )



Actually the 9800GX2 temps are cooler than Reference 8800GT temps, by 10*C most of the time.


----------



## v-zero (May 31, 2008)

I wonder if this wouldn't work better with four 3870X2s. Theoretically the 3870s are just as powerful, and are cheaper to boot (and made on a better manufacturing process). Considering that the weakness of the 3870 architecture in gaming is mostly its AA implementation I think it may be a more powerful, and certainly more parallel raw-computing design.

Very interesting though.


----------



## Mussels (May 31, 2008)

v-zero said:


> I wonder if this wouldn't work better with four 3870X2s. Theoretically the 3870s are just as powerful, and are cheaper to boot (and made on a better manufacturing process). Considering that the weakness of the 3870 architecture in gaming is mostly its AA implementation I think it may be a more powerful, and certainly more parallel raw-computing design.
> 
> Very interesting though.



well since they ran their program on CUDA, theres no information on whether their needs could even be met on ATI.




> he main reason why they configured an AMD system is because they couldn't find a motherboard for the Intel platform that could fit four GeForce 9800 GX2 graphics cards. Another interesting note is that this system doesn't need SLI, their application uses the NVIDIA CUDA programming model which makes all eight GPUs work in parallel. The researchers say they don't need SLI during a reconstruction as every graphics card communicates directly with the CPU, no inter-GPU communication is needed.


----------



## v-zero (May 31, 2008)

Mussels said:


> well since they ran their program on CUDA, theres no information on whether their needs could even be met on ATI.



AMD have a "software stack" for stream computing also...


----------



## jbunch07 (May 31, 2008)

haha i like the 1500w psu.
good c/m as well.
im betting that the cards in the middle get nice and warm though.

anyway this things is very B/A


----------



## Cold Storm (May 31, 2008)

Hey it takes a lot to power that beast! lol


----------



## lemonadesoda (May 31, 2008)

If you are interested in this sort of stuff, head over to http://www.gpgpu.org/

A review of math accelerators would be interesting, esp. performance comparison (raw FPU) for

1./ Physx
2./ CUDA
3./ Clearstream
4./ SpursEngine
5./ ...


----------



## jbunch07 (May 31, 2008)

Cold Storm said:


> Hey it takes a lot to power that beast! lol



i wonder if they could have got away with 1200w or even 1000w?


----------



## Cold Storm (May 31, 2008)

I don't think so. Corsair's little calculator says I need 1000w just for 2 9800gx2... 

and the PSU Calculator only goes up with two in sli... so who knows?!?!


----------



## jbunch07 (May 31, 2008)

Cold Storm said:


> I don't think so. Corsair's little calculator says I need 1000w just for 2 9800gx2...
> 
> and the PSU Calculator only goes up with two in sli... so who knows?!?!



hmm well the only other psu that is higher than that that i know of is the ultra 1600w

but hey...if you got to have it... why not.


----------



## ghost101 (May 31, 2008)

If each card uses say 200w, 4 of them means 800w. Add all the other components and also overclock, a 1500w psu is probably advisable.


----------



## jbunch07 (May 31, 2008)

ghost101 said:


> If each card uses say 200w, 4 of them means 800w. Add all the other components and also overclock, a 1500w psu is probably advisable.



true true.
i forget we are talking about 8gpus and 4cpus.


----------



## Bull Dog (May 31, 2008)

And of course what is even funnier is that its running on MSI's 790FX board (only board I know of that would support such a setup).


----------



## Cold Storm (May 31, 2008)

yeah, there we go! sounds good to me also.


----------



## jbunch07 (May 31, 2008)

Bull Dog said:


> And of course what is even funnier is that its running on MSI's 790FX board (only board I know of that would support such a setup).



haha true that. 
this is good advertising for MSI


----------



## wolf (May 31, 2008)

the great thing about the 8 gpu setup means that a healthy shader overclock ober 8 cores means decent gains in overall computational power.

stock shader clock 1500mhz X 128 SP X (MADD (2 FLOPs) + MUL (1 FLOP) X 2 (because theres 2 Gpu's) = 768

GeForce 9800 GX2---------------768---G/Flops---[1152 (peak theoretical)]
GeForce 9800 GX2 SLI (x4)------1536--G/Flops---[2304 (peak theoretical)]
GeForce 9800 GX2 SLI (x8)------3072--G/Flops---[4608 (peak theoretical)]


An over clock of the shader units to somewhere in the order of ~1750mhz, which i would argue is modest on a G92, would bring you these figures.

GeForce 9800 GX2------------896---G/Flops---[1344 (peak theoretical)]
GeForce 9800 GX2 SLI (x4)----1792--G/Flops---[2688 (peak theoretical)]
GeForce 9800 GX2 SLI (x8)----3584--G/Flops---[5376 (peak theoretical)]

so we can see that a modest overclock here brings you a half a teraflop jsut like that, not bad. and a shader overclock of 2000mhz would give you this nice figure on this sytem.

GeForce 9800 GX2 SLI (x8)----4096--G/Flops---[6144 (peak theoretical)

also interestingly ive made some figures based on the current GTX280 estimates.

1296mhz x 240 SP X (MADD (2 FLOPs) + MUL (1 FLOP) X 4 (because we would have four in this system) = 2488 G/flops [3732 (peak theoretical)]

now overclocking....

~1500 mhz x 240 SP X (MADD (2 FLOPs) + MUL (1 FLOP) X 4 (because we would have four in this system) = 2488 G/flops [4320 (peak theoretical)]
~1750 mhz x 240 SP X (MADD (2 FLOPs) + MUL (1 FLOP) X 4 (because we would have four in this system) = 3360 G/flops [5040 (peak theoretical)]
~2000 mhz x 240 SP X (MADD (2 FLOPs) + MUL (1 FLOP) X 4 (because we would have four in this system) = 3840 G/flops [5760 (peak theoretical)]

just imagine four GX2-280's... a man can dream......



The Tech Report


> One thing I should note: I've changed the FLOPS numbers for the GeForce cards compared to what I used in past reviews. I decided to use a more conservative method of counting FLOPS per clock, and doing so reduces theoretical GeForce FLOPS numbers by a third. I think that's a more accurate way of counting for the typical case.



Wikipedia


> For example the GeForce 8800 GTX has 518.43 GigaFLOPs theoretical performance given the fact that there are 128 stream processors at 1.35 GHz with each SP being able to run 1 Multiply-Add and 1 Multiply instruction per clock [(MADD (2 FLOPs) + MUL (1 FLOP))×1350MHz×128 SPs ＝ 518.4 GigaFLOPs][4]. This figure may not be correct because the Multiply operation is not always available[5] giving a possibly more accurate performance figure of (2×1350×128) = 345.6 GigaFLOPs.


----------



## DrPepper (May 31, 2008)

All that money and they have to sit the screens on 2 packs of A4.


----------



## jbunch07 (May 31, 2008)

DrPepper said:


> All that money and they have to sit the screens on 2 packs of A4.



haha i saw that to and tried to ignore it.


----------



## DrPepper (May 31, 2008)

Would be cool if they could use CUDA to run crysis or 3dmark Vantage


----------



## DrPepper (May 31, 2008)

http://fastra.ua.ac.be/images/pic_tones_large.jpg I thought it was a phenom or is that more crafty intel advertising.


----------



## jbunch07 (May 31, 2008)

ha!
Intel is just wishing!


----------



## DrPepper (May 31, 2008)

That would really benefit if it could use pci-e 3 it says on the website that the pci-e was a bottleneck.


----------



## Morgoth (May 31, 2008)

so this doest need a a sli chip?


----------



## DrPepper (May 31, 2008)

Morgoth said:


> so this doest need a a sli chip?



no


----------



## Morgoth (May 31, 2008)

just readed there website 
and shecked there specs when nehalem server parts are comming out it will easly crush that system


----------



## Fleekar (May 31, 2008)

Hurray for using xp haha


----------



## DrPepper (May 31, 2008)

Morgoth said:


> just readed there website
> and shecked there specs when nehalem server parts are comming out it will easly crush that system



I doubt it since gpu's are much more efficient for that kinda stuff. Unless you mean they replace the phenom with a nehalem.


----------



## eidairaman1 (May 31, 2008)

You could turn water into steam with the GPU temps. Most definitely not worth the cash for the Enthusiast Gamer.



malware said:


> Have you ever thought that gaming parts like NVIDIA's GeForce 9800 GTX video cards for instance, can be used for building a supercomputer. Maybe no, but researchers at the University of Antwerp in Belgium have proven that it's possible to build one. Using four NVIDIA GeForce 9800 GX2 graphics cards, AMD Phenom 9850 processor, 4x2GB Corsair Twinx DDR2 PC6400 memory and MSI K9A2 Platinum motherboard, FASTRA costs less than 4000EUR to build and thanks to NVIDIA's CUDA technology and delivers roughly the same performance as a supercomputer cluster consisting of hundreds of PCs. This new system is used by the ASTRA research group, part of the Vision Lab of the University of Antwerp, to develop new computational methods for tomography. Tomography is a technique used in medical scanners to create three-dimensional images of the internal organs of patients, based on a large number of X-ray photos that are acquired over a range of angles. ASTRA develops new reconstruction techniques that lead to better reconstruction quality than classical methods. You can read more about the FASTRA GPU SuperPC project over here.
> 
> 
> 
> ...


----------



## DrPepper (May 31, 2008)

eidairaman1 said:


> You could turn water into steam with the GPU temps. Most definitely not worth the cash for the Enthusiast Gamer.



Then use the steam to turn turbines to supply enough power to light a small village


----------



## eidairaman1 (May 31, 2008)

DrPepper said:


> Then use the steam to turn turbines to supply enough power to light a small village



LMAO!!! But the energy it takes to run those cards vs the energy produced is not that great


----------



## DrPepper (May 31, 2008)

the solution to the energy crisis by recycling used energy on gpu's


----------



## newtekie1 (May 31, 2008)

They do need to cnvert this setup to water cooler.  Maybe they can do what that village in Europe did with their water cooling setup for their IBM super-computer, use it to heat their public swimming pool.


----------



## DrPepper (May 31, 2008)

Just sit it in a pool and make sure its water proof and it will heat it up. Or you could use it to boil water for tea or for cooking etc.


----------



## eidairaman1 (May 31, 2008)

newtekie1 said:


> They do need to cnvert this setup to water cooler.  Maybe they can do what that village in Europe did with their water cooling setup for their IBM super-computer, use it to heat their public swimming pool.



Wow he is being all serious about this While Shadow and Dr are joking about it, LMAO!!!


----------



## Morgoth (May 31, 2008)

DrPepper said:


> I doubt it since gpu's are much more efficient for that kinda stuff. Unless you mean they replace the phenom with a nehalem.



no i'm talking abouth bandwith of ram and quikpath


----------



## DrPepper (May 31, 2008)

I'm going to make mine a reality  DrPeppers nuclear nvidia reactor.


----------



## DrPepper (May 31, 2008)

Morgoth said:


> no i'm talking abouth bandwith of ram and quikpath



Yeah the ram was the weakpoint in the system. I think nehalem with the 4 Gx2's would cure cancer.


----------



## Morgoth (May 31, 2008)

i hope so


----------



## eidairaman1 (May 31, 2008)

DrPepper said:


> I'm going to make mine a reality  DrPeppers nuclear nvidia reactor.



ya since Green is Related to Nuclear Waste, glowing green lolz.


----------



## DrPepper (May 31, 2008)

Morgoth said:


> i hope so



If it does i'l be an intel believer for life 



eidairaman1 said:


> ya since Green is Related to Nuclear Waste, glowing green lolz.



and green is good for the environment = greenpeace loves me


----------



## eidairaman1 (May 31, 2008)

not nuclear waste dude.


----------



## DrPepper (May 31, 2008)

eidairaman1 said:


> not nuclear waste dude.



just tell them its fairies not radiation


----------



## eidairaman1 (May 31, 2008)

DrPepper said:


> just tell them its fairies not radiation



Heh, thats amusing.


----------



## Haytch (Jun 1, 2008)

I dont see anything super about it.
Its actually a pretty crappy setup, except ofcourse the 4gfx cards.  I applaud CUDA being used to make the cards render images given to them for medical purposes, thats gold and very smart, but other then that, its crappy. Actually, its a total piece of shit.

Sorry, i get emotional    The only reason i call it a piece of shit computer is because someone actually thinks its SUPER . . .   Like i said, nothing super about the actual pc. The super part is the people that wrote the program to make the pc render images for the medical purposes.

Dont IBM have a Supercomputer ? Doesnt everyone have a Supercomputer ?


----------



## Disparia (Jun 1, 2008)

They have a supercomputer until someone else outdoes them


----------



## DrPepper (Jun 1, 2008)

Haytch said:


> I dont see anything super about it.
> Its actually a pretty crappy setup, except ofcourse the 4gfx cards.  I applaud CUDA being used to make the cards render images given to them for medical purposes, thats gold and very smart, but other then that, its crappy. Actually, its a total piece of shit.
> 
> Sorry, i get emotional    The only reason i call it a piece of shit computer is because someone actually thinks its SUPER . . .   Like i said, nothing super about the actual pc. The super part is the people that wrote the program to make the pc render images for the medical purposes.
> ...



"delivers roughly the same performance as a supercomputer cluster consisting of hundreds of PCs" If a single pc can do that imagine a cluster of them at roughly a fraction of the cost, if thats not super then your insane also CUDA means they can use it for almost anything a normal supercomputer can do e.g folding at hone or climate simulations etc etc


----------



## Bjorn_Of_Iceland (Jun 1, 2008)

That is one leet and neat build.


----------



## Weer (Jun 1, 2008)

Mussels said:


> faulty logic there. SLI allows two images onto one display, but the work they're doing isnt even 3D, nor does it need to be displayed. They're using them as number crunchers through CUDA, not as video cards.



How crude of you to insult my logic. I was simply mis-informed or un-knowledged.

However, I thank you for clearing that up, as it did present a form of interest to me.


----------



## Mussels (Jun 1, 2008)

Weer said:


> How crude of you to insult my logic. I was simply mis-informed or un-knowledged.
> 
> However, I thank you for clearing that up, as it did present a form of interest to me.



making assumptions without any empirical evidence or prior knowledge, fits under faulty logic to me. Dont take it personally, it wasnt intended that way.
Video cards work fine in the same system dual display without SLI/crossfire, and always have since the PCI days.


----------



## eidairaman1 (Jun 1, 2008)

ill settle it, both yall stop arguing and kiss and makeup.


----------



## Mussels (Jun 1, 2008)

eidairaman1 said:


> ill settle it, both yall stop arguing and kiss and makeup.


----------



## sumit_malwadkar (Jun 1, 2008)

*What a shame.......*

What a shame...... they have used AMD Phenom 9850 processor .
Insted of AMD Phenom 9850 processor if they have used Intel Core2Extreme they will surely get more peroformance from this system....


----------



## DrPepper (Jun 1, 2008)

Not really since that was only to control the gpu's and the flow of information, if it did they would have used a core 2


----------



## Morgoth (Jun 1, 2008)

sumit_malwadkar said:


> What a shame...... they have used AMD Phenom 9850 processor .
> Insted of AMD Phenom 9850 processor if they have used Intel Core2Extreme they will surely get more peroformance from this system....



and if the did that the would have a huge bottleneck from the FSB and ram
since amd phenom uses HT, HT gives allot more bandwith then FSB can ever offer


----------



## DrPepper (Jun 1, 2008)

Morgoth said:


> and if the did that the would have a huge bottleneck from the FSB and ram
> since amd phenom uses HT, HT gives allot more bandwith then FSB can ever offer



Cheers for explaining that


----------



## Mussels (Jun 1, 2008)

no the HT has absolutely nothing to do with it. they say in the article, and i quoted it earlier - the ONLY reason they went AMD, was because they couldnt find an Intel board with 4x PCI-E slots.


----------



## newtekie1 (Jun 1, 2008)

Mussels said:


> no the HT has absolutely nothing to do with it. they say in the article, and i quoted it earlier - the ONLY reason they went AMD, was because they couldnt find an Intel board with 4x PCI-E slots.



Correct, the CPU and FSB/HT probably sits idle 99% of the time in this setup.  They probably could have actually gotten away with using a single core processor with little to now loss in overall performance.


----------



## Disparia (Jun 1, 2008)

Fastra, version two 







Dual QC Xeon SBC / PCIe x16 backplane with enough slots for (8) 9800GX2's.


----------



## Morgoth (Jun 1, 2008)

emm skulltrail got 4 pci-e slots


----------



## newtekie1 (Jun 1, 2008)

Morgoth said:


> emm skulltrail got 4 pci-e slots



But the last two are too close together to allow 4 9800GX2's to fit.  MSI is the only company to make motherboards that have 4 PCI-E slots that all allow two slot cards.  If they wanted an Intel solution the P6N Diamond would have worked for them, but it only has PCI-E x16 1.1 slots, not 2.0, but I don't think it would have affected performance.


----------



## eidairaman1 (Jun 2, 2008)

also probably cost alot more at the time.


----------



## sumit_malwadkar (Jun 4, 2008)

Morgoth said:


> emm skulltrail got 4 pci-e slots


Yaa..... but it uses FB-DIMM which is quite slow compared to DDR2 & DDR3........(Means it is bottleneck to bandwith)


----------



## newtekie1 (Jun 4, 2008)

sumit_malwadkar said:


> Yaa..... but it uses FB-DIMM which is quite slow compared to DDR2 & DDR3........(Means it is bottleneck to bandwith)



FB-DIMMs aren't really the much slower and don't drastically affect performance and in this application it wouldn't affect performance at all as the system memory isn't used for the caculations.


----------



## Mussels (Jun 4, 2008)

newtekie1 said:


> FB-DIMMs aren't really the much slower and don't drastically affect performance and in this application it wouldn't affect performance at all as the system memory isn't used for the caculations.



it would drastically affect the price, however.

I am sure that memory and CPU power contributed something to this project, all we know is that the GPU's were the main power... we have no idea how important the ram performance truly is.


----------

