# Dual Xeon Cruncher Build



## [Ion] (Oct 31, 2014)

So I'm finally getting around to putting together something else interesting (no, the endless i5 HP SFFs don't count) so I might as well do a build log.  Parts are:

- 2x Xeon X5672 (3.2GHz + turbo, quad-core + HT, LGA1366)
- 6x2GB DDR3-10666 RAM.  More than enough for what I'm working on here
- 550w AcBel PSU from an old Sun workstation.  Not very modern of efficient, but with 375w available @ 12V, it should do just fine.  It's going to require a bit of modification, which I'll get to later
- 160GB HDD from Stinger608.  Thanks! 
- Supermicro X8DTi-F (almost certainly, just waiting to hear back if my offer on Ebay was accepted)
- nVidia Quadro 400
- 2x Supermicro heatpipe copper coolers
- Either Server 2008 R2 Enterprise or Linux Mint
- Caseless for now, but if I can find a decently-priced eATX case I might have to snap one up

First issue: the AcBel PSU only has a single 8-pin EPS12V.  However, I salved the EPS12V plug off of a past trashed PSU (perhaps the 300w Antec I burned out running the FX8350, but I don't remember).  There is a lab here with a soldering iron + supplies, however, that I believe is open to all electrical and computer engineering students.  I'm heading there tomorrow to see if I can use it, in which case I'll solder the extra EPS12V plug on.  If not I'll cut up a Molex->PCIe power adapter and tape them together.  That's not going to be my first choice, however





(and yup, that's my Model M; a damn fine keyboard!)



I'm still waiting on most of the parts; I have the RAM, GPU, and PSU, but that's it ATM.


----------



## t_ski (Oct 31, 2014)

subbed


----------



## [Ion] (Oct 31, 2014)

t_ski said:


> subbed


Thanks!  Now just to settle on which board I want to go with--seller on Ebay rejected my offer


----------



## t_ski (Oct 31, 2014)

I was very happy with the Supermicro boards I had:

http://www.newegg.com/Product/Product.aspx?Item=N82E16813182192

Some times you can find an open box deal for less than what they're listed for on the Egg.


----------



## [Ion] (Oct 31, 2014)

t_ski said:


> I was very happy with the Supermicro boards I had:
> 
> http://www.newegg.com/Product/Product.aspx?Item=N82E16813182192
> 
> Some times you can find an open box deal for less than what they're listed for on the Egg.


That's way, way above budget, especially when there are all sorts of boards available for ~$100-150 on Ebay.


----------



## t_ski (Oct 31, 2014)

Oh, I figured it would be, but if you can find one of them cheap, jump on it.  The best part is that they are regular ATX, so they fit in just about any case.


----------



## [Ion] (Oct 31, 2014)

t_ski said:


> Oh, I figured it would be, but if you can find one of them cheap, jump on it.  The best part is that they are regular ATX, so they fit in just about any case.


Yeah, regular ATX would be awesome so I could stick it in another old (and cheap) ATX mid-tower like the 2600k and 3770k, but what I'm planning on doing is putting the FX8350 in a case and then running the dual-Xeon caseless on the shelf where the FX8350 has been.


----------



## Norton (Oct 31, 2014)

This 550w XFX psu has 2x 8pin connectors for $60.99

http://www.newegg.com/Product/Product.aspx?Item=N82E16817207013

I wouldn't trust that old psu too much... may be good for toasting marshmallows though!







There is also an 8 pin EPS splitter available for $10.99
http://www.newegg.com/Product/Produ...012&cm_re=eps_splitter-_-12-706-012-_-Product

Will at least save you from a soldering and taping project


----------



## [Ion] (Oct 31, 2014)

Norton said:


> This 550w XFX psu has 2x 8pin connectors for $60.99
> 
> http://www.newegg.com/Product/Product.aspx?Item=N82E16817207013
> 
> ...


Well, the PSU was running a dual Socket 940 Opteron system until about a month ago and did fine there.  And it doesn't feel light and hollow like cheap PSUs--old it may be, but it's quite heavy as well.

Taping I really don't want to do--but soldering is fine.  I've soldered a bunch of things over the years (including soldering the power jack back together on my sister's laptop and the fan on my laptop) and I think that I can get a decent result.  I'll see what I can manage tomorrow and decide from there.  And since the DC draw should be just under 250w I'd imagine I'll be OK.  But that XFX PSU is quite cheap.


----------



## [Ion] (Oct 31, 2014)

Alright, PSU has been soldered and the two 8pin 12V plugs work (tested with the multimeter in the electronics lab at least).  It's not pretty, but it gets the job done.  I'll take a picture later today


----------



## OneMoar (Oct 31, 2014)

you could very well end up overdrawing the rail on the PSU
bad idea @[ion] never skimp on the psu and certainly never rewire one ... if the board requires two EPS8 that means double the AMPs you try and draw power for both processors over one EPS8 line and the results could be very bad 
get the xfx @Norton linked
if that old psu blows it will probably take everything with it


----------



## ThE_MaD_ShOt (Oct 31, 2014)

There's also the Evga 750 watt 80+ bronze for $60 @ the egg. Which happens to be the one I am going to pick up at some point soon for my dual 1366 rig. It is going to start life with a 650 watt unit.

http://www.newegg.com/Product/Product.aspx?Item=N82E16817438021



Also I learned my lesson with using old outdated psu's in crunching rigs when I took out the psu, gpu, and a mobo a couple years back with an old outdated 500 watt unit. Never again. All my rigs have very good psu's in them. (Corsair/ Antec  80+)


----------



## twilyth (Oct 31, 2014)

Personally, I'd be worried about the fire hazard.  I don't know how likely that is, but regardless, I wouldn't want to take the chance.  If anything ever happened, you'd get blamed for it - especially given how many rigs you have running.  I think you might want to slow down or even maybe take a step back.  It's one thing to be dedicated but there IS such a thing as going too far.


----------



## OneMoar (Oct 31, 2014)

twilyth said:


> Personally, I'd be worried about the fire hazard.  I don't know how likely that is, but regardless, I wouldn't want to take the chance.  If anything ever happened, you'd get blamed for it - especially given how many rigs you have running.  I think you might want to slow down or even maybe take a step back.  It's one thing to be dedicated but there IS such a thing as going too far.


the risk of fire is fairly low server grade PSU's have OCP in the event of a dead short it will simply power off


----------



## [Ion] (Oct 31, 2014)

Alright, so here's what I did:




I took the new 8pin and split its wires in half; half are spliced into the existing 8pin plug and the other half are hooked up to two Molex plugs.  The PSU has three 12V rails; a 16A, a 17A, and a 8A.  I'm assuming that one of the 16A or 17A is going to the existing 8pin.  16A would be barely enough for two CPUs (16x12 = 192W, two CPUs = 190W).  But running a CPU and a half on 16A seems fine, and I can run the other half of the CPU off of the Molex plugs.

It seems like this should be fine; I've run a pair of GTX470s and an OCed i7 on a 600w before (albeit, a new one)--and this is certainly far less power consumption than that setup.  The GTX470s are approx 150W each folding, and the CPU was ~100W (4.4GHz Sandy), so that's 400W DC for the CPUs+GPU; this will be ~190W tops.


----------



## ThE_MaD_ShOt (Oct 31, 2014)

Hey Ion a little side thing I use heat shrink instead of tape. Cleaner and safer then tape.


----------



## [Ion] (Oct 31, 2014)

ThE_MaD_ShOt said:


> Hey Ion a little side thing I use heat shrink instead of tape. Cleaner and safer then tape.


Yeah heat shrink would have been better, but all I had was electrical tape.  Electrical tape + cable ties (got a few more to put on) should get the job done.
Does this really seem like such a horrible way of doing things?  Obviously I don't want a fire, but I also don't want to pay for a new PSU...


----------



## ThE_MaD_ShOt (Oct 31, 2014)

For me I would at the very least go back and use heat shrink. Tape seems to lossen up and fall off with heat. When I am wiring cars up I always use solder (which you did ) and heat shrink. You do not want any chance of a short in there as it can take out the whole rig. Your work doesn't seem botched. I just tend to be overly cautious on stuff. Hence the reason I err to the side of over kill with psu's


----------



## [Ion] (Oct 31, 2014)

ThE_MaD_ShOt said:


> For me I would at the very least go back and use heat shrink. Tape seems to lossen up and fall off with heat. When I am wiring cars up I always use solder (which you did ) and heat shrink. You do not want any chance of a short in there as it can take out the whole rig. Your work doesn't seem botched. I just tend to be overly cautious on stuff. Hence the reason I err to the side of over kill with psu's


Alright, well that makes me feel better.  I think what I'll do is put on extra cable ties to make sure that the tape doesn't come off--two cable ties over each piece of tape ought to do a solid enough job--and then I'll pick up heat shrink (almost certain I have some at home at my parents house) over winter break and heatshrink it then.  Lots of cable ties ought to do a sufficient job if I keep an eye on it for the short term.

Also, I think I"ll plug it into my FX8350 + 8800GTS system and run it like that for a bit, that ought to give me an idea of how it'll do.  The FX8350 + G92 GTS ought to be similar power draw to the two Xeons.


----------



## OneMoar (Oct 31, 2014)

how are you not blowing every breaker in your house [ION]
If my math is correct you are close to pulling over 80AMP's on all your systems


----------



## [Ion] (Oct 31, 2014)

OneMoar said:


> how are you not blowing every breaker in your house [ION]
> If my math is correct you are close to pulling over 80AMP's on all your systems


I dunno how you get 80 amps....no way it's close to that.  Running on a single 20A breaker, I have: Xeon X5670 system, 2720QM system, OCed 3930k system, AMD 4P, ULV i7, OCed FX8350, OCed 2600k, 3770k, Atom, two i5-2400s, and a 4702QM.  How that all is less than 20A I don't know, but apparently it is.
On a separate 20A circuit is another i5-2400, an i7-860, an i7-4770.  The i5-520m is on yet another circuit.  As is the i7-920.  The i5-4690k is on another circuit as well.


----------



## OneMoar (Oct 31, 2014)

at 100% load using a low avg peak of ~8 amps per machine thats what I came up with lol
so either your machines draw well under 8 amps while crunching or the outlet is a lot beefier then you think it is ..
even at 3 amps per machine witch is really unrealistically low that's still 50AMP's
*counting all the machines you have listed in your sig


----------



## [Ion] (Oct 31, 2014)

OneMoar said:


> at 100% load using a low avg peak of ~8 amps per machine thats what I came up with lol
> so either your machines draw well under 8 amps while crunching or the outlet is a lot beefier then you think it is ..
> even at 3 amps per machine witch is really unrealistically low that's still 50AMP's


No way it's 8A.  Absolutely no way.  For most of them, it's not even 3A.  The three mobile i7 systems are less than 1A AC each.  The i5s are about 125W AC (HP builds them using 80+ Bronze PSUs, and there's barely anything in them other than a CPU).  The 3930k system is probably about 3A (don't remember exactly), and the AMD 4P is very approximately 4A.  The other assorted systems (X5670, FX8350, 2600k, 3770k, 860, 4770) are all 2A or just barely over.


----------



## thebluebumblebee (Oct 31, 2014)

@OneMoar , are you forgetting to convert between 12v and 120V?  8 amps at 120 volts is 960 watts (if I remember my electrical math correctly)


----------



## OneMoar (Oct 31, 2014)

I was talking at the wall ac-draw


----------



## [Ion] (Nov 6, 2014)

Alright, some project updates!

CPU coolers arrived, they're 1U Supermicro passive coolers:





Also, a pair of 92mm fans pulled from broken Celeron HPs, which fit almost perfectly.  No idea how I'm gonna attach the fans, however 
Suggestions?





Also, not planned, but I decided to buy a case! 




It's gaudy, flimsy, and huge.  _But_, it fits an eATX motherboard (at least when I add in the extra standoffs) and was $5.  Hard to argue with a price like that.  It also has none of the hard drive trays and two of the 5.25" bays are occupied by IDE optical drives, but that's not relevant 
It's an Apevia X-Jupiter


----------



## twilyth (Nov 6, 2014)

I recommend twist ties for the fans.  Strip the covering to get the wire.  Thread the wire between the heatfins and up through the mounting holes.  If you don't have really long twist ties lying about, you can get a continuous roll of them for a buck at most dollar stores.  Or you can get it at any hardware store.


----------



## [Ion] (Nov 6, 2014)

twilyth said:


> I recommend twist ties for the fans.  Strip the covering to get the wire.  Thread the wire between the heatfins and up through the mounting holes.  If you don't have really long twist ties lying about, you can get a continuous roll of them for a buck at most dollar stores.  Or you can get it at any hardware store.


Oh that's clever!  I unfortunately don't have any, but that's easy enough to acquire.  I'm pretty sure there's a dollar store within walking distance of campus


----------



## twilyth (Nov 6, 2014)

I can't guarantee that they'll have it, but it tends to be a common item - around here anyway.  You tend to get the really cheesy stuff with very thin wire, but that's probably better anyway since it's easier to work with.  It just might be a little more difficult to strip since if the wire is really thin, you can sometimes pull it apart w/o a lot of effort.


----------



## [Ion] (Nov 6, 2014)

twilyth said:


> I can't guarantee that they'll have it, but it tends to be a common item - around here anyway.  You tend to get the really cheesy stuff with very thin wire, but that's probably better anyway since it's easier to work with.  It just might be a little more difficult to strip since if the wire is really thin, you can sometimes pull it apart w/o a lot of effort.


Alternatively, I could take one of the long-ish wires from a PSU an cut off all of the insulation, and just use the individual strands of wire inside...in fact, that would be even easier, given that I have an already cut-up PSU laying around (the broken one I took the extra 8pin from).


----------



## t_ski (Nov 6, 2014)

I would just use some plastic zip ties and loop them under the mounting screws.  Use a couple together if you don't have ones that are long enough, and you probably only need ones on two opposite corners out of four.


----------



## [Ion] (Nov 6, 2014)

t_ski said:


> I would just use some plastic zip ties and loop them under the mounting screws.  Use a couple together if you don't have ones that are long enough, and you probably only need ones on two opposite corners out of four.


Regrettably I don't have any more


----------



## Toothless (Nov 6, 2014)

[Ion] said:


> Regrettably I don't have any more


I have about 200 of assorted sizes. Want some?


----------



## t_ski (Nov 6, 2014)

I could send you some, but you could pick up a small pack for a few $$ at Auto Zone or the like.

Are you going to end up with any leftover S1366 Xeons?


----------



## [Ion] (Nov 6, 2014)

t_ski said:


> I could send you some, but you could pick up a small pack for a few $$ at Auto Zone or the like.
> 
> Are you going to end up with any leftover S1366 Xeons?


I can pick up some from my parents house when I go home for Thanksgiving break--my Dad has a whole box of them and I'm sure I could get a few there.  Wire ought to do for now so long as I'm careful.

I have a single W3503--but it's a pretty horrible CPU.  That's the only one that will be leftover.


----------



## t_ski (Nov 6, 2014)

[Ion] said:


> I have a single W3503--but it's a pretty horrible CPU.  That's the only one that will be leftover.


Yep, that's worse than the E5506 I've got at the moment.


----------



## Toothless (Nov 6, 2014)

If you really need a bunch at once. I gotta say that this place is the best I've ever been to.

http://www.pchcables.com/nyloncableties.html

Amazing service and cheap! Their cable ties still work after two years inside a desktop case. (24/7 operation with lots of heat)


----------



## [Ion] (Nov 6, 2014)

Lightbulbie said:


> If you really need a bunch at once. I gotta say that this place is the best I've ever been to.
> 
> http://www.pchcables.com/nyloncableties.html
> 
> Amazing service and cheap! Their cable ties still work after two years inside a desktop case. (24/7 operation with lots of heat)


I'll just loot some of my Dad's, it'll be fine 


t_ski said:


> Yep, that's worse than the E5506 I've got at the moment.


Yup, they go for like $6 on Ebay.  It was the original CPU in the X5670 system I have; hence why it was so cheap, but also why I was so determined to replace it ASAP.


----------



## Constantine Yevseyev (Nov 6, 2014)

[Ion] said:


> Either Server 2008 R2 Enterprise or Linux Mint


I'm not familiar with the topic ("crunching" it is), but I wouldn't recommend you or anybody else using a Linux distributive meant for end-users as a 24/7 sorta system. What you should try is something more stabilized and approved for server use (Scientific Linux or any other RHEL7-based solution, there's also Debian Stable). The packages that you get are a little bit old, but most of the important things (like Kernel changes, especially task management and networking features) are being back-ported anyway.
Windows Server 2008 R2 sounds fine if you already have a license. But 2012 R2 Standard should actually cost the same, so...


----------



## ThE_MaD_ShOt (Nov 6, 2014)

Constantine Yevseyev said:


> I'm not familiar with the topic ("crunching" it is), but I wouldn't recommend you or anybody else using a Linux distributive meant for end-users as a 24/7 sorta system. What you should try is something more stabilized and approved for server use (Scientific Linux or any other RHEL7-based solution, there's also Debian Stable). The packages that you get are a little bit old, but most of the important things (like Kernel changes, especially task management and networking features) are being back-ported anyway.
> Windows Server 2008 R2 sounds fine if you already have a license. But 2012 R2 Standard should actually cost the same, so...


Ummm been using ubuntu for a couple years now with out a single issue. It has been great to us. But to each is there own.


----------



## [Ion] (Nov 6, 2014)

Constantine Yevseyev said:


> I'm not familiar with the topic ("crunching" it is), but I wouldn't recommend you or anybody else using a Linux distributive meant for end-users as a 24/7 sorta system. What you should try is something more stabilized and approved for server use (Scientific Linux or any other RHEL7-based solution, there's also Debian Stable). The packages that you get are a little bit old, but most of the important things (like Kernel changes, especially task management and networking features) are being back-ported anyway.
> Windows Server 2008 R2 sounds fine if you already have a license. But 2012 R2 Standard should actually cost the same, so...


Well, I don't have a license, but that hasn't stopped me in the past 


ThE_MaD_ShOt said:


> Ummm been using ubuntu for a couple years now with out a single issue. It has been great to us. But to each is there own.


Agreed with Mad, I've yet to see an issue, aside from my SSD wearing out...


----------



## ThE_MaD_ShOt (Nov 6, 2014)

[Ion] said:


> Well, I don't have a license, but that hasn't stopped me in the past
> 
> Agreed with Mad, I've yet to see an issue, aside from my SSD wearing out...




I use platter drives in my crunchers for that reason.


----------



## [Ion] (Nov 6, 2014)

ThE_MaD_ShOt said:


> I use platter drives in my crunchers for that reason.


Yeah, I need to clone the SSD in the quad Opty system over to another platter drive--although admittedly the SSD that it burned out was an ancient and cheap 40GB one.  The Crucial drive in there ought to do just fine until I can buy a 40GB or 80GB drive to replace it.  But everything else is using platter drives (lots cheaper, too!)


----------



## BUCK NASTY (Nov 6, 2014)

[Ion] said:


> Well, I don't have a license, but that hasn't stopped me in the past



I'm not hearing any of this, LOL. Carry on....

I remember being challenged in 2008 to find a "creative way" to populate Windoz XP on my 3rd Folding Rig. It opened up a whole new world to me.


----------



## [Ion] (Nov 6, 2014)

BUCK NASTY said:


> I'm not hearing any of this, LOL. Carry on....
> 
> I remember being challenged in 2008 to find a "creative way" to populate Windoz XP on my 3rd Folding Rig. It opened up a whole new world to me.



Still not sure what I'm going to go with; Linux gives somewhat better PPD but Windows is easier to administer (and more forgiving).  But the CPUs/board aren't here yet, so I still have some time to decide

Well, CPUs and board are now delivered, but apparently the school hasn't sorted them yet, so I can't pick them up


----------



## m&m's (Nov 6, 2014)

[Ion] said:


> Still not sure what I'm going to go with; Linux gives somewhat better PPD but Windows is easier to administer (and more forgiving).


I don't want to influence you but, GO WITH *LINUX*!
Are you going to average *100,000PPD* with this new rig added to the fleet?


----------



## [Ion] (Nov 6, 2014)

m&m's said:


> I don't want to influence you but, GO WITH *LINUX*!
> Are you going to average *100,000PPD* with this new rig added to the fleet?


Regrettably not--looks like I'll probably be at just over 90k or so.  But that's still pretty cool IMO 
And it means that if I was to get everything else I have at home and bring it to school to crunch on, I'd be at about 100k....possibilities for the future 
I'm leaning towards Mint--but why do you feel so strongly about it?


----------



## m&m's (Nov 6, 2014)

[Ion] said:


> Regrettably not--looks like I'll probably be at just over 90k or so.  But that's still pretty cool IMO
> And it means that if I was to get everything else I have at home and bring it to school to crunch on, I'd be at about 100k....possibilities for the future
> I'm leaning towards Mint--but why do you feel so strongly about it?



Some years ago, there was Windows guys and Linux guys, like Oil and Water they could not mix
There was Script Kiddies and Hackers
You can press a button or know what the button does

+1 Linux and it gives moar points

But I like Windows, I use it everyday.


----------



## [Ion] (Nov 6, 2014)

Two pictures of the build!
PSU installed and HDD test placement:





Everything set up, system running in BIOS:





The fans are attached to the heatsinks via rubber bands.  Far from ideal, however, I was unable to get the wire tight enough to hold the fans anywhere near on properly.  Given that rubber bands tend to dryrot quickly, I'll need to replace them--likely with cable ties after I'm able to get some from home.  But for the time being, it seems to work fine.

I decided to go with 6x2GB DDR3 RAM; more than is necessary, but it lets me run triple-channel on each CPU, so that's cool.  I may pull some out later, we'll see.

I'm installing Server 2008 R2 right now.  Ubuntu was showing a maximum CPU speed of 2.4GHz, which is far below what they should be doing (3.2GHz minimum + turbo to 3.46 on all cores).  I know my way around Windows better, so I'm going to see if I can sort the issue out there.


----------

