12-core Xeon engineering workstation build

[8BitZ80]

Can you raise the BLCK on that mobo? If so, 103.5 has worked well for the E5 V3's and V4's I've tried in X99. The V3's max out at 2133 and the V4's at 2400 by default.

There's some options called "Overclocking Features" in the bios but I haven't looked at them yet. I wouldn't get my hopes up because it's an American Megatrends bios and the options are pretty basic.

Edit: Yeah there's a few options in there for voltages and offsets, but nothing BCLK related.

 

[storm-chaser]

33x on two cores, well there you go. I must be confusing it with the 32x multiplier that everyone runs these chips at on all cores with the turbo hack.



Yeah the grey slots are DDR3 and the black slots are DDR4. The stickers show which Xeon chips have a DDR3 memory controller, although the E5-2678 V3 is the only affordable one. I believe the E5-2680 V3 (which is almost identical) doesn't have a DDR3 controller. Most of the Haswell-E chips out there are only DDR4 compatible, like my E5-2630L V3 that I also tried (which worked just fine on DDR4).

Depending on whether you go DDR3 or DDR4, you can get quad channel using either configuration.

That did look like 4 slots on either side of the CPU socket. Sick.
Maybe you already answered this, but why are you opting for the slower of the two, DDR3? Budget constraints? I imagine the DDR4 stuff is quite a bit more expensive, but on the flip side, it sounds like you cant go ECC after all on that motherboard?

Its a catch 22 and but I wouldn't worry about ECC. Instead, go to 1866MHz DDR3 (or as high as you can find) with tightest timings you can find.
Otherwise you can do the same with DDR4 but it will get a little pricey. And I suppose you can always upgrade to DDR4 down the road, seeing as how your CPU choice is very versatile.

Another issue with ECC that gets on my nerves is the fact that ECC memory has VERY conservative timings. Meaning running ECC may effect your latency or performance. Very difficult to find high performance ECC memory and its usually 1-3 back in CAS from it's "retail" cousins in the mainstream market.

 

[8BitZ80]

That did look like 4 slots on either side of the CPU socket. Sick.
Maybe you already answered this, but why are you opting for the slower of the two, DDR3? Budget constraints? I imagine the DDR4 stuff is quite a bit more expensive, but on the flip side, it sounds like you cant go ECC after all on that motherboard?

Its a catch 22 and but I wouldn't worry about ECC. Instead, go to 1866MHz DDR3 (or as high as you can find) with tightest timings you can find.
Otherwise you can do the same with DDR4 but it will get a little pricey. And I suppose you can always upgrade to DDR4 down the road, seeing as how your CPU choice is very versatile.

Another issue with ECC that gets on my nerves is the fact that ECC memory has VERY conservative timings. Meaning running ECC may effect your latency or performance. Very difficult to find high performance ECC memory and its usually 1-3 back in CAS from it's "retail" cousins in the mainstream market.

Yeah the DDR3 ECC registered memory is by far the cheapest option because it's no longer in demand in enterprise. Really all I'm looking for is plenty of capacity and good reliability rather than chasing outright performance. Maybe DDR5 ECC server memory will come along eventually and bring the prices of DDR4 down, but that's still a long way off at this point.

This stuff I'm currently running is 11-11-11-28. I think a CAS latency of 11 is normally what you'd have on DDR3 2133MHz, so yeah the timings are veeeeery loose for 1600MHz lol Not too worried about it though.

I got some Userbench results to gauge how the system is performing:

 

[8BitZ80]

Now it's starting to look much tidier after doing a bunch of work to properly organize all the cables. Still need to get rid of that ugly yellow sticker on the fan cable for the CPU heatsink.

I may end up ditching the white LED strip because you can't see much through the side panel when it's installed anyway. The Huananzhi motherboard apparently has some greenish-blueish LEDs underneath the plastic shroud, which you can sort of see in the bottom picture.

I've been using the rig for a few hours on an old Windows 8 install and it runs great! The onboard audio is surprisingly decent. No reason why this couldn't be used as a daily driver machine for gaming and office work.

 

[8BitZ80]

New installation of Windows 10 and added a PCI-E wifi adapter card for internet. Funnily enough I had to install the wifi adapter in the lower PCI-E x1 slot on the board because the white Huananzhi PCB shroud is too thick and prevents my particular Asus card from being fully seated in the highest one. A bit of an odd quirk.

One of the things I would like to change about this Huananzhi board is the stupid little internal speaker that beeps on POST. It's soldered onto the board and there's no way to shut it up. I might need to de-solder it.

Device Manager lists what seems to be 63 unrecognized PCI devices called "Base System Device", plus some others. No idea what they are, but all the drivers are installed (the motherboard and wireless card are plug-and-play).

Spoiler

Nothing unusual going on in the Event Viewer though and the system is perfectly stable.

I've been using this system to get a bit of experience with FreeCAD and Blender for rendering assets. Idle power consumption is around 55 watts and it's up around 190 watts during an all-core load. So it's not a power-hungry machine with those 12 cores. Load temperatures don't go past 55C. The Quadro K620 is too weak for gaming but it's good for productivity. I'm yet to see how many years this Huananzhi board will last but so far it's proving to be refined enough to be used in a daily driver build.

 

[jboydgolfer]

One of these days I need a better tool to install standoffs

Cool project

I have a multi bit screw driver that fits right over most standoffs ive encountered. The drivers are not standardized Afaik, but it works great if it fits. With no bit in the driver, put it on the standoff & its a breeze

 

[storm-chaser]

One of the things I would like to change about this Huananzhi board is the stupid little internal speaker that beeps on POST. It's soldered onto the board and there's no way to shut it up. I might need to de-solder it.

Pull it clean off the board with a vice grip.

 

[storm-chaser]

Now it's starting to look much tidier after doing a bunch of work to properly organize all the cables. Still need to get rid of that ugly yellow sticker on the fan cable for the CPU heatsink.

I may end up ditching the white LED strip because you can't see much through the side panel when it's installed anyway. The Huananzhi motherboard apparently has some greenish-blueish LEDs underneath the plastic shroud, which you can sort of see in the bottom picture.

View attachment 189802

View attachment 189803

I've been using the rig for a few hours on an old Windows 8 install and it runs great! The onboard audio is surprisingly decent. No reason why this couldn't be used as a daily driver machine for gaming and office work.

Im liking this case you've got for this project. You say it has two 5.25" drive bays? Any more that are hidden? I need a case with at least 4 drive bays as I am running two reservoirs in my custom loop 9600KF.... the price is definitely right and it's definitely sleeper status.

 

[8BitZ80]

Im liking this case you've got for this project. You say it has two 5.25" drive bays? Any more that are hidden? I need a case with at least 4 drive bays as I am running two reservoirs in my custom loop 9600KF.... the price is definitely right and it's definitely sleeper status.

Yeah it's a decent case, I got it on sale. It has an internal enclosure for two 5.25" bays and you can mount other stuff to the removable brace on the side. See what you think.

 

[8BitZ80]

Pull it clean off the board with a vice grip.

That was a good recommendation, thanks lol The little squawker has been silenced.

All it needed was a pair of pliers and with a bit of leverage action you can pull those electrical pins out pretty easily. It didn't even leave a scratch on the board!

 

[storm-chaser]

those pesky little motherboard speakers.. they always get silenced if they come into my domain lol

 

[8BitZ80]

I tried another completely different kit of DDR3 ECC REG memory on the off-chance that the ECC functionality would work, but I'm not surprised that it doesn't. So unless there's some kind of ECC functionality working that cannot be detected by any software, I'm calling it that the Huananzhi X99-TF board doesn't support error-correcting. HOWEVER, at least the memory itself works and that's a good thing when DDR3 ECC registered memory is so cheap. This 4x4GB DDR3 1333MHz kit was about US$22.

At least I got some spare memory now. The black heatspreaders look good on this Micron kit. It may be possible to transfer them onto the Samsung modules, but I'm not interested in ripping things apart just for some better aesthetics in a windowless case.

 

[jaggerwild]

For years now I've had some ideas brewing in my mind of what I'd like to use for an engineering workstation and now I finally have the parts at my disposal to make it happen. This is going to be a very interesting one.

NOTE: This build definitely is NOT how you SHOULD build a workstation. But instead for personal use it will be plenty powerful enough for my own purposes.

There's a couple of stipulations I had for this build:

1. CPU must be a Xeon on a relatively modern architecture (less than 10 years old) with lots of cores and cache.
2. ECC registered memory, something I've never personally used and would be great to have.
3. Quadro graphics card that is powerful enough to play a game or two.

For those who want to get straight into the specifications list, here you go:

Spoiler: Xeon Build Specifications

• Intel Xeon E5-2678 V3 X99 12-core Haswell processor
• Coolermaster Hyper T4 cooler
• Huananzhi X99-TF motherboard
• 4x8GB Samsung DDR3 ECC 1600MHz dual rank PC3L memory
• Nvidia Quadro K620 2GB graphics card
• Samsung 850 Evo 250GB SSD
• 1TB 7200rpm hard drive?
• Fractal Design Core 2300 case
• Zalman ZM-700SV power supply

I've already done some initial testing to make sure nothing is DOA. Everything is working and good to go!

View attachment 189423

More to come.

Weird how that board even runs with no battery in it? Usually that borkes a board right away.....

 

[8BitZ80]

Weird how that board even runs with no battery in it? Usually that borkes a board right away.....

I didn't know that normally screws things up. But yeah, I was actually running tests on the board for a few days with no CMOS battery. The only difference really was it doesn't remember any BIOS settings (which is what the battery is obviously for). The Huananzhi boards don't ship with a CR2032 battery.

 

[8BitZ80]

Changed the CPU heatsink orientation to a rear exhaust.

One of the things that strikes me about this system is that it's supremely quiet under load. I mean the 12-core maxes out in the mid-50C range, so the fan doesn't even ramp up while I'm stress testing it. It's got a grand total of six fans running at all times (PSU, GPU, CPU, case and 2 x mobo) but even at full load it's almost as quiet as my Ryzen system is at idle. Fans ramping up can get very annoying and so really this kind of quiet operation is ideal for a workstation.

 

[storm-chaser]

Sounds like you have plenty of fans to keep it cool. Is there a provision on that case for a 140mm fan above the CPU?

Are all your case/mobo/CPU fans PWM? What I've found with my rad and CPU fans is that there is a limit to effectiveness, in other words, above a certain RPM, you get diminishing returns in terms of cooling effectiveness. For example, I have all my fans set to a maximum of 40%, which is sufficient enough to keep my 5.0GHz 9600KF under 180*F under full load.

 

[8BitZ80]

Sounds like you have plenty of fans to keep it cool. Is there a provision on that case for a 140mm fan above the CPU?

Are all your case/mobo/CPU fans PWM? What I've found with my rad and CPU fans is that there is a limit to effectiveness, in other words, above a certain RPM, you get diminishing returns in terms of cooling effectiveness. For example, I have all my fans set to a maximum of 40%, which is sufficient enough to keep my 5.0GHz 9600KF under 180*F under full load.

There's provisions for 2 x 140mm fans on top but there's 120mm mounting holes as well.

The only PWM fan in the system is the CPU fan, no custom fan curves though. Not sure what speeds they're running at so I'll need to check next time I'm on there. They're just running at whatever the default is.

 

[storm-chaser]

There's provisions for 2 x 140mm fans on top but there's 120mm mounting holes as well.

The only PWM fan in the system is the CPU fan, no custom fan curves though. Not sure what speeds they're running at so I'll need to check next time I'm on there. They're just running at whatever the default is.

Maybe you have already addressed this and I missed it... does this board allow you to modify the base clock? How are the overclock options overall? Are you going to be OC this rig?

 

[ThrashZone]

Call me eccentric, but something about the Haswell-E and Broadwell-E IHS just rubs me the right way, and no other CPU since then has done quite the same

Hi,
Probably the 44 lanes tickling you

 

[Caring1]

One of the things I would like to change about this Huananzhi board is the stupid little internal speaker that beeps on POST. It's soldered onto the board and there's no way to shut it up. I might need to de-solder it.

Usually there's a Bios setting to have that on or off at startup.

 

[8BitZ80]

Maybe you have already addressed this and I missed it... does this board allow you to modify the base clock? How are the overclock options overall? Are you going to be OC this rig?

It's pretty powerful as it is so I probably won't overclock it, but there are some settings in there for voltages, power states, that kind of stuff. I don't recall seeing anything labelled for BCLK, but I haven't looked very hard either.

Edit: Just double checked and there's definitely no BCLK adjustment. Ah well.

Usually there's a Bios setting to have that on or off at startup.

Yeah I looked for a setting that would do that and I couldn't find anything. It beeps as soon as you hit the power button so I doubt it would have that capability of turning it off.

 

[storm-chaser]

Can you run a quick CPUz bench? I'm curious how this chip does in ST / MT

 

[8BitZ80]

Can you run a quick CPUz bench? I'm curious how this chip does in ST / MT

I did a CPU-Z run on the first page (post #19), but here's a re-run from just now:

Performance has gone down a little bit although I've seen better numbers recently. It might be because I've got Spectre/Meltdown protection enabled right now. Cinebench R15 still performs good at around 1713cb multi core and 128cb single core.

 

[8BitZ80]

New power supply installed. Not that the Zalman unit was bad, in fact this is more of a downgrade in terms of overall wattage.

These Be Quiet! System Power U9 400W 80+ Bronze units were being sold at US$35 each (AUD converted to USD) and it was such a deal that I bought four of them. They're cheap but far from being bad quality. Ripple is around 10 - 20mV up to 80% load and it's based on a modified CWT platform using good quality Elite capacitors.

Running both the Xeon CPU and Quadro GPU at 100% load only consumes 220 watts in this system, so having 384 watts available on the 12V rails is more than enough to allow for a bit of future expansion.

 

[8BitZ80]

The Huananzhi board is starting to show a sign of weakness. Nothing major, but one of the dual VRM heatsink fans (or potentially both) is starting to make some intermittent noise. It almost sounds like a mouse scratching away in the ceiling, except it's inside the computer. I checked all the fans and it's definitely the VRM heatsink fans making the noise. Possibly a speck of dust has contaminated a fan bearing and it's being chewed up in there.

I was worried for a second that the motherboard was frying itself after the power supply was changed, but that's not the case at all. Just a random coincidence.

I do conveniently have a spare Noctua NF-A6x25 PWM 60mm fan that I could mount in there and hook up to the CPU #2 fan header. It would actually be a worthwhile change considering that it would make the system quieter and more reliable anyway. Those VRM fans look like cheap junk and I don't expect them to last very long.