192 Core AMD Opteron system + 4 Intel dual-socket servers + 4-8 GPUs

[storm-chaser]

They are going to have to bring another nuclear reactor online to cope with this level of power demand

House was recently wired for 200 amp service (and the previous owner was an electrician, and so am I in my spare time, so we will have our ducks in a row in terms of power delivery) , and I've been wanting to do something like this for quite a while... so we are in a good place to now execute the project and move from the planning phase to the implementation phase. Especially now that we are all in a "lock down" situation, this project will continue to protect me from boredom and isolation, and now I can share it with you guys and track my progress. Quietly in the background, I have been massing RAM, hardware, networking gear, processors and servers and other equipment for this project. We are getting closer to realizing the dream!

Before you say how horrible the performance is going to be, please keep in mind that is not the chief reason for this build. That should be pretty obvious. We aren't measuring this projects performance relative to new hardware metrics. This is a sentimental build, because I am using the Opteron 6180 SE, AMDs 12 core "best in class" CPU based on the K10 architecture. As you may or may not be aware, I ran a Phenom II X6 chip for many years, much longer than most PC enthusiasts, and it was my primary rig up until about a year ago. That being said, the Phenom II left quite an impression on me and quickly became my favorite CPU of all time. Partially due to its rock solid performance, snappy responsiveness and low latency, but also because the platform was pretty well flushed out at that point, had DDR3 1T 1600 MHz support, and it was the only chip I've ever had that allowed me to unlock additional cores and take advantage of "hidden power". It was also AMDs first chip to see substantial increases in memory throughput (and performance improvements in level 3 cache as well), from overclocking the memory controller (CPU NB). All these attributes, intricate details and fine tuning have put the Phenom II very close to my heart.

And look guys, please take this into account, when I say this is a sentimental build that does not mean that we wont or cannot have an emphasis on performance. No, in truth it is actually quite the opposite. Some like to make assumptions and measure other people's hardware projects by current tech performance metrics and hold whatever they are working on "hostage" to the false comparison against recent / newly released tech and then take that as the gold standard. NO. The benchmark standard here is tech from the same class / era / epoch. We are competing against LIKE HARDWARE. That is my primary and ONLY metric in determining overall performance. We will not hold this technology hostage to a false standard, and I want to emphasize that point so you know my rational behind creating this beast.

Well, I've had my fun with the consumer grade chip. It is now time to shift gears and re-visit the K10 architecture in a server environment. Like a magnet, that quest lead me to the 6180 SE, AMD's 12 core K10 variant, a best in class processor. This is a 140W CPU that is essentially two Thuban X6 Phenom II processors in one, hence the 12 core delineation. Now... these servers are unique in their power to weight ratio. What I mean by that is at the time of their release, there was no other server that packed the same punch into a 2U chassis as the Dell PowerEdge C6145. And that's what we've got to work with. Two Dell PowerEdge C6145s. These servers offered best in class performance back in the day. And what's interesting is the C6145 is essentially two servers in one, in other words, two nodes, a little bit like a blade server. These servers have two individual motherboards and can support up to four CPUs on each board and Dell has managed to incorporate high performance components into a very condensed package. High density and massive throughput, by sharing the remaining infrastructure betwixt both nodes. That gives us a total of 8 Opteron 6180 SE processors PER SERVER to work with. And like I said, we have two Dell C6145s to work with here. 96 Cores was just not going to cut it, so we doubled our potential with the purchase of the second PowerEdge C6145. The total core count is now up to 192.

Now, I have lots of data I want to share with you guys but I need to parcel it out in segments... there is a lot going on here and I'll do my best to document the whole project clearly and concisely... but first we have to get the hardware on the table so you know what I am working with. So this is a start. I will go into more detail and reveal my overarching goals for this system a little bit later today. Hope you are well. Stay happy and heathy out there guys!

All the best to my fellow compatriots. And I mean that. You guys are great. I just want you all to know, this one is brought to you courtesy of the red white and blue. (Thanks Toby!)

 

[Toothless]

I like, big, servers and I cannot lie

Enjoy the build! There's a few things I have and will keep purely out of personal value (even when I'm told it's worthless) and honestly it's those builds that are the most fun. Keep us updated!

 

[storm-chaser]

So I spent hours and hours on amazon last night ordering more equipment for this project (we are getting closer!!!). This is what I just ordered:

- Two 1100W 110v PSUs (Both C6145s come with 220v PSUs only)
-Dell C4130 GPU server (dual E5 v3 procs)
- Five Sata III 512GB Solid State drives. These will serve as OS drives ONLY on each server that we throw in the mix. I also have more SSDs at the house here.
-Including the recently purchased PowerEdge C4130, the server count involved in this project is up to 5.
-1.8" msata/usata SSD adapter to the standard 2.5" SATA connection. This is because the PowerEdge C4130 has provisions for two 1.8" msata SSD boot drives, but I've only got 2.5" sata 6GB/s SSDs in stock at the moment. This will bridge the gap and allow me to run my existing 2.5" SSDs for booting in the GPU server.

Currently shopping for a half sized server rack since we are working with a large volume of hardware/servers here. Also the weight is very substantial so I need something strong, cant just piece this together.

We have:
*C4130 GPU host server (dual socket) (SSD powered) 16GB RAM (Handles up to 4 300W TDP GPUS)
*C6145 Server #1a (Quad socket) (SSD powered OS)
*C6145 Server #1b (Quad socket) (SSD powered OS)
*C6145 Server #2a (Quad socket) (SSD powered OS)
*C6145 Server #2b (Quad socket) (SSD powered OS)
*HP ProLiant DL360p gen 8 (dual socket) (SSD powered) 64GB RAM
*HP Microserver Gen10. (SSD powered) (We now have 3TB drives to populate all for slots for storage... t. Also upgraded the unit to 16GB of DDR4 RAM earlier today.

Not to mention the second dual proc HP z820 that I can throw into the mix if needed...

Next step is going to be tackling the issue of RAM (ideally I want to populate all 128 slots for max memory bandwidth in both C5145 Servers. Hard to fathom this level of performance. But, the Dell PowerEdge C6145 is capable of huge bandwidth numbers, with an effective 16 channels of DDR3 1600MHz RAM. And Im not talking as a whole, I'm saying 16 channels per node. We have 4 nodes in total.

Phase after that will be acquiring 4 GPUs to use in conjunction with the C4130 GPU server.

Matter of fact if you took an assessment of my current hardware you would quickly realize the majority of my rigs have more than one processor.

Also good to know my C4130 PowerEdge is rated for use in the upper stratosphere. I was really worried the SR-71 would steal my thunder, but truth be told, Dell did a fantastic job designing this machine with huge margins in mind. The C4130 stands in a class of its own, thats for sure.

 

[storm-chaser]

To give you guys an idea of some of the hardware we will be working with, take a look at this. This will be our GPU server. I will be outlining the specs on other hardware as well.

 

[storm-chaser]

It's time to get acquainted with the Dell PowerEdge C6145 and hunt down the best memory configuration/speed to use for maximum throughput. And make no mistake, that's exactly what we are going to do.

Ideally, I would populate all 32 slots in each node. That will give me a total of 128 slots to fill across both servers. One idea I had was to buy 128 (I'm saying 128 individual 2GB modules) Dell's memory spec on this server only starts with 4GB modules and goes up from there. The dell documentation says nothing about 2GB modules. But that being said, I see no reason why they wouldn't work so long as we follow Dells guidelines and spec it out correctly. And that magical spec would be:

-Dual Rank DDR3 modules
-PC3-12800R 1600MHz speed
-A voltage of 1.5
-RDIMM ECC Memory Registered





Here is the run down on memory specs. Ill just tell you now it does seem a little convoluted at first blush. I've included snips straight from the service manual that do a pretty good job explaining specific memory sets / memory configurations and their respective performance, that work with the PowerEdge C6145. According to my calculations its the RDIMM memory that I want to target for maximum bandwidth. Has to be dual rank as well.



The PowerEdge C6145 has 32 memory sockets split into 4 sets of 8 sockets with one set of memory sockets per processor. Each 8-socket set is organized into 4 channels, with 2 memory sockets per channel.

This visual representation shows the physical layout of memory slots on the motherboard. Populating ALL 32 slots with the appropriate memory modules will give me an effective 16 channels of memory bandwidth.

And keep in mind, that's 16 channels per node --- now multiply that by 4 to get your data on total memory throughput across the entire implementation, spanning my entire server landscape. It's actually off the chain and off the charts. Never in my wildest dreams did I envision I would be working with hardware that brings that level of memory performance to the table. Simply put, its astronomical.

Board layout for context:

Bottom line:
I should look into RDIMM and more specifically, RDIMM in dual rank form. I am going to go ahead and trust Dell when it comes to determining the optimal memory configuration with my given scenario. According to the performance data my choice is clear:



(In my interpretation) The 1.5v Dual rank 1600MHz RDIMMs are the modules I should use to assure maximum memory bandwidth and throughput. I may sacrifice a little latency with dual rank, but that's fine by me since we are targeting maximum bandwidth.

 

[storm-chaser]

Next Task:

CPU upgrade for the Dell PowerEdge C4130 GPU server. It's currently stocked with two Intel Xeon E5-2620 V3 2.4GHz 6 Core 15MB Cache Processors

Here is the list I am working with. At first blush it looks like an upgrade here would break the bank, as most of the top tier V3 CPUs go for well over $500

However, it looks like some of the more obscure V3 chips are not so expensive. Matter of fact, I think I can pull off two V3 CPUs with at least 20 cores total for less than $200.

 

[storm-chaser]

Update:
Spent the last hour on cpu world and ebay researching potential upgrade paths for my GPU server, the PowerEdge C4130, which currently has some of the lowest performing Intel chips in the entire E5 2600 V3 family. It has two E5 2620 v3 CPUs with a stock frequency of 2.4GHz and a poultry 6 cores apiece. A grand total of 12 cores and 24 threads. A chip like that is simply unacceptable (especially if we want to qualify as this project as HPC (high performance Computing) and I will not stand for it, especially in the confines of my GPU server that needs all the horse power it can get. It's really all about eliminating every single bottleneck. I want to see this hardware decked out and running at it's full performance potential. In addition, if we want to qualify this build as HPC (high performance computing), they've got to go directly into the garbage. And I do intend on meeting that standard with this project.

That being said, I've just pulled the trigger on two E5 2678 v3 processors. According to my research, CPU specs and performance data are identical with the Xeon E5-2680 v3, including turbo. It has a 3.3GHz turbo and 2.9GHz all core turbo. Not bad. One of the better chips out there in there

That should wake it up a bit. With the replacement of both processors we are looking at 24 cores and 48 threads in total. I can live with that! lol

*Only difference is the E5-2678 v3 can also run with DDR3 ram up to 1866 MHz.

 

[Toothless]

Man you gonna put my dual Xeon to shame with those.