Server Project

[TopHatProductions115]

Hello there! I've hinted toward this project in various forum posts, but I'll cut to the chase. I'm in the final phase of a long-awaited server project, which began back in mid-2018. However, I never got to actually acquire any of the parts until 2019/2020. With the massive gap in time, I had to change things up a bit, due to market forces and parts availability. The server project is meant to replace (and exceed) my previous workstation - a Dell Precision T7500. This is what I've ended up with as of summer 2023:

HPE ProLiant DL580 G7

    OS   :: VMware ESXi 6.5u3 Enterprise Plus
    CPU  :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
    RAM  :: 256GB (64x4GB) PC3-10600R DDR3-1333 ECC
    PCIe :: 1x HP 512843-001/591196-001 System I/O board +
                1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board
    GPU  :: 1x nVIDIA GeForce GTX Titan Xp +
                1x AMD FirePro S9300 x2 (2x "AMD Radeon Fury X's")
    SFX  :: 1x Creative Sound Blaster Audigy Rx
    NIC  :: 1x HPE NC524SFP (489892-B21) +
                2x Silicom PE310G4SPI9L-XR-CX3's
    STR  :: 1x HP Smart Array P410i Controller (integrated) +
                1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, vCenter Appliance, ISOs) +
                4x HP 507127-B21 300GB HDDs (ESXi guest datastores) +
                1x Western Digital WD Blue 3D NAND 500GB SSD +
                1x Intel 320 Series SSDSA2CW600G3 600GB SSD +
                1x Seagate Video ST500VT003 500GB HDD
    STR  :: 1x LSI SAS 9201-16e HBA SAS card +
                1x Mini-SAS SFF-8088 cable +
                        1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) +
                                4x HITACHI Ultrastar HUH728080AL4205 8TB HDDs +
                                4x IBM Storewise XIV v7000 98Y3241 4TB HDDs
    I/O  :: 1x Inateck KU8212 (USB 3.2) +
                1x Logitech K845 (Cherry MX Blue) +
                1x Dell MS819 Wired Mouse
            1x Sonnet Allegro USB3-PRO-4P10-E (USB 3.X) +
                1x LG WH16NS40 BD-RE ODD
    PRP  :: 1x Samsung ViewFinity S70A UHD 32" (S32A700)
            1x Sony Optiarc BluRay drive
    PSU  :: 4x HP 1200W PSUs (441830-001/438203-001)

The planned software configuration has been moved back to the LTT post, and will be changing often for the foreseeable future.

Product links and details can be found here.


ESXi itself is usually run from a USB thumb drive, but I have a drive dedicated to it. No harm done. A small amount of thin provisioning/overbooking (RAM only) won’t hurt. macOS and Linux would have gotten a Radeon/FirePro (ie., Rx Vega 64), for best compatibility and stability, but market forces originally prevented this. Windows 10 gets the Audigy Rx and a Titan Xp. The macOS and Linux VMs get whatever audio the Titan Z FirePro S9300 x2 can provide. The whole purpose of Nextcloud is to phase out the use of Google Drive/Photos, iCloud, Box.com, and other externally-hosted cloud services (Mega can stay though). The reason for why I’m doing this.


If you want more details about the project posted here, please let me know.


There are three other mirrors for this project, all of which get updated when new developments get made:

• Linus Tech Tips
• Level1Techs
• Overclock.net

 

[TopHatProductions115]

Replaced the Rosewill RASA-11001 with a Kingwin MKS-435TL, due to the fact that it doesn't need molex and will also have a cleaner look to it. Forgot how to edit posts here, sorry if I could have added the info to the OP above. Also planning out a long-term Tesla K80 upgrade...

 

[TopHatProductions115]

Grabbed the wired mouse. Now back to the waiting game, to see when everything will arrive in the mail...

 

[phill]

Looking forward to seeing the outcome

Although I do wonder, if 4 1200w PSUs will be enough??...

 

[TopHatProductions115]

The new drive cage arrived via Amazon. There's only one item left that hasn't arrived in the mail yet - my new mouse...

 

[TopHatProductions115]

Virtual Flash looks like fun. I might actually look into it more when the time comes...

 

[TopHatProductions115]

Looking forward to seeing the outcome

Although I do wonder, if 4 1200w PSUs will be enough??...

Thank goodness it only needs one of the PSUsat a time! Otherwise, I'd be toast XD

 

[phill]

Thank goodness it only needs one of the PSUsat a time! Otherwise, I'd be toast XD

I know that feeling!!! I have a few servers here, boy they are power hungry!! Looking forward to seeing another picture filled update!!

 

[TopHatProductions115]

So, here's the current plan:

• Have ESXi run 24/7, in a low-power/on-demand state
• Have Windows Server 2016 (VPN+AD) setup for 24/7 low-power operation as well
• Have Artix Linux (Nextcloud) setup for 24/7 low-power operation too
• Have MacOS setup for Hybrid Sleep after 20 minutes' inactivity and setup for Wake-on-LAN
• Have Windows 10 setup for ^^^ as well, since it's not running anything mission critical
• Don't power down ESXi unless I want to lose access to over half my infrastructure :3

Notes:

• https://helgeklein.com/blog/2012/03/dead-simple-wake-on-lan-for-your-windows-server-at-home/

 

[phill]

I look forward to seeing this all up and running

 

[TopHatProductions115]

So close, yet so far:
https://communities.vmware.com/message/2980320#2980320

 

[phill]

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

 

[TopHatProductions115]

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

Yes. And I'm getting ready to run a memory diagnostic, to see which sticks may be failing.

 

[TopHatProductions115]

Have you got spare server RAM you can change it out with to test @TopHatProductions115 ??

Crisis averted. From the looks of it, the issue was caused by me running a slightly older version of ESXi 6.5 (U2). VMware's servers have been a bit unreliable for downloads recently, so I had to dig for a copy of ESXi 6.5 U3 on the internet. Now troubleshooting the next issue:

• https://communities.vmware.com/message/2980646#2980646

Also have to look into this at some point (even though the server will be mostly hiding behind a VPN):

• https://kb.vmware.com/s/article/55636

 

[TopHatProductions115]

I'm getting ready to reinstall ESXi 6.5 U3, since HPE's Smart Array assumed I was gonna 'RAID all da things' and lose over half my storage to sub-optimally configured RAID arrays that it made (one of which was in perpetual recovery and included an unsupported SSD). Had to manually configure each drive as a logical volume with RAID 0 fault tolerance, treating each drive as its own virtual array of one disk. I ended up RAIDing all the things anyway, by that logic.

 

[TopHatProductions115]

Now to address the SSD issue in the background, while I look at security patches and initial VM setup.

 

[TopHatProductions115]

Currently trying to make a new datastore via SSH. More info here:

• https://communities.vmware.com/message/2980777#2980777

 

[TopHatProductions115]

So close to the final stage:

• Failed to create VMFS datastore - Cannot change the host configuration.

 

[TopHatProductions115]

While I'm waiting on comments for the previous issue, and a few ISOs to upload to my server, I can start working on investigating this:

• https://kb.vmware.com/s/article/55636
• https://kb.vmware.com/s/article/55806
• https://www.vmware.com/security/advisories/VMSA-2018-0020.html
  • https://kb.vmware.com/s/article/56547
  • https://kb.vmware.com/s/article/56563
  • https://kb.vmware.com/s/article/56896

Gotta search for the patches through this page, by entering the details mentioned in the last 3 kb pages:

• https://my.vmware.com/group/vmware/patch#search

On a side note, also ran into this when setting up my first VM:

• http://www.yellow-bricks.com/2018/0...when-powering-on-vm-on-vsphere/#comment-77449

Reserve Memory beforehand, I guess

 

[TopHatProductions115]

Here to update the Reddit mirror's link - those expire every 6 months

• https://www.reddit.com/user/TopHatProductions115/comments/ijoeg0/project_personal_datacentre/

 

[phill]

Someone has been having fun!! Gotta love trouble shooting don't you?? I hope it all settles out soon for you!!

 

[TopHatProductions115]

Someone has been having fun!! Gotta love trouble shooting don't you?? I hope it all settles out soon for you!!

Yeah. Still privately working out how to get my HBA working in ESXi as well, before I send that issue to VMware's forums as well XD

It's been hectic...

 

[phill]

Yeah. Still privately working out how to get my HBA working in ESXi as well, before I send that issue to VMware's forums as well XD

It's been hectic...

Sometimes I do ask myself the question about all that I do at home, for home and I say, is it all worth the hassle sometimes

I hope you get it sorted out mate and can report back all of the great work you've been doing, oh and don't forget a load of pictures!!

 

[TopHatProductions115]

Oh, I will :3

Broadcom Community - VMTN, Mainframe, Symantec, Carbon Black

Broadcom Community - VMTN, Mainframe, Symantec, Carbon Black

communities.vmware.com

 

[TopHatProductions115]

Sometimes I do ask myself the question about all that I do at home, for home and I say, is it all worth the hassle sometimes

I hope you get it sorted out mate and can report back all of the great work you've been doing, oh and don't forget a load of pictures!!

One step closer