• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

Server Project

Joined
Aug 14, 2017
Messages
348 (0.14/day)
Location
Edge of the Void
System Name Serious Series - Serious Server (99.99%)
Processor 4x Intel Xeon E7-8870's
Motherboard HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling HP ProLiant OEM cooling fans(s) + heatsinks
Memory 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s) AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage 1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s) Samsung ViewFinity S70A UHD 32" (S32A700)
Case HP ProLiant DL580 G7 chassis
Audio Device(s) 1x Creative Sound Blaster Audigy Rx
Power Supply 4x HP 441830-001/438203-001's (1200W PSU's)
Mouse Dell MS819
Keyboard Logitech K845 (Cherry MX Blue)
VR HMD N/a
Software VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores 3DMark won't let me post my scores publicly at this time...
It would appear that macOS Mojave is the last version that can support Radeon GPUs with just PCI Passthrough. macOS versions past Mojave won't allow simple GPU passthrough to work, even with Lilu and Whatevergreen kexts loaded. macOS VMs may now require passing vBIOS/option ROM passed to them as well. However, such a feature may only be available in vSphere 7.X.
More info on some of the features listed here:
 
Joined
Aug 14, 2017
Messages
348 (0.14/day)
Location
Edge of the Void
System Name Serious Series - Serious Server (99.99%)
Processor 4x Intel Xeon E7-8870's
Motherboard HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling HP ProLiant OEM cooling fans(s) + heatsinks
Memory 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s) AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage 1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s) Samsung ViewFinity S70A UHD 32" (S32A700)
Case HP ProLiant DL580 G7 chassis
Audio Device(s) 1x Creative Sound Blaster Audigy Rx
Power Supply 4x HP 441830-001/438203-001's (1200W PSU's)
Mouse Dell MS819
Keyboard Logitech K845 (Cherry MX Blue)
VR HMD N/a
Software VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores 3DMark won't let me post my scores publicly at this time...
The only other option I can see is using something like Clover to dynamically load a compatible vBIOS. But, that will require further research...
 
Joined
Aug 14, 2017
Messages
348 (0.14/day)
Location
Edge of the Void
System Name Serious Series - Serious Server (99.99%)
Processor 4x Intel Xeon E7-8870's
Motherboard HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling HP ProLiant OEM cooling fans(s) + heatsinks
Memory 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s) AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage 1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s) Samsung ViewFinity S70A UHD 32" (S32A700)
Case HP ProLiant DL580 G7 chassis
Audio Device(s) 1x Creative Sound Blaster Audigy Rx
Power Supply 4x HP 441830-001/438203-001's (1200W PSU's)
Mouse Dell MS819
Keyboard Logitech K845 (Cherry MX Blue)
VR HMD N/a
Software VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores 3DMark won't let me post my scores publicly at this time...
As mentioned in the previous update(s), the macOS VM is currently unable to use any AMD GPUs that I've passed through to it. This issue only appeared after upgrading from Mojave. There are two potential solutions for this issue (pertaining to device initialisation):
  • convert to OpenCore and use SSDT to inject vBIOS
  • use ESXi 7, VMX settings to load "option ROM" (vBIOS)
I've also seen a few posts online, indicating that the latter option may be available in ESXi 6.7. However, I have yet to confirm this. I'm tempted to try upgrading the DL580 G7 to ESXi 6.7u3, to see if that exposes the VMX options I'd need. In the worst case scenario, I won't be able to use AMD GPUs with this VM as long as I'm using the DL580 G7, and the VM runs Monterey.


Now for better news...


OSRM will run just as well in an application container as it would in a system container:
I can leave that in a Podman container now, and not be concerned about potential performance penalties.

I also encountered a thread yesterday, mentioning this repo:
AD CS can be made compatible with ACME clients, to allow for easier certificate renewal automation.

The vSphere version target for the DL580 Gen9 has been moved, from 6.7 to 7.0.
 
Joined
Aug 14, 2017
Messages
348 (0.14/day)
Location
Edge of the Void
System Name Serious Series - Serious Server (99.99%)
Processor 4x Intel Xeon E7-8870's
Motherboard HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling HP ProLiant OEM cooling fans(s) + heatsinks
Memory 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s) AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage 1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s) Samsung ViewFinity S70A UHD 32" (S32A700)
Case HP ProLiant DL580 G7 chassis
Audio Device(s) 1x Creative Sound Blaster Audigy Rx
Power Supply 4x HP 441830-001/438203-001's (1200W PSU's)
Mouse Dell MS819
Keyboard Logitech K845 (Cherry MX Blue)
VR HMD N/a
Software VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores 3DMark won't let me post my scores publicly at this time...
Joined
Aug 14, 2017
Messages
348 (0.14/day)
Location
Edge of the Void
System Name Serious Series - Serious Server (99.99%)
Processor 4x Intel Xeon E7-8870's
Motherboard HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling HP ProLiant OEM cooling fans(s) + heatsinks
Memory 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s) AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage 1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s) Samsung ViewFinity S70A UHD 32" (S32A700)
Case HP ProLiant DL580 G7 chassis
Audio Device(s) 1x Creative Sound Blaster Audigy Rx
Power Supply 4x HP 441830-001/438203-001's (1200W PSU's)
Mouse Dell MS819
Keyboard Logitech K845 (Cherry MX Blue)
VR HMD N/a
Software VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores 3DMark won't let me post my scores publicly at this time...
Still looking into solutions for using newer cards in the DL580 G7, until I can move to the DL580 Gen9. From what I've seen in documentation, I could try disabling unneeded PCIe devices to free up resources for other PCIe devices:
However, I'm not sure which ones to disable yet. I may have to open a support ticket with HPE:
That will take a while to investigate. Still need to get vBIOS for the FirePro S9300 X2, to re-test the VMX parameters.

hMailServer is no longer actively maintained. I'll be attempting a migration to Stalwart this year. But need a way to either migrate or archive and access e-mails handled and generated with the previous mail server. Currently looking into MailStore for that.

On a side note, I'm taking another shot at RADIUS with ClearBox Enterprise RADIUS server. As usual, the MikroTik Chateau isn't playing nice. Same results as last time, with TekRADIUS OD. I'm starting to wonder if I should just ditch the idea of having LTE failover in the future...
 
Joined
Aug 14, 2017
Messages
348 (0.14/day)
Location
Edge of the Void
System Name Serious Series - Serious Server (99.99%)
Processor 4x Intel Xeon E7-8870's
Motherboard HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling HP ProLiant OEM cooling fans(s) + heatsinks
Memory 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s) AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage 1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s) Samsung ViewFinity S70A UHD 32" (S32A700)
Case HP ProLiant DL580 G7 chassis
Audio Device(s) 1x Creative Sound Blaster Audigy Rx
Power Supply 4x HP 441830-001/438203-001's (1200W PSU's)
Mouse Dell MS819
Keyboard Logitech K845 (Cherry MX Blue)
VR HMD N/a
Software VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores 3DMark won't let me post my scores publicly at this time...
I finished installing and configuring MailStore Server, in preparation for the move from hMailServer to Stalwart. Evaluation of ClearBox Enterprise RADIUS server has been delayed indefinitely (best candidate tested). Project:ArcZ has changed a bit more, swapping LightDM for ly. Working on releasing an ISO for a small group of testers. The ISO repair for the Windows 10 VM appears to have been successful -- no issues since completion in mid-February. Swapped the current PDU for one with more outlets, since I was running out of usable ones. Too many appliances have chunky rectangular plugs, that block adjacent outlets on the PDU. The next version of the server project has moved on from 400GB SAS SSDs to 800GB ones. It appears that running TrueNAS as a VM, in production, is no longer discouraged:
If such is the case, I may no longer need the DL380 Gen9. If I had known (late last year) that such a change-up was coming, I would not have gotten a dedicated file server. But, it's here now...
 
Joined
Aug 14, 2017
Messages
348 (0.14/day)
Location
Edge of the Void
System Name Serious Series - Serious Server (99.99%)
Processor 4x Intel Xeon E7-8870's
Motherboard HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling HP ProLiant OEM cooling fans(s) + heatsinks
Memory 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s) AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage 1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s) Samsung ViewFinity S70A UHD 32" (S32A700)
Case HP ProLiant DL580 G7 chassis
Audio Device(s) 1x Creative Sound Blaster Audigy Rx
Power Supply 4x HP 441830-001/438203-001's (1200W PSU's)
Mouse Dell MS819
Keyboard Logitech K845 (Cherry MX Blue)
VR HMD N/a
Software VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores 3DMark won't let me post my scores publicly at this time...
The month of March has been very eventful. At first, I was looking into whether I should split the Windows Server VM into 2-3 different VMs instead:
During this brief period, I was also reviewing some security policy changes/software patches that were suggested in ManageEngine Endpoint Central. One of the software patches were for MariaDB, which would require me to check version compatibility with each app/service accessing it. Knowing my luck, things were bound to get complicated on day 5.

I then found multiple pages from iXsystems, stating that it's safe to virtualise TrueNAS Scale. I'd already spent money on the DL380 Gen9 for that, but I guess there's no use getting peeved about that. This simply means that I can get away with one less physical server in my rack (and less power draw), so there is a plus side to it. Most of the monetary loss is still there, but I can at least use the SSDs (and the discrete HBA) planned for it elsewhere.

On that same day, the VM for Project:ArcZ also threw warnings related to deprecated options/hooks in image build config file (initcpio). The older Artix OpenRC VM did not give the same warning. I got help from a contact on Discord, to correct the deprecated config parameters. Two days later, I was installing a service pack for Endpoint Central.

The next day, I was testing the Nextcloud Social app, and found out that I finally had to configure .well-known/webfinger (CardDAV/CalDAV related) for the instance. I started looking into how to edit the Nextcloud container's config for it. Attempts for this concluded on the 21st. I committed changes to the .htaccess file in Nextcloud itself, and the subdomain > custom location(s) in NGINX Proxy Manager (reverse proxy). Both methods did not work, leaving me with no clear path forward. I'll have to leave self-hosting federated services for later.

Five days later, I was reviewing FreePBX extension configs when I decided to buy more DID numbers to use in FreePBX. I also attempted to install Sunshine gamestream server via MacPorts, only for it to fail at the installation step. I'll have to look into that later as well.

Four days later, I was advised to move /boot/efi to its own dedicated partition (/efi) while updating GRUB on Project:ArcZ. I spent the next 2 days working on it, with help from the same Discord contact. At this point, if you couldn't tell, they're pretty amazing! Still need to write a pacman hook for auto-generating GRUB configuration whenever GRUB gets updated. I then started work on a dedicated VoIP VLAN for FreePBX the next day. Work for this concluded on the 22nd.

After that, I was applying and testing more security policy changes through Endpoint Central. On the 25th, I decided to remove the * (wildcard) user from SoftEther VPN, due to the rapid increase in reported software vulnerabilities. Now, each VPN user has to be explicitly defined with an AD-linked account. On the 26th, I started clearing out TimeShift backups on the Artix OpenRC VM (backup partition ran out of space for new backups).

This morning, the Windows Server VM reported an unexpected shutdown from the previous night -- even though I had issued the last shutdown command myself. I checked the Event Logs, and found multiple warning/error events from yesterday and today. Investigation and remediation for it is ongoing...
 
Joined
Aug 14, 2017
Messages
348 (0.14/day)
Location
Edge of the Void
System Name Serious Series - Serious Server (99.99%)
Processor 4x Intel Xeon E7-8870's
Motherboard HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling HP ProLiant OEM cooling fans(s) + heatsinks
Memory 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s) AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage 1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s) Samsung ViewFinity S70A UHD 32" (S32A700)
Case HP ProLiant DL580 G7 chassis
Audio Device(s) 1x Creative Sound Blaster Audigy Rx
Power Supply 4x HP 441830-001/438203-001's (1200W PSU's)
Mouse Dell MS819
Keyboard Logitech K845 (Cherry MX Blue)
VR HMD N/a
Software VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores 3DMark won't let me post my scores publicly at this time...
After seeing a notification in the Server Manager (mentioned unexpected power event) I had the Windows Server VM perform a check-disk on next power-on and checked Event Viewer. That's where I started seeing errors and event IDs that I hadn't encountered before. I ended up doing the same on the Windows 10 VM. Here are some (not all) of the things I had to review, mask, and/or remediate in the last 24 hours:
Still more for me to take on in the coming months. Some of these started popping up after taking actions suggested in ManageEngine Endpoint Central, as security policy/configurations (like the RPC-related one). While most of the heavy-lifting in Endpoint Central is done (may setup MDM certificate for Apple devices), I now have to start doing the same in Wazuh XDR. The work never stops.

On a side note, I also need a Redis replacement for the Nextcloud instance...
 
Joined
Aug 14, 2017
Messages
348 (0.14/day)
Location
Edge of the Void
System Name Serious Series - Serious Server (99.99%)
Processor 4x Intel Xeon E7-8870's
Motherboard HP 512843-001/591196-001 (rev 0B) + 588137-B21/591205-001
Cooling HP ProLiant OEM cooling fans(s) + heatsinks
Memory 256GB (64x4GB) DDR3-1333 PC3-10600R ECC
Video Card(s) AMD FirePro S9300 X2 + nVIDIA GeForce GTX Titan Xp
Storage 1x HGST HUSMM8040ASS200 + 4x HP 507127-B21's + 1x WD Blue 3D NAND 500GB + 1x Intel SSDSA2CW600G3
Display(s) Samsung ViewFinity S70A UHD 32" (S32A700)
Case HP ProLiant DL580 G7 chassis
Audio Device(s) 1x Creative Sound Blaster Audigy Rx
Power Supply 4x HP 441830-001/438203-001's (1200W PSU's)
Mouse Dell MS819
Keyboard Logitech K845 (Cherry MX Blue)
VR HMD N/a
Software VMware ESXi 6.5u3 Enterprise Plus (VM: Windows 10 Enterprise LTSC)
Benchmark Scores 3DMark won't let me post my scores publicly at this time...
Changed Cloudflare WAF settings recently, so that only connections from certain regions are allowed (GeoIP).

With FreePBX Distro losing support soon, I'll have to move to Debian by the end of May.

May start experimenting with ClearBox RADIUS server again, once FreePBX has been moved to Debian.

The macOS has gone back to chewing through USB cards, which means that I can't directly connect a BluRay player to that VM at this time. I'll have to see if I can start ripping and uploading DVD/BluRay images to it over the network.
 
Top