Server Project

[TopHatProductions115]

Currently getting ready to troubleshoot a VLAN setup:

• https://www.reddit.com/r/mikrotik/comments/12ybec4

 

[TopHatProductions115]

Some news and updates:

• The 8TB SAS HDD for the Windows 10 VM died recently
• The ISP router threw a fit and knocked me offline for a weekend
• The current SMTP relay decided to temp. suspend my account
• I've found a replacement SMTP relay as a result, and am switching over this week
• FreePBX can no longer do outbound calls after the ISP router spat
• Haven't determined what caused the ISP router to crap itself yet
• Got three more SAS HDDs for the next iteration of the server project
• One of them is to replace the one for the Windows 10 VM
• Bridge VLAN filtering is coming soon, and will be used to move iLO3 to a VLAN
• Still need to figure out how to move vCenter Appliance to the new VLAN
• Determining if Nextcloud needs to move to an LXD container

And there's more coming soon...

 

[TopHatProductions115]

More VLAN config troubleshooting:

• https://communities.vmware.com/t5/v...nter-Server-Appliance-to-New-VLAN/m-p/2970566

 

[TopHatProductions115]

Getting closer to migrating vCenter...

• https://communities.vmware.com/t5/v...to-New-VLAN/m-p/2971972/highlight/true#M94451

New comments added on how to manually configure VLAN adapter.

 

[TopHatProductions115]

vCenter has been successfully migrated. iLO3 is the only one left. Moving to ArcZ is next...

 

[TopHatProductions115]

hMailServer has finally been configured with the new SMTP relay. Initial configuration for ArcZ (reduced target) has been completed. Now I need to create a template out of it, for future VM deployments. In opposed to configuring ZSF on a per-VM basis, I have decided to look into getting a TrueNAS VM or appliance for an iSCSI SAN. This should reduce time and complexity cost for managing storage arrays/pools, which is a must at this point. Upgrading macOS has been set back to post-DL580Gen9 transition, due to recent PSOD's -- which were strikingly similar to what happened with BlissOS. I'm still deciding on whether to renew OnlyOffice EE, which is currently ~2.4k USD for 3 years (since I count as an org). My decision is due by the end of this month. Also trying to determine whether to move Nextcloud to an LXD container.

And for the hot take...

Due to recent events surrounding Reddit and Twitch, I will not be hosting public instances of Pleroma/Lemmy/PeerTube. If the users truly wanted to force a change, Reddit would not still be on-track for an IPO with a jerk CEO at the helm. They need to prove that they won't take BS from people in charge. Until I see that, all notions of public instances are off-the-table. VDI is also out of the question. Sorry if I sound mean, but hosting costs money and time. I'm not spending if I don't see dedication from the intended end user.

 

[TopHatProductions115]

It's been a while since I last updated you all. Here's the summary:

• Originally wanted to create a vSphere template of Project ArcZ, but ran into storage constraints
• Started looking into Linux-native cloning/templating tools
  • They all either can't clone while online, don't support BTRFS, don't support non-systemd distros, can't run in OCI container, or some combo of the former(s)
  • Sticking to TimeShift and EaseUS
• I'm going to end up buying more 8TB SAS HDDs instead, to have space for everything
• Tried upgrading macOS Mojave to Ventura - somehow cause the entire hypervisor to PSOD
  • Gonna try Catalina instead, when I have time
• I wanna have all of my large SAS HDDs managed by TrueNAS Scale, as ZFS-backed iSCSI storage pools
  • And have ESXi be the primary iSCSI initiator
• Still need to setup Cronicle Edge, Fleet MDM, and Wazuh XDR

Summer is stupidly hot, and I can't game in this heat. So, no fun. Also, software licenses for NT domain environments are expensive af.

 

[TopHatProductions115]

With the move to Cronicle Edge (Docker container), I was never able to retrieve and apply the secret_key value needed to link all Cronicle hosts across my network. No matter what I tried, I was never able to find the value that the manager host was using. As such, it became impossible for me to manage the other worker hosts. I even considered using Cronicle Edge's SSH plugin as an agentless solution in the meantime, but that was simply a bandage over the wound. Ended up wiping out Cronicle in its entirety, and am now learning crontab basics instead. The sad part is, I truly liked using Cronicle when it worked. But this lost day of progress burned me. I was supposed to be working on cloning the ArcZ VM today, but that never happened. Also considered bringing back LibreNMS, but I've eaten through too much time messing around with Cronicle. For now, I need one more 8TB SAS HDD and more RAM to kick OSRM to a dedicated VM with ~96GB RAM...

 

[TopHatProductions115]

Just grabbed a DL380 Gen9 for TrueNAS Scale on baremetal. ZFS via iSCSI coming soon...

 

[TopHatProductions115]

Since last time, I've managed to order:

• 1x ProLiant DL380 Gen9 (2x Intel Xeon E5-2670v3's, 64GB DDR3 ECC)
• 16x Toshiba THNSNJ256GCSY 256GB SSDs
• 3x HGST He8/He10 8TB SAS HDDs
• 1x Radeon Pro v340 16GB
• 1x LSI SAS9200-16E HBA

The majority of those are meant for a iSCSI/ZFS build that will be running TrueNAS Scale in the future. However, the shipping company appears to have lost my SSDs. So now I'm sitting here with no SLOG or L2ARC -- just spinning disks with small ARC. This delays the transition away from vSphere RDMs to proper VMDKs (Storage vMotion to iSCSI/ZFS backend) and replacement of the current Artix OpenRC VM with Project ArcZ (BTRFS stopgap). I haven't received the SSDs or a refund yet, because it's still under investigation.

I've managed to work on other tasks in the meantime:

• added ClamAV file scanning (Nextcloud)
• configured proper Redis cache (Nextcloud)
• added LibreSign app (Nextcloud)
• configured video calling (FreePBX)
• compose file draft for Spreed/HPB (Nextcloud)

Still need to upgrade the macOS VM from Mojave to Catalina or newer, since Chromium won't be supported on Mojave in a few months time. Haven't properly tested the Radeon Pro v340 with ESXi 6.5u3 or macOS yet either. The DL380 Gen9 will possibly need more RAM for the amount of storage it'll be managing, so that will be one of my next purchases (in a few months).

Once I finish inventorying the DL380 Gen9, I'll add the details to the project configuration.

 

[TopHatProductions115]

USPS seems to have lost the 16x Toshiba THNSNJ256GCSY 256GB SSDs, so I had to replace them. Still waiting on a refund. I'll be testing the DL380 for the first time this weekend, if time permits it.

 

[TopHatProductions115]

Focus is now split between configuring the Chateau, upgrading macOS, and the configuring the DL380 Gen9. The transition to Project:ArcZ depends on the TrueNAS Scale now. Once the Chateau is ready, all users will connect to WLAN with AD credentials only.

Current ToDo's:
 - Server/Networking:
    - Configure MikroTik Chateau 5G ax for RADIUS authentication
    - DL380 Gen9: Install TrueNAS Scale, install Wazuh agent, setup iSCSI target(s)
 - macOS Mojave:
    - Change unlockers (DrDonk => shanyungyang)
    - Upgrade to Catalina
    - Install *Arr suite
    - OPTIONAL: Test Radeon Pro v340

Upcoming ToDo's:
 - ArcZ0:
    - Clone to ArcZ1:
    - Setup Samba (DomainJoin), Podman + Compose
    - Podman Migration:
        - PortainerBE, NGINX Proxy Manager, Wazuh XDR, ClamAV
        - OnlyOfficeEE, xBrowserSync, RustDesk, IT-tools
 - ArtixServ => ArcZ1:
    - Setup Samba (DomainJoin), LXD
    - LXD Migration:
        - LXD Dashboard, Lxdocker, Nextcloud + Spreed/HPB, OSRM

Long-term ToDo's:
 - Server/Networking:
    - Migrate iLO3 to VLAN 51 (Project vNet)
    - DL580 Gen9 transition
 - FreePBX:
    - Port out Google Voice phone# to VoIP.ms (DID)
        - https://support.google.com/voice/answer/10130510?hl=en
        - https://support.google.com/voice/answer/1065667?hl=en
        - https://wiki.voip.ms/article/Porting_FAQ#How_to_port_my_Google_Voice_Number_to_VoIP.ms

Other Long-Term Tasks:

• https://linustechtips.com/status/337848/
• https://linustechtips.com/status/337856/

 

[TopHatProductions115]

Updated Task List (09/25/2023):

Current ToDo's:
- macOS Mojave:
    - Change unlockers (DrDonk => shanyungyang)
    - Upgrade to Catalina
    - Install *Arr suite
- Server/Networking:
    - RADUIS/PEAP on Chateau 5Gax
    - DL380 Gen9: Setup TrueNAS Scale, Wazuh agent, iSCSI target(s)

Upcoming ToDo's:
- ArcZ0:
    - Clone to ArcZ1:
    - Setup Samba (DomainJoin), Podman + Compose
    - Podman Migration:
        - PortainerBE, NGINX Proxy Manager, Keycloak, Wazuh XDR, ClamAV
        - OnlyOfficeEE, xBrowserSync, RustDesk, IT-tools
- ArtixServ => ArcZ1:
    - Setup Samba (DomainJoin), LXD
    - LXD Migration:
        - LXD Dashboard, Lxdocker, Nextcloud + Spreed/HPB, OSRM

Focus is currently split between upgrading the ESXi unlocker and macOS VM, configuring the Chateau (RADIUS), and configuring the DL380 Gen9 (TrueNAS Scale). The move to Project:ArcZ depends on the TrueNAS Scale. The release of Incus (LXD fork) is forcing me to consider delaying the move to LXD.


Other Long-Term Tasks:

• https://linustechtips.com/status/337848/
• https://linustechtips.com/status/337856/

 

[claes]

Why’s the *arr suite going on MacOS?

 

[TopHatProductions115]

Why’s the *arr suite going on MacOS?

It's more preference than anything in this case. The *arr suite is being used solely to augment the capabilities of PleX in my setup. Not mission critical.

 

[TopHatProductions115]

I had a small slip-up ~2 days ago (10/07/2023). The certificates for vCenter expired, before I got a chance to renew them. I was distracted with other IRL tasks, and am now unable to use vCenter until this gets resolved. This is gonna delay pretty much everything else that I'm working on.

 

[TopHatProductions115]

vCenter troubleshooting under way...

 

[TopHatProductions115]

Finished troubleshooting one issue, and now hit with another one. After updating Keycloak (Docker Compose stack), my realm settings got nuked back to defaults. Undid the AD integration config that I toiled for days over to figure out. While it would be nice to have local MFA and IAM, I can't justify using a solution if my settings aren't safe. The only Docker volume options that I saw in the official documentation were for development purposes (not for use in production). I have other things to do, and this would only delay me further. I can't afford that. Very close to just kicking Keycloak from the server project...

 

[TopHatProductions115]

I've kicked Keycloak from the project indefinitely, and am replacing RustDesk with MeshCentral. The included AD integration is a plus.

 

[TopHatProductions115]

Upgrading to Monterey on the macOS VM caused multiple PSODs - all while either shutting down or rebooting/restarting the VM. I'm going to try a clean install sometime in the next month. If that doesn't go well, I'm either attempting an upgrade to Catalina or reinstalling Mojave. The jump to Catalina is enough to make sure that the Chromium browser still gets updates, but no updates/support from Apple. Only versions past 10.X get support/patches at this point. I can't upgrade to Ventura on the current server, since:

• Ventura requires AVX2.0 support in the CPU (opens new window)for native graphics acceleration
• OpenCore Legacy Patcher only supports patching AMD Polaris and above to no longer require AVX2.0 (Navi coming soon)
• Radeon R9 Series' highest supported OS version is Monterey (until I move to Vega or newer, can't touch Ventura)

The newest OS version that can run on the DL580 G7 is Monterey, assuming I can get the GUI to work as smoothly as it did on Mojave. As of now, the macOS VM is out-of-commission. It's a good thing that nothing mission-critical runs on that VM.


On a better note, currently working on a new OpenRC script for the Wazuh Agent:

• https://github.com/wazuh/wazuh/discussions/19790

 

[TopHatProductions115]

I may attempt clean-installing macOS this coming weekend, assuming that nothing else comes up. Didn't (originally) mention this in the previous update, but the Radeon R9 doesn’t appear to have Metal support on macOS. AMD Fiji was not liked, I suppose. Polaris and newer have it better on that front. I will be disabling SSH on the Artix and macOS VMs, since I never use it. One less attack surface to worry about. From what I can remember, the Radeon Pro v340 requires Above 4G Decoding. That would prevent it from working on the DL580 G7, but not the DL980. I will probably have to put off using that card until I move out. Managing Windows (especially on baremetal) is a pain, and I can't wait to get a Linux laptop...

 

[TopHatProductions115]

I'm considering moving the host OS target to ESXi 7.X. However, that would also require me to upgrade the vCSA. May do that in summer 2024, preemptively...

 

[TopHatProductions115]

So, the current Sonnet Allegro USB card appears to be ded (causing PSODs when passed through to macOS Server) and macOS Monterey doesn't seem to like the FirePro S9300 X2 (no longer loads kexts for or recognises it). I've purchased a USB3-PRO-4P10-E to replace the current USB card, hoping that this one will last a bit longer. However, the GPU situation is a strange one. I've wanted to use the FirePro X2 so I can share it with multiple VMs (or even for VDI). However, I haven't had the free time and/or end users to make good on that. I don't think the Radeon Pro v340 will work in the DL580 G7 (Above 4G Decode). I'm tempted to just use a Vega 64 and call it even, since I'm the only one actually using the server on a regular basis. I may just save the MxGPU and SR-IOV fun for the DL580 Gen9, since no one has asked for VDI/remote desktop from me...

 

[TopHatProductions115]

Due to limitations of the DL580 G7, no other VMs or containers requiring/preferring AVX and/or GPU passthrough will be added during this project phase. That includes (but is not limited to):

• waydroid
• redroid
• blissos
• f@h
• self-hosted LLMs

These are all in the same boat as VMware Horizon (VDI) now, and will be re-evaluated once the DL580 Gen9 is the VM host. The last major hardware upgrade/change for this phase will be the GPU used (at least partially) for macOS Monterey.

 

[TopHatProductions115]

At the beginning of the month (12/01), I started researching how to configure MDM in ManageEngine Endpoint Central. I was also considering adding Overseerr as a Docker/Podman container. However, this month was not a calm one...

Changelog:

• [12/06] The Vega 64 arrived in the mail (for the macOS VM). I updated the graphics drivers for the Windows 10 VM (Titan Xp).
• [12/07] The Windows 10 VM had its first BSOD in almost half a year. Haven't run outside of stock values lately due to previous BSODs, but uninstalled MSI Afterburner to be safe.
• [12/08] The Windows 10 VM ran drive error repairs -- probably caused by the recent BSODs.
• [12/09] The primary display is now likely to change to the VMware display adapter if I open it, so I now must avoid opening it. I (re-)learned [Win] + [Ctrl] + [Shift] + B .
• [12/10] Updated Nextcloud to 26.0.9, and planning a (delayed due to BSOD troubleshooting) move to 27 later this month.
• [12/11] The Windows 10 VM had another BSOD. I couldn't run the DISM command. After checking Event Viewer and Device Manager logs, I decided to DDU and rollback graphics drivers.
• [12/12] I observed the Windows 10 VM, to make sure the BSODs stopped. The 2nd Radeon Pro v340 arrived by this time.
• [12/13] Internet connectivity for the entire server rack dropped (~01:20 EST) -- but there've been no network configuration changes since May. Had to re-configure the MikroTik Audience.
• [12/14] Attended an installation demo/evaluation for ManageEngine Log360, to compare to Wazuh XDR. Installation is incomplete, and will be continued next week.

Still need to evaluate whether to move to Windows 11 on the Windows 10 VM, whether to add Overseerr, and apply configuration changes suggested in Wazuh XDR...