Technical Issues - TPU Main Site & Forum (2024)

lexluthermiester · Nov 21, 2024

W1zzard said:
You're still having this issue?

Nope, seems good now.

RJARRRPCGP · Nov 21, 2024

48 hours later or around there, it's responding good.

lexluthermiester · Nov 21, 2024

@W1zzard
I don't know if this will be helpful with the current situation or if you've seen it, PHP 8.4.1 is out, just today. Seems it fixes a number of problems and makes some improvements.

PHP: Hypertext Preprocessor

PHP is a popular general-purpose scripting language that powers everything from your blog to the most popular websites in the world.

www.php.net

There's also been some security patches for older versions.

Macro Device · Nov 22, 2024

Still giving out some 502s here and there. Intermittently so. I assume you'll be migrating your systems for another couple days or maybe even weeks, @W1zzard?

Ruru · Nov 22, 2024

Macro Device said:
Still giving out some 502s here and there. Intermittently so. I assume you'll be migrating your systems for another couple days or maybe even weeks, @W1zzard?

Yeah, I had that also. Hope that now that's fixed.

W1zzard · Nov 22, 2024

yeah, still working on fixing things in the background, no downtime was expected though

wNotyarD · Nov 22, 2024

W1zzard said:
no downtime was expected though

As I learned in technical and in uni, this is called unifying theory and practice. It doesn't work as expected and no one knows why.

502's in our midst again.

EddieTan · Nov 22, 2024

Keep getting Bad Gateway

68Olds · Nov 22, 2024

wNotyarD said:
502's in our midst again.

Yes, I'm seeing a lot of 502s this morning as well.

Count von Schwalbe · Nov 22, 2024

A 502 all morning so far.

W1zzard · Nov 22, 2024

Yup, will take probably an hour or more

chrcoluk · Nov 22, 2024

Getting loads of 502s, after 2 or 3 refreshes it loads.

Count von Schwalbe · Nov 22, 2024

W1zzard said:
Yup, will take probably an hour or more

A wizard working on repairing a gate, cartoon, line drawing

Good luck!

trparky · Nov 22, 2024

502s, lots of 502s.

W1zzard · Nov 22, 2024

meh .. what a shit day .. things should run much smoother now

wNotyarD · Nov 22, 2024

W1zzard said:
meh .. what a shit day .. things should run much smoother now

Thanks for the hard work, @W1zzard. We know we may nag you a lot over issues, but that's because we care about these forums.

Dr. Dro · Nov 22, 2024

W1zzard said:
meh .. what a shit day .. things should run much smoother now

Appreciate the hard work, W1zz. Have a cold one now you've earned it :toast:

Frick · Nov 22, 2024

W1zzard said:
meh .. what a shit day .. things should run much smoother now

DRINK

lexluthermiester · Nov 22, 2024

W1zzard said:
meh .. what a shit day .. things should run much smoother now

Things do seem much smoother. So what was the problem? No need for nitty gritty details, just a general idea would be good. Kinda curious..

I can imagine how much of a pain this has been. Tipping my hat to you!

Frick said:
DRINK

Not a bad idea. For those inclined, tip a toast!

Bobaganoosh · Nov 22, 2024

W1zzard said:
meh .. what a shit day .. things should run much smoother now

Some behind the scenes footage of W1zzard battling it out:

Sorry for the shit day, man. hope the weekend is kinder to you.

W1zzard · Nov 23, 2024

Site is chugging along nicely now. Could still be a little bit faster

Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35), also moved our shared filesystem (Gluster) into containers.
Also no more plain Docker, and switched to Podman for the cases where I just need a fs to mess around in.
Migrated most services into the new cluster, which no longer uses Zerotier. ZT looks great on paper and I love that it can do ARP because it's L2, but under serious load it just fell apart randomly and unpredictably.

Now running with Netmaker, which uses kernel Wireguard, no more dynamic/floating IPs

Also not 100% happy with the current Cilium setup.

This new cluster setup is engineered as blue/green, so it should be easier to switch live between them.

Another interesting issue I found today, and that's probably the cause for most 502's yday, was that the control plane node serving DNS queries (CoreDNS) was overloaded, so some DNS queries would randomly fail. MySQL Router does periodical health checks and (obviously) uses DNS in the process, if such a query failed, it would mark the whole backend as failed and the DB access would just disappear.

Caring1 · Nov 23, 2024

Settle down W1zz, this is an English speaking forum.

W1zzard · Nov 23, 2024

Caring1 said:
Settle down W1zz, this is an English speaking forum.

People wanted more info what was going on

Caring1 · Nov 23, 2024

W1zzard said:
People wanted more info what was going on

I know, and I tried to understand it, I really did. :laugh:

lexluthermiester · Nov 23, 2024

W1zzard said:
Site is chugging along nicely now. Could still be a little bit faster

Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35), also moved our shared filesystem (Gluster) into containers.
Also no more plain Docker, and switched to Podman for the cases where I just need a fs to mess around in.
Migrated most services into the new cluster, which no longer uses Zerotier. ZT looks great on paper and I love that it can do ARP because it's L2, but under serious load it just fell apart randomly and unpredictably.

Now running with Netmaker, which uses kernel Wireguard, no more dynamic/floating IPs Also not 100% happy with the current Cilium setup.

This new cluster setup is engineered as blue/green, so it should be easier to switch live between them.

Another interesting issue I found today, and that's probably the cause for most 502's yday, was that the control plane node serving DNS queries (CoreDNS) was overloaded, so some DNS queries would randomly fail. MySQL Router does periodical health checks and (obviously) uses DNS in the process, if such a query failed, it would mark the whole backend as failed and the DB access would just disappear.

Wow! So alot was going on then. We could tell it was involved, but this is next level. As stated above, tipping my hat to you! :toast:

System Name	KHR-1
Processor	Ryzen 9 5900X
Motherboard	ASRock B550 PG Velocita (UEFI-BIOS P3.40)
Memory	32 GB G.Skill RipJawsV F4-3200C16D-32GVR
Video Card(s)	Sparkle Titan Arc A770 16 GB
Storage	Western Digital Black SN850 1 TB NVMe SSD
Display(s)	Alienware AW3423DWF OLED-ASRock PG27Q15R2A (backup)
Case	Corsair 275R
Audio Device(s)	Technics SA-EX140 receiver with Polk VT60 speakers
Power Supply	eVGA Supernova G3 750W
Mouse	Logitech G Pro (Hero)
Software	Windows 11 Pro x64 23H2

System Name	DLSS / YOLO-PC / FULLRETARD
Processor	i5-12400F / 10600KF / C2D E6750
Motherboard	Gigabyte B760M DS3H / Z490 Vision D / P5GC-MX/1333
Cooling	Laminar RM1 / Gammaxx 400 / 775 Box cooler
Memory	32 GB DDR4-3200 / 16 GB DDR4-3333 / 3 GB DDR2-700
Video Card(s)	RX 6700 XT / R9 380 2 GB / 9600 GT
Storage	A couple SSDs, m.2 NVMe included / 240 GB CX1 / 500 GB HDD
Display(s)	Compit HA2704 / MSi G2712 / non-existent
Case	Matrexx 55 / Junkyard special / non-existent
Audio Device(s)	Want loud, use headphones. Want quiet, use satellites.
Power Supply	Thermaltake 1000 W / Corsair CX650M / non-existent
Mouse	Don't disturb, cheese eating in progress...
Keyboard	Makes some noise. Probably onto something.
VR HMD	I live in real reality and don't need a virtual one.
Software	Windows 11 / 10 / 8

System Name	4K-gaming / console
Processor	AMD Ryzen 7 5800X / Intel Core i7-6700K
Motherboard	Asus ROG Crosshair VII Hero / Asus Z170-K
Cooling	Alphacool Eisbaer 360 / Alphacool Eisbaer 240
Memory	32GB DDR4-3466 / 16GB DDR4-3000
Video Card(s)	Asus RTX 3080 TUF OC / Powercolor RX 6700 XT
Storage	3.3TB of SSDs / several small SSDs
Display(s)	Acer 27" 4K120 IPS + Lenovo 32" 4K60 IPS
Case	Corsair 4000D AF White / DeepCool CC560 WH
Audio Device(s)	Sony WH-CN720N
Power Supply	EVGA G2 750W / Fractal ION Gold 550W
Mouse	Logitech MX518 / Logitech G400s
Keyboard	Roccat Vulcan 121 AIMO / NOS C450 Mini Pro
VR HMD	Oculus Rift CV1
Software	Windows 11 Pro / Windows 11 Pro
Benchmark Scores	They run Crysis

Processor	Ryzen 7 5700X
Memory	48 GB
Video Card(s)	RTX 4080
Storage	2x HDD RAID 1, 3x M.2 NVMe
Display(s)	30" 2560x1600 + 19" 1280x1024
Software	Windows 10 64-bit

System Name	G-Station 2.0 "YGUAZU"
Processor	AMD Ryzen 7 5700X3D
Motherboard	Gigabyte X470 Aorus Gaming 7 WiFi
Cooling	Freezemod: Pump, Reservoir, 360mm Radiator, Fittings / Bykski: Blocks / Barrow: Meters
Memory	Asgard Bragi DDR4-3600CL14 2x16GB
Video Card(s)	Sapphire PULSE RX 7900 XTX
Storage	240GB Samsung 840 Evo, 1TB Asgard AN2, 2TB Hiksemi FUTURE-LITE, 320GB+1TB 7200RPM HDD
Display(s)	Samsung 34" Odyssey OLED G8
Case	Lian Li Lancool 216
Audio Device(s)	Astro A40 TR + MixAmp
Power Supply	Cougar GEX X2 1000W
Mouse	Razer Viper Ultimate
Keyboard	Razer Huntsman Elite (Red)
Software	Windows 11 Pro

Technical Issues - TPU Main Site & Forum (2024)

lexluthermiester

RJARRRPCGP

lexluthermiester

PHP: Hypertext Preprocessor

Macro Device

Ruru

S.T.A.R.S.

W1zzard

Administrator

wNotyarD

EddieTan

68Olds

Count von Schwalbe

Nocturnus Moderatus

W1zzard

Administrator

chrcoluk

Count von Schwalbe

Nocturnus Moderatus

trparky

W1zzard

Administrator

wNotyarD

Dr. Dro

Frick

Fishfaced Nincompoop

lexluthermiester

Bobaganoosh

W1zzard

Administrator

Caring1

W1zzard

Administrator

Caring1

lexluthermiester

System Name	ZakuPC
Processor	AMD Ryzen 9 5900X 12 Core/24 Thread
Motherboard	ASUS TUF GAMING B550M (WI-FI) ZAKU II EDITION
Cooling	ASUS TUF Gaming LC 240 RGB ZAKU II EDITION AIO
Memory	VENGEANCE RGB PRO DDR4 32GB (4x8GB) 3600MHz
Video Card(s)	ROG STRIX GAMING AMD RADEON 6700XT 12GB
Storage	Corsair NVMe M.2 SSD 1TB - OS & Crucial NVMe M.2 SSD 1TB - Games, Movies, Music, Torrent
Display(s)	Samsung Gaming Monitor 27" 2560 x 1440
Case	ZAKU II Limited Edition
Audio Device(s)	Logitech G560 RGB Gaming Speakers
Power Supply	850W
Mouse	YELANDAR RGB Mouse
Keyboard	MARVO RGB Keyboard With MARVO RGB Mousepad
Software	Windows 11 Pro 24H2 64 bit

System Name	Gamer
Processor	Ryzen 7 5800X3D
Motherboard	Gigabyte X570 Aorus Elite WIFI
Cooling	Apogee XL, MCR360, MCP655, Micro-rez
Memory	2x16GB G.Skill DDR4-3600 CL16
Video Card(s)	Red Dragon RX 6800 XT
Storage	WD SN850 1TB & SN750 1TB
Display(s)	LG 32GK650F-B 32" 1440p 144Hz
Case	Rocketfish (Lian Li) e-ATX
Audio Device(s)	Harman Kardon HK695 2.1
Power Supply	Seasonic Focus GM-750
Mouse	HP Gaming wired
Keyboard	Logitech G110 wired
VR HMD	HP WMR 1440^2
Software	Win10 Pro 64bit

System Name	Work Computer \| Unfinished Computer
Processor	Core i7-6700 \| Ryzen 5 5600X
Motherboard	Dell Q170 \| Gigabyte Aorus Elite Wi-Fi
Cooling	A fan? \| Truly Custom Loop
Memory	4x4GB Crucial 2133 C17 \| 4x8GB Corsair Vengeance RGB 3600 C26
Video Card(s)	Dell Radeon R7 450 \| RTX 2080 Ti FE
Storage	Crucial BX500 2TB \| TBD
Display(s)	3x LG QHD 32" GSM5B96 \| TBD
Case	Dell \| Heavily Modified Phanteks P400
Power Supply	Dell TFX Non-standard \| EVGA BQ 650W
Mouse	Monster No-Name $7 Gaming Mouse\| TBD

System Name	Main PC
Processor	13700k
Motherboard	Asrock Z690 Steel Legend D4 - Bios 13.02
Cooling	Noctua NH-D15S
Memory	32 Gig 3200CL14
Video Card(s)	4080 RTX SUPER FE 16G
Storage	1TB 980 PRO, 2TB SN850X, 2TB DC P4600, 1TB 860 EVO, 2x 3TB WD Red, 2x 4TB WD Red
Display(s)	LG 27GL850
Case	Fractal Define R4
Audio Device(s)	Soundblaster AE-9
Power Supply	Antec HCG 750 Gold
Software	Windows 10 21H2 LTSC

System Name	My Ryzen 7 7700X Super Computer
Processor	AMD Ryzen 7 7700X
Motherboard	Gigabyte B650 Aorus Elite AX
Cooling	DeepCool AK620 with Arctic Silver 5
Memory	2x16GB G.Skill Trident Z5 NEO DDR5 EXPO (CL30)
Video Card(s)	XFX AMD Radeon RX 7900 GRE
Storage	Samsung 980 EVO 1 TB NVMe SSD (System Drive), Samsung 970 EVO 500 GB NVMe SSD (Game Drive)
Display(s)	Acer Nitro XV272U (DisplayPort) and Acer Nitro XV270U (DisplayPort)
Case	Lian Li LANCOOL II MESH C
Audio Device(s)	On-Board Sound / Sony WH-XB910N Bluetooth Headphones
Power Supply	MSI A850GF
Mouse	Logitech M705
Keyboard	Steelseries
Software	Windows 11 Pro 64-bit
Benchmark Scores	https://valid.x86.fr/liwjs3

System Name	"Icy Resurrection"
Processor	13th Gen Intel Core i9-13900KS Special Edition
Motherboard	ASUS ROG Maximus Z790 Apex Encore
Cooling	Noctua NH-D15S upgraded with 2x NF-F12 iPPC-3000 fans and Honeywell PTM7950 TIM
Memory	32 GB G.SKILL Trident Z5 RGB F5-6800J3445G16GX2-TZ5RK @ 7600 MT/s 36-44-44-52-96 1.4V
Video Card(s)	ASUS ROG Strix GeForce RTX™ 4080 16GB GDDR6X White OC Edition
Storage	500 GB WD Black SN750 SE NVMe SSD + 4 TB WD Red Plus WD40EFPX HDD
Display(s)	55-inch LG G3 OLED
Case	Pichau Mancer CV500 White Edition
Audio Device(s)	Apple USB-C + Sony MDR-V7 headphones
Power Supply	EVGA 1300 G2 1.3kW 80+ Gold
Mouse	Microsoft Classic Intellimouse
Keyboard	IBM Model M type 1391405 (distribución española)
Software	Windows 11 IoT Enterprise LTSC 24H2
Benchmark Scores	I pulled a Qiqi~

System Name	Black MC in Tokyo
Processor	Ryzen 5 7600
Motherboard	MSI X670E Gaming Plus Wifi
Cooling	Be Quiet! Pure Rock 2
Memory	2 x 16GB Corsair Vengeance @ 6000Mhz
Video Card(s)	XFX 6950XT Speedster MERC 319
Storage	Kingston KC3000 1TB \| WD Black SN750 2TB \|WD Blue 1TB x 2 \| Toshiba P300 2TB \| Seagate Expansion 8TB
Display(s)	Samsung U32J590U 4K + BenQ GL2450HT 1080p
Case	Fractal Design Define R4
Audio Device(s)	Plantronics 5220, Nektar SE61 keyboard
Power Supply	Corsair RM850x v3
Mouse	Logitech G602
Keyboard	Dell SK3205
Software	Windows 10 Pro
Benchmark Scores	Rimworld 4K ready!

System Name	Dark Palimpsest
Processor	Intel i9 13900k with Optimus Foundation Block
Motherboard	EVGA z690 Classified
Cooling	MO-RA3 420mm Custom Loop
Memory	G.Skill 6000CL30, 64GB
Video Card(s)	Nvidia 4090 FE with Heatkiller Block
Storage	3 NVMe SSDs, 2TB-each, plus a SATA SSD
Display(s)	Gigabyte FO32U2P (32" QD-OLED) , Asus ProArt PA248QV (24")
Case	Be quiet! Dark Base Pro 900
Audio Device(s)	Logitech G Pro X
Power Supply	Be quiet! Straight Power 12 1200W
Mouse	Logitech G502 X
Keyboard	GMMK Pro + Numpad

System Name	H7 Flow 2024
Processor	AMD 5800X3D
Motherboard	Asus X570 Tough Gaming
Cooling	Custom liquid
Memory	32 GB DDR4
Video Card(s)	Intel ARC A750
Storage	Crucial P5 Plus 2TB.
Display(s)	AOC 24" Freesync 1m.s. 75Hz
Mouse	Lenovo
Keyboard	Eweadn Mechanical
Software	W11 Pro 64 bit