Technical Issues - TPU Main Site & Forum (2024)

Noci · Nov 23, 2024

W1zzard said:
Site is chugging along nicely now. Could still be a little bit faster

Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35), also moved our shared filesystem (Gluster) into containers.
Also no more plain Docker, and switched to Podman for the cases where I just need a fs to mess around in.
Migrated most services into the new cluster, which no longer uses Zerotier. ZT looks great on paper and I love that it can do ARP because it's L2, but under serious load it just fell apart randomly and unpredictably.

Now running with Netmaker, which uses kernel Wireguard, no more dynamic/floating IPs Also not 100% happy with the current Cilium setup.

This new cluster setup is engineered as blue/green, so it should be easier to switch live between them.

Another interesting issue I found today, and that's probably the cause for most 502's yday, was that the control plane node serving DNS queries (CoreDNS) was overloaded, so some DNS queries would randomly fail. MySQL Router does periodical health checks and (obviously) uses DNS in the process, if such a query failed, it would mark the whole backend as failed and the DB access would just disappear.

To me, most of it sounds like IT Abracadabra alakazam, but that's probably why you're called the W1zzard :roll:

.

But from a simple forum user's point of view, it seems to be running fine now. Thank you for fixing it :respect:

.

chrcoluk · Nov 23, 2024

Appreciate the technical explanation, much more going on behind the scenes than what I assumed.

trparky · Nov 23, 2024

W1zzard said:
Site is chugging along nicely now. Could still be a little bit faster

Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35), also moved our shared filesystem (Gluster) into containers.
Also no more plain Docker, and switched to Podman for the cases where I just need a fs to mess around in.
Migrated most services into the new cluster, which no longer uses Zerotier. ZT looks great on paper and I love that it can do ARP because it's L2, but under serious load it just fell apart randomly and unpredictably.

Now running with Netmaker, which uses kernel Wireguard, no more dynamic/floating IPs Also not 100% happy with the current Cilium setup.

This new cluster setup is engineered as blue/green, so it should be easier to switch live between them.

Another interesting issue I found today, and that's probably the cause for most 502's yday, was that the control plane node serving DNS queries (CoreDNS) was overloaded, so some DNS queries would randomly fail. MySQL Router does periodical health checks and (obviously) uses DNS in the process, if such a query failed, it would mark the whole backend as failed and the DB access would just disappear.

I understood about half of that.

Wireguard, yep... know what that is. L2, ARP... that's got stuff to do with the OSI model. I know about MySQL and have worked with it from a server management point of view, but I've never had a setup MySQL clusters before.

Solaris17 · Nov 23, 2024

W1zzard said:
Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35), also moved our shared filesystem (Gluster) into containers.
Also no more plain Docker, and switched to Podman for the cases where I just need a fs to mess around in.

lets goooooooo

RJARRRPCGP · Nov 23, 2024

W1zzard said:
Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35)

Was Fedora 35 even getting security fixes?!

W1zzard · Nov 23, 2024

RJARRRPCGP said:
Was Fedora 35 even getting security fixes?!

More or less, but everything ran in Docker, all ports closed, I'm the only one with filesystem access

Onasi · Sunday at 7:29 PM

Amusing bug I noticed after updating my phone to iOS 18 finally - the forum loading bar that’s showing up at the top of the page sometimes just… stays there, scrolling endlessly. The page is fine, the Safari itself says that no loading is being done, but it just stays. No effect on functionality and doesn’t always replicate (reloading the page usually solves it), but is quite funny. In the instance of this screenshot I watched it for a minute, lol. Probably a Safari bug?

lexluthermiester · Sunday at 8:09 PM

Onasi said:
Amusing bug I noticed after updating my phone to iOS 18 finally - the forum loading bar that’s showing up at the top of the page sometimes just… stays there, scrolling endlessly. The page is fine, the Safari itself says that no loading is being done, but it just stays. No effect on functionality and doesn’t always replicate (reloading the page usually solves it), but is quite funny. In the instance of this screenshot I watched it for a minute, lol. Probably a Safari bug?
View attachment 373139

Wait, you use TPU on a phone?!? Seriously?

Onasi · Sunday at 8:10 PM

@lexluthermiester
Sometimes, sure. Like right now when I am in my kitchen drinking tea. Why not? Works fine.

trparky · Sunday at 8:37 PM

I use the site on my iPhone as well.

lexluthermiester · Sunday at 8:38 PM

Good grief! :eek:

.... Fair enough, I digress.

Solaris17 · Sunday at 9:05 PM

Onasi said:
@lexluthermiester
Sometimes, sure. Like right now when I am in my kitchen drinking tea. Why not? Works fine.

trparky said:
I use the site on my iPhone as well.

I think there are days most of my time at TPU is on my iPhone.

W1zzard · Monday at 10:18 AM

lexluthermiester said:
Wait, you use TPU on a phone?!? Seriously?

TPU is designed to work fine on a phone. Are you having any specific issues?

chrcoluk · Monday at 12:40 PM

On phone I use the TPU app.

lexluthermiester · Monday at 3:12 PM

W1zzard said:
TPU is designed to work fine on a phone. Are you having any specific issues?

Not per-se. I just don't do the internet on a phone. Never have, never will. Can't stand the small screens and the lack of an actual and real keyboard. To me that's just crazy talk..

sneekypeet · Monday at 3:36 PM

lexluthermiester said:
Not per-se. I just don't do the internet on a phone. Never have, never will. Can't stand the small screens and the lack of an actual and real keyboard. To me that's just crazy talk..

Honestly, I used to be that way as well, but the size is less of an issue than I assumed using it initially, but I do agree that the keyboards suck on mobile, and the autocorrect update blows a$$. Moderation or aproval of posts is pretty easy to do, and without it, I would assume a lot of the mods would be online a lot less.

Not harshing your stance, but through a typical browser, TPU is much the same except having to rotate the phone to see specs.

lexluthermiester · Monday at 3:45 PM

sneekypeet said:
Honestly, I used to be that way as well, but the size is less of an issue than I assumed using it initially, but I do agree that the keyboards suck on mobile, and the autocorrect update blows a$$. Moderation or aproval of posts is pretty easy to do, and without it, I would assume a lot of the mods would be online a lot less.

Not harshing your stance, but through a typical browser, TPU is much the same except having to rotate the phone to see specs.

I do try it on a regular basis or whenever there's a pinch. It's just not my jam. Can't be bother with it.

Count von Schwalbe · Monday at 4:21 PM

W1zzard said:
TPU is designed to work fine on a phone. Are you having any specific issues?

TPU does seem to hang on load more often on mobile, but I wouldn't swear that isn't my phone thrashing networks. A quick reload usually fixes it.

RJARRRPCGP · Monday at 11:51 PM

Onasi said:
Amusing bug I noticed after updating my phone to iOS 18 finally - the forum loading bar that’s showing up at the top of the page sometimes just… stays there, scrolling endlessly. The page is fine, the Safari itself says that no loading is being done, but it just stays. No effect on functionality and doesn’t always replicate (reloading the page usually solves it), but is quite funny. In the instance of this screenshot I watched it for a minute, lol. Probably a Safari bug?
View attachment 373139

I get something similar on Firefox here on my daily-driver. It looks like a hung JS. Often, when I reply. Even with no JS blockers!

lexluthermiester · Monday at 11:58 PM

RJARRRPCGP said:
Often, when I reply. Even with no JS blockers!

I've been seeing that too, it's not just mobile. I'll go to post a comment and it hangs for 10 or 15 seconds while it finishes.
EDIT: correction.

RJARRRPCGP · Tuesday at 12:48 AM

I get the issue on Windows, so it doesn't look like just mobile.

lexluthermiester · Tuesday at 7:17 AM

RJARRRPCGP said:
I get the issue on Windows, so it doesn't look like just mobile.

Sorry, see edit..

W1zzard · Tuesday at 7:52 AM

RJARRRPCGP said:
I get something similar on Firefox here on my daily-driver. It looks like a hung JS. Often, when I reply. Even with no JS blockers!

Open Devtools (F12), Network tab, which request is the one that hangs?

Frick · Tuesday at 11:06 AM

lexluthermiester said:
Not per-se. I just don't do the internet on a phone. Never have, never will. Can't stand the small screens and the lack of an actual and real keyboard. To me that's just crazy talk..

It is far more exotic to not use a phone for the internet.

Onasi · Tuesday at 6:49 PM

Okay, weird issue - the site keeps signing me out. Issue has been replicable for the last half hour or so. No idea why, I even reset the password to try and solve this, but once in a while it just hangs and then I see myself logged out.

System Name	The Dark side of the room
Processor	AMD Ryzen 9 5900X
Motherboard	MSI MEG X570 Unify
Cooling	Custom loop watercooling (Bykski CPU-XPR-POM-M-V2, Alphacool Eisblock GPX, Freezemod PU-PWM5B18W)
Memory	GSkill Ripjaws V DDR4 3600 CL16 (4 x 16GB)
Video Card(s)	XFX Speedster QICK 319 Radeon RX 6700 XT
Storage	1 x Kingston KC3000 1024GB (boot drive) + 2 x Kingston NV2 2TB (games & storage)
Display(s)	LG 34WP65C Ultrawide 3440x1440 @ 160Hz freesync premium
Case	Thermaltake Core P90 TG (slightly modded)
Audio Device(s)	onboard Realtek® ALC1220 with Logitech Z906
Power Supply	MSI MAG A850GF 80 Plus Gold
Mouse	Generic
Keyboard	Sharkoon Skiller SGK60 (with brown Kalih switches)
Software	Windows 11 pro
Benchmark Scores	It's a form of exhibitionism...;-), but fun in a way But showing off is triggering.............

System Name	Main PC
Processor	13700k
Motherboard	Asrock Z690 Steel Legend D4 - Bios 13.02
Cooling	Noctua NH-D15S
Memory	32 Gig 3200CL14
Video Card(s)	4080 RTX SUPER FE 16G
Storage	1TB 980 PRO, 2TB SN850X, 2TB DC P4600, 1TB 860 EVO, 2x 3TB WD Red, 2x 4TB WD Red
Display(s)	LG 27GL850
Case	Fractal Define R4
Audio Device(s)	Soundblaster AE-9
Power Supply	Antec HCG 750 Gold
Software	Windows 10 21H2 LTSC

System Name	My Ryzen 7 7700X Super Computer
Processor	AMD Ryzen 7 7700X
Motherboard	Gigabyte B650 Aorus Elite AX
Cooling	DeepCool AK620 with Arctic Silver 5
Memory	2x16GB G.Skill Trident Z5 NEO DDR5 EXPO (CL30)
Video Card(s)	XFX AMD Radeon RX 7900 GRE
Storage	Samsung 980 EVO 1 TB NVMe SSD (System Drive), Samsung 970 EVO 500 GB NVMe SSD (Game Drive)
Display(s)	Acer Nitro XV272U (DisplayPort) and Acer Nitro XV270U (DisplayPort)
Case	Lian Li LANCOOL II MESH C
Audio Device(s)	On-Board Sound / Sony WH-XB910N Bluetooth Headphones
Power Supply	MSI A850GF
Mouse	Logitech M705
Keyboard	Steelseries
Software	Windows 11 Pro 64-bit
Benchmark Scores	https://valid.x86.fr/liwjs3

System Name	RogueOne
Processor	Xeon W9-3495x
Motherboard	ASUS w790E Sage SE
Cooling	SilverStone XE360-4677
Memory	128gb Gskill Zeta R5 DDR5 RDIMMs
Video Card(s)	MSI SUPRIM Liquid X 4090
Storage	1x 2TB WD SN850X \| 2x 8TB GAMMIX S70
Display(s)	49" Philips Evnia OLED (49M2C8900)
Case	Thermaltake Core P3 Pro Snow
Audio Device(s)	Moondrop S8's on schitt Gunnr
Power Supply	Seasonic Prime TX-1600
Mouse	Lamzu Maya Grey
Keyboard	Monsgeek M3 Lavender, Moondrop Luna lights
VR HMD	Quest 3
Software	Windows 11 Pro Workstation
Benchmark Scores	I dont have time for that.

System Name	KHR-1
Processor	Ryzen 9 5900X
Motherboard	ASRock B550 PG Velocita (UEFI-BIOS P3.40)
Memory	32 GB G.Skill RipJawsV F4-3200C16D-32GVR
Video Card(s)	Sapphire Nitro+ Radeon RX 6750 XT
Storage	Western Digital Black SN850 1 TB NVMe SSD
Display(s)	Alienware AW3423DWF OLED-ASRock PG27Q15R2A (backup)
Case	Corsair 275R
Audio Device(s)	Technics SA-EX140 receiver with Polk VT60 speakers
Power Supply	eVGA Supernova G3 750W
Mouse	Logitech G Pro (Hero)
Software	Windows 11 Pro x64 23H2

Technical Issues - TPU Main Site & Forum (2024)

Noci

chrcoluk

trparky

Solaris17

Super Dainty Moderator

RJARRRPCGP

W1zzard

Administrator

Onasi

lexluthermiester

Onasi

trparky

lexluthermiester

Solaris17

Super Dainty Moderator

W1zzard

Administrator

chrcoluk

lexluthermiester

sneekypeet

not-so supermod

lexluthermiester

Count von Schwalbe

Moderator

RJARRRPCGP

lexluthermiester

RJARRRPCGP

lexluthermiester

W1zzard

Administrator

Frick

Fishfaced Nincompoop

Onasi

Processor	Ryzen 7 5700X
Memory	48 GB
Video Card(s)	RTX 4080
Storage	2x HDD RAID 1, 3x M.2 NVMe
Display(s)	30" 2560x1600 + 19" 1280x1024
Software	Windows 10 64-bit

System Name	The Workhorse
Processor	AMD Ryzen R9 5900X
Motherboard	Gigabyte Aorus B550 Pro
Cooling	CPU - Noctua NH-D15S Case - 3 Noctua NF-A14 PWM at the bottom, 2 Fractal Design 180mm at the front
Memory	GSkill Trident Z 3200CL14
Video Card(s)	NVidia GTX 1070 MSI QuickSilver
Storage	Adata SX8200Pro
Display(s)	LG 32GK850G
Case	Fractal Design Torrent (Solid)
Audio Device(s)	FiiO E-10K DAC/Amp, Samson Meteorite USB Microphone
Power Supply	Corsair RMx850 (2018)
Mouse	Razer Viper (Original) on a X-Raypad Equate Plus V2
Keyboard	Cooler Master QuickFire Rapid TKL keyboard (Cherry MX Black)
Software	Windows 11 Pro (24H2)

System Name	EVA-01
Processor	Intel i7 13700K
Motherboard	Asus ROG Maximus Z690 HERO EVA Edition
Cooling	ASUS ROG Ryujin III 360 with Noctua Industrial Fans
Memory	PAtriot Viper Elite RGB 96GB @ 6000MHz.
Video Card(s)	Asus ROG Strix GeForce RTX 3090 24GB OC EVA Edition
Storage	Addlink S95 M.2 PCIe GEN 4x4 2TB
Display(s)	Asus ROG SWIFT OLED PG42UQ
Case	Thermaltake Core P3 TG
Audio Device(s)	Realtek on board > Sony Receiver > Cerwin Vegas
Power Supply	be quiet DARK POWER PRO 12 1500W
Mouse	ROG STRIX Impact Electro Punk
Keyboard	ROG STRIX Scope TKL Electro Punk
Software	Windows 11

System Name	Work Computer \| Unfinished Computer
Processor	Core i7-6700 \| Ryzen 5 5600X
Motherboard	Dell Q170 \| Gigabyte Aorus Elite Wi-Fi
Cooling	A fan? \| Truly Custom Loop
Memory	4x4GB Crucial 2133 C17 \| 4x8GB Corsair Vengeance RGB 3600 C26
Video Card(s)	Dell Radeon R7 450 \| RTX 2080 Ti FE
Storage	Crucial BX500 2TB \| TBD
Display(s)	3x LG QHD 32" GSM5B96 \| TBD
Case	Dell \| Heavily Modified Phanteks P400
Power Supply	Dell TFX Non-standard \| EVGA BQ 650W
Mouse	Monster No-Name $7 Gaming Mouse\| TBD

System Name	White DJ in Detroit
Processor	Ryzen 5 5600
Motherboard	Asrock B450M-HDV
Cooling	Be Quiet! Pure Rock 2
Memory	2 x 16GB Kingston Fury 3400mhz
Video Card(s)	XFX 6950XT Speedster MERC 319
Storage	Kingston A400 240GB \| WD Black SN750 2TB \|WD Blue 1TB x 2 \| Toshiba P300 2TB \| Seagate Expansion 8TB
Display(s)	Samsung U32J590U 4K + BenQ GL2450HT 1080p
Case	Fractal Design Define R4
Audio Device(s)	Plantronics 5220, Nektar SE61 keyboard
Power Supply	Corsair RM850x v3
Mouse	Logitech G602
Keyboard	Cherry MX Board 1.0 TKL Brown
Software	Windows 10 Pro
Benchmark Scores	Rimworld 4K ready!