Technical Issues - TPU Main Site & Forum (2024)

[Noci]

Site is chugging along nicely now. Could still be a little bit faster

Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35), also moved our shared filesystem (Gluster) into containers.
Also no more plain Docker, and switched to Podman for the cases where I just need a fs to mess around in.
Migrated most services into the new cluster, which no longer uses Zerotier. ZT looks great on paper and I love that it can do ARP because it's L2, but under serious load it just fell apart randomly and unpredictably.

Now running with Netmaker, which uses kernel Wireguard, no more dynamic/floating IPs Also not 100% happy with the current Cilium setup.

This new cluster setup is engineered as blue/green, so it should be easier to switch live between them.

Another interesting issue I found today, and that's probably the cause for most 502's yday, was that the control plane node serving DNS queries (CoreDNS) was overloaded, so some DNS queries would randomly fail. MySQL Router does periodical health checks and (obviously) uses DNS in the process, if such a query failed, it would mark the whole backend as failed and the DB access would just disappear.

To me, most of it sounds like IT Abracadabra alakazam, but that's probably why you're called the W1zzard .

But from a simple forum user's point of view, it seems to be running fine now. Thank you for fixing it .

 

[chrcoluk]

Appreciate the technical explanation, much more going on behind the scenes than what I assumed.

 

[trparky]

Site is chugging along nicely now. Could still be a little bit faster

Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35), also moved our shared filesystem (Gluster) into containers.
Also no more plain Docker, and switched to Podman for the cases where I just need a fs to mess around in.
Migrated most services into the new cluster, which no longer uses Zerotier. ZT looks great on paper and I love that it can do ARP because it's L2, but under serious load it just fell apart randomly and unpredictably.

Now running with Netmaker, which uses kernel Wireguard, no more dynamic/floating IPs Also not 100% happy with the current Cilium setup.

This new cluster setup is engineered as blue/green, so it should be easier to switch live between them.

Another interesting issue I found today, and that's probably the cause for most 502's yday, was that the control plane node serving DNS queries (CoreDNS) was overloaded, so some DNS queries would randomly fail. MySQL Router does periodical health checks and (obviously) uses DNS in the process, if such a query failed, it would mark the whole backend as failed and the DB access would just disappear.

I understood about half of that.

Wireguard, yep... know what that is. L2, ARP... that's got stuff to do with the OSI model. I know about MySQL and have worked with it from a server management point of view, but I've never had a setup MySQL clusters before.

 

[Solaris17]

Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35), also moved our shared filesystem (Gluster) into containers.
Also no more plain Docker, and switched to Podman for the cases where I just need a fs to mess around in.

lets goooooooo

 

[RJARRRPCGP]

Finally got the last of our core servers reinstalled to Ubuntu (from old old Fedora 35)

Was Fedora 35 even getting security fixes?!

 

[W1zzard]

Was Fedora 35 even getting security fixes?!

More or less, but everything ran in Docker, all ports closed, I'm the only one with filesystem access

 

[Onasi]

Amusing bug I noticed after updating my phone to iOS 18 finally - the forum loading bar that’s showing up at the top of the page sometimes just… stays there, scrolling endlessly. The page is fine, the Safari itself says that no loading is being done, but it just stays. No effect on functionality and doesn’t always replicate (reloading the page usually solves it), but is quite funny. In the instance of this screenshot I watched it for a minute, lol. Probably a Safari bug?

 

[lexluthermiester]

Amusing bug I noticed after updating my phone to iOS 18 finally - the forum loading bar that’s showing up at the top of the page sometimes just… stays there, scrolling endlessly. The page is fine, the Safari itself says that no loading is being done, but it just stays. No effect on functionality and doesn’t always replicate (reloading the page usually solves it), but is quite funny. In the instance of this screenshot I watched it for a minute, lol. Probably a Safari bug?
View attachment 373139

Wait, you use TPU on a phone?!? Seriously?

 

[Onasi]

@lexluthermiester
Sometimes, sure. Like right now when I am in my kitchen drinking tea. Why not? Works fine.

 

[trparky]

I use the site on my iPhone as well.

 

[lexluthermiester]

Good grief! .... Fair enough, I digress.

 

[Solaris17]

@lexluthermiester
Sometimes, sure. Like right now when I am in my kitchen drinking tea. Why not? Works fine.

I use the site on my iPhone as well.

I think there are days most of my time at TPU is on my iPhone.

 

[W1zzard]

Wait, you use TPU on a phone?!? Seriously?

TPU is designed to work fine on a phone. Are you having any specific issues?

 

[chrcoluk]

On phone I use the TPU app.

 

[lexluthermiester]

TPU is designed to work fine on a phone. Are you having any specific issues?

Not per-se. I just don't do the internet on a phone. Never have, never will. Can't stand the small screens and the lack of an actual and real keyboard. To me that's just crazy talk..

 

[sneekypeet]

Not per-se. I just don't do the internet on a phone. Never have, never will. Can't stand the small screens and the lack of an actual and real keyboard. To me that's just crazy talk..

Honestly, I used to be that way as well, but the size is less of an issue than I assumed using it initially, but I do agree that the keyboards suck on mobile, and the autocorrect update blows a$$. Moderation or aproval of posts is pretty easy to do, and without it, I would assume a lot of the mods would be online a lot less.

Not harshing your stance, but through a typical browser, TPU is much the same except having to rotate the phone to see specs.

 

[lexluthermiester]

Honestly, I used to be that way as well, but the size is less of an issue than I assumed using it initially, but I do agree that the keyboards suck on mobile, and the autocorrect update blows a$$. Moderation or aproval of posts is pretty easy to do, and without it, I would assume a lot of the mods would be online a lot less.

Not harshing your stance, but through a typical browser, TPU is much the same except having to rotate the phone to see specs.

I do try it on a regular basis or whenever there's a pinch. It's just not my jam. Can't be bother with it.

 

[Count von Schwalbe]

TPU is designed to work fine on a phone. Are you having any specific issues?

TPU does seem to hang on load more often on mobile, but I wouldn't swear that isn't my phone thrashing networks. A quick reload usually fixes it.

 

[RJARRRPCGP]

Amusing bug I noticed after updating my phone to iOS 18 finally - the forum loading bar that’s showing up at the top of the page sometimes just… stays there, scrolling endlessly. The page is fine, the Safari itself says that no loading is being done, but it just stays. No effect on functionality and doesn’t always replicate (reloading the page usually solves it), but is quite funny. In the instance of this screenshot I watched it for a minute, lol. Probably a Safari bug?
View attachment 373139

I get something similar on Firefox here on my daily-driver. It looks like a hung JS. Often, when I reply. Even with no JS blockers!

 

[lexluthermiester]

Often, when I reply. Even with no JS blockers!

I've been seeing that too, it's not just mobile. I'll go to post a comment and it hangs for 10 or 15 seconds while it finishes.
EDIT: correction.

 

[RJARRRPCGP]

I get the issue on Windows, so it doesn't look like just mobile.

 

[lexluthermiester]

I get the issue on Windows, so it doesn't look like just mobile.

Sorry, see edit..

 

[W1zzard]

I get something similar on Firefox here on my daily-driver. It looks like a hung JS. Often, when I reply. Even with no JS blockers!

Open Devtools (F12), Network tab, which request is the one that hangs?

 

[Frick]

Not per-se. I just don't do the internet on a phone. Never have, never will. Can't stand the small screens and the lack of an actual and real keyboard. To me that's just crazy talk..

It is far more exotic to not use a phone for the internet.

 

[Onasi]

Okay, weird issue - the site keeps signing me out. Issue has been replicable for the last half hour or so. No idea why, I even reset the password to try and solve this, but once in a while it just hangs and then I see myself logged out.