Technical Issues - TPU Main Site & Forum (2023)

[Mussels]

saw the outages, but all seems good today

 

[lexluthermiester]

I'm seeing a lot of that "You have to wait X seconds before you can post". Recent change?

 

[Solaris17]

I'm seeing a lot of that "You have to wait X seconds before you can post". Recent change?

Maybe a rate limiter because your goin wild

 

[lexluthermiester]

Maybe a rate limiter because your goin wild

View attachment 305953

I'm just responding. No spamming involved, just conversation.

 

[Mussels]

And we're offline, i guess lex is posting heaps again

 

[W1zzard]

And we're offline, i guess lex is posting heaps again

Some clowns have been ddosing us over the last few days.. but only for a few minutes, so no big deal

 

[lexluthermiester]

Some clowns have been ddosing us over the last few days.. but only for a few minutes, so no big deal

Ah, ok. So this is a response to an attack. Understood. It is a bit annoying, but only slightly.

 

[W1zzard]

Ah, ok. So this is a response to an attack. Understood. It is a bit annoying, but only slightly.

More like the effect of an attack, not a planned response from our side

DDOS has never been an issue for us in almost 20 years, so I never put much effort into mitigation, and even right now, not sure if spending days of work to protect against minutes of downtime. If these get more serious though, I'm 100% committed to protect my site and keep it functional

 

[neatfeatguy]

Some clowns have been ddosing us over the last few days.. but only for a few minutes, so no big deal

Techspot has been having a lot of issues with ddos recently. I had been wondering why it's been a pain to post on their site over the past week or so. I thought it was something to do with Chrome or cookies/cache so I cleared them out, but the issue remained. I even tried the site on Firefox, but the issue was the same. When I came across this post the other night it made sense as to why the site was being finicky.

Here's a post from a forum moderator there (https://www.techspot.com/community/topics/getting-errors-today.281470/#post-2036434):

"Apologies for the site issues and other annoyances. We've been dealing with incessant DDoS attacks for the past month or so. It was slowing down the servers and causing other issues like brief downtimes. We tried a few solutions but nothing worked as effectively as enabling a WAF at the edge of our servers, however that in itself comes with problems of its own.

We are already exploring a different WAF that is faster and more effective, but transitioning to a different provider takes time. The current solution messes up the forum posting, etc. because it verifies the same user every 5 minutes or so. For now, the "trick" is opening another TechSpot page before submitting anything.

Everybody is also getting that flash/blank page (WAF verification) but the site is fast and accessible now -- so that's where we are for now. To give you further context, in the last 24 hours, the WAF says it's blocked over 2 million DDoS connection attempts to our servers.

As for proper configuration, believe us, we've tried everything with our current provider but in short, it's a horrible setup and the reason we only used them as a CDN and not for the WAF functionality. I hope we can get us on a better platform sometime next week."

 

[Assimilator]

Seen a couple of 502s myself this morning, so attacks definitely seem to be ramping up.

 

[lexluthermiester]

More like the effect of an attack, not a planned response from our side

DDOS has never been an issue for us in almost 20 years, so I never put much effort into mitigation, and even right now, not sure if spending days of work to protect against minutes of downtime. If these get more serious though, I'm 100% committed to protect my site and keep it functional

Yeah, no worries. After you mentioned what was going on, I totally understood. That kind of thing is a PITB.

 

[Mussels]

I'd be worried its to cover up a hacking attempt - we dont need to see w1zzards wand like we did naked linus.

 

[W1zzard]

I'd be worried its to cover up a hacking attempt

Not sure how that would work? This not the movies

 

[Mussels]

Not sure how that would work? This not the movies

There can be known vulnerabilities that only work under specific circumstances, like when software is booting up and a DDoS can trigger that event

something like spamming the forums to log in and brute force accounts could be a form of DDoS too - but one that's easier for you to know about

 

[Shrek]

Here we go again... 502

 

[W1zzard]

Just got hit by a DDOS attack, banned around 200 IPs and things seem better now

 

[Mussels]

It always makes me curious about things like this, because it could be someone pissed off at a particular website - but it always feels more likely its automated attacks trying to breach vulnerabilities (like those router exploits a few weeks ago, target any IP that sends an email hoping it's vulnerable)

 

[Solaris17]

It always makes me curious about things like this, because it could be someone pissed off at a particular website - but it always feels more likely its automated attacks trying to breach vulnerabilities (like those router exploits a few weeks ago, target any IP that sends an email hoping it's vulnerable)

A little of both. The majority of the time its just bots trying to time out nginx or apache. On really big attacks it always comes down "who has the bigger pipe" when we see crazy things on the carrier level thats when the NOC or networking teams work together. In most cases if you are running something like Noction or WANguard or some other commercial product rules can be setup that handle scrubbing or black holing automatically.

In the event we start taking too much traffic and the appliance is starting to take water you move further afield and start BGP filtering and blocking problematic ASNs if attacks get that large. If you have a strong CDN network and lots of edge bandwidth you can weather most things.

Once the attack enters the inner network you are really just hoping your onsite equipment can handle the load.

Thats a 30k view of what we do. Not TPU mind you. me.

 

[W1zzard]

The majority of the time its just bots trying to time out nginx or apache

Yeah seems to be the case here, too. It's just a few hundred IPs doing HTTP requests, usually not enough to trigger the Level 3 DDOS detection on our upstream. Now that you mention it, I saw an email that they did block a 15 Gbps UDP attack on Monday.

 

[Ferrum Master]

Yeah seems to be the case here, too. It's just a few hundred IPs doing HTTP requests, usually not enough to trigger the Level 3 DDOS detection on our upstream. Now that you mention it, I saw an email that they did block a 15 Gbps UDP attack on Monday.

strange question tho... but who and why?

 

[W1zzard]

strange question tho... but who and why?

no way we will know, it's not enough to disrupt our business, no demands have been sent

 

[Mussels]

strange question tho... but who and why?

For all we know, it's a DDoS botnet of HP printers trying to dial in to some server with a typo in the IP address and they'll end up becoming sentient in a few years.


It's one of those impossible to know things that fascinates me, as there would be so much wasted and malicious network traffic out there - but to do anything serious to fix it at a base level, defeats the purpose of an open internet in the first place

 

[Wirko]

In the event we start taking too much traffic and the appliance is starting to take water you move further afield and start BGP filtering and blocking problematic ASNs if attacks get that large. If you have a strong CDN network and lots of edge bandwidth you can weather most things.

Once the attack enters the inner network you are really just hoping your onsite equipment can handle the load.

A question from a 99.9% n00b: can a server signal to the ISP (or whatever is between it and the backbone), via a standard protocol, that it's fully loaded with requests?

 

[W1zzard]

or all we know, it's a DDoS botnet of HP printers trying to dial in to some server

They speak HTTP 2.0, so not some printers. Another attack had randomly crafted query strings and HTTP referers

A question from a 99.9% n00b: can a server signal to the ISP (or whatever is between it and the backbone), via a standard protocol, that it's fully loaded with requests?

Nothing to the ISP/backbone.

In theory, to the browser, and that will show an error message. But during a HTTP flood the attacker simply sends requests as fast as it can, without looking at the result or errors. Basically like when you're holding down F5 in your browser (please don't try, you might get your IP banned)

 

[Dan.G]

Hmm... CAPTCHA implementation after a certain number of requests / minute?