Website problems

I think meanwhile probably everyone noticed the issues we currently have with the website.

"internal error 500" etc.

Cause is a specific username trying to access the databases and hitting a hard limit of permitted db accesses, with the result that the server then blocks further db requests for everyone else.

I originally assumed that somehow malicous code was injected directly into the server that was using a specific username to establish a db connection, but I was not able to find such code in any of the (~390.000) files yet. The username responsible for the db access is completely unknown to me, therefore I am assuming that is a kind of outside injection (or I simply overlooked something). That username would not even have access to the databases since only very specific usernames have permission to access data. Therefore confidential user data is secure.

The main problem is that this username simply hits the limit of simultanous db links. I could in theory ask the server host to increase this limit but that is no guarantee that the problem will go away. It probably just will take that specific username a bit longer to reach the limit.

Considering that we have 3000-4000 guests online on the forum (pretty much at any time) it is higly likely that most of them are either spambots, website crawlers, or ai agents. Isolating the connection responsible for the problems is very tricky.

Therefore I have taken rough measurements to limit connections. The type of db requests was limited aswell as any kind of media cross-linking that is initiated from outside sources (as this could also trigger db accesses). I hope that this will show positive effects in the coming days.

IF not I will have to systematically check if db access through other websites hosted on this server (e.g. Lancersreactor) would be the reason for the problems. In worst case I would have to temporarily shut down these domains.

I will perform further scans of the files just in case I have overlooked a detail in the meantime.

Since I can not be online 24/7 to observe the issues, it would be great if you could report if you experience issues connecting to the forum here on this thread. Just leave a short message including which kind of error you experienced, the time it happened and your timezone of course (Keep in mind: It is always five o'clock somewhere, making it the perfect time for a drink, a nap, or a bad decision.)

Quote from flathack

error occured when i tried to reply to a thread - 21:20 (utc+2)

pasted-from-clipboard.png

that sucks

I guess my solution is not working then.

The server was down for +24 hours.

I had to let a technican take a look at it. Hopefully everything is working again.

Ive done further changes to limit db access and serveral other domains hosted on the server got access restricted temporarily narrow down the source of the problems.

Online atm is only the SWAT Portal and the CF update.

I unfortunately did.

I was scanning some files on the CF server earlier today. Maybe this explains it.

Ive spent the past evening/night restricting the website access to bots even further and have removed a good amount of content that bots left on the forum.

Additionally I have removed 20.000 user accounts created by spam bots. 4000 of them were already identified earlier by the Moderator team (a quick thanks for your work).

Another 5000 accounts so far are currently under review (unfortunatly by hand, to prevent mistakes). I hope that the removal of the bot content will reduce a bit stress on the server. Maybe it will also reduce the amount of bot spam in general (I think certain keywords trigger the creation of new spam content).

However, all this is a bit like fighting windmills atm. In the past 12 hours ~50 new accounts were created (most of them spambots).

The website error just happened (eventhough just for a short moment). My hope is that all measurements combined will make a difference.