Forum outages 15/08/2024 - 24/08/2024

[xantia_v6]

This forum was offline for several hours due to a server disk space issue. Preliminary findings are that this was due to a log file runaway, but there don't seem to be any lasting effects.
I will be investigating further.

Sorry for any inconvenience.

[xantia_v6]

Looking a bit deeper, it appears to have (a successful) Denial Of Service attack. The logfiles show that we were receiving about 200,000 requests per second for about 7 minutes before the log files filled the available disk space.

[Vic Evans]

Probably caused by a French car hating nerd with nothing better to do with their time.
Thanks for sorting it .

[mickthemaverick]

Thanks for sorting so promptly Mike, I tried to get on about 5am and hit the issue. Needless to say I went through all sorts of checks and tests to make sure it wasn't my end and eventually decided it was a forum issue and so I went back to bed!!

[jeastham]

Looking a bit deeper, it appears to have (a successful) Denial Of Service attack. The logfiles show that we were receiving about 200,000 requests per second for about 7 minutes before the log files filled the available disk space.

200,000 requests, with each request recording say 100 characters (100 bytes), makes for 20 million bytes per second, or ~8.4 billion bytes over the ~7 minutes.

Somewhere in the region of 7.8231 GiB (assuming 100 bytes per logged request, and 420 seconds duration).

Am I warm?

[xantia_v6]

Yes, there must have been a lot of incoming data. and the server exhausted the available file handles trying to handle the traffic. The surprising thing was that the system recovered completely after I deleted a few hundred error log files, I did not even need to restart the web service.

I have since tightened up a few DOS parameters, but I don't really think it will make much difference, there are some lower-level protections I could configure, but I need to read more about them first.

[jeastham]

I've used "fail2ban" in the past, which will ban hosts that cause multiple authentication errors.......but that's for stopping people ssh-ing in to your server. I doubt it could help with shed-loads of genuine http/https requests :-/

[xantia_v6]

We don't have much information about the exact traffic, but as the http server ran out of file handles, it was probably a properly formed request for static content.

I am still investigating options for hardening the server.

[Hell Razor5543]

Did we just have another outage? I ask because, for a while, I was getting a 500 error (taking too long to respond).

[myglaren]

Same here, 'internal server error' was in the notification.

[xantia_v6]

Yes we did.

It looks just like the previous one, unfortunately I was not watching when it happened, so don't have much more to go on, but I am about to install some better logging which should capture more useful data if it happens again.

[xantia_v6]

It happened again overnight, but at least I have some more diagnostic data. At first sight it looks like a server configuration error rather than an external attack, but I am not sure yet.

[Vic Evans]

It happened again overnight, but at least I have some more diagnostic data. At first sight it looks like a server configuration error rather than an external attack, but I am not sure yet.

Can't think of any reason why someone would attack this place ....unless it's some of the jokes

[Rhothgar]

Is www. down? Because I only have https://frenchcarforum.co.uk/forum come up now?

Noticed this a couple of times in the last few days.

We need to do whatever we can to protect the assets of this forum and all the hard work. Naturally, I do not know who hosts the site etc..

If the site is experiencing these numerous DOS attacks from China or Russia, can we not just exclude access from domains located in RU or CN? I do this on my business website so no-one can access from those countries.

Maybe it is worth gathering some worst case scenario site statistics and then throttling the number of requests that can be made within a set period of time. So, for instance, if the site receives 1000 hits a day then limit any incoming IP address to no more than 100 hits in say 60 seconds and then ban that IP for 24 hours or so.

It's of grave concern if this resource goes down and all the work it has taken to build it up over the years and knowledge.

If there is anything I can do to help, let me know even if it means me having a remote clone backup of all the site data I'd happily have a hard drive available for that.

Is there a facility to download the log files in future to a USB and then delete from server so they can be looked into to see what the issue is?

Of course, I have no idea how the site is hosted but I am guessing it is not hosted on someone's personal server room and is probably on the Cloud somewhere? If on someone's own servers then I'd recommend a DrayTek Vigor router for incoming connections. Never done a BB so there may be more to it.

I use a double router configuration for my setup. Virgin Media router with a DrayTek piggy-backed onto the VM router. DrayTek have superb customer support service.

You may be able to edit the .htaccess file to limit countries.

And you don't need to apologise considering all the hard work you undertake for the forum.

[myglaren]

Historically any attempts to register by Russians was disallowed but they are much better now and no worse than most other places.
Scandinavians are the least bothersome group.