Forum outage 01/10/2018

Forum News, Announcements and any issues you're having with the forum

Moderator: myglaren

User avatar
xantia_v6
Forum Admin Team
Posts: 5794
Joined: 09 Nov 2005, 23:03
x 236

Forum outage 01/10/2018

Post by xantia_v6 » 01 Oct 2018, 11:48

This forum was down for about 5 hours this morning (1/10/2018).

The exact cause of the failure is not clear yet, but while attempting to resolve a server performance issue, it was found that the forum database was corrupted and has been subsequently restored from backup. This process took rather longer than normal.

Fortunately the failure occurred shortly after the daily backup was taken so there were no more than about 2 posts lost.

My apologies to anyone inconvenienced.

User avatar
Paul-R
Posts: 2871
Joined: 07 May 2009, 16:24
x 169

Re: Forum outage 01/10/2018

Post by Paul-R » 01 Oct 2018, 11:51

Welcome back - it wasn't here five minutes ago.

User avatar
MikeT
Posts: 4628
Joined: 11 Jun 2007, 16:17
x 159

Re: Forum outage 01/10/2018

Post by MikeT » 01 Oct 2018, 13:33

Not saying it's the forum server Mike but over the weekend the forum randomly failed to load a few times for me.

User avatar
xantia_v6
Forum Admin Team
Posts: 5794
Joined: 09 Nov 2005, 23:03
x 236

Re: Forum outage 01/10/2018

Post by xantia_v6 » 01 Oct 2018, 13:42

It was probably the onset of the same issue.

I think that there were some hung threads in the http server which were gradually sapping the server performance, and I was just unlucky that when I restarted the http service this morning the database was left in a state where it became corrupted after the restart. It took a couple of hours of testing to get to the point where I was sure that it was a database corruption and not something else (the symptoms were very strange). Then the database restore (which normally takes about an hour) somehow did not work the first time, and I aborted it after 2 hours, then started it again.

User avatar
Skull
donor 2017
Posts: 386
Joined: 02 Aug 2013, 13:35
x 12

Re: Forum outage 01/10/2018

Post by Skull » 06 Oct 2018, 11:06

Contact us message sent

May be a problem my end but just seems to affect FCF :?:
fcf 10.45.png
Scrrenshot

User avatar
xantia_v6
Forum Admin Team
Posts: 5794
Joined: 09 Nov 2005, 23:03
x 236

Re: Forum outage 01/10/2018

Post by xantia_v6 » 06 Oct 2018, 13:29

Thanks again for the report. The server is getting intermittently overloaded, I have seen loads 5000% higher than expected. This is causing a number of symptoms, but the root cause is still being investigated.

User avatar
MikeT
Posts: 4628
Joined: 11 Jun 2007, 16:17
x 159

Re: Forum outage 01/10/2018

Post by MikeT » 06 Oct 2018, 14:14

Bloody Russians

:twisted:

User avatar
GiveMeABreak
Forum Admin Team
Posts: 11584
Joined: 15 Sep 2015, 19:38
x 989

Re: Forum outage 01/10/2018

Post by GiveMeABreak » 06 Oct 2018, 14:42

Good point - I'm sure our servers aren't hosted in Holland!

User avatar
xantia_v6
Forum Admin Team
Posts: 5794
Joined: 09 Nov 2005, 23:03
x 236

Re: Forum outage 01/10/2018

Post by xantia_v6 » 06 Oct 2018, 16:04

It could also be the Americans...

User avatar
GiveMeABreak
Forum Admin Team
Posts: 11584
Joined: 15 Sep 2015, 19:38
x 989

Re: Forum outage 01/10/2018

Post by GiveMeABreak » 06 Oct 2018, 18:24

I think it's more likely to be Brussels though...

Hell Razor5543
NOT Alistair or Simon
Posts: 8467
Joined: 01 Apr 2012, 09:47
x 501

Re: Forum outage 01/10/2018

Post by Hell Razor5543 » 06 Oct 2018, 21:08

What about the Chinese or North Koreans?

I doubt this is helping Mike one little bit, but we can have fun trying to guess the cause! :D

User avatar
bobins
Donor 2016
Posts: 1878
Joined: 05 Jul 2012, 18:07
x 366

Re: Forum outage 01/10/2018

Post by bobins » 07 Oct 2018, 09:04

Might be the French taking exception to all this free info on here that's prolonging the life of their cars and stopping them from selling new ones :wink: