So, for the last week you may have noticed that we have been offline. This was not due to any alcohol related trips over server power cables or anything like that.
On Sunday March 31, around mid afternoon UTC, we had reports that the site was throwing a “Error establishing database connection” issue. For those like me who are not techy, this is a bad error to get from your website. This can mean that something has majorly gone wrong and everything is on fire.
We dispatched urgent messages to our wonderful dev team, who began checking everything and found the issue fairly quickly. Now for a small bit of background info. The volume backing all of the storage for INN is a RAID1 array. RAID1 is a data storage system where the data is stored on 2 or more drives, duplicating the data and creating a redundancy. Usually this means that if there is a fault, the site would stay up.
What happened on Sunday was that one of the discs failed. This meant that there were severe performance issues as the RAID array tried to sync data to the bad disc. Our web provider investigated and confirmed that this was the issue. We were initially told that repairing this would take around a day. Unfortunately, it turned out that the failed disc would need to be replaced completely.
As this would have taken longer than we were prepared to wait (by now it was Thursday, and there was no immediate fix in sight), the decision was made to migrate the site to Platform.sh which utilises a different type of infrastructure.
Some readers were still able to access the site, this was thanks to to a supercache plugin which generates and delivers static HTML files for all our articles so that when you visit the site it loads faster. These files were not as impacted during the failure so if you were not logged into the site chances are you got to see either an HTML page or the normal article pages.
Onwards and upwards
It has been a frustrating week for everyone on staff, from myself, through the editors & copy editors and onto the writers as we have had articles and news that we have wanted to bring you and have been unable to.
We have been preparing a number of articles while we have been offline and are looking forward to bringing them to you in the next week. Handily, TEST and FRT decided to have a fight on the day the site became operational again, so we will be bringing you a write-up of that.
To give you a taster of what is coming up in the next week (outside of explosions that happen in New Eden of course!) we have a number of interviews with CCP staff, from CCP Bartender talking nerdy to us, to CCP Delegate Zero giving a peek behind the curtain of creating the lore and stories that play out in New Eden. The PLEX prices are still too damn high, and one of our writers will be weighing in on that topic. I also heard a rumour that some Goonswarm titans exploded terribly this evening, so we will be once again explaining why goons are terrible at spaceship piloting.
Thank you for your patience while we have been away, and we look forward to bringing you the news, and opinions that you have come to expect from us once again!
