Status updates May 31st to June 2nd

May 31

We had a site outage of about 20 minutes from 9:44AM PST to 10:06AM PST.  An alert was sent out to alerts@mojopages.com once I looked at the systems and determined it would take longer than 5 minutes to diagnose. This outage appears to have been related to some form of interaction between our development server and our master database as I can see a large traffic spike on both those servers at the time of the outage.  This is as the development server should not be interacting with our production systems. After speaking with GMI they reported that no such interaction should have occurred. 

The issue manifested itself by high server load with no iowait, low cpu usage, etc. This pointed to a database issue and it is the first that I have seen since May 1st. Looking in the mysql-query-log-slow there was an insert statement executing for 14 minutes at the time of the outage. Unfortunately, MySQL does not report on table locking in the MySQL query log so it is difficult to say without having looked at it while it was in process if a table lock was holding up the database.

Resolution:

If I see a similar issue in the future I will firewall off the development environment from the production servers to ensure that these systems cannot communicate directly.

June 2nd:

Over the past two days I have seen reports of an intermittent server issue where we would see long page loads and connection timeouts. It would happen in approximately 1 in 5 page loads. After speaking with Edgecast it appears that we didn't have Layer 7 HTTP checks on the servers so if an HTTP server started to go wayward it would not be removed from the pool. This was exactly the case on App8 where we have a new Solr index building.

Resolution:

I have asked Edgecast to put in place the appropriate Layer 7 checks. This should allow us to add/remove HTTP servers at will. There is no expected downtime from this change.