Site performance issues and outages

As of around 8AM PST time we experienced an outage that we are still investigating. Nagios did not alert before we received a downtime from CBS2. The reason nagios failed to report was because yesterday the development server's disk was 100% utilized and even though we cleared the disk space the nagios daemon got hung somehow. We have since cleared the disk space and restarted the nagios daemon.

Unfortunately, we had little warning about the issue. An email was sent to alerts@mojopages around 20 minutes into the issue despite repeated attempts to restart the services (http processes had grown to over 800 processes). We saw rapid growth in the number of http processes after every restart indicating there was some kind of mysterious bottleneck.

Resolution -- We did the following to resolve the issue:

* We reverted the search page changes as these appeared to be getting hung. The solr changes moved search to live. This did not start an immediate recovery. Furthermore, Solr seemed to be executing normally and most threads were idle. Not the bottleneck.
* We increased the max connections on memcached servers. This was necessary because as the http processes climbed the memcached would run out of connections and this would make things even worse.
* Checked the tracking db, master db, slave db -- No locking and only one or two queries executing. Not the bottleneck.
* No %iowait on any of the servers so disk space did not appear to be the issue. This is a good sign in the sense that we are finally releiving some pressure from the NFS mount hosting the index now with Solr.
* Checked the error log to see if there were timeouts to superpages. Not the bottleneck.

After changing the memcached and while still investigating the cause the number of HTTP processes gradually fell and the site returned to normal. I am still unsure at this time what the cause of the issue was. One remaining possibility is extra requests/second or something strange with the coding that really causes issues.

We are looking at both of these possibilities now and creating reports of requests per minute to compare yesterday with today (need to get these into cacti). I will also look at the code to see if I can identify an issue. We may attempt to re-roll Solr tonight and we'll monitor closely for signs of issues (with Nagios enabled this will help).