June 3rd -- Site interruption

After rolling Solr (tag pages only) on our scheduled rollout window Wednesday we found that, as we feared, the competition for memory on the web servers was just too high to cope. Basically there are four things on each webserver box:

• Solr (consumes about 6G of memory to keep the whole index in memory)
• Java Bridge -- Still being used for everything but tags pages which are now in Solr.
• Apache processes -- About 50M a piece when the system gets slow on disk etc the number of these grow. I still feel like we have MaxClients etc. set wayyyy too high. I'll likely scale this number down.
• Disk Buffers -- Whenever linux has free memory it starts stuffing any disk accesses into these buffers/cache. If the data gets requested again then it gets loaded from the buffers.

The issues we had this morning were kind of a viscious chain of events where each of these systems would compete for memory...

• Solr -- Solr sucks up 6G of memory and this steals away from disk buffers. Disk buffers need to be build for Solr as well because it now has a local index that needs to get put into memory.
• Apache -- Apache processes start to slow down because they are waiting for disk that just got eaten for Solr and the number of processes starts to grow and further eats away disk buffers.
• JavaBridge -- Java bridge is trying to read from a different index than Solr and it's index is competing with the Solr index and further competes for precious disk buffer space.
• Disk Buffers -- Not enough of them to go around. They get completely eaten away by the fact that the Apache processes are growing.

Now both of the NFS servers that have the index for Lucene and code for the site are 100% utilized because disk buffers just got eaten away completely. Not good. The site struggles to stay above water.

Several downtime events occur from 8AM to 10:30AM that account for a combined total of about 30 minutes. The site was mostly slow during this time period. An alert was sent out to the "alerts" email at about 9:00AM when we saw the first real downtime that wasn't slowness.

The Fix

To fix the issue we basically took one web server out of the pool and turned it into a single Solr server. The web servers were then made to point to this one Solr server and their respective internal Solr servers were shutdown. This had the following effects:

• Only one server in the system has Solr which makes it an SPOF but not a huge one because we can bring up Solr quickly on any other system.
• The one server with Solr just serves up the tag results which allows it to just disk buffer the one single Solr index instead of needing to worry about the Lucene index that is mounted on a different system.
• The web servers in the system can now freely use their disk buffer to buffer up the Lucene mounted index and they use the Solr instance to get their tag results. Not much competition for the disk buffer anymore.

I'm happy to report that the system recovered nicely and performance is also up and should even get better as the disk buffers all recover. The one Solr instance is easily handling all the requests from the rest of the system and is at a CPU usage of about 5% with <>

Typically, the responses from the Solr server searching over the index appear to be <>

Moving Forward

As we continue to roll out more Solr pages we'll deploy them to hit this single Solr instance. When all of the pages have been converted to use Solr we won't need the Lucene disk buffers anymore and can safely use the solr servers on each webserver and allow their disk buffers to fill completely with the Solr index.

Alternatively, we could also keep multiple Solr servers separate and further cannibalize the web server pool replacing them with Solr. We'll have to see how the load introduces itself as we convert each page.