Good News

Well it's been a long week of rolling out our Solr improvements in an effort to give the site more stability and perform faster. The good news is that we're starting to see the fruits of some of these labors although we still have a ways to go to get much better uptime and performance.

I have attached a graphic which is from our Gomez monitoring which is plotting the availability of the site in addition to the site response times. As one can see there were definitely some issues with site performance and availability. One will also note that the 7th saw 100% uptime (this is actually somewhat false because that was the first day we started running Gomez). One also notes that it appears that we had 0% availability for the partner tag today but that's just because of the way the tests get run (it's actually 100% today).

While we worked on Solr we had issues with database locking (we still have some but have been optimizing and have another database replicant on the way to relieve more load). We also identified a very interesting issue with the old Java bridge which caused our HTTP processes to hang onto requests and never return (this was a HUGE one). However, there is still a lot of work left to do.

In spite of having much better availability and performance as reported by Gomez and other monitoring we are still facing some issues with the GWT reporting "No Response" errors. We continue to work on these issues.