I came home from a week long company hackathon to see my newrelic error reporting going crazy …
My free bugsnag account had already maxed out of the 2000 error allocation I had for the entire month (I’ll usually get 100 in a month … maybe).
I immediately knew from the exception in new relic that something was wrong with Sphinx, and I figured it was pretty bad because on each page load of news pages, I hit sphinx to give me a list of related news stories, which meant that the crawlers that hit my site every second of the day could be generating loads of exceptions.
However, when I went to pull up one of my news pages … it loaded just fine …
I hit the search page, and got a 500 error. Weird as that may be, I’d encountered that before. I immediately logged into my vps and ran
at the console and was ready to drop my mic and moonwalk back to my sofa to catch up on “The Americans”.
Then I looked at my new relic account and noticed that the same error was still coming in 10 minutes after it should have been fixed.
I tried hitting the urls specified in the errors and got the 500 error. I realized that some urls were exhibiting the problem while others weren’t.
I’d never seen that before :\
My google-fu quickly turned up this non-upvoted gem that helped me fix the issue. Basically you just have to rotate your sphinx index because that error means parts of it have gone “stale”.
the command I used to accomplish this was
`/usr/bin/indexer –rotate xx_core –config /path/to/your/sphinx/production.sphinx.conf`
your indexer command might be located somewhere different though; to locate mine I just used the `whereis indexer` command
You’ll want to change xx_core to match the name of your index (which you can find by looking in your sphinx.conf file)
Hope this helps you out!