I was getting this as well. My background was that I installed Solr locally and ignored everything in the solr subdirectories. So the contents of solr/pids where ignored but the file structure of solr/pids remained.
I deployed this to our dev server and encountered the above errors.
The solution turned out to be simple. The solr directory needs to be created fresh in a new environment.
- rake sunspot:solr:stop
- Delete your /solr directory.
- rake sunspot:solr:start
- rake sunspot:solr:reindex
courtesy of Can't reindex Sunspot SOLR - Error - RSolr::Error::Http - 500 Internal Server Error