Searching through elmah error log files(Perhaps in 1000's)

Question

We have a folder of elmah error logs in XML format. These files will be in millions and each file might be upto 50 kb in size. We need to be able to search on the files(eg: What errors occured, what system failed etc). Do we have a open source system that will index the files and perhaps help us search through the files using keywords? I have looked at Lucene.net but it seems that I will have the code the application. Please advise.

Answer 1

If you need to have the logs in a folder in XML, elmah-loganalyzer might be of use.

You can also use Microsoft's Log Parser to perform "sql like" queries over the xml files:

LogParser -i:XML "SELECT * FROM *.xml WHERE detail like '%something%'"

EDIT:

You could use a combination of nutch+SOLR or logstash+Elastic Search as an indexing solution.

• http://wiki.apache.org/nutch/NutchTutorial
• http://lucene.apache.org/solr/tutorial.html

• http://blog.building-blocks.com/building-a-search-engine-with-nutch-and-solr-in-10-minutes

• http://www.logstash.net/

• http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/
• http://www.javacodegeeks.com/2013/02/your-logs-are-your-data-logstash-elasticsearch.html

Answer 2

We are a couple of developers doing the website http://elmah.io. elmah.io index all your errors (in ElasticSearch) and makes it possible to do funky searches, group errors, hide errors, time filter errors and more. We are currently in beta, but you will get a link to the beta site if you sign up at http://elmah.io.

Unfortunately elmah.io doesn't import your existing error logs. We will open source an implementation of the ELMAH ErrorLog type, which index your errors in your own ElasticSearch (watch https://github.com/elmahio for the project). Again this error logger will not index your existing error logs, but you could implement a parser which runs through your XML files and index everything using our open source error logger. Also you could import the errors directly to elmah.io through our API, if you don't want to implement a new UI on top of ElasticSearch.