Splunk uses its own search engine, it's not based on any 3rd party.
Its search engine is based on files only, no database behind it. It does not store fields, but raw data only. The fields are extracted during search time, and due to that are very dynamic. Its also very fast in finding keywords in the data (needle in haystack).
- Breaking the data into time-based events, attaching time for each raw event.
- Marking every word found in the events and their location across the index
- Storing the events in compressed format (tar.gz)
To be more detailed, Splunk is storing data in the following way:
- Very fast search for keywords inside the events
- Look in the original raw data
- Create new fields on the raw data and use them with statistics commands.
Source: http://www.splunk.com/web_assets/pdfs/secure/Splunk_for_BigData.pdf http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Howindexingworks
+3 Years experience Splunk architect.