Solr - How can I receive notifications of failed imports from my DataImportHandler?

Question

Our solr indexes are refreshed according to a schedule, as well as arbitrarily as needed by means of a DataImportHandler full import. We've had several occasions where the import fails for various reasons.

How can I receive a notification (preferably email) that an error has occured while performing an import with a DataImportHandler?

Answer 1

After attempting to use an EventListener with some success we reviewed our options and ultimately settled on a less-than-foolproof approach.

Instead of extending Solr we're using our existing monitoring infrastructure (primarily with Nagios) to poll the status of the dataimporthandler. It runs every minute and alerts us via our normal channels if it detects a failed import. This is good, but it still fails to detect if jobs aren't running at all--and has missed a few other corner cases.

Overall though this works well enough for now. It uses our existing infrastructure and keeps our efforts around Solr limited to its core functionality. I'll definately be revisiting this when we finally make the upgrade to Solr 4.0.

Answer 2

There is no easy config solution. But an alternative exists you might have to do little work.

You could register EventListener with DIH in data-config to listen for events EventListener.

Refer Wiki

<dataConfig>
  <document onImportStart ="com.foo.StartEventListener" onImportEnd="com.foo.EndEventListener">
   ....
   </document>
</dataConfig>

Your EventListener gives you access to Context Object, which provides access to most of DataImportHandler Objects & Event Statistics.

For instance, onImportEnd event your com.foo.EndEventListener could you use Context object handle to get Staistics such # of DocsSkipped , # of DocsFailed ... Context is a valuable object that exposes lot of DIH internals. It is up to your event listener to do what it needs to do with this information.

Perhaps a caveat, the DIH notification is mostly after the fact, you won't notified of events as it happens, you have to wait for the import process to complete for DIH to notify your listener or may be there is a workaround.

• EventListener
• Context