Lucene.Net IndexWriter multiple application

Question

I am currently working on a project to index a lot of documents (with a slow process) before inserting it into Lucene.Net.

To make the indexation faster, I would like to duplicate the windows service to keep only one thread into my service(s). This means that I will have multiple IndexWriter and this is not possible because of the lock. Also the services wont be on the same server this means that Shared memory is not possible.

Is there any way to do this safely? if yes how to do it?

Thanks in advance for your answers.

Lenny

Answer 1

Every writer should be writing to their own directory, and you can merge them together as a final step using IndexWriter.AddIndexesNoOptimize

Merges all segments from an array of indexes into this index.

This may be used to parallelize batch indexing. A large document collection can be broken into sub-collections. Each sub-collection can be indexed in parallel, on a different thread, process or machine. The complete index can then be created by merging sub-collection indexes with this method.