optimize lucene 4.6.1 for real time search

Question

I know there are some similar questions, but as I searched through none of them is suitable to real-time searching.

my case is that I have over a few millions of small text files, for indexing I was able to achieve 3 mins per million files which is ok. The problem is for searching.

In order to get real time response, my search has to be as fast as possbile. currently, it returns result abount 8 - 10 seconds somehow. The query itself is kind of large though, but that should not be the major reason.

when indexing, i have used following configurations

public static LogMergePolicy optimizeIndex() {
    LogMergePolicy mergePolicy = new LogByteSizeMergePolicy(); 
    mergePolicy.setMergeFactor(2); 
    mergePolicy.setMaxMergeDocs(50000);

    return mergePolicy;
}

and

IndexWriterConfig config =new IndexWriterConfig(Version.LUCENE_46, analyzer);
        config.setMergePolicy(optimizeIndex());
        config.setUseCompoundFile(false); 
        config.setRAMBufferSizeMB(16);
        config.setMaxBufferedDocs(50000);

So any thoughts on how should I do to get real time response from search?