This is what I do: The search method has number of results as a parameter. I pass pageSize*page. So for page 1, I get only pageSize documents.
Then I only load the document (using searcher.doc()) for the page that I need.
TopDocs hits = searcher.search(lucene_query, pageSize*(page));
ScoreDoc[] scoreDocs = hits.scoreDocs;
int j = startIndex;
int rem = 0;
while (j < scoreDocs.length && (endIndex==0 || j<endIndex)) {
ScoreDoc sd = scoreDocs[j];
Document d = searcher.doc(sd.doc);
}