Loading millions of DB row in wicket table through a list causing outofmemory error

Question

I'm loading million lines of data by sortabledataprovider .. the query returns a list(Arraylist) as I sent it to a Wicket ajax enabled table and pagination enable table. So the problem is that - If there are concurrent queries - the application might get crashed. I'm already getting Java heap space error with just 100,000 rows in DB. So what I want to achieve is this - when a user click on next page or may be number 10th page - it will load only the number 10th pages data from the DB - not the whole query which might have million lines in it.

here is how I loading the list by query

      final SyslogProvider<SyslogParsed> sysLogProviderAjax = new SyslogProvider<SyslogParsed>(new ArrayList<SyslogParsed>());

     *
     *
           daoList = syslogParsedDao.findByDatesAndHostName(utcStartDate, utcEndDate, null);
        sysLogProviderAjax.setList(daoList);

**

DB query returning the big list of all rows

           public List<logParsed> findByDatesAndHostName() {
    return getJpaTemplate().execute(new JpaCallback<List<SyslogParsed>>() {

            return query.getResultList();
        }
    });
}

========= my data provider

 public class logProvider<logParsed> extends SortableDataProvider{

/**
 * 
 */
private static final long serialVersionUID = 1L;
@SpringBean(name="logParsedDao")
private logParsedDao logParsedDao;

class SortableDataProviderComparator implements Comparator<logParsed>, Serializable {

    public int compare(logParsed log1, logParsed log2) {
        PropertyModel<Comparable> model1 = new PropertyModel<Comparable>(log1, getSort().getProperty());
        PropertyModel<Comparable> model2 = new PropertyModel<Comparable>(log1, getSort().getProperty());

        int result = model1.getObject().compareTo(model2.getObject());

        if (!getSort().isAscending()) {
            result = -result;
        }

        return result;
    }

}


private List<logParsed> list =  new ArrayList<logParsed>();
private SortableDataProviderComparator comparator = new SortableDataProviderComparator();

public logProvider(List<logParsed> sentModel){

    setSort("numberOfEntries",SortOrder.DESCENDING);
    list = sentModel;
}

public Iterator<logParsed> iterator(int first, int count) {

    //ArrayList<logParsed> newList = (ArrayList<logParsed>) logParsedDao.findAll();
    //Collections.sort(newList, comparator);
    Iterator<logParsed> iterator = null;

    try {
        if(getSort() != null) {
            Collections.sort(list, new Comparator<logParsed>() {
                private static final long serialVersionUID = 1L;

                    public int compare(logParsed sl1, logParsed sl2) {
                     int result=1;
                        PropertyModel<Comparable> model1= new PropertyModel<Comparable>(sl1, getSort().getProperty());
                        PropertyModel<Comparable> model2= new PropertyModel<Comparable>(sl2, getSort().getProperty());

                        if(model1.getObject() == null && model2.getObject() == null) 
                            result = 0;
                        else if(model1.getObject() == null) 
                            result = 1;
                        else if(model2.getObject() == null) 
                            result = -1;
                        else 
                            result = model1.getObject().compareTo(model2.getObject());

                        result = getSort().isAscending() ? result : -result;

                        return result;
                }
            });
        }

        if (list.size() > (first+count))
            iterator = list.subList(first, first+count).iterator();
        else
            iterator = list.iterator();
    } 
    catch (Exception e) {
        e.printStackTrace();
    }

    return iterator;

}

public int size() {
    // TODO Auto-generated method stub
    return list.size();
}

public IModel<logParsed> model(final Object object) {
    return new AbstractReadOnlyModel<logParsed>() {
        @Override
        public logParsed getObject() {
            return (logParsed) object;
        }
    };
}

 public void setList(List<logParsed> newList){

     list = newList;
    }

}

Answer 1

The problem with query.list() is it returns all rows at once.
Instead of query.list() you can use either:

• query.scroll(), which returns the query results as ScrollableResults
• query.iterate(), which returns the query results as an Iterator

Both of these options return one row at a time, which is what you need.

Note that the query remains "executing" for the duration of processing, so you may find that the tables are locked etc depending on the isolation level you've chosen.

Answer 2

You have to use a JPA query that sort and returns only the desired rows each time that iterator(int first, int count) of your IDataProvider is called.

Something similar to this:

public Iterator<logParsed> iterator(int first, int count) {

    Query query = entityManager.createQuery("from LogParsed m ORDER BY m.numberOfEntries DESC", LogParsed.class);
    List<LogParsed> output = query.setFirstResult(first).setMaxResults(count).getResultList(); 

    return output.iterator();
}

Have a look at this question.