Java wicket dataprovider for large tables size

Question

I'm developing a wicket front-end for an EJB3 java application with MySQL database. The website mostly is submitting forms and showing tables. For these tables I prefer to create IDataProviders which work very nice with AjaxPagingNavigation. But I have a few tables which are quite large and the size function of the IDataProvider becomes a bit tricky to implement. Usually I use a count for the query and this performs good in most cases but I have a few large table where a count takes too long (3s+). Now there are a few options I can use but I'm not really happy using any of these so that's why I ask, does anyone know an elegant solution for implementing the size() function for a large table?

The first option I found on Google was use a subquery with limit results an perform a count a that. But since I mainly use named queries and EJB3 it feels hackish and it can be hard to maintain if something changes.

The second option is use a ListDataView and just do a limited query on table and omit the need for a large count query. This solution is my favorite so far but is not the nicest since it (almost) always gets the maximum amount of records. I also have to chose between storing this List object or query the database again between requests.

The last option I found was spoofing the size option. I havn't implemented this but I could make the size() function of the IDataProvider return something like pagesize + 1. This bring a few other problems like having a pagesized number of records. There are checks to catch these but this is also messy.

Does someone know an elegant solution to use an IDataProvider in wicket for showing paged tables of large database tables?

Many thanks, Martin

Answer 1

I've implemented grid view with data provider which requires only iterator, without size information - its called IterableGridView

Here is the code: https://github.com/maciejmiklas/cyclop/tree/master/cyclop-wicket-components

Iterable Grid View is based on Wicket's GridView, however it does not work with IDataProvider but with IterableDataProvider. This new data provider relies only on plain java iterator - size information is not needed, and there is also no need to create range iterators for each page.

final List<String> myGridData = new ArrayList<>();
myGridData.add("value 1");
myGridData.add("value 2");

IterableDataProvider<String> iterableDataProvider = new IterableDataProvider<String>(10) {
    @Override
    protected Iterator<String> iterator() {
        return myGridData.iterator();
    }

    @Override
    public IModel<String> model(String s) {
        return Model.of(s);
    }

    @Override
    public void detach() {
    }
};

IterableGridView<String> myGrid = new IterableGridView<String>("myGrid", iterableDataProvider) {
    @Override
    protected void populateEmptyItem(Item<String> item) {
        item.add(new Label("myValue"));
    }

    @Override
    protected void populateItem(Item<String> item) {
        item.add(new Label("myValue", item.getModelObject()));
    }
};

add(myGrid);

myGrid.setItemsPerPage(10);

// you have to use custom pager and not AjaxPagingNavigator
IterablePagingNavigator pager = new IterablePagingNavigator("rowNamesListPager", rowNamesList);
resultTable.add(pager);

Answer 2

What about giving up on pagination completely and implement an infinite scrolling with quickview? The problem with the size will be avoided and the usage of the list will be more natural.

Answer 3

I use "original" family of objects IDataProvider, seems be good - with my own layer over JDBC, with two important features:

• auto-magically execute select [skip columns] count(*) ... where ... under wicket size() method

• execute prepared (prepared in human sense) MSSQL query, which server hasn't LIMIT/OFSET (in older versions) under iterator(), I use common trick with ROW_NUMBER()

The second isn't' important in your case, because MySQL has LIMIT/OFFSET

Code is ugly, specific to my integration targets etc, but working, so is closed-source ;) Querying is a kind of "criteria" with string substitutions, the same criteria is used twice.