OpenJPA Eager Fetching

Question

I use OpenJPA 2.3 bundled with WebSphere 8.5 and I have to read a lot of data from a table. I also have to fetch a lot of relations with the root entity.

Atm I am using the criteria API to create the search query and select the entities. I annotated all collections with EAGER. When I check the logfile it creates 5 Queries to fetch all children. That is the way I want it. The catch is that I have to filter a lot in java after the select and stop after 1000 matching entities. So I thought i specify the fetch size and stop reading entities from the db as soon I have my 1k results.

If I introduce the FetchBatchSize setting, OpenJPA creates single queries for each entity to load the children. (n+1 problem)

I also tried to use the fetch join syntax directly in my query, but without any success. So what am I doing wrong?

I tried:

1)

    query.setHint("openjpa.FetchPlan.FetchBatchSize", 1000);
    query.setHint("openjpa.FetchPlan.ResultSetType", "SCROLL_INSENSITIVE"); 

2)

        OpenJPAQuery<?> kq = OpenJPAPersistence.cast(query);
        JDBCFetchPlan fetch = (JDBCFetchPlan) kq.getFetchPlan();
        fetch.setFetchBatchSize(1000);
        fetch.setResultSetType(ResultSetType.FORWARD_ONLY);
        fetch.setFetchDirection(FetchDirection.FORWARD);
        fetch.setLRSSizeAlgorithm(LRSSizeAlgorithm.UNKNOWN);

The entity:

@Entity
@Table(name = "CONTRACT")
public class Contract {

// omitted the other properties. The other relationships are annotated the same way
    @OneToMany(fetch = FetchType.EAGER, cascade = CascadeType.ALL, mappedBy = "contract")
    private List<Vehicle> vehicles= new ArrayList<Vehicle>();

The query:

        CriteriaBuilder cb = em.getCriteriaBuilder();
        CriteriaQuery<Contract> crit = cb.createQuery(Contract.class);
        crit.distinct(true);
        Root<Contract> r = crit.from(Contract.class);

        // omited the where clause. In worst case I have a full table scan without any where clause. (the reason I need the batch size)

        Fetch<Contract, Vehicle> fetchVehicles = r.fetch("vehicles", JoinType.LEFT); // I tried to work with a fetch join as well

                TypedQuery<Contract> query = em.createQuery(crit);

//      query.setHint("openjpa.FetchPlan.FetchBatchSize", FETCH_SIZE);
//      query.setHint("openjpa.FetchPlan.ResultSetType", "SCROLL_INSENSITIVE"); 

        OpenJPAQuery<?> kq = OpenJPAPersistence.cast(query);
        JDBCFetchPlan fetch = (JDBCFetchPlan) kq.getFetchPlan();
        fetch.setFetchBatchSize(FETCH_SIZE);
        fetch.setResultSetType(ResultSetType.FORWARD_ONLY);
        fetch.setFetchDirection(FetchDirection.FORWARD);
        fetch.setLRSSizeAlgorithm(LRSSizeAlgorithm.UNKNOWN);
        fetch.setEagerFetchMode(FetchMode.PARALLEL);

        List<TPV> queryResult = query.getResultList();

        // here begins the filtering and I stop as soon I have 1000 results

Thanks for the help!

Answer 1

It seems that there are some Bugs filed which apply in my scenario. I found a workaround which scales well.

First I select only the ids (Criteria API can select skalar values) and I apply the batching there. So I have no n+1 problem due to the wrong fetching strategy anymore.

After this I select my entities with an IN() statement in batches of 1000 without limiting with fetch batch size or max results. So I do not run into this bug and OpenJPA generates one query for each relation.

So I have around 6 querys for the entity with all its dependencies.

Thanks again thobens for your help!

Answer 2

Have a look at how to deal with large result sets and you will see that EAGER is the opposite of what you should do.

As I stated in comments, EAGER means that JPA loads all results at once, so it is not recommended for large result sets. Setting the fetchBatchSize causes JPA to lazy load every x (in your case 1000) results. So it would be practically the same as if you would use @OneToMany(fetch = FetchType.LAZY, ...) (also worth a try)

Setting the fetchBatch size to a much lower number (e.g. 50) will also lower the objects that are kept in memory.

Also try

query.setHint("openjpa.FetchPlan.ResultSetType", "SCROLL_SENSITIVE");