Using Hibernate 4.1 & JPA 2.0 with an Entity Manager, what is the optimal way to load multiple entities with sub-entities down to multiple levels?

Question

We are using Hibernate 4.1.4 with an entity manager implementation and are very pleased with it. When loading a single entity or small sets of entities (10-50 + sub-entities) we get excellent performance and all our joins work great and are bi-directional. Editing/saving/deleting is a breeze!

Loading 2500 Entities & Related Data for Reporting

We are running into problems when we try to load hundreds or thousands of entities along with sub-entities. In one example we have about 2500 base entities along with two one-to-one and three one-to-many joins. This results in 2500 x 5 +1 queries. Takes about 30 seconds to execute with our database (DB2).

We use a dynamic query and the entity manager create query method:

http://docs.jboss.org/hibernate/orm/4.1/javadocs/

http://docs.oracle.com/javaee/6/api/javax/persistence/EntityManager.html

Attempting to coax Hibernate to use Joins

Our observations have been that no matter what we try, we can't seem to get hibernate to execute loads of sub-entities as joins on the primary query. When loading a single entity using the find method and by primary key hibernate uses joins, but when multiple base entities are loaded it does not.

We have tried criteria api with the same result.

Some Success Retrieving Multiple Entities in Query

Example:

select a,b from table1 a left join table2 b

We have had partial success doing the joins manually in our query and retrieving the individual entities via tuples but this bypasses our annotated entity model and is not the preferred method (if we can avoid it).

http://docs.jboss.org/hibernate/orm/4.1/manual/en-US/html_single/#objectstate-querying-executing

The problem we have encountered with tuples is they are detached from the primary entity (which seems logical as they are not loaded via annotated relationships) and there seems to be no way to re-attach them. After manually adding all the one-to-many records in the list of an entity, when hibernate hits the get method for the list, it still tries to eager/lazy load the list (despite the list already being set).

Attempted Optimization - Preloading of Data Sets

Another optimization technique we attempted was to pre-load sets of data in advance. After loading all the core entities we built a list of the pk's and fired a single query off to load all the one-to-many sets of data. But again when hibernate hits the getter for the list data it again tries to eager/lazy load the list ignoring what is already pre-loaded in the session (need to enable caching perhaps?).

Suggestions?

We are going in a number of different directions with little success and are hoping someone might be able to point us down the right path (or point out a new one!)!

Answer 1

Seems I have figured it out myself! Two strategies fixed the issue:

1. Join fetch in the query takes care of a one-to-many join! Of course you can only do one join fetch as Hibernate will throw the cannot simultaneously fetch multiple bags exception.

   Example:

   LEFT OUTER JOIN FETCH t.table3 t3
   

2. For additional joins @Fetch(FetchMode.SUBSELECT) did the trick!

Effectively reduced the total queries from thousands to just a few. In one example, 188 core entities were selected and a total of 12 queries were executed. The extra queries are mostly one-to-one look-ups and have no significant effect on the total execution time (on average 1-2 seconds). Whether 188 or 1888 records are selected, the total number of queries does not increase significantly.

If anyone would like more information, feel free to contact me or leave a comment! :)