creating 155000 entities off of a openJPA query

Question

I have a query which in the worst case will create upwards of 150K entities. Probably with an upward limit of 300,000k entities. I have tried several ways to return this set of data back to the user... I run the query just using sql Developer and it runs in about .348 seconds. So that is not a bottle neck of any kind.

I create a nested query like so...

List<Object[]> ObjList = (List<Object[]>) emf.createNativeQuery(assembleNestedQuery(query2)).getResultList();

The native query is assembled as such...

String query2 = assembleQuery(organizationIDs, 2);

else if (type == 4){ 
    queryBuilder.append("SELECT t0.RESOURCE_ID, t0.FIRST_NAME,  t0.MIDDLE_NAME, t0.LAST_NAME FROM EPCD13.Provider t0");
    if(typeArgs.length > 0){ 
        queryBuilder.append("  WHERE t0.RESOURCE_ID IN (");             
        for(int i = 0 ; i <= typeArgs.length - 1; i++){
        if(i != typeArgs.length -1)
            queryBuilder.append(typeArgs[i] +", ");
        else
            queryBuilder.append(typeArgs[i] +" ");
        if((i % 1000 == 0) && (i != 0)){
            queryBuilder.append(") OR IN (");
        }
    }
    queryBuilder.append(")");               
            }
    }

private String assembleNestedQuery(String typeArgs2){
        StringBuilder queryBuilder = new StringBuilder();
        queryBuilder.append("SELECT t0.RESOURCE_ID, t0.FIRST_NAME,  t0.MIDDLE_NAME, t0.LAST_NAME FROM EPCD13.Provider t0");
        queryBuilder.append("  WHERE t0.RESOURCE_ID IN (");
        queryBuilder.append(typeArgs2);
        queryBuilder.append(")");
        return queryBuilder.toString();
    }

That above code basically assembles this query...

SELECT t0.RESOURCE_ID, t0.FIRST_NAME,  t0.MIDDLE_NAME, t0.LAST_NAME FROM EPCD13.Provider t0  WHERE t0.RESOURCE_ID IN (SELECT DISTINCT d.RESOURCE_ID FROM EPCD13.RESOURCES d WHERE d.ORGANIZATION_ID in (...))

So the nested query is basically created dynamically... The query that I run has about 155K records returned. Now I basically run the below code to turn the results into Provider objects...

List<Provider> provList = new ArrayList<Provider>(); 
for(Object[] obj: ObjList)
{
    provList.add(this.GetProviderFromObj(obj));
}

And this is where the exception are occuring. Is this too large a set of data for JPA to handle? I could do this with some simple Java JDBC connections and some basic Java objects pretty simply. But is this set of data to large to work with?

Thanks.

UPDATE: Exception I am getting

java.lang.OutOfMemoryError: caused by: java.lang.OutOfMemoryError

Answer 1

It sounds like are you trying to use a data set that is too large for your JVM. You could either pull less number of objects into memory, or you could increase your max heap.

I'd also suggest looking at Large Result Sets.