Hibernate out of memory exception while processing large collection of elements

StackOverflow https://stackoverflow.com/questions/20869473

Domanda

I am trying to process collection of heavy weight elements (images). Size of collection varies between 8000 - 50000 entries. But for some reason after processing 1800-1900 entries my program falls with java.lang.OutOfMemoryError: Java heap space.

In my understanding each time when I call session.getTransaction().commit() program should free heap memory, but looks like it never happens. What do I do wrong? Here is the code:

private static void loadImages( LoadStrategy loadStrategy ) throws IOException {
    log.info( "Loading images for: " + loadStrategy.getPageType() );

    Session session = sessionFactory.openSession();
    session.setFlushMode( FlushMode.COMMIT );
    Query query = session.createQuery( "from PageRaw where pageType = :pageType and pageStatus = :pageStatus and sessionId = 1" );
    query.setString( "pageStatus", PageStatus.SUCCESS.name() );
    query.setString( "pageType", loadStrategy.getPageType().name() );
    query.setMaxResults( 50 );

    List<PageRaw> pages;
    int resultNum = 0;

    do {

        session.getTransaction().begin();

        log.info( "Get pages statring form " + resultNum + " position" );
        query.setFirstResult( resultNum );
        resultNum += 50;
        pages = query.list();
        log.info( "Found " + pages.size() + " pages" );


        for (PageRaw pr : pages ) {
            Set<String> imageUrls = new HashSet<>();
            for ( UrlLocator imageUrlLocator : loadStrategy.getImageUrlLocators() ) {
                imageUrls.addAll(
                        imageUrlLocator.locateUrls( StringConvector.toString( pr.getSourceHtml() ) )
                );
            }

            removeDeletedImageRaws( pr.getImages(), imageUrls );
            loadNewImageRaws( pr.getImages(), imageUrls );
        }

        session.getTransaction().commit();

    } while ( pages.size() > 0 );

    session.close();
}
È stato utile?

Soluzione

You have confused flushing with clearing:

  • flushing a session executes all pending statements against the database (it synchronizes the in-memory state with the database state);

  • clearing a session purges the session (1st-level) cache, thus freeing memory.

So you need to both flush and clear a session in order to recover the occupied memory.

In addition to that, you must disable the 2nd-level cache. Otherwise all (or most of) the objects will remain reachable even after clearing the session.

Altri suggerimenti

I don't know why you think committing a transaction frees heap memory. Running garbage collection does that.

OOM error can happen if your perm gen is exhausted.

The easy answer is to change your min and max heap sizes and perm gen size when you start the JVM and see if it goes away.

I'd recommending getting a profiler, like VisualVM, and seeing what is consuming your memory at runtime. It should be easy to fix.

I'd guess that you're trying to commit too large a chunk at once. Break it up into smaller pieces and see if that helps.

Try using session.clear() which "Completely clear the session. Evict all loaded instances and cancel all pending saves, updates and deletions. Do not close open iterators or instances of ScrollableResults"

This article solved my issue

    Session session = sessionFactory.getCurrentSession();
      ScrollableResults scrollableResults = session.createQuery("from DemoEntity").scroll(ScrollMode.FORWARD_ONLY);
      int count = 0;
      while (scrollableResults.next()) {
       if (++count > 0 && count % 100 == 0) {
        System.out.println("Fetched " + count + " entities");
       }
       DemoEntity demoEntity = (DemoEntity) scrollableResults.get()[0];
       //Process and write result
       session.evict(demoEntity);//important to add this
      }
     }

bulk fetching hibernate

  1. Use hibernate ScrollableResult
  2. Use evict

BTW I tried the statless solution it gave me this exception i did not how to solve ( may be you can improve this answer) Exception details is here

org.hibernate.SessionException: collections cannot be fetched by a stateless session

So i tuned with sleep(delay) as its a back ground process and very long with low resources on server i have to cool down cpu; with midnight work ( none rush hours).

This was probably a very obscure version of the same bug, but in my case the culprit was Eclipse. Switching to IntelliJ and running the same project solved the problem - and of course it worked just fine when running the project without an IDE. I couldn't quite figure out what was wrong with Eclipse, so stuck with IntelliJ. Full stack trace for future Google searches:

INFO: HHH000400: Using dialect: org.hibernate.dialect.SQLServerDialect
июн 12, 2022 11:37:30 AM org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PoolState stop
INFO: HHH10001008: Cleaning up connection pool [jdbc:sqlserver://localhost;encrypt=true;trustServerCertificate=true]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.hibernate.loader.PropertyPath.<init>(PropertyPath.java:45)
    at org.hibernate.loader.PropertyPath.append(PropertyPath.java:58)
    at org.hibernate.loader.plan.build.internal.returns.CollectionFetchableElementEntityGraph.<init>(CollectionFetchableElementEntityGraph.java:26)
    at org.hibernate.loader.plan.build.internal.returns.AbstractCollectionReference.buildElementGraph(AbstractCollectionReference.java:135)
    at org.hibernate.loader.plan.build.internal.returns.AbstractCollectionReference.<init>(AbstractCollectionReference.java:69)
    at org.hibernate.loader.plan.build.internal.returns.CollectionAttributeFetchImpl.<init>(CollectionAttributeFetchImpl.java:32)
    at org.hibernate.loader.plan.build.internal.returns.AbstractExpandingFetchSource.buildCollectionAttributeFetch(AbstractExpandingFetchSource.java:176)
    at org.hibernate.loader.plan.build.internal.AbstractLoadPlanBuildingAssociationVisitationStrategy.handleAssociationAttribute(AbstractLoadPlanBuildingAssociationVisitationStrategy.java:874)
    at org.hibernate.loader.plan.build.internal.FetchStyleLoadPlanBuildingAssociationVisitationStrategy.handleAssociationAttribute(FetchStyleLoadPlanBuildingAssociationVisitationStrategy.java:155)
    at org.hibernate.loader.plan.build.internal.AbstractLoadPlanBuildingAssociationVisitationStrategy.startingAttribute(AbstractLoadPlanBuildingAssociationVisitationStrategy.java:564)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributeDefinition(MetamodelGraphWalker.java:170)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributes(MetamodelGraphWalker.java:140)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitEntityDefinition(MetamodelGraphWalker.java:97)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAssociation(MetamodelGraphWalker.java:209)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributeDefinition(MetamodelGraphWalker.java:178)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributes(MetamodelGraphWalker.java:140)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitEntityDefinition(MetamodelGraphWalker.java:97)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAssociation(MetamodelGraphWalker.java:209)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributeDefinition(MetamodelGraphWalker.java:178)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributes(MetamodelGraphWalker.java:140)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitEntityDefinition(MetamodelGraphWalker.java:97)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAssociation(MetamodelGraphWalker.java:209)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributeDefinition(MetamodelGraphWalker.java:178)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributes(MetamodelGraphWalker.java:140)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitEntityDefinition(MetamodelGraphWalker.java:97)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAssociation(MetamodelGraphWalker.java:209)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributeDefinition(MetamodelGraphWalker.java:178)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributes(MetamodelGraphWalker.java:140)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitEntityDefinition(MetamodelGraphWalker.java:97)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAssociation(MetamodelGraphWalker.java:209)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributeDefinition(MetamodelGraphWalker.java:178)
    at org.hibernate.persister.walking.spi.MetamodelGraphWalker.visitAttributes(MetamodelGraphWalker.java:140)
Autorizzato sotto: CC-BY-SA insieme a attribuzione
Non affiliato a StackOverflow
scroll top