Memory Leak in WebObjects

Question

I've been battling an issue I've been having using Apple's EOF framework for quite some time. It seems that, occasionally, when an EOEnterpriseObject is created, or pulled into an editing context from the DB, EOF won't let go of the memory that object consumes, even after the relevant enterprise object, editing context, and object store are disposed and deleted. Most objects, it seems, are handled just fine by EOF, but I have 2 objects where EOF consistently holds on to the memory used by the objects until the app is restarted. Both of these EOs are potentially very large (they contain an NSData object that is used to hold a file attachment).

Using JProfiler, I've found that a reference to the problem EOs is held by the EODatabase._snapshots array.

I was wondering if anyone else might have had a similar problem with EOF and/or project Wonder. Since I consistently see the problem under 2 different scenarios, I'm hoping it is somewhat common and, therefore, has a resolution.

I'm using the latest WebObjects library (5.4.3) and the latest Wonder libraries.

The below isn't my exact code, but it's the smallest-possible example that still has the memory leak:

public WOActionResults createEmailHistoryEntry() throws MessagingException, IOException {
    File emailFile = new File("Email_with_large_attachment.eml");
    javax.mail.Message message = EmailUtils.convertEmlToMessage( emailFile );

    EOObjectStore osc = new ERXObjectStoreCoordinator(true);
    EOEditingContext ec = ERXEC.newEditingContext(osc);
    ec.lock();
    try {
        EmailHistoryEntry historyEntry = (EmailHistoryEntry) EOUtilities.createAndInsertInstance( ec, EmailHistoryEntry.class.getSimpleName() );
        EmailDataObject emailData = (EmailDataObject) EOUtilities.createAndInsertInstance( ec, EmailDataObject.class.getSimpleName() );
        emailData.setEmailHistoryEntry( historyEntry );

        ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
        message.writeTo( byteStream );
        NSData rawEmail = new NSData( byteStream.toByteArray() );
        emailData.setRawEmail( rawEmail );

        ec.saveChanges();
    }
    finally {
        ec.unlock();
        ec.dispose();
        osc.dispose();
    }
    return null;
}

I don't know that I'm doing anything unusual there. If I run it multiple times, the memory consumption will grow by about 140MB each time and eventually will hit an OutOfMemory error.

2012-12-26 Edit

I've done some more investigating on this. It appears that the problem is in the Project Wonder library, not in the EOF library. I understand the "problem" may likely be me and/or my understanding, rather than the Wonder lib. :)

I've created a test app that duplicates the issue I've been seeing and posted it on github: https://github.com/t-evans/memory-leak-test.git.

The test app is mostly just the default app that Eclipse creates when you add a new Wonder application. The changes are the addition of one line in Application.java, most of the code in Main.java, and, of course, the model file. Currently, its configured to connect to a postgres database named "memleaktest".

My app's run configuration has just two VM args: "-Xmx5m -Xmx50m". If I start the app and click the "Create Object" link about 5 times, it will hit an OutOfMemory error. Monitoring the memory using jConsole shows that the memory consumption grows by about 5MB each time, and the app never lets go of those 5MB.

My findings, so far, point to ERXObjectStoreCoordinatorSynchronizer as the culprit. In the test app, Application.java turns on synchronization. The constructor of Main.java just performs a dummy query, which ultimately causes Main._osc to be passed to ERXObjectStoreCoordinatorSynchronizer.addObjectStore() (the synchronizer needs more than 1 OSC to synchronize anything). Main.createDataStore() creates an OSC and EC, adds a DataStore object to the DB, then nukes the OSC and EC.

AFTER the new object, OSC, and EC are nuked, disposed, and fall out of scope, the synchronizer runs and adds that newly-created (but, now, obsolete) object to that other OSC, which ultimately re-adds the new object to the EODatabase._snapshots array, where it remains until the other OSC is disposed.

It seems strange that the new EO is synchronized with the other OSC after it, and it's EC and OSC, are dead and gone and out of scope. Shouldn't the synchronizer also synchronize the fact that the EO is out of scope and remove it from all other OSCs (or not add it the other OSCs in the first place)?

I know synchronization can be turned off by calling

ERXObjectStoreCoordinatorSynchronizer.synchronizer().setDefaultSettings(
    new SynchronizerSettings(false, false, false, false));

which will avoid the issue, but the default settings for the synchronizer have everything turned on, which causes a pretty big leak.

Is this a bug, or am I doing something incorrectly? I'm confused why other people don't seem to be running into this. Or maybe they are running into it, but haven't noticed the memory leak because they are not using large EOs(?)

Answer 1

The best solution I've found is to either avoid ERXObjectStoreCoordinatorSynchronizer (which means you also need to avoid ERXObjectStoreCoordinatorPool, since that uses the synchronizer), or to disable the synchronizer as follows:

ERXObjectStoreCoordinatorSynchronizer.synchronizer().setDefaultSettings(new SynchronizerSettings(false, false, false, false));

Or, you could probably get away with just disabling the InsertSnapshotProcessor:

ERXObjectStoreCoordinatorSynchronizer.synchronizer().setDefaultSettings(new SynchronizerSettings(false, true, true, true));

as that's where the memory leak seems to occur (the others may cause problems too, but I haven't specifically seen that).

After posting to the Project Wonder mail group, no one seems to have a better solution than the above.

Answer 2

I hope you might have verified all parts of your code and profiled it. But still I feel the issue is in your code only.

Worth checking the following once again: references to EOs, EC, NSData object, Components and see if by chance anywhere your huge objects and more importantly EC is not let to GC-ed.

We may need some more info to help you with debugging this issue, if issue persists!

Answer 3

Sorry to post this as an answer, but I'm new to StackOverflow and don't have enough points to add a comment. Just wanted to add a reference to an issue in the Wonder Github repository that addresses this problem, in hopes it will help lead to a solution:

https://github.com/wocommunity/wonder/issues/130 -- from user 'nullterminated' (I believe this is Ramsey Gurley, who confirmed the issue -- see http://comments.gmane.org/gmane.comp.web.webobjects.wonder-disc/19078)

"It appears that ERXObjectStoreCoordinatorPool leaks EODatabase._DatabaseRecord objects whenever the pool size is > 1 and EOs are saved. After many hours in the debugger, I think I understand why..."

"Normally, when an EC saves changes, the ObjectsChangedInStore notification is fired, the snapshot is inserted with EODatabase's _fastHashInsert and then the EC refaults the EO when processing changes (updates) or frees the snapshot on finalize (inserts). These actions fire the corresponding _fastHashRemove to free the snapshot."

"The problem appears to be that ERXObjectStoreCoordinatorSynchronizer rebroadcasts the ObjectChangedInStore notification to the other OSCs in the pool. This results in snapshots inserted, but with no EC around to clean up, the snapshots are never removed."