Why GC doesn't collect my objects?

Question

I have a java program which keeps calling java.util.zip to compress/decompress data. It runs out of memory within a few seconds. I had a memory dump with jmap and I'm viewing it with jhat.

Finalizer summary shows Total instances pending finalization: 0. If I understand it correctly, I don't have any objects which (1) have finalize() method, (2) have been marked by GC and (3) are waiting to be finalized. This seems good.

When I look at a particular object, the only reference to this object is a java.lang.ref.Finalizer. The Finalizer object is created for each object which has a finalize() method no matter if the object is GC'ed or not. So it looks like nothing is preventing this this Deflater object to be GC'ed.

Object at 0x7f4aeb7a35d0

instance of java.util.zip.Deflater@0x7f4aeb7a35d0 (51 bytes)

References to this object:

java.lang.ref.Finalizer@0x7f4aeb8607c8 (64 bytes) : field referent

The program is paused in the run by System.in.read(). The memory usage doesn't go down after a while.

UPDATE:

I should make it clear. The memory dump shows many objects were not GC'ed but no other objects (except Finalizer objects) were referencing them. I'm trying to find out why they were not GC'ed.

Answer 1

Are you positive you closed the stream and invoked end on the deflater? I'm sorry if this is a simplistic suggestion you've already tried, but there were a lot of complaints about a memory leak when using Deflater when not immediately invoking end on it, for example:

• http://www-01.ibm.com/support/docview.wss?uid=swg21227106
• http://bugs.sun.com/view_bug.do?bug_id=6734186 (1.5)
• excessive memory use by java
• http://bugs.sun.com/view_bug.do?bug_id=4797189
• http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6293787
• http://kohsuke.org/2011/11/04/quiz-answer-memory-leak-in-java/

The root cause is apparently that the collector is not able to keep up with the application when memory used by native elements is involved. That also explains the behavior you are seeing when profiling: the memory is ready to be reclaimed, but just isn't reclaimed quickly enough.

Since you wrote you don't use a Deflator directly but through Apache Thrift, try to identify which method in that library is responsible for ending the deflator, and made sure you invoked that method.

Answer 2

The base problem is that you aren't freeing objects from the C heap. Many classes in java.util.zip use Deflater. Deflater maintains a reference to data in the C heap. Your code is likely not calling close() on ZipOutputStream or DeflaterOutputStream, or it's not calling end() on Deflater.

(If you are passing your own Deflater to ZipOutputStream or DeflaterOutputStream, you are responsible for calling end() on the Deflater.)

GC is of limited help in your case because the stream(s) must be dereferenced and eventually finalized. This may take multiple GC sweeps. I experienced a similar problem with Jetty and proposed a fix for it.

Answer 3

Adding to Marian-Daniel Craciunescu's answer

I did have this kind of problem with another type of use.

The solution I found is the following :

• Put the object into a map or somewhere you can easily drop its reference
• Put also the object into a WeakReference for using it
• Use your reference as normal
• Delete the reference from the map where no more used

Be careful of inherited references

It is greatly possible your object is "hidden-inner-referenced" in another object, use the debugger and "expand" all objects using it.

You will never know when the GC is running, nor what it will handle (except objects in use).

Useful link

You should read (if it is not already done) this post about WeakReferences : Understanding Java's Reference classes: SoftReference, WeakReference, and PhantomReference

Answer 4

Firstly before thinking about memory leak we should try to understand if the memory requirement is normal for the processing you are doing. What is the size of memory available to JVM i.e -XmX. If you increase the size of memory or decrease the size of file being processed does work?

Answer 5

Maybe you should use a WeakReference for your object.In this way GC may finalize () your object much quickly

http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/ref/WeakReference.html http://weblogs.java.net/blog/2006/05/04/understanding-weak-references