Growth in old gen memory usage always means memory leak in Java?

Question

We have several VMs which are running data service in production, client sends Restful HTTP requests to data service, the load is kind of heavy (500 requests per second per host in general) and load is always balanced on each VM. We have same configuration on all hosts (2 CPUs, -Xms2048m -Xmx4096m -XX:MaxPermSize=192m -XX:NewSize=512m -XX:MaxNewSize=512M -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+HeapDumpOnOutOfMemoryError)

Two days ago, we saw old gen heap usage start growing on 5 of those VMs (300 MB per day), old gen heap usage on other ones stay the same (around 80 MB), we are trying to identify the root cause, may I ask if this is a memory leak issue or just a normal situation? Does growth in old gen memory usage always mean memory leak in Java?

Thanks.

Update: We just restarted those 5 hosts yesterday, old gen heap usage on all of them went back to normal as other ones, however, after the peak load we had this morning, the old gen heap usage on one of them started growing again...

Answer 1

Does growth in old gen memory usage always mean memory leak in java?

Not necessarily.

The concurrent mark sweep garbage collector does not compact the old gen during collection. So under sufficient memory load, it's possible to get a large amount of fragmentation, making it impossible to reclaim enough memory to allow promotion of tenured objects into the old gen space.

Try turning on these params and see what's going on:

-XX:+PrintGCDetails -XX:+PrintPromotionFailure -XX:PrintFLSStatistics=1

Look for promotion failures and frequent full GC sweeps that fail to free up a lot of memory.

If you're using Java 7 or higher you can try switching to the G1 collector (-XX:+UseG1GC instead of -XX:+UseConcMarkSweepGC). This is a compacting collector which avoids some of the above issues.

If you're still running into problems after that, then I'd look to your code to see if something is hanging onto object references when it shouldn't.

Edit: since this is happening on some hosts and not others, I'd lean towards a code issue, perhaps related to unexpected user input, since it only occurs sporadically.

Answer 2

It sounds like what you are describing is a memory leak. Because for me a memory leak is when your memory increases in time without any good reason or without developers understanding why. Which probably implies that some data you actually do not need is still kept in memory probably as a result of a bug.

I would consider using a good profiler to get to the root cause. It is probably easiest. I don't know if I'm allowed to name products here but JProfiler saved my team in multiple projects I worked on.