What do the "compacting perm gen" values represent?

Question

I'm investigating a JVM crash on one of our production systems, what do the following memory values represent in the hs_err_pid log file snippet below?

Heap
 par new generation   total 1258624K, used 955445K [0x00000005c0000000, 0x00000006155b0000, 0x000000066aaa0000)
  eden space 1118784K,  73% used [0x00000005c0000000, 0x00000005f1e52598, 0x0000000604490000)
  from space 139840K,  98% used [0x000000060cd20000, 0x00000006153db100, 0x00000006155b0000)
  to   space 139840K,   0% used [0x0000000604490000, 0x0000000604490000, 0x000000060cd20000)
 tenured generation   total 2796224K, used 1745107K [0x000000066aaa0000, 0x0000000715550000, 0x00000007c0000000)
   the space 2796224K,  62% used [0x000000066aaa0000, 0x00000006d52d4d90, 0x00000006c2e0c400, 0x0000000715550000)
 compacting perm gen  total 482944K, used 482943K [0x00000007c0000000, 0x00000007dd7a0000, 0x0000000800000000)
   the space 482944K,  99% used [0x00000007c0000000, 0x00000007dd79fff0, 0x00000007dd7a0000, 0x00000007dd7a0000)
No shared spaces configured.

My concern is with the "compacted perm gen" usage: does it mean percentage used of maximum allocated perm gen heap, or percentage used of maximum heap, or something else? The percentage provided appears to be a division of the used/total, is this the total allocated perm gen? Since our -XX:MaxPermSize is set to 1GB...

Are there any useful resources (other than the Oracle whitepaper, which does not mention hs_err files) to interpret the data dumped on a JVM crash?

Answer 1

I never found a reference that accurately described the "compacting perm gen" values, but our own investigation proved that the reported values were:

currently used PermGen / currently allocated PermGen

In the example in my question, this meant that 482944K of memory had already been allocated for PermGen and 482943K of it had been used at the point of GC (99%). Our max PermGen size was set to 1048576K (1GB), so the collection process had plenty of reserved resources to re-allocate with.

For those encountering similar issues - we solved our problem in the end. In our case it turned out to be a third-party library that made use of the sun.misc.Unsafe class, which is notoriously "unsafe" when used incorrectly.

In this case, a piece of logic for cloning objects passed a specific ClassLoader to some sun.misc.Unsafe operations to copy objects. On certain machines, the copied objects were regularly being created in a corrupted state. When the JVM attempted to garbage collect, it would eventually harvest one of these bad objects and crash. This always caused the error described in my question.

Answer 2

But just because you set MaxPermSize to 1GB doesn't mean HotSpot will honor it. I don't think I've ever gotten more than 512MB. Your JVM's 482MB is pretty close to this figure.

In any case, 512MB is plenty for PermGen since it's only used to hold classes' metadata, it won't grow in size unless more classes are loaded through ClassLoader. So the question is: do you really need more than 512MB to hold roughly the size of the bytecodes of all the necessary classes in memory?

It's very possible that your system actually has memory leaks in PermGen.