Heap dump error on Centos 64bit and openjdk 7

Question

I'm trying to generate a heap dump on my machine that is running a glassfish 3.1.2 using a open-jdk7 java.

I'm using the following command :

jmap -dump:live,format=b,file=dump.t -F 24935

But I keep getting this error :

Attaching to process ID 24935, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.7-b01
Dumping heap to dump.t ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at sun.tools.jmap.JMap.runTool(JMap.java:197)
        at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: sun.jvm.hotspot.utilities.AssertionFailure: Expecting GenCollectedHeap, G1CollectedHeap, or ParallelScavengeHeap, but got sun.jvm.hotspot.gc_interface.CollectedHeap
        at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
        at sun.jvm.hotspot.oops.ObjectHeap.collectLiveRegions(ObjectHeap.java:605)
        at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:244)
        at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51)
        at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:416)
        at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:56)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
        at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)
        ... 6 more

Here is my full java version :

[ufasoli]$ java -version
java version "1.7.0_19"
OpenJDK Runtime Environment (rhel-2.3.9.1.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

The exact CentOS version is :

CentOS release 6.3 (Final)

Any ideas?

Answer 1

Posting this so I dont keep forgetting the solution myself :P

This works:

$ sudo /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/bin/jmap -dump:format=b,file=./testDump   8894
Dumping heap to /home/<snip>/testDump ...
Heap dump file created

This fails:

$ sudo /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/bin/jmap -heap:format=b     8894
Attaching to process ID 8894, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.45-b08
Dumping heap to heap.bin ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jmap.JMap.runTool(JMap.java:197)
at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: sun.jvm.hotspot.utilities.AssertionFailure: Expecting GenCollectedHeap, G1CollectedHeap, or ParallelScavengeHeap, but got sun.jvm.hotspot.gc_interface.CollectedHeap
at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
at sun.jvm.hotspot.oops.ObjectHeap.collectLiveRegions(ObjectHeap.java:604)
at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:244)
at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:416)
at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:56)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)
... 6 more

Answer 2

This may be related, but on some Java versions, there seems to be a slight oddity on recent RHEL / Centos (version >= 6) whereby jmap, jstack and friends need to be run as the exact user that started the process - and it will not work if run as root.

A command like:

sudo -H -u <USERNAME WHO STARTED PROCESS> jmap -histo <PID>

should take care of it, assuming you have appropriate sudo privileges.

Answer 3

Try running without the -F option

Answer 4

According to http://iamadevops.tumblr.com/post/109281462774/howto-fix-jmap-f-dump-format-b-on-archlinux the solution is to "recompile your java with the debug symbols not stripped out" yikes.

Answer 5

If you don't have the option to recompile the code with debug symbols and you have JMX configured, then you may try to generate a heapdump with VisualVM.

For this, start VisualVM and connect to the host via JMX. On the topleft of the monitor panel, you'll see a "Head Dump" button.

The format is the dump file will be different, but you'll be able to open it with jhat or VisualVM.

Answer 6

jmap and jstack do need debug symbols. Fortunately, you may not need to recompile your java.

For Ubuntu:

# apt-get install openjdk-7-dbg

If someone has the command for CentOS, please edit this answer with it.

https://bugzilla.redhat.com/show_bug.cgi?id=1010786#c15

After that this worked for me.

# jmap -dump:live,file=dump -F 11316