jdk6 Problematic frame: # J java.util.LinkedHashMap.addEntry(ILjava/lang/Object;Ljava/lang/Object;I)V

Question

Is there a way to resolve this kind of error report:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fc955e66998, pid=25851, tid=140467030525696
#
# JRE version: 6.0_37-b06
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.12-b01 mixed mode linux-amd64 compressed     oops)
# Problematic frame:
# J  java.util.LinkedHashMap.addEntry(ILjava/lang/Object;Ljava/lang/Object;I)V

?

The crash occurs quite frequently (1-2 times per day in web server production), almost always with different problematic frame report.

Here are examples of some error reports:

# J  java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter()Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;
# J  java.util.LinkedHashMap.addEntry(ILjava/lang/Object;Ljava/lang/Object;I)V
# C  [libc.so.6+0x6bb34]
# C  [libgobject-2.0.so.0+0x2346f]  g_type_check_instance_is_a+0x43
# C  [libgobject-2.0.so.0+0x2346f]  g_type_check_instance_is_a+0x43
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x32d166]  CardTableRS::write_ref_field_gc_par(void*, oopDesc*)+0x26
# V  [libjvm.so+0x7a33e2]  ContiguousSpace::prepare_for_compaction(CompactPoint*)+0x242
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x76943b]  ReferenceProcessor::balance_queues(DiscoveredList*)+0x32b
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x32d166]  CardTableRS::write_ref_field_gc_par(void*, oopDesc*)+0x26
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x76943b]  ReferenceProcessor::balance_queues(DiscoveredList*)+0x32b

The only thing that seems to trigger the crashes is high memory usage approx 30gb, even though that has not always been the case (there are some crashes at instants where gc log shows low memory usage). The crashes do not occur when running in -Xint mode, but that mode is so slow that it is not an option.

Seems to be difficult to make any simple 'reproduceable code' to reproduce the error that occurs in production environment of a complex app.

What to do? I did report a bunch of these at the Oracle crash site though ...

I do not suspect hardware memory problems because nothing else ever crashes except java. And there is no custom native jni code in the application.

Our vm parameters are -server -Xss4096k -Xms32255M -Xmx32255M -Xnoclassgc -XX:+UseNUMA -XX:MaxPermSize=512m -XX:+UseGCOverheadLimit -verbose:gc -Xmaxf1 -XX:+UseCompressedOops -XX:+DisableExplicitGC -XX:+AggressiveOpts -XX:+ScavengeBeforeFullGC -XX:CMSFullGCsBeforeCompaction=10 -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:GCTimeRatio=19 -XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=500 -Xloggc:gc.log.

Answer 1

Upgraded to jdk7 Java(TM) SE Runtime Environment (build 1.7.0_09-b05) and haven't had any problems since; follwing vmargs:

-server -Xss4096k -XX:MaxPermSize=512m -Xms32255M -Xmx32255M -Xnoclassgc -XX:+UseNUMA -XX:+UseBiasedLocking -XX:+UseFastAccessorMethods -XX:ReservedCodeCacheSize=48m -XX:+UseStringCache -XX:+HeapDumpOnOutOfMemoryError -XX:+UseGCOverheadLimit -Duser.timezone=EET -Xmaxf1 -XX:+UseCompressedOops -XX:+DisableExplicitGC -XX:+AggressiveOpts -XX:CMSInitiatingOccupancyFraction=70 -XX:+ParallelRefProcEnabled -XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=100 -XX:+UseG1GC -XX:GCPauseIntervalMillis=3000 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -Xloggc:gc.log

Answer 2

While it is possible that the crash is caused by a JVM bug, it is more likely to be caused by some JNI / JNA native code that you have written, or that is part of some 3rd-party library that you are using.

What to do?

Here is a blog on the topic of how to get started with debugging a crash dump: http://www.javacodegeeks.com/2012/01/debugging-jvm.html

In your case, the fact that reports are all different is going to make the problem harder to track down. It sounds like you may have a problem with something "randomly" corrupting heap objects.

I did report a bunch of these at the Oracle crash site though ...

Unless you have a support contract, Oracle are unlikely to get back to you with a solution.

Answer 3

If the crashes are frequent with apparently random causes then I'd be thinking in terms of a possible hardware problem (e.g. dodgy RAM). I'd be inclined to run a full battery of hardware diagnostics on the server and see if that throws up anything.

Answer 4

I found this article on the web ` If you use the Java™ virtual machine (JVM) AggressiveOpts option with a Java Platform Enterprise Edition (Java EE) application that contains Enterprise JavaBeans (EJB) files, the JVM might crash. To work around this issue, disable the DoEscapeAnalysis optimization using the following arguments:

-XX:+AggressiveOpts -XX:-DoEscapeAnalysis`:

http://www-01.ibm.com/support/docview.wss?uid=swg21422605

What is strange is, that -XX:-CMSIncrementalMode made the system VERY instable, I had to enable this option.