Complete lock-up of a Java process

Question

This one is baffling. All of a sudden we've started experiencing a situation where all threads in a Java process reach a BLOCKED state and the process is completely frozen. There's no jvm.log or hs*.log file to point to what's going on.

A jstack dump lists many threads, all in BLOCKED state, some of them as baffling as this:

Thread 14766: (state = BLOCKED)
- java.lang.String.substring(int, int) @bci=62, line=1939 (Compiled frame)
- java.util.Formatter.parse(java.lang.String) @bci=73, line=2469 (Compiled frame)
- java.util.Formatter.format(java.util.Locale, java.lang.String, java.lang.Object[]) @bci=213, line=2438 (Compiled frame)
- java.util.Formatter.format(java.lang.String, java.lang.Object[]) @bci=7, line=2367 (Compiled frame)
- java.lang.String.format(java.lang.String, java.lang.Object[]) @bci=9, line=2769 (Compiled frame)

(cropped the rest)

Or this:

Thread 14794: (state = BLOCKED)
- java.lang.Throwable.toString() @bci=17, line=344 (Interpreted frame)
- java.lang.String.valueOf(java.lang.Object) @bci=10, line=2826 (Compiled frame)
- java.io.PrintWriter.println(java.lang.Object) @bci=1, line=710 (Interpreted frame)
- java.lang.Throwable.printStackTrace(java.io.PrintWriter) @bci=6, line=509 (Compiled frame)
- org.apache.log4j.DefaultThrowableRenderer.render(java.lang.Throwable) @bci=19, line=60 (Compiled frame)
- org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep() @bci=50, line=87 (Interpreted frame)
- org.apache.log4j.spi.LoggingEvent.getThrowableStrRep() @bci=13, line=413 (Compiled frame)
- org.apache.log4j.WriterAppender.subAppend(org.apache.log4j.spi.LoggingEvent) @bci=26, line=313 (Compiled frame)
- org.apache.log4j.RollingFileAppender.subAppend(org.apache.log4j.spi.LoggingEvent) @bci=2, line=276 (Interpreted frame)
- org.apache.log4j.WriterAppender.append(org.apache.log4j.spi.LoggingEvent) @bci=10, line=162 (Compiled frame)
- org.apache.log4j.AppenderSkeleton.doAppend(org.apache.log4j.spi.LoggingEvent) @bci=106, line=251 (Compiled frame)
- org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(org.apache.log4j.spi.LoggingEvent) @bci=41, line=66 (Compiled frame)
- org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=26, line=206 (Compiled frame)
- org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=391 (Interpreted frame)
- org.apache.log4j.Category.error(java.lang.Object, java.lang.Throwable) @bci=37, line=322 (Interpreted frame)

(cropped the rest)

I must admit that while I've tried to search for a cause online, I don't know where to begin looking for it.

This is happening under jdk1.6.0_31

Answer 1

Are they all blocked in string operations? Have you run out of heap perm gen space maybe (do a 'jstat -gc' on the hung process)

Answer 2

What is the OS? Multithreading issues are many times OS dependent since their implementation is OS specific. Java threads are just an abstraction of OS threading.

/Koya

Answer 3

I would run the program with -verbosegc. It appears to be blocked waiting for some free space.

Answer 4

This part of exception:

org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=26, line=206 (Compiled frame)
org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=391 (Interpreted frame)
org.apache.log4j.Category.error(java.lang.Object, java.lang.Throwable) @bci=37, line=322 (Interpreted frame)

Probably, it is related to a bug in log4j. Have a look at the following webpage:

https://issues.apache.org/bugzilla/show_bug.cgi?id=41214