Hundreds of JBoss 5.1 System Threads WAITING forever

Question

About 3 months ago we migrated to JBoss 5.1.0.GA and a few days after testing we found several Out Of Memory Errors (OOME) in the logs of our app.

Then I found an issue that described the cause as a bug in the JVM up to version 7. We updated to version 7u25 and we haven't seen more OOME but now I see an abnormally huge thread count: about 2k threads and of them 1.9k are daemon threads.

After checking in our monitoring tools I discovered that they are all threads generated by the JBoss system thread pool (they are all named JBoss System Threads(1)-XXXX). And here is the Stack Trace Details:

"JBoss System Threads(1)-1649" Id=130217 in WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@a4d1d3
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)

I have also checked in the tool for collection leaks and found the next data that I think is somewhat related:

I have checked JBoss JIRA but haven't found something related.

Can someone point me what is going on?

UPDATE:

Here is the thread pool configuration in the jboss-service.xml

<!-- A Thread pool service -->
   <mbean code="org.jboss.util.threadpool.BasicThreadPool"
      name="jboss.system:service=ThreadPool">
      <attribute name="Name">JBoss System Threads</attribute>
      <attribute name="ThreadGroupName">System Threads</attribute>
      <!-- How long a thread will live without any tasks in MS -->
      <attribute name="KeepAliveTime">60000</attribute>
      <!-- The max number of threads in the pool -->
      <attribute name="MaximumPoolSize">3200</attribute>
      <!-- The max number of tasks before the queue is full -->
      <attribute name="MaximumQueueSize">3200</attribute>
      <!-- The behavior of the pool when a task is added and the queue is full.
      abort - a RuntimeException is thrown
      run - the calling thread executes the task
      wait - the calling thread blocks until the queue has room
      discard - the task is silently discarded without being run
      discardOldest - check to see if a task is about to complete and enque
         the new task if possible, else run the task in the calling thread
      -->
      <attribute name="BlockingMode">run</attribute>
   </mbean>

Answer 1

The threads were created by the pool as the version of Java that ran the server was incompatible. We switched to an earlier version and the problem was gone.