Zombie threads eating my brainz (J2EE, Tomcat, Hibernate, Quartz)

Question

It is Hallowe'en after all.

Here's the problem: I'm maintaining some old-ish J2EE code, using Quartz, in which I'm running out of threads. jconsole tells me that there are just short of 60K threads when it goes pear-shaped, of which about 100 (!!) are actually running. Intuition and some googling (see also here) suggest that what's happening is something (I'm betting Quartz) is creating unmanaged threads that never get cleaned up.

Several subquestions:

1. It there a tool that I can use easily to trace thread creation, so I can be certain the issue is really Quartz?

2. Most everything I've found about similar problems references Weblogic; is this a false lead for Tomcat?

3. Does anyone have a known solution?

It's been years since I did J2EE, so I wouldn't be too surprised if this is something that can be solved simply.

Update: It's clearly increasing threads without bound, see this plot from jconsole.

Answer 1

• Try to increase the logging level of org.quartz.simpl.SimpleThreadPool to debug to get more information.

• If that does not work, try a logging listener. Quartz has a JobListener interface, which is specified in its tutorial. A listener can help you trace job execution. Maybe jobs just don't finish and get deadlocked.

• Configure org.quartz.threadPool.threadCount to stop running out of threads.

update:

• Also, you might want to take a thread dump and see the thread stats. visual vm has a plugin called TDA, or you can use Thread dump analyzer directly.

• Just in case, check the quartz version to see if there is no known bug.

Answer 2

Have you had a look with jvisualvm - it gives some more information.

Also, get stack traces to see what the threads are actually waiting on. You might have an aha-feeling right there.