Some JVMs on the box are showing higher CPU utilization

Question

We have a weblogic cluster across 6 boxes. Each box has 3 JVMs. 2 JVMs on box 4 are showing very high CPU utilization (in the range of 80-90%) compared to other JVMs which have less than 10% CPU utilization.

We checked load-balancing. The requests are getting uniformly distributed to all the JVMs. All the JVMs are doing GC correctly, there is no issue with Garbage collection. Each JVM has the same GC and memory configuration.

Is there any way to figure out what threads are utilizing high CPU? We can not restart the JVMs or modify any settings on them as those are the production JVMs.

Answer 1

Try this shell on you box 4.

show-busy-java-threads.sh

It will show java busy threads like this:

---

The stack of busy(0.2%) thread(3901/0xf3d) of java process(3626) of user(zuojing):
"ApplicationImpl pooled thread 16" daemon prio=10 tid=0x00007fbd54076000 nid=0xf3d waiting on condition [0x00007fbcd9636000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.intellij.util.TimeoutUtil.sleep(TimeoutUtil.java:58)
    at com.intellij.util.io.BaseOutputReader.doRun(BaseOutputReader.java:116)
    at com.intellij.util.io.BaseOutputReader$1.run(BaseOutputReader.java:57)
    at com.intellij.openapi.application.impl.ApplicationImpl$8.run(ApplicationImpl.java:454)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:701)
    at com.intellij.openapi.application.impl.ApplicationImpl$1$1.run(ApplicationImpl.java:152)

The stack of busy(0.2%) thread(3897/0xf39) of java process(3626) of user(zuojing):
"ApplicationImpl pooled thread 15" daemon prio=10 tid=0x00007fbd1c39b800 nid=0xf39 waiting on condition [0x00007fbcd9838000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.intellij.util.TimeoutUtil.sleep(TimeoutUtil.java:58)
    at com.intellij.util.io.BaseOutputReader.doRun(BaseOutputReader.java:116)
    at com.intellij.util.io.BaseOutputReader$1.run(BaseOutputReader.java:57)
    at com.intellij.openapi.application.impl.ApplicationImpl$8.run(ApplicationImpl.java:454)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:701)
    ....

---

Then trace the stack and find out what the busy threads doing in you source code.