Websphere 7 Thread Dump Analysis

Question

We are using WAS 7 and our ear is deployed on this.

Environment Details

Operating System : AIX 7.1

Processor Architecture : ppc64

Number of Processors : 8

Java version : JRE 1.6.0 AIX ppc64-64 build jvmap6460sr10fp1-20120202_101568 (pap6460sr10fp1-20120321_01(SR10 FP1))

Virtual machine version : VM build 20120202_101568 Just-In-Time(JIT) compiler switch, Ahead-Of-Time (AOT) compiler switch, Compiler version : r9_20111107_21307ifx1

Garbage collector version : GC - 20120202_AA_CMPRSS

Java Heap Information

Maximum Java heap size : 1024m

Initial Java heap size : 512m

I have tried to analyse heap dump using IBM Thread and Monitor Dump Analyser tool.

Below is summary of threads

But I am not able to analyse, whether this statics is good or needs improvement ?

As we have so many threads in Parking / Waiting on Condition consistently (taken thread sump 10 times every day), some times application takes 4 sec to process 5 messages, which suppose to be 5 messages per second.

Application works fine and achieve this SLA of 5 messages per second, but say few times in a day it takes 4 sec to process 5 messages.

Note: Concurrent processing.

And If I am trying to get thread dump at that moment, it is like I shared above.

Answer 1

Unless you have a firm reason to think thread-contention is the cause, I don't think a thread-dump is the best place to start investigating why "few times in a day it takes 4 sec to process 5 messages.". There could be many,many causes to the problem, of which thread-contention is only one.

Firstly, I would identify those messages. Presumably the speed of messages is logged somewhere and if not, I would start with that. Can web-server access logs help identify these messages?

Once you have them, is there anything unique to those messages? If you access those messages yourself, are you able to replicate the slowness? Are they always at the same time of day/are they trying to access the same entities or perform the same actions? Are the messages sent by the same user?

If it's reproducible, you're job is half-done. You can now start using profiling tools to identify why it's taking so long.

Even if the slow messages are not reproducible and you cannot identify any distinguishing feature of the messages, at least you now know when it happened and can drill down more precisely.

Once you have the times, you can then start examining different causes at those times. A not-exhausive list would include:

• was there a full-garbage collection going on at that time?
• do other processes run on the application server at that time?
• if it accesses a database, was that under load at that time?
• are there any error messages in the logs for that time?
• was there any thread-contention at that time?
• are there any slow-running database queries at that time (for example, an Oracle AWR might give you this if you used Oracle)?
• were a lot of users, using the site at this time?

Identifying these require answers in their own rights, but Stackoverflow should have many. Regarding the thread-contention - was that thread analysis taken at the time the problem was occurring. Personally, I would like a thread-dump from that moment, to try and see which threads were were blocked by other threads.