How to monitor exceptions or errors generated by other Java applications?

Question

I want to find or develop an application that can run as a daemon, notify the administrator by email or sms when the Java applications running on a host get any exceptions or errors. I know JVMTI can achieve part of my goal, but it will impact performance of the monitored applications(I don't know how much will it be, it will be acceptable if it's slight), besides it seems to be a troublesom job to develop a JVMTI agent and I'm not sure what would happen if several applications running at the same time using the same agent. Is there any better solutions? Thanks in advance.

Answer 1

One way would be to use a logging system like log4j that publishes all errors occuring on system A to a logging server on system B from which you can monitor the errors occured. This isn't a completely generic solutation however, since only exceptions propagated to log4j (or any other logging system) would be handled - but it may be a good start.

Answer 2

The best solution is to have the Java application send its errors via email/sms. The problem is that programs will generate exceptions and handle correctly in normal operation. You only want particular exception.

Failing this you could write a log reader, which reads the logs of the application. This is tricky to get right, but it can be done.

An application can generate 1000+ exception per days and still be behaving normally because the application knows how to handle these exceptions. e.g. every time a socket connection is closed an exception can be thrown.

Answer 3

IMO, the best approach is to deploy an external monitoring system. This can:

• monitor multiple applications
• monitor infrastructure services
• monitor network availability and machine accessibility,
• monitor resources such as processor and file system usage.

Applications can be monitored in a variety of ways, including:

• by processing log events,
• by watching for application restarts,
• by "pinging" the application's web apis to check service liveness, and
• by using the application's JMX interfaces.

This information can be filtered and prioritized in an intelligent fashion, and critical events can be reported by whatever means is most appropriate.

You don't want individual applications sending emails, because they don't have sufficient information to do a decent job. Furthermore, putting the reporting logic into individual applications is likely to lead to inconsistent implementation, poor configurability, and so on.

Answer 4

There is a nearby alternative to JVMTI : JPDA. This infrastructure allows you to create a remote "debugger" (yes, that's what you're planning to do) using Java code, and connect it to the VM using either local or remote connection.

There will be, like for JVMTI, an overhead to program execution. However, as the Trace.java example shows, it's quite simple to both implement and connect to target VM.

Finally, notice if you want to instrument code run by application server (JBoss, Glassfish, Tomcat, you name it) there are various other means available.

Answer 5

I follow the pattern where every exception gets logged to a table. Then an RSS feed selects from that table. I subscribe to the RSS feed in MS Outlook at work and also on my Android phone with a program called NewsRob. NewsRob let me set my phone to alert me when there is something new.

I blog about how to do this HERE. It is in .net, but you get the idea.

As a related step I found a way to notify myself when something DIDN'T happen. That blog is HERE.

Answer 6

There are loads of applications out there that do what you are looking for in a way that does not impact performance. Have you had a look at Kibana/ElasticSearch, or Splunk or Logscape for enterprise solutions ( they both also have free versions).

I'm going to echo what has already been said and highlight what java already provides and what you can do with an external monitoring system. Java already provides:

• log4j - log ERRORS, WARNINGS, FATAL and Exceptions to a file
• JMX - Create custom application metrics and you also have access to java.lang/* which will give you heap memory usage , garbage collection, thread counters etc.
• JVM gc logging - you can log all your garbage collection events to a file and watch for any long Full GC collections.

An external monitoring system will allow you to set alerts triggered off different operational scenarios. You will also get visualisation of your system performance through charts. I've used Logscape's java app in the past to monitor 30 java processes spread out over3 hosts.