Which Appenders should be used in distributed system? How to configure them?

Question

I am trying to add logging component to distributed system. It is written in AspectJ to avoid chaining current source-code. I use socket appender to send logs, but I'd like to try something more effective.

I've heard I should use JMSAppender and AsyncAppender, but I failed to configure it. Should I create Receiver which gathers logs and pass them to database and to GUI (I use ChainSaw)?

I tried to follow turorial1 and tutorial2 , but they aren't clear enough.

Edit:

In a small demo I've prepared I sent 6 logs for a request (simulation of 3 components)

[2012-08-08 15:40:28,957] [request1344433228957] [Component_A] [start]
[2012-08-08 15:40:32,050] [request1344433228957] [Component_B] [start]
[2012-08-08 15:40:32,113] [request1344433228957] [Component_C] [start]
[2012-08-08 15:40:32,113] [request1344433228957] [Component_C] [end - throwing]
[2012-08-08 15:40:32,144] [request1344433228957] [Component_B] [end]
[2012-08-08 15:40:32,175] [request1344433228957] [Component_A] [end]

Using socket Appender. So my log4j.properties is:

log4j.rootLogger=DEBUG, server

log4j.appender.server=org.apache.log4j.net.SocketAppender
log4j.appender.server.Port=4712
log4j.appender.server.RemoteHost=localhost
log4j.appender.server.ReconnectionDelay=1000

so I run

>java -classpath log4j-1.2.17.jar org.apache.log4j.net.SimpleSocketServer 4712 log4j-server.properties

with configuration

log4j.rootLogger=DEBUG, CA, FA

#
log4j.appender.CA=org.apache.log4j.ConsoleAppender
log4j.appender.CA.layout=org.apache.log4j.PatternLayout
log4j.appender.CA.layout.ConversionPattern=[%d] [%t] [%c] [%m]%n

#
log4j.appender.FA=org.apache.log4j.FileAppender
log4j.appender.FA.File=report.log
log4j.appender.FA.layout=org.apache.log4j.PatternLayout
log4j.appender.FA.layout.ConversionPattern=[%d] [%t] [%c] [%m]%n

Then I send my logs from file to Chainsaw:

It is absolutely basic, but I want to learn how to do it better. First of all, I'd like to send logs asynchronously. Then create very simple Receiver, which e.g. can pass logs to a file.

I tried to follow tutorials I listed above, but I failed. So question is: could you provide some example configuration? Example of Receiver.java and log4.properties files?

Answer 1

Finally I've found how to configure it. I put 2 files into src folder.

jndi.properties

topic.logTopic=logTopic

and log4j-jms.properties

log4j.rootLogger=INFO, stdout, jms

## Be sure that ActiveMQ messages are not logged to 'jms' appender
log4j.logger.org.apache.activemq=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern= 

## Configure 'jms' appender. You'll also need jndi.properties file in order to make it work
log4j.appender.jms=org.apache.log4j.net.JMSAppender
log4j.appender.jms.InitialContextFactoryName=org.apache.activemq.jndi.ActiveMQInitialContextFactory
log4j.appender.jms.ProviderURL=tcp://localhost:61616
log4j.appender.jms.TopicBindingName=logTopic
log4j.appender.jms.TopicConnectionFactoryBindingName=ConnectionFactory

Then I run my program with VM argument

-Dlog4j.configuration=log4j-jms.properties

and receive logs in class Receiver.java

public class Receiver implements MessageListener {

    PrintWriter pw = new PrintWriter("result.log");
    Connection conn;
    Session sess;
    MessageConsumer consumer;

    public Receiver() throws Exception {


        ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory("tcp://localhost:61616");
        Connection conn = factory.createConnection();
        Session sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
        conn.start();
        MessageConsumer consumer = sess.createConsumer(sess.createTopic("logTopic"));
        consumer.setMessageListener(this);
    }

    public static void main(String[] args) throws Exception {
        new Receiver();

    }

    public void onMessage(Message message) {
        try {
            LoggingEvent event = (LoggingEvent) ((ActiveMQObjectMessage) message).getObject();

            DateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS"); 
            String nowAsString = df.format(new Date(event.getTimeStamp())); 

            pw.println("["+ nowAsString + "]" + 
                    " [" + event.getThreadName()+"]" +
                    " ["+ event.getLoggerName() + "]" +
                    " ["+ event.getMessage()+"]");
            pw.flush();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Answer 2

I would use NFS or CDFS and mount a drive on all the machines. Have each application instance write to a different file. You will be able to find all the logs in one directory (or drive) no matter how many machines you use.

I wouldn't use NFS or CDFS over a global WAN with a high latency e.g. > 50 ms round trip. In this cause I have used JMS (but I didn't use log4j)

Answer 3

My two cents.. Whatever you do, make sure that you use asynchronous mechanism to deliver your logs to the receiver, otherwise it will eventually stall your apps. Another point, to deliver logs reliably you should consider a fail over mechanism built into the appender itself - receivers may go offline for short or long time, if you care for the logs, the fail over is definitely required. We have built similar system you describe (sorry for the add), but if you like you can use our appender (look in downloads), it's free and has the sources. There is also a video tutorial. It has fail over and flexible asynchronous mechanism plus a backup fall back.

How many appenders should you use? One appender per jvm will do all right. Config files should probably be per jvm, not sure how you intend to implement the receiver, in any case the appenders need to find your receiver which is usually host port pair at least. Regarding the database, my experience is very sour with RDBMS (we are moving to nosql) but if you don't go above couple of hundred million records, most commercial databases will do with some effort. Not a simple task I must say, took us couple of years to build commercial quality system you just drawn with few skinny rectangles :)

Answer 4

I'd recommend syslog and the built in syslog appender. Use TCP for reliable logging (+Asyc appender maybe) or UDP for fire-and-forget logging. I have a rsyslog config if you need.