Centralize logs from Python and NodeJS thanks to RabbitMQ

Question

I have a service using NodeJS, RabbitMQ and Python workers. The NodeJS brokers use MongoDB and the Python workers have only a connection to the rabbitMQ server.

I would like to be able to centralize all the logs from the different languages in a db.

My idea was to push all the logs in a rabbitMQ queue and then push them in the mongoDB used by NodeJS.

I would like to know if this is the best way to have centralized log and how can I redirect the logging python module to a pika consumer?

Answer 1

It sounds like you want to create a custom logging.Handler. You would override the emit method, and have it publish the log message to a RabbitMQ queue of your choosing. You'll also need to override close, and have it close the RabbitMQ channel/connection, etc.

Then to use the handler, do something like this (see https://docs.python.org/2/howto/logging.html for more info):

import logging

# create logger
logger = logging.getLogger('my_logger')

# create RabbitMQ handler
rh = RabbitMQHandler()  # You need to create this.

# add rh to logger
logger.addHandler(rh)

# start logging stuff
logger.error("An error!")

Answer 2

python's Logbook library has a RabbitMQHandler built into it, so rather than having to write your own, you could probably just use it to redirect logs from your workers back into the RabbitMQ exchange.

Answer 3

This is custom python logging Handler for RabbitMQ which I had written. I had inherited Handler class and had overridden emit and close methods. I used Kombu as python library to access RabbiMQ

import logging

class RabbitMQHandler(logging.Handler):
    """
     A handler that acts as a RabbitMQ publisher
     Requires the kombu module.

     Example setup::

        handler = RabbitMQHandler('amqp://guest:guest@localhost//', queue='my_log')
    """
    def __init__(self, uri=None, queue='logging'):
        logging.Handler.__init__(self)
        try:
            import kombu
        except ImportError:
            raise RuntimeError('The Kombu library is required for the RabbitMQSubscriber.')
        if uri:
            connection = kombu.Connection(uri)

        self.queue = connection.SimpleQueue(queue)

    def emit(self, record):
        self.queue.put(record.msg)

    def close(self):
        self.queue.close()

Usage :

# our own logger
my_logger = logging.getLogger('teja_logger')
my_logger.setLevel(logging.DEBUG)

# rabbitmq handler
rabbitmq_handler = RabbitMQHandler('amqp://guest:guest@localhost:5672//', queue = 'log_q')

# adding rabbitmq handler
my_logger.addHandler(rabbitmq_handler)
my_logger.debug('hello')