Why we use logstash_formatter package in Python?

Question

I am using logstash_formatter python module for sending formatted logs to logstash. The logstash_formatter is converting my passed dictionary to a JSON string. My application is then writing this JSON message to the audit_log file. logstash agent is reading the this log file and sending the JSON data into the RabbitMQ.

Before installing the logstash_formatter package from https://pypi.python.org/pypi/logstash_formatter I expected that since this formatter is passing JSON message to the RabbitMQ indexer, I don't have to add filters to my shipper.conf file for my logstash agent running on my machine. All the JSON fields will automatically be added as tags or fields to logstash and will appear same as a filter in Kibana.

But nothing like that happened, I still have to add the filters into my shipper.conf file. Actually the log message is coming as a message field/tag in the logstash reply.

Now I feel that there is no need of using this package. I would have instead create a dict on my own and converted it to JSON using json module.

Kindly guide me if I am missing something or my understanding about this formatter is totally wrong.

Answer 1

You're right -- you could write your own script that takes records and dumps them out as JSON.

However, that's the point of the module -- it prevents you from having to write and maintain this yourself.

The logstash_formatter package coerces everything to strings, does some date handling, and catches an error in the case that the source host is empty. The module coerces and converts your log file and the logstash agent is concerned with shipping it. You could tell your shipper to filter out some tags or only send some events.

Take a look at the module itself by downloading it from the Package Index -- as of this writing, it's 154 lines and well commented.