How can I log messages to generic (changing) files

Question

This question is similar to the unanswered rsyslog specific question on Server Fault. It is not clear to me if the best solution is via already existing software like rsyslog or logstash or if I have to build something myself. This is why I ask here.

Problem

I have several python processes (not handled by multiprocessing) that run in parallel. They build groups which need to write to the same logfile of that group. Logging from multiple processes to the same file has to go through some kind of serialization to avoid file corruption, so rsyslog or logstash come to mind.

What I did not manage to find out (neither for rsyslog nor for logstash) is how to filter for a field (or tag or whatever it might be called in that system) that contains a file name I can't know beforehand (contains version numbers I don't control) and write these messages to the correspondent file.

So, I know how it's done for static input filters with dynamic output. But I don't know how it's done for dynamic input and dynamic output.

My favorite fantasy solution

In the Python applications:

• Create a custom formatter which adds a field (e.g. syslogtag for rsyslog) containing the name of the target log file

• Add a SyslogHandler or something similar to my logger.

In rsyslog or logstash config add a rule like

<config of some logging system>
[filterfor] <field> -> write messages containing <value of field> to logfile /path/to/logs/<value of field>.log

done.

Question

Is this possible out of the box or (in logstash) with a (yet to be written) plugin?

EDIT: Maybe an example will clarify what I am looking for.

Example log messages with the target in brackets as exapmle:

<group_a> Message that will end up only in "group_a.log"
<group_a> Another Message for "group_a.log"
<group_b> Some interesting message for "group_b.log"
<group_c> Message for "group_c.log"

These messages based on the matched name will then end up in the corresponding log file. For the above example, the logfiles with their contents would be:

/var/log/group_logs/group_a.log
   <group_a> Message that will end up only in "group_a.log"
   <group_a> Another Message for "group_a.log"

/var/log/group_logs/group_b.log
<group_b> Some interesting message for "group_b.log"

/var/log/group_logs/group_c.log
   <group_c> Message for "group_c.log"

So it would work like a regex that saves the matched name in a group and uses it as the name of the file.

Answer 1

A working solution for rsyslog works as follows:

1. All log messages contain the name of the target file name delimited by (arbitrary) start and end delimiters. They must only occur to wrap the file name (in this case we use [[ as start and ]] as end delimiter)
2. The log messages are sent to the local0 facility to separate them from other messages.

The significant parts in the rsyslog configuration are:

$template debugFile,"/var/log/%msg:R,ERE,1,DFLT:\[\[(.*)\]\]--end%.log"

Generate absolute path of the log file from the message via regex - the Regular Expression Checker/Generator provided by the rsyslog team can be helpful here.

$template rawFormat,"%msg%\n"

Output template containing only the original log message (mainly cosmetic).

local0.* ?debugFile;rawFormat

Use this template and format for all logs send to the local0 facility.

Answer 2

The following solution is for logstash (no plugins required).

I'd make each process output to a separate file, say /path/logs/process-<unique-process-specific-id>.log. The file would contain lines like:

<group> <rest of line>

Then you could use this configuration:

input {
  file {
    path => "/path/logs/process-*.log"
    start_position => beginning
  }
}

filter {
  grok {
    match => { "message" => "\A(?<group>[^ ]++) (?<rest>.*+)\z" }
  }
}

output {
  file {
    path => "/path/logs/group-%{group}.log"
    message_format => "%{rest}"
  }
}

So if one process is writing a line to /path/logs/process-foobar.log looking like:

groupA Hello world!

.. then this would then produce a log file /path/logs/group-groupA.log with the contents

Hello world!