how to revice syslog-ng's log with line stream by tcp?

Question

I have a machine to revice syslog-ng's log; the edit the config file to write the log to a local file /data/logs/access.log

destination d_nginx_video { file("/data/logs/access.log" create_dirs(yes) template("$PROGRAM $MESSAGE\n"));}; 

then I want to send the log file to a redis queue which I can get my log use some processing

but i need read the log file first and then put the log to redis

so I use tail -F /data/logs/access.log|python process put to redis queue and it is a bad way

I would like to run a tcp server to receive the log file and how to config the syslog-ng

my tcp server script is tcp server script

the 5144 is syslog-ng port which i listen it by my script

Answer 1

Have you considered using the "program" directive in addition to "file" and just piping the log directly to your script instead of involving the network?

destination d_nginx_video { program("/path/to/python/script arg arg arg"
                                    template("$PROGRAM $MESSAGE\n"));
                            file("/data/logs/access.log" create_dirs(yes)
                                 template("$PROGRAM $MESSAGE\n")); };

I am very much not a syslog-ng expert but this seems like it would be pretty similar to your current approach except that it keeps the script running and continually processing logs from within the syslog daemon. The syslog-ng documentation contains a little more information.