Recording messages received on a port with SOCAT

Question

I have a server with an open port which receives between 50 and 1000 messages per second. By message I mean that a single line of text is sent.

Essentially we want to record these messages in a file which will be processed every hour (or x minutes).

I have created a bash script (see below) which runs in the background and it works except when I kill the socat process (so I can take the file for processing and it can start a new file) we get part of a message, plus I am sure we are losing messages during the split second that socat is down.

DELAY="3600"
while true
do  
    NEXT_STOP=`date +%s --date "$DELAY second"`
    (
        while [ "$(date +%s)" -lt "$NEXT_STOP" ]
        do
            killall socat
            socat -u TCP-LISTEN:6116,reuseaddr,keepalive,rcvbuf=131071,reuseaddr OPEN:/var/ais/out_v4.txt,creat,append
        done
    ) & sleep $DELAY ; killall socat

    mv /var/ais/out_v4.txt "/var/ais/_socat_received/"$(date +"%Y-%m-%d-%T")"__out_v4.txt"
done

Is there a way to:

1. Get socat to rotate its output file without killing the process
2. Or can we purge the content of the file whilst SOCAT is writing to it. e.g. cut the first 10000 lines into another file, so the output file remains a manageable size?

Many thanks in advance

Answer 1

For anyone interested the final solution looks like the following, the key difference to Nicholas solution below is that I needed to grep the PID of the socat process rather than use $?:

#!/bin/bash
DELAY=600
SOCAT_PID=$(/bin/ps -eo pid,args | grep "socat -u TCP-LISTEN:12456" | grep -v grep | awk '{ print $1 }')

while `kill -0 $SOCAT_PID`
do  
  touch /var/ais/out.txt
  NEXT_STOP=`date +%s --date "$DELAY second"`
  while  `kill -0 $SOCAT_PID` && [ "$(date +%s)" -lt "$NEXT_STOP" ]
  do
    head -q - >> /var/ais/out.txt
  done
  mv /var/ais/out.txt "/var/ais/_socat_received/"$(date +"%Y-%m-%d-%T")"__out.txt"
done

In addition adding the start script within an infinite while loop so that when the client disconnects we restart socat and wait for the next connection attempt:

while true
do
socat -u TCP-LISTEN:12456,keepalive,reuseaddr,rcvbuf=131071 STDOUT | /var/ais/socat_write.sh
done

Answer 2

Instead of reinventing the wheel, you could use rotatelogs or multilog, both of which read log messages on std input and write them to log files with very flexible rotation config.

Even one step higher, the functionality you described is very similar to what rsyslogd and the like do.

Answer 3

Not so obvious! socat doesn't have any options for changing what it does with the connection halfway through. That means you'll have to be a little bit sneaky. Use socat with the output as STDOUT, and pipe to this script:

#!/bin/bash
rv=0
while [ $rv -lt 1 ]
do  
  NEXT_STOP=`date +%s --date "$DELAY second"`
  while [ "$(date +%s)" -lt "$NEXT_STOP" ] && [ $rv -lt 1 ]
  do
    head -q - >> /var/ais/out_v4.txt
    rv=$?
  done
  mv /var/ais/out_v4.txt "/var/ais/_socat_received/"$(date +"%Y-%m-%d-%T")"__out_v4.txt"
done

Totally untested, but looks reasonable?

socat -u TCP-LISTEN:6116,reuseaddr,keepalive,rcvbuf=131071,reuseaddr STDOUT | ./thescript.sh