How do I parse lines from a log file?

Question

I need to extract the values of following output:

Oct  6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0

I'd need for example the value of PROTO stored in a value. tried shellscripting, my problem is that it works only if the log entry is in the same order everytime.

so this doens't work:

while read line
do
        in_if=`echo $line | cut -d ' ' -f 10 | cut -d '=' -f 2`;
        out_if=`echo $line | cut -d ' ' -f 11 | cut -d '=' -f 2`;
        src_ip=`echo $line | cut -d ' ' -f 12 | cut -d '=' -f 2`;
        dst_ip=`echo $line | cut -d ' ' -f 13 | cut -d '=' -f 2`;
        pro=`echo $line | cut -d ' ' -f 20 | cut -d '=' -f 2`;
        echo "$in_if,$out_if,$src_ip,$dst_ip,$pro" >> output.csv;
done < $tmp_file

Answer 1

You can do this without touching Perl. You were on the right track, but with a regex you can search by name, not position.

Also, you should put quotes around $line so you don't get burned by any pipes or semicolons hanging around.

pro=`echo "$line" | grep -o 'PROTO=\w+\+' | cut -d '=' -f 2`;

Of course, if you did want to use Perl, you could make a much slicker solution:

#!/usr/bin/perl
while(<>) {
    /IN=(\S*) .*OUT=(\S*) .*SRC=(\S*) .*DST=(\S*) .*PROTO=(\S*)/
       and print "$1,$2,$3,$4,$5\n";
}

Then call:

./thatScript.pl logFile.txt >>output.csv

Answer 2

Python does this conveniently. A general solution that gets all the KEY=value pairs is:

import re
import fileinput

pair_re = re.compile('([^ ]+)=([^ ]+)')  # Matches KEY=value pair

for line in fileinput.input():  # The script accepts both data from stdin or a filename 

    line = line.rstrip()  # Removes final spaces and newlines
    data = dict(pair_re.findall(line))  # Fetches all the KEY=value pairs and puts them in a dictionary

    # Example of usage:
    print "PROTO =", data['PROTO'], "SRC =", data['SRC']  # Easy access to any value

This is arguably more legible, flexible and convenient than a shell script.

Answer 3

you don't even need to cut:

grep -Po "(?<=PROTO=)\w+" yourFile

OR

 sed -r 's/.*PROTO=(\w+).*/\1/' yourFile

OR

awk -F'PROTO=' '{split($2,a," ");print a[1]}' yourfile

test:

kent$  echo "Oct  6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0"|grep -Po "(?<=PROTO=)\w+"
TCP

kent$  echo "Oct  6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0"|sed -r 's/.*PROTO=(\w+).*/\1/'
TCP

kent$  echo "Oct  6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0"|awk -F'PROTO=' '{split($2,a," ");print a[1]}'
TCP

Answer 4

A straightforward Perl solution might be the most readable one:

#!/usr/bin/env perl

use strict; use warnings;

my $s = q{Oct  6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0
SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64
ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0};

while ($s =~ /(?<k> [A-Z]+) = (?<v> \S*)/xg)  {
    print "'$+{k}' = '$+{v}'\n";
}

C:\Temp> z
'IN' = ''
'OUT' = 'eth0'
'SRC' = '192.168.1.116'
'DST' = '192.168.1.110'
'LEN' = '516'
'TOS' = '0x10'
'PREC' = '0x00'
'TTL' = '64'
'ID' = '4949'
'PROTO' = 'TCP'
'SPT' = '22'
'DPT' = '46216'
'WINDOW' = '446'
'RES' = '0x00'
'URGP' = '0'

You can also assign the information in the log line to a hash:

my %entry = ($s =~ /(?<k> [A-Z]+) = (?<v> \S*)/xg);

Answer 5

In perl this should do it

#consider the $a variable has the log file my
$a = <<log file>>;
my $desired_answer;
#regex 
if ($a =~ m/PROTO=(.*?) /ig) 
{  $desired_answer=$1; }

Answer 6

thanks for all the responses!

i chose the way of shellscripting using egrep and regex...

in_if=`echo "$line" | egrep -Eo 'IN=eth[0-9]*\b' | cut -d '=' -f 2`;
out_if=`echo "$line" | egrep -Eo 'OUT=eth[0-9]*\b' | cut -d '=' -f 2`;
src_ip=`echo "$line" | egrep -Eo 'SRC=[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | cut -d '=' -f 2`;
dst_ip=`echo "$line" | egrep -Eo 'DST=[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | cut -d '=' -f 2`;
pro=`echo "$line" | grep -o 'PROTO=[A-Z]*\b' | cut -d '=' -f 2`;