Grep from text file over the past hour

Question

I have several commands similar to:

ping -i 60 8.8.8.8 | while read pong; do echo "$(date): $pong" >> /security/latencytracking/pingcapturetest2.txt; done

output:

Tue Feb 4 15:13:39 EST 2014: 64 bytes from 8.8.8.8: icmp_seq=0 ttl=50 time=88.844 ms

I then search the results using:

cat /security/latencytracking/pingcapturetest* | egrep 'time=........ ms|time=......... ms'

I am looking for latency anomalies over X ms.

Is there a way to search better than I am doing and search over the past 1,2,3, etc. hours as opposed to from the start of the file? This could get tedious over time.

Answer 1

You could add unix timestamp to your log, and then search based on that:

ping -i 60 8.8.8.8 | while read pong; do 
   echo "$(date +"%s"): $pong" >> log.txt
done

Your log will have entries like:

1391548048: 64 bytes from 8.8.8.8: icmp_req=1 ttl=47 time=20.0 ms

Then search with a combination of date and awk:

Using GNU Date (Linux etc):

awk -F: "\$1 > $(date -d '1 hour ago' +'%s')" log.txt

or BSD Date (Mac OSX, BSD)

awk -F: "\$1 > $(date -j -v '-1H' +%s)" log.txt

The command uses date -d to translate english time-sentence (or date -v for the same task on BSD/OSX) to unix timestamp. awk then compares the logged timestamp (first field before the :) with the generated timestamp and prints all log-lines which have a higher value, ie newer.

Answer 2

If you are familiar with R: 1. I'd slurp the whole thing in with read.table(), drop the unnecessary columns 2. then do whatever calculations you like Unless you have tens of millions of records, then R might be a bit slow.

Plan B: 1. use cut to nuke anything you dont need and then goto the plan above.

Answer 3

You can also do it with bash. You can compare dates, as follows:

Crop the date field. You can convert that date into the number of seconds since midnight of 1st Jan 1970

date -d "Tue Feb 4 15:13:39 EST 2014" '+%s'

you compare that number against the number of seconds you got one hour ago,

reference=$(date --date='-1 hour' '+%s')

This way you get all records from last hour. Then you can filter after the length of the delay