How to append string to file if it is not included in the file?

Question

Protagonists

• The Admin
• Pipes
• The Cron Daemon
• A bunch of text processing utilities
• netstat
• >> the Scribe

Setting

The Cron Daemon is repeatedly performing the same job where he forces an innocent netstat to show the network status (netstat -n). Pipes then have to pick up the information and deliver it to bystanding text processing utilities (| grep tcp | awk '{ print $5 }' | cut -d "." -f-4). >> has to scribe the important results to a file. As his highness, The Admin, is a lazy and easily annoyed ruler, >> only wants to scribe new information to the file.

*/1 * * * * netstat -n | grep tcp | awk '{ print $5 }' | cut -d "." -f-4 >> /tmp/file

Soliloquy by >>

To append, or not append, that is the question: 
Whether 'tis new information to bother The Admin with 
and earn an outrageous Fortune,
Or to take Arms against `netstat` and the others, 
And by opposing, ignore them? To die: to sleep;

note by the publisher: For all those that had problems understanding Hamlet, like I did, the question is, how do I check if the string is already included in the file and if not, append it to the file?

Answer 1

Unless you are dealing with a very big file, you can use the uniq command to remove the duplicate lines from the file. This means you will also have the file sorted, I don't know if this is an advantage or disadvantage for you:

netstat -n | grep tcp | awk '{ print $5 }' | cut -d "." -f-4 >> /tmp/file && sort /tmp/file | uniq > /tmp/file.uniq

This will give you the sorted results without duplicates in /tmp/file.uniq

Answer 2

What a piece of work is piping, how easy to reason about, 
how infinite in use cases, in bash and script, 
how elegant and admirable in action,
how like a vim in flexibility, 
how like a gnu!

Here is a slightly different take:

netstat -n | awk -F"[\t .]+" '/tcp/ {print $9"."$10"."$11"."$12}' | sort -nu | while read ip; do if ! grep -q $ip /tmp/file; then echo $ip >> /tmp/file; fi; done;

Explanation:

awk -F"[\t .]+" '/tcp/ {print $9"."$10"."$11"."$12}'

Awk splits the input string by tabs and ".". The input string is filtered (instead of using a separate grep invocation) by lines containing "tcp". Finally the resulting output fields are concatenated with dots and printed out.

sort -nu

Sorts the IPs numerically and creates a set of unique entries. This eliminates the need for the separate uniq command.

if ! grep -q $ip /tmp/file; then echo $ip >> /tmp/file; fi;

Greps for the ip in the file, if it doesn't find it, the ip gets appended.

Note: This solution does not remove old entries and clean up the file after each run - it merely appends - as your question implied.