How find duplicated[s] line[s] and remove it? [duplicate]

Question

This question already has answers here:

How to delete duplicate lines in a file without sorting it in Unix (9 answers)

Closed 8 years ago.

I had to append in my nrpe.cfg file this line, in more than 200 servers:

command[check_utc]=/company/nrpe/libexec/check_utc.sh

But for some issues some machines have more than once this line. So now I would like to check if there's more than once and remove it, leaving just one line with the specified command. I need to do this in shell script.

Answer 1

Method 1: Eliminate all duplicate lines

    cat filename | sort | uniq > outfile
    mv outfile filename

The | characters feed the output of the commands on the left into the commands on the right as input.

The cat command reads the contents of the file to standard output (the screen or another file)

The sort command sorts the output alphabetically, which is needed before using uniq...

The uniq command eliminates adjacent duplicate values

The > outfile writes the output of the last step to a file named "outfile"

The mv command replaces the original file with the new data file outfile

This method will be destructive because it will remove all duplicate lines of ANY kind, not just the one you want to remove.

Method 2: Remove all instances of the specific line, then re-add once

    cat filename | sed 's/command\[check_utc\]\=\/company\/nrpe\/libexec\/check_utc.sh//' > outfile
    mv outfile filename

The sed command allows string substitutions to be performed on specific string patterns. the format for such a command is sed 's/find/replace/' to find the first instance of 'find' on each line and replace it with 'replace'. To make it work on every instance on each line you add 'g' to the end, i.e. sed 's/find/replace/g'

The \ characters cause sed to literally interpret special characters in the string that would otherwise be misread as special instructions.

The command above will completely remove all instances of that specific string (replacing it with nothing), which you will then need to follow with editing the code and adding it back in once

Method 3: Remove all instances, then re-add automatically to the end of the file

    cat filename | sed 's/command\[check_utc\]\=\/company\/nrpe\/libexec\/check_utc.sh//' > outfile
    mv outfile filename
    echo "command[check_utc]=/company/nrpe/libexec/check_utc.sh" >> filename

This is the same code as above, but after removing all instances of the string, you use echo to print the line and use >> to append the text to the end of the contents of filename

Answer 2

If the lines are not adjacent, and you do not mind sorting them, this will work.

$ cat foo.txt
wings of fire
this is a test
food that are killing you
this is a test

Result

$ sort -u foo.txt
food that are killing you
this is a test
wings of fire