Escape a string for a sed replace pattern

Question

In my bash script I have an external (received from user) string, which I should use in sed pattern.

REPLACE="<funny characters here>"
sed "s/KEYWORD/$REPLACE/g"

How can I escape the $REPLACE string so it would be safely accepted by sed as a literal replacement?

NOTE: The KEYWORD is a dumb substring with no matches etc. It is not supplied by user.

Answer 1

Warning: This does not consider newlines. For a more in-depth answer, see this SO-question instead. (Thanks, Ed Morton & Niklas Peter)

Note that escaping everything is a bad idea. Sed needs many characters to be escaped to get their special meaning. For example, if you escape a digit in the replacement string, it will turn in to a backreference.

As Ben Blank said, there are only three characters that need to be escaped in the replacement string (escapes themselves, forward slash for end of statement and & for replace all):

sed -e 's/[\/&]/\\&/g'

If you ever need to escape the KEYWORD string, the following is the one you need:

sed -e 's/[]\/$*.^[]/\\&/g'

Remember, if you use a character other than / as delimiter, you need replace the slash in the expressions above wih the character you are using. See PeterJCLaw's comment for explanation.

Edited: Due to some corner cases previously not accounted for, the commands above have changed several times. Check the edit history for details.

Answer 2

The sed command allows you to use other characters instead of / as separator:

sed 's#"http://www\.fubar\.com"#URL_FUBAR#g'

The double quotes are not a problem.

Answer 3

The only three literal characters which are treated specially in the replace clause are / (to close the clause), \ (to escape characters, backreference, &c.), and & (to include the match in the replacement). Therefore, all you need to do is escape those three characters:

sed "s/KEYWORD/$(echo $REPLACE | sed -e 's/\\/\\\\/g; s/\//\\\//g; s/&/\\\&/g')/g"

Example:

$ export REPLACE="'\"|\\/><&!"
$ echo fooKEYWORDbar | sed "s/KEYWORD/$(echo $REPLACE | sed -e 's/\\/\\\\/g; s/\//\\\//g; s/&/\\\&/g')/g"
foo'"|\/><&!bar

Answer 4

Based on Pianosaurus's regular expressions, I made a bash function that escapes both keyword and replacement.

function sedeasy {
  sed -i "s/$(echo $1 | sed -e 's/\([[\/.*]\|\]\)/\\&/g')/$(echo $2 | sed -e 's/[\/&]/\\&/g')/g" $3
}

Here's how you use it:

sedeasy "include /etc/nginx/conf.d/*" "include /apps/*/conf/nginx.conf" /etc/nginx/nginx.conf

Answer 5

It's a bit late to respond... but there IS a much simpler way to do this. Just change the delimiter (i.e., the character that separates fields). So, instead of s/foo/bar/ you write s|bar|foo.

And, here's the easy way to do this:

sed 's|/\*!50017 DEFINER=`snafu`@`localhost`\*/||g'

The resulting output is devoid of that nasty DEFINER clause.

Answer 6

It turns out you're asking the wrong question. I also asked the wrong question. The reason it's wrong is the beginning of the first sentence: "In my bash script...".

I had the same question & made the same mistake. If you're using bash, you don't need to use sed to do string replacements (and it's much cleaner to use the replace feature built into bash).

Instead of something like, for example:

function escape-all-funny-characters() { UNKNOWN_CODE_THAT_ANSWERS_THE_QUESTION_YOU_ASKED; }
INPUT='some long string with KEYWORD that need replacing KEYWORD.'
A="$(escape-all-funny-characters 'KEYWORD')"
B="$(escape-all-funny-characters '<funny characters here>')"
OUTPUT="$(sed "s/$A/$B/g" <<<"$INPUT")"

you can use bash features exclusively:

INPUT='some long string with KEYWORD that need replacing KEYWORD.'
A='KEYWORD'
B='<funny characters here>'
OUTPUT="${INPUT//"$A"/"$B"}"

Answer 7

Use awk - it is cleaner:

$ awk -v R='//addr:\\file' '{ sub("THIS", R, $0); print $0 }' <<< "http://file:\_THIS_/path/to/a/file\\is\\\a\\ nightmare"
http://file:\_//addr:\file_/path/to/a/file\\is\\\a\\ nightmare

Answer 8

Here is an example of an AWK I used a while ago. It is an AWK that prints new AWKS. AWK and SED being similar it may be a good template.

ls | awk '{ print "awk " "'"'"'"  " {print $1,$2,$3} " "'"'"'"  " " $1 ".old_ext > " $1 ".new_ext"  }' > for_the_birds

It looks excessive, but somehow that combination of quotes works to keep the ' printed as literals. Then if I remember correctly the vaiables are just surrounded with quotes like this: "$1". Try it, let me know how it works with SED.

Answer 9

I have an improvement over the sedeasy function, which WILL break with special characters like tab.

function sedeasy_improved {
    sed -i "s/$(
        echo "$1" | sed -e 's/\([[\/.*]\|\]\)/\\&/g' 
            | sed -e 's:\t:\\t:g'
    )/$(
        echo "$2" | sed -e 's/[\/&]/\\&/g' 
            | sed -e 's:\t:\\t:g'
    )/g" "$3"
}

So, whats different? $1 and $2 wrapped in quotes to avoid shell expansions and preserve tabs or double spaces.

Additional piping | sed -e 's:\t:\\t:g' (I like : as token) which transforms a tab in \t.

Answer 10

don't forget all the pleasure that occur with the shell limitation around " and '

so (in ksh)

Var=">New version of \"content' here <"
printf "%s" "${Var}" | sed "s/[&\/\\\\*\\"']/\\&/g' | read -r EscVar

echo "Here is your \"text\" to change" | sed "s/text/${EscVar}/g"

Answer 11

If the case happens to be that you are generating a random password to pass to sed replace pattern, then you choose to be careful about which set of characters in the random string. If you choose a password made by encoding a value as base64, then there is is only character that is both possible in base64 and is also a special character in sed replace pattern. That character is "/", and is easily removed from the password you are generating:

# password 32 characters log, minus any copies of the "/" character.
pass=`openssl rand -base64 32 | sed -e 's/\///g'`;

Answer 12

If you are just looking to replace Variable value in sed command then just remove Example:

sed -i 's/dev-/dev-$ENV/g' test to sed -i s/dev-/dev-$ENV/g test

Answer 13

An easier way to do this is simply building the string before hand and using it as a parameter for sed

rpstring="s/KEYWORD/$REPLACE/g"
sed -i $rpstring  test.txt