invoking sed with a shell variable

Question 1

Simple looking question with a complicated answer. Most of the issue is with the shell; it is only partly a problem with sed. (In other words, you could use a number of different commands instead of sed and would run into similar issues.)

Note that most commands documented with an option letter and a separate argument string will also work when the argument string is attached to the option. For example:

sort -t :
sort -t:

Both of these give the value : to the -t option. Similarly with sed and the -e option. That is, you can write either of these:

sed -n -e /match/p
sed -n -e/match/p

Let's look at the one of the working sed commands you wrote:

$ s="-e s/a/b/"
$ ls |  sed $s

What the sed command is passed here is two arguments (after it's command name):

-e
s/a/b/

This is a perfectly fine set of arguments for sed. What went wrong with the first one, then?

$ s="-e 's/^ *//' -e 's/ *$//'"
$ ls | sed $s

Well, this time, the sed command was passed 6 arguments:

-e
's/^
*//'
-e
's/
*$//'

You can use the al command (argument list — print each argument on its own line; it is described and implemented at the bottom of this answer) to see how arguments are presented to sed. Simply type al in place of sed in the examples.

Now, the -e option should be followed by a valid sed command, but 's/^ is not a valid command; the quote ' is not a valid sed command. When you type the command at the shell prompt, the shell processes the single quote and removes it, so sed does not normally see it, but that happens before shell variables are expanded.

Why, then, does the eval work:

$ s="-e 's/^ *//' -e 's/ *$//'"
$ ls | eval sed $s

The eval re-evaluates the command line. It sees:

eval sed -e 's/$ *//' -e 's/ *$//'

and goes through the full evaluation process. It removes the single quotes after grouping the characters, so sed sees:

-e
s/$ *//
-e
s/ *$//

which is all completely valid sed scripting.

One of your tests was:

$ s="-e s/^ *//"
$ ls |  sed $s

And this failed because sed was given the arguments:

-e
s/^
*//

The first is not a valid substitute command, and the second is unlikely to be a valid file name. Interestingly, you could rescue this by putting double quotes around the $s, as in:

$ s="-e s/^ *//"
$ ls |  sed "$s"

Now sed gets a single argument:

-e s/^ *//

but the -e can have the command attached, and leading spaces on commands are ignored, so this is all valid. You can't do that with your first attempt, though:

$ s="-e 's/^ *//' -e 's/ *$//'"
$ ls | sed "$s"

Now you get told about the ' not being recognized. You could, however, have used:

$ s="-e s/^ *//; s/ *$//"
$ ls | sed "$s"

Again, sed sees a single argument, and there are two semicolon-separated sed commands in the argument to the -e option.

You can ring the variations from here. I find the al command very useful; it quite often helps me understand where something is going wrong.

Source for `al` — argument list

#include <stdio.h>

int main(int argc, char **argv)
{
    while (*++argv)
        puts(*argv);
    return 0;
}

This is one of the smallest useful C programs you can write ('hello world' is one line shorter, but it isn't useful for much beyond demonstrating how to compile and run a program). It lists each of its arguments on a line on its own. You can also simulate it in bash and other related shells with the printf command:

printf "%s\n" "$@"

Wrap it as a function:

al()
{
    printf "%s\n" "$@"
}

Question 2

The sed worked for your normal replace pattern because it did not have any metacharacters. You had just a and b. When there are metacharacters involved, you need single quotes.

I think the only way sed would work properly for your variable assignment case is only by using eval.

invoking sed with a shell variable

Source for al — argument list

Source for `al` — argument list