Python Subprocess Command as List Not String

Question

I need to use the subprocess module in Python to create some new files by redirecting stdout. I don't want to use shell=True because of the security vulnerabilities.

I wrote some test commands to figure this out, and I found that this worked:

import subprocess as sp
filer = open("testFile.txt", 'w')
sp.call(["ls", "-lh"], stdout=filer)
filer.close()

However, when I passed the command as one long string instead of a list, it couldn't find the file. So when I wrote this:

import subprocess as sp
filer = open("testFile.txt", 'w')
sp.call("ls -lh", stdout=filer)
filer.close()

I received this error:

Traceback (most recent call last):
  File "./testSubprocess.py", line 16, in <module>
    sp.call(command2, stdout=filer)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 524, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Why does it matter if I pass the arguments as a string or as a list?

Answer 1

It is because of the way how the call happens:

With shell=True, the call is performed via the shell, and the command is given to the shell as one string.

With shell=False, the call is performed directly, via execv() and related functions. These functions expet an array of arguments.

If you only pass one string, it is treated as an abbreviation for a call with only the executable's name without arguments. But there is (probably) no executable called ls -lh on your system.

To be exact, somewhere deep inside subprocess.py, the following happens:

        if isinstance(args, types.StringTypes):
            args = [args]
        else:
            args = list(args)

So every string passed is turned into a list with one element.

        if shell:
            args = ["/bin/sh", "-c"] + args

This one I didn't know: obviously, this allows for passing additional arguments to the called shell. Although it is documented this way, don't use it as it is subject to create too much confusion.

If shell=False, we have further down below

if env is None:
    os.execvp(executable, args)
else:
    os.execvpe(executable, args, env)

which just takes a list and uses it for the call.

Answer 2

If you want your strings to be divided like they would at a shell, use shlex:

import subprocess as sp
import shlex
with open("testFile.txt", 'w') as filer:
    sp.call(shlex.split("ls -lh"), stdout=filer)

BTW, let me make the case for check_call while I'm here. Without it, you'd get empty output if you added an invalid argument, for example. You'd be left wondering why the output at filer is empty.

with open("testFile.txt", 'w') as filer:
    sp.check_call(shlex.split("ls -lh0"), stdout=filer)

With check_call you get an error that localizes the problem and prevents subsequent code from executing:

Traceback (most recent call last):
  File "go.py", line 6, in <module>
    sp.check_call(shlex.split("ls -lh0"), stdout=filer)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ls', '-lh0']' returned non-zero exit status 2

Answer 3

It is because the argument is the interpreted as the executable name. It is the same if you put "ls -lh" into shell.

luk32:~/projects/tests$ "ls -lh"
bash: ls -lh: command not found

There is an utility for this called shlex.split.

>>> import shlex, subprocess
>>> command_line = raw_input()
/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
>>> args = shlex.split(command_line)
>>> print args
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
>>> p = subprocess.Popen(args)

But I don't think you need it. Just assume that going with list is the proper way, and the tool is to support transition from the discouraged shell=True mode.

Answer 4

According to the comments in subprocess.py:

On UNIX, with shell=False (default): In this case, the Popen class uses os.execvp() to execute the child program. args should normally be a sequence. A string will be treated as a sequence with the string as the only item (the program to execute).

On UNIX, with shell=True: If args is a string, it specifies the command string to execute through the shell. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments.

On Windows: the Popen class uses CreateProcess() to execute the child program, which operates on strings. If args is a sequence, it will be converted to a string using the list2cmdline method. Please note that not all MS Windows applications interpret the command line the same way: The list2cmdline is designed for applications using the same rules as the MS C runtime.

In UNIX subprocess.call('ls -l') will fail while in Windows it will succeed. It is os.execvp() which is causing the issue. The problem is that the whole string is passed as an argument. If you execute subprocess.call('free') it will succeed in UNIX.