Bash doesn't think string is properly quoted?

Question

I'm attempting to execute a command over SSH, but bash on the other end doesn't think it's escaped properly.

Here, self._client is a paramiko.SSHClient object; args is a list of arguments, the command to execute.

def run(self, args, stdin=None, capture_stdout=False):
    """Runs a command.

    On success, returns the output, if requested, or None.

    On failure, raises CommandError, with stderr and, if captured, stdout,
    as well as the exit code.
    """
    command = ' '.join(_shell_escape(arg) for arg in args)
    print('About to run command:\n  {}'.format(command))
    print('About to run command:\n  {!r}'.format(command))
    channel = self._client.get_transport().open_session()
    channel.exec_command(command)

_shell_escape:

_SHELL_SAFE = _re.compile(r'^[-A-Za-z0-9_./]+$')
def _shell_escape(s):
    if _SHELL_SAFE.match(s):
        return s
    return '\'{}\''.format(s.replace('\'', '\'\\\'\''))

I'm attempt to run some Python through this. On stderr, I get back:

bash: -c: line 5: unexpected EOF while looking for matching `''
bash: -c: line 6: syntax error: unexpected end of file

The output from the two print statements:

About to run command:
  python -c 'import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
    path = path.decode('\''utf-8'\'')
entries = os.listdir(path)
out = b'\'''\''.join(e.encode('\''utf-8'\'') + b'\'''\'' for e in entries)
sys.stdout.write(out)
' .
About to run command:
  "python -c 'import os, sys\npath = sys.argv[1]\nif sys.version_info.major == 2:\n    path = path.decode('\\''utf-8'\\'')\nentries = os.listdir(path)\nout = b'\\'''\\''.join(e.encode('\\''utf-8'\\'') + b'\\''\x00'\\'' for e in entries)\nsys.stdout.write(out)\n' ."

If I copy and paste the output of command, and paste it into bash, it executes, so it really does appear to be properly escaped. My current understanding is that SSH, on the other end, will take command, and run [my_shell, '-c', command].

Why is bash erroring on that command?

Answer 1

The input contains an embedded nul character, which bash appears to treat as the end of the string. (I'm not sure there's any way it couldn't!). This is visible in my question, where I output command:

About to run command:
  "python -c 'import os, sys [SNIP…] + b'\\''\x00'\\'' for [SNIP…]"

That's a repr output, but notice the single slash before the x in \x00: that's an actual \x00 that made it through. My original code has this Python embedded as a snippet, which I didn't include (I didn't believe it was relevant):

_LS_CODE = """\
import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
    path = path.decode('utf-8')
entries = os.listdir(path)
out = b''.join(e.encode('utf-8') + b'\x00' for e in entries)
sys.stdout.write(out)
"""

Here, Python's """ is still processing \ as an escape character. I need to double up, or look into raw strings (r""")

Answer 2

You need to escape newlines as well. A better option is to put the program text in a here document.

Make the output of "About to run command:" to look like

python -c << EOF
import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
    path = path.decode('\''utf-8'\'')
entries = os.listdir(path)
out = b'\'''\''.join(e.encode('\''utf-8'\'') + b'\'''\'' for e in entries)
sys.stdout.write(out)
.
EOF

Maybe you wouldn't need to escape anything at all.