os.system complains about round brackets

Question

I am trying to include this line in a python script.

!#/bin/bash/env python

import os

os.system("paste <(awk '!/^($|[:space:]*#)/{print $0}' file1) <(awk '!/^($|[:space:]*#)/{print $0} file2) > out_file")

The command is perfectly fine when run from bash directly. However, inside the script, I get:

sh: -c: line0: syntax error near unexpected token `('

The problem persists when using simply:

os.system("paste <(cat file1) > output_file")

Any ideas?

Answer 1

The command is perfectly fine when run from bash directly. However, inside the script, I get:

sh: -c: line0: syntax error near unexpected token `('

That's because inside the script, you're running the command with sh rather than bash. Both this command, and the simpler one, use bash-specific features. Try running an sh shell and typing the same lines, and you'll get the same error.

The os.system call doesn't document what shell it uses, because it's:

implemented by calling the Standard C function system()

On most Unix-like systems, this calls sh. You probably shouldn't rely on that… but you definitely shouldn't rely on it calling bash!

If you want to run bash commands, use the subprocess module, and run bash explicitly:

subprocess.call(['bash', '-c', 'paste <(cat file1) > output_file'])

You could, I suppose, try to get the quoting right to run bash as a subshell within the shell system uses… but why bother?

This is one of the many reasons that the documentation repeatedly tells you that you should consider using subprocess instead of os.system.

Answer 2

Kill two birds with one awk script:

awk -v DELIM=' ' '!/^($|[[:space:]]*#)/{a[FNR]=a[FNR]DELIM$0}END{for(i=1;i<=FNR;i++)print substr(a[i],2)}' file1 file2

This eliminates the need for process substitution and is therefor sh compliant.