Piping data from python to an external command

Question

I've read everything there is on subprocess.Popen but I think I'm missing something.

I need to be able to execute a unix program which reads a data stream from a list created in the python script and write the result of that program to a file. From the bash prompt I do this all the time with no problem but now I am trying to to this from within a python script which preprocesses some binary files and a lot of data before coming to this stage.

Lets look at a simple example not including all the preprocessing:

import sys
from pylab import *
from subprocess import *
from shlex import split

# some arbitrary x,y points
points = [(11,31),(13,33),(15,37),(16,35),(17,38),(18,39.55)]

commandline = 'my_unix_prog option1 option2 .... > outfile'
command = split(commandline)

process = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE)
print process.communicate(str(points))

The way this would be executed in bash is:

echo "11 31
      13 33
      15 37
      16 35
      17 38
      18 39.55" | my_unix_prog option1 option2 .... > outfile

The way the data is fed into the unix prog is important as well, I should be formatted in 2 columns separated by whitespace.

Any help is appreciated...

Answer 1

SOLVED!

With the help of Dhara and xhainingx I was able to figure this out:

import sys
from pylab import *
from subprocess import *
from shlex import split

# some arbitrary x,y points
points = [(11,31),(13,33),(15,37),(16,35),(17,38),(18,39.55)]

commandline = 'my_unix_prog option1 option2 ....'
command = split(commandline)

process = Popen(command, stdin=PIPE, stdout=open('outfile', 'w'), stderr=PIPE)
for p in points:
    process.stdin.write(str(p[0]) + ' ' + str(p[1]) + '\n')

print process.communicate()

This works very well, thanks.

Answer 2

how about something like

for p in points:
    process.stdin.write(str(p[0]) + ' ' + str(p[1]) + '\n')

print process.communicate()

Answer 3

You need to format your input to communicate correctly.

str will keep the special characters when you print your list of tuples which isn't what you want.

>>> print str([(1,2), (3,4)]) 
[(1,2), (3,4)]

Try this:

print process.communicate("\n".join(["%s %s"%(x[0], x[1]) for x in points])