How to use subprocess Popen?

Question

I'm trying to execute a command using Popen.

The command uses some PostGIS/Postgresql utility programs to upload a raster file to a database and works when executed from the command line. It uses unix style pipes to chain 2 commands and looks like this:

"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres

When using within Python, I make it a string with the ' codes:

command = '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres'

attempting to execute it results in an error:

p = subprocess.Popen(command)

ERROR: Unable to read raster file: test

The error seems like the command was not parsed correctly (it is interpreting the wrong argument as the raster file)

Am I using Popen wrong?

Answer 1

Your command uses pipe |. It requires a shell:

p = subprocess.Popen(command, shell=True)

The command itself as far as I can tell looks ok.

Answer 2

It's not necessary to use shell=True to achieve this with pipes. This can be done programmatically with pipes even where concern about insecure input is an issue. Here, conn_params is a dictionary with PASSWORD, NAME (database name), USER, and HOST keys.

raster2pgsql_ps = subprocess.Popen([
    'raster2pgsql', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128',
    'C:\\temp\\SampleDTM\\SampleDTM.tif',
    'test'
], stdout=subprocess.PIPE)

# Connection made using conninfo parameters
# http://www.postgresql.org/docs/9.0/static/libpq-connect.html
psql_ps = subprocess.check_output([
    'psql',
    'password={PASSWORD} dbname={NAME} user={USER} host={HOST}'.format(**conn_params),
], stdin=raster2pgsql_ps.stdout)

Answer 3

The following worked for me on Windows, while avoiding shell=True

One can make use of Python's fstring formatting to make sure the commands will work in windows.

Please note that I used shp2pgsql but it should be a very similar process for raster2pgsql.

Parameters for the shp2pgsql: srid is the coordinate system of the shape file, filename is the path to the shape file to be imported, tablename is the name you'd like to give your table.

import os
import subprocess

shp2pgsql_binary = os.path.join(pgsql_dir, "bin", "shp2pgsql")
psql_binary = os.path.join(pgsql_dir, "bin", "psql")

command0 = f'\"{shp2pgsql_binary}\" -s {srid} \"{filename}\" {tablename}'
command1 = f'\"{psql_binary}\" \"dbname={databasename} user={username} password={password} host={hostname}\"'

try:
    shp2pgsql_ps = subprocess.Popen(command0, stdout=subprocess.PIPE)
    psql_ps = subprocess.check_output(command1, stdin=shp2pgsql_ps.stdout)
except:
    sys.stderr.write("An error occurred while importing data into the database, you might want to \
                        check the SQL command below:")
    sys.stderr.write(command)
    raise

To adpat to raster2pgsql, you just need to modify the string in command0, e.g. -s {srid} becomes -d -I -C -e -Y -F -t 128x128. The string for command1 can remain the same.

Answer 4

  PIPE = subprocess.PIPE
  pd = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe", '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', "C:\\temp\\SampleDTM\\SampleDTM.tif", 'test'],
    stdout=PIPE, stderr=PIPE)
  stdout, stderr = pd.communicate()

Answer 5

It will be better to use subprocess.Popen in this way:

proc = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe"', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', '"C:\\temp\\SampleDTM\\SampleDTM.tif"', 'test', '|', '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe"', '-h', 'localhost', '-p', '5432', '-d', 'adr_hazard', '-U', 'postgres'], shell = True, stdout = subprocess.pipe, stderr = subprocess.STDOUT)
proc.wait()
result = proc.stdout.readlines()#if you want to process the result of your command
proc.kill()

B.T.W, it's good to format the path first, use:

path = os.path.normalpath("C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe")

this will avoid some path problems for different OS platform.

The shell = True is important if you want to execute your command just like executing it in local shell.

Hope will help you.