How to use right subprocess Pclose in Python?

Question

I did a script in Python 3.3 which does queries to a PostgreSQL database. I am using subprocess and Popen, and it works, but I am pretty new doing this and I think I do not have much idea of how it is actually working. My question is:

Do I have to close anything after doing Popen?

I read some forums and tutorials which say yes, but when I do a kill or other command to my query result, obviously, I get messages like, for example, this one:

'list' object has no attribute 'kill'

I tried with signals and killpg, any idea? These are two of my code's queries:

database_list = subprocess.Popen('echo "select datname from pg_database" '
                                 '| psql -t -U %s -h %s template1' %
                                 (bkp_vars['user'], bkp_vars['host']),
                                 shell=True,
                                 stdout=subprocess.PIPE).stdout.readlines()

update_result = subprocess.Popen('echo "UPDATE pg_database SET datallowconn = TRUE WHERE   datname = \'template0\'" '
                                    '| psql -t -U %s -h %s template1' %
                                    (bkp_vars['user'],bkp_vars['host']), shell=True,
                                    stdout=subprocess.PIPE).stdout.readlines()

Thank you in advance!

Answer 1

First, this is a much better job for psycopg2, the Python interface to PostgreSQL, than for use of subprocess and psql.

Furthermore, if you're doing backups, you should be using pg_dump and pg_restore, not messing with psql.

How to get a database list

import psycopg2

# Replace connection parameters with those for your DB; see the psycopg2 docs
conn = psycopg2.connect("dbname=postgres user=myusername password=whatever")

curs = conn.cursor()
curs.execute("select datname from pg_database;")
dbs = curs.fetchall()

print dbs

Better use of subprocess

If you did have to do it with psql, you'd need to realise that Popen returns a subprocess.Popen object.

These represent the child process. You can use them to wait for the child process to finish, to read child process output, etc.

Most of the time, you will instead be using the wrapper interfaces like subprocess.check_call and subprocess.check_output, which automate this and make it a lot easier in exchange for reduced flexibility. In your case, I think you want subprocess.check_output.

subproces can be used to write stdin to an object, by passing a tempfile containing the desired stdin, or by directly writing to the stdin file handle returned by the Popen constructor. You shouldn't need to do hacks like echo stdin into a process opened via the subprocess module.

If you had to do this with psql you'd want something more like:

import subprocess

username = "postgres"
dbname = "postgres"

p = subprocess.Popen(
  args = ['psql', '-qAt', '-0', '-U', username, dbname],
  stdin = subprocess.PIPE,
  stdout = subprocess.PIPE
)
p.stdin.write("SELECT datname FROM pg_database;");
p.stdin.close()
dbs = p.stdout.read().split('\0')
print dbs
p.wait() 

Note the use of the null byte as a record separator, so database names with newlines in them are parsed correctly.