Confusing on subprocess.Popen

Question

This problem makes me confused

I just want to run 1 command on 18 different input file so I've wrote it like

while filenames or running:
  while filenames and len(running) < N_CORES:
    filename = filenames.pop(0)
    print 'Submiting process for %s' % filename
    cmd = COMMAND % dict(filename=filename, localdir=localdir)
    p = subprocess.Popen(cmd, shell=True)
    print 'running:', cmd
    running.append((cmd, p))

  i = 0
  while i < len(running):
    (cmd, p) = running[i]
    ret = p.poll()
    if ret is not None:
        rep = open('Crux.report.%d' % (report_number), 'w')
        rep.write('Command: %s' % cmd)
        print localdir
        print 'done!' 
        report_number += 1
        running.remove((cmd, p))
    else:
        i += 1
  time.sleep(1)

But when I've run it after 3 hours all of the process going to Sleep mode.

But if I call the command from terminal manually (for all of the different files), all of them have been Ok.

Any help would be appreciate.

Answer 1

I assume you want to run 18 processes (one process per file) with no more than N_CORES processes in parallel.

The simplest way could be to use multiprocessing.Pool here:

import multiprocessing as mp
import subprocess

def process_file(filename):
    try:
        return filename, subprocess.call([cmd, filename], cwd=localdir)
    except OSError:
        return filename, None # failed to start subprocess

if __name__ == "__main__":
    pool = mp.Pool()
    for result in pool.imap_unordered(process_file, filenames):
        # report result here

Answer 2

Whithout knowing what your subprocesses are supposed to do and how long they are supposed to be running it's hard to give an accurate answer here.

Some problems I see with your program:

• you check for i < len(running), while incrementing i and removing from running.
  Either use a counter or check if the list still contains elements, but don't do both at the same time. This way you will break out of the loop halfway.
• you increment i each time a process has not finished, you probably want to increment if a process has finished.