Queues and Threading: subprocess goes missing

Question

Morning,

I am coding a python daemon that needs to convert videos to .mp4 format. To do so, I was planning to use Handbrake via Subprocess, but i'm getting mixed results: sometimes it works, sometimes the process doesn't even show up on top.

I am not sure what is going on. I've tried some variations, like using Shell=True for instance, but the issue remains.

Thanks,

#! /usr/bin/python
# -*- coding: utf-8 -*- 
import os, time, threading, psutil, resource, logging, subprocess as sp, sys
from Queue import Queue
from threading import Thread

SERVER_LOG=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'convertCentral.log')
logging.basicConfig(format='[%(asctime)s.%(msecs).03d] %(message)s', datefmt='%Y-%m-%d %H:%M:%S', filename=SERVER_LOG, level=logging.INFO)

class QueueServer(object):

    current_video_queue = Queue(maxsize=0)
    N_WORKER_THREADS = 1
    counter = 0


    def __init__(self):
        print("[QueueServer] Initializing the video conversion queue")
        t = threading.Thread(target=self.monitor)
        t.start()


    ''' Alters the process' niceness in order to throtle the CPU usage '''
    def preexec_fn(self):
        pid = os.getpid()
        ps = psutil.Process(pid)
        ps.set_nice(10)
        resource.setrlimit(resource.RLIMIT_CPU, (1, 1))


    ''' Converts the video using Handbrake via subprocess'''
    def convertVideo(self, video):
        print("Now converting %s" % video)
        fileName, fileExtension = os.path.splitext(video)
        payload = "ulimit -t 360; nice -n 15 HandBrakeCLI -i %s -e x264 -q 15 -o %s.mp4" % (video, fileName)
        payload = payload.split(" ")

        # Fire in the hole
        pr = sp.Popen(payload, stdout=open('/dev/null', 'w'), stderr=sp.STDOUT)
        print("Fired.")
        pr.wait()

        self.counter = self.counter + 1
        print("Conversion's done. %d" % self.counter)


    ''' A worker thread '''
    def worker(self):
        while True:
            print("Getting one")
            item = self.current_video_queue.get()
            print("Firing conversion: %s" % item)
            self.convertVideo(item)
            self.current_video_queue.task_done()
            print("All done")


    def monitor(self):
        for i in range(self.N_WORKER_THREADS):
            print("Firing thread")
            t = Thread(target=self.worker)
            t.daemon = True
            t.start()


    ''' Adds a video to the video conversion queue '''
    def add(self, video):
        print("* Adding %s  to the queue" % video)
        self.current_video_queue.put(video)
        print("* Added %s  to the queue" % video)


q = QueueServer()
q.add('UNKNOWN_PARAMETER_VALUE.WMV')
#time.sleep(500)

Here's what I'm getting on the logs:

Hal@ubuntu:~/Desktop/$ python threadedqueue.py
[QueueServer] Initializing the video conversion queue
Firing thread
* Adding UNKNOWN_PARAMETER_VALUE.WMV  to the queue
* Added UNKNOWN_PARAMETER_VALUE.WMV  to the queue
Getting one
Firing conversion: UNKNOWN_PARAMETER_VALUE.WMV
Now converting UNKNOWN_PARAMETER_VALUE.WMV

So, we can tell that the subprocess actually runs, but it just dies there mysteriously... Any ideas what could be causing this?

Answer 1

First of all, the payload you're giving is probably not what you intended.

In normal operation, subprocess is not giving your command to a shell, but instead starts the process which name is at args[0], and pass all the other arguments directly to it. What you're doing, is to pass the following arguments:

["-t", "360;", "nice", "-n", "15", "HandBrakeCLI", "-i", "someInput", "-e", "x264", "-q", "15", "-o", "someOutput.mp4"]

... to a very confused ulimit process. Instead, what you want is to use shell=True, and provide args as a string instead. This tells Popen to, instead of to actually start the process you're requesting, to start a shell and dump it all as a one-liner.

payload = "ulimit -t 360; nice -n 15 HandBrakeCLI -i %s -e x264 -q 15 -o %s.mp4" % (video, fileName)
pr = sp.Popen(payload, shell=True, stdout=sp.DEVNULL, stderr=sp.STDOUT)
print('Started handbrake')
pr.wait()

If you don't get the "Started handbrake" printed, expect that something went horribly wrong at the Popen. A good way to debug that, is to take the payload, import subprocess and try to open it with Popen at an interactive console. If you're locked up inside the Popen call, a ctrl-c should give you a trace to where inside it you're hanging, which might in turn provide you some assistance. Even if it makes no sense to you, it might help others tell what's going on.

I hope that this is helpful!

Answer 2

You could emulate ulimit -t, nice -n in Python, to run HandBrakeCLI without the shell:

#!/usr/bin/env python
import os
import resource
import shlex
from subprocess import call, STDOUT

DEVNULL = open(os.devnull, 'wb', 0)

# quote filenames to allows names with spaces
path, output_path = map(shlex.quote, [video, filename + '.mp4'])
cmd = "HandBrakeCLI -i {path} -e x264 -q 15 -o {output_path}".format(**vars())

def limit_resources():
    resource.setrlimit(resource.RLIMIT_CPU, (360, 360)) # emulate ulimit -t
    os.nice(15) # emulate nice -n

rc = call(shlex.split(cmd), stdout=DEVNULL, stderr=STDOUT,
          preexec_fn=limit_resources)