running ffmpeg from Popen inside (twisted) timer.LoopingCall() stalls

Question

I have an RTSP stream which i need to re-stream as HLS. When RTSP stream goes down (e.g. camera disconnects) I put a blue screen to let the user know that the camera went offline. HLS segmenter is running separately, listening on port 22200 for incoming packets.

in python the code essentially boils down to this:

import psutil, subprocess as sb
from twisted.internet import reactor, task
from cameraControls import camStatus, camURL
ffOn = False
psRef = False
def monitor():
 print "TIMER TICK"
 if camStatus()=='ON' and not ffOn: #camera just came online
  cmd = ["ffmpeg", "-i", camURL, "-codec", "copy", "-f", "mpegts", "udp://127.0.0.1:22200"]
  ps = sb.Popen(cmd,stderr=sb.PIPE)
  psRef=ps
 #check the stream:
 psmon = psutil.Process(psRef.pid)
 if psmon.status!=psutil.STATUS_RUNNING: 
  print "FFMPEG STOPPED"

tmr = task.LoopingCall(monitor)
tmr.start(2)
reactor.run()

it works fine for 5-6 minutes, then i see the video stall and if i check the cpu usage of the ffmpeg it shows 0, the ffmpeg output doesn't change, as if paused. however psmon.status shows as running, and the timer is still going (i see "TIMER TICK" message pop up every 2 seconds in the command line.

if i simply run the ffmpeg command from the command line (not from python) then it works for hours no problem.

does anybody know if the twisted reactor is stalling the process? or is it the subprocess.Popen itself issue? or the timer itself is glitching somehow(even though it gets to the 'monitor' function)? i have other timers running also in the same reactor (same thread), could that be an issue?

Answer 1

Found the problem: the issue is stderr=subprocess.PIPE. When ffmpeg runs for a few minutes, it fills up the pipe, and since there's no ps.communicate() to clear it, eventually the process just stalls. 2 simplest ways to solve this:

read the output from the pipe:

ps = subprocess.Popen(cmd,stderr=sb.PIPE)    
ps.communicate()

or send all output to /dev/null:

devnul = open(os.devnull,"w") #this is multiplatform, rather than hardocding "/dev/null"
ps = sb.Popen(cmd,stderr=devnul)
#....
#some other code here, eventually kill the process maybe
#....
devnul.close()

(assuming subprocess was imported as sb)