How can I read/write from/to file descriptors in Python3's Popen?

Question

In Popen I can write to stdin (0) and read from stdout (1) and stderr (2).

I'd like to do something like this:

#!/usr/bin/env python3
from subprocess import Popen, PIPE
with Popen(
        [
            'ffmpeg',
            '-f', 'matroska', '-i', 'pipe:0',
            '-f', 'matroska', '-i', 'pipe:3',
            ],
        stdin=PIPE, in_3=PIPE) as p:
    p.stdin.write(b'There is nothing special.')
    p.in_3.write(b'\xf0\x9f\x99\x88\xf0\x9f\x99\x89\xf0\x9f\x99\x8a')

Answer 1

stderr, stdout and stdin are treated specially. When Popen starts a new process it has to create pipes for communicating with the child process for these, and therefor it's not that simple to specify an additional pipe to communicate with the child.

If you need another pipe, you need to set it up before you execute the subprocess.

Here is a simple example to show you how it can be done. It executes a test script which just copies data from a file descriptor given as command line argument (or stdin) to stderr (or stdout):

test.sh:

#!/bin/bash

read_fd="$1"            # get file descriptor from command line

cat                     # write stdin to stdout
cat <&"$read_fd" >&2    # read from read_fd and write to stderr

invoking program: ptest.py:

import os
import subprocess

pipe_rfd, pipe_wfd = os.pipe()

print(pipe_rfd, pipe_wfd)

p = subprocess.Popen(
    ["./test.sh", str(pipe_rfd)],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    close_fds=False,                           # make sure file descriptors are kept open in subprocess
    preexec_fn = lambda: os.close(pipe_wfd)    # make sure write end is closed in child
)

p.stdin.write(b'spam\n')
p.stdin.close()

os.write(pipe_wfd, b'eggs\n')
os.close(pipe_wfd)

print('out:', p.stdout.read())
print('err:', p.stderr.read())

In python2, you need close_fds=False and the preexec_fn to close the write end of the pipe before spawning the child, otherwise the read end won't see an EOF if the write end is closed in the parent. Starting with python3.2, you could instead use the new pass_fds argument to provide a list of file descriptor to keep open, but the code above also works (only tested on linux).

Applied to your problem, the Popen call would then look like:

...
with Popen(
        [
            'ffmpeg',
            '-f', 'matroska', '-i', 'pipe:0',
            '-f', 'matroska', '-i', 'pipe:%d' % pipe_rfd,
        ],
        stdin=PIPE, pass_fds=[pipe_rfd]) as p:
    ...