Python subprocess.popen and rdiff-backup

Question

I want to create a rdiff-backup wrapper program in python for backing up windows machines to a linux server.

I want to process the output of rdiff-backup in the wrapper program. But when executing rdiff-backup with the subprocess module and piping stdout and stderr to the wrapper, the stdout always ends up a the end of the pipe.

When using subprocess without piping the stdout and the stderr gets printed in the right order in the console.

What I also noticed when using rdiff-backup locally and not using a ssh pipe, stdout and stderr are in the right order. Rdiff-backup is also using subprocess.popen to open an ssh session and piping the data to server. I suspect that for some reason stdout get's blocked until the ssh session closes.

Here is my code, this a simplified version the real program uses threads to listen to stdout:

import sys
import subprocess

class Rdiffbackup(object):
    def __init__(self):
        #self.io_q = Queue()
        self.exe = 'F:\\workspace\\pysubprocess\\bin\\rdiff-backup\\rdiff-backup.exe'

        self.verbosity = '-v5'

        self.ssh_exe = './bin/openssh/bin/ssh'
        self.quiet = '-q'
        self.compression = '-C'
        self.port = '-p 5555'
        self.key = '-i ./keys/rdiffbackup'
        self.options = '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
        self.remote_schema = self.ssh_exe +  ' ' + self.quiet + ' ' + self.compression + ' ' + self.port + ' ' + self.key + ' ' + self.options + ' %s rdiff-backup --server'  

    def start(self,source,dest):
        with subprocess.Popen([self.exe,self.verbosity,'--remote-schema',self.remote_schema,source,dest],stdout=subprocess.PIPE,stderr=subprocess.STDOUT) as self.proc:     
            for line in self.proc.stdout:
                sys.stdout.write(line.decode("utf-8"))      

if __name__ == '__main__':
    rdb = Rdiffbackup()
    source = "C:/Users/vdrmrt/Desktop/data"    
    dest = "vdrmrt@hostname::backup"
    rdb.start(source,dest)

The output:

-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         Off
  Extended attributes                          Off
  Windows access control lists                 On
  Case sensitivity                             Off
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           Off
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Symlink permissions                          Off
  Extended filenames                           On
  Windows reserved filenames                   Off
  Access control lists                         On
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup

The correct output:

Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         Off
  Extended attributes                          Off
  Windows access control lists                 On
  Case sensitivity                             Off
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           Off
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Symlink permissions                          Off
  Extended filenames                           On
  Windows reserved filenames                   Off
  Access control lists                         On
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup

Answer 1

I finally was able to solve the problem.
Rdiff-backup buffered the stdout and stderr when it's was piped to an other program. The solution was to rebuild rdiff-backup with an extra py2exe option: 'unbuffered': True
I also had to add an extra option to make it work when building with windows 7.
Py2exe - win32api.pyc ImportError DLL load failed

Here's a snippet from the final setup.py from the rdiff-backup build.

if '--single-file' in sys.argv[1:]:
            sys.argv.remove('--single-file')
            extra_options.update({
                'options': {'py2exe': {'bundle_files': 1,
                                       'unbuffered': True,
                                       'dll_excludes': [ "mswsock.dll", "powrprof.dll" ]}},
                'zipfile': None
            })

Answer 2

Try using separate pipes for stdout and stderr. Read the stdout lines first, then the ones from stderr.