Python how to direct subprocess to output file

Question

I have created a script that receives a bunch of log files as input in order to do some pattern matching. However, my "processFiles" method is not working properly. It should write all data to "fileDest". But the created file remains empty. If I put a "print processFiles" statement under the function itself I can see lines as throughput on my terminal.

To make matters more interesting, bunzip2 reports this error while I run the script:

---

Proccessing /opt/syslog/app/applog-20140314.bz2. bunzip2: Can't open input file > : No such file or directory. bunzip2: Compressed file ends unexpectedly; perhaps it is corrupted? *Possible reason follows. bunzip2: No such file or directory Input file = /var/tmp/parsed_applog-20140314.decompressed, output file = (stdout)

---

It seems something in my code sends the output to stdout. And not to the file.

def parse_log_files(self):
    sub_dir = os.listdir(self.base_path)
    for directory in sub_dir:
        if re.search('app\d+', directory):
            fileInput = self.base_path + '/' + directory + '/applog-' + str(self.date.strftime('%Y%m%d')) + '.bz2'
            fileDest = '/var/tmp/parsed_log_files-'  + str(self.date.strftime('%Y%m%d')) + '.decompressed'
            if not os.path.isfile(fileDest):
                subprocess.Popen(['touch',fileDest]).communicate()[0]
            proccessFiles = subprocess.Popen(['/bin/bunzip2','-cd',fileInput,' > ',fileDest],stdout=subprocess.PIPE).communicate()[0]
            accessFileHandle =  open(self.file_out, 'r')
            readFileHandle = accessFileHandle.readlines()
            print "Proccessing %s." % fileInput

Answer 1

' > ' is a shell redirection syntax. Popen doesn't spawn the shell unless you ask (you shouldn't). If you want to redirect the output of a subprocess to a file then use stdout=file_object parameter e.g.:

from subprocess import check_call

with open('/path/to/output', 'wb', 0) as output_file:
    check_call(['command', 'arg1', 'arg2'], stdout=output_file)