Subprocess fails to catch the standard output

Question

I am trying to generate tree with fasta file input and Alignment with MuscleCommandline

import sys,os, subprocess
from Bio import AlignIO
from Bio.Align.Applications import MuscleCommandline
cline = MuscleCommandline(input="c:\Python26\opuntia.fasta")
child= subprocess.Popen(str(cline),
                         stdout = subprocess.PIPE,
                         stderr=subprocess.PIPE,
                        shell=(sys.platform!="win32"))
align=AlignIO.read(child.stdout,"fasta")
outfile=open('c:\Python26\opuntia.phy','w')
AlignIO.write([align],outfile,'phylip')
outfile.close()

I always encounter with these problems

Traceback (most recent call last):
  File "<string>", line 244, in run_nodebug
  File "C:\Python26\muscleIO.py", line 11, in <module>
    align=AlignIO.read(child.stdout,"fasta")
  File "C:\Python26\Lib\site-packages\Bio\AlignIO\__init__.py", line 423, in read
    raise ValueError("No records found in handle")
ValueError: No records found in handle

Answer 1

Biopython 1.54 was released today with a stable version of the Bio.Phylo module. I've updated the documentation with an example pipeline for generating trees. For simplicity, the example uses ClustalW to align sequences and generate a tree, instead of Muscle and Phylip, but most of the code is still the same or similar.

http://biopython.org/wiki/Phylo#Example_pipeline

If you've already generated a tree with Phylip (using the .phy alignment as input), you can still follow the Phylo examples in general. Phylip creates a Newick file with a name like "outttree" or "foo.tree".

(Feel free to merge this with Brad's answer; I can't write a comment in that thread yet.)

Answer 2

A couple of things are giving problems here:

1. You need a child.wait() after the subprocess call so that your code will wait until the external program is done running.

2. Muscle does not actually write to stdout, even though the help documentation says it does, at least with v3.6 that I have here. I believe the latest is v3.8 so this may be fixed.

Biopython is telling you that the stdout you are passing it is empty, which is the error you are seeing. Try running the commandline directly:

muscle -in opuntia.fasta

and see if you see FASTA output. Here is a version that fixes the wait problem and uses an intermediate output file:

import sys,os, subprocess
from Bio import AlignIO
from Bio.Align.Applications import MuscleCommandline
out_file = "opuntia.aln"
cline = MuscleCommandline(input="opuntia.fasta", out=out_file)
child= subprocess.Popen(str(cline),
                         stdout = subprocess.PIPE,
                         stderr=subprocess.PIPE,
                        shell=(sys.platform!="win32"))
child.wait()
with open(out_file) as align_handle:
    align=AlignIO.read(align_handle,"fasta")
outfile=open('opuntia.phy','w')
AlignIO.write([align],outfile,'phylip')
outfile.close()
os.remove(out_file)

Answer 3

From the documentation of the subproccess library:

Warning

Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

so maybe you could try something like:

mydata = child.communicate()[0]

Answer 4

You have an unprotected backslash in your output filename, that is never good.

Use 'r' to get raw strings, i.e. r'c:\Python26\opuntia.phy'.