How can you join two or more dictionaries created by Bio.SeqIO.index?

Question

I would like to be able to join the two "dictionaries" stored in "indata" and "pairdata", but this code,

indata = SeqIO.index(infile, infmt)
pairdata = SeqIO.index(pairfile, infmt)
indata.update(pairdata)

produces the following error:

indata.update(pairdata)
TypeError: update() takes exactly 1 argument (2 given)

I have tried using,

indata = SeqIO.to_dict(SeqIO.parse(infile, infmt))
pairdata = SeqIO.to_dict(SeqIO.parse(pairfile, infmt))
indata.update(pairdata)

which does work, but the resulting dictionaries take up too much memory to be practical for for the sizes of infile and pairfile I have.

The final option I have explored is:

indata = SeqIO.index_db(indexfile, [infile, pairfile], infmt)

which works perfectly, but is very slow. Does anyone know how/whether I can successfully join the two indexes from the first example above?

Answer 1

SeqIO.index returns a read-only dictionary-like object, so update will not work on it (apologies for the confusing error message; I just checked in a fix for that to the main Biopython repository).

The best approach is to either use index_db, which will be slower but only needs to index the file once, or to define a higher level object which acts like a dictionary over your multiple files. Here is a simple example:

from Bio import SeqIO

class MultiIndexDict:
    def __init__(self, *indexes):
        self._indexes = indexes
    def __getitem__(self, key):
        for idx in self._indexes:
            try:
                return idx[key]
            except KeyError:
                pass
        raise KeyError("{0} not found".format(key))

indata = SeqIO.index("f001", "fasta")
pairdata = SeqIO.index("f002", "fasta")
combo = MultiIndexDict(indata, pairdata)

print combo['gi|3318709|pdb|1A91|'].description
print combo['gi|1348917|gb|G26685|G26685'].description
print combo["key_failure"]

Answer 2

In you don't plan to use the index again and memory isn't a limitation (which both appear to be true in your case), you can tell Bio.SeqIO.index_db(...) to use an in memory SQLite3 index with the special index name ":memory:" like so:

indata = SeqIO.index_db(":memory:", [infile, pairfile], infmt)

where infile and pairfile are filenames, and infmt is their format type as defined in Bio.SeqIO (e.g. "fasta").

This is actually a general trick with Python's SQLite3 library. For a small set of files this should be much faster than building the SQLite index on disk.