Most efficient way of loading formatted binary files in Python

Question

I have binary files no larger than 20Mb in size that have a header section and then a data section containing sequences of uchars. I have Numpy, SciPy, etc. and each library has different ways of loading in the data. Any suggestions for the most efficient methods I should use?

Answer 1

struct should work for the header section, while numpy's memmap would be efficient for the data section if you are going to manipulate it in numpy anyways. There's no need to stress out about being inconsistent here. Both methods are compatible, just use the right tool for each job.

Answer 2

Use the struct module, or possibly a custom module written in C if performance is critical.

Answer 3

bdec seems promising.

Answer 4

I found that array.fromfile is the fastest methods for homogeneous data.