You have all the data upfront in an array in memory whereas BinaryReader
streams the bytes in one at a time from the source which I guess is a file on disk. I guess you could speed it up by passing it a stream that reads from an in-memory array:
Stream stream = new MemoryStream(byteArray);
//Pass the stream to BinaryReader
Note that with this approach you need to fill the entire file in memory at once but I guess that's ok for you.