BitConverter.ToInt32
isn't very fast in the first place. I'd simply use
public static int ToInt32BigEndian(byte[] buf, int i)
{
return (buf[i]<<24) | (buf[i+1]<<16) | (buf[i+2]<<8) | buf[i+3];
}
You could also consider reading more than 4 bytes at a time.