Fast casting in C# using BitConverter, can it be any faster?

Question

In our application, we have a very large byte-array and we have to convert these bytes into different types. Currently, we use BitConverter.ToXXXX() for this purpose. Our heavy hitters are, ToInt16 and ToUInt64.

For UInt64, our problem is that the data stream has actually 6-bytes of data to represent a large integer. Since there is no native function to convert 6-bytes of data to UInt64, we do:

UInt64 value = BitConverter.ToUInt64() & 0x0000ffffffffffff;

Our use of ToInt16 is simpler, do don't have to do any bit manipulation.

We do so many of these 2 operations that I wanted to ask the SO community whether there's a faster way to do these conversions. Right now, approximately 20% of our entire CPU cycles is consumed by these two functions.