How to shift bits in a 2-5 byte long bytes object in python?

Question

I am trying to extract data out of a byte object. For example: From b'\x93\x4c\x00' my integer hides from bit 8 to 21. I tried to do bytes >> 3 but that isn't possible with more than one byte. I also tried to solve this with struct but the byte object must have a specific length.

How can I shift the bits to the right?

Answer 1

Don't use bytes to represent integer values; if you need bits, convert to an int:

value = int.from_bytes(your_bytes_value, byteorder='big')
bits_21_to_8 = (value & 0x1fffff) >> 8

where the 0x1fffff mask could also be calculated with:

mask = 2 ** 21 - 1

Demo:

>>> your_bytes_value = b'\x93\x4c\x00'
>>> value = int.from_bytes(your_bytes_value, byteorder='big')
>>> (value & 0x1fffff) >> 8
4940

You can then move back to bytes with the int.to_bytes() method:

>>> ((value & 0x1fffff) >> 8).to_bytes(2, byteorder='big')
b'\x13L'

Answer 2

As you have a bytes string and you want to strip the right-most eight bits (i.e. one byte), you can simply it from the bytes string:

>>> b'\x93\x4c\x00'[:-1]
b'\x93L'

If you want to convert that then to an integer, you can use Python’s struct to unpack it. As you correctly said, you need a fixed size to use structs, so you can just pad the bytes string to add as many zeros as you need:

>>> data = b'\x93\x4c\x00'
>>> data[:-1]
b'\x93L'
>>> data[:-1].rjust(4, b'\x00')
b'\x00\x00\x93L'
>>> struct.unpack('>L', data[:-1].rjust(4, b'\x00'))[0]
37708

Of course, you can also convert it first, and then shift off the 8 bits from the resulting integer:

>>> struct.unpack('>Q', data.rjust(8, b'\x00'))[0] >> 8
37708

If you want to make sure that you don’t actually interpret more than those 13 bits (bits 8 to 21), you have to apply the bit mask 0x1FFF of course:

>>> 37708 & 0x1FFF
4940

(If you need big-endianness instead, just use <L or <Q respectively.)

---

If you are really counting the bits from left to right (which would be unusual but okay), then you can use that padding technique too:

>>> struct.unpack('>Q', data.ljust(8, b'\x00'))[0] >> 43
1206656

Note that we’re adding the padding to the other side, and are shifting it by 43 bits (your 3 bits plus 5 bytes for the padded data we won’t need to look at)

Answer 3

Another approach that works for arbitrarily long byte sequences is to use the bitstring library which allows for bitwise operations on bitstrings e.g.

>>> import bitstring
>>> bitstring.BitArray(bytes=b'\x93\x4c\x00') >> 3
BitArray('0x126980')

Answer 4

You could convert your bytes to an integer then multiply or divide by powers of two to accomplish the shifting