The fastest way to unpack bit (sub-byte) numbers from a file

Question

Given a binary compressed file with resolution, I would like to convert the sub-byte bits to their integer representations in python. By this, I mean that I need to interpret bits of n from a file as a whole.

I am currently reading a file into bitarray objects and converting subsets of objects to integers. The process works, but rather slow and cumbersome. Is there a better way to do this, perhaps with a struct module?

 import bitarray bits = bitarray.bitarray() with open('/dir/to/any/file.dat','r') as f: bits.fromfile(f,2) # read 2 bytes into the bitarray ## bits 0:4 represent a field field1 = int(bits[0:4].to01(), 2) # Converts to a string of 0s and 1s, then int()s the string ## bits 5:7 represent a field field2 = int(bits[4:7].to01(), 2) ## bits 8:16 represent a field field3 = int(bits[7:16].to01(), 2) print """All bits: {bits}\n\tfield1: {b1}={field1}\n\tfield2: {b2}={field2}\n\tfield3: {b3}={field3}""".format( bits=bits, b1=bits[0:4].to01(), field1=field1, b2=bits[4:7].to01(), field2=field2, b3=bits[7:16].to01(), field3=field3) 

Outputs:

 All bits: bitarray('0000100110000000') field1: 0000=0 field2: 100=4 field3: 110000000=384 

Answer 1

This should work for your specific case:

 #bitmasks of fields 1-3, they fit in 2 bytes FIELD1 = 0b1111000000000000 # first 4 bits FIELD2 = 0b0000111000000000 # next 3 bits FIELD3 = 0b0000000111111111 # last 9 bits def bytes_to_int(num): #convert bytes object to an int res = 0 num = num[::-1] # reverse the bytes for i in range(len(num)): res += num[i] * (256**i) return res def get_fields(f): chunk = bytes_to_int(f.read(2)) # read 2 bytes, f1-f3, convert to int f1 = (chunk & FIELD1) >> 12 # get each field with its bitmask f2 = (chunk & FIELD2) >> 9 f3 = chunk & FIELD3 f4 = f.read(f3) # field4 as a bytes object f4 = bytes_to_int(f4) #comment this line if you want field4 as a bytes obj return f1, f2, f3, f4 file = open('file.dat','rb') #using your sample data print(get_fields(file)) # returns 0, 4, 384, field4 as an int file.close() 

Perhaps the struct module was used, but it could not decompress more than 8 bytes into an int, which will be needed for your field 4, whose length is 384 bytes.

Answer 2

If you like to use some kind of module, it looks like the bit string module has a good representation and bit manipulation: http://pythonhosted.org/bitstring/index.html

For example, if you know the size of your fields, you can use format strings: http://pythonhosted.org/bitstring/reading.html#reading-using-format-strings

 import bitstring bitstream = bitstring.ConstBitStream(filename='testfile.bin') field1, field2, field3 = bitstream.readlist('int:4, int:3, int:9') 

If you do not know the size of the fields that you could read in all bits, then use slicing to extract all of your fields: http://pythonhosted.org/bitstring/slicing.html

 import bitstring bitstream = bitstring.ConstBitStream(filename='testfile.bin') bits = bitstring.BitArray(bitstream) field1 = bits[0:4].int field2 = bits[4:7].int field3 = bits[7:16].int 

Just a thought, you probably already found this module.