Undefined byte lengths in Python

Question

I am writing a client for a P2P application per minute, and the specification for the protocol says that the header for each packet should have each field with a specific byte length:

Version: 1 Byte Type: 1 Byte Length: 2 Bytes And then the data 

I have a way to pack and unpack the header fields (I think) as follows:

 packed = struct.pack('cch' , '1' , '1' , 26) 

This creates a header for a packet with a data length of 26, but when it comes to unpacking the data, I'm not sure how to do this after receiving the rest of the data. To unpack, we need to know the size of all the fields, if I do not miss something? I think in order to pack the data, I would use the "cch26s" format indicator, meaning:

 1 Byte char 1 Byte char 2 Byte short 26 Byte char array 

But how can I unpack the data when I do not know how much data will be included in the package?

Answer 1

As you describe the protocol, first you need to unpack the first four bytes and extract the length (16-bit int). This tells you how many bytes to unpack in the second step.

 version, type, length = struct.unpack("cch", packed[:4]) content, = struct.unpack("%ds" % length, packed[4:]) 

This is if everything is checked. unpack () requires that the packed buffer contain exactly as much data as you unpacked. Also, check if 4 bytes of the header are included in the length counter.

Answer 2

You can guess the number of characters to unpack by checking len(data) .

Here is a helper function that does this for you:

 def unpack(fmt, astr): """ Return struct.unpack(fmt, astr) with the optional single * in fmt replaced with the appropriate number, given the length of astr. """ # http://stackoverflow.com/a/7867892/190597 try: return struct.unpack(fmt, astr) except struct.error: flen = struct.calcsize(fmt.replace('*', '')) alen = len(astr) idx = fmt.find('*') before_char = fmt[idx-1] n = (alen-flen)/struct.calcsize(before_char)+1 fmt = ''.join((fmt[:idx-1], str(n), before_char, fmt[idx+1:])) return struct.unpack(fmt, astr) 

You can use it as follows:

 unpack('cchs*', data)