Efficient byte packing into integers?

Question

This is one of those questions that seem easy at first, but I have been doing research for some time and can't find the answer ....

I need to convert a list of bytes (i.e., Word8 s) to a number of arbitrary length (i.e. Integer ). for instance

 intPack::[Word8]->Integer intPack [1] = 1 intPack [1, 0] = 256 showHex (intPack [1,2,3,4,5,6,7,8,9,10,11]) "" = "102030405060708090a0b" 

A slow solution is easy to write (see answers in How to convert ByteString to Int and deal with content? )

 intPack = foldl (\v -> ((v*256) +)) 0 

.... But I compress to it all the extra multiplications and additions, as well as a whole chain of useless integers created in the middle to (possibly) get the same bytes that I started with Integers Type packed in internal structures.

Of course, I don’t know the details about how Integer stores its data (maybe it does something more complicated than keeping bytes in an array of variable length .... for example, use flags to indicate the length of the number, like utf-8 when encoding characters). At least it would be good to know that the above intPack is as good as it gets .... Then I could stop the study, bite (or rather byte :)) the bullet and move on.

Answer 1

I would look at the binary package for efficiently packing and unpacking binary data structures:

https://hackage.haskell.org/package/binary-0.7.2.1/docs/Data-Binary-Get.html

Some ideas:

• See if the Binary Integer instance can work for you:

   import Data.Binary import qualified Data.ByteString.Lazy.Char8 as LBS main = do let i = 0x0102030405060708090a0b0c0d0e0f :: Integer bs = encode i print ("before", i) LBS.writeFile "output" bs j <- fmap decode $ LBS.readFile "output" :: IO Integer print ("after", j) 

• Look at function definitions such as word64be to see if it gives you any ideas:

http://hackage.haskell.org/package/binary-0.7.2.1/docs/src/Data-Binary-Get.html#getWord64be