How to make haskell not store whole bytes?

Question

I am writing a small (relatively) haskell app for academic purposes. I implement Huffman compression based on this code http://www.haskell.org/haskellwiki/Toy_compression_implementations .

My version of this code is here https://github.com/kravitz/har/blob/a5d221f227c27fd1c5587217a29a169a377521a6/huffman.hs and it uses lazy bytes. When I implemented RLE compression, everything was smooth, because it processes the input stream in one step. But Huffman processes it twice, and as a result, I have a bytestring score stored in memory, which is bad for large files (but for relatively small files, it also allocates too much space on the heap). This is not only my suspicion, because profiling also shows that most of the heap is eaten by distributing strings.

I also serialize the length of the stream in the file, and can also cause a complete load of bytes in memory. Is there an easy way to tell ghc to be nice and re-evaluate the stream several times?

Answer 1

Instead of passing the bytestring to the encoder, you can pass something that the bytestring calculates, and then explicitly recalculate the value each time you need it.

compress :: ST s ByteString -> ST s ByteString
compress makeInput = do
  len      <- (return $!) . ByteString.length =<< makeInput
  codebook <- (return $!) . makeCodebook      =<< makeInput
  return . encode len codebook                =<< makeInput

compressIO :: IO ByteString -> IO ByteString
compressIO m = stToIO (compress (unsafeIOToST m))

The parameter compressshould actually calculate the value. Simply wrapping the value with returnwill not work. In addition, each call makeInputshould actually evaluate its result, otherwise, in the process of recounting the input, there will be a lazy, not evaluated copy of the memory entry.

The usual approach, as Barsoap said, is to simply compress one block at a time.

Answer 2

, (Huffmann-) , , , , , , . , , , .

, bytestring-mmap, , , , mmap.

( , , ) , , , 1TB .