Haskell Streaming

Question

The two resources I found offered recipes for streaming downloads using the popular Haskell libraries:

• https://haskell-lang.org/library/http-client#Streaming
• http://www.alfredodinapoli.com/posts/2013-07-20-slick-http-download-in-haskell.html

How can I change the code in the first one for (a) saving to a file and (b) printing only a (take 5) byte response, and not the whole response to stdout?

My attempt (b):

#!/usr/bin/env stack {- stack --install-ghc --resolver lts-5.13 runghc --package http-conduit -} {-# LANGUAGE OverloadedStrings #-} import Control.Monad.IO.Class (liftIO) import qualified Data.ByteString as S import qualified Data.Conduit.List as CL import Network.HTTP.Simple import System.IO (stdout) main :: IO () main = httpSink "http://httpbin.org/get" $ \response -> do liftIO $ putStrLn $ "The status code was: " ++ show (getResponseStatusCode response) CL.mapM_ (take 5) (S.hPut stdout) 

What can’t be displayed (take 5) and it offers me, among other things, I still don’t understand how the display works on monads, or liftIO.

In addition, this resource:

http://haskelliseasy.readthedocs.io/en/latest/#note-on-streaming

... gave me a warning: “I know what I'm doing, and I need finer control over resources such as streaming,” which is not easy or even supported.

Other places I watched:

• Download large files from the Internet in Haskell
• https://hackage.haskell.org/package/wreq
• https://hackage.haskell.org/package/pipes-http

If there is something in Haskellverse that makes it easier, more like Python requests:

 response = requests.get(URL, stream=True) for i,chunk in enumerate(response.iter_content(BLOCK)): f.write(chunk) 

I would appreciate a tip, or pointers to the state of the art in 2016.

Answer 1

You are httpSource looking for httpSource from the latest version of http-conduit . This behaves exactly like Python requests: you return a stream of pieces.

save to file

It's easy, just redirect the source directly to the sink of the file.

 #!/usr/bin/env stack {- stack --install-ghc --resolver nightly-2016-11-26 runghc --package http-conduit -} {-# LANGUAGE OverloadedStrings #-} import Network.HTTP.Simple (httpSource, getResponseBody) import Conduit main = runConduitRes $ httpSource "http://httpbin.org/get" getResponseBody .| sinkFile "data_file" 

print only a (take 5) byte response

Once we have the source, we take the first 5 bytes with takeCE 5 , and then print them through printC .

 #!/usr/bin/env stack {- stack --install-ghc --resolver nightly-2016-11-26 runghc --package http-conduit -} {-# LANGUAGE OverloadedStrings #-} import Network.HTTP.Simple (httpSource, getResponseBody) import Data.ByteString (unpack) import Conduit main = runConduitRes $ httpSource "http://httpbin.org/get" getResponseBody .| takeCE 5 .| printC 

save to file and print only a (take 5) byte response

To do this, you want zipSinks or for more general cases involving ZipSink several ZipSink drains:

 #!/usr/bin/env stack {- stack --install-ghc --resolver nightly-2016-11-26 runghc --package http-conduit -} {-# LANGUAGE OverloadedStrings #-} import Network.HTTP.Simple (httpSource, getResponseBody) import Data.ByteString (unpack) import Data.Conduit.Internal (zipSinks) import Conduit main = runConduitRes $ httpSource "http://httpbin.org/get" getResponseBody .| zipSinks (takeCE 5 .| printC) (sinkFile "data_file")