Em-http flow with basic auth and gzip freezes

Question

I am trying to use the Gnip PowerTrack API, which requires me to connect to an HTTPS JSON stream with basic auth. I feel that this should be pretty trivial, so I hope that some ruby ​​player who is smarter than me can point out my obvious mistake.

Here are the relevant parts of my ruby ​​1.9.3 code:

require 'eventmachine' require 'em-http' require 'json' usage = "#{$0} <user> <password>" abort usage unless user = ARGV.shift abort usage unless password = ARGV.shift GNIP_STREAMING_URL = 'https://stream.gnip.com:443/foo/bar/prod.json' http = EM::HttpRequest.new(GNIP_STREAMING_URL) EventMachine.run do s = http.get(:head => { 'Authorization' => [user, password], 'accept' => 'application/json', 'Accept-Encoding' => 'gzip,deflate' }, :keepalive => true, :connect_timeout => 0, :inactivity_timeout => 0) buffer = "" s.stream do |chunk| buffer << chunk while line = buffer.slice!(/.+\r?\n/) puts JSON.parse(line) end end end 

The stream connects (the My Gnip control panel rewrites the connection), and then simply buffers and never outputs anything. It actually seems like it never enters the s.stream do.. block. Please note that this is a GZip encoded stream.

Please note that this works:

 curl --compressed -uusername $GNIP_STREAMING_URL 

EDIT: I am sure this is implicit, but I cannot give out any login logins or the actual URL, so don't ask;)

EDIT # 2: yajl-ruby will probably work if I can figure out how to encode the credentials for a URL (a simple URL encoding does not work since I fail authentication with Gnip).

EDIT # 3: @rweald discovered that em-http does not support streaming gzip, I created a GitHub problem here.

EDIT # 4: I unblocked and fixed it in em-http-request, you can specify my plug if you want to use em-http this way. The patch has been merged into repeater support and will work in the next release.

EDIT No. 5: My patches are published in em-http-request 1.0.3, so this should no longer be a problem.

Answer 1

The problem is with the em-http-request request. If you look at https://github.com/igrigorik/em-http-request/blob/master/lib/em-http/decoders.rb

You will notice that the GZIP decompressor cannot perform streaming decompression :( https://github.com/igrigorik/em-http-request/blob/master/lib/em-http/decoders.rb#L100

You will need to fix the main problem with gzip streaming if you want to be able to read the stream using em-http-request

Answer 2

I used some code base of this Gist to connect to the Gnip console. https://gist.github.com/1468622

Answer 3

it seems that using https://github.com/brianmario/yajl-ruby could solve this problem

Answer 4

Gnip suggested using curb , and here is what I came up with from their example:

 require 'rubygems' require 'curb' # Usage: <script> username password url # prints data to stdout. usage = "#{$0} <user> <password> <url>" username, password, url = ARGV.first 3 Curl::Easy.http_get url do |c| c.http_auth_types = :basic c.username = username c.password = password c.encoding = 'gzip' c.on_body do |data| puts data data.size # required by curl api. end end 

Although I would like something that will reconnect when the connection is removed and handle the various types of failures gracefully.