Using rest-client to download a file to disk without first loading it into memory

Question

I use rest-client to load a large page (about 1.5 GB in size). The checked value is stored in memory, and not stored in a file. As a result, my program crashes with failed to allocate memory (NoMemoryError) .

But there is no need to store this data in memory, it can even be saved directly to disk.

I found "You can: (...) manually process the response (for example, work with it as a stream, and not read it all in memory). For more details, see the RestClient :: Request documentation." at https://github.com/rest-client/rest-client Unfortunately, after reading http://www.rubydoc.info/gems/rest-client/1.7.3/RestClient/Request I have no idea how it can be achieved.

I also know that I can use another library ( Using WWW: Mechanize to load a file to disk without first loading it into memory ), but my program is already using rest-client.

Simplified code:

 data = RestClient::Request.execute(:method => :get, :url => url, :timeout => 3600) file = File.new(filename, 'w') file.write data file.close 

Code - https://github.com/mkoniecz/CartoCSSHelper/blob/395deab626209bcdafd675c2d8e08d0e3dd0c7f9/downloader.rb#L126

Answer 1

Another way is to use raw_response . This is saved directly to the file, usually in /tmp . This allows you to redirect without problems. See Streaming Responses . Here is an example of them:

 >> raw = RestClient::Request.execute( method: :get, url: 'http://releases.ubuntu.com/16.04.2/ubuntu-16.04.2-desktop-amd64.iso', raw_response: true) => <RestClient::RawResponse @code=200, @file=#<Tempfile:/tmp/rest-client.20170522-5346-1pptjm1>, @request=<RestClient::Request @method="get", @url="http://releases.ubuntu.com/16.04.2/ubuntu-16.04.2-desktop-amd64.iso">> >> raw.file.size => 1554186240 >> raw.file.path => "/tmp/rest-client.20170522-5346-1pptjm1" 

Answer 2

My original answer contributed to passing the block to RestClient::Request#execute , but it only passed data to the block after the full answer was read. Thus, exercise is futile. Here's how to do it:

 File.open('/tmp/foo.iso', 'w') {|f| block = proc { |response| response.read_body do |chunk| puts "Working on response" f.write chunk end } RestClient::Request.new(method: :get, url: 'http://mirror.pnl.gov/releases/xenial/ubuntu-16.04-server-amd64.iso', block_response: block).execute } 

This is due to the problem of the client-client project .

Note. redirection does not work in this mode, and also you lose HTTP exit status, cookies, headers, etc. Hope this will be fixed some day.