How to quickly load web pages in ruby? Parallelizing download?

Question

I need to clear (using scrAPI) 400+ ruby ​​web pages, my actual code is very consistent:

data = urls.map {|url| scraper.scrape url }

Actually, the code is a little different (exception handling, etc.).

How can I do it faster? How can I parallelize downloads?

Answer 1

th = []
data = []
dlock = Mutex.new

urls.each do |url|
  th << Thread.new(url) do |url|
    d = scraper.scrape url
    dlock.synchronize { data << d }
  end
end

th.each { |t| t.join }

TA-dah! (A warning written from memory, not verified, may eat your kitten, etc.)

Edit: I decided that someone should have written a generalized version of this, and so they have: http://peach.rubyforge.org/ - enjoy!

Answer 2

, Pickaxe :

http://www.rubycentral.com/pickaxe/tut_threads.html

Pickaxe , .