Ruby Open-URI with dynamic website

Question

I am trying to use open-uri to get an html page for a website. However, the problem is that the website takes a few seconds to load for the correct correct code. Now I have:

require 'open-uri' html = open('http://hiddencode.me/dribbbucket/embed.html?key=MY_API_KEY&bucket=56024-Glassboard&delay=5000') response = html.read puts response 

If I run this right now, I get:

 <div id="slam-dunk"> <div id="loading">Loading..</div> </div> 

However, the site must load correctly first before opening in order to get the correct answer. Any ideas how to do this in ruby? I can also use the solution in another language if ruby ​​is not your experience!

Answer 1

As an example, I recently used watir-webdriver to perform a similar task. You will be able to request the DOM after javascript is executed and pull out whatever you want. If you want it to be headless, in my case I used a headless pearl.

If you want to stick with "open-uri" you will need to use something like httpfox to see which ajax javascript requests. You can do this with many other tools. But you must start httpfox, in this case, before you visit the URL. Wait until you see the information you are trying to clear, then stop httpfox and go through each request, checking each answer for things related to what you are clearing. Once you determine the correct request, you can use it with open-uri. Being the simplest, this solution is not guaranteed, since web applications vary widely depending on how they interact with the servers and manipulate the dom.