Scrambling an ASP.NET Site Using Ruby

Question

I would like to clear the search results of this ASP.NET site using Ruby and, preferably, just use Hpricot (I cannot open an instance of Firefox): http://www.ngosinfo.gov.pk/SearchResults.aspx?name=&foa= 0

However, it’s hard for me to figure out how to get through each page of results. Basically, I need to simulate a click on links like these:

<a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$2','')" class="blue_11" id="ctl00_ContentPlaceHolder1_Pager1">2</a> <a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$3','')" class="blue_11" id="ctl00_ContentPlaceHolder1_Pager1">3</a> 

and etc.

I tried using Net :: HTTP to process the message, but until it received the correct HTML, there were no search results (maybe I'm not doing it right). In addition, the page URL does not contain any parameters pointing to the page, so it is not possible to present the results in this way.

Any help would be greatly appreciated.

Answer 1

Using mechanize-1.0.0, do the following:

  agent = Mechanize.new page = agent.get('http://127.0.0.1/some.aspx') form = page.form("aspnetForm") form.add_field!('__EVENTARGUMENT', 'Page$2') form.add_field!('__EVENTTARGET', 'ctl00$ContentPlaceHolder1$gvwSomeList') page = agent.submit(form) # this gets page 2 

Answer 2

Better yet, check Mechanization. A good starting point on screen scripting is the mechanization episode of railscasts.com.

Answer 3

If you are just starting out, you can check out Nokogiri . It is lighter and more documented than Hpricot (which seems to have been abandoned).

Edit: Jakub Hampl is right - Mechanize is what you are looking for to interact with websites. He works in collaboration with Nokogiri (which parses HTML and XML).