How to click a link that has javascript: __ doPostBack in href?

Question

I am writing a scraper script screen in python with the 'mechanize' module, and I would like to use the mechanize.click_link () method on a link that has javascript: __ doPostBack in href. I believe that the page I'm trying to parse is using AJAX.

Note: mech - mechanize.Browser ()

>>> next_link.__class__.__name__ 'Link' >>> next_link Link(base_url='http://www.citius.mj.pt/Portal/consultas/ConsultasDistribuicao.aspx', url="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext','')", text='2', tag='a', attrs=[('id', 'ctl00_ContentPlaceHolder1_Pager1_lnkNext'), ('title', 'P\xc3\xa1gina seguinte: 2'), ('href', "javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext','')")]) >>> req = mech.click_link(next_link) >>> req <urllib2.Request instance at 0x025BEE40> >>> req.has_data() False 

I want to get the page source after clicking the link.

Answer 1

I don't use mechanization, but I read a lot with python.

When I come across a javascript function like __doPostBack, I do the following:
I access the website in Firefox and use the HttpFox extension to see the parameters of the POST request sent by the browser to the web server when I click on the corresponding link.
Then I create the same request in python using urllib.parse.urlencode to build the query strings and POST data that I need.
Sometimes a website also uses cookies, so I just use python http.cookiejar.

I have used this technique several times.

Answer 2

I do not think mechanize supports Javascript; To clear pages that rely on Javascript for their functionality, you may need to use another tool, such as Selenium RC .

Answer 3

 >>> next_link.__class__.__name__ 'Link' >>> next_link Link(base_url='http://www.citius.mj.pt/Portal/consultas/ConsultasDistribuicao.aspx', url="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext','')", text='2', tag='a', attrs=[('id', 'ctl00_ContentPlaceHolder1_Pager1_lnkNext'), ('title', 'P\xc3\xa1gina seguinte: 2'), ('href', "javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext','')")]) >>> req = mech.click_link(next_link) >>> req <urllib2.Request instance at 0x025BEE40> >>> req.has_data() False