This is a problem that I have been struggling with for several days. I am trying to automate downloading files from this website and hopefully other sites with a similar design. I know that the website is ASP, and I know that all download links call the doPostBack function, which takes the form:
function __doPostBack(eventTarget, eventArgument) { if (!theForm.onsubmit || (theForm.onsubmit() != false)) { theForm.__EVENTTARGET.value = eventTarget; theForm.__EVENTARGUMENT.value = eventArgument; theForm.submit(); } }
I tried just clicking links through selenium and phantomjs, which fails. Then I decided to try and emulate a function call. To do this, I used mechanization and did the following:
def python_func(self,url,target,argument): br = mechanize.Browser() br.set_handle_robots(False) br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3. 0.1')] br.open(url) form_num = 0 while True: try: inner_browser.select_form(nr=form_num) except FormNotFoundError: return False br.form.set_all_readonly(False) inner_browser["__EVENTTARGET"] = targ inner_browser["__EVENTARGUMENT"] = arg response = inner_browser.submit()
Unfortunately, the form does not have a submit control, and I get nothing from sending the request. I also tried to emulate a mail request by manually coding the four ASP fields of doPostBack. These are eventtarget, eventargument, viewstate (has a default value) and eventvalidaiton (also has a default value). That won't work either - I get the Ginnie Mae site again.
I suppose I can somehow take the doPostBack function and run it directly using a JavaScript interpreter, but I just don't know how to do it. So far I have mechanization, selenium and phantomJS work for me, but I'm not sure if I can use any of them to run the code and download the file.
source share