All geek questions in one place

Beautifulsoup and mechanize to get the result of an ajax call

Hi, I'm creating a scraper using python 2.5 and beautifulsoup but in case of a problem ... the part of the webpage is generated after the user clicks on some button, start the ajax request by calling a specific javacsript function using the correct parameters

Is there a way to simulate user interaction and get this result? I come across a mechanization module, but it seems to me that this is mainly used for working with forms ...

I would appreciate any links or some code examples thanks

+4
source share
2 answers

ok, so I realized that it was pretty simple after I realized that I can use a combination of urllib, ulrlib2 and beautifulsoup

import urllib, urllib2 from BeautifulSoup import BeautifulSoup as bs_parse data = urllib.urlencode(values) req = urllib2.Request(url, data) res = urllib2.urlopen(req) page = bs_parse(res.read())
+6
source

No, you cannot do it easily. AFAIK your options are easiest:

  • Read the AJAX javascript code yourself, as a human programmer, understand it, and then write python code to simulate AJAX calls manually. You can also use some capture software to capture requests / responses made in real time and try to play them using code;
  • Use selenium or some other browser automation tool to get the page in a real web browser;
  • Use some javascript runner for python like spidermonkey or pyv8 to run javascript code and bind it to your copy of HTML dom;
+3
source

Source: https://habr.com/ru/post/1306511/

More articles:


All Articles