All geek questions in one place

Acceleration of the beautiful

I am launching a scraper of this course website, and I am wondering if there is a faster way to clear the page as soon as I close it in beautifulsoup. It will take longer than I expected.

Tips?

from selenium import webdriver from selenium.webdriver.common.keys import Keys from selenium.webdriver.support.ui import Select from selenium.webdriver.support import expected_conditions as EC from bs4 import BeautifulSoup driver = webdriver.PhantomJS() driver.implicitly_wait(10) # seconds driver.get("https://acadinfo.wustl.edu/Courselistings/Semester/Search.aspx") select = Select(driver.find_element_by_name("ctl00$Body$ddlSchool")) parsedClasses = {} for i in range(len(select.options)): print i select = Select(driver.find_element_by_name("ctl00$Body$ddlSchool")) select.options[i].click() upperLevelClassButton = driver.find_element_by_id("Body_Level500") upperLevelClassButton.click() driver.find_element_by_name("ctl00$Body$ctl15").click() soup = BeautifulSoup(driver.page_source, "lxml") courses = soup.select(".CrsOpen") for course in courses: courseName = course.find_next(class_="ResultTable")["id"][13:] parsedClasses[courseName] = [] print courseName for section in course.select(".SecOpen"): classInfo = section.find_all_next(class_="ItemRowCenter") parsedClasses[courseName].append((int(classInfo[0].string), int(classInfo[1].string), int(classInfo[2].string))) print parsedClasses print parsedClasses['FL2014' + 'A46' + '3284'] driver.quit()
+6
source share
1 answer

Well, you can really speed it up:

Since this is an ASP.NET generated form and, thanks to it, security features, the situation becomes a little more complicated. Here is the full code, do not be afraid of it - I added comments and open questions:

 import re from bs4 import BeautifulSoup, SoupStrainer import requests # start session and get the search page session = requests.Session() response = session.get('https://acadinfo.wustl.edu/Courselistings/Semester/Search.aspx') # parse the search page using SoupStrainer and lxml strainer = SoupStrainer('form', attrs={'id': 'form1'}) soup = BeautifulSoup(response.content, 'lxml', parse_only=strainer) # get the view state, event target and validation values viewstate = soup.find('input', id='__VIEWSTATE').get('value') eventvalidation = soup.find('input', id='__EVENTVALIDATION').get('value') search_button = soup.find('input', value='Search') event_target = re.search(r"__doPostBack\('(.*?)'", search_button.get('onclick')).group(1) # configure post request parameters data = { '__EVENTTARGET': event_target, '__EVENTARGUMENT': '', '__LASTFOCUS': '', '__VIEWSTATE': viewstate, '__EVENTVALIDATION': eventvalidation, 'ctl00$Body$ddlSemester': '201405', 'ctl00$Body$ddlSession': '', 'ctl00$Body$ddlDept': '%', 'ctl00$Body$ddlAttributes': '0', 'ctl00$Body$Days': 'rbAnyDay', 'ctl00$Body$Time': 'rbAnyTime', 'ctl00$Body$cbMorning': 'on', 'ctl00$Body$cbAfternoon': 'on', 'ctl00$Body$cbEvening': 'on', 'ctl00$Body$tbStart': '9:00am', 'ctl00$Body$tbEnds': '5:00pm', 'ctl00$Body$ddlUnits': '0', 'ctl00$Body$cbHideIStudy': 'on', 'ctl00$Body$courseList$hidHoverShow': 'Y', 'ctl00$Body$courseList$hidDeptBarCnt': '', 'ctl00$Body$courseList$hidSiteURL': 'https://acadinfo.wustl.edu/Courselistings', 'ctl00$Body$courseList$hidExpandDetail': '', 'ctl00$Body$hidDay': ',1,2,3,4,5,6,7', 'ctl00$Body$hidLevel': '1234', 'ctl00$Body$hidDefLevel': '' } # get the list of options strainer = SoupStrainer('div', attrs={'id': 'Body_courseList_tabSelect'}) options = soup.select('#Body_ddlSchool > option') for option in options: print "Processing {option} ...".format(option=option.text) data['ctl00$Body$ddlSchool'] = option.get('value') # make the search post request for a particular option response = session.post('https://acadinfo.wustl.edu/Courselistings/Semester/Search.aspx', data=data) result_soup = BeautifulSoup(response.content, parse_only=strainer) print [item.text[:20].replace('&nbsp', ' ') + '...' for item in result_soup.select('div.CrsOpen')]

Print

 Processing Architecture ... [u'A46 ARCH 100...', u'A46 ARCH 111...', u'A46 ARCH 209...', u'A46 ARCH 211...', u'A46 ARCH 266...', u'A46 ARCH 305...', u'A46 ARCH 311...', u'A46 ARCH 323...', u'A46 ARCH 328...', u'A46 ARCH 336...', u'A46 ARCH 343...', u'A46 ARCH 350...', u'A46 ARCH 355...', u'A46 ARCH 411...', u'A46 ARCH 422...', u'A46 ARCH 428...', u'A46 ARCH 436...', u'A46 ARCH 445...', u'A46 ARCH 447...', u'A46 ARCH 465...', u'A48 LAND 451...', u'A48 LAND 453...', u'A48 LAND 461...'] Processing Art ... [u'F10 ART 1052...', u'F10 ART 1073...', u'F10 ART 213A...', u'F10 ART 215A...', u'F10 ART 217B...', u'F10 ART 221A...', u'F10 ART 231I...', u'F10 ART 241D...', u'F10 ART 283T...', u'F10 ART 301A...', u'F10 ART 311E...', u'F10 ART 313D...', u'F10 ART 315B...', u'F10 ART 317H...', u'F10 ART 323A...', u'F10 ART 323B...', u'F10 ART 323C...', u'F10 ART 329C...', u'F10 ART 337E...', u'F10 ART 337F...', u'F10 ART 337H...', u'F10 ART 385A...', u'F10 ART 391M...', u'F10 ART 401A...', u'F10 ART 411E...', u'F10 ART 413D...', u'F10 ART 415B...', u'F10 ART 417H...', u'F10 ART 423A...', u'F10 ART 423B...', u'F10 ART 423C...', u'F10 ART 429C...', u'F10 ART 433C...', u'F10 ART 433D...', u'F10 ART 433E...', u'F10 ART 433K...', u'F10 ART 461C...', u'F10 ART 485A...', u'F20 ART 111P...', u'F20 ART 115P...', u'F20 ART 1186...', u'F20 ART 119C...', u'F20 ART 127A...', u'F20 ART 133B...', u'F20 ART 135G...', u'F20 ART 135I...', u'F20 ART 135J...', u'F20 ART 1361...', u'F20 ART 1363...', u'F20 ART 1713...', u'F20 ART 219C...', u'F20 ART 2363...', u'F20 ART 2661...', u'F20 ART 281S...', u'F20 ART 311P...', u'F20 ART 315P...', u'F20 ART 3183...', u'F20 ART 333B...', u'F20 ART 335A...', u'F20 ART 335J...', u'F20 ART 3713...', u'F20 ART 381S...', u'F20 ART 415P...', u'F20 ART 435I...'] ...

Of course, there are things that need to be improved, for example, I hardcoded the other values ​​of the form - you probably should analyze the possible values ​​and set them accordingly.

Another improvement would be to associate this with grequests :

GRequests allows you to use queries with Gevent for asynchronous HTTP requests easily.

As you can see, when you are at a higher level and interact with the browser through webdriver, you are not worried about the actual requests coming to the server to receive data. This simplifies automation, but can be painfully slow. When you switch to low-level automation, you have more options to speed up the process, but the complexity of the implementation grows very quickly. Also, think about how reliable such a solution can be. So you can stick to the black box solution and stay with selenium ?

I also tried to solve the problem using:

but failed due to various reasons (may provide you with relevant error messages). Although all these 3 tools should help simplify the solution.

Also see related topics:

+10
source

Source: https://habr.com/ru/post/974442/

More articles:


All Articles