Web page cleaning: automatic button click

Question

I am trying to clear data from a website using Scrapy , a python framework. I can get data from the site using spiders, but the problem occurs when I try to navigate the website.

Accordingly , the Scrapy post does not support Javascript.

Also, as indicated in the accepted answer, I cannot use mechanize or lxml. He suggests using a combination Selenium and Scrapy.

Button Function:

I am looking at offers on the website. Button function to show more offers. SO when clicked on it calls a javascript function that loads the results.

I also watched CasperJS and PhantomJS. Will they work?

I just need to automate the button click. How should I do it?

Answer 1

First of all, yes - you can use PhantomJSghostdriver with python. It is built in python-selenium:

pip install selenium

Demo:

>>> from selenium import webdriver
>>> driver = webdriver.PhantomJS()
>>> driver.get('https://stackoverflow.com/questions/27813251')
>>> driver.title
u'javascript - Web scraping: Automating button click - Stack Overflow'

There are also several other threads that give examples of scrapy + selenium spiders:

• selenium with scrapy for dynamic page
• Scraper with Scrapy and Selenium
• seleniumcrawler

There is also a scrapy-webdrivermodule , which may also help with it.

---

scrapy with selenium PhantomJS.

, " ", , . , , , scrapy.http.Request .