Stop scrolling in a dynamic page using Selenium in Python

Question

Hi everyone, I'm trying to use selenium and scrapy to scrape some information from https://answers.yahoo.com/dir/index/discover?sid=396545663

I try another method, I use Selenium and install PhantomJs as a driver. To scroll a page, this is an endless scroll page, I use this instruction:

elem.send_keys(Keys.PAGE_DOWN)

To simulate a click on the Down button instead of a JavaScript function:

browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Because this "seems" loads fewer elements on the page.

Main problem is how can I find out when I reached the bottom of the page? Is the page "Infinite scroll", so I can not know when it will end, I need to scroll down, but I do not have an element at the bottom for analysis.

I actually use a time loop, but I look really stupid.

thanks

Answer 1

I would really look for this "Loading ..." indicator. Wait until it is visible on each scroll, but if you get it TimeoutException- this time there was no download indicator on it, and there is no longer loading.

Implementation Example:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

while True:
    # do the scrolling
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    try:
        wait.until(EC.visibility_of_element_located((By.XPATH, "//*[. = 'Loading...']")))
    except TimeoutException:
        break  # not more posts were loaded - exit the loop

Not tested.

Answer 2

, ajax. 10 - . .