Python - selenium webdriver stuck in .get () in a loop

Question

I have a snippet of Python code that uses Selenium Webdriver to scroll through some historical baseball odds. This first part of the code is designed to retrieve the entire separate game URL from the schedule table (consisting of 57 pages that must be encoded) and save them in a list.

The first time I tested this, it worked fine - now, for some reason, the driver.get () function does not seem to work. It happens that webdriver initiates the first .get () method in the pageRange loop (page 2), but after that at the next iteration of the loop it gets stuck and does not go to page 3. There is no error message or crash.

Some manual error checking with print () indicates that all other areas of the code are working fine. What could be the possible reasons for this problem?

EDIT 1: The code actually gets stuck right after the first call to .get (), and not until the second, as mentioned above. I also noticed that the .get () function works a bit later in the code when navigating through the game URL. For some reason, this is " http://www.oddsportal.com/baseball/usa/mlb-2017/results/#/page/2/ ", " http://www.oddsportal.com/baseball/usa/mlb- 2017 / results / # / page / 3 / "etc. that it is stuck.

season = str(2017) URL = "http://www.oddsportal.com/baseball/usa/mlb-" + season + "/results/#/" chrome_path = r"C:\Users\dansl110\Dropbox\Betting Project/chromedriver.exe" OddsList = pd.DataFrame(columns=["Date", "HomeTeam", "AwayTeam", "HomeOdds", "AwayOdds", "Accuracy"]) GameURLs = [] StartURL = 2 #Gets GameURLs and EndPage from Page 1 driver = webdriver.Chrome(chrome_path) driver.get(URL) elems = driver.find_elements_by_xpath("//a[@href]") for elem in elems: link = elem.get_attribute("href") if "/results/#/page/" in link: EndURL = int(''.join(c for c in link if c in digits)) elif "/mlb" in link and len(str(link)) > 58 and "results" not in link: GameURLs.append(link) PageRange = range(StartURL, EndURL - 5) #Gets remaining GameURLs for page in PageRange: oldURL = URL URL = "http://www.oddsportal.com/baseball/usa/mlb-" + season + "/results/#/page/" + str(page) + "/" #This .get() works only during the first iteration of the range loop driver.get(URL) time.sleep(3) elems = driver.find_elements_by_xpath("//a[@href]") for elem in elems: link = elem.get_attribute("href") if "/nhl" in link and len(str(link)) > 65 and "results" not in link: GameURLs.append(link) 

Answer 1

I had the same problem starting today. I found that on all machines on which I had running 64.- versions of Chrome, there was a problem with intermittent freezes, but machines with 63.- weren’t. go to chrome://settings/help and check which version:

If you are using this version. try downloading the version of Chromedriver here (2.35): https://sites.google.com/a/chromium.org/chromedriver/downloads

I tried this and it seems to have helped a little with hanging, but still it seems like this is happening.

The only thing fixed is to get back to creating 63.- for Chrome.

Hope this helps you.

EDIT:

I found this thread that will help! Add this to your script before creating the driver:

 from selenium import webdriver ChromeOptions = webdriver.ChromeOptions() ChromeOptions.add_argument('--disable-browser-side-navigation') driver = webdriver.Chrome('your/path/to/chromedriver.exe', chrome_options=ChromeOptions) 

After the release of Chrome 65.- it will be fixed. In the meantime, use the above if you're still on 64.-

Answer 2

Have you tried using driver = webdriver.Firefox() ? I believe it is more reliable, and you can even use the Selenium IDE .