You have some errors. The first is a “navigation” timeout, indicating that the page has not completed rendering:
Exception in callback NavigatorWatcher.waitForNavigation.<locals>.watchdog_cb(<Task finishe...> result=None>) at C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py:49 handle: <Handle NavigatorWatcher.waitForNavigation.<locals>.watchdog_cb(<Task finishe...> result=None>) at C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py:49> Traceback (most recent call last): File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\asyncio\events.py", line 145, in _run self._callback(*self._args) File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py", line 52, in watchdog_cb self._timeout) File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py", line 40, in _raise_error raise error concurrent.futures._base.TimeoutError: Navigation Timeout Exceeded: 3000 ms exceeded
This trace does not rise in the main thread, because of this, your code was not interrupted. Your page may or may not be complete; you can set a longer timeout or enter a sleep cycle for the browser in order to process AJAX responses in time.
Then the response.html.render() element returns None . It loads the HTML into the Chromium's mute browser, leaves the JavaScript rendering in that browser, and then copies the HTML page to the response.html data structure in place, and nothing needs to be returned. Thus, js set to None , and not to a new instance of HTML , causing the next trace.
Use an existing response.html object to search after rendering:
r.html.render() item = r.html.find('.MarketInfo_market-num_1lAXs', first=True)
Most likely, there is no such CSS class, because the last 5 characters are generated on each page, after the JSON data is loaded through AJAX. This makes it difficult to use CSS to find the item in question.
In addition, I found that without a sleep cycle, the browser does not have time to pick up AJAX resources and display the information you want to download. Give him, say, 10 seconds of sleep to do some work before copying the HTML. Set a longer timeout (default is 8 seconds) if you see network timeouts:
r.html.render(timeout=10, sleep=10)
You can also set timeout to 0 to remove the timeout and just wait endlessly until the page loads.
Hopefully a future API update also provides features to wait for network activity to stop .
You can use the parse nested library to find the appropriate CSS classes:
# search for CSS suffixes suffixes = [r[0] for r in r.html.search_all('MarketInfo_market-num_{:w}')] for suffix in suffixes:
Now we get the result:
169.81 EUR + 1.01 % 18,420 LTC 169.81 EUR + 1.01 % 18,420 LTC 169.81 EUR + 1.01 % 18,420 LTC 169.81 EUR + 1.01 % 18,420 LTC
Your last trace shows that the Chromium user data path cannot be cleared. The Pyppeteer base library sets up the Mute Chromium browser with a temporary user data path, and in your case the directory contains some resource with a permanent lock. You can ignore the error, although you can try to delete all remaining files in the .pyppeteer folder later.