Python - unable to retrieve data from web page table using beautiful soup or lxml xpath

Question

I am trying to extract data from “Advanced Boxing Points Statistics” from the following web page: http://www.sports-reference.com/cbb/boxscores/2016-11-11-villanova.html

I tried using BeautifulSoup very widely to get all tables:

import requests
from bs4 import BeautifulSoup

base_url = "http://www.sports-reference.com/cbb/boxscores/2016-11-11-villanova.html"
r = requests.get(base_url)
soup = BeautifulSoup(r.text, "html.parser")
tables = soup.find_all("table")
for table in tables:
       print table.get_text()

At the same time, he extracted only “Basic statistics on boxing”. However, he did not receive the “Advanced Box Score Stats,” as I had hoped.

Next, I tried to get more specific information using the lxml path:

import requests
from lxml import html
page = requests.get('http://www.sports-reference.com/cbb/boxscores/2016-11-11-villanova.html')
tree = html.fromstring(page.content)
boxscore_Advanced = tree.xpath('//*[@id="box-score-advanced-lafayette"]/tbody/tr[1]/td[1]/text()’)
print boxscore_Advanced

In doing so, it returns an empty list.

I struggled with this for a very long time, and tried to solve this problem using the following messages:

• Why does this xpath not work using lxml in python?
• Python xpath
• lxml xpath html
• http://web.stanford.edu/~zlotnick/TextAsData/Web_Scraping_with_Beautiful_Soup.html
• BeautifulSoup

!

Answer 1

selenium / PhantomJS. "Advanced Box Score Stats" HTML, HTML-. :

import requests
from bs4 import BeautifulSoup, Comment


url = "http://www.sports-reference.com/cbb/boxscores/2016-11-11-villanova.html"
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")

# find the comments containing the desired tables
tables = soup.find_all(text=lambda text: text and isinstance(text, Comment) and 'Advanced Box Score Stats' in text)

# we have 2 tables - one for an opponent team
for table in tables:
    table_soup = BeautifulSoup(table, "html.parser")
    advanced_table = table_soup.select_one("table[id^=box-score-advanced]")
    for row in advanced_table("tr")[2:]:  # skip headers
        print(row.th.get_text())
    print("-------")

:

Nick Lindner
Monty Boykins
Matt Klinewski
Paulius Zalys
Auston Evans
Reserves
Myles Cherry
Kyle Stout
Eric Stafford
Lukas Jarrett
Hunter Janacek
Jimmy Panzini
School Totals
-------
Kris Jenkins
Phil Booth
Josh Hart
Jalen Brunson
Darryl Reynolds
Reserves
Donte DiVincenzo
Mikal Bridges
Eric Paschall
Tim Delaney
Dylan Painter
Denny Grace
Tom Leibig
Matt Kennedy
School Totals
-------

Answer 2

@snakecharmerb : raw html ​​Javascript .

:

$ curl http://www.sports-reference.com/cbb/boxscores/2016-11-11-villanova.html | grep "box-score-advanced-lafayette"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9891    0  9891    0     0  45371      0 --:--:-- --:--:-- --:--:-- 48965<div id="all_box-score-advanced-lafayette" class="table_wrapper setup_commented commented">
  <span class="section_anchor" id="box-score-advanced-lafayette_link" data-label="Advanced Box Score"></span>
      <div class="overthrow table_container" id="div_box-score-advanced-lafayette">
  <table class="sortable stats_table" id="box-score-advanced-lafayette" data-cols-to-freeze=1><caption>&nbsp; Table</caption>
100  141k    0  141k    0     0   349k      0 --:--:-- --:--:-- --:--:--  363k

, , html, , .

- , Phantom.js http://phantomjs.org