Beautifulsoup and invalid html document

Question

I am trying to parse a document http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ecofin/acf8e.htm . I want to get the countries and names at the beginning of the document.

Here is my code

import urllib
import re
from bs4 import BeautifulSoup
url="http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ecofin/acf8e.htm"
soup=BeautifulSoup(urllib.urlopen(url))
attendances_table=soup.find("table", {"width":850})
print attendances_table #this works, I see the whole table
print attendances_table.find_all("tr")

I get the following error:

AttributeError: 'NoneType' object has no attribute 'next_element'

Then I tried to use the same solution as in this post (I know again: p): beautifulsoup with an invalid html document

I replaced the line:

soup=BeautifulSoup(urllib.urlopen(url))

with:

return BeautifulSoup(html, 'html.parser')

Now if I do:

print attendances_table

I only get:

<table border="0" cellpadding="10" cellspacing="0" width="850">
<tr><td valign="TOP" width="42%">
<p><b><u>Belgium</u></b></p></td></tr></table>

What should i change?

Answer 1

Solved!

I just used a different parser library lxml. Thanks, Martinne Peters, for that!

soup = BeautifulSoup(urllib.urlopen(url), 'lxml')

lxml , !

Answer 2

html5lib , :

soup = BeautifulSoup(urllib.urlopen(url), 'html5lib')

html5lib.

:

>>> from bs4 import BeautifulSoup
>>> import urllib
>>> url = "http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ecofin/acf8e.htm"
>>> soup = BeautifulSoup(urllib.urlopen(url), 'html5lib')
>>> attendances_table = soup.find("table", {"width": 850})
>>> print attendances_table
<table border="0" cellpadding="10" cellspacing="0" width="850">
<tbody><tr><td valign="TOP" width="42%">
<p><b><u>Belgium</u></b>:</p>
<p>Mr Philippe MAYSTADT</p></td>
<td valign="TOP" width="58%">
<p>Deputy Prime Minister, Minister for Finance and Foreign Trade</p></td>
</tr>
...
<tr><td valign="TOP" width="42%">
<b><u></u></b><u></u><p><u><b>Portugal</b></u>:</p>
<p>Mr António de SOUSA FRANCO</p>
<p>Mr Fernando TEIXEIRA dos SANTOS</p></td>
<td valign="TOP" width="58%">
<p>Minister for Finance</p>
<p>State Secretary for the Treasury and Finance</p></td>
</tr>
</tbody></table>

find_all('tr'):

>>> attendances_table = BeautifulSoup(str(attendances_table), 'html5lib')
>>> print attendances_table.find_all("tr")
[<tr><td valign="TOP" width="42%">
<p><b><u>Belgium</u></b>:</p>
<p>Mr Philippe MAYSTADT</p></td>
...
<tr><td valign="TOP" width="42%">
<b><u></u></b><u></u><p><u><b>Portugal</b></u>:</p>
<p>Mr António de SOUSA FRANCO</p>
<p>Mr Fernando TEIXEIRA dos SANTOS</p></td>
<td valign="TOP" width="58%">
<p>Minister for Finance</p>
<p>State Secretary for the Treasury and Finance</p></td>
</tr>]