How to extract table data in pairs using BeautifulSoup?

Question

An example of my data:

<table id = "history"> <tr class = "printCol"> <td class="name">Google</td><td class="date">07/11/2001</td><td class="state"> <span>CA</span> </td> </tr> <tr class = "printCol"> <td class="name">Apple</td><td class="date">27/08/2001</td> </tr> <tr class = "printCol"> <td class="name">Microsoft</td><td class="date">01/11/1991</td> </tr> </table> 

Beautifulsoup Code:

 table = soup.find("table", id = "history") rows = table.findAll('tr') for tr in rows: cols = tr.findAll('td') for td in cols: print td.find(text=True) 

Desired output for MySQL storage (list):

 ['Google|07/11/2001|CA', 'Apple|27/08/2001', 'Microsoft|01/11/1991'] 

Imprint I have (it is difficult to associate the correct date with the right company):

 Google 07/11/2001 Apple 27/08/2001 Microsoft 01/11/1991 

I wrote a function to extract elements from each tr, but I thought there was a much more efficient way to do all this in the original loop. I want to save them in a list as data pairs. Thoughts?

Answer 1

Clarification of the list will make it easier:

 table = soup.find("table", id = "history") rows = table.findAll('tr') data = [[td.findChildren(text=True) for td in tr.findAll("td")] for tr in rows] # data now contains: [[u'Google', u'07/11/2001'], [u'Apple', u'27/08/2001'], [u'Microsoft', u'01/11/1991']] # If the data may contain extraneous whitespace you can clean it up # Additional processing could also be done - but once you hit much more # complex than this later maintainers, yourself included, will thank you # for using a series of for loops that call clearly named functions to perform # the work. data = [[u"".join(d).strip() for d in l] for l in data] # If you want to store it joined as name | company # then simply follow that up with: data = [u"|".join(d) for d in data] 

Understanding a list is basically an inverse for loop with aggregation:

 [[td.findNext(text=True) for td in tr.findAll("td")] for tr in rows] 

means ^* :

 final_list = [] intermediate_list = [] for tr in rows: for td in tr.findAll("td") intermediate_list.append(td.findNext(text=True)) final_list.append(intermediate_list) intermediate_list = [] data = final_list 

^* Roughly - we don’t notice that generators are not building intermediate lists, since I can’t add generators right now without cluttering up the example.

Answer 2

Here is a small variation of Sean's answer, if you need exactly what you wrote,

 table = soup.find("table", id = "history") rows = table.findAll('tr') data = ['|'.join([td.findNext(text=True) for td in tr.findAll("td")]) for tr in rows] print data