All geek questions in one place

HTML and BeautifulSoup: how to iteratively parse when a structure is not always known in advance?

I started with a simple HTML structure, something like this:

enter image description here

Thanks to the help of @alecxe, I was able to create this JSON dict:

{u'Outer List': {u'Inner List': [u'info 1', u'info 2', u'info 3']}}

using his code:

from bs4 import BeautifulSoup

data = """your html goes here: see the very end of post""" 
soup = BeautifulSoup(data)

inner_ul = soup.find('ul', class_='innerUl')
inner_items = [li.text.strip() for li in inner_ul.ul.find_all('li')]

outer_ul_text = soup.ul.span.text.strip()
inner_ul_text = inner_ul.span.text.strip()

result = {outer_ul_text: {inner_ul_text: inner_items}}
print result

The code is fantastic and I am trying to rewrite it iteratively.

My “real” HTML dataset is much bigger and more annoying, and I need to scale the code so that I can handle something like this:

enter image description here

Or maybe the data looks like that:

enter image description here

To make things even worse, perhaps underneath sublistwe have one more sublist! Ultimately, this is my real situation.

: BeautifulSoup, ( " "!).

/ HTML , HTML ? BeautifulSoup? , - , , , .

, !

HTML :

<html>
 <body>
  <ul class="rootList">
   <li class="liItem endPlus">
    <span class="itemToBeAdded">
     Outer List
    </span>
   </li>
   <li class="noBulletsLi ">
    <ul class="innerUl">
     <li class="liItem crossPlus">
      <span class="itemToBeAdded">
       Inner List
      </span>
      <ul class="grayStarUl ">
       <li class="">
        <span class="phrasesToBeAdded">
         info 1
        </span>
       </li>
       <li class="">
        <span class="phrasesToBeAdded">
         info 2
         </span>
       </li>
       <li class="">
        <span class="phrasesToBeAdded">
         info 3
        </span>
             <ul class="grayStarUl">
                 <li class="">
                     <span class="phrasesToBeAdded">sublist</span>
                 </li>
             </ul>            
       </li>
      </ul>
     </li>
      </ul>
     </li>
    </ul>
 </body>
</html>
+4
2

, :

def parse_list(tag):
    return map(parse_list_item, tag.find_all('li', recursive=False))

def parse_list_item(tag):
    text = tag.find(text=True, recursive=False).strip()
    text += '\n' + tag.span.text.strip() if tag.span.parent == tag else ''
    inner = tag.find('ul', recursive=False)
    if inner is None:  # no more nesting:
        return text.strip()
    else:  # more nesting
        return {text.strip():parse_list(inner)} if text else parse_list(inner)

- class :

>>> parse_list(soup.find('ul'))
[u'Outer List', [{u'Inner List': [u'info 1', u'info 2', {u'info 3': [u'sublist']}]}]]
+3

, . , . , , .

soup = BeautifulSoup(html_doc)
spans = soup.findall(class="phrasesToBeAdded")
text = []
for element in spans:
    text.append(element.get_text())
0

Source: https://habr.com/ru/post/1533614/

More articles:


All Articles