All geek questions in one place

How to ignore blank lines when using .next_sibling in BeautifulSoup4 in python

As I want to remove duplicate placeholders on the html website, I use the .next_sibling operator for BeautifulSoup. As long as duplicates are on the same line, this works fine (see Data). But sometimes there is an empty line between them - so I want .next_sibling to ignore them (look at data2)

This is the code:

from bs4 import BeautifulSoup, Tag
data = "<p>method-removed-here</p><p>method-removed-here</p><p>method-removed-here</p>"
data2 = """<p>method-removed-here</p>

<p>method-removed-here</p>

<p>method-removed-here</p>

<p>method-removed-here</p>

<p>method-removed-here</p>
"""
soup = BeautifulSoup(data)
string = 'method-removed-here'
for p in soup.find_all("p"):
    while isinstance(p.next_sibling, Tag) and p.next_sibling.name== 'p' and p.text==string:
        p.next_sibling.decompose()
print(soup)

Output for data as expected:

<html><head></head><body><p>method-removed-here</p></body></html>

Output for data2 (this needs to be fixed):

<html><head></head><body><p>method-removed-here</p>

<p>method-removed-here</p>

<p>method-removed-here</p>

<p>method-removed-here</p>

<p>method-removed-here</p>
</body></html>

I could not find useful information for this in the BeautifulSoup4 documentation, and .next_element is also not what I am looking for.

+6
source share
4 answers

. google-group BeautifulSoup, html :

 def bs_preprocess(html):
     """remove distracting whitespaces and newline characters"""
     pat = re.compile('(^[\s]+)|([\s]+$)', re.MULTILINE)
     html = re.sub(pat, '', html)       # remove leading and trailing whitespaces
     html = re.sub('\n', ' ', html)     # convert newlines to spaces
                                        # this preserves newline delimiters
     html = re.sub('[\s]+<', '<', html) # remove whitespaces before opening tags
     html = re.sub('>[\s]+', '>', html) # remove whitespaces after closing tags
     return html

, .

+6

,

def get_sibling(element):
    sibling = element.next_sibling
    if sibling == "\n":
        return get_sibling(sibling)
    else:
        return sibling
+3

neurosnap , :

def next_elem(element, func):
    new_elem = getattr(element, func)
    if new_elem == "\n":
        return next_elem(new_elem, func)
    else:
        return new_elem

, :

next_elem(element, 'previous_sibling')
+2

find_next_sibling() next_sibling. find_previous_sibling() find_previous_sibling() previous_sibling.

: next_sibling HTML-, " ". , . find_next_sibling() , html-, .

, . , .

next_sibling , ( data data2)

from bs4 import BeautifulSoup, Tag
data = "<p>method-removed-here</p><p>method-removed-here</p><p>method-removed-here</p>"
data2 = """<p>method-removed-here</p>

<p>method-removed-here</p>

<p>method-removed-here</p>

<p>method-removed-here</p>

<p>method-removed-here</p>
"""
soup = BeautifulSoup(data, 'html.parser')
string = 'method-removed-here'
for p in soup.find_all("p"):
    while True:
        ns = p.next_sibling
        if isinstance(ns, Tag) and ns.name== 'p' and p.text==string:
            ns.decompose()
        else:
            break
print(soup)

find_next_sibling() data data2

soup = BeautifulSoup(data, 'html.parser')
string = 'method-removed-here'
for p in soup.find_all("p"):
    while True:
        ns = p.find_next_sibling()
        if isinstance(ns, Tag) and ns.name== 'p' and p.text==string:
            ns.decompose()
        else:
            break
print(soup)

( , ) Beautifulsoup: BeautifulSoup.children .content

0

Source: https://habr.com/ru/post/1537680/

More articles:


All Articles