python third-party library beautiful soup4 document learning

Traverse the document tree

A document in html or xml format will become a document tree after bs processing. The top-level node is a tag. This tag contains many child nodes, which can be strings or tags. Next, learn to traverse the document tree with an example document.

html_doc = """<html>
    <title>The Dormouse's story</title>
    <p class="title"><b>The Dormouse's story</b></p>
    <p class="story">Once upon a time there were three little sisters; and their names were
      <a href="" class="sister" id="link1">Elsie</a>,
      <a href="" class="sister" id="link2">Lacie</a>and
      <a href="" class="sister" id="link3">Tillie</a>;
      and they lived at the bottom of a well.

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'html.parser')

Child node

The child node can be a string or tag. bs provides many operations and attributes to traverse the child node, but the string itself does not support continuous traversal.

Traversal by tag name

For example, in the above example document, if you want to get the first a tag, you can directly get up. A. if there are child nodes under the child node, for example, there are string child nodes under the a tag, you can get the string child nodes under the a signature child node under the soup object by way of up.a.string.

Note: the child node obtained by. Above is the first child node found in the document

If you need to get all a child nodes in the current document, you can use the find of the beautiful soup object_ The all () method returns a list

list_a = soup.find_all('a')

# [<a class="sister" href="" id="link1">Elsie</a>, <a class="sister" href="" id="link2">Lacie</a>, <a class="sister" href="" id="link3">Tillie</a>]

Traversal through. contents and. children

Through the. contents attribute, the child nodes of a tag can be output as a list, such as:

list_tag = soup.a.contents

# ['Elsie']

Remember that the BeautifulSoup object itself is also a document, so it also has child nodes:

list_soup = soup.contents

# html

Through the tag.children attribute, you can perform a for loop on the child nodes of tag

a_tag = soup.a
for i in a_tag:

# Elsie

. descendants property

Through. descendants, you can recursively cycle through all descendant nodes under a parent tag node:

for i in soup.head.descendants:

# <title>The Dormouse's story</title>
# The Dormouse's story

. string attribute

If tag has only one child node of NavigableString type or only one child node, you can use. String to obtain the unique string, for example:

title_tag = soup.title
# The Dormouse's story

head_tag = soup.head
# <head><title>The Dormouse's story</title></head>
# The Dormouse's story

Note: if there is more than one child node under a tag (including spaces, line breaks, etc.), then. string will output None,

. strings and stripped_strings

If the tag contains multiple strings, you can use. Strings to loop through:

for stri in soup.strings:

Parent node

Each tag or string has a parent node


The parent node of the tag or string can be obtained through the. Parent attribute of the tag or string, for example:

tag = soup.title
# <head><title>The Dormouse's story</title></head>
# None


. parents, as the name suggests, can traverse all parent nodes of a tag or string until None

tag = soup.a
for ele in tag.parents:

# 'p'
# 'body'
# 'html'
# '[document]'

Sibling node

Children with the same parent and at the same level are called siblings

.next_sibling and. previous_sibling

sibling_soup = BeautifulSoup("<a><b>text1</b><c>text2</c></a>","html.parser")


# <c>text2</c>
# None
# <b>text1</b>
# None

.next_siblings and. previous_siblings

You can iteratively output the sibling nodes of the current node, for example

for sibling in soup.a.next_siblings:

# ',\n'
# <a class="sister" href="" id="link2">Lacie</a>
# ' and\n'
# <a class="sister" href="" id="link3">Tillie</a>
# ';\nand they lived at the bottom of a well.'

Back and forward

Let's look at an html document:


After parsing through the bs, the document should be opened with a tag, opened with a tag, written to the content in the head, closed with a tag, and then opened with a tag.

.next_element and. previous_element

.next_element is used to point to the next object element (a tag or string) of the element in the parsing process. The pointing result may be the same as. next_sibling is the same, but most of the time it is different.

last_a_tag = soup.find("a", id="link3")

# 'Tillie'
# ';\nand they lived at the bottom of a well.'

.previous_element and. next_element instead, it points to the previous object of the current resolved object

last_a_tag = soup.find("a", id="link3")

# ' and\n'
# <a class="sister" href="" id="link3">Tillie</a>

.next_elements and. previous_elements

Via. next_elements and. previous_elements can iterate through the objects to be parsed after and before the current element

last_a_tag = soup.find("a", id="link3")
for element in last_a_tag.next_elements:

# 'Tillie'
# ';\nand they lived at the bottom of a well.'
# '\n'
# <p class="story">...</p>
# '...'
# '\n'

Posted on Thu, 11 Nov 2021 14:55:15 -0500 by Dave96