[Tut] Searching The Parse Tree Using BeautifulSoup

[Tut] Searching The Parse Tree Using BeautifulSoup - Printable Version

+- Sick Gaming (https://www.sickgaming.net)
+-- Forum: Programming (https://www.sickgaming.net/forum-76.html)
+--- Forum: Python (https://www.sickgaming.net/forum-83.html)
+--- Thread: [Tut] Searching The Parse Tree Using BeautifulSoup (/thread-98554.html)

[Tut] Searching The Parse Tree Using BeautifulSoup - xSicKxBot - 12-01-2020

Searching The Parse Tree Using BeautifulSoup

<div><h2>Introduction</h2>
HTML (Hypertext Markup Language) consists of numerous tags and the data we need to extract lies inside those tags. Thus we need to find the right tags to extract what we need. Now, how do we find the right tags? We can do so with the help of <code data-enlighter-language="generic" class="EnlighterJSRAW">BeautifulSoup's </code>search methods.
Beautiful Soup has numerous methods for searching a parse tree. The two most popular and commonly methods are:
<ol>
<li> <code>find()</code></li>
<li> <code>find_all()</code></li>
</ol>
The other methods are quite similar in terms of their usage. Therefore, we will be focusing on the <code data-enlighter-language="generic" class="EnlighterJSRAW">find()</code> and<code data-enlighter-language="generic" class="EnlighterJSRAW"> find_all() </code>methods in this article. 
<img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f6a9.png" alt="?" class="wp-smiley" style="height: 1em; max-height: 1em;" /> The following Example will be used throughout this document while demonstrating the concepts:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">html_doc = """ <html><head><title>Searching Tree</title></head>
<body>
<h1>Searching Parse Tree In BeautifulSoup</h1> Learning <a href="https://docs.python.org/3/" class="language" id="python">Python</a>,
<a href="https://docs.oracle.com/en/java/" class="language" id="java">Java</a> and
<a href="https://golang.org/doc/" class="language" id="golang">Golang</a>;
is fun! Please subscribe!
copyright - FINXTER """
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, "html.parser")</pre>
<h2>Types Of Filters </h2>
There are different filters that can be passed into the <code data-enlighter-language="generic" class="EnlighterJSRAW">find()</code> and <code data-enlighter-language="generic" class="EnlighterJSRAW">find_all()</code> methods and it is crucial to have a clear understanding of these filters as they are used again and again, throughout the search mechanism. These filters can be used based on the tags:
<ul>
<li>name, </li>
<li>attributes, </li>
<li>on the text of a string,</li>
<li>or a mix of these.</li>
</ul>
<h3>❖ A String</h3>
When we pass a string to a search method then Beautiful Soup performs a match against that passed string. Let us have a look at an example and find the <h1> tags in the HTML document:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(soup.find_all('h1'))</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">[<h1>Searching Parse Tree In BeautifulSoup</h1>]</pre>
<h3>❖ A <a href="https://blog.finxter.com/python-regex/" target="_blank" rel="noreferrer noopener">Regular Expression</a></h3>
Passing a regular expression object allows Beautiful Soup to filter results according to that regular expression. In case you want to master the concepts of the regex module in Python, please refer to our <a href="https://blog.finxter.com/python-regex/" target="_blank" rel="noreferrer noopener">tutorial here</a>. 
Note: 
<ul>
<li>We need to import the <code>re</code> module to use a regular expression.</li>
<li>To get just the name of the tag instead of the entire content (tag+ content within the tag), use the <code data-enlighter-language="generic" class="EnlighterJSRAW">.name</code> attribute.</li>
</ul>
Example: The following code finds all instances of the tags starting with the letter “b”. 
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""># finding regular expressions
for regular in soup.find_all(re.compile("^b")): print(regular.name)</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">body
b</pre>
<h3>❖ A <a href="https://blog.finxter.com/python-lists/" target="_blank" rel="noreferrer noopener">List</a></h3>
Multiple tags can be passed into the search functions using a list a shown in the example below:
Example: The following code finds all the <a> and tags in the HTML document.
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">for tag in soup.find_all(['a','b']): print(tag)</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""><a class="language" href="https://docs.python.org/3/" id="python">Python</a>
<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a>
<a class="language" href="https://golang.org/doc/" id="golang">Golang</a>
Please subscribe!</pre>
<h3>❖ A <a href="https://blog.finxter.com/python-cheat-sheet-functions-and-tricks/" target="_blank" rel="noreferrer noopener">function</a></h3>
We can define a function and pass an element as its argument. The function returns <code>True</code> in case of a match, otherwise it returns <code>False</code>. 
Example: The following code defines a function which returns <code>True</code> for all <code data-enlighter-language="generic" class="EnlighterJSRAW">classes</code> that also have an <code data-enlighter-language="generic" class="EnlighterJSRAW">id </code>in the HTML document. We then pass this function to the <code data-enlighter-language="generic" class="EnlighterJSRAW">find_all()</code> method to get the desired output.
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">def func(tag): return tag.has_attr('class') and tag.has_attr('id') for tag in soup.find_all(func): print(tag)</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""><a class="language" href="https://docs.python.org/3/" id="python">Python</a>
<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a>
<a class="language" href="https://golang.org/doc/" id="golang">Golang</a></pre>
➠ Now that we have gone through the different kind of filters that we use with the search methods, we are well equipped to dive deep into the <code data-enlighter-language="generic" class="EnlighterJSRAW">find()</code> and <code data-enlighter-language="generic" class="EnlighterJSRAW">find_all()</code> methods. 
<h2><img src="https://s.w.org/images/core/emoji/13.0.0/72x72/2728.png" alt="✨" class="wp-smiley" style="height: 1em; max-height: 1em;" /> The find() Method</h2>
The <code data-enlighter-language="generic" class="EnlighterJSRAW">find()</code> method is used to search for the occurrence of the first instance of a tag with the needed name.
Syntax:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">find(name, attrs, recursive, string, **kwargs)</pre>
➠ <code data-enlighter-language="generic" class="EnlighterJSRAW">find()</code> returns an object of type bs4.element.Tag.
Example:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(soup.find('h1'), "\n")
print("RETURN TYPE OF find(): ",type(soup.find('h1')), "\n")
# note that only the first instance of the tag is returned
print(soup.find('a'))</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""><h1>Searching Parse Tree In BeautifulSoup</h1> RETURN TYPE OF find(): <class 'bs4.element.Tag'> <a class="language" href="https://docs.python.org/3/" id="python">Python</a></pre>
➠ The above operation is the same as done by the <code data-enlighter-language="generic" class="EnlighterJSRAW">soup.h1</code> or soup <code data-enlighter-language="generic" class="EnlighterJSRAW">soup.a</code> which also returns the first instance of the given tag. So what’s, the difference? The <code data-enlighter-language="generic" class="EnlighterJSRAW">find()</code> method helps us to find a particular instance of a given tag using key-value pairs as shown in the example below:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(soup.find('a',id='golang'))</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""><a class="language" href="https://golang.org/doc/" id="golang">Golang</a></pre>
<h2><img src="https://s.w.org/images/core/emoji/13.0.0/72x72/2728.png" alt="✨" class="wp-smiley" style="height: 1em; max-height: 1em;" /> The find_all() Method</h2>
We saw that the <code data-enlighter-language="generic" class="EnlighterJSRAW">find()</code> method is used to search for the first tag. What if we want to find all instances of a tag or numerous instances of a given tag within the HTML document? The <code data-enlighter-language="generic" class="EnlighterJSRAW">find_all()</code> method, helps us to search for all tags with the given tag name and returns a list of type bs4.element.ResultSet. Since the items are returned in a list, they can be accessed with help of their index.
Syntax:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">find_all(name, attrs, recursive, string, limit, **kwargs)</pre>
Example: Searching all instances of the ‘a’ tag in the HTML document.
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">for tag in soup.find_all('a'): print(tag)</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""><a class="language" href="https://docs.python.org/3/" id="python">Python</a>
<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a>
<a class="language" href="https://golang.org/doc/" id="golang">Golang</a></pre>
Now there are numerous other argument apart from the filters that we already discussed earlier. Let us have a look at them one by one.
<h3>❖ The name Argument</h3>
As stated earlier the name argument can be a string, a regular expression, a list, a function, or the value True.
Example:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">for tag in soup.find_all('p'): print(tag)</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">Learning <a class="language" href="https://docs.python.org/3/" id="python">Python</a>,
<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a> and
<a class="language" href="https://golang.org/doc/" id="golang">Golang</a>;
is fun!
Please subscribe!</pre>
<h3>❖ The keyword Arguments</h3>
Just like the <code data-enlighter-language="generic" class="EnlighterJSRAW">find()</code> method, <code data-enlighter-language="generic" class="EnlighterJSRAW">find_all()</code> also allows us to find particular instances of a tag. For example, if the <code>id</code> argument is passed, Beautiful Soup filters against each tag’s ‘id’ attribute and returns the result accordingly.
Example:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(soup.find_all('a',id='java'))</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">[<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a>]</pre>
You can also pass the attributes as dictionary key-value pairs using the <code data-enlighter-language="generic" class="EnlighterJSRAW">attrs </code>argument.
Example:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(soup.find_all('a', attrs={'id': 'java'}))</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">[<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a>]</pre>
<h3>❖ Search Using CSS Class</h3>
Often we need to find a tag that has a certain CSS class, but the attribute, <code data-enlighter-language="generic" class="EnlighterJSRAW">class</code>, is a reserved keyword in Python. Thus, using <code>class</code> as a keyword argument will give a syntax error. Beautiful Soup 4.1.2 allows us to search a CSS class using the keyword <code>class_</code>
Example:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(soup.find_all('p', class_='Secondary'))</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">[Please subscribe!]</pre>
❖ Note: The above search will allow you to search all instances of the p tag with the class “Secondary” . But you can also filter searches based on multiple attributes, using a dictionary. 
Example: 
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(soup.find_all('p', attrs={'class': 'Secondary', 'id': 'finxter'}))</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">[copyright - FINXTER]</pre>
<h3>❖ The string Argument</h3>
The string argument allows us to search for strings instead of tags.
Example:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(soup.find_all(string=["Python", "Java", "Golang"]))</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">['Python', 'Java', 'Golang']</pre>
<h3>❖ The limit Argument</h3>
The <code>find_all()</code> method scans through the entire HTML document and returns all the matching tags and strings. This can be extremely tedious and take a lot of time if the document is large. So, you can limit the number of results by passing in the <code>limit</code> argument. 
Example: There are three links in the example HTML document, but this code only finds the first two:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(soup.find_all("a", limit=2))</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">[<a class="language" href="https://docs.python.org/3/" id="python">Python</a>, <a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a>]</pre>
<h2><img src="https://s.w.org/images/core/emoji/13.0.0/72x72/2728.png" alt="✨" class="wp-smiley" style="height: 1em; max-height: 1em;" /> Other Search Methods</h2>
We have successfully explored the most commonly used search methods, i.e., <code data-enlighter-language="generic" class="EnlighterJSRAW">find </code>and <code data-enlighter-language="generic" class="EnlighterJSRAW">find_all()</code>. Beautiful Soup also has other methods for searching the parse tree, but they are quite similar to what we already discussed above. The only differences are where they are used. Let us have a quick look at these methods.
<ul>
<li>find_parents() and find_parent(): these methods are used to traverse the parse tree upwards and look for a tag’s/string’s parent(s).</li>
<li>find_next_siblings() and find_next_sibling(): these methods are used to find the next sibling(s) of an element in the HTML document.</li>
<li>find_previous_siblings() and find_previous_sibling(): these methods are used to find and iterate over the sibling(s) that appear before the current element.</li>
<li>find_all_next() and find_next(): these methods are used to find and iterate over the sibling(s) that appear after the current element.</li>
<li>find_all_previous and find_previous(): these methods are used to find and iterate over the tags and strings that appear before the current element in the HTML document.</li>
</ul>
Example:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">current = soup.find('a', id='java')
print(current.find_parent())
print()
print(current.find_parents())
print()
print(current.find_previous_sibling())
print()
print(current.find_previous_siblings())
print()
print(current.find_next())
print()
print(current.find_all_next())
print()</pre>
Output:
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">Learning <a class="language" href="https://docs.python.org/3/" id="python">Python</a>,
<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a> and
<a class="language" href="https://golang.org/doc/" id="golang">Golang</a>;
is fun! [Learning <a class="language" href="https://docs.python.org/3/" id="python">Python</a>,
<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a> and
<a class="language" href="https://golang.org/doc/" id="golang">Golang</a>;
is fun!, <body>
<h1>Searching Parse Tree In BeautifulSoup</h1>
Learning <a class="language" href="https://docs.python.org/3/" id="python">Python</a>,
<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a> and
<a class="language" href="https://golang.org/doc/" id="golang">Golang</a>;
is fun!
Please subscribe!
copyright - FINXTER
Please subscribe!
</body>, <html><head><title>Searching Tree</title></head>
<body>
<h1>Searching Parse Tree In BeautifulSoup</h1>
Learning <a class="language" href="https://docs.python.org/3/" id="python">Python</a>,
<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a> and
<a class="language" href="https://golang.org/doc/" id="golang">Golang</a>;
is fun!
Please subscribe!
copyright - FINXTER
Please subscribe!
</body></html>, <html><head><title>Searching Tree</title></head>
<body>
<h1>Searching Parse Tree In BeautifulSoup</h1>
Learning <a class="language" href="https://docs.python.org/3/" id="python">Python</a>,
<a class="language" href="https://docs.oracle.com/en/java/" id="java">Java</a> and
<a class="language" href="https://golang.org/doc/" id="golang">Golang</a>;
is fun!
Please subscribe!
copyright - FINXTER
Please subscribe!
</body></html>] <a class="language" href="https://docs.python.org/3/" id="python">Python</a> [<a class="language" href="https://docs.python.org/3/" id="python">Python</a>] <a class="language" href="https://golang.org/doc/" id="golang">Golang</a> [<a class="language" href="https://golang.org/doc/" id="golang">Golang</a>, Please subscribe!, Please subscribe!, copyright - FINXTER, copyright - FINXTER, Please subscribe!, Please subscribe!]</pre>
<h2>Conclusion</h2>
With that we come to the end of this article; I hope that after reading this article you can search elements within a parse tree with ease! Please <a href="https://blog.finxter.com/subscribe" target="_blank" rel="noreferrer noopener">subscribe</a> and <a href="https://blog.finxter.com/" target="_blank" rel="noreferrer noopener">stay tuned</a> for more interesting articles.
<h2>Where to Go From Here?</h2>
Enough theory, let’s get some practice!
To become successful in coding, you need to get out there and solve real problems for real people. That’s how you can become a six-figure earner easily. And that’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?
Practice projects is how you sharpen your saw in coding!
Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?
Then become a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.
Join my free webinar <a rel="noreferrer noopener" href="https://blog.finxter.com/webinar-freelancer/" target="_blank">“How to Build Your High-Income Skill Python”</a> and watch how I grew my coding business online and how you can, too—from the comfort of your own home.
<a href="https://blog.finxter.com/webinar-freelancer/" target="_blank" rel="noreferrer noopener">Join the free webinar now!</a>
The post <a href="https://blog.finxter.com/searching-the-parse-tree-using-beautifulsoup/" target="_blank" rel="noopener noreferrer">Searching The Parse Tree Using BeautifulSoup</a> first appeared on <a href="https://blog.finxter.com/" target="_blank" rel="noopener noreferrer">Finxter</a>.
</div>

https://www.sickgaming.net/blog/2020/12/01/searching-the-parse-tree-using-beautifulsoup/