[Tut] Scrape a Bookstore in 5 Steps Python [Learn Project]

[Tut] Scrape a Bookstore in 5 Steps Python [Learn Project] - Printable Version

+- Sick Gaming (https://sickgaming.net)
+-- Forum: Programming (https://sickgaming.net/forum-76.html)
+--- Forum: Python (https://sickgaming.net/forum-83.html)
+--- Thread: [Tut] Scrape a Bookstore in 5 Steps Python [Learn Project] (/thread-99586.html)

[Tut] Scrape a Bookstore in 5 Steps Python [Learn Project] - xSicKxBot - 06-17-2022

Scrape a Bookstore in 5 Steps Python [Learn Project]

5/5 – (1 vote)

Story: This series of articles assume you work in the IT Department of Mason Books. The Owner asks you to scrape the website of a competitor. He would like this information to gain insight into his pricing structure.

Note: Before continuing, we recommend you possess, at minimum, a basic knowledge of HTML and CSS and have reviewed our articles on How to Scrape HTML tables.

What You’ll Build in This Project

Let’s navigate to Books to Scrapeand review the format.

[Image: kmc-books-01a-1024x564.png]

At first glance, you will notice:

Book categories display on the left-hand side.
There are, in total, 1,000 books listed on the website.
Each web page shows 20 Books.
Each price is in £ (in this instance, the UK pound).
Each Book displays minimum details.
To view complete details for a book, click on the image or the
Code:
Book Title
hyperlink. This hyperlink forwards to a page containing additional book details for the selected item (see below).
The total number of website pages displays in the footer (
Code:
Page 1 of 50
).

Step 1: Install and Import Libraries for Project

Before any data manipulation can occur, three (3) new libraries will require installation.

The Pandas library enables access to/from a DataFrame.
The Requests library provides access to the HTTP requests in Python.
The Beautiful Souplibrary enables data extraction from HTML and XML files.

To install these libraries, navigate to an IDE terminal. At the command prompt (

Code:
$

), execute the code below. For the terminal used in this example, the command prompt is a dollar sign (

Code:
$

). Your terminal prompt may be different.

Code:
$ pip install pandas

Hit the

Code:
<Enter>

key on the keyboard to start the installation process.

Code:
$ pip install requests

Hit the

Code:
<Enter>

key on the keyboard to start the installation process.

Code:
$ pip install beautifulsoup4

Hit the

Code:
<Enter>

key on the keyboard to start the installation process.

If the installations were successful, a message displays in the terminal indicating the same.

Feel free to view the PyCharm installation guides for the required libraries.

Add the following code to the top of each code snippet. This snippet will allow the code in this article to run error-free.

Code:
import pandas as pd
import requests
from bs4 import BeautifulSoup
import time
import urllib.request
from csv import reader, writer

The
Code:
time
library is built-in with Python and does not require installation. This library contains
Code:
time.sleep()
and is used to set a delay between page scrapes.
The
Code:
urllib
library is built-in with Python and does not require installation. This library contains
Code:
urllib.request
and is used to save images.
The
Code:
csv
library is built-in
Code:
Pandas
and does not require additional installation. This library contains
Code:
reader and writer
methods to save data to a CSV file.

Step 2: Understand Basics and Scrape Your First Results

[Image: kmc-books-04a.png]

In this step, you’ll perform the following tasks:

Reviewing the website to scrape.
Understanding HTTP Status Codes.
Connecting to the Books to Scrape website using the
Code:
requests
library.
Retrieving Total Pages to Scrape
Closing the Open Connection.

Learn More: Learn everything you need to know to reproduce this step in the in-depth Finxter blog tutorial.

Step 3: Configure URL to Scrape and Avoid Spamming the Server

[Image: image-122.png]

Rule: Don’t Spam the Server!

In this step, you’ll perform the following tasks:

Configuring a page URL for scraping
Setting a delay:
Code:
time.sleep()
to pause between page scrapes.
Looping through two (2) pages for testing purposes.

Learn More: Learn everything you need to know to reproduce this step in the in-depth Finxter blog tutorial.

Step 4: Save Book Details in a Python List

[Image: image-123-1024x709.png]

In this step, you’ll perform the following tasks:

Locating Book details.
Writing code to retrieve this information for all Books.
Saving
Code:
Book
details to a List.

Learn More: Learn everything you need to know to reproduce this step in the in-depth Finxter blog tutorial.

Step 5: Clean and Save the Scraped Output

[Image: image-124-1024x340.png]

In this step, you’ll perform the following tasks:

Cleaning up the scraped code.
Saving the output to a CSVfile.

Learn More: Learn everything you need to know to reproduce this step in the in-depth Finxter blog tutorial.

Conclusion

This tutorial has guided you through the steps to create your first practical web scraping project: scraping the contents of a book store!

Now, go out and use your skills wisely and to the benefit of humanity, my friend!