Web Scraping with BeautifulSoup

Reviewed & published by Brayan K

When a website has data you want but no API, web scraping lets you download its HTML and extract exactly what you need. Combined with the requests library you already know, BeautifulSoup turns a tangled page into clean, structured data.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

You'll learn to parse HTML, find elements by tag and class, follow links, and scrape responsibly.

What You'll Learn in This Lesson

• How HTML structure maps to a parse tree
• Downloading pages with requests
• find vs find_all to select elements
• Reading text, attributes (href, src) and classes
• CSS selectors with select()
• Scraping ethics: robots.txt and rate limits

1 HTML Is Just a Tree

Every web page is HTML — nested tags forming a tree. Scraping means navigating that tree to find the branches you care about:

<div class="product">
  <h2 class="title">Wireless Mouse</h2>
  <span class="price">$24.99</span>
  <a href="/products/42">View details</a>
</div>

🌳 The vocabulary you need:

• Tag — an element like <h2> or <a>
• Attribute — extra info on a tag, like class="price" or href="..."
• Text — the visible content between tags ("Wireless Mouse")
• Class — a label used to group/style elements; your main hook for finding things

Tip: in your browser, right-click any element and choose "Inspect" to see its tag and class. That's how you find the selectors to scrape.

2 Download, Then Parse

Scraping is always two steps: requests fetches the HTML, then BeautifulSoup parses it into a searchable tree:

# pip install requests beautifulsoup4
import requests
from bs4 import BeautifulSoup

# 1) Download the page
resp = requests.get("https://example.com", timeout=10)
resp.raise_for_status()

# 2) Parse the HTML into a navigable tree
soup = BeautifulSoup(resp.text, "html.parser")

# 3) Now you can search it
print(soup.title.text)          # 'Example Domain'
print(soup.h1.text)             # 'Example Domain'
print(soup.find("p").text)      # the first paragraph's text

"html.parser" is Python's built-in parser (no extra install). Some scrapers prefer "lxml" for speed — both work the same way once the soup is built.

🔎 find and find_all — Your Two Workhorses

These examples embed the HTML directly, so they run anywhere BeautifulSoup is installed — no network needed:

3 Reading Text and Attributes

You'll often want a tag's text and its attributes (like a link's href or an image's src):

from bs4 import BeautifulSoup

html = '<a href="/products/42" data-id="42">Wireless Mouse</a>'
soup = BeautifulSoup(html, "html.parser")
link = soup.find("a")

print(link.text)            # Wireless Mouse  (the visible text)
print(link["href"])         # /products/42    (an attribute, dict-style)
print(link.get("href"))     # /products/42    (safer: returns None if missing)
print(link["data-id"])      # 42
print(link.attrs)           # {'href': '/products/42', 'data-id': '42'}

Use tag["attr"] when you're sure the attribute exists, and tag.get("attr") when it might be missing — get returns None instead of raising a KeyError.

🎯 CSS Selectors with select()

If you know CSS, select() lets you write the same selectors you'd use for styling. It's often more concise than chaining find calls:

from bs4 import BeautifulSoup

html = '''
<div class="card"><h3>Alpha</h3><span class="tag">new</span></div>
<div class="card"><h3>Beta</h3><span class="tag">sale</span></div>
'''
soup = BeautifulSoup(html, "html.parser")

# select returns a LIST (like find_all)
for h in soup.select("div.card h3"):     # 'h3 inside div.card'
    print(h.text)                        # Alpha, then Beta

# select_one returns the FIRST match (like find)
print(soup.select_one(".tag").text)      # new

📖 Worked Example: A Complete Little Scraper

Everything above was one piece at a time. This is the whole job end to end, in the shape almost every real scraper takes: build the soup → loop over the repeated cards → pull text and attributes out of each one → guard against the missing ones → do something with the results.

The HTML is embedded in the code, so it runs offline once you have beautifulsoup4 installed. Note the third event deliberately has no booking link — real pages are inconsistent, and your code has to survive that.

from bs4 import BeautifulSoup

# The HTML is embedded, so this runs with no internet: pip install beautifulsoup4
html = '''
<div id="listings">
  <div class="event">
    <h3 class="name">Python Meetup</h3>
    <span class="city">Leeds</span>
    <span class="seats">12</span>
    <a class="book" href="/book/101">Book</a>
  </div>
  <div class="event">
    <h3 class="name">Data Night</h3>
    <span class="city">Bristol</span>
    <span class="seats">0</span>
    <a class="book" href="/book/102">Book</a>
  </div>
  <div class="event">
    <h3 class="name">Async Workshop</h3>
    <span class="city">Leeds</span>
    <span class="seats">5</span>
  </div>
</div>
'''

soup = BeautifulSoup(html, "html.parser")   # step 1: text becomes a searchable tree

events = []
for card in soup.select("div.event"):       # step 2: one pass per repeated item
    name = card.select_one("h3.name").text.strip()      # .text is the visible content
    city = card.select_one("span.city").text.strip()
    seats = int(card.select_one("span.seats").text)     # .text is ALWAYS a string

    link_el = card.select_one("a.book")     # the third card has no link at all
    url = link_el.get("href") if link_el else None      # .get gives None, never KeyError

    events.append({"name": name, "city": city, "seats": seats, "url": url})

print("Parsed", len(events), "events")
for e in events:
    status = f"{e['seats']} seats" if e["seats"] else "SOLD OUT"
    print(f"{e['name']:16}{e['city']:10}{status:10}{e['url']}")

leeds = [e for e in events if e["city"] == "Leeds"]
print("Leeds events:", [e["name"] for e in leeds])
print("Total seats left:", sum(e["seats"] for e in events))

# ✅ Expected output:
# Parsed 3 events
# Python Meetup   Leeds     12 seats  /book/101
# Data Night      Bristol   SOLD OUT  /book/102
# Async Workshop  Leeds     5 seats   None
# Leeds events: ['Python Meetup', 'Async Workshop']
# Total seats left: 17

Two habits worth stealing: int(...) around anything numeric (HTML text is always a string, so "12" would sort before "5"), and .get("href") rather than ["href"] whenever a tag might be missing.

🎯 Your Turn: Scrape a Job Board

This job board has a trap in it: one <li> is an advert, not a job, and one real job has no "apply" link yet. Fill in the three ___ blanks so your code picks out only the jobs and never crashes on the missing link.

from bs4 import BeautifulSoup

# 🎯 YOUR TURN — replace the three ___ blanks

html = '''
<ul class="board">
  <li class="job"><span class="role">Backend Dev</span><a href="/jobs/1">apply</a></li>
  <li class="job"><span class="role">Data Analyst</span><a href="/jobs/2">apply</a></li>
  <li class="promo">Advertise your company here</li>
  <li class="job"><span class="role">QA Engineer</span></li>
</ul>
'''
soup = BeautifulSoup(html, "html.parser")

# 1) Collect EVERY <li> whose class is "job" — the promo one must not appear.
jobs = soup.___("li", ___="job")   # 👉 the method that returns a LIST, and the keyword that filters by CSS class

for job in jobs:
    role = job.find("span", class_="role").text
    link_el = job.find("a")        # None for the job with no link

    # 2) Read the href safely, so a missing link is a message rather than a crash.
    url = link_el.___("href") if link_el else "not open yet"   # 👉 returns None instead of raising KeyError

    print(f"{role:14}{url}")

print("Jobs found:", len(jobs))

# ✅ Expected output:
# Backend Dev   /jobs/1
# Data Analyst  /jobs/2
# QA Engineer   not open yet
# Jobs found: 3

1) find_all — find would return only the first job.

2) class_ — with the trailing underscore, because class is a reserved Python keyword.

If "Jobs found" is 4, you searched for every <li> and swept up the advert. If you got AttributeError: 'NoneType' object has no attribute 'text', you looked for a role inside the promo item.

from bs4 import BeautifulSoup

# HTML embedded inline so this runs after: pip install beautifulsoup4
html = '''
<table id="leaderboard">
  <tr><td class="name">Maya</td><td class="pts">120</td></tr>
  <tr><td class="name">Leo</td><td class="pts">95</td></tr>
  <tr><td class="name">Zara</td><td class="pts">140</td></tr>
</table>
'''
soup = BeautifulSoup(html, "html.parser")

players = []
for row in soup.find_all("tr"):
    name = row.find("td", class_="name").text
    pts = int(row.find("td", class_="pts").text)
    players.append((name, pts))

# Sort by points, highest first
players.sort(key=lambda p: p[1], reverse=True)
print("=== Leaderboard ===")
for rank, (name, pts) in enumerate(players, start=1):
    print(f"{rank}. {name}: {pts} pts")

# ✅ Expected output:
# === Leaderboard ===
# 1. Zara: 140 pts
# 2. Maya: 120 pts
# 3. Leo: 95 pts

Real-World Example: A Product Listing Scraper

A realistic scraper: download a page, loop over product cards, extract name, price, and link, and build clean records. This is the shape of nearly every real scraping job:

Notice the defensive if not (...) check — real pages have broken or missing cards. A scraper that assumes perfect HTML crashes on the first oddity.

Scrape Responsibly

Scraping carries real responsibilities. Be a good citizen of the web:

• Check robots.txt (e.g. example.com/robots.txt) and the site's Terms of Service first.
• Prefer an official API if one exists — it's faster, stabler, and explicitly allowed.
• Rate-limit yourself — add time.sleep(1) between requests so you don't hammer the server.
• Identify your bot with a clear User-Agent string.
• Never collect personal data or republish copyrighted content.

import time

for page in range(1, 6):
    url = f"https://example.com/list?page={page}"
    # ... scrape the page ...
    time.sleep(1)        # polite 1-second pause between requests

from bs4 import BeautifulSoup

# Extract every link and its text from a navigation menu
html = '''
<nav>
  <a href="/home">Home</a>
  <a href="/about">About Us</a>
  <a href="/contact">Contact</a>
  <a href="https://blog.example.com">Blog</a>
</nav>
'''
soup = BeautifulSoup(html, "html.parser")

print("Found links:")
for a in soup.find_all("a"):
    text = a.text.strip()
    href = a.get("href")
    kind = "external" if href.startswith("http") else "internal"
    print(f"  {text:10} -> {href:25} ({kind})")

# ✅ Expected output:
# Found links:
#   Home       -> /home                     (internal)
#   About Us   -> /about                    (internal)
#   Contact    -> /contact                  (internal)
#   Blog       -> https://blog.example.com  (external)

🏆 Mini-Challenge: Book Price Report

Outline only — the logic is yours. Turn this little catalogue into a price report: list every book with its price, name the cheapest one, and print the average. The one thing that trips people up is that "£44.50" is text, not a number — strip the currency symbol before converting.

from bs4 import BeautifulSoup

html = '''
<div class="catalogue">
  <div class="book"><h4 class="title">Fluent Python</h4><span class="gbp">£44.50</span></div>
  <div class="book"><h4 class="title">Automate the Boring Stuff</h4><span class="gbp">£22.99</span></div>
  <div class="book"><h4 class="title">Python Crash Course</h4><span class="gbp">£28.00</span></div>
  <div class="book"><h4 class="title">Think Python</h4><span class="gbp">£31.75</span></div>
</div>
'''

# 🎯 MINI-CHALLENGE: book price report
# 1. Build the soup with "html.parser"
# 2. Loop over soup.select("div.book"). For each card collect:
#      title = the text of h4.title, stripped
#      price = the text of span.gbp with the "£" removed, then float(...)
#    Append (title, price) to a list called books
# 3. Print one line per book:  f"{title:28}£{price:.2f}"
# 4. Print the cheapest title — min(books, key=lambda b: b[1])[0]
# 5. Print the average price to 2 decimal places
#
# ✅ Expected output:
# Fluent Python               £44.50
# Automate the Boring Stuff   £22.99
# Python Crash Course         £28.00
# Think Python                £31.75
# Cheapest: Automate the Boring Stuff
# Average: £31.81

# your code here

soup = BeautifulSoup(html, "html.parser")

books = []
for card in soup.select("div.book"):
    title = card.select_one("h4.title").text.strip()
    price = float(card.select_one("span.gbp").text.replace("£", ""))
    books.append((title, price))

for title, price in books:
    print(f"{title:28}£{price:.2f}")

cheapest = min(books, key=lambda b: b[1])
average = sum(p for _, p in books) / len(books)
print("Cheapest:", cheapest[0])
print(f"Average: £{average:.2f}")

Skip the float(...) and you are comparing text, which sorts character by character. Add a £9.99 book and sorted() puts it last — '£9.99' > '£44.50' is True in Python, because "9" comes after "4". Convert to numbers before you compare anything.

🧩 Reorder Challenge

These lines fetch a page and print its main heading, but they're scrambled. Find the order:

A) soup = BeautifulSoup(resp.text, "html.parser")
B) import requests
C) from bs4 import BeautifulSoup
D) print(soup.find("h1").text)
E) resp = requests.get("https://example.com", timeout=10)

Correct order: B → C → E → A → D

import requests
from bs4 import BeautifulSoup
resp = requests.get("https://example.com", timeout=10)
soup = BeautifulSoup(resp.text, "html.parser")
print(soup.find("h1").text)

Imports first, download with requests, build the soup from resp.text, then search the tree. You can't parse before you've downloaded.

🧠 Quick Recall — Predict the Output

from bs4 import BeautifulSoup
soup = BeautifulSoup("<p>Hi</p><p>Bye</p>", "html.parser")
print(len(soup.find_all("p")))

2 — find_all returns a list of every matching tag; there are two <p> elements.

from bs4 import BeautifulSoup
soup = BeautifulSoup('<a href="/x">Go</a>', "html.parser")
print(soup.find("a")["href"])

/x — indexing a tag with an attribute name (["href"]) returns that attribute's value, not the text. The text would be .text giving "Go".

from bs4 import BeautifulSoup
soup = BeautifulSoup("<div>hello</div>", "html.parser")
print(soup.find("span"))

None — there's no <span> in the HTML, and find returns None when nothing matches. Calling .text on that None would raise an AttributeError, which is why you guard with if checks.

❓ Frequently Asked Questions

You can turn any web page into data!

You can download HTML with requests, parse it with BeautifulSoup, select elements with find/find_all and CSS selectors, read text and attributes, and scrape ethically. This is one of the most practical skills in a Python toolkit.

Practice quiz

Continue this course

• Previous: Reading CSV & Excel Files
• Next: itertools — Iteration power tools — chain, groupby, combinations and more
• Quick reference: Python cheat sheet

Frequently asked questions

What is web scraping?

Web scraping is extracting structured data from web pages by downloading their HTML and parsing it. It's how you collect data from sites that don't offer an API — prices, listings, articles, and more.

What is BeautifulSoup?

BeautifulSoup is a third-party Python library (pip install beautifulsoup4) that parses HTML into a navigable tree of objects. You then use methods like find and find_all with CSS-style selectors to pull out the elements you want.

What is the difference between find and find_all?

find returns the FIRST matching element (or None). find_all returns a list of ALL matching elements. Use find for a single unique element like a page title, find_all to collect repeated items like a list of products.

Is web scraping legal?

It depends. Always check the site's robots.txt and Terms of Service, scrape only public data, add delays between requests so you don't overload the server, and prefer an official API if one exists. Never scrape personal data or copyrighted content for redistribution.

Why does my scraper return None or an empty list?

Usually the page loads its content with JavaScript after the initial HTML, so requests + BeautifulSoup never see it. For JavaScript-heavy sites you need a browser-automation tool like Selenium or Playwright. Also double-check your tag names, class names, and that the page structure hasn't changed.

Related lessons

• HTTP Requests & APIs