What is Web Scraping?

Web Scraping = extracting data from regular web pages

• No standard format - every website is different
• Pages can change at any time - not a stable data source

Understanding HTML

HTML = Hyper Text Markup Language

• Content displayed from your web browser is contained in html files
• HTML is a markup language, close to XML, but with less constraints
• The browser is highly fault tolerant - not many websites would display at all if the browser strictly enforced standards

HTML Structure Example

HTML elements have opening and closing tags:

<article class="product_pod">
    <h3><a href="..." title="A Light in the Attic">A Light in...</a></h3>
    <p class="price_color">£51.77</p>
    <p class="star-rating Three">
        <i class="icon-star"></i>
    </p>
</article>

We can target elements using CSS selectors or XPath expressions.

DOM vs HTML

What you see is not always what you fetch!

books.toscrape.com - A Safe Scraping Sandbox

• books.toscrape.com - a fictional bookstore designed for learning
• Stable HTML structure, no JavaScript required
• 1000 books with titles, prices, and ratings
• No robots.txt restrictions - built for scraping practice

Scraping with BeautifulSoup

import requests
from bs4 import BeautifulSoup

url = "http://books.toscrape.com"
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")

# Find all book elements
books = soup.select("article.product_pod")
for book in books[:5]:
    title = book.select_one("h3 a")["title"]
    price = book.select_one(".price_color").text
    print(f"{title}: {price}")

Same Task with lxml and XPath

from lxml import html
import requests

response = requests.get("http://books.toscrape.com")
tree = html.fromstring(response.content)

# XPath to select all book titles and prices
titles = tree.xpath("//article[@class='product_pod']//h3/a/@title")
prices = tree.xpath("//p[@class='price_color']/text()")

for title, price in zip(titles[:5], prices[:5]):
    print(f"{title}: {price}")

Exercise: Manual Data Extract

Goal: Extract HDI data as fast as possible

1. Navigate to Wikipedia HDI page
2. Think about the best way to extract the HDI table
3. Produce a CSV from it as fast as you can

Limits of Direct HTML Fetching

• Fetching the HTML page is more reliable when it works
• Sometimes JavaScript is required to render content
• In that case you need "headless" browsing - simulating browser interaction

# Using Selenium for JavaScript-heavy sites
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome(options=options)
driver.get("https://example.com")
# Now you can access JavaScript-rendered content

When to Use Scraping?

Short answer: when you cannot do otherwise

Longer answer: beware of the automation cost - verify that it is worth it!

• Use when data volume is too large to do manually
• Limit the effort to reference data that won't change often
• Always check if an API exists first!

Integrating Scraped Data with Orange

import requests
from bs4 import BeautifulSoup
from Orange.data import Table, Domain, StringVariable, ContinuousVariable

# Scrape book data
response = requests.get("http://books.toscrape.com")
soup = BeautifulSoup(response.content, "html.parser")

# Extract data
data = []
for book in soup.select("article.product_pod"):
    title = book.select_one("h3 a")["title"]
    price = float(book.select_one(".price_color").text[1:])
    data.append([title, price])

# Create Orange table
domain = Domain([StringVariable("title")], [ContinuousVariable("price")])
out_data = Table.from_list(domain, data)

Exercise: Scrape and Analyze Book Data

Scrape books.toscrape.com and analyze the data in Orange: