Scrape in Python, analyse in rakoon-ds

Step 1: Look before you scrape10 min

1. Open catalogue/page-1.html in your browser and use the element inspector on one book.
2. Find the container: article.product_pod. Twenty per page.
3. Inside it, find where each field lives:

Field	Selector	Note
title	h3 a, attribute title	the visible text is truncated with an ellipsis, the attribute is not
price	p.price_color	text is £51.77
rating	p.star-rating, second CSS class	One to Five, as a word
availability	p.instock	text, with whitespace
link	h3 a, attribute href	relative, starts with ../ on some pages

Check robots.txt: http://books.toscrape.com/robots.txt returns 404. No file means no stated rules, which is not the same thing as permission. On this site, permission is explicit: it exists to be scraped. On any other site, read the terms of use.

Step 2: Write the scraper20 min

Type it, do not paste it. Every line of this script is a decision you will have to defend.

"""Scrape books.toscrape.com into a CSV that rakoon-ds can import.

Le site existe pour cela : c'est un bac a sable public, sans robots.txt.
Sur un vrai site, on lit d'abord robots.txt et les conditions d'utilisation.
"""
import csv
import re
import time

import requests
from bs4 import BeautifulSoup

BASE = "http://books.toscrape.com/catalogue/"
STARS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
rows = []

for page in range(1, 100):                      # borne haute : on s'arrete sur un 404
    r = requests.get(f"{BASE}page-{page}.html", timeout=20)
    if r.status_code == 404:                    # la page 51 n'existe pas : 50 pages, 1000 livres
        break
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")
    pods = soup.select("article.product_pod")
    if not pods:
        break
    for pod in pods:
        link = pod.select_one("h3 a")
        price = pod.select_one("p.price_color").get_text(strip=True)
        stars = pod.select_one("p.star-rating")["class"][1]      # ex. ["star-rating", "Three"]
        rows.append({
            "title": link["title"],
            "price_gbp": float(re.sub(r"[^0-9.]", "", price)),   # "£51.77" -> 51.77
            "rating": STARS.get(stars, 0),
            "availability": pod.select_one("p.instock").get_text(strip=True),
            "url": BASE + link["href"].replace("../", ""),
        })
    print(f"page {page} : {len(pods)} livres, total {len(rows)}")
    time.sleep(0.5)                             # politesse : on ne martele pas le serveur

with open("books.csv", "w", newline="", encoding="utf-8") as fh:
    writer = csv.DictWriter(fh, fieldnames=list(rows[0]))
    writer.writeheader()
    writer.writerows(rows)

print(f"{len(rows)} livres ecrits dans books.csv")

The four decisions worth naming:

• Stop condition: a 404 on the next page, not a hard-coded 50. The site may grow.
• Timeout: 20 seconds. Without it, one hung request hangs the whole run.
• Rate limit: half a second between pages. Fifty pages, twenty-five seconds. A scraper without a sleep is a small denial of service.
• Type conversion at the source: £51.77 becomes 51.77, Three becomes 3. Doing it in Python is one line; doing it later in the studio costs two transformations.

Step 3: Run it, and check the CSV10 min

1. Run the script. Expect 50 lines of progress and 1000 livres ecrits dans books.csv.
2. Open the CSV in a text editor, not in a spreadsheet. Check the header, the first row, the last row.
3. Count the lines: wc -l books.csv should give 1001 (1 000 books plus the header). If a title contained a newline, this count would lie, exactly as it does on the California fire file of Practical work 2.

Step 4: Import and profile10 min

1. New project PW10 Books.
2. Add a dataset → ...or import a file → books.csv.
3. Check the node: 1 000 x 5.
4. Explore → Statistics: is price_gbp read as a number? Is rating?
5. Quality tab: any duplicates? any missing values? Is title flagged as an identifier?

If a numeric column arrived as text, that is what Convertir en nombre (to_number) is for, with a decimal separator selector for files written with commas. Fixing it in the scraper is better; fixing it here is possible.

Step 5: Clean and enrich, with clicks only15 min

1. Catégoriser (seuils) (categorize) on price_gbp, thresholds 20, 40, labels cheap, mid, expensive.
2. Discrétiser (bins) (bin_numeric) on price_gbp, quantile, 4 buckets. Compare the two results: the thresholds you chose against the thresholds the data chose.
3. Pin a Histogramme (histogram) of price_gbp, and a Comptage / moyenne (barres) (bar) with column = rating, value = price_gbp, aggregation mean.
4. Answer, from the chart: do five-star books cost more? Would you have bet on that before looking?

Then a model, because you can: train a Forêt aléatoire (rf_clf) to predict your price category from rating and availability, with a Modèle de référence (classe majoritaire) (dummy_clf) beside it.

Step 6: The ethics section, and it is not optional10 min

You collected this data. Write the Données and Conclusion et limites sections of the report and answer all of these in writing:

1. Where does the data come from, when did you collect it, and would you get the same thing tomorrow?
2. Did the site allow it? How do you know? What would you have checked on a commercial site?
3. What load did you put on the server, and how did you limit it?
4. Does the data contain anything personal? What would change if it did?
5. What can you legitimately publish: the numbers, the aggregates, the raw file?
6. What is this dataset not evidence of? (a generated catalogue is not a book market)