Practical Work 5: Mini exam: a full run, from question to handed-in report

Objectives

By the end of this session you will be able to:

Frame a question before touching a model
Seal a test set and understand what sealing forbids
Always produce a baseline before any real model
Read a confusion matrix and an error map, not just a score
Produce a report that a non-specialist can read
Work under time pressure without cutting the corner that matters

Prerequisites

A rakoon-ds account on https://rakoon-ds.apps.way-up.io (free, browser only, nothing to install), and the group join code your instructor gives you
A recent Chrome, Edge or Firefox. Computation runs either in your own tab (browser engine) or on the server; the header pill tells you which
The mission for this session, assigned to your group. Open a project, then the Mission button in the workshop header: the panel opens next to Report. Click Check after each step
Practical works 1 to 4 finished
A dataset you already know: the California fires of Practical works 2 and 4, or the Titanic of Practical work 1. Do not discover a dataset during an exam
Transformation, chart and algorithm names come from the rakoon-ds registry and are served in French even when the interface is in English. Every step below gives you the French label you will click and, in code font, the registry key the mission checks against.

Data

Use the dataset you prepared in Practical work 2 or 4, in a new project. Rebuilding the chain from the raw CSV is part of the exercise: if you cannot rebuild it in fifteen minutes, you did not understand it.

Instructors: this is the session where the group and the mission progress table are used for real. Create the group, hand out the join code, assign dep-5, and watch the progress page while the room works.

Timing

The steps below add up to the announced duration. If you fall behind, Step 1 to Step 3 are the ones that must be finished.

#	What you do	Time
Step 1	The question, and the sealed test set	12 min
Step 2	Prepare, and write down every decision	15 min
Step 3	The baseline, before anything else	8 min
Step 4	Two real models	15 min
Step 5	Where is it wrong?	13 min
Step 6	Hand in	12 min
Total		75 min

Instructions

Step 1: The question, and the sealed test set12 min

New project PW5 Mini exam. Import the raw CSV and rebuild the cleaning chain (drop, dates, date parts). Fifteen clicks.
Open the Report panel and write the Question et contexte section first: what is being predicted, for whom, and what would count as good enough. Six lines minimum.
Family Lignes, card Découper (proportions) (split): proportions 0.8, 0.2, part 1, shuffle oui. That is your working set. Repeat with part 2: that is your test set.
Click the test set node and press Sceller (Seal). A padlock appears on the node.

A sealed dataset refuses preview, profile, quality, charts and download with a 423 error, and stays perfectly usable as the test set of a model. It is a discipline, not a security barrier: you can unseal it whenever you like, which is exactly the point. It stops you from looking by accident.

Step 2: Prepare, and write down every decision15 min

Work only on the 80 % part. Everything you do here you will have to justify in the report, so write as you go.

Missing values: Remplir les valeurs manquantes (fillna) or Supprimer les lignes vides (dropna). Which columns, which strategy, and how many rows it costs.
Derived columns: at least one, with Colonne par formule (expression), Catégoriser (seuils) (categorize) or Discrétiser (bins) (bin_numeric).
Anything you drop: name it and give the reason in one clause.

Then write the Préparation section. Ten lines.

Step 3: The baseline, before anything else8 min

Train on your prepared dataset. Algorithm Modèle de référence (classe majoritaire) (dummy_clf) (or Modèle de référence (médiane) (dummy_reg) for a regression).
In the training form, set Jeu de test to your sealed dataset instead of using the random split.
Note the score. That number is the bar. Every model you build after this is judged against it, not against zero.

rakoon-ds makes the baseline an algorithm like any other, on purpose. In Orange it is a discipline you have to remember; here it is one click, and there is no excuse for not having one.

Step 4: Two real models15 min

Train a Forêt aléatoire (rf_clf) with the same target, the same features, the same sealed test set. Same experiment label as the baseline.
Train a second, different family: Gradient Boosting (gb_clf), Régression logistique (logreg) or Arbre de décision (tree_clf).
Open the Comparison panel from the workshop header: one row per run, the metrics side by side.

If a score jumps above what you expected, look at the banner above the metrics before you celebrate. leak_duplicate, leak_association, leak_name, id_feature and resample_before_split all mean the same thing in practice: the number on screen is not the number you will get in production.

Step 5: Where is it wrong?13 min

Open the results of your best model and scroll past the metrics. The extension tabs are below the confusion matrix, the ROC curve and the importance chart. They are worth the scroll.

Capture vs truth (classification): every test individual in the projection, filled when correct, circled when wrong; a clickable confusion matrix; a threshold slider on a binary target.
Error map (regression): actual against predicted, residuals, and the twenty worst-predicted individuals with their values.
Quality: the same report as the explorer, filtered on this model.
Coach: the rules that fired on this result, and the one-click remedy when there is one.

Find one individual the model got badly wrong. Write down its values.
Say in one sentence why you think the model missed it.
Write the Évaluation section: the metric you chose and why, the baseline, your score, and that one individual.

Step 6: Hand in12 min

Pour into the report: the metrics block, the confusion matrix (or the error map), the variable importance, and the Démarche (lignage) block, which draws your whole chain.
Click Journal at the top of the panel and insert the automatic log into the Préparation section: it rebuilds the full chronology of what you did, with the parameters.
Write the Conclusion et limites section: what you would claim, with what confidence, and what you would not claim.
Click Lecture to read the whole thing as a document, then Exporter le rapport → Markdown + images (.zip).
Click Check in the mission panel one last time.

Your instructor now opens the mission progress page and sees, per student: the working project, which steps are green, when, and how many hints were used. Nobody sends a file.

What you should have

A sealed test set used by every model
At least three trained models, one of which is a baseline
Five of the seven report sections written
One badly predicted individual, named and explained
The dep-5 mission at 6 / 6

Deliverables

Mission: dep-5 validated
Report: exported as Markdown + images (.zip)
The comparison table of your models, pasted into the Évaluation section

Bonus

Unseal the test set, look at it, then re-seal it. Write one honest paragraph on what you would have changed if you had looked before measuring. That paragraph is the whole reason sealing exists.
Run a 5-fold cross-validation from the Diagnostics tab. If the standard deviation across folds is larger than the gap between your two models, what have you actually shown?
Designate your best model as champion and download its model card (Fiche tab, Download (Markdown)). Read the "limits" section: it is computed, not recited.

Resources

Session 5 slides (the lecture this practical work follows)
Course page: both programmes, all fifteen sessions
rakoon-ds studio
missions/dep-5.json
Sessions 1 to 4 slides, all of them: this is the session that uses everything