Data quality in depth: holes, duplicates, outliers, identifiers, leaks

Step 1: Missing values, and the pattern block nobody reads12 min

1. New project PW6 Quality. Add the built-in Titanic (passagers).
2. Explore → Quality, target survivant.
3. In the Missing values card, read the per-column bars, then scroll to Most frequent patterns.

Reference figures: age 263 missing (20.1 %), cabine 1 014 (77.5 %), canot 823 (62.9 %). And the patterns:

Columns empty together	Rows	Share
cabine + canot	538	41.1 %
cabine alone	235	18.0 %
age + cabine + canot	182	13.9 %
canot alone	90	6.9 %

Answer in writing, in two sentences: what kind of passenger is described by the first pattern, and what does that imply about imputing age with the overall mean?

Step 2: Duplicates: measure them, do not assume15 min

1. Add the built-in Qualité du vin rouge (wine_quality).
2. Quality tab, target qualite. The Duplicates card reports 240 extra rows (15.0 %), 220 groups, 1 359 distinct rows, and lists the row numbers of each group.
3. Train a Forêt aléatoire (rf_clf) on qualite with every column. Note the accuracy.
4. Apply Supprimer les doublons (drop_duplicates) (family Lignes, no column selected = all columns). You get 1 359 rows.
5. Train the same forest on the deduplicated dataset. Note the accuracy again.

Dataset	Rows	Accuracy	F1
wine_quality as delivered	1 599	0.6725	0.6506
after drop_duplicates	1 359	0.5706	0.5504

Also read the Class balance card here: 681 wines rated 5, 10 wines rated 3, a majority-to-minority ratio of 68.1, and three classes flagged rare (4 with 53 rows, 8 with 18, 3 with 10). What does global accuracy tell you about a class that holds 0.6 % of the data?

Step 3: Outliers: two methods, two viewpoints12 min

Still on wine_quality, Outliers card. Each numeric column gets two counts side by side: outside the Tukey (IQR) bounds, and |z| > 3. Plus the bounds, the quartiles, and up to five example values with their row number.

Column	Outside IQR bounds
sucre_residuel	155
chlorures	112
sulfates	59
so2_total	55
acidite_fixe	49

1. Pick sucre_residuel. Click through to two of the example rows in Overview. Are they errors, or are they sweet wines?
2. Decide, and act: keep them, cap them with Filtrer des lignes (filter_rows), or bin them with Discrétiser (bins) (bin_numeric) so the extreme values fall into one bucket.
3. Write down the decision and the reason. "I removed the outliers" is not a reason.

Step 4: Identifiers, and what the studio drops behind your back10 min

A column with almost as many distinct values as rows cannot generalise. rakoon-ds does two things about it, and you need to know both.

1. On titanic, Quality tab: nom is listed under the points to watch, 1 307 distinct values out of 1 309, ratio 0.998.
2. Train any classification model on titanic and open its results. Look for the list of variables actually used.
3. You will find that nom and cabine were disabled automatically and that the studio says so.

One subtlety worth remembering: a continuous column whose values are all distinct is not treated as an identifier. Three random coordinates are not three row numbers. Only text, categorical and integer columns can be flagged.

Step 5: Leaks: the five it catches, and the one it misses16 min

rakoon-ds checks for leaks before training and shows a banner above the metrics. Five codes:

Code	Severity	What it means
leak_duplicate	high	the column is the target: same values, or a bijective recoding of it
leak_association	high	mixed association with the target at 0.98 or more
leak_name	to fix	the column name contains the target name (prix_final_estime when predicting prix)
id_feature	to fix	a near-unique feature column
constant_target	high	the target is constant, or 99 % one class

Now the counter-example. Add the built-in Vélos en libre-service (par jour) (bike_sharing). Its card says that occasionnels plus abonnes equals total, exactly.

1. Train a Forêt aléatoire (rf_reg) on total with every column.
2. Reference result: R² = 0.9974, RMSE 99.8, and no warning at all.

1. Retrain without occasionnels and abonnes. Reference: R² 0.8889 with the calendar and weather columns, against R² -0.0245 for the Modèle de référence (médiane) (dummy_reg).
2. Write both numbers down. One of them is a fact about the world; the other is a fact about your spreadsheet.

Step 6: Seal, and write the data section10 min

1. On any of the three datasets: apply Découper (proportions) (split) with 0.8, 0.2 to get two disjoint parts.
2. Click the 20 % node and press Sceller. Try to open Explore on it: preview, profile, quality, charts and download all answer with a padlock and one sentence.
3. Train a model on the 80 % part, choosing the sealed node as Jeu de test. It works: the sealed dataset is still a perfectly valid test set.

Write as you go, not at the end. In the Report panel every section has its guiding questions in grey; anything you type is saved after 0.7 s. Write the Données section: for each of the three datasets, the defect, the number, and what you did about it. Fifteen lines.