← Back to Data Exploration with rakoon-ds
Practical Work 2

Prepare the California fires, and meet your first leak

Clean a real, messy CSV at click speed: constant columns, identifiers, missing values, dates turned into features, and the two ways a score can lie to you

Duration 75 min
Level Beginner
Session Session 2
Mission dep-2

Objectives

By the end of this session you will be able to:

  • Import a CSV whose rows contain line breaks, and check what actually arrived
  • Use the quality report as a to-do list, not as a decoration
  • Drop constant and identifier columns, and say why
  • Turn a date into features that a model can use
  • Read a leak warning on the results screen and act on it
  • Understand why rebalancing classes before the split inflates the score

Prerequisites

  • A rakoon-ds account on https://rakoon-ds.apps.way-up.io (free, browser only, nothing to install), and the group join code your instructor gives you
  • A recent Chrome, Edge or Firefox. Computation runs either in your own tab (browser engine) or on the server; the header pill tells you which
  • The mission for this session, assigned to your group. Open a project, then the Mission button in the workshop header: the panel opens next to Report. Click Check after each step
  • Practical work 1 finished (you know the three explorer tabs)
  • Transformation, chart and algorithm names come from the rakoon-ds registry and are served in French even when the interface is in English. Every step below gives you the French label you will click and, in code font, the registry key the mission checks against.

Data

One public CSV, already hosted on this site.

FileLinkSizeTarget
California_Fire_Incidents.csvdownload1 636 rows, 40 columns (948 KB)MajorIncident (True / False)

Source: CAL FIRE incident feed, 2013 to 2019, as redistributed by DataCamp. One row is one wildfire.

The file has 2 338 physical lines but only 1 636 rows: the SearchKeywords column contains line breaks inside quotes. If your import reports a different number, your reader is wrong. rakoon-ds reads it correctly; a naive split on newlines does not.

Timing

The steps below add up to the announced duration. If you fall behind, Step 1 to Step 3 are the ones that must be finished.

#What you doTime
Step 1Import and count8 min
Step 2Quality first: build the to-do list12 min
Step 3Drop the dead weight10 min
Step 4Dates into features12 min
Step 5A baseline, a forest, and a warning you must not ignore15 min
Step 6Imbalance, and the trap that comes with the cure18 min
Total75 min

Instructions

Step 1: Import and count8 min

  1. New project PW2 California fires.
  2. Add a dataset → ...or import a file → pick the CSV.
  3. Check the node label: it must read 1 636 x 40. If it does not, say so before going further.

Open Explore → Overview and scroll right. Forty columns is more than a screen. That is normal and it is the point: your first job is to reduce it.

Step 2: Quality first: build the to-do list12 min

Tab Quality. Set the target to MajorIncident so the balance card appears. Write down, in this order:

FindingWhat the studio reportsYour decision
Constant columnsActive, PercentContained, Public, StructuresEvacuated (the last one is 100 % empty)drop
Identifier-like columnsCanonicalUrl, Location, SearchDescription, Started, UniqueId (all above 97 % distinct)?
Columns above 87 % missingAirTankers, ConditionStatement, ControlStatement, CrewsInvolved, Dozers, Engines, Fatalities, FuelType, Helicopters, Injuries, PersonnelInvolved, StructuresDamaged, StructuresDestroyed, StructuresThreatened, WaterTenders?
Duplicate rows0nothing to do
Class balance1 253 False (76.6 %) against 383 True (23.4 %), ratio 3.27?

Started is flagged as an identifier because 97.6 % of the timestamps are distinct. It is not an identifier, it is a date at second resolution. The detector is right about the measurement and wrong about the meaning: that is exactly why the decision column is yours.

The fifteen nearly empty columns are the resources engaged on the fire: engines, crews, helicopters. They are only known while and after the fire is being fought. Predicting "is this a major incident" from the number of engines sent is predicting the past.

Step 3: Drop the dead weight10 min

Node California_Fire_Incidents → Transformation. The panel works in three steps: 1 · Family, 2 · Transformation, 3 · Settings.

  1. Family Colonnes, card Supprimer des colonnes (drop_columns).
  2. In Colonnes à supprimer, select the four constant columns and the text columns that carry no signal: Active, PercentContained, Public, StructuresEvacuated, CanonicalUrl, Location, SearchDescription, UniqueId, SearchKeywords, Name, ConditionStatement, ControlStatement.
  3. Name the result calfire-nettoye, click Preview, then Apply → new dataset.

You should get 1 636 x 28. The original node is still there, untouched, with an arrow to the new one. That arrow is your lineage: it is what lets you change your mind later without redoing anything.

Step 4: Dates into features12 min

A timestamp is useless as such. A month, a weekday and a day of year are not.

  1. On calfire-nettoye: family Dates, card Convertir en date (to_date), columns Started and Extinguished. Preview, then apply. Name it calfire-dates.
  2. On the result: family Dates, card Décomposer une date/heure (date_parts), column Started, components year, month, weekday, is_weekend, dayofyear. Name it calfire-date-parts: 1 636 x 33.

Read the Preview notes before applying. When "Convertir en date" does not understand some values it says how many, by column. On this file it understands everything; on the exam file of Session 4 it used to lose 45 rows in silence, and that bug is the reason the notes exist.

Extracting the weekday is one checkbox here. In Orange it is a Python Script widget. Remember that when somebody tells you the two tools are equivalent.

Step 5: A baseline, a forest, and a warning you must not ignore15 min

Click calfire-date-parts → Train.

  1. Task Classification, algorithm Modèle de référence (classe majoritaire) (dummy_clf), target MajorIncident, all features. Experiment label essai-1. Create & Train.
  2. Same dataset, algorithm Forêt aléatoire (rf_clf), same target, same experiment. Create & Train.
  3. Open the results (click the History node, then the run).

Measured on the reference instance:

ModelAccuracyF1AUC
Baseline (majority class)0.76530.66350.500
Random forest, all columns0.91200.90580.9347

Above the metrics, a red banner: leak_association on FuelType, value 1.00. FuelType is filled in for only 12 fires out of 1 636, and all twelve are major incidents. The model found the shortcut. Your 0.912 is worth nothing.

  1. Train a third forest, same dataset, but keep only the columns known when the fire starts: AcresBurned, AdminUnit, ArchiveYear, CalFireIncident, Counties, CountyIds, Latitude, Longitude, Status, and the five Started_* columns.
  2. Read the score again.

Reference run: accuracy 0.7775, F1 0.7024, AUC 0.8185, and no warning. Against a baseline of 0.7653, the honest model gains barely one point of accuracy but eleven points of AUC. Both numbers are true and they say different things: that is the whole session.

Step 6: Imbalance, and the trap that comes with the cure18 min

The target is 3.27 to 1. The Coach, at the bottom of the results screen, says so by itself and offers to rebalance. Do it, then measure what it cost you.

  1. On calfire-date-parts: family Ré-échantillonnage, card Ré-échantillonner (resample), method balance, column MajorIncident, strategy over. Name it calfire-equilibre.
  2. It goes from 1 636 to 2 506 rows.
  3. Train the same honest forest on calfire-equilibre.
DatasetAccuracyF1AUCWarning
calfire-date-parts (honest features)0.77750.70240.8185none
calfire-equilibre0.82930.82930.9178resample_before_split

Five points of accuracy for free? No. Oversampling duplicates minority rows before the train / test split, so copies of the same fire end up on both sides of the wall. The model recognises rows it has already seen. rakoon-ds raises resample_before_split for exactly this. The gain is not real, and the honest number is 0.7775.

Write as you go, not at the end. In the Report panel every section has its guiding questions in grey; anything you type is saved after 0.7 s. Write the Préparation section (what you dropped and why) and the Évaluation section (the three scores above, and which one you would defend in front of a fire chief). Ten lines each.

What you should have

  • A lineage of five datasets: raw, cleaned, dates, date parts, rebalanced
  • Four trained models: baseline, leaky forest, honest forest, rebalanced forest
  • One leak warning seen and understood, one resampling warning seen and understood
  • Report sections Préparation and Évaluation written
  • The dep-2 mission at 6 / 6

Deliverables

  • Mission: dep-2 validated
  • Report: Préparation and Évaluation filled, with the table of the four scores
  • One paragraph: a colleague shows you a 0.91 accuracy on this dataset. What are the two questions you ask before believing it?

Bonus

  • Use Filtrer des lignes (filter_rows) to keep only fires above 1 000 acres, and retrain. Does the problem get easier or harder?
  • Try Gradient Boosting (gb_clf) on the honest feature set and compare in the Comparison panel.
  • Open the Diagnostics tab and run a 5-fold cross-validation on the honest forest. Is the 0.7775 stable across folds?
  • Find the fourth constant column (StructuresEvacuated) and explain why "100 % missing" and "constant" are reported as the same finding.

Resources