← Back to Data Exploration with rakoon-ds
Practical Work 8

Time series and temporal splits: the 27 points a random split gave you for free

Why shuffling rows on a dated dataset produces a number you cannot keep, how to split by time in rakoon-ds, and what an honest score looks like

Duration 75 min
Level Advanced
Session Session 8
Mission dep-8

Objectives

By the end of this session you will be able to:

  • See the time structure in a dataset before modelling it
  • Measure the difference between a random split and a temporal split on the same model
  • Configure a temporal split in the training form
  • Tell trend and seasonality apart, and know which one a model can extrapolate
  • Read the warning the studio shows when a dated dataset is split at random
  • Report the honest number, not the comfortable one

Prerequisites

  • A rakoon-ds account on https://rakoon-ds.apps.way-up.io (free, browser only, nothing to install), and the group join code your instructor gives you
  • A recent Chrome, Edge or Firefox. Computation runs either in your own tab (browser engine) or on the server; the header pill tells you which
  • The mission for this session, assigned to your group. Open a project, then the Mission button in the workshop header: the panel opens next to Report. Click Check after each step
  • Practical work 7 finished: you reuse its feature set
  • Transformation, chart and algorithm names come from the rakoon-ds registry and are served in French even when the interface is in English. Every step below gives you the French label you will click and, in code font, the registry key the mission checks against.

Data

The same bike_sharing dataset as Practical work 7, with the feature set you built there. Two full years, one row per day, no gaps: the cleanest possible setting for this lesson.

With the air-passengers exam pack, the same experiment gives R² 0.8837 at random against 0.7148 in time, on 11 128 flights. With the SNCF lost-property pack, train on 2019 to 2021 and test on 2022, with the 2022 part sealed.

Timing

The steps below add up to the announced duration. If you fall behind, Step 1 to Step 3 are the ones that must be finished.

#What you doTime
Step 1See the time12 min
Step 2The random split, and its comfortable number10 min
Step 3The temporal split13 min
Step 4Where the 27 points went12 min
Step 5What survives, what expires15 min
Step 6Report the honest number13 min
Total75 min

Instructions

Step 1: See the time12 min

Before any model, look at the shape of the series.

  1. On your velos-calendrier dataset, Explore → chart Nuage de points (scatter): X = date_dayofyear, Y = total, colour = annee.
  2. Then Comptage / moyenne (barres) (bar): column = date_month, value = total, aggregation mean, separated by annee.
  3. Pin both.

Two facts should jump out, and you must name them separately:

  • Seasonality: the same shape repeats every year, low in winter, high in summer.
  • Trend: 2012 is above 2011 everywhere. The service grew.

A model can learn seasonality, because it will see summer again. It cannot learn a trend in any useful way: annee captures "2012 is bigger than 2011" and tells you nothing about 2013. Any feature that encodes the absolute date is a feature that expires.

Step 2: The random split, and its comfortable number10 min

  1. Train Forêt aléatoire (rf_reg) on total with the eleven honest columns (weather plus calendar), leaving Découpe on Random (rows are shuffled), test share 25 %.
  2. Note the R² and the RMSE.
  3. Look just under the split selector.

Reference: R² 0.8889, RMSE 654.3.

The studio shows a line under the selector as soon as the dataset carries a date: "This dataset carries a date. A random split mixes past and future and inflates the score." Read it. It is the whole session in one sentence, and it was added because an exam trial measured seventeen points of R² disappearing when the split was fixed.

Step 3: The temporal split13 min

  1. Train the same model again: same dataset, same target, same eleven columns, same algorithm, same 25 % test share.
  2. This time set Découpe to By time on "date".
  3. Give it the same experiment label so the two runs sit side by side in the Comparison panel.

What the studio does: it sorts the rows by that date with a stable sort, takes the last 25 % as the test set, and shuffles nothing. Rows with no date stay in the training part.

SplitR²RMSEMAE
Random, 75 / 250.8889654.3429.6
Temporal on date0.61671 080.7900.3

27 points of R². The model is exactly the same. Only the question changed: "can you fill in a missing Tuesday?" against "can you predict next season?".

Step 4: Where the 27 points went12 min

Open both results and put them side by side.

  1. On the temporal run, open the Error map tab. Look at actual against predicted, and at the residuals.
  2. Sort the twenty worst-predicted days. When are they?
  3. Open View code on both runs. The generated code is different: the random one calls train_test_split, the temporal one sorts and cuts at an index. The code shown describes the training that actually happened.

Write the answer to this, in the report: the test period is the second half of 2012, the highest-traffic period of the whole dataset, and the model has never seen a level that high. A random split hid that by putting two days out of three of that same period into the training set.

Nothing here says the temporal number is "the right one" in every case. If your production use is "fill in the gaps of a historical table", the random split answers your question. If it is "predict tomorrow", it does not. The discipline is to say which question you are answering.

Step 5: What survives, what expires15 min

  1. Retrain the temporal model without annee. Does the score go down as much as you feared?
  2. Retrain it without date_dayofyear and date_week, keeping date_month. Which of the three carries the seasonality?
  3. Try the cyclical encodings of Practical work 7 in the temporal setting: mois_sin and mois_cos repeat every year, so they do not expire the way a year number does.

Make a small table of your own: feature, does it repeat next year (yes / no), R² with it, R² without it. That table is what separates a feature from a souvenir.

The Diagnostics tab does cross-validation with ordinary folds, not with time-ordered folds. On a dated dataset its standard deviation is therefore optimistic in the same way a random split is. Read it as a stability check on the training period, not as an estimate of future performance.

Step 6: Report the honest number13 min

  1. Pour the metrics block of both runs into the report.
  2. Pour the Comparaison de l'expérience block: one table, two rows, the whole argument.
  3. Write the Modélisation section: the two splits, why you chose the one you chose, and what question each answers.
  4. Write the Évaluation section: the honest number first, the random one second, and the gap explained in one sentence.
  5. Click Check in the mission panel.

The habit to leave this session with: on any dataset carrying a date, the first number you report is the temporal one. The random one, if you show it at all, is shown as a measure of how much a random split flatters you.

What you should have

  • Two pinned charts showing seasonality and trend separately
  • The same model trained twice, random and temporal, in the same experiment
  • A 27-point gap measured, located in time, and explained
  • A feature table separating what repeats from what expires
  • Report sections Modélisation and Évaluation written
  • The dep-8 mission at 5 / 5

Deliverables

  • Mission: dep-8 validated
  • Report: the comparison block, plus the two sections
  • One sentence you could say to a manager who asks why the score dropped from 0.89 to 0.62 after you "fixed" the evaluation

Bonus

  • Train on 2011 only and test on 2012 by using Filtrer des lignes (filter_rows) twice and passing the 2012 part as the test dataset. Seal it first. Compare with the last-25 % temporal split.
  • Build a lagged feature with the Python node and redo the two splits. A lag is exactly the feature that a random split turns into a leak.
  • Run the learning curve on the random model. It plateaus; what would the same curve look like if it were computed on time-ordered folds?

Resources