Benchmark forecasts and honest evaluation: backtesting, metrics and leakage

Before this lesson: lesson 01 (trend, season, the seasonal period $m$) and lesson 02 (white noise, random walk, the ACF and the Ljung–Box test). Cross-validation in ordinary machine learning helps (ml-fundamentals track). Time: about 75 minutes, plus the notebook.

---

Why this matters #

A forecast is only useful when it beats a simple rule. In the M4 competition (100,000 series), the six pure machine-learning methods were all less accurate than a simple combination of exponential smoothing methods, and only one of them beat a seasonally adjusted naive forecast (Makridakis, Spiliotis and Assimakopoulos 2020). Many published "improvements" disappear when they are compared with a seasonal naive forecast on the same backtest.

Evaluation mistakes are common and they always flatter the model:

• a random train/test split puts future values into the training set;
• one test period of a few weeks can be lucky or unlucky;
• a scaler, a decomposition or a hyperparameter search that sees the test period leaks information;
• a percentage metric quietly rewards forecasts that are too low.

Every later lesson of this track compares its method with the benchmarks of this lesson, using the rolling-origin backtest of this lesson.

1. Predict first #

2. The picture #

Four benchmark forecasts #

You stand at the forecast origin $t$: you know $y_1, \dots, y_t$ and nothing after. You want the values at $t + 1, t + 2, \dots, t + H$. The number of steps ahead, $h$, is the horizon. We write the forecast of $y_{t+h}$ made at origin $t$ as $\hat y_{t+h\mid t}$.

Four rules need no model fitting:

• mean: every future value equals the average of the past;
• naive: every future value equals the last observed value $y_t$;
• seasonal naive: every future value equals the value one season earlier (for monthly data with a yearly pattern, next March equals the last March);
• drift: draw a straight line through the first and the last observation, and extend it.

These are the benchmark forecasts (also called baselines). They are not toys: each one is the best possible forecast for one simple kind of series (section 3.2).

Interactive figure: loading…

Backtesting: many origins, not one #

One test period is one sample. It can be lucky. A backtest (also called rolling-origin evaluation or time series cross-validation) repeats the forecast at many origins:

1. Choose an origin $t_1$. Fit the method on the data up to $t_1$.
2. Forecast $t_1 + 1, \dots, t_1 + H$ and store the errors, labelled by their horizon $h$.
3. Move the origin forward by a step (for example 6 months) and repeat.
4. Average the errors over all origins, separately for each horizon $h$.

The training data either grow (expanding window: always start at $y_1$) or keep a fixed length (sliding window: drop the oldest values). A gap leaves some values out between the end of the training data and the first test value (section 3.5 says when you need one).

Interactive figure: loading…

3. Definitions and main results #

3.1 Forecasts, errors and residuals #

• The forecast $\hat y_{t+h\mid t}$ uses only $y_1, \dots, y_t$.
• The forecast error is $e_{t+h} = y_{t+h} - \hat y_{t+h\mid t}$. It is measured on data the method did not see.
• The residual is the one-step error on the training data, $e_t = y_t - \hat y_{t\mid t-1}$. For a method with fitted parameters, the residuals use the parameters fitted on all the training data, so they are in-sample.

Residuals tell you whether a method used all the information in the data. Forecast errors tell you how accurate it is. Do not report residuals as accuracy: they are almost always too small.

3.2 The four benchmarks and when each one is optimal #

Let $n$ be the number of observations known at the origin, $m$ the seasonal period and $h = 1, 2, \dots$ the horizon.

Method	Forecast $\hat y_{n+h\mid n}$	Best forecast (smallest mean squared error) for
mean	$\bar y = \frac{1}{n}\sum_{t=1}^{n} y_t$	white noise around a constant: $y_t = \mu + \varepsilon_t$
naive	$y_n$	a random walk: $y_t = y_{t-1} + \varepsilon_t$
seasonal naive	$y_{n+h-m(k+1)}$, $k = \lfloor (h-1)/m \rfloor$	a seasonal random walk: $y_t = y_{t-m} + \varepsilon_t$
drift	$y_n + h\,\dfrac{y_n - y_1}{n - 1}$	a random walk with drift: $y_t = c + y_{t-1} + \varepsilon_t$

Here $\varepsilon_t$ is white noise with mean 0 and variance $\sigma^2$ (lesson 02). In the seasonal naive formula, $k$ counts how many full years of the forecast period lie before the target: for $h \le m$, $k = 0$ and the forecast is $y_{n+h-m}$.

Why the conditional mean is the best forecast under squared error. Let $Y$ be the unknown future value and $\mathcal F$ the information at the origin. For any number $c$ computed from $\mathcal F$,

The first term does not depend on $c$, so the smallest value is at $c = \mathbb E[Y \mid \mathcal F]$. (To get the identity, write $Y - c = (Y - \mathbb E[Y \mid \mathcal F]) + (\mathbb E[Y \mid \mathcal F] - c)$ and expand; the cross term has conditional mean 0.)

Now compute the conditional mean for each model.

1. Random walk. $y_{n+h} = y_n + \varepsilon_{n+1} + \dots + \varepsilon_{n+h}$. The future $\varepsilon$ have mean 0 and are independent of the past, so $\mathbb E[y_{n+h} \mid y_1, \dots, y_n] = y_n$: the naive forecast.
2. Random walk with drift. $y_{n+h} = y_n + hc + \varepsilon_{n+1} + \dots + \varepsilon_{n+h}$, so the best forecast is $y_n + hc$. The drift $c$ is unknown. Estimate it by the average change: $\hat c = \frac{1}{n-1}\sum_{t=2}^{n} (y_t - y_{t-1}) = \frac{y_n - y_1}{n - 1}$, because the sum telescopes. This gives the drift forecast.
3. Seasonal random walk. Each season $j$ is its own random walk, so the best forecast of a future value is the last observed value of the same season: the seasonal naive forecast.
4. White noise around $\mu$. The best forecast is $\mu$; the sample mean $\bar y$ estimates it.

The fan. For the random walk, the forecast error is $e_{n+h} = \varepsilon_{n+1} + \dots + \varepsilon_{n+h}$, a sum of $h$ independent terms, so its variance is $h\sigma^2$. The error grows like $\sqrt h$: the prediction interval opens like a fan. With normal errors, a 95 % prediction interval is $\hat y_{n+h\mid n} \pm 1.96\,\hat\sigma_h$. The standard deviations $\hat\sigma_h$ of the four benchmarks (FPP3, §5.5, Table 5.2; $\hat\sigma$ is the standard deviation of the residuals):

Method	$\hat\sigma_h$
mean	$\hat\sigma\sqrt{1 + 1/n}$
naive	$\hat\sigma\sqrt{h}$
seasonal naive	$\hat\sigma\sqrt{k + 1}$, $k = \lfloor (h-1)/m \rfloor$
drift	$\hat\sigma\sqrt{h\,(1 + h/(n-1))}$

For the drift forecast the error is $\sum_{j=1}^{h} \varepsilon_{n+j} - h(\hat c - c)$ with $\hat c - c = \frac{1}{n-1}\sum_{t=2}^{n}\varepsilon_t$. The two parts are independent, so the variance is $h\sigma^2 + h^2\sigma^2/(n-1)$: the extra term is the uncertainty of the estimated slope. Lesson 10 checks whether such intervals have the coverage they promise.

3.3 Checking residuals #

A good forecasting method leaves residuals that look like white noise. Check, in this order:

1. Uncorrelated. If the residuals are correlated, the method missed information that is in the past values. Look at the ACF of the residuals and run the Ljung–Box test (lesson 02): $Q = n(n+2)\sum_{k=1}^{\ell} \hat\rho(k)^2/(n-k)$, compared with a $\chi^2$ distribution with $\ell$ degrees of freedom (minus the number of fitted parameters, for models from lesson 05 on). FPP3 suggests $\ell = 10$ for non-seasonal data and $\ell = 2m$ for seasonal data.
2. Mean zero. A nonzero mean is a bias: add the mean to the forecasts to remove it.
3. Constant variance (homoscedastic). If the spread grows with the level, try a log transform (lesson 01).
4. Roughly normal. Needed only for the normal prediction intervals above.

Properties 1 and 2 are essential. Properties 3 and 4 only matter for intervals. A method can pass all four checks and still forecast badly; residual checks never replace a backtest.

3.4 Split by time, never at random #

A holdout split for time series is a split in time order: the training set is $y_1, \dots, y_{n_0}$, the test set is $y_{n_0+1}, \dots, y_n$. A random split is wrong in general for two reasons:

• Neighbours are correlated. A test point $y_t$ with training points $y_{t-1}$ and $y_{t+1}$ on both sides can be interpolated. A real forecast must extrapolate into the future, which is harder. The error estimate becomes too small.
• The future is in the training set. Information after $t$ (a new level, a trend change) helps to predict $y_t$. In real use that information does not exist yet.

3.5 Rolling-origin evaluation #

Definition. A rolling-origin evaluation (backtest, time series cross-validation) uses forecast origins $t_1 < t_2 < \dots < t_K$. At each origin $t_k$:

• the training set is $\{1, \dots, t_k\}$ (expanding window) or $\{t_k - w + 1, \dots, t_k\}$ for a fixed length $w$ (sliding window);
• the test set is $\{t_k + g + 1, \dots, t_k + g + H\}$, where $g \ge 0$ is the gap;
• the origins are spaced by a step $s$: $t_{k+1} = t_k + s$.

The error at horizon $h$ is the average over the origins: $\text{MAE}(h) = \frac{1}{K}\sum_{k=1}^{K} |y_{t_k+h} - \hat y_{t_k+h\mid t_k}|$.

Expanding or sliding? An expanding window uses all the data and gives stable estimates. A sliding window follows a series whose behaviour changes, and it shows how the method works with a fixed amount of history. Use the scheme you will use in production.

Report each horizon separately. A one-step error and a twelve-step error answer different questions: the error usually grows with $h$. If you average all horizons into one number, say so, and also show the curve $\text{MAE}(h)$.

The gap (embargo). Leave $g$ values out between training and test when information near the origin is not really available at forecast time:

• the target is known late (sales are corrected two weeks later; a medical outcome is known after 30 days);
• features are built from windows that overlap the test period (a centred moving average, a label defined over the next 7 days);
• you will really forecast from an older origin (data arrive with a delay of 3 days, so the forecast for "tomorrow" is a 4-step forecast).

In finance this idea is called an embargo, together with purging training samples whose labels overlap the test period (López de Prado 2018).

In scikit-learn. sklearn.model_selection.TimeSeriesSplit(n_splits, max_train_size, test_size, gap) makes expanding folds whose test windows have test_size points and follow each other without overlap (the step equals test_size). max_train_size turns it into a sliding window, and gap removes that many points from the end of each training set. The folds start from the end of the series: the last fold always tests the last test_size points.

When is ordinary K-fold cross-validation acceptable?

Bergmeir, Hyndman and Koo (2018) show that standard K-fold cross-validation is valid for a purely autoregressive model (the inputs are lagged values $y_{t-1}, \dots, y_{t-p}$) fitted to a stationary series, if the model's errors are uncorrelated. Then each row $(y_{t-1}, \dots, y_{t-p}) \to y_t$ is a fair test example, and K-fold uses the data better than a single holdout. When the model underfits, its errors are correlated, and K-fold becomes too optimistic. The result does not cover non-stationary series (trend, level shifts) or models with other inputs. The last figure of this lesson shows both cases: on a series with a trend, random folds flatter a k-nearest-neighbour model a lot, but they agree with the truth for a linear model on lags.

3.6 Point-forecast metrics #

Let $e_j = y_j - \hat y_j$, $j = 1, \dots, J$, be the forecast errors of a test set (all origins and horizons you want to summarise).

Metric	Formula	Units and remarks
MAE (mean absolute error)	$\frac1J\sum_j \lvert e_j\rvert$	units of $y$
RMSE (root mean squared error)	$\sqrt{\frac1J\sum_j e_j^2}$	units of $y$; large errors count more
MAPE (mean absolute percentage error)	$\frac{100}{J}\sum_j \lvert e_j\rvert / \lvert y_j\rvert$	percent; undefined if some $y_j = 0$, huge if $y_j$ is near 0
sMAPE (symmetric MAPE, M4 version)	$\frac{100}{J}\sum_j \dfrac{2\lvert e_j\rvert}{\lvert y_j\rvert + \lvert\hat y_j\rvert}$	percent; between 0 and 200; not really symmetric
WAPE (weighted absolute percentage error)	$\sum_j \lvert e_j\rvert \big/ \sum_j \lvert y_j\rvert$	a ratio; the same as MAE divided by the mean of $\lvert y\rvert$
MASE (mean absolute scaled error)	$\frac1J\sum_j \lvert q_j\rvert$ with $q_j = e_j / s$	no units; $s$ is defined below
RMSSE (root mean squared scaled error)	$\sqrt{\frac1J\sum_j e_j^2 / s_2}$	no units; used in the M5 competition

MAE and RMSE are scale-dependent: you cannot average them over series with different units or sizes. Percentage errors have no units, but they need a meaningful zero (they make no sense for temperatures in °C) and they explode near $y = 0$.

MASE (Hyndman and Koehler 2006) divides by the in-sample MAE of the one-step seasonal naive forecast on the training data $y_1, \dots, y_n$:

For non-seasonal data use $m = 1$ (the naive forecast). For RMSSE, $s_2$ is the mean of the squared differences $(y_t - y_{t-m})^2$.

• MASE < 1: the forecasts on the test set are, on average, better than the one-step seasonal naive forecasts were on the training set.
• MASE > 1: they are worse. This is not the same as "worse than the seasonal naive method on the test set": the scale is in-sample and one-step, while the test errors are often several steps ahead. Even the seasonal naive method itself often has MASE > 1 on a test set.
• The scale $s$ uses only training data, so it is known at the origin and it is positive unless the training data are exactly periodic.

3.7 Which forecast does each metric reward? #

A point forecast is one number, but the future value is uncertain: it has a distribution. Each metric is minimised by a different summary of that distribution. Let $Y > 0$ have the predictive distribution with CDF $F$ and density $f$.

Theorem.

1. $\mathbb E[(Y - c)^2]$ is smallest at the mean $c = \mathbb E[Y]$.
2. $\mathbb E\lvert Y - c\rvert$ is smallest at the median of $Y$.
3. $\mathbb E\big[\lvert Y - c\rvert / Y\big]$ (the MAPE of a constant forecast) is smallest at the median of the distribution with density proportional to $f(y)/y$. This value is below the median of $Y$.

Proof.

1. This is the identity of section 3.2 without conditioning.
2. Write $\mathbb E\lvert Y - c\rvert = \int_{-\infty}^{c} (c - y) f(y)\,dy + \int_{c}^{\infty} (y - c) f(y)\,dy$. Its derivative in $c$ is $F(c) - (1 - F(c)) = 2F(c) - 1$. This is negative for $c$ below the median and positive above it, so the minimum is at $F(c) = 1/2$.
3. $\mathbb E[\lvert Y - c\rvert / Y] = \int \lvert y - c\rvert\, \frac{f(y)}{y}\,dy$. Up to a constant factor, this is $\mathbb E\lvert \tilde Y - c\rvert$ for a variable $\tilde Y$ with density $f(y)/y$ divided by its integral. By part 2 the minimum is the median of $\tilde Y$. The weight $1/y$ is larger for small values, so the distribution of $\tilde Y$ is shifted towards small values, and its median is below the median of $Y$. $\blacksquare$

The log-normal case: the MAPE-optimal value is $e^{\mu - \sigma^2}$

Let $\ln Y \sim \mathcal N(\mu, \sigma^2)$, so $f(y) = \frac{1}{y\sigma\sqrt{2\pi}}\exp\!\big(-\frac{(\ln y - \mu)^2}{2\sigma^2}\big)$. Then $f(y)/y = \frac{1}{y\sigma\sqrt{2\pi}}\exp\!\big(-\frac{(\ln y - \mu)^2}{2\sigma^2} - \ln y\big)$. Complete the square in $\ln y$: $(\ln y - \mu)^2 + 2\sigma^2 \ln y = \big(\ln y - (\mu - \sigma^2)\big)^2 + 2\mu\sigma^2 - \sigma^4$. The last two terms do not depend on $y$. So $f(y)/y$ is proportional to the log-normal density with parameters $\mu - \sigma^2$ and $\sigma^2$. The median of that distribution is $e^{\mu - \sigma^2}$. For $\mu = 0$, $\sigma = 1$: mean $e^{1/2} \approx 1.65$, median $1$, MAPE-optimal value $e^{-1} \approx 0.37$.

Interactive figure: loading…

Consequence: choose the metric from the decision, then the forecast. If the cost of an error grows like its square (large errors are very expensive), report the mean and evaluate with RMSE. If the cost grows linearly, report the median and evaluate with MAE. Do not use MAPE to choose between methods for skewed positive data unless you want forecasts that are too low. When you need several quantiles, use the pinball loss of lesson 10.

3.8 Leakage in evaluation #

Leakage means that the model, or any step before it, used information that would not be available at the forecast origin. The backtest then reports an error that you will never see in real use. The general rule: at each origin, simulate exactly what was known at that time.

Common forms in forecasting:

• Preprocessing on the whole series. A scaler, a log-mean, a decomposition (STL, classical) or a missing-value fill computed on all data, before the split. Fit every step inside each fold, on the training window only.
• Filling missing values from the future. Back-fill (bfill) and centred interpolation use values after the gap. Use forward-fill or a model of the past only.
• Tuning on the test period. Choosing the smoothing parameter, the lag order or the network size by looking at test errors. Tune inside the training window (a nested backtest), and keep a final test period untouched.
• Revised data (vintages). GDP, sales and sensor data are often corrected later. A backtest on today's corrected values uses information that did not exist at the origin. Use the data vintage that was available at each origin, when you have it.
• Covariates not known at forecast time. Observed weather used as an input, as if it were a weather forecast; a promotion flag that is decided after the forecast. Use the forecast of the covariate, or its value at the origin.

Interactive figure: loading…

3.9 Comparing two methods #

• Average over many origins and many series. With few origins the ranking of two methods is often noise.
• Use the same folds for both methods. Then the differences are paired, and the comparison is much more precise.
• Look at the curve per horizon, not only at the average.

The Diebold–Mariano test

To test whether two methods have the same expected loss at horizon $h$, compute the loss differential $d_t = L(e^{(1)}_t) - L(e^{(2)}_t)$ for the test errors of the two methods (for example $L(e) = e^2$ or $\lvert e\rvert$), over $T$ origins. Under the null hypothesis $\mathbb E[d_t] = 0$. The statistic is

$$\mathrm{DM} = \frac{\bar d}{\sqrt{\hat V / T}}, \qquad \hat V = \hat\gamma_d(0) + 2\sum_{k=1}^{h-1} \hat\gamma_d(k),$$

where $\hat\gamma_d(k)$ is the sample autocovariance of $d_t$ at lag $k$. For large $T$, DM is approximately standard normal (Diebold and Mariano 1995). The sum goes up to lag $h - 1$ because $h$-step errors from origins one step apart share $h - 1$ future shocks, so they are correlated. For small $T$, Harvey, Leybourne and Newbold (1997) give a corrected version with a $t$ distribution. The test compares forecasts, not models: it says nothing about which model is "true".

Answers to section 1. (1) The seasonal naive forecast. With a strong yearly pattern and no trend, "same month last year" keeps the pattern; "repeat the last value" gives a flat line and misses every seasonal swing. (2) Something below the median. MAPE divides by the true value, so errors on small values cost more; section 3.7 shows that the best constant for MAPE is the median of a distribution shifted towards small values (for a log-normal, $e^{\mu - \sigma^2}$). (3) Too low. A random fold lets the model interpolate between neighbours and see the future level, so random K-fold is usually too optimistic. It is acceptable only in the special case of the deeper box in section 3.5.

4. Worked example #

(a) Benchmarks and metrics by hand. A quarterly series ($m = 4$) has three years of training data and one year of test data:

	Q1	Q2	Q3	Q4
year 1 ($y_1..y_4$)	10	14	8	12
year 2 ($y_5..y_8$)	12	15	9	14
year 3 ($y_9..y_{12}$)	12	17	10	15
year 4, test ($y_{13}..y_{16}$)	14	19	11	16

The origin is $n = 12$ and $H = 4$.

Step 1: the four forecasts.

• Mean: $\bar y = (10 + 14 + 8 + 12 + 12 + 15 + 9 + 14 + 12 + 17 + 10 + 15)/12 = 148/12 \approx 12.33$ for every $h$.
• Naive: $y_{12} = 15$ for every $h$.
• Seasonal naive: for $h = 1, \dots, 4$, $k = 0$, so $\hat y_{12+h\mid 12} = y_{12+h-4} = y_{8+h}$: the values $12, 17, 10, 15$ of year 3.
• Drift: slope $(y_{12} - y_1)/(12 - 1) = (15 - 10)/11 \approx 0.4545$. Forecasts $15 + 0.4545\,h$: $15.45, 15.91, 16.36, 16.82$.

Step 2: errors $e = y - \hat y$ on the test year $(14, 19, 11, 16)$.

Method	$e_{13}$	$e_{14}$	$e_{15}$	$e_{16}$
mean	1.67	6.67	−1.33	3.67
naive	−1	4	−4	1
seasonal naive	2	2	1	1
drift	−1.45	3.09	−5.36	−0.82

Step 3: the MASE scale. The in-sample seasonal naive errors are $y_t - y_{t-4}$ for $t = 5, \dots, 12$: $12 - 10 = 2$, $15 - 14 = 1$, $9 - 8 = 1$, $14 - 12 = 2$, $12 - 12 = 0$, $17 - 15 = 2$, $10 - 9 = 1$, $15 - 14 = 1$. Their absolute values sum to 10, so $s = 10/8 = 1.25$.

Step 4: the metrics. For the seasonal naive forecast: $\text{MAE} = (2 + 2 + 1 + 1)/4 = 1.5$; $\text{RMSE} = \sqrt{(4 + 4 + 1 + 1)/4} = \sqrt{2.5} \approx 1.58$; $\text{MAPE} = \frac{100}{4}\big(\frac{2}{14} + \frac{2}{19} + \frac{1}{11} + \frac{1}{16}\big) \approx 10.0\,\%$; $\text{WAPE} = 6/60 = 10\,\%$; $\text{MASE} = 1.5/1.25 = 1.2$.

Method	MAE	RMSE	MAPE (%)	MASE
mean	3.33	3.95	20.5	2.67
naive	2.50	2.92	17.7	2.00
seasonal naive	1.50	1.58	10.0	1.20
drift	2.68	3.21	20.1	2.15

The seasonal naive forecast wins on every metric. Its MASE is still above 1: its test errors (one to four steps ahead, one year later, with a small upward trend) are larger than its in-sample one-step errors. The trend of about one unit per year is visible in its errors: all four are positive. A method that combines the season with a trend (lesson 04) can do better.

(b) Listing the folds. $n = 20$, expanding window, initial training size 12, $H = 4$, step 4, gap 0. The origins are 12 and 16 (the next one, 20, has no test data left):

Fold	training indices	test indices
1	1–12	13–16
2	1–16	17–20

This is exactly TimeSeriesSplit(n_splits=2, test_size=4) on 20 points (with indices from 0 in Python). With gap=1, the training sets would end at 11 and 15 instead.

5. In code #

NumPy checks the worked example. The notebook has these cells and the exercises.

import numpy as np

y_train = np.array([10, 14, 8, 12, 12, 15, 9, 14, 12, 17, 10, 15.])
y_test = np.array([14, 19, 11, 16.])
m, n, H = 4, len(y_train), len(y_test)
h = np.arange(1, H + 1)

fc = {
    "mean": np.full(H, y_train.mean()),
    "naive": np.full(H, y_train[-1]),
    "snaive": y_train[n + h - m * ((h - 1) // m + 1) - 1],     # -1: Python indices start at 0
    "drift": y_train[-1] + h * (y_train[-1] - y_train[0]) / (n - 1),
}
scale = np.mean(np.abs(y_train[m:] - y_train[:-m]))
print("scale", scale)                                          # scale 1.25
for name, f in fc.items():
    e = y_test - f
    print(f"{name:7s} MAE {np.mean(np.abs(e)):.2f}  RMSE {np.sqrt(np.mean(e**2)):.2f}  "
          f"MAPE {100 * np.mean(np.abs(e) / y_test):.1f}  MASE {np.mean(np.abs(e)) / scale:.2f}")
# mean    MAE 3.33  RMSE 3.95  MAPE 20.5  MASE 2.67
# naive   MAE 2.50  RMSE 2.92  MAPE 17.7  MASE 2.00
# snaive  MAE 1.50  RMSE 1.58  MAPE 10.0  MASE 1.20
# drift   MAE 2.68  RMSE 3.21  MAPE 20.1  MASE 2.15

from sklearn.model_selection import TimeSeriesSplit
for tr, te in TimeSeriesSplit(n_splits=2, test_size=4).split(np.arange(20)):
    print(tr[0], tr[-1], te)
# 0 11 [12 13 14 15]
# 0 15 [16 17 18 19]

6. Where it shows up #

• Forecasting competitions. The M3, M4 and M5 competitions rank methods by sMAPE, MASE and (M5) a weighted RMSSE, always against naive and seasonal naive benchmarks. The M4 results showed that simple statistical benchmarks are hard to beat (Makridakis et al. 2020).
• Demand and energy forecasting. Retailers report WAPE (it weights each product by its volume); electricity load forecasts are evaluated per horizon, because a day-ahead error and an hour-ahead error lead to different actions.
• Finance. Trading strategies are tested with walk-forward backtests (a rolling origin with refitting). Purging and an embargo remove training samples whose labels overlap the test period (López de Prado 2018). Leakage here gives strategies that only work in the past.
• Machine learning models for time series. Gradient boosting (lesson 08) and deep networks (lesson 09) are judged by the same backtests and by their MASE relative to seasonal naive. Foundation models for time series are compared on held-out series and periods; overlap between pretraining data and the test data is a form of leakage.
• ML for science. Machine-learning weather models are trained on earlier years of reanalysis data and tested on later years they never saw (for example GraphCast, Lam et al. 2023); the skill score of a forecast is often reported relative to a reference forecast such as climatology or persistence, which play the roles of the mean and naive benchmarks. Drift monitoring of deployed models uses the same idea (mlops lesson 07).

7. Common mistakes #

• No benchmark. A complex model with MAPE 8 % sounds good until the seasonal naive forecast has 7 %. Always report the benchmarks on the same folds.
• Shuffled cross-validation. KFold(shuffle=True) on time series lets the model interpolate and see the future. Use a time-ordered split, unless the special conditions of Bergmeir et al. hold.
• One test period. One split of the last 3 months is one sample. Use many origins.
• Residual error reported as accuracy. In-sample residuals are too small. Report errors on data that the method did not see.
• Preprocessing before the split. Scaling, decomposition, feature selection and imputation must be fitted inside each training window.
• Averaging errors over horizons without saying so. The error at $h = 1$ and at $h = 12$ are different quantities. Show $\text{MAE}(h)$.
• MAPE on data with zeros or values near zero. It is undefined or dominated by a few points, and it rewards forecasts that are too low. Use MASE or WAPE.
• Averaging MAE over series with different scales. The series with the largest values dominate. Use a scaled metric (MASE, RMSSE) to average across series.

8. Check yourself #

1. A daily series has a weekly pattern ($m = 7$). The origin is Sunday, day $t$. What is the seasonal naive forecast for the next Wednesday, $h = 3$, and for the Wednesday after that, $h = 10$?
   Answer

   For $h = 3$: $k = \lfloor 2/7 \rfloor = 0$, so $\hat y = y_{t+3-7} = y_{t-4}$, the last observed Wednesday. For $h = 10$: $k = \lfloor 9/7 \rfloor = 1$, so $\hat y = y_{t+10-14} = y_{t-4}$ again. The seasonal naive forecast repeats the last observed week.
2. Show that the naive forecast is the best mean-squared-error forecast for a random walk, and give the variance of its $h$-step error.
   Answer

   $y_{t+h} = y_t + \varepsilon_{t+1} + \dots + \varepsilon_{t+h}$. The future shocks have mean 0 and are independent of the past, so $\mathbb E[y_{t+h} \mid y_1, \dots, y_t] = y_t$. The conditional mean minimises the expected squared error. The error is the sum of $h$ independent shocks, so its variance is $h\sigma^2$.
3. Compute MASE for the naive forecast of the worked example, and say what it means.
   Answer

   The naive errors are $-1, 4, -4, 1$, so MAE $= 10/4 = 2.5$. The scale is $s = 1.25$, so MASE $= 2.5/1.25 = 2.0$. Its test errors are twice as large as the in-sample one-step errors of the seasonal naive method. It ignores the strong season.
4. A demand series has many days with zero sales. Your manager wants MAPE. What do you say, and what do you propose?
   Answer

   MAPE divides by $y_j$, so it is undefined on days with zero sales and very large on days with small sales; it also rewards forecasts that are too low. Propose MASE (scale-free, defined unless the training data are exactly periodic) or WAPE (total absolute error divided by total sales, which is well defined as long as total sales are positive).
5. Your features include a 7-day centred moving average of the target. You backtest with TimeSeriesSplit(gap=0). What leaks, and how large should the gap be?
   Answer

   A centred 7-day average at time $t$ uses $y_{t+1}, y_{t+2}, y_{t+3}$. For the last training rows these values lie in the test window, so test targets leak into training features. A gap of at least 3 removes this overlap. Better: use a trailing average, which only uses past values, and keep the gap for delays in data arrival.
6. Why does the seasonal naive method itself often have a MASE above 1 on the test set?
   Answer

   The MASE scale is the in-sample MAE of *one-step* seasonal naive forecasts. On the test set the seasonal naive method forecasts up to $H$ steps ahead, often with a trend or a change in level between the training and the test period. Its errors are then larger than its one-step in-sample errors.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Name the four benchmark forecasts.

   Show answer

2. Write the seasonal naive forecast $\hat y_{n+h\mid n}$.

   Show answer

3. For which model is the naive forecast optimal, and how fast does its error grow?

   Show answer

4. What is rolling-origin evaluation?

   Show answer

5. Expanding window or sliding window: what is the difference?

   Show answer

6. Why does random K-fold cross-validation usually leak on time series?

   Show answer

7. Define MASE.

   Show answer

8. Which constant forecast minimises MSE, and which minimises MAE?

   Show answer

9. Why does MAPE reward forecasts that are too low?

   Show answer

10. What does a gap (embargo) between training and test data prevent?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "How would you evaluate a forecasting model?" — A rolling-origin backtest with the same folds for all methods, refit (or update) at each origin, errors per horizon, and a comparison with naive and seasonal naive benchmarks using MASE.
• "Can you use K-fold cross-validation on time series?" — Not in general: shuffled folds interpolate and see the future. Exception: a purely autoregressive model with uncorrelated errors on a stationary series (Bergmeir et al. 2018).
• "MAPE or RMSE?" — It depends on the cost of errors. RMSE rewards the mean, MAE the median, MAPE a biased low value and fails near zero. For many series of different scales use MASE or WAPE.
• "Your model has great backtest results but fails in production. Why?" — Leakage: preprocessing on all data, future-filled missing values, revised data, covariates not known in time, tuning on the test period; or a backtest that did not match the production delay (a missing gap).

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. Write mean_forecast, naive, snaive and drift for a horizon $H$, and check them on the worked example.
2. Write rolling_origin_splits(n, initial, horizon, step, gap, expanding) and check it against TimeSeriesSplit where the two agree.
3. Write mase(y_train, y_test, y_pred, m) and check it on the worked example.
4. Backtest the four benchmarks on monthly Mauna Loa CO₂ data (1980–2001) and make a table of MASE by horizon.
5. Reproduce the optimism of random K-fold with a k-nearest-neighbour model on lags, and compare it with a rolling origin and with a true future block.

11. Watch and read #

Forecasting Principles & Practice: 5.2 Some simple forecasting methodsOTexts · Forecasting: Principles and Practice · 15 min

The mean, naive, seasonal naive and drift methods on real data.

Forecasting Principles & Practice: 5.8 Evaluating point forecast accuracyOTexts · Forecasting: Principles and Practice · 14 min

Scale-dependent, percentage and scaled errors, and why MASE was proposed.

Forecasting Principles & Practice: 5.10 Time series cross-validationOTexts · Forecasting: Principles and Practice · 7 min

The rolling origin as a picture, and the error as a function of the horizon.

Backtesting : Time Series Talkritvikmath · 15 min

Why a single split is not enough, and how to move the origin.

• Hyndman and Athanasopoulos, Forecasting: Principles and Practice, 3rd ed., free online: §5.2 simple methods, §5.4 residual diagnostics, §5.8 accuracy, §5.10 time series cross-validation.
• Hyndman and Koehler (2006), Another look at measures of forecast accuracy, International Journal of Forecasting 22(4): the paper that introduced MASE.
• Bergmeir, Hyndman and Koo (2018), A note on the validity of cross-validation for evaluating autoregressive time series prediction, Computational Statistics and Data Analysis 120.
• Tashman (2000), Out-of-sample tests of forecasting accuracy: an analysis and review, International Journal of Forecasting 16(4): rolling origins and their variants.
• Makridakis, Spiliotis and Assimakopoulos (2020), The M4 Competition: 100,000 time series and 61 forecasting methods, International Journal of Forecasting 36(1).
• Diebold and Mariano (1995), Comparing predictive accuracy, Journal of Business and Economic Statistics 13(3).
• scikit-learn, TimeSeriesSplit.