Deep learning for forecasting: RNNs, temporal CNNs, transformers and foundation models

Before this lesson: lesson 03 (baselines, backtesting, MASE), lesson 08 (windows as a table, recursive and direct strategies, global models) and lesson 07 (for the paragraph on state-space layers). From the deep-learning track: convolutions, recurrent networks and the LSTM, attention, the transformer and state-space models. Time: about 2 hours, plus the notebook.

---

Why this matters #

• Many related series. Retailers, cloud providers and energy companies forecast thousands to millions of series at once. One global network trained on all of them shares patterns between series and can forecast a new product with little history. DeepAR (Salinas et al. 2017) was built for this setting and is a built-in algorithm of Amazon SageMaker.
• Long contexts and rich covariates. Hourly load, traffic and sensor data have daily and weekly cycles, holidays and weather. Convolutions, patched transformers and models such as the Temporal Fusion Transformer handle long windows and many inputs.
• Pretrained models change the workflow. A zero-shot model (Chronos, TimesFM, Moirai, TiRex, ...) gives a probabilistic forecast for any series in one call, with no training. This is the newest and fastest-moving part of the field, so you need to know how these models work and how to test them.
• The evidence is mixed. On one short series, exponential smoothing, ARIMA (lessons 04 and 06) and gradient boosting (lesson 08) are hard to beat, and on standard long-horizon benchmarks a one-layer linear model beat several published transformers (Zeng et al. 2023). Knowing why protects you from using a big model where a small one is better.

1. Predict first #

2. The picture #

A forecasting network in one picture #

Every model in this lesson has the same outer shape:

1. Cut out the context. At the forecast origin $t$, take the last $L$ values $x = (y_{t-L+1}, \dots, y_t)$. $L$ is the lookback or context length. (In this lesson $H$ is the largest horizon, not the observation matrix of lesson 07.)
2. Scale the window with statistics of the window only, for example $\tilde x = x / s$ with $s = \tfrac1L \sum_i |x_i|$.
3. Run the network on $\tilde x$ (plus covariates such as calendar features).
4. Read the head. A point head outputs $H$ numbers; a distribution head outputs parameters such as a mean and a scale; a quantile head outputs, for example, the 10%, 50% and 90% quantiles.
5. Undo the scaling: multiply by $s$ (and add back a shift, if one was removed).

Training uses many windows: every position of the origin in every training series gives one example, as in the "window to table" reduction of lesson 08. The network either outputs all $H$ steps at once (multi-output, the direct strategy) or one step at a time and feeds its output back (autoregressive, the recursive strategy).

Recurrent networks: one step at a time, and samples instead of a single path #

A recurrent network reads $y_1, y_2, \dots$ in order and updates a hidden state $h_t$, a summary of the past (the cell equations are in deep-learning lesson 08). DeepAR (Salinas et al. 2017) puts a probability distribution on top: at each step the network outputs the parameters of a distribution for the next value, for example a mean $\mu_t$ and a standard deviation $\sigma_t$.

The important idea is the difference between training and prediction:

• Training (teacher forcing). The inputs at every step are the true previous values. Each step is a one-step-ahead prediction, and the loss is the negative log-likelihood of the true value.
• Prediction (ancestral sampling). After the forecast origin there are no true values. The model draws a value from its own distribution, feeds it back as the next input, and repeats. One run gives one possible future, a sample path. Many runs give many paths; their spread at each horizon is the forecast uncertainty.

Interactive figure: loading…

Convolutions that only look back #

A convolution layer slides a small filter along the series. For forecasting it must be causal: the output at time $t$ may use $y_t, y_{t-1}, \dots$ but never $y_{t+1}$, because that value does not exist at forecast time. A dilated convolution skips inputs: with dilation $d$ and kernel size $k$, the output at $t$ reads $y_t, y_{t-d}, \dots, y_{t-(k-1)d}$. Stacking layers with dilations $1, 2, 4, 8, \dots$ makes the window that one output can see, its receptive field, grow exponentially with depth while the number of weights grows only linearly. This is the design of WaveNet (van den Oord et al. 2016) and of the temporal convolutional network (TCN; Bai, Kolter and Koltun 2018).

Interactive figure: loading…

With $k = 2$ and dilations $1, 2, 4, 8$ the readout gives $r = 1 + 1 \cdot (1 + 2 + 4 + 8) = 16$: this answers question 1 of section 1. Four layers with dilation 1 would see only 5 values.

Tokens for a transformer: time steps, patches or bins #

A transformer works on a sequence of tokens and compares every pair of tokens with attention (deep-learning lesson 09). With $N$ tokens, one attention layer computes $N^2$ scores. There are three common ways to make tokens from a window of length $L$:

• one token per time step ($N = L$): simple, but a single number carries little information, and long contexts are expensive;
• patches (PatchTST, Nie et al. 2023): cut the window into pieces of $P$ values with stride $S$; each piece becomes one token, so $N \approx L/S$ and the attention cost drops by about $S^2$;
• quantised values (Chronos, Ansari et al. 2024): divide by the mean absolute value of the context, then replace each value by the index of the nearest of a few thousand bins. The series becomes a "sentence" over a fixed vocabulary, and a language-model architecture can be trained on it.

Interactive figure: loading…

3. Definitions and main results #

3.1 Windows, scaling and output heads #

Data. A training set has one or more series. A training window at origin $t$ is the pair (context $x = y_{t-L+1:t}$, target $y_{t+1:t+H}$), plus covariates: static ones (store id), ones known in advance (calendar, planned prices) and ones observed only in the past (weather measurements); see lesson 08. Windows from all series are mixed in mini-batches; this is a global model.

Per-window scaling. Series differ in level and spread, and one series changes its level over time. So each window is normalised with its own statistics, computed on the context only:

• mean scaling (DeepAR, Chronos): $\tilde x = x / s$ with $s = \frac1L\sum_{i}|x_i|$ (DeepAR uses $\nu = 1 + \frac1L\sum_i x_i$ for counts). It keeps zeros at zero, which matters for sales and solar power.
• standard scaling, called reversible instance normalisation (RevIN, Kim et al. 2022) when it is undone at the output: $\tilde x = (x - \mu_x)/\sigma_x$ with the mean $\mu_x$ and standard deviation $\sigma_x$ of the window; the forecast is $\hat y = \sigma_x \, g(\tilde x) + \mu_x$, where $g$ is the network.

Proposition (scaling makes the forecaster equivariant). Let $\hat y(x) = \sigma_x\, g\big((x - \mu_x)/\sigma_x\big) + \mu_x$ with any network $g$. For every $a > 0$ and $b \in \R$, $\hat y(ax + b) = a\,\hat y(x) + b$.

Proof.

1. The mean of $ax + b$ is $a\mu_x + b$, and its standard deviation is $a\sigma_x$ (because $a > 0$).
2. So $\big((ax + b) - (a\mu_x + b)\big)/(a\sigma_x) = (x - \mu_x)/\sigma_x$: the network sees the same input.
3. Then $\hat y(ax + b) = a\sigma_x\, g(\cdot) + a\mu_x + b = a\big(\sigma_x g(\cdot) + \mu_x\big) + b = a\,\hat y(x) + b$. $\blacksquare$

So a series in kilowatts and the same series in megawatts get the same forecast, in their own units. The same argument with $s$ shows that mean scaling gives $\hat y(ax) = a\,\hat y(x)$ for $a > 0$.

Output heads and losses. The head decides what the model estimates:

Head	Training loss	What the minimiser estimates
point	MSE $\frac1H\sum_h (y_{t+h} - \hat y_{t+h})^2$	the conditional mean
point	MAE $\frac1H\sum_h \lvert y_{t+h} - \hat y_{t+h}\rvert$	the conditional median
Gaussian	NLL $\tfrac12\log(2\pi\sigma^2) + \frac{(y - \mu)^2}{2\sigma^2}$	mean $\mu$ and variance $\sigma^2$
Student's t, negative binomial, mixtures	their NLL	heavy tails; counts; several shapes
quantile	pinball loss at levels $\tau$ (lesson 10)	the quantiles $q_\tau$

Why the Gaussian NLL estimates the mean and the variance

Fix the inputs and let $Y$ be the target, with mean $m$ and variance $v$. The expected loss is

$$\mathbb E\Big[\tfrac12\log(2\pi\sigma^2) + \tfrac{(Y-\mu)^2}{2\sigma^2}\Big] = \tfrac12\log(2\pi\sigma^2) + \frac{v + (m - \mu)^2}{2\sigma^2},$$

because $\mathbb E[(Y-\mu)^2] = \operatorname{Var}(Y) + (m - \mu)^2$.

1. For any $\sigma$, the only term with $\mu$ is $(m-\mu)^2/(2\sigma^2)$, smallest at $\mu = m$.
2. With $\mu = m$, the loss is $\tfrac12\log(2\pi\sigma^2) + v/(2\sigma^2)$. Its derivative in $\sigma^2$ is $\frac{1}{2\sigma^2} - \frac{v}{2\sigma^4}$, which is zero at $\sigma^2 = v$ (a minimum: negative before, positive after). $\blacksquare$

Channel independence. A multivariate series has $C$ variables (channels). A channel-independent model applies the same univariate network to each channel separately (shared weights, no information between channels). A channel-mixing model reads all channels together. Channel independence gives $C$ times more training windows and overfits less; it cannot use cross-channel effects. PatchTST is channel-independent; iTransformer and Chronos-2 mix channels with attention.

3.2 Recurrent networks and DeepAR #

The LSTM (long short-term memory) keeps a hidden state $h_t$ and a cell state $c_t$. With input $x_t$ (a row vector) and the logistic sigmoid $s$, in the notation of deep-learning lesson 08:

The forget gate $f_t$ decides how much of the old memory stays; the input gate $i_t$ decides how much of the candidate $g_t$ is written; the output gate $o_t$ decides what is shown as $h_t$. With $f_t$ near 1 the memory lasts about $1/(1 - f_t)$ steps; the interactive LSTM cell in deep-learning lesson 08 shows this. A GRU is a lighter variant with two gates.

Encoder–decoder. An encoder RNN reads the context and passes its final state to a decoder RNN, which produces the $H$ future steps (with known future covariates as decoder inputs). This sequence-to-sequence design was the standard neural forecaster before transformers.

DeepAR. Let $z_{i,t}$ be series $i$ at time $t$, with covariates $x_{i,t}$ and scale $\nu_i$. One RNN with shared weights runs over all series:

where $\ell$ is a likelihood with parameters $\theta$: Gaussian for real values ($\mu = w_\mu^\top h + b_\mu$, $\sigma = \log(1 + e^{w_\sigma^\top h + b_\sigma})$, so $\sigma > 0$), negative binomial for counts; the outputs are multiplied back by $\nu_i$. The paper uses $\nu_i = 1 + \frac{1}{t_0}\sum_{t\le t_0} z_{i,t}$.

Theorem (teacher forcing is maximum likelihood). For an autoregressive model, $\log p(z_1, \dots, z_T) = \sum_{t=1}^{T} \log p(z_t \mid z_1, \dots, z_{t-1})$. So minimising the sum of one-step negative log-likelihoods, each computed with the true previous values as inputs, is exactly maximum likelihood (see the probability-and-statistics track).

Proof. Apply the chain rule of probability $p(a, b) = p(a)\,p(b \mid a)$ $T - 1$ times and take logarithms. $\blacksquare$

Theorem (ancestral sampling gives samples of the joint forecast distribution). Draw $\hat z_{T+1} \sim p(\cdot \mid z_{1:T})$, then $\hat z_{T+2} \sim p(\cdot \mid z_{1:T}, \hat z_{T+1})$, and so on up to $\hat z_{T+H}$. Then $(\hat z_{T+1}, \dots, \hat z_{T+H})$ is one sample from $p(z_{T+1:T+H} \mid z_{1:T})$.

Proof. By induction on $H$. For $H = 1$ it is the definition. If the first $H - 1$ draws have the joint law $p(z_{T+1:T+H-1} \mid z_{1:T})$, the next draw multiplies it by $p(z_{T+H} \mid z_{1:T+H-1})$, and the chain rule says that this product is $p(z_{T+1:T+H} \mid z_{1:T})$. $\blacksquare$

So from $M$ paths you can estimate any summary: the quantiles at each horizon, the distribution of a sum over the horizon (total demand next month), or the probability of a stock-out. This is why DeepAR returns sample paths and not only marginal quantiles.

Echo state networks: a recurrent model trained without backpropagation

An echo state network (Jaeger 2001) fixes a random recurrent layer, the reservoir: $r_t = \tanh(W r_{t-1} + w_{\text{in}} u_t)$, with $W$ scaled so that its spectral radius is below 1 (then the state forgets its initial value: the "echo state property"). Only the linear readout $\hat u_{t+1} = \beta^\top [r_t; u_t; 1]$ is trained, by ridge regression (linear-algebra lesson 12 for least squares). It is a fast way to get a nonlinear recurrent forecaster in NumPy, and you build one in the notebook. On one short series it does not beat simple baselines reliably, which is the general lesson of this section.

3.3 Temporal convolutional networks #

Definition. A causal dilated convolution with kernel $w = (w_0, \dots, w_{k-1})$ and dilation $d \ge 1$ maps an input sequence $x$ to

A network stacks $n$ such layers (with nonlinearities between them), with dilations $d_1, \dots, d_n$.

Theorem (receptive field and causality). The output of layer $n$ at time $t$ depends only on the inputs $x_s$ with $t - (r - 1) \le s \le t$, where

It never depends on an input after $t$.

Proof.

1. Let $O_\ell$ be the set of offsets $t - s$ such that the output of layer $\ell$ at $t$ can depend on $x_s$. For one layer, $O_1 = \{0, d_1, \dots, (k-1)d_1\}$.
2. Layer $\ell$ at time $t$ reads layer $\ell - 1$ at times $t - d_\ell j$, $j = 0, \dots, k-1$. Each of those depends on inputs at offsets $O_{\ell-1}$ from its own time. So $O_\ell = \{\, d_\ell j + o : j \in \{0,\dots,k-1\},\ o \in O_{\ell-1}\,\}$.
3. All offsets are $\ge 0$ (every term is $\ge 0$): no future input is ever used. This is causality.
4. The largest offset grows by $(k-1)d_\ell$ per layer, so $\max O_n = (k-1)\sum_i d_i$. The inputs from offset 0 to $\max O_n$ are $\max O_n + 1 = r$ consecutive time steps. $\blacksquare$

Corollary (doubling dilations). With $d_i = 2^{i-1}$, $r = 1 + (k-1)(2^n - 1)$. The receptive field grows exponentially in the depth $n$, but the number of weights per channel is only $nk$. To cover a context of length $L$ you need about $\log_2\!\big(L/(k-1)\big)$ layers.

Counting a real TCN. In the TCN of Bai et al. (2018), one residual block contains two causal dilated convolutions with the same dilation (with weight normalisation, ReLU, dropout and a $1\times1$ convolution on the skip path when the widths differ). So a stack of blocks with dilations $d_b$ has $r = 1 + 2(k-1)\sum_b d_b$. WaveNet stacks layers with $k = 2$ and dilations $1, 2, \dots, 512$: $r = 1 + 1023 = 1024$ samples per stack.

Convolutions are fast to train (all time steps in parallel, unlike an RNN) and they see a fixed, known window. They cannot look further back than $r$: choose the depth so that $r$ covers the longest pattern you need, for example a weekly cycle of 168 hours.

3.4 Linear and MLP models: DLinear and N-BEATS #

DLinear (Zeng, Chen, Zhang and Xu, AAAI 2023). Let $A$ be the $L \times L$ moving-average matrix (kernel 25 in the reference code, the ends padded with copies of the first and last value). Split the context into a trend $Ax$ and a remainder $x - Ax$, and apply one linear layer to each:

NLinear, from the same paper, subtracts the last value of the window before one linear layer and adds it back after.

Proposition (DLinear is a linear map of the window). The set of DLinear forecasters $\{x \mapsto W_s(I - A)x + W_tAx + b\}$ equals the set of all affine maps $\{x \mapsto Wx + b\}$ from $\R^L$ to $\R^H$.

Proof.

1. Every DLinear forecaster is affine: it equals $Wx + b$ with $W = W_s(I - A) + W_tA$.
2. Every affine map $Wx + b$ is a DLinear forecaster: take $W_s = W_t = W$; then $W(I - A) + WA = W$. $\blacksquare$

So, fitted by plain least squares, DLinear and a linear regression on the window give the same forecast. The decomposition changes only the training: the penalty, the gradient-descent path and early stopping see different parameters. Its real message is different: a linear map of the last $L$ values is already a strong forecaster, because it can learn "the value one season ago, plus the trend over the last year".

Interactive figure: loading…

In their experiments on nine long-horizon benchmarks, these one-layer linear models were more accurate than the transformer models they compared (FEDformer, Autoformer, Informer, Pyraformer, LogTrans). The authors argue that attention is permutation-invariant, so ordering information must be added back and is partly lost. Later transformers (PatchTST, iTransformer) beat DLinear on most of these benchmarks, especially the large ones, so the right conclusion is not "transformers are useless" but "always include a linear baseline". TSMixer (Chen et al. 2023) stacks MLPs that mix along time and along channels.

N-BEATS (Oreshkin et al. 2020) is a deep stack of fully connected blocks. Block $\ell$ reads its input $x_\ell$ and outputs a backcast $\hat x_\ell$ (its explanation of the input) and a forecast $\hat y_\ell$. The next block gets the residual $x_{\ell+1} = x_\ell - \hat x_\ell$, and the final forecast is $\sum_\ell \hat y_\ell$. In the interpretable version, the outputs are expansions in fixed bases: low-degree polynomials (trend) and Fourier terms (seasonality), like the decomposition of lesson 01. N-BEATS was more accurate than the winner of the M4 competition on the M4 data. N-HiTS (Challu et al. 2023) adds pooling of the input at several rates and interpolation of the outputs, for long horizons.

3.5 Transformers for time series #

Self-attention over $N$ tokens of width $d$ costs $O(N^2 d)$ time and $O(N^2)$ memory for the score matrix (deep-learning lesson 11). Early long-horizon models changed the attention: Informer (Zhou et al. 2021) keeps only the most informative queries ("ProbSparse", $O(L\log L)$); Autoformer (Wu et al. 2021) builds the trend–seasonal decomposition into the network and replaces attention by an auto-correlation over time lags.

PatchTST (Nie et al., ICLR 2023) changes the tokens instead:

1. Normalise each channel's window (RevIN-style instance normalisation).
2. Pad the window with $S$ copies of its last value and cut it into patches of length $P$ with stride $S$:

3. Map each patch linearly to a $d$-dimensional token, add a position embedding, and run a transformer encoder.
4. Flatten the output tokens and map them linearly to the $H$ forecasts. All channels share the weights (channel independence).

Since $N \approx L/S$, the number of attention scores drops by a factor of about $S^2$; with $L = 336$, $P = 16$, $S = 8$ there are 42 tokens. A patch of 16 values also carries local shape (a rise, a peak), which a single value does not.

iTransformer (Liu et al., ICLR 2024) inverts the roles: each variable's whole context becomes one token, and attention runs across variables. The Temporal Fusion Transformer (TFT; Lim et al. 2021) combines LSTM layers for local processing, attention for long-range patterns, gating, and a variable-selection network for static, known-future and observed-only covariates; it outputs quantiles, and its attention weights and variable weights are used to interpret the forecast.

3.6 State-space layers #

S4 and Mamba (deep-learning lesson 16) are layers built on a linear state-space recurrence $s_t = \bar A s_{t-1} + \bar B u_t$, $o_t = C s_t$: the same form as the state equation of lesson 07, but without the noise terms and without a Kalman filter. They are deep layers trained by gradient descent; a time-invariant one can be computed as one long convolution (parallel training) or as a recurrence (cheap step-by-step prediction). In forecasting they are one more backbone, between RNNs and transformers in cost.

3.7 Foundation models for time series #

A time-series foundation model is a large network pretrained on many series from many domains (and often on synthetic series), and then used on new series. Zero-shot forecasting means: no gradient update on the target data; the model reads only the context of the target series at prediction time, like in-context learning in a language model. Most models can also be fine-tuned on the target data.

Interactive figure: loading…

The map shows three design choices:

• Input representation. Quantised values with a fixed vocabulary (Chronos); patches of values (TimesFM, Moirai, MOMENT, Chronos-Bolt, Chronos-2, TiRex, Sundial, Toto, Moirai 2.0); single values (Time-MoE); lag features (Lag-Llama); or tabular features for a pretrained tabular model (TabPFN-TS, which uses TabPFN-v2).
• Output head. A categorical distribution over bins, sampled token by token (Chronos); a point forecast (TimesFM 1.0, Time-MoE); a parametric distribution (Lag-Llama: Student's t; Moirai and Moirai-MoE: a mixture; Toto: a mixture of Student's t); quantiles for many steps at once (Chronos-Bolt, Chronos-2, TiRex, Moirai 2.0, TimesFM 2.5); or a generative flow-matching head that samples whole patches (Sundial).
• Architecture. Mostly transformers (encoder–decoder T5 for Chronos; encoder-only for Moirai and Chronos-2; decoder-only for TimesFM, Lag-Llama, Moirai 2.0, Sundial, Toto), sometimes with a sparse mixture of experts (Time-MoE, Moirai-MoE), a recurrent xLSTM (TiRex), or an in-context tabular model (TabPFN-TS). Chronos-2 adds "group attention" across related series and covariates, so it can use multivariate inputs and covariates zero-shot.

TimeGPT-1 (Garza et al. 2023) was presented by its authors as the first foundation model for time series.

Chronos tokenisation. Given a context $x_1, \dots, x_C$:

1. Mean scaling: $s = \frac1C\sum_{i=1}^C |x_i|$ (if $s = 0$, use $s = 1$), and $\tilde x_i = x_i/s$.
2. Quantisation: choose $B$ bin centres $c_1 < \dots < c_B$ evenly spaced on $[-15, 15]$, so the spacing is $\Delta = 30/(B-1)$; the token of $\tilde x_i$ is the index of the nearest centre. Chronos uses $B = 4094$ bins plus two special tokens (PAD for padding and missing values, EOS for the end), a vocabulary of 4096.
3. Model: a T5 encoder–decoder is trained with cross-entropy to predict the next token, like a language model.
4. Forecast: sample tokens autoregressively, map each token back to its centre, multiply by $s$.

Proposition (precision of the tokens). If $|x_i| \le 15s$, the value recovered from the token, $\hat x_i = s\,c_{\text{token}}$, satisfies $|x_i - \hat x_i| \le \dfrac{15\,s}{B - 1}$.

Proof. The scaled value $\tilde x_i$ lies in $[-15, 15]$, so it is between two neighbouring centres at distance $\Delta$, and the nearest centre is at most $\Delta/2$ away: $|\tilde x_i - c| \le 15/(B-1)$. Multiply by $s$. $\blacksquare$

Two consequences follow. Values outside $[-15s, 15s]$ are clipped to the end bins, so a strong trend that grows beyond 15 times the context's mean level cannot be forecast. And precision is relative to the level, not to the variation: a series near 1000 that moves by $\pm 3$ has $s \approx 1000$ and a bin width of about $30 \cdot 1000/4093 \approx 7.3$, so these movements fall into one or two bins and are lost (Ansari et al. 2024, section 5.7, discuss this limitation).

Evaluation as of 2026. Pretrained models are compared on public benchmarks such as GIFT-Eval (Aksu et al. 2024: 23 datasets, about 144,000 series, with a separate non-leaking pretraining corpus) and fev-bench (Shchur et al. 2025: 100 tasks, 46 with covariates, with bootstrapped confidence intervals). Three cautions:

• Contamination. If a model's pretraining data overlap with a benchmark's test data, the "zero-shot" score is really an in-domain score. The Chronos paper itself reports its in-domain benchmark separately from its zero-shot benchmark; GIFT-Eval publishes a non-leaking pretraining set for this reason.
• Ranks change monthly. New versions appear several times a year (for example TimesFM 1.0 in 2023 and the 2.5 checkpoint in 2025). Name the version and the date whenever you report a result.
• Your data are not the benchmark. Always backtest the model on your own series with a rolling origin (lesson 03) against seasonal naive, ETS (lesson 04) and a tuned boosting model (lesson 08).

Answers to section 1. (1) 16, shown in section 2. (2) The linear layer: in Zeng et al. (2023), DLinear and NLinear were more accurate than the long-horizon transformers they compared on all nine benchmarks. (3) Two values closer than about $30 \cdot 50/4093 \approx 0.37$ (one bin width) can fall into the same bin; each is recovered within about $0.18$.

4. Worked example #

(a) Receptive field and causality. Take $k = 3$ and dilations $1, 2, 4, 8$ (one convolution per layer).

The output at $t = 40$ depends on $x_{10}, \dots, x_{40}$: offsets 0 to 30. The offsets are $j_1 + 2j_2 + 4j_3 + 8j_4$ with $j_i \in \{0, 1, 2\}$. The largest is $2 \cdot 15 = 30$, and every number from 0 to 30 has this form. For example, for 29 take the largest coefficient that fits at each dilation, from the top: $29 - 2 \cdot 8 = 13$, $13 - 2 \cdot 4 = 5$, $5 - 2 \cdot 2 = 1$, $1 - 1 \cdot 1 = 0$, so $j = (1, 2, 2, 2)$ and $1 + 4 + 8 + 16 = 29$ ✓. No offset is negative, so no input after $t = 40$ is used. As TCN residual blocks (two convolutions per block, same dilations), the receptive field becomes $1 + 2 \cdot 2 \cdot 15 = 61$.

(b) Patches. PatchTST with $L = 336$, $P = 16$, $S = 8$:

Attention compares $42^2 = 1764$ pairs instead of $336^2 = 112{,}896$: $64 = 8^2$ times fewer.

(c) Tokenising four values. Context $x = (12, 15, 9, 14)$. To compute by hand, use a toy grid of $B = 13$ centres on $[-3, 3]$ (spacing $0.5$) instead of the real one.

1. Scale: $s = (12 + 15 + 9 + 14)/4 = 12.5$, so $\tilde x = (0.96, 1.20, 0.72, 1.12)$.
2. Nearest centres: $0.96 \to 1.0$; $1.20 \to 1.0$ (distance $0.2$, versus $0.3$ to $1.5$); $0.72 \to 0.5$ (distance $0.22$, versus $0.28$ to $1.0$); $1.12 \to 1.0$. With centres numbered from 0 at $-3$, the tokens are $(8, 8, 7, 8)$.
3. Back to values: $12.5 \cdot (1.0, 1.0, 0.5, 1.0) = (12.5, 12.5, 6.25, 12.5)$. The errors are $(0.5, 2.5, 2.75, 1.5)$, all below the bound $s \cdot 0.25 = 3.125$.
4. With the real grid ($B = 4094$ on $[-15, 15]$) the bound is $15 \cdot 12.5/4093 \approx 0.046$; the largest error for these four values is $0.044$.

(d) One LSTM step. Scalar LSTM with input $x_1 = 2$, previous $h_0 = 0$, $c_0 = 1$, and weights chosen by hand: $f = s(0 \cdot x + 2)$, $i = s(1 \cdot x - 1)$, $g = \tanh(0.5\,x)$, $o = s(0 \cdot x + 0)$ (the $h_0$ terms vanish because $h_0 = 0$).

1. Gates: $f = s(2) = 0.881$, $i = s(1) = 0.731$, $g = \tanh(1) = 0.762$, $o = s(0) = 0.5$.
2. Cell: $c_1 = f\,c_0 + i\,g = 0.881 \cdot 1 + 0.731 \cdot 0.762 = 0.881 + 0.557 = 1.438$.
3. Hidden state: $h_1 = o \tanh(c_1) = 0.5 \cdot 0.893 = 0.447$.

The old memory $c_0 = 1$ survives at 88% because the forget-gate bias is 2.

5. In code #

NumPy checks the four parts of the worked example, and the DLinear proposition on the "monthly demand" series.

import numpy as np

# (a) receptive field of stacked causal dilated convolutions, by an impulse test
def causal_conv(x, w, d):
    """y[t] = sum_j w[j] * x[t - d*j]  (zero padding on the left: only the past is used)."""
    y = np.zeros(len(x))
    for j, wj in enumerate(w):
        y[d * j:] += wj * x[: len(x) - d * j]
    return y

rng = np.random.default_rng(0)
k, dil, T = 3, [1, 2, 4, 8], 64
W = [rng.normal(size=k) for _ in dil]
def net(x):
    for w, d in zip(W, dil):
        x = np.tanh(causal_conv(x, w, d))
    return x
t = 40
seen = [s for s in range(T) if net(np.eye(T)[s])[t] != net(np.zeros(T))[t]]
print(len(seen), min(seen), max(seen))   # 31 10 40  ->  r = 1 + (k-1)*15 = 31, nothing after t

# (b) Chronos-style tokenization: mean scaling + uniform bins on [-15, 15]
def tokenize(ctx, B=4094, lo=-15., hi=15.):
    s = np.mean(np.abs(ctx)) or 1.0
    centres = np.linspace(lo, hi, B)
    ids = np.abs(ctx[:, None] / s - centres[None, :]).argmin(axis=1)
    return ids, s, centres
ctx = np.array([12., 15., 9., 14.])
ids, s, c = tokenize(ctx, B=13, lo=-3., hi=3.)          # the toy grid of the worked example
print(s, ids, c[ids] * s)                                # 12.5 [8 8 7 8] [12.5  12.5   6.25 12.5 ]
ids, s, c = tokenize(ctx)                                # the real grid: B = 4094
print(np.abs(c[ids] * s - ctx).max().round(4), round(15 * s / 4093, 4))   # 0.0436 0.0458

# (c) one LSTM step with scalar weights (gate order i, f, g, o)
sig = lambda z: 1 / (1 + np.exp(-z))
x, h, c0 = 2.0, 0.0, 1.0
i, f = sig(1 * x - 1), sig(0 * x + 2)
g, o = np.tanh(0.5 * x), sig(0 * x + 0)
c1 = f * c0 + i * g
print(np.round([i, f, g, o, c1, o * np.tanh(c1)], 3))  # [0.731 0.881 0.762 0.5   1.438 0.447]

# (d) DLinear and a plain linear map on the window give the same least-squares fit
tt = np.arange(1, 121)
y = 100 + 0.5 * tt + 10 * np.sin(2 * np.pi * tt / 12) + np.random.default_rng(0).normal(0, 3, 120)
Lw, H, kma = 24, 12, 23
A = np.zeros((Lw, Lw))                                   # moving average, ends padded by copies
for r in range(Lw):
    for q in range(r - kma // 2, r + kma // 2 + 1):
        A[r, min(max(q, 0), Lw - 1)] += 1 / kma
X = np.array([y[s:s + Lw] for s in range(108 - Lw - H + 1)])
Y = np.array([y[s + Lw:s + Lw + H] for s in range(108 - Lw - H + 1)])
one = np.ones((len(X), 1))
F_lin = np.hstack([X, one])
F_dl = np.hstack([X - X @ A.T, X @ A.T, one])            # remainder part, trend part, bias
ctx = np.r_[y[108 - Lw:108]]
f_lin = np.r_[ctx, 1] @ np.linalg.lstsq(F_lin, Y, rcond=None)[0]
f_dl = np.r_[ctx - A @ ctx, A @ ctx, 1] @ np.linalg.lstsq(F_dl, Y, rcond=None)[0]
print(np.abs(f_lin - f_dl).max() < 1e-6)                # True: same function class, same fit
scale = np.mean(np.abs(y[12:108] - y[:96]))              # MASE scale: in-sample seasonal naive
print(round(np.mean(np.abs(f_lin - y[108:])) / scale, 2),
      round(np.mean(np.abs(y[96:108] - y[108:])) / scale, 2))   # 0.51 1.2

The window here is 24 months, so the moving-average kernel is 23 (the largest odd number not above $L$). The linear model's MASE of 0.51 against 1.2 for seasonal naive comes from one origin only; the notebook repeats it with a rolling origin. The notebook builds every model of this lesson in NumPy (no deep-learning library), and an optional cell shows how to call a pretrained model in Colab.

6. Where it shows up #

• Retail and supply chains. DeepAR-style global models with negative binomial or Student's t heads forecast demand for many products; the sample paths give the distribution of total demand over a lead time, which sets safety stock. In the M5 competition (lesson 08) most top entries used gradient boosting (LightGBM), and some of them combined it with neural networks.
• Energy and traffic. Hourly load, solar and wind power, and road occupancy are the standard long-horizon benchmarks (electricity, traffic, weather) on which DLinear, PatchTST, iTransformer and N-HiTS were tested.
• Cloud and observability. Toto (Datadog, 2025) is pretrained on monitoring metrics and comes with the BOOM benchmark of observability series; forecasts and their intervals drive capacity planning and alerts (lesson 11).
• ML for science: hydrology. A single LSTM trained on 531 river basins of the CAMELS data set, with meteorological inputs and static catchment attributes, predicted river flow better than hydrological models calibrated basin by basin (Kratzert et al. 2019). This is the "global model" idea in a physical science.
• ML for science: weather and climate. Learned weather models are deep autoregressive forecasters of whole atmospheric fields: they apply the same network step after step, as in ancestral sampling, and ensembles of such runs give probabilistic forecasts (see the ml-for-science track).
• Forecasting tools. Pretrained models are available in libraries such as AutoGluon-TimeSeries (Chronos) and GluonTS (DeepAR, TFT and others), so a zero-shot forecast is often the first thing a team tries.

7. Common mistakes #

• A non-causal convolution or moving average. "Same" padding in a convolution, or a centred moving average, uses values after $t$. The backtest looks excellent and the model fails in production. Pad on the left only.
• Scaling with statistics that include the future. Normalise each window with the context only, and invert the scaling on the output. Scaling with the whole series' mean and standard deviation is leakage.
• Counting one convolution per TCN block. A TCN residual block has two convolutions, so $r = 1 + 2(k-1)\sum_b d_b$.
• Calling the "mean fed back" path a probabilistic forecast. Only sample paths (or a quantile or distribution head) give uncertainty, and for a nonlinear model the fed-back mean is not even the mean.
• Comparing a deep model only with other deep models. Include seasonal naive, ETS or ARIMA, a linear model on the window and a boosted model. Many published gains disappear against a tuned simple baseline.
• Trusting a zero-shot score on a benchmark that was in the pretraining data. Check the pretraining corpus, prefer benchmarks with non-leaking splits, and backtest on your own data.
• Feeding a trending series with a large offset to a mean-scaled tokeniser without thinking. Precision is about $0.0073\,s$ per bin, relative to the level $s$; small movements around a large level are lost, and values beyond $15s$ are clipped.
• Training a large network on one short series. With 100 monthly values there are fewer than 100 training windows. A model with thousands of weights will fit noise; use a classical model, a linear model, or a pretrained model zero-shot.

8. Check yourself #

1. A TCN has 3 residual blocks with kernel size 3 and dilations 1, 2 and 4 (two convolutions per block). What is its receptive field? How many blocks would you need for a weekly cycle in hourly data (168 steps)?
   Answer

   $r = 1 + 2(k-1)(1 + 2 + 4) = 1 + 4 \cdot 7 = 29$. With $n$ blocks and doubling dilations, $r = 1 + 4(2^n - 1)$: $n = 5$ gives $125$, $n = 6$ gives $253 \ge 168$. So 6 blocks.
2. RevIN computes $\mu_x$ and $\sigma_x$ of the context, feeds $(x - \mu_x)/\sigma_x$ to the network $g$ and outputs $\sigma_x g(\cdot) + \mu_x$. Context $(10, 12, 14, 12)$; the network outputs $(0.5, 1.0)$. What is the forecast? What is it if every context value is increased by 100?
   Answer

   $\mu_x = 12$, $\sigma_x = \sqrt{(4 + 0 + 4 + 0)/4} = \sqrt 2 \approx 1.414$. Forecast $12 + 1.414 \cdot (0.5, 1.0) = (12.71, 13.41)$. After adding 100 the normalised input is the same, so the network gives the same output and the forecast is $(112.71, 113.41)$: the forecaster is shift-equivariant.
3. PatchTST with $L = 512$, $P = 16$, $S = 8$: how many tokens, and how many attention scores per head and layer, compared with one token per time step?
   Answer

   $N = \lfloor 496/8 \rfloor + 2 = 62 + 2 = 64$ (the "PatchTST/64" of the paper). $64^2 = 4096$ scores instead of $512^2 = 262{,}144$: 64 times fewer.
4. Why can a DeepAR model trained on 10,000 products forecast a new product that has only 8 weeks of history, while an ARIMA model cannot?
   Answer

   DeepAR is a global model: its weights are shared across all series and learned from all of them, and the per-series scale $\nu$ puts the new series on the common scale. It only needs a short context to set the hidden state. ARIMA is a local model: its parameters must be estimated from the 8 values of the new series, which is far too few.
5. Write the DLinear forecast as one matrix $W$ times the window, and explain why the decomposition can still change the result when the model is trained with weight decay.
   Answer

   $\hat y = (W_s(I - A) + W_tA)\,x + b$, so $W = W_s(I - A) + W_tA$. Weight decay penalises $\|W_s\|^2 + \|W_t\|^2$, which is not a function of $W$ alone: two parameter pairs with the same $W$ have different penalties. So the penalised optimum (and the path of gradient descent) depends on the parametrisation, while the set of reachable forecasters does not.
6. A vendor reports that its foundation model beats all others on a public benchmark. List three questions you ask before you use it.
   Answer

   Was any of the benchmark's test data in the pretraining corpus (contamination)? Which model version and which date does the result refer to, and are the differences larger than their confidence intervals? How does it do on a rolling-origin backtest of *my* series against seasonal naive, ETS and a boosted model, for my horizons and my metric?

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is a causal convolution, and why must a forecaster use one?

   Show answer

2. Give the receptive field of $n$ causal convolution layers with kernel size $k$ and dilations $d_1, \dots, d_n$.

   Show answer

3. Describe DeepAR in one sentence.

   Show answer

4. Teacher forcing versus ancestral sampling: what is fed back at the next step?

   Show answer

5. What does per-window scaling such as RevIN do, and what must you not do with it?

   Show answer

6. PatchTST with $L = 336$, $P = 16$, $S = 8$: how many tokens?

   Show answer

7. What is channel independence?

   Show answer

8. How does Chronos turn a real value into a token?

   Show answer

9. What does "zero-shot forecasting" mean?

   Show answer

10. What is benchmark contamination for a foundation model?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Would you use a transformer for this forecasting problem?" — It depends on the data: with one or a few short series, no (ETS, ARIMA, a linear model or a pretrained model zero-shot); with many related series, long contexts and covariates, a global deep model can help. In every case, compare with seasonal naive and a boosted model in a rolling-origin backtest.
• "Why can a linear model beat a transformer on long-horizon benchmarks?" — The series are dominated by trend and seasonality, which a linear map of the window captures; point-wise tokens carry little information and attention is permutation-invariant; the transformers overfit. Patching and channel independence (PatchTST) fixed much of this.
• "How do you get prediction intervals from a neural forecaster?" — A distribution head trained by NLL (DeepAR), a quantile head trained with the pinball loss (TFT, Chronos-Bolt), sample paths from ancestral sampling, or conformal intervals on top of any point forecaster (lesson 10).
• "Compute the receptive field of this TCN." — $1 + (k-1)\sum d$ per convolution; count both convolutions in each residual block.
• "How do time-series foundation models tokenise numbers?" — Chronos: mean scaling plus uniform bins, then a language model with cross-entropy. Most others: patches of scaled values mapped linearly to embeddings, with a quantile, distribution or point head.

10. Practice #

The notebook for this lesson runs in Colab with NumPy and matplotlib only. It has five exercises, each with a check cell and a folded solution:

1. causal_dilated_conv(x, w, d) and receptive_field(k, dilations), checked with an impulse response: only the inputs inside the receptive field change the output, and none after $t$.
2. lstm_cell(x, h, c, W, U, b): one forward step of an LSTM, checked against the worked example and a vector case.
3. An echo state network (a fixed random reservoir with a ridge-regression readout) on the "monthly demand" series, backtested with a rolling origin against seasonal naive.
4. dlinear_fit(y, L, H) by least squares with the moving-average split, compared with seasonal naive by rolling origin, and checked against plain linear regression.
5. chronos_tokenize(context, n_bins, limits) and detokenize, with the error bound $15s/(B-1)$ and the clipping of out-of-range values.

11. Watch and read #

Long Short-Term Memory (LSTM), Clearly ExplainedStatQuest with Josh Starmer · 21 min

The gates of the LSTM with small numbers, step by step.

Lecture 5.4 - CNNs for Sequential DataDLVU (Deep Learning, Vrije Universiteit Amsterdam) · 28 min

Causal and dilated convolutions and temporal convolutional networks as an alternative to RNNs.

Building makemore Part 5: Building a WaveNetAndrej Karpathy · 56 min

Builds the WaveNet-style tree of dilated layers by hand in PyTorch; the same hierarchy as the TCN figure.

PatchTST (ICLR 2023)Yuqi Nie (first author) · 5 min

Patching and channel independence, from the author of PatchTST.

Chronos: Time series forecasting in the age of pretrained modelsAutoML Space (talk by Oleksandr Shchur, AutoML Summer School 2024) · 49 min

How Chronos tokenises values, how it is trained, and how pretrained models are evaluated.

Lag-Llama: Towards Foundation Models for Probabilistic Time Series ForecastingServiceNow (talk by Arjun Ashok) · 18 min

A decoder-only foundation model with lag features and a Student's t head.

Papers (all free on arXiv).

• Salinas, Flunkert and Gasthaus, DeepAR: Probabilistic forecasting with autoregressive recurrent networks (2017; IJF 2020).
• Bai, Kolter and Koltun, An empirical evaluation of generic convolutional and recurrent networks for sequence modeling (2018), the TCN paper; van den Oord et al., WaveNet (2016).
• Oreshkin et al., N-BEATS (ICLR 2020); Challu et al., N-HiTS (AAAI 2023).
• Zeng, Chen, Zhang and Xu, Are transformers effective for time series forecasting? (AAAI 2023), the DLinear paper; Chen et al., TSMixer (2023).
• Zhou et al., Informer (AAAI 2021); Wu et al., Autoformer (NeurIPS 2021); Lim et al., Temporal Fusion Transformers (IJF 2021).
• Nie et al., A time series is worth 64 words (PatchTST) (ICLR 2023); Liu et al., iTransformer (ICLR 2024).
• Kim et al., Reversible instance normalization (RevIN) (ICLR 2022).
• Foundation models: TimeGPT-1, Lag-Llama, TimesFM, Moirai, MOMENT, Chronos, Time-MoE, Moirai-MoE, TabPFN-TS, Sundial, Toto, TiRex, Chronos-2, Moirai 2.0. Model cards: Chronos-Bolt, TimesFM 2.5.
• Benchmarks: Aksu et al., GIFT-Eval (2024) and its leaderboard; Shchur et al., fev-bench (2025) and its leaderboard (both accessed November 2026; ranks change often).
• Survey: Benidis et al., Deep learning for time series forecasting: tutorial and literature survey (ACM Computing Surveys 2022). Library: Alexandrov et al., GluonTS (JMLR 2020).
• Hydrology: Kratzert et al., Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets (HESS 2019).
• Hyndman and Athanasopoulos, Forecasting: Principles and Practice, 3rd ed., free online: the benchmarks and evaluation that every deep model must beat.