Regularization: dropout, weight decay, label smoothing, augmentation, early stopping

Before this lesson: lesson 02, training a network (softmax, cross-entropy, SGD), lesson 03, backpropagation and lesson 05, normalization. From other tracks: L2 regularization and the bias–variance trade-off (ML fundamentals track), Adam (optimization track), and the SVD (linear algebra lesson 08) for the last section. Time: about 55 minutes, plus the notebook.

---

Why this matters #

• LLM pretraining uses AdamW with weight decay 0.1 (GPT-3, LLaMA). Using Adam(weight_decay=...) instead is a common bug: it is L2, not weight decay.
• The original Transformer used dropout 0.1 and label smoothing 0.1.
• Vision transformers trained on ImageNet only (DeiT) need heavy augmentation, label smoothing and stochastic depth; without them they overfit.
• Small scientific datasets rely on early stopping on a validation set, and on augmentations that respect the physics of the data.

1. Predict first #

2. The picture #

Dropout, simplest case #

Take one unit with value $x$ and a drop probability $p$. At training time, keep the unit with probability $1 - p$ and multiply it by $1/(1 - p)$; otherwise set it to 0. The expected output is

At eval time, use $x$ itself. So the network sees the same mean in both modes, but during training it cannot rely on any single unit: any unit may be missing.

Interactive figure: loading…

Weight decay, simplest case #

Weight decay multiplies every weight by $1 - \eta\lambda$ at every step ($\eta$ the learning rate, $\lambda$ the decay rate). Weights that the loss does not need decay towards 0. For SGD this is the same as adding $\frac\lambda2\|\theta\|^2$ to the loss (L2 regularization). Adam divides each gradient coordinate by its own running size. If the penalty's gradient goes through Adam, it is divided too, and the shrinkage is no longer the same for every weight. Section 3 shows this in a figure.

Label smoothing #

With label smoothing, the target is not one-hot. The true class gets a bit less than 1, and the other classes get a bit more than 0. The model is never asked to be 100% sure, so the logits stay finite.

Interactive figure: loading…

Augmentation and early stopping #

If the label does not change when an input is rotated, show the model rotated inputs: the invariance enters through the data. Early stopping limits how far the weights travel from their small initial values, which acts like L2 (section 3, last part).

3. Definitions and main results #

A regularizer is a change to the training procedure (the loss, the update rule, the data or the stopping time) that aims to lower the test error, possibly at the cost of a higher training error. Lesson 17 discusses why large networks generalize at all.

Inverted dropout #

With drop probability $p$, draw $m_i \sim \text{Bernoulli}(1 - p)$ independently for each entry, and set

• Expectation: $\mathbb{E}[y_i] = x_i$. Variance: $\operatorname{Var}(y_i) = x_i^2\, \frac{p}{1-p}$.
• Backward: $\bar x = \frac{m}{1-p} \odot \bar y$, with the same mask as in the forward pass.
• Why "inverted". The original dropout (Srivastava et al., 2014) did not scale during training and multiplied the weights by the keep probability at test time. Scaling during training leaves the eval network unchanged, so nothing special happens at inference.
• Ensemble view. Each mask selects one of $2^n$ thinned subnetworks that share weights. Eval mode approximates the average of their predictions. When everything after the dropout layer is linear, the average is exact (this is the figure above).

Theorem (dropout on the inputs of linear regression is a ridge penalty). Let $\hat y_i = \sum_j x_{ij} w_j m_{ij}/(1-p)$ with independent masks. Then

Proof.

1. $\mathbb{E}[m_{ij}/(1-p)] = 1$, so $\mathbb{E}[\hat y_i] = x_i^\top w$.
2. $\operatorname{Var}(m_{ij}/(1-p)) = (1-p)p/(1-p)^2 = p/(1-p)$, and the masks are independent, so $\operatorname{Var}(\hat y_i) = \frac{p}{1-p}\sum_j x_{ij}^2 w_j^2$.
3. For any random $\hat y_i$: $\mathbb{E}(y_i - \hat y_i)^2 = (y_i - \mathbb{E}\hat y_i)^2 + \operatorname{Var}(\hat y_i)$. Sum over $i$. $\blacksquare$

So dropout acts like a ridge penalty whose strength for $w_j$ grows with the energy $\sum_i x_{ij}^2$ of feature $j$.

Dropout before BatchNorm. In training, dropout inflates the second moment: $\mathbb{E}[y_i^2] = x_i^2/(1-p)$. In eval mode it is $x_i^2$. A BatchNorm layer after dropout learns running statistics from the inflated training activations, so they do not match the eval activations. Li et al. (2019) call this the variance shift. Place dropout after the last BatchNorm, or avoid mixing the two.

L2 regularization and weight decay #

Theorem (they are the same for SGD). SGD with learning rate $\eta$ on $\mathcal{L}(\theta) + \frac\lambda2\|\theta\|^2$ makes exactly the same steps as SGD on $\mathcal{L}$ with weight decay, $\theta \leftarrow (1 - \eta\lambda)\theta - \eta\nabla\mathcal{L}(\theta)$.

Proof.

1. The gradient of the penalized loss is $\nabla\mathcal{L}(\theta) + \lambda\theta$.
2. One step: $\theta - \eta(\nabla\mathcal{L} + \lambda\theta) = (1 - \eta\lambda)\theta - \eta\nabla\mathcal{L}$. $\blacksquare$

The decay per step is $\eta\lambda$: the best L2 coefficient depends on the learning rate.

Adam with L2. Adam keeps running averages of the gradient, $m_t$, and of its square, $v_t$, corrects their start-up bias ($\hat m_t, \hat v_t$), and steps $\theta \leftarrow \theta - \eta\, \hat m_t/(\sqrt{\hat v_t} + \epsilon)$. With L2, the term $\lambda\theta$ is added to the gradient before these averages. So the decay moves coordinate $i$ by about $\eta\lambda\theta_i/\sqrt{\hat v_{t,i}}$:

• weights with large past gradients (large $\hat v$) are decayed less;
• weights with small gradients are decayed more.

Loshchilov and Hutter (2019, Proposition 2) prove that, for any preconditioner that is not a multiple of $I$, no L2 coefficient makes the method equal to weight decay.

AdamW decouples the decay from the gradient step: the moments use $\nabla\mathcal{L}$ only, and

with base learning rate $\alpha$ and schedule multiplier $\eta_t$ (their Algorithm 2). Every weight now shrinks at the same rate. PyTorch's AdamW first applies $\theta \leftarrow \theta(1 - \text{lr}\cdot\lambda)$ and then the Adam step, so its decay per step is lr × weight_decay and follows the learning-rate schedule. PyTorch's Adam(weight_decay=...) is L2, not AdamW.

Interactive figure: loading…

With normalization layers. A weight followed by BatchNorm or LayerNorm is scale-invariant (lesson 05): decay does not change the function, but it keeps $\|W\|$ small and so keeps the effective learning rate $\eta/\|W\|^2$ up. Biases and normalization gains $\gamma$ are usually excluded from decay.

Label smoothing #

With $K$ classes, true class $y$ and smoothing $\varepsilon$ (PyTorch convention):

So the true class gets $1 - \varepsilon + \varepsilon/K$ and every other class gets $\varepsilon/K$. (Some texts give the true class $1 - \varepsilon$ and the others $\varepsilon/(K - 1)$.)

Gradient. Since $\log p_c = z_c - \log\sum_j e^{z_j}$,

because $\sum_c q_c = 1$. This is the same form as plain cross-entropy, with $q$ in place of the one-hot target.

Minimal loss and bounded logits. For a fixed $q$, $\ell = H(q) + \operatorname{KL}(q\,\|\,p)$, where $H(q) = -\sum_c q_c \log q_c$ is the entropy of $q$ and $\operatorname{KL} \ge 0$ (Gibbs' inequality). So the loss is minimal exactly at $p = q$, and it never goes below $H(q) > 0$. At $p = q$, the logit gap between the true class and any other class is

which is finite. Without smoothing the optimum $p = \text{onehot}$ needs an infinite gap, so the logits keep growing and the model becomes overconfident. Müller et al. (2019) found that label smoothing improves calibration, but makes the network a worse teacher for knowledge distillation.

Data augmentation encodes invariances #

Suppose the label does not change under a finite group $G$ of transformations (rotations by 90°, flips, cyclic shifts): $y(gx) = y(x)$ for every $g \in G$. With $g$ drawn uniformly from $G$, data augmentation minimizes

For a loss that is convex in the prediction, the averaged predictor $\bar f(x) = \mathbb{E}_g f(gx)$ is $G$-invariant, and by Jensen's inequality $\ell(\bar f(x), y) \le \mathbb{E}_g\, \ell(f(gx), y)$. Since $\bar f$ is invariant, $R_{\text{aug}}(\bar f) = \mathbb{E}_x\,\ell(\bar f(x), y) \le R_{\text{aug}}(f)$: the augmented objective prefers invariant predictors.

Additive input noise is another simple case. For linear regression on $n$ samples, with independent noise $e_i \sim \mathcal{N}(0, \sigma^2 I)$ added to each input $x_i$,

which is ridge again (Bishop, 1995). An equivariant architecture (a CNN, or an E(3)-equivariant graph network) builds the same symmetry into the model instead of the data (lessons 07 and 13).

An augmentation must keep the label and the physics. Rotating a handwritten "6" by 180° gives a "9". Flipping a regional weather map north–south makes cyclones turn the wrong way for their hemisphere. Rescaling a seismic waveform changes the magnitude it encodes.

Early stopping acts like ridge #

Take least squares, $\mathcal{L}(w) = \frac1{2n}\|Xw - y\|^2$, with gradient $Sw - b$, where $S = X^\top X/n$ and $b = X^\top y/n$. Assume $S$ is invertible. Write $S = V\operatorname{diag}(s)V^\top$ (its eigendecomposition; the $s_i$ are the squared singular values of $X/\sqrt n$) and $w^\star = S^{-1}b$. Ridge with strength $\lambda$ minimizes $\mathcal{L}(w) + \frac\lambda2\|w\|^2$, so it solves $(S + \lambda I)w = b$.

Theorem (filter factors). Gradient descent from $w_0 = 0$ with step $\eta < 2/s_{\max}$ gives, along each eigenvector with eigenvalue $s$, the component $\big(1 - (1 - \eta s)^t\big)\,w^\star_s$ after $t$ steps. Ridge with strength $\lambda$ gives $\frac{s}{s + \lambda}\,w^\star_s$.

Proof.

1. $w_{t+1} - w^\star = w_t - \eta(Sw_t - b) - w^\star = (I - \eta S)(w_t - w^\star)$, because $b = Sw^\star$.
2. Hence $w_t - w^\star = (I - \eta S)^t(0 - w^\star)$, that is, $w_t = \big(I - (I - \eta S)^t\big)w^\star$.
3. Ridge solves $(S + \lambda I)w = b = Sw^\star$, so $w_\lambda = (S + \lambda I)^{-1} S\, w^\star$.
4. In the eigenbasis both matrices are diagonal, which gives the two filter factors. $\blacksquare$

Reading the factors. For $s \gg 1/(\eta t)$ both factors are about 1: these directions are fitted. For $s \ll 1/(\eta t)$, GD gives about $\eta s t$ (because $(1 - \eta s)^t \approx 1 - \eta s t$) and ridge gives about $s/\lambda$. They match when

So stopping early suppresses the directions with small eigenvalues, which are the ones most affected by noise, just as ridge does. The stopping time is chosen on a validation set, never on the test set.

Interactive figure: loading…

Answers to section 1. (1) No. The minimum is $H(q)$ with $q = (0.91, 0.01, \dots, 0.01)$: $-(0.91\ln 0.91 + 9 \cdot 0.01 \ln 0.01) \approx 0.500$. (2) With Adam + L2, the weight with no gradient shrinks much faster: its $\sqrt{\hat v}$ is tiny, so the decay term is divided by a tiny number. With AdamW both shrink at the same rate, $\eta\lambda$ per step.

4. Worked example #

(a) Dropout. $x = (1, 2, 3, 4)$, $p = 0.5$, mask draw $m = (1, 0, 1, 1)$.

• Training output: $y = m \odot x/(1 - p) = 2\,m \odot x = (2, 0, 6, 8)$.
• Over many draws: $\mathbb{E}[y] = x$ and $\operatorname{Var}(y_i) = x_i^2 \cdot 0.5/0.5 = (1, 4, 9, 16)$.
• Backward with this mask: $\bar x = (2, 0, 2, 2) \odot \bar y$.
• Eval: $y = x$.

(b) Label smoothing. $K = 3$, $\varepsilon = 0.15$, true class 0, logits $z = (2, 0, 0)$.

• Target: $q = (0.85 + 0.05,\ 0.05,\ 0.05) = (0.9, 0.05, 0.05)$.
• Prediction: $p = (e^2, 1, 1)/(e^2 + 2) = (7.389, 1, 1)/9.389 = (0.7870, 0.1065, 0.1065)$.
• Loss: $-(0.9 \ln 0.7870 + 0.05 \ln 0.1065 + 0.05 \ln 0.1065) = 0.2156 + 0.2240 = 0.4395$.
• Gradient: $p - q = (-0.113, 0.0565, 0.0565)$. It still pushes the gap up, because the optimal gap is $\ln(0.9/0.05) = \ln 18 \approx 2.89 > 2$. Above 2.89 the gradient pushes back.
• Without smoothing, the gradient is $p - \text{onehot} = (-0.213, 0.1065, 0.1065)$, and its first entry is negative for every gap.

(c) Adam + L2 vs AdamW. Learning rate 0.01, $\lambda = 0.01$, 10 steps, two weights at 1. The data gradients alternate in sign, with size 0 (first weight) or 10 (second weight). Adam moves the second weight even without decay (by about 0.015 here), so we compare each run with the same run without decay. The shrinkage caused by the decay:

	no-gradient weight	large-gradient weight
Adam + L2	0.0997	0.00009
AdamW	0.0010	0.0010

AdamW shrinks both by about $10 \cdot \eta\lambda = 10^{-3}$. Adam + L2 shrinks the no-gradient weight 100 times more, and the large-gradient weight 11 times less.

(d) Early stopping. $\eta = 0.1$ and $t = 100$, so $\lambda \approx 1/(\eta t) = 0.1$.

eigenvalue $s$	GD: $1 - (1 - \eta s)^t$	ridge: $s/(s + \lambda)$
1	$1 - 0.9^{100} \approx 1.000$	$1/1.1 = 0.909$
0.1	$1 - 0.99^{100} = 0.634$	$0.1/0.2 = 0.5$
0.01	$1 - 0.999^{100} = 0.095$	$0.01/0.11 = 0.091$

Both keep the large-$s$ directions and shrink the small-$s$ ones; GD's filter is sharper.

5. In code #

PyTorch checks the worked examples (a)–(c):

import torch
import torch.nn.functional as F

torch.manual_seed(0)
x = torch.tensor([1., 2., 3., 4.], dtype=torch.float64).repeat(200_000, 1)   # 200k copies
y = F.dropout(x, p=0.5, training=True)
print(sorted(y[:, 3].unique().tolist()))           # [0.0, 8.0]: kept entries scaled by 1/(1-p)
print(round((y == 0).double().mean().item(), 3))   # 0.5: fraction dropped
print(y.mean(0).numpy().round(1))                  # [1. 2. 3. 4.]      E[y] = x
print(y.var(0).numpy().round(1))                   # [ 1.  4.  9. 16.]  Var = x^2 p/(1-p)
print(torch.equal(F.dropout(x, p=0.5, training=False), x))   # True: identity at eval

z = torch.tensor([[2.0, 0.0, 0.0]], dtype=torch.float64, requires_grad=True)
loss = F.cross_entropy(z, torch.tensor([0]), label_smoothing=0.15)
loss.backward()
print(round(loss.item(), 4), z.grad.numpy().round(4))   # 0.4395 [[-0.113   0.0565  0.0565]]

def run(opt_cls, weight_decay, steps=10):
    """Two weights start at 1. Their data gradients alternate in sign, with sizes 0 and 10."""
    theta = torch.ones(2, dtype=torch.float64, requires_grad=True)
    size = torch.tensor([0.0, 10.0], dtype=torch.float64)
    opt = opt_cls([theta], lr=0.01, weight_decay=weight_decay)
    for t in range(steps):
        theta.grad = size * (-1.0) ** t        # the data gradient of this step
        opt.step()
    return theta.detach()

for name, cls in [("Adam + L2", torch.optim.Adam), ("AdamW", torch.optim.AdamW)]:
    shrink = run(cls, 0.0) - run(cls, 0.01)    # the part of the movement caused by the decay
    print(f"{name:9s}  no-gradient weight {shrink[0]:.5f}   large-gradient weight {shrink[1]:.5f}")
# Adam + L2  no-gradient weight 0.09965   large-gradient weight 0.00009
# AdamW      no-gradient weight 0.00100   large-gradient weight 0.00099

Early stopping against ridge on a least-squares problem with a wide spread of eigenvalues:

import numpy as np

rng = np.random.default_rng(0)
n, d = 200, 20
X = rng.standard_normal((n, d)) * np.logspace(0, -1.5, d)   # feature scales from 1 down to 0.03
y = X @ rng.standard_normal(d) + 0.1 * rng.standard_normal(n)
S, b = X.T @ X / n, X.T @ y / n          # the loss (1/2n)|Xw - y|^2 has gradient S w - b

eta, t = 0.5, 400
w = np.zeros(d)
for _ in range(t):                       # gradient descent from 0, stopped after t steps
    w -= eta * (S @ w - b)
lam = 1 / (eta * t)                      # predicted equivalent ridge strength: 0.005
w_ridge = np.linalg.solve(S + lam * np.eye(d), b)
w_ls = np.linalg.solve(S, b)

s, V = np.linalg.eigh(S)                 # filter factor = component of w / component of w_ls
f_gd, f_ridge = (V.T @ w) / (V.T @ w_ls), (V.T @ w_ridge) / (V.T @ w_ls)
print(np.allclose(f_gd, 1 - (1 - eta * s) ** t), np.allclose(f_ridge, s / (s + lam)))  # True True
for i in (0, 5, 10, 19):
    print(f"s={s[i]:.4f}  GD {f_gd[i]:.3f}  ridge {f_ridge[i]:.3f}")
# s=0.0009  GD 0.158  ridge 0.147
# s=0.0070  GD 0.755  ridge 0.584
# s=0.0332  GD 0.999  ridge 0.869
# s=0.9810  GD 1.000  ridge 0.995

6. Where it shows up #

• LLM pretraining. AdamW with weight decay 0.1 (GPT-3, LLaMA), no decay on biases and norm gains, and usually no dropout: with one pass over a huge dataset, overfitting is not the main risk. Dropout returns when fine-tuning on small data.
• Vision. DeiT trains ViTs on ImageNet-1k only, with strong augmentation (RandAugment, Mixup, CutMix), label smoothing 0.1 and stochastic depth (randomly dropping whole residual branches).
• ML for science. Labelled data are often small, so early stopping on a held-out set is standard. Split the held-out set by time or region, not at random, or neighbouring samples leak. Augment only with true symmetries: rotations are fine for molecules, but not a north–south flip of a regional weather map. Learned simulators add noise to their inputs during training so that long autoregressive rollouts stay stable (Sanchez-Gonzalez et al., 2020).
• Uncertainty. Monte Carlo dropout keeps dropout on at test time and averages several predictions; their spread is a cheap uncertainty estimate (Gal and Ghahramani, 2016).

7. Common mistakes #

• Adam(weight_decay=λ) is L2, not AdamW. Use AdamW for decoupled decay.
• Tuning $\lambda$ without the learning rate. In PyTorch's AdamW the decay per step is lr × λ, so changing lr changes the decay, and the decay fades with a cosine schedule.
• Decaying biases and normalization gains. They are usually excluded; decaying $\gamma$ pulls the output scale of every normalized layer towards 0.
• Forgetting model.eval(). Dropout stays on and the predictions are noisy.
• Wrong meaning of $p$. PyTorch's $p$ is the drop probability; Srivastava et al. and TensorFlow 1's keep_prob use the keep probability.
• Dropout before BatchNorm. Dropout inflates $\mathbb{E}[y^2]$ by $1/(1-p)$ during training, so BN's running variance does not match the eval activations.
• Label-changing augmentation. Rotating a "6" by 180° gives a "9"; flipping a regional weather map breaks the physics.
• Early stopping on the test set. That leaks the test set into model selection; use a validation set.

8. Check yourself #

1. Dropout with $p = 0.25$ on a unit with value $x = 2$. Which values can $y$ take, with which probabilities? What are $\mathbb{E}[y]$ and $\operatorname{Var}(y)$?
   Answer

   $y = 2/0.75 = 8/3 \approx 2.667$ with probability 0.75, and $y = 0$ with probability 0.25. $\mathbb{E}[y] = 2$ and $\operatorname{Var}(y) = 4 \cdot 0.25/0.75 = 4/3 \approx 1.333$.
2. In PyTorch, what does a dropout layer do to its input in eval mode, and why is no rescaling needed there?
   Answer

   It returns the input unchanged. Inverted dropout already divides by $1 - p$ during training, so the expected training output equals the eval output.
3. $K = 10$, $\varepsilon = 0.1$: give the smoothed target, the minimal loss and the optimal logit gap.
   Answer

   True class $0.9 + 0.01 = 0.91$, others $0.01$. Minimal loss $H(q) = -(0.91\ln 0.91 + 9 \cdot 0.01\ln 0.01) \approx 0.500$. Gap $\ln(0.91/0.01) = \ln 91 \approx 4.51$.
4. PyTorch AdamW(lr=3e-4, weight_decay=0.1) with a constant learning rate: by which factor does the decay alone shrink a weight over 10,000 steps?
   Answer

   The factor per step is $1 - 3\cdot10^{-4} \cdot 0.1 = 1 - 3\cdot10^{-5}$, so $(1 - 3\cdot10^{-5})^{10^4} \approx e^{-0.3} \approx 0.741$.
5. Linear regression with dropout $p$ on the inputs, and standardized features ($\sum_i x_{ij}^2 = n$). Which ridge penalty $\lambda\|w\|^2$ does dropout add to the summed squared loss?
   Answer

   $\lambda = n\,p/(1-p)$. For $p = 0.5$, $\lambda = n$, which is ridge strength 1 on the mean squared loss.
6. Gradient descent with $\eta = 0.01$ is stopped after $t = 500$ steps. Which ridge strength does it resemble, and which eigen-directions of $S$ does it fit?
   Answer

   $\lambda \approx 1/(\eta t) = 0.2$. Directions with $s \gg 0.2$ are fitted almost fully; directions with $s \ll 0.2$ are shrunk to about $\eta s t = 5s$ of their value.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Write inverted dropout with drop probability $p$ (training and eval).

   Show answer

2. Why does inverted dropout divide by $1 - p$?

   Show answer

3. Dropout with $p = 0.5$ on a unit with value $x = 3$: what is $\operatorname{Var}(y)$?

   Show answer

4. Show that SGD with the L2 penalty $\frac\lambda2\|\theta\|^2$ is weight decay.

   Show answer

5. Why is L2 regularization not weight decay under Adam?

   Show answer

6. What is the AdamW update in PyTorch?

   Show answer

7. What is the label-smoothed target with $K$ classes and smoothing $\varepsilon$?

   Show answer

8. What is the minimal cross-entropy under label smoothing, and where is it reached?

   Show answer

9. Which ridge strength does early stopping after $t$ GD steps with step $\eta$ resemble?

   Show answer

10. What does data augmentation with a symmetry group $G$ of the labels encourage?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why scale by $1/(1-p)$ in dropout?" — So that $\mathbb{E}[y] = x$, and nothing changes at eval time.
• "Weight decay vs L2?" — Equal for SGD (factor $1 - \eta\lambda$ per step); different for Adam, because the L2 gradient is divided by $\sqrt{\hat v}$. AdamW decouples the decay.
• "What does label smoothing do to the logits?" — The optimum is $p = q$ with a finite gap $\log(1 + K(1-\varepsilon)/\varepsilon)$; the loss never reaches 0, and the model is less overconfident.
• "Is early stopping a regularizer?" — Yes: for least squares it acts like ridge with $\lambda \approx 1/(\eta t)$.
• "Why do LLMs often train without dropout?" — One epoch over a huge dataset; the regularization comes from data scale and weight decay.

10. Practice #

The notebook for this lesson runs in Colab. Each exercise has a check cell and a folded solution.

1. Inverted dropout (forward and backward with the same mask), with a check of the mean and the variance.
2. Adam + L2 and AdamW written by hand on two weights, reproducing the numbers of section 4 and checked against torch.optim.
3. Label-smoothed cross-entropy and its gradient, checked against F.cross_entropy(..., label_smoothing=).
4. Early-stopping and ridge filter factors on a random least-squares problem.
5. A small MLP on the scikit-learn digits: validation accuracy with and without dropout and weight decay.

11. Watch and read #

Dropout Regularization (C2W1L06)DeepLearning.AI · Andrew Ng · 9 min

Understanding Dropout (C2W1L07)DeepLearning.AI · Andrew Ng · 7 min

Lecture 7 | Training Neural Networks IIStanford CS231n (2017) · 1 h 15 min

• Simon Prince, Understanding Deep Learning, free PDF: chapter 9 (§9.1.2 L2 regularization, §9.3.1 early stopping, §9.3.3 dropout, §9.3.4 applying noise, §9.3.8 augmentation).
• d2l.ai, §5.6 Dropout.
• Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting (JMLR, 2014).
• Loshchilov and Hutter, Decoupled Weight Decay Regularization (2019), §2 (Propositions 1–3, Algorithm 2).
• Müller, Kornblith and Hinton, When Does Label Smoothing Help? (2019).
• Zhang et al., Understanding Deep Learning Requires Rethinking Generalization (2017): networks fit random labels.
• Li et al., Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift (2019).