Quick recap

01 MLPs and universal approximation

• An MLP alternates affine maps and activations: $Z_l = A_{l-1}W_l + \mathbf{1}b_l$, $A_l = \sigma(Z_l)$, with no activation on the last layer. It has $\sum_l (n_{l-1} + 1) n_l$ parameters.
• Without activations, any number of layers is one affine map $x \mapsto xW_1\cdots W_L + b$, with rank at most the smallest width.
• One ReLU unit $\operatorname{relu}(wx + b)$ is a hinge with its joint at $x = -b/w$. One hidden layer of $H$ units gives a piecewise-linear function with at most $H + 1$ pieces.
• Through any knots, the ReLU network $y_0 + \sum_k c_k \operatorname{relu}(x - x_k)$ with $c_0 = s_1$ and $c_k = s_{k+1} - s_k$ (changes of slope) interpolates exactly; the error for smooth $f$ is at most $h^2 \max|f''|/8$.
• Universal approximation (Leshno et al. 1993): with a continuous, non-polynomial activation, one hidden layer approximates any continuous function on a compact set to any accuracy in the sup norm.
• The theorem gives existence only: no width bound, no guarantee that training finds the weights, nothing about generalization or about inputs outside the compact set.
• Depth multiplies pieces: $K$ hidden layers of width $D$ give at most $(D + 1)^K$ pieces in 1-D, and the tent map composed $k$ times has $2^k$ pieces with only $2k$ units.

02 Training a network

• Training minimizes the empirical risk $\hat R(\theta) = \frac1n \sum_i \ell(f_\theta(x_i), y_i)$ on the training set; the validation set chooses hyperparameters, and the test set is used once for a final estimate.
• Losses are negative log-likelihoods: Gaussian noise gives the squared error, Bernoulli gives binary cross-entropy, and a softmax over $K$ classes gives cross-entropy $-\log p_{c^*}$.
• The gradient of softmax cross-entropy with respect to the logits is $p - y$ (and $(P - Y)/B$ for a batch mean); for a sigmoid with BCE it is $s - y$.
• Sigmoid + squared error has gradient $(s - y)s(1 - s)$, which almost vanishes when the model is confidently wrong; cross-entropy does not have this problem.
• Compute losses from logits with log-sum-exp: $\log\sum_j e^{z_j} = m + \log\sum_j e^{z_j - m}$. PyTorch's CrossEntropyLoss and BCEWithLogitsLoss take logits, not probabilities.
• A mini-batch gradient is unbiased, and with independent draws its covariance is $\Sigma/B$: the noise falls like $1/\sqrt B$ while the cost grows like $B$.
• Gradient descent on $\frac{\lambda}{2}w^2$ converges if and only if $0 < \eta < 2/\lambda$; in many dimensions the largest curvature sets the limit $2/\lambda_{\max}$.
• The training loop is forward, loss, zero_grad, backward, step; check the initial loss ($\log K$), overfit one batch first, and read the training and validation curves together.

03 Backpropagation

• Backpropagation computes adjoints $\bar u = \partial\mathcal{L}/\partial u$ from the loss backward: the adjoint of an input is the upstream gradient times the local gradient.
• On a computational graph, each node computes a vector-Jacobian product, and a variable that fans out gets the sum of the adjoints from all its uses.
• Linear layer $Z = AW + \mathbf{1}b$ with $G = \partial\mathcal{L}/\partial Z$: $\bar W = A^\top G$, $\bar A = GW^\top$, $\bar b = \mathbf{1}^\top G$. Every gradient has the shape of its variable.
• MLP with softmax cross-entropy (batch mean): $G_L = (P - Y)/B$ and $G_{l-1} = (G_lW_l^\top) \odot \sigma'(Z_{l-1})$; the factor $1/B$ enters only once.
• Reverse mode gives the full gradient of a scalar loss for about 2× the forward cost in the backward pass (about $6N$ FLOPs per token per training step), but it must store the activations.
• Activation checkpointing stores only segment inputs and recomputes the rest: about $2\sqrt{L}$ stored activations for at most one extra forward pass.
• Gradient check: central differences in float64 with $h \approx 10^{-5}$ to $10^{-6}$, relative error, no kinks. The error is truncation $O(h^2)$ plus round-off $O(u/h)$.

04 Activations and initialization

• At initialization, $\operatorname{Var}(z_{l+1}) = n_l\, v_{l+1}\, \mathbb{E}[\sigma(z_l)^2]$: each layer multiplies the signal size by a factor, so a factor $c \ne 1$ gives $c^{L}$, exponential in depth.
• He initialization $v = 2/\mathrm{fan}_{\mathrm{in}}$ keeps the variance of a ReLU network constant, because $\mathbb{E}[\operatorname{relu}(z)^2] = \tfrac12\operatorname{Var}(z)$ for symmetric $z$; LeCun $v = 1/\mathrm{fan}_{\mathrm{in}}$ is for linear or tanh units.
• The backward pass has the same recursion with $n_{l+1}$ and $\mathbb{E}[\sigma'(z)^2]$; Xavier $v = 2/(\mathrm{fan}_{\mathrm{in}} + \mathrm{fan}_{\mathrm{out}})$ is the compromise between the two conditions.
• Xavier in a deep ReLU network halves the variance in every layer: after 50 layers it is about $10^{-15}$ of the start.
• Sigmoid and tanh saturate ($s' \le 1/4$); ReLU passes gradient 1 for positive inputs but can die; GELU $x\Phi(x)$ and SiLU $x\,s(x)$ are smooth and pass some gradient for negative inputs.
• Equal (or zero) initial weights give equal units for ever (the symmetry problem); random weights break the symmetry.
• SwiGLU has three weight matrices; hidden width $d_{ff} = 8d/3$ keeps the parameter count and FLOPs of a GELU FFN of width $4d$.
• μP also scales the learning rates with width, so that hyperparameters tuned on a small model transfer to a large one.

05 Normalization and residuals

• A normalization layer computes $\hat x = (x - \mu)/\sqrt{\sigma^2 + \varepsilon}$ over a group of numbers (biased variance), then $y = \gamma\hat x + \beta$ with learned $\gamma, \beta$.
• BatchNorm groups one channel over the batch (and positions); LayerNorm groups the features of one token; RMSNorm is LayerNorm without the mean and without $\beta$.
• BatchNorm uses batch statistics in training and running statistics in eval mode (momentum 0.1, unbiased running variance in PyTorch); LayerNorm and RMSNorm are the same in both modes.
• The BatchNorm/LayerNorm backward pass is $\bar x = \frac1{\tilde\sigma}\big(g - \operatorname{mean}(g)\mathbf 1 - \hat x\operatorname{mean}(g\odot\hat x)\big)$, $g = \gamma\bar y$: it removes the components along $\mathbf 1$ and $\hat x$, so $\sum_i \bar x_i = 0$.
• Transformers use LayerNorm or RMSNorm because BatchNorm depends on the batch: small batches, batch-1 decoding, padding and causal leakage.
• A residual stack has $\partial x_L/\partial x_l = \prod_k (I + J_k) = I + \cdots$: the identity term carries the gradient to every layer.
• Without normalization, residual blocks with variance-preserving branches double the forward variance per block; fixes are normalization in the branch, zero-initialized branch ends, or $1/\sqrt{L}$ scaling.
• Pre-LN keeps a pure identity path and needs a final norm and little or no warmup; post-LN has larger gradients near the output at initialization and needs warmup.

06 Regularization

• A regularizer is a change to training that aims to lower the test error, not the training error.
• Inverted dropout: $y = m \odot x/(1-p)$ in training with $m_i \sim \text{Bernoulli}(1-p)$, $y = x$ in eval; $\mathbb{E}[y] = x$ and $\operatorname{Var}(y_i) = x_i^2\,p/(1-p)$.
• Dropout on the inputs of linear regression equals a ridge penalty $\frac{p}{1-p}\sum_j w_j^2\sum_i x_{ij}^2$.
• For SGD, L2 regularization $\frac\lambda2\|\theta\|^2$ equals weight decay with factor $1 - \eta\lambda$ per step.
• For Adam, L2 is not weight decay: the decay is divided by $\sqrt{\hat v}$, so weights with large gradients are decayed less. AdamW decouples the decay, $\theta \leftarrow \theta(1 - \text{lr}\cdot\lambda)$, then the Adam step.
• Label smoothing uses the target $q = (1-\varepsilon)\,\text{onehot} + \varepsilon/K$; the gradient is $p - q$, the minimal loss is $H(q) > 0$, and the optimal logit gap $\log(1 + K(1-\varepsilon)/\varepsilon)$ is finite.
• Data augmentation with a symmetry group of the labels makes the training objective prefer invariant predictors; it must not change the label or the physics.
• Early stopping after $t$ steps of GD on least squares has filter factors $1 - (1-\eta s)^t$, close to ridge's $s/(s+\lambda)$ with $\lambda \approx 1/(\eta t)$.

07 Convolutional networks

• A convolution layer slides one small kernel over the input: $Y[o,i,j] = b[o] + \sum_{c,u,v} W[o,c,u,v]\,X[c, is+u, js+v]$. Deep-learning "convolution" is a cross-correlation (no kernel flip).
• Locality and weight sharing: in 1-D the layer is a banded Toeplitz matrix with $k$ parameters instead of $n^2$.
• Output size $o = \lfloor (n + 2p - d(k-1) - 1)/s \rfloor + 1$; parameters $C_{\text{out}}(C_{\text{in}}k_hk_w + 1)$, independent of the image size; MACs $= H_{\text{out}}W_{\text{out}}C_{\text{out}}C_{\text{in}}k_hk_w$.
• Receptive field: $r_l = r_{l-1} + d_l(k_l - 1)j_{l-1}$ with jump $j_l = j_{l-1}s_l$; pooling layers count like convolutions.
• im2col turns a convolution into one matrix product $Y_{\text{mat}} = W_{\text{mat}}\text{cols} + b\mathbf{1}^\top$; cols is about $k_hk_w/s^2$ times larger than the input.
• Backward: $\bar W_{\text{mat}} = \bar Y_{\text{mat}}\text{cols}^\top$, $\bar b = \bar Y_{\text{mat}}\mathbf{1}$, $\bar X = \text{col2im}(W_{\text{mat}}^\top\bar Y_{\text{mat}})$: a transposed convolution, a full correlation with the $180°$-rotated kernel.
• Stride-1 convolution is translation equivariant (exactly on a circular domain); zero padding breaks it at the border, and stride $s$ keeps it only for shifts by multiples of $s$.
• A $1\times1$ convolution is a linear layer over channels at every pixel; the ResNet bottleneck ($1\times1$ down, $3\times3$, $1\times1$ up) has about $17\times$ fewer weights than a basic block at 256 channels.

08 RNNs and LSTMs

• An embedding is a lookup: $x = \operatorname{onehot}(s)\,E$ is row $s$ of $E \in \mathbb{R}^{|\mathcal V|\times D}$, with $|\mathcal V|\,D$ parameters.
• A language model is trained on next-token prediction: $\ell_t = -\log p_t[s_{t+1}]$, with the true previous tokens as inputs (teacher forcing).
• An RNN updates $h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b_h)$ with the same weights at every step; BPTT is backpropagation on the unrolled network, and gradients of shared weights are sums over time.
• The gradient from step $t$ to step $k$ is a product of $t - k$ Jacobians $J_i = \operatorname{diag}(\sigma'(a_i))\,W_{hh}^\top$, so it vanishes or explodes exponentially.
• If $\gamma\|W_{hh}\|_2 < 1$ the long-term gradients vanish (Pascanu et al.); for a linear RNN the spectral radius $\rho(W_{hh})$ decides the long run.
• Global-norm clipping $g \leftarrow g\min(1, \tau/\|g\|)$ keeps the direction and stops exploding steps; it does nothing for vanishing gradients.
• The LSTM cell $c_t = f_t\odot c_{t-1} + i_t\odot g_t$ has an additive path with Jacobian $\operatorname{diag}(f_t)$; with $f$ near 1, memory and gradients last about $1/(1-f)$ steps.
• Attention and state-space models replaced RNNs for long sequences: parallel training, short gradient paths, and (for SSMs) a constant-size state with controlled eigenvalues.

09 Attention

• Attention for one query: scores $s_j = q^\top k_j/\sqrt{d_k}$, weights $a = \operatorname{softmax}(s)$ (positive, sum 1), output $\sum_j a_jv_j$: a soft dictionary lookup.
• Self-attention: $Q = XW_Q$, $K = XW_K$, $V = XW_V$ and $\operatorname{softmax}(QK^\top/\sqrt{d_k} + M)\,V$; row $i$ of the $T \times T$ matrix $A$ holds the weights of token $i$.
• For independent unit-variance entries, $\operatorname{Var}(q^\top k) = d_k$; dividing by $\sqrt{d_k}$ keeps the logits of order 1 at initialization.
• The softmax Jacobian is $\operatorname{diag}(p) - pp^\top$; when $p$ is nearly one-hot, all its entries are small and the gradient to the logits vanishes.
• Masks add $-\infty$ to the logits before the softmax: causal ($j > i$) for decoders, padding for padded keys; a fully masked row gives NaN.
• Multi-head attention splits $D$ into $H$ heads of size $D/H$ with a view and a transpose; it has $4D^2 + 4D$ parameters for every $H$ and costs $8TD^2 + 4T^2D$ FLOPs.
• Cross-attention takes queries from one sequence and keys and values from another; it never uses a causal mask.
• Without a mask, self-attention is permutation-equivariant, $\operatorname{Attn}(PX) = P\operatorname{Attn}(X)$: it sees a set, so position information must be added.

10 Positional encodings

• Self-attention without positions is permutation-equivariant, so every sequence model needs a positional encoding.
• Learned absolute embeddings add a trained row $E_{\text{pos}}[t]$ to token $t$; they have $T_{\max}D$ parameters and no meaning beyond $T_{\max}$.
• Sinusoidal encodings: $PE(t)_{2i} = \sin\theta_it$, $PE(t)_{2i+1} = \cos\theta_it$, $\theta_i = b^{-2i/d}$; a shift by $k$ is a fixed block rotation, $PE(t + k) = M_kPE(t)$.
• Adding position to the input lets the score depend on absolute positions; it does not force relative attention.
• RoPE rotates pair $i$ of each query and key by $m\theta_i$; since $R_m^\top R_n = R_{n-m}$, the score $\langle R_mq, R_nk\rangle = q^\top R_{n-m}k$ depends on the offset only, with no parameters.
• Slow RoPE pairs never complete a turn during training, so new angles appear beyond $L$; position interpolation ($m \to mL/L'$) and base scaling (larger $b$), plus a short fine-tune, extend the context.
• ALiBi adds $-m_h(i - j)$ to the logits with fixed slopes $m_h = 2^{-8h/H}$; head $h$ has a half-life of $\ln 2/m_h$ positions, and it extrapolates well.
• The softmax ignores a constant added to a whole row, so the ALiBi biases $-m(i - j)$ and $+m\,j$ give the same weights.

11 Transformers and ViT

• A pre-LN transformer block is $X' = X + \operatorname{MHA}(\operatorname{LN}_1(X))$, $Y = X' + \operatorname{MLP}(\operatorname{LN}_2(X'))$; attention is the only part that mixes tokens, and input and output have the same shape $(B, T, D)$.
• A decoder-only language model is: token embedding plus positions, $L$ causal blocks, a final LayerNorm, and an unembedding (often tied, $W_U = E^\top$) followed by a softmax over the vocabulary.
• A block with $d_{ff} = 4D$ has $12D^2 + 13D \approx 12D^2$ parameters ($4D^2$ attention, $8D^2$ MLP), for any number of heads.
• Forward FLOPs per token are about $2N + 4LTD$ and training FLOPs about $6N + 12LTD$, where $N$ is the number of weights multiplied per token; the attention term dominates only when $T > 6D$.
• Encoder-only models (BERT, ViT) are bidirectional; decoder-only models (GPT) use a causal mask and predict the next token; encoder–decoder models (T5, Whisper) add cross-attention in the decoder.
• With a causal mask, past keys and values never change, so a KV cache gives exact results; it holds $2LTD$ numbers per sequence ($2LTH_{kv}d_k$ with grouped-query attention).
• Temperature sampling draws from $\operatorname{softmax}(z/\tau)$: $\tau \to 0$ is greedy and $\tau \to \infty$ is uniform.
• A ViT cuts an image into $T = H_{\text{in}}W_{\text{in}}/P^2$ patches, embeds them with a convolution of kernel = stride = $P$, adds a [CLS] token and learned position embeddings; for a new resolution it keeps $P$ and interpolates the position grid.

12 Training dynamics

• Gradient descent on a quadratic of curvature $\lambda$ is stable only for $\eta\lambda < 2$. The sharpness of a network is often largest early, so warmup keeps the first steps below the limit.
• A schedule is a linear warmup $\eta_{\max} t/W$ followed by a decay: cosine, linear, or warmup–stable–decay (flat at $\eta_{\max}$, then a short decay at the end).
• Global-norm clipping multiplies all gradients by $\min(1, \tau/\|g\|)$: the direction is kept and the norm becomes $\min(\|g\|, \tau)$. Log the norm before clipping.
• Attention logits grow like the square of the weight scale; qk-layernorm bounds every logit by $\sqrt{d_k}$ (Cauchy–Schwarz).
• Cross-entropy is unchanged by $s \mapsto s + a\mathbf{1}$, so nothing pulls $\log Z$ back; the z-loss $c_z(\log Z)^2$ does.
• An out-of-date Adam second moment gives huge steps after a quiet period; a lower $\beta_2$ (0.95) fixes it.
• Linear scaling rule: batch $\times k$ means learning rate $\times k$ for SGD, with a gradual warmup; the gain stops beyond the critical batch size.
• In a format with $p$ fraction bits, an update $u$ to $w \approx 2^e$ is lost when $u < 2^{e-p-1}$; mixed precision keeps fp32 master weights, accumulates in fp32, and in fp16 uses loss scaling.

13 Graph neural networks

• One message-passing layer: messages $m_{j\to i} = \psi(h_i, h_j, e_{ji})$, a symmetric aggregation over the neighbours (sum, mean, max or attention), and an update $h_i' = \phi(h_i, a_i)$. $K$ layers see the $K$-hop neighbourhood.
• The GCN layer is $H' = \sigma(\hat AHW)$ with $\hat A = \tilde D^{-1/2}(A + I)\tilde D^{-1/2}$. Self-loops keep the node's own state; the normalization keeps the eigenvalues of $\hat A$ in $(-1, 1]$; it acts as a low-pass filter on the graph.
• A message-passing layer is permutation-equivariant: $F(PH, PAP^\top) = PF(H, A)$. A sum, mean or max readout after it is permutation-invariant.
• No message-passing GNN separates two graphs that 1-WL colour refinement cannot separate (for example two triangles and a 6-ring). GIN, with sum aggregation and an MLP, reaches the 1-WL limit.
• On one-hot neighbour features, sum keeps the whole multiset, mean keeps only the proportions, and max keeps only the set of types present.
• Over-smoothing: $\hat A^Kx$ converges to $u_1u_1^\top x$ with $u_1 \propto \tilde D^{1/2}\mathbf 1$, at rate $\rho^K$. Deep plain GCNs lose everything except degree information.
• Over-squashing: an exponentially growing neighbourhood must pass through a few bottleneck edges into a fixed-size vector; the Jacobian $\partial h_i/\partial x_s$ decays with distance.

14 Self-supervised learning

• Self-supervised learning builds the training signal from unlabelled data. Without a second force, "make the two views close" collapses to a constant encoder.
• The InfoNCE loss is the mean cross-entropy of the rows of $S/\tau$, with labels $0, \dots, N-1$; CLIP averages the row (image→text) and column (text→image) losses. In-batch items are the negatives.
• The gradient of a row loss is $\partial\ell_i/\partial s_{ij} = (p_{ij} - \delta_{ij})/\tau$. A small $\tau$ puts the push on the hardest negatives; as $\tau \to 0$ the loss becomes a hinge on the hardest negative.
• For any critic, $I(X;Y) \ge \log N - \mathcal{L}_N$, and the bound can never exceed $\log N$; the optimal critic is the pointwise mutual information $\log\frac{p(y\mid x)}{p(y)}$.
• SimCLR and CLIP avoid collapse with negatives; BYOL and SimSiam with a predictor and a stop-gradient; DINO with an EMA teacher plus centering and sharpening; MAE with a reconstruction target at a 75% mask ratio.
• Use the encoder output $h$, not the projection-head output $z$. Evaluate with a linear probe or k-NN on frozen features, fine-tune when labels are plentiful, and use CLIP zero-shot when classes are given only as text.

15 Mixture of experts

• A top-$k$ MoE layer computes $y_t = \sum_{i \in S_t} g_{t,i}E_i(x_t)$: the router softmax picks the $k$ largest probabilities $S_t$, and the gates are the raw probabilities (Switch) or renormalized over $S_t$ (Mixtral), which equals a softmax over the top-$k$ logits.
• Parameters grow with $E$, compute per token with $k$: total = shared + all experts, active = shared + $k$ experts per MoE layer (Mixtral 8×7B: 46.7B total, 12.9B active).
• The gate value multiplies the expert output so that the main loss sends a gradient to the router; a renormalized top-1 gate is always 1 and sends none.
• Without balancing, the router collapses onto a few experts. The Switch loss $\alpha E\sum_i f_iP_i$ is $\alpha$ for uniform routing and $\alpha E$ for full collapse.
• With the counts $f$ held fixed, $\partial\mathcal{L}_{\text{aux}}/\partial z_{t,j} = \frac{\alpha E}{T}p_{t,j}(f_j - \bar f_t)$: probability moves from busy experts to idle ones.
• Expert capacity is $C = \lceil kT\,\mathrm{cf}/E\rceil$; assignments beyond it are dropped and the token continues through the residual connection.
• The router z-loss $\frac1T\sum_t\operatorname{lse}_t^2$ fixes the free shift of the router logits and keeps them where bf16 is precise.
• Expert parallelism needs two all-to-alls per MoE layer in the forward pass and two in the backward pass; DeepSeek-V3 balances with a per-expert selection bias instead of a large auxiliary loss.

16 State-space models

• An SSM is $x' = Ax + Bu$, $y = Cx$, sampled with step $\Delta$: $x_k = \bar Ax_{k-1} + \bar Bu_k$, $y_k = Cx_k$ (state $x$, input $u$, as in S4).
• Zero-order hold: $\bar A = e^{\Delta A}$, $\bar B = A^{-1}(e^{\Delta A} - I)B$; bilinear: $\bar A = (I - \frac\Delta2A)^{-1}(I + \frac\Delta2A)$, $\bar B = (I - \frac\Delta2A)^{-1}\Delta B$.
• With fixed parameters (LTI) the recurrence is a causal convolution, $y_k = \sum_j K_ju_{k-j}$ with $K_j = C\bar A^j\bar B$; an FFT with zero padding to $\ge 2L-1$ computes it in $O(L\log L)$.
• If $\operatorname{Re}\lambda(A) < 0$, then $|e^{\Delta\lambda}| < 1$ for every $\Delta > 0$, so the recurrence cannot explode.
• S4 initializes $A$ with HiPPO and needs structure (normal plus low rank, or diagonal as in S4D) because HiPPO's eigenvectors are exponentially ill-conditioned.
• Mamba makes $\Delta_t$, $B_t$, $C_t$ functions of the input; a small $\Delta_t$ keeps the state, a large $\Delta_t$ resets it; with $N = 1$, $A = -1$ this is the gate $x_t = (1-g_t)x_{t-1} + g_tu_t$.
• A time-varying linear recurrence has no single kernel, but the operator $(a,b)\bullet(a',b') = (a'a, a'b+b')$ is associative, so a parallel scan computes it in $O(\log L)$ depth.
• Attention trains in $O(L^2d)$ and stores a KV cache of $2Ld$; an SSM trains in about $O(L)$ and stores a state of $dN$, at the price of weaker exact recall.

17 Generalization

• When $p > n$, infinitely many weight vectors fit the training data exactly; the training algorithm, not the loss, chooses one. This is implicit regularization.
• Gradient descent from $0$ on $\tfrac12\|Xw - y\|^2$ (step $\eta < 2/s_1^2$) converges to the minimum-norm interpolant $X^+y = X^\top(XX^\top)^{-1}y$, because every update lies in the row space $C(X^\top)$.
• From a start $w_0$, gradient descent converges to $X^+y + (I - X^+X)w_0$, the interpolant closest to $w_0$. Adam, another parametrization or another loss can choose a different solution.
• The minimum-norm fit has risk $\|(I - X^+X)\beta\|^2 + \sigma^2\sum_i 1/s_i^2$; at the interpolation threshold $p \approx n$ the smallest singular value is near 0, so the noise is amplified and the test error peaks.
• Test error against model size can go down, up and down again (model-wise double descent); it also happens over training time (epoch-wise) and over sample size (more data can hurt near the threshold). Label noise makes the peak stronger.
• Ridge replaces $1/s_i$ by $s_i/(s_i^2 + \lambda) \le 1/(2\sqrt\lambda)$ and removes the peak; early stopping after $t$ steps acts like ridge with $\lambda \approx 1/(\eta t)$.
• Networks fit random labels to 100% training accuracy, so any bound that depends only on the model class is vacuous for them; an explanation must use the data and the algorithm.