Self-supervised learning: contrastive, non-contrastive and masked modelling

Before this lesson: lesson 09 (attention, the encoders) and lesson 07 (image encoders). The vision transformer and its patch tokens (used by MAE) are in lesson 11. From the information-theory track: entropy, KL divergence and cross-entropy, and mutual information $I(X;Y)$ with its chain rule. Section 3 recalls what the proof needs. Time: about 75 minutes, plus the notebook.

---

Why this matters #

Nearly every foundation model is pretrained without labels. CLIP learns from image–caption pairs and gives the image encoders of many multimodal language models and the text encoders of text-to-image diffusion models. DINO and MAE give vision backbones that work for segmentation and depth with few labels. BERT's masked-token objective is masked modelling for text. In science, SSL is the default for large unlabelled archives: satellite images, weather reanalysis fields and continuous seismic records. Interviews ask two things here: "write the CLIP loss" and "why does this method not collapse?"

1. Predict first #

2. The picture #

Two views, one image, and the trap #

Take an image $x$. Make two random views $v$ and $v'$ of it, for example a random crop and a colour change. An encoder maps each view to a vector, its embedding. We want the two embeddings to be close, because they show the same content.

The trap: collapse. "Make the two vectors close" has a trivial solution: map every input to the same vector. Then all pairs are perfectly close, and the encoder has learned nothing. This is collapse. Every SSL method needs a second force that prevents it. There are three families:

1. Contrastive (SimCLR, MoCo, CLIP): also push apart the embeddings of different images. Training becomes a classification: "which of these $N$ candidates is my partner?"
2. Non-contrastive (BYOL, SimSiam, DINO): no negatives. An asymmetry between two networks (a predictor head, a stop-gradient, a slowly moving teacher) prevents collapse.
3. Masked modelling (MAE, BERT): hide part of the input and reconstruct it. The target is the data itself, so a constant output cannot succeed.

The similarity matrix #

Simplest case: $N = 2$. Two images and their two text captions. The model computes a $2 \times 2$ table of similarities, one row per image and one column per caption. Training raises the diagonal (the matching pairs) and lowers the off-diagonal (the wrong pairs). With $N$ candidates, a random guess is right with probability $1/N$. This is where the $\log N$ in the theory comes from.

Interactive figure: loading…

Two forces. Wang and Isola (2020) split a contrastive loss into two parts. Alignment pulls the two views of one image together. Uniformity spreads all embeddings over the sphere. The negatives give the uniformity. Without them, nothing stops collapse.

Interactive figure: loading…

3. Definitions and main results #

Notation. A batch has $N$ pairs $(x_i, y_i)$. For SimCLR, $x_i$ and $y_i$ are two views of image $i$; for CLIP, an image and its caption. Encoders give embeddings $u_i, w_i \in \mathbb{R}^d$, normalized to length 1. Then $s_{ij} = u_i^\top w_j$ is the cosine similarity, $S = (s_{ij})$ is the $N \times N$ similarity matrix, and $\tau > 0$ is the temperature.

The InfoNCE loss is a cross-entropy #

Row $i$ of $S/\tau$ is a vector of logits over the $N$ candidates $w_1, \dots, w_N$, and the correct class is $i$. The InfoNCE loss is the mean cross-entropy of the rows:

In code this is cross_entropy(S / tau, labels=arange(N)). The other $N - 1$ items of the batch are the negatives. They are called in-batch negatives: no extra negative sampling is needed.

CLIP (Radford et al. 2021) uses both directions: image→text (a softmax over each row) and text→image (a softmax over each column, that is, the rows of $S^\top$):

CLIP learns $\tau$ as a scalar parameter (stored as a logarithm), starts at $\tau = 0.07$, and clips it so that the logit scale $1/\tau$ never exceeds 100. It trained on 400 million pairs with batch size 32,768.

SimCLR's NT-Xent loss (Chen et al. 2020) puts all $2N$ views in one pool. For view $k$, the positive is the other view of the same image, and the negatives are the other $2(N - 1)$ views. The view itself is excluded, so each softmax has $2N - 1$ terms.

The gradient and the temperature #

Write $p_{ij} = \operatorname{softmax}_j(s_{ij}/\tau)$ and $\ell_i$ for the loss of row $i$. The gradient of a cross-entropy with respect to its logits is "probabilities minus one-hot". With the chain rule through the division by $\tau$:

where $\delta_{ij} = 1$ if $j = i$ and 0 otherwise.

• The positive is pulled up with weight $(1 - p_{ii})/\tau$.
• Negative $j$ is pushed down with weight $p_{ij}/\tau$, which is proportional to $e^{s_{ij}/\tau}$.
• So the ratio of the weights of two negatives is $e^{(s_{ij} - s_{ik})/\tau}$. A small $\tau$ puts almost all the push on the hardest negatives: the wrong items that are most similar.
• As $\tau \to 0$, $\tau\ell_i \to \max\big(0, \max_{j\ne i} s_{ij} - s_{ii}\big)$: a hinge on the hardest negative. A large $\tau$ weights all negatives nearly equally.

Interactive figure: loading…

InfoNCE lower-bounds mutual information #

The mutual information $I(X;Y)$ measures how much knowing $Y$ tells you about $X$, in nats. It is 0 when $X$ and $Y$ are independent. Contrastive learning is often described as "maximizing the mutual information between views". The precise statement is a bound.

Theorem (InfoNCE bound; Oord et al. 2018, Poole et al. 2019). Let $(x, y) \sim p(x, y)$. Draw an index $I$ uniformly from $\{1, \dots, N\}$. Let $y_I$ be the partner of $x$ (a sample of $p(y \mid x)$), and let the other $y_j$, $j \ne I$, be independent samples of $p(y)$. For any function $f(x, y)$ (the critic), define

Then $I(X; Y) \ge \log N - \mathcal{L}_N$. The best critic is $f^*(x, y) = \log\frac{p(y\mid x)}{p(y)} + c(x)$ for any function $c$, and the bound can never exceed $\log N$.

A contrastive batch is this setting. Row $i$ uses $x = x_i$, the candidates are $w_1, \dots, w_N$, the negatives come from other pairs (so they are independent of $x_i$), and the critic is $f(x, y) = s(x, y)/\tau$. The temperature is part of the critic.

Why this is true: the loss is the log-loss of a classifier that guesses which of the $N$ candidates is the partner. No classifier can beat the true posterior. The true posterior can only beat a random guess ($\log N$) by using information that $y$ carries about $x$, and that information is at most $I(X; Y)$.

Proof (seven steps)

Write $D = (x, y_1, \dots, y_N)$ for everything the critic sees. Two facts from information theory: the cross-entropy of $q$ under $p$ is $H(p) + \mathrm{KL}(p\,\|\,q) \ge H(p)$ (Gibbs' inequality), and the chain rule $I(A; B, C) = I(A; B) + I(A; C \mid B)$.

1. The loss is a classifier's cross-entropy. The critic defines a guess for the index of the positive, $q(i \mid D) = e^{f(x, y_i)}/\sum_j e^{f(x, y_j)}$. So $\mathcal{L}_N = \mathbb{E}[-\log q(I \mid D)]$.
2. Cross-entropy is at least entropy. For every $D$, $\mathbb{E}[-\log q(I\mid D) \mid D] = H(p(\cdot\mid D)) + \mathrm{KL}\big(p(\cdot\mid D)\,\|\,q(\cdot\mid D)\big) \ge H(p(\cdot\mid D))$. Average over $D$: $\mathcal{L}_N \ge H(I \mid D)$.
3. $I$ is uniform, so $H(I) = \log N$. Hence $\log N - \mathcal{L}_N \le H(I) - H(I \mid D) = I(I; D)$.
4. The joint density. $p(i, x, y_{1:N}) = \frac1N\, p(x, y_i) \prod_{j\ne i} p(y_j) = \frac1N\, p(x \mid y_i) \prod_{j} p(y_j)$. Integrate out $x$: $p(i, y_{1:N}) = \frac1N \prod_j p(y_j)$. So $I$ is independent of $Y_{1:N}$, that is, $I(I; Y_{1:N}) = 0$. Divide the two lines: $p(x \mid i, y_{1:N}) = p(x \mid y_i)$.
5. Chain rule. $I(I; D) = I(I; Y_{1:N}) + I(I; X \mid Y_{1:N}) = I(I; X \mid Y_{1:N})$. Also $I(I, Y_{1:N}; X) = I(Y_{1:N}; X) + I(I; X \mid Y_{1:N})$, and mutual information is non-negative, so $I(I; X \mid Y_{1:N}) \le I(I, Y_{1:N}; X)$.
6. Only the partner carries information about $x$. By step 4, $X$ depends on $(I, Y_{1:N})$ only through $Y_I$, and $(x, y_I) \sim p(x, y)$. So $I(I, Y_{1:N}; X) = I(X; Y)$.
7. Combine steps 3, 5 and 6: $\log N - \mathcal{L}_N \le I(I; D) \le I(X; Y)$. $\blacksquare$

The optimal critic. Step 2 is tight when $q = p(\cdot \mid D)$. By step 4, $p(i \mid D) \propto p(x \mid y_i) = p(x)\,p(y_i\mid x)/p(y_i)$, so $p(i \mid D) \propto p(y_i\mid x)/p(y_i)$. A softmax over $j$ ignores any term that depends on $x$ only. So $f^*(x, y) = \log\frac{p(y\mid x)}{p(y)} + c(x)$: the pointwise mutual information, up to a term in $x$.

The $\log N$ ceiling. A cross-entropy is never negative, so $\log N - \mathcal{L}_N \le \log N$. When $I(X; Y) > \log N$, the estimate is biased low, whatever the critic. To certify $I$ nats you need $N \ge e^{I}$ candidates. This is one reason contrastive methods like large batches.

Bounded logits. With cosine similarity, every logit lies in $[-1/\tau, 1/\tau]$. Even with perfect embeddings ($s_{ii} = 1$, all $s_{ij} = -1$), the row loss is at least $\log\big(1 + (N - 1)e^{-2/\tau}\big)$. With a fixed $\tau = 1$ and $N = 4096$ this is 6.32, so the bound can show at most $\log 4096 - 6.32 \approx 2.0$ nats. This is one reason for small or learned temperatures.

Answers to section 1. (1) At most $\log 1024 \approx 6.93$ nats, far below 20. The estimate saturates at $\log N$. (2) Collapse is a fixed point of BYOL's loss, but the predictor, the stop-gradient and the slowly moving target make the training dynamics avoid it in practice. SimSiam shows that the stop-gradient is essential: remove it and the model collapses. There is no complete theory of why this works; the explanation is an empirical finding with partial analyses for linear models.

How each method avoids collapse #

SimCLR (contrastive). NT-Xent with in-batch negatives. If all embeddings are equal, every logit is the same and the loss is $\log(2N - 1)$, its value for a random guess. The denominator makes collapse expensive. Other findings of the SimCLR paper:

• Augmentations define the task. The model learns what stays the same across views. Random crop plus colour distortion is the key pair. With crops alone, colour histograms identify the image, and the model learns little else.
• Projection head. The loss is applied to $z = g(h)$, a 2-layer MLP on top of the encoder output $h$. Downstream you use $h$: it is more than 10% better in linear evaluation than $z$, because $z$ is trained to be invariant to the augmentations and so discards information.
• Batch size and $\tau$. Batches from 256 to 8192 (16,382 negatives per positive pair), with the LARS optimizer. With $\ell_2$-normalized embeddings and batch 4096 (100 epochs), $\tau = 0.1$ gave 64.4% top-1, against 59.7% for $\tau = 0.05$ and 60.7% for $\tau = 0.5$ (their Table 5). Global BatchNorm across devices is needed: with statistics per device, the model can recognize the positive pair by a statistics shortcut.

MoCo (contrastive with a queue). The negatives come from a queue of $K = 65{,}536$ keys from earlier batches. The keys are encoded by a momentum encoder: an exponential moving average (EMA) of the query encoder, with momentum $m = 0.999$. So the keys in the queue stay consistent. Many negatives, small batches.

BYOL (non-contrastive). An online network (encoder, projector, predictor $q_\theta$) and a target network (encoder, projector) whose weights $\xi$ are an EMA of the online weights, $\xi \leftarrow m\xi + (1 - m)\theta$, with $m$ from 0.996 up to 1. The loss is the squared distance between the normalized prediction $q_\theta(z_\theta)$ and the normalized target projection $z'_\xi$ (it equals $2 - 2\cos$), with a stop-gradient on the target: no gradient flows into the target branch. Collapse is a solution of the loss, but the training dynamics do not go there in practice. SimSiam (Chen and He 2021) removes the EMA and shows that the predictor plus the stop-gradient is enough; without the stop-gradient, the model collapses at once.

DINO (self-distillation). A student and a teacher (an EMA of the student) output a softmax over $K$ dimensions. The student is trained with the cross-entropy $H(P_t, P_s)$ to match the teacher on other views (multi-crop: two global 224-pixel views for the teacher, extra small 96-pixel local views for the student). Two operations on the teacher output prevent collapse:

• centering: subtract an EMA $c$ of the teacher's mean output over the batch. This stops one dimension from dominating, but alone it pushes towards a uniform output;
• sharpening: a low teacher temperature ($\tau_t$ from 0.04 to 0.07, student $\tau_s = 0.1$). This pushes away from uniform, but alone it lets one dimension dominate.

The two effects balance each other.

MAE (masked modelling). Cut the image into patches, as in the vision transformer. Remove a random 75% of them (uniform sampling without replacement). The ViT encoder runs only on the visible 25%, with no mask tokens. A light decoder (by default 8 blocks of width 512, under 10% of the encoder's compute per token) takes the encoded patches plus mask tokens and predicts the pixels. The loss is the mean squared error on the masked patches only; targets with the pixels normalized per patch work better.

Interactive figure: loading…

Why 75%? Images are spatially redundant: at a low ratio, a missing patch can be filled in from its neighbours with little understanding. BERT masks only 15% of the tokens, because each word carries much more information. Removing the masked patches from the encoder also makes pretraining 3 times or more faster.

Method	Negatives	What prevents collapse	Key knobs
SimCLR	in-batch, $2(N-1)$	contrastive denominator	augmentations, $N$, $\tau$, projection head
MoCo	queue of 65,536	contrastive denominator	queue size, momentum 0.999
CLIP	in-batch, $N - 1$ per direction	contrastive denominator	batch 32,768, learned $\tau$
BYOL / SimSiam	none	predictor + stop-gradient (+ EMA target)	EMA rate, predictor
DINO	none	EMA teacher + centering + sharpening	$\tau_t$, multi-crop
MAE	none	reconstruction target	mask ratio 75%, decoder size

Using the learned representation #

After pretraining, the encoder output $h$ (not the projection-head output $z$) is the representation. There are four standard ways to evaluate and use it.

• Linear probe. Freeze the encoder. Compute $h$ for the labelled examples and train only a linear classifier (logistic regression) on it. The accuracy measures how much of the label information is linearly available in $h$. This is the standard test in the SimCLR, MoCo and BYOL papers. A SimCLR ResNet-50 (4×) reached 76.5% top-1 on ImageNet under this protocol, the accuracy of a supervised ResNet-50.
• k-NN evaluation. Freeze the encoder, normalize $h$, and classify a test example by a vote of its $k$ nearest labelled neighbours (DINO uses $k = 20$, with votes weighted by similarity). No training at all, and no hyperparameters to tune, so it is a quick and honest check.
• Fine-tuning. Start from the pretrained weights and train all of them on the labelled task, with a small learning rate (for ViTs often with smaller rates for the early layers). This usually gives the best accuracy when there are enough labels. MAE features are weaker than contrastive features under a linear probe, but they fine-tune very well (He et al. 2022, §4.3). SimCLR fine-tuned on only 1% of the ImageNet labels reached 85.8% top-5 accuracy.
• Zero-shot classification (CLIP). No labelled examples at all. Write one text prompt per class ("a photo of a cat", "a photo of a dog", …), encode the prompts with the text encoder, and classify an image by the softmax over its cosine similarities with the prompts. This is one row of the CLIP similarity matrix. CLIP's best model reached 76.2% top-1 on ImageNet zero-shot, matching the original ResNet-50, which was trained on 1.28 million labelled images.

Rules of thumb: very few labels → linear probe or k-NN on frozen features; many labels → fine-tune; classes given only as names → zero-shot.

4. Worked example #

(a) The CLIP loss for $N = 3$. Cosine similarities (row = image, column = text) and the logits:

Divide the numerator and the denominator of the row softmax by $e^{s_{ii}/\tau}$. Then row $i$'s loss is $\log\big(1 + \sum_{j \ne i} e^{(s_{ij} - s_{ii})/\tau}\big)$.

• Image 1: $\log(1 + e^{1.0 - 4.5} + e^{2.0 - 4.5}) = \log(1 + e^{-3.5} + e^{-2.5}) = \log 1.1123 = 0.1064$.
• Image 2: $\log(1 + e^{0.5 - 4.0} + e^{1.5 - 4.0}) = \log(1 + e^{-3.5} + e^{-2.5}) = 0.1064$.
• Image 3: $\log(1 + e^{2.5 - 3.5} + e^{0 - 3.5}) = \log(1 + e^{-1} + e^{-3.5}) = \log 1.3981 = 0.3351$. Text 1 (similarity 0.5) is a hard negative for image 3.

So $\mathcal{L}_{I\to T} = (0.1064 + 0.1064 + 0.3351)/3 = 0.1826$. The columns give $0.1429$, $0.0659$ and $0.3064$, so $\mathcal{L}_{T\to I} = 0.1717$ and $\mathcal{L}_{\text{CLIP}} = (0.1826 + 0.1717)/2 = 0.1772$. The bound: $I \ge \log 3 - 0.1826 = 0.916$ nats, out of at most $\log 3 = 1.099$.

(b) Same embeddings, $\tau = 1$. Now $\mathcal{L}_{I\to T} = 0.7755$, so the bound is only $\log 3 - 0.7755 = 0.323$ nats. The embeddings did not change; only the critic $f = s/\tau$ did. The bound holds for every critic, and a better critic gives a tighter bound.

(c) The gradient for image 1, $\tau = 0.2$. The softmax of row 1 of $S/\tau$ is $p_{1\cdot} = (0.899, 0.027, 0.074)$. So $\partial\ell_1/\partial s_{1\cdot} = (p_{1\cdot} - e_1)/\tau = (-0.101, 0.027, 0.074)/0.2 = (-0.505, 0.136, 0.369)$. The negative with similarity 0.4 gets $e^{(0.4 - 0.2)/0.2} = e \approx 2.7$ times the push of the negative with similarity 0.2. At $\tau = 1$ the factor is only $e^{0.2} = 1.22$.

(d) Zero-shot. An image embedding has cosine similarities $(0.898, 0.673, 0.158)$ with the prompts for "cat", "dog" and "car". With $\tau = 0.07$ the logits differ from the largest by $(0, -3.21, -10.56)$, so the softmax is $(0.961, 0.039, 0.000)$: "cat".

(e) MAE token count. ViT-B/16 at 224 pixels: $(224/16)^2 = 14^2 = 196$ patches. With 75% masked, the encoder sees $196/4 = 49$. The per-token costs (MLP, projections) drop to 1/4. The cost of the attention matrix drops to $(49/196)^2 = 1/16$.

5. In code #

PyTorch checks the worked example, the gradient, the collapse value, the effect of bounded cosine logits, and zero-shot classification. The notebook for this lesson has these cells and the exercises.

import numpy as np
import torch
import torch.nn.functional as F

# 1. Worked example: CLIP loss as cross-entropy over the similarity matrix
S = torch.tensor([[0.9, 0.2, 0.4],
                  [0.1, 0.8, 0.3],
                  [0.5, 0.0, 0.7]], dtype=torch.float64)    # S[i, j] = cos(image i, text j)
labels = torch.arange(3)                                     # the positive of row i is column i
for tau in (0.2, 1.0):
    l_it = F.cross_entropy(S / tau, labels)                  # image -> text: softmax over each row
    l_ti = F.cross_entropy(S.T / tau, labels)                # text -> image: softmax over each column
    print(tau, round(l_it.item(), 4), round(l_ti.item(), 4),
          round((l_it + l_ti).item() / 2, 4), round(np.log(3) - l_it.item(), 3))
# 0.2 0.1826 0.1717 0.1772 0.916
# 1.0 0.7755 0.7745 0.775 0.323

# 2. Gradient of the row-1 loss with respect to the similarities: (p - e_1) / tau
S1 = S.clone().requires_grad_(True)
F.cross_entropy(S1[:1] / 0.2, labels[:1]).backward()
print(S1.grad[0].numpy().round(4))                           # [-0.5047  0.1357  0.369 ]

# 3. SimCLR NT-Xent on 2N views; a collapsed encoder (all embeddings equal) costs log(2N - 1)
N = 4
z = F.normalize(torch.ones(2 * N, 16, dtype=torch.float64), dim=1)
logits = z @ z.T / 0.5
logits.fill_diagonal_(float("-inf"))                         # a view is not its own negative
pos = (torch.arange(2 * N) + N) % (2 * N)                    # view i and view i + N come from one image
print(round(F.cross_entropy(logits, pos).item(), 4), round(np.log(2 * N - 1), 4))   # 1.9459 1.9459

# 4. Cosine logits lie in [-1/tau, 1/tau]: a lower bound on the loss for N = 4096
N = 4096
for tau in (1.0, 0.1):
    best = np.log1p((N - 1) * np.exp(-2 / tau))
    print(tau, f"{best:.2g}", round(np.log(N) - best, 3))
# 1.0 6.3 1.998
# 0.1 8.4e-06 8.318

# 5. Zero-shot classification = one row of the similarity matrix
img = F.normalize(torch.tensor([[0.8, 0.6, 0.1]]), dim=1)            # one image embedding
prompts = F.normalize(torch.tensor([[1.0, 0.2, 0.0],                  # "a photo of a cat"
                                    [0.1, 1.0, 0.0],                  # "a photo of a dog"
                                    [0.0, 0.1, 1.0]]), dim=1)         # "a photo of a car"
probs = F.softmax(img @ prompts.T / 0.07, dim=1)
print(probs.numpy().round(3))                                         # [[0.961 0.039 0.   ]]

Part 4 shows that a fixed $\tau = 1$ with cosine similarity can never certify more than about 2 nats with 4096 candidates, even with perfect embeddings: the logits cannot spread far enough.

6. Where it shows up #

• Vision–language models. CLIP-style contrastive pretraining gives the image encoders of many multimodal language models and the text encoders of text-to-image diffusion models.
• Vision backbones. DINO and its successors give features for segmentation and depth without labels. MAE pretraining is a standard start for large ViTs (ViT-H reached 87.8% ImageNet top-1 after fine-tuning, with ImageNet-1K data only).
• Language models. BERT's masked-token objective is masked modelling; next-token prediction (lesson 08) is also self-supervised: the target is the next word of the text itself.
• Earth observation and weather. Prithvi (NASA and IBM, 2023) pretrains a ViT as an MAE on Harmonized Landsat Sentinel-2 imagery, with 3-D (time × space) patches. Masked reconstruction is natural for gridded geophysical fields, where clouds and gaps already mask parts of the data.
• Seismology. SeisCLIP (Si et al. 2023) trains a transformer on time–frequency spectra of waveforms and an MLP on phase and source information of the same event, contrastively. Then it fine-tunes the spectrum encoder for event classification, location and focal mechanism.
• Practice. Choose augmentations that keep the physics you care about. A random crop of a weather field changes the location; a colour jitter has no physical meaning. The augmentations decide which invariances the encoder learns.

7. Common mistakes #

• Using the projection-head output $z$ downstream. Use the encoder output $h$.
• Forgetting to normalize the embeddings. The temperature then has no fixed meaning, and SimCLR's results drop a lot without $\ell_2$ normalization.
• Reading $\log N - \mathcal{L}$ as the mutual information. It is a lower bound, capped at $\log N$. A pair with high mutual information always looks like "$\log N$".
• False negatives. In-batch negatives can be true matches (two photos of the same dog, two captions that fit both images). They get pushed apart anyway.
• BatchNorm statistics per device in SimCLR, or a too small $\tau$ (unstable; all the gradient goes to a few negatives).
• Low mask ratios for images. BERT's 15% is right for text, not for pixels.
• Thinking BYOL has no collapse solution. It has one; the training dynamics avoid it.
• Judging an MAE encoder only by a linear probe. Masked models can probe poorly and still fine-tune best; report both.

8. Check yourself #

1. SimCLR with $N = 256$ images (512 views). Give the number of negatives per anchor and the loss of a collapsed encoder.
   Answer

   $2(N-1) = 510$ negatives. All logits are equal, so each softmax gives $1/(2N-1)$ to the positive: the loss is $\log 511 \approx 6.24$.
2. $\tau = 0.1$. An anchor has two negatives with cosine similarity 0.9 and 0.1. What is the ratio of their gradient weights?
   Answer

   $e^{(0.9-0.1)/0.1} = e^{8} \approx 2981$. Almost all the push goes to the hard negative.
3. In the proof of the InfoNCE bound, why is $I(I; Y_{1:N}) = 0$, and where is it used?
   Answer

   The joint density of $(I, Y_{1:N})$ is $\frac1N\prod_j p(y_j)$: without $x$, all candidates are independent samples of $p(y)$, whatever $I$ is. It makes $I(I; D) = I(I; X \mid Y_{1:N})$ in the chain rule (step 5).
4. What is the optimal critic of InfoNCE, and why can you add any $c(x)$ to it?
   Answer

   $f^* = \log\frac{p(y\mid x)}{p(y)} + c(x)$, the pointwise mutual information. The softmax is over the candidates $y_j$ for a fixed $x$, so a term that depends on $x$ only cancels.
5. CLIP with a fixed $\tau = 1$ and $N = 4096$. Even with perfect embeddings, what is the largest bound it can reach, and why?
   Answer

   The logits lie in $[-1, 1]$, so the loss is at least $\log(1 + 4095e^{-2}) \approx 6.32$, and the bound is at most $\log 4096 - 6.32 \approx 2.0$ nats. CLIP learns $\tau$ (starting at 0.07) to avoid this.
6. Why does DINO need both centering and sharpening?
   Answer

   Centering alone stops one dimension from dominating, but drives the teacher output towards uniform. Sharpening (a low $\tau_t$) alone does the opposite. Together they balance, and neither collapse mode wins.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Write the InfoNCE loss for a batch of $N$ pairs with cosine similarities $s_{ij}$.

   Show answer

2. Write the CLIP loss in terms of the similarity matrix $S$.

   Show answer

3. What is collapse in self-supervised learning?

   Show answer

4. State the InfoNCE bound on mutual information.

   Show answer

5. Why can the InfoNCE estimate never exceed $\log N$?

   Show answer

6. Compute the loss of a collapsed encoder under SimCLR's NT-Xent with $N$ images.

   Show answer

7. Give the gradient of a row loss with respect to $s_{ij}$, and say what a small $\tau$ does.

   Show answer

8. What prevents collapse in BYOL and SimSiam?

   Show answer

9. What is the MAE recipe?

   Show answer

10. What is a linear probe?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Write the CLIP loss." — Normalize both embeddings, logits $= UW^\top/\tau$, cross-entropy with labels arange(N) on the rows and on the columns, and average the two.
• "Why do contrastive methods want large batches?" — More negatives make the task harder and the gradient less noisy, and the bound is capped at $\log N$. MoCo gets many negatives from a queue instead.
• "How does BYOL avoid collapse without negatives?" — A predictor, a stop-gradient and an EMA target. Collapse is a solution, but not where the dynamics go (SimSiam: the stop-gradient is essential).
• "Why does MAE mask 75%?" — Spatial redundancy makes low ratios too easy. The encoder runs only on the visible patches, so pretraining is 3 times or more faster.
• "How do you evaluate a self-supervised model?" — Linear probe and k-NN on frozen features, fine-tuning with few labels (1%, 10%), and transfer to other tasks; zero-shot for CLIP.
• "Contrastive or masked for my scientific data?" — Contrastive learning needs augmentations that keep the meaning (hard for physical fields). Masked modelling needs no augmentations and fits gappy gridded data.

10. Practice #

The notebook for this lesson runs in Colab. It uses NumPy for the losses and PyTorch for training a small encoder (under a minute on a CPU). Five exercises, each with a check cell and a folded solution:

1. The CLIP loss from a similarity matrix, written as cross-entropy, checked on the worked example.
2. The gradient $(p - e_i)/\tau$ of a row loss, checked against finite differences.
3. SimCLR's NT-Xent loss on $2N$ views in PyTorch; a collapsed encoder costs $\log(2N - 1)$.
4. Contrastive pretraining of a small MLP on the scikit-learn digits with shift, noise and masking augmentations, then a linear probe on $h$ and on $z$, against a random encoder.
5. The smallest possible loss with cosine logits for a given $N$ and $\tau$, and the largest mutual information the bound can then certify.

11. Watch and read #

Contrastive Learning with SimCLR | Deep Learning AnimatedDeepia · 15 min

Two views, the similarity matrix, the NT-Xent loss and the temperature, animated; matches sections 2 and 3.

How AI 'Understands' Images (CLIP) - ComputerphileComputerphile · 18 min

Image–text pairs, the contrastive objective, and zero-shot classification with text prompts.

Fixing SimCLRs Main Problem - BYOL Paper ExplainedBoris Meinardus · 12 min

Why BYOL needs no negatives: the online and target networks, the predictor, the EMA and the stop-gradient.

DINO: Emerging Properties in Self-Supervised Vision Transformers (Facebook AI Research Explained)Yannic Kilcher · 39 min

Self-distillation with a teacher and a student, multi-crop, and centering plus sharpening against collapse.

Week 10 – Lecture: Self-supervised learning (SSL) in computer vision (CV)Alfredo Canziani · NYU Deep Learning · 2 h

A full lecture: pretext tasks, contrastive learning, the role of negatives and of augmentations, and evaluation.

• Lilian Weng, Contrastive Representation Learning: sections "InfoNCE", "Large batch size", "Hard negative mining", "SimCLR", "BYOL", "MoCo & MoCo-V2" and "CLIP".
• Simon Prince, Understanding Deep Learning (free PDF), chapter 14 (unsupervised learning) and the sections on self-supervised learning and transfer learning in chapter 9.
• Poole et al., On Variational Bounds of Mutual Information (2019): §2.3 (multi-sample bounds, eq. 10 InfoNCE, the $\log K$ ceiling and the optimal critic).
• Chen et al., A Simple Framework for Contrastive Learning of Visual Representations (SimCLR, 2020): §2.1 (method, NT-Xent), §3 (augmentations), §4.2 (projection head), §5.1 (temperature, Table 5), §5.2 (batch size).
• Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP, 2021): §2.3 (the training objective and its pseudocode), §3.1 (zero-shot transfer and prompt engineering).
• He et al., Masked Autoencoders Are Scalable Vision Learners (2022): §3 (method), §4.1 (mask ratio, decoder design, reconstruction target), §4.3 (partial fine-tuning vs linear probing).
• Grill et al., Bootstrap your own latent (BYOL) (2020) and Chen and He, Exploring Simple Siamese Representation Learning (SimSiam, 2021): the stop-gradient experiments.