Positional encodings: sinusoidal, learned, RoPE and ALiBi

Before this lesson: lesson 09 (attention and its permutation equivariance) and linear-algebra lesson 04 (rotations and orthogonal matrices). The deeper box on RoPE uses complex numbers $e^{\mathrm{i}\varphi}$ (foundations track). Time: about 65 minutes, plus the notebook.

---

Why this matters #

Without positions, "dog bites man" and "man bites dog" give the same set of output vectors. Every sequence model therefore needs a positional encoding, and the choice has large practical effects. RoPE is used by most open large language models (LLaMA, Mistral, Qwen) and by protein language models such as ESM-2. ALiBi is used by BLOOM and MPT. Learned absolute embeddings are used by GPT-2, BERT and the vision transformer. When a company extends a model from a 4k to a 128k context, the main change is in the positional encoding. In science, the same question decides how a model handles weather fields (latitude matters, longitude almost does not) or seismic records of different lengths.

1. Predict first #

2. The picture #

One frequency: a point on a circle #

Put position $t$ on a circle: $p(t) = (\sin\theta t, \cos\theta t)$. Moving from $t$ to $t + k$ turns the point by the angle $\theta k$, and this is the same turn for every $t$. So "$k$ steps later" is one fixed rotation. But one circle is not enough: $t$ and $t + 2\pi/\theta$ land on the same point.

Many frequencies: the hands of a clock #

A clock solves the same problem with several hands. The second hand is fast: it separates nearby times, but it repeats every minute. The hour hand is slow: it changes little from one minute to the next, but it does not repeat within a day. Together the hands give every time of the day a different reading. The sinusoidal encoding uses many circles with frequencies from fast to slow, $\theta_i = b^{-2i/d}$ with base $b = 10000$, one circle for each pair of dimensions.

Interactive figure: loading…

Two places to put position #

1. Add a position vector to the input (absolute). Token $t$ enters the network as $x_t + p_t$. The score between two tokens can then depend on both absolute positions.
2. Change the score so that it depends on the offset only (relative). RoPE rotates the query and the key; ALiBi adds a penalty to the logit.

RoPE on one pair, with numbers #

Take one pair of dimensions with frequency $\theta = 0.5$, a query content $q = (1, 0)$ and a key content $k = (0, 1)$. RoPE turns the query at position $m$ by the angle $m\theta$ and the key at position $n$ by $n\theta$, with the rotation matrix $R(\varphi) = \begin{pmatrix}\cos\varphi & -\sin\varphi \\ \sin\varphi & \cos\varphi\end{pmatrix}$.

• Positions $(m, n) = (2, 1)$: $R(1)\,q = (\cos 1, \sin 1)$ $= (0.540, 0.841)$ and $R(0.5)\,k = (-\sin 0.5, \cos 0.5)$ $= (-0.479, 0.878)$. The score is $0.540 \cdot (-0.479) + 0.841 \cdot 0.878$ $= 0.479$.
• Positions $(12, 11)$: $R(6)\,q = (0.960, -0.279)$ and $R(5.5)\,k = (0.706, 0.709)$. The score is $0.960 \cdot 0.706 - 0.279 \cdot 0.709 = 0.479$ again.

Both vectors turned by 5 more radians, so the angle between them did not change, and neither did the dot product. The score depends only on $n - m = -1$.

Interactive figure: loading…

3. Definitions and main results #

Notation. The encoding acts on vectors of even dimension $d$. Pair $i$ is the coordinates $(2i, 2i + 1)$, for $i = 0, \dots, d/2 - 1$ (counting from 0). Pair $i$ has the frequency $\theta_i = b^{-2i/d}$ with base $b = 10^4$, so $\theta_0 = 1$ is the fastest pair. The wavelength $2\pi/\theta_i$ is the number of positions for one full turn. $R(\varphi)$ is the $2 \times 2$ rotation by $\varphi$. Positions start at $t = 0$.

Learned absolute embeddings #

A table $E_{\text{pos}} \in \mathbb{R}^{T_{\max} \times D}$ is trained with the model, and row $t$ is added to token $t$ at the input (GPT-2, BERT, the vision transformer). It has $T_{\max}D$ parameters. Limit: there is no row for $t \ge T_{\max}$, so the model cannot run on longer inputs. In the original transformer, learned and sinusoidal encodings gave nearly the same results (Vaswani et al., Table 3, row E).

Sinusoidal encodings #

Definition. $PE(t)_{2i} = \sin(\theta_it)$ and $PE(t)_{2i+1} = \cos(\theta_it)$, for $i = 0, \dots, d/2 - 1$. The vector $PE(t) \in \mathbb{R}^D$ is added to token $t$ at the input ($d = D$). The wavelengths form a geometric sequence from $2\pi$ to almost $2\pi b$.

Theorem (a shift is a rotation). For every $t$ and $k$, $PE(t + k) = M_k\,PE(t)$, with

which does not depend on $t$.

Proof.

1. Fix pair $i$ and write $\theta = \theta_i$. The angle-addition formulas give $\sin(\theta(t + k)) = \sin\theta t\cos\theta k + \cos\theta t\sin\theta k$ and $\cos(\theta(t + k)) = \cos\theta t\cos\theta k - \sin\theta t\sin\theta k$.
2. In matrix form, $u(t + k) = B_k\,u(t)$ with $u(t) = (\sin\theta t, \cos\theta t)$ and $B_k = \begin{pmatrix} \cos\theta k & \sin\theta k \\ -\sin\theta k & \cos\theta k \end{pmatrix}$. The matrix $B_k$ depends on $k$ only.
3. Stack the pairs to get $M_k$. $\blacksquare$

Each block is the rotation $R(-\theta_ik)$, so $M_k$ is orthogonal and $M_aM_b = M_{a+b}$. A consequence: $PE(t)^\top PE(t + k) = \sum_i \cos(\theta_ik)$, a function of the offset only (the quiz of section 2).

Why this does not make attention relative. With additive encodings, the score of tokens $m$ and $n$ is $(x_m + p_m)^\top W(x_n + p_n)$ with $W = W_QW_K^\top/\sqrt{d_k}$. It has four terms: content–content $x_m^\top Wx_n$, content–position $x_m^\top Wp_n$, position–content $p_m^\top Wx_n$, and position–position $p_m^\top Wp_n$. Only the last one can depend on $m - n$ alone, and only for special $W$ (for example $W = I$). The shift property makes relative attention learnable; it does not force it.

RoPE (rotary position embedding) #

Definition. For a vector $x \in \mathbb{R}^d$ at position $m$, $f(x, m) = R_mx$ with

RoPE is applied in every layer, to the queries and the keys of each head after the projections ($d = d_k$, the head size). It never touches the values, and it adds nothing to the residual stream.

Theorem (RoPE scores are relative). $\langle R_mq, R_nk\rangle = q^\top R_{n-m}k$, a function of $q$, $k$ and $n - m$ only.

Proof.

1. A $2 \times 2$ rotation is orthogonal, $R(\varphi)^\top = R(-\varphi)$, and rotations compose: $R(\alpha)R(\beta) = R(\alpha + \beta)$.
2. Block-diagonal matrices are transposed and multiplied block by block. So $R_m^\top = R_{-m}$, and $R_{-m}R_n = R_{n-m}$, because in pair $i$: $R(-m\theta_i)R(n\theta_i) = R((n - m)\theta_i)$.
3. $\langle R_mq, R_nk\rangle = (R_mq)^\top R_nk$ $= q^\top R_m^\top R_nk$ $= q^\top R_{n-m}k$. $\blacksquare$

Properties.

• $R_m$ is orthogonal, so RoPE keeps the norms of $q$ and $k$, and $R_0 = I$. RoPE has no parameters.
• Why not the values: the output $\sum_n a_{mn}R_nv_n$ would carry the absolute position of every key, because the rotations of different keys do not cancel in the sum. Position should change where a head looks, not what it moves.
• Two layouts. The RoFormer paper pairs the coordinates $(2i, 2i + 1)$ ("interleaved"); GPT-NeoX and the Hugging Face LLaMA code pair $(i, i + d/2)$ ("half-split"). The mathematics is the same, but mixing the two layouts when loading weights breaks a model without any error message.
• With these frequencies, an upper bound of the score decreases as $|m - n|$ grows (RoFormer §3.4.3): a mild preference for nearby tokens.

RoPE as complex multiplication

Write pair $i$ of $x$ as one complex number $z_i = x_{2i} + \mathrm{i}\,x_{2i+1}$. The rotation by $\varphi$ is multiplication by $e^{\mathrm{i}\varphi}$, so RoPE is $z_i \mapsto z_i\,e^{\mathrm{i}m\theta_i}$. For two vectors, $\langle x, y\rangle = \sum_i \operatorname{Re}(z_i\bar w_i)$ with $w_i$ the pairs of $y$. Then

$$\langle R_mq, R_nk\rangle = \sum_i \operatorname{Re}\big[q_ie^{\mathrm{i}m\theta_i}\,\overline{k_ie^{\mathrm{i}n\theta_i}}\big] = \sum_i \operatorname{Re}\big[q_i\bar k_i\,e^{\mathrm{i}(m - n)\theta_i}\big],$$

which depends on $m - n$ only. Implementations use this form: torch.view_as_complex, multiply, view_as_real.

The frequency schedule and longer contexts #

The fastest pair ($\theta_0 = 1$) turns once every $2\pi \approx 6.3$ positions: it separates neighbours well, but it repeats. The slowest pair turns once in about $2\pi b$ positions: it is coarse, but it does not repeat. Take a LLaMA-style head ($d = 128$, $b = 10^4$) trained on $L = 4096$ tokens. The slowest pair has $\theta_{63} = 10^{-4 \cdot 126/128} = 1.15 \cdot 10^{-4}$. Up to position 4096 it turns by $4096 \cdot 1.15 \cdot 10^{-4} = 0.47$ rad, only 7.5% of a turn. 18 of the 64 pairs never complete a turn during training. Beyond $L$, these pairs show angles that the model has never seen. This is why RoPE extrapolates poorly: in the experiments of Press et al., a RoPE model trained on 512 tokens improved only up to about 200 tokens beyond its training length.

Interactive figure: loading…

Two ways to extend the context, at a high level (both are followed by a short fine-tune on long sequences):

• Position interpolation (Chen et al. 2023): for a new length $L' = sL$, use the positions $m \cdot L/L'$ instead of $m$. Every angle stays inside the range seen in training. With a short fine-tune (within 1000 steps), it extended LLaMA models to 32k tokens. The cost: neighbouring tokens are separated less by the fast pairs.
• Base scaling: raise $b$. This slows mainly the slow pairs, and the fastest pair ($\theta_0 = b^0 = 1$) does not change. Code Llama raised $b$ from $10^4$ to $10^6$, was fine-tuned on 16k-token sequences, and works with inputs up to 100k tokens. "NTK-aware" scaling uses $b' = b\,s^{d/(d-2)}$, which slows the slowest pair by exactly $s$. YaRN chooses the stretch for each frequency separately.

ALiBi (attention with linear biases) #

ALiBi uses no position embeddings at all. It adds a fixed penalty to the causal logits of head $h$:

with the slopes $m_h = 2^{-8h/H}$ for $h = 1, \dots, H$ ($H$ a power of 2; the reference code uses a slightly different rule otherwise). For $H = 8$ the slopes are $1/2, 1/4, \dots, 1/256$. The slopes are fixed, not learned.

Reading the penalty. The bias multiplies the softmax weight of a key at distance $\Delta = i - j$ by $e^{-m_h\Delta}$. So head $h$ has a half-life of $\ln 2/m_h$ positions: 1.4 for $m_h = 1/2$, and 177 for $m_h = 1/256$. Steep heads look at the last few tokens; flat heads look far back.

Theorem (the softmax ignores a row constant). Adding $+m_h\,j$ instead of $-m_h(i - j)$ to row $i$ gives the same attention weights.

Proof.

1. The two biases differ by $-m_h(i - j) - m_hj = -m_hi$, which is the same for every key $j$ of row $i$.
2. For any constant $c$, $\dfrac{e^{s_j + c}}{\sum_l e^{s_l + c}} = \dfrac{e^ce^{s_j}}{e^c\sum_l e^{s_l}} = \dfrac{e^{s_j}}{\sum_l e^{s_l}}$. $\blacksquare$

So an implementation can add the same vector $m_h \cdot (0, 1, 2, \dots)$ to every row, which is easy to precompute. In the experiments of Press et al., a 1.3B-parameter model trained on 1024 tokens reached, on 2048 tokens, the perplexity of a sinusoidal model trained on 2048 tokens, while training 11% faster and using 11% less memory. The price is a fixed bias towards recent tokens.

Interactive figure: loading…

Comparison #

Scheme	Where position enters	Parameters	The score depends on	Beyond the training length $L$
Learned absolute	added to the input	$T_{\max}D$	content and absolute positions	not defined beyond $T_{\max}$
Sinusoidal	added to the input	0	content and absolute positions	defined, but stops improving after 20–50 extra tokens (Press et al.)
RoPE	rotates $q$ and $k$ in every layer	0	content and $n - m$	degrades; extend by interpolation or base scaling, plus a short fine-tune
ALiBi	bias on the logits in every layer	0	content and $i - j$ (linear penalty)	extrapolates well (train short, test long)

Answers to section 1. (1) Learned absolute: positions 2048–4095 have no embedding, so the model cannot process them at all. RoPE: the model runs, because the rotations are defined for every position, but the slow pairs reach angles never seen in training, and the quality drops soon after 2048 tokens unless the context is extended (interpolation or a larger base, plus a short fine-tune). (2) Exactly $s$: both vectors turn by an extra $100\theta_i$ in every pair, and the score $q^\top R_{n-m}k$ depends only on $n - m = -3$.

4. Worked example #

(a) Sinusoidal, $d = 4$, $b = 10^4$. The frequencies are $\theta_0 = 1$ and $\theta_1 = 10^{-4 \cdot 2/4} = 0.01$. Then

The two blocks of $M_1$ are $\begin{pmatrix} 0.540 & 0.841 \\ -0.841 & 0.540 \end{pmatrix}$ and $\begin{pmatrix} 1.000 & 0.010 \\ -0.010 & 1.000 \end{pmatrix}$. Block 1 times $(0.841, 0.540)$: $0.540 \cdot 0.841 + 0.841 \cdot 0.540 = 0.909$ and $-0.841 \cdot 0.841 + 0.540 \cdot 0.540 = -0.416$. So $M_1PE(1) = (0.909, -0.416, 0.020, 1.000) = (\sin 2, \cos 2, \sin 0.02, \cos 0.02) = PE(2)$ ✓.

(b) RoPE with the same frequencies. Take $q = k = (1, 0, 1, 0)$: the same content at two positions. In each pair, $(1, 0)\,R(\varphi)\,(1, 0)^\top = \cos\varphi$, so the score is $\cos\Delta + \cos 0.01\Delta$ with $\Delta = n - m$:

$\Delta$	fast pair $\cos\Delta$	slow pair $\cos 0.01\Delta$	score
0	1.000	1.000	2.000
1	0.540	1.000	1.540
3	−0.990	1.000	0.010
100	0.862	0.540	1.403

Positions $(m, n) = (5, 2)$ and $(105, 102)$ give the same score, 0.010: only the offset counts. The fast pair alone scores $\Delta = 100$ almost like $\Delta = 0$ (0.862 vs 1): it aliases, because 100 is close to 16 full turns ($16 \cdot 2\pi = 100.5$). The slow pair separates them (0.540 vs 1). For this $q$ and $k$, the RoPE score equals $PE(m)^\top PE(n)$ of the sinusoidal encoding.

(c) ALiBi, $H = 8$. A key 100 positions back gets the bias $-100/2 = -50$ in head 1, a weight factor $e^{-50} \approx 2 \cdot 10^{-22}$: invisible. In head 8 the bias is $-100/256 = -0.39$, a factor of 0.68: still clearly visible.

(d) Extending a context. A head with $d = 128$, $b = 10^4$, trained on $L = 4096$, run on $L' = 16384$ ($s = 4$):

• no change: the slowest pair turns by $16384 \cdot 1.15 \cdot 10^{-4} = 1.89$ rad, four times the largest angle it saw (0.47 rad);
• position interpolation: the slowest pair turns by $16384/4 \cdot 1.15 \cdot 10^{-4} = 0.47$ rad, as in training, and the fastest pair turns $1/4 = 0.25$ rad between neighbours instead of 1 rad;
• NTK-aware base scaling: $b' = 10^4 \cdot 4^{128/126} \approx 40{,}900$; the fastest pair still turns 1 rad between neighbours, and the slowest pair turns by $0.47$ rad up to $L'$.

5. In code #

NumPy checks the worked example and the two theorems. The notebook builds the same functions as exercises.

import numpy as np
from scipy.special import softmax

def freqs(d, base=1e4):
    return base ** (-2 * np.arange(d // 2) / d)          # theta_i, i = 0 .. d/2 - 1

def sinusoidal(T, d, base=1e4):
    ang = np.arange(T)[:, None] * freqs(d, base)          # (T, d/2)
    pe = np.empty((T, d))
    pe[:, 0::2], pe[:, 1::2] = np.sin(ang), np.cos(ang)
    return pe

def rope(x, pos, base=1e4):
    """Rotate the pairs (2i, 2i+1) of row x[t] by the angle pos[t] * theta_i."""
    ang = np.asarray(pos, float)[:, None] * freqs(x.shape[1], base)
    c, s = np.cos(ang), np.sin(ang)
    out = np.empty_like(x)
    out[:, 0::2] = c * x[:, 0::2] - s * x[:, 1::2]
    out[:, 1::2] = s * x[:, 0::2] + c * x[:, 1::2]
    return out

# (a) PE(t + k) = M_k PE(t), with the 2x2 blocks [[cos, sin], [-sin, cos]]
pe, th, k = sinusoidal(50, 4), freqs(4), 7
blk = lambda a: np.array([[np.cos(a), np.sin(a)], [-np.sin(a), np.cos(a)]])
M = np.block([[blk(th[0] * k), np.zeros((2, 2))], [np.zeros((2, 2)), blk(th[1] * k)]])
print(pe[1].round(3), np.allclose(pe[k:], pe[:-k] @ M.T))    # [0.841 0.54  0.01  1.   ] True

# (b) RoPE scores depend on n - m only: the table of the worked example
q = np.array([[1., 0., 1., 0.]])
score = lambda m, n: (rope(q, [m]) @ rope(q, [n]).T).item()
print([round(score(0, d), 3) for d in (0, 1, 3, 100)])        # [2.0, 1.54, 0.01, 1.403]
print(round(score(5, 2), 3), round(score(105, 102), 3))        # 0.01 0.01

# (c) ALiBi: slopes, and -m(i - j) or +m j give the same softmax in row i
H, i = 8, 5
m = 2.0 ** (-8 * np.arange(1, H + 1) / H)
print(1 / m)                                    # [  2.   4.   8.  16.  32.  64. 128. 256.]
s, j = np.random.default_rng(0).normal(size=i + 1), np.arange(i + 1)
print(np.allclose(softmax(s - m[1] * (i - j)), softmax(s + m[1] * j)))   # True

# (d) the frequency schedule of a d = 128 head trained on L = 4096 tokens
th, L = freqs(128), 4096
print((2 * np.pi / th > L).sum(), round(th[-1] * L, 2))       # 18 0.47

6. Where it shows up #

• Language models. RoPE is the default in most open LLMs (the LLaMA family, Mistral, Qwen). Long-context versions raise the base (LLaMA 3 uses $b = 500{,}000$) or interpolate positions, then fine-tune briefly. BLOOM and MPT use ALiBi. GPT-2 and BERT use learned absolute embeddings; T5 adds a learned bias for each bucket of relative distances.
• Vision. The vision transformer uses learned position embeddings for its patches (lesson 11). Diffusion transformers (DiT) add fixed 2-D sine–cosine embeddings. A 2-D RoPE splits the pairs between the row and the column index.
• Proteins. ESM-2 and many later protein language models use RoPE, so that a model trained on short sequences runs on longer proteins.
• Weather and climate. Pangu-Weather replaces the relative position bias of the Swin transformer with an "Earth-specific" bias: separate parameters for each latitude and pressure level, shared along longitude. The dynamics are close to invariant along longitude but not along latitude or height, so a relative scheme fits one axis and an absolute scheme the others.
• Time series and seismology. When only time offsets matter and record lengths vary, a relative scheme (RoPE, ALiBi) is the natural choice.

7. Common mistakes #

• RoPE on the values or on the residual stream. The relative property holds only for the scores $q^\top k$.
• Interleaved vs half-split layout mixed between the weights and the code.
• Frequencies computed with the model width $D$ instead of the head size $d_k$. RoPE is applied per head.
• Expecting sinusoidal encodings to extrapolate because they are defined for every $t$. The network after them has never seen those vectors.
• "Sinusoidal encodings make attention relative." The dot product $PE(m)^\top PE(n)$ is relative, but the full score has three more terms that are not.
• Forgetting the row-constant argument. The ALiBi bias $+m\,j$ is not "wrong"; it differs from $-m(i - j)$ only by a constant in each row.

8. Check yourself #

1. RoPE with $d = 4$, $\theta = (1, 0.01)$, $q = (0, 1, 1, 0)$ and $k = (1, 0, 0, 1)$. Give the score for $(m, n) = (4, 1)$ and for $(m, n) = (1, 4)$.
   Answer

   Pair 0 gives $(0, 1)\,R(\varphi)\,(1, 0)^\top = \sin\varphi$ and pair 1 gives $(1, 0)\,R(\varphi')\,(0, 1)^\top = -\sin\varphi'$, with $\varphi = \Delta$, $\varphi' = 0.01\Delta$ and $\Delta = n - m$. So the score is $\sin\Delta - \sin 0.01\Delta$. For $(4, 1)$: $\Delta = -3$, score $-0.141 + 0.030 = -0.111$. For $(1, 4)$: $+0.111$. The sign of the offset matters: RoPE can tell "before" from "after".
2. Why is RoPE applied to queries and keys, but not to values?
   Answer

   Only the score $q^\top R_{n-m}k$ is relative. Rotated values would put the absolute position of each key into the output, and the rotations of different keys do not cancel in the weighted sum. Position should decide where a head looks, not change what it moves.
3. ALiBi with $H = 4$ heads: give the slopes and their half-lives.
   Answer

   $m_h = 2^{-8h/4} = 2^{-2h}$: $1/4, 1/16, 1/64, 1/256$. Half-lives $\ln 2/m_h$: 2.8, 11.1, 44.4 and 177.4 positions.
4. A head with $d = 64$ and $b = 10^4$ is trained on $L = 2048$ tokens. How many of its 32 pairs never complete a turn during training?
   Answer

   Pair $i$ has wavelength $2\pi \cdot 10^{4 \cdot 2i/64} = 2\pi \cdot 10^{i/8}$. It is longer than 2048 when $10^{i/8} > 326$, that is $i/8 > 2.51$, so $i \ge 21$: pairs 21 to 31, which is 11 pairs.
5. Write the attention score with additive position vectors as four terms. Which term can depend on the offset only?
   Answer

   $(x_m + p_m)^\top W(x_n + p_n) = x_m^\top Wx_n + x_m^\top Wp_n + p_m^\top Wx_n + p_m^\top Wp_n$. Only the position–position term $p_m^\top Wp_n$ can depend on $m - n$ alone, and only for special $W$, for example $W = I$ with sinusoidal $p$.
6. A model with $D = 768$ uses learned absolute embeddings for $T_{\max} = 2048$ positions. How many parameters do they have, and what happens at position 2048?
   Answer

   $2048 \cdot 768 = 1{,}572{,}864$ parameters. Position 2048 (counting from 0) has no row in the table, so the model cannot process it.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Write the sinusoidal positional encoding.

   Show answer

2. How is $PE(t + k)$ related to $PE(t)$ for the sinusoidal encoding?

   Show answer

3. What is the main limit of learned absolute position embeddings?

   Show answer

4. Define RoPE.

   Show answer

5. Why is a RoPE score relative?

   Show answer

6. Which RoPE pair rotates fastest, and how fast?

   Show answer

7. Why does RoPE degrade beyond the training length, and what is position interpolation?

   Show answer

8. Give the ALiBi logit and its slopes for $H$ heads.

   Show answer

9. Why may ALiBi add $m\,j$ instead of $-m(i - j)$?

   Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "How does RoPE work, and why is it relative?" — Rotate each pair of $q$ and $k$ by $m\theta_i$; $R_m^\top R_n = R_{n-m}$, so the score depends on $n - m$. No parameters, values untouched.
• "Extend a 4k-context model to 32k." — Position interpolation or base scaling (NTK-aware, YaRN), then a short fine-tune on long data; evaluate long-range retrieval (for example passkey retrieval), not only perplexity.
• "Why does ALiBi extrapolate?" — The penalty is a fixed function of distance, so longer inputs bring no new kind of position signal. The cost is a bias towards recent tokens.
• "Sinusoidal or learned embeddings?" — Nearly the same quality in the original transformer; learned ones cannot go beyond $T_{\max}$; neither extrapolates well.
• "Your RoPE model gives garbage after loading weights from another library." — Check the pair layout (interleaved vs half-split) and that the frequencies use the head size.

10. Practice #

The notebook for this lesson runs in Colab. Five exercises, each with a check cell and a folded solution:

1. The sinusoidal encoding and the shift matrix $M_k$; check $PE(t + k) = M_kPE(t)$.
2. RoPE on a $(T, d)$ array of queries or keys; check that norms are kept and that scores depend on the offset only.
3. ALiBi slopes and the $(H, T, T)$ bias tensor; check the row-constant argument.
4. Position interpolation: RoPE angles beyond $L$, with and without interpolation.
5. Build by hand a RoPE attention head that always attends $k$ tokens back, and use it to copy a token sequence; without positions the same head cannot do it.

11. Watch and read #

Rotary Positional Embeddings: Combining Absolute and RelativeEfficient NLP · 11 min

Why RoPE, the 2-D rotation picture, and the relative property.

How Rotary Position Embedding Supercharges Modern LLMs [RoPE]Jia-Bin Huang · 14 min

From sinusoidal encodings to RoPE, with clear figures of the rotations.

LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLUUmar Jamil · 1 h 11 min

From 24:31, RoPE as implemented in LLaMA, including the complex-number form.

ALiBi - Train Short, Test Long: Attention with linear biases enables input length extrapolationYannic Kilcher · 31 min

From 11:50, the ALiBi bias, the choice of slopes and the extrapolation results.

• Vaswani et al., Attention Is All You Need (2017): §3.5 (sinusoidal positional encoding) and Table 3, row (E) (learned vs sinusoidal).
• Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding: §3.2 (the 2-D case and the general form) and §3.4 (derivation, efficient implementation, long-term decay).
• EleutherAI, Rotary Embeddings: A Relative Revolution: intuition, derivation and code.
• Press, Smith, Lewis, Train Short, Test Long (ALiBi): §2.2 (how sinusoidal, rotary and T5-bias models extrapolate) and §3 (the method and its slopes).
• Chen et al., Extending Context Window of Large Language Models via Positional Interpolation (2023), and Peng et al., YaRN (2023), for context extension.
• Simon Prince, Understanding Deep Learning (free PDF), §12.3.1 (absolute and relative positional encodings).