How to learn here: active learning and spaced repetition

Before this lesson: nothing. Time: about 45 minutes, plus 20 minutes for the notebook.

---

Why this matters #

Machine learning is built in layers. Lesson 02 gives the notation, lesson 07 uses it for gradients, the Calculus track uses gradients for backpropagation, and the Deep Learning track uses backpropagation in every model. If you forget a layer, every later layer becomes hard to read. Most learners do not fail because a single idea is too difficult; they fail because earlier ideas faded.

The methods in this lesson come from cognitive psychology. A large review of ten common study methods (Dunlosky et al., 2013) rated only two of them as "high utility": practice testing and distributed practice (spacing). Re-reading, highlighting and summarizing got the rating "low utility". The same ideas run real software: the scheduler of this site is a variant of SM-2, the algorithm behind early SuperMemo and Anki, and Duolingo trains a regression model to predict when a learner forgets a word (section 6).

1. Predict first #

2. The picture #

Recognizing is not recalling. Look at this line: the sigmoid function is $\sigma(z) = 1/(1 + e^{-z})$. Tomorrow it will look familiar. That feeling is recognition: you know you have seen it. Now cover the line and try to write the formula. That is recall: you produce it from memory. An exam, an interview, or the next lesson needs recall. Re-reading trains recognition and gives a strong illusion of fluency: the text reads easily, so it feels known.

Memory fades, and reviews slow the fading. Psychologists describe forgetting with a forgetting curve: the chance that you can recall a fact drops fast just after learning, then more slowly (Ebbinghaus, 1885; repeated by Murre and Dros, 2015). Each successful recall makes the next drop slower. The figure uses a simple model of this. The retention $R$ is the chance that you can recall the fact. The stability $S$ (in days) says how slowly $R$ drops: a larger $S$ means slower forgetting.

Interactive figure: loading…

Notice three things in the figure:

• After "cram", $R$ falls almost to 0 by day 60. Reviews a few hours apart find $R$ close to 1, so each one adds little.
• After "spaced", the same four reviews keep $R$ near 0.76 on day 60. Each review comes when $R$ has dropped to about 0.6. The recall is harder, and in the model a harder (but successful) recall increases $S$ more.
• After "too late", every review fails: you have already forgotten, and you start again. So the best moment for a review is when recall is hard but still works.

3. Definitions and main results #

Five methods with strong evidence #

Retrieval practice means practicing recall: you answer a question from memory before you see the answer. Psychologists call its benefit the testing effect. In the study of Roediger and Karpicke (2006), students read short texts. One group re-read; the other group read and then wrote down what they could recall. After five minutes the re-reading group did slightly better. After two days and after one week, the recall group remembered clearly more. The students who only re-read also predicted that they would remember more: the illusion of fluency.

Spacing (distributed practice) means that the same study time is spread over several sessions with gaps between them. The spacing effect says that spaced sessions give better long-term memory than one long session (cramming). A review of hundreds of experiments (Cepeda et al., 2006) found the effect again and again; it also found that the best gap grows with how long you want to remember.

Pretesting means trying to answer a question before you study the topic. In experiments by Kornell, Hays and Bjork (2009), people who first guessed (and were mostly wrong) and then studied remembered more than people who used the whole time to study. A wrong guess still helps: it prepares the place where the right answer goes. This is why each lesson starts with "Predict first".

Interleaving means mixing different types of problems in one session, instead of doing all problems of one type in a block. In a study by Rohrer and Taylor (2007) on volumes of geometric solids, the mixed group scored lower during practice but much higher on a test one week later. The mixed practice forces you to choose the method, and that choice is what the test needs.

A desirable difficulty (Bjork, 1994) is a difficulty during practice that improves long-term learning: recalling instead of re-reading, waiting before a review, mixing problem types. Such practice feels slower and less pleasant. That feeling is a sign that the practice works, not that it fails. A difficulty is desirable only if you can still succeed with effort: a recall that fails completely, or a problem far above your level, teaches less.

The forgetting-curve model #

The figure in section 2 uses this model. It is a simplification, chosen because it is easy to compute; real data fit a power law better than an exponential, and real algorithms (section 6) use more parameters.

Definition (retention and stability). After a review at time $t_0$ (in days), the retention at time $t \ge t_0$ is

where $S > 0$ is the stability. At $t = t_0$, $R = 1$. After $S$ days, $R = e^{-1} \approx 0.37$. A fact you have just learned has stability $S_0$ (2 days in the figure).

Definition (what a review does). Let $R$ be the retention just before a review.

1. If $R \ge 0.5$, the recall works, and the stability grows: $S_{\text{new}} = S \cdot \bigl(1 + (k - 1)(1 - R)\bigr)$, with a constant $k > 1$ (5 in the figure).
2. If $R < 0.5$, the recall fails, and you relearn the fact: $S_{\text{new}} = S_0$.

In both cases $R$ jumps back to 1. The factor $1 + (k-1)(1 - R)$ is the "desirable difficulty" in one formula: it is 1 when $R = 1$ (a review right after the last one adds nothing), and it grows as $R$ drops. The threshold 0.5 is the other half of the idea: a recall that is too hard fails.

Result (same number of reviews, different outcome). With $S_0 = 2$ and $k = 5$:

Schedule (4 reviews)	$R$ just before each review	$S$ after the last review	$R$ on day 60
days 1, 1.1, 1.2, 1.3 (cram)	0.61, 0.98, 0.98, 0.98	6.3 days	0.0001
days 1, 4, 12, 32 (spaced)	0.61, 0.56, 0.57, 0.60	101 days	0.76
days 10, 20, 30, 40 (too late)	0.007 each time (all fail)	2 days	0.00005

Why. In the cram schedule only the first review increases $S$ much (from 2 to 5.1). The next three find $R \approx 0.98$, so each multiplies $S$ by only about $1 + 4 \cdot 0.02 = 1.08$. In the spaced schedule each review finds $R$ near 0.6 and multiplies $S$ by about $1 + 4 \cdot 0.4 = 2.6$. Four such factors give about $2 \cdot 2.6^4 \approx 91$ days; the exact computation gives 101 days.

Computing the first spaced review step by step

Start: $S = S_0 = 2$, learned at $t_0 = 0$.

1. Day 1: $R = e^{-(1 - 0)/2} = e^{-0.5} \approx 0.607$. This is at least 0.5, so the recall works.
2. Factor: $1 + (5 - 1)(1 - 0.607) = 1 + 4 \cdot 0.393 \approx 2.574$.
3. New stability: $S = 2 \cdot 2.574 \approx 5.15$ days, and $R$ is 1 again.
4. Day 4: $R = e^{-(4 - 1)/5.15} \approx e^{-0.583} \approx 0.558$; factor $1 + 4 \cdot 0.442 \approx 2.77$; $S \approx 14.2$.

The notebook repeats this computation for all three schedules.

Recall cards and the Review page #

Every lesson ends with recall cards: a question on the front, a short answer on the back. On the page you can test yourself at once. To get the cards back later, press "Add these cards to my review" under the cards, or "Mark this lesson as done" at the end of the lesson (this also adds its cards). The Review page can also add a whole track at once.

The Review page shows the cards that are due. A newly added card is due at once. Answer it in your head (or aloud), press Show answer (or Space), compare, and rate yourself (or press keys 1–4). The page stores four numbers for each card: the interval (days until the next review), the ease (a growth factor, 2.5 for a new card), reps (successful reviews in a row) and lapses (how often you forgot it). This is a simple variant of the SM-2 algorithm.

Rating	When to press it	What the scheduler does
Again (1)	You could not recall the answer, or it was wrong.	The card returns in 10 minutes. Reps go back to 0, so the card starts over at 1 day. Ease $-0.2$ (minimum 1.3). Lapses $+1$.
Hard (2)	You recalled it, with real effort or after a long pause.	Interval $\times 1.2$, at least 1 day. Ease $-0.15$ (minimum 1.3).
Good (3)	You recalled it correctly before you saw the answer.	1 day after the first success, 3 days after the second, then interval $\times$ ease.
Easy (4)	You knew it at once, without effort.	Ease $+0.15$. 4 days for a card with no successful review yet, otherwise interval $\times$ ease $\times 1.3$.

All intervals are rounded to whole days (halves round up).

Result (the "Good" sequence). A new card that you rate Good every time comes back after 1, 3, 8, 20, 50, 125 days. Proof. The first two intervals are 1 and 3 by the rule. From then on, ease stays 2.5 (Good does not change it), and each interval is the previous one times 2.5, rounded: $3 \cdot 2.5 = 7.5 \to 8$, $8 \cdot 2.5 = 20$, $20 \cdot 2.5 = 50$, $50 \cdot 2.5 = 125$. $\blacksquare$

Counted from the day you add the card (day 0), the reviews fall on days 0, 1, 4, 12, 32, 82 and 207: six successful recalls carry a card past half a year.

Interactive figure: loading…

Result (the daily load stays small). Suppose you add $N$ new cards every day and rate every review Good. Then on day $d$ you have $N \cdot c(d)$ reviews, where $c(d)$ is the number of numbers in $0, 1, 4, 12, 32, 82, 207, \dots$ that are at most $d$.

Proof. A card added on day $a$ is shown on the days $a + 0, a + 1, a + 4, a + 12, \dots$. So on day $d$ it is shown exactly when $d - a$ is one of these offsets. For each offset $o \le d$ there is one batch of $N$ cards with $a = d - o$, and these are the only cards shown that day. $\blacksquare$

With $N = 10$: 10 reviews on day 0, 20 on days 1–3, 30 on days 4–11, 40 on days 12–31, 50 on days 32–81. The offsets grow about 2.5 times each step, so $c(d)$ grows like $\log d$: the load grows very slowly. Forgotten cards add some extra reviews, so real numbers are a bit higher. At about 10 seconds per card, 50 reviews take under 10 minutes.

Your deck is stored only in this browser. The site has no accounts and sends nothing to a server. The deck lives in this browser's local storage. Another device, another browser, or cleared site data means an empty deck. The Review page has buttons to save the deck and your progress to a file and to load it on another device, and to export the cards for Anki.

Rate honestly. Rate Good only if you recalled the answer before you saw it. "I recognize it now" is Again. A dishonest Good sends the card away for days, and you lose it.

Using each block of a lesson #

Block	What to do	Time
After this lesson you can	Read it as a checklist. At the end of the lesson, check each line.	1 min
In plain words	Read it first. Then close your eyes and say the main idea in one sentence.	2 min
Predict first	Write down an answer, even a guess. Being wrong is useful (pretesting).	2 min
Figures	Before you move a slider, predict what will happen. Then move it.	3–5 min each
Quick check (quiz)	Choose without scrolling back. Then read "Why", also when you were right.	1 min each
Proofs	Read the steps; then cover them and try to give the reason for each step.	varies
Worked example	Cover the solution, try the first step yourself, then read on.	5–10 min
In code and notebook	Run every cell. Try each exercise for 10 minutes before you open the solution.	20–40 min
Common mistakes	For each one, ask: "Did I make this mistake?"	2 min
Check yourself	Answer on paper, then open the answers.	5 min
Recall cards	Answer each one in your head, then reveal. Add them to your review.	3 min

The quick look, done right #

The Key takeaways box at the end of every lesson, and the track's Quick recap page (all the boxes of the track on one page), are for a fast refresh. Used passively, they only train recognition. Used actively, they are retrieval practice:

1. Read only the lesson title or the heading of the box.
2. Cover the bullets (with your hand, or scroll so they are hidden).
3. Say or write what you remember: the main facts, a formula, an example.
4. Now read the bullets. Mark the ones you missed or got wrong.
5. For each missed bullet, open the lesson section once, or find its recall card.

Two minutes of this give you more than ten minutes of reading the same box. The missed bullets also tell you exactly what to study.

A routine #

• Each day: first the Review page (10–15 minutes), then new material. Reviews come first because they are due today; new material can wait a day.
• Each lesson: Predict first → read with the figures → quick checks → worked example (covered) → notebook → recall cards to your review → mark the lesson as done.
• Each week: one active quick look at the recap page of every track you study.
• Mix: when you practice, mix problems from different lessons (interleaving). The "Check yourself" questions of older lessons are good for this.

This track is the start of the learning paths. If you already read $\Sigma$-notation and NumPy with ease, skim lessons 02 and 03: do their quizzes and add their cards, then move on.

Answers to section 1. (1) Student B remembers more after one week: she practiced retrieval. Student A probably feels better prepared right after the hour, because the text has become familiar (Roediger and Karpicke, 2006, found both effects). (2) Days 1, 4, 12, 32. Reviews on the same day find the fact still fresh and add little; spaced reviews come when recall is harder but still works. In the model of the figure, $R$ on day 60 is 0.76 versus 0.0001.

Making your own cards

You can write your own cards in any spaced-repetition app (or on paper). The rules this site uses come from the research and from Andy Matuschak's guide How to write good prompts:

• One fact per card. "What are the three conditions of X?" is three cards.
• One clear answer. If two different answers are both right, rewrite the question.
• No yes/no questions. "Is the sigmoid increasing?" can be guessed. Ask "On which interval is the sigmoid increasing?" or "What is $\sigma(0)$?"
• Mix kinds: "what is", "why", and "compute this small case".
• Write it in your own words, and add a card when you made a mistake: the mistake is exactly what you need to remember.

4. Worked example #

(a) One card over a month. You add a card on day 0. You rate it Good, Good, then forget it on day 4 (Again), then rate it Good four times. Follow the rules of the table:

Review	Day	Rating	Reps before	Ease after	New interval	Next review
1	0	Good	0	2.5	1 (first success)	day 1
2	1	Good	1	2.5	3 (second success)	day 4
3	4	Again	2	$2.5 - 0.2 = 2.3$	10 minutes; reps $\to 0$	day 4, 10 minutes later
4	4	Good	0	2.3	1 (first success again)	day 5
5	5	Good	1	2.3	3	day 8
6	8	Good	2	2.3	$3 \cdot 2.3 = 6.9 \to 7$	day 15
7	15	Good	3	2.3	$7 \cdot 2.3 = 16.1 \to 16$	day 31

Compare with a card that was never forgotten: its fifth Good gives 50 days, not 16. The lapse cost three extra reviews and a lower ease, which keeps the card more frequent from now on. That is what a lapse should do: the card was hard for you, so you see it more often.

(b) The forgetting model for the spaced schedule. $S_0 = 2$, $k = 5$, reviews on days 1, 4, 12, 32. Each line uses $R = e^{-\Delta t / S}$, where $\Delta t$ is the time since the last review, and $S_{\text{new}} = S\,(1 + 4(1 - R))$:

Day	$\Delta t$	$S$ before	$R = e^{-\Delta t/S}$	Factor $1 + 4(1 - R)$	$S$ after
1	1	2	$e^{-0.5} = 0.607$	2.57	5.15
4	3	5.15	$e^{-0.583} = 0.558$	2.77	14.24
12	8	14.24	$e^{-0.562} = 0.570$	2.72	38.73
32	20	38.73	$e^{-0.516} = 0.597$	2.61	101.21

Day 60: $\Delta t = 28$, so $R = e^{-28/101.21} = e^{-0.277} \approx 0.758$.

5. In code #

The notebook builds both computations step by step. Here is the core of each, in plain Python.

import math

def js_round(x):              # JavaScript's Math.round: halves go up
    return math.floor(x + 0.5)    # (Python's round(2.5) gives 2, not 3)

def next_review(ivl, ease, reps, rating):
    """The site's rule. Returns the new (interval in days, ease, reps)."""
    if rating == "again":
        return 0, max(1.3, ease - 0.2), 0          # due again in 10 minutes
    if rating == "hard":
        ease = max(1.3, ease - 0.15)
        ivl = max(1, js_round((ivl or 1) * 1.2))
    elif rating == "good":
        ivl = 1 if reps == 0 else 3 if reps == 1 else js_round(ivl * ease)
    else:  # "easy"
        ease = ease + 0.15
        ivl = 4 if reps == 0 else js_round(max(ivl, 1) * ease * 1.3)
    return ivl, ease, reps + 1

ivl, ease, reps, days = 0, 2.5, 0, []
for _ in range(6):
    ivl, ease, reps = next_review(ivl, ease, reps, "good")
    days.append(ivl)
print(days)                           # [1, 3, 8, 20, 50, 125]

def retention_day60(reviews, S0=2.0, k=5.0):
    S, last = S0, 0.0
    for t in sorted(reviews):
        R = math.exp(-(t - last) / S)
        S = S * (1 + (k - 1) * (1 - R)) if R >= 0.5 else S0
        last = t
    return math.exp(-(60 - last) / S)

print(round(retention_day60([1, 1.1, 1.2, 1.3]), 4))   # 0.0001
print(round(retention_day60([1, 4, 12, 32]), 4))       # 0.7583

6. Where it shows up #

• Spaced-repetition software. SM-2 (Woźniak, 1990) set the pattern this site follows: intervals of 1 day and 6 days, then multiplication by an "E-Factor" that starts at 2.5 and never goes below 1.3. This site uses 3 days instead of 6 for the second step. Anki used an SM-2 variant for many years and, since version 23.10, also offers FSRS, a model fitted to millions of real reviews. FSRS tracks a difficulty, a stability and a retrievability for each card; the last two are the $S$ and $R$ of section 3.
• Forgetting as a machine-learning problem. Duolingo's half-life regression (Settles and Meeder, 2016) predicts the chance of recall as $p = 2^{-\Delta/h}$, where $\Delta$ is the time since the last practice and the half-life $h$ is predicted from features of the learner and the word. Since $2^{-\Delta/h} = e^{-\Delta \log 2 / h}$, this is the exponential model of section 3 with $S = h / \log 2$ (here $\log$ is the natural logarithm, see lesson 04). The weights are learned from data by minimizing a loss, as in the ML Fundamentals track.
• Your own learning. Interviews and exams test recall without notes. The Interview track is written as questions; answer them aloud before you read the answers.

7. Common mistakes #

• Re-reading as "review". Reading a lesson or summary again trains recognition. Cover it and recall first, then read.
• Opening the answer too early. Thirty seconds of real effort before you reveal an answer is what makes the review work. Give every card an honest attempt.
• Rating Good when you only recognized the answer. If you could not produce it before you saw it, press Again. A false Good removes the card for days.
• Cramming before an exam and nothing after. Cramming can work for tomorrow's test, but the forgetting curve then drops fast. Spaced reviews keep the knowledge for the next track.
• Skipping Predict first because you might be wrong. A wrong guess still helps you learn the right answer. Write a guess down.
• Letting the Review page pile up. If you skip a week, the due cards collect. Do the most urgent ones first and add fewer new cards until the pile is gone. Never clear the deck to "start fresh".
• Opening the notebook solution after one minute. Try for about 10 minutes, use the hints in the check messages, and only then compare with the solution.

8. Check yourself #

1. In your own words: what is the difference between recognition and recall? Which one does an exam test?
   Answer

   Recognition is seeing something and knowing that it is familiar; recall is producing it from memory without seeing it. Exams, interviews and later lessons need recall.
2. A card has interval 20 days, ease 2.5 and reps 4. You rate it Good. What is the new interval? And what would Hard give instead?
   Answer

   Good: $20 \cdot 2.5 = 50$ days. Hard: $\operatorname{round}(20 \cdot 1.2) = 24$ days, and the ease drops to $2.35$.
3. In the forgetting model, why does a review right after another review add almost nothing?
   Answer

   Just after a review, $R \approx 1$. The stability is multiplied by $1 + (k-1)(1 - R) \approx 1$, so it barely grows.
4. Why does this site ask you to predict before you read, even when you cannot know the answer?
   Answer

   Pretesting: trying to answer first, even with a wrong guess, improves how well you remember the answer you then study (Kornell, Hays and Bjork, 2009).
5. You study on your laptop and on your phone. Will your review deck be the same on both?
   Answer

   No. The deck is stored only in each browser. Use "Save deck and progress" on the Review page on one device and "Load from a file" on the other.
6. Describe a two-minute active quick look at a lesson's Key takeaways.
   Answer

   Read only the heading, cover the bullets, say or write what you remember, then read the bullets and mark the ones you missed; revisit those sections or cards.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is the testing effect (retrieval practice)?

   Show answer

2. What is the spacing effect?

   Show answer

3. Why answer the "Predict first" questions even if you will probably be wrong?

   Show answer

4. In the model $R(t) = e^{-(t - t_0)/S}$, what does the stability $S$ mean?

   Show answer

5. When is the best moment for a review, according to the desirable-difficulty idea?

   Show answer

6. A new card is rated Good every time. After how many days does it come back each time (first five)?

   Show answer

7. What does "Again" do to a card on this site?

   Show answer

8. Where is your review deck stored, and how do you move it to another device?

   Show answer

9. How do you make a quick look at a summary box an act of recall?

   Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

This lesson is about how to learn, but two questions come up in ML interviews and research discussions:

• "How would you predict when a student forgets a word?" — Model the recall probability as a decaying function of the time since the last practice, $p = 2^{-\Delta/h}$, and learn the half-life $h$ from features with a regression on logged reviews (half-life regression, Settles and Meeder, 2016). Evaluate on held-out reviews by log loss or AUC.
• "Your validation accuracy looks great but the model fails in production. A study analogy?" — Testing on the training data measures recognition, not recall. Hold out data, just as you hold the answer back during a review.

For your own interview preparation: answer questions aloud before you read the answers, space the practice over weeks, and mix topics in one session.

10. Practice #

The notebook for this lesson runs in Colab or in your browser. It needs no NumPy; every line has a comment. It has four short exercises, each with a check cell and a folded solution:

1. Warm-up: fill in a variable and a small function, run the check, and see a pass and a failure.
2. Implement next_review, the site's rule, and check the sequence 1, 3, 8, 20, 50 and the Again and Easy cases.
3. Simulate the forgetting model for "cram" and "spaced" schedules and compare retention on day 60.
4. Count how many reviews a day you need with 10 new cards a day, and see that the number grows slowly.

11. Watch and read #

Study Strategies: Retrieval PracticeThe Learning Scientists · 3 min

What retrieval practice is, and simple ways to do it when you study alone.

Study Strategies: Spaced PracticeThe Learning Scientists · 2 min

How to spread study sessions over days instead of one long session.

Learning how to learn | Barbara Oakley | TEDxOaklandUniversityTEDx Talks · 18 min

Focused and diffuse modes of thinking, and simple habits against procrastination.

• Nicky Case, How to Remember Anything Forever-ish: an interactive comic about the forgetting curve and spaced repetition.
• Andy Matuschak, How to write good prompts: how to write recall cards that work.
• Michael Nielsen, Augmenting Long-term Memory: a researcher's experience with spaced repetition for mathematics and papers.
• E. L. Bjork and R. A. Bjork (2011), Making things hard on yourself, but in a good way (free PDF): desirable difficulties for students.
• The Learning Scientists: free one-page guides on six study strategies.
• Research cited in this lesson: Dunlosky et al. (2013), doi:10.1177/1529100612453266; Roediger and Karpicke (2006), doi:10.1111/j.1467-9280.2006.01693.x; Cepeda et al. (2006), doi:10.1037/0033-2909.132.3.354; Kornell, Hays and Bjork (2009), doi:10.1037/a0015729; Rohrer and Taylor (2007), doi:10.1007/s11251-007-9015-8; Murre and Dros (2015), doi:10.1371/journal.pone.0120644 (free).
• Spaced-repetition algorithms: SM-2 (P. Woźniak); FSRS; Settles and Meeder (2016), A Trainable Spaced Repetition Model for Language Learning.
• Free course: Learning How to Learn (Coursera, free to audit).