Probability, conditional probability and Bayes' rule

Before this lesson: nothing from this track; this is the first lesson. You need sets and their notation: union $A \cup B$, intersection $A \cap B$, complement $A^c$, the empty set $\emptyset$ (foundations, lesson 2). A gentle start with chance and data is foundations, lesson 8 (optional). Time: about 80 minutes, plus the notebook.

---

Why this matters #

Every probabilistic model in machine learning is built from the rules of this lesson. A posterior over parameters, $p(\theta \mid \mathcal D) \propto p(\mathcal D \mid \theta)\,p(\theta)$, is Bayes' rule (lesson 10). A naive Bayes spam filter multiplies likelihood ratios, which is correct only under conditional independence. Hidden Markov models, Bayesian networks and the forward process of a diffusion model are lists of conditional independences.

The base-rate effect explains why a fraud detector with 99% recall can still produce mostly false alarms, and why many "significant" A/B test results are not real effects. Questions about Bayes' rule and the Monty Hall problem are also among the first questions in data science and ML interviews.

1. Predict first #

2. The picture #

Outcomes, events and counting #

Roll a fair die. The sample space $\Omega = \{1, 2, 3, 4, 5, 6\}$ is the set of all possible outcomes. An event is a subset of $\Omega$, for example "even" $= \{2, 4, 6\}$. We say that the event happens when the outcome lies in it.

When all outcomes are equally likely and $\Omega$ is finite,

So $P(\text{even}) = 3/6 = 1/2$. For two dice, $\Omega$ has $6 \times 6 = 36$ equally likely ordered pairs $(i, j)$. The sum is 7 for $(1,6), (2,5), (3,4), (4,3), (5,2), (6,1)$, so $P(\text{sum} = 7) = 6/36 = 1/6$.

Counting gets hard quickly, so here are three tools.

• Multiplication rule. If a choice has $n_1$ options and, for each of them, a second choice has $n_2$ options, there are $n_1 n_2$ pairs. Two dice: $6 \cdot 6 = 36$.
• Permutations (ordered, no repetition). The number of ordered lists of $k$ different items taken from $n$ is $n(n-1)\cdots(n-k+1) = \dfrac{n!}{(n-k)!}$. With $k = n$ this is $n!$, the number of orders of $n$ items: 3 books can stand on a shelf in $3! = 6$ orders.
• Combinations (unordered). The number of $k$-element subsets of an $n$-element set is $\displaystyle\binom{n}{k} = \frac{n!}{k!\,(n-k)!}$. Each subset appears in $k!$ orders among the permutations, so we divide by $k!$. Example: in 4 fair coin flips, there are $2^4 = 16$ equally likely sequences, and $\binom{4}{2} = 6$ of them have exactly 2 heads. So $P(\text{exactly 2 heads}) = 6/16 = 0.375$.

The birthday problem. Assume 365 equally likely birthdays, independent between people (ignore 29 February and twins). Among $n$ people, count the outcomes with no shared birthday: the first person has 365 options, the second 364, and so on. So

For $n = 23$ this gives $1 - 0.4927 = 0.5073$. The number is large because there are $\binom{23}{2} = 253$ pairs of people, and each pair is a chance for a match. This trick, "compute the complement", is often the easiest path.

Probability as mass. In general, outcomes need not be equally likely. Think of probability as mass spread over $\Omega$, with total mass 1. $P(A)$ is the mass inside $A$. There is also a frequency view: if you repeat the experiment many times, the share of repetitions in which $A$ happens gets close to $P(A)$. Roll a die 10,000 times and about 1/6 of the rolls show a 6 (the notebook does it). Lesson 06 proves this as the law of large numbers.

Conditioning: zoom in and rescale #

You learn that the outcome lies in an event $B$. Then the outcomes outside $B$ are no longer possible. The new probability of $A$ is the share of $B$'s mass that also lies in $A$:

Read $P(A \mid B)$ as "the probability of $A$ given $B$". For the die: given "even", each of 2, 4, 6 has probability $1/3$, and $P(6 \mid \text{even}) = (1/6)/(1/2) = 1/3$.

Interactive figure: loading…

Independence as a picture. If $A$ covers the same share of $B$ as of the whole square, then learning $B$ does not change the chance of $A$: $P(A \mid B) = P(A)$. Then $P(A \cap B) = P(A)P(B)$. This is independence; section 3 defines it precisely.

Bayes' rule with counts #

Now the question from section 1. Imagine 10,000 people. This is the method of natural frequencies: replace probabilities by counts of people, and Bayes' rule becomes a division.

• 1% are sick: 100 sick and 9,900 healthy people.
• The test is positive for 90% of the sick: 90 true positives (10 sick people are missed).
• The test is negative for 91% of the healthy, so positive for 9%: $0.09 \cdot 9{,}900 =$ 891 false positives.

In total $90 + 891 = 981$ people test positive, and only 90 of them are sick:

About 9%, not 90%. The test is good, but the disease is rare: the small error rate of 9% acts on a very large group (9,900 healthy people) and produces many more false alarms than there are sick people.

Interactive figure: loading…

The second test shows the power of more evidence. Of the 90 sick positives, 90% (81) test positive again. Of the 891 healthy positives, only 9% (about 80) do. Now $P(\text{sick} \mid +, +) \approx 81/161 \approx 0.50$. This assumes that the second test makes its errors independently of the first one, given the true state; section 3 makes this precise.

3. Definitions and main results #

The axioms of probability #

A probability space has three parts: a sample space $\Omega$, a collection $\mathcal F$ of events (subsets of $\Omega$), and a function $P$ that gives each event a number. Kolmogorov's axioms are:

1. $P(A) \ge 0$ for every event $A$.
2. $P(\Omega) = 1$.
3. Countable additivity: if $A_1, A_2, \dots$ are pairwise disjoint ($A_i \cap A_j = \emptyset$ for $i \ne j$), then $P\left(\bigcup_i A_i\right) = \sum_i P(A_i)$.

Everything else in probability follows from these three rules.

Why $\mathcal F$ is not always "all subsets"

For a finite $\Omega$ you can take $\mathcal F$ = all subsets. For $\Omega = [0, 1)$, there is no way to give every subset a probability such that the axioms hold, every interval has $P([a, b)) = b - a$, and a shift of a set (modulo 1) keeps its probability: Vitali's construction gives a set that breaks it. So $\mathcal F$ is a σ-algebra: a collection that contains $\Omega$ and is closed under complements and countable unions. All sets you meet in practice (intervals, their unions, sets defined by continuous functions) are in it. In this track, read "event" as "a set you are allowed to ask about".

Derived rules. Let $A, B$ be events.

1. $P(\emptyset) = 0$. The sets $\Omega, \emptyset, \emptyset, \dots$ are pairwise disjoint and their union is $\Omega$. Axiom 3 gives $P(\Omega) = P(\Omega) + \sum_{i \ge 2} P(\emptyset)$. A sum of equal nonnegative numbers is 0 only if each one is 0.
2. Finite additivity. For disjoint $A_1, \dots, A_n$, add empty sets $A_{n+1} = A_{n+2} = \dots = \emptyset$ and use axiom 3 and rule 1: $P(A_1 \cup \dots \cup A_n) = P(A_1) + \dots + P(A_n)$.
3. Complement rule $P(A^c) = 1 - P(A)$. $A$ and $A^c$ are disjoint, and their union is $\Omega$. So $P(A) + P(A^c) = P(\Omega) = 1$.
4. Monotonicity. If $A \subseteq B$, then $P(A) \le P(B)$. Write $B = A \cup (B \setminus A)$, a disjoint union, so $P(B) = P(A) + P(B \setminus A) \ge P(A)$. In particular, $P(A) \le 1$.
5. Inclusion–exclusion $P(A \cup B) = P(A) + P(B) - P(A \cap B)$. Write $A \cup B = A \cup (B \setminus A)$ and $B = (A \cap B) \cup (B \setminus A)$; both are disjoint unions. So $P(A \cup B) = P(A) + P(B \setminus A)$ and $P(B \setminus A) = P(B) - P(A \cap B)$. Substitute the second equation into the first. $\blacksquare$

The term $P(A \cap B)$ is subtracted because $P(A) + P(B)$ counts the overlap twice.

Union bound (Boole's inequality). $P(A \cap B) \ge 0$, so rule 5 gives $P(A \cup B) \le P(A) + P(B)$. By induction on $n$,

It needs no assumption about the events. It is the main tool when many things can go wrong at the same time: many tests in a study (lesson 11), many hypotheses in a learning bound (lesson 07).

Conditional probability, trees and the law of total probability #

Definition. For $P(B) > 0$, the conditional probability of $A$ given $B$ is $P(A \mid B) = P(A \cap B) / P(B)$.

For a fixed $B$, the function $A \mapsto P(A \mid B)$ satisfies all three axioms (check: it is $\ge 0$; $P(\Omega \mid B) = P(B)/P(B) = 1$; and disjoint $A_i$ give disjoint $A_i \cap B$). So every rule of this lesson also holds with "$\mid B$" added everywhere, for example $P(A^c \mid B) = 1 - P(A \mid B)$.

Multiplication rule. Multiply the definition by $P(B)$: $P(A \cap B) = P(A \mid B)\,P(B)$. Repeated, it gives the chain rule $P(A_1 \cap A_2 \cap A_3) = P(A_1)\,P(A_2 \mid A_1)\,P(A_3 \mid A_1 \cap A_2)$.

Probability trees. A tree draws the chain rule. Each branch carries a conditional probability, and the probability of a path is the product along it. Example: draw two cards from a 52-card deck without putting the first back. Write $A_1$ = "first card is an ace", $A_2$ = "second card is an ace".

The event $A_2$ is reached by two paths, so add them: $P(A_2) = \tfrac{12 + 192}{2652} = \tfrac{1}{13}$. This is exactly $4/52$: before you look at the first card, the second card is an ace with the same chance as the first. Adding the paths is the next theorem.

Theorem (law of total probability). Let $B_1, \dots, B_n$ be a partition of $\Omega$ (pairwise disjoint, with union $\Omega$) with all $P(B_i) > 0$. Then

Proof.

1. $A = \bigcup_i (A \cap B_i)$, because the $B_i$ cover $\Omega$.
2. The sets $A \cap B_i$ are pairwise disjoint, because the $B_i$ are.
3. Finite additivity gives $P(A) = \sum_i P(A \cap B_i)$.
4. The multiplication rule gives $P(A \cap B_i) = P(A \mid B_i)\,P(B_i)$. $\blacksquare$

In the medical test, the partition is {sick, healthy}, and $P(+) = 0.9 \cdot 0.01 + 0.09 \cdot 0.99 = 0.0981$: the 981 positives out of 10,000.

Bayes' rule #

Theorem (Bayes' rule). Let $P(A) > 0$ and $P(B) > 0$. Then

If also $P(B) < 1$, the denominator is $P(A) = P(A \mid B)P(B) + P(A \mid B^c)P(B^c)$. More generally, for a partition $B_1, \dots, B_n$: $P(B_j \mid A) = P(A \mid B_j)P(B_j) / \sum_i P(A \mid B_i)P(B_i)$.

Proof. Both $P(B \mid A)\,P(A)$ and $P(A \mid B)\,P(B)$ equal $P(A \cap B)$, by the multiplication rule. Divide by $P(A)$. The denominator is the law of total probability. $\blacksquare$

Write $H$ for a hypothesis (sick, fraud, spam) and $E$ for the evidence (a positive test, a flag, a word in an e-mail). The parts of Bayes' rule have standard names:

The prior is what you believe before you see $E$. The likelihood says how well $H$ predicts $E$. The posterior is the updated belief. For a test, the standard words are: sensitivity (or recall, true positive rate) $= P(+ \mid H)$, specificity $= P(- \mid H^c)$, false positive rate $= P(+ \mid H^c) = 1 - \text{specificity}$, prevalence (base rate) $= P(H)$, and precision (positive predictive value) $= P(H \mid +)$.

The base-rate fallacy is to read $P(E \mid H)$ as if it were $P(H \mid E)$. When the prior $P(H)$ is small, most positives come from the large group $H^c$, even when the false positive rate is small.

Odds and likelihood ratios #

The odds of an event are $P/(1-P)$. Probability 0.5 is odds 1 ("1 to 1"); probability 0.01 is odds $1/99$. Write Bayes' rule for $H$ and for $H^c$, and divide. The evidence $P(E)$ cancels:

The likelihood ratio (LR, also called the Bayes factor) measures the strength of the evidence. For a positive test, $\text{LR}_+ = \text{sensitivity}/(1 - \text{specificity})$. For the medical test, $\text{LR}_+ = 0.9/0.09 = 10$. The prior odds are $0.01/0.99 = 1/99$, so the posterior odds are $10/99$, and $P(\text{sick} \mid +) = 10/(10 + 99) = 10/109 \approx 0.092$, as with the counts. A negative test has $\text{LR}_- = (1 - \text{sensitivity})/\text{specificity} = 0.1/0.91 \approx 0.11$.

Take logarithms and the product becomes a sum: each piece of evidence adds $\log \text{LR}$ to the log-odds. Strong evidence is a long step; useless evidence ($\text{LR} = 1$) is no step.

Two tests. Suppose two results $E_1, E_2$ are conditionally independent given $H$ and given $H^c$ (defined below): $P(E_1 \cap E_2 \mid H) = P(E_1 \mid H)\,P(E_2 \mid H)$, and the same given $H^c$. Then the likelihood ratio of both results is the product of the two likelihood ratios. Equivalently: update with $E_1$, and use that posterior as the prior for $E_2$. Two positive medical tests: posterior odds $= (1/99) \cdot 10 \cdot 10 = 100/99$, so $P(\text{sick} \mid +, +) = 100/199 \approx 0.503$.

Interactive figure: loading…

When the product rule fails. Two tests on the same blood sample, two sensors on the same cable, or two runs of the same model are not conditionally independent. If the second test always repeats the first one, then $P(E_2 \mid E_1, H) = P(E_2 \mid E_1, H^c) = 1$: the second result has likelihood ratio 1 and adds nothing.

Independence: three kinds #

• $A$ and $B$ are independent if $P(A \cap B) = P(A)\,P(B)$. If $P(B) > 0$, this is the same as $P(A \mid B) = P(A)$. (The product form also works when $P(B) = 0$, and it is symmetric in $A$, $B$.)
• $A_1, \dots, A_n$ are mutually independent if $P\left(\bigcap_{i \in S} A_i\right) = \prod_{i \in S} P(A_i)$ for every subset $S$ of the indices. For three events: the three pair conditions and $P(A \cap B \cap C) = P(A)P(B)P(C)$.
• They are pairwise independent if only the pair conditions hold.
• $A$ and $B$ are conditionally independent given $C$ (with $P(C) > 0$) if $P(A \cap B \mid C) = P(A \mid C)\,P(B \mid C)$.

Disjoint is not independent. If $A \cap B = \emptyset$ and $P(A), P(B) > 0$, then $P(A \cap B) = 0 < P(A)P(B)$. Disjoint events are strongly dependent: if $A$ happens, $B$ cannot.

Mutual independence implies pairwise independence. Every other implication fails:

Gap	Counterexample
pairwise $\not\Rightarrow$ mutual	$X, Y$ are independent fair coins with values 0/1, and $Z = X \oplus Y$ (XOR: $Z = 1$ when exactly one of them is 1). The 4 outcomes $(X, Y, Z) \in \{(0,0,0), (0,1,1), (1,0,1), (1,1,0)\}$ are equally likely. Each pair takes its 4 values with probability $1/4$, so every pair is independent. But $P(X{=}1, Y{=}1, Z{=}1) = 0 \ne 1/8$.
conditional $\not\Rightarrow$ unconditional	A common cause. Two students take an exam that is easy ($P = 0.7$) or hard ($P = 0.3$). Each student fails with probability 0.1 on an easy exam and 0.6 on a hard one, independently given the difficulty. Then $P(F_1) = 0.7 \cdot 0.1 + 0.3 \cdot 0.6 = 0.25$, but $P(F_1 \cap F_2) = 0.7 \cdot 0.1^2 + 0.3 \cdot 0.6^2 = 0.115 \ne 0.25^2 = 0.0625$.
unconditional $\not\Rightarrow$ conditional	A collider (common effect). Two independent servers each fail with probability 0.1. An alarm rings if at least one fails. Given the alarm, learning that server 2 failed makes a failure of server 1 less likely: $0.526$ drops to $0.1$ (explaining away; check-yourself question 5).

In the common-cause example, $P(F_2 \mid F_1) = 0.115/0.25 = 0.46$, much more than 0.25. A failing student 1 is evidence for a hard exam, and a hard exam raises the chance that student 2 fails.

Four classic traps #

1. The base-rate fallacy and the prosecutor's fallacy. $P(E \mid H) \ne P(H \mid E)$. In court, "the chance of this evidence if the defendant is innocent is 1 in a million" is $P(E \mid H^c)$. It is not the chance that the defendant is innocent. In a city of 10 million people, about 10 innocent people would also match the evidence.

2. The Monty Hall problem. You pick door 1. The host knows where the car is. He always opens a door that you did not pick and that hides a goat (if he has two such doors, he picks one at random). He opens door 3. Let $C_i$ = "the car is behind door $i$", with prior $1/3$ each, and $O_3$ = "the host opens door 3". The likelihoods follow from the host's rule:

• $P(O_3 \mid C_1) = 1/2$: the car is behind your door, so both other doors hide goats;
• $P(O_3 \mid C_2) = 1$: he cannot open your door or the car's door;
• $P(O_3 \mid C_3) = 0$: he never shows the car.

Total probability: $P(O_3) = \tfrac13 \cdot \tfrac12 + \tfrac13 \cdot 1 + 0 = \tfrac12$. Bayes' rule:

Switching wins with probability 2/3. The host's action carries information because it depends on where the car is. Your own door keeps its prior 1/3: the host never opens it, whatever is behind it. With $n$ doors and a host who opens $n - 2$ goat doors, switching wins with probability $(n-1)/n$.

Interactive figure: loading…

Now change the host. He does not know where the car is, and he opens one of the two other doors at random. He opens door 3, and it shows a goat. Call this observation $O$. He chooses door 2 or door 3 with probability $1/2$ each, so:

• $P(O \mid C_1) = 1/2$: he chooses door 3, and it has a goat;
• $P(O \mid C_2) = 1/2$: he chooses door 3, and it has a goat;
• $P(O \mid C_3) = 0$: door 3 has the car.

The two likelihoods are equal, so the posterior is $1/2$ for each closed door: switching does not help. The observation (a goat behind door 3) is the same as before, but it means something different, because the process that produced it is different.

3. Simpson's paradox. A real study compared two treatments for kidney stones (Charig et al., 1986). Treatment A was better for small stones (93.1% vs 86.7% success) and for large stones (73.0% vs 68.8%). But pooled over all patients, B looked better: 78.0% vs 82.6%.

	small stones	large stones	all patients
A	81 / 87 = 93.1%	192 / 263 = 73.0%	273 / 350 = 78.0%
B	234 / 270 = 86.7%	55 / 80 = 68.8%	289 / 350 = 82.6%

There is no error in the arithmetic. By the law of total probability, a pooled rate is a weighted average of the group rates: $P(\text{success} \mid A) = P(\text{succ} \mid A, \text{small})\,P(\text{small} \mid A) + P(\text{succ} \mid A, \text{large})\,P(\text{large} \mid A)$. A treated mostly large stones (263 of 350), which are harder; B treated mostly small ones (270 of 350). So the weights differ, and the ranking can flip.

Interactive figure: loading…

Which answer is right, "A is better" or "B is better"? Probability alone cannot say; it depends on how the data were produced. Here stone size affects both the choice of treatment and the outcome, so the within-group comparison is the fair one. The causal inference track explains when to pool and when to split.

4. Selection bias: conditioning on a common effect. The collider example above is the general pattern. If you look only at cases selected by a common effect of two causes (patients in a hospital, accepted papers, hired people, alarms), you create a dependence between causes that are independent in the whole population. The causal graphs lesson shows it as a scatter plot (Berkson's paradox).

Answers to section 1. (1) About 10%: the exact value is $90/981 \approx 0.092$. (2) Switch: switching wins with probability $2/3$. (3) About 50%: the exact value is $0.507$.

4. Worked example #

Setup. A bank's fraud detector checks card transactions. Let $H$ = "the transaction is fraud", with prior $P(H) = 0.001$ (1 in 1,000). The detector flags (event $E$) with $P(E \mid H) = 0.99$ (recall) and $P(E \mid H^c) = 0.02$ (false positive rate; the specificity is 0.98). A transaction is flagged. How likely is it to be fraud?

(a) With counts. Out of 100,000 transactions:

1. $0.001 \cdot 100{,}000 = 100$ are fraud, and $99{,}900$ are legitimate.
2. The detector flags $0.99 \cdot 100 = 99$ of the frauds.
3. It flags $0.02 \cdot 99{,}900 = 1{,}998$ of the legitimate transactions.
4. So $99 + 1{,}998 = 2{,}097$ transactions are flagged, and

(b) With Bayes' rule. The evidence is $P(E) = 0.99 \cdot 0.001 + 0.02 \cdot 0.999 = 0.00099 + 0.01998 = 0.02097$. Then $P(H \mid E) = 0.00099 / 0.02097 \approx 0.0472$. The precision of the detector is under 5%, although its recall is 99%.

(c) A second detector. It has the same rates and is conditionally independent of the first given the true state (it uses different data). Use the odds form:

1. prior odds $= 0.001/0.999 = 1/999$;
2. likelihood ratio of one flag $= 0.99/0.02 = 49.5$;
3. after one flag: odds $= 49.5/999 \approx 0.0495$, so $P = 0.0495/1.0495 \approx 0.0472$ (as in (a));
4. after two flags: odds $= 49.5^2/999 = 2{,}450.25/999 \approx 2.453$, so $P(H \mid E_1, E_2) = 2.453/3.453 \approx 0.710$.

(d) Check the assumption. If both detectors read the same features, they are not conditionally independent: a false alarm of one makes a false alarm of the other likely. In the extreme case of two copies, the second flag has likelihood ratio 1, and the posterior stays at 0.047.

5. In code #

NumPy checks the worked example: exact values first, then a simulation of 2 million transactions. Simulated numbers differ from the exact ones by the Monte Carlo error: about $\pm 0.001$ for the first estimate (about 42,000 flagged transactions) and about $\pm 0.01$ for the second (about 2,800 transactions flagged twice).

import numpy as np
from itertools import product

# (a) one detector: exact value with Bayes' rule
prior, recall, fpr = 0.001, 0.99, 0.02
evidence = recall * prior + fpr * (1 - prior)          # law of total probability
print(round(evidence, 5), round(recall * prior / evidence, 4))   # 0.02097 0.0472

# (b) two conditionally independent detectors: odds form
odds = prior / (1 - prior) * (recall / fpr) ** 2
print(round(odds, 3), round(odds / (1 + odds), 3))     # 2.453 0.71

# (c) simulate many transactions and count
rng = np.random.default_rng(0)
n = 2_000_000
fraud = rng.random(n) < prior                          # true state H
p_flag = np.where(fraud, recall, fpr)                  # P(flag | state)
flag1 = rng.random(n) < p_flag                         # detector 1
flag2 = rng.random(n) < p_flag                         # detector 2: independent GIVEN the state
print(round(fraud[flag1].mean(), 3))                   # 0.048  (exact 0.0472)
print(round(fraud[flag1 & flag2].mean(), 2))           # 0.72   (exact 0.710)
# without conditioning, the flags are NOT independent (common cause)
print(round((flag1 & flag2).mean(), 5), round(flag1.mean() * flag2.mean(), 5))
# 0.00137 0.00044  (exact 0.00138 and 0.00044)

# (d) XOR: pairwise independent, not mutually independent
outcomes = [(x, y, x ^ y) for x, y in product([0, 1], repeat=2)]   # 4 equally likely outcomes
P = lambda event: sum(event(o) for o in outcomes) / 4
for i, j in [(0, 1), (0, 2), (1, 2)]:
    print(P(lambda o: o[i] == 1 and o[j] == 1), P(lambda o: o[i] == 1) * P(lambda o: o[j] == 1))
# 0.25 0.25   (three times: every pair is independent)
print(P(lambda o: o == (1, 1, 1)), 0.5 ** 3)          # 0.0 0.125

The last-but-one line of (c) shows the common-cause effect: the two flags are independent given the state, but not without conditioning, because both depend on whether the transaction is fraud.

6. Where it shows up #

• Precision depends on the base rate. Recall is $P(\text{flag} \mid \text{positive class})$; precision is $P(\text{positive class} \mid \text{flag})$. Bayes' rule links them through the class prior. On rare classes (fraud, faults, rare diseases, extreme events), a classifier with high recall and a small false positive rate can still have low precision. This is one reason to report precision–recall curves, not only ROC curves, on very imbalanced data.
• Naive Bayes classifiers. A naive Bayes spam filter assumes that the words of an e-mail are conditionally independent given the class (spam or not spam). Then the posterior odds are the prior odds times one likelihood ratio per word, exactly the "multiply the likelihood ratios" rule. The assumption is false (words are correlated), so the probabilities are overconfident, but the ranking of e-mails is often good.
• Bayesian inference. $p(\theta \mid \mathcal D) \propto p(\mathcal D \mid \theta)\,p(\theta)$ is Bayes' rule for a parameter $\theta$. With data that are independent given $\theta$, the likelihood is a product over the data points (lessons 08 and 10). Thompson sampling for bandits samples from such a posterior.
• A/B tests and false discoveries. Suppose 10% of tested ideas really work, a test detects a real effect with probability 0.8 (its power), and a useless idea passes with probability 0.05. Then $P(\text{real} \mid \text{significant}) = 0.08/(0.08 + 0.045) = 0.64$: 36% of the significant results are false discoveries (lessons 11–12, and the interview lesson on A/B testing).
• Graphical models. A Bayesian network, a hidden Markov model and the forward (noising) process of a diffusion model are all lists of conditional independences. They make a joint probability a product of small factors, by the chain rule.
• ML for science. Detecting rare events (earthquakes in continuous seismic records, extreme weather, faults in sensor networks) is a base-rate problem. Data assimilation is Bayes' rule: the forecast is the prior and the observation gives the likelihood (data assimilation lesson). Sensor fusion may multiply likelihoods only when the sensors are conditionally independent given the true state. Members of a weather ensemble share initial conditions and model physics, a common cause, so their errors are dependent (lesson 02 shows how this limits the gain from averaging).

7. Common mistakes #

• Swapping the conditioning. $P(E \mid H) \ne P(H \mid E)$. This is the base-rate fallacy (in court: the prosecutor's fallacy). Always ask "out of how many?".
• Disjoint is not independent. Disjoint events with positive probabilities are strongly dependent: if one happens, the other cannot.
• Pairwise is not mutual. For three or more events, check every subset, also the triple (the XOR example).
• Multiplying likelihood ratios of dependent evidence. Two tests on the same sample, two sensors on the same cable, or two runs of the same model count the same evidence twice.
• Assuming independence survives conditioning. Conditioning on a common effect (an alarm, "the paper was accepted", "the patient is in hospital") creates dependence between independent causes. Conditioning on a common cause can remove a dependence.
• Ignoring how the information was produced. In Monty Hall, the same observed goat means different things for a host who knows and a host who opens at random. Write down the likelihood of what you observed, under each hypothesis.
• Adding probabilities of overlapping events. $P(A \cup B) = P(A) + P(B)$ only for disjoint events; otherwise subtract $P(A \cap B)$, or use the union bound as an upper bound.

8. Check yourself #

1. $P(A) = 0.5$, $P(B) = 0.4$, $P(A \cup B) = 0.7$. Find $P(A \cap B)$. Are $A$ and $B$ independent?
   Answer

   Inclusion–exclusion: $P(A \cap B) = 0.5 + 0.4 - 0.7 = 0.2 = 0.5 \cdot 0.4$. Yes, they are independent.
2. Prove: if $A$ and $B$ are independent, then $A$ and $B^c$ are independent.
   Answer

   $A$ is the disjoint union of $A \cap B$ and $A \cap B^c$. So $P(A \cap B^c) = P(A) - P(A \cap B) = P(A) - P(A)P(B) = P(A)(1 - P(B)) = P(A)P(B^c)$.
3. A family has two children, and at least one is a girl. What is the probability that both are girls? What if you learn that the older child is a girl? (Assume boys and girls are equally likely and independent.)
   Answer

   The four equally likely outcomes (older, younger) are GG, GB, BG, BB. "At least one girl" keeps GG, GB, BG, so $P = 1/3$. "The older is a girl" keeps GG, GB, so $P = 1/2$. The exact conditioning event matters.
4. You run 20 tests, each with false positive probability 0.05. Bound the probability of at least one false positive. Which level per test guarantees at most 0.05?
   Answer

   The union bound gives $\le 20 \cdot 0.05 = 1$, which says nothing. With level $0.05/20 = 0.0025$ per test the bound is $0.05$ (the Bonferroni correction, lesson 11). For independent tests at level 0.05 the exact value is $1 - 0.95^{20} \approx 0.64$.
5. Two independent servers each fail with probability 0.1. An alarm rings if at least one fails. Compute $P(F_1 \mid \text{alarm})$ and $P(F_1 \mid \text{alarm}, F_2)$.
   Answer

   $P(\text{alarm}) = 1 - 0.9^2 = 0.19$, and $F_1 \subseteq \text{alarm}$, so $P(F_1 \mid \text{alarm}) = 0.1/0.19 \approx 0.526$. Given $F_2$, the alarm is certain, so $P(F_1 \mid \text{alarm}, F_2) = P(F_1 \mid F_2) = 0.1$. $F_1$ and $F_2$ are independent, but dependent given the alarm (explaining away).
6. In the worked example, what is the second detector's likelihood ratio if both detectors always agree (they are copies)?
   Answer

   Given the first flag, the second flag is certain under both hypotheses: $P(E_2 \mid E_1, H) = P(E_2 \mid E_1, H^c) = 1$. So its likelihood ratio is 1, and the posterior stays at 0.047.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Define the conditional probability $P(A \mid B)$.

   Show answer

2. State the law of total probability.

   Show answer

3. State Bayes' rule and name its parts.

   Show answer

4. Write Bayes' rule in odds form.

   Show answer

5. Prevalence 1%, sensitivity 90%, specificity 91%. Compute $P(\text{sick} \mid +)$ with 10,000 people.

   Show answer

6. What is the base-rate fallacy?

   Show answer

7. Give three events that are pairwise but not mutually independent.

   Show answer

8. Why does conditioning on a common effect create dependence ("explaining away")?

   Show answer

9. State the union bound.

   Show answer

10. Why does switching win with probability 2/3 in the Monty Hall game?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "A test is 99% accurate (sensitivity and specificity). The prevalence is 1 in 10,000. You test positive. What is the probability that you are ill?" — About 1%: $0.99 \cdot 10^{-4} / (0.99 \cdot 10^{-4} + 0.01 \cdot 0.9999) \approx 0.0098$. Explain it with counts: of 1,000,000 people, 99 sick positives and about 10,000 healthy positives.
• "Your fraud model has 95% recall. Why can its precision still be low?" — With a tiny base rate, most flags are false: precision depends on the prior through Bayes' rule, not on recall alone.
• "Explain the Monty Hall problem." — Stay wins only if the first pick was right, probability 1/3. Otherwise the host is forced to leave the car closed, so switch wins with 2/3. If the host opens at random, the answer becomes 1/2.
• "Give three events that are pairwise but not mutually independent." — Two fair coins and their XOR.
• "Two sensors both report an anomaly. Can you multiply their likelihood ratios?" — Only if they are conditionally independent given the true state. Shared inputs, shared calibration or a shared cable break it, and multiplying counts the same evidence twice.
• "What is Simpson's paradox?" — A comparison that holds in every group reverses after pooling, because the groups have different weights (case mix) in the two arms. The kidney-stone data are the standard example.

10. Practice #

The notebook for this lesson runs in Colab or in your browser. It has five exercises, each with a check cell and a folded solution:

1. simulate_dice(n, rng): estimate $P(\text{sum} = 7)$ for two dice by simulation, and compare with $6/36$.
2. posterior(prior, sens, spec) and posterior_two_tests(...): Bayes' rule for one and two conditionally independent tests, checked against counts from a simulation.
3. monty_hall(n_games, switch, host, rng): simulate stay and switch, for a host who knows and a host who opens at random.
4. Build the XOR triple and write is_independent to test pairwise and mutual independence.
5. birthday(n): the exact probability of a shared birthday, a simulation, and the smallest $n$ with probability at least 0.5.

11. Watch and read #

The medical test paradox, and redesigning Bayes' rule3Blue1Brown · 21 min

Bayes factors (likelihood ratios) and why "sensitivity" is not the answer a patient needs; this is the odds form of section 3.

Bayes theorem, the geometry of changing beliefs3Blue1Brown · 15 min

Bayes' rule as areas in a square, the same picture as the conditioning figure of section 2.

Lecture 6: Monty Hall, Simpson's Paradox | Statistics 110Harvard University · Joe Blitzstein, Statistics 110 · 49 min

Both classic traps of section 3, solved with conditioning and the law of total probability.

• Joseph Blitzstein and Jessica Hwang, Introduction to Probability, 2nd edition, free PDF: ch. 1 (counting, birthday problem, the axioms) and ch. 2 (conditional probability, Bayes' rule, independence, Monty Hall and Simpson's paradox). The lecture videos of Statistics 110 follow the book; lecture 4 covers conditional probability.
• Allen Downey, Think Bayes 2, ch. 1 "Probability" and ch. 2 "Bayes's Theorem": Bayes tables in Python, including the Monty Hall problem.
• MIT 6.041, Probabilistic Systems Analysis and Applied Probability (OpenCourseWare): lectures 1–3 cover the axioms, conditioning, Bayes' rule and independence.
• Kevin Murphy, Probabilistic Machine Learning: An Introduction, free PDF: §2.2 (probability, conditional independence) and §2.3 (Bayes' rule, with a COVID-19 test example).
• C. R. Charig et al., Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy, BMJ 1986: the kidney-stone data of Simpson's paradox.