Common distributions, the Poisson process and moment generating functions

Before this lesson: lesson 01 (independence, counting with $\binom{n}{k}$) and lesson 02 (PMF, PDF, CDF, expectation, variance). The gamma integral uses integration by parts (calculus, lesson 10). Time: about 75 minutes, plus the notebook.

---

Why this matters #

When you choose a loss or a prior, you choose a distribution. Binary cross-entropy is the negative log-likelihood of a Bernoulli; softmax cross-entropy is the categorical; the squared error is the Gaussian; the Poisson loss is for count targets such as clicks or photon counts. A beta prior on a click rate and a Dirichlet prior on topic proportions (LDA) are the standard Bayesian choices (lesson 10).

The relations between the distributions are how samplers are built: a gamma sampler gives beta and Dirichlet samplers. Queues, server logs, website traffic, radioactive decay and earthquake catalogues are first modelled as Poisson processes. The moment generating function leads to the central limit theorem (lesson 06) and to Chernoff bounds (lesson 07).

1. Predict first #

2. The picture #

A family tree #

Start with one trial with success probability $p$: a Bernoulli variable, 1 for success and 0 for failure. Everything in the discrete half of the family comes from repeating it independently.

• Count the successes in $n$ trials: binomial $\text{Bin}(n, p)$.
• Count the trials up to the first success: geometric $\text{Geom}(p)$.
• Let $n$ be large and $p$ small, with $np = \lambda$ fixed (many chances, each one rare): Poisson $\text{Poisson}(\lambda)$.
• Replace "success or failure" by "one of $K$ classes": categorical.

Now let time be continuous. Events happen at rate $\lambda$ per unit of time: on average $\lambda$ events per unit.

• The waiting time to the first event: exponential $\text{Exp}(\lambda)$, the continuous version of the geometric.
• The waiting time to the $k$-th event: gamma $\text{Gamma}(k, \lambda)$, a sum of $k$ exponential gaps.

Two distributions describe probabilities themselves. A beta variable is a number in $(0, 1)$, for example an unknown click rate. A Dirichlet variable is a probability vector, for example the shares of $K$ classes. Both come from normalising independent gamma variables. Finally, the Gaussian $\mathcal N(\mu, \sigma^2)$ appears when many small independent effects add up (lesson 06).

Interactive figure: loading…

From binomial to Poisson #

A shop's website gets many visitors per hour. Each visitor buys with a small probability. The number of orders in an hour is binomial with a large $n$ and a small $p$. In practice, nobody knows $n$ and $p$ separately; only the average $\lambda = np$ is easy to measure. The Poisson distribution needs only $\lambda$:

Interactive figure: loading…

Events in continuous time #

Now spread the events over time. A Poisson process with rate $\lambda$ is a random set of time points with two properties: the numbers of events in disjoint time windows are independent, and the number in a window of length $t$ is $\text{Poisson}(\lambda t)$. Equivalently, the gaps between consecutive events are independent $\text{Exp}(\lambda)$ waiting times. Section 3 proves that the two descriptions agree.

Interactive figure: loading…

3. Definitions and main results #

The table #

Notation. We use the rate for the exponential and the gamma: $\text{Exp}(\lambda)$ has mean $1/\lambda$, and $\text{Gamma}(k, \beta)$ has mean $k/\beta$ (here $\beta$ is a rate; it has nothing to do with the second parameter $b$ of the beta). $\mathcal N(\mu, \sigma^2)$ is given by its variance. The gamma function is $\Gamma(a) = \int_0^\infty x^{a-1} e^{-x}\,dx$ for $a > 0$. Integration by parts gives $\Gamma(a + 1) = a\,\Gamma(a)$, so $\Gamma(n) = (n-1)!$ for integers $n \ge 1$. The beta function is $B(a, b) = \Gamma(a)\Gamma(b)/\Gamma(a + b)$.

Distribution	Support	PMF / PDF	Mean	Variance	Models
$\text{Bernoulli}(p)$	$\{0, 1\}$	$p^x (1-p)^{1-x}$	$p$	$p(1-p)$	one yes/no outcome (a click, a correct label)
$\text{Bin}(n, p)$	$\{0, \dots, n\}$	$\binom{n}{x} p^x (1-p)^{n-x}$	$np$	$np(1-p)$	successes in $n$ independent trials
$\text{Geom}(p)$, trials	$\{1, 2, \dots\}$	$(1-p)^{x-1} p$	$1/p$	$(1-p)/p^2$	trials up to and including the first success
$\text{Poisson}(\lambda)$	$\{0, 1, \dots\}$	$e^{-\lambda} \lambda^x / x!$	$\lambda$	$\lambda$	counts of rare independent events in a window
$\text{Cat}(p_1, \dots, p_K)$	$\{1, \dots, K\}$	$p_x$	one-hot: $p$	one-hot: $\operatorname{diag}(p) - pp^\top$	one class label
$\text{Exp}(\lambda)$	$[0, \infty)$	$\lambda e^{-\lambda x}$	$1/\lambda$	$1/\lambda^2$	waiting time to the next event
$\text{Gamma}(k, \beta)$	$(0, \infty)$	$\dfrac{\beta^k x^{k-1} e^{-\beta x}}{\Gamma(k)}$	$k/\beta$	$k/\beta^2$	waiting time to the $k$-th event; positive, skewed amounts
$\text{Beta}(a, b)$	$(0, 1)$	$\dfrac{x^{a-1}(1-x)^{b-1}}{B(a, b)}$	$\dfrac{a}{a+b}$	$\dfrac{ab}{(a+b)^2(a+b+1)}$	an unknown probability or proportion
$\mathcal N(\mu, \sigma^2)$	$\mathbb{R}$	$\dfrac{1}{\sqrt{2\pi\sigma^2}} e^{-(x-\mu)^2/(2\sigma^2)}$	$\mu$	$\sigma^2$	a sum of many small effects; measurement noise
$\text{Dir}(\alpha_1, \dots, \alpha_K)$	simplex	$\dfrac{\Gamma(\alpha_0)}{\prod_i \Gamma(\alpha_i)} \prod_i x_i^{\alpha_i - 1}$	$\alpha_i / \alpha_0$	$\dfrac{\alpha_i(\alpha_0 - \alpha_i)}{\alpha_0^2(\alpha_0 + 1)}$	a probability vector (class shares, mixture weights)

All parameters are positive, and $p \in (0, 1)$. For the Dirichlet, $\alpha_0 = \sum_i \alpha_i$, and the simplex is the set of vectors with $x_i > 0$ and $\sum_i x_i = 1$. For the categorical, the mean and covariance are those of the one-hot vector $e_X$ (a 1 in position $X$, 0 elsewhere).

Special cases. $\text{Exp}(\lambda) = \text{Gamma}(1, \lambda)$. $\text{Bernoulli}(p) = \text{Bin}(1, p)$. $\text{Beta}(1, 1)$ is the uniform distribution on $(0, 1)$. $\text{Beta}(a, b)$ is the Dirichlet with $K = 2$ (the first component). Each mean and variance in the table follows from the definitions of lesson 02; the MGF below gives most of them in two lines.

Two Dirichlet facts. The components compete for a fixed total of 1, so they are negatively correlated: $\operatorname{Cov}(X_i, X_j) = -\alpha_i\alpha_j / (\alpha_0^2(\alpha_0 + 1))$ for $i \ne j$. A large $\alpha_0$ means little spread around the mean $\alpha/\alpha_0$.

Relation 1: from binomial to Poisson #

Theorem. Fix $\lambda > 0$ and $k \ge 0$. Then $P(\text{Bin}(n, \lambda/n) = k) \to e^{-\lambda}\lambda^k/k!$ as $n \to \infty$.

Proof.

1. Write the PMF with $p = \lambda/n$ and regroup the factors:

2. The first factor is a product of $k$ terms $(n - j)/n$, and each one tends to 1 ($k$ is fixed).
3. $(1 - \lambda/n)^n \to e^{-\lambda}$ (the definition of $e^x$ as a limit), and $(1 - \lambda/n)^{-k} \to 1$.
4. So the product tends to $1 \cdot \frac{\lambda^k}{k!} \cdot e^{-\lambda} \cdot 1$. $\blacksquare$

How good is it for a finite $n$? The total variation distance between two distributions $P$ and $Q$ on the integers is $d_{TV} = \frac12 \sum_k |P(k) - Q(k)|$. It is the largest difference $|P(A) - Q(A)|$ over all events $A$. Le Cam's inequality says that a sum of independent $\text{Bernoulli}(p_i)$ variables is within total variation distance $\sum_i p_i^2$ of $\text{Poisson}(\sum_i p_i)$. For $\text{Bin}(n, p)$, the bound is $np^2$. It is small when $p$ is small, even if $np$ is not. The trials may even have different $p_i$.

Why the bound is $\sum_i p_i^2$: coupling

Le Cam's inequality is proved by a coupling: build the Bernoulli $B_i$ and a $\text{Poisson}(p_i)$ variable $Q_i$ on the same probability space so that they are equal most of the time. One can do it with $P(B_i \ne Q_i) \le p_i^2$. Then $\sum_i Q_i \sim \text{Poisson}(\sum_i p_i)$ (sums of independent Poissons are Poisson, below), and the two sums differ only if some pair differs. By the union bound of lesson 01, $P(\sum B_i \ne \sum Q_i) \le \sum_i p_i^2$, and $d_{TV}$ is at most this probability.

Relation 2: memorylessness #

Theorem. If $X \sim \text{Exp}(\lambda)$, then for all $s, t \ge 0$,

Proof.

1. The survival function is $P(X > x) = \int_x^\infty \lambda e^{-\lambda u}\,du = e^{-\lambda x}$ for $x \ge 0$.
2. The event $\{X > s + t\}$ is a subset of $\{X > s\}$, so their intersection is $\{X > s + t\}$.
3. By the definition of conditional probability, $P(X > s + t \mid X > s) = e^{-\lambda(s+t)} / e^{-\lambda s} = e^{-\lambda t} = P(X > t)$. $\blacksquare$

In words: if you have already waited $s$, the remaining wait has the same distribution as a fresh wait. For a lifetime, it means the item does not age. The geometric distribution has the discrete version: $P(X > n) = (1-p)^n$ (the first $n$ trials fail), so $P(X > m + n \mid X > m) = (1-p)^n$.

These are the only memoryless distributions on $(0, \infty)$ and on $\{1, 2, \dots\}$. Memorylessness says that the survival function $G(x) = P(X > x)$ satisfies $G(s + t) = G(s)\,G(t)$. The only non-increasing solutions with $0 < G < 1$ are $G(t) = e^{-\lambda t}$ on $(0, \infty)$ and $G(n) = (1-p)^n$ on the integers.

Interactive figure: loading…

The minimum of exponentials. Let $X_1, \dots, X_m$ be independent, $X_i \sim \text{Exp}(\lambda_i)$. The minimum is larger than $t$ only if every $X_i$ is, so $P(\min_i X_i > t) = \prod_i e^{-\lambda_i t} = e^{-(\sum_i \lambda_i) t}$. Hence

Competing exponential clocks: the first one rings at the total rate, and clock $i$ wins with a probability proportional to its rate. (The second formula is $\int_0^\infty \lambda_1 e^{-\lambda_1 t}\, e^{-(\sum_{i \ge 2} \lambda_i) t}\,dt$.)

Moment generating functions #

Definition. The moment generating function (MGF) of $X$ is $M_X(t) = \mathbb{E}[e^{tX}]$. We use it when it is finite for all $t$ in an open interval $(-h, h)$ around 0, with $h > 0$.

Theorem (three MGF facts). Assume $M_X$ is finite on $(-h, h)$.

• (a) Moments: $\mathbb{E}[X^n] = M_X^{(n)}(0)$, the $n$-th derivative at 0.
• (b) Sums: if $X$ and $Y$ are independent, $M_{X+Y}(t) = M_X(t)\,M_Y(t)$.
• (c) Uniqueness: if $M_X = M_Y$ on some interval $(-h, h)$, then $X$ and $Y$ have the same distribution.

Proof of (a) and (b). (c) is a deep result; we use it without proof.

1. (a) Expand $e^{tX} = \sum_{n \ge 0} t^n X^n / n!$. When $M_X$ is finite near 0, you may take the expectation term by term: $M_X(t) = \sum_{n \ge 0} \mathbb{E}[X^n]\, t^n / n!$.
2. A power series is the Taylor series of its sum, so the coefficient of $t^n/n!$ is $M_X^{(n)}(0)$. Hence $\mathbb{E}[X^n] = M_X^{(n)}(0)$.
3. (b) $e^{t(X+Y)} = e^{tX} e^{tY}$. For independent $X$ and $Y$, the mean of a product of a function of $X$ and a function of $Y$ is the product of the means (lesson 04). $\blacksquare$

It is often easier to use the cumulant generating function $K_X(t) = \log M_X(t)$. Since $M(0) = 1$, differentiating twice gives $K_X'(0) = \mathbb{E}[X]$ and $K_X''(0) = \operatorname{Var}(X)$.

Distribution	MGF $M(t)$	Valid for
$\text{Bernoulli}(p)$	$1 - p + pe^t$	all $t$
$\text{Bin}(n, p)$	$(1 - p + pe^t)^n$	all $t$
$\text{Poisson}(\lambda)$	$\exp(\lambda(e^t - 1))$	all $t$
$\text{Geom}(p)$, trials	$pe^t / (1 - (1-p)e^t)$	$t < -\ln(1-p)$
$\text{Exp}(\lambda)$	$\lambda / (\lambda - t)$	$t < \lambda$
$\text{Gamma}(k, \beta)$	$(\beta / (\beta - t))^k$	$t < \beta$
$\mathcal N(\mu, \sigma^2)$	$\exp(\mu t + \sigma^2 t^2 / 2)$	all $t$

Example (moments of the gamma). $K(t) = -k \log(1 - t/\beta)$, so $K'(t) = k/(\beta - t)$ and $K''(t) = k/(\beta - t)^2$. At $t = 0$: mean $k/\beta$ and variance $k/\beta^2$. For the Poisson, $K(t) = \lambda(e^t - 1)$, so $K'(0) = K''(0) = \lambda$: the mean equals the variance.

Example (sums). Products of MGFs give at once, for independent summands:

• $\text{Bin}(n, p)$ is a sum of $n$ independent Bernoullis: $(1 - p + pe^t)^n$;
• $\text{Poisson}(\lambda_1) + \text{Poisson}(\lambda_2) = \text{Poisson}(\lambda_1 + \lambda_2)$;
• $\mathcal N(\mu_1, \sigma_1^2) + \mathcal N(\mu_2, \sigma_2^2) = \mathcal N(\mu_1 + \mu_2, \sigma_1^2 + \sigma_2^2)$;
• $\text{Gamma}(a, \beta) + \text{Gamma}(b, \beta) = \text{Gamma}(a + b, \beta)$, with the same rate $\beta$.

Not every distribution has an MGF

The Cauchy, the Student-$t$ and the log-normal distributions have $\mathbb{E}[e^{tX}] = \infty$ for every $t > 0$: their right tails are too heavy. The log-normal even has all moments $\mathbb{E}[X^n]$, but they do not determine its distribution. The characteristic function $\varphi_X(t) = \mathbb{E}[e^{itX}]$ always exists, because $|e^{itX}| = 1$, and it has the same product rule for sums and the same uniqueness property. Proofs of the central limit theorem often use it for this reason.

Relation 3: a sum of exponentials is gamma #

Theorem. If $X_1, \dots, X_n$ are i.i.d. $\text{Exp}(\lambda)$, then $S = X_1 + \dots + X_n \sim \text{Gamma}(n, \lambda)$.

Proof.

1. For $t < \lambda$: $M_{X_i}(t) = \int_0^\infty e^{tx} \lambda e^{-\lambda x}\,dx = \lambda \int_0^\infty e^{-(\lambda - t)x}\,dx = \dfrac{\lambda}{\lambda - t}$.
2. By independence and fact (b), $M_S(t) = \left(\dfrac{\lambda}{\lambda - t}\right)^n$.
3. The MGF of $\text{Gamma}(n, \lambda)$ is $\displaystyle\int_0^\infty e^{tx} \frac{\lambda^n x^{n-1} e^{-\lambda x}}{\Gamma(n)}\,dx = \frac{\lambda^n}{\Gamma(n)} \cdot \frac{\Gamma(n)}{(\lambda - t)^n}$, using $\int_0^\infty x^{n-1} e^{-cx}\,dx = \Gamma(n)/c^n$ for $c > 0$ (substitute $u = cx$).
4. The two MGFs agree on $(-\lambda, \lambda)$, so by uniqueness (c), $S \sim \text{Gamma}(n, \lambda)$. $\blacksquare$

So the time of the $k$-th event, when gaps are $\text{Exp}(\lambda)$, is $\text{Gamma}(k, \lambda)$, with mean $k/\lambda$. With integer $k$ the gamma is also called the Erlang distribution.

Relation 4: gamma ratios give the beta and the Dirichlet #

Theorem. Let $X \sim \text{Gamma}(a, 1)$ and $Y \sim \text{Gamma}(b, 1)$ be independent. Then

and $T$ and $S$ are independent.

Proof with a two-dimensional change of variables

We use the change of variables for a pair: the density of the new pair is the old density times $|\det J|$, where $J$ is the Jacobian of the inverse map. This is the 2-D version of the rule in lesson 02; lesson 05 treats it fully.

1. The inverse map is $x = st$, $y = s(1 - t)$, for $s > 0$ and $t \in (0, 1)$.
2. $J = \begin{pmatrix} \partial x/\partial s & \partial x/\partial t \\ \partial y/\partial s & \partial y/\partial t \end{pmatrix} = \begin{pmatrix} t & s \\ 1 - t & -s \end{pmatrix}$, so $\det J = -st - s(1 - t) = -s$ and $|\det J| = s$.
3. The joint density of $(X, Y)$ is the product $\frac{x^{a-1}e^{-x}}{\Gamma(a)} \cdot \frac{y^{b-1}e^{-y}}{\Gamma(b)}$ (independence). Substitute and multiply by $s$:

$$p(s, t) = \frac{(st)^{a-1} (s(1-t))^{b-1} e^{-s}}{\Gamma(a)\Gamma(b)}\, s = \underbrace{\frac{s^{a+b-1} e^{-s}}{\Gamma(a+b)}}_{\text{Gamma}(a+b,\,1)} \cdot \underbrace{\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} t^{a-1}(1-t)^{b-1}}_{\text{Beta}(a,\,b)}.$$

4. The joint density is a density of $s$ times a density of $t$. So $S$ and $T$ are independent, with the stated distributions. $\blacksquare$

Both factors integrate to 1, so the proof also shows $B(a, b) = \Gamma(a)\Gamma(b)/\Gamma(a+b)$.

The rate does not matter: with a common rate $\beta$, every gamma is scaled by $1/\beta$, and the scale cancels in $X/(X + Y)$.

Dirichlet. In the same way (with a $K$-dimensional Jacobian): if $G_i \sim \text{Gamma}(\alpha_i, 1)$ are independent and $S = \sum_i G_i$, then

This is how Dirichlet samples are made in practice (NumPy's rng.dirichlet does it).

Interactive figure: loading…

Relation 5: Dirichlet marginals are beta #

Theorem. If $X \sim \text{Dir}(\alpha_1, \dots, \alpha_K)$, then $X_i \sim \text{Beta}(\alpha_i, \alpha_0 - \alpha_i)$.

Proof.

1. Build $X$ from independent gammas: $X_i = G_i / (G_i + R)$, with $R = \sum_{j \ne i} G_j$.
2. $R$ is a sum of independent $\text{Gamma}(\alpha_j, 1)$ variables with the same rate, so $R \sim \text{Gamma}(\alpha_0 - \alpha_i, 1)$ (MGF sums). $R$ is independent of $G_i$.
3. Relation 4 with $a = \alpha_i$ and $b = \alpha_0 - \alpha_i$ gives $X_i \sim \text{Beta}(\alpha_i, \alpha_0 - \alpha_i)$. $\blacksquare$

The beta mean and variance then give the Dirichlet mean $\alpha_i/\alpha_0$ and the variance in the table. The same argument shows that merging components adds their $\alpha$'s: $(X_1 + X_2, X_3, \dots) \sim \text{Dir}(\alpha_1 + \alpha_2, \alpha_3, \dots)$.

The Poisson process #

Definition A (gaps). Let $G_1, G_2, \dots$ be i.i.d. $\text{Exp}(\lambda)$. The event times are $T_k = G_1 + \dots + G_k$, and $N(t)$ is the number of events in $[0, t]$.

Definition B (counts). $N(0) = 0$; the counts in disjoint intervals are independent; and $N(t) - N(s) \sim \text{Poisson}(\lambda(t - s))$ for $s < t$.

Why the two definitions agree (sketch).

1. By relation 3, $T_k \sim \text{Gamma}(k, \lambda)$.
2. There are at least $k$ events by time $t$ exactly when the $k$-th event happens by time $t$: $\{N(t) \ge k\} = \{T_k \le t\}$.
3. Integration by parts ($k - 1$ times) shows that the $\text{Gamma}(k, \lambda)$ CDF is $P(T_k \le t) = 1 - \sum_{j < k} e^{-\lambda t}(\lambda t)^j/j!$. That is $P(N(t) \ge k)$ for $N(t) \sim \text{Poisson}(\lambda t)$.
4. Independence of disjoint intervals comes from memorylessness: at any time $s$, the wait for the next event is again $\text{Exp}(\lambda)$ and does not depend on what happened before $s$.

Thinning. Keep each event independently with probability $q$. Then the kept events form a Poisson process with rate $q\lambda$, the removed events form one with rate $(1 - q)\lambda$, and the two are independent.

Proof (counts in one window of length $t$).

1. Given $N(t) = n$, the kept count $C$ is $\text{Bin}(n, q)$, and the removed count is $R = n - C$.
2. So $P(C = k, R = r) = P(N(t) = k + r)\,\binom{k + r}{k} q^k (1 - q)^r$ $= e^{-\lambda t}\dfrac{(\lambda t)^{k + r}}{(k + r)!} \cdot \dfrac{(k + r)!}{k!\,r!}\, q^k (1 - q)^r$.
3. Split $e^{-\lambda t} = e^{-q\lambda t} e^{-(1-q)\lambda t}$ and $(\lambda t)^{k+r} = (\lambda t)^k (\lambda t)^r$: $P(C = k, R = r) = e^{-q\lambda t}\dfrac{(q\lambda t)^k}{k!} \cdot e^{-(1 - q)\lambda t}\dfrac{((1 - q)\lambda t)^r}{r!}$.
4. The joint PMF is a $\text{Poisson}(q\lambda t)$ PMF times a $\text{Poisson}((1-q)\lambda t)$ PMF, so $C$ and $R$ are independent Poissons. Disjoint windows stay independent, because the coins are independent. $\blacksquare$

Superposition. The union of two independent Poisson processes with rates $\lambda_1$ and $\lambda_2$ is a Poisson process with rate $\lambda_1 + \lambda_2$: a sum of independent Poisson counts is Poisson (MGFs). Each event comes from process 1 with probability $\lambda_1/(\lambda_1 + \lambda_2)$, the probability that its exponential clock rings first.

Conditional uniformity. Given $N(T) = n$, the $n$ event times are distributed like $n$ i.i.d. $U(0, T)$ points, sorted. So, given 4 events in an hour, the number in the first 15 minutes is $\text{Bin}(4, 1/4)$.

The inspection paradox: why your bus is late

Buses arrive as a Poisson process with on average one bus every 10 minutes. You arrive at a random time. By memorylessness, your expected wait is 10 minutes, not 5. And the time since the last bus is also about 10 minutes on average (for a process that has run for a long time). So the gap you land in has mean 20 minutes, twice the average gap. This is not a contradiction: a random arrival time is more likely to fall into a long gap than into a short one. The same length-biased sampling appears when you sample "a random user's session" by picking a random time point.

Answers to section 1. (1) The count is $\text{Poisson}(2)$, so $P(N = 0) = e^{-2} \approx 0.135$. (2) $P(X > 20 \mid X > 10) = P(X > 10) = e^{-10/10} = e^{-1} \approx 0.368$, exactly the same as for a new disk. The exponential model says that the disk does not age; if real disks wear out, the exponential is the wrong model. (3) $\text{Poisson}(2)$, because $np = 1000 \cdot 0.002 = 2$ and $p$ is small (Le Cam's bound $np^2 = 0.004$).

4. Worked example #

A website gets support tickets as a Poisson process with rate $\lambda = 2$ per day.

Counts. $N$ = tickets in one day, $N \sim \text{Poisson}(2)$.

1. $P(N = 0) = e^{-2} \approx 0.1353$.
2. $P(N \ge 3) = 1 - P(N \le 2) = 1 - e^{-2}\big(1 + 2 + \tfrac{2^2}{2}\big) = 1 - 5e^{-2} \approx 0.3233$.

The binomial view. Split the day into 1440 minutes. In each minute a ticket arrives with probability $2/1440$, independently (ignore two tickets in the same minute). Then $N \sim \text{Bin}(1440, 2/1440)$, with $P(N = 0) = (1 - 2/1440)^{1440} \approx 0.13515$ and $P(N \ge 3) \approx 0.32332$. Le Cam's bound is $np^2 = 1440 \cdot (2/1440)^2 = 4/1440 \approx 0.0028$; the true total variation distance is about 0.0003.

Waiting times.

3. The wait for the first ticket is $\text{Exp}(2)$: $P(\text{wait} > 1 \text{ day}) = e^{-2}$. It is the same number as $P(N = 0)$, because it is the same event.
4. After 12 quiet hours, the remaining wait is again $\text{Exp}(2)$, with mean $1/2$ day (memorylessness).
5. The time of the third ticket is $T_3 \sim \text{Gamma}(3, 2)$: mean $3/2 = 1.5$ days, variance $3/2^2 = 0.75$, standard deviation $\approx 0.866$ days.
6. Check of the link: $P(T_3 \le 1) = P(N \ge 3) \approx 0.3233$.

Thinning. Each ticket is urgent with probability $1/4$, independently.

7. Urgent tickets form a Poisson process with rate $2 \cdot 1/4 = 0.5$ per day.
8. $P(\text{no urgent ticket in a week}) = e^{-0.5 \cdot 7} = e^{-3.5} \approx 0.030$.
9. Given 4 tickets in one day, the expected number in its first 6 hours is $4 \cdot 6/24 = 1$ (conditional uniformity: $\text{Bin}(4, 1/4)$).

A Dirichlet. A classifier is trained on data whose class shares are uncertain, modelled as $X \sim \text{Dir}(2, 3, 5)$, so $\alpha_0 = 10$.

10. Mean $(0.2, 0.3, 0.5)$.
11. $X_3 \sim \text{Beta}(5, 5)$, with variance $\dfrac{5 \cdot 5}{10^2 \cdot 11} = \dfrac{25}{1100} \approx 0.0227$.
12. $\operatorname{Cov}(X_1, X_2) = -\dfrac{2 \cdot 3}{10^2 \cdot 11} = -\dfrac{6}{1100} \approx -0.0055$.

5. In code #

SciPy checks the numbers of the worked example and the relations. Mind SciPy's parametrisation: scale is $1/\text{rate}$, and the first positional argument of stats.expon is a shift (loc), not a rate.

import numpy as np
from scipy import stats, integrate

# worked example: 2 tickets per day
N = stats.poisson(2)
print(round(N.pmf(0), 4), round(N.sf(2), 4))      # 0.1353 0.3233
print(round(stats.binom(1440, 2 / 1440).sf(2), 4))   # 0.3233 (1440 minutes)
T3 = stats.gamma(a=3, scale=1 / 2)                # shape 3, SCALE = 1/rate
print(T3.mean(), T3.var(), round(T3.cdf(1), 4))   # 1.5 0.75 0.3233 = P(N >= 3)
print(stats.expon(2).mean(), stats.expon(scale=1 / 2).mean())
# 3.0 0.5  (the first one is loc=2, a shift!)
print(round(np.exp(-0.5 * 7), 4))                 # 0.0302: no urgent ticket in a week

# Binomial(n, 2/n) -> Poisson(2): total variation distance and Le Cam's bound
for n in (10, 100, 1000):
    k = np.arange(n + 1)
    tv = 0.5 * (np.abs(stats.binom(n, 2 / n).pmf(k) - N.pmf(k)).sum() + N.sf(n))
    print(n, f"{tv:.2g}", f"{n * (2 / n) ** 2:.2g}")
# 10 0.052 0.4 | 100 0.0046 0.04 | 1000 0.00045 0.004

# Dirichlet(2, 3, 5): moments, and the marginal X_3 ~ Beta(5, 5)
alpha = np.array([2.0, 3.0, 5.0])
D = stats.dirichlet(alpha)
print(D.mean(), np.round(D.var(), 4), round(stats.beta(5, 5).var(), 4))
# [0.2 0.3 0.5] [0.0145 0.0191 0.0227] 0.0227
rng = np.random.default_rng(0)
G = rng.gamma(shape=alpha, size=(50_000, 3))      # independent Gamma(alpha_i, 1)
X = G / G.sum(axis=1, keepdims=True)              # normalise: Dirichlet samples
print(stats.kstest(X[:, 2], stats.beta(5, 5).cdf).pvalue > 0.01,
      round(np.cov(X[:, 0], X[:, 1])[0, 1], 4))   # True -0.0055

# MGF of Gamma(3, rate 2) by numerical integration: e^{tx} times the
# density 2^3 x^2 e^{-2x} / Gamma(3), written as one exponential (no overflow)
f = lambda x, t: 2**3 / 2 * x**2 * np.exp((t - 2) * x)
M = lambda t: integrate.quad(f, 0, np.inf, args=(t,))[0]
print(round(M(0.5), 6), round((2 / (2 - 0.5)) ** 3, 6))   # 2.37037 2.37037

# moments by finite differences
h = 1e-3
m1 = (M(h) - M(-h)) / (2 * h)
m2 = (M(h) - 2 * M(0) + M(-h)) / h**2
print(round(m1, 4), round(m2 - m1**2, 4))         # 1.5 0.75 (k/beta, k/beta^2)

6. Where it shows up #

• Losses are log-likelihoods. Bernoulli gives binary cross-entropy, categorical gives softmax cross-entropy (information theory, lesson 4), Gaussian with fixed variance gives the squared error, and Poisson gives the Poisson loss for count targets (torch.nn.PoissonNLLLoss). Lesson 08 derives these from maximum likelihood.
• Priors. Beta for a probability (a click rate, the success rate of an arm in Thompson sampling), Dirichlet for mixture weights and topic proportions (LDA), gamma for a precision $1/\sigma^2$. Lesson 10 shows why these are conjugate.
• Traffic and queues. Requests to a web server, calls to a help desk and jobs on a cluster are first modelled as Poisson processes. Thinning gives the rate of a subset (errors, urgent calls); superposition merges sources.
• Overdispersion. Real counts (orders per day, earthquakes per month) often have variance much larger than the mean, because the rate itself varies. A gamma mixture of Poissons, the negative binomial, is the standard fix. Earthquake catalogues cluster in time (aftershocks) and need self-exciting models.
• Simulation. The Gillespie algorithm for chemical reactions uses the minimum of exponentials: the next reaction comes after an $\text{Exp}(\text{total rate})$ wait, and it is reaction $i$ with probability $\lambda_i / \sum_j \lambda_j$.
• Diffusion models. A sum of independent Gaussians is Gaussian (MGF product). This is why the forward noising process has the closed form $q(x_t \mid x_0) = \mathcal N(\sqrt{\bar\alpha_t}\, x_0, (1 - \bar\alpha_t) I)$ (generative models, lesson 7).

7. Common mistakes #

• Rate versus scale. NumPy and SciPy use the scale $1/\lambda$: rng.exponential(scale), rng.gamma(shape, scale), stats.expon(scale=1/lam). PyTorch's Exponential and Gamma use the rate. stats.expon(2) is a shift by 2, not a rate of 2.
• Two geometric conventions. "Trials up to the first success" (support $1, 2, \dots$, mean $1/p$; SciPy and NumPy) or "failures before the first success" (support $0, 1, \dots$, mean $(1-p)/p$). The variance $(1-p)/p^2$ is the same. Always check which one a book or a library uses.
• Adding gammas with different rates. $\text{Gamma}(a, \beta_1) + \text{Gamma}(b, \beta_2)$ is not gamma when $\beta_1 \ne \beta_2$.
• The Poisson parameter is a mean, not a probability. For 5,000 trials with $p = 0.0004$, use $\text{Poisson}(2)$, not $\text{Poisson}(0.0004)$.
• Forcing a Poisson on overdispersed counts. Compare the sample variance with the sample mean; for a Poisson their ratio is close to 1.
• Assuming every variable has an MGF. The Cauchy, Student-$t$ and log-normal do not.
• Treating Dirichlet components as independent betas. Each one is beta, but they sum to 1 and are negatively correlated.
• Reading memorylessness as "nothing ever changes". It is a property of the waiting times: the past waiting gives no information about the remaining wait. Items that wear out are not memoryless.

8. Check yourself #

1. $X \sim \text{Bin}(20, 0.1)$. Give $\mathbb{E}[X]$, $\operatorname{Var}(X)$, $P(X = 0)$ and the Poisson approximation of $P(X = 0)$. Is the approximation as good as in section 4?
   Answer

   Mean $20 \cdot 0.1 = 2$, variance $2 \cdot 0.9 = 1.8$, $P(0) = 0.9^{20} \approx 0.1216$. $\text{Poisson}(2)$ gives $e^{-2} \approx 0.1353$. It is worse: $p = 0.1$ is not small, and Le Cam's bound $np^2 = 0.2$ is much larger than $0.0028$ (the true total variation distance is about 0.024).
2. Show that the geometric distribution (trials) is memoryless.
   Answer

   $P(X > n) = P(\text{the first } n \text{ trials fail}) = (1-p)^n$. So $P(X > m + n \mid X > m) = (1-p)^{m+n}/(1-p)^m = (1-p)^n = P(X > n)$.
3. Use MGFs to show that the sum of independent $\text{Poisson}(\lambda_1)$ and $\text{Poisson}(\lambda_2)$ variables is $\text{Poisson}(\lambda_1 + \lambda_2)$.
   Answer

   $M(t) = e^{\lambda_1(e^t - 1)}\, e^{\lambda_2(e^t - 1)} = e^{(\lambda_1 + \lambda_2)(e^t - 1)}$. This is the MGF of $\text{Poisson}(\lambda_1 + \lambda_2)$, finite for all $t$; uniqueness finishes the proof.
4. Get the mean and the variance of $\text{Exp}(\lambda)$ from its MGF.
   Answer

   $K(t) = \log\lambda - \log(\lambda - t)$, so $K'(t) = 1/(\lambda - t)$ and $K''(t) = 1/(\lambda - t)^2$. At $t = 0$: mean $1/\lambda$, variance $1/\lambda^2$. (Or $M'(0) = 1/\lambda$ and $M''(0) = 2/\lambda^2$, so the variance is $2/\lambda^2 - 1/\lambda^2$.)
5. $X \sim \text{Dir}(2, 3, 5)$. What is the distribution of $X_1 + X_2$?
   Answer

   Merging components adds the $\alpha$'s: $(X_1 + X_2, X_3) \sim \text{Dir}(5, 5)$, so $X_1 + X_2 \sim \text{Beta}(5, 5)$, with mean 0.5. (It is also $1 - X_3$, and $X_3 \sim \text{Beta}(5, 5)$.)
6. You have a sampler for $\text{Gamma}(a, 1)$ only. How do you sample $\text{Beta}(3, 7)$, and what are its mean and variance?
   Answer

   $X/(X + Y)$ with independent $X \sim \text{Gamma}(3, 1)$ and $Y \sim \text{Gamma}(7, 1)$. Mean $3/10 = 0.3$, variance $\dfrac{3 \cdot 7}{10^2 \cdot 11} = 21/1100 \approx 0.0191$.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Give the Poisson PMF, its mean and its variance.

   Show answer

2. When is $\text{Bin}(n, p)$ close to $\text{Poisson}(np)$, and how close?

   Show answer

3. State memorylessness of the exponential distribution.

   Show answer

4. What is the distribution of the minimum of independent $\text{Exp}(\lambda_1)$ and $\text{Exp}(\lambda_2)$?

   Show answer

5. How does the MGF of a sum of independent variables relate to their MGFs?

   Show answer

6. What is the distribution of a sum of $k$ i.i.d. $\text{Exp}(\lambda)$ variables?

   Show answer

7. Give the mean of $\text{Beta}(a, b)$, and compute it for $\text{Beta}(2, 8)$.

   Show answer

8. How do you sample a Dirichlet from gamma variables?

   Show answer

9. Give the two equivalent definitions of a Poisson process with rate $\lambda$.

   Show answer

10. What is thinning of a Poisson process?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Calls arrive as a Poisson process with 10 per hour. What is the probability of no call in 15 minutes?" — The count in 15 minutes is $\text{Poisson}(2.5)$, so $e^{-2.5} \approx 0.082$.
• "What is the distribution of the minimum of two independent exponentials?" — $\text{Exp}(\lambda_1 + \lambda_2)$; $X_1$ is the minimum with probability $\lambda_1/(\lambda_1 + \lambda_2)$.
• "Expected number of die rolls until the first 6?" — $\text{Geom}(1/6)$ (trials): mean 6, variance $(5/6)/(1/36) = 30$.
• "Your count data have variance much larger than the mean. What do you do?" — The Poisson model is wrong (overdispersion). Use a negative binomial, or model the extra variation (covariates, random effects).
• "Buses come as a Poisson process every 10 minutes on average. How long do you wait?" — 10 minutes on average, by memorylessness. The gap you land in has mean 20 minutes (inspection paradox).
• "How do you sample a Beta or a Dirichlet if you only have a gamma sampler?" — Normalise independent gammas with the same rate.

10. Practice #

The notebook for this lesson runs in Colab or in your browser. It has five exercises, each with a check cell and a folded solution:

1. sample_gamma_int(k, lam, n, rng) as a sum of exponentials, and sample_dirichlet(alpha, n, rng) by normalising gammas; check the means and variances.
2. poisson_process(lam, T, rng): event times on $[0, T]$ from running sums of exponential gaps; a chi-square test of the counts per window.
3. tv_binom_poisson(n, lam): the total variation distance between $\text{Bin}(n, \lambda/n)$ and $\text{Poisson}(\lambda)$, compared with Le Cam's bound.
4. conditional_survival(samples, s, t): estimate $P(X > s + t \mid X > s)$ by simulation, for exponential and gamma lifetimes.
5. mgf_moments(M, h): the mean and the variance from an MGF by finite differences, checked on the gamma and the Poisson.

11. Watch and read #

Lecture 11: The Poisson distribution | Statistics 110Harvard University · Joe Blitzstein, Statistics 110 · 43 min

The Poisson distribution, its link to the binomial and the "Poisson paradigm" for many rare, weakly dependent events.

Lecture 16: Exponential Distribution | Statistics 110Harvard University · Joe Blitzstein, Statistics 110 · 18 min

Memorylessness and why the exponential is the only continuous memoryless distribution (relation 2).

15. Poisson Process IIMIT OpenCourseWare · John Tsitsiklis, MIT 6.041 · 49 min

Merging and splitting Poisson processes (superposition and thinning) and the inspection paradox.

• Joseph Blitzstein and Jessica Hwang, Introduction to Probability, 2nd edition, free PDF: §3.3 (Bernoulli, binomial), §4.3 (geometric), §4.7–4.8 (Poisson and the binomial limit), §5.4–5.5 (normal, exponential), §6.4–6.6 (MGFs and sums), §8.3–8.5 (beta, gamma and their connections), ch. 13 (Poisson processes).
• Kevin Murphy, Probabilistic Machine Learning: An Introduction, free PDF: §2.4–2.7 (common univariate distributions and their use in ML).
• The Statistics 110 course page has all lecture videos, including lecture 17 (moment generating functions) and lecture 23 (beta distribution).
• Art Owen, Monte Carlo theory, methods and examples, ch. 4: how gamma, beta and Dirichlet variables are sampled in practice.