Breadth: classic ML, evaluation and A/B testing

Before this lesson: a first course in machine learning and in statistics (hypothesis tests, the normal distribution). For the code of AUC and logistic regression, see lesson 06. Depth on these topics is in a first ML course and in the Probability and Statistics track; canary releases and peeking in production are in MLOps lesson 06. Time: about 85 minutes, plus the notebook.

Why this matters #

Breadth questions come in phone screens, in "ML knowledge" rounds and in data-science loops. They test whether you give a short, correct, structured answer quickly. Common failures are an answer that runs for five minutes, a wrong basic definition (precision vs recall), or no number at all.

A good breadth answer has four parts and takes 30 seconds to 2 minutes:

1. one sentence that defines the thing;
2. the formula or the mechanism;
3. one trade-off, or when to use it;
4. one number or one example.

The same tools matter in ML for science. Evaluation design decides whether a weather or medical model is tested honestly (spatial, temporal and patient-level leakage). Calibration decides whether a probabilistic forecast can be trusted. And "is model B better than model A?" is a hypothesis test on a fixed test set.

1. Predict first #

2. The picture #

2.1 Error has three sources #

Fix one input $x$. The prediction error there comes from three sources:

1. noise in the label, which no model can remove;
2. bias: the average model (over many training sets) is offset from the truth;
3. variance: the fitted model moves when you draw a new training set.

A constant predictor has large bias and almost no variance. A 1-nearest-neighbour predictor has small bias and large variance. Most choices in classic ML (model size, regularization, the number of neighbours, ensembles) move error between bias and variance.

2.2 L1 gives zeros, L2 does not #

Regularization adds a penalty on the weights to the loss. With an L2 (ridge) penalty $\lambda\|w\|_2^2$ the weights shrink towards zero but stay nonzero. With an L1 (lasso) penalty $\lambda\|w\|_1$ many weights become exactly zero. The picture: minimize the loss inside a ball. The L1 ball is a diamond with corners on the axes, and the smallest loss contour often touches it at a corner, where one coordinate is 0.

2.3 A classifier is a ranking plus a threshold #

A classifier gives each example a score. Three different questions follow, and each has its own metrics:

• Does the score rank positives above negatives? (ROC-AUC, average precision.)
• At the chosen threshold, how many alarms are right, and how many positives are found? (precision, recall, $F_1$.)
• Is the score a probability: of the cases scored 0.8, are 80% positive? (calibration, log-loss, Brier score.)

2.4 An A/B test is two noisy coins #

In an A/B test, each user converts with some probability $p$. The rate of one arm with $n$ users has standard error $\sqrt{p(1-p)/n}$. The difference of two independent arms has standard error $\sqrt{2p(1-p)/n}$. To see a true difference $\delta$ reliably, $\delta$ must be about 2.8 standard errors: 1.96 for the test at $\alpha = 0.05$, plus 0.84 for power 0.8. Solve for $n$:

Small effects need many users, because $n$ grows like $1/\delta^2$: half the effect needs four times the users.

3. Definitions and main results #

3.1 Bias and variance #

Setting. Let $y = f(x) + \varepsilon$ with $\mathbb{E}[\varepsilon] = 0$ and $\operatorname{Var}(\varepsilon) = \sigma^2$. A model $\hat f_D$ is fitted on a random training set $D$. The new noise $\varepsilon$ at the test point is independent of $D$. Let $\bar f(x) = \mathbb{E}_D[\hat f_D(x)]$ be the average fit over training sets.

Theorem (bias–variance decomposition, squared loss). At a fixed input $x$,

Why this is true, in one line. We split the error into three pieces that are "uncorrelated", so the squares add like the sides of a right triangle.

Proof.

1. Add and subtract $f(x)$ and $\bar f(x)$: $y - \hat f_D(x) = \underbrace{\varepsilon}_{a} + \underbrace{(f(x) - \bar f(x))}_{b} + \underbrace{(\bar f(x) - \hat f_D(x))}_{c}$.
2. Square: $(a + b + c)^2 = a^2 + b^2 + c^2 + 2ab + 2ac + 2bc$. Take the expectation over $D$ and $\varepsilon$.
3. $\mathbb{E}[ab] = b\,\mathbb{E}[\varepsilon] = 0$, because $b$ is a constant.
4. $\mathbb{E}[ac] = \mathbb{E}[\varepsilon]\,\mathbb{E}[c] = 0$, because $\varepsilon$ is independent of $D$.
5. $\mathbb{E}[bc] = b\,\mathbb{E}_D[\bar f(x) - \hat f_D(x)] = b\,(\bar f(x) - \bar f(x)) = 0$.
6. What remains is $\mathbb{E}[a^2] + b^2 + \mathbb{E}[c^2] = \sigma^2 + \text{bias}^2 + \text{variance}$. $\blacksquare$

Two small examples.

• $k$-nearest neighbours (regression). Suppose the $k$ neighbours of $x$ have independent labels with noise variance $\sigma^2$. The prediction is the mean of $k$ labels, so its variance is $\sigma^2/k$. Larger $k$ lowers the variance, but uses neighbours farther from $x$, whose $f$ values differ from $f(x)$: the bias grows.
• Constant model $\hat f_D(x) = \bar y$ (the training mean). Its variance is small (about $\operatorname{Var}(y)/n$), but its bias at $x$ is $f(x) - \mathbb{E}[y]$, large wherever $f$ is far from its average.

Diagnosis from learning curves. High bias: the training error is high and the validation error is close to it; use a more flexible model or better features. High variance: the training error is low and the gap to validation is large; get more data, regularize, or average models. A large gap can also mean that training and validation data differ (shift or leakage), so check that too.

3.2 L1 vs L2 regularization #

• Ridge (L2): minimize $\|y - Xw\|^2 + \lambda\|w\|_2^2$. The solution is $w = (X^\top X + \lambda I)^{-1} X^\top y$, and the matrix is invertible for every $\lambda > 0$. Ridge shrinks all weights towards 0 but keeps them nonzero. With correlated features it shares the weight among them.
• Lasso (L1): minimize $\tfrac12\|y - Xw\|^2 + \lambda\|w\|_1$. In the simplest case, an orthonormal design ($X^\top X = I$, least-squares solution $\hat w = X^\top y$), it acts on each coordinate alone:

Every $|\hat w_j| \le \lambda$ becomes exactly 0. Ridge in the same case, with penalty $\tfrac{\lambda}{2}\|w\|^2$, gives $w_j = \hat w_j/(1 + \lambda)$: never exactly 0.

• Why zeros. The slope of $|w|$ near 0 is $\pm 1$: it does not vanish. So a small weight must "pay" $\lambda$ per unit, and if the loss gains less than that, the weight stays at 0. The slope of $w^2$ near 0 is $2w \to 0$, so a tiny nonzero weight is almost free.
• Bayesian (MAP) view. With Gaussian noise of variance $\sigma^2$, a Gaussian prior $w_j \sim \mathcal N(0, \tau^2)$ gives ridge (with $\lambda = \sigma^2/\tau^2$ for the penalty $\tfrac{\lambda}{2}\|w\|^2$). A Laplace prior $p(w_j) \propto e^{-|w_j|/b}$ gives the lasso (with $\lambda = \sigma^2/b$). The Laplace density has a sharp peak at 0, which is where the exact zeros come from.
• In practice. Standardize the features first (the penalty is not scale invariant). Do not penalize the intercept. With a group of correlated features the lasso picks one of them almost at random; the elastic net (penalty $\lambda_1\|w\|_1 + \lambda_2\|w\|_2^2$) keeps the group. Choose $\lambda$ by cross-validation.

3.3 Trees, bagging and boosting #

Decision trees (CART). A tree splits greedily. At each node it picks the feature and the threshold with the largest decrease of weighted impurity. For classification with class fractions $p_k$ in a node:

For regression the impurity is the sum of squared errors. Trees need no feature scaling, handle mixed feature types and interactions, and are easy to read. They have high variance, and their piecewise-constant predictions cannot extrapolate beyond the training range.

Proposition (variance of an average). Let $f_1, \dots, f_M$ each have variance $\sigma^2$ and pairwise correlation $\rho$. Then

Proof.

1. $\operatorname{Var}\big(\sum_m f_m\big) = \sum_m \operatorname{Var}(f_m) + \sum_{m \ne m'} \operatorname{Cov}(f_m, f_{m'}) = M\sigma^2 + M(M-1)\rho\sigma^2$.
2. Divide by $M^2$: $\sigma^2/M + (M-1)\rho\sigma^2/M = \rho\sigma^2 + (1-\rho)\sigma^2/M$. $\blacksquare$

More trees remove only the second term; the floor $\rho\sigma^2$ stays. Bagging trains deep trees on bootstrap samples (draw $n$ points with replacement). A random forest also picks each split from a random subset of the features (often $\sqrt d$ of the $d$ features for classification); this lowers $\rho$. Each point is left out of a bootstrap sample with probability $(1 - 1/n)^n \to e^{-1} \approx 0.368$. So the trees that did not see a point give a nearly free test estimate for it: the out-of-bag (OOB) error.

Gradient boosting is gradient descent in function space. Start with the best constant $F_0$. At each round $m$:

1. compute the pseudo-residuals $r_i = -\partial L(y_i, F(x_i)) / \partial F(x_i)$, the negative gradient of the loss with respect to the current prediction. For squared loss $L = \tfrac12 (y - F)^2$ this is $r_i = y_i - F(x_i)$, the ordinary residual;
2. fit a small tree $h_m$ to the pairs $(x_i, r_i)$;
3. set each leaf value by a line search on the loss;
4. update $F \leftarrow F + \nu\, h_m$ with a learning rate (shrinkage) $\nu \in (0, 1]$, often 0.05–0.1.

Why "gradient descent": if we could change each prediction $F(x_i)$ freely, the steepest-descent step would be $F(x_i) \leftarrow F(x_i) + \nu r_i$. A tree cannot fit every $r_i$ exactly, so it fits the best approximation of this step that generalizes to new $x$.

3.4 Metrics #

With true and false positives and negatives $TP, FP, FN, TN$:

• ROC-AUC is the area under the curve of TPR against FPR over all thresholds. It equals $P(s^+ > s^-) + \tfrac12 P(s^+ = s^-)$ for the score $s^+$ of a random positive and $s^-$ of a random negative. So it measures ranking only. It does not change under any increasing transform of the scores, and it does not depend on the class ratio, because TPR and FPR are each computed inside one class.
• PR curve and average precision (AP). Precision mixes the two classes, so it depends on the class ratio. A random ranking has AP about equal to the prevalence. With rare positives, report AP and the precision at the operating threshold.
• Proper scores and calibration. Log-loss $-[y \log p + (1-y)\log(1-p)]$ and the Brier score $(p - y)^2$ are minimized in expectation by the true probability. A reliability diagram plots the observed frequency of positives against the predicted probability, in bins; a calibrated model lies on the diagonal. Fix a miscalibrated model on held-out data: temperature scaling (divide the logits by one $T > 0$; standard for neural networks), Platt scaling (a logistic regression on the score), or isotonic regression.
• Multi-class: macro averaging gives each class the same weight; micro averaging pools all decisions, so large classes dominate.
• Regression: MSE is minimized by the conditional mean and punishes outliers strongly; MAE is minimized by the conditional median and is robust. MAPE, the mean of $|y - \hat y|/|y|$, is undefined at $y = 0$ and punishes over-prediction more than under-prediction.

Proposition (threshold from costs). A false alarm costs $c_{FP}$ and a missed positive costs $c_{FN}$. If $p = P(y = 1 \mid x)$ is calibrated, flagging has lower expected cost exactly when

Proof. Flagging costs $(1-p)\,c_{FP}$ in expectation (a false alarm when $y = 0$). Not flagging costs $p\,c_{FN}$ (a miss when $y = 1$). Flag when $p\,c_{FN} > (1-p)\,c_{FP}$, that is $p\,(c_{FN} + c_{FP}) > c_{FP}$. $\blacksquare$

For example, if a missed fraud costs 20 times a false alarm, flag when $p > 1/21 \approx 0.048$.

3.5 Evaluation design and leakage #

• Three sets. Train to fit, validation to choose, test to report, used once. For a model choice and an honest estimate from little data, use nested cross-validation.

• $k$-fold cross-validation. Stratify by class for classification. Split by group when examples are not independent: one patient, one user or one weather station in one fold only. For time series, train on the past and test on the future, with a gap. For spatial fields, use blocks larger than the correlation length.

• Leakage is information in the training data or features that is not available at prediction time:

  • target leakage: a feature recorded after the outcome (for example "refund issued" to predict fraud);
  • preprocessing on all data: a scaler, PCA, feature selection or target encoding fitted before the split;
  • duplicates or near-duplicates across splits;
  • a random split of time series or grouped data.

  Fix: fit every preprocessing step inside the training fold (a pipeline), and split by time or group when the data has that structure.

• Class imbalance. Keep the natural class ratio in validation and test. Class weights or resampling change the effective prior, so the predicted probabilities are no longer calibrated; recalibrate, or keep the data as it is and move the threshold.

• Offline vs online. An offline metric is a proxy for the product goal. The online A/B test is the final judge.

3.6 Probability questions #

Interviews often ask one short probability question. Three types cover most of them.

Bayes' rule with a base rate. A disease has prevalence 0.5%. A test has sensitivity 99% ($P(+ \mid D) = 0.99$) and specificity 99% ($P(- \mid \text{no } D) = 0.99$). A person tests positive. What is $P(D \mid +)$?

The natural-frequency check makes it easy to say aloud: of 100,000 people, 500 are sick and 495 of them test positive. Of the 99,500 healthy people, 1% (995) also test positive. So $495/(495 + 995) = 33\%$ of the positives are sick. The false positives win because the healthy group is 200 times larger. This is the same effect as low precision under class imbalance.

Linearity of expectation. $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y]$ always, even for dependent variables. Example: shuffle $n$ cards numbered $1, \dots, n$. The expected number of cards in their own position is 1. Let $X_i = 1$ if card $i$ is in position $i$; then $\mathbb{E}[X_i] = 1/n$, and $\mathbb{E}[\sum_i X_i] = n \cdot 1/n = 1$, for every $n$. The $X_i$ are dependent, and it does not matter.

A first-step equation. How many fair coin flips, on average, until two heads in a row? Let $E_0$ be the expected number of further flips from the start, and $E_1$ the number after one head. Condition on the next flip:

• $E_1 = 1 + \tfrac12 \cdot 0 + \tfrac12 E_0$ (a head ends the game; a tail starts over);
• $E_0 = 1 + \tfrac12 E_1 + \tfrac12 E_0$.

Substitute the first into the second: $E_0 = 1 + \tfrac12 + \tfrac14 E_0 + \tfrac12 E_0$, so $\tfrac14 E_0 = \tfrac32$ and $E_0 = 6$.

How to answer aloud. Name the events. Write the formula with symbols first. Plug in the numbers. Then check with natural frequencies or an extreme case (for example, "with prevalence 50% the answer would be 99%").

3.7 Hypothesis tests and A/B testing #

Words. The null hypothesis $H_0$ says "no difference". A type I error rejects a true $H_0$; its probability is the level $\alpha$. A type II error keeps a false $H_0$; its probability is $\beta$, and the power is $1 - \beta$. The p-value is the probability, computed under $H_0$, of a test statistic at least as extreme as the one observed. It is not the probability that $H_0$ is true. The minimum detectable effect (MDE) $\delta$ is the smallest effect that is worth detecting. It is a business choice, not the effect you hope for.

Two-proportion z-test. With $x_k$ conversions out of $n_k$ users in arm $k$, rates $\hat p_k = x_k/n_k$ and pooled rate $\bar p = (x_1 + x_2)/(n_1 + n_2)$:

Reject $H_0$ at level $\alpha$ when $|z| > z_{1-\alpha/2}$ (1.96 for $\alpha = 0.05$). For a $2 \times 2$ table this is the same test as the $\chi^2$ test without continuity correction: $\chi^2 = z^2$.

Theorem (sample size per arm). Let the metric have per-user variance $\sigma_1^2$ in control and $\sigma_2^2$ in treatment, with $n$ users per arm and true difference $\delta > 0$. With the normal approximation, a two-sided test at level $\alpha$ has power $1 - \beta$ when

where $z_q$ is the $q$-quantile of the standard normal distribution.

Proof.

1. The difference of the two means, $\hat\Delta$, has standard error $\text{se} = \sqrt{(\sigma_1^2 + \sigma_2^2)/n}$ (independent arms; central limit theorem).
2. The test rejects when $|\hat\Delta| > z_{1-\alpha/2}\,\text{se}$.
3. Under the alternative, $\hat\Delta \approx \mathcal N(\delta, \text{se}^2)$. Ignore the tiny chance of rejecting in the wrong direction. Then $\text{power} \approx P(\hat\Delta > z_{1-\alpha/2}\,\text{se}) = \Phi(\delta/\text{se} - z_{1-\alpha/2})$, with $\Phi$ the standard normal CDF.
4. Power $= 1 - \beta$ means $\delta/\text{se} - z_{1-\alpha/2} = z_{1-\beta}$, so $\delta = (z_{1-\alpha/2} + z_{1-\beta})\,\text{se}$.
5. Substitute $\text{se}$ from step 1, square, and solve for $n$. $\blacksquare$

Rule of thumb. With $\sigma_1 = \sigma_2 = \sigma$, $\alpha = 0.05$ and power 0.8: $2(1.960 + 0.842)^2 = 15.7$, so $n \approx 16\,\sigma^2/\delta^2$ per arm. For a conversion rate $p$, $\sigma^2 = p(1-p)$.

What the formula says.

• $n \propto 1/\delta^2$: half the MDE needs four times the users.
• $n \propto \sigma^2$: a metric with less variance needs fewer users (conversion instead of revenue, or variance reduction with CUPED below).
• An unequal split with a fraction $f$ in treatment needs about $1/(4f(1-f))$ times as many users in total as a 50/50 split: 2.8 times for 90/10.

Design checklist.

1. A hypothesis and one primary metric (the overall evaluation criterion), plus guardrail metrics such as latency, errors and revenue.
2. The unit of randomization, usually the user.
3. Baseline, MDE, $\alpha$ and power → $n$ per arm.
4. Duration: $n$ divided by the eligible users per arm per day, rounded up to full weeks.
5. Checks: an A/A test (two identical arms; it should reject about 5% of the time) and a sample-ratio check.
6. The analysis plan and the decision rule, fixed before the start.
7. Report the effect with a confidence interval, and ask whether it is practically large enough, not only significant.

Pitfalls and their fixes.

• Peeking. Checking every day and stopping at the first $p < 0.05$ inflates the type I error: about 0.08 with 2 looks, 0.14 with 5, 0.19 with 10, and 0.25 with 20 equally spaced looks. Fix the sample size in advance, or use a sequential design built for many looks (alpha spending, always-valid p-values).
• Sample-ratio mismatch (SRM). The observed split differs from the design more than chance allows. Test the counts with a $\chi^2$ goodness-of-fit test and raise an alarm at about $p < 0.001$. An SRM means a bug in assignment, logging or filtering; do not trust the result until it is found.
• Novelty, primacy and day-of-week effects. Users react to change itself. Run full weeks, plot the effect by day, and compare new with returning users.
• Many metrics or variants. With 20 independent metrics at $\alpha = 0.05$, $P(\text{at least one false positive}) = 1 - 0.95^{20} = 0.64$. Choose the primary metric in advance; correct the others (Holm, or Benjamini–Hochberg for the false discovery rate).
• Interference. The analysis assumes that a user's outcome does not depend on the assignment of other users (SUTVA, the stable unit treatment value assumption). Social networks and two-sided marketplaces break it. Randomize clusters or markets, or use switchback designs (switch the whole system over time).
• Unit of randomization ≠ unit of analysis. If you randomize users but compute the click rate per page view, the naive variance is too small, because the views of one user are correlated. Analyze one value per user, or use the delta method or a user-level bootstrap.
• Changing the split during the test (for example 1% → 50%) can make the pooled totals show the reverse of every daily result (Simpson's paradox). Analyze the period with a fixed split.

Answers to section 1. (1) About 60,000. The formula gives 57,760 per arm (57,763 with the pooled variance under $H_0$); the rule of thumb gives $16 \cdot 0.09/0.005^2 = 57{,}600$. (2) Ask for the precision at the operating threshold (or the AP). ROC-AUC ignores the class ratio. Suppose the threshold gives TPR 0.80 at FPR 0.05. Per 1,000,000 transactions there are 5,000 frauds and 995,000 normal ones: 4,000 frauds are caught, and $0.05 \cdot 995{,}000 = 49{,}750$ normal transactions raise false alarms. Precision is $4{,}000/53{,}750 = 7.4\%$: about 12 false alarms for every fraud caught.

WITH per_user AS (
  SELECT a.user_id, a.variant, COUNT(e.user_id) AS purchases
  FROM assignments AS a
  LEFT JOIN events AS e
    ON  e.user_id = a.user_id
    AND e.event = 'purchase'
    AND e.ts >= a.assigned_at          -- only purchases after assignment
  GROUP BY a.user_id, a.variant
)
SELECT variant, COUNT(*) AS users,
       ROUND(AVG(CASE WHEN purchases > 0 THEN 1.0 ELSE 0.0 END), 3) AS conv_rate
FROM per_user
GROUP BY variant;

4. Worked example #

A. Metrics on ten examples. Four positives with scores 0.95, 0.85, 0.70, 0.45; six negatives with scores 0.80, 0.60, 0.40, 0.30, 0.20, 0.10.

1. Threshold 0.5. The predicted positives are the scores 0.95, 0.85, 0.80, 0.70, 0.60. Of these, 3 are real positives. So $TP = 3$, $FP = 2$, $FN = 1$ (the 0.45), $TN = 4$.
2. Precision $= 3/5 = 0.60$. Recall $= 3/4 = 0.75$. $F_1 = 2 \cdot 3/(2 \cdot 3 + 2 + 1) = 6/9 = 0.667$. Accuracy $= 7/10$.
3. ROC-AUC by counting pairs. There are $4 \cdot 6 = 24$ (positive, negative) pairs. The positive 0.95 beats all 6 negatives; 0.85 beats 6 (0.80 is lower); 0.70 beats 5 (not 0.80); 0.45 beats 4 (0.40, 0.30, 0.20, 0.10). Total $6 + 6 + 5 + 4 = 21$, so AUC $= 21/24 = 0.875$.
4. Average precision. Sort by score: 0.95 (+), 0.85 (+), 0.80 (−), 0.70 (+), 0.60 (−), 0.45 (+), …. The positives arrive at ranks 1, 2, 4 and 6. The precision at those ranks is $1/1$, $2/2$, $3/4$, $4/6$. Each positive adds $1/4$ of recall, so AP $= \tfrac14 (1 + 1 + 0.75 + 0.667) = 0.854$.
5. Now add six more negatives with the same scores (the prevalence falls from 40% to 25%). Recall stays $0.75$ and ROC-AUC stays $0.875$: both are computed inside one class. Precision falls to $3/(3 + 4) = 0.43$, and AP falls to $0.775$.

B. An A/B test from plan to decision. A new checkout page. Baseline conversion $p_1 = 10\%$, MDE $\delta = 0.5$ points (a 5% relative lift), $\alpha = 0.05$ two-sided, power 0.8.

1. Sample size: $n = (1.960 + 0.842)^2\,(0.10 \cdot 0.90 + 0.105 \cdot 0.895)/0.005^2 = 7.849 \cdot 0.18398/0.000025 = 57{,}760$ per arm.
2. Duration: 20,000 eligible users per day, split 50/50, give 10,000 per arm per day. $57{,}760/10{,}000 = 5.8$ days. Round up to 7 days (one full week), so about 70,000 per arm.
3. Result: control has 7,024 conversions of 70,240 users ($10.00\%$); treatment has 7,360 of 69,760 ($10.55\%$).
4. SRM check first. The expected count per arm is 70,000. $\chi^2 = (240^2 + 240^2)/70{,}000 = 1.65$, $p = 0.20$: no alarm. (With 70,700 vs 69,300 it would be $\chi^2 = 14.0$, $p = 0.0002$: stop and debug.)
5. Test. Pooled $\bar p = 14{,}384/140{,}000 = 0.1027$. Then $z = 0.0055/\sqrt{0.1027 \cdot 0.8973 \cdot (1/70{,}240 + 1/69{,}760)} = 3.39$, and $p = 0.0007$.
6. 95% confidence interval of the difference, with the unpooled standard error 0.162 points: $0.55 \pm 1.96 \cdot 0.162 = [0.23,\ 0.87]$ points. The effect is significant, and the whole interval is above zero. Whether a lift of 0.2–0.9 points pays for the change is a product decision.

5. In code #

Libraries check the numbers of section 4. The notebook has these cells and the exercises.

import numpy as np
from scipy import stats
from sklearn.metrics import (average_precision_score, confusion_matrix,
                             precision_recall_fscore_support, roc_auc_score)

# (A) metrics on ten examples
y = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
s = np.array([0.95, 0.85, 0.70, 0.45, 0.80, 0.60, 0.40, 0.30, 0.20, 0.10])
pred = (s >= 0.5).astype(int)
print(confusion_matrix(y, pred).ravel())                   # tn fp fn tp: [4 2 1 3]
prf = precision_recall_fscore_support(y, pred, average="binary")[:3]
print(np.round(prf, 3))                                    # [0.6   0.75  0.667]
print(roc_auc_score(y, s), round(average_precision_score(y, s), 3))   # 0.875 0.854
y2, s2 = np.r_[y, y[4:]], np.r_[s, s[4:]]                  # six more negatives, same scores
p2, r2 = precision_recall_fscore_support(y2, (s2 >= 0.5).astype(int), average="binary")[:2]
print(round(p2, 3), r2, roc_auc_score(y2, s2), round(average_precision_score(y2, s2), 3))
# 0.429 0.75 0.875 0.775

# (B) sample size, SRM check, z-test and confidence interval
za, zb = stats.norm.ppf(0.975), stats.norm.ppf(0.80)
n = (za + zb) ** 2 * (0.10 * 0.90 + 0.105 * 0.895) / 0.005 ** 2
print(round(n))                                            # 57760
print(round(stats.chisquare([70240, 69760]).pvalue, 3))    # 0.2: no sample-ratio mismatch
print(round(stats.chisquare([70700, 69300]).pvalue, 5))    # 0.00018: mismatch, debug first
xa, na, xb, nb = 7024, 70240, 7360, 69760
pa, pb, pool = xa / na, xb / nb, (xa + xb) / (na + nb)
z = (pb - pa) / np.sqrt(pool * (1 - pool) * (1 / na + 1 / nb))
print(round(z, 2), round(2 * stats.norm.sf(abs(z)), 5))   # 3.39 0.00069
se = np.sqrt(pa * (1 - pa) / na + pb * (1 - pb) / nb)
print(np.round(100 * np.array([pb - pa - za * se, pb - pa + za * se]), 2))   # [0.23 0.87]

# (C) simulation: power at n = 57,760, and an A/A test
rng = np.random.default_rng(0)
def rejection_rate(p_a, p_b, n, reps=20_000):
    a, b = rng.binomial(n, p_a, reps), rng.binomial(n, p_b, reps)
    pool = (a + b) / (2 * n)
    z = (b - a) / n / np.sqrt(pool * (1 - pool) * 2 / n)
    return np.mean(np.abs(z) > za)
print(rejection_rate(0.10, 0.105, 57_760))   # 0.796 (about 0.80; Monte Carlo error 0.003)
print(rejection_rate(0.10, 0.100, 57_760))   # 0.049 (about 0.05: the A/A test)

6. Where it shows up #

• ML practice. Model selection is bias–variance in action. Offline metrics choose a candidate; an online A/B test decides, and offline and online results often disagree. Gradient-boosted trees (XGBoost, LightGBM, CatBoost) are still the default for tabular data.
• Rare events. Fraud, spam, defects and disease screening all have low prevalence. Precision at the operating point and the base-rate effect of section 3.6 decide whether the model is useful.
• ML for science. Random splits of gridded weather fields or seismic records leak through spatial and temporal correlation; use blocks in space and time, and keep all records of one station, patient or event in one fold. Probabilistic forecasts need calibration (reliability diagrams). Comparing two models on the same test cases is a paired test: bootstrap over the cases and report an interval.
• Product teams. Every launch decision for a ranking, recommendation or pricing model goes through an A/B test with the design checklist of section 3.7.

7. Common mistakes #

• "The p-value is the probability that $H_0$ is true." It is the probability of data at least this extreme, computed under $H_0$.
• Accuracy or ROC-AUC alone on rare events. Report precision and recall at the operating point, or AP.
• Preprocessing on all data before cross-validation. Scaling, feature selection and target encoding must be fitted inside each training fold.
• Random splits of grouped or time-ordered data. Split by group or by time.
• Stopping an A/B test at the first $p < 0.05$, or ignoring a sample-ratio mismatch.
• Analyzing per page view while randomizing per user. The naive variance is too small.
• "Random forests cannot overfit." More trees do not add overfitting, but deep trees on noisy data still have variance, and the floor $\rho\sigma^2$ stays. "Boosting reduces variance" is also wrong as a main claim: boosting mainly reduces bias.
• Confusing sensitivity with $P(D \mid +)$. Always bring in the base rate.

8. Check yourself #

1. 100 trees, each with variance 1 and pairwise correlation 0.3. What is the variance of their average, and how do you lower it further?
   Answer

   $0.3 + 0.7/100 = 0.307$. More trees can only remove the $0.007$. To go lower, decrease $\rho$ (fewer features per split, more randomness) or the variance of each tree.
2. Baseline conversion 5%, relative MDE 10%, $\alpha = 0.05$, power 0.8. How many users per arm (rule of thumb)? And for a relative MDE of 5%?
   Answer

   $\delta = 0.05 \cdot 0.10 = 0.005$, so $n \approx 16 \cdot 0.05 \cdot 0.95/0.005^2 = 30{,}400$ per arm. Half the MDE needs 4 times as many: $121{,}600$.
3. In worked example 4A, which of recall, precision, ROC-AUC and AP change when the test set gets six more negatives with the same scores?
   Answer

   Recall (0.75) and ROC-AUC (0.875) do not change: each is computed inside one class. Precision ($0.60 \to 0.43$) and AP ($0.854 \to 0.775$) fall, because they depend on the class ratio.
4. A missed defect costs 50 and a false alarm costs 5. Your model is calibrated. Above which probability do you flag a part?
   Answer

   $c_{FP}/(c_{FP} + c_{FN}) = 5/55 \approx 0.091$. Flag when $p > 0.091$.
5. A colleague checks the A/B test every day for 10 days and stops when $p < 0.05$. What is the false-positive rate, and what are two fixes?
   Answer

   About 0.19 instead of 0.05 for 10 equally spaced looks. Fix the sample size and analyze once; or use a sequential test with alpha spending or always-valid p-values.
6. You randomize by user, and the metric is clicks per page view. Why is the usual two-proportion test wrong here?
   Answer

   The page views of one user are correlated, so they are not independent trials. The test treats each view as independent and underestimates the variance, which gives too many false positives. Compute one value per user (clicks/views per user, or totals with the delta method) or bootstrap over users.

9. Interview angle #

Recognize the question from the wording.

Typical questions and short model answers.

1. "Explain the bias–variance trade-off." — "At a fixed input, the expected squared error splits into noise, bias squared and variance. Bias is how far the average model is from the truth; variance is how much the model changes with a new training set. Flexible models have low bias and high variance. Regularization, more data and averaging lower the variance. I diagnose it from learning curves: a large train–validation gap means variance; high error on both means bias."
2. "Bagging vs boosting?" — "Bagging trains deep trees in parallel on bootstrap samples and averages them. The average of $M$ models with correlation $\rho$ has variance $\rho\sigma^2 + (1-\rho)\sigma^2/M$, so bagging lowers variance, and random forests lower $\rho$ with random feature subsets. Boosting trains shallow trees in sequence; each tree fits the negative gradient of the loss at the current predictions. It lowers bias and needs early stopping. On tabular data, boosting usually wins on accuracy."
3. "ROC-AUC or PR-AUC for fraud?" — "ROC-AUC measures ranking and ignores the class ratio. At 0.5% fraud, a threshold with 80% recall and 5% FPR gives a precision of only 7%, about 12 false alarms per caught fraud, with an excellent ROC-AUC. So I report average precision and the precision and recall at the operating threshold, chosen from the costs of the two errors."
4. "How many users do we need for this A/B test?" — "I need the baseline rate, the minimum effect worth detecting, the level and the power. Then $n = (z_{1-\alpha/2} + z_{1-\beta})^2 \cdot 2p(1-p)/\delta^2$, about $16\,p(1-p)/\delta^2$. For 10% baseline and a 0.5-point lift that is about 58,000 per arm. I divide by the daily traffic per arm and round up to full weeks."
5. "The PM checked the dashboard on day 3 and it is significant. Can we ship?" — "Not yet. If we look every day and stop at the first significant result, the false-positive rate is much higher than 5%, about 19% with 10 looks. We fixed the sample size in advance, so we wait. If early stopping matters, we use a sequential test with alpha spending. I also check the sample ratio and the guardrail metrics first."
6. "A test is 99% sensitive and 99% specific; the disease has 0.5% prevalence. Positive result: probability of disease?" — "About one third. Of 100,000 people, 500 are sick and 495 test positive; of 99,500 healthy people, 995 test positive. So 495 of 1,490 positives are sick."

Typical follow-ups. "What if the metric is revenue, not conversion?" (higher variance: more users, CUPED, or a capped metric). "What if users talk to each other?" (interference: cluster or market randomization, switchbacks). "Why not test 20 metrics?" ($1 - 0.95^{20} = 64\%$ chance of a false positive; one primary metric, corrections for the rest). "How do you choose $k$ in $k$-fold?" (5 or 10; more folds lower bias of the estimate but cost more and have correlated folds).

Mock prompt (A/B design). "A new ranking model raises offline NDCG by 2%. Design the online test." A strong answer contains: one primary metric (for example, sessions with a purchase per user) and guardrails (latency, revenue, errors); user-level randomization; the power calculation with a baseline and an MDE, and the duration in full weeks; an A/A test and an SRM check; no peeking, or a sequential design; a check for novelty effects; the decision rule fixed in advance; and a note on interference if the product is a marketplace.

10. Practice #

The notebook for this lesson runs in Colab. Each exercise has a check cell and a folded solution:

1. bias_variance: estimate bias², variance and noise for polynomial fits by simulation, and check that they add up to the test MSE.
2. precision_recall_at and cost_threshold: precision and recall at a threshold, and the threshold that minimizes the expected cost.
3. sample_size_per_arm: the formula of section 3.7, checked against a simulation of the power.
4. two_proportion_ztest: the z-test, checked against scipy.stats.chi2_contingency.
5. srm_check: the sample-ratio check with scipy.stats.chisquare.

Timed practice (25 minutes each, talking aloud): "explain the bias–variance trade-off with a picture"; "derive the sample-size formula"; "design an A/B test for a new search ranking"; "our model is 99% accurate on fraud; what do you ask?"; "implement ROC-AUC from scratch"; "expected number of coin flips until HH"; Chip Huyen's questions on evaluation and A/B testing (see below).

Say it aloud drill. Start a 2-minute timer for each question of section 9. Answer with the four parts of "Why this matters" (definition, mechanism, trade-off, number). Record yourself, then compare with sections 3–4. Repeat the ones where you ran over time or had no number.

11. Watch and read #

• Kohavi, Longbotham, Sommerfield and Henne (2009), Controlled experiments on the web: survey and practical guide: §3.2 (sample size, the rule of 16) and §6 (A/A tests, pitfalls).
• James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning (free PDF): §2.2 (bias–variance), §5.1 (cross-validation), §6.2 (ridge and lasso), chapter 8 (trees, bagging, random forests, boosting).
• Chip Huyen, Introduction to Machine Learning Interviews (free online book): the chapters on ML fundamentals, evaluation and statistics.
• scikit-learn user guide, Probability calibration: reliability diagrams, Platt scaling and isotonic regression.
• Friedman (2001), Greedy function approximation: a gradient boosting machine: the original gradient boosting paper.
• Deng, Xu, Kohavi and Walker (2013), Improving the sensitivity of online controlled experiments by utilizing pre-experiment data (CUPED).