Monitoring in production: alerts, delayed labels, feature stores and skew

Before this lesson: lesson 07 (PSI, KS and the kinds of shift); lesson 05 (the serving path that reads features); lesson 06 (rollback and retraining, the answers to an alert); multiple testing. Time: about 60 minutes, plus the notebook.

---

Why this matters #

After a release, a model can fail in two quiet ways. The world changes (drift, lesson 07), or the pipeline feeds the model wrong inputs (skew). Both look the same from outside: the server answers, and the answers are worse. A monitor that misses them costs money or trust; a monitor that pages too often is ignored, which is just as bad.

Weather and climate ML has the same problems. ML weather models are trained on reanalysis data (ERA5) but run from operational analyses, whose statistics differ. Observation labels (rain gauges, river levels) arrive after the forecast time. Fraud labels (chargebacks) arrive weeks later. "How would you monitor this model?" and "What is training-serving skew?" are standard questions in ML system design interviews.

1. Predict first #

2. The picture #

Layers of a monitor. Watch the system from the outside in. Each layer catches problems the layers above it cannot.

Layer	Example signals	Labels?	Catches
service health	request rate, errors, p99 latency	no	outages, overload
data quality	schema, nulls, ranges, volume, freshness, feature age	no	broken pipelines, unit changes, stale features
input drift	PSI or $D$ per feature, a domain classifier	no	covariate shift and label shift
prediction drift	distribution of $\hat y$ or of the score	no	input changes that reach the output
performance per slice	accuracy, calibration per station, region or class	yes, delayed	everything, including concept drift

Every alert needs an owner and a runbook: a short list of what to check and what to do (check the upstream data, roll back, retrain; lesson 06).

A score over time. A drift monitor computes a score (here PSI against a reference) on the last $w$ values, every $s$ new values. The window $w$ sets how much data each score sees; the stride $s$ sets how often a score is computed. Then a rule decides when to page.

Interactive figure: loading…

3. Definitions and main results #

Designing a drift monitor #

• Windows. Score the last $w$ values every $s$ new values. A short window reacts fast but is noisy (its PSI noise floor $(k-1)(1/n + 1/w)$ is high); a long window is stable but slow. Use sliding windows, not cumulative statistics: a cumulative accuracy over months hides a sudden dip.
• Reference. Usually the training data. For seasonal inputs (temperature, river flow) use the same season of earlier years, or every summer looks like drift.
• Thresholds. Calibrate them on stable history: replay months without a known incident and take a high quantile (for example 99.9%) of the window scores. Prefer effect sizes (PSI, $D$) to p-values (lesson 07).
• Levels and hysteresis. Use two levels, warn and alert, and a lower clear level.

Definition (alert state machine with hysteresis). The state is ok, warn or alert, and $\text{clear} \le \text{warn} \le \text{alert}$. For each new score $x$:

1. Compute the level of $x$: alert if $x \ge \text{alert}$, warn if $x \ge \text{warn}$, else ok.
2. If this level is higher than the state, move up to it and send an event ("warn" or "alert").
3. Otherwise, if the state is not ok and $x < \text{clear}$, move down to ok and send "recovered".
4. Otherwise, stay and send nothing.

So a state goes up at once, but it comes down only below the clear level. One incident gives one page, even when the score jumps around the alert threshold. A persistence rule ("page only after 3 windows in a row above alert") is another way to get the same effect, at the cost of a later page.

Many features. Each check of a feature has a small false-alarm probability $\alpha$.

Result. With $k$ independent features, the expected number of false alarms per check is $k\alpha$, and the probability of at least one is $1 - (1 - \alpha)^k$.

Proof. Each feature gives a false alarm with probability $\alpha$; the count is $\mathrm{Binomial}(k, \alpha)$, with mean $k\alpha$. No false alarm at all means every feature is quiet, which has probability $(1-\alpha)^k$ by independence. $\blacksquare$

With 200 features and $\alpha = 0.01$: 2 false alarms per check on average, and $1 - 0.99^{200} = 0.866$, so 87% of checks page someone for nothing. (Correlated features give fewer, but still many.) So page on a few signals (prediction drift, data quality, the most important features), and show the rest on a dashboard. Alerts that nobody acts on teach people to ignore all alerts: this is alert fatigue.

Delayed labels #

A performance metric needs labels. If the label of a prediction made at time $p$ arrives at time $p + d$, then at time $t$ the metric can only use predictions with $p + d \le t$. With a window of length $w$, the monitor at time $t$ uses the predictions made in

Result. Let traffic be uniform, and let the true accuracy change from $A$ to $B$ at time $t_0$. Then the monitored accuracy is $A$ for $t \le t_0 + d$, it is $B$ for $t \ge t_0 + w$, and in between it is

Proof.

1. The usable predictions lie in $(t - w, t - d]$, an interval of length $w - d$.
2. If $t - d \le t_0$, all of them were made before the change: the metric is $A$.
3. If $t - w \ge t_0$, all of them were made after the change: the metric is $B$.
4. In between, the part after $t_0$ is $(t_0, t - d]$, of length $t - d - t_0$. With uniform traffic the metric is the weighted average of $A$ and $B$ with weight $(t - d - t_0)/(w - d)$ on $B$. $\blacksquare$

Answer to question 2 of section 1. With $d = 30$, $w = 60$ and $t_0 = 100$, the monitored accuracy starts to fall after day 130 and shows the full drop of 0.60 from day 160. At day 145 half of the usable predictions come after the drop, so it reads $0.90 - 0.30 \times 15/30 = 0.75$.

Interactive figure: loading…

Two more facts about delayed labels:

• The evaluation window must be longer than the label delay, and a quality drop shows up only after the delay. This is the main reason to also monitor inputs and predictions.
• Labels that arrive early are often not a random sample. Obvious fraud is reported first; a flood is confirmed before a "no flood". An accuracy computed on early labels is biased. Report the share of labels that have arrived next to the metric.

Feature stores #

A feature is a value computed for an entity (a user, a station, a grid cell) at an event time, for example "soil moisture of station S1 at 12:00". A feature store holds each feature definition once and provides its values both for training and for serving. Its parts:

• The offline store keeps the full history of feature values with timestamps (a data warehouse or Parquet files). It builds training sets and runs batch scoring.
• The online store keeps only the latest value per entity in a low-latency key-value store. The serving API reads it at request time, in milliseconds.
• Materialisation is the scheduled job that loads the latest values from the offline store (or a batch source) into the online store. Streaming features are pushed into the online store directly.
• The TTL (time to live) of a feature is the maximum age of a valid value. Offline, the training-set join ignores values older than the TTL. Online, an older value should count as missing too, so that both paths agree. Stores differ in whether they enforce this online, so check your store, or enforce the age limit in the serving code.

Feast, Hopsworks and the feature stores of the large cloud platforms all have these parts.

Point-in-time correct joins #

To train, we join each label row (entity $u$, time $t$, label $y$) with feature values. The values must be the ones the model could have known at time $t$.

Definition (point-in-time correct join). A label row $(u, t)$ gets the feature row of entity $u$ with the latest time $s$ such that

1. $s < t$ (strict), or $s \le t$ (inclusive), and
2. $s \ge t - \text{TTL}$ (the value is not too old).

If there is no such row, the features are missing.

• Strict or inclusive? Use strict $s < t$ when a value stamped at $t$ can contain the label event itself (for example a count of transactions that includes the current one). Inclusive $s \le t$ is right when values stamped at $t$ were really available before the prediction. pd.merge_asof is inclusive by default; allow_exact_matches=False makes it strict.
• TTL. Feast's historical join scans backward from each label time for at most one TTL; the TTL is relative to each label time, not to the time you run the query.
• The wrong join. A plain join on the entity with today's feature table gives old label rows values from their future. This is future leakage: offline metrics become too good, and online the model meets other inputs. Rule #31 of Google's Rules of ML warns about exactly this: a table you join at training time may have changed by serving time.

Interactive figure: loading…

Training-serving skew #

Google's Rules of ML define training-serving skew as a difference between performance during training and performance during serving, caused by a difference in how the two pipelines handle data, a change in the data between training and serving, or a feedback loop. This lesson uses the word skew for the pipeline cause: the same example gets different inputs offline and online. A change of the world is drift (lesson 07).

Cause	Mitigation
two code paths (SQL for training, another language at serving)	one feature definition for both paths: a feature store or a shared library (rule #32: re-use code between training and serving)
future leakage: features computed after the label time	point-in-time joins on event time (rule #31)
stale or missing online features (materialisation lag, expired TTL, other default values)	monitor feature age and null rate online; the same TTL and the same default on both paths
client-side preprocessing (the app normalises before calling the API)	move preprocessing into the model artifact or the server; validate inputs at the API
different library versions (tokenizer, scaler, pandas), float32 vs float64	one pinned image or lockfile for training and serving (lesson 03); a parity test

How to find skew. Log the features actually used at serving time (rule #29: "save the set of features used at serving time"). For the same entity and time, recompute the features with the training pipeline. Compare them feature by feature: report the match rate per feature (exact, or within a stated tolerance), then compare the model scores. This is rule #37: measure training-serving skew.

Interactive figure: loading…

Skew or drift? Skew: the same example, different inputs; fix the pipeline. Drift: the world changed; monitor, retrain or adapt. To decide, take logged serving features and recompute the training features for the same entities and times. If they differ, it is skew. If they match but the production distribution differs from the training set, it is drift.

Answer to question 1 of section 1. Future leakage. Label rows from before a feature update see the later value, so the offline metrics are too optimistic, and online the inputs differ from the training inputs. Input drift monitors compare marginal distributions of production with training, so they can miss it. The fix is a point-in-time join.

When accuracy drops: a debugging order #

Check the cheap causes first:

1. Service and data quality. Errors, timeouts, nulls, unit changes, stale features.
2. Skew. Logged serving features against recomputed training features.
3. Drift per feature and per slice. Which features moved, for which stations or regions.
4. Concept drift. Fresh labels: is the rule itself different? Then retrain on recent data or roll back (lesson 06).

4. Worked example #

A. Hysteresis. Warn 0.10, alert 0.25, clear 0.10. PSI per window:

Window	1	2	3	4	5	6	7	8
PSI	0.05	0.12	0.27	0.22	0.26	0.24	0.08	0.05
level of the score	ok	warn	alert	warn	alert	warn	ok	ok
state after the step	ok	warn	alert	alert	alert	alert	ok	ok
event	—	warn	alert	—	—	—	recovered	—

Step by step: window 2 (0.12) is above warn, so the state moves up to warn and sends "warn". Window 3 (0.27) moves it up to alert: one page. Window 4 (0.22) has level warn, which is not higher than the state, and $0.22 \ge 0.10$ (clear), so nothing happens. Windows 5 and 6 are the same. Window 7 (0.08 < 0.10) clears the alert: "recovered". A naive rule "page when the score crosses 0.25" pages at windows 3 and 5: two pages for one incident.

B. Point-in-time join. Station S1 has soil_moisture at 06:00 (0.21), 12:00 (0.25) and 18:00 (0.40). Label rows at 05:00, 12:00, 13:00, 20:00 and 23:30. Strict $s < t$, TTL 4 hours.

Label time $t$	Latest feature row with $s < t$	Age $t - s$	Value
05:00	none	—	missing
12:00	06:00 (12:00 is not strictly before)	6 h > 4 h	missing
13:00	12:00	1 h	0.25
20:00	18:00	2 h	0.40
23:30	18:00	5.5 h > 4 h	missing

With the inclusive rule, the label at 12:00 gets 0.25 (age 0). A plain join on the station with the current table gives 0.40 to every row: the rows at 05:00, 12:00 and 13:00 see a value from their future.

C. Delayed labels. Labels arrive $d = 30$ days after each prediction, and the accuracy window is $w = 60$ days. At day $t$ the metric uses the predictions of $(t - 60, t - 30]$. The true accuracy drops from 0.90 to 0.60 at day 100.

• Day 130: the usable predictions are from $(70, 100]$, all before the drop: 0.90.
• Day 145: $(85, 115]$; the part after the drop is $(100, 115]$, half of 30 days: $0.90 + (0.60 - 0.90) \times 15/30 = 0.75$.
• Day 160: $(100, 130]$, all after the drop: 0.60.

An input drift monitor can react on day 100, if the drop comes with an input change.

5. In code #

The code checks the three worked examples; the notebook asks you to write each function yourself.

import numpy as np
import pandas as pd

# A. The alert state machine with hysteresis.
ORDER = {"ok": 0, "warn": 1, "alert": 2}

def step(state, score, warn=0.10, alert=0.25, clear=0.10):
    level = "alert" if score >= alert else "warn" if score >= warn else "ok"
    if ORDER[level] > ORDER[state]:
        return level, level                       # move up, send an event
    if state != "ok" and score < clear:
        return "ok", "recovered"                  # move down only below clear
    return state, None

state, events = "ok", []
for score in [0.05, 0.12, 0.27, 0.22, 0.26, 0.24, 0.08, 0.05]:
    state, ev = step(state, score)
    events.append(ev or "-")
print(events)   # ['-', 'warn', 'alert', '-', '-', '-', 'recovered', '-']

# Many features: chance of at least one false alarm per check.
print(f"{1 - 0.99**200:.3f}")                      # 0.866

# B. Point-in-time join with a 4-hour TTL, strict s < t.
t = lambda h: pd.Timestamp("2026-07-01") + pd.Timedelta(hours=h)
features = pd.DataFrame({"station": "S1", "time": [t(6), t(12), t(18)],
                         "soil_moisture": [0.21, 0.25, 0.40]})
labels = pd.DataFrame({"station": "S1", "time": [t(5), t(12), t(13), t(20), t(23.5)]})
joined = pd.merge_asof(labels, features.assign(feature_time=features["time"]),
                       on="time", by="station", direction="backward",
                       tolerance=pd.Timedelta(hours=4), allow_exact_matches=False)
print(joined["soil_moisture"].tolist())            # [nan, nan, 0.25, 0.4, nan]

# C. Delayed labels: delay 30 days, window 60 days, accuracy 0.90 -> 0.60 on day 100.
day = np.arange(0, 250, 0.01)                      # prediction times (uniform traffic)
acc = np.where(day < 100, 0.90, 0.60)
def monitored(t_now, delay=30, window=60):
    use = (day > t_now - window) & (day <= t_now - delay)   # in the window and labelled
    return acc[use].mean()
print(", ".join(f"{monitored(t_now):.2f}" for t_now in (130, 145, 160)))   # 0.90, 0.75, 0.60

merge_asof needs both tables sorted by time. Its tolerance is inclusive ($t - s \le$ TTL), which matches $s \ge t - \text{TTL}$ in the definition.

6. Where it shows up #

• ML weather models. They are trained on ERA5 reanalysis but run from operational analyses with other statistics: the same atmospheric state gets other inputs in training and in operation, which is skew in the sense of this lesson. For this reason, the versions of Google DeepMind's weather models used operationally are fine-tuned on ECMWF HRES data and start from operational HRES initial conditions (GraphCast repository, as of 2026). An upgrade of the data assimilation system, or a new satellite instrument, shifts the input statistics overnight.
• Hydrology and observations. River levels and rain-gauge data arrive late and are quality controlled later still: a delayed-label problem. Monitor per station, the natural slice.
• Fraud and credit. Chargebacks arrive weeks later, and the early ones are the obvious cases. Feature stores were built for these systems: many features per user, computed in batch for training and read online in milliseconds.
• Recommendation and ranking. Features such as "clicks in the last hour" are pushed by streaming jobs; materialisation lag and different default values are common sources of skew.
• Tools. Metrics and alerts are often kept in Prometheus and Grafana; drift reports in Evidently or NannyML; feature stores in Feast, Hopsworks or a cloud platform. The concepts of this lesson are the same in all of them.

7. Common mistakes #

• Paging on every feature. Many features times a small false-alarm rate gives a page on most checks.
• A single threshold without hysteresis. One incident around the threshold gives many pages.
• Cumulative statistics. They average a sudden incident away. Use sliding windows.
• The training data as the reference for a seasonal feature. Every season then looks like drift.
• Thresholds chosen by hand. Calibrate them on stable history, for the window size you use.
• Trusting accuracy on early labels. Early labels are not a random sample, and the metric lags by the label delay.
• Building training data with a plain join on the current feature table. Future leakage and skew.
• Retraining before checking skew. A buggy pipeline stays buggy after retraining.

8. Check yourself #

1. Why do monitors use sliding windows instead of a cumulative statistic since the release?
   Answer

   A cumulative statistic averages over all data since the release, so a sudden incident moves it very little and very late. A sliding window of length $w$ forgets old data and reacts within about $w$ values; the stride sets how often it is checked.
2. How do you choose the alert threshold of a PSI monitor?
   Answer

   Replay a stable period (no known incident) with the same window, stride and binning, compute the window scores, and take a high quantile, for example the 99.9% quantile. The threshold then has a known false-alarm rate per window. It must be recalibrated if the window or the binning changes.
3. Labels arrive 14 days late and the accuracy window is 7 days. What does the accuracy monitor show?
   Answer

   Nothing: the usable predictions are in $(t - 7, t - 14]$, which is empty. The window must be longer than the delay, for example 28 days, and then a drop shows fully only 28 days after it starts.
4. A feature has rows at 08:00 (value 3) and 09:30 (value 7); a label row is at 09:30; TTL 1 hour. What does the label row get with $s \le t$? When is that right?
   Answer

   7 (age 0). It is right if the value stamped 09:30 was really available before the prediction at 09:30 and does not contain the labelled event. If the value is computed from events up to and including the labelled one, use strict $s < t$, which gives "missing" here.
5. Offline AUC 0.91, online the model behaves like 0.80. How do you decide between skew and drift?
   Answer

   Take logged serving features (rule #29) and recompute the training features for the same entities and timestamps. If they differ for the same example (a low match rate on some feature), it is skew: fix the pipeline. If they match but the production distributions differ from the training set, it is drift: retrain or adapt.
6. The online store's materialisation job fails for 6 hours. Which monitor sees it first, and how would the model's inputs change?
   Answer

   A data-quality monitor on feature age (freshness) sees it at once. The online store keeps serving the last materialised values, so the model gets stale inputs. If the serving code enforces the TTL, the values become missing after one TTL and the null rate rises. Drift monitors may see it later, or not at all if the stale values look normal.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is hysteresis in an alerting rule?

   Show answer

2. Why use sliding windows instead of cumulative statistics?

   Show answer

3. How do you calibrate a drift alert threshold?

   Show answer

4. 200 independent features, 1% false-alarm rate each: probability of at least one false alarm per check?

   Show answer

5. With label delay $d$ and window $w$, which predictions can an accuracy monitor use at time $t$?

   Show answer

6. What is the difference between the offline and the online feature store?

   Show answer

7. What is materialisation in a feature store?

   Show answer

8. State the point-in-time join rule for a label row at time $t$.

   Show answer

9. Define training-serving skew (in the pipeline sense).

   Show answer

10. Skew or drift: which one does retraining fix?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "How would you monitor a model whose labels arrive weeks later?" — Layers: service health, data quality (including feature freshness), input and prediction drift with calibrated thresholds and hysteresis, then delayed performance per slice with the share of arrived labels; a runbook per alert.
• "How do you avoid alert fatigue?" — Page on few signals; calibrate thresholds on stable history; hysteresis or persistence; everything else on a dashboard; an owner for every alert.
• "What is training-serving skew, and how does a feature store help?" — The same example gets different inputs; one feature definition for both paths, point-in-time joins, the same TTL and defaults; log serving features and measure the match rate.
• "Accuracy dropped. How do you find out why?" — Data quality first, then skew (logged vs recomputed features), then drift per feature and slice, then concept drift with fresh labels; then roll back or retrain.

10. Practice #

The notebook for this lesson runs in Colab. Each exercise has a check cell and a folded solution:

1. Write rolling_scores(reference, stream, window, stride, metric) with the PSI and KS functions of lesson 07 (given in the notebook).
2. Write drift_state_step(state, score, warn, alert, clear) and reproduce the hysteresis table of worked example A; count pages of the naive rule and of hysteresis on a noisy stream.
3. Write labeled_accuracy(t, pred_times, label_delays, correct, window) and reproduce worked example C (0.75 at day 145).
4. Write point_in_time_join(labels, features, ttl, strict) with pandas, without merge_asof, and compare it with merge_asof.
5. Skew hunt: a training feature function and a serving function with a planted bug; compute the match rate per feature on logged requests and find the bug.

11. Watch and read #

Lecture 06: Continual Learning (FSDL 2022)The Full Stack · Full Stack Deep Learning 2022 · 65 min

From 12:34, metrics for monitoring ML models and the tools to test them; then retraining strategies.

What is a Feature Store for Machine Learning?Hopsworks · 4 min

What a feature store does and why teams use one.

ML Drift: Identifying Issues Before You Have a ProblemFiddler AI · 15 min

Concept drift and data drift, and the early warning signs to monitor.

• Google, Rules of Machine Learning: the section "Training-Serving Skew", rules #29 to #37.
• Feast documentation, Point-in-time joins: the TTL example.
• Chip Huyen, Data distribution shifts and monitoring (2022): the section "Monitoring and Observability".
• Made With ML, Monitoring: the sections on delayed outcomes, drift and solutions.
• Breck et al., Data Validation for Machine Learning (SysML 2019): validation between batches and of training against serving data.