Monitoring¶
Drift, live calibration, and the watchdog behind pmlab monitor.
dataclass
¶
DriftConfig(
n_bins: int = 10,
min_samples: int = 300,
epsilon: float = 0.0001,
thresholds: DriftThresholds = DriftThresholds(),
numeric: tuple[str, ...] = (),
categorical: tuple[str, ...] = (),
)
Everything that is a policy choice rather than arithmetic.
n_bins is the number of reference-quantile bins for numeric features (10 is
the PSI convention). min_samples is the least number of usable values each
side needs before a distribution statistic is reported, and the least number of
rows each side needs before a missing-rate change is classified. PSI of an
unchanged feature is not zero: its noise floor is roughly (n_bins - 1) / n
for n current values. The default of 300 keeps that near 0.03 at ten bins —
in 500 simulated windows of an unchanged feature, 3 crossed the 0.10 warn line
at 300 values and 223 did at 100. Raise it if you raise n_bins.
epsilon is the floor on a bin's proportion — see the module docstring.
Numeric vs categorical is inferred from the reference column's contents
(integers and floats are numeric; strings, booleans and everything else are
categorical). Name a feature in numeric or categorical to override — for
example integer-coded categories.
class-attribute
instance-attribute
¶
thresholds: DriftThresholds = field(
default_factory=DriftThresholds
)
dataclass
¶
DriftReport(
config: DriftConfig,
features: tuple[FeatureDrift, ...],
prediction: FeatureDrift | None,
n_reference: int,
n_current: int,
absent_from_current: tuple[str, ...],
new_in_current: tuple[str, ...],
n_current_excluded_overlap: int = 0,
)
Drift of every assessed feature, plus the model's predictions if supplied.
features is ordered by name, whatever order the columns arrived in.
prediction is reported separately so a model whose inputs look unchanged
but whose output moved is visible. status is the worst of all of them.
n_reference / n_current are the row counts of the feature tables.
n_current_excluded_overlap is how many live-window markets
drift_report_from_store dropped because the reference panel already contains
them (always 0 from the pure drift_report, which sees only the tables it is
given). absent_from_current lists assessed features the current table does not have
at all (they are scored as 100% missing); new_in_current lists columns the
reference does not have (informational — they are not assessed).
The flat counts and max_psi / max_kl exist for trackers that only log
top-level numbers; they are None, not zero, when nothing was computed.
dataclass
¶
DriftThresholds(
psi_warn: float = 0.1,
psi_alert: float = 0.25,
kl_warn: float = 0.05,
kl_alert: float = 0.125,
missing_warn: float = 0.05,
missing_alert: float = 0.2,
)
Where each signal turns from ok to warn to alert.
A value at or above a threshold is in that band, with 1e-9 of slack for float round-off (a missing rate going 10% -> 30% changes by 0.19999999999999998, and is still an alert at 0.20). The defaults are conventions, not laws — tune them on your own data:
- PSI 0.10 / 0.25 is the widely used credit-scoring convention: below 0.10 no meaningful shift, 0.10-0.25 moderate, above 0.25 significant.
- KL 0.05 / 0.125 has no industry convention. These are half the PSI bands: for small shifts both KL directions are about equal, so PSI ~ 2 * KL.
- Missing-rate 0.05 / 0.20 is the absolute change in the share of missing values (either direction). No convention exists; a 0% -> 40% jump is an alert.
dataclass
¶
FeatureDrift(
name: str,
kind: FeatureKind,
status: DriftStatus,
drivers: tuple[str, ...],
n_reference: int,
n_current: int,
n_reference_valid: int,
n_current_valid: int,
reference_missing_rate: float | None,
current_missing_rate: float | None,
missing_rate_change: float | None,
psi: float | None,
kl: float | None,
unseen_rate: float | None,
n_unseen_categories: int,
unseen_categories: tuple[str, ...],
)
Drift of one feature (or of the model's predictions) against the reference.
psi and kl are None — not zero — when either side has fewer than
min_samples usable values. kl is KL(current || reference), in nats.
missing_rate_change is current - reference (signed); it is classified by
its absolute value. unseen_rate is the share of usable current values the
reference never contained: outside the reference's min/max for a numeric
feature, in a category absent from the reference for a categorical one. It is
already inside psi / kl in both cases (out-of-range mass has its own bins),
so it has no thresholds; it stays on the report because a coarse epsilon
makes that mass cheap and can hide it from the status.
drivers names the signals ("psi", "kl", "missing_rate") that set
status; empty when ok.
drift_report(
reference: Mapping[str, Any] | DataFrame,
current: Mapping[str, Any] | DataFrame,
*,
features: Sequence[str] | None = None,
reference_predictions: Sequence[float]
| ArrayLike
| None = None,
current_predictions: Sequence[float]
| ArrayLike
| None = None,
config: DriftConfig | None = None,
) -> DriftReport
Compare current against reference, feature by feature.
Pure: no I/O, no randomness; the same inputs give an identical report.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
Mapping[str, Any] | DataFrame
|
The training panel, as a DataFrame or |
required |
current
|
Mapping[str, Any] | DataFrame
|
What the model is being fed now, same shape. |
required |
features
|
Sequence[str] | None
|
Columns to assess; defaults to every reference column. Pass the model's feature list so identifiers and labels are not assessed. |
None
|
reference_predictions
|
Sequence[float] | ArrayLike | None
|
The model's predicted probabilities on the
reference panel. Supply together with |
None
|
current_predictions
|
Sequence[float] | ArrayLike | None
|
The model's live predicted probabilities. The two arrays are different samples and need not be the same length. |
None
|
config
|
DriftConfig | None
|
Bins, minimum sample, smoothing and thresholds. |
None
|
Returns:
| Type | Description |
|---|---|
DriftReport
|
A |
DriftReport
|
|
DriftReport
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
if |
drift_report_from_store(
store: FeatureSnapshotStore,
family: str,
reference: DataFrame,
*,
window_start: str,
cutoff: str,
features: Sequence[str] | None = None,
predict_fn: Callable[
[DataFrame], Sequence[float] | ArrayLike
]
| None = None,
exclude_reference_markets: bool = True,
config: DriftConfig | None = None,
) -> DriftReport
Drift of a live window in a FeatureSnapshotStore against a training panel.
The current window is built with store.as_of(family, cutoff) — the store's
own point-in-time read — so it holds, per market, the latest snapshot captured
strictly before cutoff; nothing later can leak in. It is then restricted
to snapshots captured at or after window_start, and — by default — to markets
the reference panel does not contain. Both restrictions exist for the same
reason: without them "current" would include the very markets the model was
trained on and drift would be understated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store
|
FeatureSnapshotStore
|
The point-in-time snapshot store. |
required |
family
|
str
|
Market family to read. |
required |
reference
|
DataFrame
|
The training panel (e.g. from |
required |
window_start
|
str
|
ISO-8601 start of the live window (inclusive). A naive timestamp is read as UTC. |
required |
cutoff
|
str
|
ISO-8601 end of the live window (exclusive), same convention as
|
required |
features
|
Sequence[str] | None
|
Columns to assess; defaults to the reference's |
None
|
predict_fn
|
Callable[[DataFrame], Sequence[float] | ArrayLike] | None
|
Optional. Called with the full reference panel and the live window, it returns predicted probabilities: exactly one per row, the probability of the label you trade. Use it to enable prediction drift. It is not called on an empty window. |
None
|
exclude_reference_markets
|
bool
|
Drop from the live window every market whose
|
True
|
config
|
DriftConfig | None
|
See |
None
|
An empty window yields a report whose features are all "insufficient" rather
than an error: no live data is not evidence the world is fine. The store
holding nothing at all for family is different — that is almost always a
typo — and raises.
Raises:
| Type | Description |
|---|---|
ValueError
|
if |
CalibrationTracker(
trades: list[dict[str, Any]], *, n_bins: int = 10
)
Scores realized trades from a paper/live trade log.
Only settled rows that carry a model_prob are scored. Trades written before
the prediction was persisted are counted in n_skipped rather than silently
treated as zero — an unscoreable trade is missing information, not evidence
of a bad forecast.
squared_errors() -> list[float]
Each scored trade's squared forecast error, in target_date order.
Their mean is overall().brier_score. The spread is what a significance
test needs: a live Brier above the backtest's is only a finding if it
clears the noise of the per-trade terms.
overall() -> CalibrationWindow | None
Calibration over every realized trade, or None if none are scoreable.
rolling(window: int) -> list[CalibrationWindow]
One window per trade once window trades have accumulated.
by_segment() -> dict[str, CalibrationWindow]
Per-segment calibration.
The gate promotes per segment, so an aggregate that hides one broken segment is the failure mode worth guarding against.
dataclass
¶
CalibrationWindow(
n_trades: int,
brier_score: float,
reliability: float,
resolution: float,
uncertainty: float,
skill_score: float,
base_rate: float,
mean_forecast: float,
window_start: str,
window_end: str,
reliability_curve: tuple[dict[str, float], ...],
)
Calibration over one set of realized trades.
reliability is the Murphy reliability term: the probability-weighted mean
squared gap between what was forecast and what occurred. Lower is better,
and unlike brier_score it does not improve just because the base rate is
extreme.
dataclass
¶
CalibrationThresholds(
min_trades: int = 30,
brier_warn_abs: float = 0.02,
brier_alert_abs: float = 0.05,
brier_warn_rel: float = 0.1,
brier_alert_rel: float = 0.25,
z_warn: float = 2.0,
z_alert: float = 3.0,
n_bins: int = 10,
)
When live Brier has fallen far enough behind the backtest's to act on.
A level (warn or alert) is reached only when the live Brier is worse than the
backtest's by at least the absolute and the relative margin, and the
gap clears the noise (z). Absolute alone over-fires on a tiny Brier; relative
alone over-fires on a large one. All values are conventions, not standards.
dataclass
¶
PnLThresholds(
min_trades: int = 30,
shortfall_warn: float = 0.5,
shortfall_alert: float = 1.0,
z_warn: float = 2.0,
z_alert: float = 3.0,
sequential: bool = True,
cusum_allowance: float = 0.5,
cusum_backtest_se: float = 0.5,
cusum_threshold: float = 30.0,
cusum_min_backtest_trades: int = 100,
cusum_increment_cap: float = 5.0,
mde_power: float = 0.8,
)
When live PnL per unit stake has fallen far enough behind the backtest's.
shortfall_* is the fraction of the backtest mean that is missing:
(backtest - live) / backtest. 0.5 means earning half of what was promised;
1.0 means earning nothing. A level needs the shortfall and a one-sided z of
at least z_*. The defaults of 2 and 3 are stricter than the textbook 1.64 and
2.33 on purpose: the watchdog looks every run, and repeated looks at the same
growing sample inflate the false-alarm rate far above the nominal one.
The sequential test (sequential) is a one-sided CUSUM on each trade's
standardized shortfall, in settlement order. It is the other way a PnL alert is
raised: it is above cusum_threshold while the book is in a run of shortfalls that the
pooled mean dilutes with the healthy trades before it, and it clears when the run ends.
What each trade is allowed to fall short by without it counting is
cusum_allowance times the promised edge, plus cusum_backtest_se standard errors of
the promise itself (both in units of the trade's own spread): the backtest mean is an
estimate, and a CUSUM that ignored its error would drift to an alert on any champion whose
backtest happened to flatter it. 0.5 of the edge is tuned to an edge that has vanished.
The defaults were calibrated by simulation, see docs/watchdog.md.
A promise made on very few trades is too noisy even so: cusum_min_backtest_trades is
the fewest backtest trades a cell may have been promised on for its trades to feed the
sequential test.
No single trade may move the statistic by more than cusum_increment_cap: a cell whose
backtest (almost) never lost has a tiny spread, and one ordinary total loss is then
hundreds of its standard deviations. Capped at 5, a statistic that alerts at 30 needs at
least six trades in a row, however small the spread; a spread that large is not reached by
any trade in a realistic cell, so the cap leaves the test's power as it was.
mde_power is the power the minimum detectable shortfall is quoted at.
dataclass
¶
WatchdogConfig(
drift: DriftConfig = DriftConfig(),
calibration: CalibrationThresholds = CalibrationThresholds(),
pnl: PnLThresholds = PnLThresholds(),
insufficient: InsufficientPolicy = "separate",
)
Every threshold the watchdog applies, written into every report.
insufficient is the policy for a check that could not be measured:
"separate" (default) gives the report its own status and exit code, so a
new champion with no trades yet is distinguishable from a healthy one;
"warn" counts it as a warn. There is deliberately no "ignore".
class-attribute
instance-attribute
¶
calibration: CalibrationThresholds = field(
default_factory=CalibrationThresholds
)
classmethod
¶
from_dict(data: Mapping[str, Any]) -> WatchdogConfig
Build from a possibly partial dict: absent keys keep their defaults.
Unknown keys raise. A typo'd threshold that was quietly ignored would leave the default in force while the operator believed otherwise.
Bases: ValueError
The inputs or configuration are unusable. The CLI exits 64 on this.
dataclass
¶
CheckResult(
name: str,
status: HealthStatus,
reasons: tuple[str, ...],
metrics: dict[str, Any],
cause: str | None = None,
)
One check's verdict, the reasons for it, and the numbers behind it.
reasons is never empty: a problem says what is wrong and against which
threshold, a healthy check says what it measured. cause is set only for
insufficient and says why it could not be measured:
too_few_samples, no_expectation, invalid_expectation (the recorded
expectation cannot be compared against), invalid_statistic (a number computed
from the live data is not finite) or not_evaluated.
dataclass
¶
HealthReport(
as_of: str,
champion: ChampionRef,
checks: tuple[CheckResult, ...],
skipped: tuple[str, ...],
config: WatchdogConfig,
)
The watchdog's verdict. Frozen, strict-JSON-safe, deterministic.
property
¶
status: HealthStatus
The worst check, after the configured policy for insufficient.
property
¶
problems: tuple[CheckResult, ...]
Every check that is not plainly ok, insufficient included.
render(*, verbose: bool = False) -> str
Short, actionable text for a human or a cron mail; "" when healthy.
Quiet on green is a property of the report, not of the CLI: a healthy
report renders to nothing unless verbose asks for the full picture.
classmethod
¶
from_dict(data: Mapping[str, Any]) -> HealthReport
Restore a report. status and exit_code are derived, not trusted.
health_report(
*,
champion: ChampionManifest,
trades: Sequence[Mapping[str, Any]],
drift: DriftReport | None,
as_of: str,
config: WatchdogConfig | None = None,
checks: Sequence[str] = CHECKS,
) -> HealthReport
Judge a deployed champion: drift, live calibration, live PnL.
Pure: no I/O, no clock. Everything is passed in already loaded.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
champion
|
ChampionManifest
|
The authoritative record of what was promoted — its allowed segments, publish time, cost model and backtest expectation. Never substituted by a retrain summary (AGENTS.md safety rule 5). |
required |
trades
|
Sequence[Mapping[str, Any]]
|
The paper/live trade log rows ( |
required |
drift
|
DriftReport | None
|
A |
required |
as_of
|
str
|
ISO-8601 cutoff. Nothing recorded after it, and no trade settled
after it ( |
required |
config
|
WatchdogConfig | None
|
Thresholds and the |
None
|
checks
|
Sequence[str]
|
Which of |
CHECKS
|
Raises:
| Type | Description |
|---|---|
WatchdogInputError
|
An unparseable or offset-less |