Reproducibility¶
Capture a run (seeds, versions, resolved config, data fingerprint) and re-execute it with pmlab reproduce.
dataclass
¶
ReproducibilityRecord(
kind: str,
seeds: dict[str, int | None],
environment: dict[str, str | None],
config: dict[str, Any],
data: DataFingerprint,
metrics: dict[str, Any],
schema_version: int = SCHEMA_VERSION,
created_at: str = (
lambda: datetime.now(UTC).isoformat()
)(),
)
Everything needed to re-run a run and to judge whether it came back the same.
kind names the recipe ("backtest"); config is its resolved parameters;
metrics is what the original run produced, the target a re-run is compared against.
class-attribute
instance-attribute
¶
created_at: str = field(
default_factory=lambda: datetime.now(UTC).isoformat()
)
classmethod
¶
capture(
kind: str,
seeds: Mapping[str, int | None],
config: Mapping[str, Any],
data: DataFingerprint,
metrics: Mapping[str, Any],
) -> ReproducibilityRecord
Record a run now: the live environment, plus what the caller says it used.
classmethod
¶
from_dict(d: Mapping[str, Any]) -> ReproducibilityRecord
Read a record back; ValueError names what is wrong with a malformed one.
dataclass
¶
DataFingerprint(
content_sha256: str,
n_rows: int,
columns: list[str],
source: str | None = None,
)
Enough of a dataset to tell, later, that it is not the same one.
content_sha256 decides; n_rows and columns are there so a mismatch can say how
the data differs. source is where the panel was read from, if it came from a file.
dataclass
¶
BacktestConfig(
plugin: str,
model: str,
model_params: dict[str, Any],
stride: int,
min_train_rows: int,
flat_stake: float,
costs: dict[str, float],
policy: dict[str, Any],
tuning: dict[str, Any] | None = None,
)
The resolved parameters of a backtest: nothing left to a default.
Build one with resolve, which applies the defaults; from_dict reads one back
from a record exactly as it was written.
classmethod
¶
resolve(
plugin: str,
model: str = "lgbm_baseline",
model_params: Mapping[str, Any] | None = None,
stride: int = 30,
min_train_rows: int = 20,
flat_stake: float = 1.0,
costs: CostModel | None = None,
policy: TradePolicy | str = "live",
min_edge: float | None = None,
tuning: Mapping[str, Any] | None = None,
) -> BacktestConfig
Apply every default and validate. ValueError says what is wrong.
tuning is how model_params were chosen, for the record: pass
tuner.tuning_config() after TunedForecaster.tune. It is provenance only (the
backtest re-runs with the tuned model_params and does not search again) and is
left out of the config entirely when None, so untuned records are unchanged.
capture_backtest(
panel: DataFrame,
config: BacktestConfig,
seeds: Mapping[str, int | None],
metrics: Mapping[str, Any],
source: str | None = None,
) -> ReproducibilityRecord
The reproducibility record of a finished backtest.
fingerprint_panel(
df: DataFrame, source: str | None = None
) -> DataFingerprint
Fingerprint a panel: content hash (values, column names, row order), rows, columns.
The index is ignored (a panel re-read from disk has a fresh one); row order is not, because a walk-forward backtest can depend on it.
load_record(
ref: str,
runs_dir: str | Path | None = None,
registry_dir: str | Path | None = None,
) -> ReproducibilityRecord
Resolve ref to a record. ref may be:
- a path to a JSON file — a tracker run file,
champion.json, a registry version'schampion.json, the filepmlab backtest --outputwrote, or a bare record; - a path to a directory holding
champion.json(a directory without one falls through to the run-id lookup, so a run id is never shadowed by an unrelated folder); - a tracker run id (or unique prefix) under
runs_dir(defaultruns); - a model-registry version id under
registry_dir.
Raises ReproduceError when nothing matches, or the match holds no record.
pmlab reproduce: find a recorded run, re-execute it, and say whether it came back.
The verdict is about the metrics. A re-run reproduces when every recorded metric comes back (counts exactly; floats within a documented tolerance) on the same data. Everything else is evidence for the reader:
- different data (content hash) — refused,
DataMismatchError, unless the caller allows it; a run on other data can never be called a reproduction. - different library versions — never a verdict by itself. They are listed (
report.drift) next to the outcome: a run that reproduces despite drift is reproduced; one that does not, with drift listed, tells you where to look first. - nothing to re-run (no record, unknown kind, missing panel) —
ReproduceError, with a message that says what to do about it.
Tolerance: on the machine and versions of the original run the result is bit-for-bit
identical, so the default is tight (rtol=1e-9, atol=1e-12): it absorbs floating-point
summation order across CPUs and library builds, and nothing a different seed would produce.
Integer metrics (trade counts, steps) are compared exactly whatever the tolerance.
Bases: Exception
The run cannot be re-executed. The message says why and what to do.
DataMismatchError(
recorded: DataFingerprint, current: DataFingerprint
)
Bases: ReproduceError
The panel is not the data the run was recorded on.
dataclass
¶
MetricComparison(
name: str,
recorded: Any,
reproduced: Any,
abs_diff: float | None,
ok: bool,
)
dataclass
¶
ReproduceReport(
record: ReproducibilityRecord,
status: Literal["reproduced", "not_reproduced"],
comparisons: list[MetricComparison],
drift: list[EnvDrift],
data_match: bool,
rtol: float,
atol: float,
notes: list[str] = list(),
)
The outcome of one reproduce: the verdict and the evidence behind it.
compare_metrics(
recorded: Mapping[str, Any],
reproduced: Mapping[str, Any],
rtol: float = DEFAULT_RTOL,
atol: float = DEFAULT_ATOL,
) -> list[MetricComparison]
Compare each recorded metric with its re-run value.
Integers must be equal; floats must satisfy |a-b| <= atol + rtol*|a|; None (a
non-finite metric, stored as null) must come back None. A metric the re-run no longer
produces fails.
load_record(
ref: str,
runs_dir: str | Path | None = None,
registry_dir: str | Path | None = None,
) -> ReproducibilityRecord
Resolve ref to a record. ref may be:
- a path to a JSON file — a tracker run file,
champion.json, a registry version'schampion.json, the filepmlab backtest --outputwrote, or a bare record; - a path to a directory holding
champion.json(a directory without one falls through to the run-id lookup, so a run id is never shadowed by an unrelated folder); - a tracker run id (or unique prefix) under
runs_dir(defaultruns); - a model-registry version id under
registry_dir.
Raises ReproduceError when nothing matches, or the match holds no record.
reproduce(
record: ReproducibilityRecord,
*,
panel: DataFrame | None = None,
panel_path: str | Path | None = None,
allow_data_mismatch: bool = False,
rtol: float = DEFAULT_RTOL,
atol: float = DEFAULT_ATOL,
) -> ReproduceReport
Re-execute the recorded run and compare its metrics with the recorded ones.
The panel is, in order of preference: panel (in memory), panel_path, the path the
record was made from. Its fingerprint must match the record's, else
DataMismatchError — unless allow_data_mismatch, which runs anyway and can then
only ever report not_reproduced.
resolve_seeds(
seed: int = DEFAULT_SEED, optuna: int | None = None
) -> dict[str, int]
One root seed -> the seed of every generator a backtest draws from.
optuna is the sampler seed of a TunedForecaster (tuner.random_state) whose
best parameters the backtest uses; leave it out for a run that was not tuned.
run_backtest(
panel: DataFrame,
config: BacktestConfig,
seeds: Mapping[str, int | None],
) -> BacktestOutcome
Seed, then run the walk-forward backtest the config describes.
The process-wide generators are seeded for the duration of the run only; the caller's own
numpy / random streams are left exactly where they were.
backtest_metrics(
result: RollingOriginResult,
) -> dict[str, Any]
The summary a backtest is judged (and reproduced) by.
dataclass
¶
BacktestOutcome(
trades: DataFrame,
steps: list[dict[str, Any]],
metrics: dict[str, Any],
)
build_forecaster(
config: BacktestConfig, seeds: Mapping[str, int | None]
) -> LGBMForecaster
The forecaster of the run, seeded from seeds['lightgbm'].
capture_environment() -> dict[str, str | None]
Versions of pmlab and the libraries a result depends on, plus the OS / architecture.
A library that is not installed (optuna, behind the tune extra) is recorded as None
rather than left out, so "was not there" and "was not recorded" are different statements.
compare_metrics(
recorded: Mapping[str, Any],
reproduced: Mapping[str, Any],
rtol: float = DEFAULT_RTOL,
atol: float = DEFAULT_ATOL,
) -> list[MetricComparison]
Compare each recorded metric with its re-run value.
Integers must be equal; floats must satisfy |a-b| <= atol + rtol*|a|; None (a
non-finite metric, stored as null) must come back None. A metric the re-run no longer
produces fails.
DataMismatchError(
recorded: DataFingerprint, current: DataFingerprint
)
Bases: ReproduceError
The panel is not the data the run was recorded on.
dataclass
¶
EnvDrift(
name: str, recorded: str | None, current: str | None
)
environment_drift(
recorded: Mapping[str, str | None],
current: Mapping[str, str | None] | None = None,
) -> list[EnvDrift]
What differs between the recorded environment and this one (default: the live one).
Only keys the record holds are compared, so a record that predates a newly tracked library is not accused of drifting on it.
load_panel(path: str | Path) -> pd.DataFrame
Read a .parquet or .csv panel; ValueError for any other extension.
dataclass
¶
MetricComparison(
name: str,
recorded: Any,
reproduced: Any,
abs_diff: float | None,
ok: bool,
)
Bases: Exception
The run cannot be re-executed. The message says why and what to do.
dataclass
¶
ReproduceReport(
record: ReproducibilityRecord,
status: Literal["reproduced", "not_reproduced"],
comparisons: list[MetricComparison],
drift: list[EnvDrift],
data_match: bool,
rtol: float,
atol: float,
notes: list[str] = list(),
)
The outcome of one reproduce: the verdict and the evidence behind it.
seed_everything(seeds: Mapping[str, int | None]) -> None
Pin the process-wide generators named in seeds (python_random, numpy).
Library-specific seeds (lightgbm, optuna, ...) are not global: they are handed to
the estimator that uses them. A None seed is left alone — the generator keeps whatever
state it has, and the record says so.
seeded(seeds: Mapping[str, int | None]) -> Iterator[None]
Seed the process-wide generators for the block, then give them back as they were.
A library call (run_backtest, reproduce) must not reseed the caller's own numpy or
random stream as a side effect, so it uses this rather than seed_everything.