Tutorial — discover to paper trade, offline¶

One notebook, the whole workflow, on data that ships with the package. No network, no account, no API key.

notebooks/tutorial.ipynb walks through discover → panel → tune → backtest → gate → paper → report with pmlab's public API only (every name is imported from pmlab). It doubles as living documentation: each step is a short prose cell and a few lines of code, and the three safety rules of AGENTS.md are not described but hit on purpose: a NO_GO champion refuses to publish, a price seen too late is excluded, and a trade will not settle before its truth is final.

It runs in about 15 seconds on a laptop. The notebook CI job executes it on every push (see Keeping it from rotting), so what you read is what works.

Run it¶

pip install "pmlab[tune]" jupyterlab        # `tune` brings Optuna, used by step 3
jupyter lab notebooks/tutorial.ipynb        # the file is in the repository, next to docs/

From a checkout, headless, exactly as CI does:

uv sync --extra dev
uv run pytest -m notebook

The bundled sample¶

pmlab.datasets.load_sample() returns 750 synthetic "Highest temperature in city on date?" markets: one per city per day, five cities, 150 days from 2025-04-02, nine bins each (a single degree each plus a tail at either end).

from pmlab.datasets import load_sample

sample = load_sample()
print(sample)

plugin = sample.plugin(sample.days[0])   # the real WeatherTmaxPlugin, wired to the bundled file
spec = plugin.discover_markets()[0]      # offline: no Gamma API involved
print(spec.question)
print([b.label for b in spec.outcome_bins])
print(spec.metadata["price_observed_at"])


Synthetic	generated by `scripts/make_sample_dataset.py` from a fixed seed; `sample.meta["synthetic"]` is `True`. It says nothing about real markets.
Size	39 KB gzipped JSON plus a 65-byte checksum, inside the wheel (`pmlab/datasets/data/`).
Per market	the book as seen at 07:30 UTC the day before (nine prices, with an overround), the forecast features of that moment (8 of them), and the day's official maximum in whole °C.
Integrity	`load_sample()` checks the file against its SHA-256 and raises `SampleDataError` on a mismatch or a missing file. It never half-reads.
Deterministic	the same bytes every time; two loads are equal.
Shaped to teach	the book under-reacts to the forecast in four cities, so a model can find an edge; in Lisbon the book is sharper than the forecast, so the per-city gate has a segment to refuse.

How "discover" works offline¶

WeatherTmaxPlugin takes its Gamma, forecast and truth clients by injection (that is how its own tests drive it). sample.plugin(day) passes three small clients that read the bundled file (SampleGammaClient, SampleForecastClient, SampleTruthClient). The plugin is neither subclassed nor given an offline switch: discover_markets, build_training_rows and is_truth_final run their production code. A plugin built without a Gamma client still raises, as before.

Time is replayed, not faked. plugin(day) is the plugin as it stood at 07:30 UTC that morning: it lists the markets open for the next day, stamps their prices with that time, and only sees observations that were final by then. plugin() with no day is the plugin after everything has resolved, which labels a panel and settles trades, and has nothing "open" to discover.

The steps¶

Step	API used	What it shows
0. Workspace	`WorkspaceContext`	every path comes from a workspace; two workspaces never share one
1. Discover	`load_sample`, `WeatherTmaxPlugin.discover_markets`, `is_truth_final`	the offline plugin; truth is not final the morning of
2. Panel	`build_training_rows`, `build_panel`, `check_no_leakage`	one row per bin, leakage-checked against "did this bin win" (docs/leakage.md); a price seen at the decision is excluded and counted
3. Tune	`TunedForecaster`	Optuna on the first 60 days only, scored walk-forward under the live policy and costs it will be traded with (docs/tuning.md)
4. Backtest	`BacktestConfig`, `run_backtest`, `capture_backtest`	the same live policy and costs; a reproducibility record that also says which objective tuned the parameters
5. Gate	`HoldoutGateResult`, `ChampionManifest.publish`, `reproduce`	a per-city NO_GO refuses to publish; scoped GO publishes; the record re-runs
6. Paper	`EdgeSignal`, `PaperBroker`, `SettlementEngine`	segment filter from `champion.json`; no settlement before truth is final
7. Report	`generate_report`	backtest vs paper, Brier score, an HTML report

The 150 days are split into three windows that never overlap: tune (days 1-60), backtest (61-120, trained on everything earlier, graded with the tuned parameters), live (121-150, paper traded by a champion that never saw them).

What a run prints¶

An excerpt of the executed notebook (the numbers are the sample's and stable on a given numpy/LightGBM; the sample is synthetic, so the PnL says nothing about real markets):

decision: NO_GO
          num_trades  total_pnl  passes        reason
Athens            60  33.724088    True            ok
Istanbul          60  20.360557    True            ok
Lisbon            58 -10.983657   False  negative_pnl
Madrid            60  26.411228    True            ok
Rome              60  26.145321    True            ok

refused: Cannot publish champion with NO_GO gate. Gate decision: NO_GO

GO on ['Athens', 'Istanbul', 'Madrid', 'Rome']
reproduced | 5 metrics compared | data matches the record: True

signals worth trading: {'Athens': 29, 'Istanbul': 30, 'Lisbon': 23, 'Madrid': 30, 'Rome': 30}
recorded: {'Athens': 29, 'Istanbul': 30, 'Madrid': 30, 'Rome': 30}
on the morning of 2025-08-14 {'settled': 55, 'pending': 64, 'total_pnl': 31.069619}
once the sample is complete: {'settled': 64, 'pending': 0, 'total_pnl': 36.159611}

Scoping the champion to the cities that cleared the gate is a choice made after looking at the backtest, so the notebook says so and leaves the live window to check it. Fifty or a hundred paper trades cannot tell a good strategy from a lucky one; that is what paper trading first, and pmlab monitor afterwards, are for.

Keeping it from rotting¶

Executed in CI. The notebook job runs pytest -m notebook: the notebook, top to bottom, in a fresh kernel started in a directory outside the repo, with every non-loopback connection refused. The marker is deselected from the default uv run pytest, so the main job stays fast. The test fails above 120 seconds; it takes about a tenth of that.
The executor is nbclient (with nbformat and ipykernel, all dev dependencies): it is what nbmake and nbconvert --execute drive underneath, without their other dependencies. The same small script, tests/e2e/run_notebook.py, runs in CI and in the e2e test.
Committed without outputs. Outputs carry timings, temp paths and library-version noise that would churn every diff. A default-suite test fails if a cell is committed with outputs or an execution count, so a diff shows only what was edited. The real output is the CI run's, and the excerpt above.
Public API only. Another default-suite test fails if the notebook imports from a private module, touches an underscore attribute, or names an export pmlab does not have.
The checks are live. A corrupted sample, or a notebook edited to lift the segment filter, turns the job red (both are tests). The e2e test runs the notebook from the installed wheel in a fresh venv.

Regenerating the sample¶

uv run python scripts/make_sample_dataset.py

The output is deterministic for a given numpy. Regenerating changes every number above and the checksum, so re-run pytest -m notebook, pytest and pytest -m e2e afterwards. The seed is chosen so that the tutorial's per-city gate separates Lisbon from the other four with a margin, which is the point of the sample and the reason to keep it synthetic.