Backtest Costs — `pmlab.core.costs`¶

A backtest that fills at the quoted price reports an edge the broker cannot reproduce. CostModel makes execution friction explicit, and it is opt-in: omit it and rolling_origin_eval behaves exactly as it did before this module existed, so previously published results stay reproducible.

Why this is not a detail¶

On Polymarket's mutually-exclusive (negRisk) events the gap between the mid and the executable price is structural, not incidental. Read at the mid, a field of outcomes sums to ~1.0 and looks coherent; read at the ask, it sums above 1.0, and in large fields half the outcomes have no book at all. Any edge measured against a mid price is a spread you have not paid yet.

The same applies inside a single market: market_price in the training panel is a quote, and your fill is worse than a quote by the half-spread plus whatever the book charges for your size.

Basic Usage¶

from pmlab import CostModel, rolling_origin_eval

# Default — identical to pre-0.8.3 behaviour: 30bps taker fee, fill at the quote.
result = rolling_origin_eval(panel, model)

# Realistic: pay the fee, cross the half-spread, and walk the book on size.
costs = CostModel(
    taker_bps=30.0,            # Polymarket CLOB taker fee
    slippage_bps=50.0,         # half-spread, as a fraction of price
    depth_bps_per_unit=20.0,   # extra bps per multiple of the reference stake
    depth_reference_stake=10.0,  # USDC that fills without moving the book
)
result = rolling_origin_eval(panel, model, flat_stake=25.0, costs=costs)

# Upper bound — what the strategy would earn in a frictionless world.
best_case = rolling_origin_eval(panel, model, costs=CostModel.frictionless())

Comparing the two runs is the point: the distance between them is how much of your edge is paid to the market.

The arithmetic¶

depth_penalty_bps = depth_bps_per_unit * max(0, stake - ref) / ref
total_bps         = slippage_bps + depth_penalty_bps
fill              = price * (1 + total_bps / 10_000) + slippage_fixed
fill              = min(fill, 1.0)
fee               = stake * taker_bps / 10_000

Notes on each term:

Parameter	Unit	Models
`taker_bps`	bps of notional stake	exchange fee
`slippage_bps`	bps of price	half-spread crossed on entry
`slippage_fixed`	probability units	a floor cost (0.01 = one cent)
`depth_bps_per_unit`	bps per multiple of reference	walking the book on size
`depth_reference_stake`	USDC	size that fills at top of book

slippage_bps is proportional to price, so a $0.10 outcome slips less in absolute terms than a $0.90 one — which matches how a percentage spread behaves. Use slippage_fixed when the cost is a floor rather than a percentage.

The fill is clamped at 1.0. A probability price above certainty would create a position paying more than it can settle for.

What the depth term is, and is not¶

It is a documented linear approximation, not an order-book walk. pmlab's training panel carries market_price, not L2 depth, so the model cannot know where the book thins out. The penalty grows linearly in excess stake above a reference you choose.

If you have depth data, measure it and set depth_bps_per_unit from the measurement rather than guessing. If you don't, prefer a pessimistic value: a backtest that overstates cost fails safe, one that understates it promotes a champion that loses money live.

Where costs propagate¶

Costs are not cosmetic — they flow into the promotion decision, and (since v0.8.4) into the live path too:

rolling_origin_eval fills at fill_price instead of market_price, and records that column in the trade log.
edge is computed as predicted_prob - fill_price. Measuring it against the quote would overstate every trade by the friction. Under the live policy (TradePolicy.live) the same arithmetic prices the NO side: a NO trade is a buy of the NO token, filled at costs.fill_price(1 - market_price), worth 1 - predicted_prob, and best_edge (the larger of the two sides, after the fee) is what --min-edge is held against. pmlab.core.edge.quote_edges is the one place that is computed; a live EdgeSignal.from_quote and the backtest both call it.
compute_metrics averages edge into avg_edge.
HoldoutGateResult reads those metrics. A book whose edge exists only at the mid can no longer pass the gate.
stability_report bootstraps the post-cost PnL, so the confidence interval is around what you would actually have earned.
ChampionManifest.publish(costs=...) records the model in champion.json, and ModelRegistry.record() carries it into the archived version — so a rollback restores a champion whose cost assumptions are known.
PaperBroker(costs=...) fills at the same price the backtest would, and records fill_price next to gamma_price.
SettlementEngine settles each trade at its recorded fill_price (a NO trade as a buy of the NO token at its own price), so realized_pnl carries the friction. It writes settled_basis ("fill_price", or "quote" when none was recorded) and settlement_version on the trade. Releases up to 0.9.0 settled at the quote even when a fill_price was recorded and wrote neither key; pmlab monitor leaves such trades out of its PnL comparison (see the watchdog).

Using one cost model for backtest and live¶

This is the whole point of steps 6 and 7. If the backtest models 50bps of slippage and the paper broker fills at the quote, then live-vs-backtest PnL differ for two reasons at once — the market, and your own arithmetic — and a drift alarm cannot separate them.

costs = CostModel(taker_bps=30.0, slippage_bps=50.0)

# 1. validate under that friction
result = rolling_origin_eval(panel, model, flat_stake=25.0, costs=costs)
gate = HoldoutGateResult.evaluate(result.trades, required_segments=[...])

# 2. record it with the champion, so the assumption travels with the model
manifest = ChampionManifest.publish(
    model=model, gate=gate, output_dir=out, plugin_family="weather_tmax", costs=costs,
)

# 3. trade under the same friction — read it back rather than retyping it
broker = PaperBroker(
    trades_path=trades, flat_stake=25.0,
    allowed_segments=manifest.get_allowed_segments(),
    costs=manifest.costs,
)

Step 3 is the part worth insisting on: take costs from the manifest instead of constructing a second CostModel by hand. Two hand-written models drift the moment someone edits one of them.

For the same quote and stake, both sides produce the same fill — asserted across a grid of prices and stakes in tests/execution/test_cost_model_unified.py.

manifest.costs is None for champions published before v0.8.4. Treat that as "cost assumptions unknown" rather than "no costs".

From the command line¶

backtest, promote-champion and record-trades each take --costs PATH, a JSON file of the same fields (CostModel.to_dict()):

{"taker_bps": 30.0, "slippage_bps": 50.0, "slippage_fixed": 0.0,
 "depth_bps_per_unit": 20.0, "depth_reference_stake": 10.0}

pmlab backtest --plugin my_family --panel panel.parquet --costs costs.json --output metrics.json
pmlab promote-champion model.pkl --gate-path gate.json --plugin my_family --expectation metrics.json
pmlab record-trades --plugin my_family            # fills under the champion's costs

The model is written once and then carried, not retyped:

backtest --output records the costs its numbers are net of (the default model if --costs was omitted) under a costs key.
promote-champion --expectation metrics.json publishes the champion with those costs. --costs there is only needed without an expectation file, and it must agree with the file's if both exist: a champion cannot be promoted on a backtest it did not run.
record-trades fills under champion.json's costs by default. --costs may name a model when the champion records none, but not a different one: trades filled under other costs are left out of the watchdog's PnL comparison, so recording them would waste the evidence.

A champion promoted with no costs anywhere still records costs: null, "unknown", and pmlab monitor then reports cost_basis: "unknown" instead of checking it.

Compatibility¶

No costs= argument → 30bps fee, fill at the quote. Byte-identical to the old behaviour; there is a test asserting frame equality.
taker_bps= still works on its own for fee-only adjustments, on both rolling_origin_eval and PaperBroker.
Passing both taker_bps= and costs= raises ValueError. Two sources of truth for the fee would silently double-charge it.
PaperBroker.taker_bps still exists and reads through to costs.taker_bps, so code inspecting it keeps working.

Reproducibility¶

CostModel.to_dict() returns a flat mapping that round-trips through CostModel(**payload), so log it with the run:

with tracker.run(experiment="weather_tmax", run_name="with_costs") as run:
    run.log_params(costs.to_dict())

A PnL number without its cost model is not reproducible — the same panel and the same model give a different answer under different friction.

Backtest Costs — pmlab.core.costs¶