Backtest and gate¶
Rolling-origin evaluation without lookahead, its metrics, the holdout gate that decides GO or NO_GO, and the expectation a promotion records.
rolling_origin_eval(
panel: DataFrame,
model: _ModelProtocol,
min_train_rows: int = 20,
stride: int = 10,
flat_stake: float = 1.0,
taker_bps: float | None = None,
costs: CostModel | None = None,
policy: TradePolicy | None = None,
) -> RollingOriginResult
Walk-forward evaluation on a panel dataset.
panel columns required
market_id, decision_date (str YYYY-MM-DD), outcome_label, winning_label, market_price, + feature_* columns
For each eval step
- Train on rows with decision_date < eval_date
- Predict on rows at eval_date
- Decide the trade of each (market_id, decision_date) under
policy - Compute PnL using settle_position
market_price is the label's YES price at the row's decision_date (the plugin
contract, see MarketPlugin.build_training_row). A decision uses only its own row's
price and forecast, so nothing later than the decision date reaches it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
panel
|
DataFrame
|
Training/eval panel (see column contract above). |
required |
model
|
_ModelProtocol
|
Anything with sklearn-style |
required |
min_train_rows
|
int
|
Skip an eval date until this many training rows exist. |
20
|
stride
|
int
|
Evaluate every Nth distinct decision_date. |
10
|
flat_stake
|
float
|
Notional USDC risked per trade. |
1.0
|
taker_bps
|
float | None
|
Legacy fee-only knob. Mutually exclusive with |
None
|
costs
|
CostModel | None
|
Full |
None
|
policy
|
TradePolicy | None
|
The decision rule. Omit it for the historical backtest: YES on the bin
with the highest predicted probability, every market, no threshold (the
numbers published before policies existed reproduce). Pass
|
None
|
Returns:
| Type | Description |
|---|---|
RollingOriginResult
|
RollingOriginResult with all trade records and step metadata. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If both |
dataclass
¶
RollingOriginResult(
trades: DataFrame, steps: list[dict[str, Any]] = list()
)
compute_metrics(trades: DataFrame) -> BacktestMetrics
Compute backtest metrics from a trades DataFrame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trades
|
DataFrame
|
DataFrame with column |
required |
Returns:
| Type | Description |
|---|---|
BacktestMetrics
|
BacktestMetrics with aggregated stats. All zeros if trades is empty. |
dataclass
¶
BacktestMetrics(
num_trades: int,
total_pnl: float,
hit_rate: float,
avg_pnl_per_trade: float,
avg_edge: float,
)
dataclass
¶
HoldoutGateResult(
decision: str,
segment_results: list[SegmentGateResult],
aggregate_pnl: float,
aggregate_trades: int,
)
classmethod
¶
evaluate(
trades: DataFrame,
required_segments: list[str] | None = None,
min_trades_per_segment: int = 40,
min_pnl_per_segment: float = 0.0,
) -> HoldoutGateResult
Evaluate the holdout gate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trades
|
DataFrame
|
Backtest trade log (as produced by |
required |
required_segments
|
list[str] | None
|
Segments that must each pass. When |
None
|
min_trades_per_segment
|
int
|
Minimum trades a segment needs to pass. |
40
|
min_pnl_per_segment
|
float
|
Minimum total PnL a segment needs to pass. |
0.0
|
dataclass
¶
SegmentGateResult(
segment: str,
num_trades: int,
total_pnl: float,
passes: bool,
reason: str,
)
dataclass
¶
BacktestExpectation(
n_trades: int,
flat_stake: float,
mean_pnl_per_stake: float,
std_pnl_per_stake: float,
brier_score: float | None = None,
brier_std: float | None = None,
by_direction: Mapping[str, GroupExpectation]
| None = None,
policy: TradePolicy | None = None,
by_segment: Mapping[str, SegmentExpectation | None]
| None = None,
)
Mean, spread and size of the backtest trades a champion was promoted on.
brier_score / brier_std are the mean and standard deviation of the
per-trade squared forecast error. They are None when the backtest trades did
not carry both predicted_prob and label_occurred.
by_direction holds the same numbers for the YES trades and for the NO trades on
their own: a NO trade is a buy of a different token, picked where the model is
bearish, and its PnL and forecast error are not draws from the YES distribution.
None means the backtest did not record a split (a champion published before
0.9.1, or a frame without a direction column): the pooled numbers then describe
the historical backtest, which only bought YES. A side the backtest traded fewer
than MIN_GROUP_TRADES times is absent from the mapping, unless it traded nothing
else (then it is the pooled expectation); an empty mapping means neither side traded
enough to be promised.
policy is the decision rule the backtest ran (TradePolicy), recorded so live
trading can use the same threshold. None when not recorded.
by_segment holds the same numbers for each segment on its own, each with its own
split by side. A segment the backtest traded fewer than MIN_GROUP_TRADES times is
present with None: recorded as absent, so "this segment could not be promised
anything" stays distinguishable from "nobody looked". None for the whole mapping
means no per-segment record was made (a champion published before it existed).
An expectation is only worth recording if a live run can be measured against it, so the constructor refuses one that cannot: every number finite, a spread that is not negative, a stake above zero, a Brier score in [0, 1] and at least two trades (one trade has no spread, none divides by zero). NaN compares false with everything, so it would not fail a test, it would silently switch the check off. The mean may be zero or negative; that is a finding the watchdog reports, not a malformed record.
class-attribute
instance-attribute
¶
by_direction: Mapping[str, GroupExpectation] | None = None
class-attribute
instance-attribute
¶
by_segment: (
Mapping[str, SegmentExpectation | None] | None
) = None
classmethod
¶
from_trades(
trades: DataFrame,
*,
flat_stake: float = 1.0,
policy: TradePolicy | None = None,
min_group_trades: int = MIN_GROUP_TRADES,
) -> BacktestExpectation
Summarise a rolling_origin_eval(...).trades frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trades
|
DataFrame
|
Needs |
required |
flat_stake
|
float
|
The notional each backtest trade risked — the
|
1.0
|
policy
|
TradePolicy | None
|
The trade policy the backtest ran, to record with the numbers. |
None
|
min_group_trades
|
int
|
Fewest trades a side needs to be recorded (at least 2). |
MIN_GROUP_TRADES
|
Raises:
| Type | Description |
|---|---|
ValueError
|
On fewer than two trades (no spread to measure), a
non-finite PnL, a non-positive stake, forecasts outside [0, 1] or a
|
classmethod
¶
from_dict(data: Mapping[str, Any]) -> BacktestExpectation
Restore a record; raises ValueError (KeyError if a key is absent) if unusable.
A record written before 0.9.1 has neither by_direction nor policy; both load
as None. One that has them but cannot be used raises: it must never read as
"not recorded" and quietly weaken the watchdog.
dataclass
¶
GroupExpectation(
n_trades: int,
mean_pnl_per_stake: float,
std_pnl_per_stake: float,
brier_score: float | None = None,
brier_std: float | None = None,
)
Mean, spread and size of one group of backtest trades: a side, or a segment.
The same numbers BacktestExpectation keeps for the whole backtest, per unit
stake. A group is held to the same rules as the whole: every number finite, a
spread that is not negative, at least two trades, a Brier score in [0, 1].
classmethod
¶
from_dict(data: Mapping[str, Any]) -> GroupExpectation
Restore a record; raises ValueError (KeyError if a key is absent) if unusable.
dataclass
¶
SegmentExpectation(
n_trades: int,
mean_pnl_per_stake: float,
std_pnl_per_stake: float,
brier_score: float | None = None,
brier_std: float | None = None,
by_direction: Mapping[str, GroupExpectation]
| None = None,
)
Bases: GroupExpectation
One segment's numbers, with that segment's own split by side.
A segment trades YES and NO in its own proportion, so a live segment is held to its own per-side means, weighted by the live mix, exactly as the whole book is.
purged_kfold(
samples: NDArray[int_] | ndarray,
n_splits: int = 5,
embargo: int = 0,
) -> Iterator[tuple[NDArray[np.int_], NDArray[np.int_]]]
Yield (train_idx, test_idx) for purged k-fold CV.
The samples are split into n_splits contiguous test blocks (preserving
time order). For each, the training set is every other sample minus an
embargo band immediately before and after the test block.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
NDArray[int_] | ndarray
|
1-D array whose length defines the sample count (values unused;
pass |
required |
n_splits
|
int
|
Number of folds (>= 2). |
5
|
embargo
|
int
|
Number of samples to purge on each side of the test block. |
0
|
Yields:
| Type | Description |
|---|---|
tuple[NDArray[int_], NDArray[int_]]
|
(train_idx, test_idx) as positional-index arrays into |
embargoed_split(
samples: NDArray[int_] | ndarray,
n_splits: int = 5,
embargo: int = 0,
) -> Iterator[tuple[NDArray[np.int_], NDArray[np.int_]]]
Yield (train_idx, test_idx) for walk-forward CV with an embargo gap.
Expanding-origin: fold f's training set is everything up to the start of
its test block minus an embargo gap, and the test block is the next
contiguous slice. Train is always strictly earlier than test.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
NDArray[int_] | ndarray
|
1-D array whose length defines the sample count. |
required |
n_splits
|
int
|
Number of forward test blocks (>= 2). |
5
|
embargo
|
int
|
Gap (in samples) between the end of train and the start of test. |
0
|
Yields:
| Type | Description |
|---|---|
NDArray[int_]
|
(train_idx, test_idx) positional-index arrays. Early folds may yield an |
NDArray[int_]
|
empty train_idx when the embargo consumes the available history. |