How We Validate
model diligence · provenance ledgerWe publish what our models can’t do.
One claim has survived real out-of-sample testing: the price-range bands are calibrated. Directional prediction has been tested the same way and has not earned a claim — so the terminal frames direction as views, never probabilities. Everything below is a tested verdict with its data provenance, reproducible from the repository in one command.
Every probe's pass/fail criteria are committed to the repository before the first run — results can't quietly move the goalposts.
Every prediction sees strictly-past data. A cheating-model proof runs in CI to keep the chokepoint honest.
Synthetic results are never evidence. Each claim is labeled with the data that tested it — and untestable claims say so.
Models that fail their gate are retired publicly and kept in the ledger. Failures are findings, not embarrassments.
| Verdict | Claim & result | Provenance |
|---|---|---|
| edge · real-oos | The 80% price-range band is calibrated Walk-forward coverage holds at 78-81% across all six commodities on ~10y of real daily prices. validate_vol_real · 6/6 commodities @1w · n_eff≈497 · gate: tests/test_vol_range.py (locked regression) · 2026-06-07 | real-OOS |
| edge · real-oos | The 95% band reaches nominal coverage (fat tails) Empirical fat-tail quantiles (Phase 30c) lift real-data 95% coverage to 93-95%. validate_vol_real · post-30c quantiles · gate: tests/test_vol_range.py (locked regression) · 2026-06-07 | real-OOS |
| edge · real-oos | The vol forecast carries forward-looking information Forecast vol correlates 0.44-0.59 with realized forward vol at 1w — stronger on real data than on the synthetic seed. forecast_vol_correlation · n_eff≈497 · gate: tests/test_vol_range.py (locked regression) · 2026-06-07 | real-OOS |
| edge · real-oos | log-HAR beats EWMA as the vol point-forecast (now the default) Beats the EWMA incumbent on out-of-sample R² in 5/6 commodities at 1w; the default swap was re-validated skill-neutral. estimator_skill · ~10y real, 6 commodities · gate: docs/MODEL_DILIGENCE.md §ledger (30b gate) · 2026-06-08 | real-OOS |
| no edge · tested | Price-only directional models (MA, Holt) Decisive accuracy 45-57%, below a drift-aware naive baseline in all 36 commodity/horizon cells. No confidence gradient. validate_direction_real + validate_engine_oos · 36/36 cells · gate: docs/MODEL_DILIGENCE.md §ledger · 2026-06-07 | real-OOS |
| no edge · tested | factor_composite with real COT + storage features Real alt-data made the model worse, not better: ~6 SE worse than its own price-only variant on 2,958 non-overlapping weekly decisions. The gate failed; the claim is retired. validate_engine_oos --alt-only · n=2,958 @1w (n_eff=n) · gate: docs/PHASE_31_PLAN.md §31b (gate pre-registered before the run) · 2026-07-25 | real-OOS |
| no edge · tested | logreg_directional (trained, price-only by construction) No edge as a price-only model; the conditional alt-data extension never fired — its precondition (a promising 31b) was not met, so the simpler model stands. validate_engine_oos · all horizons/commodities · gate: docs/PHASE_31_PLAN.md §sequencing (31c closed) · 2026-07-25 | real-OOS |
| no edge · tested | Ensemble confidence tiers predict accuracy No reliable out-of-sample confidence gradient at any horizon; shipped reframed as down-weighting miscalibrated models, never as a calibrated confidence claim. walk-forward ensemble-calibration harness (26c) · gate: tests/test_ensemble_calibration.py (locked) · 2026-06-06 | real-OOS |
| no edge · tested | Curve slope (carry) predicts forward direction Time-series carry on NG+CL over ~5y: worse than drift at both horizons, adequately powered. Does not refute the cross-sectional literature — the curve archive accumulates toward that re-run. validate_d2_carry · n_eff≈116 @1m (above the power floor) · gate: docs/PHASE_D2_PLAN.md (gates + power rule pre-registered) · 2026-07-25 | real-OOS |
| insufficient-n | Storage-day surprises drive a tradeable reaction The premise itself is absent: release-day correlation with our consensus-free surprise is a tight zero on 559 events. The market's expectation is analyst consensus, which is not public data — we do not fabricate it. validate_d4_storage_event · n=559 events, 14y · gate: docs/PHASE_D4_PLAN.md (gates + power rule pre-registered) · 2026-07-25 | real-OOS |
| promising | Our vol forecast vs implied vol times the variance premium Premium-timing passes on 2 of 3 asset pairs (crude, equities) with consistent direction; on equities our forecast beats VIX as a realized-vol predictor by >2 SE. Not crowned: gold fails and the EWMA robustness arm fails — a design review is scheduled, and any surface needs its own gate first. validate_d1_vol_premium · ~11y weekly, n_eff≈131/pair · gate: docs/PHASE_D1_PLAN.md (gate + tie-breaker rule pre-registered) · 2026-07-25 | real-OOS |
| benched | Raw-variance HAR estimator Failed its gate against EWMA and blew up on real crude (R² −1.06). Code and tests kept; never wired. estimator_skill · real CL @1m · gate: docs/MODEL_DILIGENCE.md §ledger (30b gate) · 2026-06-08 | real-OOS |
| collecting | Weather degree-day forecast features Forecasts can only be validated against an archive of what was forecast at the time. Daily vintages persist automatically; validation re-enters after ~180 real vintage days spanning a winter. weather_forecast_vintages · immutable daily archive · gate: docs/PHASE_D5_PLAN.md (re-entry trigger recorded) · 2026-07-25 | collecting |
| collecting | Futures-curve history (for the carry re-run) Expired contract months vanish from free sources, so the historical curve is otherwise unreconstructible. Daily snapshots persist; the cross-sectional carry test re-enters at ~2 years of vintages. futures_curve_vintages · immutable daily archive · gate: docs/PHASE_D2_PLAN.md (re-entry trigger recorded) · 2026-07-25 | collecting |
| methodology | Per-model diagnostics (bias, Brier decomposition, drift) The measurement machinery, validated on seeded data where the ground truth is known by construction. services/model_diagnostics.py · gate: docs/MODEL_DILIGENCE.md §ledger (26a) · 2026-06-06 | synthetic |
| methodology | Backtest features are symbol-scoped and release-date gated A correctness fix locked as tests: positioning context is filtered per market and gated on publication dates, so backtests can never mix commodities or peek at unpublished data. tests/db/test_context_scoping.py + look-ahead property tests · gate: docs/PHASE_31_PLAN.md §31a.0 · 2026-07-24 | methodology |
| methodology | Storage surprise on real data is a labeled, consensus-free proxy EIA publishes no analyst consensus, so the storage surprise is this week's change vs the same-week 5-year norm — always labeled a proxy, never passed off as a survey. services/storage_features.py · locked label tests · gate: docs/PHASE_31_PLAN.md §31a.0 · 2026-07-24 | methodology |
| methodology | Desk skill-vs-luck verdicts refuse to crown noise A Wilson 95% interval against the coin-flip baseline: 'skill' only when the lower bound clears chance. Blind seeded desks correctly read as luck — the demonstration that the tool cannot be flattered. desk_calibration.py · honesty-locked e2e in CI · gate: tests/db/test_desk_skill_verdict_e2e.py · 2026-06-10 | methodology |
| unvalidated | Index (ES) and rates (ZN) engine configurations Hand-set plausible scales proving the loop runs cross-asset — explicitly not a predictive claim for equities or rates. The range band self-calibrates per series; the directional configs carry no edge claim. services/asset_config.py · golden byte-identity locks · gate: docs/MODEL_DILIGENCE.md §ledger (B5) · 2026-06-28 | real-in-sample |
Every row mirrors docs/MODEL_DILIGENCE.md — a CI test fails the build if this page and the claims ledger ever disagree. Verdicts come from pre-registered gates; sample sizes are quoted as non-overlapping effective counts. Descriptive research diligence, not advice.
Sample analyst · real engine · real prices. A fictional sample analyst’s decisions, scored by the same calibration engine against real market prices — not a real analyst track record. Notice her highest-conviction calls resolved far below what she claimed (~87% claimed → ~29% realized): overconfidence that stays invisible until the engine scores it. This is what calibration looks like in Goldeneye — point it at your desk and it scores your analysts the same way.
Loading…
Loading…
What we can’t test yet, and the day we can: weather features re-enter validation after ~180 archived days spanning a winter; the cross-sectional carry test at ~2 years of curve snapshots.
Each verdict re-runs from the repository (uv run --directory apps/api …):
- python -m seeds.validate_engine_oos --alt-only
- python -m seeds.validate_vol_real
- python -m seeds.validate_d1_vol_premium
- python -m seeds.validate_d2_carry
- python -m seeds.validate_d4_storage_event
Gates live in docs/PHASE_*_PLAN.md, committed before each run · the claims ledger is docs/MODEL_DILIGENCE.md · walk-forward safety is enforced by CI, including a cheating-model proof.