Pilot iteration 3: K=256-scenario corpus, fresh held-out origins, full training
With K=256 aligned scenarios (per-action standard error ~1.3k points against a ~7k median sibling gap), the scenario-half label stability on decisive roots passes the 0.5 floor, and a fully trained distributional afterstate ranker ranks fresh held-out whole-origin siblings non-inferiorly to exact fair D4.
On this page
- Created
- Updated
Two earlier iterations could not deliver a verdict because their answer key was not reproducible enough to grade anyone. This one raised the shared runs of luck per column to 256, trained the network properly rather than briefly, and — the only diagnostic change — measured the key's reproducibility on decisive positions, ones where the columns genuinely differ by more than 20,000 points. That 20,000 cut was fixed from earlier training data before any of this experiment's data was read.
The candidate is the same learned board-scorer, reading only a resolved board, the visible next disc, and the rise clock. The comparator is exact fair D4. Again no games were played: both were asked to order the legal columns at fresh held-out positions and graded against the average 40-move outcome. "Pass" meant the key first proved reproducible, and then the network landed within two percentage points of the search on top-choice accuracy, on head-to-head column comparisons, and on regret — in both halves of the held-out set. Passing would have opened a 32-game paired SCREEN match against the search.
What happened. The key passed decisively for the first time (0.818 on decisive positions, 0.638 overall, against a 0.50 floor), so for once a real verdict was possible — and the verdict is a valid negative. The network trailed the four-move search on every preregistered comparison in both halves: top choice 0.424 against 0.499, head-to-head 0.685 against 0.737, regret 0.241 against 0.178. On the 411 decisive positions the gap narrowed but did not close. The record is careful about what this is: the model is a calibrated, genuinely long-horizon evaluator that comfortably beats the weak player whose games trained it — it is simply not a replacement for four-ply search as the thing that picks the move.
The run was valid; the outcome was fail, at the pilot (a small run to find bugs and project cost, not a strength claim) level. Of 7 preregistered checks, 4 passed and 3 failed.
With K=256 aligned scenarios the label-stability floor passed decisively (decisive-root split-half Spearman 0.818, unconditioned 0.638, both >= 0.5), so for the first time in this line the ranking target is stable enough to certify a verdict. The verdict is a valid negative for the direct-play configuration: the distributional afterstate ranker trailed exact fair D4 on every preregistered comparison in both fresh held-out half-folds (pooled top-1 0.424 vs 0.499; pairwise 0.685 vs 0.737; regret 0.241 vs 0.178). On the 411 decisive roots (spread > 20k) the gap narrowed but persisted (top-1 0.526 vs 0.584; regret 0.082 vs 0.064). The model clearly beat its D1 continuation teacher (top-1 0.424 vs 0.319) and its quantile head is calibrated (0.864 coverage). Conclusion: a compact action-free afterstate network trained on successor-closed H40 D1-continuation labels is a viable, calibrated long-horizon evaluator, but it is NOT a viable direct replacement for 4-ply expectimax; any future use must be as a companion signal inside search, and the same ranking gate applies. No gameplay tier was opened.
Technical recordLimitations recorded with the result
- Labels are H40 returns under a phase-greedy D1 continuation (a weak fixed teacher); a stronger-teacher corpus was not tested and might shift the verdict.
- Model undertrained: 11 of 20 epochs at the 2h GPU budget stop; ranking loss was still decreasing.
- Compact 3.4M-parameter ResNet; capacity and input resolution (single afterstate, no root context) were not scaled.
- Roots are harvested from fair-D1 games; deployment-distribution roots were not represented.
- Single machine profile; FP32 on the shared-memory iGPU.
What it did not rule out: this exact configuration was tested, and the record names two untested confounds — the model stopped at 11 of 20 epochs with its ranking loss still falling, and its labels came from a deliberately weak one-move continuation player. The stated conclusion is that any future use of this model belongs inside a search as a companion signal, under the same ranking gate. No gameplay tier was opened.
Technical recordThe registered protocol
- Hypothesis
- With K=256 aligned scenarios (per-action standard error ~1.3k points against a ~7k median sibling gap), the scenario-half label stability on decisive roots passes the 0.5 floor, and a fully trained distributional afterstate ranker ranks fresh held-out whole-origin siblings non-inferiorly to exact fair D4. Iterations 1-2 showed K=8/K=64 stability of 0.246/0.446 and an undertrained model; this iteration changes scenario count and training completeness, and refines only the stability diagnostic to condition on decisive roots.
- Arms
Arm Name Entry point Manifest Candidate distributional-afterstate-ranker approaches/afterstate-learning/distributional-afterstate/common.hpp– Comparator fair-d4 approaches/fair-expectimax/reference/fair-only-depth4.cppresearch/benchmarks/baselines-v1.json- Classification
- algorithmic
- Information boundary
- public-policy
- Benchmark tier
- PILOT
- Lifecycle
- completed
- Primary metric
- Fresh held-out whole-origin top-1 action accuracy of the model's scenario-averaged ranking against the scenario-mean H40 outcome ranking, compared with exact fair D4's top-1 accuracy on the same roots.
- Secondary metrics
- within-root pairwise accuracy and normalized regret versus the same target, model and fair D4 and fair D1
- top-2 accuracy
- quantile calibration (nominal 76.5% outer-interval coverage) on held-out afterstates
- label stability: per-root Spearman correlation of action rankings between independent scenario halves (128 vs 128), reported unconditioned and conditioned on decisive roots (full-K scenario-mean spread > 20,000 points; the 20,000 threshold was fixed from iteration-1 training-role data before this experiment's data was read)
- the same ranking metrics conditioned on decisive roots, as secondary diagnostics
- action completeness (fraction of legal siblings with labels; must be 1.0)
- logical work and wall time per corpus root; training wall time and peak shared memory
- Statistical unit
- whole-origin
- Uncertainty method
- Two deterministic held-out half-folds by origin-game hash (mix32 of the origin seed); a metric counts only if it moves the same direction in both halves. No complete-game confidence intervals are produced at this tier.
- Data role
- public-development
- Seed leases
SL-20260820T083000Z-5da70000
- Dataset references
DS-20260820-afterstate-h40-k8-3f2f6d29
- Whole-origin split
- yes
- Reuse disclosure
- Training corpus: seeds 0x5da70000-0x5da700ff (the same 256 origins as iterations 1-2; all folds are training-role and are never evaluated again). Evaluation corpus: fresh seeds 0x5da70200-0x5da7023f (64 origins), all held-out, read for the first time by this experiment's single gate evaluation; the 0x5da70100-0x5da7013f block was read by iteration 2 and is not reused here. No protected or final data is read.
- Pass criteria
- Corpus is successor-closed: every non-trivial harvested root (>=2 legal actions) has labels for every legal sibling under all 256 scenarios.
- Label stability floor: mean per-root Spearman correlation between scenario-half (128 vs 128) action rankings >= 0.5 on decisive held-out roots (spread > 20,000); below this the outcome is inconclusive, not a pass or a theory rejection.
- Model top-1 accuracy >= fair D4 top-1 accuracy - 0.02 on each held-out half-fold (all roots).
- Model pairwise accuracy >= fair D4 pairwise accuracy - 0.02 on each held-out half-fold (all roots).
- Model normalized regret <= fair D4 normalized regret + 0.02 on each held-out half-fold (all roots).
- Nominal 76.5% quantile interval coverage on held-out afterstates within [0.66, 0.87].
- All CHECK-tier mechanics, legality, determinism, reflection, information-boundary, and resource-bound tests pass before any label is inspected.
- On pass
- Register a successor SCREEN experiment: 32 paired public-development games of the frozen 1-ply chance-averaged afterstate policy with exact fair-D4 fallback versus unchanged fair D4. The candidate, corpus, and thresholds are not retuned on the held-out fold first.
- On fail
- Record a valid negative (or inconclusive, if the stability floor fails) result for this exact configuration. If stability passes but the model trails D4, the theory is not supported as tested for the direct-play configuration, and the appropriate conclusion is that the afterstate model may serve only as a companion signal inside a search, not as a D4 replacement. Open no gameplay tier.
- Gate fixed before controlled data
- yes
- Resources
Wall seconds 21600 CPU threads 24 Max host bytes 34359738368 Max GPU bytes 21474836480 GPU devices gfx1151 - Stop conditions
- Stop on any rules, information-boundary, legality, determinism, or parity failure.
- Stop corpus generation after 3 hours wall or 320 harvest games, whichever comes first.
- Stop fair-D4 comparator labeling after 3 hours wall.
- Stop training after 2 hours GPU wall or 20 epochs, whichever comes first.
- Stop immediately on out-of-memory or on host available memory falling below 16 GiB.
- Stop on any read of a seed outside the named lease.
- Expected artifacts
runs/<run-id>/corpus-a/*.ndjson K=256 training corpus with content manifestruns/<run-id>/corpus-c/*.ndjson K=256 fresh held-out corpus with content manifestruns/<run-id>/d4-labels-c/*.tsv comparator rankings for held-out rootsruns/<run-id>/model/*.pt frozen model bytes and training logruns/<run-id>/gate/gate-report.json with all primary and secondary metricsresearch/datasets/<dataset-id>.json content-addressed corpus manifestsresearch/results/<result-id>.json scientific assessment
- Amendments
Timestamp Before controlled data Reason 2026-08-20T14:30:00Z no lifecycle advanced preregistered -> completed after the run and result records were written; protocol content otherwise unchanged.
Technical recordResults recorded against this protocol
With K=256 aligned scenarios the label-stability floor passed decisively (decisive-root split-half Spearman 0.818, unconditioned 0.638, both >= 0.5), so for the first time in this line the ranking target is stable enough to certify a verdict. The verdict is a valid negative for the direct-play configuration: the distributional afterstate ranker trailed exact fair D4 on every preregistered comparison in both fresh held-out half-folds (pooled top-1 0.424 vs 0.499; pairwise 0.685 vs 0.737; regret 0.241 vs 0.178). On the 411 decisive roots (spread > 20k) the gap narrowed but persisted (top-1 0.526 vs 0.584; regret 0.082 vs 0.064). The model clearly beat its D1 continuation teacher (top-1 0.424 vs 0.319) and its quantile head is calibrated (0.864 coverage). Conclusion: a compact action-free afterstate network trained on successor-closed H40 D1-continuation labels is a viable, calibrated long-horizon evaluator, but it is NOT a viable direct replacement for 4-ply expectimax; any future use must be as a companion signal inside search, and the same ranking gate applies. No gameplay tier was opened.
- ✓Successor-closed corpus (all legal siblings x 256 scenarios) — observed: 24,270,592 rows over 14,009 roots across both corpora; completeness 1.0
- ✓Label stability floor: decisive-root scenario-half (128 vs 128) Spearman >= 0.5 — observed: 0.8181 on 411 decisive roots; unconditioned 0.6379 on 2,523 roots
- ✕Model top-1 >= D4 top-1 - 0.02 on each half-fold — observed: half1 0.3981 vs 0.4776; half2 0.4439 vs 0.5141
- ✕Model pairwise >= D4 pairwise - 0.02 on each half-fold — observed: half1 0.6828 vs 0.7280; half2 0.6868 vs 0.7429
- ✕Model regret <= D4 regret + 0.02 on each half-fold — observed: half1 0.2496 vs 0.1870; half2 0.2344 vs 0.1720
- ✓Nominal 76.5% quantile interval coverage within [0.66, 0.87] — observed: 0.8635
- ✓All CHECK-tier tests pass before any label is inspected — observed: SELFTEST OK (9 checks)
Technical recordRecorded metrics
- Labels are H40 returns under a phase-greedy D1 continuation (a weak fixed teacher); a stronger-teacher corpus was not tested and might shift the verdict.
- Model undertrained: 11 of 20 epochs at the 2h GPU budget stop; ranking loss was still decreasing.
- Compact 3.4M-parameter ResNet; capacity and input resolution (single afterstate, no root context) were not scaled.
- Roots are harvested from fair-D1 games; deployment-distribution roots were not represented.
- Single machine profile; FP32 on the shared-memory iGPU.
Agent contextHow to extend this record
To add a reader-facing explanation, write web/content/research/EX-20260820-afterstate-pilot-h40-k256-65a05477.mdx; it renders above this record on the next request. The registered protocol itself is in the technical record above.
Record file: research/experiments/EX-20260820-afterstate-pilot-h40-k256-65a05477.json, validated against research/schemas/experiment-v1.schema.json. Protocol hash: 325a96f742382e18dcee7d105d343f56047b98948ee8adaba19469b1ddd29efa.
Registered by OpenCode / moonshotai/Kimi-K3.