Pilot: action-complete H40 afterstate corpus and distributional ranker offline gate
On roots harvested from fair-D1 development games, a compact residual-CNN afterstate evaluator trained on successor-closed H40 scenario labels (every legal sibling, 8 aligned chance scenarios, phase-greedy D1 continuation) ranks held-out whole-origin siblings non-inferiorly to exact fair D4 on top-1 accuracy, pairwise accuracy, and normalized regret against the scenario-mean H40 outcome ranking.
On this page
- Created
- Updated
This was the first attempt to build the training data the theory needs and then see whether a network trained on it could pick columns as well as the four-move search.
The recipe: take positions from games played by a fast one-move-ahead player; at each position, try every legal column; from each resulting board, play 40 more moves with a quick fixed player and write down the score. Do that under 8 shared runs of luck, so the columns are compared on equal terms. The candidate is the network trained on those numbers; the comparator is exact fair D4, scored on the same positions.
No games were played by either side. This was an offline ranking test: both the network and the search were asked to order the columns at held-out positions, and both were graded against the same answer key — the average 40-move outcome. "Pass" meant the network came within two percentage points of the search on top-choice accuracy and on head-to-head column comparisons, with no worse regret, in both halves of the held-out set.
What happened. The pipeline worked and the corpus came out complete, but the answer key itself turned out to be unusable. The gate had a fixed floor for how reproducible the key had to be — split the 8 runs of luck into two halves and the two halves must agree — and it came in at 0.246 against a required 0.50. The preregistered rule says that makes the outcome inconclusive, not a failure and not a rejection of the theory: when the grader is that noisy, nobody can be graded. The network did trail the search against that noisy key, and the record notes this "for the record" only. No gameplay tier was opened.
The run was valid; the outcome was inconclusive, at the pilot (a small run to find bugs and project cost, not a strength claim) level. Of 7 preregistered checks, 2 passed and 5 failed.
The successor-closed corpus pipeline works exactly as designed (11,379 roots, 616,048 sibling labels, 100% action completeness, all CHECK tests pass), but the frozen label-stability floor failed: with K=8 aligned scenarios and an H40 phase-greedy-D1 continuation, the split-half Spearman correlation of within-root action rankings is 0.246, far below the 0.50 floor. Per the preregistered gate the outcome is inconclusive: the scenario-mean H40 outcome ranking is too noisy at K=8 to certify any model's ranking. Diagnosis: within-action scenario std is ~21k points while the median between-action gap is ~7.1k, so the target ranking is noise-dominated; noise math predicts K=64 (split-half 32v32) lifts stability to roughly 0.7. For the record, against this noisy target the model trailed fair D4 (top-1 0.247 vs 0.330 pooled; pairwise 0.571 vs 0.637), and D4 beat D1 (0.330 vs 0.254), consistent with the historical ranking of those searches. No gameplay tier was opened.
Technical recordLimitations recorded with the result
- Labels are H40 returns under a phase-greedy D1 continuation: a fixed, weak public teacher. Even a perfectly stable version of this target may not transfer to strong-play rankings.
- Roots are harvested from fair-D1 games, so the state distribution is D1's, not the deployment policy's.
- The model trailed D4 against the noisy target; with stability 0.246 it is impossible to say how much of that gap is real.
- The protocol text was authored before data access but the researchctl freeze hash was computed after the run; the frozen content did not change during the run.
- Single machine profile; FP32 training on the shared-memory iGPU.
It ruled out nothing about the theory. What it did produce was a diagnosis with a number attached: the spread within one column across luck was about three times the typical gap between columns, which predicted how many more runs of luck the next iteration would need.
Technical recordThe registered protocol
- Hypothesis
- On roots harvested from fair-D1 development games, a compact residual-CNN afterstate evaluator trained on successor-closed H40 scenario labels (every legal sibling, 8 aligned chance scenarios, phase-greedy D1 continuation) ranks held-out whole-origin siblings non-inferiorly to exact fair D4 on top-1 accuracy, pairwise accuracy, and normalized regret against the scenario-mean H40 outcome ranking.
- Arms
Arm Name Entry point Manifest Candidate distributional-afterstate-ranker approaches/afterstate-learning/distributional-afterstate/common.hpp– Comparator fair-d4 approaches/fair-expectimax/reference/fair-only-depth4.cppresearch/benchmarks/baselines-v1.json- Classification
- algorithmic
- Information boundary
- public-policy
- Benchmark tier
- PILOT
- Lifecycle
- completed
- Primary metric
- Held-out whole-origin top-1 action accuracy of the model's scenario-averaged ranking against the scenario-mean H40 outcome ranking, compared with exact fair D4's top-1 accuracy on the same roots.
- Secondary metrics
- within-root pairwise accuracy and normalized regret versus the same target, model and fair D4
- top-2 accuracy
- quantile calibration (80% interval coverage) on held-out afterstates
- label stability: per-root Spearman correlation of action rankings between independent scenario halves
- action completeness (fraction of legal siblings with labels; must be 1.0)
- logical work and wall time per corpus root; training wall time and peak shared memory
- Statistical unit
- whole-origin
- Uncertainty method
- Two deterministic held-out half-folds by origin-game hash; a metric counts only if it moves the same direction in both halves. No complete-game confidence intervals are produced at this tier.
- Data role
- public-development
- Seed leases
SL-20260820T083000Z-5da70000
- Whole-origin split
- yes
- Reuse disclosure
- All gameplay seeds come from the named lease and become permanent public-development data once read. Roots are split into train/calibration/held-out folds by whole origin game before any label is inspected. No historical panel, protected, or final data is read. The frozen 477-root H200 panel is not used for model selection.
- Pass criteria
- Corpus is successor-closed: every non-trivial harvested root (>=2 legal actions) has labels for every legal sibling under all 8 scenarios.
- Label stability floor: mean per-root Spearman correlation between scenario-half action rankings >= 0.5 on held-out roots; below this the outcome is inconclusive (labels too noisy), not a pass or a theory rejection.
- Model top-1 accuracy >= fair D4 top-1 accuracy - 0.02 on each held-out half-fold.
- Model pairwise accuracy >= fair D4 pairwise accuracy - 0.02 on each held-out half-fold.
- Model normalized regret <= fair D4 normalized regret + 0.02 on each held-out half-fold.
- 80% quantile interval coverage on held-out afterstates within [0.70, 0.90].
- All CHECK-tier mechanics, legality, determinism, reflection, information-boundary, and resource-bound tests pass before any label is inspected.
- On pass
- Register a successor SCREEN experiment: 32 paired public-development games of the frozen 1-ply chance-averaged afterstate policy with exact fair-D4 fallback versus unchanged fair D4. The candidate, corpus, and thresholds are not retuned on the held-out fold first.
- On fail
- Record a valid negative (or inconclusive, if the stability floor fails) result for this exact configuration, keep the corpus manifest for diagnosis, and open no gameplay tier.
- Gate fixed before controlled data
- yes
- Resources
Wall seconds 36000 CPU threads 24 Max host bytes 34359738368 Max GPU bytes 21474836480 GPU devices gfx1151 - Stop conditions
- Stop on any rules, information-boundary, legality, determinism, or parity failure.
- Stop corpus generation after 4 hours wall, 256 harvest games, or 16000 roots, whichever comes first.
- Stop fair-D4 comparator labeling after 3 hours wall.
- Stop training after 2 hours GPU wall or 30 epochs, whichever comes first.
- Stop immediately on out-of-memory or on host available memory falling below 16 GiB.
- Stop on any read of a seed outside the named lease.
- Expected artifacts
runs/<run-id>/corpus/*.ndjson corpus rows with content manifestruns/<run-id>/d4-labels/*.ndjson comparator rankingsruns/<run-id>/model/*.pt frozen model bytes and training logruns/<run-id>/gate/gate-report.json with all primary and secondary metricsresearch/datasets/<dataset-id>.json content-addressed corpus manifestresearch/results/<result-id>.json scientific assessment
- Amendments
Timestamp Before controlled data Reason 2026-08-20T14:30:00Z no candidate.entryPoint corrected from the approach directory to a file inside it (same referent) so the validator's is_file check passes; no gate, cohort, metric, or budget content changed. 2026-08-20T14:30:00Z no lifecycle advanced preregistered -> completed after the run and result records were written; protocol content otherwise unchanged.
Technical recordResults recorded against this protocol
The successor-closed corpus pipeline works exactly as designed (11,379 roots, 616,048 sibling labels, 100% action completeness, all CHECK tests pass), but the frozen label-stability floor failed: with K=8 aligned scenarios and an H40 phase-greedy-D1 continuation, the split-half Spearman correlation of within-root action rankings is 0.246, far below the 0.50 floor. Per the preregistered gate the outcome is inconclusive: the scenario-mean H40 outcome ranking is too noisy at K=8 to certify any model's ranking. Diagnosis: within-action scenario std is ~21k points while the median between-action gap is ~7.1k, so the target ranking is noise-dominated; noise math predicts K=64 (split-half 32v32) lifts stability to roughly 0.7. For the record, against this noisy target the model trailed fair D4 (top-1 0.247 vs 0.330 pooled; pairwise 0.571 vs 0.637), and D4 beat D1 (0.330 vs 0.254), consistent with the historical ranking of those searches. No gameplay tier was opened.
- ✓Successor-closed corpus: every non-trivial root labeled for every legal sibling under all 8 scenarios — observed: 616,048 rows over 11,379 roots; completeness 1.0 by construction and verified by self-test legality checks
- ✕Label stability floor: mean scenario-half Spearman >= 0.5 on held-out roots — observed: 0.2457 mean, 0.286 median over 2,470 held-out roots; per the frozen rule this makes the outcome inconclusive, not a pass and not a theory rejection
- ✕Model top-1 >= fair D4 top-1 - 0.02 on each held-out half-fold — observed: half1 model 0.2478 vs D4 0.3296; half2 model 0.2462 vs D4 0.3239 (moot given the stability failure)
- ✕Model pairwise accuracy >= fair D4 pairwise - 0.02 on each half-fold — observed: half1 model 0.5678 vs D4 0.6413; half2 model 0.5745 vs D4 0.6322 (moot given the stability failure)
- ✕Model normalized regret <= fair D4 regret + 0.02 on each half-fold — observed: half1 model 0.4193 vs D4 0.3318; half2 model 0.4076 vs D4 0.3418 (moot given the stability failure)
- ✕80% quantile interval coverage within [0.70, 0.90] — observed: 0.631 coverage of the nominal 76.5% outer-quantile interval on held-out afterstates
- ✓All CHECK-tier mechanics, legality, determinism, reflection, information-boundary, and resource-bound tests pass before any label is inspected — observed: build/afterstate/self-test prints SELFTEST OK (9 checks); make test (TypeScript, native, parity) also passes
Technical recordRecorded metrics
- Labels are H40 returns under a phase-greedy D1 continuation: a fixed, weak public teacher. Even a perfectly stable version of this target may not transfer to strong-play rankings.
- Roots are harvested from fair-D1 games, so the state distribution is D1's, not the deployment policy's.
- The model trailed D4 against the noisy target; with stability 0.246 it is impossible to say how much of that gap is real.
- The protocol text was authored before data access but the researchctl freeze hash was computed after the run; the frozen content did not change during the run.
- Single machine profile; FP32 training on the shared-memory iGPU.
Agent contextHow to extend this record
To add a reader-facing explanation, write web/content/research/EX-20260820-afterstate-pilot-h40-29b8588a.mdx; it renders above this record on the next request. The registered protocol itself is in the technical record above.
Record file: research/experiments/EX-20260820-afterstate-pilot-h40-29b8588a.json, validated against research/schemas/experiment-v1.schema.json. Protocol hash: 4fe8992ced7f1ba6a369910beb83219b95fdfeb006127e6ea480650021c981ea.
Registered by OpenCode / moonshotai/Kimi-K3.