Pilot iteration 2: K=64-scenario action-complete H40 corpus with fresh held-out origins
Raising aligned chance scenarios from K=8 to K=64 (the only change from EX-20260820-afterstate-pilot-h40-29b8588a) lifts scenario-half label stability above the frozen 0.5 Spearman floor, and the unchanged distributional afterstate ranker then ranks fresh held-out whole-origin siblings non-inferiorly to exact fair D4 on top-1 accuracy, pairwise accuracy, and normalized regret against the scenario-mean H40 outcome ranking.
On this page
- Created
- Updated
The first iteration could not grade anything, because the answer key it graded against was too noisy to reproduce itself. This experiment changed exactly one thing: instead of averaging each column's future over 8 shared runs of luck, average it over 64. Everything else — the network, the positions, the 40-move continuation, the thresholds — was held fixed, and the held-out positions came from a fresh block of game seeds that had never been read.
Same two contestants as before. The candidate is the learned board-scorer that reads only a resolved board, the visible next disc, and the rise clock; the comparator is exact fair D4, the four-move search that is this repository's reference. Neither played a game: this is an offline test of who orders the columns better at a position. "Pass" required the split-half agreement of the answer key to reach 0.50, and then the network to come within two percentage points of the search on top-choice and head-to-head accuracy, in both halves of the held-out set.
What happened. More luck samples helped, in the direction and roughly the size the noise arithmetic predicted — split-half agreement rose from 0.246 to 0.446 — but 0.446 is still under the frozen 0.50 floor, so the verdict is inconclusive for a second time. The network's confidence intervals were well calibrated this time (0.807 against a target band), and it clearly beat the weak one-move player whose games produced its labels, but it trailed the four-move search in both halves. The record also flags that training stopped at epoch 15 of 20 when the GPU budget ran out, with the ranking loss still falling. No gameplay tier was opened.
The run was valid; the outcome was inconclusive, at the pilot (a small run to find bugs and project cost, not a strength claim) level. Of 7 preregistered checks, 3 passed and 4 failed.
K=64 improved label stability from 0.246 to 0.446 mean split-half Spearman on fresh held-out origins, but still below the frozen 0.5 floor, so the preregistered verdict is again inconclusive. Quantile calibration passed (0.807 coverage of the nominal 76.5% interval). Against the still-noisy target the model trailed fair D4 consistently in both half-folds (pooled top-1 0.342 vs 0.433; pairwise 0.637 vs 0.697; regret 0.303 vs 0.232) while beating its D1 continuation teacher (top-1 0.342 vs 0.299). Training stopped at epoch 15 of 20 on the GPU time budget with the ranking loss still decreasing. No gameplay tier was opened.
Technical recordLimitations recorded with the result
- Labels remain H40 returns under a phase-greedy D1 continuation (a weak fixed teacher).
- Model undertrained: 15 of 20 epochs at the GPU budget stop.
- Iteration-1 held-out roots were folded into training data here; the gate read only fresh origins 0x5da70100-0x5da7013f.
- Single machine profile; FP32 on the shared-memory iGPU.
Nothing about the theory was ruled in or out. The useful output is that the noise model held: sample count moved the key's reproducibility about as predicted, which is what justified paying for a much larger third iteration.
Technical recordThe registered protocol
- Hypothesis
- Raising aligned chance scenarios from K=8 to K=64 (the only change from EX-20260820-afterstate-pilot-h40-29b8588a) lifts scenario-half label stability above the frozen 0.5 Spearman floor, and the unchanged distributional afterstate ranker then ranks fresh held-out whole-origin siblings non-inferiorly to exact fair D4 on top-1 accuracy, pairwise accuracy, and normalized regret against the scenario-mean H40 outcome ranking.
- Arms
Arm Name Entry point Manifest Candidate distributional-afterstate-ranker approaches/afterstate-learning/distributional-afterstate/common.hpp– Comparator fair-d4 approaches/fair-expectimax/reference/fair-only-depth4.cppresearch/benchmarks/baselines-v1.json- Classification
- algorithmic
- Information boundary
- public-policy
- Benchmark tier
- PILOT
- Lifecycle
- completed
- Primary metric
- Fresh held-out whole-origin top-1 action accuracy of the model's scenario-averaged ranking against the scenario-mean H40 outcome ranking, compared with exact fair D4's top-1 accuracy on the same roots.
- Secondary metrics
- within-root pairwise accuracy and normalized regret versus the same target, model and fair D4 and fair D1
- top-2 accuracy
- quantile calibration (nominal 76.5% outer-interval coverage) on held-out afterstates
- label stability: per-root Spearman correlation of action rankings between independent scenario halves (32 vs 32)
- action completeness (fraction of legal siblings with labels; must be 1.0)
- logical work and wall time per corpus root; training wall time and peak shared memory
- Statistical unit
- whole-origin
- Uncertainty method
- Two deterministic held-out half-folds by origin-game hash (mix32 of the origin seed); a metric counts only if it moves the same direction in both halves. No complete-game confidence intervals are produced at this tier.
- Data role
- public-development
- Seed leases
SL-20260820T083000Z-5da70000
- Dataset references
DS-20260820-afterstate-h40-k8-3f2f6d29
- Whole-origin split
- yes
- Reuse disclosure
- Training corpus: seeds 0x5da70000-0x5da700ff (the same 256 origins as iteration 1, including its former held-out roots, which are now training-only and are never evaluated again). Evaluation corpus: fresh seeds 0x5da70100-0x5da7013f (64 origins), all held-out, read for the first time by this experiment's single gate evaluation. Iteration 1's gate outcome was inconclusive on label stability; this experiment changes only the scenario count and the held-out origins. No protected or final data is read.
- Pass criteria
- Corpus is successor-closed: every non-trivial harvested root (>=2 legal actions) has labels for every legal sibling under all 64 scenarios.
- Label stability floor: mean per-root Spearman correlation between scenario-half (32 vs 32) action rankings >= 0.5 on held-out roots; below this the outcome is inconclusive (labels too noisy), not a pass or a theory rejection.
- Model top-1 accuracy >= fair D4 top-1 accuracy - 0.02 on each held-out half-fold.
- Model pairwise accuracy >= fair D4 pairwise accuracy - 0.02 on each held-out half-fold.
- Model normalized regret <= fair D4 normalized regret + 0.02 on each held-out half-fold.
- Nominal 76.5% quantile interval coverage on held-out afterstates within [0.66, 0.87].
- All CHECK-tier mechanics, legality, determinism, reflection, information-boundary, and resource-bound tests pass before any label is inspected.
- On pass
- Register a successor SCREEN experiment: 32 paired public-development games of the frozen 1-ply chance-averaged afterstate policy with exact fair-D4 fallback versus unchanged fair D4. The candidate, corpus, and thresholds are not retuned on the held-out fold first.
- On fail
- Record a valid negative (or inconclusive, if the stability floor fails again) result for this exact configuration. If stability passes but the model trails D4, the theory is not supported as tested; keep both corpus manifests for diagnosis and open no gameplay tier.
- Gate fixed before controlled data
- yes
- Resources
Wall seconds 36000 CPU threads 24 Max host bytes 34359738368 Max GPU bytes 21474836480 GPU devices gfx1151 - Stop conditions
- Stop on any rules, information-boundary, legality, determinism, or parity failure.
- Stop corpus generation after 2 hours wall or 320 harvest games, whichever comes first.
- Stop fair-D4 comparator labeling after 3 hours wall.
- Stop training after 2 hours GPU wall or 20 epochs, whichever comes first.
- Stop immediately on out-of-memory or on host available memory falling below 16 GiB.
- Stop on any read of a seed outside the named lease.
- Expected artifacts
runs/<run-id>/corpus-a/*.ndjson K=64 training corpus with content manifestruns/<run-id>/corpus-b/*.ndjson K=64 fresh held-out corpus with content manifestruns/<run-id>/d4-labels-b/*.tsv comparator rankings for held-out rootsruns/<run-id>/model/*.pt frozen model bytes and training logruns/<run-id>/gate/gate-report.json with all primary and secondary metricsresearch/datasets/<dataset-id>.json content-addressed corpus manifestsresearch/results/<result-id>.json scientific assessment
- Amendments
Timestamp Before controlled data Reason 2026-08-20T14:30:00Z no lifecycle advanced preregistered -> completed after the run and result records were written; protocol content otherwise unchanged.
Technical recordResults recorded against this protocol
K=64 improved label stability from 0.246 to 0.446 mean split-half Spearman on fresh held-out origins, but still below the frozen 0.5 floor, so the preregistered verdict is again inconclusive. Quantile calibration passed (0.807 coverage of the nominal 76.5% interval). Against the still-noisy target the model trailed fair D4 consistently in both half-folds (pooled top-1 0.342 vs 0.433; pairwise 0.637 vs 0.697; regret 0.303 vs 0.232) while beating its D1 continuation teacher (top-1 0.342 vs 0.299). Training stopped at epoch 15 of 20 on the GPU time budget with the ranking loss still decreasing. No gameplay tier was opened.
- ✓Successor-closed corpus (all legal siblings x 64 scenarios) — observed: 6,167,936 rows over 14,228 roots across both corpora; completeness 1.0
- ✕Label stability floor: mean scenario-half (32 vs 32) Spearman >= 0.5 — observed: 0.4462 on 2,750 fresh held-out roots; inconclusive per the frozen rule
- ✕Model top-1 >= D4 top-1 - 0.02 on each half-fold — observed: half1 0.3401 vs 0.4365; half2 0.3442 vs 0.4299 (moot given the stability failure)
- ✕Model pairwise >= D4 pairwise - 0.02 on each half-fold — observed: half1 0.6341 vs 0.6970; half2 0.6391 vs 0.6974 (moot)
- ✕Model regret <= D4 regret + 0.02 on each half-fold — observed: half1 0.3031 vs 0.2388; half2 0.3028 vs 0.2247 (moot)
- ✓Nominal 76.5% quantile interval coverage within [0.66, 0.87] — observed: 0.8069
- ✓All CHECK-tier tests pass before any label is inspected — observed: SELFTEST OK (9 checks) after the K-parameterization change
Technical recordRecorded metrics
- Labels remain H40 returns under a phase-greedy D1 continuation (a weak fixed teacher).
- Model undertrained: 15 of 20 epochs at the GPU budget stop.
- Iteration-1 held-out roots were folded into training data here; the gate read only fresh origins 0x5da70100-0x5da7013f.
- Single machine profile; FP32 on the shared-memory iGPU.
Agent contextHow to extend this record
To add a reader-facing explanation, write web/content/research/EX-20260820-afterstate-pilot-h40-k64-d7a9faf5.mdx; it renders above this record on the next request. The registered protocol itself is in the technical record above.
Record file: research/experiments/EX-20260820-afterstate-pilot-h40-k64-d7a9faf5.json, validated against research/schemas/experiment-v1.schema.json. Protocol hash: 769e418bc91f4d5e6a9ae96f04da80bdbad3ef32142b56e86f128fec50105d5b.
Registered by OpenCode / moonshotai/Kimi-K3.