The best dependable policy found so far is fair depth-4
expectimax — a four-move look-ahead that treats
the game's luck honestly. Even with its best chance model, its measured
means sit under half of that target. This page shows where the gap is, what has
been measured, and which directions the evidence has closed.
How score responds to search depth, under each of the two chance models:
Mean score against search depth under the five-stratum and seven-stratum chance models. Hover or focus a point for its value, bounds, cohort size and source record.Source data
All arms play the same 64-seed development cohort 0xa51d1000-0xa51d103f under corrected 17,000-point Hardcore scoring, except the depth-5 seven-stratum arm, which was stopped by decision after the first 32 seeds (0xa51d1000-0xa51d101f) and is drawn as its own one-point series so the two sample sizes are never joined by a line. Depth 2 was run only at seven strata. No confidence band is drawn on means; the paired contrasts with their bounds are in the companion bar figure. Evidence tier: development / public-development. The historical eight-game D3/D4 cohorts used 7,000-point scoring and are not shown.
Spec: web/content/figures/score-vs-depth.json · 3 source records
The same factorial as paired contrasts, which is the form the conclusions rest on:
Paired contrasts from the same factorial: seven strata minus five strata, and the fifth ply minus the fourth, with one-sided 95% lower bounds. The source table under each figure names the record every point was copied from.Source data
Whisker is the one-sided 95% whole-game bootstrap lower bound; the upper end of the whisker is the point estimate itself (no upper bound is drawn). A contrast is significant when its lower bound clears zero: both stratum contrasts (7 minus 5 strata) do; neither depth contrast (5 minus 4 plies) does, and each sits below its own detection floor (47,052 at n=64, 107,988 at n=32). Cohort 0xa51d1000-0xa51d103f; the n=32 contrasts cover its first 32 seeds. Evidence tier: public-development.
Spec: web/content/figures/strata-5-vs-7.json · 1 source record
The simulator and reference searches are mature enough to support reproducible
research, but the strategy problem is unsolved. Corrected-score fair depth-4
expectimax is the strongest dependable reference found so far.
Its average score across 64 games is 308,296 points, far below the goal of a
one-million-point average.
Exploratory work on 2026-08-20/21 reproduced that reference on fresh seeds and
completed the depth-by-chance-resolution factorial. Two results from it strongly
suggest what to do next:
The fourth ply is worth a large, significant gain — but only with an exact
chance model. With the approximate five-stratum model the same fourth ply
is worth slightly less than nothing. Chance resolution does not merely add
points; it changes the sign of the depth gradient.
The fifth ply is not measurable by this design. Both depth-5 contrasts
sit far inside their own detection floors, and the seven-stratum estimate
changed sign when the cohort grew from 16 games to 32, so the supported
statement is that any depth-5 effect is smaller than a 64-game paired cohort
can resolve — not that it is zero. What can be said is economic: even the
optimistic estimate buys its points at 35.6x the work per move.
A 64-game paired cohort in this game cannot see an effect below roughly
50,000 points. Every null result in the depth factorial is below its own
detection floor and is therefore a non-measurement rather than evidence of no
effect
(RS-20260821T205102Z-d89df4b5).
Choose experiments whose predicted effect exceeds the floor, or find a
lower-variance estimator than complete games.
A fourth day, 2026-08-23, went to learned models and label economics — the
reveal-construction probe in live play, the optimistic-state D0 gate, the
leaf-cost NNUE student C0, and the P-SOL G0 label-semantics guardrail — and
closed each of those four directions as tested; the research log for that
day tells the story, and the
closed-directions table below carries the headline numbers.
No candidate has qualified for the protected validation protocol. The frozen
record marks both the protected and one-shot final cohorts as unopened. See the
research status evidence for details.
How the research progressed
Seven eras, one sentence each; every configuration and outcome below is tabled
row by row in the experiment index.
A trustworthy simulator. TypeScript and native Hardcore rules were
aligned and a scoring audit corrected the five-move level award from 7,000
to 17,000; results made with 7,000-point scoring remain historical
Sequence-scored evidence.
Hand-written policies and shallow lookahead provided fast baselines and
showed that immediate score alone is a poor guide.
Fair expectimax became the reference: completed fair D4 consistently
improved on D3 in the corrected-score small cohorts, while deeper variants
were not automatically better.
Learning public-state values repeatedly hit sibling extrapolation: a
model learned the outcome of the action that was played, then deployment
asked it to rank actions it had not observed equally well.
Oracle and long-outcome teachers found predictive signal, but students
failed held-out sibling ranking or were too slow; oracle strength is an
upper-bound teaching signal, not a legal policy result.
Constructive cycles and explicit reservoirs look useful as features or
options, but no tested controller displaced D4.
Offline policy improvement around D4 on a locked panel of 477 public
roots underperformed or barely overrode D4; neither ranker justified a
gameplay run.
The failure mode that recurs through eras 4-7 is easier drawn than described:
The era-4 failure mode: training labels the played action's successor, deployment asks the model to rank all seven siblings — six of which it never observed equally.Source
diagram-sibling-extrapolation.svg — source and reading guide
Mechanism diagram D1 of runs/RUN-20260823T191900Z-b9f8f80d/kimi-k3-figure-plan.md
("Diagrams (mechanisms, not charts)"). Hand-written, self-contained SVG; no measured
numbers are drawn.
What it explains
The repeated failure mode named in docs/research/status.md §4 ("Learning
public-state values"): a model is trained on the outcome of the action that was
actually played, then deployment asks it to rank all seven legal columns — six
of which it never observed equally. This is failure class (iii) sibling
coverage / within-root discrimination of
docs/exploratory/audit-05-optimistic-curriculum.md §4, the largest class in the
census: 6 of 17 learned-policy experiments (experiments 5, 9, 11, 13, 14, 17).
Element-by-element
Root board (both panels): a small invented 7×7 position, drawn in engine
orientation (row 0 at top, columns 1–7 left to right). Blue circles are
numbered discs; solid gray circles are covered discs; the gray disc with a
crack mark is cracked (one hit taken). "next disc: 2 · 3 moves to rise" is the
rest of the public state a legal policy may use (docs/methodology.md,
information boundary).
Seven arrows: one per legal column, fanning from the root to the seven
successor afterstates.
Solid accent arrow + "label: H40 return" tag (column 3): the action that
was played during data collection. Its successor is the only one carrying a
training label — an H40 return under D1 continuation, the afterstate label
panel of the 2026-08-20/21 experiments (docs/research/status.md §7).
Dashed arrows + "?" tags: the six unplayed columns. No label exists for
these afterstates; any value the model assigns them is extrapolation.
Deployment panel: the same fan, but now the model must produce a ranking
of all seven. The six never-observed successors carry dashed red outlines and
"rank ?" tags; the observed one is marked "observed".
Caption strip: the failure in one line. The second line quotes the
status.md §4 conclusion that low value error on visited states did not
guarantee good root-action ranking.
Hover/focus popovers (the fig-pt/fig-pop convention of the chart
generator): drill-downs on the root board (public-state definition and board
orientation), the labelled sibling, the unlabelled siblings, the deployment
fan, and the caption.
Simplifications (stated explicitly)
Successors are drawn as root + dropped disc. Cascades, reveals and
gravity after the drop are not simulated; the diagram is about label
coverage, not mechanics. The landing cell of the dropped disc is the accent
square in each mini-board.
The root position is invented, not a recorded board; no recorded board
image exists in the repository for this purpose, and the mechanism does not
depend on the position.
"H40 return" names the label family of the afterstate experiments
(status.md §7); the diagram does not assert that every class-(iii) failure
used that exact label — the class spans played-action-only labels, noisy
labels, and labels that cannot separate siblings (audit-05 §4 definition).
The 2026-08-21 result quoted in the deployment popover (successor-closed
coverage, every legal sibling, exact search-value labels, still worse than
one-ply exact search) is from docs/research/status.md "Directions closed",
included because it relocates the obstacle from coverage to capacity.
Sources
docs/research/status.md §4 (sibling extrapolation paragraph) and §7;
"Directions closed" table (the successor-closed coverage row).
docs/exploratory/audit-05-optimistic-curriculum.md §4 — class (iii)
definition and the counts table (6 of 17; experiments 5, 9, 11, 13, 14, 17).
docs/methodology.md — the public-state definition.
CSS variables with light-theme fallbacks (var(--fig-fg, #222),
var(--fig-muted, #888), var(--fig-accent, #2563eb), --fig-grid,
--fig-pop-bg, --fig-pop-border, --fig-cover, --fig-danger) so the diagram
renders standalone and adopts the console's dark theme inside .research-fig
(web/app/globals.css). Popovers are pure SVG/CSS (fig-pt + fig-pop), the
same convention as web/content/figures/score-vs-depth.svg; they need no
JavaScript.
And every learned evaluator's ranking quality, against the exact search it tried to reproduce:
Every learned evaluator against its panel's exact-search comparator on held-out roots: top-1, pairwise accuracy and normalized regret. Panels differ across points and are named per category; compare within a panel, not across.Source data
Panels differ across points and are named per category: the afterstate models and the fair D4 / fair D1 comparators are gated on the H40 D1-continuation panel; the D4-value student on a D4-ordering panel; the planner-distill student and its comparator on the fair-planner H5K256 panel; martingale-dual B0 and its comparator on the locked 477-root H200 panel; exact D1 / exact D2 on the historical panel of RS-20260821T104500Z-77d21e90. Compare within a panel, not across. Fair D1 pairwise and regret on the afterstate panel are not recorded, and exact D1 / exact D2 are recorded top-1 only, so those series points are omitted. Full-train pairwise 0.658 is from the record's summary prose; its per-half values are not recorded. Per-point labels are omitted to keep the dot grid readable; every popover carries the value and source. All eight students sit below their panel's fair-D4 comparator column. Evidence tier: pilot (afterstate and D4-value line); see each source for its panel definition.
Spec: web/content/figures/learned-ranking-metrics.json · 8 source records
Most useful conclusions so far
Fair chance handling matters. Optimistic, worst-case, or tiny reused
reveal samples can rank moves incorrectly.
More depth is not automatically more strength. Depth and chance
resolution substitute rather than compound: with an approximate chance
model the best depth is three, with an exact chance model it is four, and
the budget frontier has a flat top at the fair-D4 operating point, reachable
from either axis
(RS-20260821T192140Z-189fe392,
finding-16).
The substitution is visible as a crossed pair of lines:
Mean score by search depth and chance resolution on the shared cohort. The sign of the depth gradient flips with the stratum count.Source data
Cohort 0xa51d1000-0xa51d103f, corrected 17,000-point Hardcore scoring, 64 games per cell except the depth-5 seven-stratum arm, which was stopped by decision after the first 32 seeds (0xa51d1000-0xa51d101f) and is final. The stopped arm is drawn as its own dashed one-point series so the two sample sizes are never joined by a line (same convention as score-vs-depth). No per-mean confidence intervals are recorded anywhere for these arms; the bounds on the paired contrasts live in detection-floor-map. The d2s5 mean is the coordinator-supplied figure printed in finding-10's position-mode table and in the 2026-08-21 log ArmTable (the log path is not an accepted sourceRecord scheme, so the finding is cited). Evidence tier: development / public-development.
Spec: web/content/figures/depth-chance-factorial.json · 4 source records
And the budget view of the same arms shows why neither axis wins on its own:
Mean score against logical work per move. Two different ways of spending the same budget — depth or reveal sampling — land in the same place, and the frontier is flat at the fair-D4 operating point.Source data
Cohort 0xa51d1000-0xa51d103f, corrected 17,000-point Hardcore scoring, n=64 per arm except the d5s7 point (32 of 64 planned games, stopped by decision, final), which is drawn as its own dashed one-point series so the two sample sizes are never joined by a line. Cache capacity differs across arms (60,000 entries for the recorded depth-4 comparators, 200,000 for the depth-5 arms, auto-sized 960,695 / 346,921 for the new reveal-sampling arms); capacity provably cannot change play but work per move is not comparable across capacities (finding-16 Limitations; RS-20260821T205102Z-d89df4b5 limitations). Equal-work pair: d3 M=6 (4,244,020; 376,442) vs d4 M=1 (4,956,614; 398,498) is statistically indistinguishable (paired -22,056 [-89,867, +46,009], RS-20260821T192140Z-189fe392 pairedD3M6MinusD4M1). The x axis is linear: the plan's log-scale recommendation is not expressible by the generator, and the d5s7 point dominates the range. Evidence tier: development / public-development.
Spec: web/content/figures/score-vs-work-frontier.json · 4 source records
Flow matters before spectacular chains. The task record repeatedly used
roughly 2.4 numbered clears and 1.4 reveals per move as the region associated
with stable long games. Treat these as diagnostic targets from limited runs,
not proven universal thresholds.
Static board potential is insufficient. Similar-looking boards can have
very different futures depending on how reachable triggers and covered discs
evolve across rises.
State-value accuracy is not action-ranking accuracy. Training data must
cover legal siblings or use an objective designed for relative action value.
As of 2026-08-21 this is necessary but demonstrably not sufficient: a student
given successor-closed coverage, every legal sibling, and exact search-value
labels still ranked worse than a one-ply exact search.
Score is heavy-tailed, and this sets a hard measurement floor. One
million-point game can coexist with a much lower average, and pairing does
not rescue small effects: before running a cohort, state the effect size the
mechanism predicts and check it against the detection floor
(RS-20260821T205102Z-d89df4b5).
Each contrast of the depth factorial, drawn against the smallest effect its cohort could have seen:
The six paired contrasts of the depth factorial against their detection floors. Every significant result clears its floor; every null sits below it and is a non-measurement, not evidence of no effect.Source data
All six contrasts from RS-20260821T205102Z-d89df4b5, cohort 0xa51d1000-0xa51d103f, corrected 17,000-point Hardcore scoring; the n=32 contrasts cover the cohort's first 32 seeds (d5s7 arm stopped by decision, final). Whiskers are one-sided 95% percentile-bootstrap lower bounds (20,000 resamples, Mulberry32 domain 0xb0075eed); upper bounds are NOT recorded for these contrasts in the record and are omitted, not zero. Detection floor = 1.645 x sd / sqrt(n), the smallest true effect whose one-sided 95% bound would clear zero (metrics.detectionFloorDefinition). The d4s7-d3s7 lower bound is carried as +26,468 from metrics.bootstrapVersusNormalApproximation; status.md and the 2026-08-21 log print +26,605, which is prose drift — the record and finding-05's interaction table both say +26,468. Evidence tier: public-development.
Spec: web/content/figures/detection-floor-map.json · 1 source record
D4 is useful but not the answer. It is a strong tactical fallback and
teacher, yet its average is not close enough to qualify.
The leaf evaluation prices what search cannot see. Refitting it toward a
quantity the search already computes exactly — achievable clears over the next
few moves — loses monotonically, and loses by destroying the upper tail rather
than the lower one. A leaf that correlates weakly with short-horizon clear
rate is behaving correctly, not failing.
Cheap proxies for this game invert. Three independent times a short-horizon
screen has ranked configurations in the opposite order from complete games: the
suite-h9-v1 scenario suite, a depth-3 screening run, and an eight-move
achievable-clear label. Screen at the depth you intend to deploy, or do not
screen.
Directions closed by exploratory evidence, 2026-08-20/23
Each entry rejects the exact configuration tested, at development or pilot
tier. None of them is a proof that the underlying idea is impossible, and each
names what would reopen it.
The screen and confirmation deltas behind these closures, with their recorded bounds:
The score-valued closures at a glance: every screen and confirmation paired delta with its recorded 95% bound. Whiskers are absent, not zero, where a record carries no upper bound.Source data
Whole-game paired deltas against each experiment's reference arm, corrected 17,000-point Hardcore scoring. Bounds are one-sided 95% bootstrap lower bounds; upper bounds are drawn only where recorded (the three reveal-sampling contrasts, CMA-ES, both survival-instinct arms). finding-03, finding-14, both learned-leaf arms and A900 record no upper bound — the whisker is absent there, not zero. finding-05's confirmation lower bound is printed +47,457 in the finding and +47,446.8 in RS-20260821T205102Z-d89df4b5; the figure uses the record (+47,447). A900's rejection rests on flat flow statistics and opposite-sign halves, not the score delta, which sits inside its 30,957 detection floor (RS-20260823T131226Z-16564ed9). Cohorts differ per bar (n carried per point; the survival-instinct arms share 128 fresh seeds, A900 256 fresh seeds); this chart assembles separate experiments, it does not pool them. Evidence tier: development / public-development screens.
Series
screen or confirmation (candidate minus reference)
Spec: web/content/figures/screen-deltas-with-bounds.json · 8 source records
The closures themselves, one dot per direction, each drawn against the detection floor its record states — the geometric difference between "rejected" and "not measurable":
Every closed direction whose headline is a points number, one row each, with its recorded bound and, where the record states one, its detection floor. A dot whose magnitude sits inside its floor is a non-measurement, not a rejection. Hover a marker for the recorded note; the non-score closures are quoted verbatim in the figure's notes.Source data
One dot per closed direction whose headline result is a points number, copied verbatim from its record; whiskers are the recorded bounds — one-sided 95% bootstrap lower bounds where that is all the record carries (both depth rows, reveal-construction A900, rollout veto, leaf refit), two-sided 95% where recorded (reveal M=2, CMA-ES, survival-instinct STRICT), and none for terminal utility, whose zero is EXACT (byte-identical games at four magnitudes from -1,000,000 to -50,000,000, 0-64-0; no statistical bound exists or is needed). Detection-floor markers (1.645*sd/sqrt(n)) are drawn only where the record carries one; no floor is recorded for reveal M=2, the rollout veto, or the leaf refit — markers omitted, not zero. Every closure rejects the exact tested configuration at development/pilot tier; none proves the underlying idea impossible, and each row's 'reopens if' condition stays in docs/research/status.md. The plan's 'depth beyond four plies' row is charted as its two recorded contrasts (s7 n=32 stopped/final, s5 n=64). Not chartable on a points axis, recorded here verbatim instead: compact afterstate model reproducing D4's ordering — top-1 0.375 against the 0.60 gate (RS-20260821T104500Z-77d21e90); the suite-h9-v1 benchmark as a strength measure — Spearman -0.257 (finding-10). status.md's table gained three further non-points closures after this figure's plan was written, also not chartable here: leaf-cost NNUE student top-1 0.296/0.3011 by half-fold against the 0.60 gate (RS-20260823T194142Z-946e3cd1), H-pool optimistic states tau -0.959 [cluster bootstrap -3.069, -0.390] (RS-20260823T205143Z-ead14c9d), and P-SOL G0 cheap-label proxy mean tau 0.370, LB95 0.283 (RS-20260823T225753Z-0fbd48c3). Cohorts differ per point (n on every marker); this chart assembles separate experiments, it does not pool them.
Series
closed direction
paired delta vs reference
Bounds
n
Source
headline paired delta (points)
depth 5 vs 4 · s7 d5s7 - d4s7; arm stopped at 32 of 64 games, final; 35.62x work; below its 107,988 floor: a non-measurement
A wider dose is tested; only 28.6% joint coverage was affordable at depth 4, and the dose that worked at depth 3 was 85.7%
Pricing the same-wave double hit on a solid gray (reveal construction) as a leaf term
Corpus: partial r −0.044 with 60% of live setups uncollected; in play, +3,204 over 256 fresh paired games, inside its floor (RS-20260823T131226Z-16564ed9)
A term that raises reveals per move at all; or a per-root counterfactual showing the uncollected setups were worth collecting
Harsher terminal (death) utility
Saturated: byte-identical play at every magnitude past the current value (finding-04)
Never, for this parameterization
A leaf-cost NNUE student reproducing D4's within-root ordering from successor-closed D4 values
A different architecture class at leaf cost, e.g. a cross-sibling set ranker; the self-play loop's leaf form stays blocked and its redesign is root-prior shaped
Optimistic states with fair labels (H-pool, the salvageable core of the oracle curriculum)
Stage D0: tau = −0.959 [−3.069, −0.390], with a degenerate denominator — 63/64 oracle games hit the 500-move cap (RS-20260823T205143Z-ead14c9d)
A D0 rerun with an uncapped horizon shows tau ≥ 0.25 and the oracle's action at least matching D4's under fair futures
Cheap M=1 continuation labels standing in for D3 N7M6 label semantics (P-SOL G0)
Guardrail kill: within-root orderings agree at only mean tau 0.370 (LB95 0.283) on 6 CRN-matched roots; the divergence is the reveal quadrature, not the engine (RS-20260823T225753Z-0fbd48c3)
Reopened 2026-08-24: E-FAST-M6 passed every equivalence gate (RS-20260824T010000Z-8f3e9b4f) at a realised 5.6x speedup (0.177 CPU-s/move in continuation duty), so a powered M=6 guardrail is now affordable; a full M=6 label corpus still costs thousands of CPU-hours and needs a P-SOL-3 design or a scale-out lease
A validated longer horizon; it remains usable as a diagnostic
The two afterstate rows matter more than their tier suggests. The program's
standing explanation for every failed learned policy was insufficient sibling
coverage. The 2026-08-21 experiment supplied successor-closed coverage, every
legal sibling, exact search-value labels, and completeness 1.0 — and the student
still ranked worse than a one-ply exact search. That relocates the obstacle from
the training data to the capacity of a compact board evaluator, and it makes
"train a bigger student" a falsifiable next step rather than a hopeful one.
Open work
The next research step should be evidence-driven rather than another broad
architecture sweep:
Re-register any resumed SHA-locked experiment against the reorganized source
tree and rerun cross-engine parity.
Refit the search leaf against remaining lifetime, the quantity that
correlates with score at r = 0.9995, rather than against achievable clears.
Test whether student capacity is the binding constraint, since that is now
the explicit claim on the table and the cheapest one to falsify.
Test long-cycle features as bounded corrections to D4 before allowing them
to control an entire game.
Resolve the three recorded divergences between this simulator and the
shipped game, which is an owner decision and not an agent decision.
Items 1, 4 and 5 are unchanged. Items 2 and 3 replace the AFBR-40 closure
sequence, whose data-feasibility question was answered directly: closure was
achieved and did not rescue the model.
AFBR-40 as originally proposed has no implementation, protocol, model, or
measurements in this repository and must not appear in a list of attempted
results. The action-free afterstate models that were built and measured on
2026-08-20/21 are separate, separately recorded, and are not AFBR-40.
For configurations and every retained entry point, use the
experiment index. For the unabridged chronological
record, use the experiment history. For alternatives and research
priority, use the strategy landscape and the staged
research roadmap. Future agents must use the standard
benchmark contract and machine-readable records under
research/; these do not retroactively upgrade the
historical evidence above.