42774542557611213241565pocketdeadcovershape, not numbers
Technique

Oracles, teachers and distillation

Let a program that can see hidden information make labels, then train a program that cannot see it to copy the labels, and only ever measure the copy.

Read the primer: Oracles, teachers and distillation

On this page

Stop trying to copy the teacher's moves. Ask something narrower: looking only at the shape of a board, can you tell whether it came from a planner that could see the future, or from an ordinary search?

reads hidden values or the future: a teacher, never a policy

If that question has a good answer, then something about a sustainable board is visible on screen, and a legal policy could be nudged toward it, without ever being asked to reproduce a choice that depended on invisible information.

Why this is a better question than "what would the teacher play?"

Copying the teacher's column failed because the choice depends on gray numbers a player cannot see. But two boards can be equally unknowable and still be very differently arranged — one with its high numbers stacked where they can never fire and its covers buried under four rows, the other with triggers reachable and covers exposed. That difference is entirely visible. So the experiment matches the two policies' positions on everything superficial — rise phase, exact occupancy, exact maximum height, and where in the game they occurred, and then asks a small network to tell them apart from the 49 cells alone.

Three programs, three stages

  1. The audit plays the privileged planner and an ordinary public search on the same seeds and pairs their positions on the matching keys above. It then looks for feature directions that separate the two, and: this is the part that makes it an audit rather than a fishing trip — requires each direction to appear independently in both halves of the seed set before it counts.
  2. The residual turns the surviving question into a learned model: a compact board-only network trained to output "oracle-like versus fair-like", then added as a small extra term to the leaf of an otherwise unchanged fair depth-4 search, with the size of that term chosen by a frozen rule.
  3. The extension verifies the frozen model byte for byte, replays the original held-out games and requires the original numbers to reproduce exactly, then collects a larger, untouched set of positions to test the same prediction at more than twice the sample size, and only then lets it play.

Nothing privileged enters the model's input. It receives a reflection-canonical 49-cell board and nothing else: no seed, no tape, no next disc, no phase, no score, no move index, and none of the matching keys used to balance the examples.

What happened

The signal is real. The audit found a set of board properties that are stable across independent halves of the data: sustainable boards reward reachable triggers (direct and latent) triggers that survive a rise, stored high numbers that can still fire, and cracked covers that are close to being revealed; they penalise solid cover count and cover altitude, projected occupancy debt, dead low numbers, low caps and adjacent ones. Two plausible candidates — generic access to a cliff, and strong repeated-column cohesion — did not survive the split-half rule and are recorded as unsupported.

The first learned version was underpowered, not wrong. Its held-out numbers looked encouraging, but a cost projection had shrunk the cohort before any model metric existed, and the run finished 30 examples and 15 pairs short of the sample size the protocol demanded. The gate was not moved to fit the favourable result; the experiment was recorded as a failed gate, and no game was played.

The prediction replicated, and the policy still lost. The extension reproduced the original metrics exactly, then independently confirmed the model on 204 fresh pairs. An eight-game screen looked like a win — until you notice each policy won four of the eight games and both confidence bounds were negative. The sixteen-game confirmation rejected it: about 42,000 points and 24 moves worse per game. The failure mode is worth stating precisely, because the mean hides it: the candidate did not lose a little everywhere; it destroyed a few very long baseline games, turning a 285-move game into a 90-move one and a 200-move game into a 55-move one, while occasionally making large gains elsewhere.

Technical record

Audit (completed, ledger-recorded diagnostic; historical 7,000-point seeds 0x3d70f000...0f): privileged planner at depth 4 / beam 128 averaged 429,182.5 points and hit the 200-move cap in all 16 games, winning every pair; public depth 3 averaged 90,273 points and 63.625 moves. Clears and reveals per move: 2.354 / 1.386 against 1.792 / 0.960.

Residual (rejected; ledger-recorded; underpowered gate failure): 738 matched training examples (369 pairs); board-only sparse network, 490 one-hot inputs with exactly 49 active, 8 accumulators, 3,937 parameters, 240 fixed Adam epochs. Held-out 170 examples / 85 pairs against a preregistered floor of 200 examples / 100 pairs. Area under curve 0.68097, exact-stratum ranking 0.67059, pair accuracy positive in both whole-seed halves (0.70270 and 0.64583). Screen and confirmation seeds were not read. Model fingerprint 0x0af6ed6f88895cfe.

Extension (rejected; ledger-recorded; prediction replicated, policy regressed): replay reproduced the original 85 pairs and every metric exactly. Extension cohort 204 pairs / 408 examples: area under curve 0.63829, matched pair accuracy 0.64216, halves 0.64103 / 0.64368. Pooled 289 pairs / 578 examples: 0.64811 and 0.65052. Held-out logistic loss was poorly calibrated at 1.94287. The frozen coefficient grid switched 8.33%, 12.5%, 20.83% and 29.17% of actions at 250, 500, 1,000 and 2,000; the frozen rule selected 500.

CohortFair depth 4With the residualPaired delta
Screen, 8 games 0x3ea90000...00785,874.5 pts / 60 moves95,697.125 / 65.625+9,822.625 / +5.625; four score pairs each; lower bounds −22,580.31 / −14.01
Confirmation, 16 games 0x3eaa0000...00f196,764.125 / 125.9375154,934.625 / 101.9375−41,829.5 / −24; lower bounds −116,357.90 / −67.68

Clears and reveals per move fell from 2.0705 / 1.1608 to 2.0159 / 1.1245; mean maximum chain fell from 7.6875 to 6.6875; no game was censored. Individual collapses: 476,511 points / 285 moves became 139,399 / 90, and 329,049 / 200 became 74,172 / 55.

audit-05 §4 classifies the residual as "underpowered, mechanism untested" and the extension as a sibling-coverage failure — "state-level classification did not become action ranking".

Records: experiment index "Oracle and curriculum research"; ledger sections "Privileged topology audit", "Privileged-oracle observable-topology residual" and "Frozen oracle-topology prediction extension". Sources: oracle-topology-audit.cpp, oracle-topology-residual.cpp, oracle-topology-residual-extension.cpp.

What this taught us, and what is still open

  • The split-stable feature list is the durable output. Those directions survived an independent-halves test and were reused directly by accessible energy. The two rejected candidates are equally useful: they are recorded as not supported.
  • Telling good boards apart is not the same as ranking moves. The model answered its own question correctly on fresh data and still lost games. This is the sibling-ranking problem again, and it is the single most repeated lesson in this repository.
  • Means hide tail damage. A change that improves typical games while occasionally amputating a 285-move game is a losing change, and only paired whole-game reporting shows it.
  • What was not ruled out. The underpowered run never tested its mechanism at the sample size it asked for; and no version of this idea has been tried as a veto on near-tied moves rather than as a term inside the leaf evaluator.
RecordsTheories, experiments and results that reference this directory

Claim: Public board states visited by the perfect-information oracle after move 50 have a higher expected remaining lifetime under FAIR, public-randomness continuation than fair-D4-visited states matched on rise phase, occupancy and maximum height. Defining R_fair(O) and R_fair(F) as the mean remaining moves over K independent public futures under a fixed public continuation policy, and R_tape(O), R_real(F) as the realised remaining moves on the states' own trajectories, the transferable fraction tau = (R_fair(O) - R_fair(F)) / (R_tape(O) - R_real(F)) is at least 0.25, with the same sign in both split halves of the O pool, and the oracle's column at oracle-visited roots is fair-top-1 under common public futures at least as often as fair D4's column is.

This theory is currently not-supported-as-tested at the pilot (a small run to find bugs and project cost, not a strength claim) level.

It compares oracle-curriculum state pool relabelled under public randomness against fair-d4 matched state pool at the CHECK (mechanics checks only, no games played) level, using no-gameplay data.

valid run outcome: fail The run was valid and the outcome was fail (pilot (a small run to find bugs and project cost, not a strength claim)). Read the result.

resultvalid runoutcome: failtier: pilotRS-20260823T205143Z-ead14c9d

The run was valid; the outcome was fail, at the pilot (a small run to find bugs and project cost, not a strength claim) level. Of 4 preregistered checks, 2 passed and 2 failed.

Stage D0 refuses the H-pool theory on two of its three preregistered criteria. Pool O was 1,984 public states sampled from 64 oracle games (depth 4, beam 128; 63 of 64 games reached the 500-move cap); 1,271 were matched 1:1 to fair-D4 states on exact (rise phase, occupancy/4, max-height/2) buckets over all 256 comparator seeds (match rate 0.641; 713 O states dropped and counted). Under 32 common public futures with fair D1 continuation at horizon 25, oracle-visited boards are only marginally better than matched fair boards: R_fair(O) = 24.783 vs R_fair(F) = 24.520 (difference +0.263 moves, cluster bootstrap 95% [+0.156, +0.367], same positive sign in both origin-game halves). The preregistered transferable fraction is negative, tau = -0.959 (95% cluster interval [-3.069, -0.390]), because its denominator degenerates at this horizon: R_tape(O) = 24.338 vs R_real(F) = 24.611 (94-95% of both realised remainders were capped at 25), so the realised-lifetime gap the fraction was defined against is -0.274 moves rather than positive. Independently of that degeneracy, the action-quality criterion fails outright: the oracle's own column is fair-top-1 at its own roots less often than fair D4's column at the same roots (0.766 vs 0.814 over 1,271 roots, difference -0.049, 95% [-0.069, -0.029]; on the 194 unique-maximum roots 0.268 vs 0.387). Blocked-flow-band fraction over matched O states is 0.0047 (F: 0.026). Per the preregistration the theory is assessed not-supported-as-tested, the pool-comparison follow-on is not registered, and audit-05's H-pool program closes.

Technical recordLimitations recorded with the resultRS-20260823T205143Z-ead14c9d
  • The preregistered tau is ill-conditioned as measured: with 63 of 64 oracle games censored at the 500-move cap and comparator states drawn from the same shape buckets, both realised remainders sit at the 25-move horizon cap (R_tape(O) 24.34, R_real(F) 24.61, 94-95% capped), so the denominator is -0.274 moves instead of the large positive gap the definition assumed. The tau < 0.25 refusal is therefore driven by a degenerate denominator, not by a negative numerator; the numerator (the fair-value advantage of oracle boards) is positive but small, +0.263 moves on a 25-move horizon.
  • The fair-top-1 criterion is unaffected by that degeneracy and fails on its own: the oracle's action is fair-best at its own roots significantly less often than fair D4's action at the same roots.
  • Match rate 0.641: the 713 dropped O states skew toward buckets fair D4 rarely visits (matched O states are shape-matched by construction), so the numerator is a lower bound on the raw O-vs-F fair-value difference over all oracle states, and the comparison is conditional on shape overlap as preregistered.
  • R_fair uses a fixed public D1 continuation and horizon 25 with 32 futures; 92-96% of futures survive the horizon, so both pools are near the measurement ceiling and the +0.263 difference is compressed by censoring at both ends.
  • F origin games contribute clustered states (mean 12.6 matched states over 101 of 256 games); uncertainty uses cluster bootstrap over O origin games carrying matched partners, with the independent-clusters sensitivity interval also recorded (it is much wider: [-9.35, +5.12]).
  • Diagnostic tier (CHECK): no gameplay evidence and no policy claim; training-role seeds only, opened once.

Full record →

Agent contextSource files, operational notes and how to reproduce