Oracle distillation

The problem

A planner that can read the hidden gray-disc numbers and the discs still to come plays very long games: the perfect-information oracle reaches the move cap in every training game it plays. No legal player can read those things. The question this experiment asks is whether the screen alone carries enough information to guess what that planner would do: show a network thousands of positions labelled with the oracle's column and see if it can reproduce the label on games it has never seen.

Proposed solution

The most direct form of distillation: copy the teacher's moves, then play the copy. It uses no hand-built features and no tie-break rules. The teacher is an oracle that reads the hidden gray-disc values and the future disc tape; the student reads only the board, the visible next disc and the moves until the next rise, and that boundary is enforced by a constructor that drops everything else before an example is written.

The labels are gated before any game is played. If the student cannot reproduce the teacher's column on held-out games, there is nothing to deploy. If it can, it plays as a hybrid: an exact public search decides, and the student may override it only when it is confident and the alternative is close in the search's own terms.

How it works

1. Play 12 games with the future-reading planner and 32 games with an ordinary public depth-3 search, capping label collection at 200 moves per game.
2. In both sets, write down the position as the student would see it and label it with the column the teacher would have chosen there. Generating positions with ordinary search makes the task "what would the teacher do here", on boards a normal player reaches, rather than only on boards the teacher's own play produced.
3. Split by whole game, never by position, so no position from a training game can appear in the held-out set.
4. Train a sparse network with 614 inputs, two hidden layers of 128, and seven outputs, one per column, to predict the teacher's column.
5. Gate the labels before any game is played: held-out top-1 accuracy (how often the student's first choice matches the teacher's) at least 0.30, top-2 at least 0.55, cross-entropy at most 1.75.
6. Only if the labels pass, deploy the hybrid: exact search decides, and the student may override it when it is at least 40% confident, at least 12 percentage points ahead of the search's own choice, and the alternative is within 10% of the search's value range. Eight fresh games screen it; sixteen more confirm it. The output at every position is one column.

What happened

The network learned its training positions and could not reproduce them on games it had not seen. On 1,629 held-out labels it picked the teacher's column 0.218 of the time against the 0.30 the protocol required in advance, and it did equally badly on positions from ordinary play, so unfamiliar boards were never the problem (rejected, ledger-recorded, ledger section "Privileged-future oracle distillation"). The gate had failed, but the eight-game screen was run anyway, and the hybrid passed it. The sixteen-game confirmation then reversed it: the same frozen hybrid lost 62,274.25 points per game to the exact search it was overriding (historical 7,000-point scoring, archival). Eight games cannot separate policies whose scores are heavy-tailed, and that reversal is one reason this site runs confirmation cohorts at all.

What we learned

The label was the problem. A training cross-entropy of 0.480 says the network had ample capacity to fit what it was shown; a held-out cross-entropy of 3.282 on positions drawn from ordinary play says the target is not a function of the public board (1,629 held-out labels, ledger). The information the teacher used sits in the hidden values and the future tape, which is where the primer says this method stops working: two positions that look the same to the student carry different labels, and no amount of data changes that.

A screen win on eight games means very little, and the predeclared confirmation is what caught this one; the ledger records both halves. The heavy tails page explains why a small cohort can pass on luck.

The rejection covers imitating the choices of a future-reading teacher. It says nothing about teachers whose advantage is itself public, which planner distillation pursued later, and nothing about learning something other than the teacher's action, which the topology experiments tried next. The open question is whether anything the oracle knows can be carried to a student as a target other than its move.

Agent contextRecords and provenanceopen ↓close ↑

Records: experiment index "Oracle and curriculum research"; ledger section "Privileged-future oracle distillation".

Frozen run: fitting seeds 0x3d7a0000... (oracle roll-ins) and 0x3d7b0000... (behaviour roll-ins); whole-game label holdouts 0x3d7c0000... and 0x3d7d0000.... 5,110 fitting labels, 1,629 held-out labels. Network 614 / 128 / 128 / 7, 384,540 bytes. Single run 296.3 seconds. Screen cohort 0x3d7e0000...07 (8 games); confirmation cohort 0x3d7f0000...0f (16 games).

The oracle fitting roll-ins all reached the 200-move label cap and averaged 829,516.75 points. That is an oracle number, produced with the future visible, under historical 7,000-point scoring, archival; it is not comparable with any policy score and does not appear in the tables below.

audit-05 §4 classifies this as failure 1 of 17, primary mode "representation / information gap", and notes that distribution shift was controlled here: the ledger's own words are that "oracle and behavior holdouts were similarly weak", and fitting was excellent, so it is not an optimisation failure either.

Agent contextFull results tableopen ↓close ↑

Label gate, 1,629 held-out labels:

Quantity	Gate	Observed
Held-out top-1	at least 0.30	0.218
Held-out top-2	at least 0.55	0.386
Held-out cross-entropy	at most 1.75	3.282
Training cross-entropy	none	0.480

Gameplay, historical 7,000-point scoring, archival:

Cohort	Exact behaviour search	Hybrid	Paired delta
Screen, 8 games 0x3d7e0000...07	261,871.875 pts / 79.25 moves	283,850.625 / 84.5	+21,978.75 / +5.25, both paired 95% lower bounds negative
Confirmation, 16 games 0x3d7f0000...0f	249,816.875 / 75.375	187,542.625 / 57.25	−62,274.25 / −18.125, lower bounds −118,363.971 / −33.322

The hybrid overrode the search on 18.45% of confirmation moves.

Agent contextValidity, gates and limitationsopen ↓close ↑

Who saw what:

	The teacher	The student
The board on screen	yes	yes
The visible next disc	yes	yes
Moves until the next rise	yes	yes
Hidden gray-disc numbers	yes	no
The discs still to come	yes	no
Game seed, score, level, move number	yes	no

The boundary is enforced in code: an example passes through a public-state constructor that keeps only the board, the visible disc and the rise phase, and drops the seed, the future tape, the reveal generator, the score, the level and the move count. Boards are canonicalized by reflection so a mirrored position cannot be treated as a different one.

The label gate failed and the screen was run anyway for completeness; the screen's paired 95% lower bounds were both negative even though the means improved. The rejection rests on the paired direction of the sixteen-game confirmation.

Agent contextScoring modeopen ↓close ↑

The ledger entry does not state which level bonus was in force, and it sits well before the corrected-scoring replay in the ledger, so the absolute point totals are treated as historical 7,000-point scoring, archival, rather than as corrected-score Hardcore evidence. The paired direction of the confirmation is what the rejection rests on.