---
title: Oracle distillation
family: oracle-curriculum
summary: Show a network thousands of positions labelled with the column a future-reading planner chose, and see whether it can guess that column from the screen alone.
status: rejected
evidence: ledger-recorded
reads: teacher
kind: strategy
technique: oracle-distillation
featured: true
---

## The problem

A planner that can read the hidden gray-disc numbers and the discs still to
come plays very long games: the
[perfect-information oracle](/approach/oracle-curriculum/perfect-information-oracle)
reaches the move cap in every training game it plays. No legal player can read
those things. The question this experiment asks is whether the screen alone
carries enough information to guess what that planner would do: show a network
thousands of positions labelled with the oracle's column and see if it can
reproduce the label on games it has never seen.

## Proposed solution

The most direct form of [distillation](/learn/glossary): copy the teacher's
moves, then play the copy. It uses no hand-built features and no tie-break
rules. The teacher is an [oracle](/learn/glossary) that reads the hidden
gray-disc values and the future disc tape; the student reads only the board, the
visible next disc and the moves until the next rise, and that boundary is
enforced by a constructor that drops everything else before an example is
written.

The labels are gated before any game is played. If the student cannot reproduce
the teacher's column on held-out games, there is nothing to deploy. If it can,
it plays as a hybrid: an exact public search decides, and the student may
override it only when it is confident and the alternative is close in the
search's own terms.

## How it works

1. Play 12 games with the future-reading planner and 32 games with an ordinary
   public depth-3 search, capping label collection at 200 moves per game.
2. In both sets, write down the position as the student would see it and label
   it with the column the teacher would have chosen there. Generating positions
   with ordinary search makes the task "what would the teacher do here", on
   boards a normal player reaches, rather than only on boards the teacher's own
   play produced.
3. Split by whole game, never by position, so no position from a training game
   can appear in the held-out set.
4. Train a sparse network with 614 inputs, two hidden layers of 128, and seven
   outputs, one per column, to predict the teacher's column.
5. Gate the labels before any game is played: held-out top-1 accuracy (how
   often the student's first choice matches the teacher's) at least 0.30, top-2
   at least 0.55, cross-entropy at most 1.75.
6. Only if the labels pass, deploy the hybrid: exact search decides, and the
   student may override it when it is at least 40% confident, at least 12
   percentage points ahead of the search's own choice, and the alternative is
   within 10% of the search's value range. Eight fresh games screen it; sixteen
   more confirm it. The output at every position is one column.

## What happened

The network learned its training positions and could not reproduce them on
games it had not seen. On 1,629 held-out labels it picked the teacher's column
0.218 of the time against the 0.30 the protocol required in advance, and it did
equally badly on positions from ordinary play, so unfamiliar boards were never
the problem (rejected, ledger-recorded, [ledger](/docs/research/history)
section "Privileged-future oracle distillation"). The gate had failed, but the
eight-game screen was run anyway, and the hybrid passed it. The sixteen-game
confirmation then reversed it: the same frozen hybrid lost 62,274.25 points per
game to the exact search it was overriding (historical 7,000-point scoring,
archival). Eight games cannot separate policies whose scores are heavy-tailed,
and that reversal is one reason this site runs confirmation cohorts at all.

## What we learned

The label was the problem. A training cross-entropy of 0.480 says the network
had ample capacity to fit what it was shown; a held-out cross-entropy of 3.282
on positions drawn from ordinary play says the target is not a function of the
public board (1,629 held-out labels, ledger). The information the teacher used
sits in the hidden values and the future tape, which is where the
[primer](/learn/techniques/oracle-distillation) says this method stops working:
two positions that look the same to the student carry different labels, and no
amount of data changes that.

A screen win on eight games means very little, and the predeclared confirmation
is what caught this one; the ledger records both halves. The
[heavy tails](/learn/concepts/heavy-tails) page explains why a small cohort can
pass on luck.

The rejection covers imitating the choices of a future-reading teacher. It says
nothing about teachers whose advantage is itself public, which
[planner distillation](/approach/lifetime-objective/planner-distill) pursued
later, and nothing about learning something other than the teacher's action,
which the [topology](/approach/oracle-curriculum/topology) experiments tried
next. The open question is whether anything the oracle knows can be carried to
a student as a target other than its move.

<AgentContext summary="Records and provenance">

Records: [experiment index](/docs/research/experiment-index) "Oracle and
curriculum research"; [ledger](/docs/research/history) section
"Privileged-future oracle distillation".

Frozen run: fitting seeds `0x3d7a0000...` (oracle roll-ins) and `0x3d7b0000...`
(behaviour roll-ins); whole-game label holdouts `0x3d7c0000...` and
`0x3d7d0000...`. 5,110 fitting labels, 1,629 held-out labels. Network 614 / 128
/ 128 / 7, 384,540 bytes. Single run 296.3 seconds. Screen cohort
`0x3d7e0000...07` (8 games); confirmation cohort `0x3d7f0000...0f` (16 games).

The oracle fitting roll-ins all reached the 200-move label cap and averaged
829,516.75 points. That is an oracle number, produced with the future visible,
under historical 7,000-point scoring, archival; it is not comparable with any
policy score and does not appear in the tables below.

[`audit-05`](/docs/exploratory/audit-05-optimistic-curriculum) §4 classifies
this as failure 1 of 17, primary mode "representation / information gap", and
notes that distribution shift was controlled here: the ledger's own words are
that "oracle and behavior holdouts were similarly weak", and fitting was
excellent, so it is not an optimisation failure either.

</AgentContext>

<AgentContext summary="Full results table">

Label gate, 1,629 held-out labels:

| Quantity | Gate | Observed |
| --- | --- | ---: |
| Held-out top-1 | at least 0.30 | 0.218 |
| Held-out top-2 | at least 0.55 | 0.386 |
| Held-out cross-entropy | at most 1.75 | 3.282 |
| Training cross-entropy | none | 0.480 |

Gameplay, historical 7,000-point scoring, archival:

| Cohort | Exact behaviour search | Hybrid | Paired delta |
| --- | ---: | ---: | ---: |
| Screen, 8 games `0x3d7e0000...07` | 261,871.875 pts / 79.25 moves | 283,850.625 / 84.5 | +21,978.75 / +5.25, both paired 95% lower bounds negative |
| Confirmation, 16 games `0x3d7f0000...0f` | 249,816.875 / 75.375 | 187,542.625 / 57.25 | −62,274.25 / −18.125, lower bounds −118,363.971 / −33.322 |

The hybrid overrode the search on 18.45% of confirmation moves.

</AgentContext>

<AgentContext summary="Validity, gates and limitations">

Who saw what:

| | The teacher | The student |
| --- | --- | --- |
| The board on screen | yes | yes |
| The visible next disc | yes | yes |
| Moves until the next rise | yes | yes |
| Hidden gray-disc numbers | yes | no |
| The discs still to come | yes | no |
| Game seed, score, level, move number | yes | no |

The boundary is enforced in code: an example passes through a public-state
constructor that keeps only the board, the visible disc and the rise phase, and
drops the seed, the future tape, the reveal generator, the score, the level and
the move count. Boards are canonicalized by reflection so a mirrored position
cannot be treated as a different one.

The label gate failed and the screen was run anyway for completeness; the
screen's paired 95% lower bounds were both negative even though the means
improved. The rejection rests on the paired direction of the sixteen-game
confirmation.

</AgentContext>

<AgentContext summary="Scoring mode">

The ledger entry does not state which level bonus was in force, and it sits
well before the corrected-scoring replay in the ledger, so the absolute point
totals are treated as historical 7,000-point scoring, archival, rather than as
corrected-score Hardcore evidence. The paired direction of the confirmation is
what the rejection rests on.

</AgentContext>