Drop7 Research

Type to search every page, record and vocabulary term.

GitHubPlay

Oracles, teachers and distillation

Let a program that can see hidden information make labels, then train a program that cannot see it to copy the labels, and only ever measure the copy.

technique primer
On this page
Approaches that use it
Group page

Oracles, teachers and distillation approaches

The idea

Some of what decides a game is on the screen and some is hidden. A program that is allowed to read the hidden part can play far better than one that cannot, which makes it useless as a player and useful as a teacher. Let it play, and at every position write down two things: what a legal player would have seen, and what the teacher did. Those pairs are labels. Then train a second program, one that reads only the visible part, to reproduce the teacher's choices from the visible part alone. The teacher is an oracle; the copy is a student; the copying is distillation, also called behaviour cloning.

The rule that keeps this honest is about measurement. The teacher's score is never reported as the student's. Only the student plays for the record, through the same interface as any other legal player.

A small example

A student walks a maze blindfolded to everything past the walls it can touch. A teacher has the map.

  1. The teacher walks many mazes and, at each junction, records what the student would see there (the local corridor) and the turn the teacher took. Those pairs are the labels.
  2. Train a small classifier from "what the student sees" to "which turn". That is distillation.
the information boundarythe teacher sees334643176632147575722211every hidden number, and the discs to comethe student sees3331731416 covered discs, values unknown464hidden values stop hereone columnthe label crosses
One position from a playground game (seed 0x5eed031b), shown as the teacher sees it on the left, with every hidden number filled in, and as the student sees it on the right, with 16 covered discs. The teacher's chosen column is the only thing allowed across the line; the hidden values stop at it.
  1. Test it on mazes the teacher never walked. If two junctions look identical from inside the corridor, and the map says turn left at one and right at the other, no classifier can get both right. The label depends on something the student cannot see.
??the mapexitwhat the student sees is identical · the map says turn left??the mapexitwhat the student sees is identical · the map says turn right
Two junctions in two mazes. From inside the corridor the student sees the same thing at both, and the map says turn left at one and turn right at the other. A student trained to copy the map's turns has to answer both from the same view, so it is wrong at one of them however it is trained.
  1. DAgger, short for dataset aggregation, fixes a different problem. A student trained only on the teacher's own paths wanders into junctions the teacher never visited and has no labels for them. So let the student walk, ask the teacher at every junction the student reaches what it would have done, add those labels, and retrain.
teacher playsonce, with everything visiblelabelspublic view, teacher's movestudent playstrained on the labels, blindteacher relabelsthe positions the student reachedfirst labelstrain, then walkwhere did it end up?more labelsevery lap adds labels at the student's own positions; the labels are still the teacher's reasons
The teacher plays once and its logged choices become the first labels. From then on the loop runs on the right: the student is trained and plays on its own, the teacher is asked what it would have done at every position the student reached, and those answers join the labels. The travelling dot follows the loop. Boxes on the teacher's side of the boundary are outlined in the oracle colour.
  1. DAgger does not fix step 3. If the teacher's reason is invisible to the student, more labels of the same kind do not help.
  2. Whatever happens, only the blindfolded walk is scored.

How it works

The method has three parts. The teacher is any program with access to privileged information: the hidden values, the future, or both. It plays, and its choices are logged next to the public view of each position. The student is a function from public view to choice, usually a small network, fitted to those logs. Between them sits the information boundary, and nothing crosses it except the label. A useful check before any game is played is the held-out label test: hold back whole games, fit on the rest, and measure how often the student picks the teacher's move on positions it has not seen. A student that cannot reproduce the labels has nothing to show in play.

The mechanism has a limit that no amount of data or capacity moves. The student can only learn a function of what it sees. Whenever the teacher's choice depends on hidden information, two positions that look the same to the student carry different labels, and the student's best answer is a guess weighted by how often each label appeared. DAgger changes which positions get labelled. It does not change what the labels depend on.

In Drop7

The hidden information in Drop7 is the number under every gray disc and the sequence of discs still to come. The oracle on this site is a beam search over the game's actual future tape: it reads the hidden values and the discs to be dealt, plans against them, and plays. Its students are sparse networks over the public board: the cells, the next disc and the rise clock. A second kind of teacher is legal but expensive: the fair depth-4 search itself, distilled into a network so that its choices can be made without running the search.

Pages that use the method:

The concept page on oracles and teachers walks through the same boundary on a real position.

What it cannot do

The oracle showed that long games exist. On twelve training-only games it reached the 500-move cap in every one, with a mean of 1,058,931.5 points under historical 7,000-point scoring, archival. That is an oracle number, produced with the future visible, and it says what the game allows (Perfect-information oracle).

The student could not copy it. On 1,629 held-out labels the direct copy picked the teacher's column 0.218 of the time against a gate of 0.30 fixed in advance, and it did equally badly on positions from ordinary play, so unfamiliar boards were never the problem. The page's own conclusion is that the information needed to reproduce the teacher's choice is not on the screen (Oracle distillation). The hybrid that let the student override an exact search then passed its eight-game screen by 21,978.75 points and failed its sixteen-game confirmation by 62,274.25 points and 18.125 moves per game, both under historical 7,000-point scoring, archival; eight games cannot separate policies whose scores are heavy-tailed, and that reversal is the reason this site runs confirmation cohorts at all. An audit of the whole curriculum asked where the oracle's advantage comes from and found most of it in knowing the tape, the one thing no legal player can have (audit-05).