Learning and training

A rule that maps what the player sees to a column.

A policy that reads only the board, the next disc, the rise clock and the terminal flag. The only kind eligible for the million-point claim.

The line between what a policy may read (public) and may not (seed, hidden values, future discs, score, level, move number).

A planner allowed to read hidden information, used only to produce training labels or diagnostics.

A public policy trained from a teacher's labels.

A prediction of how a game will go from a position.

The board after a move resolves, before the next disc is dealt. Evaluating afterstates removes the need for a separate action input.

A model trained only on played moves being asked to rank unplayed ones. → concept

Training data in which every legal sibling at every position is labelled.

An evaluator that looks up learned weights for many small patterns of cells and adds them.

"Efficiently updatable neural network": a small network designed so a board's evaluation can be updated incrementally after a move.

Reinforcement-learning methods that adjust a policy or value network from play experience.

Generating training data by having the current policy (or a search guided by it) play games.

Alternating between evaluating a policy and improving it; the loop the scale-out direction proposes. → concept

Learning a value function by nudging the prediction at one position toward the reward received plus the prediction at the next position, so later predictions teach earlier ones.

A store of past positions and outcomes that training samples from repeatedly, so one game's data is used more than once and consecutive samples are less correlated.

Two learned parts of one player: the actor picks columns and the critic predicts how the game will go, and the critic's judgment tells the actor which choices to reinforce.

Training a policy to copy a teacher's recorded choices directly, position by position, with no reward signal.

Training one model to reproduce another's outputs, so a slow or privileged teacher's judgment can be carried by a small public student.

Dataset aggregation: the student plays, the teacher labels the positions the student actually reached, and those labels join the training set, so the student is corrected on its own mistakes.

A training target that records several points of the distribution of outcomes from a position, so the model learns how wide the spread is as well as where its centre lies.

Optimisers that improve a set of weights by sampling variations, scoring each with complete games, and moving toward the variations that scored best. CEM and CMA-ES are two of them.

An optimiser that samples a population of weight settings, keeps the best-scoring fraction, refits the sampling distribution to them, and repeats.

Covariance matrix adaptation evolution strategy: an optimiser that samples weights from a Gaussian and adapts its mean and covariance toward the samples that scored best.