Evidence and results

A registered, falsifiable claim with a mechanism.

A frozen plan for testing one configuration of a theory.

One execution of an experiment: valid, partial, or invalid.

The assessment of a run: pass, fail, inconclusive, or not applicable. A valid fail is a completed contribution.

A fixed, ordered list of game seeds played by both the candidate and the reference.

Candidate and reference play the same games, so their scores can be compared game by game.

A registered range of random seeds an experiment may use, with a role: training, development, protected, or final.

The ladder of benchmark sizes and data roles, from mechanics checks (no games) to the one-shot final cohort.

The reference policy: fair expectimax, depth 4, hand-tuned leaf, five chance samples per node. Its widely quoted mean of about 308,000 over 64 games should be read with care: an audit found that figure appears once in the ledger, as an internal comparator, without a recorded seed list, dispersion, or censoring statement.

The target is the average over many games. One million-point game is an anecdote.

A confidence bound on the mean computed by resampling whole games; the qualification protocol requires it above one million.

Labels for how well a historical number is backed: by a conversation only, by the written ledger, or by a re-run in the current checkout.

A faster implementation proven to make identical decisions; it adds no evidence about strength.

Evaluating two candidates on the same sequence of dealt discs and hidden values, so that luck is shared and the difference in scores is down to the candidates.

Dividing a training set so that every position from one game lands in the same fold, which keeps near-duplicate positions from one game out of both the training and the held-out data.

The share of positions on which a model's first-choice column matches the label's first choice.

The share of pairs of columns at a position that a model orders the same way as the label does.

The value a model's chosen column gives up against the label's best column at that position, scaled by the spread of values there and averaged over positions; zero means it always picked the best.

How well a measurement agrees with itself when the data is split in two and each half is scored separately; low agreement means the measurement is mostly noise.