Why one great game proves nothing

The goal of this project is a policy whose mean score over many games is above one million points. A retained 64-game fair-D4 cohort contains one game that scored 1,017,234 points, but the same policy averaged 321,991.72 on that cohort. The single game is a milestone, not a qualification.

Sixty-four games, one dot each

Here is every game of a retained cohort: the repository's reference policy, fair depth-4 expectimax, over 64 complete games.

table view — the recorded cohort summary

quantity	score	moves
mean	321,991.719	94.0625
median	266,282	80
lower quartile	188,700.75	57.25
minimum	104,731	35
maximum	1,017,234	280
standard deviation	187,501.832	—

The shape is the point. Most games cluster low. A long thin tail runs off to the right. At the far end sits a genuine million-point game worth 1,017,234 points. Yet the same policy averaged 321,991.72, and its median was 266,282.

That single game is not a fluke to be explained away, and it is not evidence of a million-point policy either. It is what a heavy tail looks like.

Mean, median, and why they disagree

The median is the middle game: half scored more, half scored less. The mean is the total divided by the number of games. When a distribution is symmetric they are close. When a few games are enormous, the mean is dragged towards them and the median is not.

In this cohort the mean is 321,991.72 and the median 266,282: the average game is not the typical game. The standard deviation measures how widely the games are spread. Here it is about 187,502, which finding 01 notes is 58% of the mean itself. With 64 games at that spread, the mean is known to roughly ±46,000 at one standard error.

Both numbers are useful and they answer different questions. The median answers what happens in a typical game. The mean answers what happens on average over a long session, and the mean is what the qualification standard is written against, because the target is a mean. That choice makes the standard harder to meet reliably, not easier: a mean over a heavy-tailed variable is a jumpy quantity.

What a small cohort can and cannot say

Every recorded figure for the reference policy, on one axis, with the target:

Look at the first two rows. The same reference policy measured 400,675.25 over eight games and 308,295.578 over 64. Nothing about the policy changed. Eight games is not enough to pin down the mean of a distribution with a tail this long, which is exactly why the repository's benchmark contract calls a 1–8 game run a PILOT, whose entire permitted purpose is to "find bugs and project runtime/memory" with no strength claim (benchmark contract).

Now look at the last row. A single million-point game is above the target line, but the cohort mean is not.

Comparing two policies: pair the games

If score is this noisy, how does anyone tell whether a change helped? The answer is not "run more games and compare averages", or not only that. It is to make the two policies play the same games.

A cohort is a fixed, ordered list of seeds. Both the candidate and the reference play seed 1, then seed 2, and so on, and the comparison is done seed by seed: the candidate scored more on this game, less on that one. This is a paired comparison, and it removes the largest source of noise, which is that some games are simply luckier than others. The methodology requires exactly this: "per-game paired results against the reference", and asks for wins, ties and losses, not just a difference of means (methodology, benchmarks).

You have already seen a small paired comparison on this site: the two look-alike boards on what makes a board good? were replayed under the same sixteen scripted disc tapes, and the result was reported as six wins, nine ties and one loss rather than as two averages.

The same rule is why the repository refuses to promote an aggregate win when a preregistered subgroup fails: a policy that gains overall but regresses on a whole class of positions has not been shown to be better.

Bootstrap lower bounds, without the statistics

Suppose a candidate averages 1,050,000 over 256 games. Is its true long-run mean above one million, or did it get a lucky tail?

The bootstrap answers that by resampling. Take the 256 recorded game scores. Draw 256 of them at random with replacement. Some games appear twice and some not at all. Compute the mean of that imaginary cohort, then repeat thousands of times. You now have a spread of plausible means, built entirely out of the games you actually played. The 5th percentile of that spread is a one-sided 95% lower bound: a value the true mean is unlikely to be below.

Two things make this the right tool here. It assumes nothing about the shape of the distribution, which matters when the shape is a long tail. And it resamples whole games, respecting the rule that a complete game is the independent unit of measurement. Move-level or position-level statistics can diagnose a model, but cannot supply a confidence interval for a policy.

The qualification ladder is built from these bounds:

Stage	Games	What it must show
Development gate	256	observed mean above 1,050,000 and a one-sided 95% whole-game bootstrap lower bound above 1,000,000
Protected validation	256	mean above 1,000,000, with both bootstrap and Student-t one-sided lower bounds above it, no illegal moves, no runner failures
Final confirmation	256	the unchanged, frozen candidate repeats those conditions on a one-shot cohort

From the archived validation protocol as summarised in docs/methodology.md. The frozen record states that no candidate has qualified and that neither the protected nor the final seeds have been opened.

The development gate demands 1,050,000, not 1,000,000. That margin exists because a mean measured on data you were allowed to look at is optimistic, and a heavy tail makes it more optimistic still.

How to read a number on this site

• Ask how many games. Eight games is a pilot. Sixty-four is a development cohort, useful for deciding what to try next and explicitly "never fresh confirmation". Two hundred and fifty-six on unseen seeds is a gate.
• Ask whether it is a mean or a game. A single score is an anecdote no matter how large.
• Ask whether it was paired. An unpaired comparison of two policies on different seeds is mostly measuring luck.
• Ask what the label says. This repository marks figures as task-record only, ledger-recorded, repository-verified or reproduced. Those labels distinguish a retained run from a result preserved only in the historical record.

The technical record

The dot plot. Per-game rows from runs/RUN-A51D-d4/d4-standard64.json, snapshotted into web/content/learn/data/d4-cohort-scores.json by web/scripts/extract-d4-cohort.ts, which copies recorded fields and computes nothing. Policy fair-d4, 64 games, seeds 0xa51d0000–0xa51d003f, seed lease SEEDLEASE-A51D, 2,000-move cap, 0 censored games. Recorded summary: mean 321,991.71875, median 266,282, lower quartile 188,700.75, minimum 104,731, maximum 1,017,234, standard deviation 187,501.831766; mean moves 94.0625. Reported in docs/exploratory/finding-01-score-is-survival.md as an exploratory, development-tier result reproduced in this checkout on 2026-08-20. runs/ is not committed, so on a checkout without it the figure degrades to a message rather than inventing data.

The ledger figures are 400,675.25 over 8 games and 308,295.578 over 64 games, against depth-3's 235,071.25 over the same 8. These are the ledger-recorded entries in docs/research/status.md, labelled there as a small confirmation cohort and a broader reference cohort respectively. They are different cohorts from the finding-01 run above and must not be pooled.

Statistical rules. The independent unit is a complete game; report mean, median, lower quartile and minimum score, mean and lower-quartile lifetime, censor count and cap, clears and reveals per move, chain depth, per-game paired results, and whole-game confidence bounds for a qualification claim (methodology, benchmarks). A game stopped at the 2,000-move cap is censored, keeps the score already earned, and is a lower bound that must never be extrapolated. For qualification the exact bootstrap and Student-t rules of the frozen protocol apply; for earlier tiers the estimate, resampling method and seed, interval direction and number of whole-game samples must all be published.

What this taught us, and what is still open

• Drop7 scores are heavy-tailed: in a retained 64-game cohort the standard deviation was 58% of the mean, and the best game scored 1,017,234 against a mean of 321,991.72.
• A million-point game and a 300,000-point average are perfectly consistent, and this repository has both.
• Small cohorts move a lot: the same policy measured 400,675.25 over 8 games and 308,295.578 over 64.
• Pairing games and reporting wins, ties and losses removes far more noise than simply running more unpaired games.
• One game has already reached a million. The open question is whether any policy can put the average there. Nothing in this repository is close; the reference policy would need roughly three times its current mean lifetime, as score is survival works out.