A PyTorch policy network, cloned then trained by playing

The problem

A search scores boards and compares the columns afterwards. A policy network skips the scoring: the board goes in and seven numbers come out, one per column, which are turned into probabilities. Playing means taking the largest. Training means pushing probability toward the columns that led to more score over the rest of the game.

The attraction is that this learns the decision directly, so it never has to build a value that generalises across positions it will not visit. The difficulty is sample efficiency: policy-gradient methods need an enormous number of games, and Drop7 games are short until a policy is good. An earlier native C++ attempt copied a teacher first and then tried to improve by playing, and failed its imitation gate. The question here was whether a much faster environment, a bigger network and a stronger teacher would change that.

Proposed solution

Do the same two stages with better tools. Copy an exact two-move fair search first, so the network starts from a competent player, then drop the imitation anchor and let it improve from its own games. The network is small, 263,720 parameters, and averages each board with its mirror image before masking illegal columns, so its choices are symmetric by construction.

A C++ vector environment owns the exact rules and exposes to Python only the 49 board cells, the visible next disc, the rise clock, and the legal-move mask; no seed, future tape, score, level, move index or history crosses the boundary.

How it works

1. Run the fast environment. Finished games are not silently reset: a terminal transition is reported unambiguously before a slot is reused.
2. Generate teacher data. The same environment runs exact one- and two-move fair searches in parallel, with five reveal strata.
3. Clone. Train the network on the two-move teacher's chosen action, mixed with a softer target that also matches the teacher's relative scores for the other columns, then run rounds of relabelling the student's own positions.
4. Gate. The clone must agree with the teacher on 55% of held-out decisions before policy-gradient training is allowed.
5. Improve by playing. In a separately authorised experiment, drop the imitation anchor and run 32 rounds of 512 complete games, updating the policy from its own experience. The output at every step is a probability for each of the seven columns; the greedy player takes the largest.

What happened

Three attempts, one of which completed. The clone matched the teacher on 47.2% of held-out decisions against the preregistered gate, twice: a single permitted correction, aimed at one concrete defect (a catastrophic losing move can dominate a position's range of values and flatten the soft targets among the sensible columns), made held-out agreement very slightly worse and the cross-entropy worse still. The small network had overfit, and policy-gradient training did not run (ledger; rejected, ledger-recorded).

A separately preregistered direct run, which skipped the anchor and trained on its own games, exceeded its memory ceiling seconds after training began, having completed exactly one optimizer update. No candidate was frozen and no score was read. A follow-up audit projected that a fresh process would fit; an empirical preflight disproved the projection and stopped again.

The third attempt changed only the memory schedule (the same update, done in four chunks), ran to completion, and only then opened its single 64-game development cohort. There the candidate averaged 142,678 points against 241,825 for the two-move search it had been cloned from. It improved on the original clone in the mean, but the one-sided 95% lower bound of the paired difference was slightly below zero, so it did not clear "better than the thing it started from" with confidence. Every performance gate failed except zero censoring.

What we learned

Policy-gradient training did improve the clone, by about 9% over 16,384 games on the 64-game development cohort, and left it at 59% of the two-move search it was cloned from (ledger-recorded). This is the only completed reinforcement-learning run in the family that improved a policy and then measured it on unseen games, and it states the sample-efficiency problem as a number.

The clone could not reach 55% agreement with a two-move search. Two different networks in two different frameworks failed the same kind of imitation gate; whatever a fair search knows about a Drop7 position, a small feed-forward network trained on chosen actions has not been able to absorb it. Resource discipline was part of the science here: three separate stops came from memory limits, each recorded with the exact peak, the exact threshold and which seeds stayed unopened, and when an analytical memory estimate disagreed with a measurement, the measurement won.

Every stage of this line trained on the action a teacher or the policy itself chose. None trained on data covering every legal column, the sibling trap again, and none had a compute budget of the kind that makes policy-gradient methods work in other games. The open question is whether a policy network given sibling-complete data and a budget of that size would close the gap; the scale-out proposal states what that would require.

Agent contextRecords and provenanceopen ↓close ↑

• Status in the experiment index: rejected, ledger-recorded, "the warm-start gate failed; a separately authorized direct run later aborted at its resource limit." The ledger records a third stage the index row does not mention: an ordered gradient-accumulation run that completed all 32 iterations and was then rejected on its development cohort.
• Development cohorts: the clone stage read a fixed 32-game development cohort; the completed gradient-accumulation run read the single 64-game development cohort 0x3d3a0000...0x3d3a003f after its checkpoint and exports were frozen. The direct run's permitted training ranges were 0x3d340000...0x3d340fff and 0x3d350000...0x3d352fff; its reserved development cohort 0x3d360000...0x3d36003f was never opened. The completed run trained on 0x3d390000...0x3d393fff.
• Sources: torch-env.cpp (the batched native environment) and train.py (behaviour cloning, PPO, self-tests, preflights). Artifact and checkpoint SHA-256 hashes for every stage are in the ledger.

Agent contextFull results tableopen ↓close ↑

Clone stage, on a fixed 32-game development cohort:

Policy	Mean score	Mean moves
Random	79,307.875	27.969
Clone	141,986.938	45.125
Clone after the single correction	142,364.031	45.313
Exact fair D1	181,846.438	56.281
Exact fair D2 (the teacher)	191,189.344	58.688

Held-out agreement 0.471840, top-two accuracy 0.697788, cross-entropy 1.367013, against a preregistered 0.55 agreement gate; after the correction, 0.471784, 0.694836, and 1.426484. Teacher corpora: 768 games / 56,484 states for training (mean exact-D2 score 244,207.953), 256 games / 17,951 states held out (231,063.223), 512 games / 21,654 states of student-distribution relabelling (130,565.021). Zero censored games throughout.

Completed gradient-accumulation run, on the single 64-game development cohort 0x3d3a0000...0x3d3a003f:

Policy	Mean score	Mean moves
Random	77,674.406	27.484
Original clone	130,797.406	42.500
Trained candidate	142,677.781	45.656
Exact fair D1	180,713.422	56.359
Exact fair D2	241,825.203	72.594

Candidate-to-clone ratios 1.090830 (score) and 1.074265 (moves) against registered 1.15 floors; candidate-to-D2 ratios 0.590004 and 0.628928. The paired candidate-minus-clone score difference was +11,880.375 with standard error 7,201.038 and a one-sided 95% lower bound of −141.054. Training-batch means rose from 112,175.525 points / 37.168 moves at iteration 1 to 136,608.055 / 44.037 at iteration 32, over 16,384 games in 1,597.281 seconds with 371.89 MiB peak resident. No intermediate checkpoint was selected.

Agent contextValidity, gates and limitationsopen ↓close ↑

• The aborted direct run peaked at 551,567,360 bytes against a 536,870,912-byte ceiling and stopped 2.755 seconds after training began, after exactly one optimizer update; its partial checkpoint is explicitly marked non-deployable. The subsequent fresh-process preflight peaked at 503.97 MiB against a 480 MiB admission threshold and stopped.
• The gradient-accumulation change was proved numerically equivalent to the original update to within about 1e-7 on losses and 5e-9 on gradients before it was allowed to run. Exports are bit-exact between Python and raw float inference and agree with TorchScript to 1e-6.
• The completed run failed every performance gate except zero censoring: both candidate-to-clone ratios below the 1.15 floors, both candidate-to-D2 ratios below 1.0, and a negative one-sided lower bound on the paired difference. No continuation, tuning, or seed beyond the development cohort was opened.
• The clone-stage cohort is 32 games and the final cohort 64 games; both are development data and neither is confirmation evidence.

Agent contextScoring modeopen ↓close ↑

Corrected 17,000-point Hardcore award throughout, and audit-03 agrees. No historical 7,000-point number appears on this page.