---
title: A PyTorch policy network, cloned then trained by playing
family: ntuple-rl
summary: A small convolutional network copies a two-move search, then improves through 16,384 games. It finished about 40% short of its teacher.
status: rejected
evidence: ledger-recorded
reads: public
kind: strategy
technique: policy-gradient
featured: true
---
## The problem
A search scores boards and compares the columns afterwards. A
[policy network](/learn/techniques/policy-gradient) skips the scoring: the
board goes in and seven numbers come out, one per column, which are turned into
probabilities. Playing means taking the largest. Training means pushing
probability toward the columns that led to more score over the rest of the
game.
The attraction is that this learns the decision directly, so it never has to
build a value that generalises across positions it will not visit. The
difficulty is sample efficiency: policy-gradient methods need an enormous
number of games, and Drop7 games are short until a policy is good. An earlier
[native C++ attempt](/approach/ntuple-rl/native-ppo) copied a teacher first
and then tried to improve by playing, and failed its imitation gate. The
question here was whether a much faster environment, a bigger network and a
stronger teacher would change that.
<PolicyNetShape caption="A policy network's shape. The bar heights are illustrative: no retained record publishes a trained policy's per-column probabilities." />
## Proposed solution
Do the same two stages with better tools. Copy an exact two-move fair search
first, so the network starts from a competent player, then drop the imitation
anchor and let it improve from its own games. The network is small, 263,720
parameters, and averages each board with its mirror image before masking
illegal columns, so its choices are symmetric by construction.
A C++ vector environment owns the exact rules and exposes to Python only the
49 board cells, the visible next disc, the [rise clock](/learn/glossary), and
the legal-move mask; no seed, future tape, score, level, move index or history
crosses the boundary.
## How it works
1. Run the fast environment. Finished games are not silently reset: a terminal
transition is reported unambiguously before a slot is reused.
2. Generate teacher data. The same environment runs exact one- and two-move
[fair searches](/approach/fair-expectimax/reference) in parallel, with
five [reveal strata](/learn/glossary).
3. Clone. Train the network on the two-move teacher's chosen action, mixed
with a softer target that also matches the teacher's relative scores for
the other columns, then run rounds of relabelling the student's own
positions.
4. Gate. The clone must agree with the teacher on 55% of held-out decisions
before policy-gradient training is allowed.
5. Improve by playing. In a separately authorised experiment, drop the
imitation anchor and run 32 rounds of 512 complete games, updating the
policy from its own experience. The output at every step is a probability
for each of the seven columns; the greedy player takes the largest.
## What happened
Three attempts, one of which completed. The clone matched the teacher on 47.2%
of held-out decisions against the preregistered gate, twice: a single
permitted correction, aimed at one concrete defect (a catastrophic losing move
can dominate a position's range of values and flatten the soft targets among
the sensible columns), made held-out agreement very slightly worse and the
cross-entropy worse still. The small network had overfit, and policy-gradient
training did not run ([ledger](/docs/research/history); rejected,
ledger-recorded).
A separately preregistered direct run, which skipped the anchor and trained on
its own games, exceeded its memory ceiling seconds after training began,
having completed exactly one optimizer update. No candidate was frozen and no
score was read. A follow-up audit projected that a fresh process would fit; an
empirical preflight disproved the projection and stopped again.
The third attempt changed only the memory schedule (the same update, done in
four chunks), ran to completion, and only then opened its single 64-game
development cohort. There the candidate averaged 142,678 points against
241,825 for the two-move search it had been cloned from. It improved on the
original clone in the mean, but the one-sided 95% lower bound of the paired
difference was slightly below zero, so it did not clear "better than the thing
it started from" with confidence. Every performance gate failed except zero
censoring.
## What we learned
Policy-gradient training did improve the clone, by about 9% over 16,384 games
on the 64-game development cohort, and left it at 59% of the two-move search
it was cloned from (ledger-recorded). This is the only completed
reinforcement-learning run in the family that improved a policy and then
measured it on unseen games, and it states the sample-efficiency problem as a
number.
The clone could not reach 55% agreement with a two-move search. Two different
networks in two different frameworks failed the same kind of imitation gate;
whatever a fair search knows about a Drop7 position, a small feed-forward
network trained on chosen actions has not been able to absorb it. Resource
discipline was part of the science here: three separate stops came from memory
limits, each recorded with the exact peak, the exact threshold and which seeds
stayed unopened, and when an analytical memory estimate disagreed with a
measurement, the measurement won.
Every stage of this line trained on the action a teacher or the policy itself
chose. None trained on data covering every legal column, the
[sibling trap](/learn/concepts/ranking-siblings) again, and none had a compute
budget of the kind that makes policy-gradient methods work in other games. The
open question is whether a policy network given sibling-complete data and a
budget of that size would close the gap; the
[scale-out proposal](/learn/concepts/scale-out-direction) states what that
would require.
<AgentContext summary="Records and provenance">
- Status in the [experiment index](/docs/research/experiment-index): rejected,
ledger-recorded, "the warm-start gate failed; a separately authorized direct
run later aborted at its resource limit." The [ledger](/docs/research/history)
records a third stage the index row does not mention: an ordered
gradient-accumulation run that completed all 32 iterations and was then
rejected on its development cohort.
- Development cohorts: the clone stage read a fixed 32-game development
cohort; the completed gradient-accumulation run read the single 64-game
development cohort `0x3d3a0000...0x3d3a003f` after its checkpoint and
exports were frozen. The direct run's permitted training ranges were
`0x3d340000...0x3d340fff` and `0x3d350000...0x3d352fff`; its reserved
development cohort `0x3d360000...0x3d36003f` was never opened. The
completed run trained on `0x3d390000...0x3d393fff`.
- Sources: `torch-env.cpp` (the batched native environment) and `train.py`
(behaviour cloning, PPO, self-tests, preflights). Artifact and checkpoint
SHA-256 hashes for every stage are in the ledger.
</AgentContext>
<AgentContext summary="Full results table">
Clone stage, on a fixed 32-game development cohort:
| Policy | Mean score | Mean moves |
| --- | ---: | ---: |
| Random | 79,307.875 | 27.969 |
| Clone | 141,986.938 | 45.125 |
| Clone after the single correction | 142,364.031 | 45.313 |
| Exact fair D1 | 181,846.438 | 56.281 |
| Exact fair D2 (the teacher) | 191,189.344 | 58.688 |
Held-out agreement 0.471840, top-two accuracy 0.697788, cross-entropy
1.367013, against a preregistered 0.55 agreement gate; after the correction,
0.471784, 0.694836, and 1.426484. Teacher corpora: 768 games / 56,484 states
for training (mean exact-D2 score 244,207.953), 256 games / 17,951 states held
out (231,063.223), 512 games / 21,654 states of student-distribution
relabelling (130,565.021). Zero censored games throughout.
Completed gradient-accumulation run, on the single 64-game development cohort
`0x3d3a0000...0x3d3a003f`:
| Policy | Mean score | Mean moves |
| --- | ---: | ---: |
| Random | 77,674.406 | 27.484 |
| Original clone | 130,797.406 | 42.500 |
| Trained candidate | 142,677.781 | 45.656 |
| Exact fair D1 | 180,713.422 | 56.359 |
| Exact fair D2 | 241,825.203 | 72.594 |
Candidate-to-clone ratios 1.090830 (score) and 1.074265 (moves) against
registered 1.15 floors; candidate-to-D2 ratios 0.590004 and 0.628928. The
paired candidate-minus-clone score difference was +11,880.375 with standard
error 7,201.038 and a one-sided 95% lower bound of −141.054. Training-batch
means rose from 112,175.525 points / 37.168 moves at iteration 1 to
136,608.055 / 44.037 at iteration 32, over 16,384 games in 1,597.281 seconds
with 371.89 MiB peak resident. No intermediate checkpoint was selected.
</AgentContext>
<AgentContext summary="Validity, gates and limitations">
- The aborted direct run peaked at 551,567,360 bytes against a
536,870,912-byte ceiling and stopped 2.755 seconds after training began,
after exactly one optimizer update; its partial checkpoint is explicitly
marked non-deployable. The subsequent fresh-process preflight peaked at
503.97 MiB against a 480 MiB admission threshold and stopped.
- The gradient-accumulation change was proved numerically equivalent to the
original update to within about 1e-7 on losses and 5e-9 on gradients before
it was allowed to run. Exports are bit-exact between Python and raw float
inference and agree with TorchScript to 1e-6.
- The completed run failed every performance gate except zero censoring: both
candidate-to-clone ratios below the 1.15 floors, both candidate-to-D2 ratios
below 1.0, and a negative one-sided lower bound on the paired difference. No
continuation, tuning, or seed beyond the development cohort was opened.
- The clone-stage cohort is 32 games and the final cohort 64 games; both are
development data and neither is confirmation evidence.
</AgentContext>
<AgentContext summary="Scoring mode">
Corrected 17,000-point Hardcore award throughout, and
[audit-03](/docs/exploratory/audit-03-claim-arithmetic) agrees. No historical
7,000-point number appears on this page.
</AgentContext>