N-tuple networks and learning from play

Instead of hand-writing what makes a Drop7 board good, learn it — from small patches of cells, and from millions of games the program plays against itself.

This is the family that tried to replace human judgement with data. It contains the repository's first learned policy and, after it, a long line of reinforcement-learning variants: temporal-difference updates, Q-learning, policy gradients, actor-critic, and expert iteration. None of them produced a policy stronger than the hand-tuned fair depth-4 search that the project uses as its reference. For how this family compares with the other three ways of learning tried here, see four ways a program can learn Drop7.

What an n-tuple network is

Pick a small window of cells — four in a row, four in a column, or a 2×2 block. Read the contents of that window as an address, and look the address up in a table of learned numbers. Do that for every window on the board, add all the numbers together, and the sum is the network's opinion of the position.

The position in the figure is this repository's TypeScript rules engine playing a game, not a drawing. The native implementation in src/core/native/ntuple.hpp uses 92 windows — 28 horizontal, 28 vertical, and 36 square — sharing 17 underlying tables.

Nothing in that machinery knows Drop7's rules. It only knows which small patterns of cells tended to precede good games. That is the appeal: a table lookup is far cheaper than a neural network, and the tables can hold millions of distinct local shapes. In 2048, the game this idea was borrowed from, tables like these are enough to play the game very well.

And "learning from play"

No human labels the boards. The program plays a game, sees what happened, and nudges the weights it looked up toward the outcome. Play enough games and the weights are supposed to converge on something that ranks positions correctly. The variants in this family differ mostly in what they nudge toward: the score that followed, the number of moves survived, the value of the best next move, or the advantage of one column over another.

How it works, step by step

1. Read the position. Every program in this family is a public-information policy: the board, the visible next disc, and the rise clock. No seed, no hidden gray value, no score, no move number.
2. Evaluate. Look up all 92 windows (plus, in some variants, extra features for board height, disc counts, and rise phase) and add the weights.
3. Choose a column. Simulate dropping the visible disc in each legal column, evaluate the resulting board — its afterstate, and take the best. Some variants put a shallow expectimax search on top and use the network only at the leaves.
4. Learn. After the move, compare the prediction with what actually followed, and adjust every weight that contributed. Repeat for hundreds of thousands of games, or in the later experiments tens of millions of moves.
5. Freeze and test. Freeze the weights, then play a fixed cohort of complete games against a reference policy on the same seeds.

What happened

The first version worked well enough to be interesting and was never beaten by anything that came after it. Adding a two-move look-ahead on top of the learned values lifted it from about 182,000 to about 232,000 points per game on a fixed 64-game development probe. The improvement was real and measured on paired games. But by the time the hand-tuned depth-4 search was properly measured, that search was scoring around 308,000 on its own 64-game reference cohort, and the learned policies were not close.

Every reinforcement-learning variant that followed landed lower. The clearest example: a phase-aware n-tuple trained on 50 million moves scored 181,733 points per game, and the two-rise-deep search built on top of it scored less, at 113,644. A Rainbow-style Q-learner beat random play convincingly at 250,000 training steps, then lost decisively to a plain one-move fair search at one million. A PyTorch policy-gradient pipeline, an actor-critic with explicit safety constraints, and eight rounds of expert iteration all finished between roughly 116,000 and 176,000 points per game, and a second, C++ policy-gradient attempt never got past trying to imitate a one-move search.

Approach	Recorded mean score	Mean moves	Cohort	Source
Learned value plus depth-2 search	232,107.156	70.766	64-game development probe	ledger
Optimistic phase n-tuple, direct play	181,733.422	56.359	64-game burned gate	ledger
Primal-dual actor-critic	175,834	55.006	512-game calibration	ledger
PyTorch PPO, best candidate	142,677.781	45.656	64-game development cohort	ledger
Regenerative expert iteration, round 8	116,598	38.046	training roll-ins	ledger
Optimistic phase n-tuple, two-rise search	113,643.969	37.375	64-game burned gate	ledger
Fair depth-4 reference, for comparison	308,295.578	90.031	64 games	ledger

The technical record

Every figure above is ledger-recorded in the experiment history; the family's status rows are in the experiment index under "N-tuple and reinforcement learning". Five of the family's sources (bellman-ntuple.cpp, flow-curriculum-rainbow.cpp, manifold-gail-development.cpp, manifold-gail-scaled.cpp, curriculum-option-ppo.cpp) carry a task-record only verdict: an index row exists, but no ledger protocol, artifact hash, or per-game data was retained, so those rejections cannot be re-derived.

The 308,295.578 / 90.031 reference is itself weaker evidence than it looks. It appears once in the ledger, as an internal bootstrap comparator inside the regenerative expert-iteration run; the identity of its 64 seeds, its dispersion, and its censoring statistics were not retained. This is recorded as finding H2 in docs/exploratory/audit-03-claim-arithmetic.md.

The scoring-mode assignments quoted on the individual pages come from the same audit, which classifies every recorded (score, moves, n) triple in the ledger by which level bonus is arithmetically consistent with it. Where the ledger states the mode, the pages say so; where the audit infers it, the pages say that too.

No approach in this family opened a protected or final cohort. Several left their later gameplay ranges unopened, which is recorded per experiment.

What this taught us, and what is still open

The representation was never the bottleneck. The strongest single lesson is negative and repeated: more capacity, more training, and better-conditioned updates did not fix these policies. A shared-parameter bug in the temporal-difference update was found and corrected, and the corrected policy still failed. Conditioning the tables on the rise clock closed under 3% of the gap to the reference search. Training a Q-function to 50 million moves produced a policy no better than one trained far less.

The bottleneck was which moves the data covered. Almost every program here learned from the move it actually played, then, at play time, was asked to rank up to seven columns it had never been scored on. That is the sibling trap. The expert-iteration run showed that even regenerating fresh on-policy games every round did not remove it, because the targets still described only the played action.

Search on top of a weak value can hurt. The two-rise rollout over the optimistic phase n-tuple was strictly worse than just playing the network's first choice. A deeper look-ahead magnifies whatever the leaf evaluator gets wrong, and this one was wrong.

What is still open. Nobody has trained an n-tuple on data that scores every legal column at each position under shared imagined futures. The sibling-ranking and scale-out pages describe what that would take. Until it is tried, this family's negative results rule out the objectives that were tested, not pattern learning itself.