Value and policy learning

Searching ahead is expensive: to compare seven columns properly you have to imagine what happens after each one, and after the disc that follows, and so on. The alternative is to learn the answer. Play a lot of games, write down what happened, and fit a model that says "this board is worth about this much" or "from here, drop in column 4". This family is twelve attempts at that.

Eleven of them fell short, and (this is the useful part) most of them fell short for the same reason.

The recipe, and where it breaks

Every approach here follows one shape. The differences are in what gets recorded and what the model is asked to output.

The gap in the last box has a name in this repository — sibling extrapolation, and it has its own page, because it is the single most important lesson in the record: evaluating a board, and the sibling trap. In one line: a model can predict the outcome of played positions very accurately and still order the seven available columns wrongly, because six of those columns are boards nothing in its training data ever measured. Low prediction error is not the same skill as correct ranking. Terms like value, n-tuple, NNUE and DQN are all one line each in the glossary.

What is in this family

Grouped by what the model is asked to produce.

Learn how long the game will last, from a board.

• Denoised public value — the one that worked best. Instead of labelling a board with the single future that happened to follow it, label it with the average of 32 independently simulated futures. Includes two follow-ups that tried to put the model to work and did not succeed.
• Chance-state NNUE — values the board before the next disc is dealt, so the value never has to be split seven ways by which disc arrives.
• Structured NNUE — gives each of the 49 cells its own learned embedding, and a variant that labels every legal sibling rather than only the played one.
• Monte Carlo state value — label each board with the actual remaining lifetime of the game it came from, plus survival checks at 25 and 50 moves.
• Conservative fitted policy iteration — learns a whole distribution over remaining lifetime and only departs from its safe fallback when the improvement is statistically supported.

Learn the value of a move, then take the best one.

• Double-DQN: the textbook reinforcement-learning method, plus a follow-up with longer credit assignment and optional privileged demonstrations.
• Monte Carlo return — the same idea without any bootstrapping: score each move by the full realized score of the game that followed it.
• Sibling advantage — drops the absolute value entirely and learns only how much better one column is than its siblings, with every sibling continued under the same random future.

Copy a stronger player.

• D4 root-Q clone — compress the expensive fair depth-4 search into a small fast network that reproduces its ordering of columns.
• Phase distillation — the same idea against the exact depth-3 phase policy, once as a direct action copy and once as a numeric-score copy.

Skip the value function.

• Direct public policy — evolve the policy's own parameters against complete games; nothing is ever regressed on a label.
• General value model — a TypeScript workbench for fitting board and policy models from several kinds of teacher on separate seed lanes.

What happened

One approach in this family came out ahead of its own baseline: the denoised public value, which changed the labels rather than the network. It passed every prediction gate it had declared in advance and, in a small eight-game confirmation, its games lasted longer than the policy it was correcting — 79.5 to 88.125 mean moves (ledger-recorded; see the approach page for the important caveats, including that the confidence bounds were negative and no scores were retained for that cohort).

Everything else was retired. The patterns are consistent enough to state as findings:

• Predicting well and ranking well are different skills. The clearest single case is the D4 clone, which reproduced its teacher's top choice 76% of the time on training positions and 24.7% on held-out positions (ledger-recorded).
• More data and more capacity did not fix it. The scaled sibling-advantage study increased both, and the experiment index records that it still increased regret relative to exact search (task-record only).
• A model that looks right on paper can make no difference in play. The phase-5 value veto was allowed to act only at the one moment the search cannot see through; on its first paired game it switched zero times out of 22 opportunities and then ran out of its runtime budget.
• A small screen that looks positive can be a false positive. The denoised guided veto passed a four-game screen by a wide margin and reversed on the eight-game confirmation.

What this taught us, and what is still open

The family's collective negative result is what motivated the current research line. If the problem is that training data only ever measures the move that was played, the fix is to measure every legal move at each position, under the same random futures, so the comparison is fair — successor-closed data. That is exactly what the afterstate-learning family builds, and its records show the difference: with a stable enough ranking target, a learned model finally beat its own teacher, while still trailing fair depth-4 search.

Still open from this family:

• The denoised lifetime model is the best-predicting model in the repository and was never given a deployment rule on top of fair D4 that could actually fire. The exploratory blind-spot audit names this as an unresolved opportunity rather than a settled negative.
• Nothing here has been tested as a bounded correction to fair D4 with a gate wide enough to produce switches, which is the shape the research status now recommends for every learned signal.