Q-learning and value learning

The technique

Keep a number for 'how good is it to do this here', play, and after each step nudge that number toward what actually happened plus the best number available next.

Read the primer

Approaches using this technique

10 approaches, featured pages first. Each card is one theory of how to choose a column and what happened when it was tried.

Q-learning and value learningfeatured

Denoised public value

Instead of asking one future what a board was worth, simulate 32 futures from it and average them.

completedrecorded

Q-learning and value learningfeatured

Klein-Friedmann linear Q-learning

Score each column with six hand-made numbers about what the drop does right now, learn the six weights from experience, and find out how much of Drop7 that can see.

completedreproduced

Q-learning and value learning

Double-DQN and continuation

Use textbook reinforcement learning to score each column from experience, reward survival, and add a short look-ahead.

completed

Q-learning and value learning

Long outcomes

Play each column forward for twenty-five moves under shared imagined futures, then learn from those outcomes. All five models ranked moves worse than a cheap exact search.

rejectedrecorded

Q-learning and value learning

Monte Carlo return

Score every column with what whole games that started from it actually ended up earning, then always drop in the column with the highest learned number.

rejected

Q-learning and value learning

Monte Carlo state value

Label every board with how many more moves the game it came from actually lasted, then only leave the safe move when the learned numbers agree it is better.

rejectedrecorded

Q-learning and value learning

One-ply Q as a pruning prior and a refit leaf

Fit a linear action value to the depth-4 search's own sibling values, then use it not to play but to decide which siblings the search may skip, and separately refit the leaf to the search's own value.

completedreproduced

Q-learning and value learning

Panel value

Builds complete training data by playing every legal column through the same imagined futures, then learns from it. Both models still ranked columns worse than the four-move search.

rejected

Q-learning and value learning

Rainbow-style Q-learning over patterns

Learn a score for every column with Atari-style training and a pattern table instead of a deep network. It beat random play, then lost to a one-move search.

rejectedrecorded

Q-learning and value learning

Sibling network on the 200-move panel

Start a network as an exact copy of the reference search, train it on a locked panel where every legal column was measured, and watch it get worse.

rejected

Agent contextApproach directories in this group, by family

constructive-reservoir

approaches/constructive-reservoir/panel-value

d4-long-outcome

ntuple-rl

approaches/ntuple-rl/rainbow-q

value-policy-learning