Klein-Friedmann linear Q-learning

Claim: The six-feature linear Q-learner of Erez Klein and Ben Friedmann (Stanford CS221 final report 'Drop7', code github.com/ekreate/cs221-final-project at commit 8cc8a0e) is a survival heuristic whose quality is fixed by its features, not by its training length or its regularisation: (i) its published figures reproduce from its own code (uniform-random about 31 moves, the trained agent about 49-50 moves on 10,000 test games with no exploration); (ii) because the step size is 1/t over MOVES, the weights are effectively frozen after a few hundred games, so a 50,000-game run and a 300-game run yield policies within 2 moves of each other and the ridge term has no measurable effect on the shipped configuration; (iii) transplanted onto corrected five-move Hardcore rules in the repository engine with a legal-column argmax and re-learned with survival reward, the same six features outlive uniform-random play by a paired-significant margin but reach a mean corrected score below two thirds of the fair depth-4 ledger mean (below 205,530 against 308,296) and lose to fair d4s7 on paired seeds; and (iv) a slower step schedule (per-game, harmonic, or constant) does not raise the transplanted policy's mean lifetime over the upstream schedule by more than the pilot cohort's paired detection floor.

This theory is currently mixed at the pilot (a small run to find bugs and project cost, not a strength claim) level.

Claim: For the six Klein-Friedmann features on corrected five-move Hardcore rules, the policy plateau observed in EX-20260902-kf-linear-q-rust-transfer-4328a730 (about 36 moves for engine-learned weights, 40.5 for the authors' transplanted weights) was set by the temporal-difference optimizer, not by the feature set: a direct search over the six weights against whole-game mean lifetime on training seeds finds a weight vector that, frozen and read once on the same 256-game pilot cohort, outlives the transplanted weights by a paired one-sided 95% lower bound above zero; yet even this optimum stays below two thirds of the fair depth-4 ledger mean (205,530 points) and loses to fair d4s7 on the paired 32-seed subset.

This theory is currently supported-as-tested at the pilot (a small run to find bugs and project cost, not a strength claim) level.

Claim: On states visited by fair d4s7 play under corrected five-move Hardcore rules, a linear action value over cheap one-ply features (the seven-stratum mean of the eighteen fair-leaf terms of the afterstate, the mean score delta, the terminal fraction, the six Klein-Friedmann drop features, a rise-clock one-hot and a bias), fitted by ridge regression to exact fair-d4s7 sibling values on training-game roots, (a) reproduces d4's top choice on held-out roots more often than the exact one-ply search value does, but only by a small margin (top-1 stays below 0.65), because a linear one-ply evaluator cannot represent the four-ply value function; (b) nevertheless keeps d4's best action inside its top three on at least 95% of held-out roots; so that (c) a fair search that expands only the prior's top three siblings at every interior max node with two or more plies remaining reproduces the full-width d4s7 action on at least 95% of held-out roots with mean normalised regret at most 0.02, at no more than 35% of the full-width logical work; and (d) the same pruning at depth five, at no more than twice the full-width d4s7 work per root, chooses actions whose mean regret under exact d5s7 values is lower than full-width d4s7's.

This theory is currently mixed at the pilot (a small run to find bugs and project cost, not a strength claim) level.

It compares drop7-kf-linear-q (Rust crate on drop7-rs): six-feature linear Q policy, several trained weight sets against uniform-random legal policy and centre-first policy on identical seeds (primary); fair d4s7 (drop7-rs Searcher, FairLeaf, 65,536-entry depth table, terminal utility -1,000,000, policy seed 0xd7075eed) on the first 32 seeds as a diagnostic reference at the PILOT (a small run to find bugs and project cost, not a strength claim) level, using previously-evaluated-development data.

valid run outcome: pass The run was valid and the outcome was pass (pilot (a small run to find bugs and project cost, not a strength claim)). Read the result.

It compares upstream cs221-final-project Q-learner (external Python simulator, commit 8cc8a0e) against uniform-random policy in the same upstream simulator (the report's baseline) at the CHECK (mechanics checks only, no games played) level, using no-gameplay data.

valid run outcome: pass The run was valid and the outcome was pass (mechanics-only (checks only, no games played)). Read the result.

It compares drop7-kf-linear-q search (CEM over six weights), frozen optimum against transplanted authors' weights (pilot arm transplant-py-seed10), uniform-random legal play, and fair d4s7 on the paired 32-seed subset, all from runs/RUN-20260902T081659Z-d2aa6375/kf-linear-q at the PILOT (a small run to find bugs and project cost, not a strength claim) level, using previously-evaluated-development data.

valid run outcome: pass The run was valid and the outcome was pass (pilot (a small run to find bugs and project cost, not a strength claim)). Read the result.

It compares pruned fair expectimax (drop7-oneply-q evaluate --arm pruned:...) against full-width fair d4s7 (the same FairSearch policy as RUN-20260902T081659Z-d2aa6375's fair-d4s7 arm), 256 games on the same seeds at the PILOT (a small run to find bugs and project cost, not a strength claim) level, using previously-evaluated-development data.

valid run outcome: inconclusive The run was valid and the outcome was inconclusive (pilot (a small run to find bugs and project cost, not a strength claim)). Read the result.

The run was valid; the outcome was pass, at the mechanics-only (checks only, no games played) level. Of 4 preregistered checks, 3 passed and 1 failed.

The shipped Klein-Friedmann code (github.com/ekreate/cs221-final-project @ 8cc8a0edfa04f1a93088c951e217d3cd3d6013f0) reproduces its report in its own simulator: uniform-random means 31.657, 31.733, 31.851 over 5,000 games each (report 31.2), and the Q-learner's 10,000-game test means 49.080, 49.053, 49.015 moves with standard deviations near 11.7 (report 49.61, sd 11.18) after 50,000 training games each, so both preregistered bands pass. Two of the theory's clause-(ii) checks were decided here. (ii-a) Front-loading holds: a learner stopped after 300 training games (13,796 updates) tests at 49.0796 moves against 49.0795 for the 50,000-game learner (2,406,211 updates) on the same 10,000 test games, a difference of -0.0001; the per-1,000-game training curve is already at 46.8 in its first block and 48.2 in its last. (ii-b) The report's regularisation claim does NOT hold for the shipped configuration: with lambda = 0 the learner tests at 48.967 moves, indistinguishable from the lambda = 0.1 arm and far above random, where the report's Figure 5 says the unregularised agent fell below random. With eta = 1/t counted per move, the ridge step eta*lambda*w is of order 1e-5 for almost all of training, so the term cannot act; the report's divergence must come from a schedule the shipped code no longer contains. Units are the upstream simulator's (one point per surviving move, 200-move cap, empty starting board, dropping on a full column ends the game); nothing here is a corrected-Hardcore score or a repository cohort.

Full record →

The run was valid; the outcome was pass, at the pilot (a small run to find bugs and project cost, not a strength claim) level. Of 6 preregistered checks, 6 passed and 0 failed.

On corrected five-move Hardcore rules the six Klein-Friedmann features are a survival heuristic worth about ten moves and 34,000 points over uniform-random play, an order of magnitude below fair depth-4 in points. Pilot on 256 previously evaluated development seeds (0xa5277000-0xa52770ff, 2,000-move cap, no game censored, 0 illegal decisions): uniform-random legal play 26.25 moves / 73263 points; centre-first 21.01 / 55083; the six features re-learned with the upstream schedule (three exploration seeds) 35.79, 35.81, 36.27 moves and 107147, 107231, 109068 points, a paired gain over random of +9.54 moves [LB95 +8.85] and +33884 points [LB95 +31361] for seed a. The authors' own Python-learned weights (reproduction seed 10) transplanted unchanged live 40.53 moves / 123968 points, +4.74 moves over the engine-learned weights [LB95 +3.77]: same features, same update, same schedule, different opening experience (empty board vs gray row) frozen in by the per-move 1/t step size. Fair d4s7 (drop7-rs, 65,536-entry table) on the first 32 seeds: 380205 points / 110.38 moves, beating the transplanted arm by +256507 points [LB95 +182668] on 31/32 paired games. Slower schedules: per-game 1/g diverged to weights near 1e9 (32.47 moves), harmonic tau 50,000 and the score-reward harmonic arm diverged to NaN (20.90 moves, the rightmost-column policy), constant 0.001 gained +0.50 moves over the upstream schedule [LB95 +0.12], the same size as the +0.48 [+0.07] spread between two identical-configuration seeds. Gates G1-G5 pass; G6 records that a slower stable schedule clears LB95 > 0 by under a move, so theory clause (iv) is not supported at the letter while clauses (iii-a) and (iii-b) are. All CHECK gates (feature parity 54,852 pairs / 0 mismatches; update parity 250 steps / exact; determinism and legality) passed before any diagnostic seed was read. Training ran at about 1.4 million moves per second (1.3 s per 50,000-game arm).

Full record →

The run was valid; the outcome was pass, at the pilot (a small run to find bugs and project cost, not a strength claim) level. Of 4 preregistered checks, 4 passed and 0 failed.

A cross-entropy search over the six Klein-Friedmann feature weights against whole-game lifetime (population 64, elite 8, 30 generations of 256 paired training games, then re-selection of nine finalists on a fresh 1,024-game block) converged by generation ten and finished in 4.5 s. Its frozen unit-normalised optimum (0.429, 0.277, 0.145, 0.033, 0.736, 0.419 for lowest-column, row detonations, column detonations, tallest-column, a-clearing-1, the-disc-clears), read once on the 256 pilot games, lives 44.82 moves and scores 138973 points: +18.57 moves over uniform-random [LB95 +17.24] and +4.29 moves [LB95 +2.81], +15005 points [LB95 +9755] over the transplanted authors' weights, winning 136 of 256 paired games. So the plateau of the temporal-difference-learned weights (about 36 moves engine-learned, 40.5 transplanted) was set by the optimizer, not by the feature set. The ceiling of the six features under this search is about 45 moves and 139,000 points; fair d4s7 on the first 32 seeds still beats the optimum by +233362 points [LB95 +153370] on 30 of 32 games, so no SCREEN against depth-4 is warranted. The optimum's direction is 0.75 cosine from the transplanted vector and 0.55 from the engine-learned one; it puts its largest weight on a clearing 1 next to gray discs and nearly none on the tallest-column feature. All three theory clauses are supported as tested at pilot tier.

Full record →

The run was valid; the outcome was inconclusive, at the pilot (a small run to find bugs and project cost, not a strength claim) level. Of 3 preregistered checks, 1 passed and 0 failed.

On the 256 previously evaluated pilot games, full-width fair d4s7 (the same FairSearch as before; its first 32 rows reproduce RUN-20260902T081659Z-d2aa6375's checksums exactly) averages 404,497 points and 116.41 moves (median 315,158, lower quartile 211,976, maximum 1,746,375, no game censored). The pruned search selected by EX-4 (exact one-ply prior, width three at both interior layers) averages 370,889 points and 107.77 moves at 26.8% of the logical work per move (1.40M against 5.22M) and 3.4x faster wall time even under contention. Its paired score delta is -33,608 points with one-sided 95% bounds -69,418 and +1,448 (detection floor 35,114), and its paired lifetime delta is -8.64 moves (bounds -18.35, +0.89; floor 9.54); it wins 121 of 256 games and loses 135. Gate G1 is therefore inconclusive: the non-inferiority bound (-40,000) is not met and a measurable loss (upper bound below zero) is not shown either; the point estimate is 8.3% of the comparator's mean and its sign agrees with the panel (8% of decisions changed, all near-ties in value) and with the lifetime delta. The pruned depth-5 arm (one-ply prior, width two on all three interior layers, first 128 seeds, 1.80x the depth-4 work per move) averages 383,515 points and 111.17 moves against the comparator's 405,130 on the same 128 games: paired delta -21,615 (bounds -76,153, +31,957; floor 54,839), 62 wins to 66, so G2 is a non-measurement whose point estimate is negative, consistent with the panel finding that pruned depth 5 is no better than depth 4 in points. No illegal or incomplete decision in any arm. Read: the pruning is a four-times-cheaper policy that appears to give up about a twelfth of the score, not an engineering speed-up; and buying a fifth ply through pruning does not pay. Pilot tier, previously evaluated development cohort, zero new seeds.

Full record →