Reproduce the Klein-Friedmann Q-learning report in its own simulator at pinned commit 8cc8a0e
Running the upstream code unchanged (github.com/ekreate/cs221-final-project, commit 8cc8a0e; Drop7 simulator main.py, homework scaffold util.py, learner and features Drop7QLearning.py) with the report's protocol -- 50,000 epsilon-greedy training games with epsilon = t^-0.25 and eta = 1/t counted per move, lambda = 0.1 ridge, then 10,000 test games with epsilon = 0, plus 5,000 uniform-random games -- reproduces the report's figures (random 31.2 +/- 5.12; agent 49.61 +/- 11.18) within the fixed bands of theory clause (i).
On this page
- Created
- Updated
No explanation has been written for this record yet.
Technical recordThe registered protocol
- Hypothesis
- Running the upstream code unchanged (github.com/ekreate/cs221-final-project, commit 8cc8a0e; Drop7 simulator main.py, homework scaffold util.py, learner and features Drop7QLearning.py) with the report's protocol -- 50,000 epsilon-greedy training games with epsilon = t^-0.25 and eta = 1/t counted per move, lambda = 0.1 ridge, then 10,000 test games with epsilon = 0, plus 5,000 uniform-random games -- reproduces the report's figures (random 31.2 +/- 5.12; agent 49.61 +/- 11.18) within the fixed bands of theory clause (i). Two additional arms test clause (ii): the same learner stopped after 300 training games, and the same learner with lambda = 0. Only logging differs from upstream: the per-step weight history list is not kept (memory) and the ridge constant is exposed as a parameter; the update formula is upstream line 79 verbatim.
- Arms
Arm Name Entry point Manifest Candidate upstream cs221-final-project Q-learner (external Python simulator, commit 8cc8a0e) approaches/value-policy-learning/klein-friedmann-linear-q/reproduction/reproduce_kf.py– Comparator uniform-random policy in the same upstream simulator (the report's baseline) approaches/value-policy-learning/klein-friedmann-linear-q/reproduction/reproduce_kf.py– - Classification
- validation
- Information boundary
- not-applicable
- Benchmark tier
- CHECK
- Lifecycle
- preregistered
- Theories tested
- Primary metric
- mean moves survived over 10,000 test games (epsilon = 0) per training seed, in the upstream simulator's own units (one point per move, 200-move cap)
- Secondary metrics
- standard deviation, median, min, max and 200-cap count of the test games
- 5,000-game uniform-random mean and standard deviation per seed
- training curve as per-1,000-game block means over the 50,000 training games
- final weights per arm and total update count
- lambda = 0 arm test mean relative to the random mean (clause ii-b)
- 300-game arm test mean relative to the 50,000-game arm on the same test seeds (clause ii-a)
- wall seconds per arm
- Statistical unit
- whole-game
- Uncertainty method
- three independent training seeds against the report's fixed tolerance bands; standard deviation over games as the report does; no confidence interval is claimed
- Data role
- no-gameplay
- Seed leases
- none recorded
- Whole-origin split
- no
- Reuse disclosure
- No repository seed is read: the upstream simulator draws from Python's Mersenne Twister seeded by the arm's integer seed. An UNREGISTERED smoke run at the identical configuration preceded this record (Python seeds 0, 1, 2 for lambda = 0.1; seed 0 for lambda = 0) and its numbers are disclosed in the result record's limitations; the registered run uses disjoint Python seeds 10, 11, 12 (lambda = 0.1), 10 (lambda = 0) and 10 (300-game arm). The upstream repository carries no license, so its files are fetched at run time into the run directory and are never vendored into this repository.
- Pass criteria
- Random: 5,000-game uniform-random mean within [30.2, 32.2] for each of the three seeds.
- Agent: 10,000-game test mean within [47.6, 51.6] for each of the three lambda = 0.1 seeds.
- Clause (ii-a) check, recorded either way: the 300-game arm's test mean is within 2.0 moves of the seed-10 50,000-game arm's test mean on the same 10,000 test seeds.
- Clause (ii-b) check, recorded either way: the lambda = 0 arm's test mean is compared with the random mean; the report's Figure 5 predicts it falls below random.
- On pass
- Record valid + pass at CHECK/validation tier; the frozen upstream commit, weights and per-game rows become the reference for the Rust port's parity gates (EX-20260902-kf-linear-q-rust-transfer-4328a730). No repository cohort is opened by this record.
- On fail
- Record valid + fail: the shipped code does not reproduce the report; the theory's transfer clauses are then about the shipped code, not the report, and the result says so.
- Gate fixed before controlled data
- yes
- Resources
Wall seconds 3600 CPU threads 5 Max host bytes 4294967296 Max GPU bytes – GPU devices – - Stop conditions
- Stop if the fetched upstream tree is not at commit 8cc8a0e or its three files' SHA-256 differ from the values recorded in the run.
- Stop at 3,600 s wall or on any Python exception; a partial arm is recorded as partial, never summarised.
- No repository seed may be read under this record.
- Expected artifacts
approaches/value-policy-learning/klein-friedmann-linear-q/reproduction/ (fetch script, harness, requirements)runs/<run-id>/kf-reproduction/upstream-commit.txt (commit and file hashes)runs/<run-id>/kf-reproduction/seed10.json, seed11.json, seed12.json, seed10-noreg.json, seed10-300.json (per-game scores, weights, curves)runs/<run-id>/kf-reproduction/summary.json
- Amendments
- none recorded
Technical recordResults recorded against this protocol
The shipped Klein-Friedmann code (github.com/ekreate/cs221-final-project @ 8cc8a0edfa04f1a93088c951e217d3cd3d6013f0) reproduces its report in its own simulator: uniform-random means 31.657, 31.733, 31.851 over 5,000 games each (report 31.2), and the Q-learner's 10,000-game test means 49.080, 49.053, 49.015 moves with standard deviations near 11.7 (report 49.61, sd 11.18) after 50,000 training games each, so both preregistered bands pass. Two of the theory's clause-(ii) checks were decided here. (ii-a) Front-loading holds: a learner stopped after 300 training games (13,796 updates) tests at 49.0796 moves against 49.0795 for the 50,000-game learner (2,406,211 updates) on the same 10,000 test games, a difference of -0.0001; the per-1,000-game training curve is already at 46.8 in its first block and 48.2 in its last. (ii-b) The report's regularisation claim does NOT hold for the shipped configuration: with lambda = 0 the learner tests at 48.967 moves, indistinguishable from the lambda = 0.1 arm and far above random, where the report's Figure 5 says the unregularised agent fell below random. With eta = 1/t counted per move, the ridge step eta*lambda*w is of order 1e-5 for almost all of training, so the term cannot act; the report's divergence must come from a schedule the shipped code no longer contains. Units are the upstream simulator's (one point per surviving move, 200-move cap, empty starting board, dropping on a full column ends the game); nothing here is a corrected-Hardcore score or a repository cohort.
- ✓Random: 5,000-game uniform-random mean within [30.2, 32.2] for each of the three seeds — observed: means ['31.657', '31.733', '31.851']
- ✓Agent: 10,000-game test mean within [47.6, 51.6] for each of the three lambda = 0.1 seeds — observed: means ['49.080', '49.053', '49.015'] (report 49.61)
- ✓Clause (ii-a): the 300-game arm's test mean is within 2.0 moves of the seed-10 50,000-game arm on the same test seeds — observed: 50,000-game 49.080; 300-game 49.080; difference -0.000
- ✕Clause (ii-b): the lambda = 0 arm's test mean compared with the random mean (report's Figure 5 predicts below random) — observed: lambda = 0 test mean 48.967; random 31.657; lambda = 0.1 test mean 49.080; 'passed' here means the report's prediction held
Technical recordRecorded metrics
- randomMean
- 31.2000
- randomSd
- 5.1200
- agentTestMean
- 49.6100
- agentTestSd
- 11.1800
- humanMean30Games
- 73.2000
- seed10
- randomMean5000
- 31.6572
- randomSd
- 5.3286
- trainMean
- 48.1242
- testMean10000
- 49.0795
- testSd
- 11.7475
- testMedian
- 45
- testMax
- 145
- testCapped200
- 0
- updateCount
- 2,406,211
- trainBlockMeans1000First2
- 46.8300
- 47.5320
- trainBlockMeans1000Last2
- 48.3090
- 48.2310
- weights
- min_eq_elem_True
- 175.6754
- row_dets
- 33.0892
- col_dets
- 123.9736
- max_eq_elem
- 15.4247
- 1_dets
- 49.1437
- elem_det
- 138.6679
- wallSeconds
- 526.0967
- seed11
- randomMean5000
- 31.7332
- randomSd
- 5.3181
- trainMean
- 48.1982
- testMean10000
- 49.0531
- testSd
- 11.6803
- testMedian
- 45
- testMax
- 195
- testCapped200
- 0
- updateCount
- 2,409,908
- trainBlockMeans1000First2
- 47.2940
- 47.5400
- trainBlockMeans1000Last2
- 47.3910
- 48.4650
- weights
- min_eq_elem_True
- 292.4652
- row_dets
- 55.0005
- col_dets
- 206.3903
- max_eq_elem
- 24.7546
- 1_dets
- 82.4603
- elem_det
- 229.4302
- wallSeconds
- 527.6854
- seed12
- randomMean5000
- 31.8510
- randomSd
- 5.3634
- trainMean
- 48.1870
- testMean10000
- 49.0147
- testSd
- 11.7404
- testMedian
- 45
- testMax
- 125
- testCapped200
- 0
- updateCount
- 2,409,352
- trainBlockMeans1000First2
- 46.6090
- 47.2890
- trainBlockMeans1000Last2
- 48.7050
- 48.2640
- weights
- min_eq_elem_True
- 330.0664
- row_dets
- 62.0981
- col_dets
- 233.1830
- max_eq_elem
- 28.3817
- 1_dets
- 92.1482
- elem_det
- 260.4667
- wallSeconds
- 528.1099
- seed10-noreg
- randomMean5000
- randomSd
- trainMean
- 48.1237
- testMean10000
- 48.9671
- testSd
- 11.8369
- testMedian
- 45
- testMax
- 180
- testCapped200
- 0
- updateCount
- 2,406,185
- trainBlockMeans1000First2
- 46.4940
- 47.4820
- trainBlockMeans1000Last2
- 48.3950
- 48.4630
- weights
- min_eq_elem_True
- 440.5682
- row_dets
- 91.6883
- col_dets
- 331.4956
- max_eq_elem
- 32.2473
- 1_dets
- 111.6859
- elem_det
- 332.4404
- wallSeconds
- 501.3658
- seed10-300
- randomMean5000
- randomSd
- trainMean
- 45.9867
- testMean10000
- 49.0796
- testSd
- 11.6295
- testMedian
- 45
- testMax
- 130
- testCapped200
- 0
- updateCount
- 13,796
- trainBlockMeans1000First2
- 45.9867
- trainBlockMeans1000Last2
- 45.9867
- weights
- min_eq_elem_True
- 33.1054
- row_dets
- 6.2036
- col_dets
- 26.0357
- max_eq_elem
- 4.7284
- 1_dets
- 9.2904
- elem_det
- 28.6006
- wallSeconds
- 72.7772
- pythonSeeds
- 0
- 1
- 2
- testMeansLambda0p1
- 48.9150
- 49.2441
- 49.1112
- lambda0TestMeanSeed0
- 48.9674
- randomMeans5000
- 31.7044
- 31.8152
- 31.8732
- note
- unregistered scratchpad run preceding preregistration; no phase reseeding; not evidence
- CHECK/validation tier: an external simulator with its own rules (survival score, 200-move cap, empty opening board, full-column drops end the game, sequential within-wave reveal semantics). No repository seed was read and no Hardcore score exists here.
- The gate on clause (ii-b) is recorded as failed because the report's prediction (unregularised below random) did not hold; this is a finding against the report's explanation, not a defect of the run.
- The 300-game and 50,000-game arms share their 10,000 test seeds by phase reseeding of Python's global RNG, a harness convention the upstream code lacks; the upstream flow would have compared different test games.
- An unregistered smoke run at the identical configuration (Python seeds 0-2) preceded preregistration and is disclosed in metrics.smokeRunDisclosed; the registered arms used disjoint seeds 10-12.
- Five arms ran concurrently on a 12-core laptop; wall times are indicative only. The upstream repository has no license, so its files are not retained in this repository; the run directory keeps a fetched copy and the manifest records its hashes.
- Three training seeds cannot estimate the between-seed spread of the agent mean more precisely than about 0.03 moves; the bands, not the spread, are the gate.
Agent contextHow to extend this record
To add a reader-facing explanation, write web/content/research/EX-20260902-kf-report-reproduction-b7f61bf1.mdx; it renders above this record on the next request. The registered protocol itself is in the technical record above.
Record file: research/experiments/EX-20260902-kf-report-reproduction-b7f61bf1.json, validated against research/schemas/experiment-v1.schema.json. Protocol hash: 3f17b7a1df011ad14f95903d7f39e60683bd7c68dbf0426759827d65692c5a63.
Registered by Claude Code / claude-fable-5-1 (claude-q-learning).