Drop7 Research
← Experiments

Offline capacity and training sweep of the NNUE-shaped survival leaf on the existing corpus

preregisteredtier CHECKdiagnosticpreviously-evaluated-developmentprivileged-teacherEX-20260822-nnue-leaf-capacity-sweep-79632fd9
No explanation has been written for this experiment yet. Add web/content/research/EX-20260822-nnue-leaf-capacity-sweep-79632fd9.mdx and it will appear here. The registered protocol is shown below.

The registered protocol

Training approaches/lifetime-objective/learned-leaf/train_leaf.py unchanged on runs/RUN-A51D-corpus/all.states (explored == 0 records, whole-origin split seed 0x59117000) over the 13-configuration grid in sweep.sh — hidden 32/64/128/256/512, mid 16/32/64, 10 or 20 epochs, lr 1e-3 or 3e-3, plus three training seeds at the baseline — yields a best held-out lifetime Pearson within 0.01 of the h64/m32/e10/lr3e-3 baseline (finding-08 recorded 0.8564 on the same split).

Candidate
leafnet-capacity-grid
approaches/lifetime-objective/leaf-capacity-sweep/sweep.sh
Comparator
fair-d4
approaches/fair-expectimax/reference/fair-only-depth4.cpp

Primary metric

held-out lifetime Pearson correlation on the whole-origin test split (486,819 examples), best configuration minus baseline

Statistical unit: whole-origin

Pass criteria, fixed in advance

  • All 13 runs complete with finite metrics on the same split (summary.json lists 13 rows).
  • Gate verdict from summarize.py: supported-as-tested if delta <= max(0.005, 2 x seed spread); not-supported-as-tested if delta >= 0.02 and delta > 3 x spread; otherwise inconclusive.
  • Every reported number is copied from the trainer's own <tag>.json output; nothing is recomputed.

On pass: Record valid + pass (saturated); the next learned-leaf experiment must change the target or the feature space, not the width.

On fail: Record valid + fail (capacity or training length does matter offline); register a separate gameplay experiment before any strength claim; do not adopt any model.

Data and reuse

Reads only the existing training corpus runs/RUN-A51D-corpus/all.states (SEEDLEASE-A52 training ranges 0xa5200000-0xa52152ff as recorded in runs/RUN-A51D-corpus/gen.sh; the same file finding-08 trained on). No seed is opened and no game is played. Held-out metrics use the corpus's own whole-origin test split, which finding-08 also reported on; they are diagnostic, never strength evidence.

seed leases: none

Tests theory: The NNUE-shaped survival leaf is accuracy-saturated at hidden width 64 on the existing corpus

What happened

valid run · outcome: passsupported-as-testedmechanics-onlyRS-20260822T024228Z-94090db1

Offline capacity/training sweep of the NNUE-shaped survival leaf on runs/RUN-A51D-corpus/all.states (13 runs, same whole-origin test split, 486,819 held-out examples). Baseline h64/m32/e10/lr3e-3 held-out lifetime Pearson 0.8546 (finding-08 recorded 0.8564 on an earlier torch build); seed-to-seed spread 0.0023 over three seeds. Best configuration h64-m64-e10-lr3e3-s0 at 0.8568, delta +0.0022 <= max(0.005, 2 x spread): gate verdict supported-as-tested. Width hurts: Pearson by width (params, Pearson, final train loss) h32-m32-e10-lr3e3-s0 (286,447 params: 0.855, loss 0.5238); h64-m32-e10-lr3e3-s0 (572,367 params: 0.8546, loss 0.511); h128-m32-e10-lr3e3-s0 (1,144,207 params: 0.8543, loss 0.5006); h256-m32-e10-lr3e3-s0 (2,287,887 params: 0.8515, loss 0.4857); h256-m64-e20-lr3e3-s0 (2,296,591 params: 0.8446, loss 0.4596); h512-m64-e20-lr3e3-s0 (4,592,143 params: 0.8378, loss 0.4197). Training loss falls monotonically with width while held-out accuracy falls, i.e. larger students overfit this corpus; 20 epochs change nothing at h64 and hurt at h128/h256; lr 1e-3 and mid 16/64 are within spread. The accuracy of this feature space on this corpus is saturated near 0.855-0.857; it is not limited by hidden width, middle width, epochs or learning rate.

What it had to pass
  • ✓All 13 runs complete with finite metrics on the same split — observed: 13 rows, 0 FAILED lines in sweep.log
  • ✓Gate verdict from summarize.py — observed: delta +0.0022 vs threshold max(0.005, 2 x 0.0023) -> supported-as-tested
  • ✓Every number copied from the trainer's own <tag>.json output — observed: summarize.py reads heldOut/heldOutMirrored/history fields verbatim
Recorded metrics
basePearson
0.8546
seedSpread
0.0023
bestTag
h64-m64-e10-lr3e3-s0
bestPearson
0.8568
deltaBestMinusBase
0.0022
gateVerdict
supported-as-tested
runs
  1. tag
    h128-m32-e10-lr3e3-s0
    hidden
    128
    mid
    32
    epochs
    10
    lr
    0.0030
    seed
    42,282
    parameters
    1,144,207
    lifetimePearson
    0.8543
    mirroredPearson
    0.8541
    lifetimeMAE
    14.4423
    hazardAccMean
    0.8387
    finalTrainLoss
    0.5006
    trainWallSeconds
    10.7382
  2. tag
    h128-m32-e20-lr3e3-s0
    hidden
    128
    mid
    32
    epochs
    20
    lr
    0.0030
    seed
    42,282
    parameters
    1,144,207
    lifetimePearson
    0.8520
    mirroredPearson
    0.8516
    lifetimeMAE
    14.5026
    hazardAccMean
    0.8374
    finalTrainLoss
    0.4887
    trainWallSeconds
    10.7461
  3. tag
    h256-m32-e10-lr3e3-s0
    hidden
    256
    mid
    32
    epochs
    10
    lr
    0.0030
    seed
    42,282
    parameters
    2,287,887
    lifetimePearson
    0.8515
    mirroredPearson
    0.8512
    lifetimeMAE
    14.5446
    hazardAccMean
    0.8365
    finalTrainLoss
    0.4857
    trainWallSeconds
    15.5636
  4. tag
    h256-m64-e20-lr3e3-s0
    hidden
    256
    mid
    64
    epochs
    20
    lr
    0.0030
    seed
    42,282
    parameters
    2,296,591
    lifetimePearson
    0.8446
    mirroredPearson
    0.8444
    lifetimeMAE
    14.7638
    hazardAccMean
    0.8322
    finalTrainLoss
    0.4596
    trainWallSeconds
    15.2548
  5. tag
    h32-m32-e10-lr3e3-s0
    hidden
    32
    mid
    32
    epochs
    10
    lr
    0.0030
    seed
    42,282
    parameters
    286,447
    lifetimePearson
    0.8550
    mirroredPearson
    0.8548
    lifetimeMAE
    14.4879
    hazardAccMean
    0.8384
    finalTrainLoss
    0.5238
    trainWallSeconds
    7.5850
  6. tag
    h512-m64-e20-lr3e3-s0
    hidden
    512
    mid
    64
    epochs
    20
    lr
    0.0030
    seed
    42,282
    parameters
    4,592,143
    lifetimePearson
    0.8378
    mirroredPearson
    0.8379
    lifetimeMAE
    15.1885
    hazardAccMean
    0.8256
    finalTrainLoss
    0.4197
    trainWallSeconds
    13.9648
  7. tag
    h64-m16-e10-lr3e3-s0
    hidden
    64
    mid
    16
    epochs
    10
    lr
    0.0030
    seed
    42,282
    parameters
    571,087
    lifetimePearson
    0.8542
    mirroredPearson
    0.8543
    lifetimeMAE
    14.4476
    hazardAccMean
    0.8388
    finalTrainLoss
    0.5151
    trainWallSeconds
    8.3707
  8. tag
    h64-m32-e10-lr1e3-s0
    hidden
    64
    mid
    32
    epochs
    10
    lr
    0.0010
    seed
    42,282
    parameters
    572,367
    lifetimePearson
    0.8552
    mirroredPearson
    0.8552
    lifetimeMAE
    14.4757
    hazardAccMean
    0.8386
    finalTrainLoss
    0.5219
    trainWallSeconds
    3.2629
  9. tag
    h64-m32-e10-lr3e3-s0
    hidden
    64
    mid
    32
    epochs
    10
    lr
    0.0030
    seed
    42,282
    parameters
    572,367
    lifetimePearson
    0.8546
    mirroredPearson
    0.8546
    lifetimeMAE
    14.4145
    hazardAccMean
    0.8390
    finalTrainLoss
    0.5110
    trainWallSeconds
    8.5447
  10. tag
    h64-m32-e10-lr3e3-s1
    hidden
    64
    mid
    32
    epochs
    10
    lr
    0.0030
    seed
    42,283
    parameters
    572,367
    lifetimePearson
    0.8567
    mirroredPearson
    0.8565
    lifetimeMAE
    14.4525
    hazardAccMean
    0.8391
    finalTrainLoss
    0.5114
    trainWallSeconds
    7.2195
  11. tag
    h64-m32-e10-lr3e3-s2
    hidden
    64
    mid
    32
    epochs
    10
    lr
    0.0030
    seed
    42,284
    parameters
    572,367
    lifetimePearson
    0.8569
    mirroredPearson
    0.8569
    lifetimeMAE
    14.4411
    hazardAccMean
    0.8391
    finalTrainLoss
    0.5131
    trainWallSeconds
    8.8389
  12. tag
    h64-m32-e20-lr3e3-s0
    hidden
    64
    mid
    32
    epochs
    20
    lr
    0.0030
    seed
    42,282
    parameters
    572,367
    lifetimePearson
    0.8564
    mirroredPearson
    0.8563
    lifetimeMAE
    14.4448
    hazardAccMean
    0.8389
    finalTrainLoss
    0.5051
    trainWallSeconds
    8.4661
  13. tag
    h64-m64-e10-lr3e3-s0
    hidden
    64
    mid
    64
    epochs
    10
    lr
    0.0030
    seed
    42,282
    parameters
    574,927
    lifetimePearson
    0.8568
    mirroredPearson
    0.8567
    lifetimeMAE
    14.4299
    hazardAccMean
    0.8391
    finalTrainLoss
    0.5097
    trainWallSeconds
    5.2192
environment
python 3.13.5 torch 2.13.0+rocm7.1 hip 7.1.52802 device Radeon 8060S Graphics
Limitations
  • Offline diagnostic on teacher labels from already-played games; says nothing about playing strength (finding-08: a 0.008 Pearson gap did not separate two models in play).
  • Single corpus (5.26M records, whole-origin split); a corpus several times larger could move the saturation point and is the natural follow-up, since the wider models overfit rather than underfit.
  • Per-board inference cost was not measured; irrelevant to the verdict because no wider model won.
  • GPU ran concurrently with a 30-thread CPU evolution; wall times are not timing-grade; metrics are unaffected.
  • The baseline reproduction differs from finding-08's 0.8564 by 0.0018 on a newer torch build; within the measured seed spread.