Local validation of the depth matrix: the frontier scheduler saturates a 16-core workstation

The depth matrix runs locally, and the frontier scheduler uses the whole machine

Until today the merged Rust matrix workflow (PR #13) had only been exercised through its EC2 orchestrator's plan mode on this machine. The goal was a purely local end-to-end validation on a small fixed-work sample: one public root, depths up to five, and a check that within-decision parallelism actually reaches every hardware thread rather than serializing on one heavy column.

The machine had no Rust toolchain (an earlier one had evidently been removed; stale bench/decide release binaries remained), so a user-local rustup (stable, minimal profile) was installed, along with just itself via Homebrew. The justfile was then reorganized: the EC2 orchestrator moved to just run-matrix-ec2, a new just run-matrix-local wraps a new cluster/run-local-matrix.sh that exposes the analyzer's full option surface (round or roots file, depths, strata, leaves, scheduler, threads, cache, split-plies, frontier/host bounds, plan-only mode) and freezes its generated config through the existing run-matrix.sh path, and just run-matrix became an interactive dispatcher that offers EC2 only when a valid AWS credential is present and otherwise asks for confirmation before running locally.

positive

Frontier scheduler saturates the workstation through D5/S7

98.4% occupancy at D5/S7 on 32 threads

On gauntlet-01's initial public position (playground domain; no research seed opened), the frontier scheduler at 32 threads completed D2/D3/D4/D5 at S7 in 0.013 s / 0.007 s / 0.035 s / 1.400 s with worker busy fractions of 0.988 (D4) and 0.984 (D5), ~1005 registered continuation tasks per decision against a 2401-task worst case, and a per-worker busy-seconds spread of only 1.360-1.395 s at D5. A 16-thread repeat ran D5 in 1.475 s at 0.994 occupancy, so the 16 physical cores are the true ceiling and SMT adds only ~5%. Records: runs/RUN-20260827-local-frontier-d5/analytics.jsonl, runs/RUN-20260827-local-frontier-d5-t16/analytics.jsonl.

positive

Root scheduler reproduces the heavy-column idle tail, with exact value parity

0 value-bit mismatches; 7 of 32 threads used

The same cells under the legacy root scheduler used only seven workers (one per legal column) regardless of the 32 requested: D5/S7 took 2.534 s with per-worker busy seconds ranging 1.251-2.532 s — one light column finished a full core-second early while 25 hardware threads never engaged, the exact failure mode the frontier scheduler was built to remove. Every sibling value matched the frontier run bit for bit at D4 and D5, locally reproducing the scheduler-parity gate from the August 25 work. Record: runs/RUN-20260827-local-root-d5/analytics.jsonl.

negative

research-validate fails on eight pre-existing missing-artifact references

8 errors

make research-validate currently fails: two run records, two result records, and two contribution records from 2026-08-25 point at machine profiles and summaries under runs/RUN-20260825T045058Z-b79f76a2/ and runs/RUN-20260825T051728Z-e588e6d9/ that no longer exist on disk (runs/ is gitignored, so those artifacts were local-only). This predates today's work and needs a coordinator decision: either restore the artifacts from wherever they were produced or mark the records accordingly.

Later the same day the depth ladder was extended to a D6/S7 cell (runs/RUN-20260827T075718Z-local-9d85628c/analytics.jsonl: 32.05 s wall at 0.989 occupancy, action unchanged from D4/D5), and the playground gained a rust-fair-d6-s7 policy beside the existing d7 registration, with a just play-round <policy> <round> recipe that saves the leaderboard-schema game record and per-move replay into a self-contained runs/BENCH-* directory (smoke record: runs/BENCH-20260827T080553Z-expectimax-d2--gauntlet-01/). A first depth-6 gauntlet-01 game was started and deliberately interrupted before completion at the operator's request; the operator's own rerun then crashed partway through the game and exposed a real frontier-scheduler bug.

negative

The frontier planner rejected legal near-terminal positions

deep games crashed near the rise

At a gauntlet-01 position with the rise clock at 2 and every column but three stacked to the top, every continuation inside the frontier's split prefix terminates, so the deterministic plan reduces entirely to constants and registers zero worker tasks. choose_action_frontier_parallel treated that legal state as the error "frontier construction produced no continuation tasks", crashing any deep (D6/D7) policy game that survived close to a board- filling rise — shallow policies and the depth matrix's opening roots never hit it. The fix lets an empty frontier reduce over constants (the idle worker path already handled it); the crash position is now a regression test asserting action and bit-for-bit value parity with the root scheduler, and the formerly crashing decide invocation answers in under 3 ms.

Scripted-round results remain playground demonstrations, never tier evidence. Two dead-code warnings (index_of in board.rs, an unused NoTable import in gate_search.rs) were removed, leaving the release build warning-free.

The crash also motivated crash-safety for slow playground games: the bench CLI now journals every completed move to runs/bench-checkpoints/<policy>--<round>.jsonl and, on rerun, replays the journal through the engine without policy calls, verifying board and score at every step before continuing from the checkpoint; a journal that diverges from the current code is discarded with a warning, and the journal is deleted once the final replay is written. A kill-mid-game drill resumed and reproduced the exact checksum of the retained uninterrupted record (runs/BENCH-20260827T080553Z-expectimax-d2--gauntlet-01/), and two new tests cover resume equivalence and mismatch rejection.

The first attempted depth-7 game then exposed a harness limit rather than an engine one: nativeDecide spawned every decision under a fixed ten-minute spawnSync budget sized for depth 6, so the opening depth-7 decision was SIGTERMed as a raw ETIMEDOUT. The per-decision budget is now a per-policy registration option (timeoutMs, default unchanged), the depth-7 policy declares a two-hour budget, and an overrun reports the budget, depth, and board instead of a spawn error, leaving the crash journal intact.

positive

A depth-7 decision completes at full occupancy in 38 minutes

2302 s wall at 0.983 busy on 32 threads

Rerun outside the harness limit, the formerly killed invocation — the gauntlet-01 opening public position at D7/S7 under the frontier scheduler with a 16,777,216-entry shared cache — completed in 2302 s at a worker busy fraction of 0.9833 across 32 threads, registering 1007 frontier tasks and 86,132,823,356 nodes, and answered column 1. Against the same position's D6/S7 cell (32.05 s) that is a roughly 72x depth step, at the top of the 25-50x range assumed earlier, so a full ~60-move depth-7 game costs on the order of a day and a half of workstation time even before mid-game boards are considered. Record: runs/RUN-20260828T062425Z-local-d7-opening-timing/decide.log.

Open items: this was a single-root smoke validation, not a scaling study — beyond the single D6/S7 cell and the single D7/S7 opening decision above, no deeper cells have been attempted locally, and the frontier planner and reduction remain single-threaded phases whose share of wall time was negligible here (~1 ms at D5) but is unmeasured in isolation at D6/D7. The system-level mpstat capture proved too coarse for sub-two-second cells; the per-worker analytics rows are the instrument of record for occupancy claims. No complete games, strength comparisons, or research seed ranges were touched.

Review fix: the evolution's candidate freeze could rank on stale fitness

External review of PR #15 caught a crash-recovery gap in the NNUE evolution binary (stage C of EX-20260825-nnue-evolution-d3-bca7f330): evolve --select ranked the last completed generation's population using a sidecar final-fitness.json that is only written on graceful exit, so a hard kill after a gen-NNN.json completion marker would leave --select pairing the newest population with an earlier generation's means and freezing the wrong finalists. The fix removes the sidecar dependency entirely: --select now re-derives each candidate's mean from the per-game scores recorded in the completed generation's own gen-NNN.json, so the ranking is atomically consistent with the completion marker by construction (final-fitness.json remains as a preregistered informational artifact). Verified on a throwaway playground-domain smoke run: a planted stale fitness file that reorders the finalists under the old logic now changes nothing, and a deleted fitness file no longer blocks selection. No research seeds were opened and the stage C selection procedure's preregistered semantics (top eight by final-generation fitness, fresh re-selection block) are unchanged.