punkshell/goals/archive/G-092-split-longrunning-testfiles.md

G-092 Split long-running test files to lower the parallel floor of broad runs

Status: achieved 2026-07-19 Scope: src/tests/modules/punk/mix/testsuites/scriptwrap/ (multishell.test), src/tests/shell/testsuites/binscripts/ (dtplite.test), src/tests/modules/argparsingtest/testsuites/argparsingtest/ (argparsingtest.test), src/tests/shell/testsuites/punkexe/ (scriptexec.test - if assessed worthwhile), src/tests/runtests.tcl (longest-first weight table), src/tests/AGENTS.md + affected child AGENTS.md Goal: no single monolithic test file dominates the -jobs parallel critical path: the long-running suites are split into cohesive smaller files (grouped along shared-fixture boundaries, every test preserved verbatim) so a broad -jobs 8 run's slowest child process is bounded by its slowest cohesive test group rather than a whole legacy file. Acceptance: after the split, repeated -jobs 8 full-suite runs on the reference machine report no child process wall above 45s in the runner's slowest-child listing and improve full-suite wall time by at least 25% versus the G-091 verification baseline (2m16s); test conservation is proven and recorded in this file - global tallies identical to a pre-split sequential baseline and the full set of test names unchanged (name-level comparison, mechanism decided in the work - e.g. a names mode for scriptlib/developer/runtests_parity.tcl); no test is deleted, skipped, weakened, or renamed, and provenance comment lines travel with moved tests; the runtests longest-first weight table covers the new files; scriptlib/developer/runtests_parity.tcl reports PARITY ok between -jobs 8 and -jobs 1 on the split tree; affected AGENTS.md contracts (src/tests + shell/binscripts and scriptwrap references) are updated.

Context

G-091's verification (2026-07-19, archived - goals/archive/G-091-runtests-parallel-jobs.md) established the floor anatomy with per-child WALL measurements (the runner now prints them; child wall includes boot and file-level fixture cost that per-test usec misses):

• multishell.test child: ~2m wall (85s summed test time - the two ~30s roundtrip/checkfile tests plus two ~11s wrap tests - the remainder boot + shared scriptwrap fixtures). This single child IS the current -jobs floor: full-suite -jobs 8 wall 2m16s.
• dtplite.test ~25s (two ~12s tests), argparsingtest.test ~27-30s (largest single test 7.8s), scriptexec.test ~10-11s. Everything else is small.
• Post-split ceiling estimate: total child work ~330s / 8 workers ~41s, serial tail ~16s, slowest single test ~32s (scriptwrap_runtime_cmd_roundtrip_no_drift) + child boot ~3s -> slowest child ~35-40s, full-suite wall ~65-80s expected (acceptance leaves headroom for the ~30s run-to-run machine variance observed during G-091).

Split considerations gathered in the same session:

• Fixture duplication tradeoff: multishell's namespace-level fixtures are shared by all its tests; each new file re-pays fixture setup and child boot, so total CPU rises as wall falls. Split along fixture boundaries (tests sharing a fixture stay together) to bound the duplication; the tradeoff is accepted by this goal's premise (wall time of broad parallel runs is the optimisation target).
• exec.test is excluded: it is imported/adapted Tcl core material (src/tests/core contract preserves upstream compatibility intent) and its child is only ~5s.
• src/tests contracts that bind the work: do not delete, skip, or weaken tests; provenance comment lines are immutable facts that must move with their tests; child AGENTS.md files name multishell.test and dtplite.test in their contracts and need corresponding updates.
• The runtests longest-first weight table (static globs in runtests.tcl) references the pre-split filenames and must be updated with measured weights for the new files.
• None of the split candidates are in the -serial-paths default, so the serial tail is unaffected.

Notes

• Complementary, independent: the lean-children candidate (explicit package requires for the 17 preload-dependent files so children can drop the shellrun preload + runx warmup) reduces the per-file boot cost that splitting multiplies - doing it first or alongside lowers the split's total-CPU cost. Recorded in goals/archive/G-091 notes.
• Related: G-029 (packaged test:: modules built from src/tests) consumes this tree as its source of truth - file granularity changes flow into that packaging design.
• The 45s child-wall criterion is the noise-robust metric (child walls were stable across runs while full-suite wall varied up to ~30s); the 25%-vs-2m16s wall criterion is relative to the same-machine G-091 baseline rather than an absolute number for the same reason.

Progress

Achieved-flip verification record, 2026-07-19:

Splits performed (every test moved verbatim, provenance comment lines travelling; file headers carry a split note pointing at src/tests/modules/AGENTS.md for the sibling map):

• multishell.test (1 file, 10 tests, ~105s solo child) -> five files in the same directory: multishell.test (wrap + structure/LF-only + exec smoke incl WSL, 6 tests), multishell_wrapverify.test (fresh-wrap checkfile), multishell_wrapdeterminism.test (byte-identical re-wrap), runtimecmd_checkfile.test (committed bin/runtime.cmd checkfile
  • LF contract - needs no wrap fixture), runtimecmd_roundtrip.test (src/scriptapps re-wrap byte-identity - needs no wrap fixture).
• argparsingtest.test (17 tests, ~32s solo) -> argparsingtest.test (timeit/discover/ parser-contract/first_call, 11 tests) + argparsingtest_compare.test (compare_*, 6 tests).
• Assessed and left unsplit: dtplite.test (24.6s solo / ~29s under -jobs 8 load), scriptexec.test (12s solo) - both under the 45s bound; exec.test excluded as imported Tcl core material per this goal's Context.

Acceptance evidence (native tclsh 9.0.3, reference machine, piped/no-console context):

• Repeated -jobs 8 full-suite runs: 1m26.3s / 1m26.8s wall; slowest child 30.7s / 31.3s (runtimecmd_roundtrip/checkfile) - no child above 45s in either run's slowest-child listing.
• Wall improvement vs the G-091 verification baseline (2m16.2s): 36.6% (>=25% required); 4.07x vs the ~350s pre-parallel sequential baseline.
• Conservation: NAMES identical (989 test-name occurrences, multiset comparison via the new runtests_parity.tcl -names mode against a pre-split sequential capture); global tallies identical in every run (989 total / 973 passed / 15 skipped / 1 failed = exec-14.3, files 87 -> 92); PARITY ok between -jobs 8 and -jobs 1 on the split tree and between the repeated -jobs 8 runs; repo clean (git status) after every run.
• runtests longest-first weight table updated with measured post-split weights; src/tests/AGENTS.md (jobs bullet + Work Guidance numbers) and the src/tests/modules/AGENTS.md punk/mix bullet updated.

Findings:

• Child-wall contention inflation: with 50s+ dominant children in the mix (the first 4-way split), -jobs 8 load inflated child walls ~45% over solo (runtime singles 27s -> 42-46s; argparsingtest 32s -> 47.7s; wrapverify 35s -> 53s), briefly breaching the 45s bound. After the secondary splits flattened the distribution (no solo child above ~31s), inflation collapsed to ~15% - a flatter work distribution reduces not just the critical path but the contention each child experiences.
• runtests -include-paths anomaly noted during measurement (out of scope, reported to the user): a single -include-paths value carrying four space-separated deep patterns matched only one of them, and repeated -include-paths flags matched zero files, contradicting the documented accumulate semantics; file-tail glob targeting was used instead.