punkshell/goals/archive/G-091-runtests-parallel-jobs.md

G-091 runtests multiprocess parallelism (-jobs N)

Status: achieved 2026-07-19 Scope: src/tests/runtests.tcl, src/tests/testsupport/, src/tests/AGENTS.md Goal: src/tests/runtests.tcl runs multi-process mode's per-file child processes through a -jobs N worker pool (tpool; per-child tmpdirs and output files; longest-first scheduling; a serial group for console-sensitive suites; per-file report sections emitted in discovery order) so the full source-tree suite completes in a fraction of its sequential wall time with unchanged results. Acceptance: -jobs 8 produces aggregate results identical to -jobs 1 (scriptlib/developer/runtests_parity.tcl reports PARITY ok) across repeated full-suite runs on a native tclsh 9.0.x, reduces full-suite wall time by at least 2.5x versus the sequential baseline on the reference machine, leaves the repo clean after runs (git status unchanged), and src/tests/AGENTS.md documents the -jobs flag and the serial-group policy.

Context

Groundwork shipped 2026-07-18 (commit 218fbb1c): -singleproc 0 multi-process mode with a per-file child bootstrap (testsupport/child_test_runner.tcl + generated environment payload), plain-exec file-captured child spawn designed for pool reuse, honest nonzero-exit classification, and the parity harness (scriptlib/developer/runtests_parity.tcl). Full-suite parity between singleproc and multiproc modes verified same day (87 files / 989 tests; exec-14.3 the sole expected failure on native tclsh 9.0.3).

Why processes are the parallel unit: test files mutate process-global state (cd in multishell/dtplite/path suites, ::env save/mutate/restore in console suites, thread creation and tsv use, shellrun/shellfilter global channel transforms) - safe across processes, unsafe within one. Suites are already well-behaved across processes (FOSSIL_HOME isolation in fossilmove.test, port-0 ephemeral sockets, tmpdir-scoped fixtures).

Measured shape (native tclsh90, piped context, 2026-07-18): ~350s sequential wall; slowest files multishell.test ~85s, argparsingtest ~27s, dtplite ~25s, scriptexec ~10s. Parallel floor is the slowest single file (~90s incl. overhead) -> expect roughly 3-3.5x at 8 jobs with diminishing returns beyond.

Notes

• Per-child tmpdir subdirs are mandatory under parallelism (shared tcltest -tmpdir would collide on same-named makeFile helpers across concurrent files); phase-1's shared default matches singleproc and must become per-child in pool mode. Same for the per-child stdout/stderr capture file names.
• Serial-group default candidates (relaxable after parity evidence under a real console): modules/punk/console/, modules/opunk/console/, modules/punk/repl/, shell/. In piped contexts these mostly skip anyway.
• Kit-executable caveat carries over from phase 1: children of a kit exe boot with kit-stamped punk packages preloaded (runner warns; native tclsh preferred).
• Candidate independent win (either order): make the 17 preload-dependent files' punk::* package requires explicit so children can drop the shellrun preload + runx warmup and go leaner/faster - the parity tool gates the change. See child_test_runner.tcl comments for the two-stage preload detail (shellrun require + runx execution pulling punk::lib).
• Raising the ceiling beyond the multishell bound means splitting multishell.test (two 30s+ tests) and dtplite.test - suite restructure, optional, floor then ~40s (slowest single test ~32s).
• PUNK_TEST_UDPTEE watch mode (commit a65f9e5b): under a pool, the parent's per-file relay and lifecycle events still work; the cooked log format's source column plus file-start/ file-end events let one viewer demux interleaved children.
• Timing noise: -slowest per-test microseconds get noisier under CPU oversubscription; document rather than fix.

Progress

Achieved-flip verification record, 2026-07-19:

Implementation landed in src/tests/runtests.tcl: -jobs N (implies -singleproc 0; explicit -singleproc 1 conflicts with an error), -serial-paths (default trimmed during verification to {modules/punk/console/** modules/opunk/console/**} - the piped-child-driven repl/stdin suites proved parallel-safe and each serial file costs its full ~3-4s child boot strictly after the parallel phase), tpool pool with longest-first static-weight submission, per-child capture files and tcltest -tmpdir subdirs, results consumed by the unchanged processing loop in discovery order, phase-time and per-child-wall reporting, watch-mode file-collected events with completion-order output relay.

Acceptance evidence (native tclsh 9.0.3, reference machine, piped/no-console context):

• Repeated -jobs 8 full-suite runs: 2m16.2s and 2m15.6s wall; parity tool reports PARITY ok for -jobs 8 vs -jobs 1, between the two repeated -jobs 8 runs, and vs the pre-parallel singleproc baseline (87 files / 989 tests / 973-15-1, exec-14.3 the sole expected failure in every run).
• Speedup: 2.58x vs the ~350s sequential baseline this goal's Context records (5m50.8s singleproc / 5m50.1s sequential multiproc, 2026-07-18). One same-week sequential -jobs 1 run on a cooler machine measured 5m28.8s, against which the ratio is 2.41x - recorded for honesty; run-to-run machine variance during verification was up to ~30s.
• Repo clean (git status unchanged) after every run; src/tests/AGENTS.md documents the flag, serial-group policy, event semantics and measured speedup.

Two Tcl-core-level findings made during verification (both worked around in runtests_run_child_process, both candidates for upstream investigation):

1. Concurrent BLOCKING exec calls from multiple threads convoy on Windows (Tcl 9.0.3): they all return together when the longest-lived concurrent child exits - proven with plain sleepers of 2-12s from 8 tpool workers all walling at 12.0s. In early -jobs runs this manifested as the first-wave 8 children all reporting identical 139.9s walls and capped the speedup at ~2x. Workaround: spawn with 'exec ... &' (returns at spawn) and poll per-pid tcl::process status.
2. Tcl_ReapDetachedProcs races the poll: every exec call automatically reaps finished background children process-wide, so a sibling worker's next spawn purged a child's status entry before its own worker read it (nondeterministic 'key not known in dictionary' file errors). Workaround: tcl::process autopurge false plus explicit per-pid purge after the status read.

Remaining non-gating observations: verification context was piped/no-console - under a real console the -serial-paths default may need widening (escape hatch documented); raising the ceiling past the ~2m multishell.test child means splitting that file (suite restructure, out of scope); the lean-children candidate (explicit deps for the 17 preload-dependent files) remains open with the parity tool as its gate.