runtests -jobs N parallel test execution (G-091 achieved)
-jobs N (implies -singleproc 0; explicit -singleproc 1 conflicts) runs
test files through a tpool of N concurrent child processes with
per-child capture files and tcltest -tmpdir subdirs, longest-first
static-weight submission, and a -serial-paths serial tail (default: the
console-state suites modules/punk/console/** + modules/opunk/console/**;
piped-child-driven suites proved parallel-safe and each serial file
costs its full child boot strictly after the parallel phase). Cached
results feed the unchanged processing loop in discovery order, so
reports are identical in shape and order to sequential runs. Watch mode
emits file-collected events (completion order, output relayed there);
file-end (tallies) stays discovery-order. The runner now reports jobs
phase times and the ten slowest child WALL times (child wall includes
boot + file-level fixture cost that per-test usec misses).
Two Tcl-core-level behaviours found and worked around in the child
spawn (see the verification record in the archived goal file):
- concurrent BLOCKING execs from worker threads convoy on Windows (all
return at the longest concurrent child's exit; plain sleepers 2-12s
from 8 workers all walled at 12.0s) -> spawn via 'exec ... &' and
poll per-pid tcl::process status; blocking fallback retained for
runtimes without tcl::process (tcl 8.6).
- Tcl_ReapDetachedProcs runs on every exec and purges finished
background children process-wide, racing the poll (nondeterministic
'key <pid> not known in dictionary') -> tcl::process autopurge false
+ explicit per-pid purge after the status read.
Verified on native tclsh 9.0.3: repeated -jobs 8 full-suite runs
2m16.2s / 2m15.6s with PARITY ok vs -jobs 1, between repeats, and vs
the pre-parallel baseline (987-test identity, exec-14.3 sole expected
failure); 2.58x vs the ~350s sequential baseline (2.41x vs one
cooler-machine 5m28.8s sequential run - recorded in the goal file);
repo clean after every run.
G-091 flipped to achieved 2026-07-19 and archived (entry to
GOALS-archive.md, detail file to goals/archive/ with the verification
record; reference sweep updated runtests.tcl comments and the
src/tests/AGENTS.md marker). goals_lint clean.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
@ -96,3 +96,7 @@ Acceptance: the analysis command reports the two known real cases from the G-041
### G-082 [achieved 2026-07-14] punk::args single-form parse error selection: specific validation failures preferred over generic overflow → detail: goals/archive/G-082-punkargs-error-selection.md
Scope: src/modules/punk/args-999999.0a1.0.tm (get_dict/parse error selection, validation-failure capture around the choice/type screens), src/tests/modules/punk/args/testsuites/args/ (characterization + regression suite)
Acceptance: a definition with an optional choice-restricted leader or value that receives a non-matching word reports a choiceviolation-class error naming the word and the allowed choices (message and -errorcode), not toomanyarguments - pinned for the make.tcl-style dispatch shape (optional subcommand leader with choices) and a values-position equivalent; a word rejected by a typed optional argument (e.g int) that then overflows reports the type failure similarly; genuinely-surplus cases (all optional args satisfied, extra word remains) still report toomanyarguments; multiform error selection (rank_form_failures) is unchanged or its pins deliberately updated; current 0.12.x behaviour is characterized first and the pins flipped deliberately; full punk::args suite passes with no expectations weakened; on completion the G-030 -optional 0 workaround sites are revisited (make.tcl dispatch definitions, the src/modules/AGENTS.md "punk::args definition authoring ergonomics" bullet) and relaxed or re-documented.
Acceptance: -jobs 8 produces aggregate results identical to -jobs 1 (scriptlib/developer/runtests_parity.tcl reports PARITY ok) across repeated full-suite runs on a native tclsh 9.0.x, reduces full-suite wall time by at least 2.5x versus the sequential baseline on the reference machine, leaves the repo clean after runs (git status unchanged), and src/tests/AGENTS.md documents the -jobs flag and the serial-group policy.
Goal: src/tests/runtests.tcl runs multi-process mode's per-file child processes through a -jobs N worker pool (tpool; per-child tmpdirs and output files; longest-first scheduling; a serial group for console-sensitive suites; per-file report sections emitted in discovery order) so the full source-tree suite completes in a fraction of its sequential wall time with unchanged results.
Acceptance: -jobs 8 produces aggregate results identical to -jobs 1 (scriptlib/developer/runtests_parity.tcl reports PARITY ok) across repeated full-suite runs on a native tclsh 9.0.x, reduces full-suite wall time by at least 2.5x versus the sequential baseline on the reference machine, leaves the repo clean after runs (git status unchanged), and src/tests/AGENTS.md documents the -jobs flag and the serial-group policy.
@ -48,3 +48,48 @@ with diminishing returns beyond.
file-end events let one viewer demux interleaved children.
- Timing noise: -slowest per-test microseconds get noisier under CPU oversubscription;
document rather than fix.
## Progress
Achieved-flip verification record, 2026-07-19:
Implementation landed in src/tests/runtests.tcl: -jobs N (implies -singleproc 0; explicit
-singleproc 1 conflicts with an error), -serial-paths (default trimmed during verification
to {modules/punk/console/** modules/opunk/console/**} - the piped-child-driven repl/stdin
suites proved parallel-safe and each serial file costs its full ~3-4s child boot strictly
after the parallel phase), tpool pool with longest-first static-weight submission,
per-child capture files and tcltest -tmpdir subdirs, results consumed by the unchanged
processing loop in discovery order, phase-time and per-child-wall reporting, watch-mode
file-collected events with completion-order output relay.
@ -21,6 +21,7 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc
- The testinterp module path includes `src/vendormodules` (and `src/vendormodules_tcl<major>` when present) so vendored dependencies such as `voo` resolve in tests.
- Multi-process mode (`-tcltestoptions {-singleproc 0}`, completed 2026-07-18): each test file runs in a child process of the same executable via `testsupport/child_test_runner.tcl`, driven by a per-run environment payload `runtests.tcl` generates (package prefer latest, test tm paths, auto_path, modpod ifneeded definitions, tcltest options; per-file `-testdir` computed child-side). The child then mirrors the single-process testinterp preload: package require shellrun plus one no-op `shellrun::runx -tcl` call (the testinterp sources test files via runx, whose execution pulls further runtime deps - currently punk::lib). Several existing suites use punk::* commands without a package require of their own and depend on this preload (17 files / 97 tests error without shellrun, a further 7 files / 31 tests without the runx warmup; making suite dependencies explicit so children can go leaner is a candidate cleanup, verifiable with the parity tool below). The bootstrap warms `clock format` before the module-path wipe: first script-level clock use loads msgcat from the runtime's default module paths, which the test module paths do not supply (the single-process testinterp is shielded only by the runtests parent's earlier clock use - a latent gap for any test explicitly requiring msgcat in either mode).
- Multi-process failure classification: a nonzero child exit is reported as a file-level failure (compact/markdown/json carry an ERROR entry with errorcode `CHILDPROCESS exit <n>` and the child's stderr tail); exit-0 with no tcltest summary line remains the `missing-cleanupTests` warning. Prefer a native tclsh for multi-process runs: children of a kit executable boot via the kit's script dispatch with kit-stamped punk packages preloaded, which can shadow the src dev modules under test (the runner prints a warning).
- Parallel jobs (G-091, achieved 2026-07-19 - see goals/archive/G-091-runtests-parallel-jobs.md): `-jobs N` (N > 1) implies multi-process mode and runs test files through a worker pool of N concurrent child processes (tpool; per-child capture files and per-child tcltest `-tmpdir` subdirs so concurrent `makeFile` helpers cannot collide; longest-first submission using static weights from the 2026-07-18 timing measurements - the full-suite floor is the slowest single file, multishell.test). Files matching `-serial-paths` run sequentially after the parallel batch - default `{modules/punk/console/** modules/opunk/console/**}` (the console-state suites, which emit to and query the shared terminal; suites driven through piped-stdio child processes - shell/**, modules/punk/repl/** - stay parallel by default and can be added to `-serial-paths` if they prove flaky under a real console; each serial file costs its full ~3-4s child boot strictly after the parallel phase, so the list is deliberately minimal). Reports are unchanged in shape and emitted in discovery order. An explicit `-tcltestoptions {-singleproc 1}` conflicts with `-jobs > 1` (error). Watch-event semantics under `-jobs`: no `file-start` events; `file-collected` (with exitcode, output relayed at that moment) fires in completion order and the authoritative `file-end` (tallies) in discovery order. Per-test microsecond timings are noisier under CPU oversubscription - prefer sequential runs when `-slowest` matters. Child spawn under -jobs is background-exec plus `tcl::process status` polling: concurrent BLOCKING execs from worker threads convoy on Windows - they all return together at the longest-lived concurrent child's exit (proven 2026-07-19 with plain sleepers of 2-12s all walling at 12s; this cost early -jobs runs ~50s of false critical path). Runtimes without `tcl::process` (tcl 8.6) fall back to blocking exec, which re-convoys under -jobs. The runner prints phase times (parallel/serial-tail/processing) and the ten slowest child WALL times in non-json modes - child wall includes boot and file-level fixture cost that per-test usec timings miss.
- UDP watch mode (added 2026-07-18): `env(PUNK_TEST_UDPTEE)` = `<port>` or `<host>:<port>` (e.g `41197`) makes runtests emit the run over UDP via the shellfilter::log mechanism (shellthread worker threads -> tcludp, one datagram per line, cooked format whose source column carries the tag). Tag `runtests` carries lifecycle events; each event message is a well-formed tcl list `RUNTESTS-EVENT <eventname> ?key value ...?` (`run-start`, `file-start`, `file-end` with per-file tallies, `run-end`). Tags `teststdout`/`teststderr` carry test file output: streamed live per output chunk in single-process mode (`shellrun::runx -teelog` 0.2.0 - write traces on the capture variables, so captured results and reports are unchanged) and relayed per completed file by the parent in multi-process mode. No listener is required (sends to an unbound local port are discarded). The parent extends its auto_path with `<projectroot>/lib_tcl<N>/<arch>` so log workers resolve the project's tcludp 1.0.13+ (1.0.12's UDP_ExitProc is the G-036 exit-hang class). Viewer-side caveat: tcludp's RECEIVE fileevent under Tcl 9.0.3/Windows delivered only the first datagram in testing (send side unaffected; a .NET UdpClient received everything) - a viewer should use timer-driven non-blocking reads or a non-tcludp receiver until that is resolved.
- In watch mode the singleproc loop closes and terminates the shellfilter::log worker threads opened inside each per-file testinterp before deleting it (the `-teelog` tag workers plus shellrun's punkshout/punksherr teehandle workers). The terminate handshakes cost ~0.5-0.7s per file, so unwatched runs skip this and keep the pre-existing small leak of the two teehandle worker threads per file (noted for the shellfilter/shellthread audit; the process exits at run end regardless).
- Tests should run against source modules and libraries from `src/`, not installed packages or root-level build outputs.
@ -46,6 +47,7 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc
- Add `-slowest <n>` when timing outliers are relevant.
- Multi-process runs use `-tcltestoptions {-singleproc 0}` with otherwise identical flags. Check result parity between modes by capturing `-report json` stdout from each and comparing with `tclsh scriptlib/developer/runtests_parity.tcl <a.json> <b.json>` (ignores timings; exit 0 on parity).
- To watch runs live (including runs launched by agents or other processes, which inherit the environment): set `PUNK_TEST_UDPTEE=41197` and point a UDP listener/viewer at that port. No runner flags needed; results are unaffected.
- For broad runs, `-jobs 8` cuts full-suite wall time roughly 2.6x (~2m16s vs ~5m50s sequential on the reference machine, 2026-07-19; result parity verified) - e.g `<tcl_interpreter> src/tests/runtests.tcl -jobs 8 -report compact -show-passes 0`. The floor is the slowest single child (multishell.test, ~2m). Focused runs rarely need `-jobs`.
- Add `-strict-exit 1` when a nonzero shell exit code is needed for failures or parser warnings.
- Capture enough stderr or failure context to identify the failing command or assertion.
- For ERROR-status failures, the markdown report's `errorInfo` block and compact `message=` field carry the full Tcl error message; use `-report markdown` for untruncated context.
set slowest_tests [runtests_slowest_tests$all_passes$opt_slowest]
runtests_udptee_event run-end status $status total [dict get $tallydict total] passed [dict get $tallydict passed] skipped [dict get $tallydict skipped] failed [dict get $tallydict failed] files [llength$file_summaries] elapsed_seconds [expr{[clock seconds]-$ts_start}]
if{$jobs_mode}{
dict set runtests_phase_times processing_s [format%.1f [expr{([clock milliseconds]-$_phase_t2)/1000.0}]]
runtests_udptee_event run-end status $status total [dict get $tallydict total] passed [dict get $tallydict passed] skipped [dict get $tallydict skipped] failed [dict get $tallydict failed] files [llength$file_summaries] elapsed_seconds [expr{[clock seconds]-$ts_start}]{*}$runtests_phase_times
