lean test children: explicit suite deps, shellrun preload removed (-jobs ~50s)
The multiproc child bootstrap no longer preloads shellrun + the no-op
runx warmup: 17 test files gained their missing explicit package
require lines (punk::ansi for the bare ansi suites, punk::args for the
bare args suites incl dev/dynamic-cache, punk::lib for the lib suites
and for grepstr/untabify/examples whose punk::ansi/punk::args commands
reach punk::lib internally, plus textblock for examples.test - caught
by the empirical lean run: punk::args only soft-requires textblock and
punk::args::helpers::example calls textblock::join). Children now load
only tcltest plus each file's declared deps; the singleproc testinterp
keeps shellrun as the runner's own capture mechanism.
Module-web finding recorded in src/tests/AGENTS.md and the test-file
comments rather than fixed: punk::ansi and punk::args call punk::lib
without requiring it, and a hard back-require is blocked by punk::lib's
own hard require of punk::args (cycle) - deps declared at test level;
a considered module-level resolution remains open.
UDP watch-mode note folded into child_test_runner.tcl per user
direction: no interaction today (the parent relays child output at file
granularity; verified post-lean - events + relay flow); any future live
per-child tee must be conditional on PUNK_TEST_UDPTEE so unwatched
children stay lean.
Verified (native tclsh 9.0.3): full-suite PARITY ok for lean -jobs 16
vs the pre-lean baseline, between repeated lean -jobs 16 runs, and
lean -jobs 1 vs -jobs 16 (989/973/15/1, exec-14.3 sole failure).
Timings: -jobs 16 70.7s -> 49-53s; sequential multiproc ~5m30 -> 4m32;
singleproc unchanged. AGENTS scaling guidance refreshed.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
@ -19,7 +19,7 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc
- The testinterp auto_path (replaced wholesale) also includes the running kit's internal `lib`/`lib_tcl<N>` trees (zipfs app mount, tclkit `::tcl::kitpath`, cookfs `//cookit:/` — mirroring `src/vfs/_config/punk_main.tcl`), so kit-bundled packages (e.g. tcllib's `tcl::chan::fifo2`, needed by the shellrun harness) resolve when the project tree doesn't supply them. `tclPkgUnknown` scans only an entry plus immediate children, so the kit lib dirs must be explicit entries. No-op under a native tclsh.
- The single-process testinterp runs `package prefer latest` so alpha-versioned dev modules (`999999.0a1.0`) are preferred over stable bootsupport/vendored copies on unversioned `package require`.
- The testinterp module path includes `src/vendormodules` (and `src/vendormodules_tcl<major>` when present) so vendored dependencies such as `voo` resolve in tests.
- Multi-process mode (`-tcltestoptions {-singleproc 0}`, completed 2026-07-18): each test file runs in a child process of the same executable via `testsupport/child_test_runner.tcl`, driven by a per-run environment payload `runtests.tcl` generates (package prefer latest, test tm paths, auto_path, modpod ifneeded definitions, tcltest options; per-file `-testdir` computed child-side). The child then mirrors the single-process testinterp preload: package require shellrun plus one no-op `shellrun::runx -tcl` call (the testinterp sources test files via runx, whose execution pulls further runtime deps - currently punk::lib). Several existing suites use punk::* commands without a package require of their own and depend on this preload (17 files / 97 tests error without shellrun, a further 7 files / 31 tests without the runx warmup; making suite dependencies explicit so children can go leaner is a candidate cleanup, verifiable with the parity tool below). The bootstrap warms `clock format` before the module-path wipe: first script-level clock use loads msgcat from the runtime's default module paths, which the test module paths do not supply (the single-process testinterp is shielded only by the runtests parent's earlier clock use - a latent gap for any test explicitly requiring msgcat in either mode).
- Multi-process mode (`-tcltestoptions {-singleproc 0}`, completed 2026-07-18): each test file runs in a child process of the same executable via `testsupport/child_test_runner.tcl`, driven by a per-run environment payload `runtests.tcl` generates (package prefer latest, test tm paths, auto_path, modpod ifneeded definitions, tcltest options; per-file `-testdir` computed child-side). Children are LEAN (2026-07-19 cleanup): they load only tcltest plus whatever each test file declares - the previous shellrun-preload + runx-warmup mirror of the testinterp was removed after 17 files gained their missing explicit `package require` lines (the contract above always required them; the parity tool pinpointed the violators, and an empirical lean run caught the final straggler - examples.test needing textblock because punk::args only soft-requires it). The single-process testinterp still preloads shellrun, but as the RUNNER's capture mechanism (runx -tcl sourcing) - tests must not rely on it. Known module-web finding from the sweep: punk::ansi and punk::args call punk::lib internally without requiring it, and a hard back-require is blocked by punk::lib's own hard require of punk::args (cycle) - dependencies are therefore declared at test level (grepstr/untabify/examples carry comments); a considered module-level resolution remains open. Watch-mode interaction notes live in child_test_runner.tcl (none today - the parent relays; a future live per-child tee must be conditional on the env var so unwatched children stay lean). The bootstrap warms `clock format` before the module-path wipe: first script-level clock use loads msgcat from the runtime's default module paths, which the test module paths do not supply (the single-process testinterp is shielded only by the runtests parent's earlier clock use - a latent gap for any test explicitly requiring msgcat in either mode).
- Multi-process failure classification: a nonzero child exit is reported as a file-level failure (compact/markdown/json carry an ERROR entry with errorcode `CHILDPROCESS exit <n>` and the child's stderr tail); exit-0 with no tcltest summary line remains the `missing-cleanupTests` warning. Prefer a native tclsh for multi-process runs: children of a kit executable boot via the kit's script dispatch with kit-stamped punk packages preloaded, which can shadow the src dev modules under test (the runner prints a warning).
- Parallel jobs (G-091, achieved 2026-07-19 - see goals/archive/G-091-runtests-parallel-jobs.md): `-jobs N` (N > 1) implies multi-process mode and runs test files through a worker pool of N concurrent child processes (tpool; per-child capture files and per-child tcltest `-tmpdir` subdirs so concurrent `makeFile` helpers cannot collide; longest-first submission using static weights from the 2026-07-18/19 timing measurements - the full-suite floor is the slowest single child; the former ~2m multishell.test floor was removed by the G-092 split of that suite into four files, leaving all children under ~35s). Files matching `-serial-paths` run sequentially after the parallel batch - default `{modules/punk/console/** modules/opunk/console/**}` (the console-state suites, which emit to and query the shared terminal; suites driven through piped-stdio child processes - shell/**, modules/punk/repl/** - stay parallel by default and can be added to `-serial-paths` if they prove flaky under a real console; each serial file costs its full ~3-4s child boot strictly after the parallel phase, so the list is deliberately minimal). Reports are unchanged in shape and emitted in discovery order. An explicit `-tcltestoptions {-singleproc 1}` conflicts with `-jobs > 1` (error). Watch-event semantics under `-jobs`: no `file-start` events; `file-collected` (with exitcode, output relayed at that moment) fires in completion order and the authoritative `file-end` (tallies) in discovery order. Per-test microsecond timings are noisier under CPU oversubscription - prefer sequential runs when `-slowest` matters. Child spawn under -jobs is background-exec plus `tcl::process status` polling: concurrent BLOCKING execs from worker threads convoy on Windows - they all return together at the longest-lived concurrent child's exit (proven 2026-07-19 with plain sleepers of 2-12s all walling at 12s; this cost early -jobs runs ~50s of false critical path). Runtimes without `tcl::process` (tcl 8.6) fall back to blocking exec, which re-convoys under -jobs. The runner prints phase times (parallel/serial-tail/processing) and the ten slowest child WALL times in non-json modes - child wall includes boot and file-level fixture cost that per-test usec timings miss.
- UDP watch mode (added 2026-07-18): `env(PUNK_TEST_UDPTEE)` = `<port>` or `<host>:<port>` (e.g `41197`) makes runtests emit the run over UDP via the shellfilter::log mechanism (shellthread worker threads -> tcludp, one datagram per line, cooked format whose source column carries the tag). Tag `runtests` carries lifecycle events; each event message is a well-formed tcl list `RUNTESTS-EVENT <eventname> ?key value ...?` (`run-start`, `file-start`, `file-end` with per-file tallies, `run-end`). Tags `teststdout`/`teststderr` carry test file output: streamed live per output chunk in single-process mode (`shellrun::runx -teelog` 0.2.0 - write traces on the capture variables, so captured results and reports are unchanged) and relayed per completed file by the parent in multi-process mode. No listener is required (sends to an unbound local port are discarded). The parent extends its auto_path with `<projectroot>/lib_tcl<N>/<arch>` so log workers resolve the project's tcludp 1.0.13+ (1.0.12's UDP_ExitProc is the G-036 exit-hang class). Viewer-side caveat: tcludp's RECEIVE fileevent under Tcl 9.0.3/Windows delivered only the first datagram in testing (send side unaffected; a .NET UdpClient received everything) - a viewer should use timer-driven non-blocking reads or a non-tcludp receiver until that is resolved.
@ -48,7 +48,7 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc
- Add `-slowest <n>` when timing outliers are relevant.
- Multi-process runs use `-tcltestoptions {-singleproc 0}` with otherwise identical flags. Check result parity between modes by capturing `-report json` stdout from each and comparing with `tclsh scriptlib/developer/runtests_parity.tcl <a.json> <b.json>` (ignores timings; exit 0 on parity).
- To watch runs live (including runs launched by agents or other processes, which inherit the environment): set `PUNK_TEST_UDPTEE=41197` and point a UDP listener/viewer at that port. No runner flags needed; results are unaffected.
- For broad runs use `-jobs` - e.g `<tcl_interpreter> src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0`. Reference-machine full-suite scaling (2026-07-19, post G-092 split, result parity verified at each level): sequential ~5m40s, `-jobs 8` ~79-86s, `-jobs 16` ~61s (~5.6x), plateau by `-jobs 24` (~60s) - beyond 16 there is nothing left to overlap: the wall decomposes into the slowest single child (~31s, the runtimecmd pair - single tests, not further splittable), the serial console tail (~16s piped; larger under a real console where those suites actually run), and runner overhead (~10s). `-jobs 16` is the reference-machine sweet spot. Use `-jobs` for ANY run spanning more than a handful of test files - subtree runs count (e.g `-include-paths modules/punk/args/**` is 28 files: ~50s sequential vs ~11s at `-jobs 16`); only single-file or few-file runs gain nothing from it.
- For broad runs use `-jobs` - e.g `<tcl_interpreter> src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0`. Reference-machine full-suite scaling (2026-07-19, post G-092 split + lean-children cleanup, result parity verified at each level): singleproc sequential ~5m40s, multiproc sequential ~4m30s, `-jobs 16` ~50s (~6.8x), with diminishing returns past 16 - the wall decomposes into the slowest single child (~30s, the runtimecmd pair - single tests, not further splittable), the serial console tail (a few seconds piped now children boot lean; larger under a real console where those suites actually run), and runner overhead (~10s). `-jobs 16` is the reference-machine sweet spot. Use `-jobs` for ANY run spanning more than a handful of test files - subtree runs count (e.g `-include-paths modules/punk/args/**` is 28 files: ~50s sequential vs ~11s at `-jobs 16`); only single-file or few-file runs gain nothing from it.
- Add `-strict-exit 1` when a nonzero shell exit code is needed for failures or parser warnings.
- Capture enough stderr or failure context to identify the failing command or assertion.
- For ERROR-status failures, the markdown report's `errorInfo` block and compact `message=` field carry the full Tcl error message; use `-report markdown` for untruncated context.
package require punk::lib ;#untabify internals call punk::lib - not declared by the punk::ansi module (require-cycle constraints; see lean-children cleanup 2026-07-19)
#added 2026-07-14 (agent) - punk::ansi::untabify characterization, precursor to the
#planned punk::ns hygiene pass (punk::ns::corp -untabify delegates here, and
package require punk::lib ;#examples rendering internals reach punk::lib - not declared by the punk::args module (require-cycle constraints; see lean-children cleanup 2026-07-19)
package require textblock ;#punk::args::helpers::example calls textblock::join - punk::args only soft-requires textblock and the lean child env showed that does not cover this path
