From a590ff4aec2deb87a8f1cc02b36f083f3b682153 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Sun, 19 Jul 2026 07:01:59 +1000 Subject: [PATCH] src/tests/AGENTS.md: -jobs guidance updated with measured scaling curve User-measured reference-machine numbers 2026-07-19: -jobs 8 ~79-86s, -jobs 16 ~61s (~5.6x vs ~5m40s sequential), plateau by -jobs 24 (~60s). Guidance now recommends -jobs 16 and records the plateau anatomy (slowest single child ~31s runtimecmd pair, serial console tail ~16s piped / larger under a real console, ~10s runner overhead). Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- src/tests/AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/tests/AGENTS.md b/src/tests/AGENTS.md index fd56c1b4..63c1482a 100644 --- a/src/tests/AGENTS.md +++ b/src/tests/AGENTS.md @@ -47,7 +47,7 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc - Add `-slowest ` when timing outliers are relevant. - Multi-process runs use `-tcltestoptions {-singleproc 0}` with otherwise identical flags. Check result parity between modes by capturing `-report json` stdout from each and comparing with `tclsh scriptlib/developer/runtests_parity.tcl ` (ignores timings; exit 0 on parity). - To watch runs live (including runs launched by agents or other processes, which inherit the environment): set `PUNK_TEST_UDPTEE=41197` and point a UDP listener/viewer at that port. No runner flags needed; results are unaffected. -- For broad runs, `-jobs 8` cuts full-suite wall time roughly 4x (~1m26s vs ~5m40s sequential on the reference machine, 2026-07-19 post G-092 split; result parity verified) - e.g ` src/tests/runtests.tcl -jobs 8 -report compact -show-passes 0`. No child process exceeds ~31s; the wall is now work-distribution bound rather than dominated by any single file. Focused runs rarely need `-jobs`. +- For broad runs use `-jobs` - e.g ` src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0`. Reference-machine full-suite scaling (2026-07-19, post G-092 split, result parity verified at each level): sequential ~5m40s, `-jobs 8` ~79-86s, `-jobs 16` ~61s (~5.6x), plateau by `-jobs 24` (~60s) - beyond 16 there is nothing left to overlap: the wall decomposes into the slowest single child (~31s, the runtimecmd pair - single tests, not further splittable), the serial console tail (~16s piped; larger under a real console where those suites actually run), and runner overhead (~10s). `-jobs 16` is the reference-machine sweet spot; focused runs rarely need `-jobs`. - Add `-strict-exit 1` when a nonzero shell exit code is needed for failures or parser warnings. - Capture enough stderr or failure context to identify the failing command or assertion. - For ERROR-status failures, the markdown report's `errorInfo` block and compact `message=` field carry the full Tcl error message; use `-report markdown` for untruncated context.