# src/tests — Source-Tree Test Harness ## Purpose Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exercise source files under `src/` before installation or packaging, including module tests, shell behavior tests, and selected Tcl core compatibility tests. ## Ownership - Owned by the project maintainer; agents may update tests here when changing source behavior. - Do not delete, skip, or weaken existing tests without explicit user direction. - Keep build-output and installed-package tests outside this tree unless a local child AGENTS.md says otherwise. ## Local Contracts - `runtests.tcl` is the primary source-tree test entry point; sublevel `all.tcl` files are a legacy pattern and are not required. - Tests use `tcltest` unless a child AGENTS.md documents a different local harness. - `runtests.tcl` excludes `AGENTS.md` and `*.tcl` helper files when discovering `.test` files. - `runtests.tcl` defaults tcltest `-tmpdir` (the `makeFile`/`makeDirectory` location) to a fresh directory under the system temp area, deleted before exit, so test helper files never land in the source tree (aborted core exec.test runs previously littered `src/tests`). A `-tmpdir` supplied via `-tcltestoptions` overrides this. Applies to both modes: multi-process mode forwards `-tmpdir` (and all other tcltest options) to child processes via the generated environment payload. - The testinterp auto_path (replaced wholesale) also includes the running kit's internal `lib`/`lib_tcl` trees (zipfs app mount, tclkit `::tcl::kitpath`, cookfs `//cookit:/` — mirroring `src/vfs/_config/punk_main.tcl`), so kit-bundled packages (e.g. tcllib's `tcl::chan::fifo2`, needed by the shellrun harness) resolve when the project tree doesn't supply them. `tclPkgUnknown` scans only an entry plus immediate children, so the kit lib dirs must be explicit entries. No-op under a native tclsh. - The single-process testinterp runs `package prefer latest` so alpha-versioned dev modules (`999999.0a1.0`) are preferred over stable bootsupport/vendored copies on unversioned `package require`. - The testinterp module path includes `src/vendormodules` (and `src/vendormodules_tcl` when present) so vendored dependencies such as `voo` resolve in tests. - Multi-process mode (`-tcltestoptions {-singleproc 0}`, completed 2026-07-18): each test file runs in a child process of the same executable via `testsupport/child_test_runner.tcl`, driven by a per-run environment payload `runtests.tcl` generates (package prefer latest, test tm paths, auto_path, modpod ifneeded definitions, tcltest options; per-file `-testdir` computed child-side). The child then mirrors the single-process testinterp preload: package require shellrun plus one no-op `shellrun::runx -tcl` call (the testinterp sources test files via runx, whose execution pulls further runtime deps - currently punk::lib). Several existing suites use punk::* commands without a package require of their own and depend on this preload (17 files / 97 tests error without shellrun, a further 7 files / 31 tests without the runx warmup; making suite dependencies explicit so children can go leaner is a candidate cleanup, verifiable with the parity tool below). The bootstrap warms `clock format` before the module-path wipe: first script-level clock use loads msgcat from the runtime's default module paths, which the test module paths do not supply (the single-process testinterp is shielded only by the runtests parent's earlier clock use - a latent gap for any test explicitly requiring msgcat in either mode). - Multi-process failure classification: a nonzero child exit is reported as a file-level failure (compact/markdown/json carry an ERROR entry with errorcode `CHILDPROCESS exit ` and the child's stderr tail); exit-0 with no tcltest summary line remains the `missing-cleanupTests` warning. Prefer a native tclsh for multi-process runs: children of a kit executable boot via the kit's script dispatch with kit-stamped punk packages preloaded, which can shadow the src dev modules under test (the runner prints a warning). - Parallel jobs (G-091, achieved 2026-07-19 - see goals/archive/G-091-runtests-parallel-jobs.md): `-jobs N` (N > 1) implies multi-process mode and runs test files through a worker pool of N concurrent child processes (tpool; per-child capture files and per-child tcltest `-tmpdir` subdirs so concurrent `makeFile` helpers cannot collide; longest-first submission using static weights from the 2026-07-18/19 timing measurements - the full-suite floor is the slowest single child; the former ~2m multishell.test floor was removed by the G-092 split of that suite into four files, leaving all children under ~35s). Files matching `-serial-paths` run sequentially after the parallel batch - default `{modules/punk/console/** modules/opunk/console/**}` (the console-state suites, which emit to and query the shared terminal; suites driven through piped-stdio child processes - shell/**, modules/punk/repl/** - stay parallel by default and can be added to `-serial-paths` if they prove flaky under a real console; each serial file costs its full ~3-4s child boot strictly after the parallel phase, so the list is deliberately minimal). Reports are unchanged in shape and emitted in discovery order. An explicit `-tcltestoptions {-singleproc 1}` conflicts with `-jobs > 1` (error). Watch-event semantics under `-jobs`: no `file-start` events; `file-collected` (with exitcode, output relayed at that moment) fires in completion order and the authoritative `file-end` (tallies) in discovery order. Per-test microsecond timings are noisier under CPU oversubscription - prefer sequential runs when `-slowest` matters. Child spawn under -jobs is background-exec plus `tcl::process status` polling: concurrent BLOCKING execs from worker threads convoy on Windows - they all return together at the longest-lived concurrent child's exit (proven 2026-07-19 with plain sleepers of 2-12s all walling at 12s; this cost early -jobs runs ~50s of false critical path). Runtimes without `tcl::process` (tcl 8.6) fall back to blocking exec, which re-convoys under -jobs. The runner prints phase times (parallel/serial-tail/processing) and the ten slowest child WALL times in non-json modes - child wall includes boot and file-level fixture cost that per-test usec timings miss. - UDP watch mode (added 2026-07-18): `env(PUNK_TEST_UDPTEE)` = `` or `:` (e.g `41197`) makes runtests emit the run over UDP via the shellfilter::log mechanism (shellthread worker threads -> tcludp, one datagram per line, cooked format whose source column carries the tag). Tag `runtests` carries lifecycle events; each event message is a well-formed tcl list `RUNTESTS-EVENT ?key value ...?` (`run-start`, `file-start`, `file-end` with per-file tallies, `run-end`). Tags `teststdout`/`teststderr` carry test file output: streamed live per output chunk in single-process mode (`shellrun::runx -teelog` 0.2.0 - write traces on the capture variables, so captured results and reports are unchanged) and relayed per completed file by the parent in multi-process mode. No listener is required (sends to an unbound local port are discarded). The parent extends its auto_path with `/lib_tcl/` so log workers resolve the project's tcludp 1.0.13+ (1.0.12's UDP_ExitProc is the G-036 exit-hang class). Viewer-side caveat: tcludp's RECEIVE fileevent under Tcl 9.0.3/Windows delivered only the first datagram in testing (send side unaffected; a .NET UdpClient received everything) - a viewer should use timer-driven non-blocking reads or a non-tcludp receiver until that is resolved. - In watch mode the singleproc loop closes and terminates the shellfilter::log worker threads opened inside each per-file testinterp before deleting it (the `-teelog` tag workers plus shellrun's punkshout/punksherr teehandle workers). The terminate handshakes cost ~0.5-0.7s per file, so unwatched runs skip this and keep the pre-existing small leak of the two teehandle worker threads per file (noted for the shellfilter/shellthread audit; the process exits at run end regardless). - Tests should run against source modules and libraries from `src/`, not installed packages or root-level build outputs. - Test files must `package require` any extra packages explicitly. - Tcltest files must finish with `tcltest::cleanupTests`; missing cleanup produces a `missing-cleanupTests` runner warning and only untrusted observed testcase events. - tcltest compares the `-body` RETURN VALUE against `-result`. The suite convention of accumulating into `$result` via `lappend` works because `lappend` returns the list — but a body whose last command is a loop (`foreach`, `while`) returns the empty string; end such bodies with an explicit `set result`. - Agent-oriented runner output should use `-report compact -show-passes 0` for focused checks unless detailed Markdown pass lists are needed. - `-report json` emits a machine-readable final summary, but package-load warnings may still precede it on stdout/stderr, and the punk ANSI output stack may emit an SGR reset immediately before the JSON on the same line (`scriptlib/developer/runtests_parity.tcl` tolerates both). - Human-facing pass/fail/warning indicators in markdown/compact reports are ANSI-coloured only when stdout is a real windows console (`-colour auto` default: twapi `GetConsoleMode` on the STD_OUTPUT handle as the isatty-equivalent - `get_console_handle` is unsuitable, it succeeds for piped children of console shells; `NO_COLOR` honoured; `-colour on|off` overrides). Machine-facing output (the `RUNTESTS_RESULT` line, json reports) is never coloured, and piped/redirected output stays plain for agents. The runner uses raw literal SGR rather than `a+`/`a` so its indicators are independent of punk::console's process-global colour state. - ERROR-status failures now surface `errorInfo` (full Tcl error message/stack trace) in markdown, compact, and json reports. Compact mode truncates to a single line. - FAILED-status failures (result mismatch, not error) now surface `result_was` (actual) and `result_expected` (expected) in markdown, compact, and json reports. Compact mode truncates each to a single line. - `testsupport/` holds helper `.tcl` files sourced or exec'd by `.test` files or the runner (not discovered as suites). `testsupport/child_test_runner.tcl` is the multi-process-mode bootstrap `runtests.tcl` execs per test file (see the multi-process bullets above). `testsupport/repl_console_driver.tcl` (G-001) is exec'd in a child tclsh by `modules/punk/repl/testsuites/repl/consolebackends.test`: an interactive repl cannot run inside the shared testinterp (the codethread's quit/exit callbacks thread::send to the thread's MAIN interp, bypassing a child testinterp), so repl-through-console verification always drives a child process. `testsupport/wslprobe.tcl` (G-059) provides `::punktest::wsl`: a memoized capability probe yielding the `wsllinux` constraint (default distro launches, answers uname/tool probes, and round-trips a file through a native tempdir - NOT mere wsl.exe existence) plus native-filesystem staging helpers (`staging_create`/`staging_copy_in`/`run_in`/`staging_cleanup`). WSL-gated tests must execute from a staging dir on the distro's native filesystem with the shared `/mnt` path used only for one-way copy-in/out - never operate on the Windows checkout from inside WSL (DrvFs is slow and cross-boundary stat differences make git re-hash its index and fossil see phantom changes). Probe invocations use `wsl -e ` only (`wsl --status`/`-l` emit UTF-16). Known limitation: a broken-but-present WSL that hangs (rather than errors) on `wsl -e` can stall the probe. - Test provenance comments: an agent adding a test places one comment line directly above it — or one line above a contiguous group added in the same piece of work — of the form `#added (agent[, G-])[ - ]`. Date and the `agent` marker are mandatory; the goal reference is mandatory when a goal motivated the test; the note is optional (omit it when the test name/description already says why the test exists). Provenance lines record immutable facts only — never expectations such as "G-NNN will flip this" (that state lives in `_GAP_`-style test names and the owning goal's files). User-added tests may use the same form with `(user)` but are not required to. Do not retrofit existing suites in bulk; add provenance opportunistically when editing a file. Rationale: provenance must survive where VCS history does not travel — packaged test modules (G-029), the git+fossil dual history, and file regeneration/moves. ## Work Guidance - Select an appropriate interpreter, such as `tclsh`, `tclsh90`, or a built Punk executable such as `punk902z`. - Run the full source-tree suite with ` src/tests/runtests.tcl`. - Use `-include-paths` with paths relative to `src/tests/`, using forward slashes. The flag accepts a space-separated list of glob patterns and may also be repeated, with all occurrences accumulating (`-multiple 1` as of 2026-07-10; previously the last flag silently won). - Use `-tcltestoptions {-match }` for focused single-test runs. - For agent-efficient focused runs, prefer ` src/tests/runtests.tcl -report compact -show-passes 0 -include-paths `. - Multiple trailing file tails are supported and match independently: `runtests.tcl foo.test bar.test` runs both in one invocation (fixed 2026-07-17; previously multiple names collapsed into one glob matching zero files). - Treat `RUNTESTS_RESULT status=warn` and compact warning reasons such as `missing-cleanupTests` as incomplete test results, even if observed pass events are listed. - Add `-slowest ` when timing outliers are relevant. - Multi-process runs use `-tcltestoptions {-singleproc 0}` with otherwise identical flags. Check result parity between modes by capturing `-report json` stdout from each and comparing with `tclsh scriptlib/developer/runtests_parity.tcl ` (ignores timings; exit 0 on parity). - To watch runs live (including runs launched by agents or other processes, which inherit the environment): set `PUNK_TEST_UDPTEE=41197` and point a UDP listener/viewer at that port. No runner flags needed; results are unaffected. - For broad runs use `-jobs` - e.g ` src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0`. Reference-machine full-suite scaling (2026-07-19, post G-092 split, result parity verified at each level): sequential ~5m40s, `-jobs 8` ~79-86s, `-jobs 16` ~61s (~5.6x), plateau by `-jobs 24` (~60s) - beyond 16 there is nothing left to overlap: the wall decomposes into the slowest single child (~31s, the runtimecmd pair - single tests, not further splittable), the serial console tail (~16s piped; larger under a real console where those suites actually run), and runner overhead (~10s). `-jobs 16` is the reference-machine sweet spot. Use `-jobs` for ANY run spanning more than a handful of test files - subtree runs count (e.g `-include-paths modules/punk/args/**` is 28 files: ~50s sequential vs ~11s at `-jobs 16`); only single-file or few-file runs gain nothing from it. - Add `-strict-exit 1` when a nonzero shell exit code is needed for failures or parser warnings. - Capture enough stderr or failure context to identify the failing command or assertion. - For ERROR-status failures, the markdown report's `errorInfo` block and compact `message=` field carry the full Tcl error message; use `-report markdown` for untruncated context. - For FAILED-status failures (result mismatch), the markdown report's `result_was`/`result_expected` blocks and compact `actual=`/`expected=` fields show the actual-vs-expected comparison; use `-report markdown` for full multi-line values. ## Verification - ` src/tests/runtests.tcl` passes when broad source-tree test coverage is relevant. - Focused checks use ` src/tests/runtests.tcl -report compact -show-passes 0 -include-paths ` and optional file-tail globs. - Changes to `runtests.tcl` or `testsupport/child_test_runner.tcl` are verified by running the full suite in both modes with `-report json` and confirming `scriptlib/developer/runtests_parity.tcl` reports `PARITY: ok`. - Documentation-only changes are verified by reviewing the affected DOX chain and diff. ## Child DOX Index - `modules/` — Source module test suites mirroring editable module namespaces (see modules/AGENTS.md) - `core/` — Selected Tcl core compatibility tests for standard Tcl and built Punk executables (see core/AGENTS.md) - `shell/` — Shell behavior tests and command-line interaction suites (see shell/AGENTS.md)