punkshell/src/tests/AGENTS.md

src/tests — Source-Tree Test Harness

Purpose

Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exercise source files under src/ before installation or packaging, including module tests, shell behavior tests, and selected Tcl core compatibility tests.

Ownership

• Owned by the project maintainer; agents may update tests here when changing source behavior.
• Do not delete, skip, or weaken existing tests without explicit user direction.
• Keep build-output and installed-package tests outside this tree unless a local child AGENTS.md says otherwise.

Local Contracts

• runtests.tcl is the primary source-tree test entry point; sublevel all.tcl files are a legacy pattern and are not required.
• Tests use tcltest unless a child AGENTS.md documents a different local harness.
• runtests.tcl excludes AGENTS.md and *.tcl helper files when discovering .test files.
• runtests.tcl defaults tcltest -tmpdir (the makeFile/makeDirectory location) to a fresh directory under the system temp area, deleted before exit, so test helper files never land in the source tree (aborted core exec.test runs previously littered src/tests). A -tmpdir supplied via -tcltestoptions overrides this. Applies to both modes: multi-process mode forwards -tmpdir (and all other tcltest options) to child processes via the generated environment payload.
• The testinterp auto_path (replaced wholesale) also includes the running kit's internal lib/lib_tcl<N> trees (zipfs app mount, tclkit ::tcl::kitpath, cookfs //cookit:/ — mirroring src/vfs/_config/punk_main.tcl), so kit-bundled packages (e.g. tcllib's tcl::chan::fifo2, needed by the shellrun harness) resolve when the project tree doesn't supply them. tclPkgUnknown scans only an entry plus immediate children, so the kit lib dirs must be explicit entries. No-op under a native tclsh.
• The single-process testinterp runs package prefer latest so alpha-versioned dev modules (999999.0a1.0) are preferred over stable bootsupport/vendored copies on unversioned package require.
• The testinterp module path includes src/vendormodules (and src/vendormodules_tcl<major> when present) so vendored dependencies such as voo resolve in tests.
• Multi-process mode (-tcltestoptions {-singleproc 0}, completed 2026-07-18): each test file runs in a child process of the same executable via testsupport/child_test_runner.tcl, driven by a per-run environment payload runtests.tcl generates (package prefer latest, test tm paths, auto_path, modpod ifneeded definitions, tcltest options; per-file -testdir computed child-side). The child then mirrors the single-process testinterp preload: package require shellrun plus one no-op shellrun::runx -tcl call (the testinterp sources test files via runx, whose execution pulls further runtime deps - currently punk::lib). Several existing suites use punk::* commands without a package require of their own and depend on this preload (17 files / 97 tests error without shellrun, a further 7 files / 31 tests without the runx warmup; making suite dependencies explicit so children can go leaner is a candidate cleanup, verifiable with the parity tool below). The bootstrap warms clock format before the module-path wipe: first script-level clock use loads msgcat from the runtime's default module paths, which the test module paths do not supply (the single-process testinterp is shielded only by the runtests parent's earlier clock use - a latent gap for any test explicitly requiring msgcat in either mode).
• Multi-process failure classification: a nonzero child exit is reported as a file-level failure (compact/markdown/json carry an ERROR entry with errorcode CHILDPROCESS exit <n> and the child's stderr tail); exit-0 with no tcltest summary line remains the missing-cleanupTests warning. Prefer a native tclsh for multi-process runs: children of a kit executable boot via the kit's script dispatch with kit-stamped punk packages preloaded, which can shadow the src dev modules under test (the runner prints a warning).
• Parallel jobs (G-091, achieved 2026-07-19 - see goals/archive/G-091-runtests-parallel-jobs.md): -jobs N (N > 1) implies multi-process mode and runs test files through a worker pool of N concurrent child processes (tpool; per-child capture files and per-child tcltest -tmpdir subdirs so concurrent makeFile helpers cannot collide; longest-first submission using static weights from the 2026-07-18/19 timing measurements - the full-suite floor is the slowest single child; the former ~2m multishell.test floor was removed by the G-092 split of that suite into four files, leaving all children under ~35s). Files matching -serial-paths run sequentially after the parallel batch - default {modules/punk/console/** modules/opunk/console/**} (the console-state suites, which emit to and query the shared terminal; suites driven through piped-stdio child processes - shell/, modules/punk/repl/ - stay parallel by default and can be added to -serial-paths if they prove flaky under a real console; each serial file costs its full ~3-4s child boot strictly after the parallel phase, so the list is deliberately minimal). Reports are unchanged in shape and emitted in discovery order. An explicit -tcltestoptions {-singleproc 1} conflicts with -jobs > 1 (error). Watch-event semantics under -jobs: no file-start events; file-collected (with exitcode, output relayed at that moment) fires in completion order and the authoritative file-end (tallies) in discovery order. Per-test microsecond timings are noisier under CPU oversubscription - prefer sequential runs when -slowest matters. Child spawn under -jobs is background-exec plus tcl::process status polling: concurrent BLOCKING execs from worker threads convoy on Windows - they all return together at the longest-lived concurrent child's exit (proven 2026-07-19 with plain sleepers of 2-12s all walling at 12s; this cost early -jobs runs ~50s of false critical path). Runtimes without tcl::process (tcl 8.6) fall back to blocking exec, which re-convoys under -jobs. The runner prints phase times (parallel/serial-tail/processing) and the ten slowest child WALL times in non-json modes - child wall includes boot and file-level fixture cost that per-test usec timings miss.
• UDP watch mode (added 2026-07-18): env(PUNK_TEST_UDPTEE) = <port> or <host>:<port> (e.g 41197) makes runtests emit the run over UDP via the shellfilter::log mechanism (shellthread worker threads -> tcludp, one datagram per line, cooked format whose source column carries the tag). Tag runtests carries lifecycle events; each event message is a well-formed tcl list RUNTESTS-EVENT <eventname> ?key value ...? (run-start, file-start, file-end with per-file tallies, run-end). Tags teststdout/teststderr carry test file output: streamed live per output chunk in single-process mode (shellrun::runx -teelog 0.2.0 - write traces on the capture variables, so captured results and reports are unchanged) and relayed per completed file by the parent in multi-process mode. No listener is required (sends to an unbound local port are discarded). The parent extends its auto_path with <projectroot>/lib_tcl<N>/<arch> so log workers resolve the project's tcludp 1.0.13+ (1.0.12's UDP_ExitProc is the G-036 exit-hang class). Viewer-side caveat: tcludp's RECEIVE fileevent under Tcl 9.0.3/Windows delivered only the first datagram in testing (send side unaffected; a .NET UdpClient received everything) - a viewer should use timer-driven non-blocking reads or a non-tcludp receiver until that is resolved.
• In watch mode the singleproc loop closes and terminates the shellfilter::log worker threads opened inside each per-file testinterp before deleting it (the -teelog tag workers plus shellrun's punkshout/punksherr teehandle workers). The terminate handshakes cost ~0.5-0.7s per file, so unwatched runs skip this and keep the pre-existing small leak of the two teehandle worker threads per file (noted for the shellfilter/shellthread audit; the process exits at run end regardless).
• Tests should run against source modules and libraries from src/, not installed packages or root-level build outputs.
• Test files must package require any extra packages explicitly.
• Tcltest files must finish with tcltest::cleanupTests; missing cleanup produces a missing-cleanupTests runner warning and only untrusted observed testcase events.
• tcltest compares the -body RETURN VALUE against -result. The suite convention of accumulating into $result via lappend works because lappend returns the list — but a body whose last command is a loop (foreach, while) returns the empty string; end such bodies with an explicit set result.
• Agent-oriented runner output should use -report compact -show-passes 0 for focused checks unless detailed Markdown pass lists are needed.
• -report json emits a machine-readable final summary, but package-load warnings may still precede it on stdout/stderr, and the punk ANSI output stack may emit an SGR reset immediately before the JSON on the same line (scriptlib/developer/runtests_parity.tcl tolerates both).
• Human-facing pass/fail/warning indicators in markdown/compact reports are ANSI-coloured only when stdout is a real windows console (-colour auto default: twapi GetConsoleMode on the STD_OUTPUT handle as the isatty-equivalent - get_console_handle is unsuitable, it succeeds for piped children of console shells; NO_COLOR honoured; -colour on|off overrides). Machine-facing output (the RUNTESTS_RESULT line, json reports) is never coloured, and piped/redirected output stays plain for agents. The runner uses raw literal SGR rather than a+/a so its indicators are independent of punk::console's process-global colour state.
• ERROR-status failures now surface errorInfo (full Tcl error message/stack trace) in markdown, compact, and json reports. Compact mode truncates to a single line.
• FAILED-status failures (result mismatch, not error) now surface result_was (actual) and result_expected (expected) in markdown, compact, and json reports. Compact mode truncates each to a single line.
• testsupport/ holds helper .tcl files sourced or exec'd by .test files or the runner (not discovered as suites). testsupport/child_test_runner.tcl is the multi-process-mode bootstrap runtests.tcl execs per test file (see the multi-process bullets above). testsupport/repl_console_driver.tcl (G-001) is exec'd in a child tclsh by modules/punk/repl/testsuites/repl/consolebackends.test: an interactive repl cannot run inside the shared testinterp (the codethread's quit/exit callbacks thread::send to the thread's MAIN interp, bypassing a child testinterp), so repl-through-console verification always drives a child process. testsupport/wslprobe.tcl (G-059) provides ::punktest::wsl: a memoized capability probe yielding the wsllinux constraint (default distro launches, answers uname/tool probes, and round-trips a file through a native tempdir - NOT mere wsl.exe existence) plus native-filesystem staging helpers (staging_create/staging_copy_in/run_in/staging_cleanup). WSL-gated tests must execute from a staging dir on the distro's native filesystem with the shared /mnt path used only for one-way copy-in/out - never operate on the Windows checkout from inside WSL (DrvFs is slow and cross-boundary stat differences make git re-hash its index and fossil see phantom changes). Probe invocations use wsl -e <cmd> only (wsl --status/-l emit UTF-16). Known limitation: a broken-but-present WSL that hangs (rather than errors) on wsl -e can stall the probe.
• Test provenance comments: an agent adding a test places one comment line directly above it — or one line above a contiguous group added in the same piece of work — of the form #added <YYYY-MM-DD> (agent[, G-<id>])[ - <short note>]. Date and the agent marker are mandatory; the goal reference is mandatory when a goal motivated the test; the note is optional (omit it when the test name/description already says why the test exists). Provenance lines record immutable facts only — never expectations such as "G-NNN will flip this" (that state lives in _GAP_-style test names and the owning goal's files). User-added tests may use the same form with (user) but are not required to. Do not retrofit existing suites in bulk; add provenance opportunistically when editing a file. Rationale: provenance must survive where VCS history does not travel — packaged test modules (G-029), the git+fossil dual history, and file regeneration/moves.

Work Guidance

• Select an appropriate interpreter, such as tclsh, tclsh90, or a built Punk executable such as punk902z.
• Run the full source-tree suite with <tcl_interpreter> src/tests/runtests.tcl.
• Use -include-paths with paths relative to src/tests/, using forward slashes. The flag accepts a space-separated list of glob patterns and may also be repeated, with all occurrences accumulating (-multiple 1 as of 2026-07-10; previously the last flag silently won).
• Use -tcltestoptions {-match <test_name>} for focused single-test runs.
• For agent-efficient focused runs, prefer <tcl_interpreter> src/tests/runtests.tcl -report compact -show-passes 0 -include-paths <relative/path/**> <file-tail.test>.
• Multiple trailing file tails are supported and match independently: runtests.tcl foo.test bar.test runs both in one invocation (fixed 2026-07-17; previously multiple names collapsed into one glob matching zero files).
• Treat RUNTESTS_RESULT status=warn and compact warning reasons such as missing-cleanupTests as incomplete test results, even if observed pass events are listed.
• Add -slowest <n> when timing outliers are relevant.
• Multi-process runs use -tcltestoptions {-singleproc 0} with otherwise identical flags. Check result parity between modes by capturing -report json stdout from each and comparing with tclsh scriptlib/developer/runtests_parity.tcl <a.json> <b.json> (ignores timings; exit 0 on parity).
• To watch runs live (including runs launched by agents or other processes, which inherit the environment): set PUNK_TEST_UDPTEE=41197 and point a UDP listener/viewer at that port. No runner flags needed; results are unaffected.
• For broad runs use -jobs - e.g <tcl_interpreter> src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0. Reference-machine full-suite scaling (2026-07-19, post G-092 split, result parity verified at each level): sequential ~5m40s, -jobs 8 ~79-86s, -jobs 16 ~61s (~5.6x), plateau by -jobs 24 (~60s) - beyond 16 there is nothing left to overlap: the wall decomposes into the slowest single child (~31s, the runtimecmd pair - single tests, not further splittable), the serial console tail (~16s piped; larger under a real console where those suites actually run), and runner overhead (~10s). -jobs 16 is the reference-machine sweet spot. Use -jobs for ANY run spanning more than a handful of test files - subtree runs count (e.g -include-paths modules/punk/args/** is 28 files: ~50s sequential vs ~11s at -jobs 16); only single-file or few-file runs gain nothing from it.
• Add -strict-exit 1 when a nonzero shell exit code is needed for failures or parser warnings.
• Capture enough stderr or failure context to identify the failing command or assertion.
• For ERROR-status failures, the markdown report's errorInfo block and compact message= field carry the full Tcl error message; use -report markdown for untruncated context.
• For FAILED-status failures (result mismatch), the markdown report's result_was/result_expected blocks and compact actual=/expected= fields show the actual-vs-expected comparison; use -report markdown for full multi-line values.

Verification

• <tcl_interpreter> src/tests/runtests.tcl passes when broad source-tree test coverage is relevant.
• Focused checks use <tcl_interpreter> src/tests/runtests.tcl -report compact -show-passes 0 -include-paths <relative/path/**> and optional file-tail globs.
• Changes to runtests.tcl or testsupport/child_test_runner.tcl are verified by running the full suite in both modes with -report json and confirming scriptlib/developer/runtests_parity.tcl reports PARITY: ok.
• Documentation-only changes are verified by reviewing the affected DOX chain and diff.

Child DOX Index

• modules/ — Source module test suites mirroring editable module namespaces (see modules/AGENTS.md)
• core/ — Selected Tcl core compatibility tests for standard Tcl and built Punk executables (see core/AGENTS.md)
• shell/ — Shell behavior tests and command-line interaction suites (see shell/AGENTS.md)