--- name: tcl-runtests description: "Use BEFORE running, adding or editing any tests in this repo, and BEFORE choosing an interpreter to run them with. Canonical runner: the repo's zig-built tclsh under src/buildsuites/_build or a runtime-store tclsh under bin/runtime; a fresh checkout bootstraps either via one-time 'zig build bootstrap' (only the pinned zig required) or by fetching prebuilt runtimes from the punkbin artifact repo via bin/punk-runtime (user's choice). Bare 'tclsh' from an agent shell is often the MSYS/Git-for-Windows one and yields spurious failures. Quickstart invocation and targeting forms, result-reading rules, and the agent-environment traps that waste runs: NO_COLOR=1 breaking exact-SGR pins agent-side only, PowerShell-tool attached consoles, editing .test files mid-run, locked/stale kit exes (PUNK_SHELL_TEST_EXE), warn-status results that look like passes, plus .test authoring micro-traps. Cost: focused runs seconds; full suite ~50s at -jobs 16. Full harness contract: src/tests/AGENTS.md." --- # tcl-runtests Running the source-tree test suite (`src/tests/runtests.tcl`) as an agent: which interpreter to drive it with, the invocation forms worth using, and the environment traps that produce spurious failures or wasted runs. This is the quickstart + traps layer only - the full harness contract (targeting grammar, multi-process/jobs internals, watch mode, provenance rules) lives in `src/tests/AGENTS.md` and stays authoritative. ## Runner interpreter Preferred: the repo's own zig-built runtime - portable across machines, no reliance on whatever tclsh a host happens to have: src/buildsuites/_build/suite_tcl90/out/bin/tclsh90s (`.exe` on windows; Tcl 9.0.5 with Thread included, so `-jobs` works. `_build/suite_tcl86/out/bin/tclsh86ts` is the Tcl 8.6 counterpart - runtests supports an 8.6 runner.) The runtime store `bin/runtime//` is an equally good runner source when populated (e.g `tclsh9.0.5-punk.exe` - same 9.0.5+Thread family); check both before bootstrapping anything. On a fresh checkout (neither present) there are two sanctioned no-tclsh bootstrap paths - which one is the USER's choice, not an agent default: - BUILD: obtain the pinned zig with `bin/punk-getzig.cmd` (windows helper; on other hosts supply zig on PATH or via `PUNK_ZIG`), then from `src/buildsuites/suite_tcl90/` run `zig build bootstrap`. Builds and smokes the full runtime family - a substantial first build; zig caching makes reruns cheap. The suite README covers `-Dsteps` narrowing; `tclsh src/make.tcl buildsuite build suite_tcl90` is equivalent once some tclsh exists. - DOWNLOAD: fetch prebuilt runtimes from the punkbin artifact repo with the VCS-tracked polyglot `bin/punk-runtime.cmd` (`list`, `list -remote`, `use `; the same file runs from bash on unix). Fetched runtimes land in `bin/runtime//`. This pulls prebuilt binaries over the network - confirm with the user before fetching. - NEVER bare `tclsh` from the Bash tool: it commonly resolves to the MSYS/Git-for-Windows tclsh (8.6, msys path semantics) and yields spurious failures. A machine-local NATIVE tclsh90/tclsh87 is also fine as a runner when present. - Prefer a native tclsh over a punk kit exe: kit children boot with kit-stamped punk modules preloaded, which can shadow the src dev modules under test (the runner warns when this applies). ## Command forms Run from the repo root; options come BEFORE any trailing file-tail globs. `$RUNNER` below is the interpreter chosen above. $RUNNER src/tests/runtests.tcl -discover-only 1 -include-paths ?? $RUNNER src/tests/runtests.tcl -report compact -show-passes 0 -include-paths modules/punk/args/testsuites/args dynamic.test $RUNNER src/tests/runtests.tcl -report compact -show-passes 0 -include-paths "modules/punk/args/***" -jobs 16 $RUNNER src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0 Top to bottom: subsecond targeting pre-check (prints the discovered file list and exits); single-file run; subtree run; full suite (~50s at `-jobs 16` on the reference machine - use `-jobs` for anything beyond a handful of files). - `-include-paths` patterns are directory globs relative to `src/tests/`, forward slashes: bare `X` = files directly in X, `X/***` = X and everything below. Trailing bare words are independent file-tail globs. - Single test within a file: add `-tcltestoptions {-match }`. - Failure detail: `-report markdown` shows untruncated errorInfo (ERROR status) and result_was/result_expected (FAILED status). - Add `-strict-exit 1` when the shell exit code must reflect failures. ## Reading results - Trust the final tally and `RUNTESTS_RESULT` line. `status=warn` or a `missing-cleanupTests` reason means INCOMPLETE results, even when observed pass events are listed. - ERROR = the test raised an error; FAILED = result mismatch. Both carry detail fields in compact/markdown/json reports. ## Traps (agent environment) - Agent harnesses commonly export `NO_COLOR=1`. Suites that pin exact SGR sequences (punk/ansi and friends) then fail agent-side only - unset it in the same command (`env -u NO_COLOR $RUNNER ...` in bash) for those runs. - Use the Bash tool, not the PowerShell tool: PowerShell-tool children get an attached console, which can flip punkshell's colour/terminal detection and change output classification. - Never edit `.test` files or the modules under test while a run is in flight - multi-process children source them mid-run. - `shell/`ish punkexe suites exec a punk kit resolved from `env(PUNK_SHELL_TEST_EXE)`, else `/bin/punk902z.exe`. On windows, running punkshells keep `bin/` exes locked (deploys leave them stale) - point `PUNK_SHELL_TEST_EXE` at a fresh copy rather than killing shells. ## Traps (authoring .test files) - tcltest compares the body's RETURN VALUE with `-result`; a body ending in a loop returns the empty string - end such bodies with an explicit `set result`. - A `.test` file is parsed as one script: an unbalanced brace ANYWHERE, including inside a `#` comment, kills the whole file's parse. - Tcl ARE regexp: `\y` is the word boundary; `\b` is a BACKSPACE. - `package require` every extra package explicitly; finish files with `tcltest::cleanupTests` (its absence is the `missing-cleanupTests` warning above). - Agent-added tests take a `#added (agent...)` provenance comment line - format and rules in `src/tests/AGENTS.md`.