punkshell/.claude/skills/tcl-runtests/SKILL.md

name

description

tcl-runtests

Use BEFORE running, adding or editing any tests in this repo, and BEFORE choosing an interpreter to run them with. Canonical runner: the repo's zig-built tclsh under src/buildsuites/_build or a runtime-store tclsh under bin/runtime; a fresh checkout bootstraps either via one-time 'zig build bootstrap' (only the pinned zig required) or by fetching prebuilt runtimes from the punkbin artifact repo via bin/punk-runtime (user's choice). Bare 'tclsh' from an agent shell is often the MSYS/Git-for-Windows one and yields spurious failures. Quickstart invocation and targeting forms, result-reading rules, and the agent-environment traps that waste runs: NO_COLOR=1 breaking exact-SGR pins agent-side only, PowerShell-tool attached consoles, editing .test files mid-run, locked/stale kit exes (PUNK_SHELL_TEST_EXE), warn-status results that look like passes, plus .test authoring micro-traps. Cost: focused runs seconds; full suite ~50s at -jobs 16. Full harness contract: src/tests/AGENTS.md.

tcl-runtests

Running the source-tree test suite (src/tests/runtests.tcl) as an agent: which interpreter to drive it with, the invocation forms worth using, and the environment traps that produce spurious failures or wasted runs. This is the quickstart + traps layer only - the full harness contract (targeting grammar, multi-process/jobs internals, watch mode, provenance rules) lives in src/tests/AGENTS.md and stays authoritative.

Runner interpreter

Preferred: the repo's own zig-built runtime - portable across machines, no reliance on whatever tclsh a host happens to have:

src/buildsuites/_build/suite_tcl90/out/bin/tclsh90s

(.exe on windows; Tcl 9.0.5 with Thread included, so -jobs works. _build/suite_tcl86/out/bin/tclsh86ts is the Tcl 8.6 counterpart - runtests supports an 8.6 runner.) The runtime store bin/runtime/<platform>/ is an equally good runner source when populated (e.g tclsh9.0.5-punk.exe - same 9.0.5+Thread family); check both before bootstrapping anything.

On a fresh checkout (neither present) there are two sanctioned no-tclsh bootstrap paths - which one is the USER's choice, not an agent default:

• BUILD: obtain the pinned zig with bin/punk-getzig.cmd (windows helper; on other hosts supply zig on PATH or via PUNK_ZIG), then from src/buildsuites/suite_tcl90/ run zig build bootstrap. Builds and smokes the full runtime family - a substantial first build; zig caching makes reruns cheap. The suite README covers -Dsteps narrowing; tclsh src/make.tcl buildsuite build suite_tcl90 is equivalent once some tclsh exists.

• DOWNLOAD: fetch prebuilt runtimes from the punkbin artifact repo with the VCS-tracked polyglot bin/punk-runtime.cmd (list, list -remote, use <name>; the same file runs from bash on unix). Fetched runtimes land in bin/runtime/<platform>/. This pulls prebuilt binaries over the network - confirm with the user before fetching.

• NEVER bare tclsh from the Bash tool: it commonly resolves to the MSYS/Git-for-Windows tclsh (8.6, msys path semantics) and yields spurious failures. A machine-local NATIVE tclsh90/tclsh87 is also fine as a runner when present.

• Prefer a native tclsh over a punk kit exe: kit children boot with kit-stamped punk modules preloaded, which can shadow the src dev modules under test (the runner warns when this applies).

Command forms

Run from the repo root; options come BEFORE any trailing file-tail globs. $RUNNER below is the interpreter chosen above.

$RUNNER src/tests/runtests.tcl -discover-only 1 -include-paths <pattern> ?<file.test>?
$RUNNER src/tests/runtests.tcl -report compact -show-passes 0 -include-paths modules/punk/args/testsuites/args dynamic.test
$RUNNER src/tests/runtests.tcl -report compact -show-passes 0 -include-paths "modules/punk/args/***" -jobs 16
$RUNNER src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0

Top to bottom: subsecond targeting pre-check (prints the discovered file list and exits); single-file run; subtree run; full suite (~50s at -jobs 16 on the reference machine - use -jobs for anything beyond a handful of files).

• -include-paths patterns are directory globs relative to src/tests/, forward slashes: bare X = files directly in X, X/*** = X and everything below. Trailing bare words are independent file-tail globs.
• Single test within a file: add -tcltestoptions {-match <testname>}.
• Failure detail: -report markdown shows untruncated errorInfo (ERROR status) and result_was/result_expected (FAILED status).
• Add -strict-exit 1 when the shell exit code must reflect failures.

Reading results

• Trust the final tally and RUNTESTS_RESULT line. status=warn or a missing-cleanupTests reason means INCOMPLETE results, even when observed pass events are listed.
• ERROR = the test raised an error; FAILED = result mismatch. Both carry detail fields in compact/markdown/json reports.

Traps (agent environment)

• Agent harnesses commonly export NO_COLOR=1. Suites that pin exact SGR sequences (punk/ansi and friends) then fail agent-side only - unset it in the same command (env -u NO_COLOR $RUNNER ... in bash) for those runs.
• Use the Bash tool, not the PowerShell tool: PowerShell-tool children get an attached console, which can flip punkshell's colour/terminal detection and change output classification.
• Never edit .test files or the modules under test while a run is in flight - multi-process children source them mid-run.
• shell/ish punkexe suites exec a punk kit resolved from env(PUNK_SHELL_TEST_EXE), else <projectroot>/bin/punk902z.exe. On windows, running punkshells keep bin/ exes locked (deploys leave them stale) - point PUNK_SHELL_TEST_EXE at a fresh copy rather than killing shells.

Traps (authoring .test files)

• tcltest compares the body's RETURN VALUE with -result; a body ending in a loop returns the empty string - end such bodies with an explicit set result.
• A .test file is parsed as one script: an unbalanced brace ANYWHERE, including inside a # comment, kills the whole file's parse.
• Tcl ARE regexp: \y is the word boundary; \b is a BACKSPACE.
• package require every extra package explicitly; finish files with tcltest::cleanupTests (its absence is the missing-cleanupTests warning above).
• Agent-added tests take a #added <date> (agent...) provenance comment line - format and rules in src/tests/AGENTS.md.