Browse Source
tcl-runtests (new): agent quickstart for src/tests/runtests.tcl. Runner-interpreter policy: the zig-built suite tclsh (src/buildsuites/_build/suite_tcl90/out/bin/tclsh90s; tclsh86ts for 8.6 parity) or a bin/runtime store tclsh; a fresh checkout bootstraps via the no-tclsh 'zig build bootstrap' flow (pinned zig only, bin/punk-getzig) or by fetching prebuilt runtimes from punkbin via bin/punk-runtime - which path is the user's choice, not an agent default. Never bare 'tclsh' from an agent shell (commonly the MSYS/Git-for-Windows one - spurious failures); native tclsh preferred over punk kit exes (kit-stamped module shadowing). Plus canonical invocation/targeting forms, result-reading rules (status=warn = incomplete), agent-environment traps (NO_COLOR=1 breaks exact-SGR pins agent-side only; PowerShell-tool attached consoles; no mid-run .test edits; PUNK_SHELL_TEST_EXE for locked/stale kit exes) and .test authoring micro-traps (trailing 'set result', brace-in-comment parse death, \y-not-\b). Pointer-first: src/tests/AGENTS.md remains the authoritative harness contract. tcl-whatis/tcl-nslist: one sentence added to the Command section - the launcher one-liner form `bin/punk91 src script -e '<tcl>'` for RUNNING a snippet rather than introspecting it (behaviour checks, cross-version probes via the documented punk905/punksys kits). Facts verified live before writing: suite tclsh90s = Tcl 9.0.5 + Thread 3.0.7 and drove a focused runtests run (args dynamic.test 2/2 PASS, 5s); runtime-store tclsh9.0.5-punk/plain = 9.0.5 + Thread 3.0.7; -e forms on punk91/punk905/punksys; punk-runtime list output; PUNK_SHELL_TEST_EXE resolution in the punkexe suites. .claude/skills copies are byte-identical re-copies of the .agents canonicals (diff-verified) per the root AGENTS.md skill sync convention. Claude-Session: https://claude.ai/code/session_01Y1diJnhjUxKgEG6EwYAzxj Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.commaster
6 changed files with 242 additions and 8 deletions
@ -0,0 +1,113 @@ |
|||||||
|
--- |
||||||
|
name: tcl-runtests |
||||||
|
description: "Use BEFORE running, adding or editing any tests in this repo, and BEFORE choosing an interpreter to run them with. Canonical runner: the repo's zig-built tclsh under src/buildsuites/_build or a runtime-store tclsh under bin/runtime; a fresh checkout bootstraps either via one-time 'zig build bootstrap' (only the pinned zig required) or by fetching prebuilt runtimes from the punkbin artifact repo via bin/punk-runtime (user's choice). Bare 'tclsh' from an agent shell is often the MSYS/Git-for-Windows one and yields spurious failures. Quickstart invocation and targeting forms, result-reading rules, and the agent-environment traps that waste runs: NO_COLOR=1 breaking exact-SGR pins agent-side only, PowerShell-tool attached consoles, editing .test files mid-run, locked/stale kit exes (PUNK_SHELL_TEST_EXE), warn-status results that look like passes, plus .test authoring micro-traps. Cost: focused runs seconds; full suite ~50s at -jobs 16. Full harness contract: src/tests/AGENTS.md." |
||||||
|
--- |
||||||
|
|
||||||
|
# tcl-runtests |
||||||
|
|
||||||
|
Running the source-tree test suite (`src/tests/runtests.tcl`) as an agent: |
||||||
|
which interpreter to drive it with, the invocation forms worth using, and |
||||||
|
the environment traps that produce spurious failures or wasted runs. This |
||||||
|
is the quickstart + traps layer only - the full harness contract |
||||||
|
(targeting grammar, multi-process/jobs internals, watch mode, provenance |
||||||
|
rules) lives in `src/tests/AGENTS.md` and stays authoritative. |
||||||
|
|
||||||
|
## Runner interpreter |
||||||
|
|
||||||
|
Preferred: the repo's own zig-built runtime - portable across machines, no |
||||||
|
reliance on whatever tclsh a host happens to have: |
||||||
|
|
||||||
|
src/buildsuites/_build/suite_tcl90/out/bin/tclsh90s |
||||||
|
|
||||||
|
(`.exe` on windows; Tcl 9.0.5 with Thread included, so `-jobs` works. |
||||||
|
`_build/suite_tcl86/out/bin/tclsh86ts` is the Tcl 8.6 counterpart - |
||||||
|
runtests supports an 8.6 runner.) The runtime store |
||||||
|
`bin/runtime/<platform>/` is an equally good runner source when populated |
||||||
|
(e.g `tclsh9.0.5-punk.exe` - same 9.0.5+Thread family); check both before |
||||||
|
bootstrapping anything. |
||||||
|
|
||||||
|
On a fresh checkout (neither present) there are two sanctioned no-tclsh |
||||||
|
bootstrap paths - which one is the USER's choice, not an agent default: |
||||||
|
|
||||||
|
- BUILD: obtain the pinned zig with `bin/punk-getzig.cmd` (windows |
||||||
|
helper; on other hosts supply zig on PATH or via `PUNK_ZIG`), then from |
||||||
|
`src/buildsuites/suite_tcl90/` run `zig build bootstrap`. Builds and |
||||||
|
smokes the full runtime family - a substantial first build; zig caching |
||||||
|
makes reruns cheap. The suite README covers `-Dsteps` narrowing; |
||||||
|
`tclsh src/make.tcl buildsuite build suite_tcl90` is equivalent once |
||||||
|
some tclsh exists. |
||||||
|
- DOWNLOAD: fetch prebuilt runtimes from the punkbin artifact repo with |
||||||
|
the VCS-tracked polyglot `bin/punk-runtime.cmd` (`list`, `list -remote`, |
||||||
|
`use <name>`; the same file runs from bash on unix). Fetched runtimes |
||||||
|
land in `bin/runtime/<platform>/`. This pulls prebuilt binaries over |
||||||
|
the network - confirm with the user before fetching. |
||||||
|
|
||||||
|
- NEVER bare `tclsh` from the Bash tool: it commonly resolves to the |
||||||
|
MSYS/Git-for-Windows tclsh (8.6, msys path semantics) and yields |
||||||
|
spurious failures. A machine-local NATIVE tclsh90/tclsh87 is also fine |
||||||
|
as a runner when present. |
||||||
|
- Prefer a native tclsh over a punk kit exe: kit children boot with |
||||||
|
kit-stamped punk modules preloaded, which can shadow the src dev |
||||||
|
modules under test (the runner warns when this applies). |
||||||
|
|
||||||
|
## Command forms |
||||||
|
|
||||||
|
Run from the repo root; options come BEFORE any trailing file-tail globs. |
||||||
|
`$RUNNER` below is the interpreter chosen above. |
||||||
|
|
||||||
|
$RUNNER src/tests/runtests.tcl -discover-only 1 -include-paths <pattern> ?<file.test>? |
||||||
|
$RUNNER src/tests/runtests.tcl -report compact -show-passes 0 -include-paths modules/punk/args/testsuites/args dynamic.test |
||||||
|
$RUNNER src/tests/runtests.tcl -report compact -show-passes 0 -include-paths "modules/punk/args/***" -jobs 16 |
||||||
|
$RUNNER src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0 |
||||||
|
|
||||||
|
Top to bottom: subsecond targeting pre-check (prints the discovered file |
||||||
|
list and exits); single-file run; subtree run; full suite (~50s at |
||||||
|
`-jobs 16` on the reference machine - use `-jobs` for anything beyond a |
||||||
|
handful of files). |
||||||
|
|
||||||
|
- `-include-paths` patterns are directory globs relative to `src/tests/`, |
||||||
|
forward slashes: bare `X` = files directly in X, `X/***` = X and |
||||||
|
everything below. Trailing bare words are independent file-tail globs. |
||||||
|
- Single test within a file: add `-tcltestoptions {-match <testname>}`. |
||||||
|
- Failure detail: `-report markdown` shows untruncated errorInfo (ERROR |
||||||
|
status) and result_was/result_expected (FAILED status). |
||||||
|
- Add `-strict-exit 1` when the shell exit code must reflect failures. |
||||||
|
|
||||||
|
## Reading results |
||||||
|
|
||||||
|
- Trust the final tally and `RUNTESTS_RESULT` line. `status=warn` or a |
||||||
|
`missing-cleanupTests` reason means INCOMPLETE results, even when |
||||||
|
observed pass events are listed. |
||||||
|
- ERROR = the test raised an error; FAILED = result mismatch. Both carry |
||||||
|
detail fields in compact/markdown/json reports. |
||||||
|
|
||||||
|
## Traps (agent environment) |
||||||
|
|
||||||
|
- Agent harnesses commonly export `NO_COLOR=1`. Suites that pin exact SGR |
||||||
|
sequences (punk/ansi and friends) then fail agent-side only - unset it |
||||||
|
in the same command (`env -u NO_COLOR $RUNNER ...` in bash) for those |
||||||
|
runs. |
||||||
|
- Use the Bash tool, not the PowerShell tool: PowerShell-tool children |
||||||
|
get an attached console, which can flip punkshell's colour/terminal |
||||||
|
detection and change output classification. |
||||||
|
- Never edit `.test` files or the modules under test while a run is in |
||||||
|
flight - multi-process children source them mid-run. |
||||||
|
- `shell/`ish punkexe suites exec a punk kit resolved from |
||||||
|
`env(PUNK_SHELL_TEST_EXE)`, else `<projectroot>/bin/punk902z.exe`. On |
||||||
|
windows, running punkshells keep `bin/` exes locked (deploys leave them |
||||||
|
stale) - point `PUNK_SHELL_TEST_EXE` at a fresh copy rather than |
||||||
|
killing shells. |
||||||
|
|
||||||
|
## Traps (authoring .test files) |
||||||
|
|
||||||
|
- tcltest compares the body's RETURN VALUE with `-result`; a body ending |
||||||
|
in a loop returns the empty string - end such bodies with an explicit |
||||||
|
`set result`. |
||||||
|
- A `.test` file is parsed as one script: an unbalanced brace ANYWHERE, |
||||||
|
including inside a `#` comment, kills the whole file's parse. |
||||||
|
- Tcl ARE regexp: `\y` is the word boundary; `\b` is a BACKSPACE. |
||||||
|
- `package require` every extra package explicitly; finish files with |
||||||
|
`tcltest::cleanupTests` (its absence is the `missing-cleanupTests` |
||||||
|
warning above). |
||||||
|
- Agent-added tests take a `#added <date> (agent...)` provenance comment |
||||||
|
line - format and rules in `src/tests/AGENTS.md`. |
||||||
@ -0,0 +1,113 @@ |
|||||||
|
--- |
||||||
|
name: tcl-runtests |
||||||
|
description: "Use BEFORE running, adding or editing any tests in this repo, and BEFORE choosing an interpreter to run them with. Canonical runner: the repo's zig-built tclsh under src/buildsuites/_build or a runtime-store tclsh under bin/runtime; a fresh checkout bootstraps either via one-time 'zig build bootstrap' (only the pinned zig required) or by fetching prebuilt runtimes from the punkbin artifact repo via bin/punk-runtime (user's choice). Bare 'tclsh' from an agent shell is often the MSYS/Git-for-Windows one and yields spurious failures. Quickstart invocation and targeting forms, result-reading rules, and the agent-environment traps that waste runs: NO_COLOR=1 breaking exact-SGR pins agent-side only, PowerShell-tool attached consoles, editing .test files mid-run, locked/stale kit exes (PUNK_SHELL_TEST_EXE), warn-status results that look like passes, plus .test authoring micro-traps. Cost: focused runs seconds; full suite ~50s at -jobs 16. Full harness contract: src/tests/AGENTS.md." |
||||||
|
--- |
||||||
|
|
||||||
|
# tcl-runtests |
||||||
|
|
||||||
|
Running the source-tree test suite (`src/tests/runtests.tcl`) as an agent: |
||||||
|
which interpreter to drive it with, the invocation forms worth using, and |
||||||
|
the environment traps that produce spurious failures or wasted runs. This |
||||||
|
is the quickstart + traps layer only - the full harness contract |
||||||
|
(targeting grammar, multi-process/jobs internals, watch mode, provenance |
||||||
|
rules) lives in `src/tests/AGENTS.md` and stays authoritative. |
||||||
|
|
||||||
|
## Runner interpreter |
||||||
|
|
||||||
|
Preferred: the repo's own zig-built runtime - portable across machines, no |
||||||
|
reliance on whatever tclsh a host happens to have: |
||||||
|
|
||||||
|
src/buildsuites/_build/suite_tcl90/out/bin/tclsh90s |
||||||
|
|
||||||
|
(`.exe` on windows; Tcl 9.0.5 with Thread included, so `-jobs` works. |
||||||
|
`_build/suite_tcl86/out/bin/tclsh86ts` is the Tcl 8.6 counterpart - |
||||||
|
runtests supports an 8.6 runner.) The runtime store |
||||||
|
`bin/runtime/<platform>/` is an equally good runner source when populated |
||||||
|
(e.g `tclsh9.0.5-punk.exe` - same 9.0.5+Thread family); check both before |
||||||
|
bootstrapping anything. |
||||||
|
|
||||||
|
On a fresh checkout (neither present) there are two sanctioned no-tclsh |
||||||
|
bootstrap paths - which one is the USER's choice, not an agent default: |
||||||
|
|
||||||
|
- BUILD: obtain the pinned zig with `bin/punk-getzig.cmd` (windows |
||||||
|
helper; on other hosts supply zig on PATH or via `PUNK_ZIG`), then from |
||||||
|
`src/buildsuites/suite_tcl90/` run `zig build bootstrap`. Builds and |
||||||
|
smokes the full runtime family - a substantial first build; zig caching |
||||||
|
makes reruns cheap. The suite README covers `-Dsteps` narrowing; |
||||||
|
`tclsh src/make.tcl buildsuite build suite_tcl90` is equivalent once |
||||||
|
some tclsh exists. |
||||||
|
- DOWNLOAD: fetch prebuilt runtimes from the punkbin artifact repo with |
||||||
|
the VCS-tracked polyglot `bin/punk-runtime.cmd` (`list`, `list -remote`, |
||||||
|
`use <name>`; the same file runs from bash on unix). Fetched runtimes |
||||||
|
land in `bin/runtime/<platform>/`. This pulls prebuilt binaries over |
||||||
|
the network - confirm with the user before fetching. |
||||||
|
|
||||||
|
- NEVER bare `tclsh` from the Bash tool: it commonly resolves to the |
||||||
|
MSYS/Git-for-Windows tclsh (8.6, msys path semantics) and yields |
||||||
|
spurious failures. A machine-local NATIVE tclsh90/tclsh87 is also fine |
||||||
|
as a runner when present. |
||||||
|
- Prefer a native tclsh over a punk kit exe: kit children boot with |
||||||
|
kit-stamped punk modules preloaded, which can shadow the src dev |
||||||
|
modules under test (the runner warns when this applies). |
||||||
|
|
||||||
|
## Command forms |
||||||
|
|
||||||
|
Run from the repo root; options come BEFORE any trailing file-tail globs. |
||||||
|
`$RUNNER` below is the interpreter chosen above. |
||||||
|
|
||||||
|
$RUNNER src/tests/runtests.tcl -discover-only 1 -include-paths <pattern> ?<file.test>? |
||||||
|
$RUNNER src/tests/runtests.tcl -report compact -show-passes 0 -include-paths modules/punk/args/testsuites/args dynamic.test |
||||||
|
$RUNNER src/tests/runtests.tcl -report compact -show-passes 0 -include-paths "modules/punk/args/***" -jobs 16 |
||||||
|
$RUNNER src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0 |
||||||
|
|
||||||
|
Top to bottom: subsecond targeting pre-check (prints the discovered file |
||||||
|
list and exits); single-file run; subtree run; full suite (~50s at |
||||||
|
`-jobs 16` on the reference machine - use `-jobs` for anything beyond a |
||||||
|
handful of files). |
||||||
|
|
||||||
|
- `-include-paths` patterns are directory globs relative to `src/tests/`, |
||||||
|
forward slashes: bare `X` = files directly in X, `X/***` = X and |
||||||
|
everything below. Trailing bare words are independent file-tail globs. |
||||||
|
- Single test within a file: add `-tcltestoptions {-match <testname>}`. |
||||||
|
- Failure detail: `-report markdown` shows untruncated errorInfo (ERROR |
||||||
|
status) and result_was/result_expected (FAILED status). |
||||||
|
- Add `-strict-exit 1` when the shell exit code must reflect failures. |
||||||
|
|
||||||
|
## Reading results |
||||||
|
|
||||||
|
- Trust the final tally and `RUNTESTS_RESULT` line. `status=warn` or a |
||||||
|
`missing-cleanupTests` reason means INCOMPLETE results, even when |
||||||
|
observed pass events are listed. |
||||||
|
- ERROR = the test raised an error; FAILED = result mismatch. Both carry |
||||||
|
detail fields in compact/markdown/json reports. |
||||||
|
|
||||||
|
## Traps (agent environment) |
||||||
|
|
||||||
|
- Agent harnesses commonly export `NO_COLOR=1`. Suites that pin exact SGR |
||||||
|
sequences (punk/ansi and friends) then fail agent-side only - unset it |
||||||
|
in the same command (`env -u NO_COLOR $RUNNER ...` in bash) for those |
||||||
|
runs. |
||||||
|
- Use the Bash tool, not the PowerShell tool: PowerShell-tool children |
||||||
|
get an attached console, which can flip punkshell's colour/terminal |
||||||
|
detection and change output classification. |
||||||
|
- Never edit `.test` files or the modules under test while a run is in |
||||||
|
flight - multi-process children source them mid-run. |
||||||
|
- `shell/`ish punkexe suites exec a punk kit resolved from |
||||||
|
`env(PUNK_SHELL_TEST_EXE)`, else `<projectroot>/bin/punk902z.exe`. On |
||||||
|
windows, running punkshells keep `bin/` exes locked (deploys leave them |
||||||
|
stale) - point `PUNK_SHELL_TEST_EXE` at a fresh copy rather than |
||||||
|
killing shells. |
||||||
|
|
||||||
|
## Traps (authoring .test files) |
||||||
|
|
||||||
|
- tcltest compares the body's RETURN VALUE with `-result`; a body ending |
||||||
|
in a loop returns the empty string - end such bodies with an explicit |
||||||
|
`set result`. |
||||||
|
- A `.test` file is parsed as one script: an unbalanced brace ANYWHERE, |
||||||
|
including inside a `#` comment, kills the whole file's parse. |
||||||
|
- Tcl ARE regexp: `\y` is the word boundary; `\b` is a BACKSPACE. |
||||||
|
- `package require` every extra package explicitly; finish files with |
||||||
|
`tcltest::cleanupTests` (its absence is the `missing-cleanupTests` |
||||||
|
warning above). |
||||||
|
- Agent-added tests take a `#added <date> (agent...)` provenance comment |
||||||
|
line - format and rules in `src/tests/AGENTS.md`. |
||||||
Loading…
Reference in new issue