22 KiB

Raw Blame History

src/tests — Source-Tree Test Harness

Purpose

Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exercise source files under src/ before installation or packaging, including module tests, shell behavior tests, and selected Tcl core compatibility tests.

Ownership

Owned by the project maintainer; agents may update tests here when changing source behavior.
Do not delete, skip, or weaken existing tests without explicit user direction.
Keep build-output and installed-package tests outside this tree unless a local child AGENTS.md says otherwise.

Local Contracts

runtests.tcl is the primary source-tree test entry point; sublevel all.tcl files are a legacy pattern and are not required.
Tests use tcltest unless a child AGENTS.md documents a different local harness.
runtests.tcl excludes AGENTS.md and *.tcl helper files when discovering .test files.
Discovery path targeting (G-093, reworked 2026-07-20): -include-paths/-exclude-paths are passed through to punk::path::treefilenames unchanged and carry its directory-oriented pathglob semantics — identical meaning at every layer. Patterns match the DIRECTORY containing a test file (relative to src/tests/, forward slashes); the trailing file-tail globs (default *.test) select file names — independent axes. Separable segment forms: X (files directly in X only), X/* (exactly one below), X/** (strictly below X — NOT files directly in X), X/*** (X and everything below — the common subtree form; *** = zero or more segments, punk::path 0.4.0). Exclusion uses the same syntax: **|***-tail patterns prune the matched subtree, other matches exclude just that directory's files, and exclusion wins over inclusion for the same file. Single-file targeting = exact directory pattern + file tail. A received include pattern contributing no files earns a stderr advisory (the X/**-for-subtree foot-gun; results unaffected). -serial-paths uses the same directory-of-file reading (default modules/punk/console/*** modules/opunk/console/***). Pinned by runner/testsuites/discovery/pathdiscovery.test; the pathglob lattice itself by modules/punk/path/testsuites/tests/path.test.
-discover-only 1 prints the discovered test-file list and exits before any run machinery starts — subsecond, via the G-093 two-stage boot (only punk::args + punk::path load before the exit — the bootsupport punk::path must be >= 0.4.0 for ***, enforced by a versioned require in testsupport/discovery.tcl; punk/Thread/shellrun/punk::tcltestrun and the testinterp/child path assembly load after). Always emits one machine-readable RUNTESTS_FILES <path>... line (a well-formed tcl list with the marker as first element, emitted last), plus one path per line first in non-json report modes. Use it to check targeting before an expensive run.
runtests.tcl defaults tcltest -tmpdir (the makeFile/makeDirectory location) to a fresh directory under the system temp area, deleted before exit, so test helper files never land in the source tree (aborted core exec.test runs previously littered src/tests). A -tmpdir supplied via -tcltestoptions overrides this. Applies to both modes: multi-process mode forwards -tmpdir (and all other tcltest options) to child processes via the generated environment payload.
The testinterp auto_path (replaced wholesale) also includes the running kit's internal lib/lib_tcl<N> trees (zipfs app mount, tclkit ::tcl::kitpath, cookfs //cookit:/ — mirroring src/vfs/_config/punk_main.tcl), so kit-bundled packages (e.g. tcllib's tcl::chan::fifo2, needed by the shellrun harness) resolve when the project tree doesn't supply them. tclPkgUnknown scans only an entry plus immediate children, so the kit lib dirs must be explicit entries. No-op under a native tclsh.
The single-process testinterp runs package prefer latest so alpha-versioned dev modules (999999.0a1.0) are preferred over stable bootsupport/vendored copies on unversioned package require.
The testinterp module path includes src/vendormodules (and src/vendormodules_tcl<major> when present) so vendored dependencies such as voo resolve in tests.
Multi-process mode (-tcltestoptions {-singleproc 0}, completed 2026-07-18): each test file runs in a child process of the same executable via testsupport/child_test_runner.tcl, driven by a per-run environment payload runtests.tcl generates (package prefer latest, test tm paths, auto_path, modpod ifneeded definitions, tcltest options; per-file -testdir computed child-side). Children are LEAN (2026-07-19 cleanup): they load only tcltest plus whatever each test file declares - the previous shellrun-preload + runx-warmup mirror of the testinterp was removed after 17 files gained their missing explicit package require lines (the contract above always required them; the parity tool pinpointed the violators, and an empirical lean run caught the final straggler - examples.test needing textblock because punk::args only soft-requires it). The single-process testinterp still preloads shellrun, but as the RUNNER's capture mechanism (runx -tcl sourcing) - tests must not rely on it. Known module-web finding from the sweep: punk::ansi and punk::args call punk::lib internally without requiring it, and a hard back-require is blocked by punk::lib's own hard require of punk::args (cycle) - dependencies are therefore declared at test level (grepstr/untabify/examples carry comments); a considered module-level resolution remains open. Watch-mode interaction notes live in child_test_runner.tcl (none today - the parent relays; a future live per-child tee must be conditional on the env var so unwatched children stay lean). The bootstrap warms clock format before the module-path wipe: first script-level clock use loads msgcat from the runtime's default module paths, which the test module paths do not supply (the single-process testinterp is shielded only by the runtests parent's earlier clock use - a latent gap for any test explicitly requiring msgcat in either mode).
Multi-process failure classification: a nonzero child exit is reported as a file-level failure (compact/markdown/json carry an ERROR entry with errorcode CHILDPROCESS exit <n> and the child's stderr tail); exit-0 with no tcltest summary line remains the missing-cleanupTests warning. Prefer a native tclsh for multi-process runs: children of a kit executable boot via the kit's script dispatch with kit-stamped punk packages preloaded, which can shadow the src dev modules under test (the runner prints a warning).
Parallel jobs (G-091, achieved 2026-07-19 - see goals/archive/G-091-runtests-parallel-jobs.md): -jobs N (N > 1) implies multi-process mode and runs test files through a worker pool of N concurrent child processes (tpool; per-child capture files and per-child tcltest -tmpdir subdirs so concurrent makeFile helpers cannot collide; longest-first submission using static weights from the 2026-07-18/19 timing measurements - the full-suite floor is the slowest single child; the former ~2m multishell.test floor was removed by the G-092 split of that suite into four files, leaving all children under ~35s). Files matching -serial-paths run sequentially after the parallel batch - default {modules/punk/console/*** modules/opunk/console/***} (directory-of-file matching as of G-093; the console-state suites, which emit to and query the shared terminal; suites driven through piped-stdio child processes - shell/, modules/punk/repl/ - stay parallel by default and can be added to -serial-paths if they prove flaky under a real console; each serial file costs its full ~3-4s child boot strictly after the parallel phase, so the list is deliberately minimal). Reports are unchanged in shape and emitted in discovery order. An explicit -tcltestoptions {-singleproc 1} conflicts with -jobs > 1 (error). Watch-event semantics under -jobs: no file-start events; file-collected (with exitcode, output relayed at that moment) fires in completion order and the authoritative file-end (tallies) in discovery order. Per-test microsecond timings are noisier under CPU oversubscription - prefer sequential runs when -slowest matters. Child spawn under -jobs is background-exec plus tcl::process status polling: concurrent BLOCKING execs from worker threads convoy on Windows - they all return together at the longest-lived concurrent child's exit (proven 2026-07-19 with plain sleepers of 2-12s all walling at 12s; this cost early -jobs runs ~50s of false critical path). Runtimes without tcl::process (tcl 8.6) fall back to blocking exec, which re-convoys under -jobs. The runner prints phase times (parallel/serial-tail/processing) and the ten slowest child WALL times in non-json modes - child wall includes boot and file-level fixture cost that per-test usec timings miss.
UDP watch mode (added 2026-07-18): env(PUNK_TEST_UDPTEE) = <port> or <host>:<port> (e.g 41197) makes runtests emit the run over UDP via the shellfilter::log mechanism (shellthread worker threads -> tcludp, one datagram per line, cooked format whose source column carries the tag). Tag runtests carries lifecycle events; each event message is a well-formed tcl list RUNTESTS-EVENT <eventname> ?key value ...? (run-start, file-start, file-end with per-file tallies, run-end). Tags teststdout/teststderr carry test file output: streamed live per output chunk in single-process mode (shellrun::runx -teelog 0.2.0 - write traces on the capture variables, so captured results and reports are unchanged) and relayed per completed file by the parent in multi-process mode. No listener is required (sends to an unbound local port are discarded). The parent extends its auto_path with <projectroot>/lib_tcl<N>/<arch> so log workers resolve the project's tcludp 1.0.13+ (1.0.12's UDP_ExitProc is the G-036 exit-hang class). Viewer-side caveat: tcludp's RECEIVE fileevent under Tcl 9.0.3/Windows delivered only the first datagram in testing (send side unaffected; a .NET UdpClient received everything) - a viewer should use timer-driven non-blocking reads or a non-tcludp receiver until that is resolved.
In watch mode the singleproc loop closes and terminates the shellfilter::log worker threads opened inside each per-file testinterp before deleting it (the -teelog tag workers plus shellrun's punkshout/punksherr teehandle workers). The terminate handshakes cost ~0.5-0.7s per file, so unwatched runs skip this and keep the pre-existing small leak of the two teehandle worker threads per file (noted for the shellfilter/shellthread audit; the process exits at run end regardless).
Tests should run against source modules and libraries from src/, not installed packages or root-level build outputs.
Test files must package require any extra packages explicitly.
Tcltest files must finish with tcltest::cleanupTests; missing cleanup produces a missing-cleanupTests runner warning and only untrusted observed testcase events.
tcltest compares the -body RETURN VALUE against -result. The suite convention of accumulating into $result via lappend works because lappend returns the list — but a body whose last command is a loop (foreach, while) returns the empty string; end such bodies with an explicit set result.
Agent-oriented runner output should use -report compact -show-passes 0 for focused checks unless detailed Markdown pass lists are needed.
-report json emits a machine-readable final summary, but package-load warnings may still precede it on stdout/stderr, and the punk ANSI output stack may emit an SGR reset immediately before the JSON on the same line (scriptlib/developer/runtests_parity.tcl tolerates both).
Human-facing pass/fail/warning indicators in markdown/compact reports are ANSI-coloured only when stdout is a real windows console (-colour auto default: twapi GetConsoleMode on the STD_OUTPUT handle as the isatty-equivalent - get_console_handle is unsuitable, it succeeds for piped children of console shells; NO_COLOR honoured; -colour on|off overrides). Machine-facing output (the RUNTESTS_RESULT line, json reports) is never coloured, and piped/redirected output stays plain for agents. The runner uses raw literal SGR rather than a+/a so its indicators are independent of punk::console's process-global colour state.
ERROR-status failures now surface errorInfo (full Tcl error message/stack trace) in markdown, compact, and json reports. Compact mode truncates to a single line.
FAILED-status failures (result mismatch, not error) now surface result_was (actual) and result_expected (expected) in markdown, compact, and json reports. Compact mode truncates each to a single line.
testsupport/ holds helper .tcl files sourced or exec'd by .test files or the runner (not discovered as suites). testsupport/child_test_runner.tcl is the multi-process-mode bootstrap runtests.tcl execs per test file (see the multi-process bullets above). testsupport/discovery.tcl is the discovery walk + path-targeting filter sourced by runtests.tcl (parent side, before children exist) and by the runner-targeting regression suite (see the discovery bullets above). testsupport/repl_console_driver.tcl (G-001) is exec'd in a child tclsh by modules/punk/repl/testsuites/repl/consolebackends.test: an interactive repl cannot run inside the shared testinterp (the codethread's quit/exit callbacks thread::send to the thread's MAIN interp, bypassing a child testinterp), so repl-through-console verification always drives a child process. testsupport/wslprobe.tcl (G-059) provides ::punktest::wsl: a memoized capability probe yielding the wsllinux constraint (default distro launches, answers uname/tool probes, and round-trips a file through a native tempdir - NOT mere wsl.exe existence) plus native-filesystem staging helpers (staging_create/staging_copy_in/run_in/staging_cleanup). WSL-gated tests must execute from a staging dir on the distro's native filesystem with the shared /mnt path used only for one-way copy-in/out - never operate on the Windows checkout from inside WSL (DrvFs is slow and cross-boundary stat differences make git re-hash its index and fossil see phantom changes). Probe invocations use wsl -e <cmd> only (wsl --status/-l emit UTF-16). Known limitation: a broken-but-present WSL that hangs (rather than errors) on wsl -e can stall the probe.
Test provenance comments: an agent adding a test places one comment line directly above it — or one line above a contiguous group added in the same piece of work — of the form #added <YYYY-MM-DD> (agent[, G-<id>])[ - <short note>]. Date and the agent marker are mandatory; the goal reference is mandatory when a goal motivated the test; the note is optional (omit it when the test name/description already says why the test exists). Provenance lines record immutable facts only — never expectations such as "G-NNN will flip this" (that state lives in _GAP_-style test names and the owning goal's files). User-added tests may use the same form with (user) but are not required to. Do not retrofit existing suites in bulk; add provenance opportunistically when editing a file. Rationale: provenance must survive where VCS history does not travel — packaged test modules (G-029), the git+fossil dual history, and file regeneration/moves.

Work Guidance

Select an appropriate interpreter, such as tclsh, tclsh90, or a built Punk executable such as punk902z.
Native Tcl 8.6 tclsh is a supported runner interpreter (2026-07-21): runtests.tcl loads punk::lib's validated forward-compat command family (lpop etc - builtin parity pinned by modules/punk/lib/testsuites/lib/compat.test) at both boot levels (toplevel and testinterp) when the builtins are absent. The guards are no-ops on Tcl 8.7+/9, preserving the G-093 fast two-stage -discover-only boot.
Kit-hosted runner hardening (2026-07-21): the runner's own package requires carry minimum-version floors (punk::args in runtests.tcl, punk::path in testsupport/discovery.tcl, punk::lib in the 8.6 guards). Rationale: a punk-kit [interpreter] boot pre-registers its kit-stamped package versions, and an unversioned require satisfiable by such a registration never fires package unknown - the freshly-added bootsupport tm path is never even scanned, so a fossilized kit copy silently wins and then skews against modern modules pulled in by versioned requires (the historic punk86 failure: kit punk::args 0.1.0 loaded, then punk::path 0.4.0- died on invalid command name "punk::args::define"). Floored requires reject the fossil and resolve the bootsupport copy. The punk86 kit itself is mapped in src/runtime/mapvfs.config (8.6.17 dyn tclkit + punk8win.vfs, remapped 2026-07-21). Suites that spawn children of [info nameofexecutable] additionally need kit content to be current: kit children boot with kit-stamped modules preloaded from _vfscommon.vfs, so run make.tcl vfscommonupdate + make.tcl vfs after module changes such suites depend on (punkcheck's concurrency.test floors its child-side punkcheck require so a stale kit fails loud in the child log instead of mysteriously). Native tclsh remains the preferred runner interpreter.
Run the full source-tree suite with <tcl_interpreter> src/tests/runtests.tcl.
Use -include-paths with directory glob patterns relative to src/tests/, forward slashes — modules/punk/args/*** runs that whole subtree (X/*** deliberately excludes files directly in X; bare X is files-directly-in-X only). The flag accepts a space-separated list of patterns and may also be repeated, with all occurrences accumulating (-multiple 1 as of 2026-07-10; previously the last flag silently won). Single-file targeting: -include-paths modules/punk/args/testsuites/args dynamic.test.
Use -exclude-paths (same syntax and accumulation) to remove directories from discovery — **|*** tails prune the matched subtree, other matches drop only that directory's files, and exclusion wins for the same file. It gives broad runs a middle ground between full and targeted: e.g -exclude-paths shell/*** when no built kit is present, or skipping a heavy suite family unrelated to the change.
Verify targeting cheaply before an expensive run with -discover-only 1 (prints the discovered file list and exits, subsecond).
Use -tcltestoptions {-match <test_name>} for focused single-test runs.
For agent-efficient focused runs, prefer <tcl_interpreter> src/tests/runtests.tcl -report compact -show-passes 0 -include-paths <relative/path/***> <file-tail.test>.
Multiple trailing file tails are supported and match independently: runtests.tcl foo.test bar.test runs both in one invocation (fixed 2026-07-17; previously multiple names collapsed into one glob matching zero files).
Treat RUNTESTS_RESULT status=warn and compact warning reasons such as missing-cleanupTests as incomplete test results, even if observed pass events are listed.
Add -slowest <n> when timing outliers are relevant.
Multi-process runs use -tcltestoptions {-singleproc 0} with otherwise identical flags. Check result parity between modes by capturing -report json stdout from each and comparing with tclsh scriptlib/developer/runtests_parity.tcl <a.json> <b.json> (ignores timings; exit 0 on parity).
To watch runs live (including runs launched by agents or other processes, which inherit the environment): set PUNK_TEST_UDPTEE=41197 and point a UDP listener/viewer at that port. No runner flags needed; results are unaffected.
For broad runs use -jobs - e.g <tcl_interpreter> src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0. Reference-machine full-suite scaling (2026-07-19, post G-092 split + lean-children cleanup, result parity verified at each level): singleproc sequential ~5m40s, multiproc sequential ~4m30s, -jobs 16 ~50s (~6.8x), with diminishing returns past 16 - the wall decomposes into the slowest single child (~30s, the runtimecmd pair - single tests, not further splittable), the serial console tail (a few seconds piped now children boot lean; larger under a real console where those suites actually run), and runner overhead (~10s). -jobs 16 is the reference-machine sweet spot. Use -jobs for ANY run spanning more than a handful of test files - subtree runs count (e.g -include-paths modules/punk/args/*** is 28 files: ~50s sequential vs ~11s at -jobs 16); only single-file or few-file runs gain nothing from it.
Add -strict-exit 1 when a nonzero shell exit code is needed for failures or parser warnings.
Capture enough stderr or failure context to identify the failing command or assertion.
For ERROR-status failures, the markdown report's errorInfo block and compact message= field carry the full Tcl error message; use -report markdown for untruncated context.
For FAILED-status failures (result mismatch), the markdown report's result_was/result_expected blocks and compact actual=/expected= fields show the actual-vs-expected comparison; use -report markdown for full multi-line values.

Verification

<tcl_interpreter> src/tests/runtests.tcl passes when broad source-tree test coverage is relevant.
Focused checks use <tcl_interpreter> src/tests/runtests.tcl -report compact -show-passes 0 -include-paths <relative/path/***> and optional file-tail globs.
Changes to runtests.tcl or testsupport/child_test_runner.tcl are verified by running the full suite in both modes with -report json and confirming scriptlib/developer/runtests_parity.tcl reports PARITY: ok.
Documentation-only changes are verified by reviewing the affected DOX chain and diff.

Child DOX Index

modules/ — Source module test suites mirroring editable module namespaces (see modules/AGENTS.md)
core/ — Selected Tcl core compatibility tests for standard Tcl and built Punk executables (see core/AGENTS.md)
shell/ — Shell behavior tests and command-line interaction suites (see shell/AGENTS.md)
runner/ — Suites for the runtests.tcl harness itself: discovery/path-targeting (see runner/AGENTS.md)

22 KiB Raw Blame History