User request after console-context failures went unnoticed: compact
per-file status words, failure/warning list entries, the per-file
had-failures/passed/child-failed notices, the markdown
failures/warnings file lists, the tally Passed/Failed segments and a
coloured overall STATUS on the DONE line are now SGR-coloured (green
pass, bold red fail/error, bold yellow warning).
Gating: -colour auto|on|off (default auto) colours only when stdout is
a real windows console, detected via twapi GetConsoleMode on the
STD_OUTPUT handle - the isatty-equivalent; get_console_handle proved
unsuitable (it succeeds for piped children of console shells, verified
both ways). NO_COLOR honoured in auto. Machine output (RUNTESTS_RESULT,
json) is never coloured; piped output stays plain for agents (verified
zero escape bytes piped, codes present with -colour on). Raw literal
SGR is used rather than a+/a so the runner's indicators are independent
of punk::console's process-global colour state (the same 2026-07-19
console runs showed that state shifting under mixed tty-ness). The
previously UNCONDITIONAL raw-red errorInfo block in the file-error
branch now goes through the same gate, so piped error output is cleaner
than before.
Also sharpens the -jobs guidance wording: subtree runs (e.g 28-file
modules/punk/args/**: ~50s sequential vs ~11s at -jobs 16) benefit from
-jobs - only single-file/few-file runs do not.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
User-measured reference-machine numbers 2026-07-19: -jobs 8 ~79-86s,
-jobs 16 ~61s (~5.6x vs ~5m40s sequential), plateau by -jobs 24 (~60s).
Guidance now recommends -jobs 16 and records the plateau anatomy
(slowest single child ~31s runtimecmd pair, serial console tail ~16s
piped / larger under a real console, ~10s runner overhead).
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
multishell.test (10 tests, ~105s solo child - the -jobs floor) split five
ways along shared-fixture boundaries, every test moved verbatim with its
provenance lines: multishell.test keeps wrap + structure/LF-only + exec
smoke (incl WSL); multishell_wrapverify.test (fresh-wrap checkfile) and
multishell_wrapdeterminism.test (byte-identical re-wrap) each re-pay the
wrap fixture; runtimecmd_checkfile.test and runtimecmd_roundtrip.test
guard the committed bin/runtime.cmd and its source round-trip with no
wrap fixture at all. argparsingtest.test (17 tests, ~32s) split into
timeit/discover/contract/first_call (11) + argparsingtest_compare.test
(compare_*, 6). Assessed and left unsplit: dtplite.test (24.6s solo),
scriptexec.test (12s); exec.test excluded as imported core material.
File headers point at the src/tests/modules/AGENTS.md punk/mix bullet
for the sibling map; the runtests longest-first weight table carries
measured post-split weights.
runtests_parity.tcl gains a -names mode: multiset comparison of
'- testname:' lines from '-report markdown -show-passes 1' captures,
file attribution ignored - verbatim moves compare identical, lost/
duplicated/renamed tests are reported.
Verified (native tclsh 9.0.3): repeated -jobs 8 full-suite runs 1m26.3s
/ 1m26.8s (was 2m16s at G-091, ~5m40s sequential - 4.07x vs the
pre-parallel baseline, 36.6% vs the G-091 reference); slowest child
30.7s / 31.3s (bound 45s); NAMES identical (989 occurrences) vs the
pre-split sequential capture; tallies identical in every run (989/973/
15/1, exec-14.3 sole failure, files 87 -> 92); PARITY ok -jobs 8 vs
-jobs 1 and between repeats; repo clean after runs. Notable finding
recorded in the goal file: child-wall contention inflation (~45% with
50s+ dominant children present) collapsed to ~15% once the work
distribution was flattened - splitting reduced both the critical path
and the contention itself.
G-092 flipped to achieved 2026-07-19 and archived (GOALS-archive.md
record, detail file with full verification evidence to goals/archive/).
goals_lint clean. Also noted in the goal file, out of scope: runtests
-include-paths multi-pattern/repeated-flag matching misbehaved during
measurement (single value with four deep patterns matched one; repeated
flags matched zero) - contradicts documented semantics, needs its own
investigation.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
-jobs N (implies -singleproc 0; explicit -singleproc 1 conflicts) runs
test files through a tpool of N concurrent child processes with
per-child capture files and tcltest -tmpdir subdirs, longest-first
static-weight submission, and a -serial-paths serial tail (default: the
console-state suites modules/punk/console/** + modules/opunk/console/**;
piped-child-driven suites proved parallel-safe and each serial file
costs its full child boot strictly after the parallel phase). Cached
results feed the unchanged processing loop in discovery order, so
reports are identical in shape and order to sequential runs. Watch mode
emits file-collected events (completion order, output relayed there);
file-end (tallies) stays discovery-order. The runner now reports jobs
phase times and the ten slowest child WALL times (child wall includes
boot + file-level fixture cost that per-test usec misses).
Two Tcl-core-level behaviours found and worked around in the child
spawn (see the verification record in the archived goal file):
- concurrent BLOCKING execs from worker threads convoy on Windows (all
return at the longest concurrent child's exit; plain sleepers 2-12s
from 8 workers all walled at 12.0s) -> spawn via 'exec ... &' and
poll per-pid tcl::process status; blocking fallback retained for
runtimes without tcl::process (tcl 8.6).
- Tcl_ReapDetachedProcs runs on every exec and purges finished
background children process-wide, racing the poll (nondeterministic
'key <pid> not known in dictionary') -> tcl::process autopurge false
+ explicit per-pid purge after the status read.
Verified on native tclsh 9.0.3: repeated -jobs 8 full-suite runs
2m16.2s / 2m15.6s with PARITY ok vs -jobs 1, between repeats, and vs
the pre-parallel baseline (987-test identity, exec-14.3 sole expected
failure); 2.58x vs the ~350s sequential baseline (2.41x vs one
cooler-machine 5m28.8s sequential run - recorded in the goal file);
repo clean after every run.
G-091 flipped to achieved 2026-07-19 and archived (entry to
GOALS-archive.md, detail file to goals/archive/ with the verification
record; reference sweep updated runtests.tcl comments and the
src/tests/AGENTS.md marker). goals_lint clean.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
runx -teelog {-tag/-syslog/-file/-raw settings}: live tee of the
captured stdout/stderr to shellfilter::log workers (log sources
<tag>stdout/<tag>stderr) while captured/returned data is unchanged.
Implemented as write traces on the capture variables rather than extra
channel transforms: probing showed a tee_to_log float combined with the
float-locked capture junction is starved or loses its worker settings
through add/remove churn (float-placement logic is unaudited - README
'very unripe parts'), and shellthread's parked-worker adoption does not
re-apply settings (writes silently misroute to a previous target). Both
findings deliberately not addressed here - recorded for a proposed
shellfilter/shellthread audit goal. Tag workers stay open for reuse;
conflicting reconfiguration errors explicitly.
runtests.tcl watch mode: env PUNK_TEST_UDPTEE=<port>|<host>:<port>
streams runner lifecycle events (tag 'runtests', tcl-list
RUNTESTS-EVENT messages: run-start/file-start/file-end/run-end with
tallies) and test output (tags teststdout/teststderr) to a local UDP
listener via the same mechanism - live per chunk in singleproc mode
(runx -teelog inside the testinterp), relayed per completed file by the
parent in multiproc mode. Watch mode terminates the per-testinterp log
worker threads after each file; unwatched runs are unchanged (including
the pre-existing teehandle-worker leak, also noted for the audit). The
parent extends auto_path with <projectroot>/lib_tcl<N>/<arch> so
workers load tcludp 1.0.13+ (1.0.12 UDP_ExitProc = G-036 exit-hang
class).
Verified on native tclsh 9.0.3: full-suite result parity (87 files, 989
tests, exec-14.3 sole failure) for watch-off and watch-on singleproc
runs vs the pre-change baseline; focused watch runs in both modes
captured over UDP (events + live/relayed output). Viewer-side caveat
documented: tcludp's receive fileevent under Tcl 9.0.3/Windows
delivered only the first datagram (send side unaffected; .NET UdpClient
received all).
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
Multi-process mode previously execd bare test files with no environment
(children died on their first package require) and misreported the
crashes as missing-cleanupTests warnings with the stderr discarded.
- testsupport/child_test_runner.tcl (new): per-file child bootstrap.
Applies a runtests-generated environment payload (prefer latest, test
tm paths, auto_path, modpod ifneeded defs, tcltest options; per-file
-testdir computed child-side), warms 'clock format' before the
module-path wipe (first script-level clock use loads msgcat from the
runtime default paths, which the test paths do not supply), and
mirrors the testinterp preload: package require shellrun plus one
no-op runx -tcl call (runx execution pulls punk::lib; several suites
depend on preloaded punk::* commands).
- runtests.tcl: generate the payload per run; spawn children via plain
exec with file-captured stdout/stderr and immediate-EOF stdin
(mechanism reusable for future parallel -jobs scheduling); classify
nonzero child exits as file-level failures with the stderr tail
surfaced in all report styles (exit-0-no-summary remains the
missing-cleanupTests warning); warn under kit executables (children
boot with kit-stamped punk packages preloaded, shadowing src dev
modules - prefer a native tclsh).
- scriptlib/developer/runtests_parity.tcl (new): compares two
'-report json' captures on result identity (per-file counts,
failure/skip/warning identities; timings ignored), tolerating
preamble noise and the ANSI SGR reset emitted on the JSON line.
Verified on native tclsh 9.0.3: full suite singleproc vs multiproc
PARITY ok (87 files, 989 tests, 973/15/1 with the documented exec-14.3
baseline as the sole failure); crash and warning classification
exercised. Wall time ~5m50s in both modes.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
Two independent list-flattening bugs made `runtests.tcl foo.test bar.test`
match zero files, forcing agents to run files individually:
- the trailing glob value is -multiple 1, so its parsed value is a list;
plain lappend nested it as a single space-containing element
- punk::path::treefilenames declares tailglobs as a -multiple 1 positional
(one glob per argument), but the whole file_globs list was passed as one
argument, re-collapsing it (this also broke multiple globs supplied via
-tcltestoptions {-file {...}})
Single names always worked because a one-element Tcl list flattens to
itself, which masked both bugs. Verified: two file tails now discover and
run both suites (files=2, 40/40 pass).
Also note multi-file-tail support in src/tests/AGENTS.md Work Guidance.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
New modules/punk/repl/testsuites/repl/consolebackends.test drives an
interactive repl through TestConsole (chan pipe pair), SshConsole (socket
pair with a scripted remote terminal answering CSI 6n - size 100x30
resolved over the socket) and TkConsole (wired text widget; self-gates on
Tk availability in the child) to a clean exit: prompt-by-prompt
conversation, results, diverted code-interp stdout/stderr. Each case execs
testsupport/repl_console_driver.tcl in a child tclsh - a repl cannot run
inside the shared runtests testinterp because the codethread's quit/exit
callbacks thread::send to the thread's MAIN interp (documented in
src/tests/AGENTS.md).
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
- src/tests/AGENTS.md: agent-added tests carry a one-line provenance comment `#added <YYYY-MM-DD> (agent[, G-<id>])[ - <short note>]` - immutable facts only (never "G-NNN will flip this" expectations, which live in _GAP_ test names and goal files); user-added tests optional; no bulk retrofit of existing suites. Rationale: provenance must survive where VCS history does not travel - packaged test modules (G-029), git+fossil dual history, file regeneration/moves
- GOALS.md maintenance rules: marking a goal superseded/abandoned includes a test sweep - grep G-<old id> (at minimum src/tests) plus the tests named in the goal's detail-file Acceptance, and record each test's disposition (transfers to the superseding goal / stands down to plain characterization / lapses) in the appropriate detail file; the sweep never deletes, skips, or weakens tests on its own
- the provenance lines are what keep the supersession sweep's grep reliable over time; closes the dangling-GAP-pin risk analyzed for superseded goals
- no project version bump: governance docs only
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
- new src/tests/testsupport/wslprobe.tcl (::punktest::wsl): memoized capability probe for the wsllinux constraint - default distro launches and unames as Linux, tools present (bash/mktemp/wslpath + sha1 tool), native tempdir file round trip; any failure -> skip with reason, so wsl.exe-present-but-unusable installs cannot false-positive. Staging helpers (staging_create/copy_in/run_in/cleanup) enforce the design: execution on the distro NATIVE filesystem only, shared /mnt used for one-way copy-in/out, the windows checkout never operated on from inside WSL (DrvFs slowness + cross-boundary stat differences that make git re-hash its index and fossil see phantom changes). Probe uses wsl -e invocations only (wsl --status/-l emit UTF-16)
- multishell.test: scriptwrap_multishell_exec_wsl_sh - the polyglot sh payload produces its marker under the distro bash from native staging (the direct isunix twin still skips on windows)
- new shell/testsuites/binscripts/runtimebash_wsl.test (8 tests): runtime.bash first-ever execution on real unix - multi-candidate run errors listing candidates, use/list/run round trip with argument passing, PUNK_ACTIVE_RUNTIME override, unknown/unselectable use errors, stale-active guidance, single-candidate fallback, offline list -remote (crafted cached sha1sums.txt + PUNKBIN_URL forced fetch failure), and the checkout-untouched guard (git status --porcelain identical before/after per acceptance)
- green on tcl 9.0.3 + 8.7 against Ubuntu-24.04/WSL2; suites skip cleanly when the probe fails; enablement/limitations in src/tests/AGENTS.md (known limitation: a present-but-HANGING wsl can stall the probe)
- GOALS.md G-059 flipped to achieved 2026-07-11; detail file records the outcome incl the UTF-16 discovery and the 512-byte label regression this work surfaced (caught by the guardrail suite as designed)
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
- Repeated -include-paths flags previously overrode each other silently (standard
non-multiple punk::args opt semantics), quietly narrowing the test selection
while reporting green - a recurring misuse driven by Unix-CLI accumulate-flag
muscle memory (gcc -I / curl -H style).
- The option is now -multiple 1: repeated flags accumulate, the single
space-separated quoted list form is unchanged, and the unreceived default {**}
passes through (the consumer flattens occurrences with concat). Help text
documents both forms as equivalent.
- src/tests/AGENTS.md work-guidance updated. Known remaining sibling issue:
multiple trailing file-tail globs still silently match 0 files.
- Verified: repeated-flag and single-list invocations select identical file sets
(20 files / 189 tests) on Tcl 9.0.3, repeated-flag accumulation on 8.7, and a
no-flag default run with a file-tail glob unchanged. The general strict-arity
mechanism is proposed as G-053.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
The testinterp replaces auto_path wholesale (project src/lib dirs, root
lib_tcl<N>/<arch>, [info library] + parent). Under a kit executable that
parent is e.g //zipfs:/app, and tclPkgUnknown scans only an entry plus its
immediate children - so kit-bundled packages under //zipfs:/app/lib_tcl<N>/
<pkg> (e.g tcllib's tcl::chan::fifo2, required by the shellrun harness at
testinterp setup) were unreachable. Unnoticed in punkshell because the
project's own root lib_tcl9/win32-x86_64 carries tcllib2.0; projects
without that payload (e.g tomlish) failed every file at
'package require shellrun' when run under a punk kit exe.
Now the kit's internal lib/lib_tcl<N> dirs are appended when present,
mirroring the internal-path classification in src/vfs/_config/punk_main.tcl:
zipfs app mount, tclkit ::tcl::kitpath exe dir-shadow, cookfs //cookit:/
volume (default mount name only, as per punk_main.tcl). No-op under native
tclsh. Verified: punk902z + tclsh902z suites pass; classification probed OK
under tclkit 8.6.17, punk9cook (cookit), punk91 (zipfs 9.1), native Tcl903;
tomlish suite under punk902z/punk91 goes 0 -> 149/149 with the same edit.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
makeFile/makeDirectory output previously landed in the working directory -
aborted core exec.test runs littered src/tests with untracked helper files
(cat, echo, gorp.file ...). Unless -tcltestoptions supplies -tmpdir, a
fresh directory from 'file tempdir' (TMPDIR/TEMP/TMP + pid fallback for
tcl 8.6) is configured and deleted before exit; a hard-aborted run now
leaves litter in the OS temp area rather than the source tree.
Takes effect in single-process mode; multi-process mode still does not
forward tcltest options to child processes (pre-existing todo).
Contract documented in src/tests/AGENTS.md.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
- fixed "@opts -any 1" (arbitrary/adhoc option passthrough): an option not in
the definition crashed get_dict with "can't read argname" at the name-mapping
stage - adhoc opts now fall back to their raw supplied name, under which
argstate/arg_checks were already populated at scan time (real consumers:
vendored tomlish definitions declare -any 1)
- commented out a debug puts stderr (">>>_get_dict_can_assign_value NOT
alloc_ok...") that fired on every failed clause type assignment (visible on
any multiform parse miss; its happy-path twin was already commented)
- new forms.test (8 tests): @form spec structure (form_names, shared
{a b}-prologue blocks), explicit -form by name/index with cross-form
rejection, multiform synopsis rendering, and GAP pins for G-041 - default
parse is effectively form-0-only, -form rejects the documented list-of-forms
usage, and @form -synopsis overrides are stored in the spec but ignored by
synopsis rendering
- new validation.test (11 tests): -type validation matrix, -range/-minsize/
-maxsize, -nocase choice normalization, option -multiple accumulation,
required-option enforcement, unknown-option rejection vs -any adhoc
passthrough (the bugfix's regression test), -- end-of-options marker,
@values -max enforcement, received-key clause ordinals, solos list, and
default-vs-received distinction
- src/tests/AGENTS.md: note that tcltest compares the -body return value -
bodies ending in a loop must end with an explicit 'set result'
- suites: punk/args 93 pass + 1 skip (was 70+1), punk/ns 18 pass; full
source-tree run 569/581 with only the known exec-14.3 baseline failure
- project 0.4.5 (CHANGELOG entry)
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
runtests_failure_summaries now includes errorinfo, result_was, and result_expected when present in the parsed testdict, so the json report (which iterates keys) and downstream consumers get the full failure context. runtests_print_failure_details prints an errorInfo fenced block for ERROR-status failures and result_was/result_expected fenced blocks for FAILED-status mismatches, with full multi-line text. The compact report appends one-line truncated message=, actual=, and expected= fields (120/80/80 chars) so compact mode stays compact while still being actionable for agents. src/tests/AGENTS.md Local Contracts and Work Guidance updated to document the new failure-detail surfacing.
Julian Noble