From 5002371864cfadbd863f75668fa20d5d9a6315f5 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Sun, 19 Jul 2026 06:25:24 +1000 Subject: [PATCH] Split long-running test files to lower the -jobs parallel floor (G-092 achieved) multishell.test (10 tests, ~105s solo child - the -jobs floor) split five ways along shared-fixture boundaries, every test moved verbatim with its provenance lines: multishell.test keeps wrap + structure/LF-only + exec smoke (incl WSL); multishell_wrapverify.test (fresh-wrap checkfile) and multishell_wrapdeterminism.test (byte-identical re-wrap) each re-pay the wrap fixture; runtimecmd_checkfile.test and runtimecmd_roundtrip.test guard the committed bin/runtime.cmd and its source round-trip with no wrap fixture at all. argparsingtest.test (17 tests, ~32s) split into timeit/discover/contract/first_call (11) + argparsingtest_compare.test (compare_*, 6). Assessed and left unsplit: dtplite.test (24.6s solo), scriptexec.test (12s); exec.test excluded as imported core material. File headers point at the src/tests/modules/AGENTS.md punk/mix bullet for the sibling map; the runtests longest-first weight table carries measured post-split weights. runtests_parity.tcl gains a -names mode: multiset comparison of '- testname:' lines from '-report markdown -show-passes 1' captures, file attribution ignored - verbatim moves compare identical, lost/ duplicated/renamed tests are reported. Verified (native tclsh 9.0.3): repeated -jobs 8 full-suite runs 1m26.3s / 1m26.8s (was 2m16s at G-091, ~5m40s sequential - 4.07x vs the pre-parallel baseline, 36.6% vs the G-091 reference); slowest child 30.7s / 31.3s (bound 45s); NAMES identical (989 occurrences) vs the pre-split sequential capture; tallies identical in every run (989/973/ 15/1, exec-14.3 sole failure, files 87 -> 92); PARITY ok -jobs 8 vs -jobs 1 and between repeats; repo clean after runs. Notable finding recorded in the goal file: child-wall contention inflation (~45% with 50s+ dominant children present) collapsed to ~15% once the work distribution was flattened - splitting reduced both the critical path and the contention itself. G-092 flipped to achieved 2026-07-19 and archived (GOALS-archive.md record, detail file with full verification evidence to goals/archive/). goals_lint clean. Also noted in the goal file, out of scope: runtests -include-paths multi-pattern/repeated-flag matching misbehaved during measurement (single value with four deep patterns matched one; repeated flags matched zero) - contradicts documented semantics, needs its own investigation. Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- GOALS-archive.md | 4 + GOALS.md | 4 - .../G-092-split-longrunning-testfiles.md | 48 ++++++++- scriptlib/developer/runtests_parity.tcl | 87 ++++++++++++++- src/tests/AGENTS.md | 4 +- src/tests/modules/AGENTS.md | 2 +- .../argparsingtest/argparsingtest.test | 67 +----------- .../argparsingtest_compare.test | 78 ++++++++++++++ .../mix/testsuites/scriptwrap/multishell.test | 89 ++-------------- .../multishell_wrapdeterminism.test | 100 ++++++++++++++++++ .../scriptwrap/multishell_wrapverify.test | 95 +++++++++++++++++ .../scriptwrap/runtimecmd_checkfile.test | 62 +++++++++++ .../scriptwrap/runtimecmd_roundtrip.test | 70 ++++++++++++ src/tests/runtests.tcl | 23 ++-- 14 files changed, 565 insertions(+), 168 deletions(-) rename goals/{ => archive}/G-092-split-longrunning-testfiles.md (60%) create mode 100644 src/tests/modules/argparsingtest/testsuites/argparsingtest/argparsingtest_compare.test create mode 100644 src/tests/modules/punk/mix/testsuites/scriptwrap/multishell_wrapdeterminism.test create mode 100644 src/tests/modules/punk/mix/testsuites/scriptwrap/multishell_wrapverify.test create mode 100644 src/tests/modules/punk/mix/testsuites/scriptwrap/runtimecmd_checkfile.test create mode 100644 src/tests/modules/punk/mix/testsuites/scriptwrap/runtimecmd_roundtrip.test diff --git a/GOALS-archive.md b/GOALS-archive.md index 77ef3235..4fc45e96 100644 --- a/GOALS-archive.md +++ b/GOALS-archive.md @@ -100,3 +100,7 @@ Acceptance: a definition with an optional choice-restricted leader or value that ### G-091 [achieved 2026-07-19] runtests multiprocess parallelism (-jobs N) → detail: goals/archive/G-091-runtests-parallel-jobs.md Scope: src/tests/runtests.tcl, src/tests/testsupport/, src/tests/AGENTS.md Acceptance: -jobs 8 produces aggregate results identical to -jobs 1 (scriptlib/developer/runtests_parity.tcl reports PARITY ok) across repeated full-suite runs on a native tclsh 9.0.x, reduces full-suite wall time by at least 2.5x versus the sequential baseline on the reference machine, leaves the repo clean after runs (git status unchanged), and src/tests/AGENTS.md documents the -jobs flag and the serial-group policy. + +### G-092 [achieved 2026-07-19] Split long-running test files to lower the parallel floor of broad runs → detail: goals/archive/G-092-split-longrunning-testfiles.md +Scope: src/tests/modules/punk/mix/testsuites/scriptwrap/ (multishell.test), src/tests/shell/testsuites/binscripts/ (dtplite.test), src/tests/modules/argparsingtest/testsuites/argparsingtest/ (argparsingtest.test), src/tests/shell/testsuites/punkexe/ (scriptexec.test - if assessed worthwhile), src/tests/runtests.tcl (longest-first weight table), src/tests/AGENTS.md + affected child AGENTS.md +Acceptance: after the split, repeated -jobs 8 full-suite runs on the reference machine report no child process wall above 45s in the runner's slowest-child listing and improve full-suite wall time by at least 25% versus the G-091 verification baseline (2m16s); test conservation is proven and recorded in this file - global tallies identical to a pre-split sequential baseline and the full set of test names unchanged (name-level comparison, mechanism decided in the work - e.g. a names mode for scriptlib/developer/runtests_parity.tcl); no test is deleted, skipped, weakened, or renamed, and provenance comment lines travel with moved tests; the runtests longest-first weight table covers the new files; scriptlib/developer/runtests_parity.tcl reports PARITY ok between -jobs 8 and -jobs 1 on the split tree; affected AGENTS.md contracts (src/tests + shell/binscripts and scriptwrap references) are updated. diff --git a/GOALS.md b/GOALS.md index 29985c1c..f1178197 100644 --- a/GOALS.md +++ b/GOALS.md @@ -341,7 +341,3 @@ Detail: goals/G-089-scriptlib-kits-and-modes.md ### G-090 [proposed] shellfilter/shellthread stack and log-worker lifecycle audit Scope: src/modules/shellfilter-999999.0a1.0.tm (stack placement, insert_transform, junctions), src/modules/shellthread-999999.0a1.0.tm (worker park/adoption/termination), src/tests/modules/ (new characterization suites), README.md ('very unripe parts' note) Detail: goals/G-090-shellfilter-shellthread-audit.md - -### G-092 [proposed] Split long-running test files to lower the parallel floor of broad runs -Scope: src/tests/modules/punk/mix/testsuites/scriptwrap/ (multishell.test), src/tests/shell/testsuites/binscripts/ (dtplite.test), src/tests/modules/argparsingtest/testsuites/argparsingtest/ (argparsingtest.test), src/tests/shell/testsuites/punkexe/ (scriptexec.test - if assessed worthwhile), src/tests/runtests.tcl (longest-first weight table), src/tests/AGENTS.md + affected child AGENTS.md -Detail: goals/G-092-split-longrunning-testfiles.md diff --git a/goals/G-092-split-longrunning-testfiles.md b/goals/archive/G-092-split-longrunning-testfiles.md similarity index 60% rename from goals/G-092-split-longrunning-testfiles.md rename to goals/archive/G-092-split-longrunning-testfiles.md index 79bd96cc..75afb5a6 100644 --- a/goals/G-092-split-longrunning-testfiles.md +++ b/goals/archive/G-092-split-longrunning-testfiles.md @@ -1,6 +1,6 @@ # G-092 Split long-running test files to lower the parallel floor of broad runs -Status: proposed +Status: achieved 2026-07-19 Scope: src/tests/modules/punk/mix/testsuites/scriptwrap/ (multishell.test), src/tests/shell/testsuites/binscripts/ (dtplite.test), src/tests/modules/argparsingtest/testsuites/argparsingtest/ (argparsingtest.test), src/tests/shell/testsuites/punkexe/ (scriptexec.test - if assessed worthwhile), src/tests/runtests.tcl (longest-first weight table), src/tests/AGENTS.md + affected child AGENTS.md Goal: no single monolithic test file dominates the -jobs parallel critical path: the long-running suites are split into cohesive smaller files (grouped along shared-fixture boundaries, every test preserved verbatim) so a broad -jobs 8 run's slowest child process is bounded by its slowest cohesive test group rather than a whole legacy file. Acceptance: after the split, repeated -jobs 8 full-suite runs on the reference machine report no child process wall above 45s in the runner's slowest-child listing and improve full-suite wall time by at least 25% versus the G-091 verification baseline (2m16s); test conservation is proven and recorded in this file - global tallies identical to a pre-split sequential baseline and the full set of test names unchanged (name-level comparison, mechanism decided in the work - e.g. a names mode for scriptlib/developer/runtests_parity.tcl); no test is deleted, skipped, weakened, or renamed, and provenance comment lines travel with moved tests; the runtests longest-first weight table covers the new files; scriptlib/developer/runtests_parity.tcl reports PARITY ok between -jobs 8 and -jobs 1 on the split tree; affected AGENTS.md contracts (src/tests + shell/binscripts and scriptwrap references) are updated. @@ -50,3 +50,49 @@ Split considerations gathered in the same session: runs while full-suite wall varied up to ~30s); the 25%-vs-2m16s wall criterion is relative to the same-machine G-091 baseline rather than an absolute number for the same reason. + +## Progress + +Achieved-flip verification record, 2026-07-19: + +Splits performed (every test moved verbatim, provenance comment lines travelling; file +headers carry a split note pointing at src/tests/modules/AGENTS.md for the sibling map): +- multishell.test (1 file, 10 tests, ~105s solo child) -> five files in the same + directory: multishell.test (wrap + structure/LF-only + exec smoke incl WSL, 6 tests), + multishell_wrapverify.test (fresh-wrap checkfile), multishell_wrapdeterminism.test + (byte-identical re-wrap), runtimecmd_checkfile.test (committed bin/runtime.cmd checkfile + + LF contract - needs no wrap fixture), runtimecmd_roundtrip.test (src/scriptapps + re-wrap byte-identity - needs no wrap fixture). +- argparsingtest.test (17 tests, ~32s solo) -> argparsingtest.test (timeit/discover/ + parser-contract/first_call, 11 tests) + argparsingtest_compare.test (compare_*, 6 tests). +- Assessed and left unsplit: dtplite.test (24.6s solo / ~29s under -jobs 8 load), + scriptexec.test (12s solo) - both under the 45s bound; exec.test excluded as imported + Tcl core material per this goal's Context. + +Acceptance evidence (native tclsh 9.0.3, reference machine, piped/no-console context): +- Repeated -jobs 8 full-suite runs: 1m26.3s / 1m26.8s wall; slowest child 30.7s / 31.3s + (runtimecmd_roundtrip/checkfile) - no child above 45s in either run's slowest-child + listing. +- Wall improvement vs the G-091 verification baseline (2m16.2s): 36.6% (>=25% required); + 4.07x vs the ~350s pre-parallel sequential baseline. +- Conservation: NAMES identical (989 test-name occurrences, multiset comparison via the + new runtests_parity.tcl -names mode against a pre-split sequential capture); global + tallies identical in every run (989 total / 973 passed / 15 skipped / 1 failed = + exec-14.3, files 87 -> 92); PARITY ok between -jobs 8 and -jobs 1 on the split tree and + between the repeated -jobs 8 runs; repo clean (git status) after every run. +- runtests longest-first weight table updated with measured post-split weights; + src/tests/AGENTS.md (jobs bullet + Work Guidance numbers) and the + src/tests/modules/AGENTS.md punk/mix bullet updated. + +Findings: +- Child-wall contention inflation: with 50s+ dominant children in the mix (the first + 4-way split), -jobs 8 load inflated child walls ~45% over solo (runtime singles 27s -> + 42-46s; argparsingtest 32s -> 47.7s; wrapverify 35s -> 53s), briefly breaching the 45s + bound. After the secondary splits flattened the distribution (no solo child above + ~31s), inflation collapsed to ~15% - a flatter work distribution reduces not just the + critical path but the contention each child experiences. +- runtests -include-paths anomaly noted during measurement (out of scope, reported to + the user): a single -include-paths value carrying four space-separated deep patterns + matched only one of them, and repeated -include-paths flags matched zero files, + contradicting the documented accumulate semantics; file-tail glob targeting was used + instead. diff --git a/scriptlib/developer/runtests_parity.tcl b/scriptlib/developer/runtests_parity.tcl index 4d872093..f2af84b4 100644 --- a/scriptlib/developer/runtests_parity.tcl +++ b/scriptlib/developer/runtests_parity.tcl @@ -4,10 +4,12 @@ # #Usage: # tclsh scriptlib/developer/runtests_parity.tcl +# tclsh scriptlib/developer/runtests_parity.tcl -names # -#Each input file is the captured stdout of a runtests.tcl run with -report json (or -#markdown+json): the last line starting with {"runner" is taken as the report, so preamble -#lines (argv echo, package-load warnings) and fenced markdown context are tolerated. +#Default (json) mode: each input file is the captured stdout of a runtests.tcl run with +#-report json (or markdown+json): the last line starting with {"runner" is taken as the +#report, so preamble lines (argv echo, package-load warnings) and fenced markdown context +#are tolerated. # #Compared (deterministic result identity): # - top-level status and total/passed/skipped/failed tallies @@ -17,6 +19,13 @@ #Ignored (expected to vary between runs/modes): microseconds, the slowest list, #observed_* event counts. # +#-names mode (added 2026-07-19 for G-092 test-file splits): each input is the captured +#stdout of a runtests.tcl run with '-report markdown -show-passes 1'. Every +#'- testname: ' line (pass, failure and constraint-skip sections all use this form) +#contributes to a MULTISET of test names with file attribution deliberately ignored, so +#moving tests verbatim between files compares identical while any lost, duplicated or +#renamed test is reported. NAMES: identical -> exit 0, differences -> exit 1. +# #Exit codes: 0 = parity, 1 = differences found, 2 = usage or parse error. #No package dependencies (runs under any tclsh 8.6+); contains a minimal JSON parser #sufficient for the runner's machine-generated report format. @@ -257,10 +266,80 @@ proc list_diff_lines {label lista listb} { return $lines } +proc load_names {filename} { + if {![file exists $filename]} { + puts stderr "runtests_parity: file not found: $filename" + exit 2 + } + set fd [open $filename r] + set data [read $fd] + close $fd + set names [list] + foreach ln [split $data \n] { + if {[regexp {^- testname: (.*?)\s*$} $ln -> nm]} { + if {$nm ne ""} { + lappend names $nm + } + } + } + if {![llength $names]} { + puts stderr "runtests_parity: no '- testname:' lines found in $filename (expected captured stdout of: runtests.tcl -report markdown -show-passes 1)" + exit 2 + } + return $names +} + +proc names_multiset {names} { + set counts [dict create] + foreach n $names { + dict incr counts $n + } + return $counts +} + +if {[lindex $argv 0] eq "-names"} { + lassign [lrange $argv 1 end] filea fileb + if {$filea eq "" || $fileb eq "" || [llength $argv] != 3} { + puts stderr "usage: tclsh runtests_parity.tcl -names " + puts stderr " where each file is captured stdout of: src/tests/runtests.tcl -report markdown -show-passes 1 ..." + exit 2 + } + set namesa [load_names $filea] + set namesb [load_names $fileb] + set counta [names_multiset $namesa] + set countb [names_multiset $namesb] + puts "runtests test-name comparison (multiset, file attribution ignored)" + puts "A: $filea ([llength $namesa] name occurrences, [dict size $counta] distinct)" + puts "B: $fileb ([llength $namesb] name occurrences, [dict size $countb] distinct)" + set diffs [list] + foreach nm [lsort [dict keys $counta]] { + set ca [dict get $counta $nm] + set cb [expr {[dict exists $countb $nm] ? [dict get $countb $nm] : 0}] + if {$ca != $cb} { + lappend diffs "$nm: A=$ca B=$cb" + } + } + foreach nm [lsort [dict keys $countb]] { + if {![dict exists $counta $nm]} { + lappend diffs "$nm: A=0 B=[dict get $countb $nm]" + } + } + if {[llength $diffs] == 0} { + puts "NAMES: identical ([llength $namesa] test name occurrences)" + exit 0 + } + foreach d $diffs { + puts "- $d" + } + puts "NAMES: DIFFER ([llength $diffs] names with differing counts)" + exit 1 +} + lassign $argv filea fileb if {$filea eq "" || $fileb eq "" || [llength $argv] != 2} { puts stderr "usage: tclsh runtests_parity.tcl " - puts stderr " where each file is captured stdout of: src/tests/runtests.tcl -report json ..." + puts stderr " tclsh runtests_parity.tcl -names " + puts stderr " where each json file is captured stdout of: src/tests/runtests.tcl -report json ..." exit 2 } diff --git a/src/tests/AGENTS.md b/src/tests/AGENTS.md index 761e2c4b..fd56c1b4 100644 --- a/src/tests/AGENTS.md +++ b/src/tests/AGENTS.md @@ -21,7 +21,7 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc - The testinterp module path includes `src/vendormodules` (and `src/vendormodules_tcl` when present) so vendored dependencies such as `voo` resolve in tests. - Multi-process mode (`-tcltestoptions {-singleproc 0}`, completed 2026-07-18): each test file runs in a child process of the same executable via `testsupport/child_test_runner.tcl`, driven by a per-run environment payload `runtests.tcl` generates (package prefer latest, test tm paths, auto_path, modpod ifneeded definitions, tcltest options; per-file `-testdir` computed child-side). The child then mirrors the single-process testinterp preload: package require shellrun plus one no-op `shellrun::runx -tcl` call (the testinterp sources test files via runx, whose execution pulls further runtime deps - currently punk::lib). Several existing suites use punk::* commands without a package require of their own and depend on this preload (17 files / 97 tests error without shellrun, a further 7 files / 31 tests without the runx warmup; making suite dependencies explicit so children can go leaner is a candidate cleanup, verifiable with the parity tool below). The bootstrap warms `clock format` before the module-path wipe: first script-level clock use loads msgcat from the runtime's default module paths, which the test module paths do not supply (the single-process testinterp is shielded only by the runtests parent's earlier clock use - a latent gap for any test explicitly requiring msgcat in either mode). - Multi-process failure classification: a nonzero child exit is reported as a file-level failure (compact/markdown/json carry an ERROR entry with errorcode `CHILDPROCESS exit ` and the child's stderr tail); exit-0 with no tcltest summary line remains the `missing-cleanupTests` warning. Prefer a native tclsh for multi-process runs: children of a kit executable boot via the kit's script dispatch with kit-stamped punk packages preloaded, which can shadow the src dev modules under test (the runner prints a warning). -- Parallel jobs (G-091, achieved 2026-07-19 - see goals/archive/G-091-runtests-parallel-jobs.md): `-jobs N` (N > 1) implies multi-process mode and runs test files through a worker pool of N concurrent child processes (tpool; per-child capture files and per-child tcltest `-tmpdir` subdirs so concurrent `makeFile` helpers cannot collide; longest-first submission using static weights from the 2026-07-18 timing measurements - the full-suite floor is the slowest single file, multishell.test). Files matching `-serial-paths` run sequentially after the parallel batch - default `{modules/punk/console/** modules/opunk/console/**}` (the console-state suites, which emit to and query the shared terminal; suites driven through piped-stdio child processes - shell/**, modules/punk/repl/** - stay parallel by default and can be added to `-serial-paths` if they prove flaky under a real console; each serial file costs its full ~3-4s child boot strictly after the parallel phase, so the list is deliberately minimal). Reports are unchanged in shape and emitted in discovery order. An explicit `-tcltestoptions {-singleproc 1}` conflicts with `-jobs > 1` (error). Watch-event semantics under `-jobs`: no `file-start` events; `file-collected` (with exitcode, output relayed at that moment) fires in completion order and the authoritative `file-end` (tallies) in discovery order. Per-test microsecond timings are noisier under CPU oversubscription - prefer sequential runs when `-slowest` matters. Child spawn under -jobs is background-exec plus `tcl::process status` polling: concurrent BLOCKING execs from worker threads convoy on Windows - they all return together at the longest-lived concurrent child's exit (proven 2026-07-19 with plain sleepers of 2-12s all walling at 12s; this cost early -jobs runs ~50s of false critical path). Runtimes without `tcl::process` (tcl 8.6) fall back to blocking exec, which re-convoys under -jobs. The runner prints phase times (parallel/serial-tail/processing) and the ten slowest child WALL times in non-json modes - child wall includes boot and file-level fixture cost that per-test usec timings miss. +- Parallel jobs (G-091, achieved 2026-07-19 - see goals/archive/G-091-runtests-parallel-jobs.md): `-jobs N` (N > 1) implies multi-process mode and runs test files through a worker pool of N concurrent child processes (tpool; per-child capture files and per-child tcltest `-tmpdir` subdirs so concurrent `makeFile` helpers cannot collide; longest-first submission using static weights from the 2026-07-18/19 timing measurements - the full-suite floor is the slowest single child; the former ~2m multishell.test floor was removed by the G-092 split of that suite into four files, leaving all children under ~35s). Files matching `-serial-paths` run sequentially after the parallel batch - default `{modules/punk/console/** modules/opunk/console/**}` (the console-state suites, which emit to and query the shared terminal; suites driven through piped-stdio child processes - shell/**, modules/punk/repl/** - stay parallel by default and can be added to `-serial-paths` if they prove flaky under a real console; each serial file costs its full ~3-4s child boot strictly after the parallel phase, so the list is deliberately minimal). Reports are unchanged in shape and emitted in discovery order. An explicit `-tcltestoptions {-singleproc 1}` conflicts with `-jobs > 1` (error). Watch-event semantics under `-jobs`: no `file-start` events; `file-collected` (with exitcode, output relayed at that moment) fires in completion order and the authoritative `file-end` (tallies) in discovery order. Per-test microsecond timings are noisier under CPU oversubscription - prefer sequential runs when `-slowest` matters. Child spawn under -jobs is background-exec plus `tcl::process status` polling: concurrent BLOCKING execs from worker threads convoy on Windows - they all return together at the longest-lived concurrent child's exit (proven 2026-07-19 with plain sleepers of 2-12s all walling at 12s; this cost early -jobs runs ~50s of false critical path). Runtimes without `tcl::process` (tcl 8.6) fall back to blocking exec, which re-convoys under -jobs. The runner prints phase times (parallel/serial-tail/processing) and the ten slowest child WALL times in non-json modes - child wall includes boot and file-level fixture cost that per-test usec timings miss. - UDP watch mode (added 2026-07-18): `env(PUNK_TEST_UDPTEE)` = `` or `:` (e.g `41197`) makes runtests emit the run over UDP via the shellfilter::log mechanism (shellthread worker threads -> tcludp, one datagram per line, cooked format whose source column carries the tag). Tag `runtests` carries lifecycle events; each event message is a well-formed tcl list `RUNTESTS-EVENT ?key value ...?` (`run-start`, `file-start`, `file-end` with per-file tallies, `run-end`). Tags `teststdout`/`teststderr` carry test file output: streamed live per output chunk in single-process mode (`shellrun::runx -teelog` 0.2.0 - write traces on the capture variables, so captured results and reports are unchanged) and relayed per completed file by the parent in multi-process mode. No listener is required (sends to an unbound local port are discarded). The parent extends its auto_path with `/lib_tcl/` so log workers resolve the project's tcludp 1.0.13+ (1.0.12's UDP_ExitProc is the G-036 exit-hang class). Viewer-side caveat: tcludp's RECEIVE fileevent under Tcl 9.0.3/Windows delivered only the first datagram in testing (send side unaffected; a .NET UdpClient received everything) - a viewer should use timer-driven non-blocking reads or a non-tcludp receiver until that is resolved. - In watch mode the singleproc loop closes and terminates the shellfilter::log worker threads opened inside each per-file testinterp before deleting it (the `-teelog` tag workers plus shellrun's punkshout/punksherr teehandle workers). The terminate handshakes cost ~0.5-0.7s per file, so unwatched runs skip this and keep the pre-existing small leak of the two teehandle worker threads per file (noted for the shellfilter/shellthread audit; the process exits at run end regardless). - Tests should run against source modules and libraries from `src/`, not installed packages or root-level build outputs. @@ -47,7 +47,7 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc - Add `-slowest ` when timing outliers are relevant. - Multi-process runs use `-tcltestoptions {-singleproc 0}` with otherwise identical flags. Check result parity between modes by capturing `-report json` stdout from each and comparing with `tclsh scriptlib/developer/runtests_parity.tcl ` (ignores timings; exit 0 on parity). - To watch runs live (including runs launched by agents or other processes, which inherit the environment): set `PUNK_TEST_UDPTEE=41197` and point a UDP listener/viewer at that port. No runner flags needed; results are unaffected. -- For broad runs, `-jobs 8` cuts full-suite wall time roughly 2.6x (~2m16s vs ~5m50s sequential on the reference machine, 2026-07-19; result parity verified) - e.g ` src/tests/runtests.tcl -jobs 8 -report compact -show-passes 0`. The floor is the slowest single child (multishell.test, ~2m). Focused runs rarely need `-jobs`. +- For broad runs, `-jobs 8` cuts full-suite wall time roughly 4x (~1m26s vs ~5m40s sequential on the reference machine, 2026-07-19 post G-092 split; result parity verified) - e.g ` src/tests/runtests.tcl -jobs 8 -report compact -show-passes 0`. No child process exceeds ~31s; the wall is now work-distribution bound rather than dominated by any single file. Focused runs rarely need `-jobs`. - Add `-strict-exit 1` when a nonzero shell exit code is needed for failures or parser warnings. - Capture enough stderr or failure context to identify the failing command or assertion. - For ERROR-status failures, the markdown report's `errorInfo` block and compact `message=` field carry the full Tcl error message; use `-report markdown` for untruncated context. diff --git a/src/tests/modules/AGENTS.md b/src/tests/modules/AGENTS.md index ab19d9e4..82b12ea1 100644 --- a/src/tests/modules/AGENTS.md +++ b/src/tests/modules/AGENTS.md @@ -44,7 +44,7 @@ Unit tests for editable source modules under `src/modules/`, `src/modules_tcl8/` - `punk/nav/ns/` — punk::nav::ns tests (`testsuites/nav/navns.test`): the n/ n// n/// navigation state machine (ns/ transitions absolute/relative/glob-no-nav, failed-nav state preservation, quad-colon normalization, v-form content selection, ensemble annotation) and the ::punk::nav::ns::ns_current variable contract the repl/codethread/subshell seeding all consume; display content is covered in punk/ns nslist.test - `punk/repl/` — punk::repl tests (`testsuites/repl/`): opunk console backend integration (`consolebackends.test`) and repl current-namespace retention (`nscurrent.test`: real codethread via repl::init driven by synchronous runscript sends - inscope evaluation of ns_current, retention across submissions, n/-navigation retained, auto-create-with-notice for missing namespaces, the 2026-07-14 stray-namespace seeding fix pinned behaviourally plus a source-text guard on repl::start's inline template; the end-to-end piped subshell session is covered at shell level by shell/testsuites/punkexe/shellnavns.test - which found the first-subshell shared-code-interp asymmetry and the piped-inscope gap recorded there) - `punk/ns/` — punk::ns tests (`testsuites/ns/`): cmdwhich/cmdinfo/cmd_traverse doc-lookup flow (`cmdflow.test`, G-040 parity), n/ display machinery characterization (`nslist.test`: tier A get_ns_dicts classification buckets as the machine contract for display reworks - incl package tail/prefix derivation, alias edge cases, usageinfo scan-dependence; tier B per-element layout-agnostic marking - underline/underdouble/underdotted namespace package styles, command type tag colours, exported/imported markers, the punkargs doc icon; tier C REWORK-flagged pins of the current hardcoded 2-col/4-col layout and nspath subtables, to flip deliberately with the planned punk-tables/width-responsive rework; plus the flipped nslist_types_default pin - bare nslist without -types displays all member types since punk::ns 0.7.1 fixed the braced-literal -types default), corp proc-retrieval and syntax/untabify interplay (`corp.test`: name edge cases, -ranges/-n line handling, basic-highlight ansistrip equivalence, -untabify spaces/unicode tab-free output, the KNOWN-DEFICIENCY pin for default -untabify none on tabbed bodies - grepstr warns per pass and brace overlays mangle tabbed lines, deterministic under mocked console tabstops - and a ::tcl::CopyDirectory -untabify spaces smoke test; precursor coverage for the planned punk::ns hygiene pass), cmdtrace characterization (`cmdtrace.test`: -pause 0 non-interactive runs, linedict line-mark keys for flat and 2-word-form nested switches as correct-mark guards, and GAP pins for the upstream nested-switch mismark - core.tcl-lang.org tktview 5d5b1052280c976ea3d4, arm bodies whose split-list index lands on a literal switch-command word report container-relative lines; mark tests gated on have_tclcoredocs because cmdtrace's arm-offset correction parses against the ::switch argdoc; plus the fixed-canary asserting punk::lib::check::has_tclbug_nestedswitch_tracelines still reports the bug - a live behavioural probe, so a fixed Tcl release fails the canary first and triggers the documented flip workflow), cmdhelp usage-rendering integration (`cmdhelp.test`: scheme selection, goodarg/badarg marking incl type/allocation failures, goodchoice highlighting of supplied/default choice words, alias path, cmdinfo result shape, queried-command failure attribution, and `-return dict` parse-status returns (G-049 - its GAP pins flipped 2026-07-10); remaining GAP pins for pseudo-command cmdtype + space-form docid prefixes (G-051, real `string is` pins behind the have_tclcoredocs constraint), TclOO undocumented-method fallback (G-052), and synopsis marking absence (G-050)), and name/path primitive characterization (`nsprimitives.test`: string pins for nsparts/nsprefix/nstail/nsjoin/nsjoinall incl weird colon-run (`:::`) splitting, the trailing-colon parse ambiguity (`::x:` + `y` joins to the same string as `::x` + `:y` and reparses leading-colon-greedy), and prefix/tail/join round-trip and its absolutizing exceptions (the original twin-divergence pins for nsparts1/nsprefix1/nsprefix_orig/nstail1/nstail_orig served as safe-deletion evidence and were removed with the twins in the punk::ns 0.7.0 hygiene pass - divergence record in this file's git history, commit 0c7168a1); plus nseval fq-requirement/create-on-eval/evaluator-proc caching, the native-vs-punk `p:::x` resolution divergence (native namespace eval reaches child `x`, nseval creates/reaches literal `:x`), nseval_ifexists no-create + error propagation on plain and genuinely weird namespaces, nsexists/nschildren/nstree_raw weird-ns and relative-resolution pins, globmatchns `*`/`**`/`?` semantics (incl `*` matching a single inner colon - a formerly stale 'should be fixed' comment above nsglob_as_re was corrected in the 0.7.0 hygiene pass), and nspath_to_absolute/nspath_here_absolute caller-resolution pins; the nsjoinall error-message wart pin ('nsjoin:' prefix) flipped when punk::ns 0.7.1 fixed it) -- `punk/mix/` — punk::mix::cli tests (prune helpers, punkcheck virtual sources), punk::mix::commandset::repo fossil move/rename characterization tests (`testsuites/repo/`, FOSSIL_HOME-isolated; GAP-marked tests pin behaviour G-022 will change), punk::mix::commandset::loadedlib tests (`testsuites/loadedlib/libsearch.test`: 'dev lib.search' match semantics via -return list — wrap-glob default, =exact prefix, case rules, explicit globs, version aggregation — plus the loadedlib 0.2.0 contract: deep discovery by default (deep .tm modules found without -refresh, registration persists), -refresh = genuine re-scan (epoch incr + rediscovery picks up .tm files added to already-scanned dirs), and highlight working without the shell-global a+ alias; shared provisioned child interp sourcing the source-tree libunknown directly — see the file's ORDERING NOTE), and the MULTISHELL polyglot build machinery (`testsuites/scriptwrap/multishell.test`: scriptset wrap via the punk.multishell.cmd template - structure/LF-only/determinism, checkfile 512-byte label validation of fresh wraps AND the committed bin/runtime.cmd, the runtime scriptset round-trip byte-identity pin, and platform-gated execution smoke: cmd.exe→powershell payload on windows, sh payload on unix or via the `wsllinux` capability constraint from `src/tests/testsupport/wslprobe.tcl` - staged to the WSL distro's native filesystem, G-059) +- `punk/mix/` — punk::mix::cli tests (prune helpers, punkcheck virtual sources), punk::mix::commandset::repo fossil move/rename characterization tests (`testsuites/repo/`, FOSSIL_HOME-isolated; GAP-marked tests pin behaviour G-022 will change), punk::mix::commandset::loadedlib tests (`testsuites/loadedlib/libsearch.test`: 'dev lib.search' match semantics via -return list — wrap-glob default, =exact prefix, case rules, explicit globs, version aggregation — plus the loadedlib 0.2.0 contract: deep discovery by default (deep .tm modules found without -refresh, registration persists), -refresh = genuine re-scan (epoch incr + rediscovery picks up .tm files added to already-scanned dirs), and highlight working without the shell-global a+ alias; shared provisioned child interp sourcing the source-tree libunknown directly — see the file's ORDERING NOTE), and the MULTISHELL polyglot build machinery (`testsuites/scriptwrap/`, split 2026-07-19 per G-092 so no single file dominates the -jobs parallel floor - tests moved verbatim: `multishell.test` = scriptset wrap via the punk.multishell.cmd template with structure/LF-only pins plus platform-gated execution smoke (cmd.exe→powershell payload on windows, sh payload on unix or via the `wsllinux` capability constraint from `src/tests/testsupport/wslprobe.tcl` - staged to the WSL distro's native filesystem, G-059); `multishell_wrapverify.test` = checkfile 512-byte label validation of a fresh wrap; `multishell_wrapdeterminism.test` = byte-identical re-wrap pin; `runtimecmd_checkfile.test` = checkfile + LF contract of the committed bin/runtime.cmd; `runtimecmd_roundtrip.test` = the runtime scriptset round-trip byte-identity pin) - `punk/lib/` — punk::lib tests (`testsuites/lib/`): range/index/parse/compat/interp_sync utilities, G-058 static-baseline seeding (`staticseed.test`: interp_sync_package_paths/snapshot_package_paths propagate a simulated ::punkboot static baseline and seed `load {} ` ifneeded mappings; no-op without a baseline), and the repl command-completeness engine (`commandcomplete.test`: punk::lib::system::incomplete pending-opener stacks - the info-complete quoting quirk progression (`set x "{*}{"` standalone vs in-proc-body), single openers, tabs, escapes, incomplete<->info-complete parity property; pre-repl-refactor characterization, see goals/G-044 detail preserve-list) - `punk/packagepreference/` — punk::packagepreference tests (`testsuites/packagepreference/`): G-058 static-vs-bundled policy (`staticpolicy.test`: require of a baseline package triggers the index scan before resolution so a newer bundled copy wins, static beats older bundled, exact requires of bundled versions stay reachable, missing static mappings get seeded) - `punk/libunknown/` — .tm same-version shadowing pin-tests (`testsuites/shadowing/`): tcl::tm::add prepend rule, head-of-tm-list wins exact-version ties, version beats order, punk::libunknown parity — shipped behaviour depends on these (runtests tm ordering, punk_main package-mode precedence, G-033); mixed .tm/pkgIndex.tcl characterization is goal G-035; discovery/epoch-cache characterization (`testsuites/discovery/discovery.test`): sibling .tm registration happens only at the requested namespace depth (deeper modules invisible to 'package names' until requested), register_all_tm deep discovery (all-depths registration, per-epoch cached no-op, head-of-tm-list precedence parity), 'package epoch' command shape, trace-driven epoch increments on tm/auto_path changes, the epoch-index short-circuit (a .tm added to an already-scanned dir needs 'package epoch incr'), and the pkgIndex.tcl sourcing-scope contract (source_pkgindex, fixed 2026-07-11 - formerly a GAP pin of the global-'dir' clobber): user global 'dir' survives pkg-unknown fallthrough, index scripts see $dir and reach the real ::auto_path (tcllib extension pattern), and their stray unqualified sets no longer leak to ::. Suite conventions learned the hard way: child probes source the SOURCE-TREE libunknown directly by path — 'package require punk::libunknown' tie-breaks the same-version copies (src/modules vs bootsupport) by tm path order, which favours bootsupport in the testinterp and machine env paths in children; and probe scripts must avoid global variable names the handlers use (notably 'dir') — that clobbering misdirected a fixture write into the real Tcl install during test development diff --git a/src/tests/modules/argparsingtest/testsuites/argparsingtest/argparsingtest.test b/src/tests/modules/argparsingtest/testsuites/argparsingtest/argparsingtest.test index 8a35cffd..f6463de0 100644 --- a/src/tests/modules/argparsingtest/testsuites/argparsingtest/argparsingtest.test +++ b/src/tests/modules/argparsingtest/testsuites/argparsingtest/argparsingtest.test @@ -1,7 +1,6 @@ package require tcltest package require argparsingtest -package require json namespace eval ::testspace { namespace import ::tcltest::* @@ -11,6 +10,9 @@ namespace eval ::testspace { #added 2026-07-18 (agent) - suite rewritten for the argparsingtest 1.0.0 restructure: #parser procs live in per-style namespaces (opts/tkstyle/tclstyle), harness results are keyed by style + #Split 2026-07-19 (G-092 - parallel-floor reduction, tests moved verbatim): this file holds + #timeit/discover_parsers/parser-contract/first_call coverage; the compare_* tests live in + #argparsingtest_compare.test in this directory. #added 2026-07-18 (agent) test timeit_basic {Test that timeit -style opts returns style-keyed dict with numeric timing} -body { @@ -108,60 +110,6 @@ namespace eval ::testspace { set ok } -result 1 - #added 2026-07-18 (agent) - N/A rows: unsupported library/style combos stay visible in the table - test compare_table_sections {Test that compare -format table emits all three style sections with N/A rows} -body { - set r [argparsingtest::compare -iterations 20 -format table -parsers {manual_switch punkargs cmdline_typed argp}] - set ok 1 - foreach expected {"options only" "tk style (positionals first)" "tcl style (positionals last)" "N/A" "not loaded:" "Parser"} { - if {[string first $expected $r] < 0} { set ok "missing: $expected" ; break } - } - set ok - } -result 1 - - #added 2026-07-18 (agent) - test compare_dict {Test that compare -format dict returns style-keyed results/first_call/not_loaded/unsupported} -body { - set r [argparsingtest::compare -style opts -iterations 20 -parsers {manual_switch} -format dict] - set ok 1 - foreach key {results first_call not_loaded unsupported} { - if {![dict exists $r $key]} { set ok "missing $key" ; break } - } - if {$ok eq 1 && ![dict exists $r results opts manual_switch us_per_call]} { set ok "missing results.opts.manual_switch" } - if {$ok eq 1 && [dict get $r unsupported opts] ne ""} { set ok "unexpected unsupported list" } - set ok - } -result 1 - - #added 2026-07-18 (agent) - test compare_json {Test that compare -format json round-trips with style-keyed unsupported and first_call structure} -body { - set r [argparsingtest::compare -style tkstyle -iterations 20 -parsers {manual_switch cmdline_typed} -format json] - set ok 1 - if {[catch {set d [json::json2dict $r]} _err]} { set ok "json parse: $_err" } - if {$ok eq 1 && ![dict exists $d results tkstyle manual_switch us_per_call]} { set ok "missing results" } - if {$ok eq 1 && [dict get $d unsupported tkstyle] ne "cmdline_typed"} { set ok "unsupported: [dict get $d unsupported tkstyle]" } - if {$ok eq 1 && ![dict exists $d first_call tkstyle manual_switch cold]} { set ok "missing first_call cold" } - if {$ok eq 1 && ![dict exists $d first_call tkstyle manual_switch warm]} { set ok "missing first_call warm" } - set ok - } -result 1 - - #added 2026-07-17 (agent) - test compare_badformat {Test that compare -format bogus raises an error containing choiceviolation} -body { - set caught [catch {argparsingtest::compare -format bogus} err] - set ok 0 - if {$caught && ([string first "choiceviolation" $err] >= 0 || [string first "must be one of" $err] >= 0)} { - set ok 1 - } - set ok - } -result 1 - - #added 2026-07-18 (agent) - test compare_badstyle {Test that compare -style bogus raises an error containing choiceviolation} -body { - set caught [catch {argparsingtest::compare -style bogus} err] - set ok 0 - if {$caught && ([string first "choiceviolation" $err] >= 0 || [string first "must be one of" $err] >= 0)} { - set ok 1 - } - set ok - } -result 1 - #added 2026-07-17 (agent) test first_call_handrolled {Test that first_call returns cold and warm float timings for a hand-rolled parser} -body { set r [argparsingtest::first_call -style opts -parsers {manual_switch}] @@ -218,14 +166,5 @@ namespace eval ::testspace { } set ok } -result 1 - - #added 2026-07-17 (agent) - test compare_table_first_columns {Test that compare table output has first_cold and first_warm columns} -body { - set r [argparsingtest::compare -style opts -iterations 20 -format table -parsers {manual_switch}] - set ok 1 - if {[string first "first_cold" $r] < 0} { set ok 0 } - if {$ok && [string first "first_warm" $r] < 0} { set ok 0 } - set ok - } -result 1 } tcltest::cleanupTests diff --git a/src/tests/modules/argparsingtest/testsuites/argparsingtest/argparsingtest_compare.test b/src/tests/modules/argparsingtest/testsuites/argparsingtest/argparsingtest_compare.test new file mode 100644 index 00000000..9fcecdd2 --- /dev/null +++ b/src/tests/modules/argparsingtest/testsuites/argparsingtest/argparsingtest_compare.test @@ -0,0 +1,78 @@ + +package require tcltest +package require argparsingtest +package require json + +namespace eval ::testspace { + namespace import ::tcltest::* + + #added 2026-07-18 (agent) - suite rewritten for the argparsingtest 1.0.0 restructure: + #parser procs live in per-style namespaces (opts/tkstyle/tclstyle), harness results are keyed by style + #Split 2026-07-19 (G-092 - parallel-floor reduction, tests moved verbatim from + #argparsingtest.test): this file holds the compare_* coverage; timeit/discover_parsers/ + #parser-contract/first_call tests live in argparsingtest.test in this directory. + + #added 2026-07-18 (agent) - N/A rows: unsupported library/style combos stay visible in the table + test compare_table_sections {Test that compare -format table emits all three style sections with N/A rows} -body { + set r [argparsingtest::compare -iterations 20 -format table -parsers {manual_switch punkargs cmdline_typed argp}] + set ok 1 + foreach expected {"options only" "tk style (positionals first)" "tcl style (positionals last)" "N/A" "not loaded:" "Parser"} { + if {[string first $expected $r] < 0} { set ok "missing: $expected" ; break } + } + set ok + } -result 1 + + #added 2026-07-18 (agent) + test compare_dict {Test that compare -format dict returns style-keyed results/first_call/not_loaded/unsupported} -body { + set r [argparsingtest::compare -style opts -iterations 20 -parsers {manual_switch} -format dict] + set ok 1 + foreach key {results first_call not_loaded unsupported} { + if {![dict exists $r $key]} { set ok "missing $key" ; break } + } + if {$ok eq 1 && ![dict exists $r results opts manual_switch us_per_call]} { set ok "missing results.opts.manual_switch" } + if {$ok eq 1 && [dict get $r unsupported opts] ne ""} { set ok "unexpected unsupported list" } + set ok + } -result 1 + + #added 2026-07-18 (agent) + test compare_json {Test that compare -format json round-trips with style-keyed unsupported and first_call structure} -body { + set r [argparsingtest::compare -style tkstyle -iterations 20 -parsers {manual_switch cmdline_typed} -format json] + set ok 1 + if {[catch {set d [json::json2dict $r]} _err]} { set ok "json parse: $_err" } + if {$ok eq 1 && ![dict exists $d results tkstyle manual_switch us_per_call]} { set ok "missing results" } + if {$ok eq 1 && [dict get $d unsupported tkstyle] ne "cmdline_typed"} { set ok "unsupported: [dict get $d unsupported tkstyle]" } + if {$ok eq 1 && ![dict exists $d first_call tkstyle manual_switch cold]} { set ok "missing first_call cold" } + if {$ok eq 1 && ![dict exists $d first_call tkstyle manual_switch warm]} { set ok "missing first_call warm" } + set ok + } -result 1 + + #added 2026-07-17 (agent) + test compare_badformat {Test that compare -format bogus raises an error containing choiceviolation} -body { + set caught [catch {argparsingtest::compare -format bogus} err] + set ok 0 + if {$caught && ([string first "choiceviolation" $err] >= 0 || [string first "must be one of" $err] >= 0)} { + set ok 1 + } + set ok + } -result 1 + + #added 2026-07-18 (agent) + test compare_badstyle {Test that compare -style bogus raises an error containing choiceviolation} -body { + set caught [catch {argparsingtest::compare -style bogus} err] + set ok 0 + if {$caught && ([string first "choiceviolation" $err] >= 0 || [string first "must be one of" $err] >= 0)} { + set ok 1 + } + set ok + } -result 1 + + #added 2026-07-17 (agent) + test compare_table_first_columns {Test that compare table output has first_cold and first_warm columns} -body { + set r [argparsingtest::compare -style opts -iterations 20 -format table -parsers {manual_switch}] + set ok 1 + if {[string first "first_cold" $r] < 0} { set ok 0 } + if {$ok && [string first "first_warm" $r] < 0} { set ok 0 } + set ok + } -result 1 +} +tcltest::cleanupTests diff --git a/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell.test b/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell.test index ca781f93..ec1540dd 100644 --- a/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell.test +++ b/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell.test @@ -9,15 +9,16 @@ package require punk::mix::commandset::scriptwrap #fragility, so the machinery that generates bin/*.cmd (e.g bin/runtime.cmd from #src/scriptapps/runtime.ps1 + runtime.bash + runtime_wrap.toml) needs pins. # -#Covered: +#Split 2026-07-19 (G-092 - parallel-floor reduction, tests moved verbatim): this file holds +#the wrap-and-run coverage (wrap a scriptset, assert output structure/LF-only, execution +#smoke under cmd.exe/sh/WSL). The other .test files in this directory hold the rest of the +#multishell/scriptwrap coverage - see src/tests/modules/AGENTS.md (punk/mix bullet). +# +#Covered here: # - wrapping a scriptset (payload scripts + _wrap.toml) via the module-provided # punk.multishell.cmd template produces the polyglot with payloads embedded verbatim # - the output is LF-only (the cmd.exe label scanner reads 512-byte chunks; the whole # mechanism depends on unix line endings - see checkfile comments in the module) -# - wrapping is deterministic (byte-identical on re-wrap) -# - checkfile (the 512-byte label/boundary validator) reports no label location errors -# for a fresh wrap AND for the committed bin/runtime.cmd - guarding against hand-edits -# or line-ending damage to the shipped artifact # - execution smoke: the same file runs under cmd.exe on windows (dispatching to the # configured win32 nextshell - powershell) and under sh/bash on unix. On windows the # shell layer deliberately relaunches via the win32 nextshell, so the sh payload is @@ -35,8 +36,6 @@ namespace eval ::testspace { } variable testdir [file dirname [file normalize [info script]]] - #/src/tests/modules/punk/mix/testsuites/scriptwrap -> 7 levels up - variable projectroot [file normalize [file join $testdir .. .. .. .. .. .. ..]] variable ps_payload {Write-Host "MARKER_POWERSHELL_OK"} variable sh_payload {echo "MARKER_SHELL_OK"} @@ -132,82 +131,6 @@ namespace eval ::testspace { }\ -result [list 1] - test scriptwrap_multishell_wrap_deterministic {re-wrapping the same scriptset produces byte-identical output}\ - -constraints wrapok\ - -setup $common -body { - variable fixdir - variable outfile - variable startdir - set outdir2 [makeDirectory scriptwrap_fixture_rewrap] - cd $fixdir - set res2 [punk::mix::commandset::scriptwrap::multishell wrapset -outputfolder $outdir2 -askme 0 -force 1] - cd $startdir - set outfile2 [dict get $res2 filename] - lappend result [file exists $outfile2] - lappend result [expr {[readbytes $outfile] eq [readbytes $outfile2]}] - }\ - -cleanup { - cd $startdir - }\ - -result [list 1 1] - - test scriptwrap_checkfile_generated_no_label_errors {checkfile reports no 512-byte label location errors for a fresh wrap}\ - -constraints wrapok\ - -setup $common -body { - variable outfile - set summary [punk::mix::commandset::scriptwrap::checkfile $outfile] - lappend result [string match "*ERROR: label location errors*" $summary] - #sanity: the analysis actually ran (labels enumerated) - lappend result [string match "*call-labels-found:*" $summary] - }\ - -cleanup { - }\ - -result [list 0 1] - - test scriptwrap_checkfile_committed_runtime_cmd {the committed bin/runtime.cmd passes checkfile - no label/boundary damage from hand-edits or line-ending conversion}\ - -setup $common -body { - variable projectroot - set target [file join $projectroot bin runtime.cmd] - if {![file exists $target]} { - #not built/present in this checkout - treat as vacuous pass with marker - lappend result no_runtime_cmd - } else { - set data [readbytes $target] - #LF-only is part of the artifact's contract - lappend result [expr {[string first \r $data] == -1}] - set summary [punk::mix::commandset::scriptwrap::checkfile $target] - lappend result [string match "*ERROR: label location errors*" $summary] - } - }\ - -cleanup { - }\ - -result [list 1 0] - - test scriptwrap_runtime_cmd_roundtrip_no_drift {re-wrapping the runtime scriptset from src/scriptapps/bin reproduces the committed bin/runtime.cmd byte for byte - the artifact is in sync with its sources and un-hand-edited}\ - -setup $common -body { - variable projectroot - variable startdir - set target [file join $projectroot bin runtime.cmd] - set scriptapps [file join $projectroot src scriptapps bin] - if {![file exists $target] || ![file exists $scriptapps/runtime_wrap.toml]} { - lappend result no_runtime_sources - } else { - set outdir [makeDirectory scriptwrap_runtime_roundtrip] - cd $scriptapps - set res [punk::mix::commandset::scriptwrap::multishell runtime -outputfolder $outdir -askme 0 -force 1] - cd $startdir - set fresh [dict get $res filename] - lappend result [expr {[readbytes $fresh] eq [readbytes $target]}] - #a failure here means either bin/runtime.cmd was edited directly (fix the - #src/scriptapps payloads instead and re-wrap), or the payloads/template - #changed without regenerating bin/runtime.cmd (re-wrap and commit it) - } - }\ - -cleanup { - cd $startdir - }\ - -result [list 1] - test scriptwrap_multishell_exec_windows_cmd {on windows, cmd.exe runs the polyglot and dispatches to the configured win32 nextshell (powershell payload executes)}\ -constraints {wrapok iswindows}\ -setup $common -body { diff --git a/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell_wrapdeterminism.test b/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell_wrapdeterminism.test new file mode 100644 index 00000000..7f24422d --- /dev/null +++ b/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell_wrapdeterminism.test @@ -0,0 +1,100 @@ +package require tcltest + +package require Thread ;#punk::fileline textinfo (used by scriptwrap machinery) calls thread::id +package require punk::mix::commandset::scriptwrap + +#Characterization of the punk MULTISHELL polyglot build machinery +#(punk::mix::commandset::scriptwrap::multishell + checkfile) - added 2026-07-10 at the +#user's direction: the polyglot technique is deliberately maintained despite its +#fragility, so the machinery that generates bin/*.cmd needs pins. +# +#Split 2026-07-19 (G-092 - parallel-floor reduction, test moved verbatim from +#multishell.test via multishell_wrapverify.test): this file holds the wrap determinism +#pin. The other .test files in this directory hold the rest of the multishell/scriptwrap +#coverage - see src/tests/modules/AGENTS.md (punk/mix bullet). +# +#Covered here: +# - wrapping is deterministic (byte-identical on re-wrap) +# +#NOTE for agents: bin/*.cmd files are GENERATED - fix the payloads under src/scriptapps/ +#(.ps1/.bash/.tcl + _wrap.toml) and re-wrap; do not edit the polyglot output +#(see bin/AGENTS.md and src/scriptapps/AGENTS.md). + +namespace eval ::testspace { + namespace import ::tcltest::* + variable common { + set result "" + } + + variable ps_payload {Write-Host "MARKER_POWERSHELL_OK"} + variable sh_payload {echo "MARKER_SHELL_OK"} + + #write with explicit lf translation - payload/toml files are consumed byte-wise + proc writefile_lf {path content} { + set fd [open $path w] + fconfigure $fd -translation lf + puts $fd $content + close $fd + } + variable fixdir [makeDirectory scriptwrap_fixture] + writefile_lf $fixdir/wrapset.ps1 $ps_payload + writefile_lf $fixdir/wrapset.sh $sh_payload + writefile_lf $fixdir/wrapset_wrap.toml { +[application] + template="punk.multishell.cmd" + scripts=[ + "wrapset.ps1", + "wrapset.sh" + ] + default_outputfile="wrapset.cmd" + default_nextshellpath="/usr/bin/env bash" + default_nextshelltype="bash" + win32.nextshellpath="cmd.exe /c powershell -nop -nol -ExecutionPolicy bypass -File" + win32.nextshelltype="powershell" + win32.outputfile="wrapset.cmd" +} + + #wrap once at load - the test re-wraps and compares against this output (multishell + #chats on stdout; that output is not part of test comparisons) + variable wrapresult "" + variable outfile "" + variable wrap_ok 0 + variable startdir [pwd] + cd $fixdir + if {![catch {punk::mix::commandset::scriptwrap::multishell wrapset -outputfolder $fixdir -askme 0 -force 1} wrapresult]} { + if {[dict exists $wrapresult filename]} { + set outfile [dict get $wrapresult filename] + set wrap_ok [file exists $outfile] + } + } + cd $startdir + testConstraint wrapok $wrap_ok + + proc readbytes {path} { + set fd [open $path r] + fconfigure $fd -translation binary + set data [read $fd] + close $fd + return $data + } + + test scriptwrap_multishell_wrap_deterministic {re-wrapping the same scriptset produces byte-identical output}\ + -constraints wrapok\ + -setup $common -body { + variable fixdir + variable outfile + variable startdir + set outdir2 [makeDirectory scriptwrap_fixture_rewrap] + cd $fixdir + set res2 [punk::mix::commandset::scriptwrap::multishell wrapset -outputfolder $outdir2 -askme 0 -force 1] + cd $startdir + set outfile2 [dict get $res2 filename] + lappend result [file exists $outfile2] + lappend result [expr {[readbytes $outfile] eq [readbytes $outfile2]}] + }\ + -cleanup { + cd $startdir + }\ + -result [list 1 1] +} +tcltest::cleanupTests ;#needed to produce test summary line. diff --git a/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell_wrapverify.test b/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell_wrapverify.test new file mode 100644 index 00000000..7abf9a83 --- /dev/null +++ b/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell_wrapverify.test @@ -0,0 +1,95 @@ +package require tcltest + +package require Thread ;#punk::fileline textinfo (used by scriptwrap checkfile) calls thread::id +package require punk::mix::commandset::scriptwrap + +#Characterization of the punk MULTISHELL polyglot build machinery +#(punk::mix::commandset::scriptwrap::multishell + checkfile) - added 2026-07-10 at the +#user's direction: the polyglot technique is deliberately maintained despite its +#fragility, so the machinery that generates bin/*.cmd needs pins. +# +#Split 2026-07-19 (G-092 - parallel-floor reduction, tests moved verbatim from +#multishell.test): this file holds checkfile validation of a fresh wrap. The other .test +#files in this directory hold the rest of the multishell/scriptwrap coverage - see +#src/tests/modules/AGENTS.md (punk/mix bullet). +# +#Covered here: +# - checkfile (the 512-byte label/boundary validator) reports no label location errors +# for a fresh wrap +# +#NOTE for agents: bin/*.cmd files are GENERATED - fix the payloads under src/scriptapps/ +#(.ps1/.bash/.tcl + _wrap.toml) and re-wrap; do not edit the polyglot output +#(see bin/AGENTS.md and src/scriptapps/AGENTS.md). + +namespace eval ::testspace { + namespace import ::tcltest::* + variable common { + set result "" + } + + variable ps_payload {Write-Host "MARKER_POWERSHELL_OK"} + variable sh_payload {echo "MARKER_SHELL_OK"} + + #write with explicit lf translation - payload/toml files are consumed byte-wise + proc writefile_lf {path content} { + set fd [open $path w] + fconfigure $fd -translation lf + puts $fd $content + close $fd + } + variable fixdir [makeDirectory scriptwrap_fixture] + writefile_lf $fixdir/wrapset.ps1 $ps_payload + writefile_lf $fixdir/wrapset.sh $sh_payload + writefile_lf $fixdir/wrapset_wrap.toml { +[application] + template="punk.multishell.cmd" + scripts=[ + "wrapset.ps1", + "wrapset.sh" + ] + default_outputfile="wrapset.cmd" + default_nextshellpath="/usr/bin/env bash" + default_nextshelltype="bash" + win32.nextshellpath="cmd.exe /c powershell -nop -nol -ExecutionPolicy bypass -File" + win32.nextshelltype="powershell" + win32.outputfile="wrapset.cmd" +} + + #wrap once at load - the tests examine this output (multishell chats on stdout; that + #output is not part of test comparisons) + variable wrapresult "" + variable outfile "" + variable wrap_ok 0 + variable startdir [pwd] + cd $fixdir + if {![catch {punk::mix::commandset::scriptwrap::multishell wrapset -outputfolder $fixdir -askme 0 -force 1} wrapresult]} { + if {[dict exists $wrapresult filename]} { + set outfile [dict get $wrapresult filename] + set wrap_ok [file exists $outfile] + } + } + cd $startdir + testConstraint wrapok $wrap_ok + + proc readbytes {path} { + set fd [open $path r] + fconfigure $fd -translation binary + set data [read $fd] + close $fd + return $data + } + + test scriptwrap_checkfile_generated_no_label_errors {checkfile reports no 512-byte label location errors for a fresh wrap}\ + -constraints wrapok\ + -setup $common -body { + variable outfile + set summary [punk::mix::commandset::scriptwrap::checkfile $outfile] + lappend result [string match "*ERROR: label location errors*" $summary] + #sanity: the analysis actually ran (labels enumerated) + lappend result [string match "*call-labels-found:*" $summary] + }\ + -cleanup { + }\ + -result [list 0 1] +} +tcltest::cleanupTests ;#needed to produce test summary line. diff --git a/src/tests/modules/punk/mix/testsuites/scriptwrap/runtimecmd_checkfile.test b/src/tests/modules/punk/mix/testsuites/scriptwrap/runtimecmd_checkfile.test new file mode 100644 index 00000000..a12db07f --- /dev/null +++ b/src/tests/modules/punk/mix/testsuites/scriptwrap/runtimecmd_checkfile.test @@ -0,0 +1,62 @@ +package require tcltest + +package require Thread ;#punk::fileline textinfo (used by scriptwrap checkfile) calls thread::id +package require punk::mix::commandset::scriptwrap + +#Characterization of the punk MULTISHELL polyglot build machinery +#(punk::mix::commandset::scriptwrap::multishell + checkfile) - added 2026-07-10 at the +#user's direction: the polyglot technique is deliberately maintained despite its +#fragility, so the machinery that generates bin/*.cmd needs pins. +# +#Split 2026-07-19 (G-092 - parallel-floor reduction, test moved verbatim from +#multishell.test): this file guards the COMMITTED bin/runtime.cmd artifact (no wrap +#fixture needed). The other .test files in this directory hold the rest of the +#multishell/scriptwrap coverage - see src/tests/modules/AGENTS.md (punk/mix bullet). +# +#Covered here: +# - checkfile (the 512-byte label/boundary validator) reports no label location errors +# for the committed bin/runtime.cmd, and the artifact is LF-only - guarding against +# hand-edits or line-ending damage to the shipped artifact +# +#NOTE for agents: bin/*.cmd files are GENERATED - fix the payloads under src/scriptapps/ +#(.ps1/.bash/.tcl + _wrap.toml) and re-wrap; do not edit the polyglot output +#(see bin/AGENTS.md and src/scriptapps/AGENTS.md). + +namespace eval ::testspace { + namespace import ::tcltest::* + variable common { + set result "" + } + + variable testdir [file dirname [file normalize [info script]]] + #/src/tests/modules/punk/mix/testsuites/scriptwrap -> 7 levels up + variable projectroot [file normalize [file join $testdir .. .. .. .. .. .. ..]] + + proc readbytes {path} { + set fd [open $path r] + fconfigure $fd -translation binary + set data [read $fd] + close $fd + return $data + } + + test scriptwrap_checkfile_committed_runtime_cmd {the committed bin/runtime.cmd passes checkfile - no label/boundary damage from hand-edits or line-ending conversion}\ + -setup $common -body { + variable projectroot + set target [file join $projectroot bin runtime.cmd] + if {![file exists $target]} { + #not built/present in this checkout - treat as vacuous pass with marker + lappend result no_runtime_cmd + } else { + set data [readbytes $target] + #LF-only is part of the artifact's contract + lappend result [expr {[string first \r $data] == -1}] + set summary [punk::mix::commandset::scriptwrap::checkfile $target] + lappend result [string match "*ERROR: label location errors*" $summary] + } + }\ + -cleanup { + }\ + -result [list 1 0] +} +tcltest::cleanupTests ;#needed to produce test summary line. diff --git a/src/tests/modules/punk/mix/testsuites/scriptwrap/runtimecmd_roundtrip.test b/src/tests/modules/punk/mix/testsuites/scriptwrap/runtimecmd_roundtrip.test new file mode 100644 index 00000000..898fed10 --- /dev/null +++ b/src/tests/modules/punk/mix/testsuites/scriptwrap/runtimecmd_roundtrip.test @@ -0,0 +1,70 @@ +package require tcltest + +package require Thread ;#punk::fileline textinfo (used by scriptwrap machinery) calls thread::id +package require punk::mix::commandset::scriptwrap + +#Characterization of the punk MULTISHELL polyglot build machinery +#(punk::mix::commandset::scriptwrap::multishell + checkfile) - added 2026-07-10 at the +#user's direction: the polyglot technique is deliberately maintained despite its +#fragility, so the machinery that generates bin/*.cmd needs pins. +# +#Split 2026-07-19 (G-092 - parallel-floor reduction, test moved verbatim from +#multishell.test): this file holds the source-to-artifact round-trip pin (no wrap fixture +#needed - it wraps the real runtime scriptset from src/scriptapps/bin). The other .test +#files in this directory hold the rest of the multishell/scriptwrap coverage - see +#src/tests/modules/AGENTS.md (punk/mix bullet). +# +#Covered here: +# - re-wrapping the runtime scriptset from src/scriptapps/bin reproduces the committed +# bin/runtime.cmd byte for byte - the artifact is in sync with its sources and +# un-hand-edited +# +#NOTE for agents: bin/*.cmd files are GENERATED - fix the payloads under src/scriptapps/ +#(.ps1/.bash/.tcl + _wrap.toml) and re-wrap; do not edit the polyglot output +#(see bin/AGENTS.md and src/scriptapps/AGENTS.md). + +namespace eval ::testspace { + namespace import ::tcltest::* + variable common { + set result "" + } + + variable testdir [file dirname [file normalize [info script]]] + #/src/tests/modules/punk/mix/testsuites/scriptwrap -> 7 levels up + variable projectroot [file normalize [file join $testdir .. .. .. .. .. .. ..]] + variable startdir [pwd] + + proc readbytes {path} { + set fd [open $path r] + fconfigure $fd -translation binary + set data [read $fd] + close $fd + return $data + } + + test scriptwrap_runtime_cmd_roundtrip_no_drift {re-wrapping the runtime scriptset from src/scriptapps/bin reproduces the committed bin/runtime.cmd byte for byte - the artifact is in sync with its sources and un-hand-edited}\ + -setup $common -body { + variable projectroot + variable startdir + set target [file join $projectroot bin runtime.cmd] + set scriptapps [file join $projectroot src scriptapps bin] + if {![file exists $target] || ![file exists $scriptapps/runtime_wrap.toml]} { + lappend result no_runtime_sources + } else { + set outdir [makeDirectory scriptwrap_runtime_roundtrip] + cd $scriptapps + set res [punk::mix::commandset::scriptwrap::multishell runtime -outputfolder $outdir -askme 0 -force 1] + cd $startdir + set fresh [dict get $res filename] + lappend result [expr {[readbytes $fresh] eq [readbytes $target]}] + #a failure here means either bin/runtime.cmd was edited directly (fix the + #src/scriptapps payloads instead and re-wrap), or the payloads/template + #changed without regenerating bin/runtime.cmd (re-wrap and commit it) + } + }\ + -cleanup { + cd $startdir + }\ + -result [list 1] +} +tcltest::cleanupTests ;#needed to produce test summary line. diff --git a/src/tests/runtests.tcl b/src/tests/runtests.tcl index 62b98885..503b8d0b 100644 --- a/src/tests/runtests.tcl +++ b/src/tests/runtests.tcl @@ -850,15 +850,20 @@ if {$jobs_mode} { #on the reference machine); unlisted files weight 0 and keep discovery order among #themselves (lsort is stable). set weight_globs { - */multishell.test 85 - */argparsingtest.test 27 - */dtplite.test 25 - */scriptexec.test 10 - */shellexit.test 6 - */exec.test 5 - */staticruntime.test 3 - */runtimebash_wsl.test 3 - */shellnavns.test 2 + */runtimecmd_roundtrip.test 36 + */runtimecmd_checkfile.test 35 + */argparsingtest_compare.test 30 + */multishell_wrapverify.test 26 + */multishell_wrapdeterminism.test 26 + */dtplite.test 25 + */argparsingtest.test 22 + */multishell.test 20 + */scriptexec.test 10 + */shellexit.test 6 + */exec.test 5 + */staticruntime.test 3 + */runtimebash_wsl.test 3 + */shellnavns.test 2 } set decorated [list] foreach tf $parallel_files {