diff --git a/src/tests/AGENTS.md b/src/tests/AGENTS.md index 36071455..c59ad5de 100644 --- a/src/tests/AGENTS.md +++ b/src/tests/AGENTS.md @@ -17,6 +17,8 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc - `runtests.tcl` excludes `AGENTS.md` and `*.tcl` helper files when discovering `.test` files. - Tests should run against source modules and libraries from `src/`, not installed packages or root-level build outputs. - Test files must `package require` any extra packages explicitly. +- Agent-oriented runner output should use `-report compact -show-passes 0` for focused checks unless detailed Markdown pass lists are needed. +- `-report json` emits a machine-readable final summary, but package-load warnings may still precede it on stdout/stderr. ## Work Guidance @@ -24,12 +26,15 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc - Run the full source-tree suite with ` src/tests/runtests.tcl`. - Use `-include-paths` with paths relative to `src/tests/`, using forward slashes. - Use `-tcltestoptions {-match }` for focused single-test runs. +- For agent-efficient focused runs, prefer ` src/tests/runtests.tcl -report compact -show-passes 0 -include-paths `. +- Add `-slowest ` when timing outliers are relevant. +- Add `-strict-exit 1` when a nonzero shell exit code is needed for failures or parser warnings. - Capture enough stderr or failure context to identify the failing command or assertion. ## Verification - ` src/tests/runtests.tcl` passes when broad source-tree test coverage is relevant. -- Focused checks use ` src/tests/runtests.tcl -include-paths ` and optional file-tail globs. +- Focused checks use ` src/tests/runtests.tcl -report compact -show-passes 0 -include-paths ` and optional file-tail globs. - Documentation-only changes are verified by reviewing the affected DOX chain and diff. ## Child DOX Index diff --git a/src/tests/core/AGENTS.md b/src/tests/core/AGENTS.md index 81e61ddb..e4ac4930 100644 --- a/src/tests/core/AGENTS.md +++ b/src/tests/core/AGENTS.md @@ -1,31 +1,31 @@ -# src/tests/core — Tcl Core Compatibility Tests - -## Purpose - -Selected tests taken from or modeled after Tcl core tests. These tests should produce compatible results under a standard `tclsh` and an appropriate built Punk executable. - -## Ownership - -- Agents may update this subtree when changing compatibility expectations or test harness behavior for core Tcl commands. -- Preserve the intent of upstream Tcl compatibility coverage when editing imported or adapted tests. - -## Local Contracts - -- Tests under this subtree are compatibility tests, not general module behavior tests. -- Keep command behavior expectations explicit enough to compare standard Tcl and built Punk results. -- Avoid mixing source-module feature tests into this subtree; use `src/tests/modules/` instead. - -## Work Guidance - -- Use `src/tests/runtests.tcl` from the repository root for source-tree harness runs. -- Prefer focused `-include-paths core/**` checks when editing only this subtree. -- Document any deliberate divergence from standard Tcl behavior in the nearest relevant test file or child AGENTS.md. - -## Verification - -- ` src/tests/runtests.tcl -include-paths core/**` passes for changes in this subtree. -- Compare results with standard `tclsh` and the target Punk executable when compatibility behavior is the subject of the change. - -## Child DOX Index - -- `tcl/testsuites/tests/` — Imported/adapted Tcl core test files (see tcl/testsuites/tests/AGENTS.md) +# src/tests/core — Tcl Core Compatibility Tests + +## Purpose + +Selected tests taken from or modeled after Tcl core tests. These tests should produce compatible results under a standard `tclsh` and an appropriate built Punk executable. + +## Ownership + +- Agents may update this subtree when changing compatibility expectations or test harness behavior for core Tcl commands. +- Preserve the intent of upstream Tcl compatibility coverage when editing imported or adapted tests. + +## Local Contracts + +- Tests under this subtree are compatibility tests, not general module behavior tests. +- Keep command behavior expectations explicit enough to compare standard Tcl and built Punk results. +- Avoid mixing source-module feature tests into this subtree; use `src/tests/modules/` instead. + +## Work Guidance + +- Use `src/tests/runtests.tcl` from the repository root for source-tree harness runs. +- Prefer focused `-report compact -show-passes 0 -include-paths core/**` checks when editing only this subtree. +- Document any deliberate divergence from standard Tcl behavior in the nearest relevant test file or child AGENTS.md. + +## Verification + +- ` src/tests/runtests.tcl -report compact -show-passes 0 -include-paths core/**` passes for changes in this subtree. +- Compare results with standard `tclsh` and the target Punk executable when compatibility behavior is the subject of the change. + +## Child DOX Index + +- `tcl/testsuites/tests/` — Imported/adapted Tcl core test files (see tcl/testsuites/tests/AGENTS.md) diff --git a/src/tests/core/tcl/testsuites/tests/AGENTS.md b/src/tests/core/tcl/testsuites/tests/AGENTS.md index 39ed1486..f1f0421b 100644 --- a/src/tests/core/tcl/testsuites/tests/AGENTS.md +++ b/src/tests/core/tcl/testsuites/tests/AGENTS.md @@ -1,27 +1,27 @@ -# src/tests/core/tcl/testsuites/tests — Tcl Core Test Files - -## Purpose - -A selection of unit tests taken from or modeled after the Tcl core distribution. These tests, designed for a standard `tclsh`, should produce the same results for a suitable built Punk executable. - -## Ownership - -- Preserve Tcl core compatibility intent when editing tests in this directory. -- Keep broader core-test rules in `src/tests/core/AGENTS.md`. - -## Local Contracts - -- Tests here target Tcl command compatibility, not general Punk module behavior. -- Deliberate Punk divergences from standard Tcl behavior must be documented near the affected test. - -## Work Guidance - -- Use focused runs through `src/tests/runtests.tcl` with `-include-paths core/tcl/testsuites/tests/**`. - -## Verification - -- ` src/tests/runtests.tcl -include-paths core/tcl/testsuites/tests/**` passes for changes in this directory. - -## Child DOX Index - - +# src/tests/core/tcl/testsuites/tests — Tcl Core Test Files + +## Purpose + +A selection of unit tests taken from or modeled after the Tcl core distribution. These tests, designed for a standard `tclsh`, should produce the same results for a suitable built Punk executable. + +## Ownership + +- Preserve Tcl core compatibility intent when editing tests in this directory. +- Keep broader core-test rules in `src/tests/core/AGENTS.md`. + +## Local Contracts + +- Tests here target Tcl command compatibility, not general Punk module behavior. +- Deliberate Punk divergences from standard Tcl behavior must be documented near the affected test. + +## Work Guidance + +- Use focused runs through `src/tests/runtests.tcl` with `-report compact -show-passes 0 -include-paths core/tcl/testsuites/tests/**`. + +## Verification + +- ` src/tests/runtests.tcl -report compact -show-passes 0 -include-paths core/tcl/testsuites/tests/**` passes for changes in this directory. + +## Child DOX Index + + diff --git a/src/tests/modules/AGENTS.md b/src/tests/modules/AGENTS.md index 6bb01682..00989cbf 100644 --- a/src/tests/modules/AGENTS.md +++ b/src/tests/modules/AGENTS.md @@ -1,34 +1,36 @@ -# src/tests/modules — Source Module Test Suites - -## Purpose - -Unit tests for editable source modules under `src/modules/`, `src/modules_tcl8/`, and `src/modules_tcl9/`. These tests use `tcltest` and run against the uninstalled source tree through `src/tests/runtests.tcl`. - -## Ownership - -- Agents should add or update tests here when changing module behavior in `src/modules/` or Tcl-version-specific module trees. -- Installed-module test packages under `src/modules/test/` are packaging artifacts and are not the default place for source-tree behavior tests. -- Do not delete, skip, or weaken existing tests without explicit user direction. - -## Local Contracts - -- The directory hierarchy mirrors module namespace paths, such as `src/tests/modules/punk/args/` for `punk::args`. -- Test files use `.test` extension and must `package require` any extra packages explicitly. -- Test files may commonly use line-continuation backslashes, and existing Tcl expand-syntax structures in tests should not be refactored into line continuations. -- Tests must exercise source modules as loaded by `src/tests/runtests.tcl`, not installed or built output modules. - -## Work Guidance - -- Put new module tests under `src/tests/modules//testsuites//` following nearby layout. -- Prefer `try { ... } on error {result options} { ... }` for structured error capture in test bodies. -- Focused verification should mirror production pipelines inside tests using Punk pipeline syntax when that parity matters. -- Capture enough stderr or failure context to identify the failing command or assertion. - -## Verification - -- Run targeted module tests with ` src/tests/runtests.tcl -include-paths modules//**`. -- Run a single test by name with ` src/tests/runtests.tcl -tcltestoptions {-match } -include-paths modules//**`. -- Run a specific test file by adding its file tail, such as ` src/tests/runtests.tcl -include-paths modules/punk/args/** parsekey.test`. - -## Child DOX Index - +# src/tests/modules — Source Module Test Suites + +## Purpose + +Unit tests for editable source modules under `src/modules/`, `src/modules_tcl8/`, and `src/modules_tcl9/`. These tests use `tcltest` and run against the uninstalled source tree through `src/tests/runtests.tcl`. + +## Ownership + +- Agents should add or update tests here when changing module behavior in `src/modules/` or Tcl-version-specific module trees. +- Installed-module test packages under `src/modules/test/` are packaging artifacts and are not the default place for source-tree behavior tests. +- Do not delete, skip, or weaken existing tests without explicit user direction. + +## Local Contracts + +- The directory hierarchy mirrors module namespace paths, such as `src/tests/modules/punk/args/` for `punk::args`. +- Test files use `.test` extension and must `package require` any extra packages explicitly. +- Test files may commonly use line-continuation backslashes, and existing Tcl expand-syntax structures in tests should not be refactored into line continuations. +- Tests must exercise source modules as loaded by `src/tests/runtests.tcl`, not installed or built output modules. + +## Work Guidance + +- Put new module tests under `src/tests/modules//testsuites//` following nearby layout. +- For efficient agent runs, use `-report compact -show-passes 0` and include the narrowest namespace path plus file tail when known. +- Prefer `try { ... } on error {result options} { ... }` for structured error capture in test bodies. +- Focused verification should mirror production pipelines inside tests using Punk pipeline syntax when that parity matters. +- Capture enough stderr or failure context to identify the failing command or assertion. + +## Verification + +- Run targeted module tests with ` src/tests/runtests.tcl -report compact -show-passes 0 -include-paths modules//**`. +- Run a single test by name with ` src/tests/runtests.tcl -report compact -show-passes 0 -tcltestoptions {-match } -include-paths modules//**`. +- Run a specific test file by adding its file tail, such as ` src/tests/runtests.tcl -report compact -show-passes 0 -include-paths modules/punk/args/** parsekey.test`. +- Use `-report markdown+json` when both detailed human-readable failure sections and a structured summary are useful. + +## Child DOX Index + diff --git a/src/tests/modules/punk/path/testsuites/tests/path.test b/src/tests/modules/punk/path/testsuites/tests/path.test index 843e2936..faf4702a 100644 --- a/src/tests/modules/punk/path/testsuites/tests/path.test +++ b/src/tests/modules/punk/path/testsuites/tests/path.test @@ -471,9 +471,8 @@ namespace eval ::testspace { } expr {$results eq {1 1 1}} } \ - -constraints foobar \ -cleanup $treefilenames_cleanup \ - -result 1a + -result 1 } tcltest::cleanupTests ;#needed to produce test summary. \ No newline at end of file diff --git a/src/tests/runtests.tcl b/src/tests/runtests.tcl index 93eb6f1a..4dddc94f 100644 --- a/src/tests/runtests.tcl +++ b/src/tests/runtests.tcl @@ -1,359 +1,646 @@ -#!tclsh -#This script uses shellfilter::run calls under the hood -lassign [split [info tclversion] .] tcl_major tcl_minor -set test_base [file dirname [file normalize [info script]]] -set test_base_parent [file dirname $test_base] -if {[file tail $test_base_parent] eq "src"} { - set project_root [file dirname $test_base_parent] -} else { - set msg "Error: test script is not under a src/ directory: $test_base" - append msg \n "To run tests against the built modules, run src/make.tcl packages and then see the modules/test folder within this project" - puts stderr $msg - exit 2 -} - -#------------------------------------ -#For the toplevel script, use the bootsupport modules. -set original_tmlist [tcl::tm::list] -tcl::tm::remove {*}$original_tmlist -tcl::tm::add [file normalize $project_root/src/bootsupport/modules] ;#ie /src/modules -tcl::tm::add [file normalize $project_root/src/bootsupport/modules_tcl$tcl_major] -tcl::tm::add {*}[lreverse $original_tmlist] -package require platform - -set arch [platform::generic] - -set libdir [file normalize $project_root/src/bootsupport/lib] -if {$libdir ni $::auto_path} { - lappend ::auto_path $libdir -} -set libdir_arch [file normalize $project_root/src/bootsupport/lib/tcl$tcl_major/$arch] -if {$libdir_arch ni $::auto_path} { - lappend ::auto_path $libdir_arch -} -#------------------------------------ -package require punk -package require punk::args -package require shellrun -package require punk::tcltestrun - - - - - -#------------------------------------ -#for the tests running in child processes, -#use the unbuilt modules/libraries under development in preference to the installed versions. -set test_tmlist [list] -set ifneeded_script "" -lappend test_tmlist [file normalize $test_base/../modules] ;#ie /src/modules -append ifneeded_script [punk::tcltestrun::tm_path_additional_ifneeded [file normalize $test_base/../modules]] -lappend test_tmlist [file normalize $test_base/../modules_tcl$tcl_major] -append ifneeded_script [punk::tcltestrun::tm_path_additional_ifneeded [file normalize $test_base/../modules]] -lappend test_tmlist [file normalize $test_base/../bootsupport/modules] -set test_auto_path [list] -lappend test_auto_path [file normalize $test_base/../lib] -lappend test_auto_path [file normalize $test_base/../lib/tcl$tcl_major] -lappend test_auto_path [file normalize $test_base/../../lib_tcl$tcl_major/$arch] -lappend test_auto_path [info library] -#------------------------------------ - - -punk::args::define { - @id -id (script)::runtests - @cmd -name runtests\ - -summary\ - "Run the source test suites for this project."\ - -help\ - "Run the test suites for this project. - By default, all test files under src/tests/ will be included - - but you can specify particular test files or paths to include using the arguments below. - - These tests run against the unbuilt pure-tcl modules under development rather than any installed versions - - so you can test the latest code without installing it. - - To run tests against the installed versions, see the modules/test folder within this project or - package require the approprate test:: package and call its RUN function. - " - @opts - -tcltestoptions -type dict -default {-singleproc 1 -verbose {body pass skip start error line usec}} -help\ - "Pairs of flags/values that will be passed to tcltest::configure before running the tests. - The supplied flags will be merged with the default flags." - -show-raw-output -type boolean -default 0 -help\ - "If true, the raw stdout and stderr produced by tcltest from the test files will be shown in addition to the parsed results. - This will be in markdown code blocks under headings: - stdout from - and - stderr from - " - -include-paths -type list -default {**} -help\ - "List of glob patterns for paths. - Paths are relative to the test base and should use forward slashes as separators. - - Only test files under paths matching these patterns will be included. - For example, to only include test files under /src/modules/punk - -include-paths {modules/punk/**}" - @values -min 0 -max -1 - glob -type string -multiple 1 -optional 1 -help\ - "Names or glob patterns of test files to run. - This matches against the file tail - so should not include path segments. - The default if not supplied is a single *.test entry." -} - -if {"-h" in $::argv || "--help" in $::argv || "-help" in $::argv} { - puts stderr [punk::args::usage (script)::runtests] - exit 0 -} - -#puts "argv: $::argv" - -set argd [punk::args::parse $::argv withid (script)::runtests] -lassign [dict values $argd] leaders opts values received -set opt_tcltestoptions [dict get $opts -tcltestoptions] -set default_tcltestoptions {-singleproc 1 -verbose {body pass skip start error line usec}} - -set opt_show_raw_output [dict get $opts -show-raw-output] - - -set include_paths [dict get $opts -include-paths] - - -set tcltestoptions [dict merge $default_tcltestoptions $opt_tcltestoptions] - -if {[dict exists $tcltestoptions -file]} { - set file_globs [dict get $tcltestoptions -file] -} else { - #set file_globs [list *.test] - set file_globs [list] -} - -if {[dict exists $received glob]} { - lappend file_globs [dict get $values glob] -} -#fall back to default if still empty -if {[llength $file_globs] == 0} { - set file_globs [list *.test] -} - - -dict set tcltestoptions -file $file_globs - - -#puts "tcltestoptions: $tcltestoptions" -#puts "file_globs: $file_globs" -#puts "test_base: $test_base" - -set thisexecutable [info nameofexecutable] -#puts "executable: $thisexecutable" - -set exclude_files [list AGENTS.md *.tcl] - - -set tallydict [dict create] -dict set tallydict total 0 -dict set tallydict passed 0 -dict set tallydict skipped 0 -dict set tallydict failed 0 -dict set tallydict pkgs_with_fails [list] -dict set tallydict pkgs_without_fails [list] -dict set tallydict pkgs_with_warnings [list] - -set testfiles [punk::path::treefilenames -dir $test_base -tailbase $test_base -exclude-files $exclude_files -include-paths $include_paths $file_globs] - -if {[dict exists $tcltestoptions -singleproc] && [dict get $tcltestoptions -singleproc]} { - puts "Running tests in single process mode" - set singleproc 1 -} else { - puts "Running tests in multiple processes" - set singleproc 0 -} - -puts "---------------" -puts "test auto_path: $test_auto_path" -puts "---------------" -puts "tcltestoptions: $tcltestoptions" -foreach testfile_relative $testfiles { - set testfile [file normalize [file join $test_base $testfile_relative]] - puts stdout "" - #puts stdout "running test file $testfile" - puts stdout "## runtests $testfile_relative" - puts stdout "" - - dict set tcltestoptions -testdir [file dirname $testfile] - - if {$singleproc} { - #in single process mode we can just source the test file in an interp - but we need to set up the environment for the test file - such as the module search path to ensure it picks up the unbuilt modules under development rather than any installed versions. - interp create testinterp - testinterp eval {tcl::tm::remove [tcl::tm::list]} - testinterp eval [list tcl::tm::add {*}$test_tmlist] - testinterp eval [list set ::auto_path $test_auto_path] - testinterp eval $ifneeded_script - testinterp eval [list set ::argv $tcltestoptions] - testinterp eval [list set ::argc [llength $tcltestoptions]] - testinterp eval {package require tcltest} - testinterp eval [list tcltest::configure {*}$tcltestoptions] - testinterp eval {package require shellrun} - #puts ">>>> [testinterp eval { - # set result "" - # foreach key [tcltest::configure] { - # append result "$key: [tcltest::configure $key]\n" - # } - # return $result - # }] <<<<" - set result [testinterp eval [list shellrun::runx -tcl source $testfile]] - interp delete testinterp - puts stdout "sourced $testfile " - flush stdout - } else { - #todo - we need to set up the environment for the test file - such as the module search path to ensure it picks up the unbuilt modules under development rather than any installed versions. - #we also need to pass the tcltestoptions to the test file. - #one way would be to prepend this data to the testfiledata in a temporary file and then run that temporary file. - #Another way might be to pipe the whole script through stdin to the child process - but that may cause issues with tcltest's use of stdin for user input. - if {[catch { - set result [shellrun::runx $thisexecutable $testfile] - #set result [shellrun::runx ls] - } errM]} { - puts stderr "error calling 'runout $thisexecutable $testfile' $errM"; flush stderr - set result {none ""} - } - puts stdout "executed $thisexecutable $testfile " - puts stdout "result keys: [dict keys $result] " - flush stdout - } - - if {$opt_show_raw_output} { - puts stdout "" - puts stdout "### stdout from $testfile_relative" - puts stdout "" - puts stdout "```" - puts stdout [dict get $result stdout] - puts stdout "```" - puts stdout "" - puts stdout "### stderr from $testfile_relative" - puts stdout "" - puts stdout "```" - puts stdout [dict get $result stderr] - puts stdout "```" - puts stdout "" - } - - if {[dict exists $result error]} { - puts stderr "error running test file $testfile: [dict get $result error]" - flush stderr - dict lappend tallydict pkgs_with_fails $testfile - } else { - set resultdict [punk::tcltestrun::parse_testrun $result $testfile] - if {[dict get $resultdict summaryline_detected] == 0} { - puts stderr "WARNING: No results parsed from test file $testfile - test file may be missing tcltest::cleanupTests call" - dict lappend tallydict pkgs_with_warnings $testfile - } elseif {[dict get $resultdict summaryline_detected] > 1} { - puts stderr "WARNING: More than one summary line detected in test file $testfile - test file may be calling tcltest::cleanupTests multiple times" - dict lappend tallydict pkgs_with_warnings $testfile - } else { - dict incr tallydict total [dict get $resultdict summaryline_total] - dict incr tallydict passed [dict get $resultdict summaryline_passed] - dict incr tallydict skipped [dict get $resultdict summaryline_skipped] ;#more than just skipped due to constraints - also includes skipped due to -skip and -match values. - dict incr tallydict failed [dict get $resultdict summaryline_failed] - #puts stdout [dict get $resultdict out] - if {[dict get $resultdict summaryline_failed] > 0} { - puts stdout "test file $testfile had failures" - dict lappend tallydict pkgs_with_fails $testfile - } else { - puts stdout "test file $testfile passed" - dict lappend tallydict pkgs_without_fails $testfile - } - } - - - #puts stdout "resultdict: $resultdict" - puts stdout "" - puts stdout "### testcase passes for $testfile" - puts stdout "" - dict for {testname testdict} [dict get $resultdict testcase_passes] { - puts stdout "- testname: $testname " - if {[dict exists $testdict microseconds]} { - puts stdout " microseconds: [dict get $testdict microseconds] " - } - } - puts stdout "" - - - puts stdout "### testcase failures for $testfile" - puts stdout "" - dict for {testname testdict} [dict get $resultdict testcase_fails] { - puts stdout "- testname: $testname " - puts stdout " test_description: [dict get $testdict test_description] " - puts stdout " test_body : " - puts stdout " ```tcl" - puts stdout " [dict get $testdict test_body]" - puts stdout " ```" - puts stdout " test_status : [dict get $testdict test_status] " - if {[dict get $testdict test_status] eq "ERROR"} { - puts stdout " returncode : [dict get $testdict returncode] " - puts stdout " errorcode : [dict get $testdict errorcode] " - } - } - puts stdout "" - puts stdout "### testcase_constraintskips" - puts stdout "" - dict for {testname testdict} [dict get $resultdict testcase_constraintskips] { - puts stdout "- testname: $testname " - puts stdout " skipreason: [dict get $testdict reason] " - } - puts -nonewline stdout "\n" - flush stdout - } - - set i 0 -} -puts stdout "## runtests summary" -puts stdout "" - -puts stdout "### testfiles without failures" -puts stdout "" -foreach pkg [dict get $tallydict pkgs_without_fails] { - puts stdout " - $pkg" -} -puts stdout "" - -puts stdout "### testfiles with failures" -puts stdout "" -foreach pkg [dict get $tallydict pkgs_with_fails] { - puts stdout " - $pkg" -} -puts stdout "" - -puts stdout "### testfiles with warnings" -puts stdout "" -foreach pkg [dict get $tallydict pkgs_with_warnings] { - puts stdout " - $pkg" -} -puts stdout "" - -puts stdout "### runtests tally" -puts stdout "" -puts stdout "Total [dict get $tallydict total] Passed [dict get $tallydict passed] Skipped [dict get $tallydict skipped] Failed [dict get $tallydict failed] " -puts stdout "" -puts stdout "runtests DONE " - -exit 0 - -##don't package require tcltest too early or it may examine and respond to ::argv itself. (e.g to respond to --help, but we have our own help) -#package require tcltest -# -#set ::argv $tcltestoptions -#set ::argc [llength $tcltestoptions] -##set ::argv {} -##set ::argc 0 -# -#tcltest::configure -verbose "body pass skip error usec" -#tcltest::configure -testdir $script_dir -#tcltest::configure -file $file_globs -##review - single process has less isolation - but works better in this case. -##(some tclsh shells can hang when running with -singleproc false - needs investigation) -##tclte::configure -singleproc true -#tcltest::configure -singleproc true -#dict for {k v} $tcltestoptions { -# tcltest::configure $k $v -#} -#tcltest::runAllTests \ No newline at end of file +#!tclsh +#This script uses shellfilter::run calls under the hood +lassign [split [info tclversion] .] tcl_major tcl_minor +set test_base [file dirname [file normalize [info script]]] +set test_base_parent [file dirname $test_base] +if {[file tail $test_base_parent] eq "src"} { + set project_root [file dirname $test_base_parent] +} else { + set msg "Error: test script is not under a src/ directory: $test_base" + append msg \n "To run tests against the built modules, run src/make.tcl packages and then see the modules/test folder within this project" + puts stderr $msg + exit 2 +} + +#------------------------------------ +#For the toplevel script, use the bootsupport modules. +set original_tmlist [tcl::tm::list] +tcl::tm::remove {*}$original_tmlist +tcl::tm::add [file normalize $project_root/src/bootsupport/modules] ;#ie /src/modules +tcl::tm::add [file normalize $project_root/src/bootsupport/modules_tcl$tcl_major] +tcl::tm::add {*}[lreverse $original_tmlist] +package require platform + +set arch [platform::generic] + +set libdir [file normalize $project_root/src/bootsupport/lib] +if {$libdir ni $::auto_path} { + lappend ::auto_path $libdir +} +set libdir_arch [file normalize $project_root/src/bootsupport/lib/tcl$tcl_major/$arch] +if {$libdir_arch ni $::auto_path} { + lappend ::auto_path $libdir_arch +} +#------------------------------------ +package require punk +package require punk::args +package require shellrun +package require punk::tcltestrun + +proc runtests_json_string {text} { + set text [string map [list \\ \\\\ \" \\\" \n \\n \r \\r \t \\t] $text] + return "\"$text\"" +} + +proc runtests_json_string_array {values} { + set parts [list] + foreach value $values { + lappend parts [runtests_json_string $value] + } + return "\[[join $parts ,]\]" +} + +proc runtests_json_testdict_array {testdicts} { + set parts [list] + foreach testdict $testdicts { + set fields [list] + foreach {key value} $testdict { + if {[string is entier -strict $value]} { + lappend fields "[runtests_json_string $key]:$value" + } else { + lappend fields "[runtests_json_string $key]:[runtests_json_string $value]" + } + } + lappend parts "{[join $fields ,]}" + } + return "\[[join $parts ,]\]" +} + +proc runtests_json_report {status tallydict file_summaries slowest_tests} { + set files_failed [dict get $tallydict files_with_fails] + set files_warned [dict get $tallydict files_with_warnings] + set fields [list] + lappend fields "\"runner\":\"src/tests/runtests.tcl\"" + lappend fields "\"status\":[runtests_json_string $status]" + lappend fields "\"total\":[dict get $tallydict total]" + lappend fields "\"passed\":[dict get $tallydict passed]" + lappend fields "\"skipped\":[dict get $tallydict skipped]" + lappend fields "\"failed\":[dict get $tallydict failed]" + lappend fields "\"warnings\":[llength $files_warned]" + lappend fields "\"files\":{\"total\":[llength $file_summaries],\"failed\":[runtests_json_string_array $files_failed],\"warnings\":[runtests_json_string_array $files_warned]}" + + set file_parts [list] + foreach file_summary $file_summaries { + set file_fields [list] + lappend file_fields "\"path\":[runtests_json_string [dict get $file_summary path]]" + lappend file_fields "\"status\":[runtests_json_string [dict get $file_summary status]]" + lappend file_fields "\"total\":[dict get $file_summary total]" + lappend file_fields "\"passed\":[dict get $file_summary passed]" + lappend file_fields "\"skipped\":[dict get $file_summary skipped]" + lappend file_fields "\"failed\":[dict get $file_summary failed]" + lappend file_fields "\"summaryline_detected\":[dict get $file_summary summaryline_detected]" + lappend file_fields "\"failures\":[runtests_json_testdict_array [dict get $file_summary failures]]" + lappend file_fields "\"skips\":[runtests_json_testdict_array [dict get $file_summary skips]]" + lappend file_parts "{[join $file_fields ,]}" + } + lappend fields "\"testfiles\":\[[join $file_parts ,]\]" + lappend fields "\"slowest\":[runtests_json_testdict_array $slowest_tests]" + return "{[join $fields ,]}" +} + +proc runtests_status {tallydict} { + if {[dict get $tallydict failed] > 0 || [llength [dict get $tallydict files_with_fails]] > 0} { + return fail + } + if {[llength [dict get $tallydict files_with_warnings]] > 0} { + return warn + } + return pass +} + +proc runtests_emit_result_line {status tallydict file_summaries} { + puts stdout "RUNTESTS_RESULT status=$status total=[dict get $tallydict total] passed=[dict get $tallydict passed] skipped=[dict get $tallydict skipped] failed=[dict get $tallydict failed] warnings=[llength [dict get $tallydict files_with_warnings]] files=[llength $file_summaries] files_failed=[llength [dict get $tallydict files_with_fails]]" +} + +proc runtests_failure_summaries {resultdict} { + set failures [list] + dict for {testname testdict} [dict get $resultdict testcase_fails] { + set failure [dict create name $testname status [dict get $testdict test_status]] + if {[dict exists $testdict errorcode]} { + dict set failure errorcode [dict get $testdict errorcode] + } + lappend failures $failure + } + return $failures +} + +proc runtests_skip_summaries {resultdict} { + set skips [list] + dict for {testname testdict} [dict get $resultdict testcase_constraintskips] { + lappend skips [dict create name $testname reason [dict get $testdict reason]] + } + return $skips +} + +proc runtests_pass_summaries {testfile_relative resultdict} { + set passes [list] + dict for {testname testdict} [dict get $resultdict testcase_passes] { + set pass [dict create file $testfile_relative name $testname] + if {[dict exists $testdict microseconds]} { + dict set pass microseconds [dict get $testdict microseconds] + } + lappend passes $pass + } + return $passes +} + +proc runtests_print_failure_details {testfile_relative resultdict} { + dict for {testname testdict} [dict get $resultdict testcase_fails] { + puts stdout "- testname: $testname " + puts stdout " file : $testfile_relative " + puts stdout " test_description: [dict get $testdict test_description] " + puts stdout " test_body : " + puts stdout " ```tcl" + puts stdout " [dict get $testdict test_body]" + puts stdout " ```" + puts stdout " test_status : [dict get $testdict test_status] " + if {[dict get $testdict test_status] eq "ERROR"} { + puts stdout " returncode : [dict get $testdict returncode] " + puts stdout " errorcode : [dict get $testdict errorcode] " + } + } +} + +proc runtests_compare_microseconds_desc {left right} { + set left_usec 0 + set right_usec 0 + if {[dict exists $left microseconds]} { + set left_usec [dict get $left microseconds] + } + if {[dict exists $right microseconds]} { + set right_usec [dict get $right microseconds] + } + if {$left_usec > $right_usec} { + return -1 + } + if {$left_usec < $right_usec} { + return 1 + } + return 0 +} + +proc runtests_slowest_tests {passes count} { + if {$count <= 0} { + return [list] + } + set sorted [lsort -command runtests_compare_microseconds_desc $passes] + return [lrange $sorted 0 [expr {$count - 1}]] +} + + + + + +#------------------------------------ +#for the tests running in child processes, +#use the unbuilt modules/libraries under development in preference to the installed versions. +set test_tmlist [list] +set ifneeded_script "" +lappend test_tmlist [file normalize $test_base/../modules] ;#ie /src/modules +append ifneeded_script [punk::tcltestrun::tm_path_additional_ifneeded [file normalize $test_base/../modules]] +lappend test_tmlist [file normalize $test_base/../modules_tcl$tcl_major] +append ifneeded_script [punk::tcltestrun::tm_path_additional_ifneeded [file normalize $test_base/../modules]] +lappend test_tmlist [file normalize $test_base/../bootsupport/modules] +set test_auto_path [list] +lappend test_auto_path [file normalize $test_base/../lib] +lappend test_auto_path [file normalize $test_base/../lib/tcl$tcl_major] +lappend test_auto_path [file normalize $test_base/../../lib_tcl$tcl_major/$arch] +lappend test_auto_path [info library] +#------------------------------------ + + +punk::args::define { + @id -id (script)::runtests + @cmd -name runtests\ + -summary\ + "Run the source test suites for this project."\ + -help\ + "Run the test suites for this project. + By default, all test files under src/tests/ will be included + - but you can specify particular test files or paths to include using the arguments below. + + These tests run against the unbuilt pure-tcl modules under development rather than any installed versions + - so you can test the latest code without installing it. + + To run tests against the installed versions, see the modules/test folder within this project or + package require the approprate test:: package and call its RUN function. + " + @opts + -tcltestoptions -type dict -default {-singleproc 1 -verbose {body pass skip start error line usec}} -help\ + "Pairs of flags/values that will be passed to tcltest::configure before running the tests. + The supplied flags will be merged with the default flags." + -show-raw-output -type boolean -default 0 -help\ + "If true, the raw stdout and stderr produced by tcltest from the test files will be shown in addition to the parsed results. + This will be in markdown code blocks under headings: + stdout from + and + stderr from + " + -include-paths -type list -default {**} -help\ + "List of glob patterns for paths. + Paths are relative to the test base and should use forward slashes as separators. + + Only test files under paths matching these patterns will be included. + For example, to only include test files under /src/modules/punk + -include-paths {modules/punk/**}" + -report -type string -default markdown -choices {markdown compact json markdown+json} -help\ + "Output report style. markdown preserves the detailed human-readable report. + compact prints a short Markdown report with failures, warnings, and optional slow tests. + json prints a machine-readable JSON summary. + markdown+json prints the detailed Markdown report plus a fenced JSON summary." + -show-passes -type boolean -default 1 -help\ + "If true, include per-test pass lists in markdown reports. Use false for shorter agent-oriented output." + -show-timings -type boolean -default 1 -help\ + "If true, include microsecond timings where available." + -slowest -type int -default 0 -help\ + "Number of slowest passing tests to include in compact and JSON-oriented summaries. Use 0 to suppress." + -strict-exit -type boolean -default 0 -help\ + "If true, exit 1 when tests fail and exit 2 when files produce parser warnings." + @values -min 0 -max -1 + glob -type string -multiple 1 -optional 1 -help\ + "Names or glob patterns of test files to run. + This matches against the file tail - so should not include path segments. + The default if not supplied is a single *.test entry." +} + +if {"-h" in $::argv || "--help" in $::argv || "-help" in $::argv} { + puts stderr [punk::args::usage (script)::runtests] + exit 0 +} + +set ts_start [clock seconds] +#puts "argv: $::argv" + +set argd [punk::args::parse $::argv withid (script)::runtests] +lassign [dict values $argd] leaders opts values received +set opt_tcltestoptions [dict get $opts -tcltestoptions] +set default_tcltestoptions {-singleproc 1 -verbose {body pass skip start error line usec}} + +set opt_show_raw_output [dict get $opts -show-raw-output] +set opt_report [dict get $opts -report] +set opt_show_passes [dict get $opts -show-passes] +set opt_show_timings [dict get $opts -show-timings] +set opt_slowest [dict get $opts -slowest] +set opt_strict_exit [dict get $opts -strict-exit] +set report_detailed_markdown [expr {$opt_report in {markdown markdown+json}}] +set report_compact [expr {$opt_report eq "compact"}] +set report_json_only [expr {$opt_report eq "json"}] +set report_emit_json [expr {$opt_report in {json markdown+json}}] + + +set include_paths [dict get $opts -include-paths] + + +set tcltestoptions [dict merge $default_tcltestoptions $opt_tcltestoptions] + +if {[dict exists $tcltestoptions -file]} { + set file_globs [dict get $tcltestoptions -file] +} else { + #set file_globs [list *.test] + set file_globs [list] +} + +if {[dict exists $received glob]} { + lappend file_globs [dict get $values glob] +} +#fall back to default if still empty +if {[llength $file_globs] == 0} { + set file_globs [list *.test] +} + + +dict set tcltestoptions -file $file_globs + + +#puts "tcltestoptions: $tcltestoptions" +#puts "file_globs: $file_globs" +set thisexecutable [info nameofexecutable] + +if {!$report_json_only} { + puts "" + puts "# runtests.tcl started at [clock format $ts_start -format "%Y-%m-%d %H:%M:%S"]" + puts "" + puts "test_base: $test_base " + puts "executable: $thisexecutable " + puts "" +} + +set exclude_files [list AGENTS.md *.tcl] + + +set tallydict [dict create] +dict set tallydict total 0 +dict set tallydict passed 0 +dict set tallydict skipped 0 +dict set tallydict failed 0 +dict set tallydict files_with_fails [list] +dict set tallydict files_without_fails [list] +dict set tallydict files_with_warnings [list] +set file_summaries [list] +set all_passes [list] + +set testfiles [punk::path::treefilenames -dir $test_base -tailbase $test_base -exclude-files $exclude_files -include-paths $include_paths $file_globs] + +if {[dict exists $tcltestoptions -singleproc] && [dict get $tcltestoptions -singleproc]} { + if {!$report_json_only} { + puts "Running tests in single process mode" + } + set singleproc 1 +} else { + if {!$report_json_only} { + puts "Running tests in multiple processes" + } + set singleproc 0 +} + +if {!$report_json_only} { + puts "test auto_path: $test_auto_path" + puts "test tmlist: $test_tmlist" + puts "tcltestoptions: $tcltestoptions" +} +foreach testfile_relative $testfiles { + set testfile [file normalize [file join $test_base $testfile_relative]] + if {$report_detailed_markdown} { + puts stdout "" + #puts stdout "running test file $testfile" + puts stdout "## runtests $testfile_relative" + puts stdout "" + } + + dict set tcltestoptions -testdir [file dirname $testfile] + + if {$singleproc} { + #in single process mode we can just source the test file in an interp - but we need to set up the environment for the test file - such as the module search path to ensure it picks up the unbuilt modules under development rather than any installed versions. + interp create testinterp + testinterp eval {tcl::tm::remove [tcl::tm::list]} + testinterp eval [list tcl::tm::add {*}$test_tmlist] + testinterp eval [list set ::auto_path $test_auto_path] + testinterp eval $ifneeded_script + testinterp eval [list set ::argv $tcltestoptions] + testinterp eval [list set ::argc [llength $tcltestoptions]] + testinterp eval {package require tcltest} + testinterp eval [list tcltest::configure {*}$tcltestoptions] + testinterp eval {package require shellrun} + #puts ">>>> [testinterp eval { + # set result "" + # foreach key [tcltest::configure] { + # append result "$key: [tcltest::configure $key]\n" + # } + # return $result + # }] <<<<" + set result [testinterp eval [list shellrun::runx -tcl source $testfile]] + interp delete testinterp + if {$report_detailed_markdown} { + puts stdout "sourced $testfile " + } + flush stdout + } else { + #todo - we need to set up the environment for the test file - such as the module search path to ensure it picks up the unbuilt modules under development rather than any installed versions. + #we also need to pass the tcltestoptions to the test file. + #one way would be to prepend this data to the testfiledata in a temporary file and then run that temporary file. + #Another way might be to pipe the whole script through stdin to the child process - but that may cause issues with tcltest's use of stdin for user input. + if {[catch { + set result [shellrun::runx $thisexecutable $testfile] + #set result [shellrun::runx ls] + } errM]} { + puts stderr "error calling 'runout $thisexecutable $testfile' $errM"; flush stderr + set result {none ""} + } + if {$report_detailed_markdown} { + puts stdout "executed $thisexecutable $testfile " + puts stdout "result keys: [dict keys $result] " + } + flush stdout + } + + if {$opt_show_raw_output} { + puts stdout "" + puts stdout "### stdout from $testfile_relative" + puts stdout "" + puts stdout "```" + puts stdout [dict get $result stdout] + puts stdout "```" + puts stdout "" + puts stdout "### stderr from $testfile_relative" + puts stdout "" + puts stdout "```" + puts stdout [dict get $result stderr] + puts stdout "```" + puts stdout "" + } + + if {[dict exists $result error]} { + puts stderr "error running test file $testfile: [dict get $result error]" + flush stderr + dict lappend tallydict files_with_fails $testfile_relative + lappend file_summaries [dict create path $testfile_relative status error total 0 passed 0 skipped 0 failed 1 summaryline_detected 0 failures [list [dict create name $testfile_relative status ERROR errorcode [dict get $result error]]] skips [list]] + } else { + set resultdict [punk::tcltestrun::parse_testrun $result $testfile] + set file_status pass + if {[dict get $resultdict summaryline_detected] == 0} { + puts stderr "WARNING: No results parsed from test file $testfile - test file may be missing tcltest::cleanupTests call" + dict lappend tallydict files_with_warnings $testfile_relative + set file_status warning + } elseif {[dict get $resultdict summaryline_detected] > 1} { + puts stderr "WARNING: More than one summary line detected in test file $testfile - test file may be calling tcltest::cleanupTests multiple times" + dict lappend tallydict files_with_warnings $testfile_relative + set file_status warning + } else { + dict incr tallydict total [dict get $resultdict summaryline_total] + dict incr tallydict passed [dict get $resultdict summaryline_passed] + dict incr tallydict skipped [dict get $resultdict summaryline_skipped] ;#more than just skipped due to constraints - also includes skipped due to -skip and -match values. + dict incr tallydict failed [dict get $resultdict summaryline_failed] + #puts stdout [dict get $resultdict out] + if {[dict get $resultdict summaryline_failed] > 0} { + if {!$report_json_only} { + puts stdout "test file $testfile_relative had failures" + } + dict lappend tallydict files_with_fails $testfile_relative + set file_status fail + } else { + if {$report_detailed_markdown} { + puts stdout "test file $testfile_relative passed" + } + dict lappend tallydict files_without_fails $testfile_relative + } + } + + set failures [runtests_failure_summaries $resultdict] + set skips [runtests_skip_summaries $resultdict] + set passes [runtests_pass_summaries $testfile_relative $resultdict] + set all_passes [concat $all_passes $passes] + lappend file_summaries [dict create {*}{ + } path $testfile_relative {*}{ + } status $file_status {*}{ + } total [dict get $resultdict summaryline_total] {*}{ + } passed [dict get $resultdict summaryline_passed] {*}{ + } skipped [dict get $resultdict summaryline_skipped] {*}{ + } failed [dict get $resultdict summaryline_failed] {*}{ + } summaryline_detected [dict get $resultdict summaryline_detected] {*}{ + } failures $failures {*}{ + } skips $skips {*}{ + } + ] + + + #puts stdout "resultdict: $resultdict" + if {$report_detailed_markdown} { + if {$opt_show_passes} { + puts stdout "" + puts stdout "### testcase passes for $testfile_relative" + puts stdout "" + dict for {testname testdict} [dict get $resultdict testcase_passes] { + puts stdout "- testname: $testname " + if {$opt_show_timings && [dict exists $testdict microseconds]} { + puts stdout " microseconds: [dict get $testdict microseconds] " + } + } + puts stdout "" + } + + + puts stdout "### testcase failures for $testfile_relative" + puts stdout "" + runtests_print_failure_details $testfile_relative $resultdict + puts stdout "" + puts stdout "### testcase_constraintskips for $testfile_relative" + puts stdout "" + dict for {testname testdict} [dict get $resultdict testcase_constraintskips] { + puts stdout "- testname: $testname " + puts stdout " skipreason: [dict get $testdict reason] " + } + puts -nonewline stdout "\n" + } + flush stdout + } + + set i 0 +} +set status [runtests_status $tallydict] +set slowest_tests [runtests_slowest_tests $all_passes $opt_slowest] + +if {!$report_json_only} { + puts stdout "## runtests summary" + puts stdout "" + runtests_emit_result_line $status $tallydict $file_summaries + puts stdout "" +} + +if {$report_compact} { + puts stdout "### testfiles" + puts stdout "" + foreach file_summary $file_summaries { + set line "- [dict get $file_summary status] `[dict get $file_summary path]` total=[dict get $file_summary total] passed=[dict get $file_summary passed] skipped=[dict get $file_summary skipped] failed=[dict get $file_summary failed]" + puts stdout $line + } + puts stdout "" + puts stdout "### failures" + puts stdout "" + foreach file_summary $file_summaries { + foreach failure [dict get $file_summary failures] { + set line "- `[dict get $file_summary path]` [dict get $failure name] status=[dict get $failure status]" + if {[dict exists $failure errorcode] && [dict get $failure errorcode] ne ""} { + append line " errorcode=[dict get $failure errorcode]" + } + puts stdout $line + } + } + puts stdout "" + puts stdout "### warnings" + puts stdout "" + foreach f [dict get $tallydict files_with_warnings] { + puts stdout "- $f" + } + puts stdout "" + if {$opt_slowest > 0} { + puts stdout "### slowest passing tests" + puts stdout "" + foreach testdict $slowest_tests { + set line "- `[dict get $testdict file]` [dict get $testdict name]" + if {$opt_show_timings && [dict exists $testdict microseconds]} { + append line " microseconds=[dict get $testdict microseconds]" + } + puts stdout $line + } + puts stdout "" + } +} + +if {$report_detailed_markdown} { + puts stdout "### testfiles without failures" + puts stdout "" + foreach f [dict get $tallydict files_without_fails] { + puts stdout " - $f" + } + puts stdout "" + + puts stdout "### testfiles with failures" + puts stdout "" + foreach f [dict get $tallydict files_with_fails] { + puts stdout " - $f" + } + puts stdout "" + + puts stdout "### testfiles with warnings" + puts stdout "" + foreach f [dict get $tallydict files_with_warnings] { + puts stdout " - $f" + } + puts stdout "" +} + +if {!$report_json_only} { + puts stdout "### runtests tally" + puts stdout "" + puts stdout "Total [dict get $tallydict total] Passed [dict get $tallydict passed] Skipped [dict get $tallydict skipped] Failed [dict get $tallydict failed] " + puts stdout "" +} + +if {$report_emit_json} { + if {!$report_json_only} { + puts stdout "### runtests json" + puts stdout "" + puts stdout "```json" + } + puts stdout [runtests_json_report $status $tallydict $file_summaries $slowest_tests] + if {!$report_json_only} { + puts stdout "```" + puts stdout "" + } +} + +if {!$report_json_only} { + puts stdout "### runtests DONE at [clock format [clock seconds] -format "%Y-%m-%d %H:%M:%S"]" + puts stdout "" +} + +if {$opt_strict_exit} { + if {[dict get $tallydict failed] > 0 || [llength [dict get $tallydict files_with_fails]] > 0} { + exit 1 + } + if {[llength [dict get $tallydict files_with_warnings]] > 0} { + exit 2 + } +} + +exit 0 + +##don't package require tcltest too early or it may examine and respond to ::argv itself. (e.g to respond to --help, but we have our own help) +#package require tcltest +# +#set ::argv $tcltestoptions +#set ::argc [llength $tcltestoptions] +##set ::argv {} +##set ::argc 0 +# +#tcltest::configure -verbose "body pass skip error usec" +#tcltest::configure -testdir $script_dir +#tcltest::configure -file $file_globs +##review - single process has less isolation - but works better in this case. +##(some tclsh shells can hang when running with -singleproc false - needs investigation) +##tclte::configure -singleproc true +#tcltest::configure -singleproc true +#dict for {k v} $tcltestoptions { +# tcltest::configure $k $v +#} +#tcltest::runAllTests diff --git a/src/tests/shell/AGENTS.md b/src/tests/shell/AGENTS.md index ad80e783..8191e561 100644 --- a/src/tests/shell/AGENTS.md +++ b/src/tests/shell/AGENTS.md @@ -1,29 +1,29 @@ -# src/tests/shell — Shell Behavior Tests - -## Purpose - -Tests for shell-level behavior, command-line execution, and stdin/stdout interaction that are not owned by a specific source module namespace. - -## Ownership - -- Agents may update this subtree when changing shell behavior, command execution, or interactive stream handling. -- Keep module API behavior tests in `src/tests/modules/` unless shell invocation is the behavior under test. - -## Local Contracts - -- Tests here may exercise process execution, stdin, stdout, stderr, and shell filtering behavior. -- Keep command inputs and expected outputs explicit so failures can be reproduced outside the aggregate runner. -- Avoid relying on installed packages or root-level build outputs unless the test explicitly targets a built executable. - -## Work Guidance - -- Use focused harness runs with `-include-paths shell/**` when editing this subtree. -- Include enough failure output to identify the launched command and stream comparison. -- Keep platform-sensitive assumptions visible in the test body. - -## Verification - -- ` src/tests/runtests.tcl -include-paths shell/**` passes for changes in this subtree. - -## Child DOX Index - +# src/tests/shell — Shell Behavior Tests + +## Purpose + +Tests for shell-level behavior, command-line execution, and stdin/stdout interaction that are not owned by a specific source module namespace. + +## Ownership + +- Agents may update this subtree when changing shell behavior, command execution, or interactive stream handling. +- Keep module API behavior tests in `src/tests/modules/` unless shell invocation is the behavior under test. + +## Local Contracts + +- Tests here may exercise process execution, stdin, stdout, stderr, and shell filtering behavior. +- Keep command inputs and expected outputs explicit so failures can be reproduced outside the aggregate runner. +- Avoid relying on installed packages or root-level build outputs unless the test explicitly targets a built executable. + +## Work Guidance + +- Use focused harness runs with `-report compact -show-passes 0 -include-paths shell/**` when editing this subtree. +- Include enough failure output to identify the launched command and stream comparison. +- Keep platform-sensitive assumptions visible in the test body. + +## Verification + +- ` src/tests/runtests.tcl -report compact -show-passes 0 -include-paths shell/**` passes for changes in this subtree. + +## Child DOX Index +