diff --git a/src/tests/AGENTS.md b/src/tests/AGENTS.md index 63c1482a..579d646d 100644 --- a/src/tests/AGENTS.md +++ b/src/tests/AGENTS.md @@ -30,6 +30,7 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc - tcltest compares the `-body` RETURN VALUE against `-result`. The suite convention of accumulating into `$result` via `lappend` works because `lappend` returns the list — but a body whose last command is a loop (`foreach`, `while`) returns the empty string; end such bodies with an explicit `set result`. - Agent-oriented runner output should use `-report compact -show-passes 0` for focused checks unless detailed Markdown pass lists are needed. - `-report json` emits a machine-readable final summary, but package-load warnings may still precede it on stdout/stderr, and the punk ANSI output stack may emit an SGR reset immediately before the JSON on the same line (`scriptlib/developer/runtests_parity.tcl` tolerates both). +- Human-facing pass/fail/warning indicators in markdown/compact reports are ANSI-coloured only when stdout is a real windows console (`-colour auto` default: twapi `GetConsoleMode` on the STD_OUTPUT handle as the isatty-equivalent - `get_console_handle` is unsuitable, it succeeds for piped children of console shells; `NO_COLOR` honoured; `-colour on|off` overrides). Machine-facing output (the `RUNTESTS_RESULT` line, json reports) is never coloured, and piped/redirected output stays plain for agents. The runner uses raw literal SGR rather than `a+`/`a` so its indicators are independent of punk::console's process-global colour state. - ERROR-status failures now surface `errorInfo` (full Tcl error message/stack trace) in markdown, compact, and json reports. Compact mode truncates to a single line. - FAILED-status failures (result mismatch, not error) now surface `result_was` (actual) and `result_expected` (expected) in markdown, compact, and json reports. Compact mode truncates each to a single line. - `testsupport/` holds helper `.tcl` files sourced or exec'd by `.test` files or the runner (not discovered as suites). `testsupport/child_test_runner.tcl` is the multi-process-mode bootstrap `runtests.tcl` execs per test file (see the multi-process bullets above). `testsupport/repl_console_driver.tcl` (G-001) is exec'd in a child tclsh by `modules/punk/repl/testsuites/repl/consolebackends.test`: an interactive repl cannot run inside the shared testinterp (the codethread's quit/exit callbacks thread::send to the thread's MAIN interp, bypassing a child testinterp), so repl-through-console verification always drives a child process. `testsupport/wslprobe.tcl` (G-059) provides `::punktest::wsl`: a memoized capability probe yielding the `wsllinux` constraint (default distro launches, answers uname/tool probes, and round-trips a file through a native tempdir - NOT mere wsl.exe existence) plus native-filesystem staging helpers (`staging_create`/`staging_copy_in`/`run_in`/`staging_cleanup`). WSL-gated tests must execute from a staging dir on the distro's native filesystem with the shared `/mnt` path used only for one-way copy-in/out - never operate on the Windows checkout from inside WSL (DrvFs is slow and cross-boundary stat differences make git re-hash its index and fossil see phantom changes). Probe invocations use `wsl -e ` only (`wsl --status`/`-l` emit UTF-16). Known limitation: a broken-but-present WSL that hangs (rather than errors) on `wsl -e` can stall the probe. @@ -47,7 +48,7 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc - Add `-slowest ` when timing outliers are relevant. - Multi-process runs use `-tcltestoptions {-singleproc 0}` with otherwise identical flags. Check result parity between modes by capturing `-report json` stdout from each and comparing with `tclsh scriptlib/developer/runtests_parity.tcl ` (ignores timings; exit 0 on parity). - To watch runs live (including runs launched by agents or other processes, which inherit the environment): set `PUNK_TEST_UDPTEE=41197` and point a UDP listener/viewer at that port. No runner flags needed; results are unaffected. -- For broad runs use `-jobs` - e.g ` src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0`. Reference-machine full-suite scaling (2026-07-19, post G-092 split, result parity verified at each level): sequential ~5m40s, `-jobs 8` ~79-86s, `-jobs 16` ~61s (~5.6x), plateau by `-jobs 24` (~60s) - beyond 16 there is nothing left to overlap: the wall decomposes into the slowest single child (~31s, the runtimecmd pair - single tests, not further splittable), the serial console tail (~16s piped; larger under a real console where those suites actually run), and runner overhead (~10s). `-jobs 16` is the reference-machine sweet spot; focused runs rarely need `-jobs`. +- For broad runs use `-jobs` - e.g ` src/tests/runtests.tcl -jobs 16 -report compact -show-passes 0`. Reference-machine full-suite scaling (2026-07-19, post G-092 split, result parity verified at each level): sequential ~5m40s, `-jobs 8` ~79-86s, `-jobs 16` ~61s (~5.6x), plateau by `-jobs 24` (~60s) - beyond 16 there is nothing left to overlap: the wall decomposes into the slowest single child (~31s, the runtimecmd pair - single tests, not further splittable), the serial console tail (~16s piped; larger under a real console where those suites actually run), and runner overhead (~10s). `-jobs 16` is the reference-machine sweet spot. Use `-jobs` for ANY run spanning more than a handful of test files - subtree runs count (e.g `-include-paths modules/punk/args/**` is 28 files: ~50s sequential vs ~11s at `-jobs 16`); only single-file or few-file runs gain nothing from it. - Add `-strict-exit 1` when a nonzero shell exit code is needed for failures or parser warnings. - Capture enough stderr or failure context to identify the failing command or assertion. - For ERROR-status failures, the markdown report's `errorInfo` block and compact `message=` field carry the full Tcl error message; use `-report markdown` for untruncated context. diff --git a/src/tests/runtests.tcl b/src/tests/runtests.tcl index e86b461c..c3bd99d1 100644 --- a/src/tests/runtests.tcl +++ b/src/tests/runtests.tcl @@ -410,6 +410,49 @@ proc runtests_run_child_process {exe bootstrap payloadfile testfile outfile errf return [dict create exitcode $exitcode stdout $out stderr $err wall_ms [expr {[clock milliseconds] - $wall_t0}]] } +proc runtests_colour_enabled {opt} { + #opt: auto|on|off. auto = colour only when stdout is a real console - piped/redirected + #output (agents, captures) must stay plain - and NO_COLOR is honoured. + #Detection uses a twapi console handle probe (the reliable check on windows - see + #punk::console usage); deliberately NOT punk::console colour state or a+/a, whose + #process-global state proved context-sensitive under mixed tty-ness (2026-07-19 + #console-run findings) - the runner's own indicators must be deterministic. + switch -- $opt { + on {return 1} + off {return 0} + } + if {[info exists ::env(NO_COLOR)] && $::env(NO_COLOR) ne ""} { + return 0 + } + if {$::tcl_platform(platform) eq "windows"} { + #GetConsoleMode on the actual STD_OUTPUT handle is the isatty-equivalent: it errors + #("handle is invalid") when stdout is a pipe/file. twapi::get_console_handle stdout + #is NOT usable here - it succeeds whenever the process has an attached console, + #which is true even for piped children of console shells (verified 2026-07-19). + if {![catch {package require twapi}] + && ![catch {twapi::GetConsoleMode [twapi::GetStdHandle -11]}]} { + return 1 + } + return 0 + } + #non-windows: no reliable in-core tty test here - default off in auto (use -colour on) + return 0 +} + +proc runtests_clr {style text} { + #wrap text in a status colour when colour is enabled (raw literal SGR - see + #runtests_colour_enabled for why not a+/a) + if {!$::runtests_do_colour} { + return $text + } + switch -- $style { + pass {return "\x1b\[32m$text\x1b\[0m"} + fail - error {return "\x1b\[31;1m$text\x1b\[0m"} + warn - warning {return "\x1b\[33;1m$text\x1b\[0m"} + default {return $text} + } +} + proc runtests_udptee_event {args} { #emit a runner lifecycle event to the 'runtests' log tag when the UDP tee is active. #Each event message is a well-formed tcl list: RUNTESTS-EVENT ?key value ...? @@ -587,6 +630,12 @@ punk::args::define { Reports are unchanged and emitted in discovery order; per-test microsecond timings get noisier under CPU oversubscription. Conflicts with an explicit -tcltestoptions {-singleproc 1}." + -colour -type string -default auto -choices {auto on off} -help\ + "ANSI colour for the human-facing pass/fail/warning indicators in markdown and + compact reports. auto (default) colours only when stdout is a real console + (never piped/redirected output) and honours the NO_COLOR environment variable. + Machine-facing output (the RUNTESTS_RESULT line, json reports) is never + coloured." -serial-paths -type list -default {modules/punk/console/** modules/opunk/console/**} -help\ "Glob patterns (relative to the test base, forward slashes - same form as -include-paths) for test files that must NOT run concurrently when -jobs @@ -625,6 +674,7 @@ set opt_slowest [dict get $opts -slowest] set opt_strict_exit [dict get $opts -strict-exit] set opt_jobs [dict get $opts -jobs] set opt_serial_paths [dict get $opts -serial-paths] +set runtests_do_colour [runtests_colour_enabled [dict get $opts -colour]] if {![string is integer -strict $opt_jobs] || $opt_jobs < 1} { puts stderr "runtests: invalid -jobs value '$opt_jobs' (expected an integer >= 1)" exit 2 @@ -1088,9 +1138,7 @@ foreach testfile_relative $testfiles { puts stdout "errorCode: [dict get $result errorCode] " puts stdout "errorInfo: " puts stdout "```" - puts stdout "\x1b\[31m" - puts stdout [dict get $result errorInfo] - puts stdout "\x1b\[m" + puts stdout [runtests_clr fail [dict get $result errorInfo]] puts stdout "```" puts stdout "" flush stdout @@ -1111,7 +1159,7 @@ foreach testfile_relative $testfiles { set child_exit_failures [list [dict create name $testfile_relative status ERROR errorcode [list CHILDPROCESS exit $child_exitcode] errorinfo $stderr_tail]] puts stderr "ERROR: test file child process exited with code $child_exitcode: $testfile" if {!$report_json_only} { - puts stdout "test file $testfile_relative child process failed (exitcode $child_exitcode)" + puts stdout [runtests_clr fail "test file $testfile_relative child process failed (exitcode $child_exitcode)"] } dict lappend tallydict files_with_fails $testfile_relative set file_status error @@ -1142,13 +1190,13 @@ foreach testfile_relative $testfiles { #puts stdout [dict get $resultdict out] if {[dict get $resultdict summaryline_failed] > 0} { if {!$report_json_only} { - puts stdout "test file $testfile_relative had failures" + puts stdout [runtests_clr fail "test file $testfile_relative had failures"] } dict lappend tallydict files_with_fails $testfile_relative set file_status fail } else { if {$report_detailed_markdown} { - puts stdout "test file $testfile_relative passed" + puts stdout [runtests_clr pass "test file $testfile_relative passed"] } dict lappend tallydict files_without_fails $testfile_relative } @@ -1276,7 +1324,8 @@ if {$report_compact} { puts stdout "### testfiles" puts stdout "" foreach file_summary $file_summaries { - set line "- [dict get $file_summary status] `[dict get $file_summary path]` total=[dict get $file_summary total] passed=[dict get $file_summary passed] skipped=[dict get $file_summary skipped] failed=[dict get $file_summary failed]" + set fstatus [dict get $file_summary status] + set line "- [runtests_clr $fstatus $fstatus] `[dict get $file_summary path]` total=[dict get $file_summary total] passed=[dict get $file_summary passed] skipped=[dict get $file_summary skipped] failed=[dict get $file_summary failed]" if {[dict get $file_summary status] eq "warning"} { set warnings [dict get $file_summary warnings] if {[llength $warnings] > 0} { @@ -1319,7 +1368,7 @@ if {$report_compact} { } append line " expected=$first_line" } - puts stdout $line + puts stdout [runtests_clr fail $line] } } puts stdout "" @@ -1327,7 +1376,7 @@ if {$report_compact} { puts stdout "" foreach file_summary $file_summaries { foreach warning [dict get $file_summary warnings] { - puts stdout "- `[dict get $file_summary path]` reason=[dict get $warning code] summaryline_detected=[dict get $warning summaryline_detected] observed_passes=[dict get $file_summary observed_passes]" + puts stdout [runtests_clr warning "- `[dict get $file_summary path]` reason=[dict get $warning code] summaryline_detected=[dict get $warning summaryline_detected] observed_passes=[dict get $file_summary observed_passes]"] } } puts stdout "" @@ -1347,14 +1396,14 @@ if {$report_detailed_markdown} { puts stdout "### testfiles with failures" puts stdout "" foreach f [dict get $tallydict files_with_fails] { - puts stdout " - $f" + puts stdout " - [runtests_clr fail $f]" } puts stdout "" puts stdout "### testfiles with warnings" puts stdout "" foreach f [dict get $tallydict files_with_warnings] { - puts stdout " - $f" + puts stdout " - [runtests_clr warning $f]" } puts stdout "" @@ -1366,7 +1415,14 @@ if {$report_detailed_markdown} { if {!$report_json_only} { puts stdout "### runtests tally" puts stdout "" - puts stdout "Total [dict get $tallydict total] Passed [dict get $tallydict passed] Skipped [dict get $tallydict skipped] Failed [dict get $tallydict failed] " + set tally_passed "Passed [dict get $tallydict passed]" + set tally_failed "Failed [dict get $tallydict failed]" + if {[dict get $tallydict failed] > 0 || [llength [dict get $tallydict files_with_fails]]} { + set tally_failed [runtests_clr fail $tally_failed] + } else { + set tally_passed [runtests_clr pass $tally_passed] + } + puts stdout "Total [dict get $tallydict total] $tally_passed Skipped [dict get $tallydict skipped] $tally_failed " puts stdout "" } @@ -1384,7 +1440,7 @@ if {$report_emit_json} { } if {!$report_json_only} { - puts stdout "### runtests DONE at [clock format [clock seconds] -format "%Y-%m-%d %H:%M:%S"] (elapsed ${elapsed_seconds}s)" + puts stdout "### runtests DONE at [clock format [clock seconds] -format "%Y-%m-%d %H:%M:%S"] (elapsed ${elapsed_seconds}s) - [runtests_clr $status [string toupper $status]]" puts stdout "" }