Browse Source

runtests.tcl: surface full error messages and actual-vs-expected results in failure reports

runtests_failure_summaries now includes errorinfo, result_was, and result_expected when present in the parsed testdict, so the json report (which iterates keys) and downstream consumers get the full failure context. runtests_print_failure_details prints an errorInfo fenced block for ERROR-status failures and result_was/result_expected fenced blocks for FAILED-status mismatches, with full multi-line text. The compact report appends one-line truncated message=, actual=, and expected= fields (120/80/80 chars) so compact mode stays compact while still being actionable for agents. src/tests/AGENTS.md Local Contracts and Work Guidance updated to document the new failure-detail surfacing.
master
Julian Noble 2 weeks ago
parent
commit
98d75d0716
  1. 4
      src/tests/AGENTS.md
  2. 52
      src/tests/runtests.tcl

4
src/tests/AGENTS.md

@ -22,6 +22,8 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc
- Tcltest files must finish with `tcltest::cleanupTests`; missing cleanup produces a `missing-cleanupTests` runner warning and only untrusted observed testcase events.
- Agent-oriented runner output should use `-report compact -show-passes 0` for focused checks unless detailed Markdown pass lists are needed.
- `-report json` emits a machine-readable final summary, but package-load warnings may still precede it on stdout/stderr.
- ERROR-status failures now surface `errorInfo` (full Tcl error message/stack trace) in markdown, compact, and json reports. Compact mode truncates to a single line.
- FAILED-status failures (result mismatch, not error) now surface `result_was` (actual) and `result_expected` (expected) in markdown, compact, and json reports. Compact mode truncates each to a single line.
## Work Guidance
@ -34,6 +36,8 @@ Top-level test harness and source-tree tests for ShellSpy/Punk. Tests here exerc
- Add `-slowest <n>` when timing outliers are relevant.
- Add `-strict-exit 1` when a nonzero shell exit code is needed for failures or parser warnings.
- Capture enough stderr or failure context to identify the failing command or assertion.
- For ERROR-status failures, the markdown report's `errorInfo` block and compact `message=` field carry the full Tcl error message; use `-report markdown` for untruncated context.
- For FAILED-status failures (result mismatch), the markdown report's `result_was`/`result_expected` blocks and compact `actual=`/`expected=` fields show the actual-vs-expected comparison; use `-report markdown` for full multi-line values.
## Verification

52
src/tests/runtests.tcl

@ -155,6 +155,15 @@ proc runtests_failure_summaries {resultdict} {
if {[dict exists $testdict errorcode]} {
dict set failure errorcode [dict get $testdict errorcode]
}
if {[dict exists $testdict errorinfo] && [dict get $testdict errorinfo] ne ""} {
dict set failure errorinfo [dict get $testdict errorinfo]
}
if {[dict exists $testdict result_was] && [dict get $testdict result_was] ne ""} {
dict set failure result_was [dict get $testdict result_was]
}
if {[dict exists $testdict result_expected] && [dict get $testdict result_expected] ne ""} {
dict set failure result_expected [dict get $testdict result_expected]
}
lappend failures $failure
}
return $failures
@ -193,6 +202,25 @@ proc runtests_print_failure_details {testfile_relative resultdict} {
if {[dict get $testdict test_status] eq "ERROR"} {
puts stdout " returncode : [dict get $testdict returncode] "
puts stdout " errorcode : [dict get $testdict errorcode] "
if {[dict exists $testdict errorinfo] && [dict get $testdict errorinfo] ne ""} {
puts stdout " errorInfo : "
puts stdout " ```"
puts stdout " [string trimright [dict get $testdict errorinfo]]"
puts stdout " ```"
}
} elseif {[dict get $testdict test_status] eq "FAILED"} {
if {[dict exists $testdict result_was] && [dict get $testdict result_was] ne ""} {
puts stdout " result_was : "
puts stdout " ```"
puts stdout " [string trimright [dict get $testdict result_was]]"
puts stdout " ```"
}
if {[dict exists $testdict result_expected] && [dict get $testdict result_expected] ne ""} {
puts stdout " result_expected : "
puts stdout " ```"
puts stdout " [string trimright [dict get $testdict result_expected]]"
puts stdout " ```"
}
}
}
}
@ -660,6 +688,30 @@ if {$report_compact} {
if {[dict exists $failure errorcode] && [dict get $failure errorcode] ne ""} {
append line " errorcode=[dict get $failure errorcode]"
}
if {[dict exists $failure errorinfo] && [dict get $failure errorinfo] ne ""} {
set msg [string trim [dict get $failure errorinfo]]
set first_line [lindex [split $msg \n] 0]
if {[string length $first_line] > 120} {
set first_line "[string range $first_line 0 119]..."
}
append line " message=$first_line"
}
if {[dict exists $failure result_was] && [dict get $failure result_was] ne ""} {
set actual [string trim [dict get $failure result_was]]
set first_line [lindex [split $actual \n] 0]
if {[string length $first_line] > 80} {
set first_line "[string range $first_line 0 79]..."
}
append line " actual=$first_line"
}
if {[dict exists $failure result_expected] && [dict get $failure result_expected] ne ""} {
set expected [string trim [dict get $failure result_expected]]
set first_line [lindex [split $expected \n] 0]
if {[string length $first_line] > 80} {
set first_line "[string range $first_line 0 79]..."
}
append line " expected=$first_line"
}
puts stdout $line
}
}

Loading…
Cancel
Save