jn
/
punkshell
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1443 Commits
1 Branch
1 Tag
223 MiB
 
 
 
 
 
 
Tree: 977eade569
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '977eade569'
${ noResults }
punkshell/goals/archive/G-161-tcltestrun-multiline-...

9.0 KiB
Raw Blame History

G-161 tcltestrun failure parsing robust to multi-line test descriptions

Status: achieved 2026-08-04 Scope: src/modules/punk/tcltestrun-999999.0a1.0.tm (parse_testrun failure-banner state machine); src/tests/runner/ (parser characterization suite - synthetic captured-output + end-to-end failing-fixture coverage); src/tests/AGENTS.md (single-line-description contract note relaxed to style guidance once tolerated) Goal: a failing tcltest whose description contains embedded newlines is reported with full fidelity - failure event with name and description, result_was/result_expected, and every subsequent event including the summary line - instead of the parser missing the multi-line opening banner, misreading the closing banner as an opener, and wedging in failure-capture state so the file degrades to warn/missing-cleanupTests with no failure detail. Acceptance: a committed runner suite covers both a synthetic captured-output fixture (multi-line-description failure followed by passes and a summary line, parsed to the correct event set) and an end-to-end fixture .test file whose deliberately failing multi-line-description test is reported by runtests.tcl as status=FAILED with actual/expected populated and a trusted summary (summaryline_detected=1, correct totals) in BOTH single-process and multi-process modes; single-line-description failure reports keep their existing shape, verified by scriptlib/developer/runtests_parity.tcl comparing -report json output of an identical full-suite run before and after the change; the src/tests/AGENTS.md single-line-description bullet is updated to style guidance reflecting the new tolerance.

Context

Discovered 2026-08-03 while authoring the commandstack characterisation suite. tcltest prints a failing test's opening banner as ONE puts of "==== FAILED" and only trims the description's ENDS, so a description with embedded newlines makes the banner span lines. punk::tcltestrun's per-line scanner recognises an opening banner only as a line both starting "==== " and ending " FAILED": the multi-line opener never matches, the later single-line closing banner "==== FAILED" is then treated as an OPENER, and the parser waits forever for a "==== Contents of test case:" line that already went past - swallowing every subsequent event, including the pass events of later tests and the cleanupTests summary line. The file then reports warn/missing-cleanupTests with observed passes stopping at the failure and no failure detail at all. The trap is recorded as a hard contract in src/tests/AGENTS.md (descriptions must be single content line); this goal makes the parser tolerant so the contract can stand down to a style preference.

Approach

Two candidate mechanisms, settled in the work:

  • Banner buffering: a line starting "==== " that does not end " FAILED" opens a bounded lookahead buffer joined until a line ending " FAILED" completes the banner (or the buffer is abandoned as ordinary output).
  • Start-event anchoring: the runner always passes -verbose start, so "---- start" precedes every test; carrying the current test name makes the closing-banner match exact and the opening banner recognisable by prefix alone. Either way the existing state machine's confounding-output heuristics (emulated banner lines inside test output) must keep their current behaviour for single-line cases.

Notes

  • Overlap survey 2026-08-03: goals_xref paths over src/modules/punk/tcltestrun-999999.0a1.0.tm, src/tests/runner and src/tests/runtests.tcl found no intersecting live goals.
  • Activation-freshness survey 2026-08-03 (goals_xref score G-161, run at the user-directed activation): no related live goals - every overlap pair is unlinked with zero path intersection, lexical noise only (top scores: G-085 0.55 cmdtrace, G-162 0.43 bakelist freshness, G-014 0.42 config toml; the interval-drafted G-163 tty drain 0.39 is likewise unrelated). Reconfirms the drafting-time survey; no Related: lines warranted.
  • Reproduction record: the commandstack suite's pre-rewrite failure mode (24-of-29 observed passes, no failure event, no summary) in both runner modes, 2026-08-03; zipper.test's single-line-description failure reported correctly under the same conditions - the discriminating experiment.
  • punk::tcltestrun is also consumed by kit-baked copies; a mint (and later promotion) carries the fix to kits per the normal channels.

Progress

  • 2026-08-03..04, one session (activation git 5bf057a3 through this flip): implemented, verified, archived.
  • Mechanism settled: the two Approach candidates COMBINED - start-event anchoring gates entry (a "==== ..." line not ending " FAILED" opens banner capture; exact prefix compare via string equal -length, no glob; "---- start" tracking extracts the name at the LAST " start" token), and banner buffering accumulates continuation lines until the " FAILED" terminator completes the banner. Abandonment at recognisable event lines (took/PASSED/SKIPPED/start/summary-line) reprocesses that line through normal handling; a stray "==== Contents of test case:" completes the banner instead of wedging; 100-line bound backstop. Streams without start events keep historic behaviour (pinned by parsetestrun-nostart-6.0).
  • Second defect fixed under this goal's acceptance surface ("actual/expected populated"): result_expected was NEVER captured - the "---- Result should have been" opener always arrives while the result_was capture is active, and the active-capture reset branch fell through past the opener elseif; expected values were absent from every FAILED report (verified in the pre-change full-suite baselines: both real failing files carried result_was only) and the expected-value lines leaked into test_body. Fixed by a direct result_was -> result_expected transition in the active-capture branch.
  • Acceptance evidence (2026-08-04, driver tclsh90 = native Tcl 9.0.3):
    • Synthetic suite runner/testsuites/parser/parsetestrun.test 6/6 green on tclsh90 and native tclsh 8.6.17; fabricated streams byte-checked against a live tcltest 2.5 capture of the real fixture (banner/verbose-line/summary shapes).
    • E2E suite runner/testsuites/parser/multilinebanner.test 2/2 green (tclsh90, tclsh86, and under -jobs): fixtures/multilinedesc.testfixture driven through real child runtests.tcl runs in BOTH single-process and multi-process modes; json failure entry FAILED with result_was "actual-value\n" / result_expected "expected-value\n", summaryline_detected 1, totals 2/1/0/1.
    • Full-suite parity in BOTH modes: scriptlib/developer/runtests_parity.tcl PARITY ok (126 files, totals 1422/1400/20/2 both sides) comparing -report json runs before (tcltestrun 0.3.1) vs after (0.4.0 promoted to bootsupport; the new parser dir excluded from the after-runs to keep the file set identical). Inclusive full run: 1430 total, the same two pre-existing failing files (core exec.test, punkboot offsetstyle.test), 0 warnings.
    • src/tests/AGENTS.md single-line-description bullet relaxed to style guidance; runtests.tcl floors its require at punk::tcltestrun 0.4.0-.
  • Consumption note: runtests.tcl's PARENT loads the bootsupport tcltestrun (src/modules is not on the parent's tm path) - the bootsupport promotion is what put the tolerant parser in front of the runner; _vfscommon/kit promotion is left to the normal channels (the 0.4.0- floor makes a stale kit copy fail loud rather than parse wrong).
  • Commits (git/fossil): activation 5bf057a3/87487711; parser+synthetic suite 08804abf/319c8f86; bootsupport batch 11dbe02d/a6226938 (also swept pending console 0.8.1
    • shellfilter 0.2.5 promotions); e2e+floor+docs+0.49.6 e6ed4bb5/ea94476a; flip commit is this change-set.
  • Traps for future parser/suite work: Tcl's brace scan counts braces inside COMMENTS in braced proc bodies (a lone open-brace in a .test comment kills the whole file's parse with "missing close-brace: possible unbalanced brace in comment"); lean -jobs children expose load-time dependencies (punk::tcltestrun's about section calls punk::args::lib::tstr/resolved_def at source time, so tests must require punk::args before punk::tcltestrun - declared per the lean-children convention).

Follow-ons

Follow-on: anchor-verify single-line opening banners too - any "==== * FAILED" line still opens failure capture unanchored (the historic emulated-banner confounder cost, pinned by parsetestrun-confounder-5.0); anchoring would close it but changes characterized behaviour => open Follow-on: closing-banner prefix match uses glob string match on the raw name, so a test NAME containing glob metacharacters can mis-match its opening text (latent, pre-existing, unexercised in the corpus) => open Follow-on: errorInfo/errorCode capture retains a historic leading space (capture ranges from index 15 of the 16-char "---- errorCode: " prefix); consumers tolerate it and the parser suite trims - normalizing is a report-shape change => landed 2026-08-04 (tcltestrun 0.4.1 parse_testrun index-16 capture, parsetestrun-multiline-error-3.0 pins the normalized shape exact; punkshell 0.49.7)