Browse Source
Root cause (recorded in the goal file): punk::path::treefilenames matches -include-paths against the DIRECTORY containing files, and its documented X/** boundary semantics exclude files directly in X - so suites keeping .test files at the subtree root (scriptwrap, binscripts, punkexe) were silently missed by X/** targeting: the G-092 four-pattern (2/13 files) and repeated-flag (zero files) failures. punk::args -multiple accumulation probed healthy - repeated flags and a single space-separated value reach discovery identically; no punk change was needed or made. runtests.tcl now filters the tailbase-relative FILE paths after an unrestricted walk (new testsupport/discovery.tcl: globmatchpath, any-include-match, exclusion wins): X/** includes files directly in X, an exact relative path targets a single file, a bare directory path matches nothing (append /**). -exclude-paths exposed (same syntax and accumulation, exclusion wins over inclusion). -discover-only prints the discovered list plus a RUNTESTS_FILES tcl-list line and exits before any run machinery - subsecond via a two-stage boot (discovery phase loads only punk::args + punk::path; punk/Thread/shellrun/punk::tcltestrun and the testinterp/child path assembly load after the exit). New regression suite runner/testsuites/discovery/pathdiscovery.test (23 tests: pure filter, fixture-tree walk+filter, end-to-end -discover-only children including both G-092 characterization forms; ~7s wall, parallel-safe). src/tests/AGENTS.md targeting wording reconciled; new src/tests/runner/AGENTS.md. G-093 index status proposed -> active (user-directed). Verification (native tclsh 9.0.3, piped Bash context): pre-change baseline default -jobs 16 (93 files / 1010 tests / failed=1 exec-14.3 only) vs post-change -jobs 16 with -exclude-paths runner/**: PARITY ok; post-change default run differs from baseline only by the added suite (+23 passing); mode parity -jobs 16 vs singleproc sequential: PARITY ok (94/1033/1). Claude-Session: https://claude.ai/code/session_01BNUVVkYq9vHa6G3S9a3XTZ Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.commaster
7 changed files with 623 additions and 90 deletions
@ -0,0 +1,25 @@
|
||||
# src/tests/runner — Runner Harness Suites |
||||
|
||||
## Purpose |
||||
|
||||
Test suites for the `src/tests/runtests.tcl` harness itself (discovery/path-targeting, runner options) — as opposed to suites for source modules (`modules/`), shell behaviour (`shell/`) or Tcl core compatibility (`core/`). |
||||
|
||||
## Ownership |
||||
|
||||
- Same rules as the parent `src/tests/AGENTS.md`: maintainer-owned; agents may update tests here when changing runner behaviour; do not delete, skip, or weaken existing tests without explicit user direction. |
||||
|
||||
## Local Contracts |
||||
|
||||
- Suites here may exec `runtests.tcl -discover-only 1` child processes of `[info nameofexecutable]` (subsecond since the G-093 two-stage boot — discovery loads only punk::args + punk::path before the early exit). Keep such children cheap and few: a suite file's child-process cost lands in its `-jobs` child wall. |
||||
- `testsuites/discovery/pathdiscovery.test` is the G-093 runner-targeting regression suite: it pins the `-include-paths`/`-exclude-paths` file-relpath matching semantics at three levels — pure `runtests_discovery::filter_paths`, composed `discover_testfiles` over a fixture tree, and end-to-end `-discover-only` runs against the real test tree, including the two 2026-07-19 G-092 measurement failures as characterization cases. |
||||
- Fixture trees go under tcltest's `temporaryDirectory` (the runner's run-scoped `-tmpdir`), never into the source tree. |
||||
- Suites here must stay parallel-safe (no console interaction) — they are not in `-serial-paths`. |
||||
|
||||
## Work Guidance |
||||
|
||||
- Run: `<tcl_interpreter> src/tests/runtests.tcl -report compact -show-passes 0 -include-paths runner/**` |
||||
- LF line endings (per root AGENTS.md user preferences). |
||||
|
||||
## Verification |
||||
|
||||
- The run above passes. For changes to runner discovery/targeting behaviour, also apply the parent `src/tests/AGENTS.md` verification bullet (full suite in both modes plus `scriptlib/developer/runtests_parity.tcl`). |
||||
@ -0,0 +1,247 @@
|
||||
# -*- tcl -*- |
||||
# Regression suite for runtests.tcl path-glob discovery targeting (G-093): |
||||
# the -include-paths/-exclude-paths semantics implemented by |
||||
# testsupport/discovery.tcl (runtests_discovery::filter_paths and |
||||
# runtests_discovery::discover_testfiles), plus end-to-end |
||||
# 'runtests.tcl -discover-only' child runs against the real test tree, |
||||
# including the two 2026-07-19 G-092 measurement failures as characterization |
||||
# cases (a four-pattern single value that matched only one subtree, and |
||||
# repeated -include-paths flags that matched zero files). |
||||
# |
||||
# Contract pinned here (G-093 root cause: punk::path::treefilenames' |
||||
# directory-oriented -include-paths excluded files directly at an X/** |
||||
# boundary, so suites keeping .test files at the subtree root were silently |
||||
# missed; runtests now filters the tailbase-relative FILE paths instead): |
||||
# 1. Patterns match each test file's path relative to the test base |
||||
# (forward slashes): * within a segment, ? single char, ** spans segments. |
||||
# 2. X/** includes files directly in X as well as deeper ones. |
||||
# 3. An exact relative path is a valid single-file pattern; a bare directory |
||||
# path without a glob tail matches nothing. |
||||
# 4. A single -include-paths value may carry multiple space-separated |
||||
# patterns; repeated flags accumulate; both forms are equivalent. |
||||
# 5. -exclude-paths uses the same syntax and accumulation semantics; |
||||
# exclusion wins over inclusion for the same file. |
||||
# 6. File-tail globs (trailing positional args) compose with path patterns. |
||||
# |
||||
# Run: tclsh src/tests/runtests.tcl -report compact -show-passes 0 -include-paths runner/** pathdiscovery.test |
||||
|
||||
package require tcltest |
||||
package require punk::path |
||||
|
||||
namespace eval ::testspace { |
||||
namespace import ::tcltest::* |
||||
|
||||
#src/tests, located relative to this test file (four levels up), so the suite |
||||
#exercises the SOURCE-TREE runner and discovery helper regardless of cwd. |
||||
variable testbase [file dirname [file dirname [file dirname [file dirname [file normalize [info script]]]]]] |
||||
source [file join $testbase testsupport discovery.tcl] |
||||
|
||||
#the runner's own discovery exclude-files list (mirrored from runtests.tcl) |
||||
variable runner_exclude_files {AGENTS.md *.tcl} |
||||
|
||||
proc runtests_discovered {args} { |
||||
#run runtests.tcl -discover-only in a child process of this interpreter's |
||||
#executable and return the discovered relative-path list parsed from the |
||||
#RUNTESTS_FILES line (a well-formed tcl list with the marker as its first |
||||
#element). stderr is merged - parent boot noise lines are tolerated. |
||||
variable testbase |
||||
set out [exec [info nameofexecutable] [file join $testbase runtests.tcl] -discover-only 1 {*}$args 2>@1] |
||||
foreach ln [split $out \n] { |
||||
set ln [string trimright $ln \r] |
||||
if {[string match {RUNTESTS_FILES *} $ln]} { |
||||
return [lrange $ln 1 end] |
||||
} |
||||
} |
||||
error "no RUNTESTS_FILES line in runtests.tcl -discover-only output:\n$out" |
||||
} |
||||
|
||||
proc with_prefix {prefix paths} { |
||||
#the subset of paths lying below prefix (prefix without trailing slash) |
||||
set matched [list] |
||||
foreach p $paths { |
||||
if {[string match "$prefix/*" $p]} { |
||||
lappend matched $p |
||||
} |
||||
} |
||||
return $matched |
||||
} |
||||
|
||||
# ------------------------------------------------------------------------- |
||||
# Fixture tree (under tcltest's temporaryDirectory - the runner points that |
||||
# at a run-scoped system temp dir): |
||||
# fixbase/direct.test (file directly at the base) |
||||
# fixbase/helper.tcl (must be dropped by the *.tcl exclude-files rule) |
||||
# fixbase/AGENTS.md (must be dropped by the AGENTS.md exclude-files rule) |
||||
# fixbase/sub1/x.test (file directly at an X/** pattern boundary) |
||||
# fixbase/sub1/x.other (tail-glob discrimination) |
||||
# fixbase/sub2/deep/y.test (file below a deeper boundary) |
||||
# fixbase/sub2/other.test |
||||
# ------------------------------------------------------------------------- |
||||
variable fixture_error "" |
||||
variable fixbase "" |
||||
try { |
||||
set fixbase [file join [temporaryDirectory] g093_pathdiscovery_fixture] |
||||
file delete -force $fixbase |
||||
file mkdir [file join $fixbase sub1] [file join $fixbase sub2 deep] |
||||
foreach relpath { |
||||
direct.test |
||||
helper.tcl |
||||
AGENTS.md |
||||
sub1/x.test |
||||
sub1/x.other |
||||
sub2/deep/y.test |
||||
sub2/other.test |
||||
} { |
||||
set fd [open [file join $fixbase {*}[file split $relpath]] w] |
||||
puts $fd "#g093 pathdiscovery fixture file" |
||||
close $fd |
||||
} |
||||
} on error {result} { |
||||
set fixture_error $result |
||||
} |
||||
testConstraint g093fixture [expr {$fixture_error eq ""}] |
||||
if {$fixture_error ne ""} { |
||||
puts stderr "pathdiscovery.test fixture setup failed (fixture tests will be skipped): $fixture_error" |
||||
} |
||||
|
||||
proc fixdiscover {includes excludes {tails {*.test}}} { |
||||
#composed walk+filter over the fixture tree, result sorted for |
||||
#order-independent comparison |
||||
variable fixbase |
||||
variable runner_exclude_files |
||||
lsort [runtests_discovery::discover_testfiles $fixbase $includes $excludes $tails $runner_exclude_files] |
||||
} |
||||
|
||||
#added 2026-07-19 (agent, G-093) |
||||
test pathdiscovery-1.0 {filter_paths: single value carrying multiple patterns includes every pattern's matches} -body { |
||||
runtests_discovery::filter_paths {a/f.test b/g.test c/h.test} {a/** b/**} {} |
||||
} -result {a/f.test b/g.test} |
||||
|
||||
test pathdiscovery-1.1 {filter_paths: a literal ** include pattern matches every path} -body { |
||||
runtests_discovery::filter_paths {direct.test a/b/c.test} {**} {} |
||||
} -result {direct.test a/b/c.test} |
||||
|
||||
test pathdiscovery-1.2 {filter_paths: exclusion wins over inclusion for the same file} -body { |
||||
runtests_discovery::filter_paths {a/f.test a/g.test} {a/**} {a/g.test} |
||||
} -result {a/f.test} |
||||
|
||||
test pathdiscovery-1.3 {filter_paths: a literal ** exclude pattern removes everything} -body { |
||||
runtests_discovery::filter_paths {a/f.test b/g.test} {**} {**} |
||||
} -result {} |
||||
|
||||
test pathdiscovery-1.4 {filter_paths: a bare directory-like pattern matches no file paths} -body { |
||||
runtests_discovery::filter_paths {a/f.test} {a} {} |
||||
} -result {} |
||||
|
||||
test pathdiscovery-2.0 {fixture: default ** discovers every .test file at all depths} -constraints g093fixture -body { |
||||
fixdiscover {**} {} |
||||
} -result {direct.test sub1/x.test sub2/deep/y.test sub2/other.test} |
||||
|
||||
test pathdiscovery-2.1 {fixture: X/** includes a file directly at the pattern boundary (G-093 regression)} -constraints g093fixture -body { |
||||
fixdiscover {sub1/**} {} |
||||
} -result {sub1/x.test} |
||||
|
||||
test pathdiscovery-2.2 {fixture: deeper X/** boundary} -constraints g093fixture -body { |
||||
fixdiscover {sub2/deep/**} {} |
||||
} -result {sub2/deep/y.test} |
||||
|
||||
test pathdiscovery-2.3 {fixture: single value carrying two patterns includes both subtrees} -constraints g093fixture -body { |
||||
fixdiscover {sub1/** sub2/**} {} |
||||
} -result {sub1/x.test sub2/deep/y.test sub2/other.test} |
||||
|
||||
test pathdiscovery-2.4 {fixture: a bare directory path matches nothing (append /** for a subtree)} -constraints g093fixture -body { |
||||
fixdiscover {sub1} {} |
||||
} -result {} |
||||
|
||||
test pathdiscovery-2.5 {fixture: an exact relative path targets a single file} -constraints g093fixture -body { |
||||
fixdiscover {sub1/x.test} {} |
||||
} -result {sub1/x.test} |
||||
|
||||
test pathdiscovery-2.6 {fixture: explicit file-tail glob composes with a path pattern} -constraints g093fixture -body { |
||||
fixdiscover {sub1/**} {} {x.other} |
||||
} -result {sub1/x.other} |
||||
|
||||
test pathdiscovery-2.7 {fixture: multiple file-tail globs match independently under a path pattern} -constraints g093fixture -body { |
||||
fixdiscover {sub1/**} {} {*.test *.other} |
||||
} -result {sub1/x.other sub1/x.test} |
||||
|
||||
test pathdiscovery-2.8 {fixture: exclude a subtree from a full run} -constraints g093fixture -body { |
||||
fixdiscover {**} {sub2/**} |
||||
} -result {direct.test sub1/x.test} |
||||
|
||||
test pathdiscovery-2.9 {fixture: exclusion wins inside an included subtree} -constraints g093fixture -body { |
||||
fixdiscover {sub2/**} {sub2/deep/**} |
||||
} -result {sub2/other.test} |
||||
|
||||
test pathdiscovery-2.10 {fixture: exact-path exclusion beats an including pattern} -constraints g093fixture -body { |
||||
fixdiscover {sub1/**} {sub1/x.test} |
||||
} -result {} |
||||
|
||||
test pathdiscovery-2.11 {fixture: exclude-files rules drop AGENTS.md and *.tcl helpers regardless of tails} -constraints g093fixture -body { |
||||
fixdiscover {**} {} {*} |
||||
} -result {direct.test sub1/x.other sub1/x.test sub2/deep/y.test sub2/other.test} |
||||
|
||||
test pathdiscovery-3.0 {end-to-end: the G-092 four-pattern single value discovers files from all four subtrees (2026-07-19 failure (a))} -body { |
||||
set files [runtests_discovered -include-paths {modules/punk/mix/testsuites/scriptwrap/** shell/testsuites/binscripts/** modules/argparsingtest/** shell/testsuites/punkexe/**}] |
||||
set result [list] |
||||
foreach prefix { |
||||
modules/punk/mix/testsuites/scriptwrap |
||||
shell/testsuites/binscripts |
||||
modules/argparsingtest |
||||
shell/testsuites/punkexe |
||||
} { |
||||
lappend result [expr {[llength [with_prefix $prefix $files]] > 0}] |
||||
} |
||||
set result |
||||
} -result {1 1 1 1} |
||||
|
||||
test pathdiscovery-3.1 {end-to-end: repeated -include-paths flags accumulate and equal the single-value form (2026-07-19 failure (b))} -body { |
||||
set a [runtests_discovered -include-paths modules/punk/mix/testsuites/scriptwrap/** -include-paths shell/testsuites/binscripts/**] |
||||
set b [runtests_discovered -include-paths {modules/punk/mix/testsuites/scriptwrap/** shell/testsuites/binscripts/**}] |
||||
list [expr {$a eq $b}]\ |
||||
[expr {[llength [with_prefix modules/punk/mix/testsuites/scriptwrap $a]] > 0}]\ |
||||
[expr {[llength [with_prefix shell/testsuites/binscripts $a]] > 0}] |
||||
} -result {1 1 1} |
||||
|
||||
test pathdiscovery-3.2 {end-to-end: default discovery covers boundary-direct, deep and this suite's own file} -body { |
||||
set files [runtests_discovered] |
||||
set result [list] |
||||
foreach path { |
||||
modules/punk/mix/testsuites/scriptwrap/multishell.test |
||||
modules/argparsingtest/testsuites/argparsingtest/argparsingtest.test |
||||
runner/testsuites/discovery/pathdiscovery.test |
||||
} { |
||||
lappend result [expr {$path in $files}] |
||||
} |
||||
set result |
||||
} -result {1 1 1} |
||||
|
||||
test pathdiscovery-3.3 {end-to-end: -exclude-paths yields exactly the default list minus the excluded subtree, order preserved} -body { |
||||
set full [runtests_discovered] |
||||
set excl [runtests_discovered -exclude-paths modules/punk/mix/testsuites/scriptwrap/**] |
||||
set manual [list] |
||||
foreach p $full { |
||||
if {![string match modules/punk/mix/testsuites/scriptwrap/* $p]} { |
||||
lappend manual $p |
||||
} |
||||
} |
||||
list [expr {$excl eq $manual}] [expr {[llength $full] > [llength $excl]}] |
||||
} -result {1 1} |
||||
|
||||
test pathdiscovery-3.4 {end-to-end: path patterns compose with an explicit file tail (json report mode)} -body { |
||||
runtests_discovered -report json -include-paths {modules/punk/mix/testsuites/scriptwrap/** shell/testsuites/binscripts/** modules/argparsingtest/** shell/testsuites/punkexe/**} multishell.test |
||||
} -result {modules/punk/mix/testsuites/scriptwrap/multishell.test} |
||||
|
||||
test pathdiscovery-3.5 {end-to-end: an exact relative path discovers exactly that file} -body { |
||||
runtests_discovered -include-paths runner/testsuites/discovery/pathdiscovery.test |
||||
} -result {runner/testsuites/discovery/pathdiscovery.test} |
||||
|
||||
# cleanup fixture |
||||
variable fixbase |
||||
if {$fixbase ne "" && [file isdirectory $fixbase]} { |
||||
catch {file delete -force $fixbase} |
||||
} |
||||
} |
||||
|
||||
tcltest::cleanupTests |
||||
namespace delete ::testspace |
||||
@ -0,0 +1,71 @@
|
||||
#testsupport/discovery.tcl - test-file discovery and path-targeting filter for runtests.tcl |
||||
#added 2026-07-19 (agent, G-093) |
||||
# |
||||
#Discovery contract (the runner's -include-paths/-exclude-paths semantics): |
||||
# - the tree below the test base is walked unrestricted (punk::path::treefilenames with its |
||||
# default match-everything include, tailglobs applied per directory as before), then the |
||||
# tailbase-RELATIVE FILE PATHS are filtered against the include/exclude patterns using |
||||
# punk::path::globmatchpath (* within a segment, ? single char, ** spans segments). |
||||
# - a file is included when ANY include pattern matches its relative path, then dropped when |
||||
# ANY exclude pattern matches it - exclusion wins over inclusion for the same file. |
||||
# - matching FILE paths (not the directory containing them) means modules/punk/mix/** includes |
||||
# files sitting directly in modules/punk/mix as well as deeper ones, and an exact relative |
||||
# path is a valid single-file pattern. A bare directory path without a glob tail matches |
||||
# nothing (append /** for a subtree). |
||||
# |
||||
#Why the filter exists here rather than passing the patterns to treefilenames (G-093 root |
||||
#cause, 2026-07-19): treefilenames' own -include-paths matches the DIRECTORY containing |
||||
#files, and its documented X/** boundary semantics exclude files directly in X - so suites |
||||
#keeping .test files at the subtree root (e.g modules/punk/mix/testsuites/scriptwrap) were |
||||
#silently missed by X/** targeting. runtests' contract - like its -serial-paths matching - |
||||
#is file-path oriented, so the patterns are applied to the discovered relative paths instead. |
||||
#The walk is unpruned, but the default full run already walked the whole tree; the test tree |
||||
#is small. |
||||
# |
||||
#Sourced by runtests.tcl (parent side - discovery happens once, before any child processes) |
||||
#and by the runner-targeting regression suite (runner/testsuites/discovery/pathdiscovery.test). |
||||
#Caller must have module paths established before sourcing (bootsupport for the runner, |
||||
#the generated payload for suite child processes). |
||||
|
||||
package require punk::path |
||||
|
||||
namespace eval ::runtests_discovery {} |
||||
|
||||
proc ::runtests_discovery::filter_paths {relpaths include_patterns exclude_patterns} { |
||||
#pure filter over relative file paths - include when any include pattern matches, |
||||
#then exclusion wins. A literal ** include pattern matches every path (fast path). |
||||
set include_all [expr {"**" in $include_patterns}] |
||||
set kept [list] |
||||
foreach relpath $relpaths { |
||||
set include $include_all |
||||
if {!$include} { |
||||
foreach pat $include_patterns { |
||||
if {[punk::path::globmatchpath $pat $relpath]} { |
||||
set include 1 |
||||
break |
||||
} |
||||
} |
||||
} |
||||
if {!$include} { |
||||
continue |
||||
} |
||||
foreach pat $exclude_patterns { |
||||
if {[punk::path::globmatchpath $pat $relpath]} { |
||||
set include 0 |
||||
break |
||||
} |
||||
} |
||||
if {$include} { |
||||
lappend kept $relpath |
||||
} |
||||
} |
||||
return $kept |
||||
} |
||||
|
||||
proc ::runtests_discovery::discover_testfiles {base include_patterns exclude_patterns tailglobs exclude_files} { |
||||
#walk everything below base (treefilenames' own include filtering deliberately unused - |
||||
#see header), then filter the relative paths. Result keeps treefilenames' natural sort |
||||
#order, so runner discovery order is unchanged from pre-G-093 default discovery. |
||||
set relpaths [punk::path::treefilenames -directory $base -tailbase $base -include-paths {**} -exclude-files $exclude_files {*}$tailglobs] |
||||
return [filter_paths $relpaths $include_patterns $exclude_patterns] |
||||
} |
||||
Loading…
Reference in new issue