From bc958300b125ba2603559b61472b3674052f3d5d Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Mon, 20 Jul 2026 00:44:32 +1000 Subject: [PATCH] G-093 rework: punk::path *** pathglob + separable lattice; runtests passthrough targeting User-approved contract amendment replacing the day-1 file-relpath filter: one directory-oriented pathglob semantics at every layer, with the ergonomics gap closed in the syntax itself. punk::path 0.4.0: - *** as a whole segment matches ZERO or more segments (join-aware regex in pathglob_as_re: an adjacent separator is absorbed into an optional group), so X/*** = X and everything below - the subtree spelling; ***/f.txt matches at any depth including the root. - include separability fix: an exact or single-segment-glob full match no longer sets allbelow (the deep walk previously dragged the whole subtree in for X and X/* patterns; the zipfs walk was already separable). - exclude separability fix (new exclude_state classifier, both walks): only **|***-tail patterns prune the matched subtree; other matches exclude just that directory's files - the **/_aside vs **/_aside/** argdoc distinction is now real. Dead _path_matches_any removed. - legacy bare-* include collapse removed (a * in -include-paths is the one-below lattice form, not a match-everything alias; no caller used it). - globmatchpath default-nocase driveletter exception rewritten as a direct regex-head rewrite (the regex-text file-split broke on c:/***'s glued group and accepted non-alpha drives). - path.test +5 lattice/*** pins (40 total). runtests.tcl: - -include-paths/-exclude-paths pass through to treefilenames unchanged; argdocs teach the separable X | X/* | X/** | X/*** lattice; directory and file-name axes independent (tails); single-file targeting = exact dir pattern + file tail. - -serial-paths reconciled to the same directory-of-file reading (default now modules/punk/console/*** modules/opunk/console/*** via runtests_discovery::dir_matches_any) - the pre-existing file-path serial matching was the other same-option-different-referent divergence. - per-pattern zero-match stderr advisory (the X/**-for-subtree foot-gun points at X/***). - json report emitter: whitespace-padded entier-looking values now take the quoted-escaped path ('string is entier -strict' accepts surrounding whitespace - a failed test's "0\n" result_was embedded a raw newline and truncated the parseable report line). - testsupport/discovery.tcl now a thin passthrough (requires punk::path 0.4.0-); pathdiscovery.test rewritten to the directory-semantics contract (22 tests incl. the X/** zero-match advisory characterization); AGENTS.md targeting guidance swept to /*** across src/tests and src/modules/opunk. Verification (native tclsh 9.0.3, piped, serialized): 45-probe matrix, path suite 40/40, runner suite 22/22; vs the pre-G-093 baseline the default discovery FILE SET differs by exactly the added runner suite and result parity only by the five new path tests (+ that suite); mode parity -jobs 16 vs singleproc: PARITY ok (94/1037/failed=1 exec-14.3). Goal contract amended in the detail file; full record there. Claude-Session: https://claude.ai/code/session_01BNUVVkYq9vHa6G3S9a3XTZ Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- .../G-093-runtests-includepaths-discovery.md | 157 +++++++++++--- src/modules/opunk/AGENTS.md | 2 +- src/modules/punk/path-999999.0a1.0.tm | 195 ++++++++++++----- src/modules/punk/path-buildversion.txt | 3 +- src/tests/AGENTS.md | 16 +- src/tests/core/AGENTS.md | 4 +- src/tests/core/tcl/testsuites/tests/AGENTS.md | 4 +- src/tests/modules/AGENTS.md | 6 +- .../punk/path/testsuites/tests/path.test | 82 +++++++ src/tests/runner/AGENTS.md | 4 +- .../testsuites/discovery/pathdiscovery.test | 203 ++++++++++-------- src/tests/runtests.tcl | 108 ++++++---- src/tests/shell/AGENTS.md | 4 +- src/tests/testsupport/discovery.tcl | 106 ++++----- 14 files changed, 598 insertions(+), 296 deletions(-) diff --git a/goals/G-093-runtests-includepaths-discovery.md b/goals/G-093-runtests-includepaths-discovery.md index 9bf3990b..4ca9781d 100644 --- a/goals/G-093-runtests-includepaths-discovery.md +++ b/goals/G-093-runtests-includepaths-discovery.md @@ -2,8 +2,8 @@ Status: active Scope: src/tests/runtests.tcl, src/modules/punk/path-999999.0a1.0.tm (treefilenames/globmatchpath as implicated), src/modules/punk/args-999999.0a1.0.tm (only if -multiple implicated), src/tests/ (runner-targeting regression suite), src/tests/AGENTS.md -Goal: runtests.tcl -include-paths discovery honours its documented contract - a single value carrying multiple space-separated glob patterns includes every pattern's matches, repeated -include-paths flags accumulate, and deep ** patterns match regardless of whether explicit file-tail globs are supplied - and a new -exclude-paths option (treefilenames' existing capability exposed, or an equivalent post-discovery globmatchpath filter over the relative paths) gives broad development runs a middle ground between full and targeted by excluding glob-matched files, with both directions' semantics pinned by the same regression suite and the include-side root cause recorded. -Acceptance: the two 2026-07-19 failures reproduce as characterization tests before the fix and pass after it - (a) a single -include-paths value listing four deep patterns (the G-092 measurement set) discovers files from all four subtrees under the default *.test tail glob, and (b) two repeated -include-paths flags naming distinct subtrees discover both (observed matching zero files); the root cause is identified and recorded in this file; -exclude-paths is exposed on runtests (argdoc + src/tests/AGENTS.md documented, accumulate semantics matching -include-paths, exclusion winning over inclusion for the same file) and a full run excluding a named subtree discovers exactly the default-discovery file list minus that subtree; a regression suite under src/tests covers both options across single-value multi-pattern, repeated-flag, deep-pattern and tail-glob-present/absent combinations plus include/exclude interaction; a full-suite default-discovery run before and after the changes compares PARITY ok via scriptlib/developer/runtests_parity.tcl (default '**' discovery with no exclusions must be unaffected); src/tests/AGENTS.md path-targeting wording is reconciled with the verified semantics. +Goal: runtests.tcl path targeting shares one directory-oriented pathglob contract with punk::path::treefilenames - patterns match the directory containing a test file, a single -include-paths value carrying multiple space-separated patterns includes every pattern's matches, repeated flags accumulate, the pathglob syntax gains *** (zero or more segments) so X/*** expresses a base-inclusive subtree, and a new -exclude-paths option (same syntax and accumulation, exclusion winning) gives broad development runs a middle ground between full and targeted - with the include and exclude lattices (X, X/*, X/**, X/***) separable, pinned by regression suites at both the punk::path and runner levels, and the root cause of the 2026-07-19 failures recorded in this file. +Acceptance: the G-092 four-subtree measurement set spelled X/*** discovers files from all four subtrees under the default *.test tail glob, and the X/** spelling of a base-direct suite matches zero files while emitting a stderr advisory (both pinned as characterization tests); two repeated -include-paths flags naming distinct subtrees in X/*** form discover both and equal the single-value form; the root cause of the original failures is identified and recorded in this file; punk::path 0.4.0 implements *** across pathglob_as_re/globmatchpath/treefilenames with the include and exclude lattice forms separable and pinned in the punk::path suite; -exclude-paths is exposed on runtests (argdoc + src/tests/AGENTS.md documented, accumulate semantics matching -include-paths, exclusion winning over inclusion for the same file) and a full run excluding a named subtree discovers exactly the default-discovery file list minus that subtree; -serial-paths classifies by the same directory-of-file reading; a regression suite under src/tests covers both options across single-value multi-pattern, repeated-flag, lattice-form and tail-glob-present/absent combinations plus include/exclude interaction; a full-suite default-discovery run before and after the changes compares PARITY ok via scriptlib/developer/runtests_parity.tcl (default '**' discovery with no exclusions must be unaffected); src/tests/AGENTS.md path-targeting wording is reconciled with the verified semantics. ## Context @@ -66,24 +66,37 @@ not multi-pattern interaction and not punk::args: - The with-tails invocation that "worked" during G-092 worked because its subtrees kept files deeper than the pattern roots - tail globs were never a factor in the failure. -Fix shape: runtests' contract (matching each test file's RELATIVE PATH, consistent with -its pre-existing -serial-paths matching at the same layer) is implemented as a post-walk -filter in src/tests/testsupport/discovery.tcl - treefilenames walks the whole tree -(match-everything include, tailglobs as before) and runtests_discovery::filter_paths -applies include/exclude patterns to the tailbase-relative file paths via globmatchpath -(include on any match, exclusion wins). punk::path is deliberately untouched. +Fix shape (final, 2026-07-20): the ergonomics gap is closed at the pathglob-syntax +layer - punk::path 0.4.0 adds *** (zero or more segments as a whole segment), so X/*** +is the base-inclusive subtree spelling and runtests passes its patterns through to +treefilenames unchanged: one directory-oriented semantics at every layer, with the +file-name axis staying on the trailing tail globs. Two engine separability defects +found en route are fixed in the same punk::path release: an exact or +single-segment-glob include match no longer drags the whole subtree in (the deep walk +previously set allbelow on ANY full match; the zipfs walk was already separable), and +non-spanning exclude matches now drop only the matched directory's files instead of +pruning (making the argdoc's **/_aside vs **/_aside/** distinction real). The +2026-07-19 interim fix (file-relpath post-filtering at the runner layer) was reworked +away at user direction - see Alternatives considered. ## Alternatives considered -- Passing translated patterns down to treefilenames (X/** expanded to {X X/**} to pull in - boundary-direct files) - rejected: cannot express exact-file patterns or per-pattern - tail constraints, and leaves runtests coupled to directory-oriented semantics. -- Changing treefilenames' own X/** semantics to include boundary-direct files - rejected: - its documented directory-oriented contract is self-consistent and other callers depend - on the module; a file-path matching mode for punk::path remains a possible future - enhancement (out of scope here). -- Walk pruning is lost for targeted runs (the walk is now always full-tree) - accepted: - the default '**' run already walked the whole tree, and src/tests is small. +- File-relpath post-filtering at the runner layer (SHIPPED 2026-07-19, reworked away + 2026-07-20 at user direction): treefilenames walked everything and + runtests_discovery::filter_paths matched include/exclude patterns against the + tailbase-relative FILE paths. It fixed the boundary miss and made exact file paths + valid patterns, but gave -include-paths a different meaning at the runtests layer + than in treefilenames (same name, same syntax, different referent), made the trailing + tail-glob axis partially redundant (a pattern's final segment also constrained the + filename), and lost walk pruning. The user's review framed the root problem as syntax + ergonomics (no zero-or-more-segments form) rather than the matching target - hence + the *** rework. Single-file targeting is now a directory pattern plus a file tail. +- Passing translated patterns down to treefilenames (X/** expanded to {X X/**} to pull + in boundary-direct files) - rejected: two spellings for one intent, and unable to + express per-pattern filename constraints. +- Changing treefilenames' X/** semantics to include boundary-direct files - rejected: + the in-dir / one-below / one-or-more-below separability is a deliberate design; *** + completes the lattice instead of collapsing it. ## Notes @@ -96,23 +109,38 @@ applies include/exclude patterns to the tailbase-relative file paths via globmat motivation examples: excluding shell/** when no built kit is present, runtimebash_wsl without WSL, or a heavy suite family unrelated to the change - most valuable for sequential/tcl8.6 contexts where full runs still cost minutes (-jobs 8 full runs are - ~1m26s post G-092). RESOLVED: the include-side root cause made the post-discovery - globmatchpath filter the home for both directions (see Root cause above). -- Behaviour change pinned deliberately: a bare directory path without a glob tail - (e.g -include-paths modules/punk/args) now matches NOTHING (append /** for a subtree). - Under the old directory-oriented semantics it matched files directly in that directory. - In exchange, an exact relative FILE path is now a valid single-file targeting pattern. - Neither form was documented at the runtests level before; all documented usage is X/**. + ~1m26s post G-092). RESOLVED (final form 2026-07-20): a thin passthrough to + treefilenames' own -exclude-paths, which gained the exclude-side separability fix + (**|***-tail prunes, other matches drop only that directory's files). +- Final semantics notes: a bare directory path X includes exactly the files directly in + X (the lattice in-dir form); X/** deliberately excludes files directly in X, and a + received include pattern contributing no files earns a stderr advisory pointing at + X/***; single-file targeting is an exact directory pattern plus a file tail + (-include-paths modules/punk/args/testsuites/args dynamic.test). The interim + 2026-07-19 file-relpath semantics (exact FILE paths as patterns; bare dir matching + nothing) lasted one day and are superseded - see Alternatives considered. +- punk::path 0.4.0 ride-alongs (same release, see path-buildversion.txt): the legacy + bare-* include collapse is removed (a * in treefilenames -include-paths no longer + means match-everything - it is the one-below lattice form; no repo caller used it), + and the globmatchpath default-nocase driveletter exception is now a direct regex-head + rewrite (the old file-split of the regex text broke on c:/***'s glued group and + accepted non-alpha drives). subfolders/subfolders1 rule-compiler machinery untouched + (its own inadequacy TODO stands in the module). +- -serial-paths reconciled to the same directory-of-file reading (default switched to + modules/punk/console/*** + modules/opunk/console/*** - same file set as the old + file-path-oriented console/** default): the pre-existing serial-vs-discovery + divergence was the codebase's other same-option-different-referent case; both gone. - runtests.tcl -discover-only 1 (added with the fix): prints the discovered list and exits before any run machinery - the observation channel used by the regression suite and for cheap human/agent targeting checks. Emits a well-formed tcl-list line 'RUNTESTS_FILES ...' (marker first element, emitted last). - Two-stage boot (added for -discover-only usability): discovery-phase loads only - punk::args + punk::path (via testsupport/discovery.tcl); punk/Thread/shellrun/ - punk::tcltestrun and the testinterp/child path assembly load after the -discover-only - exit. Discover-only child processes dropped ~11.2s -> ~0.5s, and the regression suite's - 8 end-to-end children dropped its file wall 93s -> ~7s (parallel-safe, well under the - post-G-092 ~31s max-child bound). + punk::args + punk::path (via testsupport/discovery.tcl, which requires punk::path + 0.4.0- - the runner parent resolves it from the refreshed bootsupport snapshot); + punk/Thread/shellrun/punk::tcltestrun and the testinterp/child path assembly load + after the -discover-only exit. Discover-only child processes dropped ~11.2s -> ~0.5s, + and the regression suite's end-to-end children dropped its file wall 93s -> ~7s + (parallel-safe, well under the post-G-092 ~31s max-child bound). - Verification context note: run verification through a context whose children have no attached Windows console (e.g the agent Bash tool; Git Bash/mintty). Under the agent PowerShell tool the children inherit an attached console with redirected stdio and the @@ -170,5 +198,72 @@ on the console-context requirement): under the stage-1 boot (textblock soft-require resolves from bootsupport, ~0.9s); goals_lint clean; no runtests_tmp leak from -discover-only. -Remaining for acceptance: none known - all acceptance clauses verified above; the -achieved flip is the user's call. +Status at the time: all original acceptance clauses verified. Superseded the next day by +the user-approved contract amendment and *** rework below. + +### 2026-07-20 rework to *** directory-oriented pass-through (user-approved contract amendment) + +User review of the 2026-07-19 increment identified the divergence cost (-include-paths +meaning different things at the runner than in treefilenames) and proposed the *** +syntax; the Goal/Acceptance contract was amended to the *** spelling with user approval +and the increment reworked the same day. + +Landed: + +- punk::path 0.4.0 (buildversion + changelog; root modules, bootsupport and + _vfscommon.vfs regenerated via make.tcl modules + bootsupport + vfscommonupdate): + *** zero-or-more-segments in pathglob_as_re (join-aware regex construction - a + trailing/medial/leading *** absorbs one adjacent separator inside an optional group) + and globmatchpath; prefix-viability treats *** like **; include separability fix + (an exact or single-segment-glob full match no longer sets allbelow); exclude + separability fix via the new exclude_state classifier (subtree prune only for **|*** + tails, both walks; dead _path_matches_any removed); legacy bare-* include collapse + removed; driveletter default-nocase exception now a direct regex-head rewrite. +- path.test +5 (40 total): *** matrix, *** drive-letter case, include lattice + separability, *** subsumes the {X/** X} pattern pair, exclude lattice separability. +- runtests.tcl: -include-paths/-exclude-paths pass through to treefilenames (one + directory-oriented semantics everywhere); argdocs rewritten to the lattice; + -serial-paths default modules/punk/console/*** + modules/opunk/console/*** with + directory-of-file classification (runtests_discovery::dir_matches_any - the + pre-existing file-path serial matching was the codebase's other + same-option-different-referent case); per-pattern zero-match stderr advisory + pointing X/** misses at X/***; json emitter guard for whitespace-padded + entier-looking values (a failed test's "0\n" result_was embedded a raw newline in + the json report - string is entier -strict accepts surrounding whitespace). +- testsupport/discovery.tcl: thin treefilenames passthrough + dir_matches_any + (requires punk::path 0.4.0-). +- pathdiscovery.test rewritten to the directory-semantics contract (22 tests): + classifier lattice, fixture-walk lattice including files-level vs subtree exclusion, + e2e X/*** four-subtree set, repeated-flag equivalence, exclude-minus-subtree, + dir+tail single-file targeting, and the X/** zero-match + advisory characterization. +- AGENTS.md sweep: src/tests contract bullets, Work Guidance and the -jobs serial + default mention; runner/AGENTS.md; and the /** -> /*** targeting examples in + src/tests/modules, src/tests/shell, src/tests/core, src/tests/core/tcl/testsuites/ + tests and src/modules/opunk AGENTS.md files. + +Verification (2026-07-20, native tclsh 9.0.3, Bash context, runs serialized with no +concurrent checkout edits - an earlier battery invalidated itself when +runtimebash_wsl_leaves_checkout_untouched failed under mid-run goal-file edits, which +is also what surfaced the json emitter bug): + +- Probes: 45-case globmatchpath/treefilenames matrix (both lattice directions, + trailing/medial/leading ***, drive case) all pass; path suite 40/40; runner suite + 22/22 (both driven through runtests with -include-paths /*** targeting). +- Acceptance (a): the four-subtree G-092 set in X/*** spelling discovers 13 files + across all four subtrees (pathdiscovery-3.0); the X/** spelling on the base-direct + scriptwrap suite matches zero files and emits the stderr advisory + (pathdiscovery-3.6). +- Acceptance (b): repeated X/*** flags accumulate and equal the single-value form + (pathdiscovery-3.1). +- Exclude: a full run with -exclude-paths X/*** equals the default list minus that + subtree, order preserved (pathdiscovery-3.3). +- Parity battery vs the pre-G-093 baseline (default -jobs 16: 93 files / 1010 tests / + failed=1 exec-14.3 only): clean exclude-runner run differs ONLY by path.test's five + new passing tests; clean default run differs ONLY by that plus the added runner + suite file; the baseline-vs-current default DISCOVERY SET differs by exactly the one + added suite file - default '**' discovery is unaffected. Mode parity post-rework: + -jobs 16 vs singleproc sequential PARITY ok (94 files / 1037 tests / failed=1). +- discover-only child ~0.5s under the two-stage boot; -help renders the new argdocs. + +Remaining for acceptance: none known - all amended acceptance clauses verified above; +the achieved flip is the user's call. diff --git a/src/modules/opunk/AGENTS.md b/src/modules/opunk/AGENTS.md index ad54926d..1da29ed5 100644 --- a/src/modules/opunk/AGENTS.md +++ b/src/modules/opunk/AGENTS.md @@ -33,7 +33,7 @@ Modules under the `opunk::*` namespace explore value-based class implementations ## Verification -- `tclsh src/tests/runtests.tcl -report compact -show-passes 0 -include-paths modules/opunk//**` passes for the touched module. +- `tclsh src/tests/runtests.tcl -report compact -show-passes 0 -include-paths modules/opunk//***` passes for the touched module. - `tclsh src/make.tcl modules` completes without errors after module additions. ## Child DOX Index diff --git a/src/modules/punk/path-999999.0a1.0.tm b/src/modules/punk/path-999999.0a1.0.tm index 5e9c9937..00b95114 100644 --- a/src/modules/punk/path-999999.0a1.0.tm +++ b/src/modules/punk/path-999999.0a1.0.tm @@ -725,9 +725,10 @@ namespace eval punk::path { proc pathglob_as_re {pathglob} { #*** !doctools #[call [fun pathglob_as_re] [arg pathglob]] - #[para] Returns a regular expression for matching a path to a glob pattern which can contain glob chars *|? in any segment of the path structure + #[para] Returns a regular expression for matching a path to a glob pattern which can contain glob chars *|**|***|? in any segment of the path structure #[para] Does not support square bracket globs or character classes. - #[para] ** matches any number of subdirectories. + #[para] ** as a whole segment matches one or more segments. + #[para] *** as a whole segment matches zero or more segments - e.g /etc/*** will match /etc itself as well as anything below it. (added 2026-07-20, G-093) #[para] e.g /etc/**/*.txt will match any .txt files at any depth below /etc (except directly within /etc itself) #[para] e.g /etc/**.txt will match any .txt files at any depth below /etc #[para] any segment that does not contain ** must match exactly one segment in the path @@ -738,6 +739,7 @@ namespace eval punk::path { #todo - consider whether a way to escape the glob chars ? * is practical - to allow literals ? * # - would require counting immediately-preceding backslashes + set zom "\x00zom\x00" ;#zero-or-more-segments marker for whole-segment *** (segment regexes never contain \x00) set pats [list] foreach seg [file split $pathglob] { if {[string range $seg end end] eq "/"} { @@ -746,6 +748,12 @@ namespace eval punk::path { switch -- $seg { * {lappend pats {[^/]*}} ** {lappend pats {.*}} + *** { + #zero or more whole segments - adjacent *** duplicates collapse to one + if {[lindex $pats end] ne $zom} { + lappend pats $zom + } + } default { set seg [string map [list ^ {\^} $ {\$} \[ {\[} \] {\]} ( {\(} ) {\)} \{ \\\{ \\ {\\}] $seg] ;#treat regex characters (or tcl glob square bracket chars) in the input as literals #set seg [string map [list . {[.]}] $seg] @@ -759,7 +767,34 @@ namespace eval punk::path { } } } - return "^[join $pats /]\$" + #join with / - except around zero-or-more markers, which must absorb one adjacent + #separator (a plain join cannot express 'zero segments'): + # X/*** -> ^X(?:/.*)?$ (trailing marker absorbs the preceding separator) + # ***/X a/***/b -> (?:.*/)?X ... (marker carries its trailing separator inside the + # optional group; no separator follows the marker) + # *** -> ^.*$ + set n [llength $pats] + set out "" + for {set i 0} {$i < $n} {incr i} { + set pat [lindex $pats $i] + set is_zom [expr {$pat eq $zom}] + set is_last [expr {$i == $n - 1}] + if {$i > 0 && [lindex $pats $i-1] ne $zom && !($is_zom && $is_last)} { + append out / + } + if {$is_zom} { + if {$n == 1} { + append out {.*} + } elseif {$is_last} { + append out {(?:/.*)?} + } else { + append out {(?:.*/)?} + } + } else { + append out $pat + } + } + return "^${out}\$" } punk::args::define { @@ -775,10 +810,12 @@ namespace eval punk::path { with 1 or more segments in between (so it will not match /usr/bin). A pattern such as /usr/** will match any path that has /usr as the first segment, with 1 or more segments following (so it will not match /usr itself). + A pattern such as /usr/*** will match /usr itself as well as any path below /usr + - *** as a whole segment matches zero or more segments (added 2026-07-20, G-093). A pattern such as **/*.txt will match any path that ends with .txt, with 1 or more leading segments - (so it will not match test.txt or .txt). + (so it will not match test.txt or .txt). Use ***/*.txt to also match a bare test.txt. A pattern such as ** will match any path. - The glob characters * and ? are the only special characters in the pathglob syntax. + The glob characters * and ? (and the whole-segment forms ** and ***) are the only special characters in the pathglob syntax. - they are treated as glob characters regardless of where they appear in the pathglob string. Note that this is different from other Tcl glob contexts where square brackets can be used. The pathglob syntax treats other characters, including square brackets as literals. @@ -849,12 +886,12 @@ namespace eval punk::path { set ismatch [regexp $re $path] ;#explicit -nocase 0 - require exact match of path literals including driveletter } else { #caller is using default for -nocase - which indicates case sensitivity - but we have an exception for the driveletter. - set re_segments [file split $re] ;#Note that file split c:/etc gives {c:/ etc} but file split ^c:/etc gives {^c: etc} - set first_seg [lindex $re_segments 0] - if {[regexp {^\^(.{1}):$} $first_seg _match driveletter]} { - #first part of re is like "^c:" i.e a drive letter - set chars [string tolower $driveletter][string toupper $driveletter] - set re [join [concat "^\[$chars\]:" [lrange $re_segments 1 end]] /] ;#rebuild re with case insensitive driveletter only - use join - not file join. file join will misinterpret leading re segment. + #Direct head rewrite of the anchored regex: if the pattern begins with a windows + #drive segment (^c:...) make just the drive letter class-insensitive. + #(The previous file-split of the regex text broke on forms where a group is glued + # to the drive - e.g ^c:(?:/.*)?$ from c:/*** - and allowed non-alpha 'drives'.) + if {[regexp {^\^([A-Za-z]):(.*)$} $re _match driveletter rest]} { + set re "^\[[string tolower $driveletter][string toupper $driveletter]\]:$rest" } #puts stderr "-->re: $re" set ismatch [regexp $re $path] @@ -1142,7 +1179,10 @@ namespace eval punk::path { set pattern_head [lindex $pattern_parts 0] set path_head [lindex $path_parts 0] - if {$pattern_head eq "**"} { + if {$pattern_head eq "**" || $pattern_head eq "***"} { + #both spanning forms admit consuming zero path segments here (viability only + #asks whether a match below remains possible - *** additionally full-matches + #with zero segments, which globmatchpath handles) if {[_pattern_prefix_viable_parts [lrange $pattern_parts 1 end] $path_parts]} { return 1 } @@ -1160,6 +1200,10 @@ namespace eval punk::path { } proc pattern_boundary {pattern} { + #the directory at which a trailing-** pattern's subtree begins (that directory + #itself does not full-match X/** - the boundary branch grants descent+allbelow + #without including its files). Trailing-*** patterns need no boundary: they + #full-match their base directory directly. set parts [file split $pattern] if {[llength $parts] >= 2 && [lindex $parts end] eq "**"} { return [file join {*}[lrange $parts 0 end-1]] @@ -1167,6 +1211,37 @@ namespace eval punk::path { return "" } + proc pattern_tail_spans {pattern} { + #1 when the pattern's final segment is ** or *** - a directory matching such a + #pattern implies every deeper directory also matches it (basis for the include + #allbelow shortcut and the exclude subtree prune). An exact or single-segment-glob + #tail does not extend below its match, keeping the in-dir (X), one-below (X/*), + #one-or-more-below (X/**) and zero-or-more-below (X/***) forms separable. + set last [lindex [file split $pattern] end] + return [expr {$last eq "**" || $last eq "***"}] + } + + proc exclude_state {exclude_paths path} { + #Directory-oriented exclusion classification (separability fix 2026-07-20, G-093 - + #previously ANY matching pattern pruned the whole subtree, so an exact exclude + #like **/_aside also killed everything below _aside, contradicting the argdoc's + #distinct **/_aside vs **/_aside/** forms and the punk::path::subfolders precedent): + # subtree - a matching pattern's final segment is **|*** : prune path and all below + # files - matched by non-spanning pattern(s) only: exclude the files directly in + # path, but keep walking below it + # none - not excluded + set state none + foreach xp $exclude_paths { + if {[::punk::path::globmatchpath $xp $path]} { + if {[pattern_tail_spans $xp]} { + return subtree + } + set state files + } + } + return $state + } + proc directory_state {glob_paths path inherited_allbelow} { if {$inherited_allbelow} { return [dict create include_files 1 recurse_below 1 next_allbelow 1] @@ -1180,8 +1255,17 @@ namespace eval punk::path { if {[::punk::path::globmatchpath $gp $path]} { set include_files 1 set recurse_below 1 - set next_allbelow 1 - break + if {[pattern_tail_spans $gp]} { + #trailing **|*** - every deeper directory also matches this pattern + set next_allbelow 1 + break + } + #exact or single-segment-glob tail: this match includes only the files + #directly here - later patterns may still grant descent/allbelow + #(previously any full match set next_allbelow and broke, dragging the + # whole subtree in for exact/X-star patterns - separability fix 2026-07-20, + # G-093; the zipfs walk already behaved separably) + continue } set boundary [pattern_boundary $gp] @@ -1233,15 +1317,6 @@ namespace eval punk::path { } } - proc _path_matches_any {patterns path} { - foreach pattern $patterns { - if {[::punk::path::globmatchpath $pattern $path]} { - return 1 - } - } - return 0 - } - proc _tailbase_relative {tailbase path} { if {$tailbase eq ""} { return $path @@ -1297,10 +1372,10 @@ namespace eval punk::path { set directory [pwd] } + #-include-paths patterns are taken as-is: a bare * is the one-below lattice form, + #not a legacy match-everything alias (special-case collapse removed 2026-07-20, + #G-093 - use ** or *** for match-everything) set glob_paths [dict get $opts -include-paths] - if {"*" in $glob_paths} { - set glob_paths {*} - } set sortmode [dict get $opts -sort] if {$sortmode eq "natural"} { @@ -1341,13 +1416,14 @@ namespace eval punk::path { return [walk_treefilenames_zipfs $state] } set opt_dir_match [_tailbase_match_path $opt_tailbase $opt_dir] - if {[_path_matches_any $opt_exclude_paths $opt_dir_match]} { + set dir_exclude_state [exclude_state $opt_exclude_paths $opt_dir_match] + if {$dir_exclude_state eq "subtree"} { return [list] } set files [list] set dir_state [directory_state $opt_glob_paths $opt_dir_match $callallbelow] - if {[dict get $dir_state include_files]} { + if {[dict get $dir_state include_files] && $dir_exclude_state eq "none"} { if {[catch {glob -nocomplain -dir $opt_dir -type f -- {*}$tailglobs} matches]} { puts stderr "treefilenames error while listing files in dir $opt_dir\n $matches" set dirfiles [list] @@ -1389,19 +1465,17 @@ namespace eval punk::path { } set okdirs [list] foreach dir $dirdirs { - if {![_path_matches_any $opt_exclude_paths [_tailbase_match_path $opt_tailbase $dir]]} { + #only subtree exclusions prune descent here - a child matched by a + #non-spanning exclude still walks (its own entry pass drops its files) + if {[exclude_state $opt_exclude_paths [_tailbase_match_path $opt_tailbase $dir]] ne "subtree"} { lappend okdirs $dir } } - if {$opt_glob_paths eq "*"} { - set matchdirs $okdirs - } else { - set matchdirs [list] - foreach dir $okdirs { - if {$callallbelow || [child_path_state $opt_glob_paths [_tailbase_match_path $opt_tailbase $dir] $callallbelow]} { - lappend matchdirs $dir - } + set matchdirs [list] + foreach dir $okdirs { + if {$callallbelow || [child_path_state $opt_glob_paths [_tailbase_match_path $opt_tailbase $dir] $callallbelow]} { + lappend matchdirs $dir } } @@ -1434,7 +1508,7 @@ namespace eval punk::path { error "treefilenames_zipfs can only be used on paths beginning with [zipfs root] on this systems" } set dir [string trimright $opt_dir "/"] - if {[_path_matches_any $opt_exclude_paths [_tailbase_match_path $opt_tailbase $dir]]} { + if {[exclude_state $opt_exclude_paths [_tailbase_match_path $opt_tailbase $dir]] eq "subtree"} { return [list] } set dirlen [string length $dir] @@ -1455,7 +1529,7 @@ namespace eval punk::path { break } if {$superpath ni $dirlist} { - if {[_path_matches_any $opt_exclude_paths [_tailbase_match_path $opt_tailbase $superpath]]} { + if {[exclude_state $opt_exclude_paths [_tailbase_match_path $opt_tailbase $superpath]] eq "subtree"} { lappend skipdirs $superpath set skipdir 1 break @@ -1481,10 +1555,15 @@ namespace eval punk::path { } else { set match 1 } - if {$match && $opt_glob_paths ne "*"} { + if {$match} { set file_dir_match [_tailbase_match_path $opt_tailbase [file dirname $finalpart]] set file_dir_state [directory_state $opt_glob_paths $file_dir_match 0] set match [dict get $file_dir_state include_files] + if {$match && [exclude_state $opt_exclude_paths $file_dir_match] ne "none"} { + #files-level exclusion of the immediate directory + #(subtree exclusions were pruned via the superpath loop) + set match 0 + } } if {$match} { set skipfile 0 @@ -1500,7 +1579,7 @@ namespace eval punk::path { } } else { if {$finalpart ni $dirlist} { - if {[_path_matches_any $opt_exclude_paths [_tailbase_match_path $opt_tailbase $finalpart]]} { + if {[exclude_state $opt_exclude_paths [_tailbase_match_path $opt_tailbase $finalpart]] eq "subtree"} { lappend skipdirs $finalpart } else { lappend dirlist $finalpart @@ -1525,9 +1604,10 @@ namespace eval punk::path { "List of filenames below path. The resulting list is unsorted. - The path globbing syntax supports *, ** and ? as glob characters in any segment of the path, with the following semantics: + The path globbing syntax supports *, **, *** and ? as glob characters in any segment of the path, with the following semantics: * matches any single segment in the path - ** matches 1 or more segments in the path (so /usr/**/bin will match /usr/x/bin and user/x/y/bin but not /usr/bin ) + ** as a whole segment matches 1 or more segments in the path (so /usr/**/bin will match /usr/x/bin and user/x/y/bin but not /usr/bin ) + *** as a whole segment matches 0 or more segments in the path (so /usr/*** will match /usr itself as well as anything below it) ? matches any single character in a single segment of the path (so /usr/te?t will match /usr/test and /usr/text but not /usr/texxt) " -directory -type directory -help\ @@ -1546,22 +1626,35 @@ namespace eval punk::path { " -sort -type any -default natural -choices {none ascii dictionary natural} -exclude-paths -default {} -help\ - "list of path patterns to exclude - may include * and ** path segments e.g - /usr/** (exclude subfolders based at /usr but not + "list of directory path patterns to exclude + may include *, ** and *** path segments e.g + /usr (exclude files directly within /usr; subfolders + are still walked) + /usr/* (exclude files in folders exactly one below /usr) + /usr/** (exclude the subtrees based at /usr but not files within /usr itself) + /usr/*** (exclude files within /usr and all its subfolders) **/_aside (exclude files where _aside is last segment) - **/_aside/* (exclude folders one below an _aside folder) - **/_aside/** (exclude files in all folders with _aside as a segment)" + **/_aside/** (exclude everything below any _aside segment) + Patterns whose final segment is ** or *** prune the matched + subtree; other matches exclude only the files directly in the + matching folder. Exclusion wins over inclusion for the same file." -exclude-files -default {} -include-paths -default {**} -help\ - "list of path patterns to include - may include * and ** path segments e.g + "list of directory path patterns to include + may include *, ** and *** path segments e.g + /usr (include files directly within /usr only) + /usr/* (include files in folders exactly one below /usr) /usr/** (include files in subfolders based at /usr but not files within /usr itself) + /usr/*** (include files within /usr and all its subfolders) **/_aside (include files where _aside is last segment in the folder) **/_aside/* (include files in folders one below an _aside folder) - **/_aside/** (include all files in folders with _aside as a segment)" + **/_aside/** (include all files in folders with _aside as a segment) + A directory matching a pattern contributes the files directly within + it; patterns whose final segment is ** or *** also include everything + below their match. The /usr, /usr/*, /usr/** and /usr/*** forms are + deliberately separable." @values -min 0 -max -1 -optional 1 -type string tailglobs -default * -multiple 1 -help\ "Patterns to match against filename portion (last segment) of each file path diff --git a/src/modules/punk/path-buildversion.txt b/src/modules/punk/path-buildversion.txt index 2ee20848..92118a84 100644 --- a/src/modules/punk/path-buildversion.txt +++ b/src/modules/punk/path-buildversion.txt @@ -1,5 +1,6 @@ -0.3.0 +0.4.0 #First line must be a semantic version number #all other lines are ignored. +#0.4.0 - pathglob syntax: *** as a whole segment matches ZERO or more segments (X/*** = X and everything below; ***/f.txt matches f.txt at any depth incl root) - globmatchpath/pathglob_as_re/treefilenames (G-093); treefilenames include separability fix: an exact or single-segment-glob full match no longer sets allbelow (previously X or X/* dragged the whole subtree in via the deep walk; zipfs walk already separable); treefilenames exclude separability fix: only **|***-tail exclude patterns prune the subtree - other matches exclude just that folder's files and the walk continues below (both walks; previously any dir match pruned, contradicting the **/_aside vs **/_aside/** argdoc distinction); globmatchpath default-nocase driveletter rewrite now a direct regex-head rewrite (handles c:/*** glued-group form, restricts to alpha drives); treefilenames legacy bare-* include collapse removed (a * in -include-paths no longer means match-everything - it is the one-below lattice form; use **|***) #0.3.0 - scriptlib_resolve: extensionless tcl scripts resolvable via first-line identification (new system::scriptfile_is_tcl helper - '# tcl' comment, tclsh/tclkit/wish shebang, or sh-trampoline); per-directory spelling order (exact name as called wins over .tcl-appended within each location, kit-internal dir precedence preserved); .kit added to known extensions (previously lib:x.kit wrongly searched x.kit.tcl); return dict keys reworked: scripttype (tcl/kit/py/pl/ps1/sh or empty) + notes added, scriptname now the matched relative name; directories no longer match a spelling (file isfile); now also used by app-punkshell/app_shellrun lib: handling (previously app-punkscript only) #0.2.2 - added scriptlib_resolve: shared resolution policy for the lib: