diff --git a/CHANGELOG.md b/CHANGELOG.md index 332b4276..d9fd30d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,19 @@ The latest `## [X.Y.Z]` header must match the `version` field in `punkproject.to Entries are newest-first; one bullet per notable change. See the root `AGENTS.md` "Project Versioning" section for the bump policy. +## [0.27.1] - 2026-07-27 + +- Third-party `pkgIndex.tcl` files that define commands at index-source time for their own + `package ifneeded` scripts to call now work again (punk::libunknown 0.2.2 -> 0.2.3). + 0.2.0 moved index sourcing into a proc of the `punk::libunknown` namespace to stop index + helper variables leaking into the global namespace; that also changed the *current + namespace*, so an index's relative `namespace eval foo` created + `::punk::libunknown::foo` instead of `::foo` - and stock `tclPkgUnknown`, being a proc in + `::`, has always given indexes global resolution. Such packages failed at require time + with `invalid command name`. The index body now runs via `apply` with `::` as its + namespace, keeping the frame isolation and restoring the resolution context. + Found on twapi 4.7.2 as shipped in a third-party 8.6 kit; the pattern is common. + ## [0.27.0] - 2026-07-27 - A kit that cannot boot is no longer deployed (G-125). `make.tcl bake` now checks each diff --git a/GOALS.md b/GOALS.md index 7e7222d5..baf7b7d4 100644 --- a/GOALS.md +++ b/GOALS.md @@ -198,7 +198,7 @@ Scope: src/modules/punk/mix/ (modpod mount path); vfs::zip availability in the r Detail: goals/G-034-modpod-codeinterp-tcl86.md ### G-035 [proposed] Characterise mixed .tm / pkgIndex.tcl provision of the same package -Scope: src/tests/modules/punk/libunknown/testsuites/ (characterization suite); src/modules/punk/libunknown-0.2.2.tm and src/modules/punk/packagepreference-999999.0a1.0.tm (as characterised, fixed only if outright bugs surface); src/modules/AGENTS.md + src/lib/AGENTS.md (resulting guidance) +Scope: src/tests/modules/punk/libunknown/testsuites/ (characterization suite); src/modules/punk/libunknown-*.tm and src/modules/punk/packagepreference-999999.0a1.0.tm (as characterised, fixed only if outright bugs surface); src/modules/AGENTS.md + src/lib/AGENTS.md (resulting guidance) Detail: goals/G-035-mixed-tm-pkgindex-provision.md ### G-038 [proposed] Piped-to-interactive restart continues the same session (context preserved) @@ -362,7 +362,7 @@ Scope: src/buildsuites/suite_tcl90/ (build905.zig debug knobs + tcl::test wiring Detail: goals/G-108-buildsuite-debug-tier.md ### G-109 [proposed] libunknown manifest-declared multi-name discovery for zip-based .tm modules -Scope: src/modules/punk/libunknown-0.2.2.tm (scan/registration); modpod (manifest emission + load stub; canonical source c:/repo/jn/tclmodules/modpod, vendored copies as consumed); src/modules/punk/mix/ (modpod wrap tooling as touched); src/tests/modules/punk/libunknown/ (new suites + scan-performance characterization) +Scope: src/modules/punk/libunknown-*.tm (scan/registration); modpod (manifest emission + load stub; canonical source c:/repo/jn/tclmodules/modpod, vendored copies as consumed); src/modules/punk/mix/ (modpod wrap tooling as touched); src/tests/modules/punk/libunknown/ (new suites + scan-performance characterization) Detail: goals/G-109-libunknown-manifest-multiname-tm.md ### G-110 [proposed] Shared-lib extraction cache: single content-addressed user cache vs per-run temp copies (investigation) diff --git a/goals/G-035-mixed-tm-pkgindex-provision.md b/goals/G-035-mixed-tm-pkgindex-provision.md index a3e319bc..67e22c95 100644 --- a/goals/G-035-mixed-tm-pkgindex-provision.md +++ b/goals/G-035-mixed-tm-pkgindex-provision.md @@ -1,7 +1,7 @@ # G-035 Characterise mixed .tm / pkgIndex.tcl provision of the same package Status: proposed -Scope: src/tests/modules/punk/libunknown/testsuites/ (characterization suite); src/modules/punk/libunknown-0.2.2.tm and src/modules/punk/packagepreference-999999.0a1.0.tm (as characterised, fixed only if outright bugs surface); src/modules/AGENTS.md + src/lib/AGENTS.md (resulting guidance) +Scope: src/tests/modules/punk/libunknown/testsuites/ (characterization suite); src/modules/punk/libunknown-*.tm and src/modules/punk/packagepreference-999999.0a1.0.tm (as characterised, fixed only if outright bugs surface); src/modules/AGENTS.md + src/lib/AGENTS.md (resulting guidance) Goal: the behaviour when the same package is provided both as a .tm module and as a pkgIndex.tcl-based library - same or differing versions, under the standard package unknown, punk::libunknown and punk::packagepreference - is characterised by committed tests, and the currently informal working rule ("avoid mixing provision forms for one package - unexpected behaviour even with libunknown's improvements") is either substantiated with the specific failure modes named in AGENTS.md guidance, or retired if the characterisation shows the machinery now handles mixing predictably. Acceptance: a committed test suite (extending src/tests/modules/punk/libunknown/testsuites/) characterises at least: same name+version provided via .tm and via pkgIndex.tcl (which registration wins, and whether it is deterministic across scan-trigger orderings) under the standard scanner, under punk::libunknown, and with punk::packagepreference active; differing versions across the two forms (version selection integrity including package prefer latest, and whether the losing form's registration lingers); re-registration effects (package forget then re-require crossing forms); surprising-but-accepted behaviours are pinned with GAP/known-quirk comments (the fossilmove characterization pattern), outright bugs fixed or filed as goals; the resulting do/don't guidance lands in src/modules/AGENTS.md and src/lib/AGENTS.md naming the characterised failure modes (or explicitly lifting the avoid-mixing rule if unwarranted). @@ -73,3 +73,82 @@ is a realistic hazard - and same-version-different-content drift (the on the swept text (no live edit-window coordination remains). - Out of scope: redesigning the loading machinery. This goal is characterise-document-and-pin; behavioural changes only for outright bugs. +- Baseline moved 2026-07-27 (before activation): one such outright bug was found and + fixed under this Scope's "fixed only if outright bugs surface" clause - punk::libunknown + 0.2.2 -> 0.2.3. Index scripts were being sourced in a proc of the `punk::libunknown` + namespace, so a pkgIndex.tcl's relative `namespace eval foo` created + `::punk::libunknown::foo` rather than `::foo`; indexes that define commands at source + time for their own ifneeded scripts to call then failed at require time with `invalid + command name` (found on twapi 4.7.2 in a third-party 8.6 kit). Pinned by + `pkgindex_relative_namespace_is_global` in + `src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test`. Relevant to + this goal's characterization in two ways: the *namespace context* of index sourcing is + now a pinned property to preserve alongside the frame isolation, and one of the + behaviours the "avoid mixing provision forms" folklore may have been describing has + turned out to be a plain bug rather than a mixing hazard. + +### The folklore's mechanism, identified 2026-07-27 + +The developer's recollection (the rule came from `$satisfied` in the default +`::tcl::tm::UnknownHandler` aborting the search too early) is correct, and the mechanism +is now pinned down. Stock `tm.tcl` ends its scan with: + + if {$satisfied} { + return + } + # Fallback to previous command, if existing. + if {[llength $original]} { + uplevel 1 $original [::linsert $args 0 $name] + } + +`satisfied` is set as soon as a .tm candidate matches the requested name AND version +requirement, and `$original` is the rest of the unknown chain - i.e. `::tclPkgUnknown`, +the auto_path/pkgIndex.tcl scanner. So **a require satisfied from a .tm suppresses the +entire library scan for that require**. The in-source comment shows it is deliberate +("We abort in this unknown handler only if we got a satisfying candidate ... Otherwise we +still have to fallback to the regular package search"), i.e. a performance optimisation - +but it couples an unrelated question (did the tm scan happen to satisfy this request?) to +whether the library half of the search runs at all. + +Reproduced with a synthetic fixture - one `.tm` providing `mixpkg 1.0`, one auto_path +library providing `mixpkg 2.0` plus a sibling package and an index side effect, under +`package prefer latest`. Identical results on Tcl 8.6.10 and 9.0.3: + +| observation after `package require mixpkg` | stock chain | libunknown chain | +|---|---|---| +| version returned | **1.0** | 2.0 | +| `package versions mixpkg` | **1.0** | 1.0 2.0 | +| index side effect ran | **no** | yes | +| sibling `sibpkg` registered | **no** | yes | + +The first row is the sharp one: with both forms present, stock Tcl silently returns the +OLDER .tm version while a newer library version exists and `prefer latest` is in force - +no error, no warning. The other rows explain the spooky-action reports: whether a given +auto_path directory has been indexed at all depends on whether some earlier, unrelated +require happened to be satisfied by a .tm, so behaviour varies with require ORDER. + +punk::libunknown already neutralises this: `zipfs_tm_UnknownHandler` still computes +`satisfied` but its early return is commented out, so the chain ALWAYS falls through to +`zipfs_tclPkgUnknown`. That is the other half of the developer's recollection - and it is +what the epoch cache buys: unconditional fallthrough is only affordable because +already-scanned directories are not re-globbed within an epoch. The two mechanisms are +one design. + +Consequences for this goal's framing (the Goal line contemplates either substantiating the +avoid-mixing rule or retiring it): + +- The rule is SUBSTANTIATED for stock Tcl, with a named failure mode (silent older-version + selection) rather than vague "unexpected behaviour". +- It is NEUTRALISED under the punkshell chain, so the characterization should be + per-chain rather than a single verdict. +- Stock behaviour stays reachable inside punkshell's own world - any interp that has not + run `punk::libunknown::init` (child interps created without it, including deliberate + test probes; a plain tclsh consuming punk .tm trees). So guidance cannot simply say + "libunknown handles it". +- Still open for the characterization proper: same-version (rather than newer-library) + collisions, whether the losing form's registration lingers and can resurface after + `package forget`, and the interaction with punk::packagepreference's `package require` + overload. + +Reproduction script kept out of tree (session scratchpad `folklore.tcl`); the fixture is +small enough to re-create from the table above when the suite is written. diff --git a/goals/G-109-libunknown-manifest-multiname-tm.md b/goals/G-109-libunknown-manifest-multiname-tm.md index efe66b5b..c24bede1 100644 --- a/goals/G-109-libunknown-manifest-multiname-tm.md +++ b/goals/G-109-libunknown-manifest-multiname-tm.md @@ -1,7 +1,7 @@ # G-109 libunknown manifest-declared multi-name discovery for zip-based .tm modules Status: proposed -Scope: src/modules/punk/libunknown-0.2.2.tm (scan/registration); modpod (manifest emission + load stub; canonical source c:/repo/jn/tclmodules/modpod, vendored copies as consumed); src/modules/punk/mix/ (modpod wrap tooling as touched); src/tests/modules/punk/libunknown/ (new suites + scan-performance characterization) +Scope: src/modules/punk/libunknown-*.tm (scan/registration); modpod (manifest emission + load stub; canonical source c:/repo/jn/tclmodules/modpod, vendored copies as consumed); src/modules/punk/mix/ (modpod wrap tooling as touched); src/tests/modules/punk/libunknown/ (new suites + scan-performance characterization) Goal: a zip-based .tm (modpod) can declare additional package-require names - case variants (Thread/thread) and sub-packages - in a manifest at a well-known internal location, and punkshell's libunknown discovers those declarations during module scanning and registers them so package require of ANY declared name loads the single .tm file; real-disk scan performance is preserved: libunknown exists to fix slow-drive/large-multi-path-tree scan cost, so with manifest support enabled the worst-case target is parity with plain tclsh's builtin tm handler on the same tree (zipfs trees remain the fast path). Acceptance: a demonstration modpod .tm declaring a case-variant name and a sub-package name resolves via package require of each declared name on the tcl9 kit (8.6 covered, or its limitation recorded per G-034), from both a real-disk module path and a zipfs module path; the manifest location and format are documented and emitted by the modpod wrap tooling; plain-tclsh behaviour is documented prominently (the filename-derived name loads anywhere zipfs exists; extra names are punkshell/libunknown-only); a scan-performance characterization test compares libunknown-with-manifest-support against plain tclsh's builtin handler over a generated large multi-path real-disk tree and records the numbers in this file showing no worse than parity; extensive suites under src/tests/modules/punk/libunknown/ cover registration, case-variant and sub-package loading, refresh/epoch staleness, and the no-manifest fast path. diff --git a/punkproject.toml b/punkproject.toml index 80a3138e..c3c6d304 100644 --- a/punkproject.toml +++ b/punkproject.toml @@ -1,4 +1,4 @@ [project] name = "punkshell" -version = "0.27.0" +version = "0.27.1" license = "BSD-2-Clause" diff --git a/src/modules/punk/libunknown-0.2.2.tm b/src/modules/punk/libunknown-0.2.3.tm similarity index 98% rename from src/modules/punk/libunknown-0.2.2.tm rename to src/modules/punk/libunknown-0.2.3.tm index 471c5e0e..22a9599b 100644 --- a/src/modules/punk/libunknown-0.2.2.tm +++ b/src/modules/punk/libunknown-0.2.3.tm @@ -7,7 +7,7 @@ # (C) 2025 # # @@ Meta Begin -# Application punk::libunknown 0.2.2 +# Application punk::libunknown 0.2.3 # Meta platform tcl # Meta license MIT # @@ Meta End @@ -18,6 +18,17 @@ # from src/modules/AGENTS.md "Versioning And Releases" - bumping means # renaming the file AND updating the Meta line above, the manpage_begin line # below and the provide-block version, then appending a line here): +#0.2.3 - source_pkgindex: index scripts run in the GLOBAL namespace again (in a +# local frame, as 0.2.0 made them). 0.2.0 moved the source into a proc of +# THIS namespace, so a pkgIndex.tcl's relative 'namespace eval foo' created +# ::punk::libunknown::foo instead of ::foo - stock tclPkgUnknown lives in :: +# and indexes have always resolved relative names against the global +# namespace. Indexes that define commands at source time for their own +# ifneeded scripts to call therefore failed at require time with 'invalid +# command name'. Found 2026-07-27 on twapi 4.7.2 (its index defines +# twapi::set_scriptdir and its ifneeded script calls it); the pattern is +# common in third-party indexes. Frame isolation is preserved - the body +# runs via apply with :: as its namespace. #0.2.2 - zipfs_tm_UnknownHandler: the builtin-static shortcircuit probe interp no # longer leaks (G-120). The probe child interp was created under the fixed # name ptest and deleted only on the non-error paths; an unversioned @@ -55,7 +66,7 @@ # doctools header # ++ +++ +++ +++ +++ +++ +++ +++ +++ +++ +++ #*** !doctools -#[manpage_begin punkshell_module_punk::libunknown 0 0.2.2] +#[manpage_begin punkshell_module_punk::libunknown 0 0.2.3] #[copyright "2025"] #[titledesc {Module API}] [comment {-- Name section and table of contents description --}] #[moddesc {-}] [comment {-- Description at end of page heading --}] @@ -142,8 +153,9 @@ tcl::namespace::eval ::punk::libunknown { } } - #Execute a pkgIndex.tcl script in this proc's frame (tcl_Pkg_source uplevels - #the actual 'source' into its caller). + #Execute a pkgIndex.tcl script in a local frame whose CURRENT NAMESPACE is :: + #(tcl_Pkg_source uplevels the actual 'source' into its caller - here, the apply + #body's frame). #The pkgIndex.tcl contract is that $dir holds the index file's directory, and #stock tclPkgUnknown additionally exposes the auto_path and env globals (some #indexes, e.g tcllib's, extend auto_path with an unqualified lappend). @@ -152,9 +164,18 @@ tcl::namespace::eval ::punk::libunknown { #via 'namespace eval ::', which clobbered any user global named 'dir' (via the #handler's since-removed 'global dir') and leaked each index's helper #variables (ver, pkg, script, ...) into the global namespace. + #The namespace matters as much as the frame: stock tclPkgUnknown is a proc in + #::, so an index's relative 'namespace eval foo' has always created ::foo. Run + #here in a proc of THIS namespace it would create ::punk::libunknown::foo, and + #an index that defines commands at source time for its own ifneeded scripts to + #call (twapi 4.7.2's twapi::set_scriptdir is one) would then fail at require + #time with 'invalid command name'. apply's namespace argument gives us the + #local frame AND the global resolution context. proc source_pkgindex {dir indexfile} { - global auto_path env - tcl_Pkg_source $indexfile + apply [list {dir indexfile} { + global auto_path env + ::punk::libunknown::tcl_Pkg_source $indexfile + } ::] $dir $indexfile } namespace eval argdoc { @@ -2157,7 +2178,7 @@ namespace eval ::punk::args::register { package provide punk::libunknown [tcl::namespace::eval ::punk::libunknown { variable pkg punk::libunknown variable version - set version 0.2.2 + set version 0.2.3 }] return diff --git a/src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test b/src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test index 49da9a41..d5ec1732 100644 --- a/src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test +++ b/src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test @@ -65,6 +65,7 @@ namespace eval ::testspace { set libbase [file join $base libbase] file mkdir [file join $fixdir pkluns subns] $mutdir $mut2dir $sdirA $sdirB $epochextra\ [file join $libbase pklu_pkgidx]\ + [file join $libbase pklu_nsindex]\ [file join $libbase pklu_extender sublib pklu_ext] foreach {relpath content} { {pklu_top-1.0.tm} {package provide pklu_top 1.0} @@ -95,6 +96,17 @@ namespace eval ::testspace { {package ifneeded pklu_pkgidx 1.0 [list source [file join $dir pklu_pkgidx.tcl]]} {pklu_pkgidx/pklu_pkgidx.tcl} {package provide pklu_pkgidx 1.0} + {pklu_nsindex/pkgIndex.tcl} + {namespace eval pklu_nsindex { + variable scriptdir + proc set_scriptdir dir {variable scriptdir ; set scriptdir $dir} + } + package ifneeded pklu_nsindex 1.0 [subst { + pklu_nsindex::set_scriptdir [list $dir] + source [list [file join $dir pklu_nsindex.tcl]] + }]} + {pklu_nsindex/pklu_nsindex.tcl} + {package provide pklu_nsindex 1.0} {pklu_extender/pkgIndex.tcl} {lappend auto_path [file join $dir sublib] set pklu_leak_probe leaked_value} @@ -359,6 +371,29 @@ namespace eval ::testspace { }] } -result {1.0 0} + #added 2026-07-27 (agent) - punk::libunknown 0.2.3 regression pin. 0.2.0 moved index + #sourcing into a proc of the punk::libunknown namespace, so an index's RELATIVE + #'namespace eval foo' created ::punk::libunknown::foo instead of ::foo. Indexes that + #define commands at source time for their own ifneeded scripts to call then failed at + #require time with 'invalid command name' - found on twapi 4.7.2 (its index defines + #twapi::set_scriptdir; its ifneeded script calls it), whose pattern this fixture copies. + test pkgindex_relative_namespace_is_global {a pkgIndex.tcl's relative 'namespace eval' creates a GLOBAL namespace, so commands it defines for its own ifneeded script resolve at require time}\ + -constraints {discoveryfixture libunknownavailable} -body { + variable fixdir + variable libbase + libu_probe [list $fixdir] [string map [list %LIBBASE% [list $libbase]] { + set ::auto_path [list %LIBBASE%] + set ::t_result [list] + lappend ::t_result [package require pklu_nsindex] + #the index-defined command must live where the ifneeded script looks for it + lappend ::t_result [info commands ::pklu_nsindex::set_scriptdir] + #and the sourcing namespace must not have collected it instead + lappend ::t_result [namespace exists ::punk::libunknown::pklu_nsindex] + #the variable the index set through its own command is readable too + lappend ::t_result [expr {[info exists ::pklu_nsindex::scriptdir] ? 1 : 0}] + }] + } -result {1.0 ::pklu_nsindex::set_scriptdir 0 1} + test pkgindex_auto_path_extension {a pkgIndex extending auto_path with an unqualified lappend still works (tcllib pattern)}\ -constraints {discoveryfixture libunknownavailable} -body { variable fixdir