diff --git a/goals/G-035-mixed-tm-pkgindex-provision.md b/goals/G-035-mixed-tm-pkgindex-provision.md index 8d1679be..a3d32ea3 100644 --- a/goals/G-035-mixed-tm-pkgindex-provision.md +++ b/goals/G-035-mixed-tm-pkgindex-provision.md @@ -2,8 +2,8 @@ Status: proposed Scope: src/tests/modules/punk/libunknown/testsuites/ (characterization suite); src/modules/punk/libunknown-*.tm and src/modules/punk/packagepreference-999999.0a1.0.tm (as characterised, fixed only if outright bugs surface); src/modules/AGENTS.md + src/lib/AGENTS.md (resulting guidance) -Goal: the behaviour when the same package is provided both as a .tm module and as a pkgIndex.tcl-based library - same or differing versions, under the standard package unknown, punk::libunknown and punk::packagepreference - is characterised by committed tests, and the currently informal working rule ("avoid mixing provision forms for one package - unexpected behaviour even with libunknown's improvements") is either substantiated with the specific failure modes named in AGENTS.md guidance, or retired if the characterisation shows the machinery now handles mixing predictably. -Acceptance: a committed test suite (extending src/tests/modules/punk/libunknown/testsuites/) characterises at least: same name+version provided via .tm and via pkgIndex.tcl (which registration wins, and whether it is deterministic across scan-trigger orderings) under the standard scanner, under punk::libunknown, and with punk::packagepreference active; differing versions across the two forms (version selection integrity including package prefer latest, and whether the losing form's registration lingers); re-registration effects (package forget then re-require crossing forms); surprising-but-accepted behaviours are pinned with GAP/known-quirk comments (the fossilmove characterization pattern), outright bugs fixed or filed as goals; the resulting do/don't guidance lands in src/modules/AGENTS.md and src/lib/AGENTS.md naming the characterised failure modes (or explicitly lifting the avoid-mixing rule if unwarranted). +Goal: the per-chain behaviour when the same package is provided both as a .tm module and as a pkgIndex.tcl-based library is characterised by committed crossform tests. The avoid-mixing rule is already substantiated for stock Tcl (named mechanism: `$satisfied`-suppressed chain fallthrough producing silent older-version selection and order-dependent indexing; see the Identified Mechanism section) and neutralised under the punkshell chain (libunknown always falls through; the 0.2.3 namespace-context bug is fixed). The remaining work is the crossform characterization suite covering the still-open cases (same-version collisions, `package forget` crossing forms, packagepreference interaction) and AGENTS.md guidance naming the failure mode for stock Tcl and the safe condition for the punkshell chain. +Acceptance: a committed crossform test suite under src/tests/modules/punk/libunknown/testsuites/ (new suite, complementing the existing shadowing/ and discovery/ suites which cover .tm-vs-.tm tie-breaking and pkgIndex.tcl sourcing scope respectively) characterises the identified-mechanism cases and the still-open cases: (1) the identified mechanism (tm `$satisfied` suppressing the library scan) reproduced under the standard scanner and confirmed absent under the punkshell chain - a .tm providing an older version silently winning over a newer pkgIndex.tcl version under `prefer latest`; (2) same name+version via both forms (which registration wins and whether it is deterministic across trigger orderings) under each chain; (3) differing versions with the newer in pkgIndex.tcl (version selection integrity under `prefer latest`, and whether the losing form's ifneeded registration lingers and can resurface after `package forget then re-require`); (4) interaction with punk::packagepreference's `package require` overload; (5) the pkgIndex.tcl side-effect and sibling-registration consequences of suppressed vs unsuppressed library scans; all three chains (standard scanner, libunknown, packagepreference) are covered by the same fixture matrix; surprising-but-accepted behaviours are pinned with GAP/known-quirk comments; outright bugs are fixed or filed; the resulting guidance lands in src/modules/AGENTS.md and src/lib/AGENTS.md naming the stock-Tcl failure mode (silent older-version selection + order-dependent indexing suppression) and the punkshell safe condition (libunknown chain always falls through; uninit'd child interps and plain tclsh still see stock behaviour). ## Context @@ -19,45 +19,72 @@ Tcl packages arrive by two registration routes with different machinery: semantics, `auto_path` ordering rather than tm-list ordering. When the SAME package is provided by both routes - same version or different -versions - the interaction is not well characterised. The user's standing -informal rule (recalled 2026-07-07): **avoid mixing .tm and pkgIndex.tcl -provision for one package** - unexpected behaviour was observed even after -punk::libunknown's improvements to the unknown-handler chain. That rule is -currently folklore: undocumented, unproven, and untested. Meanwhile the punk -ecosystem genuinely straddles both forms (src/modules .tm trees vs src/lib -pkgIndex libraries; kits carry both; punk::packagepreference overloads +versions - the interaction has a now-identified mechanism (see below). The +avoid-mixing rule is substantiated for stock Tcl with a named failure mode +(silent older-version selection via the `$satisfied` early return; see the +Identified Mechanism section) and neutralised under the punkshell chain +(libunknown always falls through). Two co-causes of the original "unexpected +behaviour" reports are now pinned: (1) the `$satisfied` suppression, and +(2) the libunknown 0.2.2 namespace-context bug (pkgIndex.tcl's relative +`namespace eval` creating `::punk::libunknown::foo` instead of `::foo`), fixed +in 0.2.3 and pinned by `pkgindex_relative_namespace_is_global` in the +discovery suite. What remains uncharacterised is the per-chain crossform +matrix (same-version collisions, `package forget` crossing forms, +packagepreference interaction) and its AGENTS.md guidance. Meanwhile the +punk ecosystem genuinely straddles both forms (src/modules .tm trees vs +src/lib pkgIndex libraries; kits carry both; punk::packagepreference overloads `package require` on top of the unknown-handler chain), so accidental mixing is a realistic hazard - and same-version-different-content drift (the 2026-07-06 textblock incident class) would interact with it unpredictably. -## Questions the characterisation must answer - -- Same name+version via .tm and via pkgIndex.tcl: which registration wins? - Is it deterministic, or dependent on which scan ran first (tm scan vs - auto_path scan can be triggered in either order by unrelated requires)? -- Different versions across forms: does version selection stay correct - (including under `package prefer latest` and with alpha/dev versions)? - Does the losing form's `package ifneeded` registration linger, and can it - resurface via `package forget` + re-require? -- How do punk::libunknown and punk::packagepreference each change the - answers? (libunknown replaces/wraps the unknown chain; packagepreference - overloads `package require` itself - three layers that can each reorder - scans.) -- Are there differences across Tcl 8.6 and 9 (tm.tcl and tclPkgUnknown have - evolved)? +## Resolved questions (identified mechanism, 2026-07-27) + +- **Stock Tcl silently selects the older .tm version over a newer pkgIndex.tcl + version.** Reproduced under `prefer latest` with mixpkg 1.0 (.tm) vs 2.0 + (pkgIndex.tcl) on both 8.6.10 and 9.0.3: the standard chain returns 1.0, + `package versions` shows only 1.0, and the auto_path scan never runs for that + require (suppressed by `$satisfied`). Under the punkshell chain, both + versions register and `prefer latest` correctly selects 2.0. +- **The suppression is order-dependent at inter-package scope.** Whether a + given auto_path directory gets indexed at all depends on whether some + earlier, unrelated require happened to be satisfied by a .tm - so sibling + packages and side effects in the same pkgIndex.tcl may or may not run + depending on the order of unrelated `package require` calls. +- **The libunknown chain always falls through.** `zipfs_tm_UnknownHandler` + computes `$satisfied` but its early return is commented out; the chain + always proceeds to `zipfs_tclPkgUnknown`. This is affordable because the + epoch cache avoids re-scanning already-indexed directories. + +## Questions the characterisation suite must still answer + +- Same name+version via both forms: which registration wins, and is it + deterministic across trigger orderings? (The `$satisfied` mechanism does + not apply here - both scans must run since neither form satisfies alone + when versions match.) +- `package forget` + re-require crossing forms: does the losing form's + `ifneeded` registration linger, and can it resurface? +- How does punk::packagepreference's `package require` overload interact + with the crossform matrix? (It wraps `package require` itself, adding a + third layer on top of the two unknown-handler chains.) +- Are there differences across Tcl 8.6 and 9 in the still-open cases? ## Approach -- Extend the shadowing suite's child-interp probe pattern: scratch dirs - providing the same test package as a .tm and as a pkgIndex.tcl library, all - combinations of {same version, tm newer, pkgIndex newer} x {standard - scanner, libunknown, packagepreference} x scan-trigger orderings. +- New crossform test suite under src/tests/modules/punk/libunknown/testsuites/ + (complementing shadowing/ and discovery/), using the child-interp probe pattern + from both existing suites: scratch dirs providing the same test package as a + .tm and as a pkgIndex.tcl library, all combinations of {same version, tm + newer, pkgIndex newer} x {standard scanner, libunknown, packagepreference} + x scan-trigger orderings. The existing suites cover .tm-vs-.tm shadowing and + pkgIndex.tcl sourcing scope respectively; crossform is the missing matrix. - Pin observed behaviour; mark surprising-but-tolerable outcomes with GAP/known-quirk comments (the fossilmove characterization pattern) rather than encoding wishes; anything outright broken is fixed or filed. -- Convert the folklore into documentation: AGENTS.md guidance in src/modules - and src/lib stating either the substantiated avoid-mixing rule with its - named failure modes, or the conditions under which mixing is safe. +- Land the guidance: src/modules/AGENTS.md and src/lib/AGENTS.md name the + stock-Tcl failure mode (silent older-version selection + order-dependent + indexing suppression) and the punkshell safe condition (libunknown chain + always falls through; epoch cache makes this affordable; caveat: uninit'd + child interps and plain tclsh still see stock behaviour). ## Notes @@ -84,25 +111,26 @@ is a realistic hazard - and same-version-different-content drift (the installguard.test sources the source-tree commandstack+packagepreference pair into child interps by path (resolution-proof) - see goals/archive/G-160-commandstack-hygiene.md. -- Baseline moved 2026-07-27 (before activation): one such outright bug was found and - fixed under this Scope's "fixed only if outright bugs surface" clause - punk::libunknown - 0.2.2 -> 0.2.3. Index scripts were being sourced in a proc of the `punk::libunknown` - namespace, so a pkgIndex.tcl's relative `namespace eval foo` created - `::punk::libunknown::foo` rather than `::foo`; indexes that define commands at source - time for their own ifneeded scripts to call then failed at require time with `invalid - command name` (found on twapi 4.7.2 in a third-party 8.6 kit). Pinned by +- Co-cause of the original "unexpected behaviour" reports (identified alongside the + `$satisfied` mechanism): punk::libunknown 0.2.2 -> 0.2.3 fixed a namespace-context + bug where pkgIndex.tcl scripts were sourced in a proc of the `punk::libunknown` + namespace, so a relative `namespace eval foo` created `::punk::libunknown::foo` + rather than `::foo` (found on twapi 4.7.2 in a third-party 8.6 kit). Pinned by `pkgindex_relative_namespace_is_global` in - `src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test`. Relevant to - this goal's characterization in two ways: the *namespace context* of index sourcing is - now a pinned property to preserve alongside the frame isolation, and one of the - behaviours the "avoid mixing provision forms" folklore may have been describing has - turned out to be a plain bug rather than a mixing hazard. + `src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test`. The two + co-causes together - the stock-Tcl `$satisfied` suppression and the libunknown + namespace-context bug - explain the original folklore: (1) stock Tcl silently + suppressing the library scan made mixing unreliable, and (2) even when the library + scan did run under libunknown, the wrong namespace context broke pkgIndex.tcl + scripts that defined helper commands. Both are now fixed in the punkshell chain; + neither is a mixing-hazard inherent to having both forms present simultaneously. -### The folklore's mechanism, identified 2026-07-27 +### Identified mechanism (2026-07-27 investigation) -The developer's recollection (the rule came from `$satisfied` in the default -`::tcl::tm::UnknownHandler` aborting the search too early) is correct, and the mechanism -is now pinned down. Stock `tm.tcl` ends its scan with: +Two co-causes explain the original "avoid mixing" rule, both now pinned: + +**Co-cause 1: stock Tcl's `$satisfied` early return suppresses the library scan.** +Stock `tm.tcl` ends its scan with: if {$satisfied} { return @@ -121,6 +149,11 @@ still have to fallback to the regular package search"), i.e. a performance optim but it couples an unrelated question (did the tm scan happen to satisfy this request?) to whether the library half of the search runs at all. +punk::libunknown neutralises this: `zipfs_tm_UnknownHandler` still computes `$satisfied` +but its early return is commented out, so the chain ALWAYS falls through to +`zipfs_tclPkgUnknown`. This is what the epoch cache buys: unconditional fallthrough is +only affordable because already-scanned directories are not re-globbed within an epoch. + Reproduced with a synthetic fixture - one `.tm` providing `mixpkg 1.0`, one auto_path library providing `mixpkg 2.0` plus a sibling package and an index side effect, under `package prefer latest`. Identical results on Tcl 8.6.10 and 9.0.3: @@ -138,25 +171,25 @@ no error, no warning. The other rows explain the spooky-action reports: whether auto_path directory has been indexed at all depends on whether some earlier, unrelated require happened to be satisfied by a .tm, so behaviour varies with require ORDER. -punk::libunknown already neutralises this: `zipfs_tm_UnknownHandler` still computes -`satisfied` but its early return is commented out, so the chain ALWAYS falls through to -`zipfs_tclPkgUnknown`. That is the other half of the developer's recollection - and it is -what the epoch cache buys: unconditional fallthrough is only affordable because -already-scanned directories are not re-globbed within an epoch. The two mechanisms are -one design. +**Co-cause 2: libunknown 0.2.2 namespace-context bug (fixed 0.2.3).** Even when the +library scan ran under libunknown, pkgIndex.tcl scripts were sourced in a proc of the +`punk::libunknown` namespace, so a relative `namespace eval foo` created +`::punk::libunknown::foo` rather than `::foo`. Indexes that defined helper commands for +their ifneeded scripts then failed at require time with `invalid command name` (found on +twapi 4.7.2). Pinned by `pkgindex_relative_namespace_is_global` in the discovery suite. +This was a plain bug, not a mixing hazard inherent to having both forms present. -Consequences for this goal's framing (the Goal line contemplates either substantiating the -avoid-mixing rule or retiring it): +**Consequences for the avoid-mixing rule:** - The rule is SUBSTANTIATED for stock Tcl, with a named failure mode (silent older-version - selection) rather than vague "unexpected behaviour". -- It is NEUTRALISED under the punkshell chain, so the characterization should be - per-chain rather than a single verdict. + selection + order-dependent indexing suppression) rather than vague "unexpected behaviour". +- It is NEUTRALISED under the punkshell chain (libunknown always falls through; the + namespace-context bug is fixed). - Stock behaviour stays reachable inside punkshell's own world - any interp that has not run `punk::libunknown::init` (child interps created without it, including deliberate test probes; a plain tclsh consuming punk .tm trees). So guidance cannot simply say "libunknown handles it". -- Still open for the characterization proper: same-version (rather than newer-library) +- Still open for the characterization suite: same-version (rather than newer-library) collisions, whether the losing form's registration lingers and can resurface after `package forget`, and the interaction with punk::packagepreference's `package require` overload.