punkshell/goals/G-035-mixed-tm-pkgindex-provision.md

G-035 Characterise mixed .tm / pkgIndex.tcl provision of the same package

Status: proposed Scope: src/tests/modules/punk/libunknown/testsuites/ (characterization suite); src/modules/punk/libunknown-*.tm and src/modules/punk/packagepreference-999999.0a1.0.tm (as characterised, fixed only if outright bugs surface); src/modules/AGENTS.md + src/lib/AGENTS.md (resulting guidance) Goal: the per-chain behaviour when the same package is provided both as a .tm module and as a pkgIndex.tcl-based library is characterised by committed crossform tests. The avoid-mixing rule is already substantiated for stock Tcl (named mechanism: $satisfied-suppressed chain fallthrough producing silent older-version selection and order-dependent indexing; see the Identified Mechanism section) and neutralised under the punkshell chain (libunknown always falls through; the 0.2.3 namespace-context bug is fixed). The remaining work is the crossform characterization suite covering the still-open cases (same-version collisions, package forget crossing forms, packagepreference interaction) and AGENTS.md guidance naming the failure mode for stock Tcl and the safe condition for the punkshell chain. Acceptance: a committed crossform test suite under src/tests/modules/punk/libunknown/testsuites/ (new suite, complementing the existing shadowing/ and discovery/ suites which cover .tm-vs-.tm tie-breaking and pkgIndex.tcl sourcing scope respectively) characterises the identified-mechanism cases and the still-open cases: (1) the identified mechanism (tm $satisfied suppressing the library scan) reproduced under the standard scanner and confirmed absent under the punkshell chain - a .tm providing an older version silently winning over a newer pkgIndex.tcl version under prefer latest; (2) same name+version via both forms (which registration wins and whether it is deterministic across trigger orderings) under each chain; (3) differing versions with the newer in pkgIndex.tcl (version selection integrity under prefer latest, and whether the losing form's ifneeded registration lingers and can resurface after package forget then re-require); (4) interaction with punk::packagepreference's package require overload; (5) the pkgIndex.tcl side-effect and sibling-registration consequences of suppressed vs unsuppressed library scans; all three chains (standard scanner, libunknown, packagepreference) are covered by the same fixture matrix; surprising-but-accepted behaviours are pinned with GAP/known-quirk comments; outright bugs are fixed or filed; the resulting guidance lands in src/modules/AGENTS.md and src/lib/AGENTS.md naming the stock-Tcl failure mode (silent older-version selection + order-dependent indexing suppression) and the punkshell safe condition (libunknown chain always falls through; uninit'd child interps and plain tclsh still see stock behaviour).

Context

Tcl packages arrive by two registration routes with different machinery:

• .tm modules: tcl::tm path scan during package unknown; registration keyed on name-version from the filename; the same-version tie-break rules are now pinned by src/tests/modules/punk/libunknown/testsuites/shadowing/ (2026-07-07): tcl:™️:add prepends, head of tcl:™️:list wins exact-version ties, version beats order, punk::libunknown preserves all of it.
• pkgIndex.tcl libraries: auto_path directory scan (tclPkgUnknown) evaluating index scripts; different scan timing, different overwrite semantics, auto_path ordering rather than tm-list ordering.

When the SAME package is provided by both routes - same version or different versions - the interaction has a now-identified mechanism (see below). The avoid-mixing rule is substantiated for stock Tcl with a named failure mode (silent older-version selection via the $satisfied early return; see the Identified Mechanism section) and neutralised under the punkshell chain (libunknown always falls through). Two co-causes of the original "unexpected behaviour" reports are now pinned: (1) the $satisfied suppression, and (2) the libunknown 0.2.2 namespace-context bug (pkgIndex.tcl's relative namespace eval creating ::punk::libunknown::foo instead of ::foo), fixed in 0.2.3 and pinned by pkgindex_relative_namespace_is_global in the discovery suite. What remains uncharacterised is the per-chain crossform matrix (same-version collisions, package forget crossing forms, packagepreference interaction) and its AGENTS.md guidance. Meanwhile the punk ecosystem genuinely straddles both forms (src/modules .tm trees vs src/lib pkgIndex libraries; kits carry both; punk::packagepreference overloads package require on top of the unknown-handler chain), so accidental mixing is a realistic hazard - and same-version-different-content drift (the 2026-07-06 textblock incident class) would interact with it unpredictably.

Resolved questions (identified mechanism, 2026-07-27)

• Stock Tcl silently selects the older .tm version over a newer pkgIndex.tcl version. Reproduced under prefer latest with mixpkg 1.0 (.tm) vs 2.0 (pkgIndex.tcl) on both 8.6.10 and 9.0.3: the standard chain returns 1.0, package versions shows only 1.0, and the auto_path scan never runs for that require (suppressed by $satisfied). Under the punkshell chain, both versions register and prefer latest correctly selects 2.0.
• The suppression is order-dependent at inter-package scope. Whether a given auto_path directory gets indexed at all depends on whether some earlier, unrelated require happened to be satisfied by a .tm - so sibling packages and side effects in the same pkgIndex.tcl may or may not run depending on the order of unrelated package require calls.
• The libunknown chain always falls through. zipfs_tm_UnknownHandler computes $satisfied but its early return is commented out; the chain always proceeds to zipfs_tclPkgUnknown. This is affordable because the epoch cache avoids re-scanning already-indexed directories.

Questions the characterisation suite must still answer

• Same name+version via both forms: which registration wins, and is it deterministic across trigger orderings? (The $satisfied mechanism does not apply here - both scans must run since neither form satisfies alone when versions match.)
• package forget + re-require crossing forms: does the losing form's ifneeded registration linger, and can it resurface?
• How does punk::packagepreference's package require overload interact with the crossform matrix? (It wraps package require itself, adding a third layer on top of the two unknown-handler chains.)
• Are there differences across Tcl 8.6 and 9 in the still-open cases?

Approach

• New crossform test suite under src/tests/modules/punk/libunknown/testsuites/ (complementing shadowing/ and discovery/), using the child-interp probe pattern from both existing suites: scratch dirs providing the same test package as a .tm and as a pkgIndex.tcl library, all combinations of {same version, tm newer, pkgIndex newer} x {standard scanner, libunknown, packagepreference} x scan-trigger orderings. The existing suites cover .tm-vs-.tm shadowing and pkgIndex.tcl sourcing scope respectively; crossform is the missing matrix.
• Pin observed behaviour; mark surprising-but-tolerable outcomes with GAP/known-quirk comments (the fossilmove characterization pattern) rather than encoding wishes; anything outright broken is fixed or filed.
• Land the guidance: src/modules/AGENTS.md and src/lib/AGENTS.md name the stock-Tcl failure mode (silent older-version selection + order-dependent indexing suppression) and the punkshell safe condition (libunknown chain always falls through; epoch cache makes this affordable; caveat: uninit'd child interps and plain tclsh still see stock behaviour).

Notes

• Related: the shadowing pin-tests (same suite family - this goal's tests extend them), G-033 (achieved 2026-07-28, goals/archive/G-033-proj-mode-cwd-project.md - visitor-mode path mixing is now live, making cross-form collisions more reachable: a kit's internal libs vs a visited project's .tm trees), G-026/G-024 (same-version drift provenance; G-024 achieved 2026-07-31 - goals/archive/G-024-mapvfs-toml.md), punk::packagepreference 0.1.1 (its moduledoc auto-load hook lives in exactly this machinery).
• G-112 relationship (recorded 2026-07-24; G-112 achieved 2026-07-25 - see goals/archive/G-112-maketcl-subcommand-rename.md): its rename sweep updated the shared src/modules/AGENTS.md + src/lib/AGENTS.md guidance to the bake/bakehouse vocabulary - this goal's characterization outcomes now land on the swept text (no live edit-window coordination remains).
• Out of scope: redesigning the loading machinery. This goal is characterise-document-and-pin; behavioural changes only for outright bugs.
• G-160 (archived) fixed the sibling packagepreference surface this goal will drive: the install re-install guard reads the record key that exists (renamer - a second install returns 0 silently instead of throwing, packagepreference 0.2.1) and commandstack 0.6.0 gives packagepreference's ::package rename unique tokenids, an O(1) dispatch map, and debug-gated warnings. New probe pattern available: testsuites/packagepreference/ installguard.test sources the source-tree commandstack+packagepreference pair into child interps by path (resolution-proof) - see goals/archive/G-160-commandstack-hygiene.md.
• Co-cause of the original "unexpected behaviour" reports (identified alongside the $satisfied mechanism): punk::libunknown 0.2.2 -> 0.2.3 fixed a namespace-context bug where pkgIndex.tcl scripts were sourced in a proc of the punk::libunknown namespace, so a relative namespace eval foo created ::punk::libunknown::foo rather than ::foo (found on twapi 4.7.2 in a third-party 8.6 kit). Pinned by pkgindex_relative_namespace_is_global in src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test. The two co-causes together - the stock-Tcl $satisfied suppression and the libunknown namespace-context bug - explain the original folklore: (1) stock Tcl silently suppressing the library scan made mixing unreliable, and (2) even when the library scan did run under libunknown, the wrong namespace context broke pkgIndex.tcl scripts that defined helper commands. Both are now fixed in the punkshell chain; neither is a mixing-hazard inherent to having both forms present simultaneously.

Identified mechanism (2026-07-27 investigation)

Two co-causes explain the original "avoid mixing" rule, both now pinned:

Co-cause 1: stock Tcl's $satisfied early return suppresses the library scan. Stock tm.tcl ends its scan with:

if {$satisfied} {
    return
}
# Fallback to previous command, if existing.
if {[llength $original]} {
    uplevel 1 $original [::linsert $args 0 $name]
}

satisfied is set as soon as a .tm candidate matches the requested name AND version requirement, and $original is the rest of the unknown chain - i.e. ::tclPkgUnknown, the auto_path/pkgIndex.tcl scanner. So a require satisfied from a .tm suppresses the entire library scan for that require. The in-source comment shows it is deliberate ("We abort in this unknown handler only if we got a satisfying candidate ... Otherwise we still have to fallback to the regular package search"), i.e. a performance optimisation - but it couples an unrelated question (did the tm scan happen to satisfy this request?) to whether the library half of the search runs at all.

punk::libunknown neutralises this: zipfs_tm_UnknownHandler still computes $satisfied but its early return is commented out, so the chain ALWAYS falls through to zipfs_tclPkgUnknown. This is what the epoch cache buys: unconditional fallthrough is only affordable because already-scanned directories are not re-globbed within an epoch.

Reproduced with a synthetic fixture - one .tm providing mixpkg 1.0, one auto_path library providing mixpkg 2.0 plus a sibling package and an index side effect, under package prefer latest. Identical results on Tcl 8.6.10 and 9.0.3:

observation after package require mixpkg	stock chain	libunknown chain
version returned	1.0	2.0
package versions mixpkg	1.0	1.0 2.0
index side effect ran	no	yes
sibling sibpkg registered	no	yes

The first row is the sharp one: with both forms present, stock Tcl silently returns the OLDER .tm version while a newer library version exists and prefer latest is in force - no error, no warning. The other rows explain the spooky-action reports: whether a given auto_path directory has been indexed at all depends on whether some earlier, unrelated require happened to be satisfied by a .tm, so behaviour varies with require ORDER.

Co-cause 2: libunknown 0.2.2 namespace-context bug (fixed 0.2.3). Even when the library scan ran under libunknown, pkgIndex.tcl scripts were sourced in a proc of the punk::libunknown namespace, so a relative namespace eval foo created ::punk::libunknown::foo rather than ::foo. Indexes that defined helper commands for their ifneeded scripts then failed at require time with invalid command name (found on twapi 4.7.2). Pinned by pkgindex_relative_namespace_is_global in the discovery suite. This was a plain bug, not a mixing hazard inherent to having both forms present.

Consequences for the avoid-mixing rule:

• The rule is SUBSTANTIATED for stock Tcl, with a named failure mode (silent older-version selection + order-dependent indexing suppression) rather than vague "unexpected behaviour".
• It is NEUTRALISED under the punkshell chain (libunknown always falls through; the namespace-context bug is fixed).
• Stock behaviour stays reachable inside punkshell's own world - any interp that has not run punk::libunknown::init (child interps created without it, including deliberate test probes; a plain tclsh consuming punk .tm trees). So guidance cannot simply say "libunknown handles it".
• Still open for the characterization suite: same-version (rather than newer-library) collisions, whether the losing form's registration lingers and can resurface after package forget, and the interaction with punk::packagepreference's package require overload.

Reproduction script kept out of tree (session scratchpad folklore.tcl); the fixture is small enough to re-create from the table above when the suite is written.