punkshell/goals/G-035-mixed-tm-pkgindex-provision.md

G-035 Characterise mixed .tm / pkgIndex.tcl provision of the same package

Status: proposed Scope: src/tests/modules/punk/libunknown/testsuites/ (characterization suite); src/modules/punk/libunknown-*.tm and src/modules/punk/packagepreference-999999.0a1.0.tm (as characterised, fixed only if outright bugs surface); src/modules/AGENTS.md + src/lib/AGENTS.md (resulting guidance) Goal: the behaviour when the same package is provided both as a .tm module and as a pkgIndex.tcl-based library - same or differing versions, under the standard package unknown, punk::libunknown and punk::packagepreference - is characterised by committed tests, and the currently informal working rule ("avoid mixing provision forms for one package - unexpected behaviour even with libunknown's improvements") is either substantiated with the specific failure modes named in AGENTS.md guidance, or retired if the characterisation shows the machinery now handles mixing predictably. Acceptance: a committed test suite (extending src/tests/modules/punk/libunknown/testsuites/) characterises at least: same name+version provided via .tm and via pkgIndex.tcl (which registration wins, and whether it is deterministic across scan-trigger orderings) under the standard scanner, under punk::libunknown, and with punk::packagepreference active; differing versions across the two forms (version selection integrity including package prefer latest, and whether the losing form's registration lingers); re-registration effects (package forget then re-require crossing forms); surprising-but-accepted behaviours are pinned with GAP/known-quirk comments (the fossilmove characterization pattern), outright bugs fixed or filed as goals; the resulting do/don't guidance lands in src/modules/AGENTS.md and src/lib/AGENTS.md naming the characterised failure modes (or explicitly lifting the avoid-mixing rule if unwarranted).

Context

Tcl packages arrive by two registration routes with different machinery:

• .tm modules: tcl::tm path scan during package unknown; registration keyed on name-version from the filename; the same-version tie-break rules are now pinned by src/tests/modules/punk/libunknown/testsuites/shadowing/ (2026-07-07): tcl:™️:add prepends, head of tcl:™️:list wins exact-version ties, version beats order, punk::libunknown preserves all of it.
• pkgIndex.tcl libraries: auto_path directory scan (tclPkgUnknown) evaluating index scripts; different scan timing, different overwrite semantics, auto_path ordering rather than tm-list ordering.

When the SAME package is provided by both routes - same version or different versions - the interaction is not well characterised. The user's standing informal rule (recalled 2026-07-07): avoid mixing .tm and pkgIndex.tcl provision for one package - unexpected behaviour was observed even after punk::libunknown's improvements to the unknown-handler chain. That rule is currently folklore: undocumented, unproven, and untested. Meanwhile the punk ecosystem genuinely straddles both forms (src/modules .tm trees vs src/lib pkgIndex libraries; kits carry both; punk::packagepreference overloads package require on top of the unknown-handler chain), so accidental mixing is a realistic hazard - and same-version-different-content drift (the 2026-07-06 textblock incident class) would interact with it unpredictably.

Questions the characterisation must answer

• Same name+version via .tm and via pkgIndex.tcl: which registration wins? Is it deterministic, or dependent on which scan ran first (tm scan vs auto_path scan can be triggered in either order by unrelated requires)?
• Different versions across forms: does version selection stay correct (including under package prefer latest and with alpha/dev versions)? Does the losing form's package ifneeded registration linger, and can it resurface via package forget + re-require?
• How do punk::libunknown and punk::packagepreference each change the answers? (libunknown replaces/wraps the unknown chain; packagepreference overloads package require itself - three layers that can each reorder scans.)
• Are there differences across Tcl 8.6 and 9 (tm.tcl and tclPkgUnknown have evolved)?

Approach

• Extend the shadowing suite's child-interp probe pattern: scratch dirs providing the same test package as a .tm and as a pkgIndex.tcl library, all combinations of {same version, tm newer, pkgIndex newer} x {standard scanner, libunknown, packagepreference} x scan-trigger orderings.
• Pin observed behaviour; mark surprising-but-tolerable outcomes with GAP/known-quirk comments (the fossilmove characterization pattern) rather than encoding wishes; anything outright broken is fixed or filed.
• Convert the folklore into documentation: AGENTS.md guidance in src/modules and src/lib stating either the substantiated avoid-mixing rule with its named failure modes, or the conditions under which mixing is safe.

Notes

• Related: the shadowing pin-tests (same suite family - this goal's tests extend them), G-033 (achieved 2026-07-28, goals/archive/G-033-proj-mode-cwd-project.md - visitor-mode path mixing is now live, making cross-form collisions more reachable: a kit's internal libs vs a visited project's .tm trees), G-026/G-024 (same-version drift provenance), punk::packagepreference 0.1.1 (its moduledoc auto-load hook lives in exactly this machinery).
• G-112 relationship (recorded 2026-07-24; G-112 achieved 2026-07-25 - see goals/archive/G-112-maketcl-subcommand-rename.md): its rename sweep updated the shared src/modules/AGENTS.md + src/lib/AGENTS.md guidance to the bake/bakehouse vocabulary - this goal's characterization outcomes now land on the swept text (no live edit-window coordination remains).
• Out of scope: redesigning the loading machinery. This goal is characterise-document-and-pin; behavioural changes only for outright bugs.
• Baseline moved 2026-07-27 (before activation): one such outright bug was found and fixed under this Scope's "fixed only if outright bugs surface" clause - punk::libunknown 0.2.2 -> 0.2.3. Index scripts were being sourced in a proc of the punk::libunknown namespace, so a pkgIndex.tcl's relative namespace eval foo created ::punk::libunknown::foo rather than ::foo; indexes that define commands at source time for their own ifneeded scripts to call then failed at require time with invalid command name (found on twapi 4.7.2 in a third-party 8.6 kit). Pinned by pkgindex_relative_namespace_is_global in src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test. Relevant to this goal's characterization in two ways: the namespace context of index sourcing is now a pinned property to preserve alongside the frame isolation, and one of the behaviours the "avoid mixing provision forms" folklore may have been describing has turned out to be a plain bug rather than a mixing hazard.

The folklore's mechanism, identified 2026-07-27

The developer's recollection (the rule came from $satisfied in the default ::tcl::tm::UnknownHandler aborting the search too early) is correct, and the mechanism is now pinned down. Stock tm.tcl ends its scan with:

if {$satisfied} {
    return
}
# Fallback to previous command, if existing.
if {[llength $original]} {
    uplevel 1 $original [::linsert $args 0 $name]
}

satisfied is set as soon as a .tm candidate matches the requested name AND version requirement, and $original is the rest of the unknown chain - i.e. ::tclPkgUnknown, the auto_path/pkgIndex.tcl scanner. So a require satisfied from a .tm suppresses the entire library scan for that require. The in-source comment shows it is deliberate ("We abort in this unknown handler only if we got a satisfying candidate ... Otherwise we still have to fallback to the regular package search"), i.e. a performance optimisation - but it couples an unrelated question (did the tm scan happen to satisfy this request?) to whether the library half of the search runs at all.

Reproduced with a synthetic fixture - one .tm providing mixpkg 1.0, one auto_path library providing mixpkg 2.0 plus a sibling package and an index side effect, under package prefer latest. Identical results on Tcl 8.6.10 and 9.0.3:

observation after package require mixpkg	stock chain	libunknown chain
version returned	1.0	2.0
package versions mixpkg	1.0	1.0 2.0
index side effect ran	no	yes
sibling sibpkg registered	no	yes

The first row is the sharp one: with both forms present, stock Tcl silently returns the OLDER .tm version while a newer library version exists and prefer latest is in force - no error, no warning. The other rows explain the spooky-action reports: whether a given auto_path directory has been indexed at all depends on whether some earlier, unrelated require happened to be satisfied by a .tm, so behaviour varies with require ORDER.

punk::libunknown already neutralises this: zipfs_tm_UnknownHandler still computes satisfied but its early return is commented out, so the chain ALWAYS falls through to zipfs_tclPkgUnknown. That is the other half of the developer's recollection - and it is what the epoch cache buys: unconditional fallthrough is only affordable because already-scanned directories are not re-globbed within an epoch. The two mechanisms are one design.

Consequences for this goal's framing (the Goal line contemplates either substantiating the avoid-mixing rule or retiring it):

• The rule is SUBSTANTIATED for stock Tcl, with a named failure mode (silent older-version selection) rather than vague "unexpected behaviour".
• It is NEUTRALISED under the punkshell chain, so the characterization should be per-chain rather than a single verdict.
• Stock behaviour stays reachable inside punkshell's own world - any interp that has not run punk::libunknown::init (child interps created without it, including deliberate test probes; a plain tclsh consuming punk .tm trees). So guidance cannot simply say "libunknown handles it".
• Still open for the characterization proper: same-version (rather than newer-library) collisions, whether the losing form's registration lingers and can resurface after package forget, and the interaction with punk::packagepreference's package require overload.

Reproduction script kept out of tree (session scratchpad folklore.tcl); the fixture is small enough to re-create from the table above when the suite is written.