punk::libunknown 0.2.3: pkgIndex.tcl scripts run in the global namespace again (punkshell 0.27.1)
A third-party 8.6 kit under evaluation (Lean Undroidwish Construction Kit, 32-bit windows)
could not load its own bundled twapi 4.7.2 inside punkshell:
P% package require twapi
invalid command name "twapi::set_scriptdir"
("package ifneeded twapi 4.7.2" script)
twapi's pkgIndex.tcl defines 'namespace eval twapi { proc set_scriptdir ... }' when the
INDEX is sourced, and its ifneeded script calls that command at require time. The command
existed - as ::punk::libunknown::twapi::set_scriptdir. punk::libunknown 0.2.0 moved index
sourcing out of 'namespace eval ::' into source_pkgindex, a proc of the punk::libunknown
namespace, to stop each index's helper variables leaking into the global namespace and to
stop a user global named 'dir' being clobbered. That fixed the frame, but a proc body
executes in ITS OWN namespace, so an index's RELATIVE 'namespace eval foo' started
creating ::punk::libunknown::foo instead of ::foo. Stock tclPkgUnknown is a proc in ::,
so indexes have always had global resolution; the ifneeded script, evaluated later at
global scope, then looked for a command that was never there.
The body now runs via apply with :: as its namespace argument, which gives both
properties at once: a local frame (index helper variables still discarded, user 'dir'
still safe) and the global resolution context indexes are written against.
Not twapi-specific: any index that defines commands or namespaces at source time for its
own ifneeded scripts to use hits this, and the pattern is common in third-party indexes.
punkshell's own vendored twapi 5.0b1 happens to avoid it (self-contained 'apply' ifneeded
scripts), which is why the tree never surfaced it.
Verified on the LUCK 8.6.10 runtime end-to-end - 'package require twapi' at the punkshell
prompt now returns 4.7.2 with ::twapi::scriptdir correctly set - and pinned generically by
pkgindex_relative_namespace_is_global in the discovery suite, whose new fixture copies
twapi's pattern so no third-party kit is needed. The pin was proven to bite: driven
against the pre-fix body the same fixture fails with 'invalid command name' and leaves
::punk::libunknown::pklu_nsindex behind; against 0.2.3 it resolves and leaves nothing.
Full suite 1165 tests, 1144 passed, 1 failure - core/tcl exec-14.3, the documented
baseline.
Manual-version bump per src/modules/AGENTS.md: file renamed, Meta line, manpage_begin and
provide-block updated, version-history line appended. The rename sweep converted the two
goal Scope references (G-035, G-109) from the exact filename to the glob form
src/modules/punk/libunknown-*.tm that GOALS.md's own Scope-authoring guidance recommends,
so the reference stops drifting at every bump.
Also records in G-035 the mechanism behind its "avoid mixing .tm and pkgIndex.tcl
provision" folklore, which this investigation identified (non-contract Notes). Stock
::tcl:™️:UnknownHandler returns early when $satisfied is set and therefore never falls
through to ::tclPkgUnknown, so a require satisfied from a .tm suppresses the whole library
scan. Measured with a synthetic fixture on both 8.6.10 and 9.0.3: with mixpkg 1.0 as a .tm
and mixpkg 2.0 in an auto_path library under 'package prefer latest', stock returns 1.0
and reports only 1.0 from 'package versions', runs no index side effect and registers no
sibling package; the libunknown chain returns 2.0 and does all three. libunknown already
neutralises it - its handler still computes $satisfied but the early return is commented
out - and that unconditional fallthrough is exactly what the epoch cache pays for. The
folklore is therefore substantiated for stock Tcl with a named failure mode (silent
older-version selection) and neutralised under the punkshell chain, with stock behaviour
still reachable in any interp that has not run punk::libunknown::init.
Assisted-by: harness=claude; primary-model=claude-opus-5[1m]; api-location=anthropic.com
# G-035 Characterise mixed .tm / pkgIndex.tcl provision of the same package
# G-035 Characterise mixed .tm / pkgIndex.tcl provision of the same package
Status: proposed
Status: proposed
Scope: src/tests/modules/punk/libunknown/testsuites/ (characterization suite); src/modules/punk/libunknown-0.2.2.tm and src/modules/punk/packagepreference-999999.0a1.0.tm (as characterised, fixed only if outright bugs surface); src/modules/AGENTS.md + src/lib/AGENTS.md (resulting guidance)
Scope: src/tests/modules/punk/libunknown/testsuites/ (characterization suite); src/modules/punk/libunknown-*.tm and src/modules/punk/packagepreference-999999.0a1.0.tm (as characterised, fixed only if outright bugs surface); src/modules/AGENTS.md + src/lib/AGENTS.md (resulting guidance)
Goal: the behaviour when the same package is provided both as a .tm module and as a pkgIndex.tcl-based library - same or differing versions, under the standard package unknown, punk::libunknown and punk::packagepreference - is characterised by committed tests, and the currently informal working rule ("avoid mixing provision forms for one package - unexpected behaviour even with libunknown's improvements") is either substantiated with the specific failure modes named in AGENTS.md guidance, or retired if the characterisation shows the machinery now handles mixing predictably.
Goal: the behaviour when the same package is provided both as a .tm module and as a pkgIndex.tcl-based library - same or differing versions, under the standard package unknown, punk::libunknown and punk::packagepreference - is characterised by committed tests, and the currently informal working rule ("avoid mixing provision forms for one package - unexpected behaviour even with libunknown's improvements") is either substantiated with the specific failure modes named in AGENTS.md guidance, or retired if the characterisation shows the machinery now handles mixing predictably.
Acceptance: a committed test suite (extending src/tests/modules/punk/libunknown/testsuites/) characterises at least: same name+version provided via .tm and via pkgIndex.tcl (which registration wins, and whether it is deterministic across scan-trigger orderings) under the standard scanner, under punk::libunknown, and with punk::packagepreference active; differing versions across the two forms (version selection integrity including package prefer latest, and whether the losing form's registration lingers); re-registration effects (package forget then re-require crossing forms); surprising-but-accepted behaviours are pinned with GAP/known-quirk comments (the fossilmove characterization pattern), outright bugs fixed or filed as goals; the resulting do/don't guidance lands in src/modules/AGENTS.md and src/lib/AGENTS.md naming the characterised failure modes (or explicitly lifting the avoid-mixing rule if unwarranted).
Acceptance: a committed test suite (extending src/tests/modules/punk/libunknown/testsuites/) characterises at least: same name+version provided via .tm and via pkgIndex.tcl (which registration wins, and whether it is deterministic across scan-trigger orderings) under the standard scanner, under punk::libunknown, and with punk::packagepreference active; differing versions across the two forms (version selection integrity including package prefer latest, and whether the losing form's registration lingers); re-registration effects (package forget then re-require crossing forms); surprising-but-accepted behaviours are pinned with GAP/known-quirk comments (the fossilmove characterization pattern), outright bugs fixed or filed as goals; the resulting do/don't guidance lands in src/modules/AGENTS.md and src/lib/AGENTS.md naming the characterised failure modes (or explicitly lifting the avoid-mixing rule if unwarranted).
@ -73,3 +73,82 @@ is a realistic hazard - and same-version-different-content drift (the
on the swept text (no live edit-window coordination remains).
on the swept text (no live edit-window coordination remains).
- Out of scope: redesigning the loading machinery. This goal is
- Out of scope: redesigning the loading machinery. This goal is
characterise-document-and-pin; behavioural changes only for outright bugs.
characterise-document-and-pin; behavioural changes only for outright bugs.
- Baseline moved 2026-07-27 (before activation): one such outright bug was found and
fixed under this Scope's "fixed only if outright bugs surface" clause - punk::libunknown
0.2.2 -> 0.2.3. Index scripts were being sourced in a proc of the `punk::libunknown`
namespace, so a pkgIndex.tcl's relative `namespace eval foo` created
`::punk::libunknown::foo` rather than `::foo`; indexes that define commands at source
time for their own ifneeded scripts to call then failed at require time with `invalid
command name` (found on twapi 4.7.2 in a third-party 8.6 kit). Pinned by
`pkgindex_relative_namespace_is_global` in
`src/tests/modules/punk/libunknown/testsuites/discovery/discovery.test`. Relevant to
this goal's characterization in two ways: the *namespace context* of index sourcing is
now a pinned property to preserve alongside the frame isolation, and one of the
behaviours the "avoid mixing provision forms" folklore may have been describing has
turned out to be a plain bug rather than a mixing hazard.
### The folklore's mechanism, identified 2026-07-27
The developer's recollection (the rule came from `$satisfied` in the default
`::tcl::tm::UnknownHandler` aborting the search too early) is correct, and the mechanism
is now pinned down. Stock `tm.tcl` ends its scan with:
if {$satisfied} {
return
}
# Fallback to previous command, if existing.
if {[llength $original]} {
uplevel 1 $original [::linsert $args 0 $name]
}
`satisfied` is set as soon as a .tm candidate matches the requested name AND version
requirement, and `$original` is the rest of the unknown chain - i.e. `::tclPkgUnknown`,
the auto_path/pkgIndex.tcl scanner. So **a require satisfied from a .tm suppresses the
entire library scan for that require**. The in-source comment shows it is deliberate
("We abort in this unknown handler only if we got a satisfying candidate ... Otherwise we
still have to fallback to the regular package search"), i.e. a performance optimisation -
but it couples an unrelated question (did the tm scan happen to satisfy this request?) to
whether the library half of the search runs at all.
Reproduced with a synthetic fixture - one `.tm` providing `mixpkg 1.0`, one auto_path
library providing `mixpkg 2.0` plus a sibling package and an index side effect, under
`package prefer latest`. Identical results on Tcl 8.6.10 and 9.0.3:
# G-109 libunknown manifest-declared multi-name discovery for zip-based .tm modules
# G-109 libunknown manifest-declared multi-name discovery for zip-based .tm modules
Status: proposed
Status: proposed
Scope: src/modules/punk/libunknown-0.2.2.tm (scan/registration); modpod (manifest emission + load stub; canonical source c:/repo/jn/tclmodules/modpod, vendored copies as consumed); src/modules/punk/mix/ (modpod wrap tooling as touched); src/tests/modules/punk/libunknown/ (new suites + scan-performance characterization)
Scope: src/modules/punk/libunknown-*.tm (scan/registration); modpod (manifest emission + load stub; canonical source c:/repo/jn/tclmodules/modpod, vendored copies as consumed); src/modules/punk/mix/ (modpod wrap tooling as touched); src/tests/modules/punk/libunknown/ (new suites + scan-performance characterization)
Goal: a zip-based .tm (modpod) can declare additional package-require names - case variants (Thread/thread) and sub-packages - in a manifest at a well-known internal location, and punkshell's libunknown discovers those declarations during module scanning and registers them so package require of ANY declared name loads the single .tm file; real-disk scan performance is preserved: libunknown exists to fix slow-drive/large-multi-path-tree scan cost, so with manifest support enabled the worst-case target is parity with plain tclsh's builtin tm handler on the same tree (zipfs trees remain the fast path).
Goal: a zip-based .tm (modpod) can declare additional package-require names - case variants (Thread/thread) and sub-packages - in a manifest at a well-known internal location, and punkshell's libunknown discovers those declarations during module scanning and registers them so package require of ANY declared name loads the single .tm file; real-disk scan performance is preserved: libunknown exists to fix slow-drive/large-multi-path-tree scan cost, so with manifest support enabled the worst-case target is parity with plain tclsh's builtin tm handler on the same tree (zipfs trees remain the fast path).
Acceptance: a demonstration modpod .tm declaring a case-variant name and a sub-package name resolves via package require of each declared name on the tcl9 kit (8.6 covered, or its limitation recorded per G-034), from both a real-disk module path and a zipfs module path; the manifest location and format are documented and emitted by the modpod wrap tooling; plain-tclsh behaviour is documented prominently (the filename-derived name loads anywhere zipfs exists; extra names are punkshell/libunknown-only); a scan-performance characterization test compares libunknown-with-manifest-support against plain tclsh's builtin handler over a generated large multi-path real-disk tree and records the numbers in this file showing no worse than parity; extensive suites under src/tests/modules/punk/libunknown/ cover registration, case-variant and sub-package loading, refresh/epoch staleness, and the no-manifest fast path.
Acceptance: a demonstration modpod .tm declaring a case-variant name and a sub-package name resolves via package require of each declared name on the tcl9 kit (8.6 covered, or its limitation recorded per G-034), from both a real-disk module path and a zipfs module path; the manifest location and format are documented and emitted by the modpod wrap tooling; plain-tclsh behaviour is documented prominently (the filename-derived name loads anywhere zipfs exists; extra names are punkshell/libunknown-only); a scan-performance characterization test compares libunknown-with-manifest-support against plain tclsh's builtin handler over a generated large multi-path real-disk tree and records the numbers in this file showing no worse than parity; extensive suites under src/tests/modules/punk/libunknown/ cover registration, case-variant and sub-package loading, refresh/epoch staleness, and the no-manifest fast path.
proc set_scriptdir dir {variable scriptdir ; set scriptdir $dir}
}
package ifneeded pklu_nsindex 1.0 [subst {
pklu_nsindex::set_scriptdir [list $dir]
source [list [file join $dir pklu_nsindex.tcl]]
}]}
{pklu_nsindex/pklu_nsindex.tcl}
{package provide pklu_nsindex 1.0}
{pklu_extender/pkgIndex.tcl}
{pklu_extender/pkgIndex.tcl}
{lappend auto_path [file join $dir sublib]
{lappend auto_path [file join $dir sublib]
set pklu_leak_probe leaked_value}
set pklu_leak_probe leaked_value}
@ -359,6 +371,29 @@ namespace eval ::testspace {
}]
}]
} -result {1.0 0}
} -result {1.0 0}
#added 2026-07-27 (agent) - punk::libunknown 0.2.3 regression pin. 0.2.0 moved index
#sourcing into a proc of the punk::libunknown namespace, so an index's RELATIVE
#'namespace eval foo' created ::punk::libunknown::foo instead of ::foo. Indexes that
#define commands at source time for their own ifneeded scripts to call then failed at
#require time with 'invalid command name' - found on twapi 4.7.2 (its index defines
#twapi::set_scriptdir; its ifneeded script calls it), whose pattern this fixture copies.
test pkgindex_relative_namespace_is_global {a pkgIndex.tcl's relative 'namespace eval' creates a GLOBAL namespace, so commands it defines for its own ifneeded script resolve at require time}\
