punkshell/goals/archive/G-058-static-runtime-packages.md

G-058 Boot honours statically-linked runtime packages (static baseline seeding + packagepreference static-awareness)

Status: achieved 2026-07-10 Scope: src/vfs/_config/punk_main.tcl (boot auto_path/tm path filtering), src/modules/punk/packagepreference-999999.0a1.0.tm, src/modules/punk/repl-999999.0a1.0.tm + src/modules/punk/repl/codethread-999999.0a1.0.tm (code interp / codethread bootstrap), src/modules/shellthread-999999.0a1.0.tm (punkshell-created worker threads) as applicable, src/tests/ (un-gated unit tests + constraint-gated shell/kit integration tests), punkbin artifact repo (separate git repo, local checkout c:/repo/jn/punkbin - pinned runtime additions) Goal: punkshell running on a runtime with statically-linked/builtin packages (tclsfe-x64: Thread/twapi/sqlite3/tdbc; punkbin runtimes' builtin Thread/tcllibc/vfs/vlerq; the expected shape of future zig-built static runtimes, G-005) keeps those packages resolvable in every interp and thread punkshell fabricates: boot captures the static baseline (info loaded entries with empty filename, plus their provided versions) before replacing package search paths, code interps and punkshell-created threads are seeded with ifneeded scripts mapping each static package to 'load {} ', and punk::packagepreference resolves static-vs-bundled by a documented version-aware policy instead of blindly loading a bundled dll over an already-provided static package. Acceptance: a punk91-style kit (tclsfe-x64 + punk9win.vfs, no thread dll in the vfs) boots to a working repl with no "can't find package Thread" - punk::console loads in the code interp, and package require Thread succeeds there and in a punkshell-created worker thread, resolving to the static version; in the same kit, twapi resolves per the documented policy (no repeat of the observed static-Twapi-masked-by-older-vfs-twapi-5.0.2 double load; a genuinely newer bundled copy remains reachable by that policy); dll-based kits (punk902z) boot and pass their existing shell test baseline unchanged, as does a plain tclsh dev launch; the seeding mechanism is generic - driven by the captured baseline, no runtime-specific package naming - and the boot-time static baseline is introspectable at the repl; the seeding/preference logic is covered by un-gated unit tests against simulated baselines (runnable under plain tclsh), while kit-boot integration tests are gated behind a capability-probed tcltest constraint (a built kit whose baseline shows static entries including Thread) that skips cleanly when no such kit is present; the runtimes used for verification (tclsfe-x64.exe at minimum) are added to the punkbin artifact repository under win32-x86_64 with sha1sums.txt updated, so the constraint is satisfiable on other machines via the existing runtime-retrieval path; the punk91 code-interp vfs/vfs::zip load failure is re-diagnosed after the fix and either resolved or recorded as a distinct issue/candidate goal.

Context

Discovered 2026-07-10 wrapping punk9win.vfs onto the tclsfe-x64.exe runtime (punk91, part of the tk-loading experiment). tclsfe (by twapi author and Tcl core member Ashok P. Nadkarni; TEMP_REFERENCE/tcl-sfe) statically links Thread, twapi, sqlite3 and tdbc::odbc into a Tcl 9.1b0 stub. The punk91 kit's main interp works, but the repl's code interp fails at boot:

repl code interp FAILED to load vfs,vfs::zip
failed to load punk::console - can't find package Thread

Probe evidence (2026-07-10)

• Plain tclsfe-x64 stub: static registrations visible as empty-filename entries in info loaded ({} Thread, {} Twapi, {} Tdbcodbc, {} Tdbc, {} Sqlite3, {} Dde); package require Thread resolves 3.0.5 via a chained ifneeded (package require -exact thread 3.0.5); a fresh thread::create WORKER resolves Thread 3.0.5 fine and tsv round-trips. The stub seeds its own thread interps correctly (its _sfeinit.tcl mechanism initializes primary/secondary/thread interps) - the runtime is not the problem.
• punk91 (same stub + punk9win.vfs): main interp shows the static baseline, but the code interp cannot package require Thread. punkshell's boot replaces auto_path/module paths wholesale (punk_main.tcl filtering) and overloads package (punk::packagepreference), discarding the runtime's seeding; the interps/threads punkshell fabricates get punkshell-controlled paths only, so nothing maps Thread -> load {} Thread even though that load is available.
• punk902z never hits this: its vfs bundles lib_tcl9/thread3.0.4/tcl9thread304.dll so punkshell's own paths satisfy the require.
• Shadowing variant: punk91's main interp showed BOTH {} Twapi (static, newer) and //zipfs:/app/lib_tcl9/twapi5.0.2/...dll Twapi loaded - the bundled OLDER twapi loaded over the newer static one. Suspected pre-existing instance: the punkbin runtime README declares builtin Thread/tcllibc/vfs/vlerq, yet punk902z loads thread3.0.4 from the punk vfs (//zipfs:/app/...) - the bundled-over- provided pattern likely occurs quietly on the recommended runtime too.

Precedent (same bug class)

src/tests/runtests.tcl testinterp previously replaced auto_path wholesale and lost kit-internal package trees; fixed by re-adding the running kit's internal lib/lib_tcl dirs (see src/tests/AGENTS.md). Static packages are the flavour with NO directory to re-add - ifneeded seeding is required instead.

Approach

1. Capture-before-replace: at boot, before punk_main.tcl filters package search paths, record the static baseline - [info loaded] entries with empty filename plus their provided versions (package present in the pristine interp). Keep it introspectable at the repl (diagnostics for future static runtimes).
2. Seed fabricated interps/threads: codethread bootstrap, shellthread workers and any punkshell-created child interps get package ifneeded $pkg $ver [list load {} $pkg] for each baseline entry. Generic by construction - driven by the captured baseline, no runtime-specific package naming (zig-built static runtimes, G-005, inherit it).
3. packagepreference static-awareness: static-vs-bundled resolved by a documented version-aware policy (e.g latest-version-wins, static preferred on ties - exact rule recorded here when settled) instead of loading whatever the vfs paths find over an already-provided static package.
4. Re-diagnose the punk91 code-interp vfs/vfs::zip load failure after the Thread fix (may be cascade, may be an independent 9.1b0/dll issue) - resolve or record as a distinct issue/candidate goal.

Testing and artifacts

• Un-gated unit tests: the seeding generator and preference policy tested against SIMULATED baselines ({{} Thread} {{} Twapi} + versions; static/bundled version pairs) under plain tclsh in the normal suite - the logic has coverage on every machine.
• Constraint-gated integration tests: boot the real static kit and probe the code interp + a worker thread. The tcltest constraint is CAPABILITY-probed (a built kit whose baseline shows static entries including Thread), not filename-hardcoded - skips cleanly when absent, extends automatically to zig-era static runtimes.
• Artifact pinning: bin/runtime/ binaries are not git-tracked and tclsfe-x64.exe is a moving upstream beta (github.com/apnadkarni/tcl-sfe releases). Add the verification runtimes (tclsfe-x64.exe at minimum) to the punkbin artifact repo (local checkout c:/repo/jn/punkbin, per-platform dirs, sha1sums.txt checksums) under win32-x86_64 so the integration constraint is satisfiable on other machines via the existing runtime-retrieval path (bin/runtime.cmd; G-006 direction). The .vfs side needs no preservation work - src/vfs/*.vfs is git-tracked.

Notes

• tclsfe's _sfeinit.tcl hook is a runtime-specific customization channel; it is deliberately NOT the fix - it cannot help interps punkshell fabricates with its own paths, and the punkshell-side mechanism must stay runtime-agnostic.
• Related: G-005 (zig static runtimes make this foundational), G-006 (artifact retrieval), the tk-loading experiment that surfaced it (punk91 = tclsfe-x64 + punk9win.vfs per src/runtime/mapvfs.config).

Outcome (achieved 2026-07-10)

Shipped as punk::lib 0.4.0, punk::repl 0.3.0, punk::packagepreference 0.2.0, shellthread 1.7.0 + punk_main.tcl capture (punkshell 0.6.0).

Implementation-discovered refinements to the Approach:

• The tcl-sfe stub does NOT seed ifneeded entries for its statics: it registers them process-wide via Tcl_StaticLibrary(NULL,...) and relies on pkgIndex.tcl files inside its own //zipfs:/app zip, resolved through a per-interp PostInit hook (tclSfeMainHook.c) forcing auto_path to //zipfs:/app. A punk kit's appended vfs REPLACES that mount, so even the pristine boot interp has no name->load mapping - the capture therefore probe-loads each empty-filename [info loaded] prefix in a throwaway interp and records only packages the load actually PROVIDES (diff on provided packages, not 'package names' - a probe load can trigger an index scan that registers many names without providing them).
• Composite statics exclude themselves: static twapi's C init provides nothing (its script layer lived in the replaced stub zip), so it is never seeded and the bundled complete twapi is correctly the resolution - the observed "static Twapi + twapi dll" double listing was registration+load, not a double load, and the bundled copy was the only usable one. A configurable denylist (::punkboot::static_probe_denylist, default: tk* vfs mk4tcl vlerq tdbc*) guards side-effecting (tk creates '.') and known-composite inits from probing at all.
• Policy mechanics: seeding alone would HIDE newer bundled copies (a satisfiable ifneeded suppresses the package unknown scan), so packagepreference triggers the index scan for baseline names before resolution; and static REGISTRATIONS (empty-filename info loaded entries) are excluded from the pre-existing loaded-shared-object same-version pinning, which would otherwise force the static version even when bundled is newer. Net policy: all candidates registered, highest acceptable version wins; identical versions resolve to the bundled copy (later-registered ifneeded entry).
• vfs/vfs::zip re-diagnosis: NOT a seeding defect - the tclsfe runtime doesn't bundle tclvfs at all (tclkit-family runtimes carry vfs1.5.0+vlerq in their tcl_library; sfe carries thread/twapi/sqlite3/tdbc only), so the punk91 kit genuinely lacks the package and the repl's existing soft-fail warning stands. Candidate follow-up if wanted: bundle a tcl9 tclvfs build in punk9win.vfs, or accept vfs-less zipfs kits (vfs is primarily needed for .kit interop).
• Verification: capture confirmed on the stub (thread/Thread 3.0.5, sqlite3 3.53.0, dde 1.5b0 seeded; twapi excluded); punk91 rebuilt - repl boots with no "can't find package Thread", code interp resolves Thread and introspects ::punkboot::static_packages, fabricated interp + snapshot-seeded worker resolve Thread, twapi resolves (staticruntime.test 4/4 behind the capability probe); staticseed.test 4/4 + staticpolicy.test 4/4 un-gated (9.0.3 and 8.7); punk902z + full shell tree unchanged (23 pass/2 constraint skips); full punk/lib tree green. tclsfe-x64.exe pinned in punkbin (commit b2dbbe6) with sha1sums and new AGENTS.md scaffolding.