From 40f3ef2bcaea91b20be174ac74f4e1d76b10f221 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Fri, 10 Jul 2026 18:31:35 +1000 Subject: [PATCH] GOALS: add G-058 [proposed] boot honours statically-linked runtime packages - diagnosis from the punk91 tk-loading experiment (tclsfe-x64 + punk9win.vfs): plain tclsfe stub worker threads resolve static Thread 3.0.5 fine, but punkshell boot replaces package search paths wholesale and discards the runtime seeding - the repl code interp cannot find Thread ("failed to load punk::console"); punk902z unaffected only because its vfs bundles a thread dll - fix shape: capture the static baseline (empty-filename info loaded entries + versions) before path replacement, seed fabricated interps/threads with 'load {} ' ifneeded scripts, and make punk::packagepreference resolve static-vs-bundled by a documented version-aware policy (punk91 loaded the OLDER bundled twapi 5.0.2 dll over the newer static Twapi) - testing: un-gated unit tests against simulated baselines + kit-boot integration tests behind a capability-probed constraint (static-entries-incl-Thread kit present) - skips cleanly, extends to zig-era static runtimes (G-005) - artifacts: verification runtimes (tclsfe-x64.exe at minimum) to be pinned in the punkbin artifact repo (win32-x86_64 + sha1sums.txt) so the constraint is satisfiable off this machine; .vfs folders already git-tracked - detail file records probe evidence, the runtests-testinterp precedent (same bug class, directory flavour), the suspected quiet thread-over-builtin instance on punk902z, and the vfs/vfs::zip re-diagnosis follow-up Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- GOALS.md | 6 ++ goals/G-058-static-runtime-packages.md | 95 ++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100644 goals/G-058-static-runtime-packages.md diff --git a/GOALS.md b/GOALS.md index 88e30d6d..931e7fc3 100644 --- a/GOALS.md +++ b/GOALS.md @@ -391,3 +391,9 @@ Scope: src/make.tcl (kit/zipkit wrap steps), src/runtime/punk1.ico (project defa Detail: goals/G-057-kit-icon-embedding.md Goal: Windows kit/zipkit builds produce executables carrying an embedded icon chosen at build time - defaulting to the project icon src/runtime/punk1.ico, overridable per kit by its .vfs folder - by replacing the icon resources in the built executable using the twapi-based mechanism demonstrated in tcl-sfe (TEMP_REFERENCE/tcl-sfe, by twapi author and Tcl core member Ashok P. Nadkarni): RT_ICON/RT_GROUP_ICON replacement via twapi resource-update APIs, applied so the appended vfs payload stays intact (icon the stub before appending, or sfe-style split/update/reattach). Acceptance: a Windows `make.tcl project` build produces kit executables whose embedded icon resources are the project default punk1.ico, and a kit whose .vfs supplies an override icon gets that icon instead (verified by resource inspection, e.g twapi::extract_resources, not just Explorer eyeballing); the icon-replaced executables still boot to a working punk shell reading their vfs payload for the kit types we build (kit, zip, zipcat per mapvfs.config); runtimes under src/runtime are never modified - replacement applies to the built copies only; twapi unavailable or non-Windows platform skips the icon step with a notice and the build otherwise completes unchanged; rebuilds are idempotent (re-wrapping an already-iconed build copy converges, no resource accumulation); the override convention (filename/location in the kit's custom .vfs folder vs a mapvfs.config element) and the stub-vs-split ordering decision are recorded in the detail file with the tcl-sfe attribution. + +### G-058 [proposed] Boot honours statically-linked runtime packages (static baseline seeding + packagepreference static-awareness) +Scope: src/vfs/_config/punk_main.tcl (boot auto_path/tm path filtering), src/modules/punk/packagepreference-999999.0a1.0.tm, src/modules/punk/repl-999999.0a1.0.tm + src/modules/punk/repl/codethread-999999.0a1.0.tm (code interp / codethread bootstrap), src/modules/shellthread-999999.0a1.0.tm (punkshell-created worker threads) as applicable, src/tests/ (un-gated unit tests + constraint-gated shell/kit integration tests), punkbin artifact repo (separate git repo, local checkout c:/repo/jn/punkbin - pinned runtime additions) +Detail: goals/G-058-static-runtime-packages.md +Goal: punkshell running on a runtime with statically-linked/builtin packages (tclsfe-x64: Thread/twapi/sqlite3/tdbc; punkbin runtimes' builtin Thread/tcllibc/vfs/vlerq; the expected shape of future zig-built static runtimes, G-005) keeps those packages resolvable in every interp and thread punkshell fabricates: boot captures the static baseline (info loaded entries with empty filename, plus their provided versions) before replacing package search paths, code interps and punkshell-created threads are seeded with ifneeded scripts mapping each static package to 'load {} ', and punk::packagepreference resolves static-vs-bundled by a documented version-aware policy instead of blindly loading a bundled dll over an already-provided static package. +Acceptance: a punk91-style kit (tclsfe-x64 + punk9win.vfs, no thread dll in the vfs) boots to a working repl with no "can't find package Thread" - punk::console loads in the code interp, and package require Thread succeeds there and in a punkshell-created worker thread, resolving to the static version; in the same kit, twapi resolves per the documented policy (no repeat of the observed static-Twapi-masked-by-older-vfs-twapi-5.0.2 double load; a genuinely newer bundled copy remains reachable by that policy); dll-based kits (punk902z) boot and pass their existing shell test baseline unchanged, as does a plain tclsh dev launch; the seeding mechanism is generic - driven by the captured baseline, no runtime-specific package naming - and the boot-time static baseline is introspectable at the repl; the seeding/preference logic is covered by un-gated unit tests against simulated baselines (runnable under plain tclsh), while kit-boot integration tests are gated behind a capability-probed tcltest constraint (a built kit whose baseline shows static entries including Thread) that skips cleanly when no such kit is present; the runtimes used for verification (tclsfe-x64.exe at minimum) are added to the punkbin artifact repository under win32-x86_64 with sha1sums.txt updated, so the constraint is satisfiable on other machines via the existing runtime-retrieval path; the punk91 code-interp vfs/vfs::zip load failure is re-diagnosed after the fix and either resolved or recorded as a distinct issue/candidate goal. diff --git a/goals/G-058-static-runtime-packages.md b/goals/G-058-static-runtime-packages.md new file mode 100644 index 00000000..26891e88 --- /dev/null +++ b/goals/G-058-static-runtime-packages.md @@ -0,0 +1,95 @@ +# G-058 Boot honours statically-linked runtime packages (static baseline seeding + packagepreference static-awareness) + +Status: proposed +Scope: src/vfs/_config/punk_main.tcl (boot auto_path/tm path filtering), src/modules/punk/packagepreference-999999.0a1.0.tm, src/modules/punk/repl-999999.0a1.0.tm + src/modules/punk/repl/codethread-999999.0a1.0.tm (code interp / codethread bootstrap), src/modules/shellthread-999999.0a1.0.tm (punkshell-created worker threads) as applicable, src/tests/ (un-gated unit tests + constraint-gated shell/kit integration tests), punkbin artifact repo (separate git repo, local checkout c:/repo/jn/punkbin - pinned runtime additions) +Acceptance: see GOALS.md index entry (canonical). + +## Context + +Discovered 2026-07-10 wrapping punk9win.vfs onto the tclsfe-x64.exe runtime +(punk91, part of the tk-loading experiment). tclsfe (by twapi author and Tcl core +member Ashok P. Nadkarni; TEMP_REFERENCE/tcl-sfe) statically links Thread, twapi, +sqlite3 and tdbc::odbc into a Tcl 9.1b0 stub. The punk91 kit's main interp works, +but the repl's code interp fails at boot: + + repl code interp FAILED to load vfs,vfs::zip + failed to load punk::console - can't find package Thread + +### Probe evidence (2026-07-10) + +- Plain tclsfe-x64 stub: static registrations visible as empty-filename entries in + info loaded ({} Thread, {} Twapi, {} Tdbcodbc, {} Tdbc, {} Sqlite3, {} Dde); + `package require Thread` resolves 3.0.5 via a chained ifneeded + (`package require -exact thread 3.0.5`); a fresh thread::create WORKER resolves + Thread 3.0.5 fine and tsv round-trips. The stub seeds its own thread interps + correctly (its _sfeinit.tcl mechanism initializes primary/secondary/thread + interps) - the runtime is not the problem. +- punk91 (same stub + punk9win.vfs): main interp shows the static baseline, but + the code interp cannot `package require Thread`. punkshell's boot replaces + auto_path/module paths wholesale (punk_main.tcl filtering) and overloads + `package` (punk::packagepreference), discarding the runtime's seeding; the + interps/threads punkshell fabricates get punkshell-controlled paths only, so + nothing maps Thread -> `load {} Thread` even though that load is available. +- punk902z never hits this: its vfs bundles lib_tcl9/thread3.0.4/tcl9thread304.dll + so punkshell's own paths satisfy the require. +- Shadowing variant: punk91's main interp showed BOTH {} Twapi (static, newer) + and //zipfs:/app/lib_tcl9/twapi5.0.2/...dll Twapi loaded - the bundled OLDER + twapi loaded over the newer static one. Suspected pre-existing instance: the + punkbin runtime README declares builtin Thread/tcllibc/vfs/vlerq, yet punk902z + loads thread3.0.4 from the punk vfs (//zipfs:/app/...) - the bundled-over- + provided pattern likely occurs quietly on the recommended runtime too. + +### Precedent (same bug class) + +src/tests/runtests.tcl testinterp previously replaced auto_path wholesale and +lost kit-internal package trees; fixed by re-adding the running kit's internal +lib/lib_tcl dirs (see src/tests/AGENTS.md). Static packages are the flavour +with NO directory to re-add - ifneeded seeding is required instead. + +## Approach + +1. Capture-before-replace: at boot, before punk_main.tcl filters package search + paths, record the static baseline - [info loaded] entries with empty filename + plus their provided versions (package present in the pristine interp). Keep it + introspectable at the repl (diagnostics for future static runtimes). +2. Seed fabricated interps/threads: codethread bootstrap, shellthread workers and + any punkshell-created child interps get + `package ifneeded $pkg $ver [list load {} $pkg]` for each baseline entry. + Generic by construction - driven by the captured baseline, no runtime-specific + package naming (zig-built static runtimes, G-005, inherit it). +3. packagepreference static-awareness: static-vs-bundled resolved by a documented + version-aware policy (e.g latest-version-wins, static preferred on ties - + exact rule recorded here when settled) instead of loading whatever the vfs + paths find over an already-provided static package. +4. Re-diagnose the punk91 code-interp vfs/vfs::zip load failure after the Thread + fix (may be cascade, may be an independent 9.1b0/dll issue) - resolve or + record as a distinct issue/candidate goal. + +## Testing and artifacts + +- Un-gated unit tests: the seeding generator and preference policy tested against + SIMULATED baselines ({{} Thread} {{} Twapi} + versions; static/bundled version + pairs) under plain tclsh in the normal suite - the logic has coverage on every + machine. +- Constraint-gated integration tests: boot the real static kit and probe the code + interp + a worker thread. The tcltest constraint is CAPABILITY-probed (a built + kit whose baseline shows static entries including Thread), not + filename-hardcoded - skips cleanly when absent, extends automatically to + zig-era static runtimes. +- Artifact pinning: bin/runtime/ binaries are not git-tracked and tclsfe-x64.exe + is a moving upstream beta (github.com/apnadkarni/tcl-sfe releases). Add the + verification runtimes (tclsfe-x64.exe at minimum) to the punkbin artifact repo + (local checkout c:/repo/jn/punkbin, per-platform dirs, sha1sums.txt checksums) + under win32-x86_64 so the integration constraint is satisfiable on other + machines via the existing runtime-retrieval path (bin/runtime.cmd; G-006 + direction). The .vfs side needs no preservation work - src/vfs/*.vfs is + git-tracked. + +## Notes + +- tclsfe's _sfeinit.tcl hook is a runtime-specific customization channel; it is + deliberately NOT the fix - it cannot help interps punkshell fabricates with its + own paths, and the punkshell-side mechanism must stay runtime-agnostic. +- Related: [[G-005]] (zig static runtimes make this foundational), [[G-006]] + (artifact retrieval), the tk-loading experiment that surfaced it (punk91 = + tclsfe-x64 + punk9win.vfs per src/runtime/mapvfs.config).