punkshell/goals/G-037-vendorlib-vfs-propagation.md

G-037 Propagate platform vendor libraries into kit vfs lib_tcl trees via make.tcl

Status: achieved 2026-07-08 Scope: src/make.tcl (new or extended step), src/vendorlib_tcl8 + src/vendorlib_tcl9 (sources), src/vfs/.vfs/lib_tcl8 + lib_tcl9 (targets), punkcheck tracking Acceptance: with a newer package version placed under src/vendorlib_tcl9/, one documented make.tcl invocation updates the participating src/vfs/*/lib_tcl9 trees - installing the new package and removing or explicitly retiring the superseded version (no silent mixed-version provision, per the G-035 concerns) - with punkcheck-tracked provenance; which vfs folders participate is explicitly declared per kit rather than blanket-copied (kit vfs package sets may intentionally differ), with the declaration mechanism recorded (candidate home: the G-024 mapvfs toml); a subsequent make.tcl project yields kits loading the new version (provable via the tcludp case: built punk902z reports package require udp == 1.0.13 with no udp1.0.12 folder remaining in its vfs); the lib_tcl8 tree gets the same treatment or an explicit exclusion rationale in the goal record.

Context

Motivating case (2026-07-08, the G-036 wedge remedy): tcludp 1.0.13 was placed in src/vendorlib_tcl9/win32-x86_64 and make.tcl libs, make.tcl vfscommonupdate, and make.tcl project were all run - yet the rebuilt punk902z still loaded udp 1.0.12, because every kit vfs still carried lib_tcl9/udp1.0.12. The upgrade had to be hand-copied into each kit vfs folder (done for punk9win.vfs and punk9win_for_tkruntime.vfs in commit a1b232dc).

What the existing targets actually cover (verified in make.tcl at the time of writing):

• libs/project (VENDORLIB steps, make.tcl ~lines 2420-2475): copy src/vendorlib -> <projectroot>/lib, and src/vendorlib_tcl<N>/{allplatforms,<platform>} -> <projectroot>/lib_tcl<N>/{allplatforms,<platform>} (punkcheck-tracked). These feed tclsh-based use of the project tree - never the kit vfs folders.
• vfscommonupdate (~lines 1934-1963): wholesale-replaces src/vfs/_vfscommon.vfs/modules and .../lib from <projectroot>/modules and <projectroot>/lib. _vfscommon.vfs carries no lib_tcl<N> platform tree at all.
• Kit vfs lib_tcl<N> folders (punk9win.vfs, punk9win_for_tkruntime.vfs, punk8win.vfs, etc.) are hand-maintained; project merges _vfscommon.vfs over each kit vfs but nothing sources the platform-lib payloads from vendorlib.

Approach (design points to settle)

• Per-kit participation must be declarative, not blanket. Kit vfs package sets intentionally differ (e.g. punk9win vs punk9win_for_tkruntime vs slim kits), and a vendorlib drop must not force a package into every kit. Candidate home for the declaration: the mapvfs config, especially once G-024 moves it to toml (a per-kit package/include list or a vendorlib participation flag).
• Supersession semantics: installing foo<newver> alongside foo<oldver> creates mixed-version provision (G-035 territory) - the step should remove or explicitly retire the superseded folder, or refuse with a clear message. Folder-name conventions currently differ (udp1.0.12 in vfs vs tcludp1.0.13 in vendorlib) - the sync should make the target name match the source, and tolerate legacy names when detecting supersession (pkgIndex-declared package name/version, not folder name, is the truth).
• Platform axis: vendorlib platform folders are per-<platform_generic> (win32-x86_64 today); kit vfs folders are platform-specific by kit. The step should map vendorlib platform -> the kits built for that platform (mapvfs already knows runtime -> vfs -> output mappings).
• punkcheck provenance as with the other install steps (skip unchanged, record source).
• lib_tcl8: same mechanism should serve the tcl8 kits (punk8win.vfs still bundles udp 1.0.12 as of 0.4.3 - see G-036 loose ends), or the exclusion is recorded here.

Implementation (2026-07-08)

Mechanism as landed (design points above settled as follows):

• Declaration: src/runtime/vendorlib_vfs.toml, parsed with the bootsupport tomlish package (1.1.10) - no ad-hoc parsing. Named tables [install.<name>] with keys: source (package folder relative to src/), targets (list of kit vfs lib folders relative to src/vfs/), optional supersedes (folder names removed from each target before install). G-024's mapvfs toml conversion was NOT required first - this file is recorded as a candidate for absorption into it. (make.tcl's own help already carried a matching intent: "update the src/vendorlib based on src/vendorlib/config.toml (todo)".)
• Step: make.tcl vfslibs (new subcommand), also run automatically inside make.tcl project before kit vfs assembly. Malformed/unresolvable entries fail loudly naming the entry (exit 3).
• Supersession: explicit supersedes folder-name list per entry (the simpler option; pkgIndex-declared package identity detection remains a possible refinement). The step never removes the folder it is about to install, and removal happens before install.
• Direction/authority: declaration-driven only - nothing syncs unless declared, so the observed twapi drift (vendorlib 5.0b1 older than kit copies) is untouched until someone declares a twapi entry. This resolves the "explicit direction/authority semantics" question: authority is the declaration, not file timestamps/versions.
• Cross-generation sources: allowed when the package supports it - the motivating tcludp 1.0.13 folder is dual-generation (tcl9udp1013.dll + udp1013t.dll, pkgIndex selects by [package vsatisfies]), so the single vendorlib_tcl9 source serves the punk8win.vfs/lib_tcl8 target (no duplicate binary committed under vendorlib_tcl8).
• punkcheck provenance: records at src/vfs/.punkcheck via -punkcheck-folder (kept outside the .vfs payload folders so no records are baked into kits); skip-if-unchanged verified (second run: 0 copied, 9 skipped per target).
• lib_tcl8: same mechanism, exercised immediately - punk8win.vfs/lib_tcl8 had udp1.0.12 removed and tcludp1.0.13 installed via the declaration (the G-036 loose-end decision, made 2026-07-08).

Verification (2026-07-08, acceptance met - goal flipped to achieved):

• make.tcl vfslibs: installed tcludp1.0.13 into punk8win.vfs/lib_tcl8 (9 files), removed superseded udp1.0.12; re-recorded the two tcl9 kit targets (previously hand-copied). Second run: 0 copied / 9 skipped per target (punkcheck skip-unchanged).
• make.tcl project (with the vfslibs step now inside it): INSTALLED KITS punkbi.exe, punksys.exe, punk902z.exe. punk9win_for_tkruntime.vfs correctly detected as unchanged (its 1.0.13 payload predated the step) - cached kit redeployed.
• Built kits: punksys 8.6.13 package require udp = 1.0.13 loading lib_tcl8/tcludp1.0.13/udp1013t.dll; punkbi likewise; punk902z 9.0.2 = 1.0.13 loading zipfs lib_tcl9/tcludp1.0.13/tcl9udp1013.dll. No udp1.0.12 folder remains anywhere under src/vfs. Piped-shell exit smoke on punksys clean (PUNK_PIPE_EOF=exit).

Alternatives considered

• Extending vfscommonupdate to also sync platform libs into _vfscommon.vfs - rejected as-is: _vfscommon.vfs is deliberately platform-neutral common payload; platform libs are per-kit and per-platform, so the declaration belongs at the kit/mapvfs level.
• Documented manual copy as the permanent process - rejected: silent staleness (the motivating case shipped kits with a known-buggy tcludp through three make invocations with no warning).

Notes

• Related goals: G-024 (mapvfs toml - natural home for per-kit declarations), G-026 (vendor provenance policy), G-035 (mixed-version provision characterisation), G-004/G-005 (the no-committed-binaries endgame would reshape where these payloads come from), G-036 (the motivating tcludp upgrade).
• Interim state: tcl9 kits carry tcludp1.0.13 (hand-copied, commit a1b232dc); punk8win.vfs still carries udp1.0.12.
• Further drift evidence (2026-07-08): src/vendorlib_tcl9/win32-x86_64 carries twapi-5.0b1 while the punk9win kit vfs bundles twapi 5.2.0 - vendorlib and kit vfs trees have already diverged in BOTH directions (vendorlib newer for udp, older for twapi), so the sync step needs explicit direction/authority semantics, not just "copy newer over older".