punkshell/goals/archive/G-139-punkbin-library-consumption.md

G-139 Consume punkbin library artifacts: fetch/materialize tier replacing vendored tcllib/tcllibc

Status: achieved 2026-07-30 Scope: src/make.tcl (consent-gated library fetch entrypoint; libs step consuming the packages tier; vfslibs as touched); bin/packages// (new untracked materialization tier mirroring bin/runtime//); src/runtime/vendorlib_vfs.toml (tcllib/tcllibc kit-vfs declarations with supersedes removal of the legacy versions); src/vendorlib_tcl8/win32-x86_64/tcllib1.21 + src/vendorlib_tcl9/win32-x86_64/tcllib2.0 (retirement targets); src/vfs/*.vfs lib_tcl trees (kit payload switchover incl. the punk86-family tcllib 2.0 upgrade); lib_tcl8/ + lib_tcl9/ (deployed-tree materialization targets); user-config consent (server-trust pattern shared with G-123/G-006) Goal: punkshell obtains tcllib and tcllibc from the punkbin library tier (G-138) instead of committed vendored copies - a consent-gated fetch materializes verified library zips into an untracked bin/packages/ input tier from which make.tcl installs the deployed lib_tcl trees and the declared kit vfs payloads - retiring the hand-dropped src/vendorlib tcllib trees in the G-004 direction, with every materialized package self-describing via its embedded artifact record, and the derived-project story recorded (fetch as the default path; in-project suite build as the alternative). Acceptance: a make.tcl entrypoint fetches declared library artifacts from the punkbin lib tier into bin/packages// with sha1 verification and beside-toml metadata (consent keyed to server trust per the G-123 posture - fetches from the canonical origin prompt nothing new); the libs step materializes lib_tcl/allplatforms and lib_tcl/ from that tier with punkcheck-tracked provenance; tcllib and tcllibc enter src/runtime/vendorlib_vfs.toml (or its G-115/G-024 successor) declarations so the declared kit vfs trees receive them, with supersedes removing tcllib1.21 (including its nested tcllibc 0.4 and md5c) from the 8.6-family kits; the punk86-family upgrade to tcllib 2.0 + a current tcllibc is validated by kit-hosted runtests against a same-day baseline with every difference dispositioned; the vendored src/vendorlib_tcl8/win32-x86_64/tcllib1.21 and src/vendorlib_tcl9/win32-x86_64/tcllib2.0 trees are removed from the tracked tree once a bake fed from the artifact path reproduces working kits; a bake on a checkout without those trees (fetch consented or bin/packages pre-populated) ships kits whose tcllib/tcllibc provenance is verifiable from the embedded records; the derived-project decision is recorded in this file (fetch via the same entrypoint as the default - the copy-and-tweak in-project suite build per G-104's self-description contract remains the documented alternative).

Context

The publish half is G-138 (achieved 2026-07-30 - see goals/archive/G-138-punkbin-library-artifact-tier.md, including the original provenance investigation that motivated both). This goal owns the consuming side and the repo-changing consequences: what src/vendorlib_tcl means afterwards (it shrinks to genuinely hand-managed exceptions), how kit vfs trees stop drifting independently, and how derived projects get the same packages.

State at drafting worth recording:

• The deployed lib_tcl8|9 trees are pure punkcheck mirrors of src/vendorlib_tcl8|9 (make.tcl VENDORLIB_tcl step); src/lib_tcl8|9 are empty. So switching the vendorlib source of truth switches the deployed trees with no second mechanism.
• Kit vfs copies of tcllib (punk86/punk8win/punk8min/punk86bawt/ punk8_statictwapi lib_tcl8/tcllib1.21; mkzipfix/punk9cook-family lib_tcl9/tcllib2.0) are INDEPENDENT hand-drops from the same 2024 import - vendorlib_vfs.toml declares only tcludp today. Bringing tcllib/tcllibc under the G-037 declaration mechanism is part of this goal, not an aside.
• The 8.6 side is a version upgrade, not a swap: vendored is tcllib 1.21 + tcllibc 0.4 (2023, critcl 3.2); the suite builds tcllib 2.0 + tcllibc (critcl 3.3.1, zig cc). G-100 (achieved 2026-07-26) already ran punkshell runtests on the suite-built 8.6 runtime - this goal's validation burden is the KIT payload upgrade, using the established kit-hosted runtests method (vfscommonupdate-current kits, per the recorded traps).
• Orphaned stray copies were cleared 2026-07-30 ahead of this work: lib_tcl8/test_old + lib_tcl9/test_old moved (untracked, never on auto_path) to TEMP_REFERENCE/retired_lib_tcl_test_old/. Remaining known strays to sweep during the switchover: bin/test2/lib_tcl9/tcllib2.0, scratch/tclkit86bi.vfs/lib/tcllibc.

Approach

• bin/packages// mirrors the bin/runtime// input-tier idiom (untracked working copies + beside-toml metadata, sha1-verified on materialization); allplatforms is a target value here exactly as in the punkbin lib tier.
• Unzip via the punk::zip/punkzip machinery already in the tree; installs out of bin/packages into lib_tcl ride the existing punkcheck::install path so provenance/skip-unchanged behaviour matches the current vendorlib step.
• vendorlib_vfs.toml gains install entries for tcllib + tcllibc per generation, source paths pointing at the materialized tier; supersedes handles the legacy-version removal (the explicit no-silent-mixed-version mechanism it was built for). If G-115/G-024 land their toml consolidation first, the declarations ride the successor surface.
• Consent: no new mechanism - the server-trust posture of G-123 (canonical punkbin origin trusted by default; G-006's mechanism when landed, else the interim documented flag). Unattended bakes stay unattended.
• Retirement sequencing: artifact path proven (bake + kit checks green) BEFORE the vendored trees are removed; removal is its own commit so the fallback is a revert.
• Derived projects: default is the fetch path (their make.tcl copy gains the same entrypoint via the layout/G-027 update channels); projects wanting source-level control copy a suite (G-104 self-description contract) and build in-project - a tcllib/tcllibc-only suite needs no Tcl source tree (critcl bundles its own headers; any punk runtime can drive it), which keeps that alternative genuinely light. The stale samplesuite1 download_and_build.config stub in the project-0.1 layout should be replaced or removed as part of recording this story.

Alternatives considered

• Keep vendoring (suite output committed into src/vendorlib) - rejected: G-004 direction; see the archived G-138 Alternatives (goals/archive/G-138-punkbin-library-artifact-tier.md).
• Fetch directly into lib_tcl with no bin/packages tier - rejected: the input-tier split (immutable verified artifact copy vs punkcheck-installed deployment) is what makes re-bakes, freshness checks and multi-kit propagation coherent; it mirrors the proven runtime-tier design.
• Blanket-copy fetched packages into every kit vfs - rejected: per-package per-kit declaration is the settled G-037 design (kit package sets intentionally differ).

Notes

• Related: G-138 - the publish half; this goal consumes its tier. ACHIEVED 2026-07-30 (see goals/archive/G-138-punkbin-library-artifact-tier.md). What it settled that this goal consumes: artifact naming is <installed folder>-<tcl8|tcl9>-r<N>.zip (generation tag is identity - the tcl8/tcl9 builds are different bytes under identical folder names); each zip carries the package folder in installed shape + embedded schema-v2 class=library record at <folder>/punkbin-artifact.toml; md5c is DROPPED (tcllib's md5 prefers tcllibc, which bundles the accelerator) - so the supersedes removal of tcllib1.21's nested md5c needs no replacement package; emitted-but-unpublished artifacts stage under each suite's out/library/punkbin/lib/ (r1, nothing published yet - publication is the user step this goal's fetch consumes afterwards); punkbin build_sha1sums.tcl already handles the lib tier and its AGENTS.md documents the record schema.
• Related: G-123 - shared consent posture and materialization idiom (bin/runtime/ : runtimes :: bin/packages/ : libraries).
• Related: G-004 - the vendored-tree retirement is its first major payoff.
• Related: G-006 - consent surface, as coordinated through G-123.
• Related: G-065 - declarative vendoring generally; this goal implements the artifact-fetch shape for the tcllib case specifically, and G-065 should absorb or reference the resulting pattern rather than duplicate it.
• Related: G-115/G-024 - vendorlib_vfs.toml consolidation surfaces; the declarations here ride whichever surface is current.
• Related: G-027 - how derived projects receive the updated make.tcl entrypoint.
• Anchors (archived): G-037 (vendorlib -> kit vfs propagation mechanism and its supersedes design), G-100 (8.6 suite runtime validation this builds on), G-104 (buildsuite self-description contract the in-project-build alternative leans on - goals/archive/G-104-maketcl-buildsuite-surface.md), G-119 (freshness/row vocabulary if library listing surfaces are added - goals/archive/G-119-runtime-active-freshness.md).
• Overlap survey 2026-07-30 (drafting): shared with G-138 (see its Notes); additionally src/vfs intersects G-115/G-127/G-065 (payload declaration surfaces - carried as Related) and lib_tcl/make.tcl show no live-goal intersections.
• Activation survey 2026-07-30: goals_xref score G-139 re-run at activation (user-directed, same day as drafting - no goals drafted in the interval); no overlaps beyond the drafting survey.

Progress

• 2026-07-30 phase 1 landed: the fetch/materialize input tier and its make.tcl entrypoint (project 0.31.0).
  • make.tcl libfetch (argdoc'd subcommand, workflow-text + plain-help updated): reads src/runtime/libpackages.toml declarations ([artifact.] name/target), fetches from /lib// into bin/packages/<target>/ - zip + .toml sidecar, each sha1-verified against the server's fetched per-target sha1sums.txt (kept as the local tier copy, the punk-runtime store idiom) - then materializes each verified zip via punk::zip::unzip (defaults: crc-verified, mtimes restored, punkzip-accelerator-eligible) into bin/packages/<target>/tcl<N>/<pkgfolder>/ (generation subdirs: both generations install identically-named folders). Idempotent: present artifacts re-verify and are skipped; a materialized tree is skipped while its embedded record's name matches the declared artifact (the G-138 self-description payoff); -force re-does both.
  • Consent (per this goal's acceptance + the G-123 posture): canonical punkbin origin (the punk-runtime raw-base convention) prompts nothing; ANY other origin - -serverurl flag or PUNKBIN_URL env - refuses before network access without the explicit -trust-server flag. Never an interactive prompt. file:// origins supported natively (local mirrors, fixtures). Transport for http(s): Tcl http (+tls for https) when loadable, else curl, else PowerShell.
  • Verified 2026-07-30 against a file:// mirror assembled from BOTH suites' out/library/punkbin staging (the shape publication will produce): trust-gate refusal without the flag; consented fetch of all 4 declared r1 artifacts with sha1 verification; materialization of all 4 trees; full idempotent re-run (verified-present + tree-current); package-load proof from the tier (md5 2.0.9 + tcllibc from bin/packages/*/tcl9 under the plain family kit, accelerator engaged, ifneeded source path in the tier).
  • Emission-side amendment (both suites' library_artifacts.tcl): sidecar tomls now ride the per-tier staging sha1sums.txt (the punkbin repo's build_sha1sums.tcl already sums them; the staging mirror now matches), so sidecars are verifiable from a staging-derived mirror too.
  • bin/packages is ignored by both VCS via the existing /bin/* rules (no ignore edits needed). Docs: bin/AGENTS.md "Library package input tier", src/runtime/AGENTS.md libpackages.toml bullet, workflow diagram block.
  • Session note: the suite_tcl86 staging r1 zips changed bytes when its installer/critcl Run steps re-ran under this session's different PATH (zig Run cache keys include the env) - immaterial pre-publication, and exactly the deliberate-publish revision discipline's point; the tcl9 zips stayed byte-identical across three independent emissions.
• 2026-07-30 r1 PUBLISHED (user-directed): all four artifacts + sidecars copied into the punkbin repo's lib tier, build_sha1sums.tcl generated the per-tier sha1sums (every hash byte-identical to the staging emission; platforms.txt untouched), committed (punkbin ee571ed) and PUSHED to the gitea origin together with the earlier lib-tier docs commit. The canonical fetch path verified LIVE immediately after: 'make.tcl libfetch -force' with no server flags fetched, sha1-verified and materialized all four artifacts from the canonical origin - no consent gate, per the acceptance.
• 2026-07-30 phase 2 increment 1 (0.31.1): PACKAGES_tcl deployment phase (bin/packages tier -> lib_tcl/allplatforms + lib_tcl/, punkcheck-tracked, embedded records riding along - verified deployed); vfslibs schema gains source_root="packages" (missing-source errors name libfetch) + replace=true (clean-slate same-name refresh - the old tcl9 hand-drops nested md5c/tcllibc INSIDE tcllib2.0); tcllib/tcllibc kit-vfs declarations added for the 13 trees that carried hand-drops (8 win32 tcl9 trees get the pair + supersedes of the stray tcllibc2.0 name; punk9linux gets only platform-neutral tcllib2.0 - its linux-built tcllibc siblings stay, a linux lib-tier emission being future work; punk86/punk8win/ punk8_statictwapi upgrade via supersedes of tcllib1.21; punk86bawt refreshes its bare tcllibc).
• DERIVED-PROJECT DECISION (recorded per acceptance, 2026-07-30): fetch via the same entrypoint is the DEFAULT - generated projects receive make.tcl (and thus libfetch + the PACKAGES phase) through the established layout sync/G-027 channels and declare their artifacts in their own src/runtime/libpackages.toml; the copy-and-tweak in-project suite build per G-104's self-description contract remains the documented alternative (a tcllib/tcllibc-only suite needs no Tcl source tree - critcl bundles its headers). The stale samplesuite1 sketch in the project-0.1 layout was REMOVED and replaced by src/buildsuites/README.md in the layout, documenting both routes (the goal-named disposition).
• 2026-07-30 kit switchover + VALIDATION: vfslibs processed all 13 declared trees (45 installs; clean-slate replaces verified - nested md5c/tcllibc gone from every tcl9 tcllib2.0, tcllib1.21 + its nested tcllibc 0.4 + md5c superseded out of the tcl8 kits, stray tcllibc2.0 names removed where declared, punk9linux's linux tcllibc siblings untouched). Selective bake punk86 + punk905 (punk91/punksys/punk9_beta are live user shells - their vfs trees switched now, bakes at the next natural rebake): both built and deployed; punk905 arch scan OK with tcllibc exempt under its platform subdir; punk86's only BUILD-WARNING is the recorded pre-existing zint i386 defect (known, not a regression). Kit-level provenance verified BOTH container types: punkzip central-directory read of punk905.exe (zip kit) lists both embedded records without executing it; an in-vfs probe of punk86.exe (metakit kit) reads both records naming their source artifacts + build_ids. KIT-HOSTED RUNTESTS VALIDATION (same-day baseline bracket isolating exactly the kit payload switchover): baseline (pre-switchover punk86) 1125 total / 1099 passed / 18 skipped / 8 failed in 5 files (exec.test, ns/nslist, ns/nsprimitives, zip/zipaccel, stdin/runstdin; 1960s); post-switchover run IDENTICAL - same totals, same 5 files, and the full per-test outcome extraction (1119 lines) diffs EMPTY against baseline. Differences to disposition: none (the 8 failures pre-exist on the old payload and are unrelated to tcllib).
• 2026-07-30 retirement + post-removal verification: vendored src/vendorlib_tcl8/win32-x86_64/tcllib1.21 and src/vendorlib_tcl9/win32-x86_64/tcllib2.0 removed in their own commit (git 70d1838c - removal-only, fallback is a revert) AFTER the artifact path proved out; stray sweep done (bin/test2/lib_tcl9/tcllib2.0, scratch/tclkit86bi.vfs/lib/tcllibc, deployed lib_tcl stale vendored mirrors - all untracked build-output areas). Post-removal verification bake (punk86 + punk905, tier pre-populated + canonical fetch consented): both kits shipped from the tcllib-vendorless checkout with NO provenance warnings (clean tracked tree) and only the recorded pre-existing zint i386 warning; embedded records re-verified in both container types (in-vfs probe / punkzip central-directory read).
• ACHIEVED FLIP 2026-07-30. Acceptance walk: libfetch entrypoint with sha1 verification + beside-toml metadata + server-trust consent, proven LIVE against the canonical origin post-publication (no prompts/flags); PACKAGES_tcl libs-step materialization into lib_tcl/allplatforms + lib_tcl/win32-x86_64 with punkcheck provenance; tcllib/tcllibc vendorlib_vfs.toml declarations (source_root=packages + replace) with supersedes removing tcllib1.21 incl. nested tcllibc 0.4 + md5c from the 8.6-family kits; punk86 kit-hosted runtests validation vs the same-day baseline - IDENTICAL results, zero differences to disposition; vendored trees removed in their own commit after the proven bake; the post-removal bake ships kits with verifiable embedded provenance; derived-project decision recorded (fetch default / G-104 suite alternative; layout samplesuite1 sketch replaced by the routes README). Follow-through notes for the future: punk9linux's linux-built tcllibc/tcllibc2.0 siblings await a linux-x86_64 lib-tier emission (candidate goal - not written); kits other than punk86/punk905 switch payloads at their next natural rebake (punk91/punksys/punk9_beta were live user shells this session); a piped binscripts characterization test for libfetch (httpfixture pattern) is a natural test-coverage follow-up.