punkshell/goals/archive/G-165-driver-invariant-kit-assembly.md

G-165 Driver-invariant kit assembly: single zip writer path with punkzip acceleration, file-relative by explicit opt-in only

Status: achieved 2026-08-04 Scope: src/make.tcl (zip-kit assembly branch); src/modules/punk/zip-999999.0a1.0.tm (mkzip write-path acceleration seam); src/tools/punkzip/ (re-vendor after upstream enabling changes); src/runtime/mapvfs.toml (per-kit offsetstyle opt-in); src/tests/ (accelerated/floor parity + driver-vintage characterization); src/AGENTS.md; bin/AGENTS.md Goal: every make.tcl zip-kit bake emits an ARCHIVE-relative kit regardless of the driving interpreter (legacy file-relative mkimg in 8.7/9.0.0, fixed mkimg in 9.0.1+, or no mkimg at all) and produces equivalent kit zip payloads for a given merged tree across drivers, platforms and architectures - supporting bake servers running sizeable platform/content matrices: zip-kit assembly uses a single writer layer, punkzip (zig) when its binary is present with punk::zip::mkzip (pure Tcl) as the always-available floor, semantically equivalent, with tcl::zipfs::mkimg retired from the pipeline; file-relative output is available only as an explicit per-kit opt-in recorded in mapvfs.toml - the G-134 deliberate-creation carve-out expressed as configuration, never ambient driver behaviour. Acceptance: a kit baked from the same source tree under all three driver classes on the reference machine - tclsh 8.7a6 (legacy file-relative mkimg, the reported failure case), tclsh90 9.0.x (fixed mkimg), and a mkimg-less tclsh (the zipcat-only class) - probes offsetstyle=archive via punk::zip::archive_info with identical member sets and member CRCs across the three runs; with bin/punkzip present the bake assembles via punkzip (visible in bake output) and a write-path parity suite proves member-set, CRC and offsetstyle equivalence between the accelerated and pure-Tcl paths over the G-124 fixture set plus a real .vfs payload, while a no-punkzip bake of the same tree produces an equivalent archive; a kit declaring offsetstyle="file" in its mapvfs.toml entry probes file - the opt-in recorded, and the offsetstyle.test baseline exempts declared kits - while G-134's BAKE-WARNING never fires on default (non-opt-in) bakes under any driver class; upstream punkzip gains the root-strip and repeatable-exclude options the bake's call shape requires (zig build test green, entries rooted at the payload tree with mkzip's default exclusion set honoured identically), re-vendored with PROVENANCE.md updated and tools-tier publication recorded or explicitly user-deferred; make.tcl's kit assembly path contains no tcl::zipfs::mkimg call; full punkexe + punkboot utils suites green in both runner modes; and src/AGENTS.md plus bin/AGENTS.md state the single-writer contract (consistent kits regardless of driving interpreter; file-relative by explicit opt-in only).

Context

The zip-kit assembly branch in make.tcl selects its writer on the PRESENCE of tcl::zipfs::mkimg in the driving interpreter (~line 9135), so the kit's zip offset convention is a property of which tclsh drives the bake, not of the pipeline. mkimg's convention changed upstream: pre-2021 archive-relative, then file-relative (internal preamble) from 2021 until 2024-12-05, when the fix argued for by the developer landed (Tcl ticket aaa84fbbc5, submitted juliannoble2 with patch; fixed in 3d62ee540d by jan.nijtmans; the same change retired the build system's zip -A step). Drivers therefore split by vintage: 8.7 alphas and 9.0.0 emit the legacy file-relative convention (measured on the reference machine 2026-08-04: prefixed-mkimg probe under tclsh 8.7a6 -> offsetstyle=file), 9.0.1+ and 9.1 emit archive-relative (same probe under tclsh90 9.0.3 -> offsetstyle=archive), and mkimg-less drivers take the zipcat concatenation path (archive-relative by mkzip default). An 8.7-driven bake today produces file-relative kits: G-134's advisory BAKE-WARNING fires (by design, but signalling "pipeline regression" when the cause is driver vintage), and offsetstyle.test's _bake baseline goes permanently red for such users. The kits themselves mount and boot on either convention (G-129 measured); the material consumer is G-128's punkres stamper, which refuses file-relative payloads by default because appending a PE section shifts the overlay.

User intent (stated 2026-08-04): punkshell produces consistent kits regardless of the driving tclsh; file-relative kits are by explicit user configuration/opt-in, not a default; legacy file-offset mkimg runtimes will exist in the wild for many years. Server-driven bakes across a platform/architecture/content matrix are in scope, so assembly throughput matters beyond the single-bake case.

Write-path performance measured 2026-08-04 (reference machine; whole-tree assembly of src/_bake/punk91.exe.vfs and a 3x duplication; all writers C-zlib deflate with store-if-larger; output sizes within 1.5%):

payload	mkimg (9.0.3)	punk::zip::mkzip (Tcl)	punkzip build (zig)
132MB / 3,645 files	11.4s	8.1s	6.7s
396MB / 10,935 files	35.9s	24.8s	20.0s

Linear scaling in all three (mkimg's figure includes its unavoidable per-member progress printing, also what a real bake emits). No wall short of multi-GB payloads. Conclusion: the pure-Tcl floor is viable as the sole path; punkzip acceleration is a matrix-scale dividend, not a correctness prerequisite. (Scratch benchmark artifacts: temp zipbench dir, regenerable.)

punkzip (v2.3.1) gaps for kit-payload writing, measured the same day: build carries the input path into entry names (no root-strip option - entries must root at the payload tree) and has no exclude-pattern option (mkzip's default exclusion set has no equivalent). Non-gaps already present: sorted deterministic entries, directory entries (zipfs mounts by trailing slash), comment flag, deflate + store-if-larger equivalence.

Approach

Two workstreams, ordered by user decision 2026-08-04 (upstream first as the enabling step):

1. Upstream punkzip (developer's repo c:/repo/jn/zig/punkzip, re-vendored per src/tools/AGENTS.md): add root-strip (entries rooted at a named base, the mkzip -base semantics) and repeatable exclude patterns; verify store-if-larger and determinism unchanged; version bump; zig build test green; tools-tier publication per the punkres route (G-123, achieved - see goals/archive/G-123-thirdparty-runtime-tiers.md) or explicit user deferral.
2. punkshell: a write-path acceleration seam inside punk::zip::mkzip mirroring G-126's read-path pattern (detect bin/punkzip -> shell out with the equivalent strip/exclude arguments; pure-Tcl path unchanged as the floor, with parity as the contract), so all mkzip consumers accelerate, not just the bake; make.tcl's kit assembly retires the tcl::zipfs::mkimg branch in favour of the single zipcat compose point (runtime prefix concatenated outside the writer, as assemble_zipcat_image already does); per-kit offsetstyle key in mapvfs.toml (exact key name settled at implementation; opt-in kits always take the floor with -offsettype file, which mkzip already supports - punkzip needs no offset-adjustment scope); offsetstyle.test's baseline reads the mapping and exempts declared kits; G-134's pin and advisory posture stay, its warning text accurate again (file means opt-in or genuine regression, never driver vintage); docs in src/AGENTS.md (G-122/G-134 bullets) and bin/AGENTS.md.

Alternatives considered

• Normalize old-mkimg output to archive-relative post-assembly (conversion) - rejected: keeps two assembly paths and requires the parked in-place conversion machinery (G-128's open follow-on); manages the sensitivity rather than removing it.
• Probe the driver's mkimg convention once per bake and specialize warnings/test gating - rejected for the same reason: driver-invariant output is the intent, not driver-aware messaging.
• Retire the mkimg branch WITHOUT the punkzip acceleration layer - viable on the measured numbers (the floor beats mkimg at both measured sizes); the acceleration layer was user-directed for the bake-matrix case and adopted.
• Acceleration seam in make.tcl (bake-only) rather than inside punk::zip::mkzip - rejected: the mkzip seam accelerates every consumer with one parity contract and mirrors G-126 exactly.
• Bake CLI flag for the file-relative opt-in - rejected: session-scoped; the per-kit mapvfs declaration is durable configuration (user decision, opt-in shape (a), 2026-08-04).

Progress

• 2026-08-04 (workstream 1, upstream punkzip - LANDED): root-strip build -b <base> (mkzip -base semantics: entry names rooted at base, base's own entry not stored, fileless directories pruned) and repeatable build -x <pattern> (Tcl string match against the entry name; '*' crosses '/', '?', '[...]' ranges, no class negation - verified empirically against tclsh) landed upstream as v2.4.0, committed 2026-08-04 as c:/repo/jn/zig/punkzip 2599127fa98dae2df8e9ccf3647f51c08dc8f3c0 (user's commit word: "Commit upstream punkzip as necessary to do this work"). Entry name and disk path split (MemberPathInfo.diskpath); add_entry opens by disk path. Tests: new zipper_test.zig (5 unit: glob matcher, rel_to_base, prune) plus 5 new CLI tests and the usage pin update; zig build test exit 0, all suites green standalone (test_punkzip 23/23, test_zipper 5/5; 140/140 total). Differential probe against the pure-Tcl floor: mkzip (default exclusion set) vs punkzip -b/-x produced identical member names, CRCs, directory flags and method choices over a fixture covering all five default patterns at two depths plus empty dirs (compressed bytes within 1 byte - legitimate deflate implementation difference). Diagnosed and documented in upstream AGENTS.md a pre-existing zig 0.16.0 windows test-runner teardown artifact ("failed command ... --listen=-" with overall exit 0; gate is the exit code per src/tools/AGENTS.md). Re-vendored into src/tools/punkzip/ the same day per PROVENANCE.md (vendored-state line, lineage, and the src/tools/AGENTS.md 140/140 count updated); vendored tree zig build test exit 0; make.tcl tool build punkzip test gate passed and installed bin/punkzip.exe v2.4.0 (ReleaseSafe; smoke-verified: usage shows -b/-x and the deployed binary reproduces the floor's parity member set).
• 2026-08-04 (workstream 2, punkshell - LANDED, all acceptance evidence in): (a) write-path acceleration seam inside punk::zip::mkzip mirroring G-126 (walk stays authoritative for member set and returned names; exec punkzip build -b -x ...; produced archive read back and member names compared against the walk result; any failure/unreadable/mismatch deletes the file and falls back silently; -offsettype file and -runtime/-zipkit prefixes always floor; diagnostics last_write_engine/last_write_note) - punk::zip 0.4.0, bootsupport-promoted. (b) make.tcl: the tcl::zipfs::mkimg branch retired (zip kits always assemble via assemble_zipcat_image, itself routed through punk::zip::mkzip with the project's bin/punkzip.exe pointed at the write seam - the driving tclsh is not beside it); G-134 warning now fires only on UNDECLARED file-relative output and exempts declared kits; workflow text updated; bake/bakehouse -force flag added (user-endorsed 2026-08-04: bypasses the 'no change detected' skip - used for the per-driver forced re-assembly); punkkit-stamp.toml now written with pinned LF translation (the default text translation followed the driver - crlf vs lf - making the stamp's bytes driver-dependent; caught by the four-driver parity run's only initial CRC diff). (c) mapvfs.toml: offsetstyle key documented in the header spec and validated in entry_reader ("file" only, zip-type only, entry-named errors); [kit.punkfiledemo] declared specimen (bake_default=false). (d) offsetstyle.test baseline reads the declared set from mapvfs.toml and exempts declared kits (asserting they DO probe file)
  • 7/7. (e) docs: src/AGENTS.md (G-122 single-writer bullet, G-134 opt-in-or-regression bullet), bin/AGENTS.md (G-129 emit-vs-mount note, G-134 opt-in exemption), src/runtime/AGENTS.md (offsetstyle key). (f) zipwriteaccel.test (14/14; #added 2026-08-04 (agent, G-165)): accelerated-vs-floor parity over the G-124 tree shape, an exclusion-heavy tree (default set honoured identically), custom excludes, comment, floor-when-absent, ineligible-call floors, failure fallback, return contract, and the committed _vfscommon.vfs payload (65.8MB); method choice on already-compressed members documented engine-dependent (crc/name/isdir are the contract). ACCEPTANCE RUNS (all on the reference machine): punkluck86 baked under tclsh 8.7a6 (legacy mkimg), tclsh90 9.0.3 (fixed mkimg), tclsh86 8.6.17 (native, mkimg-less) and msys2 /usr/bin/tclsh8.6 8.6.12 (unix platform, mkimg-less - the G-122 class): all four probe offsetstyle=archive with IDENTICAL member sets and CRCs (3319 members), each visibly assembled via the accelerated seam, zero offsetstyle BAKE-WARNINGs; a no-punkzip bake produced an equivalent archive (floor path); punkfiledemo probes file with no warning (declared); stale file-relative punk9_beta (pre-G-165 8.7 artifact) rebaked to archive, clearing the offsetstyle baseline. Suites: zip 53/53 (tclsh90), punkboot utils 81/81 both runner modes, punkexe 146/150 (4 constraint skips, 0 failures) both runner modes, modpod roundtrip 7/7, maketclhelp 12/12 (bakehouse synopsis pin updated for -force). punkproject 0.52.0 + CHANGELOG; projectversion check OK.
• 2026-08-04 (tools-tier publication - user's word "publish"): punkzip 2.4.0 published to the punkbin tools tier as win32-x86_64/tools/punkzip-x86_64-windows-2.4.0.exe (sha1 2008b51ffa26e31faa0e76ea1e229ee1cec16395, recorded in the tier's sha1sums.txt via src/build_sha1sums.tcl - NEW, no CHANGED), punkbin git a22e16b, pushed to the canonical origin. Non-windows cross-builds remain a follow-on (see Follow-ons). This satisfies the acceptance's final item (publication recorded - not deferred).

Notes

• Related: G-122 (achieved - see goals/archive/G-122-host-target-platform-split.md) - introduced the zipcat concatenation fallback (assemble_zipcat_image) this goal makes the only assembly path.
• Related: G-124 (achieved - see goals/archive/G-124-punkzip-reader.md) - the punk::zip reader/archive_info classification instrument used by the G-134 pin and this goal's acceptance; its bootsupport >= 0.2.0 extraction requirement is long satisfied.
• Related: G-126 (achieved - see goals/archive/G-126-punkzip-accelerator.md) - the read-path acceleration pattern (detect binary -> fast path, pure-Tcl floor, parity suite) mirrored for writes; the vendored-tool build step and provenance/publication route this goal reuses.
• Related: G-128 (achieved - see goals/archive/G-128-portable-pe-resource-stamping.md) - the stamper whose default refusal makes archive-relative the load-bearing convention. Its open in-place-conversion follow-on is NOT enacted here (mkzip -offsettype file serves the opt-in) and stays open.
• Related: G-129 (achieved - see goals/archive/G-129-kitboot-mountpoint-derivation.md) - runtime zipfs requirements; both offset conventions mount on both verified runtime families, which is why driver-vintage kits booted despite being off-convention.
• Related: G-134 (achieved - see goals/archive/G-134-baked-kits-pinned-archive-relative.md) - the pin this goal makes driver-independent by construction; its deliberate-creation carve-out (user decision 2026-07-27) becomes the per-kit mapvfs opt-in.
• Related: G-131 (live) - deliberate non-overlap: container autodetection (metakit or zipfs) is orthogonal to zip offset convention; the single-writer change touches only the zipfs kit path and must not disturb G-131's metakit branches or its make.tcl check reconciliation.
• Related: G-158 (live) - sibling surface: the 8.6 'kit' (metakit/sdx) lane is a different container family, untouched here.
• Related: G-005/G-006 (live) - binary build/retrieval infrastructure; the bake-server punkzip distribution rides the tools-tier route (punkres precedent), G-105-family cross-builds for non-windows bake servers.
• Related: G-162 (live) - sibling bake-surface goal: its content-digest freshness stamps meet this goal's writer swap inside make.tcl's bake/bakelist path. Classified orthogonal at activation: the digests run over bake inputs, which this goal does not change (the writer layer is not an input) - re-check if G-162 ever digests bake OUTPUTS, since the accelerated and floor writers are member-set/CRC-equivalent, not byte-pinned.
• Related: G-142 (live) - punkbin curated listing manifests: the tools-tier publication this goal's workstream 1 rides (punkzip cross-builds for non-windows bake servers, a follow-on candidate) would surface artifacts in punkbin, whose per-tier listing manifests are G-142's subject.
• Overlap survey 2026-08-04 (drafting): goals_xref.tcl paths src/make.tcl src/modules/punk src/tools/punkzip src/runtime/mapvfs.toml - no live goals intersect the first three; mapvfs.toml surfaces path-mention hits (G-131, G-158, G-005, G-006, G-023, G-141), all read and classified orthogonal to offset convention (container type, kit naming, libfetch target selection, 8.6 kit parity). goals_xref.tcl report unlinked-pair list reviewed: no pair intersects this goal's scope.
• Activation survey 2026-08-04 (proposed -> active flip, user-confirmed): goals_xref.tcl score G-165 re-run at activation; new Related lines recorded for G-162 and G-142 (above). One-directional pairs (G-158, G-131, G-141, G-105, G-023, G-005, G-006) already carry this goal's pointer from drafting - back-pointers optional, not added. Remaining top unlinked pairs (G-137, G-073, G-147, G-157, G-034) reviewed and classified lexical noise (bin/AGENTS.md adjacency, generic term overlap). goals_lint clean after the flip (82 active-index, 83 archived).
• The offsetstyle.test baseline's premise ("today's bakes produce no warning") was environment-dependent when written (the reference driver has fixed mkimg); this goal makes it driver-independent again - the baseline stays a true tripwire once opt-in kits are exempted by declaration.
• mkzip's per-member punk::args::parse in Addentry was profiled as a suspected hot spot before the benchmark; measured all-in per-member cost ~2.2ms at 3.6k members - not a blowup, left as-is (floor stays simple and correct).
• The same vintage sensitivity exists one level down in the buildsuites' runtime wrapper (src/buildsuites/suite_tcl90/tools/zipfs_mkimg.tcl): a runtime built with a legacy-mkimg driver comes out file-relative (the store's tclsh90b4_piperepl.exe is exactly such an artifact). Tolerated on input by the G-124 reader; recorded here as hygiene, not in this goal's scope - a candidate follow-on if runtime-side consistency is later wanted.

Follow-ons

Follow-on: tools-tier cross-builds of punkzip for non-windows bake servers (linux/macosx/freebsd and other arches; the G-105-family cross-build route, punkres publication precedent) => open Follow-on: buildsuites runtime-side zipfs_mkimg.tcl wrapper carries the same mkimg vintage sensitivity (a runtime built with a legacy-mkimg driver comes out file-relative; tolerated on input by the G-124 reader) - hygiene, runtime-side consistency if ever wanted => landed 2026-08-05 (src/buildsuites/suite_tcl90/tools/zipfs_wrap.tcl: payload written by the driving shell's builtin tcl::zipfs::mkzip - convention-stable across mkimg vintages, probed 8.7a6 vs 9.0.3 - with the prefix executable concatenated outside the writer, plus an EOCD archive-relative self-check per wrap; build905.zig wrap lane renamed zipwrap_*, suite README updated; suite make-zipfs/smoke/kit-family green, all 5 products probe archive-relative and boot)