G-165 proposed: driver-invariant kit assembly (single zip writer path, punkzip acceleration, file-relative opt-in)

Every make.tcl zip-kit bake is to emit an ARCHIVE-relative kit regardless
of the driving interpreter's zipfs vintage (legacy file-relative mkimg in
8.7/9.0.0, fixed mkimg in 9.0.1+, or no mkimg at all): zip-kit assembly
moves to a single writer layer - punkzip (zig) when its binary is present,
punk::zip::mkzip (pure Tcl) as the always-available floor - with
tcl::zipfs::mkimg retired from the pipeline, and file-relative output only
via an explicit per-kit mapvfs.toml opt-in. Workstream order (user
decision): upstream punkzip root-strip/exclude enabling changes first,
then the punkshell seam. The detail file carries the driver-vintage probe
evidence (8.7a6=file, 9.0.3=archive), the upstream fix provenance (ticket
aaa84fbbc5, core 3d62ee540d 2024-12-05), the write-path benchmark (mkimg
11.4/35.9s vs mkzip 8.1/24.8s vs punkzip 6.7/20.0s at 132MB/396MB), the
drafting-time overlap survey and Related lines.

Assisted-by: harness=opencode; primary-model=openrouter/moonshotai/kimi-k3; api-location=openrouter.ai

GOALS.md

@@ -404,3 +404,7 @@ Detail: goals/G-162-bakelist-kit-freshness.md
 Scope: src/modules/punk/repl-999999.0a1.0.tm (exit/eof done paths, reader-window blocking-restore discipline); src/modules/punk/console-999999.0a1.0.tm (drain helper, -blocking 0 window sites); src/lib/app-punkshell/punkshell.tcl (final-exit seam); ARCHITECTURE.md (Process exit and shutdown section); src/tests/shell/testsuites/punkexe/ (unix-pty-gated drain pin)
 Detail: goals/G-163-unix-exit-output-drain.md
 
+### G-165 [proposed] Driver-invariant kit assembly: single zip writer path with punkzip acceleration, file-relative by explicit opt-in only
+Scope: src/make.tcl (zip-kit assembly branch); src/modules/punk/zip-999999.0a1.0.tm (mkzip write-path acceleration seam); src/tools/punkzip/ (re-vendor after upstream enabling changes); src/runtime/mapvfs.toml (per-kit offsetstyle opt-in); src/tests/ (accelerated/floor parity + driver-vintage characterization); src/AGENTS.md; bin/AGENTS.md
+Detail: goals/G-165-driver-invariant-kit-assembly.md
+

goals/G-165-driver-invariant-kit-assembly.md

@@ -0,0 +1,158 @@
+# G-165 Driver-invariant kit assembly: single zip writer path with punkzip acceleration, file-relative by explicit opt-in only
+
+Status: proposed
+Scope: src/make.tcl (zip-kit assembly branch); src/modules/punk/zip-999999.0a1.0.tm (mkzip write-path acceleration seam); src/tools/punkzip/ (re-vendor after upstream enabling changes); src/runtime/mapvfs.toml (per-kit offsetstyle opt-in); src/tests/ (accelerated/floor parity + driver-vintage characterization); src/AGENTS.md; bin/AGENTS.md
+Goal: every make.tcl zip-kit bake emits an ARCHIVE-relative kit regardless of the driving interpreter (legacy file-relative mkimg in 8.7/9.0.0, fixed mkimg in 9.0.1+, or no mkimg at all) and produces equivalent kit zip payloads for a given merged tree across drivers, platforms and architectures - supporting bake servers running sizeable platform/content matrices: zip-kit assembly uses a single writer layer, punkzip (zig) when its binary is present with punk::zip::mkzip (pure Tcl) as the always-available floor, semantically equivalent, with tcl::zipfs::mkimg retired from the pipeline; file-relative output is available only as an explicit per-kit opt-in recorded in mapvfs.toml - the G-134 deliberate-creation carve-out expressed as configuration, never ambient driver behaviour.
+Acceptance: a kit baked from the same source tree under all three driver classes on the reference machine - tclsh 8.7a6 (legacy file-relative mkimg, the reported failure case), tclsh90 9.0.x (fixed mkimg), and a mkimg-less tclsh (the zipcat-only class) - probes offsetstyle=archive via punk::zip::archive_info with identical member sets and member CRCs across the three runs; with bin/punkzip present the bake assembles via punkzip (visible in bake output) and a write-path parity suite proves member-set, CRC and offsetstyle equivalence between the accelerated and pure-Tcl paths over the G-124 fixture set plus a real .vfs payload, while a no-punkzip bake of the same tree produces an equivalent archive; a kit declaring offsetstyle="file" in its mapvfs.toml entry probes file - the opt-in recorded, and the offsetstyle.test baseline exempts declared kits - while G-134's BAKE-WARNING never fires on default (non-opt-in) bakes under any driver class; upstream punkzip gains the root-strip and repeatable-exclude options the bake's call shape requires (zig build test green, entries rooted at the payload tree with mkzip's default exclusion set honoured identically), re-vendored with PROVENANCE.md updated and tools-tier publication recorded or explicitly user-deferred; make.tcl's kit assembly path contains no tcl::zipfs::mkimg call; full punkexe + punkboot utils suites green in both runner modes; and src/AGENTS.md plus bin/AGENTS.md state the single-writer contract (consistent kits regardless of driving interpreter; file-relative by explicit opt-in only).
+
+## Context
+
+The zip-kit assembly branch in make.tcl selects its writer on the PRESENCE of
+`tcl::zipfs::mkimg` in the driving interpreter (~line 9135), so the kit's zip
+offset convention is a property of which tclsh drives the bake, not of the
+pipeline. mkimg's convention changed upstream: pre-2021 archive-relative, then
+file-relative (internal preamble) from 2021 until 2024-12-05, when the fix
+argued for by the developer landed (Tcl ticket aaa84fbbc5, submitted
+juliannoble2 with patch; fixed in 3d62ee540d by jan.nijtmans; the same change
+retired the build system's `zip -A` step). Drivers therefore split by vintage:
+8.7 alphas and 9.0.0 emit the legacy file-relative convention (measured on the
+reference machine 2026-08-04: prefixed-mkimg probe under tclsh 8.7a6 ->
+offsetstyle=file), 9.0.1+ and 9.1 emit archive-relative (same probe under
+tclsh90 9.0.3 -> offsetstyle=archive), and mkimg-less drivers take the zipcat
+concatenation path (archive-relative by mkzip default). An 8.7-driven bake
+today produces file-relative kits: G-134's advisory BAKE-WARNING fires (by
+design, but signalling "pipeline regression" when the cause is driver vintage),
+and offsetstyle.test's _bake baseline goes permanently red for such users.
+The kits themselves mount and boot on either convention (G-129 measured); the
+material consumer is G-128's punkres stamper, which refuses file-relative
+payloads by default because appending a PE section shifts the overlay.
+
+User intent (stated 2026-08-04): punkshell produces consistent kits regardless
+of the driving tclsh; file-relative kits are by explicit user
+configuration/opt-in, not a default; legacy file-offset mkimg runtimes will
+exist in the wild for many years. Server-driven bakes across a
+platform/architecture/content matrix are in scope, so assembly throughput
+matters beyond the single-bake case.
+
+Write-path performance measured 2026-08-04 (reference machine; whole-tree
+assembly of src/_bake/punk91.exe.vfs and a 3x duplication; all writers
+C-zlib deflate with store-if-larger; output sizes within 1.5%):
+
+| payload                    | mkimg (9.0.3) | punk::zip::mkzip (Tcl) | punkzip build (zig) |
+|----------------------------|---------------|------------------------|---------------------|
+| 132MB / 3,645 files        | 11.4s         | 8.1s                   | 6.7s                |
+| 396MB / 10,935 files       | 35.9s         | 24.8s                  | 20.0s               |
+
+Linear scaling in all three (mkimg's figure includes its unavoidable
+per-member progress printing, also what a real bake emits). No wall short of
+multi-GB payloads. Conclusion: the pure-Tcl floor is viable as the sole path;
+punkzip acceleration is a matrix-scale dividend, not a correctness
+prerequisite. (Scratch benchmark artifacts: temp zipbench dir, regenerable.)
+
+punkzip (v2.3.1) gaps for kit-payload writing, measured the same day:
+`build` carries the input path into entry names (no root-strip option -
+entries must root at the payload tree) and has no exclude-pattern option
+(mkzip's default exclusion set has no equivalent). Non-gaps already present:
+sorted deterministic entries, directory entries (zipfs mounts by trailing
+slash), comment flag, deflate + store-if-larger equivalence.
+
+## Approach
+
+Two workstreams, ordered by user decision 2026-08-04 (upstream first as the
+enabling step):
+
+1. Upstream punkzip (developer's repo c:/repo/jn/zig/punkzip, re-vendored per
+   src/tools/AGENTS.md): add root-strip (entries rooted at a named base, the
+   mkzip -base semantics) and repeatable exclude patterns; verify
+   store-if-larger and determinism unchanged; version bump; zig build test
+   green; tools-tier publication per the punkres route (G-123, achieved - see
+   goals/archive/G-123-thirdparty-runtime-tiers.md) or explicit user
+   deferral.
+2. punkshell: a write-path acceleration seam inside punk::zip::mkzip mirroring
+   G-126's read-path pattern (detect bin/punkzip -> shell out with the
+   equivalent strip/exclude arguments; pure-Tcl path unchanged as the floor,
+   with parity as the contract), so all mkzip consumers accelerate, not just
+   the bake; make.tcl's kit assembly retires the tcl::zipfs::mkimg branch in
+   favour of the single zipcat compose point (runtime prefix concatenated
+   outside the writer, as assemble_zipcat_image already does); per-kit
+   offsetstyle key in mapvfs.toml (exact key name settled at implementation;
+   opt-in kits always take the floor with -offsettype file, which mkzip
+   already supports - punkzip needs no offset-adjustment scope);
+   offsetstyle.test's baseline reads the mapping and exempts declared kits;
+   G-134's pin and advisory posture stay, its warning text accurate again
+   (file means opt-in or genuine regression, never driver vintage); docs in
+   src/AGENTS.md (G-122/G-134 bullets) and bin/AGENTS.md.
+
+## Alternatives considered
+
+- Normalize old-mkimg output to archive-relative post-assembly (conversion) -
+  rejected: keeps two assembly paths and requires the parked in-place
+  conversion machinery (G-128's open follow-on); manages the sensitivity
+  rather than removing it.
+- Probe the driver's mkimg convention once per bake and specialize
+  warnings/test gating - rejected for the same reason: driver-invariant
+  output is the intent, not driver-aware messaging.
+- Retire the mkimg branch WITHOUT the punkzip acceleration layer - viable on
+  the measured numbers (the floor beats mkimg at both measured sizes); the
+  acceleration layer was user-directed for the bake-matrix case and adopted.
+- Acceleration seam in make.tcl (bake-only) rather than inside punk::zip::mkzip -
+  rejected: the mkzip seam accelerates every consumer with one parity
+  contract and mirrors G-126 exactly.
+- Bake CLI flag for the file-relative opt-in - rejected: session-scoped;
+  the per-kit mapvfs declaration is durable configuration (user decision,
+  opt-in shape (a), 2026-08-04).
+
+## Notes
+
+- Related: G-122 (achieved - see goals/archive/G-122-host-target-platform-split.md) -
+  introduced the zipcat concatenation fallback (assemble_zipcat_image) this
+  goal makes the only assembly path.
+- Related: G-124 (achieved - see goals/archive/G-124-punkzip-reader.md) - the
+  punk::zip reader/archive_info classification instrument used by the G-134
+  pin and this goal's acceptance; its bootsupport >= 0.2.0 extraction
+  requirement is long satisfied.
+- Related: G-126 (achieved - see goals/archive/G-126-punkzip-accelerator.md) -
+  the read-path acceleration pattern (detect binary -> fast path, pure-Tcl
+  floor, parity suite) mirrored for writes; the vendored-tool build step and
+  provenance/publication route this goal reuses.
+- Related: G-128 (achieved - see goals/archive/G-128-portable-pe-resource-stamping.md) -
+  the stamper whose default refusal makes archive-relative the load-bearing
+  convention. Its open in-place-conversion follow-on is NOT enacted here
+  (mkzip -offsettype file serves the opt-in) and stays open.
+- Related: G-129 (achieved - see goals/archive/G-129-kitboot-mountpoint-derivation.md) -
+  runtime zipfs requirements; both offset conventions mount on both verified
+  runtime families, which is why driver-vintage kits booted despite being
+  off-convention.
+- Related: G-134 (achieved - see goals/archive/G-134-baked-kits-pinned-archive-relative.md) -
+  the pin this goal makes driver-independent by construction; its
+  deliberate-creation carve-out (user decision 2026-07-27) becomes the
+  per-kit mapvfs opt-in.
+- Related: G-131 (live) - deliberate non-overlap: container autodetection
+  (metakit or zipfs) is orthogonal to zip offset convention; the single-writer
+  change touches only the zipfs kit path and must not disturb G-131's metakit
+  branches or its make.tcl check reconciliation.
+- Related: G-158 (live) - sibling surface: the 8.6 'kit' (metakit/sdx) lane is
+  a different container family, untouched here.
+- Related: G-005/G-006 (live) - binary build/retrieval infrastructure; the
+  bake-server punkzip distribution rides the tools-tier route (punkres
+  precedent), G-105-family cross-builds for non-windows bake servers.
+- Overlap survey 2026-08-04 (drafting): `goals_xref.tcl paths src/make.tcl
+  src/modules/punk src/tools/punkzip src/runtime/mapvfs.toml` - no live goals
+  intersect the first three; mapvfs.toml surfaces path-mention hits (G-131,
+  G-158, G-005, G-006, G-023, G-141), all read and classified orthogonal to
+  offset convention (container type, kit naming, libfetch target selection,
+  8.6 kit parity). `goals_xref.tcl report` unlinked-pair list reviewed: no
+  pair intersects this goal's scope.
+- The offsetstyle.test baseline's premise ("today's bakes produce no warning")
+  was environment-dependent when written (the reference driver has fixed
+  mkimg); this goal makes it driver-independent again - the baseline stays a
+  true tripwire once opt-in kits are exempted by declaration.
+- mkzip's per-member punk::args::parse in Addentry was profiled as a suspected
+  hot spot before the benchmark; measured all-in per-member cost ~2.2ms at
+  3.6k members - not a blowup, left as-is (floor stays simple and correct).
+- The same vintage sensitivity exists one level down in the buildsuites'
+  runtime wrapper (src/buildsuites/suite_tcl90/tools/zipfs_mkimg.tcl): a
+  runtime built with a legacy-mkimg driver comes out file-relative (the
+  store's tclsh90b4_piperepl.exe is exactly such an artifact). Tolerated on
+  input by the G-124 reader; recorded here as hygiene, not in this goal's
+  scope - a candidate follow-on if runtime-side consistency is later wanted.