From 45fd5e7c870eec6763513cf539a6e8db19b02ef6 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Tue, 4 Aug 2026 18:53:40 +1000 Subject: [PATCH] G-165 proposed: driver-invariant kit assembly (single zip writer path, punkzip acceleration, file-relative opt-in) Every make.tcl zip-kit bake is to emit an ARCHIVE-relative kit regardless of the driving interpreter's zipfs vintage (legacy file-relative mkimg in 8.7/9.0.0, fixed mkimg in 9.0.1+, or no mkimg at all): zip-kit assembly moves to a single writer layer - punkzip (zig) when its binary is present, punk::zip::mkzip (pure Tcl) as the always-available floor - with tcl::zipfs::mkimg retired from the pipeline, and file-relative output only via an explicit per-kit mapvfs.toml opt-in. Workstream order (user decision): upstream punkzip root-strip/exclude enabling changes first, then the punkshell seam. The detail file carries the driver-vintage probe evidence (8.7a6=file, 9.0.3=archive), the upstream fix provenance (ticket aaa84fbbc5, core 3d62ee540d 2024-12-05), the write-path benchmark (mkimg 11.4/35.9s vs mkzip 8.1/24.8s vs punkzip 6.7/20.0s at 132MB/396MB), the drafting-time overlap survey and Related lines. Assisted-by: harness=opencode; primary-model=openrouter/moonshotai/kimi-k3; api-location=openrouter.ai --- GOALS.md | 4 + goals/G-165-driver-invariant-kit-assembly.md | 158 +++++++++++++++++++ 2 files changed, 162 insertions(+) create mode 100644 goals/G-165-driver-invariant-kit-assembly.md diff --git a/GOALS.md b/GOALS.md index 1651bcfe..a710aeee 100644 --- a/GOALS.md +++ b/GOALS.md @@ -404,3 +404,7 @@ Detail: goals/G-162-bakelist-kit-freshness.md Scope: src/modules/punk/repl-999999.0a1.0.tm (exit/eof done paths, reader-window blocking-restore discipline); src/modules/punk/console-999999.0a1.0.tm (drain helper, -blocking 0 window sites); src/lib/app-punkshell/punkshell.tcl (final-exit seam); ARCHITECTURE.md (Process exit and shutdown section); src/tests/shell/testsuites/punkexe/ (unix-pty-gated drain pin) Detail: goals/G-163-unix-exit-output-drain.md +### G-165 [proposed] Driver-invariant kit assembly: single zip writer path with punkzip acceleration, file-relative by explicit opt-in only +Scope: src/make.tcl (zip-kit assembly branch); src/modules/punk/zip-999999.0a1.0.tm (mkzip write-path acceleration seam); src/tools/punkzip/ (re-vendor after upstream enabling changes); src/runtime/mapvfs.toml (per-kit offsetstyle opt-in); src/tests/ (accelerated/floor parity + driver-vintage characterization); src/AGENTS.md; bin/AGENTS.md +Detail: goals/G-165-driver-invariant-kit-assembly.md + diff --git a/goals/G-165-driver-invariant-kit-assembly.md b/goals/G-165-driver-invariant-kit-assembly.md new file mode 100644 index 00000000..0de727db --- /dev/null +++ b/goals/G-165-driver-invariant-kit-assembly.md @@ -0,0 +1,158 @@ +# G-165 Driver-invariant kit assembly: single zip writer path with punkzip acceleration, file-relative by explicit opt-in only + +Status: proposed +Scope: src/make.tcl (zip-kit assembly branch); src/modules/punk/zip-999999.0a1.0.tm (mkzip write-path acceleration seam); src/tools/punkzip/ (re-vendor after upstream enabling changes); src/runtime/mapvfs.toml (per-kit offsetstyle opt-in); src/tests/ (accelerated/floor parity + driver-vintage characterization); src/AGENTS.md; bin/AGENTS.md +Goal: every make.tcl zip-kit bake emits an ARCHIVE-relative kit regardless of the driving interpreter (legacy file-relative mkimg in 8.7/9.0.0, fixed mkimg in 9.0.1+, or no mkimg at all) and produces equivalent kit zip payloads for a given merged tree across drivers, platforms and architectures - supporting bake servers running sizeable platform/content matrices: zip-kit assembly uses a single writer layer, punkzip (zig) when its binary is present with punk::zip::mkzip (pure Tcl) as the always-available floor, semantically equivalent, with tcl::zipfs::mkimg retired from the pipeline; file-relative output is available only as an explicit per-kit opt-in recorded in mapvfs.toml - the G-134 deliberate-creation carve-out expressed as configuration, never ambient driver behaviour. +Acceptance: a kit baked from the same source tree under all three driver classes on the reference machine - tclsh 8.7a6 (legacy file-relative mkimg, the reported failure case), tclsh90 9.0.x (fixed mkimg), and a mkimg-less tclsh (the zipcat-only class) - probes offsetstyle=archive via punk::zip::archive_info with identical member sets and member CRCs across the three runs; with bin/punkzip present the bake assembles via punkzip (visible in bake output) and a write-path parity suite proves member-set, CRC and offsetstyle equivalence between the accelerated and pure-Tcl paths over the G-124 fixture set plus a real .vfs payload, while a no-punkzip bake of the same tree produces an equivalent archive; a kit declaring offsetstyle="file" in its mapvfs.toml entry probes file - the opt-in recorded, and the offsetstyle.test baseline exempts declared kits - while G-134's BAKE-WARNING never fires on default (non-opt-in) bakes under any driver class; upstream punkzip gains the root-strip and repeatable-exclude options the bake's call shape requires (zig build test green, entries rooted at the payload tree with mkzip's default exclusion set honoured identically), re-vendored with PROVENANCE.md updated and tools-tier publication recorded or explicitly user-deferred; make.tcl's kit assembly path contains no tcl::zipfs::mkimg call; full punkexe + punkboot utils suites green in both runner modes; and src/AGENTS.md plus bin/AGENTS.md state the single-writer contract (consistent kits regardless of driving interpreter; file-relative by explicit opt-in only). + +## Context + +The zip-kit assembly branch in make.tcl selects its writer on the PRESENCE of +`tcl::zipfs::mkimg` in the driving interpreter (~line 9135), so the kit's zip +offset convention is a property of which tclsh drives the bake, not of the +pipeline. mkimg's convention changed upstream: pre-2021 archive-relative, then +file-relative (internal preamble) from 2021 until 2024-12-05, when the fix +argued for by the developer landed (Tcl ticket aaa84fbbc5, submitted +juliannoble2 with patch; fixed in 3d62ee540d by jan.nijtmans; the same change +retired the build system's `zip -A` step). Drivers therefore split by vintage: +8.7 alphas and 9.0.0 emit the legacy file-relative convention (measured on the +reference machine 2026-08-04: prefixed-mkimg probe under tclsh 8.7a6 -> +offsetstyle=file), 9.0.1+ and 9.1 emit archive-relative (same probe under +tclsh90 9.0.3 -> offsetstyle=archive), and mkimg-less drivers take the zipcat +concatenation path (archive-relative by mkzip default). An 8.7-driven bake +today produces file-relative kits: G-134's advisory BAKE-WARNING fires (by +design, but signalling "pipeline regression" when the cause is driver vintage), +and offsetstyle.test's _bake baseline goes permanently red for such users. +The kits themselves mount and boot on either convention (G-129 measured); the +material consumer is G-128's punkres stamper, which refuses file-relative +payloads by default because appending a PE section shifts the overlay. + +User intent (stated 2026-08-04): punkshell produces consistent kits regardless +of the driving tclsh; file-relative kits are by explicit user +configuration/opt-in, not a default; legacy file-offset mkimg runtimes will +exist in the wild for many years. Server-driven bakes across a +platform/architecture/content matrix are in scope, so assembly throughput +matters beyond the single-bake case. + +Write-path performance measured 2026-08-04 (reference machine; whole-tree +assembly of src/_bake/punk91.exe.vfs and a 3x duplication; all writers +C-zlib deflate with store-if-larger; output sizes within 1.5%): + +| payload | mkimg (9.0.3) | punk::zip::mkzip (Tcl) | punkzip build (zig) | +|----------------------------|---------------|------------------------|---------------------| +| 132MB / 3,645 files | 11.4s | 8.1s | 6.7s | +| 396MB / 10,935 files | 35.9s | 24.8s | 20.0s | + +Linear scaling in all three (mkimg's figure includes its unavoidable +per-member progress printing, also what a real bake emits). No wall short of +multi-GB payloads. Conclusion: the pure-Tcl floor is viable as the sole path; +punkzip acceleration is a matrix-scale dividend, not a correctness +prerequisite. (Scratch benchmark artifacts: temp zipbench dir, regenerable.) + +punkzip (v2.3.1) gaps for kit-payload writing, measured the same day: +`build` carries the input path into entry names (no root-strip option - +entries must root at the payload tree) and has no exclude-pattern option +(mkzip's default exclusion set has no equivalent). Non-gaps already present: +sorted deterministic entries, directory entries (zipfs mounts by trailing +slash), comment flag, deflate + store-if-larger equivalence. + +## Approach + +Two workstreams, ordered by user decision 2026-08-04 (upstream first as the +enabling step): + +1. Upstream punkzip (developer's repo c:/repo/jn/zig/punkzip, re-vendored per + src/tools/AGENTS.md): add root-strip (entries rooted at a named base, the + mkzip -base semantics) and repeatable exclude patterns; verify + store-if-larger and determinism unchanged; version bump; zig build test + green; tools-tier publication per the punkres route (G-123, achieved - see + goals/archive/G-123-thirdparty-runtime-tiers.md) or explicit user + deferral. +2. punkshell: a write-path acceleration seam inside punk::zip::mkzip mirroring + G-126's read-path pattern (detect bin/punkzip -> shell out with the + equivalent strip/exclude arguments; pure-Tcl path unchanged as the floor, + with parity as the contract), so all mkzip consumers accelerate, not just + the bake; make.tcl's kit assembly retires the tcl::zipfs::mkimg branch in + favour of the single zipcat compose point (runtime prefix concatenated + outside the writer, as assemble_zipcat_image already does); per-kit + offsetstyle key in mapvfs.toml (exact key name settled at implementation; + opt-in kits always take the floor with -offsettype file, which mkzip + already supports - punkzip needs no offset-adjustment scope); + offsetstyle.test's baseline reads the mapping and exempts declared kits; + G-134's pin and advisory posture stay, its warning text accurate again + (file means opt-in or genuine regression, never driver vintage); docs in + src/AGENTS.md (G-122/G-134 bullets) and bin/AGENTS.md. + +## Alternatives considered + +- Normalize old-mkimg output to archive-relative post-assembly (conversion) - + rejected: keeps two assembly paths and requires the parked in-place + conversion machinery (G-128's open follow-on); manages the sensitivity + rather than removing it. +- Probe the driver's mkimg convention once per bake and specialize + warnings/test gating - rejected for the same reason: driver-invariant + output is the intent, not driver-aware messaging. +- Retire the mkimg branch WITHOUT the punkzip acceleration layer - viable on + the measured numbers (the floor beats mkimg at both measured sizes); the + acceleration layer was user-directed for the bake-matrix case and adopted. +- Acceleration seam in make.tcl (bake-only) rather than inside punk::zip::mkzip - + rejected: the mkzip seam accelerates every consumer with one parity + contract and mirrors G-126 exactly. +- Bake CLI flag for the file-relative opt-in - rejected: session-scoped; + the per-kit mapvfs declaration is durable configuration (user decision, + opt-in shape (a), 2026-08-04). + +## Notes + +- Related: G-122 (achieved - see goals/archive/G-122-host-target-platform-split.md) - + introduced the zipcat concatenation fallback (assemble_zipcat_image) this + goal makes the only assembly path. +- Related: G-124 (achieved - see goals/archive/G-124-punkzip-reader.md) - the + punk::zip reader/archive_info classification instrument used by the G-134 + pin and this goal's acceptance; its bootsupport >= 0.2.0 extraction + requirement is long satisfied. +- Related: G-126 (achieved - see goals/archive/G-126-punkzip-accelerator.md) - + the read-path acceleration pattern (detect binary -> fast path, pure-Tcl + floor, parity suite) mirrored for writes; the vendored-tool build step and + provenance/publication route this goal reuses. +- Related: G-128 (achieved - see goals/archive/G-128-portable-pe-resource-stamping.md) - + the stamper whose default refusal makes archive-relative the load-bearing + convention. Its open in-place-conversion follow-on is NOT enacted here + (mkzip -offsettype file serves the opt-in) and stays open. +- Related: G-129 (achieved - see goals/archive/G-129-kitboot-mountpoint-derivation.md) - + runtime zipfs requirements; both offset conventions mount on both verified + runtime families, which is why driver-vintage kits booted despite being + off-convention. +- Related: G-134 (achieved - see goals/archive/G-134-baked-kits-pinned-archive-relative.md) - + the pin this goal makes driver-independent by construction; its + deliberate-creation carve-out (user decision 2026-07-27) becomes the + per-kit mapvfs opt-in. +- Related: G-131 (live) - deliberate non-overlap: container autodetection + (metakit or zipfs) is orthogonal to zip offset convention; the single-writer + change touches only the zipfs kit path and must not disturb G-131's metakit + branches or its make.tcl check reconciliation. +- Related: G-158 (live) - sibling surface: the 8.6 'kit' (metakit/sdx) lane is + a different container family, untouched here. +- Related: G-005/G-006 (live) - binary build/retrieval infrastructure; the + bake-server punkzip distribution rides the tools-tier route (punkres + precedent), G-105-family cross-builds for non-windows bake servers. +- Overlap survey 2026-08-04 (drafting): `goals_xref.tcl paths src/make.tcl + src/modules/punk src/tools/punkzip src/runtime/mapvfs.toml` - no live goals + intersect the first three; mapvfs.toml surfaces path-mention hits (G-131, + G-158, G-005, G-006, G-023, G-141), all read and classified orthogonal to + offset convention (container type, kit naming, libfetch target selection, + 8.6 kit parity). `goals_xref.tcl report` unlinked-pair list reviewed: no + pair intersects this goal's scope. +- The offsetstyle.test baseline's premise ("today's bakes produce no warning") + was environment-dependent when written (the reference driver has fixed + mkimg); this goal makes it driver-independent again - the baseline stays a + true tripwire once opt-in kits are exempted by declaration. +- mkzip's per-member punk::args::parse in Addentry was profiled as a suspected + hot spot before the benchmark; measured all-in per-member cost ~2.2ms at + 3.6k members - not a blowup, left as-is (floor stays simple and correct). +- The same vintage sensitivity exists one level down in the buildsuites' + runtime wrapper (src/buildsuites/suite_tcl90/tools/zipfs_mkimg.tcl): a + runtime built with a legacy-mkimg driver comes out file-relative (the + store's tclsh90b4_piperepl.exe is exactly such an artifact). Tolerated on + input by the G-124 reader; recorded here as hygiene, not in this goal's + scope - a candidate follow-on if runtime-side consistency is later wanted.