Browse Source
Every make.tcl zip-kit bake is to emit an ARCHIVE-relative kit regardless of the driving interpreter's zipfs vintage (legacy file-relative mkimg in 8.7/9.0.0, fixed mkimg in 9.0.1+, or no mkimg at all): zip-kit assembly moves to a single writer layer - punkzip (zig) when its binary is present, punk::zip::mkzip (pure Tcl) as the always-available floor - with tcl::zipfs::mkimg retired from the pipeline, and file-relative output only via an explicit per-kit mapvfs.toml opt-in. Workstream order (user decision): upstream punkzip root-strip/exclude enabling changes first, then the punkshell seam. The detail file carries the driver-vintage probe evidence (8.7a6=file, 9.0.3=archive), the upstream fix provenance (ticket aaa84fbbc5, core 3d62ee540d 2024-12-05), the write-path benchmark (mkimg 11.4/35.9s vs mkzip 8.1/24.8s vs punkzip 6.7/20.0s at 132MB/396MB), the drafting-time overlap survey and Related lines. Assisted-by: harness=opencode; primary-model=openrouter/moonshotai/kimi-k3; api-location=openrouter.aimaster
2 changed files with 162 additions and 0 deletions
@ -0,0 +1,158 @@
|
||||
# G-165 Driver-invariant kit assembly: single zip writer path with punkzip acceleration, file-relative by explicit opt-in only |
||||
|
||||
Status: proposed |
||||
Scope: src/make.tcl (zip-kit assembly branch); src/modules/punk/zip-999999.0a1.0.tm (mkzip write-path acceleration seam); src/tools/punkzip/ (re-vendor after upstream enabling changes); src/runtime/mapvfs.toml (per-kit offsetstyle opt-in); src/tests/ (accelerated/floor parity + driver-vintage characterization); src/AGENTS.md; bin/AGENTS.md |
||||
Goal: every make.tcl zip-kit bake emits an ARCHIVE-relative kit regardless of the driving interpreter (legacy file-relative mkimg in 8.7/9.0.0, fixed mkimg in 9.0.1+, or no mkimg at all) and produces equivalent kit zip payloads for a given merged tree across drivers, platforms and architectures - supporting bake servers running sizeable platform/content matrices: zip-kit assembly uses a single writer layer, punkzip (zig) when its binary is present with punk::zip::mkzip (pure Tcl) as the always-available floor, semantically equivalent, with tcl::zipfs::mkimg retired from the pipeline; file-relative output is available only as an explicit per-kit opt-in recorded in mapvfs.toml - the G-134 deliberate-creation carve-out expressed as configuration, never ambient driver behaviour. |
||||
Acceptance: a kit baked from the same source tree under all three driver classes on the reference machine - tclsh 8.7a6 (legacy file-relative mkimg, the reported failure case), tclsh90 9.0.x (fixed mkimg), and a mkimg-less tclsh (the zipcat-only class) - probes offsetstyle=archive via punk::zip::archive_info with identical member sets and member CRCs across the three runs; with bin/punkzip present the bake assembles via punkzip (visible in bake output) and a write-path parity suite proves member-set, CRC and offsetstyle equivalence between the accelerated and pure-Tcl paths over the G-124 fixture set plus a real .vfs payload, while a no-punkzip bake of the same tree produces an equivalent archive; a kit declaring offsetstyle="file" in its mapvfs.toml entry probes file - the opt-in recorded, and the offsetstyle.test baseline exempts declared kits - while G-134's BAKE-WARNING never fires on default (non-opt-in) bakes under any driver class; upstream punkzip gains the root-strip and repeatable-exclude options the bake's call shape requires (zig build test green, entries rooted at the payload tree with mkzip's default exclusion set honoured identically), re-vendored with PROVENANCE.md updated and tools-tier publication recorded or explicitly user-deferred; make.tcl's kit assembly path contains no tcl::zipfs::mkimg call; full punkexe + punkboot utils suites green in both runner modes; and src/AGENTS.md plus bin/AGENTS.md state the single-writer contract (consistent kits regardless of driving interpreter; file-relative by explicit opt-in only). |
||||
|
||||
## Context |
||||
|
||||
The zip-kit assembly branch in make.tcl selects its writer on the PRESENCE of |
||||
`tcl::zipfs::mkimg` in the driving interpreter (~line 9135), so the kit's zip |
||||
offset convention is a property of which tclsh drives the bake, not of the |
||||
pipeline. mkimg's convention changed upstream: pre-2021 archive-relative, then |
||||
file-relative (internal preamble) from 2021 until 2024-12-05, when the fix |
||||
argued for by the developer landed (Tcl ticket aaa84fbbc5, submitted |
||||
juliannoble2 with patch; fixed in 3d62ee540d by jan.nijtmans; the same change |
||||
retired the build system's `zip -A` step). Drivers therefore split by vintage: |
||||
8.7 alphas and 9.0.0 emit the legacy file-relative convention (measured on the |
||||
reference machine 2026-08-04: prefixed-mkimg probe under tclsh 8.7a6 -> |
||||
offsetstyle=file), 9.0.1+ and 9.1 emit archive-relative (same probe under |
||||
tclsh90 9.0.3 -> offsetstyle=archive), and mkimg-less drivers take the zipcat |
||||
concatenation path (archive-relative by mkzip default). An 8.7-driven bake |
||||
today produces file-relative kits: G-134's advisory BAKE-WARNING fires (by |
||||
design, but signalling "pipeline regression" when the cause is driver vintage), |
||||
and offsetstyle.test's _bake baseline goes permanently red for such users. |
||||
The kits themselves mount and boot on either convention (G-129 measured); the |
||||
material consumer is G-128's punkres stamper, which refuses file-relative |
||||
payloads by default because appending a PE section shifts the overlay. |
||||
|
||||
User intent (stated 2026-08-04): punkshell produces consistent kits regardless |
||||
of the driving tclsh; file-relative kits are by explicit user |
||||
configuration/opt-in, not a default; legacy file-offset mkimg runtimes will |
||||
exist in the wild for many years. Server-driven bakes across a |
||||
platform/architecture/content matrix are in scope, so assembly throughput |
||||
matters beyond the single-bake case. |
||||
|
||||
Write-path performance measured 2026-08-04 (reference machine; whole-tree |
||||
assembly of src/_bake/punk91.exe.vfs and a 3x duplication; all writers |
||||
C-zlib deflate with store-if-larger; output sizes within 1.5%): |
||||
|
||||
| payload | mkimg (9.0.3) | punk::zip::mkzip (Tcl) | punkzip build (zig) | |
||||
|----------------------------|---------------|------------------------|---------------------| |
||||
| 132MB / 3,645 files | 11.4s | 8.1s | 6.7s | |
||||
| 396MB / 10,935 files | 35.9s | 24.8s | 20.0s | |
||||
|
||||
Linear scaling in all three (mkimg's figure includes its unavoidable |
||||
per-member progress printing, also what a real bake emits). No wall short of |
||||
multi-GB payloads. Conclusion: the pure-Tcl floor is viable as the sole path; |
||||
punkzip acceleration is a matrix-scale dividend, not a correctness |
||||
prerequisite. (Scratch benchmark artifacts: temp zipbench dir, regenerable.) |
||||
|
||||
punkzip (v2.3.1) gaps for kit-payload writing, measured the same day: |
||||
`build` carries the input path into entry names (no root-strip option - |
||||
entries must root at the payload tree) and has no exclude-pattern option |
||||
(mkzip's default exclusion set has no equivalent). Non-gaps already present: |
||||
sorted deterministic entries, directory entries (zipfs mounts by trailing |
||||
slash), comment flag, deflate + store-if-larger equivalence. |
||||
|
||||
## Approach |
||||
|
||||
Two workstreams, ordered by user decision 2026-08-04 (upstream first as the |
||||
enabling step): |
||||
|
||||
1. Upstream punkzip (developer's repo c:/repo/jn/zig/punkzip, re-vendored per |
||||
src/tools/AGENTS.md): add root-strip (entries rooted at a named base, the |
||||
mkzip -base semantics) and repeatable exclude patterns; verify |
||||
store-if-larger and determinism unchanged; version bump; zig build test |
||||
green; tools-tier publication per the punkres route (G-123, achieved - see |
||||
goals/archive/G-123-thirdparty-runtime-tiers.md) or explicit user |
||||
deferral. |
||||
2. punkshell: a write-path acceleration seam inside punk::zip::mkzip mirroring |
||||
G-126's read-path pattern (detect bin/punkzip -> shell out with the |
||||
equivalent strip/exclude arguments; pure-Tcl path unchanged as the floor, |
||||
with parity as the contract), so all mkzip consumers accelerate, not just |
||||
the bake; make.tcl's kit assembly retires the tcl::zipfs::mkimg branch in |
||||
favour of the single zipcat compose point (runtime prefix concatenated |
||||
outside the writer, as assemble_zipcat_image already does); per-kit |
||||
offsetstyle key in mapvfs.toml (exact key name settled at implementation; |
||||
opt-in kits always take the floor with -offsettype file, which mkzip |
||||
already supports - punkzip needs no offset-adjustment scope); |
||||
offsetstyle.test's baseline reads the mapping and exempts declared kits; |
||||
G-134's pin and advisory posture stay, its warning text accurate again |
||||
(file means opt-in or genuine regression, never driver vintage); docs in |
||||
src/AGENTS.md (G-122/G-134 bullets) and bin/AGENTS.md. |
||||
|
||||
## Alternatives considered |
||||
|
||||
- Normalize old-mkimg output to archive-relative post-assembly (conversion) - |
||||
rejected: keeps two assembly paths and requires the parked in-place |
||||
conversion machinery (G-128's open follow-on); manages the sensitivity |
||||
rather than removing it. |
||||
- Probe the driver's mkimg convention once per bake and specialize |
||||
warnings/test gating - rejected for the same reason: driver-invariant |
||||
output is the intent, not driver-aware messaging. |
||||
- Retire the mkimg branch WITHOUT the punkzip acceleration layer - viable on |
||||
the measured numbers (the floor beats mkimg at both measured sizes); the |
||||
acceleration layer was user-directed for the bake-matrix case and adopted. |
||||
- Acceleration seam in make.tcl (bake-only) rather than inside punk::zip::mkzip - |
||||
rejected: the mkzip seam accelerates every consumer with one parity |
||||
contract and mirrors G-126 exactly. |
||||
- Bake CLI flag for the file-relative opt-in - rejected: session-scoped; |
||||
the per-kit mapvfs declaration is durable configuration (user decision, |
||||
opt-in shape (a), 2026-08-04). |
||||
|
||||
## Notes |
||||
|
||||
- Related: G-122 (achieved - see goals/archive/G-122-host-target-platform-split.md) - |
||||
introduced the zipcat concatenation fallback (assemble_zipcat_image) this |
||||
goal makes the only assembly path. |
||||
- Related: G-124 (achieved - see goals/archive/G-124-punkzip-reader.md) - the |
||||
punk::zip reader/archive_info classification instrument used by the G-134 |
||||
pin and this goal's acceptance; its bootsupport >= 0.2.0 extraction |
||||
requirement is long satisfied. |
||||
- Related: G-126 (achieved - see goals/archive/G-126-punkzip-accelerator.md) - |
||||
the read-path acceleration pattern (detect binary -> fast path, pure-Tcl |
||||
floor, parity suite) mirrored for writes; the vendored-tool build step and |
||||
provenance/publication route this goal reuses. |
||||
- Related: G-128 (achieved - see goals/archive/G-128-portable-pe-resource-stamping.md) - |
||||
the stamper whose default refusal makes archive-relative the load-bearing |
||||
convention. Its open in-place-conversion follow-on is NOT enacted here |
||||
(mkzip -offsettype file serves the opt-in) and stays open. |
||||
- Related: G-129 (achieved - see goals/archive/G-129-kitboot-mountpoint-derivation.md) - |
||||
runtime zipfs requirements; both offset conventions mount on both verified |
||||
runtime families, which is why driver-vintage kits booted despite being |
||||
off-convention. |
||||
- Related: G-134 (achieved - see goals/archive/G-134-baked-kits-pinned-archive-relative.md) - |
||||
the pin this goal makes driver-independent by construction; its |
||||
deliberate-creation carve-out (user decision 2026-07-27) becomes the |
||||
per-kit mapvfs opt-in. |
||||
- Related: G-131 (live) - deliberate non-overlap: container autodetection |
||||
(metakit or zipfs) is orthogonal to zip offset convention; the single-writer |
||||
change touches only the zipfs kit path and must not disturb G-131's metakit |
||||
branches or its make.tcl check reconciliation. |
||||
- Related: G-158 (live) - sibling surface: the 8.6 'kit' (metakit/sdx) lane is |
||||
a different container family, untouched here. |
||||
- Related: G-005/G-006 (live) - binary build/retrieval infrastructure; the |
||||
bake-server punkzip distribution rides the tools-tier route (punkres |
||||
precedent), G-105-family cross-builds for non-windows bake servers. |
||||
- Overlap survey 2026-08-04 (drafting): `goals_xref.tcl paths src/make.tcl |
||||
src/modules/punk src/tools/punkzip src/runtime/mapvfs.toml` - no live goals |
||||
intersect the first three; mapvfs.toml surfaces path-mention hits (G-131, |
||||
G-158, G-005, G-006, G-023, G-141), all read and classified orthogonal to |
||||
offset convention (container type, kit naming, libfetch target selection, |
||||
8.6 kit parity). `goals_xref.tcl report` unlinked-pair list reviewed: no |
||||
pair intersects this goal's scope. |
||||
- The offsetstyle.test baseline's premise ("today's bakes produce no warning") |
||||
was environment-dependent when written (the reference driver has fixed |
||||
mkimg); this goal makes it driver-independent again - the baseline stays a |
||||
true tripwire once opt-in kits are exempted by declaration. |
||||
- mkzip's per-member punk::args::parse in Addentry was profiled as a suspected |
||||
hot spot before the benchmark; measured all-in per-member cost ~2.2ms at |
||||
3.6k members - not a blowup, left as-is (floor stays simple and correct). |
||||
- The same vintage sensitivity exists one level down in the buildsuites' |
||||
runtime wrapper (src/buildsuites/suite_tcl90/tools/zipfs_mkimg.tcl): a |
||||
runtime built with a legacy-mkimg driver comes out file-relative (the |
||||
store's tclsh90b4_piperepl.exe is exactly such an artifact). Tolerated on |
||||
input by the G-124 reader; recorded here as hygiene, not in this goal's |
||||
scope - a candidate follow-on if runtime-side consistency is later wanted. |
||||
Loading…
Reference in new issue