From e1711a7b2aa248ace5d06e12fd26d353b3fb74ad Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Sun, 26 Jul 2026 16:35:34 +1000 Subject: [PATCH] G-124/G-125/G-126 drafted (user-approved): punk::zip reader, unbootable-kit deploy gate, vendored punkzip accelerator Three goals arising from the G-122 work, each drafted with its overlap survey and approved before writing. G-124 - dependency-free zip reading in punk::zip. Nothing in the tree can READ a zip without zipfs, vfs::zip or tcllib's zipfile::decode, which is a system package punkshell does not vendor and which has exactly one consumer: make.tcl's zipfs-less kit extraction. The goal removes that requirement outright (not as an optional fallback), adds an introspection surface so punk::zip is useful to script programmers, and gives the writer its first round-trip coverage. Measured motivation recorded in the detail file: msys2's tclsh8.6 has no tcllib and so bakes an unbootable kit, and extract_preamble's live '#todo - need to adjust offsets' means a FILE-relative prefixed executable (tclsh90b4_piperepl.exe) splits into a .zip tcllib rejects with "Bad closure" - reading the original file at its derived base offset removes that failure mode by construction. G-125 - a kit that cannot boot is not deployed. Today a failed extraction emits a recapped BUILD-WARNING and then deletes and replaces the deployed kit anyway; field- observed 2026-07-26 replacing a working punk91.exe with one that dies on init.tcl. The gate tests the boot precondition on the assembled vfs, so a kit whose payload legitimately supplies its own tcl library still builds. G-126 - punkzip as a VENDORED zip accelerator behind G-124's pure-Tcl floor, with zig kept optional (a tclsh-only clean checkout must still build and bake). Feasibility measured and recorded: the source builds clean under its declared zig 0.15.2, and under 0.16 fails on one surfaced error (std.process.argsAlloc), with the OS-facing surface confined to 3 of ~20 files while the ~10k-line compression core is untouched; zig 0.16.0 is already unpacked in bin/tools. Benchmarks captured for the speedup claim. Non-contract Notes back-pointers added to G-034 (the reader is a candidate for its own Acceptance's "alternative modpod mount path" branch) and G-101 (an 8.6 container has no zipfs, so it needs a reader that works without one). Assisted-by: harness=claude; primary-model=claude-opus-5[1m]; api-location=anthropic.com --- GOALS.md | 12 ++ goals/G-034-modpod-codeinterp-tcl86.md | 7 ++ goals/G-101-tcl86-kit-container-strategy.md | 4 + goals/G-124-punkzip-reader.md | 106 ++++++++++++++++ goals/G-125-unbootable-kit-deploy-gate.md | 76 ++++++++++++ goals/G-126-punkzip-accelerator.md | 127 ++++++++++++++++++++ 6 files changed, 332 insertions(+) create mode 100644 goals/G-124-punkzip-reader.md create mode 100644 goals/G-125-unbootable-kit-deploy-gate.md create mode 100644 goals/G-126-punkzip-accelerator.md diff --git a/GOALS.md b/GOALS.md index d0b9719f..e33bb3e8 100644 --- a/GOALS.md +++ b/GOALS.md @@ -389,3 +389,15 @@ Detail: goals/G-116-suite-built-tcltls.md Scope: src/scriptapps/punk-runtime.ps1 + src/scriptapps/bin/punk-runtime.bash + punk-runtime_wrap.toml (named-tier operations in both payloads; bin/punk-runtime.cmd via the established re-wrap channel); punkbin artifact repo (external, c:/repo/jn/punkbin - per-tier layout, defaults, sha1sums, schema-v2 provenance sidecars incl. retroactive backfill, layout docs); src/buildsuites/suite_tcl90/ (family_artifacts.tcl schema-v2 emission for suite-built artifacts); alternative-server override + trusted-server set (PUNKBIN_URL pattern; consent surface coordinated with G-006); src/tests/shell/testsuites/binscripts/ (fixture-server characterization); bin/runtime// stores (as materialization targets) Detail: goals/G-123-thirdparty-runtime-tiers.md + +### G-124 [proposed] Dependency-free zip reading in punk::zip: archives, runtime-prefixed executables, and a scripter-facing introspection surface +Scope: src/modules/punk/zip-999999.0a1.0.tm (reader + introspection procs, base-offset derivation factored out of extract_preamble for shared use, buildversion); src/make.tcl (zipfs-less kit extraction path as first consumer - the zipfile::decode requirement is removed, not made optional); src/tests/modules/punk/zip/ (new testsuite - mkzip round-trip, prefixed-archive cases, introspection, unsupported-input behaviour); bootsupport + src/vfs/_vfscommon.vfs copies via established sync channels +Detail: goals/G-124-punkzip-reader.md + +### G-125 [proposed] A kit that cannot boot is not deployed: extraction failure gates the bake instead of warning past it +Scope: src/make.tcl (kit extraction outcome handling, deploy step gating, the no-extraction BUILD-WARNING as the current behaviour being replaced); src/tests/shell/testsuites/punkexe/ (characterization of the gate); src/AGENTS.md (bake failure-mode documentation) +Detail: goals/G-125-unbootable-kit-deploy-gate.md + +### G-126 [proposed] punkzip as a vendored zip accelerator: zig 0.16 port, reproducible build into bin/, punk::zip fast path +Scope: src/tools/punkzip/ (vendored punkzip source tree - root name settled in the work - with provenance and licence records); src/make.tcl (tool build step); bin/tools/zig* (pinned toolchain as consumed); bin/punkzip.exe (untracked build output); src/modules/punk/zip-999999.0a1.0.tm (accelerator detection + fast path behind the G-124 pure-Tcl floor); src/tests/modules/punk/zip/ (parity suite - accelerated and pure-Tcl paths must agree); upstream maintained checkout c:/repo/jn/zig/punkzip (re-vendor source, read-only from this repo) +Detail: goals/G-126-punkzip-accelerator.md diff --git a/goals/G-034-modpod-codeinterp-tcl86.md b/goals/G-034-modpod-codeinterp-tcl86.md index 27f66d90..3c7e43fd 100644 --- a/goals/G-034-modpod-codeinterp-tcl86.md +++ b/goals/G-034-modpod-codeinterp-tcl86.md @@ -4,3 +4,10 @@ Status: proposed Scope: src/modules/punk/mix/ (modpod mount path); vfs::zip availability in the repl code interp; src/modules/punk/cap/ (templates capability handler) Goal: zip-based `#modpod` modules (e.g. `punk::mix::templates`) mount and their punk::cap handlers register in the `shell` subcommand's code interp on Tcl 8.6, matching the `script`/main-interp context - so `dev module.templates` and other `punk.templates`-capability consumers work in an interactive 8.6 shell instead of failing with `invalid command name vfs::RegisterMount`. Acceptance: `dev module.templates` in an interactive 8.6 punk shell (`shell` subcommand) lists the template providers with no `failed to load ZIP archive-based module` / `invalid command name vfs::RegisterMount` / `Unable to register any template providers` / `invalid command name ::punk::cap::handlers::templates::api_punk.templates` errors, matching the `script`-subcommand output on the same binary (verified 2026-07-07: script works, shell fails); root cause fixed (the code interp lacks the vfs::zip library that provides vfs::RegisterMount for pre-zipfs Tcl, present in the main interp - restore it to the code interp or use an alternative modpod mount path there); Tcl 9 (built-in zipfs) behaviour is unchanged. + +## Notes + +- Related: G-124 - a dependency-free zip READER in punk::zip (no zipfs, no vfs::zip, no + tcllib) is a concrete candidate for the "or use an alternative modpod mount path there" + branch of this goal's Acceptance: extract the modpod rather than mount it. Recorded + 2026-07-26 at G-124's drafting; it does not narrow this goal's options. diff --git a/goals/G-101-tcl86-kit-container-strategy.md b/goals/G-101-tcl86-kit-container-strategy.md index 1042615c..b3455acf 100644 --- a/goals/G-101-tcl86-kit-container-strategy.md +++ b/goals/G-101-tcl86-kit-container-strategy.md @@ -45,6 +45,10 @@ unblocked; activation is the user's call. ## Notes +- Related: G-124 - an 8.6 container has no zipfs by definition, so whichever container + this goal picks needs a reader that works without it; that goal supplies the + dependency-free pure-Tcl one (and G-126 an optional accelerator). Recorded 2026-07-26 + at G-124's drafting. - Related: G-116 (suite-built tcltls with a zig-built crypto backend) - the payload question is shared: whatever container this goal picks carries the battery set, and G-116's bi-family payload contract is generation-shaped, not diff --git a/goals/G-124-punkzip-reader.md b/goals/G-124-punkzip-reader.md new file mode 100644 index 00000000..6c3041a4 --- /dev/null +++ b/goals/G-124-punkzip-reader.md @@ -0,0 +1,106 @@ +# G-124 Dependency-free zip reading in punk::zip + +Status: proposed +Scope: src/modules/punk/zip-999999.0a1.0.tm (reader + introspection procs, base-offset derivation factored out of extract_preamble for shared use, buildversion); src/make.tcl (zipfs-less kit extraction path as first consumer - the zipfile::decode requirement is removed, not made optional); src/tests/modules/punk/zip/ (new testsuite - mkzip round-trip, prefixed-archive cases, introspection, unsupported-input behaviour); bootsupport + src/vfs/_vfscommon.vfs copies via established sync channels +Goal: punkshell can read a zip archive - including one prefixed by an executable, with either archive-relative or file-relative offsets - using only stock Tcl, and can do so as a documented utility a Tcl script programmer would reach for: list what is inside without extracting, extract all or part of it, on any tclsh with no zipfs, no vfs::zip and no tcllib. make.tcl's zipfs-less kit extraction stops depending on tcllib's zipfile::decode (absent from e.g msys2's /usr/bin/tclsh8.6) and stops depending on a split intermediate whose offsets can be wrong, and punk::zip's writer gains the round-trip verification it has never had. +Acceptance: punk::zip extracts to a target directory from (a) a plain zip, (b) an executable-prefixed archive with archive-relative offsets and (c) one with file-relative offsets - bin/runtime/win32-x86_64/tclsh90b4_piperepl.exe is the recorded (c) case, on which the current split-then-tcllib path fails with "Bad zip file. Bad closure." - producing CRC-verified byte-identical members with directory entries materialized as directories, on a tclsh with no zipfs, no vfs::zip and no tcllib reachable (the interpreter used is recorded here); an introspection call lists members without extracting, returning per-entry name, size, compressed size, method, mtime, crc and directory-vs-file, over the same three input shapes, and both surfaces carry punk::args definitions with worked examples so 'i punk::zip::' documents them; unsupported inputs (zip64, encrypted, unknown compression method) fail naming the reason rather than emitting partial or garbled output; make.tcl's zipfs-less kit extraction consumes the new surface and NO zipfile::decode requirement remains anywhere in the shipped tree, verified by a tree grep, and demonstrated by an 8.6 zipfs-less bake of a zip kit on a tcllib-less tclsh (msys2 /usr/bin/tclsh8.6 as reference host) producing an artifact that boots with its tcl_library present; a new src/tests/modules/punk/zip/ suite covers a mkzip -> read round trip asserting names, content, directory-vs-file classification and stored attributes - recording the answer to the standing zipfs dir-misidentification question noted in punk::mix::cli and src/vfs/mkzipfix.vfs - plus the three prefixed-archive cases, the introspection output, and the unsupported-input failures; punk::zip buildversion minor-bumped with its changelog line. + +## Context + +punk::zip is a write-plus-split module: `walk`, `mkzip`/`Addentry`, `extract_preamble`. +Nothing in the tree can READ a zip without zipfs (Tcl 8.7+/9), vfs::zip, or tcllib's +`zipfile::decode`. That last one is a SYSTEM package punkshell does not vendor, and it +has exactly one consumer: make.tcl's zipfs-less kit-extraction fallback (shipped under +G-122, achieved - see goals/archive/G-122-host-target-platform-split.md). + +Measured 2026-07-26: msys2's `/usr/bin/tclsh8.6` has no tcllib, so a bake driven from +that host cannot lift the runtime's own zip (its tcl_library) into the kit and the +artifact does not boot. The same tclsh, given a tcllib on its path, splits and decodes +byte-identically to a native 8.6 host (baseoffset 5267968, 1429659 zip bytes, 1784 +entries from both) - so the gap is purely the absent package, not an msys defect. + +The dependency is not the only defect. `extract_preamble` carries +`#todo - if it was internal preamble - need to adjust offsets to fix the split off +zipfile`. That todo is live: an archive whose offsets are FILE-relative (the historical +`zipfs mkimg` convention) splits into a .zip whose offsets still assume the removed +preamble. `bin/runtime/win32-x86_64/tclsh90b4_piperepl.exe` is such a file +(filerel_cdirstart == recorded_offset == 4085188); the current pipeline splits it +correctly and tcllib then fails on the result with "Bad zip file. Bad closure.". A reader +that works on the ORIGINAL file at the derived base offset never produces the broken +intermediate - the failure mode disappears rather than needing offset-rewriting code. +(Survey of the current store: every other runtime and every built kit is +archive-relative, so this is a real but presently-unexercised shape.) + +Reading has value well beyond the build. punk::zip is in the bootsupport manifest and +ships in every kit (`_vfscommon.vfs/modules/punk/zip-*.tm`), so a reader there is +available to any script the shell runs - and punkshell today has no answer at all for +"what is inside this .tm modpod / kit / zip" on a runtime without zipfs. + +## Approach + +- Factor the base-offset derivation out of `extract_preamble` into a shared private + helper so the reader accepts a plain zip or a prefixed executable through one code + path, and `extract_preamble` stops being the only thing that knows how to find the + archive within a file. +- Walk ALL central-directory records. `extract_preamble` binary-scans only the first, + with its own `#todo! loop through all cdr file headers` note. Sizes and offsets taken + from the CD sidestep data descriptors entirely. +- Inflate with the stock primitives the writer already uses: `zlib inflate` / + `zlib stream inflate` mirror `Addentry`'s `zlib deflate` / `zlib stream deflate`, and + `zlib crc32` gives verification tcllib never performed (its `CopyFile` carries + `# FUTURE: Run crc checksum on created file`). Mirror `Addentry`'s existing 2MB + threshold for choosing whole-member vs streaming. +- Support store and deflate; fail by name on zip64, encryption (GPBF bit 0) and unknown + methods rather than emitting partial output. +- The introspection surface is the same CD walk without extraction, so listing costs no + decompression - which is what makes it usable as a general scripting utility. + +## Alternatives considered + +- Vendor tcllib's `zipfile::decode` (plus its `fileutil::decode` dependency) through + G-065 - rejected: it imports a byte-oriented decode framework to obtain one capability, + does not verify CRCs, cannot read a prefixed executable at all (the caller must split + first, which is exactly where the file-relative bug lives), and leaves punk::zip's + writer with no round-trip coverage. Cost for comparison, measured 2026-07-26: 830ms to + extract 892 files that punkzip does in 490ms. +- Require zipfs, i.e. declare 8.6-without-tcllib unsupported - rejected: 8.6 is a real + target (src/AGENTS.md), G-101's 8.6 container work has no zipfs by definition, and + G-034 is an existing 8.6 read failure in the shell itself. +- Shell out to an external zip tool as the ONLY path - rejected as a floor, since it + would make a build tool a hard dependency of module-level functionality. Adopted as an + optional accelerator instead - see G-126. + +## Notes + +- Related: G-034 - its Acceptance permits "an alternative modpod mount path" for the 8.6 + code interp, which currently fails for want of `vfs::RegisterMount`. A dependency-free + reader is that alternative: extract rather than mount. +- Related: G-101 - an 8.6 kit container has no zipfs by definition, so whatever container + is chosen needs a reader that works without it. This goal is that primitive. +- Related: G-109 - manifest-declared discovery reads members out of zip-based .tm + modules; the introspection surface here is the natural mechanism. +- Related: G-111 - modpod IS the zip .tm format and that goal establishes its first test + baseline. The round-trip suite here is adjacent coverage of the writer, not a + substitute for modpod's own. +- Related: G-065 - declarative vendoring is the mechanism the rejected alternative above + would have used. +- Related: G-066 - `lib.copyasmodule` emits zip .tm modules; its converter testsuite + could verify output with this reader. +- Related: G-110 - adjacent, not a dependency: it studies caching of EXTRACTED shared + libs, where extraction today is done by zipfs/vfs::zip. +- Related: G-126 - the optional zig accelerator that sits behind this pure-Tcl floor. +- Related: G-125 - the deploy gate that turns an extraction failure into a failed kit + rather than a deployed-but-unbootable one. Independent of this goal, which removes the + most common cause of that failure. +- Related: G-122 (achieved) - shipped the zipfs-less assembly half and recorded the + tcllib gap that motivates this goal; see goals/archive/G-122-host-target-platform-split.md. +- Test baseline as at drafting: punk::zip has NO tests (`grep -rn "punk::zip" src/tests/` + finds only log noise). The `zipper` modpod - a separate low-level writer - has four + write-only smoke tests in `src/tests/modules/zipper/testsuites/zipper/zipper.test` + (version, initialize, addentry, finalize-produces->100-bytes) with no content + assertions. So nothing today verifies that anything punkshell WRITES is readable. +- Standing question the round-trip suite should settle: `punk::mix::cli` records that + "punk::zip::mkzip stores permissions - (unix style) - which zipfs mkzip doesn't" and + `src/vfs/mkzipfix.vfs`'s copy records that this "confuses zipfs when reading - it + misidentifies dirs as files". The reader makes that assertable in-tree for the first + time; record the finding here. diff --git a/goals/G-125-unbootable-kit-deploy-gate.md b/goals/G-125-unbootable-kit-deploy-gate.md new file mode 100644 index 00000000..3e1ca0e5 --- /dev/null +++ b/goals/G-125-unbootable-kit-deploy-gate.md @@ -0,0 +1,76 @@ +# G-125 A kit that cannot boot is not deployed + +Status: proposed +Scope: src/make.tcl (kit extraction outcome handling, deploy step gating, the no-extraction BUILD-WARNING as the current behaviour being replaced); src/tests/shell/testsuites/punkexe/ (characterization of the gate); src/AGENTS.md (bake failure-mode documentation) +Goal: a bake that could not lift a runtime's own payload (its tcl_library) into the kit fails that kit instead of building and deploying an artifact that cannot boot - so the worst outcome of a missing extraction capability is a red kit, never a silently replaced working executable. +Acceptance: when extraction from the source runtime yields nothing and the kit's .vfs does not itself supply a bootable tcl library, the kit is recorded in FAILED KITS with a reason naming the cause and NEITHER src/_build nor bin receives a new artifact - the previously deployed kit is left untouched, verified by mtime and byte comparison across such a run; a kit whose .vfs legitimately supplies its own tcl library still builds and deploys, so the gate tests for the boot precondition rather than for the extraction step having run; the check is cheap enough to run on every kit and executes no artifact; the condition and its remedy are documented in src/AGENTS.md; existing punkexe suites pass unchanged and the gate itself is characterized. + +## Context + +The kit machinery already knows when it failed to extract anything from a source runtime. +Under G-122 (achieved - see goals/archive/G-122-host-target-platform-split.md) that +notice was promoted from an inline stderr block to a recapped `BUILD-WARNING:` naming the +kit and what was tried - but the build still proceeds, assembles an artifact, DELETES the +deployed kit and copies the new one over it. The pre-deploy process sweep will even close +running instances of the kit first. + +Field-observed 2026-07-26: an msys2-hosted bake of `punk91` produced a 49,501,792-byte +`bin/punk91.exe` that failed at startup with + + application-specific initialization failed: Cannot find a usable init.tcl + +because the host's tclsh had no tcllib and therefore could not extract the runtime's own +zip. The previously working kit had already been replaced by then. G-124 removes the +particular cause, but not the class: a runtime with no attached payload at all, an +extraction that fails for any other reason, or a future container type reaches the same +state, and the build's response is still "warn and deploy". + +The existing precedent in the same code path is `vfs_startup_script_warning`, which warns +about a missing `main.tcl` and continues - deliberately, because a kit without a startup +script is legal. A kit without a resolvable tcl library is not in the same category: it +cannot initialise at all. + +## Approach + +- Gate on the BOOT PRECONDITION, not on the extraction step having run. Some .vfs folders + legitimately supply their own tcl library (the merged `targetvfs` is what matters), so + the check inspects the assembled tree rather than the extraction outcome - a runtime + that needed no extraction must keep building. +- Fail the kit the same way every other failure in that loop does (append to + `failed_kits` with a reason, end the punkcheck event FAILED) BEFORE the artifact is + renamed into place, so neither `src/_build/` nor `bin/` is touched and the + punkcheck records stay consistent with what is on disk. +- Keep it cheap and non-executing: a structural check of the merged vfs, no artifact + launch, so it can run for every kit on every bake. +- The recapped BUILD-WARNING wording changes from "will not initialise unless the vfs + supplies one" to a failure reason, since the gate now enforces what the warning + described. + +## Alternatives considered + +- Keep warning and deploying, on the grounds that the developer sees the recap - rejected: + the recap arrives after the deployed kit has already been replaced, and the failure + surfaces later as an unexplained broken shell. A build tool should not make a working + executable worse. +- Boot the freshly built artifact as the check - rejected: it cannot work for + cross-target kits (G-122 makes those routine), costs a process launch per kit, and a + kit that boots into an interactive shell is awkward to probe safely. +- Emit to `src/_build` but refuse only the deploy - rejected as a half-measure that + leaves a punkcheck record claiming a good build product; a later run would consider it + current. + +## Notes + +- Related: G-124 - removes the most common cause (no zip reader on a tcllib-less tclsh). + This gate is independent of it and remains valuable afterwards. +- Related: G-122 (achieved) - promoted the no-extraction notice to a recapped + BUILD-WARNING and recorded the field incident that motivates this goal; see + goals/archive/G-122-host-target-platform-split.md. +- Related: G-101 - a new 8.6 container type is exactly the situation where an extraction + path can silently produce nothing; the gate is what keeps that honest. +- Related: G-028 - sibling failure-reporting surface in the same deploy step (that goal + names the process holding a target the build cannot replace; this one refuses to + replace a target with something unbootable). +- Overlap survey also considered G-057 (icon embedding rides the same wrap steps) and + G-115 (declarative .vfs composition decides what the vfs supplies) - neither is a + dependency: this gate reads whatever the merged tree contains, however it was composed. diff --git a/goals/G-126-punkzip-accelerator.md b/goals/G-126-punkzip-accelerator.md new file mode 100644 index 00000000..94067e3e --- /dev/null +++ b/goals/G-126-punkzip-accelerator.md @@ -0,0 +1,127 @@ +# G-126 punkzip as a vendored zip accelerator: zig 0.16 port, reproducible build into bin/, punk::zip fast path + +Status: proposed +Scope: src/tools/punkzip/ (vendored punkzip source tree - root name settled in the work - with provenance and licence records); src/make.tcl (tool build step); bin/tools/zig* (pinned toolchain as consumed); bin/punkzip.exe (untracked build output); src/modules/punk/zip-999999.0a1.0.tm (accelerator detection + fast path behind the G-124 pure-Tcl floor); src/tests/modules/punk/zip/ (parity suite - accelerated and pure-Tcl paths must agree); upstream maintained checkout c:/repo/jn/zig/punkzip (re-vendor source, read-only from this repo) +Goal: punkshell carries punkzip as vendored source, builds it from the repo's pinned zig toolchain, and uses the resulting binary as an optional accelerator for zip reading - with the pure-Tcl path as the always-available floor - so a machine with zig gets fast archive listing and extraction (materially so for bakes with large library payloads) while a clean checkout with only tclsh still builds and bakes exactly as before. +Acceptance: the vendored source builds under the repo's pinned zig 0.16 toolchain with 'zig build test' green (the port's own 10 test modules), driven by a make.tcl step that produces bin/punkzip.exe reproducibly on a clean checkout - and zig stays OPTIONAL: a tclsh-only clean checkout completes packages + bake unchanged, with the step's absence reported rather than fatal; punkzip gains an explicit extract-to-directory argument and a machine-readable listing mode whose format is pinned by a test (the current fixed-width table stays the human default); punk::zip detects the accelerator and uses it, with a parity suite proving accelerated and pure-Tcl paths return identical listings and produce byte-identical extraction trees over the G-124 fixture set including the file-relative prefixed executable, and proving the pure-Tcl path is still taken when the binary is absent; a recorded benchmark on a representative kit payload states the speedup and the entry count at which it becomes material; the vendored tree records its upstream origin, the commit or state it was taken from, the re-vendor procedure, and its licensing (public-domain hwzip lineage plus MIT musl time.zig) per G-063 and the G-026 provenance direction. + +## Context + +`punkzip` is a fork of hwzip.zig (Hadrien Dorio's zig port of Hans Wennborg's hwzip 2.1), +maintained by the developer at `c:/repo/jn/zig/punkzip` and adapted for the case standard +zip tools handle inconsistently: an executable or script concatenated with a zip whose +offsets were never adjusted. Its README records the motivation - 7z and peazip can read +and even edit catenated exezips with unadjusted offsets, but not when offsets have been +rewritten file-relative, and behaviour varies with the file extension used. That is +precisely the axis punkshell cares about (see G-124's Context and the archived G-122). + +Measured 2026-07-26 with the existing `bin/punkzip.exe` (v2.1.0, Nov 2024, untracked - +`bin/*.exe` is gitignored - and already stale against the 2.1.1 source): + +- lists `bin/runtime/win32-x86_64/tclsh90b4_piperepl.exe`, the FILE-relative prefixed + executable on which tcllib's decoder fails with "Bad zip file. Bad closure." - 846 + entries, no error +- lists a 49MB kit (`punk91.exe`, 3629 entries) in 96ms whole-process, against 46ms for a + native Tcl 9 zipfs mount plus recursive walk in-process +- extracts 892 files from the 1.4MB tclsfe tcl_library payload in 493ms from a split zip, + and 490ms STRAIGHT FROM THE PREFIXED EXECUTABLE with no split step - against tcllib's + 52ms parse plus 830ms unzip for the same set +- honest caveat: much of that 490ms is filesystem writes, which a pure-Tcl reader also + pays. The tcllib comparison is inflated by its byte-oriented `fileutil::decode` + framework, so the pure-Tcl-vs-punkzip gap will be smaller. The accelerator's value + grows with entry count - the large-library-payload bake is the case that motivates it. + +Zig-version feasibility, measured the same day: the source builds CLEAN under its +declared zig 0.15.2 (`zig build`, exit 0, 2.0MB exe), which means the hard part - the +0.15.1 Writer/Reader redesign - is already absorbed. Under zig 0.16.0 it fails on one +surfaced error, `std.process.argsAlloc` having been removed; 0.16 continues the same +architectural move, threading an explicit `Io` through `std.process`/`std.fs` +(`Args`+`Iterator`, `currentPath(io,...)`, `spawn(io,...)`, `Io.Dir`) and handing `main` +an `Init` carrying args/environ/io. The blast radius is bounded: 17 `std.process.`, 24 +`std.fs.`, 1 `std.io.`, 1 `std.os.`, 4 ArrayList and 8 writer/reader call sites, confined +to 3 of roughly 20 files (`punkzip.zig` 715 lines, `zipper.zig`, `punkzip_test.zig`). The +~10k-line compression core (deflate, huffman, lz77, shrink, reduce, implode, bits, +bitstream, tables, zip) is pure computation and untouched by the Io redesign. There are +no external dependencies (`.dependencies = .{}`), and `zig build test` already runs ten +test modules. + +Toolchain is on hand: `bin/tools/` already carries unpacked zig 0.16.0 alongside 0.15.1, +0.15.2 and 0.14.1 (`bin/tools/zig` is currently 0.15.2), with `bin/punk-getzig.cmd` for +per-version fetch. + +Upstream state at drafting: the 0.15.2 migration is IN PROGRESS and uncommitted +(`build.zig`, `build.zig.zon`, `src/punkzip.zig`, `src/punkzip_test.zig`, +`src/zipper.zig` modified; `AGENTS.md`, `ZIG_IO.md`, `ZIG_ARRAYLIST.md`, +`refactor_2026-01-19.txt` untracked). Vendoring must take a deliberate, recorded upstream +state rather than a working tree. + +## Approach + +- VENDOR the source into punkshell (developer decision, 2026-07-26). It is small, + dependency-free and public domain, so vendoring makes clean-checkout reproducibility + trivial and removes any build-time reach outside the repo. The cost - a maintenance + fork from `c:/repo/jn/zig/punkzip` - is accepted, and mitigated by recording the + upstream origin and a re-vendor procedure beside the tree. +- Keep zig OPTIONAL. The tool build is its own make.tcl step, not part of `bakehouse`: + a clean checkout with only tclsh must still build and bake, since making zig a + prerequisite for an ordinary build would be a significant regression in what punkshell + needs to bootstrap. +- punk::zip decides per call whether the accelerator is usable and silently falls back; + the pure-Tcl floor from G-124 is never optional, and parity between the two paths is a + test obligation rather than an assumption. +- Two small upstream-side changes are prerequisites for use as a build tool: `extract` + currently writes to the current directory only (no target-directory argument), and + there is no explicit machine-readable listing mode - the human table is parseable + (banner lines prefixed `# `, fixed columns, name last) but nothing pins it. +- Port order is a work decision: porting upstream first and then vendoring the ported + state keeps the fork boundary clean; vendoring first and porting in-tree gets the + in-tree build gate working sooner. Record which was done and why. + +## Alternatives considered + +- Build from the external checkout at `c:/repo/jn/zig/punkzip` (buildsuite-style fetch or + a path reference) - rejected by developer decision: a clean checkout would depend on a + machine-local path or a network fetch for a tool that is a few hundred KB of + dependency-free public-domain source. +- Put it under `src/buildsuites/` - not chosen: the buildsuites are the arm's-length + factory for building EXTERNAL sources (Tcl runtimes) fetched per `sources.config`, with + a fetch and test-gate contract that a vendored first-party tool does not need. Revisit + only if the tool build grows a fetch step. +- Keep hand-building `bin/punkzip.exe` as today - rejected: the current binary is already + a year stale against its own source, nothing in the repo can rebuild it, and its + provenance is unrecorded. +- Build it as part of the default `bakehouse` path - rejected: makes zig a prerequisite + for an ordinary build (see Approach). +- Use punkzip as the only zip reader, with no pure-Tcl path - rejected: it would make a + build tool a hard dependency of module-level functionality, and would not help a kit + running on a platform we have no punkzip build for. G-124 is the floor. + +## Notes + +- Depends on: G-124 - the pure-Tcl floor and the fixture set this goal's parity suite + reuses. This goal is additive to it and must not weaken it. +- Related: G-005 - zig-based build infrastructure for binary dependencies from source; + this is a small, self-contained instance of that direction and the first zig build in + the tree that is not a Tcl runtime. +- Related: G-004 - `bin/*.exe` is gitignored, so the built accelerator is an untracked + local artifact, consistent with the no-committed-binaries direction: the source is + committed and the binary is built. +- Related: G-063 - licence tracking for the vendored tree (public-domain hwzip lineage + plus the MIT musl `time.zig`). +- Related: G-026 - provenance for the vendored copy (which upstream state, when, and how + to re-vendor). +- Related: G-123 / G-006 - the other way to put a punkzip binary on a machine is to + PUBLISH one and fetch it, riding the artifact-server and consent machinery those goals + build, rather than requiring zig locally. Not chosen here (the source is tiny and + dependency-free, and a build keeps bin/ binaries reproducible from committed source per + G-004) but it is the natural extension if a no-zig fast path is ever wanted. Recorded + from the drafting overlap survey. +- Related: G-105 - if cross-target punkzip builds are ever wanted, the toolchain and + target-naming work lives there rather than here. +- Related: G-101 - an 8.6 container decision may make archive reading hot enough for the + accelerator to matter to it. +- punkzip CLI as at drafting: `list `, `extract `, + `create [-m ] [-c ] `, plus an undocumented + `build` subcommand using `zipper.add_pathinfo_items`. Compression methods store, + shrink, reduce, implode, deflate - the legacy methods are implemented in-tree, so it + reads archives zlib-based readers cannot.