punkshell/goals/G-126-punkzip-accelerator.md

G-126 punkzip as a vendored zip accelerator: zig 0.16 port, reproducible build into bin/, punk::zip fast path

Status: proposed Scope: src/tools/punkzip/ (vendored punkzip source tree - root name settled in the work - with provenance and licence records); src/make.tcl (tool build step); bin/tools/zig* (pinned toolchain as consumed); bin/punkzip.exe (untracked build output); src/modules/punk/zip-999999.0a1.0.tm (accelerator detection + fast path behind the G-124 pure-Tcl floor); src/tests/modules/punk/zip/ (parity suite - accelerated and pure-Tcl paths must agree); upstream maintained checkout c:/repo/jn/zig/punkzip (re-vendor source, read-only from this repo) Goal: punkshell carries punkzip as vendored source, builds it from the repo's pinned zig toolchain, and uses the resulting binary as an optional accelerator for zip reading - with the pure-Tcl path as the always-available floor - so a machine with zig gets fast archive listing and extraction (materially so for bakes with large library payloads) while a clean checkout with only tclsh still builds and bakes exactly as before. Acceptance: the vendored source builds under the repo's pinned zig 0.16 toolchain with 'zig build test' green (the port's own 10 test modules), driven by a make.tcl step that produces bin/punkzip.exe reproducibly on a clean checkout - and zig stays OPTIONAL: a tclsh-only clean checkout completes packages + bake unchanged, with the step's absence reported rather than fatal; punkzip gains an explicit extract-to-directory argument and a machine-readable listing mode whose format is pinned by a test (the current fixed-width table stays the human default); punk::zip detects the accelerator and uses it, with a parity suite proving accelerated and pure-Tcl paths return identical listings and produce byte-identical extraction trees over the G-124 fixture set including the file-relative prefixed executable, and proving the pure-Tcl path is still taken when the binary is absent; a recorded benchmark on a representative kit payload states the speedup and the entry count at which it becomes material; the vendored tree records its upstream origin, the commit or state it was taken from, the re-vendor procedure, and its licensing (public-domain hwzip lineage plus MIT musl time.zig) per G-063 and the G-026 provenance direction.

Context

punkzip is a fork of hwzip.zig (Hadrien Dorio's zig port of Hans Wennborg's hwzip 2.1), maintained by the developer at c:/repo/jn/zig/punkzip and adapted for the case standard zip tools handle inconsistently: an executable or script concatenated with a zip whose offsets were never adjusted. Its README records the motivation - 7z and peazip can read and even edit catenated exezips with unadjusted offsets, but not when offsets have been rewritten file-relative, and behaviour varies with the file extension used. That is precisely the axis punkshell cares about (see G-124's Context and the archived G-122).

Measured 2026-07-26 with the existing bin/punkzip.exe (v2.1.0, Nov 2024, untracked - bin/*.exe is gitignored - and already stale against the 2.1.1 source):

• lists bin/runtime/win32-x86_64/tclsh90b4_piperepl.exe, the FILE-relative prefixed executable on which tcllib's decoder fails with "Bad zip file. Bad closure." - 846 entries, no error
• lists a 49MB kit (punk91.exe, 3629 entries) in 96ms whole-process, against 46ms for a native Tcl 9 zipfs mount plus recursive walk in-process
• extracts 892 files from the 1.4MB tclsfe tcl_library payload in 493ms from a split zip, and 490ms STRAIGHT FROM THE PREFIXED EXECUTABLE with no split step - against tcllib's 52ms parse plus 830ms unzip for the same set
• honest caveat: much of that 490ms is filesystem writes, which a pure-Tcl reader also pays. The tcllib comparison is inflated by its byte-oriented fileutil::decode framework, so the pure-Tcl-vs-punkzip gap will be smaller. The accelerator's value grows with entry count - the large-library-payload bake is the case that motivates it.

Zig-version feasibility, measured the same day: the source builds CLEAN under its declared zig 0.15.2 (zig build, exit 0, 2.0MB exe), which means the hard part - the 0.15.1 Writer/Reader redesign - is already absorbed. Under zig 0.16.0 it fails on one surfaced error, std.process.argsAlloc having been removed; 0.16 continues the same architectural move, threading an explicit Io through std.process/std.fs (Args+Iterator, currentPath(io,...), spawn(io,...), Io.Dir) and handing main an Init carrying args/environ/io. The blast radius is bounded: 17 std.process., 24 std.fs., 1 std.io., 1 std.os., 4 ArrayList and 8 writer/reader call sites, confined to 3 of roughly 20 files (punkzip.zig 715 lines, zipper.zig, punkzip_test.zig). The ~10k-line compression core (deflate, huffman, lz77, shrink, reduce, implode, bits, bitstream, tables, zip) is pure computation and untouched by the Io redesign. There are no external dependencies (.dependencies = .{}), and zig build test already runs ten test modules.

Toolchain is on hand: bin/tools/ already carries unpacked zig 0.16.0 alongside 0.15.1, 0.15.2 and 0.14.1 (bin/tools/zig is currently 0.15.2), with bin/punk-getzig.cmd for per-version fetch.

Upstream state: the drafting-time 0.15.2 migration (then in progress and uncommitted) was completed, made fully green and committed 2026-07-27 as upstream e9b5b553, with the localtime banner restored dependency-free in 78453fc0 - see Notes for the pre-activation baseline record. Vendoring must take a deliberate, recorded upstream state rather than a working tree; one now exists.

Approach

• VENDOR the source into punkshell (developer decision, 2026-07-26). It is small, dependency-free and public domain, so vendoring makes clean-checkout reproducibility trivial and removes any build-time reach outside the repo. The cost - a maintenance fork from c:/repo/jn/zig/punkzip - is accepted, and mitigated by recording the upstream origin and a re-vendor procedure beside the tree.
• Keep zig OPTIONAL. The tool build is its own make.tcl step, not part of bakehouse: a clean checkout with only tclsh must still build and bake, since making zig a prerequisite for an ordinary build would be a significant regression in what punkshell needs to bootstrap.
• punk::zip decides per call whether the accelerator is usable and silently falls back; the pure-Tcl floor from G-124 is never optional, and parity between the two paths is a test obligation rather than an assumption.
• Two small upstream-side changes are prerequisites for use as a build tool: extract currently writes to the current directory only (no target-directory argument), and there is no explicit machine-readable listing mode - the human table is parseable (banner lines prefixed # , fixed columns, name last) but nothing pins it.
• Port order is a work decision: porting upstream first and then vendoring the ported state keeps the fork boundary clean; vendoring first and porting in-tree gets the in-tree build gate working sooner. Record which was done and why.

Alternatives considered

• Build from the external checkout at c:/repo/jn/zig/punkzip (buildsuite-style fetch or a path reference) - rejected by developer decision: a clean checkout would depend on a machine-local path or a network fetch for a tool that is a few hundred KB of dependency-free public-domain source.
• Put it under src/buildsuites/ - not chosen: the buildsuites are the arm's-length factory for building EXTERNAL sources (Tcl runtimes) fetched per sources.config, with a fetch and test-gate contract that a vendored first-party tool does not need. Revisit only if the tool build grows a fetch step.
• Keep hand-building bin/punkzip.exe as today - rejected: the current binary is already a year stale against its own source, nothing in the repo can rebuild it, and its provenance is unrecorded.
• Build it as part of the default bakehouse path - rejected: makes zig a prerequisite for an ordinary build (see Approach).
• Use punkzip as the only zip reader, with no pure-Tcl path - rejected: it would make a build tool a hard dependency of module-level functionality, and would not help a kit running on a platform we have no punkzip build for. G-124 is the floor.

Notes

• Depends on: G-124 (achieved 2026-07-26 - see goals/archive/G-124-punkzip-reader.md) - the pure-Tcl floor and the fixture set this goal's parity suite reuses. This goal is additive to it and must not weaken it. What landed and what this goal must match: punk::zip::archive_info / punk::zip::members / punk::zip::unzip in punk::zip 0.2.0, with the member-dict keys the parity suite has to reproduce (name, isdirectory, size, csize, method, methodname, mtime, dostime, crc, offset, fileoffset, flags, encrypted, attributes, iattributes, madeby, hostsystem, version, comment) and the fixture set now built in src/tests/modules/punk/zip/testsuites/zip/zipreader.test (the three archive shapes from one source tree, plus hand-built raw zips for the refusal cases via that file's rawzip helper). Recorded pure-Tcl timing to beat: 841 members of the file-relative tclsh90b4_piperepl.exe extracted crc-verified in 735ms on Tcl 9.0.3.
• Related: G-128 - back-pointer added at the G-124 archive sweep (2026-07-26), where G-124 was the pair's only recorded bridge. The overlap is direct and named in G-128's own Scope: it is the second vendored zig tool and shares THIS goal's make.tcl tool build step and pinned bin/tools/zig* toolchain, so the build step designed here has to serve both.
• Related: G-005 - zig-based build infrastructure for binary dependencies from source; this is a small, self-contained instance of that direction and the first zig build in the tree that is not a Tcl runtime.
• Related: G-004 - bin/*.exe is gitignored, so the built accelerator is an untracked local artifact, consistent with the no-committed-binaries direction: the source is committed and the binary is built.
• Related: G-063 - licence tracking for the vendored tree (public-domain hwzip lineage plus the MIT musl time.zig).
• Related: G-026 - provenance for the vendored copy (which upstream state, when, and how to re-vendor).
• Related: G-123 / G-006 - the other way to put a punkzip binary on a machine is to PUBLISH one and fetch it, riding the artifact-server and consent machinery those goals build, rather than requiring zig locally. Not chosen here (the source is tiny and dependency-free, and a build keeps bin/ binaries reproducible from committed source per G-004) but it is the natural extension if a no-zig fast path is ever wanted. Recorded from the drafting overlap survey.
• Related: G-105 - if cross-target punkzip builds are ever wanted, the toolchain and target-naming work lives there rather than here.
• Related: G-101 - an 8.6 container decision may make archive reading hot enough for the accelerator to matter to it.
• punkzip CLI as at drafting: list <zipfile>, extract <zipfile>, create [-m <method>] [-c <comment>] <zipfile> <files...>, plus an undocumented build subcommand using zipper.add_pathinfo_items. Compression methods store, shrink, reduce, implode, deflate - the legacy methods are implemented in-tree, so it reads archives zlib-based readers cannot.
• Pre-activation upstream baseline (user-directed, 2026-07-27): the maintained checkout c:/repo/jn/zig/punkzip now carries commit e9b5b553 "zig 0.15.2 migration complete: build + test green (122/122)" and 78453fc0 "restore localtime diagnostic banner, dependency-free, on stderr". zig build test under the pinned 0.15.2 toolchain runs all ten test modules - 122/122 tests green on Windows, including the previously mis-wired bitstream_test.zig (its 4 tests had never run anywhere: build.zig pointed test_bitstream at bits_test.zig) and the 13 exact-stdout CLI tests, whose expected outputs now PIN the current human table listing and creating:/extracting: wording. Port-order decision per Approach: UPSTREAM-FIRST, adopted in practice - port and stabilise upstream, then vendor the ported state.
• zeit decision (user, 2026-07-27): the zeit timezone dependency stays out - the localtime diagnostic returns dependency-free instead (Windows kernel32 GetLocalTime/GetTimeZoneInformation with numeric offset + zone name; honest UTC line elsewhere), emitted on STDERR. Contract worth keeping: stdout is the parseable product, diagnostics go to stderr - the machine-readable listing mode this goal adds must preserve that split. Fallback if real tz fidelity is ever wanted: zeit v0.9.0 is zig-0.16-compatible, MIT, zero transitive deps - vendor it as a path dep (never a build-time URL fetch, which conflicts with this goal's no-network-build premise).
• Known upstream gaps carried into the goal work: the catenated/prefixed-zip base_offset read path (src/zip.zig) - the fork's raison d'etre - has NO direct zig-side test; add coverage alongside the machine-listing pin (the parity suite covers it from the Tcl side only). The undocumented build subcommand remains unfinished scaffolding (wrong arg indexing, writes only an EOCD; its finalize call was restored at the checkpoint) - finish or remove it during the port. 0.16 first surfaced error re-confirmed 2026-07-27: std.process.argsAlloc removed.