# G-124 Dependency-free zip reading in punk::zip Status: achieved 2026-07-26 Scope: src/modules/punk/zip-999999.0a1.0.tm (reader + introspection procs, base-offset derivation factored out of extract_preamble for shared use, buildversion); src/make.tcl (zipfs-less kit extraction path as first consumer - the zipfile::decode requirement is removed, not made optional); src/tests/modules/punk/zip/ (new testsuite - mkzip round-trip, prefixed-archive cases, introspection, unsupported-input behaviour); bootsupport + src/vfs/_vfscommon.vfs copies via established sync channels Goal: punkshell can read a zip archive - including one prefixed by an executable, with either archive-relative or file-relative offsets - using only stock Tcl, and can do so as a documented utility a Tcl script programmer would reach for: list what is inside without extracting, extract all or part of it, on any tclsh with no zipfs, no vfs::zip and no tcllib. make.tcl's zipfs-less kit extraction stops depending on tcllib's zipfile::decode (absent from e.g msys2's /usr/bin/tclsh8.6) and stops depending on a split intermediate whose offsets can be wrong, and punk::zip's writer gains the round-trip verification it has never had. Acceptance: punk::zip extracts to a target directory from (a) a plain zip, (b) an executable-prefixed archive with archive-relative offsets and (c) one with file-relative offsets - bin/runtime/win32-x86_64/tclsh90b4_piperepl.exe is the recorded (c) case, on which the current split-then-tcllib path fails with "Bad zip file. Bad closure." - producing CRC-verified byte-identical members with directory entries materialized as directories, on a tclsh with no zipfs, no vfs::zip and no tcllib reachable (the interpreter used is recorded here); an introspection call lists members without extracting, returning per-entry name, size, compressed size, method, mtime, crc and directory-vs-file, over the same three input shapes, and both surfaces carry punk::args definitions with worked examples so 'i punk::zip::' documents them; unsupported inputs (zip64, encrypted, unknown compression method) fail naming the reason rather than emitting partial or garbled output; make.tcl's zipfs-less kit extraction consumes the new surface and NO zipfile::decode requirement remains anywhere in the shipped tree, verified by a tree grep, and demonstrated by an 8.6 zipfs-less bake of a zip kit on a tcllib-less tclsh (msys2 /usr/bin/tclsh8.6 as reference host) producing an artifact that boots with its tcl_library present; a new src/tests/modules/punk/zip/ suite covers a mkzip -> read round trip asserting names, content, directory-vs-file classification and stored attributes - recording the answer to the standing zipfs dir-misidentification question noted in punk::mix::cli and src/vfs/mkzipfix.vfs - plus the three prefixed-archive cases, the introspection output, and the unsupported-input failures; punk::zip buildversion minor-bumped with its changelog line. ## Context punk::zip is a write-plus-split module: `walk`, `mkzip`/`Addentry`, `extract_preamble`. Nothing in the tree can READ a zip without zipfs (Tcl 8.7+/9), vfs::zip, or tcllib's `zipfile::decode`. That last one is a SYSTEM package punkshell does not vendor, and it has exactly one consumer: make.tcl's zipfs-less kit-extraction fallback (shipped under G-122, achieved - see goals/archive/G-122-host-target-platform-split.md). Measured 2026-07-26: msys2's `/usr/bin/tclsh8.6` has no tcllib, so a bake driven from that host cannot lift the runtime's own zip (its tcl_library) into the kit and the artifact does not boot. The same tclsh, given a tcllib on its path, splits and decodes byte-identically to a native 8.6 host (baseoffset 5267968, 1429659 zip bytes, 1784 entries from both) - so the gap is purely the absent package, not an msys defect. The dependency is not the only defect. `extract_preamble` carries `#todo - if it was internal preamble - need to adjust offsets to fix the split off zipfile`. That todo is live: an archive whose offsets are FILE-relative (the historical `zipfs mkimg` convention) splits into a .zip whose offsets still assume the removed preamble. `bin/runtime/win32-x86_64/tclsh90b4_piperepl.exe` is such a file (filerel_cdirstart == recorded_offset == 4085188); the current pipeline splits it correctly and tcllib then fails on the result with "Bad zip file. Bad closure.". A reader that works on the ORIGINAL file at the derived base offset never produces the broken intermediate - the failure mode disappears rather than needing offset-rewriting code. (Survey of the current store: every other runtime and every built kit is archive-relative, so this is a real but presently-unexercised shape.) Reading has value well beyond the build. punk::zip is in the bootsupport manifest and ships in every kit (`_vfscommon.vfs/modules/punk/zip-*.tm`), so a reader there is available to any script the shell runs - and punkshell today has no answer at all for "what is inside this .tm modpod / kit / zip" on a runtime without zipfs. ## Approach - Factor the base-offset derivation out of `extract_preamble` into a shared private helper so the reader accepts a plain zip or a prefixed executable through one code path, and `extract_preamble` stops being the only thing that knows how to find the archive within a file. - Walk ALL central-directory records. `extract_preamble` binary-scans only the first, with its own `#todo! loop through all cdr file headers` note. Sizes and offsets taken from the CD sidestep data descriptors entirely. - Inflate with the stock primitives the writer already uses: `zlib inflate` / `zlib stream inflate` mirror `Addentry`'s `zlib deflate` / `zlib stream deflate`, and `zlib crc32` gives verification tcllib never performed (its `CopyFile` carries `# FUTURE: Run crc checksum on created file`). Mirror `Addentry`'s existing 2MB threshold for choosing whole-member vs streaming. - Support store and deflate; fail by name on zip64, encryption (GPBF bit 0) and unknown methods rather than emitting partial output. - The introspection surface is the same CD walk without extraction, so listing costs no decompression - which is what makes it usable as a general scripting utility. ## Alternatives considered - Vendor tcllib's `zipfile::decode` (plus its `fileutil::decode` dependency) through G-065 - rejected: it imports a byte-oriented decode framework to obtain one capability, does not verify CRCs, cannot read a prefixed executable at all (the caller must split first, which is exactly where the file-relative bug lives), and leaves punk::zip's writer with no round-trip coverage. Cost for comparison, measured 2026-07-26: 830ms to extract 892 files that punkzip does in 490ms. - Require zipfs, i.e. declare 8.6-without-tcllib unsupported - rejected: 8.6 is a real target (src/AGENTS.md), G-101's 8.6 container work has no zipfs by definition, and G-034 is an existing 8.6 read failure in the shell itself. - Shell out to an external zip tool as the ONLY path - rejected as a floor, since it would make a build tool a hard dependency of module-level functionality. Adopted as an optional accelerator instead - see G-126. ## Notes - Related: G-034 - its Acceptance permits "an alternative modpod mount path" for the 8.6 code interp, which currently fails for want of `vfs::RegisterMount`. A dependency-free reader is that alternative: extract rather than mount. - Related: G-101 - an 8.6 kit container has no zipfs by definition, so whatever container is chosen needs a reader that works without it. This goal is that primitive. - Related: G-109 - manifest-declared discovery reads members out of zip-based .tm modules; the introspection surface here is the natural mechanism. - Related: G-111 - modpod IS the zip .tm format and that goal establishes its first test baseline. The round-trip suite here is adjacent coverage of the writer, not a substitute for modpod's own. - Related: G-065 - declarative vendoring is the mechanism the rejected alternative above would have used. - Related: G-066 - `lib.copyasmodule` emits zip .tm modules; its converter testsuite could verify output with this reader. - Related: G-110 - adjacent, not a dependency: it studies caching of EXTRACTED shared libs, where extraction today is done by zipfs/vfs::zip. - Related: G-126 - the optional zig accelerator that sits behind this pure-Tcl floor. - Related: G-128 (recorded at activation) - shares the archive-relative vs file-relative offset distinction from the other side: this goal READS a zip at a derived base offset, that one MOVES a prefixed executable's resources and must not invalidate the offsets a reader derives. - Related: G-125 - the deploy gate that turns an extraction failure into a failed kit rather than a deployed-but-unbootable one. Independent of this goal, which removes the most common cause of that failure. - Related: G-122 (achieved) - shipped the zipfs-less assembly half and recorded the tcllib gap that motivates this goal; see goals/archive/G-122-host-target-platform-split.md. - Test baseline as at drafting: punk::zip has NO tests (`grep -rn "punk::zip" src/tests/` finds only log noise). The `zipper` modpod - a separate low-level writer - has four write-only smoke tests in `src/tests/modules/zipper/testsuites/zipper/zipper.test` (version, initialize, addentry, finalize-produces->100-bytes) with no content assertions. So nothing today verifies that anything punkshell WRITES is readable. - Standing question the round-trip suite should settle: `punk::mix::cli` records that "punk::zip::mkzip stores permissions - (unix style) - which zipfs mkzip doesn't" and `src/vfs/mkzipfix.vfs`'s copy records that this "confuses zipfs when reading - it misidentifies dirs as files". The reader makes that assertable in-tree for the first time; record the finding here. ANSWERED 2026-07-26 (Tcl 9.0.3): the permission bits are not the problem and zipfs does NOT misidentify punk::zip::mkzip directories. A mkzip archive mounted under zipfs reports its directory entries as `file isdirectory` 1 / `file type` directory. What zipfs keys on is the TRAILING SLASH, exactly as the current `cli-999999.0a1.0.tm` wording says ("Directory ident in zipfs relies on folders ending with trailing slash ... it can't use permissions/attributes alone"); `punk::zip::walk` always emits one, so the classification holds. The older, less precise wording survives only in frozen module snapshots under `src/vfs/mkzipfix.vfs` and `src/vfs/project.vfs` (cli-0.3/0.3.1) - stale copies, not live guidance. The real difference from `zipfs mkzip` is that zipfs writes NO directory entries at all (its members carry external attributes 0x00000000 and version-made-by 0x0014); punk::zip::mkzip writes them with 0x41ff0010 (unix 0o040777 in the high word, FAT directory bit in the low byte) and version-made-by 0x0017 (host 0 = MS-DOS/FAT). Pinned by `zipreader_roundtrip_mkzip` (attributes) and the zipfs-gated `zipreader_zipfs_sees_mkzip_directories`. ## Progress 2026-07-26 - implemented; every Acceptance clause verified. Detail: - `punk::zip` 0.1.1 -> 0.2.0. New public surface: `archive_info` (geometry + offset-style derivation), `members` (central-directory introspection, no decompression), `unzip` (crc-verified extraction of all or part of an archive). Private machinery: `Eocd_scan` / `Eocd_try` / `Cdir_records` / `Archive_read` / `Extract_member` plus `Dos_to_timet`, `Decode_text`, `Is_directory_entry`, `Method_name`, `Select_members`, `Safe_member_path`, `Member_unsupported_reason`, `Open_archive`. - `extract_preamble` now consumes `Archive_read` instead of carrying its own scan; its five `puts stdout` debug lines are gone. Two defects went with the refactor: it read 28 bytes of a 30-byte local file header, and it derived the file-relative base offset from the FIRST central directory record only (its own `#todo!`). The shared walk takes the minimum over ALL records and then validates that a `PK\3\4` really sits there. - Offset derivation, stated once: `offsetbase = cdiroffset - diroffset`. Positive means an external preamble with archive-relative offsets; zero means the offsets are file-relative (or there is no preamble). `dataoffset` (the split point) is `offsetbase` in the first case and the minimum recorded member offset in the second. Reported as `offsetstyle` `archive` | `file` | `plain` | `none`. - Store and deflate only, mirroring the writer's primitives (`zlib inflate` / `zlib stream inflate`, `zlib crc32`), with the writer's 2MB whole-member-vs-streaming threshold. TRAP found and pinned: `zlib stream inflate`'s `get` returns only what the stream has already produced, so one `get` per `put` silently truncates a large member to the first few chunks (measured: 327680 of 5048890 bytes on both 8.6.12 and 9.0.3). The inner drain loop is load-bearing. - Refusals are decided for every selected member BEFORE any file is opened, so an unsupported archive cannot leave a half-populated directory. Encrypted (GPBF bit 0/6), zip64 (saturated EOCD fields or per-member 0xFFFFFFFF), unknown compression method, and member paths that escape the target are each named. crc failure deletes the partial file. - `make.tcl`'s zipfs-less extraction now calls `punk::zip::unzip` on the ORIGINAL runtime and no longer creates the `extracted_.zip` intermediate. No punkshell code requires `zipfile::decode` any more (tree grep). Two categories of hit remain and are deliberate: tcllib's own `zip/decode.tcl` shipped as a LIBRARY inside some `.vfs` payloads, and one guarded optional probe inside the vendored third-party `ooxml1.10/ooxml.tcl` - neither is a punkshell requirement. ### Verification records (2026-07-26) - Reference host for the dependency-free claim: msys2 `tclsh8.6.exe` 8.6.12 (`C:/Users/sleek/scoop/apps/msys2/current/usr/bin/tclsh8.6.exe`), probed as zipfs ABSENT, `vfs::zip` ABSENT, `zipfile::decode` ABSENT. Full suite driven there harness-free: 28 passed, 2 skipped (the zipfs-gated cross-checks only), 0 failed. - `src/tests/modules/punk/zip/testsuites/zip/zipreader.test`: 30/30 under Tcl 9.0.3, 28 passed + 2 zipfs-skipped under native Tcl 8.6. Full source-tree suite after the change: 1142 tests, 1123 passed, 18 skipped, 1 failed - `core/tcl exec-14.3`, the documented pre-existing baseline failure. - The three shapes, verified on real artifacts as well as built fixtures. Every runtime in `bin/runtime/win32-x86_64` classified: 15 archive-relative, 1 file-relative (`tclsh90b4_piperepl.exe`, dataoffset 2873856, 841 members), 4 with no zip at all (the tclkits - starkits, not zipfs images). The recorded (c) case extracts all 841 members crc-verified in 735ms and yields a working `tcl_library/init.tcl`; the old split-then-tcllib path on the same file still fails with exactly "Bad zip file. Bad closure.", reproduced this session. - Independent cross-check: 868 members of `tclsh905.exe` extracted by punk::zip compare byte-identical against the same archive mounted with tcl zipfs. - Bake demonstration: `make.tcl bake -confirm 0 punk905` driven by the msys2 tclsh8.6 (host=msys-x86_64, target=win32-x86_64). The extraction produced `src/_build/extracted_tclsh905.exe/{tcl_library,lib}` with no intermediate .zip, the kit assembled by concatenation, and the deployed `bin/punk905.exe` boots: tcl_library at `//zipfs:/app/tcl_library` with init.tcl, 95 encodings incl. cp437, msgcat 1.7.1, http 2.10.2. The kit also reads ITSELF with the new reader (3642 members, offsetstyle archive, dataoffset 2320384). - `i punk::zip::members` / `i punk::zip::unzip` / `i punk::zip::archive_info` render through `punk::ns::cmdhelp` with their worked examples intact. ### Findings for other goals - G-034 / G-101: on the msys2 tclsh8.6, `make.tcl` cannot mount its own `punk::mix::templates` modpod - "Unable to load vfs::zip package to mount module templates-0.2.0 (and zipfs not available either)". Non-fatal (the package reports `broken` and the build continues), and it is exactly the failure `punk::zip::unzip` can now serve as the extract-rather-than-mount alternative. - Unrelated make.tcl limitation surfaced while syncing bootsupport: `make.tcl bootsupport` cannot replace a bootsupport modpod that the running make.tcl has itself mounted - `BOOTSUPPORT module update FAILED: ... templates-0.2.0.tm (invalid argument)`. A plain tclsh copies the identical file without complaint. Worked around by hand this session; candidate goal material, not addressed here.