punkshell/goals/archive/G-124-punkzip-reader.md

G-124 Dependency-free zip reading in punk::zip

Status: achieved 2026-07-26 Scope: src/modules/punk/zip-999999.0a1.0.tm (reader + introspection procs, base-offset derivation factored out of extract_preamble for shared use, buildversion); src/make.tcl (zipfs-less kit extraction path as first consumer - the zipfile::decode requirement is removed, not made optional); src/tests/modules/punk/zip/ (new testsuite - mkzip round-trip, prefixed-archive cases, introspection, unsupported-input behaviour); bootsupport + src/vfs/_vfscommon.vfs copies via established sync channels Goal: punkshell can read a zip archive - including one prefixed by an executable, with either archive-relative or file-relative offsets - using only stock Tcl, and can do so as a documented utility a Tcl script programmer would reach for: list what is inside without extracting, extract all or part of it, on any tclsh with no zipfs, no vfs::zip and no tcllib. make.tcl's zipfs-less kit extraction stops depending on tcllib's zipfile::decode (absent from e.g msys2's /usr/bin/tclsh8.6) and stops depending on a split intermediate whose offsets can be wrong, and punk::zip's writer gains the round-trip verification it has never had. Acceptance: punk::zip extracts to a target directory from (a) a plain zip, (b) an executable-prefixed archive with archive-relative offsets and (c) one with file-relative offsets - bin/runtime/win32-x86_64/tclsh90b4_piperepl.exe is the recorded (c) case, on which the current split-then-tcllib path fails with "Bad zip file. Bad closure." - producing CRC-verified byte-identical members with directory entries materialized as directories, on a tclsh with no zipfs, no vfs::zip and no tcllib reachable (the interpreter used is recorded here); an introspection call lists members without extracting, returning per-entry name, size, compressed size, method, mtime, crc and directory-vs-file, over the same three input shapes, and both surfaces carry punk::args definitions with worked examples so 'i punk::zip::' documents them; unsupported inputs (zip64, encrypted, unknown compression method) fail naming the reason rather than emitting partial or garbled output; make.tcl's zipfs-less kit extraction consumes the new surface and NO zipfile::decode requirement remains anywhere in the shipped tree, verified by a tree grep, and demonstrated by an 8.6 zipfs-less bake of a zip kit on a tcllib-less tclsh (msys2 /usr/bin/tclsh8.6 as reference host) producing an artifact that boots with its tcl_library present; a new src/tests/modules/punk/zip/ suite covers a mkzip -> read round trip asserting names, content, directory-vs-file classification and stored attributes - recording the answer to the standing zipfs dir-misidentification question noted in punk::mix::cli and src/vfs/mkzipfix.vfs - plus the three prefixed-archive cases, the introspection output, and the unsupported-input failures; punk::zip buildversion minor-bumped with its changelog line.

Context

punk::zip is a write-plus-split module: walk, mkzip/Addentry, extract_preamble. Nothing in the tree can READ a zip without zipfs (Tcl 8.7+/9), vfs::zip, or tcllib's zipfile::decode. That last one is a SYSTEM package punkshell does not vendor, and it has exactly one consumer: make.tcl's zipfs-less kit-extraction fallback (shipped under G-122, achieved - see goals/archive/G-122-host-target-platform-split.md).

Measured 2026-07-26: msys2's /usr/bin/tclsh8.6 has no tcllib, so a bake driven from that host cannot lift the runtime's own zip (its tcl_library) into the kit and the artifact does not boot. The same tclsh, given a tcllib on its path, splits and decodes byte-identically to a native 8.6 host (baseoffset 5267968, 1429659 zip bytes, 1784 entries from both) - so the gap is purely the absent package, not an msys defect.

The dependency is not the only defect. extract_preamble carries #todo - if it was internal preamble - need to adjust offsets to fix the split off zipfile. That todo is live: an archive whose offsets are FILE-relative (the historical zipfs mkimg convention) splits into a .zip whose offsets still assume the removed preamble. bin/runtime/win32-x86_64/tclsh90b4_piperepl.exe is such a file (filerel_cdirstart == recorded_offset == 4085188); the current pipeline splits it correctly and tcllib then fails on the result with "Bad zip file. Bad closure.". A reader that works on the ORIGINAL file at the derived base offset never produces the broken intermediate - the failure mode disappears rather than needing offset-rewriting code. (Survey of the current store: every other runtime and every built kit is archive-relative, so this is a real but presently-unexercised shape.)

Reading has value well beyond the build. punk::zip is in the bootsupport manifest and ships in every kit (_vfscommon.vfs/modules/punk/zip-*.tm), so a reader there is available to any script the shell runs - and punkshell today has no answer at all for "what is inside this .tm modpod / kit / zip" on a runtime without zipfs.

Approach

• Factor the base-offset derivation out of extract_preamble into a shared private helper so the reader accepts a plain zip or a prefixed executable through one code path, and extract_preamble stops being the only thing that knows how to find the archive within a file.
• Walk ALL central-directory records. extract_preamble binary-scans only the first, with its own #todo! loop through all cdr file headers note. Sizes and offsets taken from the CD sidestep data descriptors entirely.
• Inflate with the stock primitives the writer already uses: zlib inflate / zlib stream inflate mirror Addentry's zlib deflate / zlib stream deflate, and zlib crc32 gives verification tcllib never performed (its CopyFile carries # FUTURE: Run crc checksum on created file). Mirror Addentry's existing 2MB threshold for choosing whole-member vs streaming.
• Support store and deflate; fail by name on zip64, encryption (GPBF bit 0) and unknown methods rather than emitting partial output.
• The introspection surface is the same CD walk without extraction, so listing costs no decompression - which is what makes it usable as a general scripting utility.

Alternatives considered

• Vendor tcllib's zipfile::decode (plus its fileutil::decode dependency) through G-065 - rejected: it imports a byte-oriented decode framework to obtain one capability, does not verify CRCs, cannot read a prefixed executable at all (the caller must split first, which is exactly where the file-relative bug lives), and leaves punk::zip's writer with no round-trip coverage. Cost for comparison, measured 2026-07-26: 830ms to extract 892 files that punkzip does in 490ms.
• Require zipfs, i.e. declare 8.6-without-tcllib unsupported - rejected: 8.6 is a real target (src/AGENTS.md), G-101's 8.6 container work has no zipfs by definition, and G-034 is an existing 8.6 read failure in the shell itself.
• Shell out to an external zip tool as the ONLY path - rejected as a floor, since it would make a build tool a hard dependency of module-level functionality. Adopted as an optional accelerator instead - see G-126.

Notes

• Related: G-034 - its Acceptance permits "an alternative modpod mount path" for the 8.6 code interp, which currently fails for want of vfs::RegisterMount. A dependency-free reader is that alternative: extract rather than mount.
• Related: G-101 - an 8.6 kit container has no zipfs by definition, so whatever container is chosen needs a reader that works without it. This goal is that primitive.
• Related: G-109 - manifest-declared discovery reads members out of zip-based .tm modules; the introspection surface here is the natural mechanism.
• Related: G-111 - modpod IS the zip .tm format and that goal establishes its first test baseline. The round-trip suite here is adjacent coverage of the writer, not a substitute for modpod's own.
• Related: G-065 - declarative vendoring is the mechanism the rejected alternative above would have used.
• Related: G-066 - lib.copyasmodule emits zip .tm modules; its converter testsuite could verify output with this reader.
• Related: G-110 - adjacent, not a dependency: it studies caching of EXTRACTED shared libs, where extraction today is done by zipfs/vfs::zip.
• Related: G-126 - the optional zig accelerator that sits behind this pure-Tcl floor.
• Related: G-128 (recorded at activation) - shares the archive-relative vs file-relative offset distinction from the other side: this goal READS a zip at a derived base offset, that one MOVES a prefixed executable's resources and must not invalidate the offsets a reader derives.
• Related: G-125 - the deploy gate that turns an extraction failure into a failed kit rather than a deployed-but-unbootable one. Independent of this goal, which removes the most common cause of that failure.
• Related: G-122 (achieved) - shipped the zipfs-less assembly half and recorded the tcllib gap that motivates this goal; see goals/archive/G-122-host-target-platform-split.md.
• Test baseline as at drafting: punk::zip has NO tests (grep -rn "punk::zip" src/tests/ finds only log noise). The zipper modpod - a separate low-level writer - has four write-only smoke tests in src/tests/modules/zipper/testsuites/zipper/zipper.test (version, initialize, addentry, finalize-produces->100-bytes) with no content assertions. So nothing today verifies that anything punkshell WRITES is readable.
• Standing question the round-trip suite should settle: punk::mix::cli records that "punk::zip::mkzip stores permissions - (unix style) - which zipfs mkzip doesn't" and src/vfs/mkzipfix.vfs's copy records that this "confuses zipfs when reading - it misidentifies dirs as files". The reader makes that assertable in-tree for the first time; record the finding here. ANSWERED 2026-07-26 (Tcl 9.0.3): the permission bits are not the problem and zipfs does NOT misidentify punk::zip::mkzip directories. A mkzip archive mounted under zipfs reports its directory entries as file isdirectory 1 / file type directory. What zipfs keys on is the TRAILING SLASH, exactly as the current cli-999999.0a1.0.tm wording says ("Directory ident in zipfs relies on folders ending with trailing slash ... it can't use permissions/attributes alone"); punk::zip::walk always emits one, so the classification holds. The older, less precise wording survives only in frozen module snapshots under src/vfs/mkzipfix.vfs and src/vfs/project.vfs (cli-0.3/0.3.1) - stale copies, not live guidance. The real difference from zipfs mkzip is that zipfs writes NO directory entries at all (its members carry external attributes 0x00000000 and version-made-by 0x0014); punk::zip::mkzip writes them with 0x41ff0010 (unix 0o040777 in the high word, FAT directory bit in the low byte) and version-made-by 0x0017 (host 0 = MS-DOS/FAT). Pinned by zipreader_roundtrip_mkzip (attributes) and the zipfs-gated zipreader_zipfs_sees_mkzip_directories.

Progress

2026-07-26 - implemented; every Acceptance clause verified. Detail:

• punk::zip 0.1.1 -> 0.2.0. New public surface: archive_info (geometry + offset-style derivation), members (central-directory introspection, no decompression), unzip (crc-verified extraction of all or part of an archive). Private machinery: Eocd_scan / Eocd_try / Cdir_records / Archive_read / Extract_member plus Dos_to_timet, Decode_text, Is_directory_entry, Method_name, Select_members, Safe_member_path, Member_unsupported_reason, Open_archive.
• extract_preamble now consumes Archive_read instead of carrying its own scan; its five puts stdout debug lines are gone. Two defects went with the refactor: it read 28 bytes of a 30-byte local file header, and it derived the file-relative base offset from the FIRST central directory record only (its own #todo!). The shared walk takes the minimum over ALL records and then validates that a PK\3\4 really sits there.
• Offset derivation, stated once: offsetbase = cdiroffset - diroffset. Positive means an external preamble with archive-relative offsets; zero means the offsets are file-relative (or there is no preamble). dataoffset (the split point) is offsetbase in the first case and the minimum recorded member offset in the second. Reported as offsetstyle archive | file | plain | none.
• Store and deflate only, mirroring the writer's primitives (zlib inflate / zlib stream inflate, zlib crc32), with the writer's 2MB whole-member-vs-streaming threshold. TRAP found and pinned: zlib stream inflate's get returns only what the stream has already produced, so one get per put silently truncates a large member to the first few chunks (measured: 327680 of 5048890 bytes on both 8.6.12 and 9.0.3). The inner drain loop is load-bearing.
• Refusals are decided for every selected member BEFORE any file is opened, so an unsupported archive cannot leave a half-populated directory. Encrypted (GPBF bit 0/6), zip64 (saturated EOCD fields or per-member 0xFFFFFFFF), unknown compression method, and member paths that escape the target are each named. crc failure deletes the partial file.
• make.tcl's zipfs-less extraction now calls punk::zip::unzip on the ORIGINAL runtime and no longer creates the extracted_<runtime>.zip intermediate. No punkshell code requires zipfile::decode any more (tree grep). Two categories of hit remain and are deliberate: tcllib's own zip/decode.tcl shipped as a LIBRARY inside some .vfs payloads, and one guarded optional probe inside the vendored third-party ooxml1.10/ooxml.tcl - neither is a punkshell requirement.

Verification records (2026-07-26)

• Reference host for the dependency-free claim: msys2 tclsh8.6.exe 8.6.12 (C:/Users/sleek/scoop/apps/msys2/current/usr/bin/tclsh8.6.exe), probed as zipfs ABSENT, vfs::zip ABSENT, zipfile::decode ABSENT. Full suite driven there harness-free: 28 passed, 2 skipped (the zipfs-gated cross-checks only), 0 failed.
• src/tests/modules/punk/zip/testsuites/zip/zipreader.test: 30/30 under Tcl 9.0.3, 28 passed + 2 zipfs-skipped under native Tcl 8.6. Full source-tree suite after the change: 1142 tests, 1123 passed, 18 skipped, 1 failed - core/tcl exec-14.3, the documented pre-existing baseline failure.
• The three shapes, verified on real artifacts as well as built fixtures. Every runtime in bin/runtime/win32-x86_64 classified: 15 archive-relative, 1 file-relative (tclsh90b4_piperepl.exe, dataoffset 2873856, 841 members), 4 with no zip at all (the tclkits - starkits, not zipfs images). The recorded (c) case extracts all 841 members crc-verified in 735ms and yields a working tcl_library/init.tcl; the old split-then-tcllib path on the same file still fails with exactly "Bad zip file. Bad closure.", reproduced this session.
• Independent cross-check: 868 members of tclsh905.exe extracted by punk::zip compare byte-identical against the same archive mounted with tcl zipfs.
• Bake demonstration: make.tcl bake -confirm 0 punk905 driven by the msys2 tclsh8.6 (host=msys-x86_64, target=win32-x86_64). The extraction produced src/_build/extracted_tclsh905.exe/{tcl_library,lib} with no intermediate .zip, the kit assembled by concatenation, and the deployed bin/punk905.exe boots: tcl_library at //zipfs:/app/tcl_library with init.tcl, 95 encodings incl. cp437, msgcat 1.7.1, http 2.10.2. The kit also reads ITSELF with the new reader (3642 members, offsetstyle archive, dataoffset 2320384).
• i punk::zip::members / i punk::zip::unzip / i punk::zip::archive_info render through punk::ns::cmdhelp with their worked examples intact.

Findings for other goals

• G-034 / G-101: on the msys2 tclsh8.6, make.tcl cannot mount its own punk::mix::templates modpod - "Unable to load vfs::zip package to mount module templates-0.2.0 (and zipfs not available either)". Non-fatal (the package reports broken and the build continues), and it is exactly the failure punk::zip::unzip can now serve as the extract-rather-than-mount alternative.
• Unrelated make.tcl limitation surfaced while syncing bootsupport: make.tcl bootsupport cannot replace a bootsupport modpod that the running make.tcl has itself mounted - BOOTSUPPORT module update FAILED: ... templates-0.2.0.tm (invalid argument). A plain tclsh copies the identical file without complaint. Worked around by hand this session; candidate goal material, not addressed here.