punkshell/goals/G-066-pkgindex-tm-repackaging.md

G-066 pkgIndex.tcl-to-.tm repackaging: lib.copyasmodule expansion with embedded metadata and distribution-unit tracking

Status: proposed Scope: src/modules/punk/mix/commandset/loadedlib-999999.0a1.0.tm (lib.copyasmodule), src/modules/punk/mix/ (modpod/zipkit tooling as needed), src/tests/modules/punk/mix/ (converter testsuite) Goal: third-party pkgIndex.tcl-based packages can be repackaged as single-file .tm modules (zip-based where the payload warrants it) that embed upstream documents (LICENSE, README) and a punkshell metadata datafile giving a consistent description of upstream name, version, license and distribution-unit membership - so packages split out of a multi-package upstream (tcllib-style) record that they shipped together at upstream version X and should be upgrade-checked as a unit - with the converter handling a substantially broader class of pkgIndex.tcl scripts than today's lib.copyasmodule and refusing clearly on scripts it cannot model rather than emitting a broken module. Acceptance: the converter repackages a proving set of at least three packages - src/vendorlib/tcl_oauth2_library plus two tcllib packages, one of which has a non-trivial pkgIndex.tcl (multiple statements, computed version, or multi-file source list) - and each resulting .tm loads via package require on the primary target runtimes (Tcl 9 kit and 8.6, or a recorded limitation referencing the G-034 code-interp constraint); each repackaged module embeds the upstream LICENSE and a metadata datafile carrying upstream name, upstream version, license indication (G-063-resolvable) and distribution-unit fields; converting several packages from one upstream project in one run records a shared distribution-unit id and version queryable from the packaged artifacts (surface decided in the work); a pkgIndex.tcl construct outside the converter's modelled class produces an explicit refusal message naming the construct; converter behaviour is covered by a testsuite under src/tests/modules/punk/mix/.

Context

punkshell prefers .tm modules; the wider Tcl ecosystem mostly ships pkgIndex.tcl libraries and may continue to. Zip-based .tm modules get single-file distribution and can carry license/readme/metadata inside the artifact - which also makes them the natural payload for an artifact server (G-067). dev lib.copyasmodule already performs a basic conversion and has worked on some tcllib modules, but pkgIndex.tcl scripts are arbitrary Tcl and the current handling is narrow. Multi-package upstreams introduce the unit problem: once tcllib (or similar) packages are split into individual .tm files, nothing records that they came from one release and should be upgraded together - no current goal touches this.

Approach

• Grow lib.copyasmodule's modelled class of pkgIndex.tcl scripts incrementally (ifneeded lines with source/load lists, simple computed versions), with an explicit refusal path for everything else - correctness over coverage.
• Metadata datafile format: toml, schema shared with / derived from the G-065 manifest vocabulary so vendored-in-place and repackaged artifacts describe themselves consistently.
• Distribution-unit: unit id = upstream project identity (e.g. "tcllib"), unit version = upstream release; recorded per artifact and aggregable ("what units are present, are any mixed-version"). Upgrade-together enforcement is a consumer concern (G-065 sync / G-067 retrieval) - this goal only guarantees the data exists.
• Loading on 8.6: zip-based .tm viability in the shell code interp is G-034's subject; this goal records the limitation rather than solving mounting.

Notes

• Related: G-034 (zip modpod mounting on 8.6), G-035 (mixed .tm/pkgIndex provision characterization - what happens when both shapes of the same package are present), G-063 (license fields in the datafile), G-065 (manifest schema sharing), G-067 (repackaged artifacts as the publish payload).
• 2026-07-21 (user direction: support binary-bearing zip-based .tm as single-file downloadable libs, and generate them for punkbin). CHARACTERIZED the binary-load path of a zip-based .tm (modpod) on windows tcl 9.0.5 - specimen: a #modpod-mybintest-0.1 wrapping a real tcl9thread307.dll, built via modpod::lib::make_zip_modpod. Findings: the .tm self-mounts (its prepended stub runs tcl::zipfs::mount //zipfs:/#modpod/...#mounted-modpod--), then the inner binary 'load's from that zipfs mount and EXTRACTS to %TEMP%\TCL<dll> - the SAME copy-to-temp machinery as a direct-from-kit dll load, so the TIP-741 non-cleanup/accumulation flaw applies equally (one temp dir per binary, persists on exit; see G-103 Notes). Three loading locations all converge on this one mechanism: (1) dll direct in a kit's zipfs, (2) standalone .tm on disk, (3) .tm nested INSIDE a kit's zipfs - the nested (zip-in-zip) mount works IN PLACE with NO pre-extraction of the .tm itself; only the dll load hits temp. Note: LOADING a modpod .tm needs only zipfs (the stub is self-contained - no punk/modpod package), so a single-name modpod .tm is plain-tclsh loadable.
• 2026-07-21 (user): MULTI-package-require-name modpod .tm (case variants like Thread/thread, or sub-packages from one file) needs a manifest at a well-known internal location discovered by libunknown during module scanning - NOT YET IMPLEMENTED. Accepted consequence: a multi-name .tm is punkshell/libunknown-only, NOT loadable in plain tclsh - fine PROVIDED the repo system/docs indicate this clearly. TIP 590 ('recommend lowercase package names') makes the SINGLE-lowercase- name path the happy case (plain-tclsh loadable, no libunknown, no Thread/thread duplication) - the recommended shape for a downloadable single-file binary lib when one require name suffices. New/complex mechanisms are acceptable in punkshell IFF backed by extensive tests. Ties to G-035 (mixed .tm/pkgIndex discovery), G-034 (modpod mount path), G-067 (punkbin publish of these artifacts). UPDATE 2026-07-21: the multi-name mechanism is now its own goal - G-109 (manifest-declared multi-name discovery, with the scan-performance parity target); the extraction-path question spun into G-110 (content-addressed shared-lib cache investigation).