From 534bc58dc397a997e70774113f93c9d3890ab03c Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Tue, 21 Jul 2026 15:53:20 +1000 Subject: [PATCH] G-109 + G-110 proposed: multi-name .tm discovery; shared-lib extraction cache Two goals split out of the modpod/TIP-741 characterization work, user-approved. G-109 (libunknown manifest-declared multi-name discovery): a zip-based .tm declares case-variant and sub-package require names in a manifest at a well-known internal location, discovered by libunknown at scan time - with real-disk scan performance preserved (worst-case parity with plain tclsh's builtin tm handler; libunknown exists to fix slow-drive/large-tree scan cost). Multi-name .tm are punkshell-only (documented); TIP 590 lowercase single-name stays the plain-tclsh happy path; extensive-tests bar recorded. G-110 (extraction cache investigation): whether per-run fresh-temp-dir copies are necessary or replaceable by a single per-user content-addressed cache (hash-keyed parent dir). Carries the 2026-07-21 sniff evidence: concurrent sharing of one copy works; classic Win32 delete refused while mapped; POSIX- semantics delete succeeds with mappings unharmed (a third option next to TIPs 741/709: unlink-after-load on modern NTFS). Fallback requirement recorded: punkshell wants a win32/old-Windows fallback (industrial systems on isolated VMs/old hardware) - capability-probe, classic-safe GC degradation. Precedents (.NET single-file, critcl cache), sibling-dll sets, AV interplay, and the cheapest-first implementation surfaces (modpod stub -> suite core patch -> upstream) recorded. G-066 note pointed at the new goals. Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- GOALS.md | 8 ++ goals/G-066-pkgindex-tm-repackaging.md | 4 + .../G-109-libunknown-manifest-multiname-tm.md | 57 ++++++++++++ goals/G-110-sharedlib-extraction-cache.md | 86 +++++++++++++++++++ 4 files changed, 155 insertions(+) create mode 100644 goals/G-109-libunknown-manifest-multiname-tm.md create mode 100644 goals/G-110-sharedlib-extraction-cache.md diff --git a/GOALS.md b/GOALS.md index f5d03052..dd204111 100644 --- a/GOALS.md +++ b/GOALS.md @@ -365,3 +365,11 @@ Detail: goals/G-106-powershell-consolemode-fallback.md ### G-108 [proposed] Debug buildsuite tier: tcl::test-enabled runtimes (-dbg) with recorded constrained-test runs Scope: src/buildsuites/suite_tcl90/ (build905.zig debug knobs + tcl::test wiring, suite.tcl debug variant steps, tools/test_gate.tcl record runs); kit-family debug members coordinate with G-103 Detail: goals/G-108-buildsuite-debug-tier.md + +### G-109 [proposed] libunknown manifest-declared multi-name discovery for zip-based .tm modules +Scope: src/modules/punk/libunknown-0.2.1.tm (scan/registration), modpod (manifest emission + load stub; canonical source c:/repo/jn/tclmodules/modpod, vendored copies as consumed), src/modules/punk/mix/ (modpod wrap tooling as touched), src/tests/modules/punk/libunknown/ (new suites + scan-performance characterization) +Detail: goals/G-109-libunknown-manifest-multiname-tm.md + +### G-110 [proposed] Shared-lib extraction cache: single content-addressed user cache vs per-run temp copies (investigation) +Scope: investigation + decision record (goals/G-110-sharedlib-extraction-cache.md); prototype surface: punk/modpod-side loaders (modpod stub, punk boot loading - no core change required); src/buildsuites/suite_tcl90/patches/ (candidate zipfs-load core patch if adopted); TIP 741/709 relationship record +Detail: goals/G-110-sharedlib-extraction-cache.md diff --git a/goals/G-066-pkgindex-tm-repackaging.md b/goals/G-066-pkgindex-tm-repackaging.md index 8da01b06..8be94044 100644 --- a/goals/G-066-pkgindex-tm-repackaging.md +++ b/goals/G-066-pkgindex-tm-repackaging.md @@ -64,3 +64,7 @@ current goal touches this. when one require name suffices. New/complex mechanisms are acceptable in punkshell IFF backed by extensive tests. Ties to G-035 (mixed .tm/pkgIndex discovery), G-034 (modpod mount path), G-067 (punkbin publish of these artifacts). + UPDATE 2026-07-21: the multi-name mechanism is now its own goal - G-109 + (manifest-declared multi-name discovery, with the scan-performance parity + target); the extraction-path question spun into G-110 (content-addressed + shared-lib cache investigation). diff --git a/goals/G-109-libunknown-manifest-multiname-tm.md b/goals/G-109-libunknown-manifest-multiname-tm.md new file mode 100644 index 00000000..44fff8c8 --- /dev/null +++ b/goals/G-109-libunknown-manifest-multiname-tm.md @@ -0,0 +1,57 @@ +# G-109 libunknown manifest-declared multi-name discovery for zip-based .tm modules + +Status: proposed +Scope: src/modules/punk/libunknown-0.2.1.tm (scan/registration), modpod (manifest emission + load stub; canonical source c:/repo/jn/tclmodules/modpod, vendored copies as consumed), src/modules/punk/mix/ (modpod wrap tooling as touched), src/tests/modules/punk/libunknown/ (new suites + scan-performance characterization) +Goal: a zip-based .tm (modpod) can declare additional package-require names - case variants (Thread/thread) and sub-packages - in a manifest at a well-known internal location, and punkshell's libunknown discovers those declarations during module scanning and registers them so package require of ANY declared name loads the single .tm file; real-disk scan performance is preserved: libunknown exists to fix slow-drive/large-multi-path-tree scan cost, so with manifest support enabled the worst-case target is parity with plain tclsh's builtin tm handler on the same tree (zipfs trees remain the fast path). +Acceptance: a demonstration modpod .tm declaring a case-variant name and a sub-package name resolves via package require of each declared name on the tcl9 kit (8.6 covered, or its limitation recorded per G-034), from both a real-disk module path and a zipfs module path; the manifest location and format are documented and emitted by the modpod wrap tooling; plain-tclsh behaviour is documented prominently (the filename-derived name loads anywhere zipfs exists; extra names are punkshell/libunknown-only); a scan-performance characterization test compares libunknown-with-manifest-support against plain tclsh's builtin handler over a generated large multi-path real-disk tree and records the numbers in this file showing no worse than parity; extensive suites under src/tests/modules/punk/libunknown/ cover registration, case-variant and sub-package loading, refresh/epoch staleness, and the no-manifest fast path. + +## Context + +Drafted 2026-07-21 from the modpod binary-.tm characterization session (see +G-066 Notes for the characterization: self-mounting zip .tm verified working, +including nested inside a kit's zipfs; the multi-name mechanism was identified +there as NOT YET IMPLEMENTED and split out to this goal at user request). + +Why multi-name: a single-file binary-bearing .tm can only advertise the one +package name its filename encodes under plain tm scanning. Case variants +(thread's dll provides both 'Thread' and 'thread') would otherwise need the +.tm duplicated per name - wasteful and drift-prone - and a library shipping +sub-packages (foo, foo::bar) cannot expose them from one file at all. A +manifest at a well-known internal location, read at scan time by punkshell's +libunknown, lets one .tm register every name it provides. + +Accepted trade (user 2026-07-21): multi-name .tm are punkshell/libunknown-only +- plain tclsh will not discover the extra names (it has no manifest reader). +That is acceptable PROVIDED repo docs indicate it clearly. TIP 590 ('recommend +lowercase package names') makes the single-lowercase-name modpod the +plain-tclsh-loadable happy path (the self-mount stub needs only zipfs); the +manifest mechanism is for when one require name is not enough. New/complex +mechanisms are acceptable in punkshell IFF backed by extensive tests (user +bar, same session). + +Performance frame (user 2026-07-21): libunknown was designed to avoid +performance problems that primarily occurred on a machine with slower drives +and large os-level package paths (multiple tm paths, many packages). Scanning +zipfs trees is much faster; real-disk scans must stay performant. Worst-case +target: parity with plain tclsh's builtin tm handler on the same tree. Design +consequences to settle in the work: + +- reading a manifest inside a zip .tm without mounting/extracting every + candidate: zip end-of-central-directory tail read vs zipfs mount per file; + gate the read to modpod-recognizable files (the stub prefix is detectable) + so plain .tm files cost nothing extra; +- cache manifest results keyed by file identity (mtime/size) under the + existing libunknown epoch/refresh machinery (lib.search -refresh semantics) + so repeat scans skip re-reads; +- registration shape: package ifneeded entries for each declared name mapping + to the one .tm (source it - the stub self-mounts and provides). + +Relationships: G-066 (modpod generation + embedded metadata - the manifest +may share its schema/datafile), G-067 (punkbin publish of these artifacts), +G-035 (mixed .tm/pkgIndex provision characterization), G-034 (8.6 modpod +mount path - the 8.6 leg of acceptance), TIP 590 (lowercase naming), G-110 +(the binary inside the .tm loads via the extraction path investigated there). + +## Notes + +- 2026-07-21: drafted and approved (user: "approved - apply both"). diff --git a/goals/G-110-sharedlib-extraction-cache.md b/goals/G-110-sharedlib-extraction-cache.md new file mode 100644 index 00000000..7bfbd233 --- /dev/null +++ b/goals/G-110-sharedlib-extraction-cache.md @@ -0,0 +1,86 @@ +# G-110 Shared-lib extraction cache: single content-addressed user cache vs per-run temp copies (investigation) + +Status: proposed +Scope: investigation + decision record (goals/G-110-sharedlib-extraction-cache.md); prototype surface: punk/modpod-side loaders (modpod stub, punk boot loading - no core change required); src/buildsuites/suite_tcl90/patches/ (candidate zipfs-load core patch if adopted); TIP 741/709 relationship record +Goal: a recorded determination of whether Tcl's per-run fresh-temp-dir copies when loading shared libs from zip-based storage are necessary, or replaceable by a single per-user semi-persistent content-addressed cache (parent dir keyed by the lib's content hash) - with a working prototype if viable - so accumulation is bounded to one copy per distinct binary (vs the characterized 15k+ per-run dirs) and cache-hit loads skip extraction entirely. +Acceptance: the necessity/safety questions are answered with recorded evidence (concurrent multi-process sharing of one copy; classic vs POSIX-semantics delete-while-mapped; atomic populate under racing first extractions; verify-on-use policy for pre-existing entries; sibling-dll sets; GC policy; AV/permissions interplay), incorporating the 2026-07-21 sniff-test results; a punk/modpod-side prototype (no core change) demonstrates a second run reusing the cached copy with no new extraction and two racing first-runs producing one valid entry with both processes loading; the relationship to TIP 741 (delayed delete), TIP 709 (no extraction), and the third option the sniff surfaced (immediate POSIX-semantics unlink after load on modern NTFS) is recorded; a decision lands - adopt with named follow-ons (modpod stub, punk loaders, candidate suite core patch / upstream TIP) or reject with reasons. + +## Context + +Drafted 2026-07-21 from the TIP-741 characterization session. Today's +mechanism (characterized on the suite's 9.0.5 runtime - G-103/G-066 Notes): +every 'load' of a binary from zip-based storage (kit zipfs or modpod .tm) +extracts to a FRESH %TEMP%\TCL\ dir, never cleaned up - 15,596 +accumulated dirs (~5 months of kit runs) were observed and reclaimed on this +machine. The user's question: are there good reasons (locking/safety) for +multiple copies, or could it be a single copy in a user-owned semi-persistent +cache dir with the parent folder named by the lib's content hash? + +SNIFF-TEST EVIDENCE (2026-07-21, suite tclsh90s.exe 9.0.5, Win11 NTFS): + +- concurrent sharing WORKS: two processes loaded the same cached dll file + simultaneously (both got Thread 3.0.7) - LoadLibrary takes no exclusive + lock (system dlls are shared this way constantly); +- classic Win32 delete (cmd del) while mapped: REFUSED (Access is denied) - + this is the lock people remember; it blocks delete-on-exit schemes, not + sharing; +- POSIX-semantics delete (msys rm) while mapped: SUCCEEDED - the name + vanished immediately and the holder process kept running off the deleted + mapping (thread::id still worked at exit). Modern NTFS (Win10 1809+). + +Interpretation: nothing forces multiple copies of IDENTICAL content. The +fresh-dir-per-run design avoids same-path-different-content collisions (a +classically-loaded dll cannot be overwritten) and gets uniqueness without +coordination - content-hash keying eliminates that collision class by +construction (same content = same path = reuse; different content = +different path). A backwards-compat reading (user 2026-07-21): core Tcl +maintains compatibility with old Windows - some distributors still release +win32 variants - and on old Windows (pre-1809, non-NTFS) delete-while-mapped +is impossible and collision management is harder, so fresh dirs are the +lowest-common-denominator-safe choice. + +FALLBACK REQUIREMENT (user 2026-07-21): punkshell wants a fallback for a +possible win32 release - industrial systems commonly run old Windows in +isolated VMs or on old hardware. Any adopted design must capability-probe +and degrade gracefully where POSIX-semantics delete / modern NTFS behaviour +is unavailable: the content-hash cache itself needs neither (it never +overwrites a live path), but GC and any unlink-after-load trick must fall +back to classic-safe behaviour (skip in-use entries / delayed cleanup). + +Design sketch to evaluate: cache root per user (candidates: +~/.punkshell/dllcache aligning with the PUNK_FOSSIL_STORE precedent, or +%LOCALAPPDATA%); entry = //; populate +atomically (write sibling temp + rename into place; racing writers converge +or discard); on hit optionally re-hash before load (verify-on-use policy - +same-user trust domain, so a documented paranoia dial); sibling-dll sets +(e.g. a future tcltls + libssl/libcrypto) keyed by primary hash with +siblings extracted alongside - note today's mechanism does NOT handle +sibling deps from zip storage at all, so the cache could fix rather than +regress that; GC = age/LRU sweep, safe even against in-use entries where +POSIX delete exists, skip-if-locked elsewhere. + +Precedents: .NET single-file apps extract to a persistent per-app/hash dir +and REUSE it (the exact model proposed); critcl's ~/.critcl per-platform +cache is the in-Tcl-ecosystem precedent; PyInstaller onefile's per-run +_MEI* dirs (leak on crash) are the anti-pattern matching today's behaviour. +AV interplay: a stable persistent path is friendlier than a fresh random +dir per run (reputation/scan caching; fresh random %TEMP% dirs are also a +classic malware pattern and get rescanned every run - a per-run latency +cost today). + +Implementation surfaces, cheapest first: (1) modpod stub / punk-side +loaders - we control those load calls, no core change needed (the +prototype); (2) a suite core patch to the zipfs load path +(suite_tcl90/patches/, piperepl precedent) making kit-internal battery +loads use the cache; (3) upstream conversation (a third option next to +TIPs 741/709). + +Relationships: G-103 (family runtimes are the heaviest consumer - every kit +battery load), G-101 (Metakit 8.6 kits share the copy-to-temp shape), +G-066/G-067/G-109 (binary modpod .tm artifacts), G-105/G-060 (per-platform +characterization of the existing behaviour), Tcl TIPs 741/709. + +## Notes + +- 2026-07-21: drafted and approved (user: "approved - apply both"); win32/ + old-Windows fallback requirement added from the approval message.