punkshell/goals/G-110-sharedlib-extraction-cache.md

G-110 Shared-lib extraction cache: single content-addressed user cache vs per-run temp copies (investigation)

Status: proposed Scope: investigation + decision record (goals/G-110-sharedlib-extraction-cache.md); prototype surface: punk/modpod-side loaders (modpod stub, punk boot loading - no core change required); src/buildsuites/suite_tcl90/patches/ (candidate zipfs-load core patch if adopted); TIP 741/709 relationship record Goal: a recorded determination of whether Tcl's per-run fresh-temp-dir copies when loading shared libs from zip-based storage are necessary, or replaceable by a single per-user semi-persistent content-addressed cache (parent dir keyed by the lib's content hash) - with a working prototype if viable - so accumulation is bounded to one copy per distinct binary (vs the characterized 15k+ per-run dirs) and cache-hit loads skip extraction entirely. Acceptance: the necessity/safety questions are answered with recorded evidence (concurrent multi-process sharing of one copy; classic vs POSIX-semantics delete-while-mapped; atomic populate under racing first extractions; verify-on-use policy for pre-existing entries; sibling-dll sets; GC policy; AV/permissions interplay), incorporating the 2026-07-21 sniff-test results; a punk/modpod-side prototype (no core change) demonstrates a second run reusing the cached copy with no new extraction and two racing first-runs producing one valid entry with both processes loading; the relationship to TIP 741 (delayed delete), TIP 709 (no extraction), and the third option the sniff surfaced (immediate POSIX-semantics unlink after load on modern NTFS) is recorded; a decision lands - adopt with named follow-ons (modpod stub, punk loaders, candidate suite core patch / upstream TIP) or reject with reasons.

Context

Drafted 2026-07-21 from the TIP-741 characterization session. Today's mechanism (characterized on the suite's 9.0.5 runtime - G-103/G-066 Notes): every 'load' of a binary from zip-based storage (kit zipfs or modpod .tm) extracts to a FRESH %TEMP%\TCL\ dir, never cleaned up - 15,596 accumulated dirs (~5 months of kit runs) were observed and reclaimed on this machine. The user's question: are there good reasons (locking/safety) for multiple copies, or could it be a single copy in a user-owned semi-persistent cache dir with the parent folder named by the lib's content hash?

SNIFF-TEST EVIDENCE (2026-07-21, suite tclsh90s.exe 9.0.5, Win11 NTFS):

• concurrent sharing WORKS: two processes loaded the same cached dll file simultaneously (both got Thread 3.0.7) - LoadLibrary takes no exclusive lock (system dlls are shared this way constantly);
• classic Win32 delete (cmd del) while mapped: REFUSED (Access is denied) - this is the lock people remember; it blocks delete-on-exit schemes, not sharing;
• POSIX-semantics delete (msys rm) while mapped: SUCCEEDED - the name vanished immediately and the holder process kept running off the deleted mapping (thread::id still worked at exit). Modern NTFS (Win10 1809+).

Interpretation: nothing forces multiple copies of IDENTICAL content. The fresh-dir-per-run design avoids same-path-different-content collisions (a classically-loaded dll cannot be overwritten) and gets uniqueness without coordination - content-hash keying eliminates that collision class by construction (same content = same path = reuse; different content = different path). A backwards-compat reading (user 2026-07-21): core Tcl maintains compatibility with old Windows - some distributors still release win32 variants - and on old Windows (pre-1809, non-NTFS) delete-while-mapped is impossible and collision management is harder, so fresh dirs are the lowest-common-denominator-safe choice.

FALLBACK REQUIREMENT (user 2026-07-21): punkshell wants a fallback for a possible win32 release - industrial systems commonly run old Windows in isolated VMs or on old hardware. Any adopted design must capability-probe and degrade gracefully where POSIX-semantics delete / modern NTFS behaviour is unavailable: the content-hash cache itself needs neither (it never overwrites a live path), but GC and any unlink-after-load trick must fall back to classic-safe behaviour (skip in-use entries / delayed cleanup).

Design sketch to evaluate: cache root per user (candidates: ~/.punkshell/dllcache aligning with the PUNK_FOSSIL_STORE precedent, or %LOCALAPPDATA%); entry = //; populate atomically (write sibling temp + rename into place; racing writers converge or discard); on hit optionally re-hash before load (verify-on-use policy - same-user trust domain, so a documented paranoia dial); sibling-dll sets (e.g. a future tcltls + libssl/libcrypto) keyed by primary hash with siblings extracted alongside - note today's mechanism does NOT handle sibling deps from zip storage at all, so the cache could fix rather than regress that; GC = age/LRU sweep, safe even against in-use entries where POSIX delete exists, skip-if-locked elsewhere.

Precedents: .NET single-file apps extract to a persistent per-app/hash dir and REUSE it (the exact model proposed); critcl's ~/.critcl per-platform cache is the in-Tcl-ecosystem precedent; PyInstaller onefile's per-run _MEI* dirs (leak on crash) are the anti-pattern matching today's behaviour. AV interplay: a stable persistent path is friendlier than a fresh random dir per run (reputation/scan caching; fresh random %TEMP% dirs are also a classic malware pattern and get rescanned every run - a per-run latency cost today).

Implementation surfaces, cheapest first: (1) modpod stub / punk-side loaders - we control those load calls, no core change needed (the prototype); (2) a suite core patch to the zipfs load path (suite_tcl90/patches/, piperepl precedent) making kit-internal battery loads use the cache; (3) upstream conversation (a third option next to TIPs 741/709).

Relationships: G-103 (family runtimes are the heaviest consumer - every kit battery load), G-101 (Metakit 8.6 kits share the copy-to-temp shape), G-066/G-067/G-109 (binary modpod .tm artifacts), G-105/G-060 (per-platform characterization of the existing behaviour), Tcl TIPs 741/709.

Notes

• 2026-07-21: drafted and approved (user: "approved - apply both"); win32/ old-Windows fallback requirement added from the approval message.
• 2026-07-21 in-house precedent (user recollection): tarjar - modpod's tar-archive-based predecessor - had a shared .tclsocache for dlls, though without content-hash keying or copy-out collision handling. No .tclsocache code survives in any current tree (verified by grep across the modpod project and shellspy src), so it stands as prior art for the concept rather than reusable code. Old tarjar material lives at c:/tclmodules/main - the user intends to preserve and partly migrate it MANUALLY (explicitly not a goal); tarjar artifacts elsewhere are not free-to-delete (see the modpod-tidy goal when it lands).