punkshell/goals/G-110-sharedlib-extraction-cache.md

G-110 Shared-lib extraction cache: single content-addressed user cache vs per-run temp copies (investigation)

Status: proposed Scope: investigation + decision record (goals/G-110-sharedlib-extraction-cache.md); prototype surface: punk/modpod-side loaders (modpod stub, punk boot loading - no core change required); src/buildsuites/suite_tcl90/patches/ (candidate zipfs-load core patch if adopted); TIP 741/709 relationship record Goal: a recorded determination of whether Tcl's per-run fresh-temp-dir copies when loading shared libs from zip-based storage are necessary, or replaceable by a single per-user semi-persistent content-addressed cache (parent dir keyed by the lib's content hash) - with a working prototype if viable - so accumulation is bounded to one copy per distinct binary (vs the characterized 15k+ per-run dirs) and cache-hit loads skip extraction entirely. Acceptance: the necessity/safety questions are answered with recorded evidence (concurrent multi-process sharing of one copy; classic vs POSIX-semantics delete-while-mapped; atomic populate under racing first extractions; verify-on-use policy for pre-existing entries; sibling-dll sets; GC policy; AV/permissions interplay), incorporating the 2026-07-21 sniff-test results; a punk/modpod-side prototype (no core change) demonstrates a second run reusing the cached copy with no new extraction and two racing first-runs producing one valid entry with both processes loading; the relationship to TIP 741 (delayed delete), TIP 709 (no extraction), and the third option the sniff surfaced (immediate POSIX-semantics unlink after load on modern NTFS) is recorded; a decision lands - adopt with named follow-ons (modpod stub, punk loaders, candidate suite core patch / upstream TIP) or reject with reasons.

Context

Drafted 2026-07-21 from the TIP-741 characterization session. Today's mechanism (characterized on the suite's 9.0.5 runtime - G-103/G-066 Notes): every 'load' of a binary from zip-based storage (kit zipfs or modpod .tm) extracts to a FRESH %TEMP%\TCL\ dir, never cleaned up - 15,596 accumulated dirs (~5 months of kit runs) were observed and reclaimed on this machine. The user's question: are there good reasons (locking/safety) for multiple copies, or could it be a single copy in a user-owned semi-persistent cache dir with the parent folder named by the lib's content hash?

SNIFF-TEST EVIDENCE (2026-07-21, suite tclsh90s.exe 9.0.5, Win11 NTFS):

• concurrent sharing WORKS: two processes loaded the same cached dll file simultaneously (both got Thread 3.0.7) - LoadLibrary takes no exclusive lock (system dlls are shared this way constantly);
• classic Win32 delete (cmd del) while mapped: REFUSED (Access is denied) - this is the lock people remember; it blocks delete-on-exit schemes, not sharing;
• POSIX-semantics delete (msys rm) while mapped: SUCCEEDED - the name vanished immediately and the holder process kept running off the deleted mapping (thread::id still worked at exit). Modern NTFS (Win10 1809+).

Interpretation: nothing forces multiple copies of IDENTICAL content. The fresh-dir-per-run design avoids same-path-different-content collisions (a classically-loaded dll cannot be overwritten) and gets uniqueness without coordination - content-hash keying eliminates that collision class by construction (same content = same path = reuse; different content = different path). A backwards-compat reading (user 2026-07-21): core Tcl maintains compatibility with old Windows - some distributors still release win32 variants - and on old Windows (pre-1809, non-NTFS) delete-while-mapped is impossible and collision management is harder, so fresh dirs are the lowest-common-denominator-safe choice.

FALLBACK REQUIREMENT (user 2026-07-21): punkshell wants a fallback for a possible win32 release - industrial systems commonly run old Windows in isolated VMs or on old hardware. Any adopted design must capability-probe and degrade gracefully where POSIX-semantics delete / modern NTFS behaviour is unavailable: the content-hash cache itself needs neither (it never overwrites a live path), but GC and any unlink-after-load trick must fall back to classic-safe behaviour (skip in-use entries / delayed cleanup).

Design sketch to evaluate: cache root per user (candidates: ~/.punkshell/dllcache aligning with the PUNK_FOSSIL_STORE precedent, or %LOCALAPPDATA%); entry = //; populate atomically (write sibling temp + rename into place; racing writers converge or discard); on hit optionally re-hash before load (verify-on-use policy - same-user trust domain, so a documented paranoia dial); sibling-dll sets (e.g. a future tcltls + libssl/libcrypto) keyed by primary hash with siblings extracted alongside - note today's mechanism does NOT handle sibling deps from zip storage at all, so the cache could fix rather than regress that; GC = age/LRU sweep, safe even against in-use entries where POSIX delete exists, skip-if-locked elsewhere.

Precedents: .NET single-file apps extract to a persistent per-app/hash dir and REUSE it (the exact model proposed); critcl's ~/.critcl per-platform cache is the in-Tcl-ecosystem precedent; PyInstaller onefile's per-run _MEI* dirs (leak on crash) are the anti-pattern matching today's behaviour. AV interplay: a stable persistent path is friendlier than a fresh random dir per run (reputation/scan caching; fresh random %TEMP% dirs are also a classic malware pattern and get rescanned every run - a per-run latency cost today).

Implementation surfaces, cheapest first: (1) modpod stub / punk-side loaders - we control those load calls, no core change needed (the prototype); (2) a suite core patch to the zipfs load path (suite_tcl90/patches/, piperepl precedent) making kit-internal battery loads use the cache; (3) upstream conversation (a third option next to TIPs 741/709).

Relationships: G-103 (family runtimes are the heaviest consumer - every kit battery load), G-101 (Metakit 8.6 kits share the copy-to-temp shape), G-066/G-067/G-109 (binary modpod .tm artifacts), G-105/G-060 (per-platform characterization of the existing behaviour), Tcl TIPs 741/709.

Notes

• 2026-07-21: drafted and approved (user: "approved - apply both"); win32/ old-Windows fallback requirement added from the approval message.
• 2026-07-21 in-house precedent: tarjar - modpod's tar-archive-based predecessor - had a shared .TclSoCache for dlls, and THE CODE SURVIVES IN-TREE (an earlier note claimed otherwise from a head-truncated grep - corrected after the user spotted .TclSoCache inside the vendored tm): src/vendormodules/tarjar-2.4.3.tm (~L746 check_or_make_cached_sofile, ~L834 cachedir), _aside/tarjar-2.3.tm, src/modules/_build/tarjar-2.3.tm
  • the unwrapped #tarjar-tarjar-2.3 loadscript, and at c:/tclmodules/main (which also holds the intermediate 'tarpack' generation, tarpack-1.1.x). Its actual mechanism, worth recording as the design contrast: per-user ~/.TclSoCache/<in-archive-dir>/<tail> - keyed by the file's IN-ARCHIVE RELATIVE PATH (not content, not origin archive); populate-if-missing with mtime copied from the packaged file; staleness check = size+mtime compare that only WARNS on mismatch and loads the stale cached copy anyway ("remove $sofile and retry"); concurrency = retry-once-after-100ms. The path-key collision class (two archives shipping same-named files; upgraded packages silently loading stale binaries) is precisely what content-hash keying eliminates - tarjar is prior art for the cache-location/persistence half and a demonstration of why the keying half matters. Old tarjar material at c:/tclmodules/main is preserve/partly-migrate, handled MANUALLY by the user (explicitly not a goal); tarjar artifacts elsewhere are not free-to-delete (see the modpod-tidy goal when it lands).