Browse Source
Two goals split out of the modpod/TIP-741 characterization work, user-approved. G-109 (libunknown manifest-declared multi-name discovery): a zip-based .tm declares case-variant and sub-package require names in a manifest at a well-known internal location, discovered by libunknown at scan time - with real-disk scan performance preserved (worst-case parity with plain tclsh's builtin tm handler; libunknown exists to fix slow-drive/large-tree scan cost). Multi-name .tm are punkshell-only (documented); TIP 590 lowercase single-name stays the plain-tclsh happy path; extensive-tests bar recorded. G-110 (extraction cache investigation): whether per-run fresh-temp-dir copies are necessary or replaceable by a single per-user content-addressed cache (hash-keyed parent dir). Carries the 2026-07-21 sniff evidence: concurrent sharing of one copy works; classic Win32 delete refused while mapped; POSIX- semantics delete succeeds with mappings unharmed (a third option next to TIPs 741/709: unlink-after-load on modern NTFS). Fallback requirement recorded: punkshell wants a win32/old-Windows fallback (industrial systems on isolated VMs/old hardware) - capability-probe, classic-safe GC degradation. Precedents (.NET single-file, critcl cache), sibling-dll sets, AV interplay, and the cheapest-first implementation surfaces (modpod stub -> suite core patch -> upstream) recorded. G-066 note pointed at the new goals. Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.commaster
4 changed files with 155 additions and 0 deletions
@ -0,0 +1,57 @@
|
||||
# G-109 libunknown manifest-declared multi-name discovery for zip-based .tm modules |
||||
|
||||
Status: proposed |
||||
Scope: src/modules/punk/libunknown-0.2.1.tm (scan/registration), modpod (manifest emission + load stub; canonical source c:/repo/jn/tclmodules/modpod, vendored copies as consumed), src/modules/punk/mix/ (modpod wrap tooling as touched), src/tests/modules/punk/libunknown/ (new suites + scan-performance characterization) |
||||
Goal: a zip-based .tm (modpod) can declare additional package-require names - case variants (Thread/thread) and sub-packages - in a manifest at a well-known internal location, and punkshell's libunknown discovers those declarations during module scanning and registers them so package require of ANY declared name loads the single .tm file; real-disk scan performance is preserved: libunknown exists to fix slow-drive/large-multi-path-tree scan cost, so with manifest support enabled the worst-case target is parity with plain tclsh's builtin tm handler on the same tree (zipfs trees remain the fast path). |
||||
Acceptance: a demonstration modpod .tm declaring a case-variant name and a sub-package name resolves via package require of each declared name on the tcl9 kit (8.6 covered, or its limitation recorded per G-034), from both a real-disk module path and a zipfs module path; the manifest location and format are documented and emitted by the modpod wrap tooling; plain-tclsh behaviour is documented prominently (the filename-derived name loads anywhere zipfs exists; extra names are punkshell/libunknown-only); a scan-performance characterization test compares libunknown-with-manifest-support against plain tclsh's builtin handler over a generated large multi-path real-disk tree and records the numbers in this file showing no worse than parity; extensive suites under src/tests/modules/punk/libunknown/ cover registration, case-variant and sub-package loading, refresh/epoch staleness, and the no-manifest fast path. |
||||
|
||||
## Context |
||||
|
||||
Drafted 2026-07-21 from the modpod binary-.tm characterization session (see |
||||
G-066 Notes for the characterization: self-mounting zip .tm verified working, |
||||
including nested inside a kit's zipfs; the multi-name mechanism was identified |
||||
there as NOT YET IMPLEMENTED and split out to this goal at user request). |
||||
|
||||
Why multi-name: a single-file binary-bearing .tm can only advertise the one |
||||
package name its filename encodes under plain tm scanning. Case variants |
||||
(thread's dll provides both 'Thread' and 'thread') would otherwise need the |
||||
.tm duplicated per name - wasteful and drift-prone - and a library shipping |
||||
sub-packages (foo, foo::bar) cannot expose them from one file at all. A |
||||
manifest at a well-known internal location, read at scan time by punkshell's |
||||
libunknown, lets one .tm register every name it provides. |
||||
|
||||
Accepted trade (user 2026-07-21): multi-name .tm are punkshell/libunknown-only |
||||
- plain tclsh will not discover the extra names (it has no manifest reader). |
||||
That is acceptable PROVIDED repo docs indicate it clearly. TIP 590 ('recommend |
||||
lowercase package names') makes the single-lowercase-name modpod the |
||||
plain-tclsh-loadable happy path (the self-mount stub needs only zipfs); the |
||||
manifest mechanism is for when one require name is not enough. New/complex |
||||
mechanisms are acceptable in punkshell IFF backed by extensive tests (user |
||||
bar, same session). |
||||
|
||||
Performance frame (user 2026-07-21): libunknown was designed to avoid |
||||
performance problems that primarily occurred on a machine with slower drives |
||||
and large os-level package paths (multiple tm paths, many packages). Scanning |
||||
zipfs trees is much faster; real-disk scans must stay performant. Worst-case |
||||
target: parity with plain tclsh's builtin tm handler on the same tree. Design |
||||
consequences to settle in the work: |
||||
|
||||
- reading a manifest inside a zip .tm without mounting/extracting every |
||||
candidate: zip end-of-central-directory tail read vs zipfs mount per file; |
||||
gate the read to modpod-recognizable files (the stub prefix is detectable) |
||||
so plain .tm files cost nothing extra; |
||||
- cache manifest results keyed by file identity (mtime/size) under the |
||||
existing libunknown epoch/refresh machinery (lib.search -refresh semantics) |
||||
so repeat scans skip re-reads; |
||||
- registration shape: package ifneeded entries for each declared name mapping |
||||
to the one .tm (source it - the stub self-mounts and provides). |
||||
|
||||
Relationships: G-066 (modpod generation + embedded metadata - the manifest |
||||
may share its schema/datafile), G-067 (punkbin publish of these artifacts), |
||||
G-035 (mixed .tm/pkgIndex provision characterization), G-034 (8.6 modpod |
||||
mount path - the 8.6 leg of acceptance), TIP 590 (lowercase naming), G-110 |
||||
(the binary inside the .tm loads via the extraction path investigated there). |
||||
|
||||
## Notes |
||||
|
||||
- 2026-07-21: drafted and approved (user: "approved - apply both"). |
||||
@ -0,0 +1,86 @@
|
||||
# G-110 Shared-lib extraction cache: single content-addressed user cache vs per-run temp copies (investigation) |
||||
|
||||
Status: proposed |
||||
Scope: investigation + decision record (goals/G-110-sharedlib-extraction-cache.md); prototype surface: punk/modpod-side loaders (modpod stub, punk boot loading - no core change required); src/buildsuites/suite_tcl90/patches/ (candidate zipfs-load core patch if adopted); TIP 741/709 relationship record |
||||
Goal: a recorded determination of whether Tcl's per-run fresh-temp-dir copies when loading shared libs from zip-based storage are necessary, or replaceable by a single per-user semi-persistent content-addressed cache (parent dir keyed by the lib's content hash) - with a working prototype if viable - so accumulation is bounded to one copy per distinct binary (vs the characterized 15k+ per-run dirs) and cache-hit loads skip extraction entirely. |
||||
Acceptance: the necessity/safety questions are answered with recorded evidence (concurrent multi-process sharing of one copy; classic vs POSIX-semantics delete-while-mapped; atomic populate under racing first extractions; verify-on-use policy for pre-existing entries; sibling-dll sets; GC policy; AV/permissions interplay), incorporating the 2026-07-21 sniff-test results; a punk/modpod-side prototype (no core change) demonstrates a second run reusing the cached copy with no new extraction and two racing first-runs producing one valid entry with both processes loading; the relationship to TIP 741 (delayed delete), TIP 709 (no extraction), and the third option the sniff surfaced (immediate POSIX-semantics unlink after load on modern NTFS) is recorded; a decision lands - adopt with named follow-ons (modpod stub, punk loaders, candidate suite core patch / upstream TIP) or reject with reasons. |
||||
|
||||
## Context |
||||
|
||||
Drafted 2026-07-21 from the TIP-741 characterization session. Today's |
||||
mechanism (characterized on the suite's 9.0.5 runtime - G-103/G-066 Notes): |
||||
every 'load' of a binary from zip-based storage (kit zipfs or modpod .tm) |
||||
extracts to a FRESH %TEMP%\TCL<hex>\ dir, never cleaned up - 15,596 |
||||
accumulated dirs (~5 months of kit runs) were observed and reclaimed on this |
||||
machine. The user's question: are there good reasons (locking/safety) for |
||||
multiple copies, or could it be a single copy in a user-owned semi-persistent |
||||
cache dir with the parent folder named by the lib's content hash? |
||||
|
||||
SNIFF-TEST EVIDENCE (2026-07-21, suite tclsh90s.exe 9.0.5, Win11 NTFS): |
||||
|
||||
- concurrent sharing WORKS: two processes loaded the same cached dll file |
||||
simultaneously (both got Thread 3.0.7) - LoadLibrary takes no exclusive |
||||
lock (system dlls are shared this way constantly); |
||||
- classic Win32 delete (cmd del) while mapped: REFUSED (Access is denied) - |
||||
this is the lock people remember; it blocks delete-on-exit schemes, not |
||||
sharing; |
||||
- POSIX-semantics delete (msys rm) while mapped: SUCCEEDED - the name |
||||
vanished immediately and the holder process kept running off the deleted |
||||
mapping (thread::id still worked at exit). Modern NTFS (Win10 1809+). |
||||
|
||||
Interpretation: nothing forces multiple copies of IDENTICAL content. The |
||||
fresh-dir-per-run design avoids same-path-different-content collisions (a |
||||
classically-loaded dll cannot be overwritten) and gets uniqueness without |
||||
coordination - content-hash keying eliminates that collision class by |
||||
construction (same content = same path = reuse; different content = |
||||
different path). A backwards-compat reading (user 2026-07-21): core Tcl |
||||
maintains compatibility with old Windows - some distributors still release |
||||
win32 variants - and on old Windows (pre-1809, non-NTFS) delete-while-mapped |
||||
is impossible and collision management is harder, so fresh dirs are the |
||||
lowest-common-denominator-safe choice. |
||||
|
||||
FALLBACK REQUIREMENT (user 2026-07-21): punkshell wants a fallback for a |
||||
possible win32 release - industrial systems commonly run old Windows in |
||||
isolated VMs or on old hardware. Any adopted design must capability-probe |
||||
and degrade gracefully where POSIX-semantics delete / modern NTFS behaviour |
||||
is unavailable: the content-hash cache itself needs neither (it never |
||||
overwrites a live path), but GC and any unlink-after-load trick must fall |
||||
back to classic-safe behaviour (skip in-use entries / delayed cleanup). |
||||
|
||||
Design sketch to evaluate: cache root per user (candidates: |
||||
~/.punkshell/dllcache aligning with the PUNK_FOSSIL_STORE precedent, or |
||||
%LOCALAPPDATA%); entry = <root>/<content-hash>/<original-basename>; populate |
||||
atomically (write sibling temp + rename into place; racing writers converge |
||||
or discard); on hit optionally re-hash before load (verify-on-use policy - |
||||
same-user trust domain, so a documented paranoia dial); sibling-dll sets |
||||
(e.g. a future tcltls + libssl/libcrypto) keyed by primary hash with |
||||
siblings extracted alongside - note today's mechanism does NOT handle |
||||
sibling deps from zip storage at all, so the cache could fix rather than |
||||
regress that; GC = age/LRU sweep, safe even against in-use entries where |
||||
POSIX delete exists, skip-if-locked elsewhere. |
||||
|
||||
Precedents: .NET single-file apps extract to a persistent per-app/hash dir |
||||
and REUSE it (the exact model proposed); critcl's ~/.critcl per-platform |
||||
cache is the in-Tcl-ecosystem precedent; PyInstaller onefile's per-run |
||||
_MEI* dirs (leak on crash) are the anti-pattern matching today's behaviour. |
||||
AV interplay: a stable persistent path is friendlier than a fresh random |
||||
dir per run (reputation/scan caching; fresh random %TEMP% dirs are also a |
||||
classic malware pattern and get rescanned every run - a per-run latency |
||||
cost today). |
||||
|
||||
Implementation surfaces, cheapest first: (1) modpod stub / punk-side |
||||
loaders - we control those load calls, no core change needed (the |
||||
prototype); (2) a suite core patch to the zipfs load path |
||||
(suite_tcl90/patches/, piperepl precedent) making kit-internal battery |
||||
loads use the cache; (3) upstream conversation (a third option next to |
||||
TIPs 741/709). |
||||
|
||||
Relationships: G-103 (family runtimes are the heaviest consumer - every kit |
||||
battery load), G-101 (Metakit 8.6 kits share the copy-to-temp shape), |
||||
G-066/G-067/G-109 (binary modpod .tm artifacts), G-105/G-060 (per-platform |
||||
characterization of the existing behaviour), Tcl TIPs 741/709. |
||||
|
||||
## Notes |
||||
|
||||
- 2026-07-21: drafted and approved (user: "approved - apply both"); win32/ |
||||
old-Windows fallback requirement added from the approval message. |
||||
Loading…
Reference in new issue