From d3cc914a52a4432d921f97288965a5cdc9e5f9b8 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Mon, 20 Jul 2026 14:29:56 +1000 Subject: [PATCH] punk::libunknown epoch-copy unsoundness fix - tkconsole repl failure root-caused (G-098) Copying ::punk::libunknown::epoch verbatim into a fresh interp/thread carries per-epoch 'added' maps that drive zipfs_tclPkgUnknown's can_skip_sourcing short-circuit - sound only in the interp that performed the scans. Packages the donor obtained at an earlier epoch (or with no unknown call: Thread is baked into thread::create'd threads) are absent from the donor's current-epoch map, so recipients skip their pkgIndex sourcing and requires fail where stock tclPkgUnknown would succeed. Bit deterministically as repl_console_backend_tkconsole under the suite 9.0.5 runtime: 'package require Tk' bumps the epoch pre-init, the codethread inherits the post-bump sparse map, the code interp then can't find vfs/Thread/registry, its init aborts before the G-001 conredirect junction vars exist, and the first pending-collect error left the repl reader inactive until the driver failsafe. Tcl903 escaped only by an accidental extra epoch bump in its code interp (fresh map, full rescan). Fix: punk::lib::libunknown_epoch_export (PUNKARGS-documented) - epoch record with every 'added' map emptied; counters/untracked/scripts/index globs preserved. Used at all four propagation sites (interp_sync, snapshot, repl %lib_epoch%, repl safe-arm). Hardening: the repl pending-collect now tolerates missing junction vars so a code-interp init failure can't wedge the reader. Also: test socketpairs pinned to -myaddr 127.0.0.1 (all-interfaces listens raised windows firewall consent popups for each newly built test exe - user datapoint). Verification: consolebackends 3/3 both runtimes (suite 42s->14s); full runtests sweeps identical on suite runtime and Tcl903: 1037/1021/15/1 with the sole failure exec-14.3 (documented machine baseline). G-098 acceptance sweep met. Pre-existing PUNK_TEST_TK opt-in quirk noted in the goal (silent early end, both runtimes, out of scope). Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- goals/G-098-suite-tk-tcllib-critcl.md | 49 ++++++++++++++++-- src/modules/punk/lib-999999.0a1.0.tm | 51 ++++++++++++++++++- src/modules/punk/repl-999999.0a1.0.tm | 20 ++++++-- .../console/testsuites/console/backends.test | 4 +- src/tests/testsupport/repl_console_driver.tcl | 4 +- 5 files changed, 115 insertions(+), 13 deletions(-) diff --git a/goals/G-098-suite-tk-tcllib-critcl.md b/goals/G-098-suite-tk-tcllib-critcl.md index 89e6f044..96dc4368 100644 --- a/goals/G-098-suite-tk-tcllib-critcl.md +++ b/goals/G-098-suite-tk-tcllib-critcl.md @@ -188,10 +188,51 @@ user direction 2026-07-20: configured for building from mapvfs.config. Also planned: move at least tclvfs + thread dlls into the zip attached to the runtime exe itself rather than the .vfs. -Remaining for acceptance: - -- Diagnose/fix repl_console_backend_tkconsole under the suite runtime, then a full - punkshell runtests sweep on it (Tk suites unskipped) recorded here. +### 2026-07-20 increment 6: tkconsole root-caused and fixed (punk::libunknown epoch-copy unsoundness); full-sweep PARITY with Tcl903 + +- repl_console_backend_tkconsole failure root cause (NOT a suite recipe defect - a + latent punkshell bug the suite runtime's epoch phasing exposed deterministically): + copying ::punk::libunknown::epoch VERBATIM into a fresh interp/thread carries the + per-epoch per-searchpath 'added' maps that drive zipfs_tclPkgUnknown's + can_skip_sourcing short-circuit. Those maps are only sound in the interp that did + the scans: packages the donor obtained at an EARLIER epoch or with no unknown call + at all (Thread baked into thread::create'd threads) are absent from the donor's + CURRENT epoch map, so the recipient skips their pkgIndex sourcing and the require + fails where stock tclPkgUnknown would succeed. Trigger chain in tk mode: parent + 'package require Tk' bumps the epoch (both runtimes do this) -> codethread pastes + the post-bump sparse dict -> code interp inherits it -> vfs/Thread fail -> + punk::console absent -> 'registry' require inside punk's auto_execok chain errors + the code-interp init BEFORE the G-001 conredirect junction vars are created -> + first prompt's pending-collect reads a missing var -> repl reader left inactive -> + driver timeout-failsafe. Tcl903 escaped by luck: its code interp got an extra + epoch bump to a fresh epoch (no inherited added map -> full first-scan). Suite + test/ssh modes escaped by inheriting the pre-bump comprehensive map. +- FIX: new punk::lib::libunknown_epoch_export (PUNKARGS-documented) returns the + epoch record with every per-epoch 'added' map emptied - counters, untracked, + scripts and index-glob caches preserved; recipients do their own first sourcing + pass per searchpath. Applied at all four propagation sites: punk::lib + interp_sync_package_paths + snapshot_package_paths, punk::repl init_script + %lib_epoch% + the safe-arm direct epoch copy. +- HARDENING: repl_process_data's foreign-console pending-collect now guards for the + junction vars' existence (init aborted before junction install => treat as no + pending output) so a code-interp init failure can no longer wedge the repl reader. +- Windows firewall popups during punk91/punk905_beta test runs (user datapoint): + test socketpairs listened on all interfaces - each newly built exe prompts. + Pinned to -myaddr 127.0.0.1 (loopback binds don't prompt) in + repl_console_driver.tcl + opunk backends.test socketpair. Residual: tcludp-based + PUNK_TEST_UDPTEE teelog binds remain INADDR_ANY (tcludp has no myaddr control). +- VERIFICATION: consolebackends 3/3 on suite runtime (42s->14s, failsafe gone) and + 3/3 on Tcl903; FULL runtests sweeps IDENTICAL on both runtimes: Total 1037 / + Passed 1021 / Skipped 15 / Failed 1 = exec-14.3 (documented machine baseline), + same skip census. Tk-dependent child-process suites run unskipped on the suite + runtime. ACCEPTANCE SWEEP MET. +- Noted pre-existing quirk (both runtimes, out of scope): PUNK_TEST_TK=1 opt-in + (backends.test Tk-in-testinterp widget test) silently ends the runtests run right + after the file header - exit 0, no tally. Candidate small follow-up. +- Deployed punk905_beta.exe still carries the pre-fix repl/lib modules (kit payload + built from _vfscommon before this increment) - unaffected for default-console + interactive use (collect path is foreign-console only); picks the fix up at the + next vfscommonupdate + project cycle. - (superseded note kept for history below: the clone store subsequently moved from the suite-owned _build location to the machine-level ~/.punkshell/fossils - see increment 1 and the suite README for current truth) diff --git a/src/modules/punk/lib-999999.0a1.0.tm b/src/modules/punk/lib-999999.0a1.0.tm index 613f81ac..cc3a9836 100644 --- a/src/modules/punk/lib-999999.0a1.0.tm +++ b/src/modules/punk/lib-999999.0a1.0.tm @@ -8415,7 +8415,9 @@ namespace eval punk::lib { if {$do_libunknown} { if {[info exists ::punk::libunknown::epoch]} { append script "namespace eval ::punk::libunknown {}\n" - append script "set ::punk::libunknown::epoch [list $::punk::libunknown::epoch]\n" + #reduced copy - a verbatim copy carries 'added' short-circuit state that is + #only sound in the donor interp (see libunknown_epoch_export) + append script "set ::punk::libunknown::epoch [list [libunknown_epoch_export]]\n" } #find and source libunknown, then call init append script { @@ -8541,7 +8543,9 @@ namespace eval punk::lib { if {$do_libunknown} { if {[info exists ::punk::libunknown::epoch]} { interp eval $interp {namespace eval ::punk::libunknown {}} - interp eval $interp [list set ::punk::libunknown::epoch $::punk::libunknown::epoch] + #reduced copy - a verbatim copy carries 'added' short-circuit state that is + #only sound in the donor interp (see libunknown_epoch_export) + interp eval $interp [list set ::punk::libunknown::epoch [libunknown_epoch_export]] } #use snapshot's libunknown sourcing logic via interp eval set libunknown_script [punk::lib::snapshot_package_paths -libunknown 1] @@ -8581,6 +8585,49 @@ namespace eval punk::lib { } } + namespace eval argdoc { + variable PUNKARGS + lappend PUNKARGS [list { + @id -id ::punk::lib::libunknown_epoch_export + @cmd -name punk::lib::libunknown_epoch_export\ + -summary\ + "Reduced copy of the punk::libunknown epoch record, safe to seed into a fresh interp or thread."\ + -help\ + "Returns ::punk::libunknown::epoch with every per-epoch 'added' map emptied. + + The 'added' maps record which package names an interp's own scans registered from + each searchpath during each epoch - zipfs_tclPkgUnknown / zipfs_tm_UnknownHandler + consult the CURRENT epoch's map to skip re-sourcing a searchpath's indexes for a + name the path didn't provide before. That short-circuit is only sound in the interp + that performed the scans: packages the donor obtained at an EARLIER epoch, or with + no package unknown call at all (e.g Thread baked into a thread::create'd thread), + are absent from the donor's current-epoch map, so a fresh interp inheriting the + record verbatim skips their pkgIndex/tm sourcing and fails requires that stock + Tcl's tclPkgUnknown would satisfy (G-098 tkconsole diagnosis 2026-07-20). + + Epoch counters, untracked markers, ifneeded-script records and index-glob caches + are preserved; the recipient performs its own first sourcing pass per searchpath + and accumulates its own 'added' state. Errors if ::punk::libunknown::epoch is not + set - callers that tolerate an uninitialised libunknown should guard with + info exists as they would for the raw variable." + @values -min 0 -max 0 + }] + } + proc libunknown_epoch_export {} { + if {![info exists ::punk::libunknown::epoch]} { + error "punk::lib::libunknown_epoch_export - ::punk::libunknown::epoch not set (punk::libunknown not initialised in this interp?)" + } + set e $::punk::libunknown::epoch + foreach domain {tm pkg} { + if {![dict exists $e $domain epochs]} {continue} + dict for {ep state} [dict get $e $domain epochs] { + dict set state added [dict create] + dict set e $domain epochs $ep $state + } + } + return $e + } + proc valcopy {obj} { append obj2 $obj {} } diff --git a/src/modules/punk/repl-999999.0a1.0.tm b/src/modules/punk/repl-999999.0a1.0.tm index 812cc132..1b713b12 100644 --- a/src/modules/punk/repl-999999.0a1.0.tm +++ b/src/modules/punk/repl-999999.0a1.0.tm @@ -2946,11 +2946,19 @@ proc repl::repl_process_data {inputchan chunktype chunk stdinlines prompt_config #to the selected console's channels (rputs maps stdout/stderr to conout/conerr). #Interleaving between the two streams within a run is not preserved (single #terminal stream; per-console err semantics are G-011). + #The vars are guarded: if the code-interp init aborted before installing the + #junction (e.g a package resolution failure) they don't exist - treat as no + #pending output rather than erroring here, which would leave the repl reader + #disabled and wedge the session (G-098 tkconsole diagnosis 2026-07-20). set pending [thread::send $codethread { interp eval code { - set _pendinglist [list $::codeinterp::console_pending_out $::codeinterp::console_pending_err] - set ::codeinterp::console_pending_out "" - set ::codeinterp::console_pending_err "" + if {[info exists ::codeinterp::console_pending_out]} { + set _pendinglist [list $::codeinterp::console_pending_out $::codeinterp::console_pending_err] + set ::codeinterp::console_pending_out "" + set ::codeinterp::console_pending_err "" + } else { + set _pendinglist [list "" ""] + } set _pendinglist } }] @@ -3540,7 +3548,7 @@ namespace eval repl { } %replthread_interp% [list $opt_callback_interp] {*}{ } %tmlist% [list [tcl::tm::list]] {*}{ } %autopath% [list $::auto_path] {*}{ - } %lib_epoch% [list $::punk::libunknown::epoch] {*}{ + } %lib_epoch% [list [punk::lib::libunknown_epoch_export]] {*}{ } %packageprefer% [list [package prefer]] {*}{ } %staticprefixes% [list [expr {[info exists ::punkboot::static_prefixes] ? $::punkboot::static_prefixes : ""}]] {*}{ } %staticpackages% [list [expr {[info exists ::punkboot::static_packages] ? $::punkboot::static_packages : ""}]] {*}{ @@ -3946,7 +3954,9 @@ namespace eval repl { catch {punk::lib::interp_sync_package_paths code} code eval [list namespace eval ::punk::libunknown {}] catch { - code eval [list set ::punk::libunknown::epoch $::punk::libunknown::epoch] + #reduced copy - 'added' short-circuit state is only sound in the donor + #interp (see punk::lib::libunknown_epoch_export) + code eval [list set ::punk::libunknown::epoch [punk::lib::libunknown_epoch_export]] } } safebase { diff --git a/src/tests/modules/opunk/console/testsuites/console/backends.test b/src/tests/modules/opunk/console/testsuites/console/backends.test index 15732a94..4f7c5c71 100644 --- a/src/tests/modules/opunk/console/testsuites/console/backends.test +++ b/src/tests/modules/opunk/console/testsuites/console/backends.test @@ -38,7 +38,9 @@ namespace eval ::testspace { proc socketpair {} { #returns {localend farend} of a connected tcp pair - set srv [socket -server {apply {{ch args} {set ::testspace::accepted $ch}}} 0] + #-myaddr 127.0.0.1: an all-interfaces listen triggers the windows firewall + #consent popup for each newly built test executable - loopback binds don't + set srv [socket -server {apply {{ch args} {set ::testspace::accepted $ch}}} -myaddr 127.0.0.1 0] set port [lindex [chan configure $srv -sockname] 2] set cli [socket 127.0.0.1 $port] vwait ::testspace::accepted diff --git a/src/tests/testsupport/repl_console_driver.tcl b/src/tests/testsupport/repl_console_driver.tcl index d3ce6b25..551b8bd8 100644 --- a/src/tests/testsupport/repl_console_driver.tcl +++ b/src/tests/testsupport/repl_console_driver.tcl @@ -52,7 +52,9 @@ foreach pkg [list punk::repl opunk::console::$backend] { } proc socketpair {} { - set srv [socket -server {apply {{ch args} {set ::accepted $ch}}} 0] + #-myaddr 127.0.0.1: an all-interfaces listen triggers the windows firewall + #consent popup for each newly built test executable - loopback binds don't + set srv [socket -server {apply {{ch args} {set ::accepted $ch}}} -myaddr 127.0.0.1 0] set port [lindex [chan configure $srv -sockname] 2] set cli [socket 127.0.0.1 $port] vwait ::accepted