# G-036 Root-cause the Tcl 9 console+udp worker-thread event-loop wedge; minimal repro for possible upstream reporting Status: achieved 2026-07-08 Scope: src/modules/shellthread-999999.0a1.0.tm, src/bootsupport/modules/shellthread-1.6.2.tm, src/modules/shellfilter-999999.0a1.0.tm (as characterised - no product-code changes required by this goal; the punkshell mitigations are separate fixes) Acceptance: (reworked 2026-07-08 after the root cause was found) the wedge mechanism is identified and written up in the detail file - DONE: bundled tcludp 1.0.12's Windows per-thread UDP_ExitProc closes the process-global synchronization events, proven by dump handle-table analysis plus a live CloseHandle breakpoint and confirmed by the upstream 1.0.12->1.0.13 diff, which already fixes it (so no upstream report is required; the standalone minimal repro originally required here is waived as moot by the user); the tcl9 kits bundle tcludp >= 1.0.13 with the in-context batch harness baseline resolved - DONE 2026-07-08 (run-2 syslog workers alive vs the 4/4-wedged 1.0.12 baseline); REMAINING: a has_bug-style detection in the punkshell check machinery (in the vein of punk::lib::check::has_tclbug_* / punk::console::check::has_bug_*, surfaced through the same reporting as 'help tcl'/'help console') reports the vulnerable combination - simple version-based detection (loaded/bundled tcludp < 1.0.13 on Tcl 9 Windows) is sufficient, no behavioural probe needed; loose-end decisions (punk8win 8.6 kit's udp 1.0.12 swap; optional upstream tickets for residual tcludp trunk weaknesses) recorded in the detail file when made. ## Context Investigated 2026-07-07/08 from a user-visible symptom: piping a small script into shell mode (`'puts test' | punk902z shell` from pwsh), pressing Enter at the first interactive prompt (after the app-layer CONIN$ restart), then typing `exit` at the dim `P%` prompt intermittently froze the shell - the `exit` had no effect, one further command (e.g. `pwd`) echoed but never ran, then echo stopped entirely and only ctrl-c killed the process. Diagnosis chain (all confirmed by timestamped instrumentation; see Notes for instrumented sites): 1. `exit`/`quit` unwind the repl completely (both teardown phases finish, `repl::start` returns `exit 0`). 2. `shellfilter::run`'s tail then calls `::shellfilter::log::close` for the `shellfilter-run` log tag -> `shellthread::manager::unsubscribe` -> a **bare synchronous `thread::send`** (settings-reset when returning the worker to the free list) to that tag's worker thread. 3. That worker had silently stopped servicing its event queue ~1-2s after birth: scheduled `after` heartbeats never fired and queued async `send_info` events were never processed. The sync send therefore blocks the main thread forever -> dead event loop -> all repl symptoms downstream (the echoed-then-dead `pwd` line is the console driver's cooked read buffering one line that nobody consumes; no further read is posted so echo stops). 4. Only `exit`/`quit` touch that worker synchronously - every other command is unaffected, matching the observed "only exit/quit lock up". `exit` at the first prompt usually wins the race because the sync send happens ~600ms after the worker's first syslog write, before the wedge sets in; the Enter-then-exit detour adds the seconds the wedge needs. The wedge itself - a worker thread event loop that stops processing timers and cross-thread sends - is the residual unknown this goal exists to resolve. The punkshell hang is mitigated separately (unsubscribe robustness + un-hardcoding the syslog default), but syslog-to-localhost must remain a *supported* configuration, so the wedge must be understood, not just avoided. ## Evidence so far (2026-07-08, Windows 11, punk902z = Tcl 9.0.2, punksys = Tcl 8.6, tcludp 1.0.12, workers load bootsupport shellthread 1.6.2) Established by heartbeat scoring (each worker schedules `after` heartbeats at init; a worker with zero heartbeats that was not legitimately torn down is wedged): - Wedged workers are exactly those configured with `-syslog 127.0.0.1:514` (they open a tcludp socket and send datagrams on first log write). Workers without syslog never wedge. - **Tcl 9 only**: hidden-console batch (4 runs per shell, judging the post-restart run-2 workers, which live on): punk902z wedged 4/4; punksys (8.6) alive 4/4. One earlier punk902z hidden-console run came up clean, so treat it as near-deterministic, not absolute. - **Console required**: the same piped run with no console attached (harness/service context) never wedges - 8s-delayed-exit runs completed cleanly with heartbeats throughout. - Nothing listens on UDP 514 locally, so every datagram draws an ICMP port-unreachable. - CPU of a hung specimen stays near zero -> the wedged threads are blocked, not spinning. - Scoring caveat: each shell's run-1 workers are torn down in the run-1 -> run-2 transition before their heartbeats come due - their hb=0 is normal, judge run-2 workers only. - Diag-file caveat: the shared append-mode diag file loses/garbles occasional lines under concurrent writers; single missing lines are not evidence, multi-line absences are. Ingredients ruled out by standalone probes (all clean under a hidden console on Tcl 9): - udp_open + datagram send in a worker thread (plain probe) - + twapi loaded with a console-control handler registered in the main thread - + the punk worker package environment (punk::packagepreference + shellthread 1.6.2) - + 30 datagrams and a later send on the ICMP-flagged socket - + CONIN$ opened in the main thread with a readable handler armed - + twapi::SetStdHandle(-10, console) before worker creation - + main thread closing stdin and ADOPTING the CONIN$ channel into the stdin std-channel slot (channel name becomes 'stdin'), SetStdHandle, and an active gets-on-readable reader armed - i.e. the same visible stdin arrangement the restarted repl has ('adopt' probe variant, 2026-07-08) Upstream reconnaissance (2026-07-08, TEMP_REFERENCE fossil checkouts: Tcl core trunk-side branch @ 2026-07-07, tcludp trunk @ 2026-07-05): - core-9-0-branch has had NO tclWinConsole.c change since 9.0.2 beyond an include-hygiene commit (2025-10-24) - the wedge, if in the console driver, is likely still present in 9.0.x maintenance. Trunk has console-channel activity by apnadkarni (2025-12-29 metakey-notification change + revert; branch apn-winconsole-noreadahead) worth reading. - Current tclWinConsole.c architecture confirms a credible wedge surface: process-global gConsoleLock, a console-handle list SHARED ACROSS ALL Tcl THREADS, per-handle SRWLOCKs, reader/writer threads coordinated via CONDITION_VARIABLEs (consoleThreadCV / interpThreadCV "for awakening interp thread(s)"), and Tcl_ThreadAlert fan-out to interested channel-owning threads. A missed CV wakeup leaves an interp thread blocked inside a console-driver call - matching the observed "blocked not spinning, timers and thread::sends unserviced". - Sharpened worker-side ingredient hypothesis: a fresh thread's std channels initialize on its FIRST channel creation (udp_open, or a -file log open) - with the process stdin handle pointing at the console (post SetStdHandle), that first channel creation attaches the worker to the shared console machinery. Fits the data exactly: run-2 workers that never create any channel (-file {} -syslog {}) survive; channel-creating ones wedge; run-1 workers (stdin handle = dead pipe at creation) are immune. NOT yet reproduced by probes, so at least one more live-shell ingredient is still missing (candidates below). Untested ingredients (next candidates): - Real console *use*: main thread actually reading CONIN$ (gets/read pumping, as the repl reader does), console *output* channels (all probes redirected stdout/stderr to files), punk::console SetConsoleMode changes (VT input/output flags, raw/line flips) - The shellfilter tee stack: tcl::chan::fifo2 reflected channels + thread::transfer to pipe workers (cross-thread reflected-channel forwarding blocks in the owner thread's event loop) - The repl codethread; sheer thread/channel count - Anomaly to re-examine: probe workers created *from the codethread* inside a live shell wedged even without udp (both udp and control probes silent) - possibly artifact, possibly a lead Suggested tools for the mechanism write-up: native stacks of a wedged specimen (procdump / windbg on the hung punk902z), then map the blocked wait to tclWinConsole.c (Tcl 9 rewrite: per-handle reader/writer threads + gConsoleLock) or tcludp's Windows event source. ## Approach 1. [round 1 done 2026-07-08, see Notes] Capture native thread stacks of a wedged in-context specimen. Pipeline proven: spawn specimen via the hidden-console repro (post-fix, punkshout/punksherr still wedge in run 2 - their syslog comes from the logging config), `procdump64 -accepteula -ma ` (c:/cmdfiles/SysinternalsSuite), then `WinDbgX -y srv*C:\symbols*https://msdl.microsoft.com/download/symbols -z -c "~*k 30; .logclose; q" -logo ` (winget Microsoft.WinDbg; cdb is ACL-buried in WindowsApps but WinDbgX accepts classic scripted args). ROUND-1 LIMITATION: on the committed (fix-applied, uninstrumented) tree, workers carry no heartbeat timers, so a wedged worker is stack-indistinguishable from an idle one (all show the notifier MsgWaitForMultipleObjectsEx wait). 2. NEXT: re-add the shellthread instrumentation (listings below), spawn a specimen whose diag identifies the wedged worker tids (the tidXXXX hex in diag lines = dump thread Ids), dump, then read the wedged thread's exact stack, its MsgWait timeout argument (5th arg, on the stack frame - a permanently-zero or INFINITE timeout each discriminate a theory), and per-thread CPU (spin vs block decides the tcludp packetNum poll-mode theory below). 3. Read tcludp SocketThread's alert path and the packetNum consumption paths fully against the round-2 evidence; then reduce to a minimal repro. 4. Write up the mechanism here. 4. Re-verify against a Tcl 9 built from current core sources before any upstream report (the kit's 9.0.2 may carry an already-fixed bug). Building Tcl 9 from source is G-005 territory; this step waits on that capability (or a manually obtained current build). 5. Upstream filing (Tcl core or tcludp) is the user's manual decision and step. ## Findings round 1: native stacks + tcludp source (2026-07-08) Uninstrumented specimen dump (15 threads): 11 threads in the Tcl notifier's win32u!NtUserMsgWaitForMultipleObjectsEx (event loops - workers indistinguishable wedged vs idle without timers); one thread parked in SleepConditionVariableSRW inside static Tcl (consistent with a console reader thread on its CV); twapi's GetMessage thread; and - notable - **one thread living entirely inside tcl9udp1012.dll blocked in WaitForSingleObjectEx**: tcludp runs its own global helper thread on Windows. tcludp generic/udp_tcl.c (trunk 2026-07-05) Windows architecture - a rich wedge surface that is udp-specific by construction (matching "only udp-using workers wedge"): - `sockListLock` is an auto-reset EVENT used as a lock (CreateEvent auto-reset initially signaled; acquire = WaitForSingleObject INFINITE, release = SetEvent). No ownership, no recursion, lost-SetEvent = every udp-owning thread blocks INFINITE at its next event-loop pass (setup/check procs take it every pass) - deaf to timers and thread::send, blocked not spinning. Fingerprint-compatible. - One global SocketThread: acquire lock -> build fd_set skipping sockets with packetNum>0 or threadId NULL -> release -> if no sockets, park INFINITE on waitForSock event (the observed WaitForSingleObjectEx park) -> else select with 50ms timeout -> on readable: packetNum++ and (presumably - to verify) Tcl_ThreadAlert to the owning interp thread. - UDP_CheckProc (every event-loop pass of every udp-owning thread) recvfroms each owned socket unconditionally under the lock. Socket is non-blocking (ioctlsocket FIONBIO at open), so no blocking-read wedge - THAT theory is dead. But on error (e.g. WSAECONNRESET from our ICMP port-unreachable datagrams to 127.0.0.1:514-with-no-listener) it just frees the buffer: **packetNum is apparently NOT decremented on the error path** (decrement lives in the consumed-packet read path, ~line 1035). - UdpSetupProc: while packetNum>0 for an owned socket, forces Tcl_SetMaxBlockTime({0,0}) - ZERO block time - on EVERY pass. A packetNum stuck >0 (ICMP-flagged socket whose "packet" can never be consumed as data) therefore locks the owning thread's event loop into permanent zero-timeout poll mode. How that manifests under the Tcl 9 Windows notifier (busy spin? degenerate wait that starves timer/alert dispatch? interaction with the console-attached notifier mode?) is the round-2 question - note the separately-observed one-core-spinning orphaned interactive punk902z (dead console) may be this same packetNum poll mode showing as a spin in a different notifier state. - Tcl 8.6 immunity and console-dependence remain to be explained by whichever mechanism survives round 2 (8.6 vs 9 notifier differences, and console-attached notifier behaviour, are the candidate discriminators). Dump artifact: %TEMP%/punk_wedge.dmp (244MB, round-1 specimen) + %TEMP%/punk_wedge_stacks.txt. ## Findings round 2: object-level proof of the wedge mechanism (2026-07-08) Instrumented traffic-enabled specimen (syslog force-toggle in shellfilter::run to make the runtag log worker send datagrams again): run-2's three syslog workers completed init then wedged in <1s, blocked not spinning (~30-47ms CPU, Wait state). Their dump stacks all show **WaitForSingleObjectEx inside tcl9udp1012 called from the Tcl event loop** - the `WaitForSingleObject(sockListLock, INFINITE)` in tcludp's per-thread event-source procs (disassembly confirms both worker wait sites load the sockListLock static; run-1's two surviving syslog workers block at the sibling call site). The SocketThread's park site loads the waitForSock static immediately after a SetEvent(sockListLock) - it released properly. **Handle-table proof**: the sockListLock static holds handle 0x38c which the dump's handle data shows is now an **IoCompletion object**, and waitForSock (0x37c) shows as a **manual-reset event, set** - tcludp only ever creates auto-reset events. Both tcludp global event objects were **CloseHandle'd mid-session** (waiters keep an object alive but no handle remains to ever signal it; the freed handle VALUES were recycled by unrelated code). Every udp-loaded thread that touches the event-source procs after that blocks forever. 16-thread census: no thread holds the lock doing work; the previously-terminated run-1 shellfilter-run worker (the only udp thread that died mid-session, in the run-1->run-2 transition) is the prime closer suspect - `ExitSockets` (the only code closing these handles, registered via Tcl_CreateExitHandler in every Udp_Init) apparently runs during that thread's teardown. Downstream picture is now complete: closed-lock wedge -> udp-loaded workers freeze -> punkshell's old synchronous teardown send to a frozen worker -> the original exit/quit hang (mitigated). "Console dependence" dissolved: the CONIN$ restart was simply the only scenario where a udp-loaded worker died mid-session. Tcl 8.6 immunity presumably = its finalize-at-thread-death path not running the process exit handler (to verify; also check which tcludp version the 8.6 kit bundles). Minimal-repro attempts that did NOT reproduce (probe 'die' variants, consoleless script mode): a released udp-loaded worker; a terminate-mimic (open socket + datagrams + close + self thread::release -wait, verified thread death); same with the full punk package env (packagepreference + shellthread). So the ExitSockets trigger needs something more from the live shell. Next candidates: 1. terminate via `thread::send -async