punkshell/goals/archive/G-039-orphan-console-spin.md

G-039 Investigate the orphaned-shell one-core spin on a dead console

Status: achieved 2026-07-12 Scope: src/modules/punk/repl-999999.0a1.0.tm (console reader/event loop and EOF/error paths), src/modules/punk/console-999999.0a1.0.tm; investigation-first Goal: the observed failure mode - an interactive punk902z left running after its hosting terminal/console went away spins roughly a full core indefinitely (observed 2026-07-08: a 37-minute orphan with a single hard-looping thread) - is reliably reproduced and root-caused, then fixed or mitigated so a shell whose console dies exits or reaches zero-CPU idle cleanly. Acceptance: a documented procedure reproduces the spin on the current kit (e.g. launch an interactive shell in a terminal, then kill/close the hosting terminal or conhost), or the investigation records the attempts made and what evidence would reopen it; the spinning code path is identified (prime suspect: a console read/event loop treating a dead console's immediate EOF/error as retryable without backoff or termination - adjacent to the console-EOF restart path G-038 takes ownership of); after fix/mitigation, the same procedure shows the orphaned process exiting or settling at effectively zero CPU within a short grace period, with live-console interactive behaviour unchanged; the wedge-scoring hazard note (orphans polluting process-liveness checks in test harnesses) is updated to match the outcome.

Context

Observation (2026-07-08, during the G-036 investigation): a punk902z process from an earlier interactive session (started ~37 minutes prior; its hosting terminal presumed closed) was found consuming CPU continuously - ~1550 CPU-seconds accumulating at roughly +0.5-1 core-seconds per wall second, with exactly ONE thread in Running state (1546s of the total) and every other thread idle in normal waits. The process had to be killed manually. It also polluted the wedge-scoring checks of the day (process-liveness was being used as a hang signal), which is how it was noticed.

Not diagnosed at the time (killed to unblock the G-036 work); no dump was taken. The suspicion: when the hosting console goes away, a console channel read/event path returns immediately (EOF or error) and the surrounding loop retries without backoff or a give-up-and-exit decision. Candidate sites: the repl reader loop, the EOF branch behaviour when reopen/restart fails or loops (note tcl_interactive would be true for a real console, enabling the after 1 reopen_stdin path - see the race notes in the G-038 detail file), or punk::console query/read helpers.

Approach

1. Reproduce: launch an interactive shell in a disposable terminal and kill the host (close the tab/window; Stop-Process on the terminal; for classic conhost, killing conhost.exe; also try Windows Terminal vs conhost - behaviour may differ). Watch the orphan's CPU and thread states ((Get-Process punk902z).Threads sorted by TotalProcessorTime). Try both idle-at-prompt and mid-command states at kill time.
2. If reproduced: procdump + WinDbgX scripted stack capture of the spinning thread (the G-036 detail file documents the working tooling pipeline and pitfalls - WinDbgX process name is DbgX.Shell, cdb is ACL-buried, -z/-p -c -logo scripting works). With the spin thread's stack, map to the retry loop and decide fix: treat dead-console EOF/error as terminal (exit per PUNK_PIPE_EOF-like policy), or add backoff + detection.
3. Coordinate with G-038: its caller-driven restart owns the console-EOF path; the dead console case is the failure branch of the same decision point (reopen CONIN$ fails or the console is gone entirely) and should be designed together.

Notes

• Related: G-038 (console-EOF restart ownership, reopen_stdin race notes), G-036 (dump/ debugging tooling pipeline), tcl86-console-parked-read prior art (different mechanism, same neighbourhood).
• Harness hygiene (resolved 2026-07-12): kits were rebuilt with punk::repl 0.5.0 the same day (plain-kit kill test re-verified: exit within 2.0s), so the stray-orphan precaution applies only when deliberately testing pre-0.5.0 kits.
• Upstream: the root cause is a Tcl 9 core defect pair in win/tclWinConsole.c (see Progress below): (a) ConsoleEventProc drops the error/EOF notification that ConsoleSetupProc/ConsoleCheckProc generate for lastError != 0 (only ring-buffer data produces a Tcl_NotifyChannel), so a script can never see a dead console via fileevent; (b) ConsoleReaderThread's persistent-error branch (inputLen==0 && lastError!=0) wakes CVs + NudgeWatchers and continues with no wait, busy-looping one core (and via the wakeups a second) until the channel is closed. Both defects verified against a plain stock tclsh 9.0.3 (repro script with a stdin fileevent + vwait: ~1.85 cores after conhost kill, readable handler never fires) and the code is unchanged in the 9.1b1 reference sources. A ready-to-file ticket draft with the standalone repro lives at TEMP_REFERENCE/tcl9-dead-console-spin-TICKET-DRAFT.md. A kit-side Tcl source patch is an alternative once G-005/G-006 build infrastructure exists; the script-level watchdog stays valid regardless.
• Archived-goal references in this file: G-036 achieved 2026-07-08 (goals/archive/G-036-tcl9-udp-console-worker-wedge.md).

Progress

2026-07-12 reproduced, root-caused, fixed in source (punk::repl 0.5.0)

Reproduction procedure (reliable, first-try, current punk902z kit 2026-07-11 build):

1. Launch the shell interactively under a dedicated classic conhost (isolates the kill from the user's terminal): Start-Process -WindowStyle Hidden conhost.exe -ArgumentList 'C:\repo\jn\shellspy\bin\punk902z.exe' (optionally via cmd.exe with > out.txt 2> err.txt redirection to capture the trail; punk902z src for dev modules).
2. Confirm the tree (punk902z is a child of the new conhost) and let it settle at the idle prompt (CPU delta 0 over several seconds).
3. Stop-Process -Force the conhost pid. punk902z survives and immediately spins: ~1.7-1.9 cores, matching the 2026-07-08 observation (that one showed ~1 core; both hot threads here are part of the same mechanism). Both idle-at-prompt kill state and the redirected-stdio variant reproduce.

Spin mechanism (dump: procdump64 -ma + WinDbgX -z <dmp> -c "~*k 30; .logclose; q" -logo, per the G-036 tooling notes; mapped against TEMP_REFERENCE/tcl9/win/tclWinConsole.c):

• Hot thread 1 = the Tcl console reader thread (stack: thread start -> ConsoleReaderThread -> SetEvent storm). After the console dies, ConsoleDataAvailable returns -1 (PeekConsoleInputW fails) which the caller treats as truthy, ReadConsoleChars fails (observed error: broken pipe - ERROR_BROKEN_PIPE, not ERROR_INVALID_HANDLE) and handleInfoPtr->lastError is set. From then on the reader thread's top-of-loop branch (inputLen>0 || lastError!=0) runs every iteration - WakeAllConditionVariable + NudgeWatchers(SetEvent/Tcl_ThreadAlert) + continue - with no sleep on that path, until channel close drops numRefs to 1. lastError is never cleared because the script never reads again (see next point).
• Hot thread 2 = the repl/main interp thread: each nudge alerts its notifier; ConsoleSetupProc counts lastError as "readable" and sets max block time 0, and ConsoleCheckProc queues a console event for it - but ConsoleEventProc only calls Tcl_NotifyChannel when the ring buffer has DATA and ignores lastError entirely, so the queued event is discarded and the armed readable fileevent (repl_handler) NEVER fires. The event loop spins at zero block time queueing and discarding events.
• Script level is therefore completely blind: instrumented src-mode run confirmed repl_handler never runs after console death (no entry, no EOF branch, no reopen_stdin, no bgerror, vwait never returns), so nothing ever closes the channel and both spins are permanent. The reopen_stdin restart loop suspected in the original goal statement is NOT the mechanism - it never gets the chance to run.

Fix (src/modules/punk/repl-999999.0a1.0.tm, punk::repl 0.5.0):

• New repl::console_watchdog self-rescheduling liveness poll (default 5000ms, repl::console_watchdog_ms), armed by repl::start before its vwait, only when platform is windows AND the repl serves the process-default console AND the input channel's chan configure dict has -inputmode (i.e. a tcl9 console channel; piped stdin, foreign consoles and tcl 8.6 console channels never arm it). The probe is a read-only chan configure $inchan -inputmode (live GetConsoleMode). On probe failure: disarm, close the input channel (this is what stops the reader-thread spin - numRefs drops and the thread exits), and set ::repl::done {eof <chan>} so the repl finishes via the normal eof path; app-punkshell's eof handling then finds CONIN$ unopenable ("broken pipe") and exits cleanly (heuristic policy, exit 0). repl::start's post-vwait chan event $inchan readable {} deregistration is now guarded for the watchdog-closed-channel case (previously an unguarded call raised a traceback on this path). Scheduling state per channel name in repl::console_watchdog_afterids; cancelled after vwait by the arming frame.

Verification (2026-07-12, punk902z kit 2026-07-11 in src mode - dev modules; Tcl 9.0.2 kit runtime):

• Same kill procedure post-fix: watchdog message on stderr, orphan EXITED 1.5s after the conhost kill (worst case is ~watchdog interval + teardown), no traceback. Remaining teardown noise (disableRaw warning, codethread tsv warning) is cosmetic, printed to an already-dead console in the real scenario.
• Live-console soak: 25s at the idle prompt (>=4 watchdog probes) - process alive, CPU delta 0, prompt intact; no spurious trigger.
• Piped stdin ('set ::x 1' | punk902z src with PUNK_PIPE_EOF=exit): unaffected, exit 0 (watchdog not armed for pipes).
• runtests (native tclsh 9.0.3): repl consolebackends.test 3/3 pass; punk::console suites (console/*.test) 88 pass / 1 skip / 0 fail.
• tclsh src/make.tcl modules builds clean (provenance warnings = expected dirty tree).

2026-07-12 acceptance closed (kits rebuilt, user verification)

All remaining items resolved:

• Kits rebuilt 2026-07-12 with punk::repl 0.5.0 (punkbi, punk91, punk902z). Plain-kit kill procedure re-verified on the rebuilt punk902z: orphan exited 2.0s after the conhost kill via the clean eof path (watchdog message, app-punkshell "no console available", exit 0), no traceback.
• User verified interactive sessions on the rebuilt kits (punkbi, punk91, punk902z, punk::repl confirmed at 0.5.0): behaviour unchanged with the 5s probe - "all seems well".
• Tcl 8.6 (punksys) confirmed OUT OF SCOPE by the user 2026-07-12: different console driver, watchdog deliberately not armed there; reopen as a new goal if an 8.6 orphan spin is ever observed.
• Wedge-scoring hazard note updated (see Notes - precaution now applies only to pre-0.5.0 kits).
• Root cause verified independent of punkshell with plain tclsh 9.0.3; upstream ticket draft written to TEMP_REFERENCE/tcl9-dead-console-spin-TICKET-DRAFT.md (filing is a user decision, not gating this goal).