Browse Source
An interactive shell orphaned by its hosting console dying (killed conhost/terminal) previously spun ~2 cores forever: the Tcl 9 windows console driver never delivers the dead-console state to the script as a fileevent (tclWinConsole.c ConsoleEventProc only notifies on buffered data) while its reader thread busy-loops on the persistent channel error, so the script level is completely blind and nothing ever closes the channel. Fix: repl::console_watchdog - a self-rescheduling liveness poll (default 5s, repl::console_watchdog_ms; read-only GetConsoleMode probe via chan configure -inputmode) armed by repl::start only for a tcl9 console input channel serving the process-default console on windows. On probe failure it closes the input channel (stopping the driver's reader thread) and finishes the repl via the normal eof done-path; app-punkshell then finds no console reopenable and exits cleanly. Piped, foreign-console and tcl 8.6 inputs never arm it. repl::start's post-vwait reader deregistration now tolerates a watchdog-closed channel. Verification (2026-07-12): kill procedure post-fix exits the orphan in 1.5-2.0s with no traceback (src mode and rebuilt kits punkbi/punk91/ punk902z); 25s live-console soak shows no spurious trigger, CPU delta 0; piped stdin unaffected; repl consolebackends 3/3 and punk::console suites 88 pass / 1 skip; root cause verified against plain tclsh 9.0.3. Upstream ticket filed: https://core.tcl-lang.org/tcl/tktview/f10d91c2d3 Included: - goals: G-039 flipped achieved 2026-07-12 and archived (index entry to GOALS-archive.md, detail file to goals/archive/ with full root-cause, repro and verification record); reference sweep adds the eof done-value producer note to G-038 - src/modules/punk/AGENTS.md: watchdog contract documented - project version 0.12.1 -> 0.12.2 + CHANGELOG entry - vfs payload sync: _vfscommon.vfs punk/repl 0.4.0 -> 0.5.0 Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.commaster
11 changed files with 343 additions and 59 deletions
@ -1,49 +0,0 @@
|
||||
# G-039 Investigate the orphaned-shell one-core spin on a dead console |
||||
|
||||
Status: proposed |
||||
Scope: src/modules/punk/repl-999999.0a1.0.tm (console reader/event loop and EOF/error paths), src/modules/punk/console-999999.0a1.0.tm; investigation-first |
||||
Goal: the observed failure mode - an interactive punk902z left running after its hosting terminal/console went away spins roughly a full core indefinitely (observed 2026-07-08: a 37-minute orphan with a single hard-looping thread) - is reliably reproduced and root-caused, then fixed or mitigated so a shell whose console dies exits or reaches zero-CPU idle cleanly. |
||||
Acceptance: a documented procedure reproduces the spin on the current kit (e.g. launch an interactive shell in a terminal, then kill/close the hosting terminal or conhost), or the investigation records the attempts made and what evidence would reopen it; the spinning code path is identified (prime suspect: a console read/event loop treating a dead console's immediate EOF/error as retryable without backoff or termination - adjacent to the console-EOF restart path G-038 takes ownership of); after fix/mitigation, the same procedure shows the orphaned process exiting or settling at effectively zero CPU within a short grace period, with live-console interactive behaviour unchanged; the wedge-scoring hazard note (orphans polluting process-liveness checks in test harnesses) is updated to match the outcome. |
||||
|
||||
## Context |
||||
|
||||
Observation (2026-07-08, during the G-036 investigation): a punk902z process from an earlier |
||||
interactive session (started ~37 minutes prior; its hosting terminal presumed closed) was |
||||
found consuming CPU continuously - ~1550 CPU-seconds accumulating at roughly +0.5-1 |
||||
core-seconds per wall second, with exactly ONE thread in Running state (1546s of the total) |
||||
and every other thread idle in normal waits. The process had to be killed manually. It also |
||||
polluted the wedge-scoring checks of the day (process-liveness was being used as a hang |
||||
signal), which is how it was noticed. |
||||
|
||||
Not diagnosed at the time (killed to unblock the G-036 work); no dump was taken. The |
||||
suspicion: when the hosting console goes away, a console channel read/event path returns |
||||
immediately (EOF or error) and the surrounding loop retries without backoff or a |
||||
give-up-and-exit decision. Candidate sites: the repl reader loop, the EOF branch behaviour |
||||
when reopen/restart fails or loops (note tcl_interactive would be true for a real console, |
||||
enabling the `after 1 reopen_stdin` path - see the race notes in the G-038 detail file), or |
||||
punk::console query/read helpers. |
||||
|
||||
## Approach |
||||
|
||||
1. Reproduce: launch an interactive shell in a disposable terminal and kill the host |
||||
(close the tab/window; `Stop-Process` on the terminal; for classic conhost, killing |
||||
conhost.exe; also try Windows Terminal vs conhost - behaviour may differ). Watch the |
||||
orphan's CPU and thread states (`(Get-Process punk902z).Threads` sorted by |
||||
TotalProcessorTime). Try both idle-at-prompt and mid-command states at kill time. |
||||
2. If reproduced: procdump + WinDbgX scripted stack capture of the spinning thread (the |
||||
G-036 detail file documents the working tooling pipeline and pitfalls - WinDbgX process |
||||
name is DbgX.Shell, cdb is ACL-buried, `-z/-p -c -logo` scripting works). With the spin |
||||
thread's stack, map to the retry loop and decide fix: treat dead-console EOF/error as |
||||
terminal (exit per PUNK_PIPE_EOF-like policy), or add backoff + detection. |
||||
3. Coordinate with G-038: its caller-driven restart owns the console-EOF path; the dead |
||||
console case is the failure branch of the same decision point (reopen CONIN$ fails or |
||||
the console is gone entirely) and should be designed together. |
||||
|
||||
## Notes |
||||
|
||||
- Related: G-038 (console-EOF restart ownership, reopen_stdin race notes), G-036 (dump/ |
||||
debugging tooling pipeline), tcl86-console-parked-read prior art (different mechanism, |
||||
same neighbourhood). |
||||
- Harness hygiene until fixed: verify no stray punk902z processes before scoring any |
||||
liveness-based test run. |
||||
- Archived-goal references in this file: G-036 achieved 2026-07-08 (goals/archive/G-036-tcl9-udp-console-worker-wedge.md). |
||||
@ -0,0 +1,157 @@
|
||||
# G-039 Investigate the orphaned-shell one-core spin on a dead console |
||||
|
||||
Status: achieved 2026-07-12 |
||||
Scope: src/modules/punk/repl-999999.0a1.0.tm (console reader/event loop and EOF/error paths), src/modules/punk/console-999999.0a1.0.tm; investigation-first |
||||
Goal: the observed failure mode - an interactive punk902z left running after its hosting terminal/console went away spins roughly a full core indefinitely (observed 2026-07-08: a 37-minute orphan with a single hard-looping thread) - is reliably reproduced and root-caused, then fixed or mitigated so a shell whose console dies exits or reaches zero-CPU idle cleanly. |
||||
Acceptance: a documented procedure reproduces the spin on the current kit (e.g. launch an interactive shell in a terminal, then kill/close the hosting terminal or conhost), or the investigation records the attempts made and what evidence would reopen it; the spinning code path is identified (prime suspect: a console read/event loop treating a dead console's immediate EOF/error as retryable without backoff or termination - adjacent to the console-EOF restart path G-038 takes ownership of); after fix/mitigation, the same procedure shows the orphaned process exiting or settling at effectively zero CPU within a short grace period, with live-console interactive behaviour unchanged; the wedge-scoring hazard note (orphans polluting process-liveness checks in test harnesses) is updated to match the outcome. |
||||
|
||||
## Context |
||||
|
||||
Observation (2026-07-08, during the G-036 investigation): a punk902z process from an earlier |
||||
interactive session (started ~37 minutes prior; its hosting terminal presumed closed) was |
||||
found consuming CPU continuously - ~1550 CPU-seconds accumulating at roughly +0.5-1 |
||||
core-seconds per wall second, with exactly ONE thread in Running state (1546s of the total) |
||||
and every other thread idle in normal waits. The process had to be killed manually. It also |
||||
polluted the wedge-scoring checks of the day (process-liveness was being used as a hang |
||||
signal), which is how it was noticed. |
||||
|
||||
Not diagnosed at the time (killed to unblock the G-036 work); no dump was taken. The |
||||
suspicion: when the hosting console goes away, a console channel read/event path returns |
||||
immediately (EOF or error) and the surrounding loop retries without backoff or a |
||||
give-up-and-exit decision. Candidate sites: the repl reader loop, the EOF branch behaviour |
||||
when reopen/restart fails or loops (note tcl_interactive would be true for a real console, |
||||
enabling the `after 1 reopen_stdin` path - see the race notes in the G-038 detail file), or |
||||
punk::console query/read helpers. |
||||
|
||||
## Approach |
||||
|
||||
1. Reproduce: launch an interactive shell in a disposable terminal and kill the host |
||||
(close the tab/window; `Stop-Process` on the terminal; for classic conhost, killing |
||||
conhost.exe; also try Windows Terminal vs conhost - behaviour may differ). Watch the |
||||
orphan's CPU and thread states (`(Get-Process punk902z).Threads` sorted by |
||||
TotalProcessorTime). Try both idle-at-prompt and mid-command states at kill time. |
||||
2. If reproduced: procdump + WinDbgX scripted stack capture of the spinning thread (the |
||||
G-036 detail file documents the working tooling pipeline and pitfalls - WinDbgX process |
||||
name is DbgX.Shell, cdb is ACL-buried, `-z/-p -c -logo` scripting works). With the spin |
||||
thread's stack, map to the retry loop and decide fix: treat dead-console EOF/error as |
||||
terminal (exit per PUNK_PIPE_EOF-like policy), or add backoff + detection. |
||||
3. Coordinate with G-038: its caller-driven restart owns the console-EOF path; the dead |
||||
console case is the failure branch of the same decision point (reopen CONIN$ fails or |
||||
the console is gone entirely) and should be designed together. |
||||
|
||||
## Notes |
||||
|
||||
- Related: G-038 (console-EOF restart ownership, reopen_stdin race notes), G-036 (dump/ |
||||
debugging tooling pipeline), tcl86-console-parked-read prior art (different mechanism, |
||||
same neighbourhood). |
||||
- Harness hygiene (resolved 2026-07-12): kits were rebuilt with punk::repl 0.5.0 the same |
||||
day (plain-kit kill test re-verified: exit within 2.0s), so the stray-orphan precaution |
||||
applies only when deliberately testing pre-0.5.0 kits. |
||||
- Upstream: the root cause is a Tcl 9 core defect pair in win/tclWinConsole.c (see |
||||
Progress below): (a) ConsoleEventProc drops the error/EOF notification that |
||||
ConsoleSetupProc/ConsoleCheckProc generate for lastError != 0 (only ring-buffer data |
||||
produces a Tcl_NotifyChannel), so a script can never see a dead console via fileevent; |
||||
(b) ConsoleReaderThread's persistent-error branch (inputLen==0 && lastError!=0) wakes |
||||
CVs + NudgeWatchers and continues with no wait, busy-looping one core (and via the |
||||
wakeups a second) until the channel is closed. Both defects verified against a plain |
||||
stock tclsh 9.0.3 (repro script with a stdin fileevent + vwait: ~1.85 cores after |
||||
conhost kill, readable handler never fires) and the code is unchanged in the 9.1b1 |
||||
reference sources. A ready-to-file ticket draft with the standalone repro lives at |
||||
`TEMP_REFERENCE/tcl9-dead-console-spin-TICKET-DRAFT.md`. A kit-side Tcl source patch |
||||
is an alternative once G-005/G-006 build infrastructure exists; the script-level |
||||
watchdog stays valid regardless. |
||||
- Archived-goal references in this file: G-036 achieved 2026-07-08 (goals/archive/G-036-tcl9-udp-console-worker-wedge.md). |
||||
|
||||
## Progress |
||||
|
||||
### 2026-07-12 reproduced, root-caused, fixed in source (punk::repl 0.5.0) |
||||
|
||||
Reproduction procedure (reliable, first-try, current punk902z kit 2026-07-11 build): |
||||
|
||||
1. Launch the shell interactively under a dedicated classic conhost (isolates the kill |
||||
from the user's terminal): |
||||
`Start-Process -WindowStyle Hidden conhost.exe -ArgumentList 'C:\repo\jn\shellspy\bin\punk902z.exe'` |
||||
(optionally via cmd.exe with `> out.txt 2> err.txt` redirection to capture the trail; |
||||
`punk902z src` for dev modules). |
||||
2. Confirm the tree (`punk902z` is a child of the new conhost) and let it settle at the |
||||
idle prompt (CPU delta 0 over several seconds). |
||||
3. `Stop-Process -Force` the conhost pid. punk902z survives and immediately spins: |
||||
~1.7-1.9 cores, matching the 2026-07-08 observation (that one showed ~1 core; both |
||||
hot threads here are part of the same mechanism). Both idle-at-prompt kill state and |
||||
the redirected-stdio variant reproduce. |
||||
|
||||
Spin mechanism (dump: procdump64 -ma + `WinDbgX -z <dmp> -c "~*k 30; .logclose; q" -logo`, |
||||
per the G-036 tooling notes; mapped against TEMP_REFERENCE/tcl9/win/tclWinConsole.c): |
||||
|
||||
- Hot thread 1 = the Tcl console reader thread (stack: thread start -> ConsoleReaderThread |
||||
-> SetEvent storm). After the console dies, ConsoleDataAvailable returns -1 |
||||
(PeekConsoleInputW fails) which the caller treats as truthy, ReadConsoleChars fails |
||||
(observed error: broken pipe - ERROR_BROKEN_PIPE, not ERROR_INVALID_HANDLE) and |
||||
handleInfoPtr->lastError is set. From then on the reader thread's top-of-loop branch |
||||
(inputLen>0 || lastError!=0) runs every iteration - WakeAllConditionVariable + |
||||
NudgeWatchers(SetEvent/Tcl_ThreadAlert) + continue - with no sleep on that path, until |
||||
channel close drops numRefs to 1. lastError is never cleared because the script never |
||||
reads again (see next point). |
||||
- Hot thread 2 = the repl/main interp thread: each nudge alerts its notifier; |
||||
ConsoleSetupProc counts lastError as "readable" and sets max block time 0, and |
||||
ConsoleCheckProc queues a console event for it - but ConsoleEventProc only calls |
||||
Tcl_NotifyChannel when the ring buffer has DATA and ignores lastError entirely, so the |
||||
queued event is discarded and the armed readable fileevent (repl_handler) NEVER fires. |
||||
The event loop spins at zero block time queueing and discarding events. |
||||
- Script level is therefore completely blind: instrumented src-mode run confirmed |
||||
repl_handler never runs after console death (no entry, no EOF branch, no reopen_stdin, |
||||
no bgerror, vwait never returns), so nothing ever closes the channel and both spins are |
||||
permanent. The reopen_stdin restart loop suspected in the original goal statement is NOT |
||||
the mechanism - it never gets the chance to run. |
||||
|
||||
Fix (src/modules/punk/repl-999999.0a1.0.tm, punk::repl 0.5.0): |
||||
|
||||
- New `repl::console_watchdog` self-rescheduling liveness poll (default 5000ms, |
||||
`repl::console_watchdog_ms`), armed by repl::start before its vwait, only when platform |
||||
is windows AND the repl serves the process-default console AND the input channel's |
||||
`chan configure` dict has `-inputmode` (i.e. a tcl9 console channel; piped stdin, |
||||
foreign consoles and tcl 8.6 console channels never arm it). The probe is a read-only |
||||
`chan configure $inchan -inputmode` (live GetConsoleMode). On probe failure: disarm, |
||||
close the input channel (this is what stops the reader-thread spin - numRefs drops and |
||||
the thread exits), and set `::repl::done {eof <chan>}` so the repl finishes via the |
||||
normal eof path; app-punkshell's eof handling then finds CONIN$ unopenable ("broken |
||||
pipe") and exits cleanly (heuristic policy, exit 0). repl::start's post-vwait |
||||
`chan event $inchan readable {}` deregistration is now guarded for the |
||||
watchdog-closed-channel case (previously an unguarded call raised a traceback on this |
||||
path). Scheduling state per channel name in `repl::console_watchdog_afterids`; |
||||
cancelled after vwait by the arming frame. |
||||
|
||||
Verification (2026-07-12, punk902z kit 2026-07-11 in `src` mode - dev modules; Tcl 9.0.2 |
||||
kit runtime): |
||||
|
||||
- Same kill procedure post-fix: watchdog message on stderr, orphan EXITED 1.5s after the |
||||
conhost kill (worst case is ~watchdog interval + teardown), no traceback. Remaining |
||||
teardown noise (disableRaw warning, codethread tsv warning) is cosmetic, printed to an |
||||
already-dead console in the real scenario. |
||||
- Live-console soak: 25s at the idle prompt (>=4 watchdog probes) - process alive, CPU |
||||
delta 0, prompt intact; no spurious trigger. |
||||
- Piped stdin (`'set ::x 1' | punk902z src` with PUNK_PIPE_EOF=exit): unaffected, exit 0 |
||||
(watchdog not armed for pipes). |
||||
- runtests (native tclsh 9.0.3): repl consolebackends.test 3/3 pass; punk::console suites |
||||
(console/*.test) 88 pass / 1 skip / 0 fail. |
||||
- `tclsh src/make.tcl modules` builds clean (provenance warnings = expected dirty tree). |
||||
|
||||
### 2026-07-12 acceptance closed (kits rebuilt, user verification) |
||||
|
||||
All remaining items resolved: |
||||
|
||||
- Kits rebuilt 2026-07-12 with punk::repl 0.5.0 (punkbi, punk91, punk902z). Plain-kit |
||||
kill procedure re-verified on the rebuilt punk902z: orphan exited 2.0s after the |
||||
conhost kill via the clean eof path (watchdog message, app-punkshell "no console |
||||
available", exit 0), no traceback. |
||||
- User verified interactive sessions on the rebuilt kits (punkbi, punk91, punk902z, |
||||
punk::repl confirmed at 0.5.0): behaviour unchanged with the 5s probe - "all seems |
||||
well". |
||||
- Tcl 8.6 (punksys) confirmed OUT OF SCOPE by the user 2026-07-12: different console |
||||
driver, watchdog deliberately not armed there; reopen as a new goal if an 8.6 orphan |
||||
spin is ever observed. |
||||
- Wedge-scoring hazard note updated (see Notes - precaution now applies only to |
||||
pre-0.5.0 kits). |
||||
- Root cause verified independent of punkshell with plain tclsh 9.0.3; upstream ticket |
||||
draft written to `TEMP_REFERENCE/tcl9-dead-console-spin-TICKET-DRAFT.md` (filing is a |
||||
user decision, not gating this goal). |
||||
@ -1,4 +1,4 @@
|
||||
[project] |
||||
name = "punkshell" |
||||
version = "0.12.1" |
||||
version = "0.12.2" |
||||
license = "BSD-2-Clause" |
||||
|
||||
Loading…
Reference in new issue