punkshell/goals/archive/G-106-powershell-consolemode-fallback.md

G-106 powershell console-mode fallback: maintained raw-mode path for twapi-less runtimes

Status: achieved 2026-07-22 Scope: src/modules/punk/console-999999.0a1.0.tm (enableRaw_powershell/disableRaw_powershell + the persistent server lifecycle); scriptlib/utils/pwsh/ (consolemode_server_async.ps1 canonical + experiment-variant reconciliation); ps-script resolution/packaging across launch contexts (argv0-derived pstooldir; kits via the G-089 scriptlib-in-kits interplay); src/make.tcl shell (the verified launch context) Goal: the powershell console-mode fallback - a persistent pwsh/powershell named-pipe server driven by punk::console when twapi is absent - is a MAINTAINED, dependable raw-mode path for twapi-less runtimes (the suite-built shells and kits, until/unless zig-built twapi lands): quiet in normal operation, reliable across the session lifecycle, and resolvable from every supported launch shape rather than only argv0-relative luck. Acceptance: on a twapi-less suite runtime, raw enable/disable work via the fallback from (a) the make.tcl shell launch and (b) a kit / plain-tclsh repl launch, with no stderr noise in normal operation; the server lifecycle is verified - starts once per session, stays up for the session's raw transitions (the 2026-07-20 observed early-shutdown mode diagnosed and fixed), and shuts down with the session leaving no orphan pwsh processes; ps-script resolution no longer depends solely on argv0 parent-dir derivation (works from kits and unusual cwds, with the fallback-to-pwd branch replaced by something principled); scriptlib/utils/pwsh is reconciled to one canonical server script with the experiment variants retired or explicitly labelled; the fallback's role as the no-twapi contingency is documented where G-103's twapi investigation will find it, and a repeatable verification recipe is recorded in this file.

Context

User verification 2026-07-20 on the freshly built tclsh90spr (piperepl variant from G-096, achieved 2026-07-20): with tcl::tm::add .../modules + package require repl; repl::init; repl::start stdin, basic punk features are functional on the bare suite runtime - except raw mode, which is twapi-reliant. Launched instead via ./tclsh90spr .../src/make.tcl shell, the powershell fallback engaged and raw mode WORKED - modulo consolemode_server_async.ps1 noise and an early server shutdown. That working-but-rough state is what this goal hardens. User direction: "we would like to look at maintaining this powershell fallback - especially if we can't get zig-based twapi builds going."

Mechanism today (punk::console): when twapi is absent, enableRaw/disableRaw are aliased to *_powershell variants that talk over a named pipe to a persistent consolemode_server_async.ps1 process (templated per-console id; pwsh.exe probed first, then powershell.exe). Known weak points at goal creation:

• Resolution: pstooldir = argv0's grandparent + /scriptlib/utils/pwsh, else [pwd] - works for make.tcl shell (argv0 = src/make.tcl -> repo root) and little else by construction. Kits need the script carried/resolved properly (G-089 scriptlib in kits).
• Lifecycle: observed early shutdown of the server mid-session; startup noise on stderr ("twapi not present, using persistent powershell process: ..." plus ps1 output).
• Hygiene: scriptlib/utils/pwsh holds several sibling experiments (consolemode_server.ps1, *_async1.ps1, *_async.2ps1, consolemode.ps1, consolemode_enableraw.ps1) with no marking of which is canonical.

Relationships: G-103 (bi-kit twapi under the zig-only policy is needs-investigation - THIS fallback is the mitigation; if zig-built twapi proves impractical the fallback becomes the primary raw path on suite runtimes and this goal's priority rises), G-013 (raw mode as the repl default ultimately requires raw mode to be dependable on twapi-less runtimes), G-089 (scriptlib in kits - the delivery vehicle for the ps script), G-061-era console-context findings (agent console-context traps around attached consoles apply to verifying this).

Approach

Design decisions of the 2026-07-22 overhaul (punk::console 0.8.0):

• Root cause of the 2026-07-20 early shutdown - two compounding defects in the old server script: (1) a 20s keepalive killed the server when no ping arrived, but only enableraw messages refreshed the ping and NO tcl-side pinger ever existed, so the server always died ~20s after the last raw enable ("ping stale for pipe ... - exiting" was this kill's console noise); (2) the per-connection StreamReader.Close() disposed the underlying pipe stream, so the following Disconnect()/Dispose() threw and killed the listener runspace after the FIRST message - later toggles could not reach the server even while its process lived.
• Keepalive replaced by a parent-process watch: the tcl side templates its pid into the script; the server holds a System.Diagnostics.Process object obtained once by that pid (handle-based, immune to pid reuse) and exits when HasExited reports the owner gone (<=5s poll cadence). Orphan prevention needs no cooperation from the tcl side - hard kills included.
• Lazy singleton server: spawn moved from module load to first enableRaw/disableRaw use, so loading punk::console in piped/non-console or worker-thread contexts spawns nothing and prints nothing. Server identity lives in tsv punk_console (ps_server_pipename/ps_server_pid/ps_server_spawntime) under tsv::lock, so all interps/threads of a process share ONE server (the codethread and the repl thread previously would each have spawned their own).
• Quiet by default: the server child is spawned -nop -nol -noni with stdout/stderr redirected to NUL (stdin MUST stay inherited - the console input handle is how the server reaches the console); all ps1 output is debug-gated. Env PUNK_PS_CONSOLEMODE_DEBUG=1 enables diagnostics on both sides (tcl spawn note on stderr + unredirected ps1 write-host trail). Failure paths (server unreachable, spawn failure) still emit one actionable stderr line - that is not normal operation.
• Resolution chain (punk::console::system::ps_consolemode_script_get): env PUNK_PS_CONSOLEMODE_SCRIPT override (used only when the file exists) -> argv0-derived project root -> module-location-derived roots (2-up for built modules/, 3-up for src/modules/; a zipfs kit module path just fails the probes) -> embedded copy of the script carried in the module (ps_consolemode_script_embedded). The [pwd] fallback is gone. The embedded copy makes kits and arbitrary cwds resolution-proof with no scriptlib delivery dependency (the G-089 interplay is thereby optional for THIS script: a resolvable scriptlib file wins - dev-editable - and the embedded text is the floor).
• Delivery stays -c <script text> (not a dropped file): needs nothing on disk at run time and is not subject to ExecutionPolicy restrictions that can block -File runs. The ps1's semicolon-after-each-command convention supports that delivery.
• Protocol: one line per named-pipe connection - enableraw | disableraw | ping | exit. The listener now forwards disableraw (previously swallowed - on tcl 8.6, where -inputmode is unavailable, cooked mode could never be restored via the fallback); a null-message connection (a probe) no longer shuts the server down; AutoResetEvent replaces ManualResetEvent+late-Reset (lost-wakeup race).
• enableRaw returns only after the flip is observable where the runtime can read live mode (tcl9 -inputmode poll, 750ms cap, result note carries confirmed 0|1|unknown) - closes the fire-and-forget race between the pipe write and the server applying the console flags.
• ps_consolemode_send retries connection to a deadline (15s within 15s of spawn - powershell startup - else 2.5s), and on an unreachable recorded server clears the recorded state and respawns once. ps_consolemode_server_stop is an explicit best-effort shutdown for tests/tools (not needed for orphan prevention).
• stty remains the last-resort branch after the server path (unchanged semantics: useful only in msys/mintty-without-winpty environments and when powershell is entirely absent).
• posh-git MIT attribution for the NativeConsoleMethods C# snippet now rides in the canonical script (and thus the embedded copy) - the third-party source copy (consolemode.ps1) could then be retired with the other variants.

Notes

• Verification recipe (repeatable):
  1. Sync + resolution suite (any platform, no server spawned): tclsh90 src/tests/runtests.tcl -report compact -show-passes 0 -include-paths "modules/punk/console/***"
     • psfallback.test pins the embedded-copy/canonical-file sync, placeholder presence, and the resolution chain (env override + fallthrough).
  2. Functional lifecycle selftest on a twapi-less runtime. The selftest must run with a REAL console that is not the driving shell's console: launch it via Start-Process <twapi-less tclsh> <selftest.tcl> <resultfile> <moduledir> -WindowStyle Hidden (Start-Process gives a console app a fresh console; Hidden keeps it invisible; do NOT redirect stdin). Selftest body (write results to <resultfile>, tm-path add <moduledir>, package require punk::console, assert has_twapi=0): enableRaw -> assert chan configure stdin -inputmode reads raw (tcl9's live GetConsoleMode read is the independent cross-check); disableRaw -> normal; repeat; idle 25s (the old keepalive died at 20s) then another cycle; punk::console::system::ps_consolemode_server_stop (expect stopped 1); enableRaw again (respawn) + disableRaw; record tsv::get punk_console ps_server_pid after each spawn. Driver afterwards: wait >=12s past process exit and assert every recorded server pid is gone (parent-watch reaping). Force the embedded resolution variant by setting set ::punk::console::system::module_dir C:/nonexistent before the first enableRaw with the selftest located outside any checkout, and assert ps_consolemode_script_get reports source=embedded.
  3. Live-repl spot check (the acceptance launch shapes): boot <twapi-less tclsh> src/make.tcl shell (shape a) or <twapi-less kit> <script: tcl::tm::path add <repo>/modules; package require punk::repl; repl::init; repl::start stdin> (shape b) hidden as above with stderr redirected to a file; from a helper process do FreeConsole + AttachConsole() + open CONIN$ + WriteConsoleInput to type punk::console::enableRaw + Enter at the live prompt; GetConsoleMode(CONIN$) before/after shows ENABLE_LINE_INPUT|ECHO cleared (observed 0x1E7 -> 0x1E1); a pwsh child of the shell with punkshell_ps_consolemode in its command line confirms engagement; type punk::console::disableRaw (mode returns, observed 0x1E1 -> 0x1E7); kill the shell and assert the server exits <=12s; assert the stderr capture has no fallback-related lines (grep: twapi not present|pipename|Started named pipe|consolemode_server|ping stale|persistent powershell).
  4. Debug knob when anything misbehaves: PUNK_PS_CONSOLEMODE_DEBUG=1 in the tcl process's environment (tcl-side spawn note + unredirected ps1 diagnostics).
• Residue - tcl 8.6 twapi-less: the no--inputmode disableraw-via-server path is now CODED (previously stty-or-error) but only tcl9-verified; verify when the G-099 8.6 suite produces a twapi-less 8.6 runtime.
• Residue - punk kits with twapi (e.g punk905): unaffected by design (twapi branch wins). Note their kit-stamped punk::console pre-registration satisfies a plain package require punk::console in bare tcl::tm::add contexts, shadowing newer built modules - the known kit-registration behaviour, relevant to verification setups only.
• Residue - src/vfs/project.vfs's pinned punk::console 0.1.1 copy still carries the old load-time spawn referencing the retired consolemode_server.ps1 (user-curated vfs payload for generated projects; not touched by this goal).
• Repl integration note: raw engages on demand (lazy spawn) - the first enableRaw of a twapi-less session pays the server spawn (~0.6-0.7s with pwsh 7 observed; powershell.exe cold is slower), later toggles are ~1-30ms. An eager warm-up at repl init on twapi-less interactive sessions is a possible future nicety if the first-prompt delay is felt.

Progress

2026-07-22: overhaul landed (punk::console 0.8.0) + full verification - acceptance met

Landed: punk::console 0.8.0 (see console-buildversion.txt changelog for the per-change list; design in ## Approach), canonical scriptlib/utils/pwsh/consolemode_server_async.ps1 rewrite, five experiment variants retired (consolemode_server.ps1, consolemode_server_async1.ps1, consolemode_server_async.2ps1, consolemode.ps1, consolemode_enableraw.ps1) with scriptlib/utils/pwsh/README.md labelling the canonical + echotest.ps1, new console testsuite psfallback.test (embedded-sync + resolution pins), punk::repl stale comment fix (no behaviour change), project 0.17.8.

Verification evidence (all on suite build 9.0.5 runtimes, zig 0.16.0; driver scripts per the ## Notes recipe):

• Console testsuite: 76/76 pass incl the 5 new psfallback tests (native tclsh 9.0.3 runner).
• Hidden-console lifecycle selftest, tclsh90spr (has_twapi=0), against src modules AND built modules, plus zipfs kit tclsh90sprzip against built modules - all SELFTEST-PASS: first enable ~0.6-0.7s incl server spawn with confirmed 1 and -inputmode reading raw; warm toggles ~30ms; survived 25s idle then toggled in ~30ms (old keepalive died at 20s); explicit stop stopped 1; respawn cycle ok; both server generations reaped (explicit stop + parent-watch after process kill), zero orphans.
• Embedded-resolution forced run (bogus module_dir, scratchpad argv0/cwd): source=embedded, full enable/disable cycle + stop PASS - kits/unusual cwds need no scriptlib on disk.
• Acceptance shape (a): tclsh90spr src/make.tcl shell live repl in a hidden console; keystrokes injected via AttachConsole+WriteConsoleInput typing punk::console::enableRaw / punk::console::disableRaw at the prompt; external GetConsoleMode reads flipped 0x1E7 -> 0x1E1 -> 0x1E7 (LINE|ECHO cleared and restored); server child engaged; hard-kill of the shell -> server gone <=12s; stderr capture free of fallback noise (8 lines, all unrelated boot messages).
• Acceptance shape (b): tclsh90sprzip (zipfs kit) running the bare-runtime repl formula (tcl::tm::path add <repo>/modules; package require punk::repl; repl::init; repl::start stdin) - same injection sequence, same mode flips, engagement, reaping and quiet stderr.
• punk905 (punk kit WITH twapi): twapi path unaffected - enableRaw flips modes via twapi with no server spawned.
• ps1 standalone lifecycle smoke (pwsh -f, debug mode): start/ping/null-probe/bogus message/multi-connection survival/exit-message shutdown + watched-parent-death shutdown all pass; debug output only.

Remaining manual item: none gating. The user's interactive feel of raw mode on the suite runtimes (typing experience) is the natural follow-up confirmation; the mechanical acceptance criteria are all verified above. 8.6-runtime verification is recorded as residue against the G-099 arc (see ## Notes).