From 89daa99269e31611c771473f4c15f7f852dd8fdd Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Sat, 11 Jul 2026 02:27:05 +1000 Subject: [PATCH] GOALS: add G-061 [proposed] pseudoconsole expect-alternative for interactive shell testing - ConPTY (windows) + pty (unix) expect-like harness: spawn a built punkshell on a size-controlled pseudo-terminal, send keys incl control/arrow sequences, await patterns in the rendered ANSI byte stream with timeouts and forced teardown, full transcript capture - the durable byte-accurate tier of the repl verification model recorded in goals/G-044 detail: pure layer (commandcomplete.test, done), console-seam unit tier (G-001 as enabling refactor), rendered tiers (G-020 screen capture near-term windows bridge; this goal headless + cross-platform) - first verification targets = preserve-list behaviours: the in-proc {set x "{*}{"} continuation-hint sequence, raw-mode tab-marker rendering/deletion, up-arrow recall + edit of a multiline history entry - detail file records open implementation questions (driving ConPTY from tcl: twapi surface vs helper exe; harness/shell synchronisation incl the inter-subshell beep-protocol idea vs prompt sentinels) and the G-059 WSL / G-060 QEMU-guest execution extensions Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- GOALS.md | 6 +++ goals/G-061-pseudoconsole-expect.md | 78 +++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+) create mode 100644 goals/G-061-pseudoconsole-expect.md diff --git a/GOALS.md b/GOALS.md index 252f4dc2..783ef013 100644 --- a/GOALS.md +++ b/GOALS.md @@ -409,3 +409,9 @@ Scope: build/test orchestration config and scripts (location to be settled with Detail: goals/G-060-qemu-test-matrix.md Goal: comprehensive cross-platform verification (linux variants, FreeBSD/NetBSD/OpenBSD, arm architectures) is achievable from a single dev machine by driving QEMU guests as strictly external tooling - punkshell invokes qemu as a separate process with declarative per-guest config, and nothing in punkshell or its artifact repos bundles, links against, derives from or otherwise couples to QEMU (GPLv2) - so the project's BSD licensing posture is unaffected and QEMU remains a swappable convenience: the guest-driving contract is push-based artifact staging and result collection (the same pattern G-059 establishes for WSL - guests never share a working tree with the host) and must be satisfiable by real hardware, WSL or another hypervisor. Acceptance: a documented, repeatable workflow provisions at least one non-Windows QEMU guest (e.g FreeBSD x86_64) that fetches a punkbin runtime and runs the source-tree suite inside the guest, driven from the Windows dev machine with results collected back; the guest-driving interface is hypervisor-agnostic (documented contract; QEMU is one provider); the license posture is recorded in the detail file (external-process invocation only, no QEMU binaries or derived code committed to punkshell or punkbin, guest OS images not redistributed by the project - fetched/built per machine); suite runs on machines without QEMU are unaffected. + +### G-061 [proposed] Pseudoconsole expect-alternative for interactive shell testing (ConPTY + unix pty) +Scope: test-harness support (location TBD during the work: src/tests/testsupport/ and/or a small punk module), src/tests/shell/ (capability-gated interactive suites), goals/G-061-pseudoconsole-expect.md +Detail: goals/G-061-pseudoconsole-expect.md +Goal: interactive punkshell behaviour is testable headlessly and byte-accurately by driving a real shell under a pseudoconsole - ConPTY on windows, pty on unix - through an expect-like harness (spawn, send literal keys and control/arrow sequences, await patterns in the rendered ANSI byte stream with timeouts, collect transcript), giving the goals/G-044-detail preserve-list behaviours (continuation hints, raw-mode colour staging, tab markers and space dots, multiline history navigation) their durable cross-platform verification tier - complementary to G-020 capture testing (which verifies what reaches the screen; this verifies the byte stream) and to the G-001 console-seam unit tier. +Acceptance: a capability-gated harness spawns a built punkshell under ConPTY on windows (and under a unix pty when run on unix or via the G-059 WSL staging pattern), sends keys including control/arrow sequences, and awaits expected patterns with timeouts and forced teardown on expiry (hang-proof per the shell-test conventions); at least three preserve-list behaviours are verified through it (e.g the in-proc `set x "{*}{"` continuation-hint sequence, raw-mode tab-marker rendering and deletion, up-arrow recall and edit of a multiline history entry); tests skip cleanly where no pseudoconsole is available; the harness API is platform-agnostic with per-OS backends; candidate synchronisation/communication mechanisms between harness and shell (including the inter-subshell beep-protocol idea) are evaluated and the chosen approach documented in the detail file. diff --git a/goals/G-061-pseudoconsole-expect.md b/goals/G-061-pseudoconsole-expect.md new file mode 100644 index 00000000..f1220a65 --- /dev/null +++ b/goals/G-061-pseudoconsole-expect.md @@ -0,0 +1,78 @@ +# G-061 Pseudoconsole expect-alternative for interactive shell testing (ConPTY + unix pty) + +Status: proposed +Scope: test-harness support (location TBD during the work: src/tests/testsupport/ and/or a small punk module), src/tests/shell/ (capability-gated interactive suites), goals/G-061-pseudoconsole-expect.md +Acceptance: see GOALS.md index entry (canonical). + +## Context + +punkshell's interactive behaviour has no automated verification. The preserve-list +recorded in goals/G-044-repl-command-completion.md (2026-07-11, user-specified) covers +what any repl refactor must not break: info-complete-parity continuation with the Tcl +quoting quirk, closing-prompt hints, raw-mode colour staging (in-progress vs submitted), +literal tab acceptance with raw-mode tab markers and edit smarts, dim grey space dots +(display-only - never in the submitted string), and up/down navigation + editing of +multiline editbuf history entries. + +The verification tier model established 2026-07-11: + +1. PURE layer - testable now: punk::lib::system::incomplete characterized in + src/tests/modules/punk/lib/testsuites/lib/commandcomplete.test. +2. Console-seam UNIT tier - blocked on a seam: class_editbuf is console-coupled at its + core (add_chunk renders via overtype::renderline against live terminal metrics - + cursor-position size probing, DECRQPSR tabstops). An ::opunk::Console test double + ([[G-001]]) or injectable metrics is the enabling refactor. +3. RENDERED tier: [[G-020]] screencap+input injection is the near-term windows-only + bridge (verifies what reaches the screen); THIS goal is the durable, headless, + cross-platform, byte-accurate mechanism (verifies the byte stream the shell emits). + +## Approach + +### Harness contract (platform-agnostic; per-OS backends) + +expect-like primitives over a REAL pseudo-terminal: +- spawn: launch a built punkshell attached to a pseudoconsole (ConPTY on windows, + openpty/forkpty on unix) with controlled size (cols x rows) - size control matters + because raw-mode rendering is width-dependent. +- send: literal keys plus control/arrow/escape sequences (up-arrow history recall, + backspace over a tab marker, etc). +- expect: await patterns in the rendered ANSI byte stream with timeouts; forced + teardown on expiry (hang-proof, per src/tests/shell/AGENTS.md conventions - the + punk_run pattern generalized to a live conversation). +- transcript: full byte capture for post-hoc assertions (colour staging = SGR + sequences around the in-progress vs submitted line; tab markers/space dots = the + substituted glyph bytes). + +### Open implementation questions (deliberately left to the work) + +- Driving ConPTY from Tcl: candidates include a twapi surface (if CreatePseudoConsole + is reachable), a small helper executable owning the ConPTY pair and relaying bytes + over stdio/sockets, or an existing tool at arm's length. The unix side is simpler + (openpty via a small C/critcl shim, or driving through an existing pty-capable tool). +- Synchronisation between harness and shell: pattern-awaiting alone can be racy for + timing-sensitive raw-mode behaviour. Candidate: the inter-subshell communication / + beep-protocol idea (user, 2026-07-11) - an in-band signalling channel the shell can + emit for the harness to key off. Evaluate against simpler prompt-sentinel approaches; + document the choice here. +- Unix execution from the windows dev machine: the [[G-059]] WSL staging pattern + (native-filesystem staging, capability constraint) extends naturally - a pty harness + run inside the distro against a linux punkbin runtime/kit. + +### Sequencing + +- [[G-001]] console-seam unit tier and [[G-020]] capture tier are complementary, not + prerequisites - but preserve-list items verified at a cheaper tier need not be + re-verified here beyond smoke coverage. +- First verification targets (per acceptance): the in-proc `set x "{*}{"` + continuation-hint sequence, raw-mode tab-marker rendering + deletion, up-arrow recall + and edit of a multiline history entry. + +## Notes + +- Related: [[G-044]] (completion/hinting - consumes this for its own verification), + [[G-001]] (fake-console unit tier), [[G-020]] (screen-capture tier), [[G-059]] (WSL + execution pattern), [[G-060]] (guest-driving contract - a pty harness inside a QEMU + guest is the eventual cross-platform matrix combination). +- ConPTY availability gates the windows side to Win10 1809+ (fine for punkshell's + supported baseline); the capability probe should verify actual ConPTY function, not + windows version.