punkshell/goals/G-020-screencap-input-module.md

G-020 Screen capture and input injection module with per-platform backends (Windows first)

Status: proposed Scope: src/modules/punk/ (new module - name TBD), src/vfs/punk9win.vfs/lib_tcl9/ (existing treectrl/Img/twapi payloads); scriptlib/aloupe.tcl stays untouched as a standalone app Goal: a punk module drives screen/window capture and mouse/keyboard injection from scripts via per-platform backends - Windows (treectrl loupe capture + twapi input/window-location) is the initial complete target, with the backend contract designed so X11 (Linux/FreeBSD) and macOS backends can be added without changing callers, and Wayland-native sessions explicitly out of scope. Acceptance: on Windows from a punk shell or script: a screen region and a window located by title/class pattern are each captured to a Tk photo and written as a valid PNG; mouse movement/click and key events injected into a located test window produce their observable effect (typed text arrives, click acts); window location returns the handle and geometry for a pattern; a capability-introspection call reports per-feature support and an unsupported platform/backend yields a clean capability-based refusal, not a crash; the backend interface (capture / input / window-locate) is documented well enough that a non-Windows backend can be added without modifying callers; the aloupe script remains functional and unmodified.

Context

Punk already runs Tk from tclsh-style kits with the console retained (Tk is a loadable package in the kit; the repl event loop services it - no wish executable involved, and wish binaries are not a build target). The standalone scriptlib/aloupe.tcl (aloupe 1.8.2, MIT) demonstrates the capture primitive: treectrl's loupe command copies an arbitrary screen region into a Tk photo (aloupe.tcl ~454), with Img for extra file formats. The Windows kit already ships every needed binary: src/vfs/punk9win.vfs/lib_tcl9/ has tk9.0, treectrl 2.5.2 (tcl9treectrl252.dll), Img 2.0.1/2.1.0, twapi 5.0.2, and cffi 2.0.3.

This goal turns that demonstration into a scriptable module: capture plus mouse/keyboard injection plus window location, so scripts (and, via G-021, agents) can drive and observe GUI state. aloupe.tcl remains a standalone app; the module is a separate development, not a refactor of it.

Platform assessment (investigated 2026-07-06)

• Windows (initial primary target) - no show-stoppers, all deps already in the kit. Capture: treectrl loupe (GDI). Input: twapi send_input (SendInput). Window locate: twapi enumeration/geometry/foreground. Gotchas to handle/document: desktop session must be unlocked and interactive (locked workstation / disconnected RDP -> black captures, failed injection); per-monitor DPI scaling affects coordinate mapping; UIPI blocks injecting into elevated windows from a non-elevated process; multi-monitor negative coordinates.
• Linux (X11) - supported: treectrl loupe X11 impl (XGetImage); injection via XTest (xdotool external tool, or cffi binding to libXtst); locate via xdotool/wmctrl. Xvfb gives a fully headless mode - the preferred arrangement for automated visual verification.
• Wayland-native - OUT OF SCOPE (decided 2026-07-06): XGetImage cannot see native Wayland surfaces and XTest cannot inject into them by design; the escape hatches (xdg-desktop-portal Screenshot/ScreenCast D-Bus APIs, ydotool via uinput) are permission-heavy with no ready Tcl bindings. Revisit via portals if/when needed; X11/Xvfb is the supported Linux mode.
• FreeBSD - same X11 story; tktreectrl/tkimg/xdotool in ports; twapi N/A.
• macOS - supportable, permission-gated: TCC requires Screen Recording (capture) and Accessibility (injection) grants to the hosting process - interactive one-time grants, awkward for unattended CI. treectrl loupe aqua support is unverified and non-blocking: the built-in screencapture CLI (region/window-id to file) is the natural capture backend; injection options are osascript/System Events, cliclick (third-party), or cffi -> CGEventPost. src/vfs/macosx-arm/ already exists in the tree.

Approach

• Three backend roles behind one API: capture, input, window-locate. Per-platform backend selection with capability introspection - callers ask what is supported rather than probing by failure. Echoes the G-001 pluggable pattern.
• Windows backend first and complete (acceptance gates on it - confirmed 2026-07-07: a full testable system on Windows must be reachable without first implementing the other platforms). X11 and macOS backends are in-scope follow-ons behind the same contract; their absence on a platform must produce a clean capability-based refusal.
• Output normalization: capture lands in a Tk photo; PNG via Tk's core photo support (Img optional for other formats). File save and in-memory (base64) both supported - the G-021 consumer needs both.
• cffi (bundled) is the escape hatch for gaps without new C extensions - e.g. Windows PrintWindow for capturing occluded windows (region-based loupe cannot), libXtst on X11 if avoiding the xdotool external-tool dependency, CGEventPost on macOS.
• Binary deps (treectrl, Img, twapi, cffi) follow the G-004/G-005/G-006 provisioning story on each platform; nothing new is committed to the repo.

Alternatives considered

• Refactoring aloupe.tcl into the module - rejected: it is a vendored-style standalone app under user-only scriptlib/; keeping it untouched preserves a known-working reference implementation.
• Pure-twapi capture on Windows (skip treectrl) - not pursued initially: twapi has no ready BitBlt/screen-grab wrapper; treectrl loupe is already shipped and proven. cffi/PrintWindow may supplement later for occluded windows.
• Wayland-native support via portals/ydotool - out of scope (see platform assessment); recorded for possible future revisit.

Notes

• Launch path (2026-07-07): the script subcommand (G-015) services a registered Tk main loop after the script, matching tclsh - so a Tk app can be launched via <punkexe> script app.tcl and stays alive to be snapshotted and driven, without needing the shell subcommand. This is the intended entry point for GUI automation here.
• Related: G-018 (kit composition; Tk as loadable package, no wish binaries), G-019 (a trimmed capture-capable executable is a plausible scan target), G-021 (agent-facing surface over this module), G-001 (backend plugin pattern).
• Archived-goal references in this file: G-001 achieved 2026-07-11 (goals/archive/G-001-pluggable-console-backends.md); G-015 achieved 2026-07-07 (goals/archive/G-015-script-subcommand-piped-stdin.md).

Additional driving use case: repl interactive-behaviour verification (2026-07-11)

Characterizing punkshell's interactive repl behaviour (raw-mode colour staging, tab markers and dim space dots in the editbuf, closing-prompt hints, multiline history navigation - see the preserve-list in goals/G-044 detail) currently has no automated harness: the underlying editbuf is console-coupled and there is no expect-like system. This module is the practical near-term bridge on windows: inject keystrokes at a live punkshell window and capture/compare the rendered region. Coarser than a pseudoconsole expect-alternative (the durable successor - candidate goal drafted 2026-07-11) but available sooner, and it can verify exactly the rendered-behaviour tier that unit tests cannot reach. Worth weighting this goal's priority accordingly when sequencing repl refactor work.