punkshell/goals/G-021-agent-visual-verification.md

G-021 Agent-drivable visual/UI verification via piped snapshot and interaction calls

Status: proposed Scope: src/modules/punk/ (G-020 module's agent-facing surface), AGENTS.md guidance (post G-015 pattern), src/tests/ (visual-verification test hooks) Acceptance: as in root GOALS.md index (canonical).

Context

Some behaviours can only be verified visually (rendering, theming, layout, ANSI-vs-Tk display parity) or by driving real input (mouse/keyboard-dependent flows). Today an agent in a session has no sanctioned way to see or drive GUI state. With G-020 providing capture/input/window-location as a module and G-015 providing reliable piped one-shot script calls, the combination lets a tool-calling agent: determine the applicable UI window, snapshot it, analyze the image (agents are multimodal), and drive interactions - all from ordinary tool calls during a session.

Approach

• Thin agent-facing surface over the G-020 module, exercised through the G-015 script subcommand: each operation is a single piped call with machine-parseable output and G-015 exit-code semantics (nonzero + stderr on failure, never an interactive shell).
• Operations: window list/match by pattern; snapshot a located window or region to a caller-specified PNG path; optional base64-on-stdout emission for direct image ingestion by the calling agent; scripted interaction sequences (focus, click at offset, key/type, snapshot) composable in one piped script.
• Windows first, per G-020. Capability introspection flows through, so an agent on an unsupported platform gets a clean "not supported here" rather than a hang or crash.
• Documentation for agents lands alongside the G-017 guidance (the same "use the punk mechanism, not ad-hoc scanning/screenshotting" pattern): exact invocations, output shapes, and the session prerequisites (unlocked desktop, DPI notes) from G-020's platform assessment.
• At least one real verification consumer in src/tests/ or a runbook (e.g. a Tk app smoke test asserting on a snapshot) proves the loop end-to-end.

Alternatives considered

• OS-level screenshot tooling driven directly by agents (PowerShell/GDI one-liners, screencapture, import/scrot) - rejected as the sanctioned path: per-platform incantations with no window-location or input story, no capability model, and nothing reusable by punk scripts; the punk module route gives one contract everywhere punk runs.
• Building agent tooling (MCP server or similar) directly into punk now - deferred: piped G-015 calls already fit existing agent Bash/exec tools; a richer integration can layer later without changing the module.

Notes

• Depends on G-020 (mechanism) and G-015 (reliable piped calls); guidance placement follows G-017.
• Image handoff sizing: agents ingest images with dimension/size limits; snapshot operations should support downscale/crop options eventually - acceptance keeps to save + base64, with sizing noted as a natural follow-on.