# G-021 Agent-drivable visual/UI verification via piped snapshot and interaction calls

Status: proposed
Scope: src/modules/punk/ (G-020 module's agent-facing surface), AGENTS.md guidance (post G-015 pattern), src/tests/ (visual-verification test hooks)
Goal: a tool-calling agent can, during a session, use piped script calls (G-015) to a punk executable to locate the applicable UI window, snapshot it to a PNG file and/or base64 output suitable for AI image analysis, and drive mouse/keyboard interactions - enabling tests whose verification is visual-only and/or input-driven.
Acceptance: on Windows, single piped script calls (no interactive session) can: list/match windows for a pattern with machine-parseable output; save a located window's snapshot to a caller-specified path and optionally emit it base64 on stdout; run a scripted interaction sequence (focus, click at offset, type text, snapshot) end-to-end; failures exit nonzero with the error on stderr per G-015 semantics; the invocation patterns are documented for agents alongside the G-017 guidance; at least one real visual-or-input-driven verification (e.g. a Tk app smoke test) is exercised through this path.

## Context

Some behaviours can only be verified visually (rendering, theming, layout,
ANSI-vs-Tk display parity) or by driving real input (mouse/keyboard-dependent
flows). Today an agent in a session has no sanctioned way to see or drive GUI
state. With G-020 providing capture/input/window-location as a module and
G-015 providing reliable piped one-shot script calls, the combination lets a
tool-calling agent: determine the applicable UI window, snapshot it, analyze
the image (agents are multimodal), and drive interactions - all from ordinary
tool calls during a session.

## Approach

- Thin agent-facing surface over the G-020 module, exercised through the G-015
  `script` subcommand: each operation is a single piped call with
  machine-parseable output and G-015 exit-code semantics (nonzero + stderr on
  failure, never an interactive shell).
- Operations: window list/match by pattern; snapshot a located window or region
  to a caller-specified PNG path; optional base64-on-stdout emission for direct
  image ingestion by the calling agent; scripted interaction sequences (focus,
  click at offset, key/type, snapshot) composable in one piped script.
- Windows first, per G-020. Capability introspection flows through, so an agent
  on an unsupported platform gets a clean "not supported here" rather than a
  hang or crash.
- Documentation for agents lands alongside the G-017 guidance (the same "use
  the punk mechanism, not ad-hoc scanning/screenshotting" pattern): exact
  invocations, output shapes, and the session prerequisites (unlocked desktop,
  DPI notes) from G-020's platform assessment.
- At least one real verification consumer in src/tests/ or a runbook (e.g. a
  Tk app smoke test asserting on a snapshot) proves the loop end-to-end.

## Alternatives considered

- OS-level screenshot tooling driven directly by agents (PowerShell/GDI
  one-liners, screencapture, import/scrot) - rejected as the sanctioned path:
  per-platform incantations with no window-location or input story, no
  capability model, and nothing reusable by punk scripts; the punk module route
  gives one contract everywhere punk runs.
- Building agent tooling (MCP server or similar) directly into punk now -
  deferred: piped G-015 calls already fit existing agent Bash/exec tools; a
  richer integration can layer later without changing the module.

## Notes

- Depends on G-020 (mechanism) and G-015 (reliable piped calls - achieved 2026-07-07); guidance
  placement follows G-017.
- Image handoff sizing: agents ingest images with dimension/size limits;
  snapshot operations should support downscale/crop options eventually -
  acceptance keeps to save + base64, with sizing noted as a natural follow-on.