punkshell/goals/G-021-agent-visual-verification.md

G-021 Agent-drivable visual/UI verification via piped snapshot and interaction calls

Status: proposed Scope: src/modules/punk/ (G-020 module's agent-facing surface), AGENTS.md guidance (post G-015 pattern), src/tests/ (visual-verification test hooks) Goal: a tool-calling agent can, during a session, use piped script calls (G-015) to a punk executable to locate the applicable UI window, snapshot it, and drive mouse/keyboard interactions - with capture disposition governed by user-configured privacy policy: by default the agent receives an opaque capture token and text-only analysis results relayed to user-approved endpoints, and raw pixel access (file path or base64) only where the user has explicitly enabled it - enabling tests whose verification is visual-only and/or input-driven. Acceptance: on Windows, single piped script calls (no interactive session) can: list/match windows for a pattern with machine-parseable output; snapshot a located window receiving an opaque capture token by default, with no image path or pixel data in any output; submit a token plus prompt for analysis and receive text-only results, the image going only to an endpoint the user's policy config permits (local-only or allowlist) and a policy-refused endpoint failing nonzero; obtain the raw image (caller-specified PNG path and/or base64 on stdout) only when the user's policy config enables pixel release, the default policy refusing; every capture and disposition is recorded in an audit log; run a scripted interaction sequence (focus, click at offset, type text, snapshot) end-to-end; failures exit nonzero with the error on stderr per G-015 semantics; the invocation patterns and privacy model are documented for agents alongside the G-017 guidance; at least one real visual-or-input-driven verification (e.g. a Tk app smoke test) is exercised through this path.

Context

Some behaviours can only be verified visually (rendering, theming, layout, ANSI-vs-Tk display parity) or by driving real input (mouse/keyboard-dependent flows). Today an agent in a session has no sanctioned way to see or drive GUI state. With G-020 providing capture/input/window-location as a module and G-015 providing reliable piped one-shot script calls, the combination lets a tool-calling agent: determine the applicable UI window, snapshot it, analyze the image (agents are multimodal), and drive interactions - all from ordinary tool calls during a session.

Approach

• Thin agent-facing surface over the G-020 module, exercised through the G-015 script subcommand: each operation is a single piped call with machine-parseable output and G-015 exit-code semantics (nonzero + stderr on failure, never an interactive shell).
• Operations: window list/match by pattern; snapshot a located window or region, yielding an opaque capture token by default; analysis of a token via user-approved endpoints returning text only; raw-image release (caller-specified PNG path, base64-on-stdout) as policy-gated modes per the privacy model below; scripted interaction sequences (focus, click at offset, key/type, snapshot) composable in one piped script.
• Windows first, per G-020. Capability introspection flows through, so an agent on an unsupported platform gets a clean "not supported here" rather than a hang or crash.
• Documentation for agents lands alongside the G-017 guidance (the same "use the punk mechanism, not ad-hoc scanning/screenshotting" pattern): exact invocations, output shapes, and the session prerequisites (unlocked desktop, DPI notes) from G-020's platform assessment.
• At least one real verification consumer in src/tests/ or a runbook (e.g. a Tk app smoke test asserting on a snapshot) proves the loop end-to-end.

Privacy model (capture disposition)

Captured images are private data from the driven machine; the agent must not get incidental access to pixels or their storage location. Enforcement lives in the mechanism, not in AGENTS.md instructions - guidance is an additional layer, not the gate.

• Opaque capture tokens: the snapshot operation returns a token (e.g. cap:<random>), never a path. Images are held by the capture-owning process (in memory, or in a spool location with randomized names that never appears in any output). No operation echoes a path or pixel data unless policy permits it.
• Analysis relay: an analyze operation takes a token plus a prompt; the mechanism sends the image to a vision endpoint and returns only the text result. The endpoint set is user policy: local-only (loopback addresses / named pipes only, e.g. a local vision model) or an explicit allowlist of specific endpoints. The mechanism resolves and checks the destination itself; the agent cannot redirect it. A policy-refused endpoint fails nonzero per G-015 semantics.
• Pixel-release policy: a user config setting on the order of expose_pixels = never | analyze-only | allow gates whether the raw image (file path or base64) can ever be returned to the caller. Default analyze-only.
• Policy config outside agent-writable scope: the policy lives in user-profile configuration - never in the repo, never in AGENTS.md. (A same-user agent with arbitrary exec could still edit a user-profile file; see the hardening note for the stronger boundary.)
• Audit trail: every capture and every disposition is logged (timestamp, window matched, image hash, destination endpoint or release mode), so circumvention is detectable even where it is not preventable.
• AGENTS.md as an additional layer: the agent guidance names the punk capture path as the only sanctioned way to observe GUI state and prohibits OS-level screenshot one-liners and spool-directory hunting. This keeps a compliant agent on the private-by-default path; the mechanism plus audit log handle the rest.
• Honest enforcement ceiling: an agent with unrestricted same-user command execution on the driven machine can capture the screen outside punk entirely; no in-punk mechanism can prevent that. The design target is: the sanctioned path is private by default, and circumvention requires deliberate, visible, auditable steps rather than incidental access. The ceiling rises substantially when the agent's exec is sandboxed/permission-gated or the agent drives the machine remotely (no local filesystem access) - the capture interface should stay transport-agnostic so the remote topology works unchanged.

Alternatives considered

• OS-level screenshot tooling driven directly by agents (PowerShell/GDI one-liners, screencapture, import/scrot) - rejected as the sanctioned path: per-platform incantations with no window-location or input story, no capability model, and nothing reusable by punk scripts; the punk module route gives one contract everywhere punk runs.
• Building agent tooling (MCP server or similar) directly into punk now - deferred: piped G-015 calls already fit existing agent Bash/exec tools; a richer integration can layer later without changing the module.

Notes

• Depends on G-020 (mechanism) and G-015 (reliable piped calls - achieved 2026-07-07); guidance placement follows G-017.
• Image handoff sizing: agents ingest images with dimension/size limits; snapshot operations should support downscale/crop options eventually - acceptance keeps to token + policy-gated release, with sizing noted as a natural follow-on.
• Privacy hardening beyond this goal's scope - potential down-the-track task (candidate goal if/when wanted): (a) run the capture broker as a separate principal (Windows service or second account) with admin-owned policy config, so reading the spool or editing policy requires elevation rather than mere same-user access; (b) user consent prompts for pixel release. Design trap for (b), recorded now because it constrains any future prompt mode: G-020's own input injection (SendInput) can click an ordinary consent dialog on the driven desktop - a real consent gate must live where SendInput cannot reach (secure desktop, UAC-style, or out-of-band confirmation).