From 456fa80500f4af1a475317f2c472d2aff3f5b276 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Sun, 12 Jul 2026 00:40:57 +1000 Subject: [PATCH] goals: G-021 fold in capture-privacy model (Tier 1) Contract update (user-approved): Goal/Acceptance now make capture disposition policy-governed - snapshots yield an opaque capture token by default (no path or pixels in output), analysis relays token+prompt to user-policy-permitted endpoints (local-only or allowlist) returning text only, raw-image release (path/base64) is gated on explicit user config with the default refusing, and every capture/disposition is audit-logged. New "Privacy model (capture disposition)" section specifies the mechanism: opaque tokens, mechanism-side endpoint checking, expose_pixels policy (default analyze-only), policy config in user-profile scope outside the repo, audit trail, AGENTS.md guidance as an additional layer rather than the gate, and the honest enforcement ceiling (unrestricted same-user exec can capture outside punk; target is private-by-default sanctioned path + deliberate/auditable circumvention). Notes record further hardening as a potential future task: separate- principal broker with admin-owned config, and consent prompts - including the trap that G-020 input injection can click an ordinary consent dialog, so any prompt mode needs a surface SendInput cannot reach. Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- goals/G-021-agent-visual-verification.md | 70 +++++++++++++++++++++--- 1 file changed, 63 insertions(+), 7 deletions(-) diff --git a/goals/G-021-agent-visual-verification.md b/goals/G-021-agent-visual-verification.md index 60775899..e555fb64 100644 --- a/goals/G-021-agent-visual-verification.md +++ b/goals/G-021-agent-visual-verification.md @@ -2,8 +2,8 @@ Status: proposed Scope: src/modules/punk/ (G-020 module's agent-facing surface), AGENTS.md guidance (post G-015 pattern), src/tests/ (visual-verification test hooks) -Goal: a tool-calling agent can, during a session, use piped script calls (G-015) to a punk executable to locate the applicable UI window, snapshot it to a PNG file and/or base64 output suitable for AI image analysis, and drive mouse/keyboard interactions - enabling tests whose verification is visual-only and/or input-driven. -Acceptance: on Windows, single piped script calls (no interactive session) can: list/match windows for a pattern with machine-parseable output; save a located window's snapshot to a caller-specified path and optionally emit it base64 on stdout; run a scripted interaction sequence (focus, click at offset, type text, snapshot) end-to-end; failures exit nonzero with the error on stderr per G-015 semantics; the invocation patterns are documented for agents alongside the G-017 guidance; at least one real visual-or-input-driven verification (e.g. a Tk app smoke test) is exercised through this path. +Goal: a tool-calling agent can, during a session, use piped script calls (G-015) to a punk executable to locate the applicable UI window, snapshot it, and drive mouse/keyboard interactions - with capture disposition governed by user-configured privacy policy: by default the agent receives an opaque capture token and text-only analysis results relayed to user-approved endpoints, and raw pixel access (file path or base64) only where the user has explicitly enabled it - enabling tests whose verification is visual-only and/or input-driven. +Acceptance: on Windows, single piped script calls (no interactive session) can: list/match windows for a pattern with machine-parseable output; snapshot a located window receiving an opaque capture token by default, with no image path or pixel data in any output; submit a token plus prompt for analysis and receive text-only results, the image going only to an endpoint the user's policy config permits (local-only or allowlist) and a policy-refused endpoint failing nonzero; obtain the raw image (caller-specified PNG path and/or base64 on stdout) only when the user's policy config enables pixel release, the default policy refusing; every capture and disposition is recorded in an audit log; run a scripted interaction sequence (focus, click at offset, type text, snapshot) end-to-end; failures exit nonzero with the error on stderr per G-015 semantics; the invocation patterns and privacy model are documented for agents alongside the G-017 guidance; at least one real visual-or-input-driven verification (e.g. a Tk app smoke test) is exercised through this path. ## Context @@ -22,10 +22,12 @@ tool calls during a session. `script` subcommand: each operation is a single piped call with machine-parseable output and G-015 exit-code semantics (nonzero + stderr on failure, never an interactive shell). -- Operations: window list/match by pattern; snapshot a located window or region - to a caller-specified PNG path; optional base64-on-stdout emission for direct - image ingestion by the calling agent; scripted interaction sequences (focus, - click at offset, key/type, snapshot) composable in one piped script. +- Operations: window list/match by pattern; snapshot a located window or region, + yielding an opaque capture token by default; analysis of a token via + user-approved endpoints returning text only; raw-image release (caller-specified + PNG path, base64-on-stdout) as policy-gated modes per the privacy model below; + scripted interaction sequences (focus, click at offset, key/type, snapshot) + composable in one piped script. - Windows first, per G-020. Capability introspection flows through, so an agent on an unsupported platform gets a clean "not supported here" rather than a hang or crash. @@ -36,6 +38,50 @@ tool calls during a session. - At least one real verification consumer in src/tests/ or a runbook (e.g. a Tk app smoke test asserting on a snapshot) proves the loop end-to-end. +## Privacy model (capture disposition) + +Captured images are private data from the driven machine; the agent must not get +incidental access to pixels or their storage location. Enforcement lives in the +mechanism, not in AGENTS.md instructions - guidance is an additional layer, not +the gate. + +- **Opaque capture tokens**: the snapshot operation returns a token + (e.g. `cap:`), never a path. Images are held by the capture-owning + process (in memory, or in a spool location with randomized names that never + appears in any output). No operation echoes a path or pixel data unless policy + permits it. +- **Analysis relay**: an `analyze` operation takes a token plus a prompt; the + mechanism sends the image to a vision endpoint and returns only the text + result. The endpoint set is user policy: `local-only` (loopback addresses / + named pipes only, e.g. a local vision model) or an explicit allowlist of + specific endpoints. The mechanism resolves and checks the destination itself; + the agent cannot redirect it. A policy-refused endpoint fails nonzero per + G-015 semantics. +- **Pixel-release policy**: a user config setting on the order of + `expose_pixels = never | analyze-only | allow` gates whether the raw image + (file path or base64) can ever be returned to the caller. Default + `analyze-only`. +- **Policy config outside agent-writable scope**: the policy lives in + user-profile configuration - never in the repo, never in AGENTS.md. (A + same-user agent with arbitrary exec could still edit a user-profile file; + see the hardening note for the stronger boundary.) +- **Audit trail**: every capture and every disposition is logged (timestamp, + window matched, image hash, destination endpoint or release mode), so + circumvention is detectable even where it is not preventable. +- **AGENTS.md as an additional layer**: the agent guidance names the punk + capture path as the only sanctioned way to observe GUI state and prohibits + OS-level screenshot one-liners and spool-directory hunting. This keeps a + compliant agent on the private-by-default path; the mechanism plus audit log + handle the rest. +- **Honest enforcement ceiling**: an agent with unrestricted same-user command + execution on the driven machine can capture the screen outside punk entirely; + no in-punk mechanism can prevent that. The design target is: the sanctioned + path is private by default, and circumvention requires deliberate, visible, + auditable steps rather than incidental access. The ceiling rises substantially + when the agent's exec is sandboxed/permission-gated or the agent drives the + machine remotely (no local filesystem access) - the capture interface should + stay transport-agnostic so the remote topology works unchanged. + ## Alternatives considered - OS-level screenshot tooling driven directly by agents (PowerShell/GDI @@ -53,4 +99,14 @@ tool calls during a session. placement follows G-017. - Image handoff sizing: agents ingest images with dimension/size limits; snapshot operations should support downscale/crop options eventually - - acceptance keeps to save + base64, with sizing noted as a natural follow-on. + acceptance keeps to token + policy-gated release, with sizing noted as a + natural follow-on. +- Privacy hardening beyond this goal's scope - potential down-the-track task + (candidate goal if/when wanted): (a) run the capture broker as a separate + principal (Windows service or second account) with admin-owned policy config, + so reading the spool or editing policy requires elevation rather than mere + same-user access; (b) user consent prompts for pixel release. Design trap for + (b), recorded now because it constrains any future prompt mode: G-020's own + input injection (SendInput) can click an ordinary consent dialog on the driven + desktop - a real consent gate must live where SendInput cannot reach (secure + desktop, UAC-style, or out-of-band confirmation).