goals: G-021 fold in capture-privacy model (Tier 1)
Contract update (user-approved): Goal/Acceptance now make capture
disposition policy-governed - snapshots yield an opaque capture token by
default (no path or pixels in output), analysis relays token+prompt to
user-policy-permitted endpoints (local-only or allowlist) returning text
only, raw-image release (path/base64) is gated on explicit user config
with the default refusing, and every capture/disposition is audit-logged.
New "Privacy model (capture disposition)" section specifies the
mechanism: opaque tokens, mechanism-side endpoint checking, expose_pixels
policy (default analyze-only), policy config in user-profile scope
outside the repo, audit trail, AGENTS.md guidance as an additional layer
rather than the gate, and the honest enforcement ceiling (unrestricted
same-user exec can capture outside punk; target is private-by-default
sanctioned path + deliberate/auditable circumvention).
Notes record further hardening as a potential future task: separate-
principal broker with admin-owned config, and consent prompts - including
the trap that G-020 input injection can click an ordinary consent dialog,
so any prompt mode needs a surface SendInput cannot reach.
Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
Goal: a tool-calling agent can, during a session, use piped script calls (G-015) to a punk executable to locate the applicable UI window, snapshot it to a PNG file and/or base64 output suitable for AI image analysis, and drive mouse/keyboard interactions - enabling tests whose verification is visual-only and/or input-driven.
Acceptance: on Windows, single piped script calls (no interactive session) can: list/match windows for a pattern with machine-parseable output; save a located window's snapshot to a caller-specified path and optionally emit it base64 on stdout; run a scripted interaction sequence (focus, click at offset, type text, snapshot) end-to-end; failures exit nonzero with the error on stderr per G-015 semantics; the invocation patterns are documented for agents alongside the G-017 guidance; at least one real visual-or-input-driven verification (e.g. a Tk app smoke test) is exercised through this path.
Goal: a tool-calling agent can, during a session, use piped script calls (G-015) to a punk executable to locate the applicable UI window, snapshot it, and drive mouse/keyboard interactions - with capture disposition governed by user-configured privacy policy: by default the agent receives an opaque capture token and text-only analysis results relayed to user-approved endpoints, and raw pixel access (file path or base64) only where the user has explicitly enabled it - enabling tests whose verification is visual-only and/or input-driven.
Acceptance: on Windows, single piped script calls (no interactive session) can: list/match windows for a pattern with machine-parseable output; snapshot a located window receiving an opaque capture token by default, with no image path or pixel data in any output; submit a token plus prompt for analysis and receive text-only results, the image going only to an endpoint the user's policy config permits (local-only or allowlist) and a policy-refused endpoint failing nonzero; obtain the raw image (caller-specified PNG path and/or base64 on stdout) only when the user's policy config enables pixel release, the default policy refusing; every capture and disposition is recorded in an audit log; run a scripted interaction sequence (focus, click at offset, type text, snapshot) end-to-end; failures exit nonzero with the error on stderr per G-015 semantics; the invocation patterns and privacy model are documented for agents alongside the G-017 guidance; at least one real visual-or-input-driven verification (e.g. a Tk app smoke test) is exercised through this path.
## Context
@ -22,10 +22,12 @@ tool calls during a session.
`script` subcommand: each operation is a single piped call with
machine-parseable output and G-015 exit-code semantics (nonzero + stderr on
failure, never an interactive shell).
- Operations: window list/match by pattern; snapshot a located window or region
to a caller-specified PNG path; optional base64-on-stdout emission for direct
image ingestion by the calling agent; scripted interaction sequences (focus,
click at offset, key/type, snapshot) composable in one piped script.
- Operations: window list/match by pattern; snapshot a located window or region,
yielding an opaque capture token by default; analysis of a token via
user-approved endpoints returning text only; raw-image release (caller-specified
PNG path, base64-on-stdout) as policy-gated modes per the privacy model below;
scripted interaction sequences (focus, click at offset, key/type, snapshot)
composable in one piped script.
- Windows first, per G-020. Capability introspection flows through, so an agent
on an unsupported platform gets a clean "not supported here" rather than a
hang or crash.
@ -36,6 +38,50 @@ tool calls during a session.
- At least one real verification consumer in src/tests/ or a runbook (e.g. a
Tk app smoke test asserting on a snapshot) proves the loop end-to-end.
## Privacy model (capture disposition)
Captured images are private data from the driven machine; the agent must not get
incidental access to pixels or their storage location. Enforcement lives in the
mechanism, not in AGENTS.md instructions - guidance is an additional layer, not
the gate.
- **Opaque capture tokens**: the snapshot operation returns a token
(e.g. `cap:<random>`), never a path. Images are held by the capture-owning
process (in memory, or in a spool location with randomized names that never
appears in any output). No operation echoes a path or pixel data unless policy
permits it.
- **Analysis relay**: an `analyze` operation takes a token plus a prompt; the
mechanism sends the image to a vision endpoint and returns only the text
result. The endpoint set is user policy: `local-only` (loopback addresses /
named pipes only, e.g. a local vision model) or an explicit allowlist of
specific endpoints. The mechanism resolves and checks the destination itself;
the agent cannot redirect it. A policy-refused endpoint fails nonzero per
G-015 semantics.
- **Pixel-release policy**: a user config setting on the order of
`expose_pixels = never | analyze-only | allow` gates whether the raw image
(file path or base64) can ever be returned to the caller. Default
`analyze-only`.
- **Policy config outside agent-writable scope**: the policy lives in
user-profile configuration - never in the repo, never in AGENTS.md. (A
same-user agent with arbitrary exec could still edit a user-profile file;
see the hardening note for the stronger boundary.)
- **Audit trail**: every capture and every disposition is logged (timestamp,
window matched, image hash, destination endpoint or release mode), so
circumvention is detectable even where it is not preventable.
- **AGENTS.md as an additional layer**: the agent guidance names the punk
capture path as the only sanctioned way to observe GUI state and prohibits
OS-level screenshot one-liners and spool-directory hunting. This keeps a
compliant agent on the private-by-default path; the mechanism plus audit log
handle the rest.
- **Honest enforcement ceiling**: an agent with unrestricted same-user command
execution on the driven machine can capture the screen outside punk entirely;
no in-punk mechanism can prevent that. The design target is: the sanctioned
path is private by default, and circumvention requires deliberate, visible,
auditable steps rather than incidental access. The ceiling rises substantially
when the agent's exec is sandboxed/permission-gated or the agent drives the
machine remotely (no local filesystem access) - the capture interface should
stay transport-agnostic so the remote topology works unchanged.
## Alternatives considered
- OS-level screenshot tooling driven directly by agents (PowerShell/GDI
@ -53,4 +99,14 @@ tool calls during a session.
placement follows G-017.
- Image handoff sizing: agents ingest images with dimension/size limits;
snapshot operations should support downscale/crop options eventually -
acceptance keeps to save + base64, with sizing noted as a natural follow-on.
acceptance keeps to token + policy-gated release, with sizing noted as a
