punkshell/goals/G-065-declarative-vendoring.md

G-065 Declarative vendoring: toml-declared external packages with pinning, provenance and binary gating

Status: proposed Scope: punkproject.toml or sibling vendor manifest (schema - settled in the work), src/modules/punk/mix/ (vendor-sync command surface), src/vendorlib/ + src/vendormodules/ + src/vfs/ (materialization targets), src/make.tcl (integration) Goal: vendoring an external third-party package into punkshell (and, via the G-027 channel, derived projects) is driven by a toml declaration - upstream source, optional version/commit pin, trim rules - materialized and updated by one command that records retrieval provenance (upstream URL, commit/tag, retrieval date, license-verification provenance per G-063) and scans for executable binaries, refusing or warning unless punkproject.toml carries an explicit binaries-allowed override; manual drop-ins (a library folder or .tm file placed by hand, in vendor dirs or vfs) remain supported and are surfaced by audit as undeclared rather than rejected; basic vendoring requires no agent/LLM tooling. Acceptance: a documented toml manifest schema exists and declaring a package plus running the sync command materializes it under src/vendorlib or src/vendormodules (and a vfs target where declared) on a system without agents; re-running after a pin change updates the materialized copy and an unpinned declaration records its resolved version at sync time; per-package provenance (upstream URL, commit/tag or version, retrieval date) is recorded and queryable; a declared package containing executable binaries is refused (warn-only selectable as a configured mode) unless the punkproject.toml override is present, exercised by a test or documented manual verification against a scratch package; a manually dropped-in library remains loadable and is reported as undeclared by the audit/status surface, not deleted; the ecosystem-metadata compatibility survey (teapot Meta headers, relevant TIPs, wiki.tcl-lang.org / tip.tcl.tk scan) is recorded in this file with the schema decisions it informed; src/vendorlib/tcl_oauth2_library is vendored via a declaration as the proving case; derived-project applicability (how the manifest and sync travel via G-027) has a recorded design decision - implemented or deferred with rationale.

Context

Vendoring today is copy-shaped: clone the upstream (e.g. TEMP_REFERENCE/tcl_oauth2_library, 2026-07-12), trim by hand, drop the result into src/vendorlib. Nothing records where the copy came from, what version/commit it represents, or what was trimmed - the same provenance gap G-026 closes for local-project pulls, but for external upstreams, which G-026 deliberately does not cover. The desirable end state resembles npm/mix/zig dependency declaration: declare the package, optionally pin it, let tooling materialize and update it. This is not an attempt at a general Tcl package manager, but where the Tcl ecosystem has established metadata conventions (teapot Meta headers - already reused by G-063 - and TIP-era distribution metadata) the schema should stay compatible; toml remains the punkshell format (G-024 direction).

Approach

• Manifest location: punkproject.toml section vs sibling file (e.g. vendor.toml) decided in the work; either way parsed by the vendored tomlish (G-014/G-024 pattern).
• Declaration fields (candidate set): name, target area (vendorlib / vendormodules / vfs path), upstream kind + URL (git / fossil / http archive), pin (tag / commit / version), trim/keep rules (license, readme, examples kept by default), license expectation (cross-checked by G-063 audit), binaries-allowed (default false).
• Sync command: punk::mix dev commandset or make.tcl subcommand (decided in the work); agent-free by design - retrieval + trim + provenance recording only. Moduledoc generation is explicitly out of scope here (G-068).
• Mixed mode is a contract, not a transition state: hand-dropped libraries stay legal; the audit surface (shared with G-063's enumeration) classifies each vendored entry as declared-in-sync / declared-stale / undeclared.
• Binary scan piggybacks the G-004 policy (root AGENTS.md binaries rule): scan the materialized payload for executable binaries (shared libs, exes, zip-based .tm embedding executables) before accepting it.
• Ecosystem survey is a bounded research step recorded here (Notes), informing field names and metadata mapping - not an open-ended standards effort.

Alternatives considered

• Extending G-026 to cover external upstreams - rejected: G-026 is a policy about pulls from local sibling checkouts (dirty-checkout enforcement); external retrieval, pinning and trim rules are a different mechanism sharing only the provenance vocabulary.
• Requiring all vendoring to go through declarations (rejecting drop-ins) - rejected: the user explicitly wants the mixed approach; drop-ins are surfaced, not blocked.

Notes

• Related: G-004 (binary policy this enforces at vendor time), G-024 (toml direction), G-026 (local-pull provenance sibling; include_modules.config -> toml note), G-027 (derived-project travel), G-063 (license recording hook at vendor time), G-066 (repackaging consumes declared packages), G-067 (artifact retrieval as an alternative upstream kind), G-068 (moduledoc status tracked against the manifest).
• Proving case: tcl_oauth2_library. A hand-trimmed copy (keeping LICENSE/license.terms/README/examples) was placed in src/vendorlib and load-tested with the punk91 src shell on 2026-07-12, then deliberately removed before ever being committed (2026-07-12) so the first real vendoring of it happens end-to-end through the declaration - retrieval, trim rules, provenance, materialization - rather than as a retrofit over an existing copy. The git clone in TEMP_REFERENCE/tcl_oauth2_library remains the upstream stand-in; the hand trim is the reference expectation for the declared trim rules' output.
• The adoption scenario (an existing manual drop-in surfaced as undeclared, then brought under a declaration) still needs testing despite the removal - recreate it at test time with a scratch drop-in copy.