punkshell/goals/G-065-declarative-vendoring.md

G-065 Declarative vendoring: toml-declared external packages with pinning, provenance and binary gating

Status: proposed Scope: punkproject.toml or sibling vendor manifest (schema - settled in the work); src/modules/punk/mix/ (vendor-sync command surface); src/vendorlib/ + src/vendormodules/ + src/vfs/ (materialization targets); src/make.tcl (integration) Goal: vendoring an external third-party package into punkshell (and, via the G-027 channel, derived projects) is driven by a toml declaration - upstream source, optional version/commit pin, trim rules - materialized and updated by one command that records retrieval provenance (upstream URL, commit/tag, retrieval date, license-verification provenance per G-063) and scans for executable binaries, refusing or warning unless punkproject.toml carries an explicit binaries-allowed override; manual drop-ins (a library folder or .tm file placed by hand, in vendor dirs or vfs) remain supported and are surfaced by audit as undeclared rather than rejected; basic vendoring requires no agent/LLM tooling. Acceptance: a documented toml manifest schema exists and declaring a package plus running the sync command materializes it under src/vendorlib or src/vendormodules (and a vfs target where declared) on a system without agents; re-running after a pin change updates the materialized copy and an unpinned declaration records its resolved version at sync time; per-package provenance (upstream URL, commit/tag or version, retrieval date) is recorded and queryable; a declared package containing executable binaries is refused (warn-only selectable as a configured mode) unless the punkproject.toml override is present, exercised by a test or documented manual verification against a scratch package; a manually dropped-in library remains loadable and is reported as undeclared by the audit/status surface, not deleted; the ecosystem-metadata compatibility survey (teapot Meta headers, relevant TIPs, wiki.tcl-lang.org / tip.tcl.tk scan) is recorded in this file with the schema decisions it informed; src/vendorlib/tcl_oauth2_library is vendored via a declaration as the proving case; derived-project applicability (how the manifest and sync travel via G-027) has a recorded design decision - implemented or deferred with rationale.

Context

Vendoring today is copy-shaped: clone the upstream (e.g. TEMP_REFERENCE/tcl_oauth2_library, 2026-07-12), trim by hand, drop the result into src/vendorlib. Nothing records where the copy came from, what version/commit it represents, or what was trimmed - the same provenance gap G-026 closes for local-project pulls, but for external upstreams, which G-026 deliberately does not cover. The desirable end state resembles npm/mix/zig dependency declaration: declare the package, optionally pin it, let tooling materialize and update it. This is not an attempt at a general Tcl package manager, but where the Tcl ecosystem has established metadata conventions (teapot Meta headers - already reused by G-063 - and TIP-era distribution metadata) the schema should stay compatible; toml remains the punkshell format (G-024 direction).

Approach

• Manifest location: punkproject.toml section vs sibling file (e.g. vendor.toml) decided in the work; either way parsed by the vendored tomlish (G-014/G-024 pattern).
• Declaration fields (candidate set): name, target area (vendorlib / vendormodules / vfs path), upstream kind + URL (git / fossil / http archive), pin (tag / commit / version), trim/keep rules (license, readme, examples kept by default), license expectation (cross-checked by G-063 audit), binaries-allowed (default false).
• Sync command: punk::mix dev commandset or make.tcl subcommand (decided in the work); agent-free by design - retrieval + trim + provenance recording only. Moduledoc generation is explicitly out of scope here (G-068).
• Mixed mode is a contract, not a transition state: hand-dropped libraries stay legal; the audit surface (shared with G-063's enumeration) classifies each vendored entry as declared-in-sync / declared-stale / undeclared.
• Binary scan piggybacks the G-004 policy (root AGENTS.md binaries rule): scan the materialized payload for executable binaries (shared libs, exes, zip-based .tm embedding executables) before accepting it.
• Ecosystem survey is a bounded research step recorded here (Notes), informing field names and metadata mapping - not an open-ended standards effort.

Alternatives considered

• Extending G-026 to cover external upstreams - rejected: G-026 is a policy about pulls from local sibling checkouts (dirty-checkout enforcement); external retrieval, pinning and trim rules are a different mechanism sharing only the provenance vocabulary.
• Requiring all vendoring to go through declarations (rejecting drop-ins) - rejected: the user explicitly wants the mixed approach; drop-ins are surfaced, not blocked.

Notes

• Related: G-004 (binary policy this enforces at vendor time), G-024 (toml direction), G-026 (local-pull provenance sibling; include_modules.config -> toml note), G-027 (derived-project travel), G-063 (license recording hook at vendor time), G-066 (repackaging consumes declared packages), G-067 (artifact retrieval as an alternative upstream kind), G-068 (moduledoc status tracked against the manifest), G-115 (sibling toml declaration surface for .vfs payloads - keep the schemas aligned, per the 2026-07-24 overlap review), G-047 (punkproject.toml [workflow] schema co-tenant - if the manifest lives in punkproject.toml, the two schemas must not collide).
• Proving case: tcl_oauth2_library. A hand-trimmed copy (keeping LICENSE/license.terms/README/examples) was placed in src/vendorlib and load-tested with the punk91 src shell on 2026-07-12, then deliberately removed before ever being committed (2026-07-12) so the first real vendoring of it happens end-to-end through the declaration - retrieval, trim rules, provenance, materialization - rather than as a retrofit over an existing copy. The git clone in TEMP_REFERENCE/tcl_oauth2_library remains the upstream stand-in; the hand trim is the reference expectation for the declared trim rules' output.
• The adoption scenario (an existing manual drop-in surfaced as undeclared, then brought under a declaration) still needs testing despite the removal - recreate it at test time with a scratch drop-in copy.
• G-139 (archived 2026-07-30) implemented the artifact-fetch shape for the tcllib case this goal anticipated: declared artifacts (src/runtime/libpackages.toml) -> sha1-verified fetch (make.tcl libfetch) -> untracked bin/packages tier -> punkcheck-installed lib_tcl + kit-vfs declarations (vendorlib_vfs.toml source_root="packages" + replace/supersedes). Per its drafting note, this goal should absorb or reference that pattern rather than duplicate it - see goals/archive/G-139-punkbin-library-consumption.md.
• Bootsupport tcllib-subset survey (assessment 2026-07-31, user-directed; candidate adoption case for this goal): the deliberately-partial tcllib content punkshell boot machinery relies on exists in TWO forms, neither with upstream provenance. (a) tcllib-origin .tm modules (cksum, cmdline, md5, sha1, struct, textutil, uuid, ...) ride src/vendormodules -> src/bootsupport/modules via the include_modules.config pull (punkcheck records the COPY only). (b) the pkgIndex-form src/bootsupport/lib tree (base64, control, debug, fileutil, snit, struct, tar, term, vfszip, virtchannel_*; ~119 files) has NO pull mechanism at all - src/bootsupport/tclpackages.toml (2026-06-23 "initial outline of format", no reader anywhere in the tree; described by the archived G-087 notes as checkout-level governance only) SKETCHES governing it, and the vendormodules config's own header records the toml-conversion desire ("#todo - change to include_modules.toml ... programatically editable whilst retaining comments"). Candidate resolution when this goal activates: declare the subset as per-module trims of the tcllib upstream in this goal's manifest (keeping the subset posture - the whole-package artifact tier serves the lib_tcl/kit consumers, G-139/G-141 lane); G-066's pkgIndex->tm conversion with embedded metadata is the natural tool for adopting form (b) into the modules pull; G-026 keeps the pull-hygiene enforcement. Also recorded: tclpackages.toml's [libs..] BINARY entries (thread dll trees, tcllib incl. binary parts; source lines now stale post-G-139 removal - inert, no reader) are a pre-zig-era mis-step per the user - binaries arrive via the bin/runtime + bin/packages consent-gated fetch tiers or zig builds (G-004 names src/bootsupport in its no-committed-binaries scope; G-102/G-123/G-139 are the live routes); the empty src/bootsupport/lib_tcl/ + bin scaffolds are UNTRACKED local dirs only (git tracks no files there; the project-0.1 layout ships bootsupport without them), so retiring the sketch's [libs] platform sections retires the committed footprint.