Browse Source
New goals (user-approved drafts, all proposed) covering the declarative vendoring direction: - G-065 declarative vendoring: toml-declared external packages with version/commit pinning, retrieval provenance, binary-scan gating with punkproject.toml override; manual drop-ins stay supported (surfaced as undeclared); agent-free by design - G-066 pkgIndex.tcl-to-.tm repackaging: lib.copyasmodule expansion with embedded LICENSE/metadata datafile and distribution-unit tracking for multi-package upstreams (tcllib-style) - G-067 module artifact channel: publish/retrieve prepared .tm modules against configurable artifact servers, sharing the G-006 consent-gating and source-override pattern - G-068 agent-assisted moduledoc generation workflow for vendored third-party libraries (G-055 pattern generalized), decoupled from basic vendoring G-063 contract extended (user-approved): vendored-package license indications additionally carry verification provenance (method, verifier, date), reported by the audit surface. Proving case recorded in G-065 notes: the hand-trimmed tcl_oauth2_library copy was load-tested then removed before ever being committed, so the first vendoring of it runs end-to-end through the declaration; the TEMP_REFERENCE clone remains the upstream stand-in. goals_lint: clean. Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.commaster
6 changed files with 206 additions and 3 deletions
@ -0,0 +1,67 @@
|
||||
# G-065 Declarative vendoring: toml-declared external packages with pinning, provenance and binary gating |
||||
|
||||
Status: proposed |
||||
Scope: punkproject.toml or sibling vendor manifest (schema - settled in the work), src/modules/punk/mix/ (vendor-sync command surface), src/vendorlib/ + src/vendormodules/ + src/vfs/ (materialization targets), src/make.tcl (integration) |
||||
Goal: vendoring an external third-party package into punkshell (and, via the G-027 channel, derived projects) is driven by a toml declaration - upstream source, optional version/commit pin, trim rules - materialized and updated by one command that records retrieval provenance (upstream URL, commit/tag, retrieval date, license-verification provenance per G-063) and scans for executable binaries, refusing or warning unless punkproject.toml carries an explicit binaries-allowed override; manual drop-ins (a library folder or .tm file placed by hand, in vendor dirs or vfs) remain supported and are surfaced by audit as undeclared rather than rejected; basic vendoring requires no agent/LLM tooling. |
||||
Acceptance: a documented toml manifest schema exists and declaring a package plus running the sync command materializes it under src/vendorlib or src/vendormodules (and a vfs target where declared) on a system without agents; re-running after a pin change updates the materialized copy and an unpinned declaration records its resolved version at sync time; per-package provenance (upstream URL, commit/tag or version, retrieval date) is recorded and queryable; a declared package containing executable binaries is refused (warn-only selectable as a configured mode) unless the punkproject.toml override is present, exercised by a test or documented manual verification against a scratch package; a manually dropped-in library remains loadable and is reported as undeclared by the audit/status surface, not deleted; the ecosystem-metadata compatibility survey (teapot Meta headers, relevant TIPs, wiki.tcl-lang.org / tip.tcl.tk scan) is recorded in this file with the schema decisions it informed; src/vendorlib/tcl_oauth2_library is vendored via a declaration as the proving case; derived-project applicability (how the manifest and sync travel via G-027) has a recorded design decision - implemented or deferred with rationale. |
||||
|
||||
## Context |
||||
|
||||
Vendoring today is copy-shaped: clone the upstream (e.g. TEMP_REFERENCE/tcl_oauth2_library, |
||||
2026-07-12), trim by hand, drop the result into src/vendorlib. Nothing records where the |
||||
copy came from, what version/commit it represents, or what was trimmed - the same |
||||
provenance gap G-026 closes for local-project pulls, but for external upstreams, which |
||||
G-026 deliberately does not cover. The desirable end state resembles npm/mix/zig |
||||
dependency declaration: declare the package, optionally pin it, let tooling materialize |
||||
and update it. This is not an attempt at a general Tcl package manager, but where the |
||||
Tcl ecosystem has established metadata conventions (teapot Meta headers - already reused |
||||
by G-063 - and TIP-era distribution metadata) the schema should stay compatible; toml |
||||
remains the punkshell format (G-024 direction). |
||||
|
||||
## Approach |
||||
|
||||
- Manifest location: punkproject.toml section vs sibling file (e.g. vendor.toml) decided |
||||
in the work; either way parsed by the vendored tomlish (G-014/G-024 pattern). |
||||
- Declaration fields (candidate set): name, target area (vendorlib / vendormodules / vfs |
||||
path), upstream kind + URL (git / fossil / http archive), pin (tag / commit / version), |
||||
trim/keep rules (license, readme, examples kept by default), license expectation |
||||
(cross-checked by G-063 audit), binaries-allowed (default false). |
||||
- Sync command: punk::mix dev commandset or make.tcl subcommand (decided in the work); |
||||
agent-free by design - retrieval + trim + provenance recording only. Moduledoc |
||||
generation is explicitly out of scope here (G-068). |
||||
- Mixed mode is a contract, not a transition state: hand-dropped libraries stay legal; |
||||
the audit surface (shared with G-063's enumeration) classifies each vendored entry as |
||||
declared-in-sync / declared-stale / undeclared. |
||||
- Binary scan piggybacks the G-004 policy (root AGENTS.md binaries rule): scan the |
||||
materialized payload for executable binaries (shared libs, exes, zip-based .tm |
||||
embedding executables) before accepting it. |
||||
- Ecosystem survey is a bounded research step recorded here (Notes), informing field |
||||
names and metadata mapping - not an open-ended standards effort. |
||||
|
||||
## Alternatives considered |
||||
|
||||
- Extending G-026 to cover external upstreams - rejected: G-026 is a policy about pulls |
||||
from local sibling checkouts (dirty-checkout enforcement); external retrieval, |
||||
pinning and trim rules are a different mechanism sharing only the provenance |
||||
vocabulary. |
||||
- Requiring all vendoring to go through declarations (rejecting drop-ins) - rejected: |
||||
the user explicitly wants the mixed approach; drop-ins are surfaced, not blocked. |
||||
|
||||
## Notes |
||||
|
||||
- Related: G-004 (binary policy this enforces at vendor time), G-024 (toml direction), |
||||
G-026 (local-pull provenance sibling; include_modules.config -> toml note), |
||||
G-027 (derived-project travel), G-063 (license recording hook at vendor time), |
||||
G-066 (repackaging consumes declared packages), G-067 (artifact retrieval as an |
||||
alternative upstream kind), G-068 (moduledoc status tracked against the manifest). |
||||
- Proving case: tcl_oauth2_library. A hand-trimmed copy (keeping |
||||
LICENSE/license.terms/README/examples) was placed in src/vendorlib and load-tested |
||||
with the punk91 src shell on 2026-07-12, then deliberately removed before ever being |
||||
committed (2026-07-12) so the first real vendoring of it happens end-to-end through |
||||
the declaration - retrieval, trim rules, provenance, materialization - rather than as |
||||
a retrofit over an existing copy. The git clone in TEMP_REFERENCE/tcl_oauth2_library |
||||
remains the upstream stand-in; the hand trim is the reference expectation for the |
||||
declared trim rules' output. |
||||
- The adoption scenario (an existing manual drop-in surfaced as undeclared, then |
||||
brought under a declaration) still needs testing despite the removal - recreate it |
||||
at test time with a scratch drop-in copy. |
||||
@ -0,0 +1,40 @@
|
||||
# G-066 pkgIndex.tcl-to-.tm repackaging: lib.copyasmodule expansion with embedded metadata and distribution-unit tracking |
||||
|
||||
Status: proposed |
||||
Scope: src/modules/punk/mix/commandset/loadedlib-999999.0a1.0.tm (lib.copyasmodule), src/modules/punk/mix/ (modpod/zipkit tooling as needed), src/tests/modules/punk/mix/ (converter testsuite) |
||||
Goal: third-party pkgIndex.tcl-based packages can be repackaged as single-file .tm modules (zip-based where the payload warrants it) that embed upstream documents (LICENSE, README) and a punkshell metadata datafile giving a consistent description of upstream name, version, license and distribution-unit membership - so packages split out of a multi-package upstream (tcllib-style) record that they shipped together at upstream version X and should be upgrade-checked as a unit - with the converter handling a substantially broader class of pkgIndex.tcl scripts than today's lib.copyasmodule and refusing clearly on scripts it cannot model rather than emitting a broken module. |
||||
Acceptance: the converter repackages a proving set of at least three packages - src/vendorlib/tcl_oauth2_library plus two tcllib packages, one of which has a non-trivial pkgIndex.tcl (multiple statements, computed version, or multi-file source list) - and each resulting .tm loads via package require on the primary target runtimes (Tcl 9 kit and 8.6, or a recorded limitation referencing the G-034 code-interp constraint); each repackaged module embeds the upstream LICENSE and a metadata datafile carrying upstream name, upstream version, license indication (G-063-resolvable) and distribution-unit fields; converting several packages from one upstream project in one run records a shared distribution-unit id and version queryable from the packaged artifacts (surface decided in the work); a pkgIndex.tcl construct outside the converter's modelled class produces an explicit refusal message naming the construct; converter behaviour is covered by a testsuite under src/tests/modules/punk/mix/. |
||||
|
||||
## Context |
||||
|
||||
punkshell prefers .tm modules; the wider Tcl ecosystem mostly ships pkgIndex.tcl |
||||
libraries and may continue to. Zip-based .tm modules get single-file distribution and |
||||
can carry license/readme/metadata inside the artifact - which also makes them the |
||||
natural payload for an artifact server (G-067). `dev lib.copyasmodule` already performs |
||||
a basic conversion and has worked on some tcllib modules, but pkgIndex.tcl scripts are |
||||
arbitrary Tcl and the current handling is narrow. Multi-package upstreams introduce the |
||||
unit problem: once tcllib (or similar) packages are split into individual .tm files, |
||||
nothing records that they came from one release and should be upgraded together - no |
||||
current goal touches this. |
||||
|
||||
## Approach |
||||
|
||||
- Grow lib.copyasmodule's modelled class of pkgIndex.tcl scripts incrementally |
||||
(ifneeded lines with source/load lists, simple computed versions), with an explicit |
||||
refusal path for everything else - correctness over coverage. |
||||
- Metadata datafile format: toml, schema shared with / derived from the G-065 manifest |
||||
vocabulary so vendored-in-place and repackaged artifacts describe themselves |
||||
consistently. |
||||
- Distribution-unit: unit id = upstream project identity (e.g. "tcllib"), unit version = |
||||
upstream release; recorded per artifact and aggregable ("what units are present, are |
||||
any mixed-version"). Upgrade-together enforcement is a consumer concern (G-065 sync / |
||||
G-067 retrieval) - this goal only guarantees the data exists. |
||||
- Loading on 8.6: zip-based .tm viability in the shell code interp is G-034's subject; |
||||
this goal records the limitation rather than solving mounting. |
||||
|
||||
## Notes |
||||
|
||||
- Related: G-034 (zip modpod mounting on 8.6), G-035 (mixed .tm/pkgIndex provision |
||||
characterization - what happens when both shapes of the same package are present), |
||||
G-063 (license fields in the datafile), G-065 (manifest schema sharing), G-067 |
||||
(repackaged artifacts as the publish payload). |
||||
@ -0,0 +1,37 @@
|
||||
# G-067 Module artifact channel: publish prepared .tm modules to and retrieve from configurable artifact servers |
||||
|
||||
Status: proposed |
||||
Scope: src/make.tcl or punk::mix dev commandset (publish/retrieve surface - settled in the work), user-config (consent flag + server list, G-006 pattern), src/vendormodules/ + src/vendorlib/ (retrieval targets) |
||||
Goal: prepared .tm module artifacts (typically G-066 repackaged zipkits) can be published to a punkshell official artifact server and retrieved into punkshell or derived projects on declaration, following G-006's established patterns - consent gating by default, a default official source, user-configured alternative servers as replacement or addition, version pinning encoded in the artifact addressing, checksum verification where provided - sharing rather than duplicating G-006 machinery where it exists. |
||||
Acceptance: a retrieve operation fetches a named module artifact at a pinned version from a configured server into the project's vendor area and the module loads via package require; no retrieval happens without explicit consent (config flag or interactive prompt; non-interactive use without the flag fails with an actionable message, never downloads silently); multiple servers are configurable with a documented precedence and both replacement and additional semantics available; artifact checksums are verified when the server supplies them and a mismatch aborts the install; a publish path to the official server is documented and demonstrated (authentication mechanism decided in the work); retrieval works from a derived project, or derived-project support is recorded as deferred with rationale; the relationship to G-006's downloader (shared implementation vs parallel with recorded justification) is a recorded design decision. |
||||
|
||||
## Context |
||||
|
||||
G-006 establishes consent-gated artifact download for binary build artifacts (the |
||||
zig-built set). Modules are a distinct artifact class with the same retrieval-shaped |
||||
needs, and G-027's remote-pull question already names the G-006 channel as a candidate |
||||
transport - three goals converge on one artifact-retrieval substrate. Rather than |
||||
widening G-006's tightly binary-scoped acceptance, this goal gives modules their own |
||||
contract on the same design pattern. The publish side is what makes the official server |
||||
populatable: repackage (G-066), then push, so developers on other machines can declare |
||||
and pull (G-065) instead of re-vendoring by hand. |
||||
|
||||
## Approach |
||||
|
||||
- Addressing scheme encodes name + version (and unit id/version where the artifact |
||||
belongs to a G-066 distribution unit) so pinned retrieval is a URL construction, not |
||||
a server-side search. |
||||
- Consent and server-list configuration reuse the G-006 config surface (same keys or a |
||||
documented sibling namespace) - one consent story for all remote artifact fetching. |
||||
- Retrieval integrates as an upstream kind in the G-065 manifest ("from artifact server |
||||
X" alongside "from git repo Y"), so declared vendoring and artifact retrieval are one |
||||
developer experience. |
||||
- Official-server operation (hosting, retention, signing policy) is out of scope beyond |
||||
what publish/retrieve need to interoperate with it, mirroring G-006's stance on the |
||||
binary-artifacts repo. |
||||
|
||||
## Notes |
||||
|
||||
- Related: G-006 (pattern source and candidate shared implementation), G-027 (remote |
||||
infrastructure-pull transport candidate), G-065 (manifest integration), G-066 |
||||
(artifact payloads and distribution-unit metadata). |
||||
@ -0,0 +1,41 @@
|
||||
# G-068 Agent-assisted moduledoc generation workflow for vendored third-party libraries |
||||
|
||||
Status: proposed |
||||
Scope: goals/G-068-vendored-moduledoc-workflow.md (workflow doc), src/modules/punk/args/moduledoc/ (generated companion modules), src/tests/modules/punk/args/ (probe verification where feasible) |
||||
Goal: a vendored third-party library that neither ships nor references punk::args documentation can be given a punk::args::moduledoc companion through a documented, repeatable agent-assisted workflow - the G-055 pattern generalized beyond core.tcl-lang.org projects: upstream doc text (man pages, README, doctools) carried verbatim under the G-055 fidelity policy, synopses translated into punkshell's synopsis syntax, safe probe verification applied where feasible, upstream source/version provenance recorded with the definitions - run as a separate optional step after vendoring, so basic vendoring (G-065) never requires agent availability, with each vendored package's moduledoc status (present / absent / stale against the vendored version) trackable. |
||||
Acceptance: the workflow is documented in this file (inputs, fidelity and synopsis-translation policy by reference to G-055, probe-verification gate including the criteria for declaring probing infeasible for a command, provenance recording, and how the workflow consumes a vendored payload rather than a core source tree); a moduledoc companion produced through the workflow exists for at least one vendored library - src/vendorlib/tcl_oauth2_library as the proving case - loading alongside the untouched vendored source and surfacing in the help system; the vendoring path demonstrably completes without this step (a package vendored with no moduledoc loads and audits normally); moduledoc status per vendored package is queryable from a defined surface (G-065 manifest field or the G-063/G-064 audit surface - decided in the work) and distinguishes present, absent and stale-against-vendored-version. |
||||
|
||||
## Context |
||||
|
||||
External Tcl libraries will almost never carry punk::args documentation unless future |
||||
authors adopt it; giving them punkshell-quality help means authoring a moduledoc |
||||
companion. G-055 built the workflow shape for tclcore (verbatim fidelity, synopsis |
||||
translation, probe gates, provenance) and explicitly anticipated extension to other |
||||
projects while keeping them out of scope. Doc authoring needs judgement, so this is |
||||
likely always agent-assisted - which is exactly why it must be decoupled from basic |
||||
vendoring: a developer on a machine without agents vendors now and generates (or |
||||
receives) the moduledoc later. A published artifact (G-067) can carry its moduledoc |
||||
with it, so the generation cost is paid once per ecosystem, not once per developer. |
||||
|
||||
## Approach |
||||
|
||||
- Inputs per run: the vendored payload path + recorded upstream version (from G-065 |
||||
provenance), plus whatever docs the upstream ships (man pages like oauth2.man, |
||||
README, doctools sources). |
||||
- Companion placement: a moduledoc module under src/modules/punk/args/moduledoc/ |
||||
loading on 'package require <lib>' (tkcore pattern noted in G-055), or embedded in a |
||||
G-066 repackaged artifact - placement decision recorded in the work; the vendored |
||||
source itself is never modified (src/vendorlib contract). |
||||
- Probe verification: where commands are safe/pure enough, the G-055 real-vs-model |
||||
error/ok agreement gate applies; for network-touching or stateful commands (the |
||||
oauth2 case) the workflow documents the infeasibility criteria and falls back to |
||||
doc-fidelity review only. |
||||
- Staleness: moduledoc records the upstream version it was authored against; status |
||||
compares that to the currently vendored version. |
||||
|
||||
## Notes |
||||
|
||||
- Related: G-055 (workflow pattern source; its modelability-gap scan discipline applies |
||||
here too), G-063/G-064 (audit surfaces that could report moduledoc status), G-065 |
||||
(manifest tracking, agent-free vendoring guarantee), G-066/G-067 (carrying moduledocs |
||||
inside repackaged/published artifacts). |
||||
Loading…
Reference in new issue