diff --git a/GOALS.md b/GOALS.md index c72d8810..5d8d8ba9 100644 --- a/GOALS.md +++ b/GOALS.md @@ -406,3 +406,7 @@ Detail: goals/G-137-kit-version-resources.md Scope: src/make.tcl (libfetch target-set derivation + narrow/widen options; VFSLIBS missing-source hint; PACKAGES phase as the host-need input); src/runtime/libpackages.toml (declaration surface as consumed); src/runtime/vendorlib_vfs.toml (source_root="packages" entries as the kit-consumption input - format coordinated with G-115/G-127); src/tests/shell/testsuites/punkexe/maketcllibfetch.test (characterization: derivation, narrowing, widening); bin/AGENTS.md + src/runtime/AGENTS.md (derivation rule documentation) Detail: goals/G-141-libfetch-target-selection.md +### G-142 [proposed] Punkbin curated listing manifests: per-tier artifact grouping with named groups +Scope: punkbin artifact repo (external c:/repo/jn/punkbin - layout contract, src/build_sha1sums.tcl, per-tier listing manifests); src/scriptapps/bin/ (punk-runtime payloads); bin/punk-runtime.cmd (regenerated output) + bin/AGENTS.md; src/tests/shell/testsuites/binscripts/; src/make.tcl (libfetch coordination) +Detail: goals/G-142-punkbin-listing-manifests.md + diff --git a/goals/G-141-libfetch-target-selection.md b/goals/G-141-libfetch-target-selection.md index 06ca4e8e..ebc377c1 100644 --- a/goals/G-141-libfetch-target-selection.md +++ b/goals/G-141-libfetch-target-selection.md @@ -99,6 +99,10 @@ has no equivalent of "materialize what this checkout needs". to consumption. - Related: G-027 - derived projects receive libfetch via layout sync and inherit selection unchanged. +- Related: G-142 - punkbin per-tier listing manifests; its libfetch-side + "declared artifact not in the server's current listing" warning coordinates + with this goal's libfetch surface (whichever lands second adopts the + other's vocabulary). - Overlap survey 2026-07-31 (drafting): goals_xref paths over src/make.tcl, src/runtime/libpackages.toml, src/runtime/vendorlib_vfs.toml, bin/packages, src/tests/shell/testsuites/punkexe - strongest live intersections G-127 and diff --git a/goals/G-142-punkbin-listing-manifests.md b/goals/G-142-punkbin-listing-manifests.md new file mode 100644 index 00000000..3575d109 --- /dev/null +++ b/goals/G-142-punkbin-listing-manifests.md @@ -0,0 +1,45 @@ +# G-142 Punkbin curated listing manifests: per-tier artifact grouping with named groups + +Status: proposed +Scope: punkbin artifact repo (external c:/repo/jn/punkbin - layout contract, src/build_sha1sums.tcl, per-tier listing manifests); src/scriptapps/bin/ (punk-runtime payloads); bin/punk-runtime.cmd (regenerated output) + bin/AGENTS.md; src/tests/shell/testsuites/binscripts/; src/make.tcl (libfetch coordination) +Goal: punkbin artifact tiers (runtime platform folders, tools/, lib//) gain a curated per-tier listing manifest in the defaults.txt idiom (hand-curated, generator-validated, mirror-curable) that separates the listing surface from the integrity ledger: sha1sums.txt stays the complete integrity authority covering every retained immutable artifact, while a per-tier manifest names the listed subset in named groups with tool-understood semantics (current, lts, vulnerable carve-outs, operator-invented groups) - so an operator can retain old immutable artifacts indefinitely without punkshell-side listing tools drowning the current set in history, while every retained artifact stays fetchable and sha1-verifiable by explicit name. +Acceptance: (1) the punkbin layout contract (AGENTS.md + README) documents the per-tier listing manifest: file format (line-based, '#' comments/blanks ignored, two-column ' '), the group vocabulary with reserved-group semantics (current = default listing; lts = listed in own section; vulnerable = hidden from default listing with a fetch-time warning; archive = hidden, fetch-silent; unknown groups listed as own sections; absent-from-manifest = retained-unlisted, fetchable by explicit name) and the mirror-curation posture; (2) build_sha1sums.tcl validates each manifest against its tier's sha1sums.txt (listed names must exist there) and reports per-tier listed/hidden/unlisted-retained counts; (3) punk-runtime list -remote renders the curated view by default (grouped sections; vulnerable + archive groups hidden) with an -all escape to the full sha1sums view, byte-agreeing across the ps1/bash payloads; (4) fetch of a vulnerable-group artifact succeeds with a warning naming the group; fetch of any retained unlisted artifact still verifies against the full sha1sums.txt; (5) binscripts characterization tests pin the listing/filtering/warning behaviours against the httpfixture server in both payloads; (6) libfetch warns when a declared artifact is absent from the server's current listing (surface coordinated with G-141). + +## Context + +The punkbin artifact repo's tiers are deliberately flat below their levels (runtime platform folders, tools/, lib//), and artifacts are immutable -r names retained indefinitely by operator policy. Today sha1sums.txt is both the integrity authority AND the de-facto listing source: punk-runtime list -remote renders every row including remote-only entries, so retaining old revisions indefinitely grows the listing monotonically while the signal (what should I fetch?) drowns in history. defaults.txt already proves the curated-manifest idiom in this layout (hand-curated release decision, generator-validated, mirror-curable) but names only ONE artifact per platform - it cannot express "these N artifacts are the current set", nor named groups. + +User direction shaping the design (2026-07-31): grouping is a first-class need - anticipated groups include LTS tracks and carve-outs for releases with known vulnerabilities (still servable by explicit name, excluded from the main listing, warning on fetch); there are NO punkbin mirrors yet and the operator is currently free to rearrange the repo; backwards compatibility is NOT a requirement for now (single corporate group, JN-controlled - it becomes a concern only once punkshell/punkbin have external consumers). + +## Approach + +- Per-tier curated manifest (filename settled in the work - listing.txt the working name) beside each tier's sha1sums.txt, in the defaults.txt idiom: line-based, '#' comments/blanks ignored, two-column ' '; hand-curated as a release decision in the publication change-set, never generated; mirrors may curate their own (defaults.txt precedent). +- Group vocabulary is OPEN with reserved-group semantics understood by tools: current = the default listing; lts = listed in its own section (a still-recommended track); vulnerable = hidden from the default listing, fetchable by explicit name with a fetch-time warning naming the group (the known-vulnerability carve-out - still servable, never silently listed); archive = hidden, fetch-silent (deliberately-retained history marker); operator-invented groups are listed as their own sections; names absent from the manifest are retained-unlisted - fetchable and sha1-verifiable by explicit name, never listed (the zero-curation bucket for bulk history). +- sha1sums.txt stays the COMPLETE integrity authority: every retained artifact, listed or not, stays fetchable by explicit name and sha1-verifiable. The manifest subsets listing, never integrity. +- build_sha1sums.tcl validates each manifest against its tier's sha1sums.txt (every listed name must exist there - the same drift discipline as defaults.txt) and reports per-tier listed/hidden/unlisted-retained counts as an audit surface. +- punk-runtime list -remote renders the curated view by default (grouped sections; vulnerable + archive groups hidden) with an -all escape to the full sha1sums view; fetch of a vulnerable-group artifact succeeds with a warning naming the group; fetch of unlisted artifacts verifies against the full sha1sums.txt unchanged. Both payloads agree byte-for-byte (the established ps1/bash parity discipline + re-wrap roundtrip test). +- libfetch warns when a declared artifact is absent from the server's current listing (drift signal for the declaring project; surface coordinated with G-141). +- Backwards compatibility is deliberately NOT a design constraint (user direction 2026-07-31). Tolerant defaults are kept only where free: an absent manifest means list-everything - the natural uncurated-server behaviour, which also serves future third-party mirrors using the layout. + +## Alternatives considered + +- Physical archive/ subfolder per tier - reconsidered given the recorded freedom to rearrange punkbin (no mirrors yet), still rejected: an artifact's URL should be as immutable as its bytes (locator stability for pinned references), and fetch-by-name stays a single fixed path per tier instead of a main-then-archive probe chain. Revisit only if actual pruning (not listing) becomes the need. +- Display-side grouping only (tools collapse old -r revisions by parsed name series) - deferred complement, rejected as the mechanism: no operator curation (cannot express LTS tracks or vulnerability carve-outs) and sha1sums.txt keeps growing as the transfer unit. +- Sharded integrity (sha1sums.txt + sha1sums-archive.txt) - rejected: splits the integrity authority and forces every consumer to answer "which file do I verify against"; the manifest subsets listing, never integrity. +- Single root-level manifest covering all tiers (defaults.txt's own shape) - rejected: per-tier files sit beside the sha1sums.txt they subset (locality), let mirrors curate per tier, and keep each tier's curation surface proportional to its churn. +- Single-column names with '#' comment group headers - rejected: two-column ' ' is machine-renderable (grouped sections, reserved-group semantics) at the same curation cost; comment headers are human-only grouping. + +## Notes + +- Related: G-123 (achieved 2026-07-31 - see goals/archive/G-123-thirdparty-runtime-tiers.md) - runtime-tier twin; this goal's manifest sits beside its retroactive sidecars and schema-v2 records. +- Related: G-138/G-139/G-140 (archived) - the lib tier's emission/consumption; the manifest mechanism serves lib// identically to runtime tiers, and libfetch is a consumer. +- Related: G-141 - libfetch target selection; the libfetch-side "declared artifact not in the server's current listing" warning is a coordinated surface (whichever lands second adopts the other's vocabulary), not a dependency. +- Related: G-067 - the future .tm module artifact channel inherits this listing idiom (its artifact sets have the same growth shape); recorded here so the channel design adopts rather than reinvents. +- Related: G-006 - absorbs fetch/consent mechanisms when it lands; the listing mechanism must survive that absorption. +- Related: G-119 (archived) - the list -remote freshness/row vocabulary this goal extends. +- Related: G-105/G-130/G-116 - growth drivers (more targets, 32-bit lanes, tcltls artifacts) multiplying artifacts per tier; this goal keeps their listings bounded. +- Related: G-004 - the retention posture driver (immutable artifacts accumulate; nothing is deleted). +- Anchors (archived): G-117/G-103 (the -r immutable-name pattern). +- Decision points settled in the work: the manifest filename (listing.txt the working name - it names more than "current": vulnerable/archive groups too); the exact reserved-group set (current/lts/vulnerable/archive the working vocabulary); whether tools/ normalizes to a top-level tier (tools//) for symmetry with lib// - the operator's recorded freedom to rearrange punkbin makes this cheap now. +- Posture (user direction 2026-07-31): no backwards-compat requirement for now (single corporate group, JN-controlled; compat becomes a concern with external consumers); no punkbin mirrors exist yet and the operator is free to rearrange the repo. A consent gate on vulnerable-group fetches (beyond the warning) is a possible later hardening, deliberately not in this goal. +- Overlap survey 2026-07-31 (drafting): goals_xref paths over src/scriptapps, src/tests/shell/testsuites/binscripts, src/make.tcl, bin - surfaced G-078/G-081 (src/scriptapps/tools doc tooling, domain-unrelated) and G-061 (binscripts testsuite path adjacency - pseudoconsole testing, domain-unrelated); no contract-level overlaps among live goals. Related set above from the domain sweep (punkbin layout + listing consumers + growth drivers).