punkshell/goals/G-115-declarative-vfs-composition.md

G-115 Declarative .vfs composition: toml-defined kit payloads with drop-in preservation

Status: achieved 2026-07-31 Scope: src/make.tcl (vfs assembly); src/runtime/vendorlib_vfs.toml (existing per-package declaration surface - fold/supersede settled in the work); src/vfs/ (per-.vfs declaration files + README); punk::mix machinery as touched; src/project_layouts (seeding for derived projects); coordinates with G-067 (artifact sources); G-006 (consent); G-004 (binary-free committed tree) Goal: a .vfs folder's payload can be DECLARED in a toml definition (sources: suite build products, punkbin runtime/library artifacts via the consent-gated channels, vendor trees) and materialized into the folder by the build - while the folder remains the operative assembly area: undeclared dropped-in files (binary libs/modules included) are preserved with documented precedence, so derived projects with permissive binary policies and gitignored test .vfs folders keep drop-in simplicity with no declaration required. Acceptance: a punkshell kit .vfs (or demonstration .vfs) builds from a toml declaration reproducing its payload on a clean tree, with declared binary content arriving via consented retrieval or local build products; an undeclared dropped-in file survives re-materialization per the documented precedence; a .vfs with NO declaration builds exactly as today (pure drop-in mode unchanged); the declaration format and precedence rules are documented; the vendorlib_vfs.toml relationship is settled with rationale; the layout store seeds the convention for derived projects.

Context

Drafted 2026-07-22 (user-approved wording) from the binary-policy framing discussion: with G-004 removing checked-in binaries from punkshell, the kit .vfs payloads' binary content must come from SOMEWHERE reproducible - suite build products (G-103-class batteries and library builds), punkbin artifact classes retrieved through the G-006/G-067 consent gates, or vendor trees. A per-.vfs toml declaration is the glue that makes a kit payload buildable on a clean binary-free tree, while the answer to "can we still just drop a dll in?" must stay YES - the folder remains operative, declarations are optional, and undeclared drop-ins are preserved (the operate-in-tree requirement recorded in G-004 Notes 2026-07-22: local experimentation and permissive derived projects are first-class forever).

Precedents already in the tree that make this credible:

• src/runtime/vendorlib_vfs.toml - per-package per-kit declarations ALREADY drive vendorlib-to-vfs inclusion (with superseded-version removal); this goal generalizes that surface (fold or supersede - settled in the work).
• layout_materialize (G-087, achieved) - the declarative-plus-folder hybrid with overlay/anti mechanics is proven machinery in this codebase.
• tomlish is vendored, and toml is the accepted format for punkshell-context configuration (the 2026-07-20 toml drop applies only to dependency-free BUILDSUITE BOOTSTRAP configs - recorded in G-103 Context; achieved 2026-07-22, archived at goals/archive/G-103-runtime-kit-family.md).

Relationship to the punkbin third-class question (G-004 Notes 2026-07-22): declarative composition pulling per-package artifacts is the preferred-assessment alternative to storing whole zipped binary .vfs archives - if this goal lands, the third class likely stays unnecessary except as a release-snapshot convenience.

Precedence design (the drop-in guarantee) is the core design work: materialization must never clobber an undeclared file, and the documented rules must state what happens when a declaration and a drop-in collide on the same path (drop-in wins vs declared wins vs error - decided in the work, with punkcheck-style tracking of materialized content the likely mechanism for telling the two apart).

Notes

• G-024 relationship (recorded 2026-07-24 after overlap review; neither file previously referenced the other): G-024 converts mapvfs.config to toml, and archived G-037 named that file as the natural home for per-kit vendorlib payload declarations - this goal's territory. The two toml surfaces (mapping/schemes there, payload declarations here) are designed as sibling grammars in one family; the vendorlib_vfs.toml fold/supersede decision (Acceptance) cites the G-024 scheme format.
• G-019 relationship: G-019's dependency-scan closure plus declared dynamic extras is a payload-determination INPUT this goal's materialization could consume - a trimmed kit is then a declared kit whose declaration was generated rather than hand-authored.
• G-065 relationship: G-065's declarative vendoring manifest and this goal's .vfs declaration are the same design space (toml-declared, pinned, provenance-tracked external content) - G-065's upstreams feed the sources, this goal's declarations materialize the folder. Keep the schemas aligned.
• G-121 (archived) noted per-kit payload detail as a candidate later addition to bakelist's per-kit view (name filtering doubles as detail) once this goal's declaration surface exists - see goals/archive/G-121-bakelist-selective-bake.md
• Related (activation overlap survey 2026-07-31, goals_xref score): G-127 - owns the platform axis for payload declarations (whichever of the two lands second adopts the other); this goal's format lands without a platform axis and records the extension point. G-141 - its libfetch target derivation consumes whichever declaration surface is current; if this goal supersedes vendorlib_vfs.toml the derivation reads the successor surface (recorded in its own Notes).

Progress

ACHIEVED 2026-07-31 (activated and implemented same day, alongside G-024; punkshell 0.38.0). Delivered:

• Per-.vfs declaration files src/vfs/.vfs.toml (SIBLING of the folder, so never part of the payload) with [payload.] entries: source, source_root (src|packages), target (relative, inside-the-folder validated), supersedes, replace - the key vocabulary carried over from the proven vendorlib_vfs.toml surface (sibling grammar to mapvfs.toml per the G-024 coordination note). Processed by the make.tcl VFSPAYLOAD phase (commands bakehouse/bake/vfslibs, replacing the old VFSLIBS block; G-121 selective bakes narrow to the selected kits' folders; strict entry-named errors for unknown keys, missing sources, bad roots/targets, parse failures).
• PRECEDENCE (the core design work) - drop-in wins, via punkcheck -overwrite synced-targets (records in src/vfs/.punkcheck, outside payloads): undeclared files are never touched by construction; a file at a declared path is overwritten only when punkcheck's records show this mechanism installed it AND it is unmodified since - a foreign pre-existing copy (no record) or hand-modified install is preserved, with punkcheck per-file skip lines plus a phase NOTE counting changed-but-preserved files (computed as files_skipped minus sources_unchanged, so routine unchanged-source skips stay quiet). supersedes/replace are the explicit declared exceptions (replace semantics preserved from the old surface: unconditional per-run clean-slate). A .vfs with NO declaration file is untouched - the phase only globs *.vfs.toml.
• vendorlib_vfs.toml relationship SETTLED: fold/supersede. All six [install.*] entries migrated into per-.vfs files for the 13 participating folders (identical source/supersedes/replace semantics; per-target vfs lists inverted into per-kit files); src/runtime/vendorlib_vfs.toml deleted; a leftover copy in any tree is ignored with a migration warning. Rationale: per-kit files keep each kit's composition readable in one place and give new kits a copy-a-sibling starting point; per-package cross-kit lists made adding a kit an every-entry edit; G-141's derivation reads the successor surface (its Notes anticipated exactly this) and G-127 gets one place to add the platform axis. The DRY cost (shared packages declared per kit) is accepted - entries are 4-6 lines.
• Consent linkage: source_root="packages" resolves against the untracked bin/packages tier populated by the consent-gated 'make.tcl libfetch' (declared + sha1-verified; G-139/G-006 lineage) - declared binary content arrives on a clean tree without being committed. source_root="src" covers vendor trees and suite build products.
• Demonstration kit punkdeclare (mapvfs.toml [kit.punkdeclare], group declare-demo, bake_default=false, smokerequire udp+tcllibc): VCS carries ONLY the boot fauxlink; the entire lib_tcl9 binary payload (tcludp1.0.13 from vendorlib_tcl9, tcllibc from win32-x86_64/tcl9 packages tier) is declaration-materialized and ignored in both VCS (/src/vfs/punkdeclare.vfs/lib_tcl9/ in .gitignore + derived ignore-glob; ignore-sync verification run clean).
• Documentation: src/vfs/README.md (format + precedence spec), src/vfs/AGENTS.md, src/runtime/AGENTS.md, src/AGENTS.md, ARCHITECTURE.md (+lint clean), make.tcl workflow text + vfslibs summary/helptext. Layout seeding: project-0.1 layout gains src/vfs/README.md (generic convention doc) + src/runtime/mapvfs.toml; the MACHINERY reaches derived projects automatically via the established make.tcl thin-layout sync, and the modpod templates copy follows via the LAYOUT->MODPOD sync on the next modules-class build.

Verification (win32-x86_64 host):

• Migration parity: first full VFSPAYLOAD run over all 13 migrated declarations produced ZERO tracked-content changes under src/vfs (the per-.vfs files reproduce the vendorlib_vfs.toml-installed payloads exactly).
• Clean-tree reproduction: punkdeclare.vfs/lib_tcl9 deleted, 'bake -confirm 0 punkdeclare' rematerializes the payload from both source classes; smoke-require OK for udp 1.0.13 AND tcllibc 2.0 inside the built artifact; the deployed kit boots and plain-requires both (piped tclsh probe).
• Drop-in survival: an undeclared DROPIN-note.txt AND a foreign hand-placed pkgIndex.tcl at a declared path both survived re-materialization; the foreign pkgIndex genuinely won (the kit's udp smoke failed with a recapped BUILD-WARNING until the drop-in was removed
  • the G-133 probe catching exactly the payload defect a preserved drop-in can cause).
• No-declaration parity: punk8min.vfs/punk9min.vfs (undeclared) untouched by the phase; punkluck86 (their consumer) force-rebuilt content-identical (see the G-024 record: identical prefix + 2993-member payload).
• Suite: punkexe 98 tests 0 failures (includes the updated smoke-declared pin carrying punkdeclare).

Remaining manual items: none. Extension points recorded for successors: G-127 adds the platform axis to these declarations (one file per kit, ready for per-target entries); G-141's derivation input is now the per-.vfs files' source_root="packages" entries; per-kit payload detail in bakelist (archived G-121 note) can now read the declaration.