From 3369ab9c9d9ccb33c49570adb189d781f761e533 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Fri, 17 Jul 2026 00:52:13 +1000 Subject: [PATCH] G-087: thin project layouts goal - index entry + detail file (active) New goal from the 2026-07-16 layout-system review: thin layouts with generation-time infrastructure injection, fauxlink layout refs, and a working overlay merge primitive. Detail file carries the review findings (unimplemented merge, dead module-pathtype refs, stale fat snapshots, latent bugs), the staged approach (hygiene first), and the hygiene-stage completion record. Activated at user direction 2026-07-17. Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- GOALS.md | 4 + goals/G-087-thin-project-layouts.md | 202 ++++++++++++++++++++++++++++ 2 files changed, 206 insertions(+) create mode 100644 goals/G-087-thin-project-layouts.md diff --git a/GOALS.md b/GOALS.md index 133943f6..93138939 100644 --- a/GOALS.md +++ b/GOALS.md @@ -325,3 +325,7 @@ Detail: goals/G-085-cmdtrace-attribution-and-machine-output.md ### G-086 [proposed] Light project layout: punkshell-generated minimal projects carrying startup/test module-path semantics Scope: src/project_layouts/ (new light layout), src/modules/punk/mix/commandset/project-999999.0a1.0.tm (project.new layout selection), layout-seeded test harness + startup preamble (no bootsupport/vfs/build trees in generated output) Detail: goals/G-086-light-project-layout.md + +### G-087 [active] Thin project layouts: injected infrastructure, fauxlink layout refs, working overlay merge +Scope: src/project_layouts/ (payload store restructure), src/decktemplates/ (layout_refs migration to fauxlink), src/modules/punk/mix/commandset/project-999999.0a1.0.tm (project.new generation path), src/modules/punk/mix/commandset/layout-999999.0a1.0.tm, src/modules/punk/cap/handlers/templates-999999.0a1.0.tm (ref resolution), src/modules/punk/mix/#modpod-templates-999999.0a1.0/ (module-shipped refs/layouts), src/make.tcl (layout sync steps) +Detail: goals/G-087-thin-project-layouts.md diff --git a/goals/G-087-thin-project-layouts.md b/goals/G-087-thin-project-layouts.md new file mode 100644 index 00000000..b4679930 --- /dev/null +++ b/goals/G-087-thin-project-layouts.md @@ -0,0 +1,202 @@ +# G-087 Thin project layouts: injected infrastructure, fauxlink layout refs, working overlay merge + +Status: active +Scope: src/project_layouts/ (payload store restructure), src/decktemplates/ (layout_refs migration to fauxlink), src/modules/punk/mix/commandset/project-999999.0a1.0.tm (project.new generation path), src/modules/punk/mix/commandset/layout-999999.0a1.0.tm, src/modules/punk/cap/handlers/templates-999999.0a1.0.tm (ref resolution), src/modules/punk/mix/#modpod-templates-999999.0a1.0/ (module-shipped refs/layouts), src/make.tcl (layout sync steps) +Goal: project.new generates projects from thin layouts (structure, scripts, docs - no stored bootsupport/module/vfs snapshot trees) whose heavy infrastructure is injected at generation time from the generating shell's own current trees; layout references are fauxlink files (one ref concept repo-wide, no bespoke .ref grammar, no real symlinks); and the documented overlay semantics (custom-over-vendor, .anti deletions, name@base derived layouts, _project staging) are implemented by a single staging-directory materialization primitive - so a bare built kit can generate a current project, and the committed multi-megabyte stale layout snapshots are retired. +Acceptance: hygiene stage - in a punkshell checkout `dev project.new` with defaulted -layout resolves an exact layout name (no tcl::prefix ambiguity); the no-handler warning paths execute without error (put/puts typos fixed) and an absolute-pathtype provider registration does not raise an expr syntax error; test/junk refs (test2, sample-0.1 duplicate, the a.txt-only punk.shell-0.1) no longer present as selectable layouts; src/project_layouts/README.md describes the implemented punk.templates+refs mechanism including the ref filename grammar. Pivot stage - a project generated from the thin punk.project layout carries bootsupport modules matching the generating shell's current src/bootsupport (verified by comparing a marker module version such as punk::mix::cli, not a stored snapshot vintage); src/project_layouts contains no embedded bootsupport/vfs module trees and make.tcl's layout bootsupport-sync step is retired; layout refs are .fauxlink files resolved via the fauxlink module with the bespoke .ref parser removed, every layout ref carrying a non-empty nominal name (leading-# punkcheck-exclusion hazard documented in this file); a bare built kit run outside any project lists at least the module-shipped punk.project layout and generates a working project from it; and one overlay case exercising custom-over-vendor override, a .anti deletion, and a name@base derived layout (the existing sample-0.1/othersample fixtures) materializes correctly through the staging primitive into a generated project. + +## Context + +Review of the layout system (2026-07-16) found a coherent three-layer design whose +centerpiece was never implemented: + +1. **punk.templates capability** (punk::mix::templates declarations resolved by + punk::cap::handlers::templates) locates template base folders in priority order + (module < shellproject < currentproject < absolute < adhoc). +2. **Layout references** - zero-byte `.ref` files under `decktemplates/*/layout_refs/` + whose filename encodes alias + path into `/src/project_layouts` + (`alias@vendor+punk+project-0.1.ref`). This indirection is the recursion firewall: + modules ship only refs; layout payloads (which embed module trees, including + punk::mix::templates itself) live once, outside any module. +3. **Payload store** `src/project_layouts/{vendor,custom}/{,_project}/` with + overlay semantics documented in `custom/punk/sample-0.1/extra.txt` (custom over + vendor, `.anti` deletion markers, `newname@base` derived layouts, `_project` + staging pseudo-vendor). + +Unimplemented/broken as found: + +- **No merging at all** - project.new copies exactly one resolved directory + (punkcheck::install); `.anti`, custom overlays, and `name@base` folders + (e.g. `custom/punk/othersample@sample-0.1`) are documented but inert. +- **module-pathtype refs can never resolve** - their `base` is the module directory, + and `get_itemdict_projectlayouts` looks for `/src/project_layouts`, which + cannot exist inside a module. Consequence: the only source of the exact name + `punk.project` is dead, and a bare kit outside any project has an empty layout list + (project.new unusable standalone). +- **Default -layout is prefix-ambiguous** in-checkout: no exact `punk.project` exists; + `tcl::prefix::match` sees `punk.project-0.1` and `punk.project-0.1_overrides`. +- **Stale fat snapshots**: `vendor/punk/project-0.1` is 559 files / ~19MB frozen at the + punk::mix::cli 0.3/0.3.1 era (both versions side by side); `custom/_project/*` + duplicates ~23MB more. make.tcl refreshes only the `custom/_project` copies + (bootsupport + make/build sync) - treating drift symptomatically. +- **Names lie**: `@vendor+_project+punk.shell-0.1.ref` presents as `punk.shell-0.1` + but targets a folder containing a single junk `a.txt`; the real 12MB shell layout is + exposed only as `punk.shell-0.1_overrides`. Test refs (`test2`, `sample-0.1`) show as + selectable layouts. +- **Child materialization missing**: the vendored layout carries a marker folder + `src/PROJECT_LAYOUTS_CREATE_FROM_DECKTEMPLATES_LAYOUT_REFERENCES/` (and project.new + excludes `src/PROJECT_LAYOUTS_*`); the step it names was never written, so generated + projects hold refs pointing at a folder that never exists. +- **Design-generation residue**: README describes a `punk.projectlayout` capability + that exists nowhere in code (punk.templates + refs superseded it); the modpod's + `templates/layouts/project` subtree has no reader; make.tcl still carries + `old_layout_update_list` referencing `src/mixtemplates`. +- **Latent bugs**: `put stderr` (missing s) in no-handler warning paths + (commandset/layout.tm ~lines 208/219, commandset/project.tm ~line 255); non-braced + `if {!file exists $normpath}` in the templates handler's `absolute` branch + (~line 175) errors on any absolute-pathtype registration. + +The pivot insight: the recursion problem and the snapshot weight have the same cause - +layouts *store* module trees. Stop storing them (inject at generation time from the +generating shell, which always has current bootsupport) and: layouts become small +enough to ship inside the punk::mix::templates module without recursion (a bare kit +becomes self-sufficient); ~40MB of committed snapshots leaves the repo; overlay merging +becomes cheap; and generated projects start current instead of at snapshot vintage, +with later refresh belonging to G-027's pull. + +## Approach + +Staged; hygiene first since it is independently shippable. + +1. **Hygiene** (no design changes): fix the `put`/`!file exists` bugs; restore an exact + default layout name (rename `@vendor+punk+project-0.1.ref` to + `project@vendor+punk+project-0.1.ref`, yielding exactly `punk.project`); quarantine + or ignore-prefix the test/junk refs; rewrite `src/project_layouts/README.md` to + describe the implemented system and the ref filename grammar; remove the dead modpod + `templates/layouts` subtree and make.tcl `old_layout_update_list` residue. +2. **Fauxlink refs** (user decision 2026-07-16: one ref concept repo-wide, no real + symlinks): migrate `layout_refs/*.ref` to `.fauxlink` files resolved by the vendored + `fauxlink` module - `alias@vendor+punk+project-0.1.ref` becomes + `alias#vendor+punk+project-0.1.fauxlink` (same `+` path encoding; `@` in target + segments like `othersample@sample-0.1` is legal literal fauxlink content). Target + paths stay application-resolved against `/src/project_layouts` (fauxlink + is app-driven by design). Zero-byte property preserved (target fully in the name). + Clean cutover preferred over dual-grammar support - all refs are in-repo. + Constraint: layout refs MUST carry a non-empty nominal name - an alias-less ref + (`#vendor+...fauxlink`) starts with `#`, which this project's punkcheck core + excludes (`#*`, `**/#*`) treat as aside/hidden, so it would silently vanish from + punkcheck-driven installs. Opportunities: the fauxlink tags segment can carry layout + metadata (e.g. staging markers); the reserved TOML body can carry per-ref update + policy (the `allowupdates`/`repo`/`reposubdir` keys currently parked in the + punk::mix::templates declarations) - feeding G-027 provenance. +3. **Thin layouts + injection**: strip stored bootsupport/vfs/module trees from layout + payloads; project.new copies the thin layout then injects bootsupport from the + generating shell's own current trees (same mechanism make.tcl already uses to sync + `custom/_project` layouts, relocated to generation time). Retire make.tcl's + layout bootsupport-sync step and the fat `custom/_project` copies. Ship the thin + `punk.project` layout (plus its ref) inside the punk::mix::templates module so + module-pathtype refs resolve to module-carried payloads and a bare kit works. +4. **Overlay merge primitive**: a `layout_materialize` lib proc resolving a layout name + to an ordered chain of source dirs (base from `name@base` folder naming if present, + then vendor//, then custom//, then `_project` overlays), composed + into a staging directory applying `.anti` deletions; project.new keeps its + single-path contract and receives the staged path. Brings every documented-but-dead + concept to life at once. +5. **Retirement/decision**: drop or implement the child-side + `PROJECT_LAYOUTS_CREATE_FROM_...` marker (with thin layouts, children need only + refs + thin payloads; heavy parts arrive by the same injection and refresh via + G-027); decide whether the vendor/custom x `_project` matrix at both levels still + earns its complexity once overlays are cheap (vendor/_project and custom/_project + may collapse into one override level). + +## Alternatives considered + +- Real filesystem symlinks for refs - rejected: platform-finicky (Windows privilege + and VCS behaviour), and fauxlink exists precisely to avoid them (user preference, + 2026-07-16). +- Keeping the bespoke `.ref` grammar - rejected: it is a hand-rolled subset of + fauxlink's naming scheme; two ref concepts in one repo with near-identical encodings + invites drift. +- Teaching punkcheck::install multi-source layered installs - rejected in favour of + staging-directory composition: simpler, independently testable, preserves + project.new's single-path contract. +- Keeping fat vendor snapshots and fixing drift via pull updates alone - rejected as + end state: the snapshots would still be redundant third/fourth copies of bootsupport + content, still 40MB in-repo, and the recursion constraint on module-shipped layouts + would remain. + +## Notes + +- fauxlink upstream source is directly accessible at C:/repo/jn/tclmodules/fauxlink + (fossil checkout, punkshell-style layout; user confirmed 2026-07-17): fauxlink tests + or module changes needed for the stage-2 ref migration can be made upstream there and + re-vendored, rather than patching the vendored copy. +- Related goals: G-086 (light project layout - becomes just another thin layout with + less injected; this goal generalizes its premise to all layouts), G-027 + (derived-project pull updates - takes over post-generation refresh; fauxlink TOML + body is a candidate carrier for its provenance/update-policy records), G-012 + (template payload safety - inert VCS-config payload rules apply to thin layouts + unchanged), G-004 (no committed binaries - snapshot retirement aligns), G-026 + (vendor provenance policy - source-side cleanliness for injection). +- Naming rules to enforce with the merge primitive: visible name = vendor.name derived + from the ref; report colliding visible names in `dev project.layouts`; versionless + request picks highest version (existing todo in commandset/project.tm); default + -layout must be an exact name. +- The `#n` duplicate-suffix mechanism in `_get_itemdict` and the priority ordering + (module < shellproject < currentproject < absolute < adhoc) are sound and carry over + unchanged. + +## Progress + +### 2026-07-17 hygiene stage complete (increment 1) + +Landed (punk::mix::commandset::layout 0.1.1, project 0.2.2, module 0.1.1, +scriptwrap 0.1.1; punk::mix::base 0.1.1; punk::cap::handlers::templates 0.1.1; +punk::mix::templates 0.1.4; punkshell 0.12.38): + +- put -> puts fixed at all six no-handler warning sites. The closeout sweep found + three sites beyond the review's list (mix/base get_template_basefolders, + commandset::module templates_dict, commandset::scriptwrap templates_dict). Three + procs also dereferenced an unset result variable after the warning - the + no-handler paths now warn and return an empty dict; layout/module warning + messages no longer misattribute their owning proc. +- templates handler: the non-braced 'if {!file exists $normpath}' expr bug fixed. + Verification exposed an adjacent latent bug, also fixed: pkg_register's + duplicate-registration guard did 'dict get $provider_info $pkg' without a + dict-exists check, erroring for any second provider package registering the + same capability. +- @vendor+punk+project-0.1.ref renamed to project@vendor+punk+project-0.1.ref: + exact layout name 'punk.project' now exists and the defaulted -layout + prefix-resolves unambiguously. +- Junk refs deleted: test2, the custom/_project duplicate sample-0.1 ref, and the + alias-less punk.shell-0.1 ref together with its a.txt-only target folder + (project_layouts/vendor/_project/punk.shell-0.1). +- src/project_layouts/README.md rewritten: implemented punk.templates + layout_refs + mechanism, ref filename grammar, unimplemented-overlay status, G-087 pointer; + dead punk.projectlayout capability description removed. +- Dead modpod templates/layouts/project payload removed (punk::mix::templates + 0.1.4); make.tcl old_layout_update_list and stale mixtemplates help text removed. + +Verification (2026-07-17, native tclsh 8.7/9 runtests + rebuilt root modules): + +- Scratch script (g087_hygiene_verify.tcl) 13/13: exact punk.project present and + tcl::prefix::match resolves it; test2 / bare sample-0.1 / punk.shell-0.1 absent + from layouts_dict; absolute-pathtype registration returns 1 for an existing path + and warn+0 for a missing one; with the handler entry blanked, all five + no-handler paths warn and return empty without error. +- runtests: prune 19/19, fossilmove 10/10, multishell 9 pass + 1 skip, + libsearch 8/8. make.tcl modules builds clean; rebuilt templates modpod 0.1.4 + mounts and its module-shipped ref lists as punk.project#2 (module-pathtype + resolution still dead pending stage 3). +- Resulting layout names: punk.sample-0.1 punk.project punk.minimal punk.basic + punk.shell-0.1_overrides punk.project-0.1_overrides. + +Remaining for acceptance (pivot stage - Approach stages 2-5): + +- fauxlink ref migration (stage 2). +- thin layouts + generation-time bootsupport injection, retire make.tcl layout + bootsupport-sync, module-shipped thin punk.project layout for bare kits (stage 3). +- staging-directory overlay merge primitive exercising the sample-0.1/othersample + fixtures (stage 4). +- child-side PROJECT_LAYOUTS_CREATE_FROM_... marker retirement and the + vendor/custom x _project matrix decision (stage 5).