punkshell/goals/G-087-thin-project-layouts.md

G-087 Thin project layouts: injected infrastructure, fauxlink layout refs, working overlay merge

Status: active Scope: src/project_layouts/ (payload store restructure), src/decktemplates/ (layout_refs migration to fauxlink), src/modules/punk/mix/commandset/project-999999.0a1.0.tm (project.new generation path), src/modules/punk/mix/commandset/layout-999999.0a1.0.tm, src/modules/punk/cap/handlers/templates-999999.0a1.0.tm (ref resolution), src/modules/punk/mix/#modpod-templates-999999.0a1.0/ (module-shipped refs/layouts), src/make.tcl (layout sync steps) Goal: project.new generates projects from thin layouts (structure, scripts, docs - no stored bootsupport/module/vfs snapshot trees) whose heavy infrastructure is injected at generation time from the generating shell's own current trees; layout references are fauxlink files (one ref concept repo-wide, no bespoke .ref grammar, no real symlinks); and the documented overlay semantics (custom-over-vendor, .anti deletions, name@base derived layouts, _project staging) are implemented by a single staging-directory materialization primitive - so a bare built kit can generate a current project, and the committed multi-megabyte stale layout snapshots are retired. Acceptance: hygiene stage - in a punkshell checkout dev project.new with defaulted -layout resolves an exact layout name (no tcl::prefix ambiguity); the no-handler warning paths execute without error (put/puts typos fixed) and an absolute-pathtype provider registration does not raise an expr syntax error; test/junk refs (test2, sample-0.1 duplicate, the a.txt-only punk.shell-0.1) no longer present as selectable layouts; src/project_layouts/README.md describes the implemented punk.templates+refs mechanism including the ref filename grammar. Pivot stage - a project generated from the thin punk.project layout carries bootsupport modules matching the generating shell's current src/bootsupport (verified by comparing a marker module version such as punk::mix::cli, not a stored snapshot vintage); src/project_layouts contains no embedded bootsupport/vfs module trees and make.tcl's layout bootsupport-sync step is retired; layout refs are .fauxlink files resolved via the fauxlink module with the bespoke .ref parser removed, every layout ref carrying a non-empty nominal name (leading-# punkcheck-exclusion hazard documented in this file); a bare built kit run outside any project lists at least the module-shipped punk.project layout and generates a working project from it; and one overlay case exercising custom-over-vendor override, a .anti deletion, and a name@base derived layout (the existing sample-0.1/othersample fixtures) materializes correctly through the staging primitive into a generated project.

Context

Review of the layout system (2026-07-16) found a coherent three-layer design whose centerpiece was never implemented:

1. punk.templates capability (punk::mix::templates declarations resolved by punk::cap::handlers::templates) locates template base folders in priority order (module < shellproject < currentproject < absolute < adhoc).
2. Layout references - zero-byte .ref files under decktemplates/*/layout_refs/ whose filename encodes alias + path into <projectroot>/src/project_layouts (alias@vendor+punk+project-0.1.ref). This indirection is the recursion firewall: modules ship only refs; layout payloads (which embed module trees, including punk::mix::templates itself) live once, outside any module.
3. Payload store src/project_layouts/{vendor,custom}/{<vendor>,_project}/ with overlay semantics documented in custom/punk/sample-0.1/extra.txt (custom over vendor, .anti deletion markers, newname@base derived layouts, _project staging pseudo-vendor).

Unimplemented/broken as found:

• No merging at all - project.new copies exactly one resolved directory (punkcheck::install); .anti, custom overlays, and name@base folders (e.g. custom/punk/othersample@sample-0.1) are documented but inert.
• module-pathtype refs can never resolve - their base is the module directory, and get_itemdict_projectlayouts looks for <base>/src/project_layouts, which cannot exist inside a module. Consequence: the only source of the exact name punk.project is dead, and a bare kit outside any project has an empty layout list (project.new unusable standalone).
• Default -layout is prefix-ambiguous in-checkout: no exact punk.project exists; tcl::prefix::match sees punk.project-0.1 and punk.project-0.1_overrides.
• Stale fat snapshots: vendor/punk/project-0.1 is 559 files / ~19MB frozen at the punk::mix::cli 0.3/0.3.1 era (both versions side by side); custom/_project/* duplicates ~23MB more. make.tcl refreshes only the custom/_project copies (bootsupport + make/build sync) - treating drift symptomatically.
• Names lie: @vendor+_project+punk.shell-0.1.ref presents as punk.shell-0.1 but targets a folder containing a single junk a.txt; the real 12MB shell layout is exposed only as punk.shell-0.1_overrides. Test refs (test2, sample-0.1) show as selectable layouts.
• Child materialization missing: the vendored layout carries a marker folder src/PROJECT_LAYOUTS_CREATE_FROM_DECKTEMPLATES_LAYOUT_REFERENCES/ (and project.new excludes src/PROJECT_LAYOUTS_*); the step it names was never written, so generated projects hold refs pointing at a folder that never exists.
• Design-generation residue: README describes a punk.projectlayout capability that exists nowhere in code (punk.templates + refs superseded it); the modpod's templates/layouts/project subtree has no reader; make.tcl still carries old_layout_update_list referencing src/mixtemplates.
• Latent bugs: put stderr (missing s) in no-handler warning paths (commandset/layout.tm ~lines 208/219, commandset/project.tm ~line 255); non-braced if {!file exists $normpath} in the templates handler's absolute branch (~line 175) errors on any absolute-pathtype registration.

The pivot insight: the recursion problem and the snapshot weight have the same cause - layouts store module trees. Stop storing them (inject at generation time from the generating shell, which always has current bootsupport) and: layouts become small enough to ship inside the punk::mix::templates module without recursion (a bare kit becomes self-sufficient); ~40MB of committed snapshots leaves the repo; overlay merging becomes cheap; and generated projects start current instead of at snapshot vintage, with later refresh belonging to G-027's pull.

Approach

Staged; hygiene first since it is independently shippable.

1. Hygiene (no design changes): fix the put/!file exists bugs; restore an exact default layout name (rename @vendor+punk+project-0.1.ref to project@vendor+punk+project-0.1.ref, yielding exactly punk.project); quarantine or ignore-prefix the test/junk refs; rewrite src/project_layouts/README.md to describe the implemented system and the ref filename grammar; remove the dead modpod templates/layouts subtree and make.tcl old_layout_update_list residue.
2. Fauxlink refs (user decision 2026-07-16: one ref concept repo-wide, no real symlinks): migrate layout_refs/*.ref to .fauxlink files resolved by the vendored fauxlink module - alias@vendor+punk+project-0.1.ref becomes alias#vendor+punk+project-0.1.fauxlink (same + path encoding; @ in target segments like othersample@sample-0.1 is legal literal fauxlink content). Target paths stay application-resolved against <projectroot>/src/project_layouts (fauxlink is app-driven by design). Zero-byte property preserved (target fully in the name). Clean cutover preferred over dual-grammar support - all refs are in-repo. Constraint: layout refs MUST carry a non-empty nominal name - an alias-less ref (#vendor+...fauxlink) starts with #, which this project's punkcheck core excludes (#*, **/#*) treat as aside/hidden, so it would silently vanish from punkcheck-driven installs. Opportunities: the fauxlink tags segment can carry layout metadata (e.g. staging markers); the reserved TOML body can carry per-ref update policy (the allowupdates/repo/reposubdir keys currently parked in the punk::mix::templates declarations) - feeding G-027 provenance.
3. Thin layouts + injection: strip stored bootsupport/vfs/module trees from layout payloads; project.new copies the thin layout then injects bootsupport from the generating shell's own current trees (same mechanism make.tcl already uses to sync custom/_project layouts, relocated to generation time). Retire make.tcl's layout bootsupport-sync step and the fat custom/_project copies. Ship the thin punk.project layout (plus its ref) inside the punk::mix::templates module so module-pathtype refs resolve to module-carried payloads and a bare kit works.
4. Overlay merge primitive: a layout_materialize lib proc resolving a layout name to an ordered chain of source dirs (base from name@base folder naming if present, then vendor//, then custom//, then _project overlays), composed into a staging directory applying .anti deletions; project.new keeps its single-path contract and receives the staged path. Brings every documented-but-dead concept to life at once.
5. Retirement/decision: drop or implement the child-side PROJECT_LAYOUTS_CREATE_FROM_... marker (with thin layouts, children need only refs + thin payloads; heavy parts arrive by the same injection and refresh via G-027); decide whether the vendor/custom x _project matrix at both levels still earns its complexity once overlays are cheap (vendor/_project and custom/_project may collapse into one override level).

Alternatives considered

• Real filesystem symlinks for refs - rejected: platform-finicky (Windows privilege and VCS behaviour), and fauxlink exists precisely to avoid them (user preference, 2026-07-16).
• Keeping the bespoke .ref grammar - rejected: it is a hand-rolled subset of fauxlink's naming scheme; two ref concepts in one repo with near-identical encodings invites drift.
• Teaching punkcheck::install multi-source layered installs - rejected in favour of staging-directory composition: simpler, independently testable, preserves project.new's single-path contract.
• Keeping fat vendor snapshots and fixing drift via pull updates alone - rejected as end state: the snapshots would still be redundant third/fourth copies of bootsupport content, still 40MB in-repo, and the recursion constraint on module-shipped layouts would remain.

Notes

• fauxlink upstream source is directly accessible at C:/repo/jn/tclmodules/fauxlink (fossil checkout, punkshell-style layout; user confirmed 2026-07-17): fauxlink tests or module changes needed for the stage-2 ref migration can be made upstream there and re-vendored, rather than patching the vendored copy.
• fauxlink 0.1.2 upstream work (2026-07-17, user-directed): complete test suite (src/tests/fauxlink.test, 62 tests), punk::args documentation blocks (inert PUNKARGS + registration, no punk::args dependency) including a (package)fauxlink id - the id registers and resolves once its namespace is scanned; automatic id-derived scan and user-facing rendering await G-075. Fixes: decode_unicode_escapes first-char drop on hexless %U, 3-segment comment validation/decoding parity, link_as stub errors instead of silent no-op. link_as (the encoder counterpart of resolve) remains unimplemented - stage 2 migration tooling will want it (candidate follow-on).
• punkshell fauxlink consumer survey (2026-07-17): punk::mix::util fcat follows .fauxlink (and .lnk via punk::winlnk) unless -noredirect - note fcat matches only the .fauxlink extension, not .fxlnk; make.tcl merge_over does fauxlink-aware recursive copy (follows 'punk::boot,merge_over'-tagged links, copies others as files) and the startup-script check resolves root main.tcl fauxlinks; src/scriptapps/punk.tcl kit boot fallback follows a main.tcl#*.fauxlink/.fxlnk with the same tag; vendored via src/vendormodules/include_modules.config from c:/repo/jn/tclmodules/fauxlink/modules.
• Intention (user, 2026-07-17): punk::nav::fs should gain sensible fauxlink handling in future - analogous to its optional punk::winlnk treatment of windows .lnk shortcuts (resolve for target/type classification and display) - so fauxlinks present usefully in directory listings/navigation. Candidate goal when picked up.
• Related goals: G-086 (light project layout - becomes just another thin layout with less injected; this goal generalizes its premise to all layouts), G-027 (derived-project pull updates - takes over post-generation refresh; fauxlink TOML body is a candidate carrier for its provenance/update-policy records), G-012 (template payload safety - inert VCS-config payload rules apply to thin layouts unchanged), G-004 (no committed binaries - snapshot retirement aligns), G-026 (vendor provenance policy - source-side cleanliness for injection).
• Naming rules to enforce with the merge primitive: visible name = vendor.name derived from the ref; report colliding visible names in dev project.layouts; versionless request picks highest version (existing todo in commandset/project.tm); default -layout must be an exact name.
• The #n duplicate-suffix mechanism in _get_itemdict and the priority ordering (module < shellproject < currentproject < absolute < adhoc) are sound and carry over unchanged.

Progress

2026-07-17 hygiene stage complete (increment 1)

Landed (punk::mix::commandset::layout 0.1.1, project 0.2.2, module 0.1.1, scriptwrap 0.1.1; punk::mix::base 0.1.1; punk::cap::handlers::templates 0.1.1; punk::mix::templates 0.1.4; punkshell 0.12.38):

• put -> puts fixed at all six no-handler warning sites. The closeout sweep found three sites beyond the review's list (mix/base get_template_basefolders, commandset::module templates_dict, commandset::scriptwrap templates_dict). Three procs also dereferenced an unset result variable after the warning - the no-handler paths now warn and return an empty dict; layout/module warning messages no longer misattribute their owning proc.
• templates handler: the non-braced 'if {!file exists $normpath}' expr bug fixed. Verification exposed an adjacent latent bug, also fixed: pkg_register's duplicate-registration guard did 'dict get $provider_info $pkg' without a dict-exists check, erroring for any second provider package registering the same capability.
• @vendor+punk+project-0.1.ref renamed to project@vendor+punk+project-0.1.ref: exact layout name 'punk.project' now exists and the defaulted -layout prefix-resolves unambiguously.
• Junk refs deleted: test2, the custom/_project duplicate sample-0.1 ref, and the alias-less punk.shell-0.1 ref together with its a.txt-only target folder (project_layouts/vendor/_project/punk.shell-0.1).
• src/project_layouts/README.md rewritten: implemented punk.templates + layout_refs mechanism, ref filename grammar, unimplemented-overlay status, G-087 pointer; dead punk.projectlayout capability description removed.
• Dead modpod templates/layouts/project payload removed (punk::mix::templates 0.1.4); make.tcl old_layout_update_list and stale mixtemplates help text removed.

Verification (2026-07-17, native tclsh 8.7/9 runtests + rebuilt root modules):

• Scratch script (g087_hygiene_verify.tcl) 13/13: exact punk.project present and tcl::prefix::match resolves it; test2 / bare sample-0.1 / punk.shell-0.1 absent from layouts_dict; absolute-pathtype registration returns 1 for an existing path and warn+0 for a missing one; with the handler entry blanked, all five no-handler paths warn and return empty without error.
• runtests: prune 19/19, fossilmove 10/10, multishell 9 pass + 1 skip, libsearch 8/8. make.tcl modules builds clean; rebuilt templates modpod 0.1.4 mounts and its module-shipped ref lists as punk.project#2 (module-pathtype resolution still dead pending stage 3).
• Resulting layout names: punk.sample-0.1 punk.project punk.minimal punk.basic punk.shell-0.1_overrides punk.project-0.1_overrides.

2026-07-17 stage 2 complete: fauxlink layout refs (increment 2)

Landed (punk::cap::handlers::templates 0.2.0, punk::mix::templates 0.1.5; fauxlink 0.2.0 vendored from upstream):

• fauxlink::link_as implemented upstream (fauxlink 0.2.0, checkin 1e6b4ee8): name + forward-slash target (+ optional -tags/-comments/-extension) -> fauxlink filename with a resolve round-trip guarantee. 83-test upstream suite.
• Handler cutover: get_itemdict_projectlayoutrefs selects #.fauxlink/.fxlnk and derives names via fauxlink::resolve; get_itemdict_projectlayouts derives the layout folder from the resolved targetpath; the bespoke @.ref parser is gone and .ref files are no longer recognised. Alias-less (leading-#) refs and unresolvable refs are skipped with a warning (punkcheck hazard per Approach stage 2). Bonus fix: the ignore* skip previously matched against the full glob path so it never applied.
• All 7 live-tier refs migrated via a link_as-driven script (each new name round-trip-verified before rename): decktemplates vendor/punk x4 (alias-less refs gained their target-tail aliases: basic, minimal, sample-0.1), custom/_project x2, module-shipped modpod ref x1. Zero-byte property kept.
• src/project_layouts/README.md ref-grammar section rewritten for fauxlink.

Verification (2026-07-17): stage-2 scratch script 7/7 - layout names identical to the pre-migration set with exact punk.project resolution; refs dict paths all .fauxlink; planted alias-less decoy warned + skipped; planted old-grammar .ref decoy ignored; module-shipped ref lists as punk.project#2 (resolution still dead pending stage 3). runtests prune/fossilmove/multishell/libsearch all pass. End-to-end dev project.new <abs-path> -confirm 0 (punk902z src, piped, FOSSIL_HOME diverted) resolved the default punk.project layout through fauxlink refs and materialized the full project tree; the run's trailing in-project fossil commit failed on a binary-data prompt (layout payload ships no binary-glob - G-012 territory, unrelated to refs).

Notes from the work:

• Stale-snapshot payload copies of the old .ref grammar remain under src/project_layouts/*/src/bootsupport/... and the fat vendor snapshot's inner trees - stale-by-design, retired with the snapshots in stage 3. Generated projects' decktemplates payloads currently carry no layout_refs at all.
• project.new from a cwd outside any project has an empty layout list even with a shell built from a project: the punk::mix::templates shellproject_multivendor declaration points at /decktemplates (repo root - nonexistent), not src/decktemplates. Pre-existing; natural stage-3 fix alongside module-ref resolution.
• punkcheck::install source reads can exceed Windows MAX_PATH when the relative source chain is long (deep target dir + many ../ segments, unnormalized path); surfaced as 'could not read ... no such file' during a deep-scratchpad generation. Pre-existing, candidate for a punkcheck normalization fix.
• make.tcl bootsupport prune of punk/mix/templates-.tm fails with permission denied when the running make.tcl process itself has that modpod mounted (self-lock); the stale copy must be deleted manually afterwards. Candidate small fix: unmount or defer-prune modpods the process mounted.
• 2026-07-19 punkcheck lifecycle/locking assessment (three parallel record lifecycle implementations with drift; unguarded read-modify-write windows) spun out as G-094 (single OO lifecycle) and G-095 (concurrent-writer safety); the MAX_PATH normalization candidate above is flagged in G-094's notes as a natural rider on its load/save consolidation.

2026-07-19 stage 3 complete: thin layouts + generation-time injection (increment 3)

Landed (punk::cap::handlers::templates 0.3.0, punk::mix::templates 0.2.0, punk::mix::commandset::project 0.3.0, punkcheck 0.4.0, punk::mix::base 0.1.2 (manual rename), make.tcl reworked, punkshell 0.16.0):

• Store thinned: vendor/punk/project-0.1 lost its src/bootsupport (10.6MB/254 files) and src/vfs/project.vfs lib+modules snapshots (7.7MB/230 files); vendor/punk/basic lost its 2023-era bootsupport snapshot; the fat custom/_project fragments (punk.project-0.1, punk.shell-0.1, punk.basic - pure bootsupport+make/build overlay fragments, never standalone-coherent) were deleted with their two _overrides refs. Thin layouts carry structure, scripts, docs, fossil/git settings and the three bootsupport include_modules.config manifests. New README markers document injection. Stale pre-modpod residue cleaned: unpacked punk/mix/templates trees (with an old-grammar .ref) in src/bootsupport + _vfscommon.vfs + root modules, and a superseded templates-0.1.0.tm in _vfscommon.vfs.
• Injection: project.new calls new lib::bootsupport_inject after the layout copies - resolves each manifest entry (make.tcl bootsupport semantics: latest non-glob version, glob keeps all; 999999.* never injected) from the shell project's src/bootsupport tree, or a standalone kit's tcl:™️:list paths; carries adjacent non-.tm support files (struct sets_.tcl, textutil .tex); tree-copies bootsupport lib (fileutil etc.) from a shell project (kit mode: notice - lib set has no manifest); punkcheck-recorded into the generated project's src/bootsupport/.punkcheck with plain-copy degradation. -layout now prefix-matches the live layout set (load-time LAYOUTNAMES snapshot could reject layouts appearing after package load).
• Module-carried layouts: handler resolves module-pathtype refs against the module's own templates/project_layouts (sibling of layout_refs); the templates modpod ships the thin punk.project payload (synced, excludes bin/sdx.kit per no-new-binaries and the src/PROJECT_LAYOUTS_* marker; .fossil-custom/.fossil-settings included via core-exclude override); shellproject_multivendor decl fixed decktemplates -> src/decktemplates.
• make.tcl: bootsupport-command layout loop retired; sync_layouts retargeted to vendor layouts carrying src/make.tcl (scripts + manifests, punkcheck db at src/project_layouts/.punkcheck) + store->modpod payload sync + sourcevanished prune; workflow text updated (DIAGRAM 1b); bootsupport copy onto a self-mounted modpod records OK when content already identical (FAILED-record churn fix - the pod is mounted by the running make.tcl itself).
• Latent bugs found and fixed en route:
  • punkcheck start_installer_event wrote LITERAL '$eventid'/'$rel_source'/ '$rel_target'/'$config' into every install event header (braced-literal {*}{} misuse, since inception; invisible until installtrack + install shared one db - duplicate '$eventid' keys crashed the constructor).
  • punkcheck cross-root sources: get_relativecksum was called with the punkcheck folder as base for //zipfs:/ sources -> 'don't share a common root' error; now uses the helpers' independent-absolute (empty base) mode.
  • punk::mix::base cksum_path hung FOREVER on vfs files: tcllib -file digest modes read via fileevent+vwait, and zipfs channels never deliver fileevents (core: ZipChannelWatchChannel is an empty stub, no OS handle either; native Windows FileWatchProc fakes readiness via Tcl_SetMaxBlockTime2(0)). Non-native files are now slurped and digested in data mode (0.1.2). Upstream ticket candidate.
  • bootsupport manifest was INCOMPLETE vs the working tree: cmdline (fileutil dep), struct::set (make.tcl accelerators), textutil umbrella, expander, patch existed only as recordless legacy files - invisible to injection. Entries added; generated projects' make.tcl now boots under plain tclsh.

Verification (2026-07-19, native tclsh 9.0.3):

• runtests: full suite 974 pass / 15 skip / 1 fail = the documented exec-14.3 baseline; punkcheck install+installtrack 64/64; prune/fossilmove/multishell/ libsearch suites pass.
• In-checkout (punk902z src, piped, FOSSIL_HOME+HOME diverted): layouts list punk.project (store) + punk.project#2 (modpod, now with a REAL resolvable //zipfs:/ path) + basic/minimal/sample; no _overrides ghosts; generated project's bootsupport marker punk::mix::cli-0.5.0 matches the checkout src/bootsupport exactly; handler 0.3.0/templates 0.2.0/punkcheck 0.4.0 injected; no 999999 versions; fossil-settings present; plain-tclsh 'src/make.tcl info' in the generated project succeeds.
• Bare kit (fresh punk902z.exe copied outside the repo, cwd outside any project): lists exactly punk.project resolving into the mounted modpod payload; generates a full project through zipfs-source punkcheck::install (102-107 modules injected from kit module paths, current versions, adjacent files included); kit-hosted ' src/make.tcl info' in the generated project succeeds (plain-tclsh path needs the lib-set notice honoured).
• make.tcl bootsupport converges to 0 copies on the second pass again.

Notes / residue from stage 3:

• Generated-project trailing fossil commit still aborts on binary data (bin/sdx.kit from checkout-generation; injected zip-based modpod .tm files in all cases) - G-012 territory: thin layouts ship no binary-glob.
• Standalone-kit generation cannot inject bootsupport lib packages (fileutil etc.) - no manifest exists for the lib set (bootsupport libs are governed by tclpackages.toml at the checkout level only). Candidate: an include_libs manifest, or fold into G-027 pull. Kit-hosted make.tcl runs are unaffected.
• The repodb-folder prompt in punk::repo::fossil_get_repository_folder_for_project ignores -confirm 0 when more than one .fossils candidate exists (or any conflict) - unattended kit-mode generation needs a single unambiguous candidate (e.g FOSSIL_HOME with a .fossils subdir). Candidate improvement: honour a non-interactive flag by auto-selecting the first non-conflicted choice.
• modpod packing is byte-unstable across packs (zip timestamps), so a repack without payload change churns downstream punkcheck copies once; and a make.tcl process cannot overwrite a modpod it has mounted (the convergence fix records OK on identical content; a real content change to a same-named bootsupport pod requires a second run or manual copy while no make.tcl is running). Candidate: deterministic pack timestamps + unmount-before-copy.
• sdx.kit remains in the store layout (grandfathered committed binary, excluded from the modpod) - G-004/G-006 decide its future.

Remaining for acceptance (Approach stages 4-5):

• staging-directory overlay merge primitive exercising the sample-0.1/othersample fixtures (stage 4).
• child-side PROJECT_LAYOUTS_CREATE_FROM_... marker retirement and the vendor/custom x _project matrix decision (stage 5).