jn
/
punkshell
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1292 Commits
1 Branch
1 Tag
217 MiB
 
 
 
 
 
 
Tree: a70ddfbdb3
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'a70ddfbdb3'
${ noResults }
punkshell/goals/G-127-crosstarget-vfs-bake.md

22 KiB
Raw Blame History

G-127 Cross-target bake of a custom .vfs

Status: active Scope: src/make.tcl (target-keyed output location and deploy folder, kit-name disambiguation by target, payload/target mismatch reporting, retirement of the kit-loop cross-platform TODO); src/vfs/*.vfs.toml (per-target payload declaration - the achieved G-115 surface this goal adds the platform axis to); src/runtime/mapvfs.toml (one vfs definition paired with several targets); src/vfs/ (per-target payload convention with punk9linux.vfs as the migration case); bin/kits// (non-native kit output tier, beside the existing bin/runtime// input tier); src/tests/shell/testsuites/punkexe/ (characterization - two-target bake, no-collision, mismatch report); bin/AGENTS.md + src/AGENTS.md (where cross-target kits land) Goal: one kit definition bakes for several declared target platforms from a single source of truth - each target getting ITS binary payload and an output location that cannot be overwritten by another target's kit of the same name - so building a custom .vfs for this machine and for FreeBSD is a declaration, not a duplicated .vfs folder and a naming convention held in the developer's head. G-122 made the build read the right runtime for a target; this makes it write the right kit for one. Acceptance: two mapvfs entries pairing the SAME vfs definition with two different targets bake in one run to distinct artifacts - a native-target kit stays at bin/ exactly where it lands today, a non-native one at bin/kits// with src/_build separated the same way - and neither is overwritten when the two are baked as separate selective single-kit runs (the current defect: exe_names_seen resets per run, so the second write lands on the first); two NON-NATIVE targets sharing a kit name coexist as distinct artifacts, where today they collide or are disambiguated by runtime name rather than by platform; the binary payload installed into each target's kit is selected for that target from src/vendorlib_tcl/ through one declaration rather than by duplicating a .vfs folder per target, and a kit declared for a single target builds byte-comparably to today; a payload binary that would be LOADED on the kit's target but was built for another platform is reported naming the file and both platforms, while per-platform subdirectory layouts that the loader resolves correctly (e.g tcllib's //) are explicitly not flagged; bakelist shows each row's output location; the existing punk9linux.vfs is either migrated to the declaration or recorded here as deliberately staying hand-curated, and the linux punkshell902 kit's move from bin/ to bin/kits/linux-x86_64/ is recorded as an intended relocation; the G-023 reconciliation is stated (version-named outputs compose with the platform directory rather than competing with it); native win32 kit outputs are unchanged in name and location and the existing punkexe suites pass unchanged; the kit-loop TODO at src/make.tcl "allow building of kits for other platforms" is retired pointing here.

Context

G-122 (achieved - see goals/archive/G-122-host-target-platform-split.md) made the build READ correctly for a target: which bin/runtime/<tier> a runtime comes from, its filename suffix, the artifact's suffix, presence checks, process-sweep applicability. It deliberately did not touch the output side. make.tcl still carries the standing TODO in the kit loop, of which G-122 satisfied only the second half:

#TODO - allow building of kits for other platforms
# - we need to use <project>/bin for only kits targetting current platform,
#   and use <project>/bin/<platform> for others
#   that we we can have same target executable for multiple platforms
# - to do this we need to change runtime/mapvfs.config to have platform names   <-- done

Three gaps remain, all measured 2026-07-26:

  1. Output is flat. src/_build/<name> and bin/<name>, no platform dimension. targetkit = <appname><suffix> and both linux-x86_64 and freebsd-x86_64 yield an empty suffix, so two POSIX targets of the same kit collide. Within one run the duplicate guard renames the second to <appname>_<runtimename> - RUNTIME-keyed, not platform-keyed. Across separate SELECTIVE runs exe_names_seen starts empty, so the second simply overwrites the first in both src/_build and bin. win32-vs-posix is safe today only because of the .exe suffix - naming luck, not design, and the reason the linux punkshell902 currently coexists with punk91.exe without trouble.

  2. The payload model is one hand-curated folder per target. punk9linux.vfs IS the convention: a separate folder carrying linux .so files (tcllibc, tcltls, tdom, thread). It is not a vfslibs participant, so its lib tree is maintained by hand. vendorlib_vfs.toml hardcodes the platform into the source path (source = "vendorlib_tcl9/win32-x86_64/tcludp1.0.13") with explicit per-vfs target lists - a shape inherited from G-037 (achieved 2026-07-08), which established "which vfs folders participate is explicitly declared per kit" back when there was only one target. So "bake the same custom .vfs here and on FreeBSD" is not expressible; you duplicate the folder and re-copy the pure-Tcl payload. Two mitigating facts: _vfscommon.vfs (merged into every kit) carries no platform binaries, so the common payload is already target-agnostic - the problem is confined to the kit-specific vfs. And leakage runs both ways: punk9linux.vfs currently contains tcllib2.0/md5c/win32-x86_64/md5c.dll and tcllib2.0/tcllibc/win32-x86_64/tcllibc.dll, harmless (the loader picks by platform dir) but dead weight from a wholesale tree copy.

  3. Nothing verifies payload against target. The cross-target punkshell902 bake produced a valid ELF, but the build would not have complained had that vfs been full of DLLs.

The runtime side is already in place for a FreeBSD experiment: bin/runtime/freebsd-x86_64/ exists locally (empty but for sha1sums.txt) and punkbin upstream carries freebsd-x86_64/tclkit-851-freebsd7-x86_64, fetchable via punk-runtime.cmd fetch -platform freebsd-x86_64.

Approach

  • Adopt the developer's recorded split, with the directory named explicitly: native-target kits stay at bin/<name> (every launcher, test and habit expects bin/punk91.exe there), non-native kits go to bin/kits/<platform>/<name>. bin/kits/ rather than bin/<platform>/ because bin/runtime/<platform>/ already means build INPUTS - having outputs in a sibling shape one level up would read as the same thing. No ignore-rule work: /bin/* already covers bin/kits/ (verified), so neither .gitignore nor the fossil ignore-glob needs a change.
  • Supply the PLATFORM AXIS to whatever payload-declaration mechanism exists at the time (today vendorlib_vfs.toml's explicit per-kit lists; G-115's toml if it lands first). This goal does not own the declaration FORMAT - it owns the requirement that a payload can be declared per target and selected by the kit's target.
  • Keep the mismatch check load-path-relevant: report a binary that would actually be LOADED on the target, and stay silent on per-platform subdirectory layouts the loader resolves correctly. A check that fires on every existing tree is noise, and the current trees have benign leakage.

Alternatives considered

  • Put ALL kits under bin/kits/<platform>/, including native - rejected: breaks every launcher, test, doc and habit that expects bin/punk91.exe, for uniformity's sake.
  • Keep one .vfs folder per target and fix only the output collision - rejected: leaves in place exactly the duplication that motivates the goal.
  • Make a payload/target mismatch a hard build failure - rejected as the DEFAULT: the current tree would fail immediately on known-benign leakage. Offer it as a strict flag.
  • Disambiguate colliding names by runtime rather than by platform (extending today's <appname>_<runtimename> behaviour) - rejected: it encodes the wrong axis, and it does nothing for the cross-run overwrite, which is the sharper half of the defect.

Notes

  • Depends on: G-122 (achieved) - the input half; this is the output half of the same split, and retires the TODO that goal left standing.
  • Related: G-125 (achieved 2026-07-27 - see goals/archive/G-125-unbootable-kit-deploy-gate.md; pointer pushed at its archive sweep, where the pair were each other's only recorded bridge). It added a boot-precondition gate that refuses a kit BEFORE any build product is written - immediately after the vfs merges, before the kit-type assembly switch. Two consequences for this goal: whatever output location it settles on for non-native kits (bin/kits//) inherits the never-written guarantee for free as long as the gate stays upstream of the artifact write, and the payload/target mismatch reporting this goal adds should read as one vocabulary with the gate's refusal rather than a second, differently-worded failure surface. The gate is deliberately non-executing precisely so it covers cross-target kits this host could never run.
  • Related: G-115 (achieved 2026-07-31 - see goals/archive/G-115-declarative-vfs-composition.md) - owned the payload DECLARATION FORMAT, which is now live as per-.vfs files (src/vfs/.vfs.toml, [payload.*] entries with source/source_root/target/supersedes/replace) deliberately WITHOUT a platform axis. This goal lands second and adopts that format: it supplies the axis (per-target payload entries in those files), the format supplies the surface.
  • Related: G-023 - owns output NAMES and its Acceptance hardcodes punk9-0.5.0.exe. This goal owns output LOCATION and the suffix; the two must compose (bin/kits/<platform>/punk9-0.5.0) rather than compete.
  • Related: G-114 - the same platform axis one layer down (per-platform tm module roots registered by the boot for the running platform). A cross-target kit's tm roots must be the TARGET's. Note the xref shows G-114 and G-115 unlinked despite overlapping: the platform axis and the payload axis have never been connected, and this goal is that connection.
  • Related: G-024 (achieved 2026-07-31 - see goals/archive/G-024-mapvfs-toml.md) - the mapvfs.toml format home for expressing one vfs definition against several targets is live (per-kit [kit.] tables each carrying their own target already allow one vfs under several targets; the output-collision half stays this goal's).
  • Related: G-057 (achieved 2026-07-29, archived - see goals/archive/G-057-kit-icon-embedding.md) - icon embedding is a windows-target-only wrap step keyed off a per-vfs override, so "which kits get an icon" becomes a target question once one vfs definition serves several targets. Settled there 2026-07-26 and implemented so: the override is shared across a kit definition's targets and carries no target dimension
    • used where embedding applies, ignored elsewhere. Every target's kit gets a .resources.toml sidecar either way (a linux-target kit records not-applicable; a win32 kit cross-baked from a non-windows host records unavailable with the G-128 remedy named (G-128 achieved 2026-07-29 - see goals/archive/G-128-portable-pe-resource-stamping.md; cross-host kits now EMBED via the punkres arm when bin/punkres is present, demonstrated by the linux cross-bake in its Progress) - both proven live 2026-07-29, WSL cross-bake included), so this goal's per-target outputs inherit working record semantics.
  • Related: G-105 - produces the cross-target RUNTIMES this consumes for custom kits.
  • Related: G-005 / G-006 - where a target's binary payload comes from: built from source or downloaded with consent.
  • Related: G-116 / G-020 - concrete per-platform binary payloads inside kit vfs lib trees (suite-built tcltls; the screen-capture backends under punk9wintk903.vfs/lib_tcl9).
  • Related: G-101 - 8.6 containers ride the same mapping and output surfaces.
  • Related: G-037 (achieved 2026-07-08) - established the per-kit declared vfslibs propagation whose single-target assumption this goal generalizes; see goals/archive/G-037-vendorlib-vfs-propagation.md.
  • Tension to hold in view: G-004 (no committed binaries) - per-target payloads multiply the binary count in the tree unless they arrive by build or fetch.
  • G-133 (archived) landed second-guessing-free target addressing for its checks (arch scan keyed by the entry's rt_target; smoke probe gated on host process-family runnability with a stated-reason skip) - per its cross-reference this goal, landing second, adopts that target-addressing; its output-location move must also keep the smoke probe pointed at wherever the freshly built artifact lands - see goals/archive/G-133-bake-payload-consistency-checks.md.
  • Acceptance redundancy (recorded 2026-07-28, user-directed): the payload/target mismatch clause in this goal's Acceptance - a payload binary built for another platform reported naming the file and both platforms, with platform-discriminated subdirs explicitly not flagged - is ALREADY DELIVERED by G-133 (achieved 2026-07-27). Its acceptance requires a recapped BUILD-WARNING naming file, found architecture and kit target; no warning for multi-arch platform-subdir packages or recognised vendor spellings; and the scan running for cross-target kits. The clause is deliberately left in place rather than trimmed (that would be a contract edit): read it as a regression pin on behaviour that already exists, not as work to do. The genuine remainder of this goal is the other two gaps in Context - the bin/kits// output split and per-target payload declaration - and, per the note above, keeping G-133's smoke probe pointed at wherever the artifact now lands.
  • Related: G-131 - sibling surface in the same kit loop (its kit_type_mismatch rework and container autodetection read the same mapvfs/runtime_caps machinery whose outputs this goal relocates); their previously recorded bridge was G-133, now archived.
  • Related: G-123 (achieved 2026-07-31 - see goals/archive/G-123-thirdparty-runtime-tiers.md) - a published third-party runtime tier is a consumer of the cross-target artifacts this goal gives a home to; recorded directly now that the shared G-122/G-133 references are both archive-tier.
  • G-123 (archived) recorded a finding this goal's make.tcl surface should note: its end-to-end demonstration (fixture-tier runtime -> non-default-target mapvfs entry -> bakelist row runtime-present) had to be LIVE-REVERSIBLE because make.tcl had no alternate-mapvfs-config seam for characterization tests. RETIRED by G-024 (achieved 2026-07-31): the PUNK_MAPVFS_CONFIG env var now points bakelist/bake at an alternate mapping file (used by the G-024 fixture tests in maketclbakelist.test) - this goal's cross-target characterizations can pin behaviour without editing the real config.
  • Related: G-060 (recorded at activation, 2026-07-31) - the runnability split, settled as part of the activation decision: every Acceptance clause here is host-verifiable (artifact locations and cross-run coexistence are filesystem facts, payload selection is verified by inspecting kit contents, the G-125 gate is non-executing by design, and G-133's smoke probe skips non-runnable targets with a stated reason), so acceptance does not depend on EXECUTING a non-native kit on its platform - which this host cannot do for the FreeBSD case. On-target execution of cross-target kits belongs to G-060's guest matrix (its Alternatives survey, same date, records the VMM weighing: crosvm rejected, Firecracker review candidate, QEMU lean stands). Beyond contract, linux-x86_64 kits can get an opportunistic WSL execution smoke (the punkshell902 pattern from G-140, achieved 2026-07-30 - see goals/archive/G-140-linux-lib-tier-emission.md).
  • Related: G-141 - libfetch consumption-derived target selection: the per-target payload declarations this goal adds to src/vfs/*.vfs.toml are that goal's derivation input, and its Acceptance's bake-sufficiency clause leans on them (its Notes record the pairing - whichever lands second adopts the other's surface). Back-pointer added at the activation freshness survey 2026-07-31 (goals_xref score: strongest one-directional pair).
  • Related: G-137 - RT_VERSION stamping rides the same kit-icon wrap seam whose outputs this goal relocates: its stamp + .resources.toml sidecar emission must follow non-native artifacts to bin/kits// (the same per-target wrap-seam composition question the G-057 icon note above records; bridge recorded from its side 2026-07-31, back-pointer added at the activation freshness survey).

Progress

  • 2026-07-31 increment 1 (output split, make.tcl): kit outputs are TARGET-keyed. Kits for the host's default target keep the flat locations byte-identically (punk86 rebake's deploy step skipped the install as content-identical to the pre-change deployed kit); any other target's kit builds to src/_build/kits// and deploys to bin/kits// with its own per-tier bin .punkcheck ledger - the merged .vfs image, icon/resources sidecar, arch scan, offset probe, smoke probe, release-gate and self-build guards all follow the per-kit locations. The duplicate-name guard is per (target, name) in BOTH the kit loop and the bakelist enumerator (mapvfs_kit_outputs), so same-named kits for different targets coexist instead of being runtime-renamed or overwritten; records gain out_tier; bakelist cross-target rows carry out=kits// and the per-kit detail block prints the tiered build/deploy paths (maketclplatform.test crosstarget pins updated to the new surface - 5/5 pass; maketclbakelist.test untouched 9/9 pass). The kit-loop cross-platform TODO is retired pointing here. LIVE: 'make.tcl bake punkshell902' relocated the linux kit to bin/kits/linux-x86_64/punkshell902 (valid ELF, sidecar beside it) - the INTENDED RELOCATION per Acceptance. The pre-move flat copies (bin/punkshell902, src/_build/punkshell902 + .vfs image) remain as orphans pending manual cleanup. CONSEQUENCE surfaced by the same rule: the G-129 verification kits (punkluck86, punk91ix86 - target win32-ix86 is not the host default win32-x86_64) will relocate to bin/kits/win32-ix86/ on their next bake; their old flat bin copies become the same class of orphan. (G-129 achieved - see goals/archive/G-129-kitboot-mountpoint-derivation.md; the kits stay active per their mapvfs group description.)
  • 2026-07-31 increment 2 (per-target payload axis, VFSPAYLOAD + kit loop): a payload 'source' containing %platform% is a PER-PLATFORM entry - the platform set is DERIVED from the kit mapping (every mapvfs kit output pairing the vfs, bake_default included; PUNK_MAPVFS_CONFIG honoured via mapvfs_locate), never declared twice. Each platform's copy materializes punkcheck-tracked under _targets/// in the .vfs folder (VCS-ignored staging - rules added to .gitignore + .fossil-settings/ignore-glob); at bake time the kit's merged image gets ONLY its own target's subtree (selection overlay upstream of the arch scan and G-125 gate, staging dropped from the image; missing _targets/ in a staged vfs earns a recapped BUILD-WARNING). supersedes/replace apply within each platform subtree; %platform% in 'target' is rejected (staging prefix is implicit); a %platform% entry on an unconsumed vfs warns and skips. Spec extended in src/vfs/README.md. MIGRATION: the punk9linux.vfs tcllibc entry now reads source = "%platform%/tcl9/tcllibc" (was hardcoded linux-x86_64) and the flat COMMITTED lib_tcl9/tcllibc tree was removed from both VCS (declaration + packages tier govern it; committed-binary reduction in the G-004 direction). Verified live: bake punkshell902 stages _targets/linux-x86_64, selects it into the image (lib_tcl9/tcllibc present, no _targets residue, arch scan 6 scanned/6 match - same composition as the pre-axis bake), deploys to bin/kits/linux-x86_64/; bake punkdeclare (platform-neutral entries) unchanged, smoke probe resolves udp 1.0.13 + tcllibc 2.0 from the freshly built native artifact.
  • 2026-07-31 increment 3 (characterization): new src/tests/shell/testsuites/punkexe/maketclkitlocations.test - fixture kit mappings through the PUNK_MAPVFS_CONFIG seam (nothing built, per the G-125 precedent of keeping heavy verdicts out of this suite): one vfs under two targets resolves to flat vs kits// locations in one report (location = pure function of (name, target) - the construction that makes the cross-run overwrite impossible); two NON-NATIVE targets sharing a kit name coexist as two rows split by tier and target-driven suffix with NO runtime-name rename; a same-name SAME-target duplicate keeps the historical _ disambiguation. The mismatch clause stays a REGRESSION PIN on G-133 coverage (verdicts: modules/punkboot/utils/testsuites/utils/binaryarch.test; advisory contract: maketclpayloadcheck.test) - deliberately not re-implemented. SURFACE FIX flushed out by the characterization: mapvfs_match_outputs stopped at the FIRST record matching a plain kit name (a pre-G-127 uniqueness assumption) - a name now selects ALL its records (one per target), matching the @group branch, so filtered bakelist detail and selective bake cover every target of a shared name. Full punkexe subtree green: 15 files, 101 tests, 97 pass / 4 constraint-skips / 0 fail (88s, native Tcl903 runner).
  • Remaining for acceptance: byte-comparability spot-check for a zip-type native kit; bin/AGENTS.md + src/AGENTS.md output-location docs; G-023 reconciliation statement recorded here; punkshell902 + win32-ix86 relocation records finalized; punk9linux.vfs remaining drop-ins (tcltls/tdom/thread .so) recorded as deliberately hand-curated or migrated.