From 04d2a51b671728707ba3914dd30b439264aba353 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Sun, 26 Jul 2026 16:34:33 +1000 Subject: [PATCH] G-122 achieved: make.tcl bakes by target platform, not host personality (punkshell 0.25.0) Everything a bake EMITS is now keyed by the artifact's TARGET platform rather than the driving tclsh's personality: the bin/runtime/ store a runtime is read from, .exe suffixing of runtime files and kit outputs, presence checks, and whether the pre-deploy process sweep uses tasklist/taskkill or ps/kill. Host semantics - copy commands, path handling, filesystem case rules, prompts - stay keyed to the host. The default target is the host's platform canon EXCEPT for a cygwin-family host (an msys2/cygwin-runtime tclsh, which reports tcl_platform(platform) unix on windows), which targets win32-x86_64. Such a host now drives the identical kit set, names and store addressing as a native tclsh; before, it reported every kit runtime=missing against a nonexistent mingw64-x86_64 store. mapvfs.config entries take an optional 4th element naming the kit's target platform, so a runtime kept in another platform's tier is a first-class mapping entry. The linux tclkit entry now declares linux-x86_64 and bakes to a suffixless bin/punkshell902. Zip-type kits assemble without zipfs in the driving tcl: when tcl::zipfs::mkimg is absent (any 8.6) the image is built by raw-runtime split + punk::zip::mkzip + concatenation - the same helper the zipcat type uses. Both mount identically (archive-start-relative offsets). Fixes exposed by the above: the runtime capability probe looked for a suffixless filename and so reported find-fail for every runtime on windows; with real capabilities arriving, the extraction try-list was found to test capability presence rather than value, which would send a plain zipfs runtime down the kit path where a failed sdx mksplit replaces raw_runtime with the un-split original; BUILDCOPY names now follow the runtime's own filename; runtime map keys strip only .exe (file rootname would eat the last dotted segment of tclsh9.0.5); and native windows command lines run through a guard suppressing msys2's posix-path argument rewriting - without it taskkill /PID reached taskkill as taskkill C://PID , so the sweep could find a running kit but never replace it. punk::platform folds the msys2 family (mingw64/mingw32/ucrt64/clang*) onto one stable 'msys' tag and CYGWIN_NT onto 'cygwin' - msys2's uname renders that token from MSYSTEM, so one binary otherwise canonizes differently per launching shell. New normalize_os; cygwin-x86_64 added to the platform table; 'help platforms' documents the fold. make.tcl check reports the derivation on one line; bakelist rows carry target= for non-default targets and the detail block names the tier; workflow text gains key note [K9]. New characterization suite maketclplatform.test (5 tests, two self-gating on discovering and probing a real msys/cygwin tclsh). Full suite 1112 tests: 1093 passed, 18 skipped, 1 failed = the documented exec-14.3 baseline. Verified: msys2 tclsh8.6 bakelist output identical to native Tcl 9; a changed punk91 baked and deployed from that host with the sweep killing a running instance msys ps cannot see; a real cross-target linux kit baked from windows to a 24MB ELF; a zipfs-less 8.6 bake producing a bootable zip kit. Assisted-by: harness=claude; primary-model=claude-opus-5[1m]; api-location=anthropic.com --- ARCHITECTURE.md | 1 + CHANGELOG.md | 40 + GOALS-archive.md | 4 + GOALS.md | 4 - bin/AGENTS.md | 5 + goals/G-013-raw-mode-default.md | 2 +- goals/G-122-host-target-platform-split.md | 104 --- goals/G-123-thirdparty-runtime-tiers.md | 12 +- .../G-122-host-target-platform-split.md | 224 +++++ punkproject.toml | 2 +- src/AGENTS.md | 4 +- src/make.tcl | 873 +++++++++++++----- src/modules/punk-999999.0a1.0.tm | 9 +- src/modules/punk-buildversion.txt | 3 +- src/modules/punk/platform-999999.0a1.0.tm | 79 +- src/modules/punk/platform-buildversion.txt | 3 +- src/runtime/AGENTS.md | 5 +- src/runtime/mapvfs.config | 22 +- src/tests/shell/AGENTS.md | 1 + .../testsuites/punkexe/maketclplatform.test | 245 +++++ src/vfs/_config/project_main.tcl | 17 +- src/vfs/_config/punk_main.tcl | 17 +- 22 files changed, 1287 insertions(+), 389 deletions(-) delete mode 100644 goals/G-122-host-target-platform-split.md create mode 100644 goals/archive/G-122-host-target-platform-split.md create mode 100644 src/tests/shell/testsuites/punkexe/maketclplatform.test diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 18903239..9be39258 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -81,6 +81,7 @@ Two layers with a deliberate dependency direction (the class never depends on th ## Build, provenance and kits - **`src/make.tcl` is the single build entry.** Key subcommands: `modules`/`libs`/`packages` (repo-root build outputs), `vfscommonupdate` (regenerate `_vfscommon.vfs`), `bake` (kit assembly into `bin/`; optional kit names confine the run to those kits - G-121), `bakelist` (report the configured kit matrix with runtime/vfs presence and deployed state), `bin`, `bakehouse` (packages + bake for a clean checkout; `project`/`vfs` remain as deprecated aliases - G-112 rename, achieved), `bootsupport` (snapshot refresh), `vendorupdate`, `check`, `workflow`, `projectversion`. Dispatch and help dogfood `punk::args` (G-030, achieved) and degrade to plain scan/help when bootsupport `punk::args` is stale (`PUNKBOOT_PLAIN=1` forces the degraded mode). Source: `src/AGENTS.md`. +- **Host vs target platform (G-122).** The kit surfaces separate what the driving tclsh IS from what a bake is FOR. Target-keyed: the `bin/runtime/` store a runtime is read from, `.exe` suffixing of runtimes and kit outputs, presence checks, and the pre-deploy process sweep's tooling (`tasklist`/`taskkill` vs `ps`/`kill`, skipped when the target's processes cannot exist on this host). Host-keyed: copy commands, path handling, filesystem case rules, prompts. The default target is the host canon except for cygwin-family hosts (msys2/cygwin-runtime tclsh - `tcl_platform(platform)` `unix` on windows), which target `win32-x86_64`; a `mapvfs.config` entry may declare its own target and is then addressed in that platform's tier. `make.tcl check` prints the derivation. Zip-type kits assemble without zipfs in the driving tcl (raw-runtime split + `punk::zip::mkzip` + concatenation). Sources: `src/AGENTS.md`, `src/runtime/AGENTS.md`, `src/modules/punk/platform-999999.0a1.0.tm`. - **punkcheck** records every install/delete as events in per-folder `.punkcheck` directories - the basis for skip/copy change detection, superseded-module pruning and provenance. Single OO record lifecycle (G-094, achieved) with atomic saves and advisory event-scoped locking for concurrent writers (G-095, achieved). - **Provenance gates.** Build/promotion commands warn on uncommitted `src/` changes (column-0 `PROVENANCE-WARNING:` token, `-dirty-abort` for strict mode); `vendorupdate` warns for dirty source-project checkouts. - **Buildsuites and the kit family.** `src/buildsuites/suite_tcl90/` builds Tcl/Tk/tcllib from source with a pinned zig toolchain and produces the runtime kit family (plain / punk / bi) plus artifact metadata (G-096-G-117 era: see archived goals G-096, G-098, G-102, G-103, G-107); artifacts publish to the punkbin repo that `bin/punk-runtime.cmd` fetches from. `src/buildsuites/suite_tcl86/` is the 8.6 sibling (G-099 + G-100, both achieved): a Tcl 8.6 windows runtime (static + dynamic shells, `tcl86t.dll`, on-disk lib tree - no zipfs, so no self-contained kit) with thread 2.8 + tclvfs + Tk 8.6 + tklib + tcllib(+tcllibc critcl accelerators) companions, gated against the 8.6 core/thread/tclvfs testsuites with tcllib/tklib on a record tier and an opt-in `test-tk`; punkshell's own runtests is censused on that runtime at parity with a same-day native-8.6 baseline. Suite child shells scrub `TCL__TM_PATH` as well as TCLLIBPATH/TCL_LIBRARY/TK_LIBRARY - without it a census measures the machine's module trees. The 8.6 kit container strategy remains in-flux - see "In-flux areas". diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e8c969c..86c67ee8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,46 @@ The latest `## [X.Y.Z]` header must match the `version` field in `punkproject.to Entries are newest-first; one bullet per notable change. See the root `AGENTS.md` "Project Versioning" section for the bump policy. +## [0.25.0] - 2026-07-26 + +- make.tcl host/target platform split (G-122): everything a bake EMITS is now keyed by the + artifact's TARGET platform rather than the driving tclsh's personality - the + bin/runtime/ store a runtime is read from, .exe suffixing of runtime files and kit + outputs, presence checks, and whether the pre-deploy process sweep uses tasklist/taskkill + or ps/kill. Host semantics (copy commands, path handling, filesystem case rules, prompts) + stay keyed to the host. The default target is the host's platform canon EXCEPT for a + cygwin-family host (msys2/cygwin-runtime tclsh: reports tcl_platform(platform) unix on + windows), which targets win32-x86_64 - such a host now drives the identical kit set, + names and store addressing as a native tclsh, where before it reported every kit + runtime=missing against a nonexistent mingw64-x86_64 store. +- mapvfs.config entries take an optional 4th element naming the kit's target platform, so a + runtime kept in another platform's store tier is a first-class mapping entry: it is read + from that tier, named with that platform's executable convention, and skipped by the + process sweep. The existing linux tclkit entry now declares linux-x86_64 and bakes to a + suffixless bin/punkshell902. +- make.tcl check reports the derivation on one line: `platform (G-122): host=... target=... + store=... exe-suffix=... process-tooling=...`, with the tcl_platform values it came from. +- Zip-type kits assemble without zipfs in the driving tcl: when tcl::zipfs::mkimg is absent + (any 8.6) the image is built by raw-runtime split + punk::zip::mkzip + concatenation - the + same helper the zipcat kit type uses. Both mount identically (archive-start-relative + offsets). Note the runtime's own zip is still EXTRACTED via tcllib's zipfile::decode, + which is not vendored; without it the bake emits a recapped BUILD-WARNING that the kit + will carry no tcl_library. +- Fixes exposed by the above: the runtime capability probe looked for a suffixless filename + and so reported find-fail for every runtime on windows (its results now feed the + extraction-type choice, which additionally now tests each capability's VALUE rather than + its mere presence); BUILDCOPY names follow the runtime's own filename; and native windows + command lines run through a guard that suppresses msys2's posix-path argument rewriting - + without it `taskkill /PID ` reached taskkill as `taskkill C://PID ` and + the pre-deploy sweep could find a running kit but never replace it. +- make.tcl diagnoses being handed a windows-style script path by a cygwin-family (POSIX) + tclsh, which silently resolves it relative to the cwd, and names the path spelling to use. +- punk::platform: the msys2 family (mingw64/mingw32/ucrt64/clang*) folds onto one stable + `msys` tag and CYGWIN_NT onto `cygwin` - msys2's uname renders that token from MSYSTEM, + so one binary otherwise canonizes differently per launching shell. New + punk::platform::normalize_os; cygwin-x86_64 added to the platform table; 'help platforms' + documents the fold. + ## [0.24.1] - 2026-07-26 - make.tcl bakelist/bake: a nonexistent per-platform runtime store folder is now flagged diff --git a/GOALS-archive.md b/GOALS-archive.md index 02c27118..807abd94 100644 --- a/GOALS-archive.md +++ b/GOALS-archive.md @@ -194,3 +194,7 @@ Acceptance: from the pinned zig and fetched core-8-6-branch sources (hermetic cl ### G-100 [achieved 2026-07-26] suite_tcl86 dependency set: Tk 8.6 + tklib, tcllib with critcl accelerators, punkshell runtests on the 8.6 runtime → detail: goals/archive/G-100-suite-tcl86-tk-tcllib.md Scope: src/buildsuites/suite_tcl86/ (recipe extensions, driver steps); src/buildsuites/_build/ (tk/tklib/tcllib/critcl sources and outputs); bin/tools/ (pinned zig as consumed); src/tests/ (punkshell suites exercised against the suite-built 8.6 runtime) Acceptance: from the pinned zig and fetched sources the suite produces a Tk 8.6 the suite tclsh loads (package require Tk with a create/destroy-window verification recorded here) and a matching tklib whose packages resolve (a representative require recorded); a tcllib install via its installer whose accelerators load (package require tcllibc) with the accelerated-vs-pure comparison recorded; punkshell runtests runs against the suite 8.6 runtime and its census is recorded here against a same-day native/kit 8.6 baseline run (same include set), with every difference dispositioned - Tk-dependent suites run unskipped, and any 8.6-runtime-only failures are root-caused or baselined with reasons. + +### G-122 [achieved 2026-07-26] make.tcl host/target platform split: bake by target, not host personality → detail: goals/archive/G-122-host-target-platform-split.md +Scope: src/make.tcl (target-platform derivation; store folder, exe suffixing, kit naming, presence checks and process-sweep tooling keyed by target; platform-canon inline copy; zipfs-less zip-type assembly fallback); src/modules/punk/platform-999999.0a1.0.tm (stable canon tags for cygwin-family hosts, help platforms doc); src/runtime/mapvfs.config (per-entry target platform via minimal compatible extension - format ownership coordinated with G-024); bin/runtime// store tiers (as consumed; fixture tier for tests); src/tests/shell/testsuites/punkexe/ (characterization) +Acceptance: an msys/cygwin-runtime tclsh driving bakelist/bake on windows reports - and for a changed kit, bakes and deploys - the same win32-x86_64 kit set with identical names and store addressing as a native tclsh, with the pre-deploy process sweep using windows tooling for win32 targets regardless of host personality; punk::platform and the make.tcl inline canon map MSYS_NT/MINGW64_NT/CYGWIN_NT/UCRT64-class hosts to stable documented tags (MSYSTEM-variance documented, no windows build number in the tag) and 'help platforms' lists them; a mapvfs entry declaring a non-default target platform lists in bakelist with its own tier's store presence and bakes to a correctly suffixed artifact, verified against a fixture store tier (a real third-party runtime is not required); a zipfs-less driving tcl (8.6) assembles a zip-type kit via runtime split + punk::zip::mkzip + concatenation (archive-start-relative offsets) and the artifact boots on a zipfs-capable runtime; the linuxulator expectation (a linux-personality tclsh on FreeBSD 14/15 addresses the linux-x86_64 store unchanged) is recorded here with verification deferred to the planned linux-kit-on-FreeBSD trial; native-windows behaviour stays characterization-stable (existing punkexe maketcl tests pass unchanged). diff --git a/GOALS.md b/GOALS.md index 03f940c0..d0b9719f 100644 --- a/GOALS.md +++ b/GOALS.md @@ -385,10 +385,6 @@ Detail: goals/G-115-declarative-vfs-composition.md Scope: src/buildsuites/suite_tcl90/ (build_tcltls module + crypto-backend build, sources.config + build.zig.zon source records, kit-family bi payload + metadata extension, test_gate record step); src/vfs kit payloads carrying vendored tcltls binaries (current-state reference + recorded disposition only - removal stays G-004-era work); punkbin (as eventual artifact destination via the G-067 library class); consumers punk::imap4 / punk::netbox (package require tls) as verification context Detail: goals/G-116-suite-built-tcltls.md -### G-122 [proposed] make.tcl host/target platform split: bake by target, not host personality -Scope: src/make.tcl (target-platform derivation; store folder, exe suffixing, kit naming, presence checks and process-sweep tooling keyed by target; platform-canon inline copy; zipfs-less zip-type assembly fallback); src/modules/punk/platform-999999.0a1.0.tm (stable canon tags for cygwin-family hosts, help platforms doc); src/runtime/mapvfs.config (per-entry target platform via minimal compatible extension - format ownership coordinated with G-024); bin/runtime// store tiers (as consumed; fixture tier for tests); src/tests/shell/testsuites/punkexe/ (characterization) -Detail: goals/G-122-host-target-platform-split.md - ### G-123 [proposed] Third-party runtime tiers via artifact servers: per-target publish, list, fetch, materialize Scope: src/scriptapps/punk-runtime.ps1 + src/scriptapps/bin/punk-runtime.bash + punk-runtime_wrap.toml (named-tier operations in both payloads; bin/punk-runtime.cmd via the established re-wrap channel); punkbin artifact repo (external, c:/repo/jn/punkbin - per-tier layout, defaults, sha1sums, schema-v2 provenance sidecars incl. retroactive backfill, layout docs); src/buildsuites/suite_tcl90/ (family_artifacts.tcl schema-v2 emission for suite-built artifacts); alternative-server override + trusted-server set (PUNKBIN_URL pattern; consent surface coordinated with G-006); src/tests/shell/testsuites/binscripts/ (fixture-server characterization); bin/runtime// stores (as materialization targets) Detail: goals/G-123-thirdparty-runtime-tiers.md diff --git a/bin/AGENTS.md b/bin/AGENTS.md index f399dbe9..2bd50c74 100644 --- a/bin/AGENTS.md +++ b/bin/AGENTS.md @@ -101,6 +101,11 @@ The first argument to a built punk shell may be a dash-delimited package mode co `bin/runtime//` (retrieved from the punkbin artifact repo with sha1 verification against its `sha1sums.txt` - both the powershell payload on windows and the bash payload on unix verify; a fetch whose checksum fails leaves only a `.tmp`). +The `` folders are store TIERS keyed by the platform a runtime is FOR, and the +kit build reads them the same way: `src/make.tcl` picks a tier per kit from that kit's +target platform, not from the driving tclsh's own personality (G-122 - see +`src/AGENTS.md` and `src/runtime/AGENTS.md`), so a tier holding runtimes for another +platform is a first-class thing to populate here. Hashing is deliberately dependency-free in both payloads: the powershell payload computes sha1 via .NET (`Get-PunkFileSha1`) rather than `Get-FileHash`, because Get-FileHash is a script-defined function in Windows PowerShell 5.1 resolved through PSModulePath and diff --git a/goals/G-013-raw-mode-default.md b/goals/G-013-raw-mode-default.md index 1b7b598a..d0aacf86 100644 --- a/goals/G-013-raw-mode-default.md +++ b/goals/G-013-raw-mode-default.md @@ -84,4 +84,4 @@ stays (see inventory item 4). the suite runtimes and kit shape; first enable of a session costs ~0.6-0.7s server spawn - an eager warm-up at repl init is noted there as a future nicety if raw becomes the default). See goals/archive/G-106-powershell-consolemode-fallback.md. -- 2026-07-26 field finding (msys/cygwin-runtime tclsh: scoop msys2 tclsh8.6 driving 'src/make.tcl shell' on windows): line mode is largely functional, but RAW mode places the cursor at row 1 after each command - the raw-mode cursor bookkeeping (cursor-position query/screen-state handling) does not yet cater for the msys pty terminal class. Recorded as a raw-mode-default blocker dimension alongside the existing cursor-move issues; the make.tcl host/target platform-split goal (G-122, proposed the same day) is build-tooling only and deliberately excludes this console/repl work. +- 2026-07-26 field finding (msys/cygwin-runtime tclsh: scoop msys2 tclsh8.6 driving 'src/make.tcl shell' on windows): line mode is largely functional, but RAW mode places the cursor at row 1 after each command - the raw-mode cursor bookkeeping (cursor-position query/screen-state handling) does not yet cater for the msys pty terminal class. Recorded as a raw-mode-default blocker dimension alongside the existing cursor-move issues; the make.tcl host/target platform-split goal (G-122, achieved 2026-07-26 - see goals/archive/G-122-host-target-platform-split.md) was build-tooling only and deliberately excluded this console/repl work, so the msys pty raw-mode behaviour remains entirely this goal's. diff --git a/goals/G-122-host-target-platform-split.md b/goals/G-122-host-target-platform-split.md deleted file mode 100644 index 1d958cde..00000000 --- a/goals/G-122-host-target-platform-split.md +++ /dev/null @@ -1,104 +0,0 @@ -# G-122 make.tcl host/target platform split: bake by target, not host personality - -Status: proposed -Scope: src/make.tcl (target-platform derivation; store folder, exe suffixing, kit naming, presence checks and process-sweep tooling keyed by target; platform-canon inline copy; zipfs-less zip-type assembly fallback); src/modules/punk/platform-999999.0a1.0.tm (stable canon tags for cygwin-family hosts, help platforms doc); src/runtime/mapvfs.config (per-entry target platform via minimal compatible extension - format ownership coordinated with G-024); bin/runtime// store tiers (as consumed; fixture tier for tests); src/tests/shell/testsuites/punkexe/ (characterization) -Goal: make.tcl's kit surfaces (bake/bakelist/bin) operate on an explicit target platform rather than the driving tclsh's personality: any capable host (native windows tclsh, msys/cygwin-runtime tclsh, a FreeBSD host running linux binaries) addresses the same per-target runtime store, artifact naming and process handling for the kits it reports/bakes; non-native-personality runtime tiers (e.g third-party mingw64 builds served from punkbin or an alternative artifact server) are representable as first-class mapping entries with their own store tier; and zip-type kit assembly no longer requires zipfs in the driving tcl. -Acceptance: an msys/cygwin-runtime tclsh driving bakelist/bake on windows reports - and for a changed kit, bakes and deploys - the same win32-x86_64 kit set with identical names and store addressing as a native tclsh, with the pre-deploy process sweep using windows tooling for win32 targets regardless of host personality; punk::platform and the make.tcl inline canon map MSYS_NT/MINGW64_NT/CYGWIN_NT/UCRT64-class hosts to stable documented tags (MSYSTEM-variance documented, no windows build number in the tag) and 'help platforms' lists them; a mapvfs entry declaring a non-default target platform lists in bakelist with its own tier's store presence and bakes to a correctly suffixed artifact, verified against a fixture store tier (a real third-party runtime is not required); a zipfs-less driving tcl (8.6) assembles a zip-type kit via runtime split + punk::zip::mkzip + concatenation (archive-start-relative offsets) and the artifact boots on a zipfs-capable runtime; the linuxulator expectation (a linux-personality tclsh on FreeBSD 14/15 addresses the linux-x86_64 store unchanged) is recorded here with verification deferred to the planned linux-kit-on-FreeBSD trial; native-windows behaviour stays characterization-stable (existing punkexe maketcl tests pass unchanged). - -## Context - -Field session 2026-07-26: the scoop msys2 tclsh8.6 driving `src/make.tcl -bakelist` on windows showed every configured kit as runtime=missing (and -deployed absent) from two compounding causes: the store folder derives from the -HOST's platform canon (observed `mingw64-x86_64` under a MINGW64_NT os string - -note the tag follows the MSYSTEM environment, so the same binary can canonize -as msys/ucrt64/mingw64 in different shells), and the platform=unix branch -probes suffixless filenames - an assumption that is false for the entire -cygwin family, whose executables are `.exe` files (the msys tclsh itself is -`tclsh8.6.exe`). The interim self-diagnosis shipped in punkshell 0.24.1 -(bakelist store-folder WARNING with the canon derivation + `(FOLDER MISSING)` -header mark; bake's no-runtimes exit note) tells the user what happened; this -goal makes the tooling actually cater for it. - -The underlying defect is a conflation: make.tcl's scattered -`$::tcl_platform(platform) eq "windows"` tests mix three separable concerns - -host process semantics (cp vs `cmd /c copy`, `ps`/`kill` vs -tasklist/taskkill, path personality), target artifact naming (`.exe` -suffixing, kit/output names), and store addressing (`bin/runtime/`). -An msys host pulls them apart. The process sweep is the sharpest edge: msys -`ps` sees only msys-descendant processes, so a natively-launched kit exe is -invisible to the pre-deploy sweep (taskkill remains exec-able from msys). - -Direction (user, 2026-07-26): even if msys never becomes a zig buildsuite -target, third-party-built runtime tiers (e.g an msys2-packaged tclsh) served -from punkbin or an alternative artifact server must be supportable as kit -runtimes - REPRESENTATION (mapping entry + store tier) belongs here; -publication/fetch is the flagged follow-on goal. - -FreeBSD linuxulator parallel (user): a linux-personality tclsh running on -FreeBSD via the linux compat layer is expected to self-identify as linux and -address `bin/runtime/linux-x86_64` correctly - the planned -linux-kit-on-FreeBSD-14/15 trial doubles as verification that target-keyed -stores follow runtime personality, not host OS identity. - -Zip offsets (user correction, 2026-07-26): tcl zipfs kits mount with -archive-start-relative offsets - cat-style construction (split the original -runtime first, then append a freshly written zip) is valid; `zipfs mkimg`'s -file-relative offset adjustment is one convention, not a mounting -requirement. The existing zipcat kit type is the in-repo proof, and its -machinery already has zipfs-less fallbacks. - -## Approach - -- One `target_platform` derivation: default from the host canon, with - cygwin-family hosts on windows defaulting to target `win32-x86_64`; - per-entry mapping override for non-default tiers (a minimal compatible - extension of the current vfsconfig entry - e.g a fourth element - whose - schema home is G-024's toml conversion; the parsed-model seam (G-121, - achieved - see goals/archive/G-121-bakelist-selective-bake.md) keeps every - consumer format-agnostic while the format moves). -- Key store folder, suffixing, kit/artifact naming, presence checks and the - process-sweep tool selection by TARGET; keep copy commands, path handling - and prompt behaviour host-driven. -- punk::platform canon: explicit MSYS_NT/MINGW64_NT/CYGWIN_NT/UCRT64 mapping - to stable documented tags with the MSYSTEM-variance caveat; the make.tcl - inline canon copy stays in sync per its existing comment contract. -- zip-type fallback for zipfs-less driving tcls: reuse the zipcat machinery - (extract_preamble split + punk::zip::mkzip + concatenation). -- Characterization against a fixture store tier so no real third-party - runtime is required; the msys-hosted acceptance run uses the dev machine's - msys tclsh where present (else recorded env-gated, per punkexe suite - conventions). - -## Alternatives considered - -- Per-site host patches (special-case each windows test) - rejected: the - three-concern conflation just re-accumulates with each new host class. -- Full self-hosted msys platform lineage (msys-personality kits, accelerator - builds, own test baselines) - out of scope: the third-party tier direction - covers the need without a new first-party build lineage. -- Status quo (require a native tclsh to drive make.tcl) - rejected: field - friction (the all-missing bakelist), and upcoming linux/FreeBSD hosts make - host-personality assumptions untenable regardless of msys. - -## Notes - -- Related: G-024 - owns the mapping format; the per-entry target platform is - drafted as a minimal compatible extension and its schema home is that - goal's toml conversion (make.tcl's own comments already pair the platform - column with the toml change). -- Related: G-105 - cross-target zig builds populate first-party non-native - store tiers; this goal is the consuming/addressing side. -- Related: G-114 - same punk::platform canon vocabulary, applied to - per-platform tm module roots. -- Related: G-101 - 8.6 kit containers ride the same mapping/store surfaces; - lister/bake stay kit-type-agnostic (per the archived G-121 decision). -- Related: G-057 - per-entry mapping extension precedent (per-vfs icon - override rides the same entry shape). -- Related: G-023 - kit naming schemes; target-keyed naming must not collide - with its version-named binaries direction. -- Follow-on: G-123 (drafted 2026-07-26, user-approved) - third-party runtime - tier publication/fetch via punkbin or alternative servers, with schema-v2 - provenance labeling and server-trust-keyed consent. -- Out of scope: msys pty raw-mode console/repl behaviour - the cursor-row-1 - field finding of 2026-07-26 is recorded in G-013's Notes. diff --git a/goals/G-123-thirdparty-runtime-tiers.md b/goals/G-123-thirdparty-runtime-tiers.md index 493b4823..b940cbe4 100644 --- a/goals/G-123-thirdparty-runtime-tiers.md +++ b/goals/G-123-thirdparty-runtime-tiers.md @@ -75,7 +75,17 @@ see goals/archive/G-058-static-runtime-packages.md). - Depends on: G-122 (canon tags, target-keyed stores, non-default-target mapping entries - the end-to-end acceptance item consumes its fixture-tier - support). + support). ACHIEVED 2026-07-26 - see goals/archive/G-122-host-target-platform-split.md; + the consuming side is in place, so this goal's end-to-end item is now purely + about getting a served artifact into `bin/runtime//`. What that archived + goal leaves for whoever builds on it: `msys-x86_64` and `cygwin-x86_64` are + distinct canon tags (msys2 and cygwin are different runtime/mount worlds - one + tier does not serve the other, and MSYSTEM-derived names like mingw64/ucrt64 + fold onto `msys`, so a server tier must never be named from MSYSTEM); a + mapvfs entry addresses its tier through a declared target platform (4th + element); and the zipfs-less kit path depends on tcllib's `zipfile::decode`, + which punkshell does not vendor - a served third-party runtime whose consumer + host lacks tcllib cannot have its own zip extracted into a kit. - Related: G-006 - consent-gated binary downloads; the untrusted-server gate here is a concrete consumer (interim documented flag if this lands first). - Related: G-067 - artifact-channel kinship (module tier vs runtime tier); diff --git a/goals/archive/G-122-host-target-platform-split.md b/goals/archive/G-122-host-target-platform-split.md new file mode 100644 index 00000000..a74c348c --- /dev/null +++ b/goals/archive/G-122-host-target-platform-split.md @@ -0,0 +1,224 @@ +# G-122 make.tcl host/target platform split: bake by target, not host personality + +Status: achieved 2026-07-26 +Scope: src/make.tcl (target-platform derivation; store folder, exe suffixing, kit naming, presence checks and process-sweep tooling keyed by target; platform-canon inline copy; zipfs-less zip-type assembly fallback); src/modules/punk/platform-999999.0a1.0.tm (stable canon tags for cygwin-family hosts, help platforms doc); src/runtime/mapvfs.config (per-entry target platform via minimal compatible extension - format ownership coordinated with G-024); bin/runtime// store tiers (as consumed; fixture tier for tests); src/tests/shell/testsuites/punkexe/ (characterization) +Goal: make.tcl's kit surfaces (bake/bakelist/bin) operate on an explicit target platform rather than the driving tclsh's personality: any capable host (native windows tclsh, msys/cygwin-runtime tclsh, a FreeBSD host running linux binaries) addresses the same per-target runtime store, artifact naming and process handling for the kits it reports/bakes; non-native-personality runtime tiers (e.g third-party mingw64 builds served from punkbin or an alternative artifact server) are representable as first-class mapping entries with their own store tier; and zip-type kit assembly no longer requires zipfs in the driving tcl. +Acceptance: an msys/cygwin-runtime tclsh driving bakelist/bake on windows reports - and for a changed kit, bakes and deploys - the same win32-x86_64 kit set with identical names and store addressing as a native tclsh, with the pre-deploy process sweep using windows tooling for win32 targets regardless of host personality; punk::platform and the make.tcl inline canon map MSYS_NT/MINGW64_NT/CYGWIN_NT/UCRT64-class hosts to stable documented tags (MSYSTEM-variance documented, no windows build number in the tag) and 'help platforms' lists them; a mapvfs entry declaring a non-default target platform lists in bakelist with its own tier's store presence and bakes to a correctly suffixed artifact, verified against a fixture store tier (a real third-party runtime is not required); a zipfs-less driving tcl (8.6) assembles a zip-type kit via runtime split + punk::zip::mkzip + concatenation (archive-start-relative offsets) and the artifact boots on a zipfs-capable runtime; the linuxulator expectation (a linux-personality tclsh on FreeBSD 14/15 addresses the linux-x86_64 store unchanged) is recorded here with verification deferred to the planned linux-kit-on-FreeBSD trial; native-windows behaviour stays characterization-stable (existing punkexe maketcl tests pass unchanged). + +## Context + +Field session 2026-07-26: the scoop msys2 tclsh8.6 driving `src/make.tcl +bakelist` on windows showed every configured kit as runtime=missing (and +deployed absent) from two compounding causes: the store folder derives from the +HOST's platform canon (observed `mingw64-x86_64` under a MINGW64_NT os string - +note the tag follows the MSYSTEM environment, so the same binary can canonize +as msys/ucrt64/mingw64 in different shells), and the platform=unix branch +probes suffixless filenames - an assumption that is false for the entire +cygwin family, whose executables are `.exe` files (the msys tclsh itself is +`tclsh8.6.exe`). The interim self-diagnosis shipped in punkshell 0.24.1 +(bakelist store-folder WARNING with the canon derivation + `(FOLDER MISSING)` +header mark; bake's no-runtimes exit note) tells the user what happened; this +goal makes the tooling actually cater for it. + +The underlying defect is a conflation: make.tcl's scattered +`$::tcl_platform(platform) eq "windows"` tests mix three separable concerns - +host process semantics (cp vs `cmd /c copy`, `ps`/`kill` vs +tasklist/taskkill, path personality), target artifact naming (`.exe` +suffixing, kit/output names), and store addressing (`bin/runtime/`). +An msys host pulls them apart. The process sweep is the sharpest edge: msys +`ps` sees only msys-descendant processes, so a natively-launched kit exe is +invisible to the pre-deploy sweep (taskkill remains exec-able from msys). + +Direction (user, 2026-07-26): even if msys never becomes a zig buildsuite +target, third-party-built runtime tiers (e.g an msys2-packaged tclsh) served +from punkbin or an alternative artifact server must be supportable as kit +runtimes - REPRESENTATION (mapping entry + store tier) belongs here; +publication/fetch is the flagged follow-on goal. + +FreeBSD linuxulator parallel (user): a linux-personality tclsh running on +FreeBSD via the linux compat layer is expected to self-identify as linux and +address `bin/runtime/linux-x86_64` correctly - the planned +linux-kit-on-FreeBSD-14/15 trial doubles as verification that target-keyed +stores follow runtime personality, not host OS identity. + +Zip offsets (user correction, 2026-07-26): tcl zipfs kits mount with +archive-start-relative offsets - cat-style construction (split the original +runtime first, then append a freshly written zip) is valid; `zipfs mkimg`'s +file-relative offset adjustment is one convention, not a mounting +requirement. The existing zipcat kit type is the in-repo proof, and its +machinery already has zipfs-less fallbacks. + +## Approach + +- One `target_platform` derivation: default from the host canon, with + cygwin-family hosts on windows defaulting to target `win32-x86_64`; + per-entry mapping override for non-default tiers (a minimal compatible + extension of the current vfsconfig entry - e.g a fourth element - whose + schema home is G-024's toml conversion; the parsed-model seam (G-121, + achieved - see goals/archive/G-121-bakelist-selective-bake.md) keeps every + consumer format-agnostic while the format moves). +- Key store folder, suffixing, kit/artifact naming, presence checks and the + process-sweep tool selection by TARGET; keep copy commands, path handling + and prompt behaviour host-driven. +- punk::platform canon: explicit MSYS_NT/MINGW64_NT/CYGWIN_NT/UCRT64 mapping + to stable documented tags with the MSYSTEM-variance caveat; the make.tcl + inline canon copy stays in sync per its existing comment contract. +- zip-type fallback for zipfs-less driving tcls: reuse the zipcat machinery + (extract_preamble split + punk::zip::mkzip + concatenation). +- Characterization against a fixture store tier so no real third-party + runtime is required; the msys-hosted acceptance run uses the dev machine's + msys tclsh where present (else recorded env-gated, per punkexe suite + conventions). + +## Alternatives considered + +- Per-site host patches (special-case each windows test) - rejected: the + three-concern conflation just re-accumulates with each new host class. +- Full self-hosted msys platform lineage (msys-personality kits, accelerator + builds, own test baselines) - out of scope: the third-party tier direction + covers the need without a new first-party build lineage. +- Status quo (require a native tclsh to drive make.tcl) - rejected: field + friction (the all-missing bakelist), and upcoming linux/FreeBSD hosts make + host-personality assumptions untenable regardless of msys. + +## Notes + +- Related: G-024 - owns the mapping format; the per-entry target platform is + drafted as a minimal compatible extension and its schema home is that + goal's toml conversion (make.tcl's own comments already pair the platform + column with the toml change). +- Related: G-105 - cross-target zig builds populate first-party non-native + store tiers; this goal is the consuming/addressing side. +- Related: G-114 - same punk::platform canon vocabulary, applied to + per-platform tm module roots. +- Related: G-101 - 8.6 kit containers ride the same mapping/store surfaces; + lister/bake stay kit-type-agnostic (per the archived G-121 decision). +- Related: G-057 - per-entry mapping extension precedent (per-vfs icon + override rides the same entry shape). +- Related: G-023 - kit naming schemes; target-keyed naming must not collide + with its version-named binaries direction. +- Follow-on: G-123 (drafted 2026-07-26, user-approved) - third-party runtime + tier publication/fetch via punkbin or alternative servers, with schema-v2 + provenance labeling and server-trust-keyed consent. +- Related: G-028 - the pre-deploy process sweep is the shared surface: this + goal picks the sweep TOOLING by target (windows tooling for win32 targets + from any host personality), that goal names the process actually holding a + target the build cannot replace. A locker-report helper must make the same + host-vs-target tooling choice. (activation-freshness survey, 2026-07-26) +- Related: G-060 - the eventual vehicle for exercising non-native target + output: this goal makes cross-target kits addressable/bakeable from one + host, a guest matrix is what verifies the artifacts actually boot (and is + the natural home for the deferred linuxulator trial below). + (activation-freshness survey, 2026-07-26) +- Out of scope: msys pty raw-mode console/repl behaviour - the cursor-row-1 + field finding of 2026-07-26 is recorded in G-013's Notes. +- FreeBSD linuxulator expectation (recorded per the acceptance; verification + deferred to the planned linux-kit-on-FreeBSD-14/15 trial, G-060 being the + natural harness). A linux-personality tclsh running on FreeBSD via the linux + compat layer self-identifies as linux, so `platform_punk` yields + `linux-x86_64`, `platform_target_default` returns it unchanged (the + cygwin-family exception is keyed on the os token being msys/cygwin, which a + linuxulator tclsh never reports), and the bake addresses + `bin/runtime/linux-x86_64` with suffixless artifact names and posix + (`ps`/`kill`) process tooling. Target-keyed stores follow RUNTIME + personality, not host OS identity: that is the same rule that makes an msys + host bake win32 kits, applied in the other direction. What the trial must + actually check is that `tcl_platform(os)` reports Linux (not FreeBSD) under + the compat layer - if it reported FreeBSD, the store addressed would be + `bin/runtime/freebsd-x86_64` and the expectation would need revisiting. +- Finding (2026-07-26, kept for whoever meets it next): a cygwin-family tclsh + is a POSIX Tcl - `C:/repo/.../src/make.tcl` is not an absolute path to it, so + `file normalize` prefixes the cwd and the run dies in an unrecognisable + cascade (missing bootsupport, then no project root). make.tcl now diagnoses + exactly that shape and prints the `/c/repo/...` spelling to use. This is + caller-side spelling, deliberately not rewritten by make.tcl. +- Finding (2026-07-26): the zipfs-less EXTRACTION half depends on tcllib's + `zipfile::decode`, a SYSTEM package that punkshell does not vendor. msys2's + `/usr/bin/tclsh8.6` has no tcllib, so a bake there cannot lift the runtime's + own zip (its `tcl_library`) into the kit and the artifact will not boot; the + bake now emits a recapped BUILD-WARNING saying so, and still deploys. With a + tcllib on the path the same msys tclsh splits and decodes byte-identically to + a native 8.6 host (verified: baseoffset 5267968, 1429659 zip bytes, 1784 + entries from both). Candidate follow-ons flagged to the user rather than + written as goals: (a) vendor or write a pure-Tcl zip decoder so the + zipfs-less path is self-contained, (b) refuse to DEPLOY a kit whose + extraction produced no tcl_library instead of warning past it. +- Finding (2026-07-26): the `bin/punkshell902` name is now producible on + windows (cross-target bake), and several punkexe suites use it as the + second-choice `PUNK_SHELL_TEST_EXE` candidate after `bin/punk902z.exe`. On a + windows machine with no windows kit but a cross-baked linux one, that + fallback would select an ELF binary. Left as-is (pre-existing candidate-list + logic, narrow state); `PUNK_SHELL_TEST_EXE` overrides it. + +## Progress + +Achieved 2026-07-26 in one work unit. What landed and how each acceptance +clause was verified: + +- **Target-platform derivation** (`::punkboot::lib::platform_*` in + `src/make.tcl`): `platform_target_default` (cygwin-family host -> win32), + `platform_store_tier` (macosx-* -> macosx), `platform_exe_suffix`, + `platform_process_family`, `platform_name_problem`, plus + `platform_normalize_os` mirroring the module. Globals + `::punkboot::host_platform` / `target_platform` / `host_windows` separate the + two axes; `this_platform_generic` stays the HOST key for bootsupport and + vendorlib platform dirs. +- **Mapping model keyed by target**: `mapvfs_parse` takes the store ROOT plus a + default target and resolves each entry's own tier/suffix (4th element; + conflicting or malformed targets are config errors); `mapvfs_kit_outputs` + records carry `target`/`store_tier`/`runtime_dir`. The bake path parses the + mapping BEFORE runtime discovery, since the mapping is what says which tiers + to look in; cross-target runtimes are picked up only when named. +- **msys-host acceptance** (msys2 `/usr/bin/tclsh8.6`, canon `msys-x86_64`, + `tcl_platform(platform)` unix): `bakelist` output is identical to the native + Tcl 9.0.3 run (line endings aside) - same 10 kits, same names, same + `runtimes: bin/runtime/win32-x86_64` addressing, `punk91` detail resolving + `bin/runtime/win32-x86_64/tclsfe-x64.exe (present)`, `target: win32-x86_64 + (host default)`. Before the split every row read `runtime=missing` against a + nonexistent `mingw64-x86_64` store. A changed `punk91` then baked and + deployed from that host with a hidden `punk91.exe` (pid 10120) running: the + sweep chose tasklist, FOUND it - msys `ps` reports 0 matches for the same + process, measured side by side - killed it with taskkill and replaced + `bin/punk91.exe`. +- **msys2 argument mangling** (found by that run): msys2 rewrites arguments + that look like absolute posix paths when spawning a native windows program, + so `taskkill /PID 10120` arrived as `taskkill C://PID 10120` and + the kill failed the kit. Native windows command lines now go through + `::punkboot::exec_nativeargs` (`MSYS2_ARG_CONV_EXCL=*`, inert on other + hosts); the re-run killed and deployed cleanly. +- **Cross-target entry**: the linux tclkit entry declares `linux-x86_64` and is + read from `bin/runtime/linux-x86_64/` with no `.exe`. `bakelist` lists it with + a `target=linux-x86_64` note and `(present)` store state; `make.tcl bake + punkshell902` from a windows host produced a 24,200,032-byte ELF + `bin/punkshell902` (sdx wrap over the linux runtime), with the process sweep + correctly skipped as not-visible-from-this-host. A real cross-target runtime + was used - the acceptance's fixture-tier allowance was not needed. +- **Zipfs-less zip assembly**: native Tcl 8.6.17 (no zipfs) baked `punk91` via + raw-runtime split + `punk::zip::mkzip` + concatenation (5,267,968 + 44,242,534 + = 49,510,502 bytes) and the artifact BOOTS - `//zipfs:/app/tcl_library` + mounted, 4 zipfs mounts, punk::lib 0.5.1 loaded. The `zipcat` kit type now + shares the same `::punkboot::assemble_zipcat_image` helper. +- **Latent defects the split exposed and fixed**: the runtime capability probe + tested a suffixless filename and so reported `find-fail` for every runtime on + windows (it now probes the target's own tier and suffix, and skips + cross-target runtimes as unrunnable); with real capabilities arriving, the + extraction try-list was found to test capability PRESENCE rather than value, + which would send a plain zipfs runtime down the kit path where a failed `sdx + mksplit` replaces `raw_runtime` with the un-split original - a silently + unbootable kit; BUILDCOPY naming now follows the runtime's own filename; and + runtime map keys strip only `.exe` (`file rootname` would eat the last dotted + segment of `tclsh9.0.5`). +- **Characterization**: new `src/tests/shell/testsuites/punkexe/maketclplatform.test` + (5 tests: the `check` derivation line, default-target detail, the cross-target + entry's tier/target/suffixless naming, and two cygwin-host tests that + self-gate on discovering and probing a real msys/cygwin tclsh - the + derivation line and full bakelist row-set parity against the native run). + Native behaviour stayed characterization-stable: full suite 1112 tests, + 1093 passed, 18 skipped, 1 failed = the documented `exec-14.3` baseline; the + pre-existing `maketclbakelist`/`maketclcolour` tests pass unchanged. +- **Reported surface**: `make.tcl check` prints + `platform (G-122): host=... target=... store=... exe-suffix=... process-tooling=...`; + `bakelist` rows carry `target=` for non-default targets and the + detail block names the tier and whether the target is the host default or + declared; the workflow text gained key note [K9] and a target-keyed store + line. punkshell 0.24.1 -> 0.25.0. diff --git a/punkproject.toml b/punkproject.toml index 9432f0ac..33704cbd 100644 --- a/punkproject.toml +++ b/punkproject.toml @@ -1,4 +1,4 @@ [project] name = "punkshell" -version = "0.24.1" +version = "0.25.0" license = "BSD-2-Clause" diff --git a/src/AGENTS.md b/src/AGENTS.md index c5847988..2c922584 100644 --- a/src/AGENTS.md +++ b/src/AGENTS.md @@ -72,7 +72,9 @@ Recovery after a wrong path guess: - make.tcl subcommands and flags are punk::args-declared (G-030): `tclsh src/make.tcl help ?subcommand?` or ` -help` shows tabled usage; invalid arguments produce punk::args usage errors. The dispatch degrades to plain scan/help when the bootsupport punk::args (or its rendering stack) is stale or unavailable; `PUNKBOOT_PLAIN=1` forces the degraded mode for troubleshooting. The `::punkboot::argdoc` definitions are constructed (string-built) and use the G-045 authoring mechanisms — block-form help values + `@normalize`, explicit `@form -synopsis` — see src/modules/AGENTS.md "punk::args definition authoring ergonomics" before editing them or authoring similar definitions. - Uncommitted `make.tcl`-generated outputs may be batched into one commit, regardless of how many `make.tcl` invocations produced them. This covers punkcheck-managed build outputs that are VCS-tracked: `src/bootsupport/`, `_vfscommon.vfs/modules` + `/lib`, declared per-kit `*.vfs/lib_tcl/` subfolders, the thin-layout sync copies (`src/project_layouts/vendor/punk//src/{make.tcl,build.tcl}` + bootsupport manifest copies), and the module-shipped layout payload under `src/modules/punk/mix/#modpod-templates-*/templates/project_layouts/` (G-087: synced from `src/project_layouts` - never hand-edit the modpod copy). (Root `modules/`, `lib/`, `modules_tcl/`, `lib_tcl/` are gitignored and never committed.) Each module's old→new version rename is atomic; a regeneration reflects one build-output refresh, not independent hand-edits. This is a carve-out from generic "split aggressively" commit guidance for punkcheck-managed build outputs only — it does not cover user-curated VFS content (`_config/`, per-kit `*.vfs/` outside declared `lib_tcl` subfolders, `_aside/`, `mkzipfix.vfs`, `_vfscommon.vfs/doc`), which commits separately per its own concerns. - Kit bakes consume the punk-runtime WORKING COPIES under `bin/runtime//`: `make.tcl bake`/`bakehouse` emit a `BUILD-WARNING:` (recapped at end of run) when a wrapped runtime's beside-toml revision is older than an `-r` artifact present in the same folder - the forgot-to-switch-back guard for deliberate `punk-runtime use ` testing excursions. Heed it before trusting freshly baked family kits; `bin/punk-runtime.cmd use >` (`.exe` suffix optional) re-materializes the working copy. -- `tclsh src/make.tcl bakelist ?kitname ...?` (G-121) reports the kit outputs configured in `src/runtime/mapvfs.config`: kit name, kit type, runtime (with presence in the runtime store), vfs folder, and the deployed state of `bin/` vs the `src/_build` build product (`current|stale|absent|nobuild`), with anomalies (`runtime=missing`, `vfs=missing`, `rtrev=r` materialization staleness) in a trailing notes column. A nonexistent per-platform store folder is flagged loudly (header `(FOLDER MISSING)` + stderr canon derivation; bake's no-runtimes exit self-diagnoses the same way) - the classic trigger is an msys/cygwin-runtime tclsh whose platform canon (e.g `mingw64-x86_64`) is not the windows store the kits use. Name arguments filter the report and add a per-kit detail block. `make.tcl bake ?kitname ...?` bakes and deploys only the named kits - other kits' `_build`/`bin` artifacts and punkcheck records are untouched and the vfslibs phase narrows to the named kits' vfs folders; an unknown name errors before any build, listing the configured names; flags go before kit names (`make.tcl bake -confirm 0 punk91`). Bare `bake` processes all configured kits as before. Both surfaces consume the shared parsed-mapping helpers (`::punkboot::lib::mapvfs_*`) rather than the file format, so the G-024 config-format work swaps the reader underneath. Piped characterization: `src/tests/shell/testsuites/punkexe/maketclbakelist.test`. +- `tclsh src/make.tcl bakelist ?kitname ...?` (G-121) reports the kit outputs configured in `src/runtime/mapvfs.config`: kit name, kit type, runtime (with presence in the runtime store), vfs folder, and the deployed state of `bin/` vs the `src/_build` build product (`current|stale|absent|nobuild`), with anomalies (`runtime=missing`, `vfs=missing`, `rtrev=r` materialization staleness) and the cross-target marker `target=` in a trailing notes column. A nonexistent store tier is flagged loudly per tier actually referenced (header `(FOLDER MISSING)` for the default tier + stderr derivation; bake's no-runtimes exit self-diagnoses the same way). Name arguments filter the report and add a per-kit detail block (resolved store tier, target and provenance of the target, paths/sizes/mtimes). `make.tcl bake ?kitname ...?` bakes and deploys only the named kits - other kits' `_build`/`bin` artifacts and punkcheck records are untouched and the vfslibs phase narrows to the named kits' vfs folders; an unknown name errors before any build, listing the configured names; flags go before kit names (`make.tcl bake -confirm 0 punk91`). Bare `bake` processes all configured kits as before. Both surfaces consume the shared parsed-mapping helpers (`::punkboot::lib::mapvfs_*`) rather than the file format, so the G-024 config-format work swaps the reader underneath. Piped characterization: `src/tests/shell/testsuites/punkexe/maketclbakelist.test`. +- **Host vs target platform (G-122).** Everything a bake EMITS is keyed by the artifact's TARGET platform, never by the driving tclsh's personality: the `bin/runtime/` store the runtime is read from, `.exe` suffixing of runtime files and kit outputs, the presence checks behind `runtime=missing`, and whether the pre-deploy process sweep uses `tasklist`/`taskkill` or `ps`/`kill`. Host semantics - copy commands, path handling, filesystem case rules, prompts - stay keyed to the host. The default target is the host's own platform canon, EXCEPT a cygwin-family host (an msys2/cygwin-runtime tclsh, which reports `tcl_platform(platform)` `unix` on windows and canonizes as `msys-x86_64`/`cygwin-x86_64`), which targets `win32-x86_64`: such a host now drives the identical kit set, names and store addressing as a native tclsh. `tclsh src/make.tcl check` prints the derivation on one `platform (G-122): host=... target=... store=... exe-suffix=... process-tooling=...` line. A mapvfs entry may declare its own target platform (4th element - see `src/runtime/AGENTS.md`), which is then read from that platform's tier, named with that platform's executable convention, and skipped by the process sweep (its processes are not visible to this host). Traps this closes, all field-verified 2026-07-26: msys `ps` cannot see a natively-launched kit (only `tasklist` can), and msys2 rewrites arguments that look like absolute posix paths when spawning a native windows program - so `taskkill /PID ` arrived as `taskkill C://PID `; native-windows command lines therefore go through `::punkboot::exec_nativeargs` (sets `MSYS2_ARG_CONV_EXCL=*`, inert elsewhere). Also note a cygwin-family tclsh is a POSIX Tcl: name the script the way that shell spells paths (`tclsh /c/repo/.../src/make.tcl`, not `C:/repo/...`, which it resolves relative to the cwd) - make.tcl diagnoses the windows-spelling case. Piped characterization: `src/tests/shell/testsuites/punkexe/maketclplatform.test` (the cygwin-host tests self-gate on finding and probing a real msys/cygwin tclsh; `PUNK_MSYS_TCLSH` names one explicitly). +- Zip-type kit assembly does not require zipfs in the driving tcl (G-122): with `tcl::zipfs::mkimg` present it is used as before, otherwise the image is assembled by concatenation (raw-runtime split + `punk::zip::mkzip` + append, the same `::punkboot::assemble_zipcat_image` helper the `zipcat` kit type uses). Both mount identically - tcl zipfs reads the archive with archive-start-relative offsets. Caveat for zipfs-less hosts: EXTRACTING the runtime's own attached zip (needed to carry its `tcl_library` into the kit) uses tcllib's `zipfile::decode`, which is a system package and is NOT vendored - a tclsh without tcllib (e.g msys2's `/usr/bin/tclsh8.6`) cannot take that path and the bake emits a recapped `BUILD-WARNING` that the kit will get no `tcl_library`. Heed that warning: the build still produces and deploys an artifact. - Use `punk make.tcl bakehouse` or `punk902z make.tcl bakehouse` inside Punk shell when building binaries through Punk (the deprecated `project` alias still maps). Driving make.tcl from a built punk executable is supported for informational/update subcommands and for kit builds of *other* kits — the kit whose deployed executable is running the build is skipped with a warning (it cannot be replaced while running, and the pre-deploy process sweep must not kill the build itself; the sweep also excludes the build's own pid in all cases). Rebuild that kit from tclsh or a different kit. - **Punk-exe-hosted make.tcl runs in a pre-loaded interp, not a virgin one** — the kit's script-mode boot has already loaded much of the punk stack (punk, punk::lib, punk::repl, punk::console, punk::du, flagfilter, struct::set ...) and set process state (app-punkscript forces `::tcl_interactive 0` for script semantics; libunknown/packagepreference are active). Consequences make.tcl must (and now does) handle explicitly: `package require` of an already-provided package is a no-op, so anything make.tcl breaks in the interp (the accelerator-reload block forgets+destroys sha1/md5/struct::* — it re-requires what was loaded), and anything computed at package-load time (punk::repl's `::tcl_interactive` probe — the shell branch recomputes it before `repl::start`), must be restored deliberately rather than relying on later loads to re-fire. Also note the copies that run in this mode are the *kit's* pre-loaded modules, not the bootsupport snapshots make.tcl's paths would otherwise prefer — silent provenance mixing when versions diverge. When adding interp-surgery or load-time-state assumptions to make.tcl, test under both `tclsh src/make.tcl ...` and ` src/make.tcl ...`. - Binary images are platform-specific; build on each target platform rather than expecting a cross-platform flag. diff --git a/src/make.tcl b/src/make.tcl index 88652cfd..7d08906c 100644 --- a/src/make.tcl +++ b/src/make.tcl @@ -444,16 +444,29 @@ namespace eval ::punkboot::lib { return "${plat}-${cpu}" } + proc platform_normalize_os {os} { + #INLINE COPY of punk::platform::normalize_os - see platform_punk below. + switch -glob -- [string tolower $os] { + macos {return macosx} + msys - msys_nt-* {return msys} + mingw32 - mingw64 - mingw_nt-* - mingw32_nt-* - mingw64_nt-* {return msys} + ucrt64 - ucrt64_nt-* {return msys} + clang32 - clang64 - clangarm64 - clang32_nt-* - clang64_nt-* - clangarm64_nt-* {return msys} + cygwin_nt-* {return cygwin} + } + return $os + } proc platform_punk {} { #canonical punkshell platform-dir name: platform_generic normalized. #INLINE COPY of punk::platform::normalize (src/modules/punk/platform-*.tm; #'help platforms' documents the canon) - the boot stage cannot package #require, so keep this mapping in sync with that module: - #amd64->x86_64, aarch64->arm64, macos->macosx, macosx arm->arm64. + #amd64->x86_64, aarch64->arm64, macos->macosx, macosx arm->arm64, + #msys2 family (mingw64/mingw32/ucrt64/clang*) -> msys, CYGWIN_NT -> cygwin. set parts [split [platform_generic] -] set cpu [lindex $parts end] set os [join [lrange $parts 0 end-1] -] - if {$os eq "macos"} {set os macosx} + set os [platform_normalize_os $os] switch -- $cpu { amd64 {set cpu x86_64} aarch64 {set cpu arm64} @@ -462,6 +475,93 @@ namespace eval ::punkboot::lib { return "${os}-${cpu}" } + # -- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- + # G-122 host/target platform split. + # platform_punk answers "what is THIS tclsh" (the HOST canon). Everything a + # build EMITS is a separate question keyed by TARGET: which bin/runtime/ + # store holds the runtime, whether artifacts carry .exe, and which process + # tooling can see/kill a running artifact. The two diverge for the whole + # cygwin family: an msys2/cygwin-runtime tclsh on windows reports + # tcl_platform(platform) 'unix' and canonizes as msys-x86_64/cygwin-x86_64, + # yet the kits it bakes are ordinary win32-x86_64 .exe binaries that only + # tasklist/taskkill can manage (msys 'ps' sees only msys-descendant + # processes). Host process semantics - copy commands, path handling, prompts, + # filesystem case rules - stay keyed to the host. + # -- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- + + #os token of a canonical platform-dir name (win32-x86_64 -> win32; macosx -> macosx) + proc platform_os {platform} { + set parts [split $platform -] + if {[llength $parts] < 2} { + return $platform + } + return [join [lrange $parts 0 end-1] -] + } + #cpu token of a canonical platform-dir name ("" for single-token names like macosx) + proc platform_cpu {platform} { + set parts [split $platform -] + if {[llength $parts] < 2} { + return "" + } + return [lindex $parts end] + } + #posix-personality runtimes hosted on windows (msys2 / cygwin) + proc platform_is_cygwin_family {platform} { + return [expr {[platform_os $platform] in {msys cygwin}}] + } + #Windows-family platforms: native win32 AND the cygwin-family personalities. + #They share what matters to a build: executables are .exe files and running + #processes are windows processes (tasklist/taskkill), whatever tcl_platform says. + proc platform_is_windows_family {platform} { + return [expr {[platform_os $platform] in {win32 msys cygwin}}] + } + #Default TARGET platform for a host canon. A cygwin-family host builds native + #windows kits (there is no separate msys kit lineage - third-party msys + #runtimes are addressed by an explicit per-entry target instead). + proc platform_target_default {hostplatform} { + if {[platform_is_cygwin_family $hostplatform]} { + set cpu [platform_cpu $hostplatform] + if {$cpu eq ""} {set cpu x86_64} + return "win32-$cpu" + } + return $hostplatform + } + #bin/runtime store tier folder name for a target: macOS runtimes are universal + #(multi-arch) binaries kept in one 'macosx' folder - see punk::platform. + proc platform_store_tier {target} { + if {[string match "macosx-*" $target]} { + return macosx + } + return $target + } + #executable filename suffix artifacts for this target carry + proc platform_exe_suffix {target} { + if {[platform_is_windows_family $target]} { + return ".exe" + } + return "" + } + #which process tooling manages a running artifact of this platform + proc platform_process_family {platform} { + if {[platform_is_windows_family $platform]} { + return windows + } + return posix + } + #A target platform name must look like a platform-dir name so it can address a + #store tier. Returns "" when acceptable, else a reason string. + proc platform_name_problem {platform} { + if {$platform eq ""} { + return "empty" + } + if {[regexp {[\\/:*?\"<>|]} $platform]} { + return "not a platform-dir name (contains a path/wildcard character)" + } + if {[llength [split $platform -]] < 2 && $platform ne "macosx"} { + return "not an - platform-dir name ('help platforms' lists the canon)" + } + return "" + } } #------------------------------------------------------------------------------ @@ -533,27 +633,35 @@ proc ::punkboot::lib::bootsupport_prompt_yesno {prompt} { # directly, so a config-format change (G-024 toml direction) swaps in underneath # by reimplementing mapvfs_parse alone. # -# mapvfs_parse: pure parse - no output, no exits. Returns a dict: +# mapvfs_parse: pure parse - no output, no exits. Every store address, presence +# probe and artifact suffix it computes is keyed by the entry's TARGET platform +# (G-122), never by the driving tclsh's personality: $rtbase is the store ROOT +# (bin/runtime) and each entry addresses $rtbase/. Returns a dict: # exists 0|1 (mapfile present) # mapfile path as supplied -# runtime_vfs_map runtime -> raw vfs_specs as listed (appname/type may be "") -# vfs_runtime_map vfstail -> list of {runtime appname type} (defaults applied; -# only vfs folders that exist on disk - matching the historical -# inline parse the kit loop was built around) +# default_target target platform for entries that declare none +# runtime_vfs_map runtime -> raw vfs_specs as listed (appname/type/target may be "") +# vfs_runtime_map vfstail -> list of {runtime appname type target} (defaults +# applied; only vfs folders that exist on disk - matching the +# historical inline parse the kit loop was built around) +# runtime_target runtime -> resolved target platform (all runtimes, mapped order) # missing names of referenced-but-absent runtimes/vfs folders # warnings warning lines in encounter order, ready to print verbatim # (missing-item lines carry their historical WARNING: prefix) # badentries subset of warnings: malformed entry lines (these have no kit # record to carry them, so bakelist prints them; missing-item # warnings are instead surfaced as per-row state there) -# configerrors fatal config problems (duplicate runtime line; caller decides -# whether to abort - the bake path keeps its historical exit 3) -proc ::punkboot::lib::mapvfs_parse {mapfile rtfolder sourcefolder} { +# configerrors fatal config problems (duplicate runtime line; unusable or +# self-contradicting target platform; caller decides whether to +# abort - the bake path keeps its historical exit 3) +proc ::punkboot::lib::mapvfs_parse {mapfile rtbase sourcefolder default_target} { set model [dict create\ exists 0\ mapfile $mapfile\ + default_target $default_target\ runtime_vfs_map [dict create]\ vfs_runtime_map [dict create]\ + runtime_target [dict create]\ missing [list]\ warnings [list]\ badentries [list]\ @@ -570,6 +678,7 @@ proc ::punkboot::lib::mapvfs_parse {mapfile rtfolder sourcefolder} { set mapdata [string map [list \r\n \n] $mapdata] set runtime_vfs_map [dict create] set vfs_runtime_map [dict create] + set runtime_target [dict create] set missing [list] set warnings [list] set badentries [list] @@ -585,19 +694,45 @@ proc ::punkboot::lib::mapvfs_parse {mapfile rtfolder sourcefolder} { #drop windows .exe suffix so same config can work cross platform - extension will be re-added if necessary later set runtime [string range $runtime 0 end-4] } - if {$runtime ne "-"} { - set runtime_test $runtime - if {"windows" eq $::tcl_platform(platform)} { - set runtime_test $runtime.exe + #Target platform is declared per kit-config entry (4th element) but is a + #property of the RUNTIME: one binary lives in one store tier and has one + #executable convention. Entries on a line may therefore repeat the + #declaration but must not disagree. + set line_target "" + foreach vfsconfig $vfs_specs { + if {[llength $vfsconfig] < 4} { + continue + } + set declared [lindex $vfsconfig 3] + if {$declared eq ""} { + continue } - if {![file exists [file join $rtfolder $runtime_test]]} { - lappend warnings "WARNING: Missing runtime file $rtfolder/$runtime_test (line in mapvfs.config: $ln)" + if {$line_target eq ""} { + set problem [platform_name_problem $declared] + if {$problem ne ""} { + lappend configerrors "CONFIG FILE ERROR. runtime: $runtime declares target platform '$declared' - $problem." + continue + } + set line_target $declared + } elseif {$declared ne $line_target} { + lappend configerrors "CONFIG FILE ERROR. runtime: $runtime declares conflicting target platforms ('$line_target' and '$declared') in $mapfile - a runtime has one target." + } + } + if {$line_target eq ""} { + set line_target $default_target + } + set rtdir [file join $rtbase [platform_store_tier $line_target]] + set exe_suffix [platform_exe_suffix $line_target] + if {$runtime ne "-"} { + set runtime_test $runtime$exe_suffix + if {![file exists [file join $rtdir $runtime_test]]} { + lappend warnings "WARNING: Missing runtime file $rtdir/$runtime_test (line in mapvfs.config: $ln)" lappend missing $runtime } } foreach vfsconfig $vfs_specs { switch -- [llength $vfsconfig] { - 1 - 2 - 3 { + 1 - 2 - 3 - 4 { lassign $vfsconfig vfstail appname target_kit_type if {![file isdirectory [file join $sourcefolder vfs $vfstail]]} { lappend warnings "WARNING: Missing vfs folder [file join $sourcefolder vfs $vfstail] specified in mapvfs.config for runtime $runtime" @@ -606,11 +741,11 @@ proc ::punkboot::lib::mapvfs_parse {mapfile rtfolder sourcefolder} { if {$appname eq ""} { set appname [file rootname $vfstail] } - dict lappend vfs_runtime_map $vfstail [list $runtime $appname $target_kit_type] + dict lappend vfs_runtime_map $vfstail [list $runtime $appname $target_kit_type $line_target] } } default { - set badline "bad entry in mapvfs.config - expected each entry after the runtime name to be of length 1 or length 2. got: $vfsconfig ([llength $vfsconfig])" + set badline "bad entry in mapvfs.config - expected each entry after the runtime name to be a list of 1 to 4 elements {vfsfolder ?kitname? ?kittype? ?targetplatform?}. got: $vfsconfig ([llength $vfsconfig])" lappend warnings $badline lappend badentries $badline } @@ -620,10 +755,12 @@ proc ::punkboot::lib::mapvfs_parse {mapfile rtfolder sourcefolder} { lappend configerrors "CONFIG FILE ERROR. runtime: $runtime was specified more than once in $mapfile." } else { dict set runtime_vfs_map $runtime $vfs_specs + dict set runtime_target $runtime $line_target } } dict set model runtime_vfs_map $runtime_vfs_map dict set model vfs_runtime_map $vfs_runtime_map + dict set model runtime_target $runtime_target dict set model missing $missing dict set model warnings $warnings dict set model badentries $badentries @@ -635,24 +772,30 @@ proc ::punkboot::lib::mapvfs_parse {mapfile rtfolder sourcefolder} { # Replicates the kit loop's iteration + naming (vfs_tails glob order with map # extras appended, per-vfs runtime order from the mapping, appname defaulting to # the vfs rootname, kit_type defaulting to 'kit', '-' runtime -> .kit, -# windows .exe suffix, duplicate appname -> _) so listed names +# TARGET .exe suffix, duplicate appname -> _) so listed names # match what a bake produces. Entries whose vfs folder is missing on disk never # reach the kit loop - they are appended after the buildable set so 'bakelist' # can surface them as broken config. Duplicate-name disambiguation assumes the # config keeps appnames unique (as the kit loop itself effectively does). -# Returns a list of record dicts: +# $rtbase is the store ROOT (bin/runtime): each record resolves its own tier +# under it, so entries targeting different platforms report their own store +# presence (G-122). Returns a list of record dicts: # kitname base output name (selection key, e.g punk91) # targetkit platform artifact name (e.g punk91.exe, or .kit for '-') # kit_type kit|zip|zipcat|cookfs... as configured (defaulted to kit) # runtime runtime name as configured (unsuffixed), or "-" -# runtime_file platform runtime filename ("" for "-") +# runtime_file target runtime filename ("" for "-") # runtime_present 1|0 (1 for "-": no runtime needed) +# runtime_dir store tier folder holding the runtime ("" for "-") +# target target platform (e.g win32-x86_64) +# store_tier bin/runtime tier folder name for the target # vfs vfs folder tail (e.g punk9wintk903.vfs) # vfs_present 1|0 -proc ::punkboot::lib::mapvfs_kit_outputs {model rtfolder sourcefolder} { +proc ::punkboot::lib::mapvfs_kit_outputs {model rtbase sourcefolder} { set runtime_vfs_map [dict get $model runtime_vfs_map] set vfs_runtime_map [dict get $model vfs_runtime_map] - set on_windows [expr {"windows" eq $::tcl_platform(platform)}] + set runtime_target [dict get $model runtime_target] + set default_target [dict get $model default_target] set vfs_tails [glob -nocomplain -dir $sourcefolder/vfs -types d -tail *.vfs] dict for {vfstail -} $vfs_runtime_map { if {$vfstail ni $vfs_tails} { @@ -662,33 +805,37 @@ proc ::punkboot::lib::mapvfs_kit_outputs {model rtfolder sourcefolder} { set records [list] set exe_names_seen [list] set record_pair {{rtname vfstail appname target_kit_type} { - upvar 1 records records exe_names_seen exe_names_seen rtfolder rtfolder on_windows on_windows sourcefolder sourcefolder + upvar 1 records records exe_names_seen exe_names_seen rtbase rtbase sourcefolder sourcefolder\ + runtime_target runtime_target default_target default_target if {$appname eq ""} { set appname [file rootname $vfstail] } if {$target_kit_type eq ""} { set target_kit_type "kit" } + if {[dict exists $runtime_target $rtname]} { + set target [dict get $runtime_target $rtname] + } else { + set target $default_target + } + set store_tier [::punkboot::lib::platform_store_tier $target] if {$rtname eq "-"} { set targetkit $appname.kit set kitname $appname set runtime_file "" + set runtime_dir "" set runtime_present 1 ;#no runtime needed } else { - if {$on_windows} { - set runtime_file $rtname.exe - set targetkit ${appname}.exe - } else { - set runtime_file $rtname - set targetkit $appname - } + set runtime_dir [file join $rtbase $store_tier] + set runtime_file $rtname[::punkboot::lib::platform_exe_suffix $target] + set targetkit ${appname}[::punkboot::lib::platform_exe_suffix $target] set kitname $appname if {$targetkit in $exe_names_seen} { #duplicate appname configured - kit loop disambiguates the same way set targetkit ${appname}_$rtname set kitname ${appname}_$rtname } - set runtime_present [file exists [file join $rtfolder $runtime_file]] + set runtime_present [file exists [file join $runtime_dir $runtime_file]] } lappend exe_names_seen $targetkit lappend records [dict create\ @@ -698,6 +845,9 @@ proc ::punkboot::lib::mapvfs_kit_outputs {model rtfolder sourcefolder} { runtime $rtname\ runtime_file $runtime_file\ runtime_present $runtime_present\ + runtime_dir $runtime_dir\ + target $target\ + store_tier $store_tier\ vfs $vfstail\ vfs_present [file isdirectory [file join $sourcefolder vfs $vfstail]]\ ] @@ -723,7 +873,7 @@ proc ::punkboot::lib::mapvfs_kit_outputs {model rtfolder sourcefolder} { #entries whose vfs folder is missing (excluded from vfs_runtime_map by the parse) dict for {rtname vfs_specs} $runtime_vfs_map { foreach vfsconfig $vfs_specs { - if {[llength $vfsconfig] < 1 || [llength $vfsconfig] > 3} { + if {[llength $vfsconfig] < 1 || [llength $vfsconfig] > 4} { continue } lassign $vfsconfig vfstail appname target_kit_type @@ -738,11 +888,13 @@ proc ::punkboot::lib::mapvfs_kit_outputs {model rtfolder sourcefolder} { # Match user-supplied kit names against mapvfs_kit_outputs records. # Accepts the base kit name (punk91) or the platform artifact name (punk91.exe, -# name.kit); matching is case-insensitive on windows. Returns a dict: +# name.kit); matching is case-insensitive when the HOST filesystem is (a name +# the user types is a host-side convenience - a cygwin-family host is as +# case-insensitive as a native windows one, whatever it targets). Returns a dict: # selected matched records (kit-output order, deduplicated) # unknown supplied names that matched nothing proc ::punkboot::lib::mapvfs_match_outputs {kit_outputs names} { - set on_windows [expr {"windows" eq $::tcl_platform(platform)}] + set on_windows [expr {[info exists ::punkboot::host_windows] ? $::punkboot::host_windows : ("windows" eq $::tcl_platform(platform))}] set selected_idx [list] set unknown [list] foreach name $names { @@ -918,6 +1070,21 @@ set startdir [pwd] set bootsupport_module_paths [list] set bootsupport_library_paths [list] set this_platform_generic [punkboot::lib::platform_punk] ;#normalized punkshell platform-dir name (punk::platform canon) +# -- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- +#G-122 host/target split. $this_platform_generic is the HOST canon and stays the +#key for everything that describes THIS interpreter (bootsupport/vendorlib +#platform dirs below, diagnostics). What the build EMITS is keyed by target: +# host_platform - canon of the driving tclsh (same value, named for clarity) +# target_platform - default target for kit outputs; a cygwin-family host +# (msys2/cygwin runtime on windows) builds win32 kits. +# Per-entry mapvfs.config declarations override it per runtime. +# host_windows - host process/filesystem semantics are windows-like +# (native windows OR cygwin family): drives copy commands, +# path case comparison, prompt behaviour - NOT artifact naming. +set ::punkboot::host_platform $this_platform_generic +set ::punkboot::target_platform [punkboot::lib::platform_target_default $this_platform_generic] +set ::punkboot::host_windows [expr {[punkboot::lib::platform_process_family $this_platform_generic] eq "windows"}] +# -- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- #we always create these lists in order of desired precedence. # - this is the same order when adding to auto_path - but will need to be reversed when using tcl:tm::add if {[file exists [file join $::punkboot::scriptfolder bootsupport]]} { @@ -931,6 +1098,17 @@ if {[file exists [file join $::punkboot::scriptfolder bootsupport]]} { lappend bootsupport_library_paths [file join $bootsupportdir lib] } else { puts stderr "No bootsupport dir for script [info script] at [file join $::punkboot::scriptfolder bootsupport]" + #G-122 host-personality diagnosis: a cygwin-family tclsh (msys2/cygwin runtime on + #windows) is a POSIX Tcl - 'C:/path/make.tcl' is not an absolute path to it, so + #'file normalize' silently prefixes the cwd and every derived path is wrong. The + #failure that follows (missing bootsupport, then an unresolvable project root) is + #unrecognisable without this note. Caller-side spelling, not something make.tcl can + #safely rewrite - name the script in the host's own path syntax. + if {[punkboot::lib::platform_is_cygwin_family $this_platform_generic] && [regexp {^([A-Za-z]):[/\\](.*)$} [info script] -> _cygdrive _cygrest]} { + puts stderr " NOTE: this tclsh is a $this_platform_generic (msys2/cygwin-runtime) build - a POSIX Tcl, to which the windows-style path '[info script]' is NOT absolute, so it was resolved relative to the current directory." + puts stderr " Re-run naming the script in this shell's own path syntax: /[string tolower $_cygdrive]/[string map {\\ /} $_cygrest]" + unset -nocomplain _cygdrive _cygrest + } #lappend bootsupport_module_paths [file join $startdir bootsupport modules_tcl$::tclmajorv] #lappend bootsupport_module_paths [file join $startdir bootsupport modules] #lappend bootsupport_library_paths [file join $startdir bootsupport lib_tcl$::tclmajorv/allplatforms] @@ -2086,12 +2264,14 @@ DIAGRAM 1b - THIN-LAYOUT SYNC (part of modules/libs/packages/bakehouse runs) DIAGRAM 2 - KIT ASSEMBLY DETAIL (the 'make.tcl bake' stage; incl. vfslibs phase) ---------------------------------------------------------------------------- - src/runtime/mapvfs.config which vfs folder pairs with which runtime -> kit name + src/runtime/mapvfs.config which vfs folder pairs with which runtime -> kit name, + and optionally the TARGET PLATFORM of that runtime [K9] 'make.tcl bakelist ?kitname ...?' reports the configured kit outputs (runtime/vfs presence + deployed state); 'make.tcl bake ?kitname ...?' bakes only the named kits (bare bake = all; unknown names error before any build) - src/runtime/.exe bare Tcl runtime (tclkit / tclsfe build) + bin/runtime// bare Tcl runtime (tclkit / tclsfe / suite build), in the + store tier of the TARGET platform it builds kits for [K9] src/vfs/_vfscommon.vfs/ common payload (built modules + libs, main boot support) src/vfs/.vfs/ per-kit overlay (main.tcl, kit-only modules/config) @@ -2102,7 +2282,9 @@ DIAGRAM 2 - KIT ASSEMBLY DETAIL (the 'make.tcl bake' stage; incl. vfslibs phase) v src/_build/.exe.vfs | - | tcl::zipfs::mkimg (vfs image appended to runtime exe) + | zip image appended to the runtime exe: tcl::zipfs::mkimg, + | or (driving tcl without zipfs, e.g 8.6) raw-runtime split + | + punk::zip::mkzip + concatenation - both mount the same v src/_build/.exe | @@ -2160,6 +2342,18 @@ KEY / NOTES untracked: modules*/ lib*/ at projectroot, src/_build/, bin/.exe (bin/ scripts/tools are tracked; kit executables are not) +[K9] HOST vs TARGET platform (G-122). What a bake EMITS is keyed by the target + platform, never by the driving tclsh's personality: which bin/runtime/ + holds the runtime, whether the runtime file and the built kit carry .exe, and + whether the pre-deploy sweep uses tasklist/taskkill or ps/kill. The default + target is this host's own platform - except a cygwin-family host (an + msys2/cygwin-runtime tclsh, which reports platform 'unix' on windows), which + targets win32. A mapvfs.config entry may declare its own target as a 4th + element and is then read from that platform's tier, named with its executable + convention, and (being unrunnable here) skipped by the process sweep. + 'make.tcl check' prints the derivation; 'make.tcl bakelist' shows per-kit + targets, with a target= note on rows that are not on the default target. + MAINTENANCE (agents take note) ---------------------------------------------------------------------------- This text is embedded in make.tcl (::punkboot::workflow_text). When the build @@ -2339,7 +2533,8 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO List the kit outputs configured in src/runtime/mapvfs.config (the same parsed mapping the bake subcommand consumes): kit name, kit type (kit|zip|zipcat...), runtime with presence in the runtime - store (/bin/runtime/), vfs folder, and the + store of that kit's TARGET platform + (/bin/runtime/), vfs folder, and the deployed state of /bin/ vs the src/_build build product: current - deployed copy is identical to the build product @@ -2349,9 +2544,12 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO A trailing notes column flags anomalies (runtime=missing, vfs=missing, rtrev=r when the runtime working copy is materialized from an older revision than a -r artifact beside - it). + it) and marks cross-target entries (target= on any kit + whose mapvfs entry declares a target other than this host's + default - 'make.tcl check' shows that default). kitname arguments filter the report to the named entries (per-kit - detail, with resolved paths). Reporting only - never builds." + detail, with resolved store tier, target and paths). Reporting + only - never builds." vfslibs " Propagate declared vendored platform-library packages into kit vfs @@ -2394,8 +2592,11 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO " Show module/library paths and any potentially problematic packages for running this script. - Also reports bootsupport staleness and the src provenance status (what - the build/promotion commands would do)." + Also reports the colour policy, the host/target platform derivation + (this tclsh's platform canon vs the default kit target, its runtime + store tier, executable suffix and process tooling), bootsupport + staleness and the src provenance status (what the build/promotion + commands would do)." projectversion " Advisory check: verify CHANGELOG.md matches punkproject.toml and warn @@ -2527,7 +2728,7 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO #still works when the mapping cannot be parsed here) stays authoritative. variable KITNAME_CHOICES [list] catch { - set _mapmodel [punkboot::lib::mapvfs_parse [file join $::punkboot::scriptfolder runtime mapvfs.config] "" $::punkboot::scriptfolder] + set _mapmodel [punkboot::lib::mapvfs_parse [file join $::punkboot::scriptfolder runtime mapvfs.config] "" $::punkboot::scriptfolder $::punkboot::target_platform] foreach _rec [punkboot::lib::mapvfs_kit_outputs $_mapmodel "" $::punkboot::scriptfolder] { lappend KITNAME_CHOICES [dict get $_rec kitname] } @@ -2955,23 +3156,15 @@ set ::punkboot::bake_selected_targetkits [list] ;#platform artifact names (e.g p set ::punkboot::bake_selected_vfs [list] ;#vfs folder tails the selected kits use set ::punkboot::bake_selected_runtimes [list] ;#unsuffixed runtime names the selected kits wrap if {$::punkboot::command eq "bake" && [llength $::punkboot::opt_kitnames]} { - switch -glob -- $this_platform_generic { - macosx-* { - set _rt_os_arch macosx - } - default { - set _rt_os_arch $this_platform_generic - } - } - set _rtfolder $binfolder/runtime/$_rt_os_arch - set _mapmodel [punkboot::lib::mapvfs_parse $sourcefolder/runtime/mapvfs.config $_rtfolder $sourcefolder] + set _rtbase $binfolder/runtime + set _mapmodel [punkboot::lib::mapvfs_parse $sourcefolder/runtime/mapvfs.config $_rtbase $sourcefolder $::punkboot::target_platform] if {[llength [dict get $_mapmodel configerrors]]} { foreach _errline [dict get $_mapmodel configerrors] { puts stderr $_errline } exit 3 } - set _kit_outputs [punkboot::lib::mapvfs_kit_outputs $_mapmodel $_rtfolder $sourcefolder] + set _kit_outputs [punkboot::lib::mapvfs_kit_outputs $_mapmodel $_rtbase $sourcefolder] set _matchinfo [punkboot::lib::mapvfs_match_outputs $_kit_outputs $::punkboot::opt_kitnames] if {[llength [dict get $_matchinfo unknown]]} { puts stderr "make.tcl bake: unknown kit name(s): [join [dict get $_matchinfo unknown] {, }] - nothing built" @@ -2995,7 +3188,7 @@ if {$::punkboot::command eq "bake" && [llength $::punkboot::opt_kitnames]} { exit 1 } if {[dict get $_rec runtime] ne "-" && ![dict get $_rec runtime_present]} { - puts stderr "make.tcl bake: kit '[dict get $_rec kitname]' cannot be built - runtime [dict get $_rec runtime_file] not present in bin/runtime/$_rt_os_arch - nothing built" + puts stderr "make.tcl bake: kit '[dict get $_rec kitname]' cannot be built - runtime [dict get $_rec runtime_file] not present in bin/runtime/[dict get $_rec store_tier] (target [dict get $_rec target]) - nothing built" exit 1 } } @@ -3010,7 +3203,7 @@ if {$::punkboot::command eq "bake" && [llength $::punkboot::opt_kitnames]} { } } puts stdout "selective bake (G-121): [join $::punkboot::bake_selected_kitnames {, }]" - unset -nocomplain _rt_os_arch _rtfolder _mapmodel _kit_outputs _matchinfo _rec _r _errline + unset -nocomplain _rtbase _mapmodel _kit_outputs _matchinfo _rec _r _errline } # ---------------------------------------- @@ -3140,6 +3333,15 @@ if {$::punkboot::command eq "check"} { puts stdout "colour policy (G-113): mode=$::punkboot::colour_mode colour_disabled=$::punk::console::colour_disabled ansistrip=[expr {[llength $::punkboot::ansistrip_pushed] ? [join $::punkboot::ansistrip_pushed +] : 0}]" puts stdout " (precedence: NO_COLOR > PUNK_FORCE_COLOR/FORCE_COLOR > stdout tty probe incl. the 8.6 windows console-encoding signature; piped/unknown output is fully ESC-stripped per channel unless forced - utf-16 console channels are never wrapped)" puts stdout $sep + #G-122 host/target platform split - the self-diagnosis for "why is my kit set + #addressed the way it is". One stable single line; the columns are key=value. + set _chk_suffix [punkboot::lib::platform_exe_suffix $::punkboot::target_platform] + if {$_chk_suffix eq ""} {set _chk_suffix "(none)"} + puts stdout "platform (G-122): host=$::punkboot::host_platform target=$::punkboot::target_platform store=bin/runtime/[punkboot::lib::platform_store_tier $::punkboot::target_platform] exe-suffix=$_chk_suffix process-tooling=[punkboot::lib::platform_process_family $::punkboot::target_platform]" + puts stdout " (host = this tclsh's punk::platform canon, from tcl_platform(os)='$::tcl_platform(os)' platform='$::tcl_platform(platform)'; target = default kit target - a cygwin-family host (msys2/cygwin runtime on windows) bakes win32 kits)" + puts stdout " (mapvfs.config entries may declare their own target platform as a 4th element - 'make.tcl bakelist' shows each kit's target, store tier and artifact name)" + unset -nocomplain _chk_suffix + puts stdout $sep puts stdout "module/library checks - paths from bootsupport only" puts stdout $sep puts stdout "- tcl::tm::list" @@ -3493,28 +3695,17 @@ if {$::punkboot::command eq "bakelist"} { #grepping; expected state sits in the columns, anomalies surface in a trailing #sparse notes column (key=value tags). Kit name arguments filter the report and #add a per-kit detail block (resolved paths/sizes/mtimes). - switch -glob -- $this_platform_generic { - macosx-* { - set rt_os_arch macosx - } - default { - set rt_os_arch $this_platform_generic - } - } - set rtfolder $binfolder/runtime/$rt_os_arch + #G-122: store addressing is keyed by TARGET platform. The default target is + #this host's own platform (a cygwin-family host targets win32) and mapvfs + #entries may declare their own - so each row resolves its own tier under + #bin/runtime and the report names the tiers it consulted. + set rt_default_target $::punkboot::target_platform + set rt_os_arch [punkboot::lib::platform_store_tier $rt_default_target] + set rtbase $binfolder/runtime + set rtfolder $rtbase/$rt_os_arch set buildfolder $sourcefolder/_build set mapfile $sourcefolder/runtime/mapvfs.config - #Loud store-folder guard: a nonexistent per-platform runtime store makes every row - #read runtime=missing for a reason that is not per-entry. Classic trigger: an - #msys/cygwin-runtime tclsh reports tcl_platform(platform)=unix with an MSYS_NT*/ - #CYGWIN_NT* os, deriving a store name that is not the windows store the kits use. - set rt_store_missing [expr {![file isdirectory $rtfolder]}] - if {$rt_store_missing} { - puts stderr "WARNING: runtime store folder bin/runtime/$rt_os_arch does not exist - every configured runtime below will read runtime=missing." - puts stderr " The folder name derives from THIS tclsh's platform canon: tcl_platform(os)='$::tcl_platform(os)' tcl_platform(platform)='$::tcl_platform(platform)' -> '$this_platform_generic'." - puts stderr " An msys/cygwin-runtime tclsh reports platform unix and derives a non-windows store name - drive make.tcl with a native tclsh (or a punk kit executable) to inspect/bake this platform's kits." - } - set mapmodel [punkboot::lib::mapvfs_parse $mapfile $rtfolder $sourcefolder] + set mapmodel [punkboot::lib::mapvfs_parse $mapfile $rtbase $sourcefolder $rt_default_target] foreach errline [dict get $mapmodel configerrors] { puts stderr $errline } @@ -3525,11 +3716,32 @@ if {$::punkboot::command eq "bakelist"} { puts stdout "no kit mapping config at $mapfile - no kit outputs configured" exit 0 } - set kit_outputs [punkboot::lib::mapvfs_kit_outputs $mapmodel $rtfolder $sourcefolder] + set kit_outputs [punkboot::lib::mapvfs_kit_outputs $mapmodel $rtbase $sourcefolder] if {![llength $kit_outputs]} { puts stdout "no kit outputs configured in $mapfile" exit 0 } + #Loud store-tier guard: a nonexistent store folder makes every row addressing it + #read runtime=missing for a reason that is not per-entry. Reported per tier + #actually referenced, since entries may target different platforms (G-122). + set report_tiers [list] + foreach rec $kit_outputs { + if {[dict get $rec runtime] eq "-"} {continue} + set t [dict get $rec store_tier] + if {$t ni $report_tiers} {lappend report_tiers $t} + } + set rt_store_missing [expr {![file isdirectory $rtfolder]}] + if {$rt_os_arch ni $report_tiers} {lappend report_tiers $rt_os_arch} + foreach t $report_tiers { + if {[file isdirectory $rtbase/$t]} {continue} + if {$t eq $rt_os_arch} { + puts stderr "WARNING: runtime store folder bin/runtime/$t does not exist - every configured runtime on the default target will read runtime=missing." + puts stderr " Default target '$rt_default_target' derives from THIS tclsh: tcl_platform(os)='$::tcl_platform(os)' tcl_platform(platform)='$::tcl_platform(platform)' -> host canon '$::punkboot::host_platform' -> target '$rt_default_target'." + puts stderr " Populate it with 'bin/punk-runtime.cmd' (or by hand), or declare a per-entry target platform in mapvfs.config for runtimes stored elsewhere." + } else { + puts stderr "WARNING: runtime store folder bin/runtime/$t does not exist - configured entries targeting $t will read runtime=missing." + } + } set reportset $kit_outputs if {[llength $::punkboot::opt_kitnames]} { set matchinfo [punkboot::lib::mapvfs_match_outputs $kit_outputs $::punkboot::opt_kitnames] @@ -3555,8 +3767,13 @@ if {$::punkboot::command eq "bakelist"} { if {![dict get $rec vfs_present]} { lappend notes vfs=missing } + if {[dict get $rec target] ne $rt_default_target} { + #cross-target entry: its runtime, store tier and artifact naming follow + #its own platform, not this host's default + lappend notes target=[dict get $rec target] + } if {$runtime ne "-" && [dict get $rec runtime_present]} { - set rtstale [punkboot::lib::runtime_materialization_check $rtfolder [dict get $rec runtime_file]] + set rtstale [punkboot::lib::runtime_materialization_check [dict get $rec runtime_dir] [dict get $rec runtime_file]] if {[dict size $rtstale]} { lappend notes rtrev=r[dict get $rtstale current_rev] references stay silent; one warning per runtime per run. #G-121: the comparison core lives in ::punkboot::lib::runtime_materialization_check, #shared with the bakelist report's rtrev= note. +#G-122: run a command whose arguments must reach a NATIVE WINDOWS program +#verbatim. The msys2 runtime rewrites any argument that looks like an absolute +#posix path when it spawns a native windows executable, so 'taskkill /PID 1234' +#arrives as 'taskkill C:/msys64-root/PID 1234' - field-verified 2026-07-26: the +#pre-deploy process sweep found the running kit with tasklist and then could not +#kill it, failing the whole kit. MSYS2_ARG_CONV_EXCL=* is msys2's documented +#opt-out and is inert on every other host (real cygwin does not convert arguments +#at all), so one command spelling stays correct from every host personality. +proc ::punkboot::exec_nativeargs {cmdlist} { + set had [info exists ::env(MSYS2_ARG_CONV_EXCL)] + if {$had} { + set prev $::env(MSYS2_ARG_CONV_EXCL) + } + set ::env(MSYS2_ARG_CONV_EXCL) * + try { + return [exec {*}$cmdlist] + } finally { + if {$had} { + set ::env(MSYS2_ARG_CONV_EXCL) $prev + } else { + unset -nocomplain ::env(MSYS2_ARG_CONV_EXCL) + } + } +} + set ::punkboot::runtime_rev_warned [dict create] proc ::punkboot::runtime_materialization_warning {rtfolder runtime_fullname} { if {[dict exists $::punkboot::runtime_rev_warned $runtime_fullname]} { @@ -4768,16 +5023,75 @@ proc ::punkboot::runtime_materialization_warning {rtfolder runtime_fullname} { return } +# -- --- --- --- --- --- --- --- --- --- +#load mapvfs.config file (if any) in runtime folder to map runtimes to vfs folders. +#build a dict keyed on runtime executable name. +#If no mapfile (or no mapfile entry for that runtime) - the runtime will be paired with a matching .vfs folder in src folder. e.g punk.exe to src/punk.vfs +#If vfs folders or runtime executables which are explicitly listed in the mapfile don't exist - warn on stderr - but continue. if such nonexistants found; prompt user for whether to continue or abort. +#G-121: parsing lives in ::punkboot::lib::mapvfs_parse - the shared model consumed here, +#by 'bakelist' and by the selective-bake name resolution (never the file format directly). +#G-122: parsed BEFORE runtime discovery, because the mapping is what says which +#store tiers this bake must look in - an entry declaring a target platform other +#than the host default addresses its own bin/runtime/. +set mapfile $rt_sourcefolder/mapvfs.config +set mapmodel [punkboot::lib::mapvfs_parse $mapfile $rtbase $sourcefolder $rt_default_target] +foreach warnline [dict get $mapmodel warnings] { + puts stderr $warnline +} +if {[llength [dict get $mapmodel configerrors]]} { + foreach errline [dict get $mapmodel configerrors] { + puts stderr $errline + } + exit 3 +} +set runtime_vfs_map [dict get $mapmodel runtime_vfs_map] +set vfs_runtime_map [dict get $mapmodel vfs_runtime_map] +set runtime_target [dict get $mapmodel runtime_target] +set missing [dict get $mapmodel missing] +if {[llength $missing]} { + puts stderr "WARNING [llength $missing] missing items from $mapfile. (TODO - prompt user to continue/abort)" + foreach m $missing { + puts stderr " $m" + } + puts stderr "continuing..." +} +# -- --- --- --- --- --- --- --- --- --- +puts "-- runtime_vfs_map --" +punk::lib::pdict runtime_vfs_map +puts "---------------------" +puts "-- vfs_runtime_map--" +punk::lib::pdict vfs_runtime_map +puts "---------------------" + +# -- --- --- --- --- --- --- --- --- --- +#Runtime discovery. The default-target store is scanned wholesale (a runtime +#there needs no mapping entry - it can pair with a same-named .vfs folder); +#cross-target runtimes are only picked up when the mapping names them, since +#scanning every tier would pull in runtimes nothing asked to be built. +#$rtfolder_of resolves a discovered runtime FILE name to the store folder it +#came from (runtime_target_of gives its target platform) - every later store +#address, suffix and process-tooling decision goes through these rather than +#assuming the driving tclsh's platform (G-122). set runtimes [list] +set rtfolder_of [dict create] +proc ::punkboot::lib::runtime_target_of {rtname} { + #target platform of a runtime name (unsuffixed), from the parsed mapping + upvar #0 runtime_target runtime_target rt_default_target rt_default_target + if {[info exists runtime_target] && [dict exists $runtime_target $rtname]} { + return [dict get $runtime_target $rtname] + } + return $rt_default_target +} if {[file dirname [info nameofexecutable]] eq $rtfolder && [file isdirectory [info nameofexecutable]]} { #info name will only appear to be a directory when the runtime is a tclkit that is mounted at the same path as the physical file. #This is not a problem for zip based kits. #fix so that we can find and use a runtime tclkit that is currently in use as the runtime that is running this script. # - when the current runtime is using a runtime in the rtfolder that is a tclkit, it is mounted at the same path as the physical file and so appears to Tcl as if it's a directory. # - use external filesystem tools to make a copy of the file - if {$::tcl_platform(platform) eq "windows"} { + #(copy tooling is a HOST concern - the running executable is by definition a host binary) + if {$::punkboot::host_windows} { #exec cmd /c copy [info nameofexecutable] $rtfolder/[file rootname [file tail [info nameofexecutable]]]_BUILDCOPY.exe - exec cmd /c copy [file nativename [info nameofexecutable]] [file nativename $rtfolder/[punkboot::lib::runtime_buildcopyname [info nameofexecutable]]] + ::punkboot::exec_nativeargs [list cmd /c copy [file nativename [info nameofexecutable]] [file nativename $rtfolder/[punkboot::lib::runtime_buildcopyname [info nameofexecutable]]]] } else { #exec cp [info nameofexecutable] $rtfolder/[file tail [info nameofexecutable]]_BUILDCOPY exec cp [info nameofexecutable] $rtfolder/[punkboot::lib::runtime_buildcopyname [info nameofexecutable]] @@ -4785,12 +5099,14 @@ if {[file dirname [info nameofexecutable]] eq $rtfolder && [file isdirectory [in #we need to add the base name of the current exe to the runtimes list because the next section will look for files in the runtime folder #and won't find the mounted runtime since it's mounted as a directory. lappend runtimes [file tail [info nameofexecutable]] + dict set rtfolder_of [file tail [info nameofexecutable]] $rtfolder } set rtfolder_files [glob -nocomplain -dir $rtfolder -types {f} -tail *] set exclusions {.config .md .ico .txt .doc .pdf .htm .html} ;#we don't encourage other files in runtime folder aside from mapvfs.config - but lets ignore some common possibilities lappend exclusions .zip .7z .pea .bz2 .tar .gz .tgz .z .xz ;#don't allow archives to directly be treated as runtimes - tolerate presence but require user to unpack or rename if they're to be used as runtimes lappend exclusions .tail ;#result of running sdx mksplit on a kit - in theory the .head could be used - review/test +lappend exclusions .toml ;#G-103 runtime artifact metadata sits beside its runtime foreach f $rtfolder_files { if {[string match "*_BUILDCOPY*" $f]} { continue @@ -4800,15 +5116,42 @@ foreach f $rtfolder_files { } if {$f ni $runtimes} { lappend runtimes $f + dict set rtfolder_of $f $rtfolder } } +#cross-target runtimes named by the mapping (their own tier, their own suffix) +set cross_target_tiers [list] +dict for {_rtname _rttarget} $runtime_target { + if {$_rtname eq "-" || $_rttarget eq $rt_default_target} { + continue + } + set _tier [punkboot::lib::platform_store_tier $_rttarget] + set _dir $rtbase/$_tier + set _file $_rtname[punkboot::lib::platform_exe_suffix $_rttarget] + if {$_tier ni $cross_target_tiers} { + lappend cross_target_tiers $_tier + } + if {![file exists [file join $_dir $_file]]} { + continue ;#absent cross-target runtime already warned by the parse + } + if {[dict exists $rtfolder_of $_file]} { + puts stderr "WARNING: runtime file $_file is present in more than one store tier ([dict get $rtfolder_of $_file] and $_dir) - using the first found. Rename one of them to build both." + continue + } + lappend runtimes $_file + dict set rtfolder_of $_file $_dir +} +if {[llength $cross_target_tiers]} { + puts stdout "cross-target runtime tiers in use (mapvfs.config target platforms): [join $cross_target_tiers {, }]" +} +unset -nocomplain _rtname _rttarget _tier _dir _file if {![llength $runtimes]} { puts stderr "No executable runtimes found in $rtfolder - unable to build any .vfs folders into executables." puts stderr "Add runtimes to $rtfolder if required" if {![file isdirectory $rtfolder]} { - #same store-folder self-diagnosis as bakelist (msys/cygwin-runtime tclsh trigger) - puts stderr "NOTE: that runtime store folder does not exist. The name derives from THIS tclsh's platform canon: tcl_platform(os)='$::tcl_platform(os)' tcl_platform(platform)='$::tcl_platform(platform)' -> '$this_platform_generic'." - puts stderr " An msys/cygwin-runtime tclsh reports platform unix and derives a non-windows store name - drive make.tcl with a native tclsh (or a punk kit executable) to bake this platform's kits." + #same store-tier self-diagnosis as bakelist + puts stderr "NOTE: that runtime store folder does not exist. Default target '$rt_default_target' derives from THIS tclsh: tcl_platform(os)='$::tcl_platform(os)' tcl_platform(platform)='$::tcl_platform(platform)' -> host canon '$::punkboot::host_platform' -> target '$rt_default_target'." + puts stderr " Populate it with 'bin/punk-runtime.cmd' (or by hand), or declare a per-entry target platform in mapvfs.config for runtimes stored elsewhere." } #todo - don't exit - it is valid to use runtime of - to just build a .kit/.zipkit ? exit 0 @@ -4817,10 +5160,10 @@ if {![llength $runtimes]} { #other runtimes get no BUILDCOPY refresh, no capability probe and no punkcheck events. if {[llength $::punkboot::bake_selected_kitnames]} { set runtimes [lmap _rtfile $runtimes { - set _rtkey $_rtfile - if {[string match *.exe $_rtkey]} { - set _rtkey [string range $_rtkey 0 end-4] - } + #NOTE: only an .exe suffix may be stripped to recover the mapping key - + #'file rootname' would eat the last dotted version segment of names like + #tclsh9.0.5 (suffixless on non-windows targets). + set _rtkey [punkboot::lib::runtime_mapkey $_rtfile] if {$_rtkey ni $::punkboot::bake_selected_runtimes} { continue } @@ -4830,41 +5173,6 @@ if {[llength $::punkboot::bake_selected_kitnames]} { } #previous have_sdx test location -# -- --- --- --- --- --- --- --- --- --- -#load mapvfs.config file (if any) in runtime folder to map runtimes to vfs folders. -#build a dict keyed on runtime executable name. -#If no mapfile (or no mapfile entry for that runtime) - the runtime will be paired with a matching .vfs folder in src folder. e.g punk.exe to src/punk.vfs -#If vfs folders or runtime executables which are explicitly listed in the mapfile don't exist - warn on stderr - but continue. if such nonexistants found; prompt user for whether to continue or abort. -#G-121: parsing lives in ::punkboot::lib::mapvfs_parse - the shared model consumed here, -#by 'bakelist' and by the selective-bake name resolution (never the file format directly). -set mapfile $rt_sourcefolder/mapvfs.config -set mapmodel [punkboot::lib::mapvfs_parse $mapfile $rtfolder $sourcefolder] -foreach warnline [dict get $mapmodel warnings] { - puts stderr $warnline -} -if {[llength [dict get $mapmodel configerrors]]} { - foreach errline [dict get $mapmodel configerrors] { - puts stderr $errline - } - exit 3 -} -set runtime_vfs_map [dict get $mapmodel runtime_vfs_map] -set vfs_runtime_map [dict get $mapmodel vfs_runtime_map] -set missing [dict get $mapmodel missing] -if {[llength $missing]} { - puts stderr "WARNING [llength $missing] missing items from $mapfile. (TODO - prompt user to continue/abort)" - foreach m $missing { - puts stderr " $m" - } - puts stderr "continuing..." -} -# -- --- --- --- --- --- --- --- --- --- -puts "-- runtime_vfs_map --" -punk::lib::pdict runtime_vfs_map -puts "---------------------" -puts "-- vfs_runtime_map--" -punk::lib::pdict vfs_runtime_map -puts "---------------------" #only test the runtime capabilities for runtimes that are actually in our map @@ -4913,27 +5221,34 @@ foreach runtime [dict keys $runtime_vfs_map] { exit 0 } - #temp - #build for current platform only for now - if {![file exists $rtfolder/$runtime]} { - dict set runtime_caps $runtime exitcode -1 error "find-fail $rtfolder/$runtime" + #G-122: address the runtime in ITS target's store tier, with ITS target's + #executable suffix. (Before the split this probed a suffixless name in the + #host-derived folder, so on windows every runtime silently reported + #find-fail and no capability was ever learned.) + set rt_target [punkboot::lib::runtime_target_of $runtime] + set rt_dir $rtbase/[punkboot::lib::platform_store_tier $rt_target] + set rt_file $runtime[punkboot::lib::platform_exe_suffix $rt_target] + if {![file exists $rt_dir/$rt_file]} { + dict set runtime_caps $runtime exitcode -1 error "find-fail $rt_dir/$rt_file" + continue + } + if {[punkboot::lib::platform_process_family $rt_target] ne [punkboot::lib::platform_process_family $::punkboot::host_platform]} { + #cross-target runtime: this host cannot execute it, so its capabilities + #stay unknown (the kit machinery falls back to trying the configured type) + dict set runtime_caps $runtime exitcode -1 error "cross-target ($rt_target) - not executable on this host" continue } #invoke can fail if runtime not an executable file for the current platform - if {[file type $rtfolder/$runtime] eq "directory"} { + if {[file type $rt_dir/$rt_file] eq "directory"} { #assume it's a mounted tclkit (because tcl kits mount in same place as their underlying path and mask the executable as a directory) #use the BUILDCOPY created above - REVIEW - if {$::tcl_platform(platform) eq "windows"} { - set useruntime [file rootname $runtime]_BUILDCOPY.exe - } else { - set useruntime ${runtime}_BUILDCOPY - } + set useruntime [punkboot::lib::runtime_buildcopyname $rt_file] } else { - set useruntime $runtime + set useruntime $rt_file } if {![catch { - lassign [punk::lib::invoke [list $rtfolder/$useruntime <<$capscript]] stdout stderr exitcode + lassign [punk::lib::invoke [list $rt_dir/$useruntime <<$capscript]] stdout stderr exitcode } errM]} { if {$exitcode == 0} { dict set runtime_caps $runtime $stdout @@ -4975,7 +5290,7 @@ set vfs_folder_changes [dict create] ;#cache whether each .vfs folder has change # -- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- #make build copy for all runtimes - not just those in the map - because even without a mapvfs.config file entry we build an exe for a runtime that matches .vfs folder name - REVIEW. foreach runtimefile $runtimes { - #runtimefile e.g tclkit86bi.exe on windows tclkit86bi on other platforms + #runtimefile e.g tclkit86bi.exe for a win32 target, tclkit86bi for a posix one #sdx *may* be pointed to use the runtime we use to build the kit, or the user may manually use this runtime if they don't have tclsh #sdx will complain if the same runtime is used for the shell as is used in the -runtime argument - so we make a copy (REVIEW) @@ -4983,23 +5298,19 @@ foreach runtimefile $runtimes { # file copy $rtfolder/$runtimefile $buildfolder/buildruntime.exe #} - #---------------------- - #todo - #temp - only build for current platform - need changes to runtime/mapvfs.config etc - if {![file exists $rtfolder/$runtimefile]} { - puts stderr " >> skipping $runtimefile not present in $rtfolder (wrong platform?) x-platform kit build TODO." + #G-122: each runtime is read from the tier its target platform names, not + #from one host-derived folder. Build copies keep the flat build_ name - + #the mapping cannot hold one runtime name in two tiers, so they cannot collide. + set rt_dir [dict get $rtfolder_of $runtimefile] + if {![file exists $rt_dir/$runtimefile]} { + puts stderr " >> skipping $runtimefile not present in $rt_dir" continue } - #---------------------- - if {[file type $rtfolder/$runtimefile] eq "directory"} { + if {[file type $rt_dir/$runtimefile] eq "directory"} { #assume it's a mounted tclkit (because tcl kits mount in same place as their underlying path and mask the executable) #use the BUILDCOPY created above - REVIEW - if {$::tcl_platform(platform) eq "windows"} { - set useruntimefile [file tail $runtimefile]_BUILDCOPY.exe - } else { - set useruntimefile ${runtimefile}_BUILDCOPY - } + set useruntimefile [punkboot::lib::runtime_buildcopyname $runtimefile] } else { set useruntimefile $runtimefile } @@ -5010,11 +5321,11 @@ foreach runtimefile $runtimes { }] #---------- set installer [punkcheck::installtrack new $installername $basedir/.punkcheck] - $installer set_source_target $rtfolder $buildfolder + $installer set_source_target $rt_dir $buildfolder set event [$installer start_event $config] $event targetset_init INSTALL $buildfolder/build_$runtimefile - #$event targetset_addsource $rtfolder/$runtimefile - $event targetset_addsource $rtfolder/$useruntimefile ;#possibly the _BUILDCOPY in the runtime folder (created for when mounted in current executable as a tclkit) + #$event targetset_addsource $rt_dir/$runtimefile + $event targetset_addsource $rt_dir/$useruntimefile ;#possibly the _BUILDCOPY in the runtime folder (created for when mounted in current executable as a tclkit) $event targetset_addsource $buildfolder/build_$runtimefile ;#self as source for change detection #---------- @@ -5026,9 +5337,9 @@ foreach runtimefile $runtimes { $event targetset_started # -- --- --- --- --- --- #This is the full runtime - *possibly* with some sort of vfs attached. - puts stdout "Copying runtime (as is) from $rtfolder/$useruntimefile to $buildfolder/build_$runtimefile" + puts stdout "Copying runtime (as is) from $rt_dir/$useruntimefile to $buildfolder/build_$runtimefile" if {[catch { - file copy -force $rtfolder/$useruntimefile $buildfolder/build_$runtimefile ;#becomes building_runtime + file copy -force $rt_dir/$useruntimefile $buildfolder/build_$runtimefile ;#becomes building_runtime } errM]} { puts stderr " >> copy runtime to $buildfolder/build_$runtimefile FAILED" $event targetset_end FAILED @@ -5066,12 +5377,16 @@ if {$sdxpath eq ""} { #see if the runtime (or any runtime *for current platform*) has tclkit in the name or has_starkit(?) and use that #todo - review #picking first found for now - fix + #G-122: only a runtime this HOST can execute can drive sdx - the + #capability probe already skipped cross-target runtimes, so a + #has_starkit record means the runtime ran here. dict for {rtname rtprops} $runtime_caps { if {[dict exists $rtprops has_starkit] && [dict get $rtprops has_starkit]} { - set testpath $rtfolder/$rtname + set _rt_target [punkboot::lib::runtime_target_of $rtname] + set testpath $rtbase/[punkboot::lib::platform_store_tier $_rt_target]/$rtname[punkboot::lib::platform_exe_suffix $_rt_target] if {[file exists $testpath]} { if {[file type $testpath] eq "directory"} { - set alt_tclkitpath $rtfolder/[punkboot::lib::runtime_buildcopyname $testpath] + set alt_tclkitpath [file dirname $testpath]/[punkboot::lib::runtime_buildcopyname $testpath] } else { set alt_tclkitpath $testpath } @@ -5262,6 +5577,43 @@ proc vfs_startup_script_warning {vfsfolder} { } return "vfs folder $vfsfolder has NO STARTUP SCRIPT: no root main.tcl and no root fauxlink resolving to main.tcl with an existing .tcl target - kits built from it will have no boot script (build proceeding anyway)$detail" } +#Cat-style zipkit assembly: a raw (vfs-free) runtime followed by a freshly built +#zip, concatenated. This is the zipfs-less assembly path for BOTH zip-type kits +#(G-122): Tcl's zipfs locates an image by scanning back from the END of the file +#and reads the central directory with ARCHIVE-START-RELATIVE offsets, so a plain +#concatenation mounts. 'zipfs mkimg's file-relative offset rewrite is one +#convention, not a mounting requirement - the zipcat kit type is the in-repo +#proof, and this lets a driving tcl with no zipfs at all (any 8.6) assemble zip +#kits. The zip itself is written by tcl::zipfs::mkzip when the driving tcl has +#it, else by punk::zip::mkzip (pure Tcl). +#Returns the number of zip bytes appended. +proc ::punkboot::assemble_zipcat_image {raw_runtime wrapvfs outfile zipfile} { + file copy -force $raw_runtime $outfile + #runtime in runtime folder may not have write perm set - ensure the copy does as we need to append + catch {exec chmod +w $outfile} + file delete $zipfile + if {[info commands ::tcl::zipfs] ne ""} { + puts stdout "tcl::zipfs::mkzip $zipfile $wrapvfs $wrapvfs" + ::tcl::zipfs::mkzip $zipfile $wrapvfs $wrapvfs + } else { + puts stdout "punk::zip::mkzip -directory $wrapvfs -base $wrapvfs $zipfile *" + package require punk::zip + punk::zip::mkzip -directory $wrapvfs -base $wrapvfs $zipfile * + } + puts stderr "concatenating zip to executable.." + set fdout [open $outfile a] + chan conf $fdout -translation binary + puts stderr "runtime bytes: [tell $fdout]" + set fdzip [open $zipfile r] + chan conf $fdzip -translation binary + set zipbytes [fcopy $fdzip $fdout] + close $fdzip + puts stderr "zip bytes: $zipbytes" + puts stderr "exezip bytes: [tell $fdout]" + close $fdout + return $zipbytes +} + set startdir [pwd] puts stdout "Found [llength $vfs_tails] .vfs folders - checking each for executables that may need to be built" cd [file dirname $buildfolder] @@ -5291,24 +5643,26 @@ foreach vfstail $vfs_tails { }] + #G-122: a runtime's filename and store tier come from ITS target platform. + #The '-' pseudo-runtime (unwrapped .kit output) carries no file at all. set runtimes [list] if {[dict exists $vfs_runtime_map $vfstail]} { - #set runtimes [dict get $vfs_runtime_map $vfstail] + #set runtimes [dict get $vfs_runtime_map $vfstail] #runtimes in vfs_runtime_map_vfs dict are unsuffixed (.exe stripped or was not present) set applist [dict get $vfs_runtime_map $vfstail] - if {"windows" eq $::tcl_platform(platform)} { - foreach rt_app $applist { - set rt [lindex $rt_app 0] - if {$rt ne "-" && [file exists $rtfolder/$rt.exe]} { - lappend runtimes $rt.exe - } + foreach rt_app $applist { + set rt [lindex $rt_app 0] + if {$rt eq "-"} { + #the '-' (runtime-less .kit) pseudo-runtime has never reached this + #list - both historical branches filtered it out - so the loop + #below's '-' handling is currently unreachable. Left as-is: + #enabling it is a separate, untested change. + continue } - } else { - foreach rt_app $applist { - set rt [lindex $rt_app 0] - if {[file exists $rtfolder/$rt]} { - lappend runtimes $rt - } + set rt_target [punkboot::lib::runtime_target_of $rt] + set rt_file $rt[punkboot::lib::platform_exe_suffix $rt_target] + if {[file exists $rtbase/[punkboot::lib::platform_store_tier $rt_target]/$rt_file]} { + lappend runtimes $rt_file } } } else { @@ -5317,18 +5671,14 @@ foreach vfstail $vfs_tails { #but conversely, adding an extra entry shouldn't stop default builds that used to run.. set matchrt [file rootname [file tail $vfstail]] ;#e.g project.vfs -> project if {![dict exists $runtime_vfs_map $matchrt]} { - if {"windows" eq $::tcl_platform(platform)} { - if {[file exists $rtfolder/$matchrt.exe]} { - lappend runtimes $matchrt.exe - } - } else { - if {[file exists $rtfolder/$matchrt]} { - lappend runtimes $matchrt - } + #unmapped: the default-target store is the only place a name-match is looked for + if {[file exists $rtfolder/$matchrt$rt_exe_suffix]} { + lappend runtimes $matchrt$rt_exe_suffix } } } - #assert $runtimes is a list of executable names suffixed with .exe if on windows - whether or not specified with .exe in the mapvfs.config + #assert $runtimes is a list of runtime filenames as they exist in their target's store tier + #(suffixed with .exe for windows-family targets, whether or not mapvfs.config spelled it that way) puts " vfs: $vfstail runtimes to process ([llength $runtimes]): $runtimes" if {[llength $runtimes]} { @@ -5342,13 +5692,17 @@ foreach vfstail $vfs_tails { #todo - non kit based - zipkit? # $runtimes may now include a dash entry "-" (from mapvfs.config file) foreach runtime_fullname $runtimes { - set rtname [file rootname $runtime_fullname] + set rtname [punkboot::lib::runtime_mapkey $runtime_fullname] if {[llength $::punkboot::bake_selected_kitnames] && $rtname ni $::punkboot::bake_selected_runtimes} { continue ;#G-121 selective bake - no requested kit wraps this runtime } + #G-122: everything this iteration emits is keyed by the runtime's TARGET + set rt_target [punkboot::lib::runtime_target_of $rtname] + set rt_tierdir $rtbase/[punkboot::lib::platform_store_tier $rt_target] + set rt_suffix [punkboot::lib::platform_exe_suffix $rt_target] #rtname of "-" indicates build a kit without a runtime if {$runtime_fullname ne "-"} { - ::punkboot::runtime_materialization_warning $rtfolder $runtime_fullname + ::punkboot::runtime_materialization_warning $rt_tierdir $runtime_fullname } #first configured runtime will be the one to use the same name as .vfs folder for output executable. Additional runtimes on this .vfs will need to suffix the runtime name to disambiguate. @@ -5372,11 +5726,7 @@ foreach vfstail $vfs_tails { if {$rtname eq "-"} { set targetkit $appname.kit } else { - if {$::tcl_platform(platform) eq "windows"} { - set targetkit ${appname}.exe - } else { - set targetkit $appname - } + set targetkit ${appname}$rt_suffix if {$targetkit in $exe_names_seen} { #duplicate appname configured? #todo - consider creating as ${appname}(2) etc? @@ -5400,9 +5750,10 @@ foreach vfstail $vfs_tails { #cannot replace a running image anyway. Informational/update subcommands are #unaffected (they exit before the kit machinery), so driving make.tcl from a built #punk executable remains supported for everything except rebuilding that executable. + #(a HOST filesystem comparison - case rules follow the host, not the target) set self_exe [file normalize [info nameofexecutable]] set self_deploy_target [file normalize $binfolder/$targetkit] - if {$::tcl_platform(platform) eq "windows"} { + if {$::punkboot::host_windows} { set is_self_build [string equal -nocase $self_exe $self_deploy_target] } else { set is_self_build [string equal $self_exe $self_deploy_target] @@ -5467,15 +5818,23 @@ foreach vfstail $vfs_tails { #to consider: - allow specifying runtime kits as if they are vfs folders in the normal xxx.vfs list - and autodetect and extract #would need to detect UPX, cookfs,zipfs,tclkit set rtmountpoint "" + #Test the capability VALUE, not just its presence: a probe that ran + #reports has_starkit/has_cookfs 0 for a plain zipfs runtime, and + #trying to unwrap such a runtime as a kit ends in sdx mksplit + #failing and REPLACING raw_runtime with the un-split original - + #which silently yields a zip kit with two zips and no tcl_library. + #(Latent until G-122 made the capability probe actually run on + #windows: before that every probe reported find-fail, so this list + #was always empty.) set caps [dict get $runtime_caps $rtname] set source_kit_caps [list] - if {[dict exists $caps has_zipfs]} { + if {[dict exists $caps has_zipfs] && [dict get $caps has_zipfs]} { lappend source_kit_caps zip } - if {[dict exists $caps has_starkit]} { + if {[dict exists $caps has_starkit] && [dict get $caps has_starkit]} { lappend source_kit_caps kit } - if {[dict exists $caps has_cookfs]} { + if {[dict exists $caps has_cookfs] && [dict get $caps has_cookfs]} { lappend source_kit_caps cookfs } @@ -5592,11 +5951,18 @@ foreach vfstail $vfs_tails { zipfile::decode::open $extractedzipfile set archiveinfo [zipfile::decode::archive] zipfile::decode::unzip $archiveinfo $extractedzipfolder - }]} { + } extracterr]} { set extraction_done 1 set extract_kit_type $extract_kit_try #todo - verify that init.tcl etc are present? merge_over $extractedzipfolder $targetvfs + } else { + #Do not swallow the reason: without the runtime's own + #zip contents the built kit has no tcl_library. Note + #zipfile::decode is a system tcllib package, not a + #vendored one - a tclsh without tcllib (e.g the msys2 + #/usr/bin/tclsh8.6) cannot take this path at all. + puts stderr "zipfs-less extraction of $runtime_fullname FAILED: $extracterr" } } @@ -5684,10 +6050,11 @@ foreach vfstail $vfs_tails { if {!$extraction_done} { #TODO: if not extracted - use a default tcl_library for patchlevel and platform? - puts stderr "--------------------------------------------" - puts stderr "[punkboot::sgr 31]WARNING: No extraction done from runtime $runtime_fullname[punkboot::sgr]" - puts stderr "If no init.tcl provided in the vfs at the proper location (containing init.tcl) - the resulting kit will probably not initialise properly!" - puts stderr "--------------------------------------------" + #Recapped BUILD-WARNING (not just an inline note): the build continues + #and produces an artifact, but unless the .vfs itself supplies a + #tcl_library that artifact cannot boot - and this line would otherwise + #scroll away behind thousands of merge lines. + ::punkboot::print_build_warnings [list "No extraction done from runtime $runtime_fullname (tried: $target_kit_type[expr {[llength $source_kit_caps] ? " + runtime caps $source_kit_caps" : ""}]) - kit $targetkit gets no tcl_library from the runtime and will not initialise unless src/vfs/$vfstail supplies one"] file mkdir $targetvfs } @@ -5772,9 +6139,17 @@ foreach vfstail $vfs_tails { puts stderr "RUNTIME capabilities unknown. Unsure if zip supported. trying anyway.." } } - #note - as at 2024-08 - there is some discussion about the interface to mkimg - it is considered unstable (may change to -option value syntax) - puts stderr "calling: tcl::zipfs::mkimg $buildfolder/$vfsname.new $wrapvfs $wrapvfs \"\" $raw_runtime" - tcl::zipfs::mkimg $buildfolder/$vfsname.new $wrapvfs $wrapvfs "" $raw_runtime + if {[info commands ::tcl::zipfs::mkimg] ne ""} { + #note - as at 2024-08 - there is some discussion about the interface to mkimg - it is considered unstable (may change to -option value syntax) + puts stderr "calling: tcl::zipfs::mkimg $buildfolder/$vfsname.new $wrapvfs $wrapvfs \"\" $raw_runtime" + tcl::zipfs::mkimg $buildfolder/$vfsname.new $wrapvfs $wrapvfs "" $raw_runtime + } else { + #G-122: the driving tcl has no zipfs (8.6) - assemble the same + #kind of image by concatenation. The result mounts on any + #zipfs-capable runtime (archive-start-relative offsets). + puts stderr "WARNING: tcl shell '[info nameofexecutable]' has no zipfs - assembling zip kit $targetkit by concatenation (raw runtime + mkzip)" + ::punkboot::assemble_zipcat_image $raw_runtime $wrapvfs $buildfolder/$vfsname.new $buildfolder/$vfsname.zip + } } result ]} { set failmsg "zipfs mkimg failed with msg: $result" puts stderr "tcl::zipfs::mkimg $targetkit failed" @@ -5784,7 +6159,7 @@ foreach vfstail $vfs_tails { $vfs_installer destroy continue } else { - puts stdout "ok - finished tcl::zipfs::mkimg" + puts stdout "ok - finished zip image assembly" set separator [string repeat = 40] puts stdout $separator puts stdout $result @@ -5810,32 +6185,7 @@ foreach vfstail $vfs_tails { } #'archive' based zip offsets - editable in 7z,peazip - file copy $raw_runtime $buildfolder/$vfsname.new - #runtime in runtime folder may not have write perm set - ensure the copy does as we need to append - catch {exec chmod +w $buildfolder/$vfsname.new} - file delete $buildfolder/$vfsname.zip - - if {[info commands ::tcl::zipfs] ne ""} { - puts stdout "tcl::zipfs::mkzip $buildfolder/$vfsname.zip $wrapvfs $wrapvfs" - ::tcl::zipfs::mkzip $buildfolder/$vfsname.zip $wrapvfs $wrapvfs - } else { - puts stdout "punk::zip::mkzip -directory $wrapvfs -base $wrapvfs $buildfolder/$vfsname.zip *" - package require punk::zip - punk::zip::mkzip -directory $wrapvfs -base $wrapvfs $buildfolder/$vfsname.zip * - } - - - puts stderr "concatenating executable to zip.." - set fdout [open $buildfolder/$vfsname.new a] - chan conf $fdout -translation binary - puts stderr "runtime bytes: [tell $fdout]" - set fdzip [open $buildfolder/$vfsname.zip r] - chan conf $fdzip -translation binary - set zipbytes [fcopy $fdzip $fdout] - close $fdzip - puts stderr "zip bytes: $zipbytes" - puts stderr "exezip bytes: [tell $fdout]" - close $fdout + ::punkboot::assemble_zipcat_image $raw_runtime $wrapvfs $buildfolder/$vfsname.new $buildfolder/$vfsname.zip } result ]} { set failmsg "creating zipcat image failed with msg: $result" puts stderr "creating image (zipcat) $targetkit failed" @@ -5958,18 +6308,34 @@ foreach vfstail $vfs_tails { #still - seems to be under 1s on a 2018 era i9 running in 2025 - which is not a big problem in this context. #using a filter in tasklist or get-command can make things a little faster - if {$::tcl_platform(platform) eq "windows"} { + #G-122: the tooling follows the TARGET, not the driving tclsh. + #A win32 kit is a windows process whatever baked it - and msys 'ps' + #sees only msys-descendant processes, so an msys-hosted build using + #'ps' would report a natively-launched kit as not running and then + #fail to replace it. tasklist/taskkill exec fine from an msys tclsh. + #When target and host process worlds differ (a cross-target bake) no + #local process can be holding the artifact - skip the sweep rather + #than run tooling that cannot see the target's processes anyway. + set target_proc_family [punkboot::lib::platform_process_family $rt_target] + set host_proc_family [punkboot::lib::platform_process_family $::punkboot::host_platform] + if {$target_proc_family eq "windows"} { set pscmd "tasklist" + set pid_field 1 } else { set pscmd "ps" + set pid_field 0 + } + set sweep_applicable [expr {$target_proc_family eq $host_proc_family}] + if {!$sweep_applicable} { + puts stdout " (skipping running-process sweep for $targetkit - target $rt_target processes are not visible to this $::punkboot::host_platform host)" } - #killing process doesn't apply to .kit build - if {$rtname ne "-"} { + #killing process doesn't apply to .kit build + if {$rtname ne "-" && $sweep_applicable} { #exec $pscmd | grep $targetkit ;#wrong - fails on at least some OSes because grep itself is in the list #exec ps | grep $targetkit | grep -v grep ;#exclude grep process if {![catch { - exec $pscmd + ::punkboot::exec_nativeargs [list $pscmd] } ps_lines]} { #we will process without grep for for now - may not be avail on windows? set still_running_lines [list] @@ -5982,11 +6348,7 @@ foreach vfstail $vfs_tails { #itself (e.g a copy of the target executable driving this build from #a path other than the deploy target, which the self-build guard #above doesn't catch). - if {$::tcl_platform(platform) eq "windows"} { - set ln_pid [lindex $ln 1] - } else { - set ln_pid [lindex $ln 0] - } + set ln_pid [lindex $ln $pid_field] if {$ln_pid eq [pid]} {continue} lappend still_running_lines $ln } @@ -5999,15 +6361,14 @@ foreach vfstail $vfs_tails { foreach ln $still_running_lines { puts stdout " $ln" - if {$::tcl_platform(platform) eq "windows"} { - set pid [lindex $ln 1] + set pid [lindex $ln $pid_field] + if {$target_proc_family eq "windows"} { if {$forcekill} { set killcmd [list taskkill /F /PID $pid] } else { set killcmd [list taskkill /PID $pid] } } else { - set pid [lindex $ln 0] #review! if {$forcekill} { set killcmd [list kill -9 $pid] @@ -6017,7 +6378,7 @@ foreach vfstail $vfs_tails { } puts stdout " pid: $pid (attempting to kill now using '$killcmd')" if {[catch { - exec {*}$killcmd + ::punkboot::exec_nativeargs $killcmd } errMsg]} { puts stderr "$killcmd returned an error:" puts stderr $errMsg diff --git a/src/modules/punk-999999.0a1.0.tm b/src/modules/punk-999999.0a1.0.tm index 2a1a8f06..515a60b3 100644 --- a/src/modules/punk-999999.0a1.0.tm +++ b/src/modules/punk-999999.0a1.0.tm @@ -9002,7 +9002,14 @@ namespace eval punk { runtime-artifact metadata target field. The raw Tcl platform-package identifiers are shown for comparison (punk::platform::normalize folds their version-dependent - aliases - amd64/aarch64/macos - into the canonical names)." + aliases - amd64/aarch64/macos - into the canonical names, and + folds the msys2 family - mingw64/mingw32/ucrt64/clang* - onto + the one 'msys' tag, since msys2's uname renders that token from + MSYSTEM rather than from the binary). + These are HOST names. What a name implies for build OUTPUT + (.exe suffixing, process tooling, which bin/runtime tier holds + a runtime) is a TARGET question src/make.tcl derives from the + name - a cygwin-family host targets win32-x86_64 by default." @values -min 0 -max 0 } proc platforms {context args} { diff --git a/src/modules/punk-buildversion.txt b/src/modules/punk-buildversion.txt index c303b26a..9a74b22b 100644 --- a/src/modules/punk-buildversion.txt +++ b/src/modules/punk-buildversion.txt @@ -1,6 +1,7 @@ -0.2.7 +0.2.8 #First line must be a semantic version number #all other lines are ignored. +#0.2.8 - 'help platforms' help text documents the G-122 msys2-family fold (mingw64/mingw32/ucrt64/clang* -> the one 'msys' tag, since msys2's uname renders that token from MSYSTEM rather than from the binary) and states that these are HOST names: what a name implies for build OUTPUT (.exe suffixing, process tooling, bin/runtime store tier) is a TARGET question src/make.tcl derives from it, a cygwin-family host targeting win32-x86_64 by default. Rendered table unchanged (it is generated from punk::platform::platforms, which gained cygwin-x86_64). #0.2.7 - new 'help platforms' topic (aliases: platform): canonical punkshell platform-dir names from punk::platform::platforms rendered with status/tiers/buildsuite/notes (buildsuite = zig-buildsuite runtime production, a separate axis from hosting), the running interpreter's lib-tier and runtime-tier names marked, and the raw Tcl platform-package identifiers (generic/identify) shown for comparison. Degrades cleanly when punk::platform is unavailable. All platform/platforms prefixes are mutually ambiguous so they fall through to command lookup (env/environment precedent). #0.2.6 - G-076: 'help tcl' renders the mitigated/mitigation buginfo axis (punk::lib 0.4.3+): a triggered check reporting mitigated keeps its severity level but displays 'warning level: (mitigated)' in subdued grey (term-grey foreground) instead of the level colour, followed by an indented 'mitigated: ' block when mitigation text is supplied. Unmitigated warnings render unchanged. #0.2.5 - G-045: ::punk::helptopic::define_docs converts from interim left-margin authoring to indented block-form values under the new punk::args @normalize directive (the constructed-definition normalization consumer proof): basehelp/topichelp/help_chunks-extra are braced indented blocks with a structural leading newline, the generated definitions declare @normalize, and the -unindentedfields declarations from punk 0.2.4 are dropped. Rendered 'i help'/'i help_chunks' output unchanged (verified aligned incl. the blank-line separator in help_chunks). diff --git a/src/modules/punk/platform-999999.0a1.0.tm b/src/modules/punk/platform-999999.0a1.0.tm index 75f1a5a3..9e092264 100644 --- a/src/modules/punk/platform-999999.0a1.0.tm +++ b/src/modules/punk/platform-999999.0a1.0.tm @@ -40,6 +40,29 @@ namespace eval punk::platform { # the darwin 'arm64' machine name to 'arm'; Apple silicon is 64-bit only, # so the arm token on macosx always means arm64. Elsewhere 'arm' is a # genuine 32-bit token and is preserved.) + # - os mingw64/mingw32/ucrt64/clang64/clangarm64 -> msys (G-122; see the + # MSYSTEM-variance note below) + # + #MSYSTEM VARIANCE (cygwin-family hosts on windows). An msys2-runtime tclsh + #reports tcl_platform(platform) 'unix' with an os string msys2's uname + #renders from the MSYSTEM environment variable, so ONE binary canonizes + #differently per launching shell - verified 2026-07-26 with msys2's + #/usr/bin/tclsh8.6: MSYSTEM=MSYS -> MSYS_NT-10.0-26200 (msys-x86_64), and + #MSYSTEM=MINGW64, =UCRT64 or unset (msys2 defaults it) -> MINGW64_NT-10.0-26200 + #(mingw64-x86_64). The MSYSTEM token names the shell that launched the + #process, not the binary's ABI: a genuinely MINGW-built tclsh is a native + #windows binary reporting os 'Windows NT' and canonizes as win32-*. So every + #msys2-runtime token folds to the one stable tag 'msys'. Real cygwin keeps + #its own tag 'cygwin' - a separate runtime dll and mount world, so a store + #tier serving one does not serve the other. Windows build numbers never + #reach a tag (platform::generic cuts the os at its first _ or -; normalize_os + #re-folds a raw uname-style string anyway). + #These are HOST canon names. What a tag implies for BUILD OUTPUT - .exe + #suffixing, which process tooling manages a running artifact, which + #bin/runtime store tier holds a runtime - is a TARGET question that + #src/make.tcl derives from the tag (G-122 host/target split): cygwin-family + #hosts default their build target to win32-x86_64, since the kits they bake + #are native windows binaries. #The special name 'macosx' (no cpu suffix) is the RUNTIME-tier convention for #universal (multi-arch) macOS binaries - punkbin and bin/runtime keep one #universal folder; the per-arch macosx-x86_64/macosx-arm64 names serve the @@ -83,7 +106,8 @@ namespace eval punk::platform { macosx-arm64 {status supported tiers {lib} buildsuite none notes "Apple silicon; would follow a macosx runtime arc"} freebsd-x86_64 {status supported tiers {runtime lib} buildsuite candidate notes ""} freebsd-arm64 {status supported tiers {runtime lib} buildsuite candidate notes "no artifacts yet"} - msys-x86_64 {status dormant tiers {lib} buildsuite none notes "msys/cygwin-built tclsh runtimes; utility under review"} + msys-x86_64 {status dormant tiers {runtime lib} buildsuite none notes "msys2-runtime tclsh on windows (posix personality, .exe files); host canon for every MSYSTEM shell - see normalize"} + cygwin-x86_64 {status recognized tiers {runtime lib} buildsuite none notes "cygwin-runtime tclsh on windows; separate dll/mount world from msys2, so a distinct tier"} openbsd-x86_64 {status recognized tiers {runtime} buildsuite none notes "hosted third-party runtimes possible"} netbsd-x86_64 {status recognized tiers {runtime} buildsuite none notes "hosted third-party runtimes possible"} dragonflybsd-x86_64 {status recognized tiers {runtime} buildsuite none notes "hosted third-party runtimes possible"} @@ -124,6 +148,45 @@ namespace eval punk::platform { return $platforms } + namespace eval argdoc { + variable PUNKARGS + lappend PUNKARGS [list { + @id -id ::punk::platform::normalize_os + @cmd -name punk::platform::normalize_os\ + -summary\ + "Fold an os token to its canonical punkshell spelling."\ + -help\ + "Fold a single os token (the half of an - platform + identifier) to its canonical punkshell spelling: + macos -> macosx + mingw32|mingw64|ucrt64|clang* -> msys + MSYS_NT- -> msys + CYGWIN_NT- -> cygwin + The msys2 family folds to one tag because msys2's uname renders + the os string from the MSYSTEM environment variable: the same + binary reports MSYS_NT or MINGW64_NT depending on which shell + launched it, while a genuinely MINGW-built (native windows) + tclsh reports 'Windows NT' and canonizes as win32. Raw + uname-style strings carrying a windows build number fold too - + a build number never belongs in a platform tag. + Unrecognized tokens pass through unchanged." + @values -min 1 -max 1 + os -type string -help\ + "os token, e.g the half of platform::generic's result." + }] + } + proc normalize_os {os} { + switch -glob -- [string tolower $os] { + macos {return macosx} + msys - msys_nt-* {return msys} + mingw32 - mingw64 - mingw_nt-* - mingw32_nt-* - mingw64_nt-* {return msys} + ucrt64 - ucrt64_nt-* {return msys} + clang32 - clang64 - clangarm64 - clang32_nt-* - clang64_nt-* - clangarm64_nt-* {return msys} + cygwin_nt-* {return cygwin} + } + return $os + } + namespace eval argdoc { variable PUNKARGS lappend PUNKARGS [list { @@ -137,6 +200,9 @@ namespace eval punk::platform { platform-dir name: cpu amd64 -> x86_64, aarch64 -> arm64 os macos -> macosx (Tcl platform 1.1.x renamed modern macOS) + os mingw64/mingw32/ucrt64/clang* -> msys (one stable tag for + the msys2 runtime family - the reported name follows MSYSTEM, + not the binary; see punk::platform::normalize_os) macosx cpu arm -> arm64 (platform::generic's arm* glob folds the darwin arm64 machine name to arm; Apple silicon is 64-bit only - non-macosx 'arm' stays 32-bit arm) @@ -150,18 +216,13 @@ namespace eval punk::platform { proc normalize {platform} { set parts [split $platform -] if {[llength $parts] < 2} { - #single-token names (e.g the universal 'macosx') - fold macos only - if {$platform eq "macos"} { - return macosx - } - return $platform + #single-token names (e.g the universal 'macosx') - fold the os token only + return [normalize_os $platform] } #os may itself contain a dash in principle - treat last token as cpu set cpu [lindex $parts end] set os [join [lrange $parts 0 end-1] -] - switch -- $os { - macos {set os macosx} - } + set os [normalize_os $os] switch -- $cpu { amd64 {set cpu x86_64} aarch64 {set cpu arm64} diff --git a/src/modules/punk/platform-buildversion.txt b/src/modules/punk/platform-buildversion.txt index be3e43d5..094ffe49 100644 --- a/src/modules/punk/platform-buildversion.txt +++ b/src/modules/punk/platform-buildversion.txt @@ -1,4 +1,5 @@ -0.1.0 +0.2.0 #First line must be a semantic version number #all other lines are ignored. +#0.2.0 - G-122 host/target platform split groundwork: new normalize_os folds the msys2 family (msys/mingw32/mingw64/ucrt64/clang*, plus raw MSYS_NT-*/MINGW*_NT-* uname strings carrying a windows build number) onto the one stable 'msys' tag and CYGWIN_NT-* onto 'cygwin' - msys2's uname renders that token from MSYSTEM, so one binary otherwise canonizes differently per launching shell while a genuinely MINGW-built (native) tclsh reports 'Windows NT' and is win32. normalize routes both its shapes through it. platforms table gains cygwin-x86_64 and moves msys-x86_64 to tiers {runtime lib} (third-party runtimes for these are representable as kit mapping entries with their own store tier). Module comments document the MSYSTEM variance and that .exe suffixing / process tooling / store tier are TARGET questions src/make.tcl derives from these HOST names. #0.1.0 - initial release (2026-07-22 platform-folder synchronization; G-105 groundwork): the canonical punkshell platform naming layer over Tcl's platform package - platforms (canonical records: status/tiers/buildsuite/notes; buildsuite = whether the punkshell zig buildsuites produce runtimes for the platform, deliberately a separate axis from the tiers a name organizes - punkbin-structured repos can host runtimes built by any mechanism, so runtime-tier hosting can coexist with buildsuite=none), normalize (amd64->x86_64, aarch64->arm64, macos->macosx, macosx arm->arm64), local ?-tier lib|runtime? (runtime tier collapses macosx per-arch to the universal macosx runtime-store name). Canon consumed by: punkbin platform folders, bin/runtime//, lib_tclX/vendorlib_tclX platform dirs, G-103 artifact metadata targets, 'help platforms' topic. The boot machinery (punkboot::lib platform_generic snips in punk_main.tcl/project_main.tcl/make.tcl) carries an inline copy of the normalization mapping - keep in sync. diff --git a/src/runtime/AGENTS.md b/src/runtime/AGENTS.md index ef211207..e557f10e 100644 --- a/src/runtime/AGENTS.md +++ b/src/runtime/AGENTS.md @@ -12,16 +12,19 @@ Houses the `mapvfs.config` that maps VFS payloads to platform runtimes, plus the ## Local Contracts - `mapvfs.config` defines which `src/vfs/*.vfs` folders combine with which runtime binaries (stored under `bin/runtime//`). +- Entry grammar: one line per runtime, ` {vfs ?kitname? ?kittype? ?targetplatform?} ...`. The file's own header comment is the user-facing spec; keep the two in step. +- **Target platform (4th element, G-122).** A kit config entry may name the canonical punkshell platform its runtime is for (`help platforms` lists the names). It decides which `bin/runtime//` tier the runtime is read from (macosx-* collapsing to the universal `macosx` folder), whether the runtime file and the built kit carry `.exe`, and whether the pre-deploy process sweep applies. It is a property of the RUNTIME: entries on one runtime line may repeat it but must not disagree (a conflict, or a name that is not an `-` platform-dir name, is a fatal config error). Omitted means the build host's default target - which for an msys2/cygwin-runtime tclsh is `win32-x86_64`, not that host's own canon. The live example is the `tclkit-902-Linux64-intel-dyn` entry (target `linux-x86_64`, suffixless `punkshell902` artifact). The schema's eventual home is the G-024 toml conversion; consumers read the parsed model (`::punkboot::lib::mapvfs_*`), never the file format. - Runtime executables are placed in `bin/runtime//` by the `bin/punk-runtime.cmd` helper, manually, or (G-103 family runtimes) copied from the suite_tcl90 `kit-family` build products under `src/buildsuites/_build/suite_tcl90/out/family/`. - The `_build/` subdirectory holds build intermediates; it can be safely deleted. ## Work Guidance -- When adding a new platform: create the runtime directory under `bin/runtime//`, add a VFS under `src/vfs/`, and update `mapvfs.config`. +- When adding a new platform: create the runtime directory under `bin/runtime//`, add a VFS under `src/vfs/`, and update `mapvfs.config` - declaring the target platform on the entry when it is not the build host's default. - Do not commit large binary runtimes to version control unless specifically required. ## Verification +- `tclsh src/make.tcl bakelist` resolves every entry: each row's runtime presence, target and store tier (add a kit name for the per-kit detail block). - `tclsh src/make.tcl bake` uses `mapvfs.config` correctly. - Built binaries launch and boot the Punk REPL. diff --git a/src/runtime/mapvfs.config b/src/runtime/mapvfs.config index c06860e2..68418a5e 100644 --- a/src/runtime/mapvfs.config +++ b/src/runtime/mapvfs.config @@ -1,12 +1,22 @@ #single line per runtime executable. Name of runtime followed by list of kit configuration lists. -#Each kit config list is from 1 to 3 elements +#Each kit config list is from 1 to 4 elements #1) .vfs folder with path relative to src folder. -#2) name of wrapped kit binary to create (default to rootname of .vfs folder e.g myapp.vfs will create a kit named myapp (or myapp.exe) on windows) +#2) name of wrapped kit binary to create (default to rootname of .vfs folder e.g myapp.vfs will create a kit named myapp (or myapp.exe) for a windows target) #3) Type of kit to create (zip or kit) - default kit. kit for metakit or vlerq based kit, zip for more modern zipkit. +#4) TARGET PLATFORM (G-122) - canonical punkshell platform-dir name ('help platforms' lists them). +# Defaults to the build host's own target: win32-x86_64 on windows INCLUDING from an +# msys2/cygwin-runtime tclsh, whose own canon (msys-x86_64) is a host identity, not a kit target. +# The declared target decides everything the entry emits: which store tier holds the runtime +# (bin/runtime//, macosx-* collapsing to the universal macosx folder), whether the runtime +# file and the built kit carry .exe, and which process tooling the pre-deploy sweep uses. +# It is a property of the RUNTIME: entries on one runtime line may repeat it but must not disagree. # # e.g The following creates 2 zipkits from the same runtime and 2 different .vfs folder # mytcl90kit.exe {mainapp.vfs myapp.exe zip} {test.vfs testapp.exe zip} # +# e.g a runtime kept in bin/runtime/linux-x86_64/ producing a suffixless linux kit: +# mylinuxruntime {myapp.vfs myapp kit linux-x86_64} +# #if runtime has no entry - it will only match a .vfs folder with a matching filename e.g runtime1.exe runtime1.vfs #Use a runtime with a name of dash (-) to build a .kit file from the .vfs folder using no runtime #e.g @@ -96,5 +106,9 @@ tclsh9.0.5-punk-bi.exe {punk9win_for_tkruntime.vfs punk9bi_beta zip} #testing wrong target kit type for runtime #tclkit902.exe {punk9win_for_tkruntime.vfs punkwrongruntime zip} -#only tclkit based runtime avail for linux for now - todo make zip-based runtime and install as artifact on gitea1.intx.com.au -tclkit-902-Linux64-intel-dyn {punk9linux.vfs punkshell902 kit} +#only tclkit based runtime avail for linux for now - todo make zip-based runtime and install as artifact on gitea1.intx.com.au +#Cross-target entry (G-122): declares linux-x86_64 so the runtime is read from +#bin/runtime/linux-x86_64/ with no .exe suffix and the kit deploys as bin/punkshell902. +#Before the target declaration this row read runtime=missing from a windows host, +#because the runtime was looked for as tclkit-902-Linux64-intel-dyn.exe in the windows store. +tclkit-902-Linux64-intel-dyn {punk9linux.vfs punkshell902 kit linux-x86_64} diff --git a/src/tests/shell/AGENTS.md b/src/tests/shell/AGENTS.md index 4aa83454..d4eb309e 100644 --- a/src/tests/shell/AGENTS.md +++ b/src/tests/shell/AGENTS.md @@ -16,6 +16,7 @@ Tests for shell-level behavior, command-line execution, and stdin/stdout interac - Avoid relying on installed packages or root-level build outputs unless the test explicitly targets a built executable. - `testsuites/punkexe/` targets a built punk executable: resolved from `env(PUNK_SHELL_TEST_EXE)`, else `/bin/punk902z.exe`, else `/bin/punkshell902`; tests auto-skip via the `punkexeavailable` constraint when none is found. - Tests that spawn the built executable must be hang-proof: use the event-loop `punk_run` pattern (pipe stdin, half-close for EOF, timeout then force-kill) rather than plain `exec`, because known failure modes leave the child waiting on a reopened console. +- Three punkexe suites drive the WORKING TREE's `src/make.tcl` through the built executable's `script` subcommand with output captured through a pipe, pinning its reporting surfaces rather than running builds: `maketclcolour.test` (G-113 piped colour policy), `maketclbakelist.test` (G-121 kit matrix report + the selective-bake no-build guard) and `maketclplatform.test` (G-122 host/target platform split - the `check` derivation line, per-kit store tier/target, and cross-target artifact naming). Their per-kit row expectations are characterization of the current `src/runtime/mapvfs.config` matrix, so a deliberate config change legitimately updates them. `maketclplatform.test` additionally carries self-gating cygwin-host tests: they discover an msys2/cygwin-runtime tclsh (`env(PUNK_MSYS_TCLSH)`, else the usual msys2/cygwin roots) and PROBE it - a candidate only counts when it really reports `tcl_platform(platform)` `unix` with a cygwin-family os string - then assert that host derives the same win32 target, store and kit set as the native run. Such a tclsh is a POSIX Tcl: pass it a `/c/...`-spelled script path, never `C:/...`. - `testsuites/punkexe/shellexit.test` guards piped-stdin termination behaviour of the built executable (no `invalid thread handle` on shutdown races, no console-reopen hang on eof/error paths, exitcode propagation); do not weaken these tests - a red result means a regression in punk::repl shutdown or app-punkshell eof handling. - `testsuites/punkexe/shellnavns.test` proves the namespace-navigation session contract end-to-end over a piped 'src'-mode session (dev modules - deliberate, so current punk::repl/nav source is exercised without a rebuild): n/ sets ns_current, the current namespace continues into `subshell punk` and back out (currently via the documented first-subshell shared-code-interp asymmetry - the SHARED marker pins it; the repl::start seeding template for genuinely separate interps is proven in modules/punk/repl nscurrent.test), the punk::repl 0.5.2 stray-namespace fix holds live, and the session exits cleanly. KNOWN GAP pinned there: piped submissions evaluate at :: despite ns_current (interactive and module-level inscope behaviour differ) - investigate alongside G-038. - `testsuites/punkexe/scriptexec.test` covers the `script` subcommand (app-punkscript, G-015): piped execution + honest exit codes, result echo, file-form argv, the `lib:` scriptlib-resolution mechanism (incl. punk::path 0.3.0 extensionless-tcl matching: first-line identification, called-spelling precedence over a `.tcl` sibling, sh-trampoline shebang acceptance, and the non-tcl skip note on not-found), and the tclsh-matching Tk main-loop behaviour. The `lib:` and Tk cases resolve committed fixtures under `scriptlib/_punktest/` (test-owned; a `lib:`-resolvable fixture cannot live in a tcltest temp dir since resolution only searches scriptlib locations relative to the exe). Tk cases are gated by the `punktk` constraint, auto-detected by probing whether the kit can load Tk. diff --git a/src/tests/shell/testsuites/punkexe/maketclplatform.test b/src/tests/shell/testsuites/punkexe/maketclplatform.test new file mode 100644 index 00000000..e08b286e --- /dev/null +++ b/src/tests/shell/testsuites/punkexe/maketclplatform.test @@ -0,0 +1,245 @@ +package require tcltest + +#Piped characterization of make.tcl's HOST/TARGET platform split (goal G-122). +# +#make.tcl's kit surfaces are keyed by the TARGET platform an artifact is built for, +#not by the driving tclsh's own personality. This file pins the observable contract: +# - 'check' reports the derivation on one stable key=value line +# (host / target / store tier / exe suffix / process tooling) +# - 'bakelist ' detail names the store tier and target of each kit, and a +# default-target kit reports "(host default)" +# - a mapvfs.config entry declaring its own target platform resolves its runtime in +# THAT platform's store tier, names its artifact with that platform's executable +# convention (no .exe for linux-x86_64), and carries a target= note in the report +# - (env-gated) an msys2/cygwin-runtime tclsh - which reports tcl_platform(platform) +# 'unix' on windows and canonizes as msys-x86_64 - drives the SAME win32-x86_64 +# kit set, with identical names and store addressing, as the native run +# +#The per-kit row pins are characterization of the current src/runtime/mapvfs.config +#matrix (punk91 = tclsfe-x64 + punk9wintk903.vfs zip, default target; punkshell902 = +#tclkit-902-Linux64-intel-dyn + punk9linux.vfs kit, target linux-x86_64) - a deliberate +#config change legitimately updates them. +# +#Target executable resolved from env(PUNK_SHELL_TEST_EXE), else /bin/punk902z.exe +#then /bin/punkshell902. Skipped (constraint punkexeavailable) if none found. +#The msys host tests additionally need a cygwin-family tclsh: env(PUNK_MSYS_TCLSH), else a +#scan of the usual msys2 roots, VERIFIED by probing that it really reports platform unix +#with a cygwin-family os string (constraint cygwinfamilytclsh). + +namespace eval ::testspace { + namespace import ::tcltest::* + + variable testdir [file dirname [file normalize [info script]]] + #/src/tests/shell/testsuites/punkexe -> 5 levels up to + variable projectroot [file normalize [file join $testdir .. .. .. .. ..]] + variable maketcl [file join $projectroot src make.tcl] + + variable punkexe "" + if {[info exists ::env(PUNK_SHELL_TEST_EXE)] && $::env(PUNK_SHELL_TEST_EXE) ne ""} { + set punkexe [file normalize $::env(PUNK_SHELL_TEST_EXE)] + } else { + foreach candidate [list [file join $projectroot bin punk902z.exe] [file join $projectroot bin punkshell902]] { + if {[file exists $candidate]} { + set punkexe $candidate + break + } + } + } + testConstraint punkexeavailable [expr {$punkexe ne "" && [file exists $punkexe]}] + + variable maketcl_run_timeout_ms 60000 + + variable runstate + array set runstate {} + + proc maketcl_run_read {chan} { + variable runstate + append runstate(output) [read $chan] + if {[chan eof $chan]} { + chan event $chan readable {} + set runstate(done) eof + } + } + + #Run a command line with output captured through a pipe (stdin half-closed for + #immediate EOF - make.tcl must never wait on stdin for these subcommands). + #Returns dict: timedout 0|1, exitcode , output . + proc piped_run {cmdline} { + variable runstate + variable maketcl_run_timeout_ms + array unset runstate + set runstate(output) "" + set runstate(done) "" + + set chan [open |[list {*}$cmdline 2>@1] r+] + chan configure $chan -blocking 0 -translation binary + catch {chan close $chan write} ;#no stdin for the child - immediate EOF + set timerid [after $maketcl_run_timeout_ms [list set [namespace current]::runstate(done) timeout]] + chan event $chan readable [list [namespace current]::maketcl_run_read $chan] + while {$runstate(done) eq ""} { + vwait [namespace current]::runstate(done) + } + after cancel $timerid + set timedout [expr {$runstate(done) eq "timeout"}] + set exitcode "" + if {$timedout} { + catch {exec {*}[auto_execok taskkill] /F /T /PID [lindex [pid $chan] 0]} + catch {chan close $chan} + } else { + chan configure $chan -blocking 1 + if {[catch {chan close $chan} errdata errdict]} { + set exitcode [lindex [dict get $errdict -errorcode] end] + } else { + set exitcode 0 + } + } + return [dict create timedout $timedout exitcode $exitcode output $runstate(output)] + } + + #make.tcl driven by the built punk executable's 'script' subcommand + proc maketcl_run {cmdargs} { + variable punkexe + variable maketcl + return [piped_run [list $punkexe script $maketcl {*}$cmdargs]] + } + + #make.tcl driven directly by an arbitrary tclsh. A cygwin-family tclsh is a POSIX + #Tcl: 'C:/x/make.tcl' is not absolute to it and would be resolved relative to the + #cwd (make.tcl diagnoses that case), so the script path is spelled the way that + #shell spells paths - exactly what a user in an msys/cygwin shell types. + proc maketcl_run_with {interp cmdargs {posixpaths 0}} { + variable maketcl + set script $maketcl + if {$posixpaths && [regexp {^([A-Za-z]):[/\\](.*)$} $script -> drive rest]} { + set script "/[string tolower $drive]/[string map {\\ /} $rest]" + } + return [piped_run [list $interp $script {*}$cmdargs]] + } + + #Kit data rows of a bakelist report, whitespace-normalized for comparison. + proc bakelist_rows {text} { + set rows [list] + foreach ln [split [string map [list \r\n \n \r \n] $text] \n] { + if {[regexp {^[A-Za-z][\w.-]*\s+(kit|zip|zipcat|cookfs|cookit)\s+\S} $ln]} { + lappend rows [regsub -all {\s+} [string trim $ln] " "] + } + } + return $rows + } + + # -- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- + # cygwin-family (msys2/cygwin runtime) tclsh discovery for the host-parity tests + # -- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- + variable cygwintclsh "" + proc find_cygwinfamily_tclsh {} { + set candidates [list] + if {[info exists ::env(PUNK_MSYS_TCLSH)] && $::env(PUNK_MSYS_TCLSH) ne ""} { + lappend candidates $::env(PUNK_MSYS_TCLSH) + } else { + set roots [list] + if {[info exists ::env(USERPROFILE)]} { + lappend roots [file join [string map {\\ /} $::env(USERPROFILE)] scoop apps msys2 current usr bin] + } + lappend roots C:/msys64/usr/bin C:/tools/msys64/usr/bin C:/cygwin64/bin + foreach root $roots { + foreach name {tclsh8.6.exe tclsh.exe} { + set cand [file join $root $name] + if {[file exists $cand]} { + lappend candidates $cand + } + } + } + } + #An existing tclsh is not enough - it must really be a posix-personality + #runtime on windows (that is the whole point of the host/target split). + foreach cand $candidates { + set probe {puts "PROBE $::tcl_platform(platform) [lindex $::tcl_platform(os) 0]"} + if {[catch {exec $cand << $probe} out]} { + continue + } + if {[regexp {PROBE\s+unix\s+(\S+)} $out -> osname]} { + if {[string match -nocase "*_NT-*" $osname] || [string match -nocase "cygwin*" $osname]} { + return $cand + } + } + } + return "" + } + if {$::tcl_platform(platform) eq "windows" || [string match "*_NT-*" $::tcl_platform(os)]} { + catch {set cygwintclsh [find_cygwinfamily_tclsh]} + } + testConstraint cygwinfamilytclsh [expr {$cygwintclsh ne ""}] + + #added 2026-07-26 (agent, G-122) + test maketcl_check_platform_report {make.tcl check reports the host/target derivation on one key=value line; a win32 kit host targets win32-x86_64} -constraints {punkexeavailable} -body { + set r [maketcl_run {check}] + set out [dict get $r output] + set result [list] + lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode] + lappend result line [regexp -line {^platform \(G-122\): host=win32-x86_64 target=win32-x86_64 store=bin/runtime/win32-x86_64 exe-suffix=\.exe process-tooling=windows\s*$} $out] + set result + } -result {timedout 0 exitcode 0 line 1} + + #added 2026-07-26 (agent, G-122) + test maketcl_bakelist_default_target_detail {a default-target kit resolves its runtime in the host default store tier and reports target (host default)} -constraints {punkexeavailable} -body { + set r [maketcl_run {bakelist punk91}] + set out [dict get $r output] + set result [list] + lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode] + lappend result store [regexp {(?n)^\s+runtime file:\s+bin/runtime/win32-x86_64/tclsfe-x64\.exe \(present\)} $out] + lappend result target [regexp {(?n)^\s+target:\s+win32-x86_64 \(host default\)} $out] + #a default-target row carries no target= note (the column is for the minority case) + lappend result nonote [expr {![regexp {(?n)^punk91\s.*target=} $out]}] + set result + } -result {timedout 0 exitcode 0 store 1 target 1 nonote 1} + + #added 2026-07-26 (agent, G-122) + test maketcl_bakelist_crosstarget_entry {a mapvfs entry declaring a non-default target lists with a target= note, resolves its own store tier and names a suffixless artifact} -constraints {punkexeavailable} -body { + set r [maketcl_run {bakelist punkshell902}] + set out [dict get $r output] + set result [list] + lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode] + lappend result row [regexp {(?n)^punkshell902\s+kit\s+tclkit-902-Linux64-intel-dyn\s+punk9linux\.vfs\s+\S+\s+target=linux-x86_64} $out] + lappend result store [regexp {(?n)^\s+runtime file:\s+bin/runtime/linux-x86_64/tclkit-902-Linux64-intel-dyn \(present\)} $out] + lappend result target [regexp {(?n)^\s+target:\s+linux-x86_64 \(declared in mapvfs\.config\)} $out] + #linux target -> no .exe on the build product or the deployed artifact + lappend result nosuffix [regexp {(?n)^\s+build product: src/_build/punkshell902 } $out] + lappend result nosuffix2 [regexp {(?n)^\s+deployed:\s+bin/punkshell902 } $out] + set result + } -result {timedout 0 exitcode 0 row 1 store 1 target 1 nosuffix 1 nosuffix2 1} + + #added 2026-07-26 (agent, G-122) + test maketcl_check_platform_report_cygwin_host {a cygwin-family host canonizes as msys/cygwin but still targets - and addresses the store of - win32-x86_64} -constraints {cygwinfamilytclsh} -body { + variable cygwintclsh + set r [maketcl_run_with $cygwintclsh {check} 1] + set out [dict get $r output] + set result [list] + lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode] + lappend result hostcanon [regexp -line {^platform \(G-122\): host=(msys|cygwin)-\S+ } $out] + lappend result targetline [regexp -line {^platform \(G-122\): host=\S+ target=win32-x86_64 store=bin/runtime/win32-x86_64 exe-suffix=\.exe process-tooling=windows\s*$} $out] + #the host really is the posix-personality one (this is what used to derive a non-windows store) + lappend result posixhost [regexp {platform='unix'} $out] + set result + } -result {timedout 0 exitcode 0 hostcanon 1 targetline 1 posixhost 1} + + #added 2026-07-26 (agent, G-122) + test maketcl_bakelist_cygwin_host_parity {an msys/cygwin-runtime tclsh reports the identical kit set - names, runtimes, targets and store addressing - as the native host} -constraints {punkexeavailable cygwinfamilytclsh} -body { + variable cygwintclsh + set native [maketcl_run {bakelist}] + set cyg [maketcl_run_with $cygwintclsh {bakelist} 1] + set result [list] + lappend result timedout [expr {[dict get $native timedout] || [dict get $cyg timedout]}] + lappend result exitcodes [list [dict get $native exitcode] [dict get $cyg exitcode]] + set nrows [bakelist_rows [dict get $native output]] + set crows [bakelist_rows [dict get $cyg output]] + lappend result haverows [expr {[llength $nrows] > 1}] + lappend result identical [expr {$nrows eq $crows}] + #both must address the same store folder in the report header + lappend result store [expr {[regexp {runtimes: bin/runtime/win32-x86_64} [dict get $native output]] + && [regexp {runtimes: bin/runtime/win32-x86_64} [dict get $cyg output]]}] + set result + } -result {timedout 0 exitcodes {0 0} haverows 1 identical 1 store 1} + + cleanupTests +} +namespace delete ::testspace diff --git a/src/vfs/_config/project_main.tcl b/src/vfs/_config/project_main.tcl index df81c784..9c84a78f 100644 --- a/src/vfs/_config/project_main.tcl +++ b/src/vfs/_config/project_main.tcl @@ -124,16 +124,29 @@ apply { args { return "${plat}-${cpu}" } + proc platform_normalize_os {os} { + #INLINE COPY of punk::platform::normalize_os - see platform_punk below. + switch -glob -- [string tolower $os] { + macos {return macosx} + msys - msys_nt-* {return msys} + mingw32 - mingw64 - mingw_nt-* - mingw32_nt-* - mingw64_nt-* {return msys} + ucrt64 - ucrt64_nt-* {return msys} + clang32 - clang64 - clangarm64 - clang32_nt-* - clang64_nt-* - clangarm64_nt-* {return msys} + cygwin_nt-* {return cygwin} + } + return $os + } proc platform_punk {} { #canonical punkshell platform-dir name: platform_generic normalized. #INLINE COPY of punk::platform::normalize (src/modules/punk/platform-*.tm; #'help platforms' documents the canon) - the boot stage cannot package #require, so keep this mapping in sync with that module: - #amd64->x86_64, aarch64->arm64, macos->macosx, macosx arm->arm64. + #amd64->x86_64, aarch64->arm64, macos->macosx, macosx arm->arm64, + #msys2 family (mingw64/mingw32/ucrt64/clang*) -> msys, CYGWIN_NT -> cygwin. set parts [split [platform_generic] -] set cpu [lindex $parts end] set os [join [lrange $parts 0 end-1] -] - if {$os eq "macos"} {set os macosx} + set os [platform_normalize_os $os] switch -- $cpu { amd64 {set cpu x86_64} aarch64 {set cpu arm64} diff --git a/src/vfs/_config/punk_main.tcl b/src/vfs/_config/punk_main.tcl index bb981307..f33cba16 100644 --- a/src/vfs/_config/punk_main.tcl +++ b/src/vfs/_config/punk_main.tcl @@ -219,16 +219,29 @@ apply { args { return "${plat}-${cpu}" } + proc platform_normalize_os {os} { + #INLINE COPY of punk::platform::normalize_os - see platform_punk below. + switch -glob -- [string tolower $os] { + macos {return macosx} + msys - msys_nt-* {return msys} + mingw32 - mingw64 - mingw_nt-* - mingw32_nt-* - mingw64_nt-* {return msys} + ucrt64 - ucrt64_nt-* {return msys} + clang32 - clang64 - clangarm64 - clang32_nt-* - clang64_nt-* - clangarm64_nt-* {return msys} + cygwin_nt-* {return cygwin} + } + return $os + } proc platform_punk {} { #canonical punkshell platform-dir name: platform_generic normalized. #INLINE COPY of punk::platform::normalize (src/modules/punk/platform-*.tm; #'help platforms' documents the canon) - the boot stage cannot package #require, so keep this mapping in sync with that module: - #amd64->x86_64, aarch64->arm64, macos->macosx, macosx arm->arm64. + #amd64->x86_64, aarch64->arm64, macos->macosx, macosx arm->arm64, + #msys2 family (mingw64/mingw32/ucrt64/clang*) -> msys, CYGWIN_NT -> cygwin. set parts [split [platform_generic] -] set cpu [lindex $parts end] set os [join [lrange $parts 0 end-1] -] - if {$os eq "macos"} {set os macosx} + set os [platform_normalize_os $os] switch -- $cpu { amd64 {set cpu x86_64} aarch64 {set cpu arm64}