diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index c231adf5..43e16947 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -86,6 +86,7 @@ Two layers with a deliberate dependency direction (the class never depends on th - **punkcheck** records every install/delete as events in per-folder `.punkcheck` directories - the basis for skip/copy change detection, superseded-module pruning and provenance. Single OO record lifecycle (G-094, achieved) with atomic saves and advisory event-scoped locking for concurrent writers (G-095, achieved). - **Provenance gates.** Build/promotion commands warn on uncommitted `src/` changes (column-0 `PROVENANCE-WARNING:` token, `-dirty-abort` for strict mode); `vendorupdate` warns for dirty source-project checkouts. - **Boot-precondition gate (G-125).** A bake refuses a kit whose merged `.vfs` supplies no tcl library rather than deploying an artifact that cannot initialise: the kit lands in `FAILED KITS` and nothing is written, so the previously deployed `bin/` survives. Structural and non-executing (so cross-target kits are covered), reading the merged tree rather than the extraction outcome. The predicate is `punkboot::utils::vfs_boot_library_report`, called through the same guarded require as the provenance check so a stale bootsupport snapshot degrades it to a notice; `make.tcl check` reports ACTIVE/UNAVAILABLE. Sources: `src/AGENTS.md`, `src/modules/punkboot/utils-999999.0a1.0.tm`. +- **Payload/target consistency checks (G-133), both advisory.** At the same post-merge seam, a bake classifies each binary library in the merged tree by header (`punkboot::utils::binary_arch_classify` - PE/ELF/Mach-O, honest unknowns) against the kit's target platform; wrong-arch libraries outside platform-discriminated subdirs (canonical `-` names, vendor spellings like `win-x64`) earn recapped `BUILD-WARNING`s. After assembly, a kit declaring smoke-require packages (`mapvfs.config` 5th entry element) has each one plain-`package require`d inside the freshly built artifact via its tclsh subcommand - the only check that sees resolution-order defects (wrong-arch version shadowing); cross-target kits skip with a stated reason. Same guarded-require degradation; `make.tcl check` reports both. Neither guarantees statics, pure-tcl packages with binary deps, or version preference beyond the declared smoke set. Sources: `src/AGENTS.md`, `src/runtime/AGENTS.md`, `src/modules/punkboot/utils-999999.0a1.0.tm`. - **Buildsuites and the kit family.** `src/buildsuites/suite_tcl90/` builds Tcl/Tk/tcllib from source with a pinned zig toolchain and produces the runtime kit family (plain / punk / bi) plus artifact metadata (G-096-G-117 era: see archived goals G-096, G-098, G-102, G-103, G-107); artifacts publish to the punkbin repo that `bin/punk-runtime.cmd` fetches from. `src/buildsuites/suite_tcl86/` is the 8.6 sibling (G-099 + G-100, both achieved): a Tcl 8.6 windows runtime (static + dynamic shells, `tcl86t.dll`, on-disk lib tree - no zipfs, so no self-contained kit) with thread 2.8 + tclvfs + Tk 8.6 + tklib + tcllib(+tcllibc critcl accelerators) companions, gated against the 8.6 core/thread/tclvfs testsuites with tcllib/tklib on a record tier and an opt-in `test-tk`; punkshell's own runtests is censused on that runtime at parity with a same-day native-8.6 baseline. Suite child shells scrub `TCL__TM_PATH` as well as TCLLIBPATH/TCL_LIBRARY/TK_LIBRARY - without it a census measures the machine's module trees. The 8.6 kit container strategy remains in-flux - see "In-flux areas". - **Project generation.** `dev project.new` composes thin layouts from `src/project_layouts/` (overlay chain with `.anti` deletion markers and `name@base` derivation - G-087, achieved) and injects bootsupport modules from the generating shell at generation time. - **Workflow overview.** `tclsh src/make.tcl workflow` prints the embedded ASCII data-flow overview of the build/release pipeline, with its own update contract in `src/AGENTS.md`. This section deliberately summarises rather than copies it. diff --git a/CHANGELOG.md b/CHANGELOG.md index 591f117d..ddc89d2f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,18 @@ The latest `## [X.Y.Z]` header must match the `version` field in `punkproject.to Entries are newest-first; one bullet per notable change. See the root `AGENTS.md` "Project Versioning" section for the bump policy. +## [0.28.2] - 2026-07-27 + +- Bakes verify payload/target consistency (G-133, both checks advisory - recapped + BUILD-WARNINGs, the kit still builds and deploys): a structural binary-arch scan of + each merged kit tree (PE/ELF/Mach-O headers vs the kit's target platform, platform + subdirs like win32-ix86/ and vendor spellings like win-x64/ exempt, cross-target kits + included) plus a post-build smoke-require probe (packages declared as a mapvfs.config + 5th entry element are plain-'package require'd inside the freshly built artifact via + its tclsh subcommand - the only check that sees wrong-arch version shadowing). + `make.tcl check` reports both. First real sweep flagged zint.dll (32-bit) carried by + the x64 tcl8 kits punk86/punkbi/punksys - a genuine latent load failure. + ## [0.28.1] - 2026-07-27 - The win32-ix86 verification kits (punkluck86 / punk91ix86) wrap new MINIMAL payloads diff --git a/bin/AGENTS.md b/bin/AGENTS.md index 7b7cab47..669510ec 100644 --- a/bin/AGENTS.md +++ b/bin/AGENTS.md @@ -133,6 +133,23 @@ and `tclsh9.1b0-tclsfe.exe` (modern `//zipfs:/app`; apnadkarni/tcl-sfe releases) wrapped as `punkluck86` / `punk91ix86` in `src/runtime/mapvfs.config`. Characterization: `src/tests/shell/testsuites/punkexe/kitmountpoint.test`. +### What a bake verifies about a deployed kit (G-125, G-133) + +A `bin/` that a bake deployed has passed the G-125 boot-precondition gate (its +merged vfs supplies a tcl library - a kit failing that is never deployed) and the +G-133 advisory checks have run: a structural binary-arch scan of its payload (a +wrong-arch `.dll`/`.so`/`.dylib` outside a platform-discriminated subdir earned a +recapped `BUILD-WARNING` - advisory, so heed the bake recap before trusting a warned +kit) and, when its `src/runtime/mapvfs.config` entry declares smoke-require packages +and the kit is host-runnable, a plain `package require` of each inside the built +artifact via its tclsh subcommand. What that does NOT guarantee: statically linked +packages and pure-tcl packages with binary dependencies are invisible to the arch +scan; package-resolution outcomes (a wrong-arch higher-versioned copy shadowing a +working one) are visible only to the smoke probe and only for the declared package +set; and an unclassifiable binary is silence, not a pass. A kit whose bake recap +showed no `BUILD-WARNING`s still only proves what those checks measure - not that +every payload package loads. + ### Runtime fetch/selection (`punk-runtime.cmd`) `bin/punk-runtime.cmd {fetch|list|use|run}` manages the plain runtimes under diff --git a/punkproject.toml b/punkproject.toml index d744607c..2bab8e25 100644 --- a/punkproject.toml +++ b/punkproject.toml @@ -1,4 +1,4 @@ [project] name = "punkshell" -version = "0.28.1" +version = "0.28.2" license = "BSD-2-Clause" diff --git a/src/AGENTS.md b/src/AGENTS.md index 001fdb47..f80cc015 100644 --- a/src/AGENTS.md +++ b/src/AGENTS.md @@ -72,6 +72,7 @@ Recovery after a wrong path guess: - make.tcl subcommands and flags are punk::args-declared (G-030): `tclsh src/make.tcl help ?subcommand?` or ` -help` shows tabled usage; invalid arguments produce punk::args usage errors. The dispatch degrades to plain scan/help when the bootsupport punk::args (or its rendering stack) is stale or unavailable; `PUNKBOOT_PLAIN=1` forces the degraded mode for troubleshooting. The `::punkboot::argdoc` definitions are constructed (string-built) and use the G-045 authoring mechanisms — block-form help values + `@normalize`, explicit `@form -synopsis` — see src/modules/AGENTS.md "punk::args definition authoring ergonomics" before editing them or authoring similar definitions. - Uncommitted `make.tcl`-generated outputs may be batched into one commit, regardless of how many `make.tcl` invocations produced them. This covers punkcheck-managed build outputs that are VCS-tracked: `src/bootsupport/`, `_vfscommon.vfs/modules` + `/lib`, declared per-kit `*.vfs/lib_tcl/` subfolders, the thin-layout sync copies (`src/project_layouts/vendor/punk//src/{make.tcl,build.tcl}` + bootsupport manifest copies), and the module-shipped layout payload under `src/modules/punk/mix/#modpod-templates-*/templates/project_layouts/` (G-087: synced from `src/project_layouts` - never hand-edit the modpod copy). (Root `modules/`, `lib/`, `modules_tcl/`, `lib_tcl/` are gitignored and never committed.) Each module's old→new version rename is atomic; a regeneration reflects one build-output refresh, not independent hand-edits. This is a carve-out from generic "split aggressively" commit guidance for punkcheck-managed build outputs only — it does not cover user-curated VFS content (`_config/`, per-kit `*.vfs/` outside declared `lib_tcl` subfolders, `_aside/`, `mkzipfix.vfs`, `_vfscommon.vfs/doc`), which commits separately per its own concerns. - **Boot-precondition gate (G-125).** A bake refuses any kit whose merged `.vfs` supplies no tcl library: the kit is listed under `FAILED KITS` with a reason naming the cause, and NOTHING is written - no `src/_build/`, no deploy, and the previously deployed `bin/` is left byte-identical. This is deliberately a failure rather than a warning: the deploy step deletes the old kit before copying the new one, so the pre-gate behaviour replaced a working shell with an artifact that died at startup with `application-specific initialization failed: Cannot find a usable init.tcl`. The check is structural and non-executing (so it covers cross-target kits too): `init.tcl` in `tcl_library/`, `lib/tcl./` or `tcl./` - the third for runtimes whose archive mounts at the executable's own path rather than `//zipfs:/app`, so `[info library]` is `/tcl8.6` (the androwish/undroidwish zipfs backport for 8.6 does this) - plus one companion file a real Tcl library carries beside it (`tm.tcl`, `package.tcl`, `auto.tcl`, `clock.tcl`, `history.tcl`, `word.tcl`) or an `encoding/` directory - without that second test the `lib/BWidget1.10.1/init.tcl` every punkshell kit carries would answer for a tcl library it cannot provide. It reads the MERGED tree rather than asking whether extraction ran, so a `.vfs` that legitimately supplies its own tcl library (`src/vfs/punk8_statictwapi.vfs`, `src/vfs/punk9test.vfs`) builds and deploys unchanged. **Remedy when a kit is refused:** the usual cause is that the runtime's own payload could not be extracted - read the recapped `BUILD-WARNING` naming what was tried, then either fix extraction (a driving tclsh with no zipfs needs punk::zip >= 0.2.0 in bootsupport - see the extraction bullet above), point the kit at a runtime that carries a payload, or have the kit's `.vfs` supply a tcl library itself. The predicate is `punkboot::utils::vfs_boot_library_report` called through a guarded require, exactly as the dirty-src provenance check is: a stale or missing bootsupport snapshot degrades the gate to a `NOTE` rather than failing every kit. `tclsh src/make.tcl check` reports whether the gate is `ACTIVE` or `UNAVAILABLE`. +- **Payload/target consistency checks (G-133), both ADVISORY** - warnings are recapped `BUILD-WARNING`s and the kit still builds and deploys (unlike the G-125 gate). (a) **Binary-arch scan**, at the same post-merge seam as the gate: every binary library (`*.dll`/`*.so`/`*.dylib`) in the merged tree is classified by header bytes (`punkboot::utils::binary_arch_classify` - PE Machine field, ELF e_machine, Mach-O incl. universal; unclassifiable content is honestly `unknown` and never warned about) against the kit's target platform. Structural and non-executing, so it runs for cross-target kits too (measured ~30-140ms per kit, avg ~93ms across the 12 assembled trees). Libraries under a platform-discriminated subdir - canonical `-` platform-dir names, the universal `macosx` folder, or a recognised vendor spelling (`win-x64` etc; the lists are namespace variables in `punkboot::utils`) - are exempt: multi-arch payloads are legitimate there. First real sweep (2026-07-27) found a true latent instance: `zint.dll` 2.13.0 in `src/vfs/punk8win.vfs/lib_tcl8/` is 32-bit (i386) and can never load in the x64 tcl8 kits (punk86/punkbi/punksys) that carry it - those bakes warn until the payload is fixed. (b) **Smoke-require probe**: a kit declaring packages in `src/runtime/mapvfs.config` (5th entry element - see `src/runtime/AGENTS.md`) has each one plain-`package require`d INSIDE the freshly built artifact via its tclsh subcommand, which exits cleanly even on runtimes whose full repl teardown is fragile. This is the only check that observes actual package RESOLUTION - the punkluck86 2026-07-27 incident class (a higher-versioned wrong-arch Thread shadowing the runtime's working copy on plain require) is invisible to structural checks because the working file exists too. Failures are recapped naming kit, package and the actual error; cross-target kits skip with a stated reason; undeclared kits run nothing new; the probe only runs when the kit actually rebuilds. **What the checks do NOT guarantee:** statically linked packages, pure-tcl packages whose binary dependency lives elsewhere, and version-preference outcomes are visible only to the smoke probe - and only for the declared package set; the arch scan proves nothing about loadability beyond architecture, and an `unknown` classification is silence, not a pass. Both degrade to a NOTE when the bootsupport `punkboot::utils` snapshot is stale (same guarded require as the G-125 gate); `tclsh src/make.tcl check` reports the scan's ACTIVE/UNAVAILABLE state and the currently declared smoke-require kits. - Kit bakes consume the punk-runtime WORKING COPIES under `bin/runtime//`: `make.tcl bake`/`bakehouse` emit a `BUILD-WARNING:` (recapped at end of run) when a wrapped runtime's beside-toml revision is older than an `-r` artifact present in the same folder - the forgot-to-switch-back guard for deliberate `punk-runtime use ` testing excursions. Heed it before trusting freshly baked family kits; `bin/punk-runtime.cmd use >` (`.exe` suffix optional) re-materializes the working copy. - `tclsh src/make.tcl bakelist ?kitname ...?` (G-121) reports the kit outputs configured in `src/runtime/mapvfs.config`: kit name, kit type, runtime (with presence in the runtime store), vfs folder, and the deployed state of `bin/` vs the `src/_build` build product (`current|stale|absent|nobuild`), with anomalies (`runtime=missing`, `vfs=missing`, `rtrev=r` materialization staleness) and the cross-target marker `target=` in a trailing notes column. A nonexistent store tier is flagged loudly per tier actually referenced (header `(FOLDER MISSING)` for the default tier + stderr derivation; bake's no-runtimes exit self-diagnoses the same way). Name arguments filter the report and add a per-kit detail block (resolved store tier, target and provenance of the target, paths/sizes/mtimes). `make.tcl bake ?kitname ...?` bakes and deploys only the named kits - other kits' `_build`/`bin` artifacts and punkcheck records are untouched and the vfslibs phase narrows to the named kits' vfs folders; an unknown name errors before any build, listing the configured names; flags go before kit names (`make.tcl bake -confirm 0 punk91`). Bare `bake` processes all configured kits as before. Both surfaces consume the shared parsed-mapping helpers (`::punkboot::lib::mapvfs_*`) rather than the file format, so the G-024 config-format work swaps the reader underneath. Piped characterization: `src/tests/shell/testsuites/punkexe/maketclbakelist.test`. - **Host vs target platform (G-122).** Everything a bake EMITS is keyed by the artifact's TARGET platform, never by the driving tclsh's personality: the `bin/runtime/` store the runtime is read from, `.exe` suffixing of runtime files and kit outputs, the presence checks behind `runtime=missing`, and whether the pre-deploy process sweep uses `tasklist`/`taskkill` or `ps`/`kill`. Host semantics - copy commands, path handling, filesystem case rules, prompts - stay keyed to the host. The default target is the host's own platform canon, EXCEPT a cygwin-family host (an msys2/cygwin-runtime tclsh, which reports `tcl_platform(platform)` `unix` on windows and canonizes as `msys-x86_64`/`cygwin-x86_64`), which targets `win32-x86_64`: such a host now drives the identical kit set, names and store addressing as a native tclsh. `tclsh src/make.tcl check` prints the derivation on one `platform (G-122): host=... target=... store=... exe-suffix=... process-tooling=...` line. A mapvfs entry may declare its own target platform (4th element - see `src/runtime/AGENTS.md`), which is then read from that platform's tier, named with that platform's executable convention, and skipped by the process sweep (its processes are not visible to this host). Traps this closes, all field-verified 2026-07-26: msys `ps` cannot see a natively-launched kit (only `tasklist` can), and msys2 rewrites arguments that look like absolute posix paths when spawning a native windows program - so `taskkill /PID ` arrived as `taskkill C://PID `; native-windows command lines therefore go through `::punkboot::exec_nativeargs` (sets `MSYS2_ARG_CONV_EXCL=*`, inert elsewhere). Also note a cygwin-family tclsh is a POSIX Tcl: name the script the way that shell spells paths (`tclsh /c/repo/.../src/make.tcl`, not `C:/repo/...`, which it resolves relative to the cwd) - make.tcl diagnoses the windows-spelling case. Piped characterization: `src/tests/shell/testsuites/punkexe/maketclplatform.test` (the cygwin-host tests self-gate on finding and probing a real msys/cygwin tclsh; `PUNK_MSYS_TCLSH` names one explicitly). diff --git a/src/make.tcl b/src/make.tcl index 67f7d37c..b3263dba 100644 --- a/src/make.tcl +++ b/src/make.tcl @@ -641,9 +641,10 @@ proc ::punkboot::lib::bootsupport_prompt_yesno {prompt} { # mapfile path as supplied # default_target target platform for entries that declare none # runtime_vfs_map runtime -> raw vfs_specs as listed (appname/type/target may be "") -# vfs_runtime_map vfstail -> list of {runtime appname type target} (defaults -# applied; only vfs folders that exist on disk - matching the -# historical inline parse the kit loop was built around) +# vfs_runtime_map vfstail -> list of {runtime appname type target smokerequires} +# (defaults applied; only vfs folders that exist on disk - matching +# the historical inline parse the kit loop was built around; +# smokerequires is the G-133 smoke-require package list, may be "") # runtime_target runtime -> resolved target platform (all runtimes, mapped order) # missing names of referenced-but-absent runtimes/vfs folders # warnings warning lines in encounter order, ready to print verbatim @@ -732,8 +733,10 @@ proc ::punkboot::lib::mapvfs_parse {mapfile rtbase sourcefolder default_target} } foreach vfsconfig $vfs_specs { switch -- [llength $vfsconfig] { - 1 - 2 - 3 - 4 { - lassign $vfsconfig vfstail appname target_kit_type + 1 - 2 - 3 - 4 - 5 { + #5th element (G-133): smoke-require package list for the built kit - + #interim line format; the schema's eventual home is the G-024 toml conversion + lassign $vfsconfig vfstail appname target_kit_type - smokerequires if {![file isdirectory [file join $sourcefolder vfs $vfstail]]} { lappend warnings "WARNING: Missing vfs folder [file join $sourcefolder vfs $vfstail] specified in mapvfs.config for runtime $runtime" lappend missing $vfstail @@ -741,11 +744,11 @@ proc ::punkboot::lib::mapvfs_parse {mapfile rtbase sourcefolder default_target} if {$appname eq ""} { set appname [file rootname $vfstail] } - dict lappend vfs_runtime_map $vfstail [list $runtime $appname $target_kit_type $line_target] + dict lappend vfs_runtime_map $vfstail [list $runtime $appname $target_kit_type $line_target $smokerequires] } } default { - set badline "bad entry in mapvfs.config - expected each entry after the runtime name to be a list of 1 to 4 elements {vfsfolder ?kitname? ?kittype? ?targetplatform?}. got: $vfsconfig ([llength $vfsconfig])" + set badline "bad entry in mapvfs.config - expected each entry after the runtime name to be a list of 1 to 5 elements {vfsfolder ?kitname? ?kittype? ?targetplatform? ?smokerequires?}. got: $vfsconfig ([llength $vfsconfig])" lappend warnings $badline lappend badentries $badline } @@ -791,6 +794,7 @@ proc ::punkboot::lib::mapvfs_parse {mapfile rtbase sourcefolder default_target} # store_tier bin/runtime tier folder name for the target # vfs vfs folder tail (e.g punk9wintk903.vfs) # vfs_present 1|0 +# smokerequire G-133 smoke-require package list as configured ("" when undeclared) proc ::punkboot::lib::mapvfs_kit_outputs {model rtbase sourcefolder} { set runtime_vfs_map [dict get $model runtime_vfs_map] set vfs_runtime_map [dict get $model vfs_runtime_map] @@ -804,7 +808,7 @@ proc ::punkboot::lib::mapvfs_kit_outputs {model rtbase sourcefolder} { } set records [list] set exe_names_seen [list] - set record_pair {{rtname vfstail appname target_kit_type} { + set record_pair {{rtname vfstail appname target_kit_type smokerequires} { upvar 1 records records exe_names_seen exe_names_seen rtbase rtbase sourcefolder sourcefolder\ runtime_target runtime_target default_target default_target if {$appname eq ""} { @@ -850,6 +854,7 @@ proc ::punkboot::lib::mapvfs_kit_outputs {model rtbase sourcefolder} { store_tier $store_tier\ vfs $vfstail\ vfs_present [file isdirectory [file join $sourcefolder vfs $vfstail]]\ + smokerequire $smokerequires\ ] }} foreach vfstail $vfs_tails { @@ -862,25 +867,25 @@ proc ::punkboot::lib::mapvfs_kit_outputs {model rtbase sourcefolder} { continue } foreach vfs_app [dict get $runtime_vfs_map $rtname] { - lassign $vfs_app configured_vfs appname target_kit_type + lassign $vfs_app configured_vfs appname target_kit_type - smokerequires if {$configured_vfs ne $vfstail} { continue } - apply $record_pair $rtname $vfstail $appname $target_kit_type + apply $record_pair $rtname $vfstail $appname $target_kit_type $smokerequires } } } #entries whose vfs folder is missing (excluded from vfs_runtime_map by the parse) dict for {rtname vfs_specs} $runtime_vfs_map { foreach vfsconfig $vfs_specs { - if {[llength $vfsconfig] < 1 || [llength $vfsconfig] > 4} { + if {[llength $vfsconfig] < 1 || [llength $vfsconfig] > 5} { continue } - lassign $vfsconfig vfstail appname target_kit_type + lassign $vfsconfig vfstail appname target_kit_type - smokerequires if {[dict exists $vfs_runtime_map $vfstail]} { continue ;#already enumerated above } - apply $record_pair $rtname $vfstail $appname $target_kit_type + apply $record_pair $rtname $vfstail $appname $target_kit_type $smokerequires } } return $records @@ -2282,6 +2287,8 @@ DIAGRAM 2 - KIT ASSEMBLY DETAIL (the 'make.tcl bake' stage; incl. vfslibs phase) v src/_build/.exe.vfs | + | advisory payload/target binary-arch scan of the merged + | tree (wrong-arch libraries -> recapped BUILD-WARNING) [K11] | boot-precondition gate: merged vfs must supply a tcl | library (tcl_library/, lib/tcl./, tcl./) [K10] v @@ -2294,6 +2301,8 @@ DIAGRAM 2 - KIT ASSEMBLY DETAIL (the 'make.tcl bake' stage; incl. vfslibs phase) v src/_build/.exe | + | smoke-require probe (host-runnable kits with declared + | packages): plain 'package require' inside the artifact [K11] | deploy step: delete old bin/.exe, copy new one in v bin/.exe <-- FAILS (kit listed under FAILED KITS) if a @@ -2377,6 +2386,22 @@ KEY / NOTES own payload could not be extracted - see the BUILD-WARNING naming what was tried. 'make.tcl check' reports whether the gate is ACTIVE. +[K11] PAYLOAD/TARGET consistency checks (G-133), both ADVISORY - warnings recap at + end of run, the kit still builds and deploys. (a) Binary-arch scan at the same + seam as [K10]: each binary library (*.dll/*.so/*.dylib) in the merged tree is + classified by header (PE/ELF/Mach-O - structural, nothing executed, so + cross-target kits are covered) against the kit's target platform; wrong-arch + libraries outside platform-discriminated subdirs (win32-ix86/, win-x64/ etc - + multi-arch payloads are legitimate there) earn recapped BUILD-WARNINGs. + (b) Smoke-require probe: a kit declaring packages (mapvfs.config 5th entry + element) has each one plain-'package require'd INSIDE the freshly built + artifact via its tclsh subcommand - the only check that sees resolution-order + defects (a wrong-arch higher-versioned package shadowing a working copy); + cross-target kits skip with a stated reason, undeclared kits run nothing new. + What neither check guarantees: statically linked packages, pure-tcl packages + with binary dependencies, and version-preference outcomes are visible only to + the smoke probe - and only for declared packages. 'make.tcl check' reports both. + MAINTENANCE (agents take note) ---------------------------------------------------------------------------- This text is embedded in make.tcl (::punkboot::workflow_text). When the build @@ -3356,6 +3381,95 @@ proc ::punkboot::get_vfs_boot_library_report {vfsfolder} { } return [list 1 [::punkboot::utils::vfs_boot_library_report $vfsfolder]] } +#Availability probe + advisory payload/target binary-arch scan of an assembled kit vfs +#(G-133), shared by the bake loop and the 'check' command report. Same guarded-require +#treatment as the boot-precondition check above: a stale bootsupport snapshot degrades +#the scan to a NOTE. Returns {available report}. +proc ::punkboot::get_vfs_binary_arch_report {vfsfolder targetplatform} { + set available [expr {\ + ![catch {package require punkboot::utils}]\ + && [llength [info commands ::punkboot::utils::vfs_binary_arch_report]]\ + }] + if {!$available} { + return [list 0 [dict create]] + } + return [list 1 [::punkboot::utils::vfs_binary_arch_report $vfsfolder $targetplatform]] +} +#G-133 smoke-require probe: execute a freshly built kit artifact via its 'tclsh' +#subcommand and plain-'package require' each declared package inside it. This is the +#only check that observes actual package RESOLUTION in the artifact - the punkluck86 +#incident class (a higher-versioned wrong-arch Thread shadowing the runtime's working +#copy on plain require) is invisible to every structural check because the working +#file exists. The tclsh subcommand is used deliberately: it exits cleanly even on +#runtimes whose full repl teardown is fragile, and a script argument with drained +#stdin cannot reach any console-reopen path. ADVISORY: failures become recapped +#BUILD-WARNINGs naming kit, package and the actual error - the kit still deploys. +#Returns 1 when every declared package resolved, else 0. +proc ::punkboot::kit_smoke_require_probe {kitpath targetkit packages} { + set probescript [file rootname $kitpath]_smokerequire.tcl + if {[catch { + set fd [open $probescript w] + chan configure $fd -translation lf + puts $fd "#generated by make.tcl bake (G-133) - smoke-require probe for $targetkit" + puts $fd "set packages [list $packages]" + puts $fd {foreach pkg $packages { + if {[catch {package require $pkg} v]} { + puts stdout [list SMOKE-REQUIRE $pkg FAIL $v] + } else { + puts stdout [list SMOKE-REQUIRE $pkg OK $v] + } +} +exit 0} + close $fd + } errM]} { + catch {close $fd} + ::punkboot::print_build_warnings [list "smoke-require for kit $targetkit could not write its probe script ($probescript): $errM"] + return 0 + } + if {[catch { + lassign [punk::lib::invoke [list [file nativename $kitpath] tclsh [file nativename $probescript] << ""]] p_stdout p_stderr p_exitcode + } errM]} { + ::punkboot::print_build_warnings [list "smoke-require FAILED for kit $targetkit: could not execute freshly built artifact via its tclsh subcommand ($errM)"] + return 0 + } + set seen [dict create] + foreach line [split $p_stdout \n] { + set line [string trim $line] + if {![string match "SMOKE-REQUIRE *" $line]} { + continue + } + #each line is a well-formed tcl list: SMOKE-REQUIRE OK|FAIL + if {[catch {lassign $line - r_pkg r_status r_detail}]} { + continue + } + dict set seen $r_pkg [list $r_status $r_detail] + } + set allok 1 + set warnings [list] + foreach pkg $packages { + if {![dict exists $seen $pkg]} { + set allok 0 + set tail [string trim [join [lrange [split [string trim $p_stderr] \n] end-2 end] " | "]] + lappend warnings "smoke-require FAILED for kit $targetkit: no result for package '$pkg' (artifact exited $p_exitcode before reporting[expr {$tail ne "" ? "; stderr tail: $tail" : ""}])" + continue + } + lassign [dict get $seen $pkg] r_status r_detail + if {$r_status eq "OK"} { + puts stdout " smoke-require OK for kit $targetkit: package require $pkg -> $r_detail" + } else { + set allok 0 + lappend warnings "smoke-require FAILED for kit $targetkit: package require $pkg -> $r_detail" + } + } + if {$p_exitcode != 0} { + set allok 0 + lappend warnings "smoke-require for kit $targetkit: artifact exited with code $p_exitcode via its tclsh subcommand (expected 0)" + } + if {[llength $warnings]} { + ::punkboot::print_build_warnings $warnings + } + return $allok +} #Wrap ::exit so accumulated provenance warnings are recapped at the very end of output no matter #which of make.tcl's many exit points a command takes. if {![llength [info commands ::punkboot::exit_original]]} { @@ -3612,6 +3726,40 @@ if {$::punkboot::command eq "check"} { puts stdout "boot-precondition gate (G-125): UNAVAILABLE (punkboot::utils vfs_boot_library_report not loadable from bootsupport)" puts stdout " bake will proceed with a NOTE and cannot refuse an unbootable kit - run 'make.tcl modules' then 'make.tcl bootsupport'." } + # Payload/target consistency checks (G-133) - advisory arch scan + smoke-require probe + puts stdout $sep + #availability probe only - the path deliberately does not exist + lassign [::punkboot::get_vfs_binary_arch_report [file join $projectroot __punkboot_archscan_probe__] $::punkboot::target_platform] _arch_available _arch_report + if {$_arch_available} { + puts stdout "payload/target arch scan (G-133): ACTIVE (advisory)" + puts stdout " bake classifies each merged kit's binary libraries (*.dll/*.so/*.dylib header bytes - structural," + puts stdout " nothing executed, cross-target kits included) against the kit's target platform and emits" + puts stdout " recapped BUILD-WARNINGs for wrong-arch libraries outside platform-discriminated subdirs" + puts stdout " (canonical - names and recognised vendor spellings such as win-x64 are exempt)." + } else { + puts stdout "payload/target arch scan (G-133): UNAVAILABLE (punkboot::utils vfs_binary_arch_report not loadable from bootsupport)" + puts stdout " bake will proceed with a NOTE and cannot flag wrong-arch payload libraries - run 'make.tcl modules' then 'make.tcl bootsupport'." + } + set _smoke_declared [list] + if {[file exists $sourcefolder/runtime/mapvfs.config]} { + set _smokemodel [punkboot::lib::mapvfs_parse $sourcefolder/runtime/mapvfs.config [file dirname $sourcefolder]/bin/runtime $sourcefolder $::punkboot::target_platform] + foreach _rec [punkboot::lib::mapvfs_kit_outputs $_smokemodel [file dirname $sourcefolder]/bin/runtime $sourcefolder] { + if {[llength [dict get $_rec smokerequire]]} { + lappend _smoke_declared "[dict get $_rec kitname] ([join [dict get $_rec smokerequire] {, }])" + } + } + unset -nocomplain _smokemodel _rec + } + puts stdout "smoke-require probe (G-133, advisory): freshly built host-runnable kits declaring packages" + puts stdout " (mapvfs.config 5th entry element) have each one plain-'package require'd inside the built" + puts stdout " artifact via its tclsh subcommand - the only check that sees resolution-order defects such as" + puts stdout " a wrong-arch higher-versioned package shadowing a working copy. Failures are recapped" + puts stdout " BUILD-WARNINGs; cross-target kits skip with a stated reason; undeclared kits run nothing new." + if {[llength $_smoke_declared]} { + puts stdout " declared: [join $_smoke_declared {; }]" + } else { + puts stdout " declared: (none)" + } puts stdout $sep exit 0 } @@ -5840,7 +5988,8 @@ foreach vfstail $vfs_tails { if {[dict exists $runtime_vfs_map $rtname]} { set applist [dict get $runtime_vfs_map $rtname] foreach vfs_app $applist { - lassign $vfs_app configured_vfs appname target_kit_type + #5th element = G-133 smoke-require package list ("" when undeclared) + lassign $vfs_app configured_vfs appname target_kit_type - kit_smokerequires if {$configured_vfs ne $vfstail} { continue } @@ -5862,7 +6011,7 @@ foreach vfstail $vfs_tails { } } lappend exe_names_seen $targetkit - lappend targetkits [list $targetkit $target_kit_type] + lappend targetkits [list $targetkit $target_kit_type $kit_smokerequires] } } puts stdout " vfs: $vfstail runtime: $rtname targetkits: $targetkits" @@ -5871,7 +6020,7 @@ foreach vfstail $vfs_tails { continue ;#G-121 selective bake - not a requested kit output } puts stdout " processing targetkit: $targetkit_info" - lassign $targetkit_info targetkit target_kit_type + lassign $targetkit_info targetkit target_kit_type kit_smokerequires #Self-build guard: never process the kit whose deployed executable is running this #build - the process sweep below would kill this very process, and the deploy step #cannot replace a running image anyway. Informational/update subcommands are @@ -6240,6 +6389,37 @@ foreach vfstail $vfs_tails { set sourcevfs [file join $sourcefolder vfs $vfstail] merge_over $sourcevfs $targetvfs + #G-133 advisory payload/target binary-arch scan, at the same post-merge + #seam as the G-125 gate below. Structural (header bytes only, nothing + #executed), so it runs for every kit including cross-target ones. A + #binary library under a platform-discriminated subdir (win32-ix86/, + #win-x64/ etc) is exempt - multi-arch payloads are legitimate there. + #ADVISORY by design: vendor layouts make false certainty unavoidable, + #so mismatches warn (recapped) and the kit still builds - the G-125 + #gate below stays the only refusal at this seam. + lassign [::punkboot::get_vfs_binary_arch_report $targetvfs $rt_target] archscan_available archscan + if {!$archscan_available} { + puts stderr "NOTE: kit payload binary-arch scan unavailable (punkboot::utils vfs_binary_arch_report not loadable from bootsupport) - continuing without it" + } elseif {[dict get $archscan reason] ne ""} { + puts stdout " payload binary-arch scan not applicable for kit $targetkit: [dict get $archscan reason]" + } elseif {[llength [dict get $archscan mismatches]]} { + set archscan_mismatches [dict get $archscan mismatches] + set archwarnings [list] + set shown 0 + foreach m $archscan_mismatches { + if {$shown >= 8} {break} + incr shown + lappend archwarnings "payload/target arch mismatch in kit $targetkit (target $rt_target): [dict get $m file] is [dict get $m detail] outside any platform-discriminated subdir - a $rt_target runtime cannot load it" + } + if {[llength $archscan_mismatches] > $shown} { + lappend archwarnings "payload/target arch mismatch in kit $targetkit (target $rt_target): [expr {[llength $archscan_mismatches] - $shown}] further mismatching binary libraries not listed above ([llength $archscan_mismatches] mismatches total, [dict get $archscan scanned] binary libraries scanned)" + } + ::punkboot::print_build_warnings $archwarnings + unset archwarnings archscan_mismatches shown + } else { + puts stdout " payload binary-arch scan for kit $targetkit (target $rt_target): OK - [dict get $archscan scanned] binary libraries scanned ([dict get $archscan matched] match target, [dict get $archscan exempt] exempt under platform subdirs, [dict get $archscan unknown] unclassifiable)" + } + #G-125 boot-precondition gate. A kit with no tcl library cannot initialise #at all - 'Cannot find a usable init.tcl' - so refuse it HERE: before any #build product is written, and long before the deploy step deletes the @@ -6588,6 +6768,23 @@ foreach vfstail $vfs_tails { # -- --- --- --- --- --- $vfs_event targetset_end OK + #G-133 post-build smoke-require probe. Only an actual 'package require' + #inside the built artifact observes package RESOLUTION - the wrong-arch + #version-shadowing class is invisible to every structural check because + #the working file exists too. Runs only for kits that declare + #smoke-require packages (mapvfs.config 5th entry element) and only when + #this host can execute the artifact; undeclared kits run nothing new. + if {[llength $kit_smokerequires]} { + if {$rtname eq "-"} { + puts stdout " smoke-require skipped for kit $targetkit: a runtimeless .kit artifact is not directly executable" + } elseif {[punkboot::lib::platform_process_family $rt_target] ne [punkboot::lib::platform_process_family $::punkboot::host_platform]} { + puts stdout " smoke-require skipped for kit $targetkit: cross-target ($rt_target) - the artifact is not executable on this $::punkboot::host_platform host" + } else { + puts stdout " smoke-require probe for kit $targetkit (packages: $kit_smokerequires) - executing freshly built artifact via its tclsh subcommand" + ::punkboot::kit_smoke_require_probe $buildfolder/$targetkit $targetkit $kit_smokerequires + } + } + } else { set skipped_vfs_build 1 puts stderr "." diff --git a/src/modules/punkboot/utils-999999.0a1.0.tm b/src/modules/punkboot/utils-999999.0a1.0.tm index 9bf8330f..5b188262 100644 --- a/src/modules/punkboot/utils-999999.0a1.0.tm +++ b/src/modules/punkboot/utils-999999.0a1.0.tm @@ -23,7 +23,7 @@ package require Tcl 8.6- # ++ +++ +++ +++ +++ +++ +++ +++ +++ +++ +++ namespace eval punkboot::utils { variable version 0.1.0 - namespace export parse_punkproject_version read_punkproject_version read_changelog_latest_version vcs_dirty_warnings vfs_boot_library_report + namespace export binary_arch_classify parse_punkproject_version read_punkproject_version read_changelog_latest_version vcs_dirty_warnings vfs_binary_arch_report vfs_boot_library_report namespace eval argdoc { @@ -335,6 +335,407 @@ namespace eval punkboot::utils { } return $report } + + #G-133 binary-arch scan support. Recognition data is deliberately plain namespace + #variables so extending it (a new vendor spelling, a new cpu token) is a visible + #one-line edit rather than a code change. + # + #PE Machine field values (IMAGE_FILE_HEADER.Machine, 2 bytes LE at e_lfanew+4) + variable pe_machine_map { + 014c i386 + 8664 x86_64 + aa64 arm64 + 01c0 arm + 01c4 arm + 0200 ia64 + 0166 mips + 01f0 ppc + 5032 riscv32 + 5064 riscv64 + } + #ELF e_machine values (2 bytes at offset 18, endianness per EI_DATA) + variable elf_machine_map { + 3 i386 + 62 x86_64 + 183 arm64 + 40 arm + 8 mips + 20 ppc + 21 ppc64 + 22 s390x + 2 sparc + 43 sparc64 + 243 riscv + } + #Mach-O cputype base values (CPU_ARCH_ABI64 flag 0x01000000 selects the 64-bit variant) + variable macho_cputype_map { + 7 {i386 x86_64} + 12 {arm arm64} + 18 {ppc ppc64} + } + #os / cpu tokens accepted in a CANONICAL platform-dir segment (-). The os and + #cpu lists follow the punk::platform canon plus common alias tokens, so a platform- + #discriminated subdir is recognised whichever spelling a package author normalised to. + variable platform_segment_os { + win32 windows linux macosx freebsd netbsd openbsd dragonfly solaris sunos + aix hpux irix qnx haiku android msys cygwin + } + variable platform_segment_cpu { + x86_64 amd64 x64 ix86 i386 i486 i586 i686 x86 + arm64 aarch64 arm armv6 armv7 armv6l armv7l + ppc ppc64 ppc64le powerpc mips mipsel mips64 + riscv32 riscv64 s390 s390x sparc sparc64 universal + } + #Recognised VENDOR spellings observed (or near-certain) as platform-discriminating + #directory names that do not parse as -. blend2d, + #extrafont and tclMuPDF ship win-x64 today. + variable platform_segment_vendor { + win-x64 win-x86 win-arm64 win-arm win32 win64 + linux-x64 linux-arm64 macos-x64 macos-arm64 osx-x64 osx-arm64 + } + #File tails treated as binary libraries by vfs_binary_arch_report + variable binary_library_globs {*.dll *.so *.so.[0-9]* *.dylib} + + namespace eval argdoc { + lappend PUNKARGS [list { + @id -id ::punkboot::utils::binary_arch_classify + @cmd -name "::punkboot::utils::binary_arch_classify"\ + -summary\ + "Classify a binary library file's executable format and cpu architecture"\ + -help\ + "Reads only the file's header bytes (a couple of small reads - nothing + is executed or loaded) and classifies the executable format and cpu + architecture: + PE Machine field (2 bytes at e_lfanew+4): i386, x86_64, arm64... + ELF e_machine honouring EI_DATA endianness + Mach-O thin (cputype) and fat/universal (per-member cputypes) + Unrecognised content is reported honestly as format 'unknown' rather + than guessed. + + Returned dict keys: + format pe | elf | macho | unknown + arch i386 | x86_64 | arm64 | ... | universal | unknown | + unknown-0x (a real header whose machine value is not + in the recognition map - definite, just unmapped) + archs member architectures (macho universal only; absent otherwise) + detail short human-readable classification" + @leaders + libfile -type string -optional 0 -help\ + "Path of the binary library file to classify" + }] + } + proc binary_arch_classify {libfile} { + variable pe_machine_map + variable elf_machine_map + variable macho_cputype_map + if {[catch { + set fd [open $libfile r] + chan configure $fd -translation binary + set head [read $fd 64] + } errM]} { + catch {close $fd} + return [dict create format unknown arch unknown detail "unreadable: $errM"] + } + set result [dict create format unknown arch unknown detail "not a recognised binary library format"] + try { + if {[string length $head] < 8} { + dict set result detail "too short to carry an executable header ([string length $head] bytes)" + return $result + } + set magic4 [string range $head 0 3] + if {[string range $head 0 1] eq "MZ"} { + #PE: e_lfanew at 0x3c points at "PE\0\0" + IMAGE_FILE_HEADER + if {[binary scan $head @60iu e_lfanew] != 1} { + dict set result detail "MZ header too short for e_lfanew" + return $result + } + seek $fd $e_lfanew + set pehdr [read $fd 6] + if {[string length $pehdr] < 6 || [string range $pehdr 0 3] ne "PE\x00\x00"} { + dict set result detail "MZ header without PE signature (DOS executable?)" + return $result + } + binary scan $pehdr @4su machine + set hexmachine [format %04x $machine] + if {[dict exists $pe_machine_map $hexmachine]} { + set arch [dict get $pe_machine_map $hexmachine] + set result [dict create format pe arch $arch detail "PE $arch"] + } else { + set result [dict create format pe arch unknown-0x$hexmachine detail "PE machine 0x$hexmachine (not in recognition map)"] + } + } elseif {$magic4 eq "\x7fELF"} { + binary scan $head @4cu elfclass + binary scan $head @5cu elfdata + if {$elfdata == 2} { + binary scan $head @18Su e_machine ;#big-endian + } else { + binary scan $head @18su e_machine ;#little-endian (and honest-enough default) + } + if {[dict exists $elf_machine_map $e_machine]} { + set arch [dict get $elf_machine_map $e_machine] + if {$arch eq "riscv"} { + set arch [expr {$elfclass == 2 ? "riscv64" : "riscv32"}] + } + set result [dict create format elf arch $arch detail "ELF $arch"] + } else { + set result [dict create format elf arch unknown-$e_machine detail "ELF e_machine $e_machine (not in recognition map)"] + } + } else { + binary scan $head Iu be32 + switch -- [format %08x $be32] { + feedface - feedfacf - cefaedfe - cffaedfe { + #thin Mach-O; cputype follows the magic in the file's own byte order + if {[string index $head 0] eq "\xfe"} { + binary scan $head @4Iu cputype ;#big-endian file + } else { + binary scan $head @4iu cputype + } + set is64 [expr {($cputype & 0x01000000) != 0}] + set base [expr {$cputype & 0x00ffffff}] + if {[dict exists $macho_cputype_map $base]} { + set arch [lindex [dict get $macho_cputype_map $base] $is64] + set result [dict create format macho arch $arch detail "Mach-O $arch"] + } else { + set result [dict create format macho arch unknown-cputype-$base detail "Mach-O cputype $base (not in recognition map)"] + } + } + cafebabe - cafebabf { + #fat/universal (big-endian header). A Java class file shares the + #magic - its version field where nfat_arch sits is >= 45, far above + #any credible member count, so an implausible count stays unknown. + binary scan $head @4Iu nfat + if {$nfat == 0 || $nfat > 30} { + dict set result detail "cafebabe magic with implausible member count $nfat (java class file?)" + return $result + } + set entrysize [expr {$be32 == 0xcafebabf ? 32 : 20}] + seek $fd 8 + set fatdata [read $fd [expr {$nfat * $entrysize}]] + set archs [list] + for {set i 0} {$i < $nfat} {incr i} { + if {[binary scan $fatdata @[expr {$i * $entrysize}]Iu cputype] != 1} { + break + } + set is64 [expr {($cputype & 0x01000000) != 0}] + set base [expr {$cputype & 0x00ffffff}] + if {[dict exists $macho_cputype_map $base]} { + lappend archs [lindex [dict get $macho_cputype_map $base] $is64] + } else { + lappend archs unknown-cputype-$base + } + } + set result [dict create format macho arch universal archs $archs detail "Mach-O universal ([join $archs {, }])"] + } + } + } + return $result + } finally { + catch {close $fd} + } + } + + namespace eval argdoc { + lappend PUNKARGS [list { + @id -id ::punkboot::utils::platform_expected_binary + @cmd -name "::punkboot::utils::platform_expected_binary"\ + -summary\ + "Expected binary format/arch for a canonical target platform name"\ + -help\ + "Maps a canonical punkshell platform name (-, e.g + win32-ix86) to the executable format and classifier arch token a + native binary library for that platform carries. Returns a dict + {format F arch A}, or an empty dict when the os or cpu token has + no recognised mapping (callers treat that as scan-not-applicable + rather than guessing)." + @leaders + targetplatform -type string -optional 0 -help\ + "Canonical platform name, e.g win32-x86_64, linux-x86_64, macosx-arm64" + }] + } + proc platform_expected_binary {targetplatform} { + set idx [string first - $targetplatform] + if {$idx <= 0} { + return [dict create] + } + set os [string range $targetplatform 0 [expr {$idx - 1}]] + set cpu [string range $targetplatform [expr {$idx + 1}] end] + switch -- $os { + win32 - windows {set format pe} + linux - freebsd - netbsd - openbsd - dragonfly - solaris - sunos {set format elf} + macosx - darwin - macos {set format macho} + default {return [dict create]} + } + switch -- $cpu { + ix86 - i386 - i486 - i586 - i686 - x86 {set arch i386} + x86_64 - amd64 - x64 {set arch x86_64} + arm64 - aarch64 {set arch arm64} + arm - armv6 - armv7 {set arch arm} + ppc - ppc64 - riscv64 - mips - s390x - sparc - sparc64 {set arch $cpu} + default {return [dict create]} + } + return [dict create format $format arch $arch] + } + + namespace eval argdoc { + lappend PUNKARGS [list { + @id -id ::punkboot::utils::platform_discriminated_segment + @cmd -name "::punkboot::utils::platform_discriminated_segment"\ + -summary\ + "Whether a directory name is a recognised platform-discriminating segment"\ + -help\ + "True when the segment is a canonical - platform-dir name + (os and cpu each from the recognised token lists), the bare + universal 'macosx' folder, or one of the recognised vendor + spellings (blend2d-style win-x64 etc). Binary libraries under such + a directory are selected per-platform by their package's own index + logic, so a multi-arch payload is legitimate there." + @leaders + segment -type string -optional 0 -help\ + "One directory name (a single path segment, not a path)" + }] + } + proc platform_discriminated_segment {segment} { + variable platform_segment_os + variable platform_segment_cpu + variable platform_segment_vendor + set segment [string tolower $segment] + if {$segment eq "macosx"} { + return 1 ;#the universal store-dir convention + } + if {$segment in $platform_segment_vendor} { + return 1 + } + set idx [string first - $segment] + if {$idx > 0} { + set os [string range $segment 0 [expr {$idx - 1}]] + set cpu [string range $segment [expr {$idx + 1}] end] + if {$os in $platform_segment_os && $cpu in $platform_segment_cpu} { + return 1 + } + } + return 0 + } + + namespace eval argdoc { + lappend PUNKARGS [list { + @id -id ::punkboot::utils::vfs_binary_arch_report + @cmd -name "::punkboot::utils::vfs_binary_arch_report"\ + -summary\ + "Advisory scan of an assembled vfs tree for binary libraries that mismatch the kit's target"\ + -help\ + "Walks a merged .vfs tree, classifies each binary library file + (*.dll, *.so, *.so., *.dylib) by header (binary_arch_classify) + and compares it against the kit's declared target platform. A + library under a platform-discriminated subdirectory (canonical + - names, the universal macosx folder, or a recognised + vendor spelling such as win-x64) is EXEMPT - multi-arch payloads + are legitimate when a platform segment lets the package's own + index logic pick per platform. Purely structural: header bytes are + read, nothing is executed or loaded, so the scan also covers + cross-target kits the build host could never run. + + A mismatch is reported only when the classification is DEFINITE + (a real PE/ELF/Mach-O header whose format or architecture differs + from the target's); unclassifiable files are counted as unknown, + never warned about. What this cannot see: statically linked + packages, pure-tcl packages whose binary dependency is elsewhere, + and package resolution order (a wrong-arch copy shadowing a + working one by version) - only an in-artifact 'package require' + smoke probe observes those. + + Returned dict keys: + ok 1 when no mismatches were found (advisory) + target the target platform as supplied + expected {format F arch A} for the target ({} if unmapped) + scanned binary library files classified + matched files matching the expected format/arch + exempt files under platform-discriminated subdirs (not read) + unknown files whose classification was indefinite + mismatches list of {file format F arch A detail D} + reason why the scan did not apply; empty when it ran" + @leaders + vfsfolder -type string -optional 0 -help\ + "Path of the assembled vfs tree to scan" + targetplatform -type string -optional 0 -help\ + "Canonical target platform of the kit, e.g win32-ix86" + }] + } + proc vfs_binary_arch_report {vfsfolder targetplatform} { + variable binary_library_globs + set report [dict create ok 1 target $targetplatform expected {} scanned 0 matched 0 exempt 0 unknown 0 mismatches {} reason ""] + if {![file isdirectory $vfsfolder]} { + dict set report ok 0 + dict set report reason "no such directory: $vfsfolder" + return $report + } + set expected [platform_expected_binary $targetplatform] + dict set report expected $expected + if {![dict size $expected]} { + dict set report reason "target platform '$targetplatform' has no recognised binary format/arch mapping - scan not applicable" + return $report + } + set expected_format [dict get $expected format] + set expected_arch [dict get $expected arch] + set scanned 0 + set matched 0 + set exempt 0 + set unknown 0 + set mismatches [list] + set rootlen [string length $vfsfolder] + #iterative walk - plain Tcl 8.6, no deps. Exemption is decided per DIRECTORY + #(one platform_discriminated_segment test as each dir is queued), so files + #under an exempt subtree are counted without being read. + set pending [list [list $vfsfolder 0]] + while {[llength $pending]} { + set entry [lindex $pending end] + set pending [lrange $pending 0 end-1] + lassign $entry dir dir_exempt + foreach sub [glob -nocomplain -types d -directory $dir *] { + set sub_exempt $dir_exempt + if {!$sub_exempt && [platform_discriminated_segment [file tail $sub]]} { + set sub_exempt 1 + } + lappend pending [list $sub $sub_exempt] + } + foreach libfile [glob -nocomplain -types f -directory $dir -- {*}$binary_library_globs] { + if {$dir_exempt} { + incr exempt + continue + } + incr scanned + set class [binary_arch_classify $libfile] + set cformat [dict get $class format] + set carch [dict get $class arch] + if {$cformat eq "unknown"} { + incr unknown + continue + } + set rel [string map {\\ /} [string range $libfile [expr {$rootlen + 1}] end]] + if {$cformat ne $expected_format} { + lappend mismatches [dict create file $rel format $cformat arch $carch detail [dict get $class detail]] + continue + } + if {$carch eq $expected_arch} { + incr matched + continue + } + if {$carch eq "universal" && [dict exists $class archs] && $expected_arch in [dict get $class archs]} { + incr matched + continue + } + #definite finding either way: a differing mapped arch, or a real header + #whose unmapped machine value is provably not the expected architecture + lappend mismatches [dict create file $rel format $cformat arch $carch detail [dict get $class detail]] + } + } + dict set report scanned $scanned + dict set report matched $matched + dict set report exempt $exempt + dict set report unknown $unknown + dict set report mismatches $mismatches + dict set report ok [expr {[llength $mismatches] == 0}] + return $report + } } namespace eval ::punk::args::register { diff --git a/src/modules/punkboot/utils-buildversion.txt b/src/modules/punkboot/utils-buildversion.txt index f8cb8a1d..c85f47cb 100644 --- a/src/modules/punkboot/utils-buildversion.txt +++ b/src/modules/punkboot/utils-buildversion.txt @@ -1,6 +1,7 @@ -0.4.0 +0.5.0 #First line must be a semantic version number #all other lines are ignored. +#0.5.0 - added binary_arch_classify (PE/ELF/Mach-O header classifier, honest unknowns), platform_expected_binary, platform_discriminated_segment and vfs_binary_arch_report (advisory payload/target binary-arch scan of a merged .vfs tree with platform-subdir exemption; backs make.tcl's G-133 payload/target consistency checks) #0.4.0 - vfs_boot_library_report recognises a third tcl-library convention: tcl./ at the vfs root, used by runtimes whose archive mounts at the executable's own path rather than //zipfs:/app (the androwish/undroidwish zipfs backport for 8.6). Such a runtime was being refused as unbootable. 'checked' gains the new location. #0.3.0 - added vfs_boot_library_report (structural boot-precondition check of an assembled .vfs tree; backs make.tcl's G-125 unbootable-kit deploy gate) #0.2.0 - vcs_dirty_warnings: optional scope arg limiting which uncommitted changes count (e.g scope src for make.tcl build/promotion dirty-src gate) diff --git a/src/runtime/AGENTS.md b/src/runtime/AGENTS.md index e557f10e..bd30b9b9 100644 --- a/src/runtime/AGENTS.md +++ b/src/runtime/AGENTS.md @@ -12,7 +12,8 @@ Houses the `mapvfs.config` that maps VFS payloads to platform runtimes, plus the ## Local Contracts - `mapvfs.config` defines which `src/vfs/*.vfs` folders combine with which runtime binaries (stored under `bin/runtime//`). -- Entry grammar: one line per runtime, ` {vfs ?kitname? ?kittype? ?targetplatform?} ...`. The file's own header comment is the user-facing spec; keep the two in step. +- Entry grammar: one line per runtime, ` {vfs ?kitname? ?kittype? ?targetplatform? ?smokerequires?} ...`. The file's own header comment is the user-facing spec; keep the two in step. +- **Smoke-require packages (5th element, G-133).** A kit entry may name packages the freshly built artifact must be able to plain-`package require`; the bake executes host-runnable kits via their tclsh subcommand and requires each one inside the real artifact (the only check that sees package resolution order - e.g a higher-versioned wrong-arch copy shadowing a working one). Advisory: failures are recapped `BUILD-WARNING`s; cross-target kits skip with a stated reason; undeclared kits run nothing new. Interim line format - the schema's eventual home is the G-024 toml conversion. Element positions are fixed, so declaring smoke-requires means also spelling out the target platform. - **Target platform (4th element, G-122).** A kit config entry may name the canonical punkshell platform its runtime is for (`help platforms` lists the names). It decides which `bin/runtime//` tier the runtime is read from (macosx-* collapsing to the universal `macosx` folder), whether the runtime file and the built kit carry `.exe`, and whether the pre-deploy process sweep applies. It is a property of the RUNTIME: entries on one runtime line may repeat it but must not disagree (a conflict, or a name that is not an `-` platform-dir name, is a fatal config error). Omitted means the build host's default target - which for an msys2/cygwin-runtime tclsh is `win32-x86_64`, not that host's own canon. The live example is the `tclkit-902-Linux64-intel-dyn` entry (target `linux-x86_64`, suffixless `punkshell902` artifact). The schema's eventual home is the G-024 toml conversion; consumers read the parsed model (`::punkboot::lib::mapvfs_*`), never the file format. - Runtime executables are placed in `bin/runtime//` by the `bin/punk-runtime.cmd` helper, manually, or (G-103 family runtimes) copied from the suite_tcl90 `kit-family` build products under `src/buildsuites/_build/suite_tcl90/out/family/`. - The `_build/` subdirectory holds build intermediates; it can be safely deleted. diff --git a/src/runtime/mapvfs.config b/src/runtime/mapvfs.config index 1acb8fa8..39c89831 100644 --- a/src/runtime/mapvfs.config +++ b/src/runtime/mapvfs.config @@ -1,5 +1,5 @@ #single line per runtime executable. Name of runtime followed by list of kit configuration lists. -#Each kit config list is from 1 to 4 elements +#Each kit config list is from 1 to 5 elements #1) .vfs folder with path relative to src folder. #2) name of wrapped kit binary to create (default to rootname of .vfs folder e.g myapp.vfs will create a kit named myapp (or myapp.exe) for a windows target) #3) Type of kit to create (zip or kit) - default kit. kit for metakit or vlerq based kit, zip for more modern zipkit. @@ -10,6 +10,15 @@ # (bin/runtime//, macosx-* collapsing to the universal macosx folder), whether the runtime # file and the built kit carry .exe, and which process tooling the pre-deploy sweep uses. # It is a property of the RUNTIME: entries on one runtime line may repeat it but must not disagree. +#5) SMOKE-REQUIRE packages (G-133) - a list of package names the freshly built kit must be able to +# plain-'package require'. After building, a host-runnable kit is executed via its tclsh +# subcommand and each named package is required inside the real artifact - the only check that +# sees package RESOLUTION (e.g a higher-versioned wrong-arch copy shadowing a working one). +# ADVISORY: failures become recapped BUILD-WARNINGs naming kit, package and error; the kit still +# builds and deploys. Cross-target kits skip with a stated reason; kits with no declaration run +# nothing new. Interim line format - the schema's eventual home is the G-024 toml conversion. +# e.g myruntime.exe {myapp.vfs myapp zip win32-x86_64 {Thread sqlite3}} +# (element positions are fixed: declaring smoke-requires means also spelling out the target) # # e.g The following creates 2 zipkits from the same runtime and 2 different .vfs folder # mytcl90kit.exe {mainapp.vfs myapp.exe zip} {test.vfs testapp.exe zip} @@ -111,7 +120,9 @@ tclsh9.0.5-punk-bi.exe {punk9win_for_tkruntime.vfs punk9bi_beta zip} #bin/runtime/linux-x86_64/ with no .exe suffix and the kit deploys as bin/punkshell902. #Before the target declaration this row read runtime=missing from a windows host, #because the runtime was looked for as tclkit-902-Linux64-intel-dyn.exe in the windows store. -tclkit-902-Linux64-intel-dyn {punk9linux.vfs punkshell902 kit linux-x86_64} +#Smoke-require Thread (G-133): on a non-linux host this SKIPS with a stated reason (the +# artifact is not executable here) - a linux-hosted bake would probe the tclkit's Thread. +tclkit-902-Linux64-intel-dyn {punk9linux.vfs punkshell902 kit linux-x86_64 Thread} #G-129 verification runtimes: third-party 32-bit windows, bin/runtime/win32-ix86/ tier # (see goals/archive/G-129-kitboot-mountpoint-derivation.md). Kept active so both zipfs @@ -126,8 +137,14 @@ tclkit-902-Linux64-intel-dyn {punk9linux.vfs punkshell902 kit linux-x86_64} #androwish/undroidwish 8.6 zipfs backport (LUCK build): mounts its attached archive at the # EXECUTABLE'S OWN PATH and has no tcl::zipfs::root - the boot derives the mount point # from the runtime's own mount table. -tclsh8.6.10-luck-zip.exe {punk8min.vfs punkluck86 zip win32-ix86} +#Smoke-require Thread (G-133): the 2026-07-27 incident kit - a payload x64 Thread 2.8.12 +# outversioned the runtime's working 32-bit 2.8.5, so plain 'package require Thread' +# selected the unloadable copy and the repl's tsv/codethread machinery died. The probe +# plain-requires Thread in the built artifact, so that class is caught by construction. +tclsh8.6.10-luck-zip.exe {punk8min.vfs punkluck86 zip win32-ix86 Thread} #apn tcl-sfe 9.1b0 32-bit: modern //zipfs:/app convention - proves the 32-bit target # itself needs nothing special. Runtime carries no Tk; Thread/twapi/sqlite3/tdbc are # STATIC in this runtime (32-bit builtins - G-058 capture resolves them). -tclsh9.1b0-tclsfe.exe {punk9min.vfs punk91ix86 zip win32-ix86} +#Smoke-require (G-133): Thread resolves via the static builtin; iocp exercises the +# payload's platform-subdir (win32-ix86) dll selection inside the 32-bit artifact. +tclsh9.1b0-tclsfe.exe {punk9min.vfs punk91ix86 zip win32-ix86 {Thread iocp}} diff --git a/src/tests/modules/punkboot/utils/testsuites/utils/binaryarch.test b/src/tests/modules/punkboot/utils/testsuites/utils/binaryarch.test new file mode 100644 index 00000000..f537915f --- /dev/null +++ b/src/tests/modules/punkboot/utils/testsuites/utils/binaryarch.test @@ -0,0 +1,353 @@ +# -*- tcl -*- +# Tests for the punkboot::utils G-133 payload/target binary-arch machinery: +# - binary_arch_classify: PE Machine-field classification (i386/x86_64/arm64), ELF and +# Mach-O (thin + fat/universal) classification, and honest 'unknown' for everything +# else (text named .dll, MZ without PE, java-class cafebabe decoy, short files) +# - platform_expected_binary / platform_discriminated_segment mapping + recognition +# - vfs_binary_arch_report: the 2026-07-27 punkluck86 case reproduced as a fixture (an +# x64 thread dll under plain lib_tcl8/ in a win32-ix86 kit), the platform-subdir +# exemptions (iocp-2.0.2's win32-ix86 + win32-x86_64 pair, blend2d-style win-x64), +# and the real assembled trees in src/_build when a build has been run +# All binary fixtures are GENERATED here with 'binary format' - header bytes only, no +# committed binaries (root AGENTS.md binary policy), nothing executable. The scan reads +# headers only and never executes anything - what makes it usable for cross-target kits. +# Run: tclsh src/tests/runtests.tcl -report compact -show-passes 0 -include-paths modules/punkboot/utils/*** binaryarch.test + +package require tcltest +package require punkboot::utils + +#added 2026-07-27 (agent, G-133) + +namespace eval ::testspace { + namespace import ::tcltest::* + + variable BASE [makeDirectory g133_binaryarch] + + #/src/tests/modules/punkboot/utils/testsuites/utils -> 7 levels up + variable projectroot [file normalize [file join [file dirname [info script]] .. .. .. .. .. .. ..]] + variable buildfolder [file join $projectroot src _build] + testConstraint builtvfsavailable [expr {[llength [glob -nocomplain -type d -directory $buildfolder *.exe.vfs]] > 0}] + + proc writebytes {path bytes} { + file mkdir [file dirname $path] + set fd [open $path w] + chan configure $fd -translation binary + puts -nonewline $fd $bytes + close $fd + } + #minimal PE: MZ header, e_lfanew(0x3c)=128, PE\0\0 + Machine at 128 + proc pebytes {machine} { + return [binary format a2x58ix64a4sx20 MZ 128 PE\x00\x00 $machine] + } + #minimal ELF: magic, EI_CLASS, EI_DATA, e_machine at 18 (endianness per EI_DATA) + proc elfbytes {class data machine} { + if {$data == 2} { + return [binary format a4ccx10x2Sx12 \x7fELF $class $data $machine] + } + return [binary format a4ccx10x2sx12 \x7fELF $class $data $machine] + } + proc mktree {name} { + variable BASE + set root [file join $BASE $name] + file delete -force $root + file mkdir $root + return $root + } + proc classify {path} { + return [punkboot::utils::binary_arch_classify $path] + } + #distinct name - a proc named 'scan' would shadow the Tcl builtin in this namespace + proc archscan {root target} { + return [punkboot::utils::vfs_binary_arch_report $root $target] + } + proc fmtarch {class} { + return [list [dict get $class format] [dict get $class arch]] + } + + variable common { + set result [list] + } + + # -- --- --- classifier: PE --- --- -- + + test binaryarch_classify_pe_fixtures {PE i386, x86_64 and arm64 fixtures classify by Machine field}\ + -setup $common -body { + variable BASE + foreach {name machine} {i386 0x014c x86_64 0x8664 arm64 -0x559c} { + #-0x559c is 0xaa64 as a signed 16-bit value - 'binary format s' takes either + set f [file join $BASE pe_$name.dll] + writebytes $f [pebytes [expr {$machine}]] + lappend result [fmtarch [classify $f]] + } + set result + }\ + -result {{pe i386} {pe x86_64} {pe arm64}} + + test binaryarch_classify_pe_unknown_machine {a real PE with an unmapped Machine value is definite-but-unmapped, not silently unknown}\ + -setup $common -body { + variable BASE + set f [file join $BASE pe_odd.dll] + writebytes $f [pebytes 0x1234] + set c [classify $f] + lappend result [dict get $c format] [dict get $c arch] + }\ + -result {pe unknown-0x1234} + + test binaryarch_classify_mz_without_pe {an MZ header without a PE signature (DOS executable) is honestly unknown}\ + -setup $common -body { + variable BASE + set f [file join $BASE dos.dll] + writebytes $f [binary format a2x58ix64a4x20 MZ 128 XX\x00\x00] + lappend result [fmtarch [classify $f]] + }\ + -result {{unknown unknown}} + + # -- --- --- classifier: ELF and Mach-O --- --- -- + + test binaryarch_classify_elf_fixtures {ELF i386, x86_64 and arm64 fixtures classify via e_machine}\ + -setup $common -body { + variable BASE + foreach {name class data machine} {i386 1 1 3 x86_64 2 1 62 arm64 2 1 183} { + set f [file join $BASE elf_$name.so] + writebytes $f [elfbytes $class $data $machine] + lappend result [fmtarch [classify $f]] + } + set result + }\ + -result {{elf i386} {elf x86_64} {elf arm64}} + + test binaryarch_classify_elf_bigendian {a big-endian ELF (EI_DATA=2) reads e_machine big-endian}\ + -setup $common -body { + variable BASE + set f [file join $BASE elf_be.so] + writebytes $f [elfbytes 2 2 21] + lappend result [fmtarch [classify $f]] + }\ + -result {{elf ppc64}} + + test binaryarch_classify_macho_fixtures {thin Mach-O x86_64 and arm64 (little-endian on-disk form) classify via cputype}\ + -setup $common -body { + variable BASE + foreach {name cputype} {x86_64 0x01000007 arm64 0x0100000C} { + set f [file join $BASE macho_$name.dylib] + writebytes $f [binary format iix56 [expr {0xfeedfacf}] [expr {$cputype}]] + lappend result [fmtarch [classify $f]] + } + set result + }\ + -result {{macho x86_64} {macho arm64}} + + test binaryarch_classify_macho_universal {a fat/universal Mach-O reports its member architectures}\ + -setup $common -body { + variable BASE + set f [file join $BASE macho_fat.dylib] + #fat header (big-endian): magic, nfat=2, then 20-byte entries (cputype first) + writebytes $f [binary format IIIIIIIIIIIIx8 [expr {0xcafebabe}] 2\ + [expr {0x01000007}] 3 4096 100 12\ + [expr {0x0100000C}] 0 8192 100 12] + set c [classify $f] + lappend result [dict get $c format] [dict get $c arch] [dict get $c archs] + }\ + -result {macho universal {x86_64 arm64}} + + test binaryarch_classify_java_class_decoy {a java class file (cafebabe with implausible member count) stays unknown}\ + -setup $common -body { + variable BASE + set f [file join $BASE decoy.dll] + writebytes $f [binary format IIx56 [expr {0xcafebabe}] 52] + lappend result [fmtarch [classify $f]] + }\ + -result {{unknown unknown}} + + test binaryarch_classify_nonbinary_content {a text file named .dll and a too-short file are honestly unknown}\ + -setup $common -body { + variable BASE + set f1 [file join $BASE textstub.dll] + writebytes $f1 "#this is a tcl stub pretending to be a dll\n" + set f2 [file join $BASE tiny.dll] + writebytes $f2 MZ + lappend result [fmtarch [classify $f1]] [fmtarch [classify $f2]] + }\ + -result {{unknown unknown} {unknown unknown}} + + # -- --- --- target mapping + segment recognition --- --- -- + + test binaryarch_expected_mapping {canonical target platforms map to expected format/arch; unknown os yields empty}\ + -setup $common -body { + foreach t {win32-x86_64 win32-ix86 win32-arm64 linux-x86_64 freebsd-x86_64 macosx-arm64 weirdos-x86 noarch} { + lappend result [punkboot::utils::platform_expected_binary $t] + } + set result + }\ + -result {{format pe arch x86_64} {format pe arch i386} {format pe arch arm64} {format elf arch x86_64} {format elf arch x86_64} {format macho arch arm64} {} {}} + + test binaryarch_segment_recognition {canonical platform-dir names, the universal macosx folder and recognised vendor spellings discriminate; package dirs do not}\ + -setup $common -body { + foreach seg {win32-x86_64 win32-ix86 linux-x86_64 macosx win-x64 win64 WIN32-X86_64 tcl8.6 BWidget1.10.1 thread2.8.5 x86_64-linux-gnu} { + lappend result [punkboot::utils::platform_discriminated_segment $seg] + } + set result + }\ + -result {1 1 1 1 1 1 1 0 0 0 0} + + # -- --- --- the scan: the motivating case and the exemptions --- --- -- + + test binaryarch_scan_punkluck86_case {the 2026-07-27 punkluck86 case: an x64 thread dll under plain lib_tcl8/ in a win32-ix86 kit is named as a mismatch}\ + -setup $common -body { + set root [mktree luckcase] + #the payload shape that broke the repl: wrong-arch Thread outside any platform + #subdir, beside the runtime's own working 32-bit copy + writebytes [file join $root thread2.8.5 thread285.dll] [pebytes 0x014c] + writebytes [file join $root lib_tcl8 thread2.8.12 thread28.dll] [pebytes [expr {0x8664}]] + set r [archscan $root win32-ix86] + lappend result ok [dict get $r ok] scanned [dict get $r scanned] matched [dict get $r matched] + set m [lindex [dict get $r mismatches] 0] + lappend result file [dict get $m file] arch [dict get $m arch] + }\ + -result {ok 0 scanned 2 matched 1 file lib_tcl8/thread2.8.12/thread28.dll arch x86_64} + + test binaryarch_scan_platform_subdir_exempt {the iocp-2.0.2 multi-arch pair layout produces no mismatch - platform subdirs are exempt without being read}\ + -setup $common -body { + set root [mktree iocppair] + writebytes [file join $root lib_tcl8 iocp-2.0.2 win32-ix86 iocp202.dll] [pebytes 0x014c] + writebytes [file join $root lib_tcl8 iocp-2.0.2 win32-x86_64 iocp202.dll] [pebytes [expr {0x8664}]] + set r [archscan $root win32-ix86] + lappend result ok [dict get $r ok] scanned [dict get $r scanned] exempt [dict get $r exempt] mismatches [llength [dict get $r mismatches]] + }\ + -result {ok 1 scanned 0 exempt 2 mismatches 0} + + test binaryarch_scan_vendor_spelling_exempt {a blend2d-style win-x64 vendor subdir is recognised - no warning for its wrong-arch dll in an ix86 kit}\ + -setup $common -body { + set root [mktree vendorspelling] + writebytes [file join $root lib_tcl9 blend2d1.5 win-x64 blend2d.dll] [pebytes [expr {0x8664}]] + set r [archscan $root win32-ix86] + lappend result ok [dict get $r ok] exempt [dict get $r exempt] mismatches [llength [dict get $r mismatches]] + }\ + -result {ok 1 exempt 1 mismatches 0} + + test binaryarch_scan_exemption_covers_subtree {everything below a platform-discriminated dir is exempt, not just its direct files}\ + -setup $common -body { + set root [mktree subtree] + writebytes [file join $root lib win32-x86_64 sub deeper thing.dll] [pebytes [expr {0x8664}]] + set r [archscan $root win32-ix86] + lappend result ok [dict get $r ok] exempt [dict get $r exempt] + }\ + -result {ok 1 exempt 1} + + test binaryarch_scan_format_mismatch {a wrong-FORMAT library (ELF .so in a windows-target kit) outside platform subdirs is a mismatch}\ + -setup $common -body { + set root [mktree formatmismatch] + writebytes [file join $root lib foreign libfoo.so] [elfbytes 2 1 62] + set r [archscan $root win32-x86_64] + lappend result ok [dict get $r ok] + set m [lindex [dict get $r mismatches] 0] + lappend result file [dict get $m file] format [dict get $m format] + }\ + -result {ok 0 file lib/foreign/libfoo.so format elf} + + test binaryarch_scan_unknown_not_warned {unclassifiable files are counted unknown and never reported as mismatches}\ + -setup $common -body { + set root [mktree unknowns] + writebytes [file join $root lib stub.dll] "#tcl stub\n" + set r [archscan $root win32-x86_64] + lappend result ok [dict get $r ok] scanned [dict get $r scanned] unknown [dict get $r unknown] mismatches [llength [dict get $r mismatches]] + }\ + -result {ok 1 scanned 1 unknown 1 mismatches 0} + + test binaryarch_scan_matching_tree {a tree whose libraries all match the target reports ok with counts}\ + -setup $common -body { + set root [mktree allmatch] + writebytes [file join $root bin tcl9tk90.dll] [pebytes [expr {0x8664}]] + writebytes [file join $root tcl_library dde tcl9dde14.dll] [pebytes [expr {0x8664}]] + set r [archscan $root win32-x86_64] + lappend result ok [dict get $r ok] scanned [dict get $r scanned] matched [dict get $r matched] + }\ + -result {ok 1 scanned 2 matched 2} + + test binaryarch_scan_macho_universal_matches {a universal Mach-O carrying the target's arch matches a macosx target}\ + -setup $common -body { + set root [mktree fattree] + writebytes [file join $root lib libuni.dylib] [binary format IIIIIIIIIIIIx8 [expr {0xcafebabe}] 2\ + [expr {0x01000007}] 3 4096 100 12\ + [expr {0x0100000C}] 0 8192 100 12] + set r1 [archscan $root macosx-arm64] + set r2 [archscan $root macosx-x86_64] + lappend result [dict get $r1 ok] [dict get $r1 matched] [dict get $r2 ok] [dict get $r2 matched] + }\ + -result {1 1 1 1} + + test binaryarch_scan_unmapped_target {a target with no recognised binary mapping reports not-applicable rather than guessing}\ + -setup $common -body { + set root [mktree unmappedtarget] + writebytes [file join $root lib foo.dll] [pebytes 0x014c] + set r [archscan $root weirdos-x86] + lappend result ok [dict get $r ok] scanned [dict get $r scanned] + lappend result reason_says [string match "*no recognised binary format/arch mapping*" [dict get $r reason]] + }\ + -result {ok 1 scanned 0 reason_says 1} + + test binaryarch_scan_missing_directory {a nonexistent folder is refused by name rather than erroring}\ + -setup $common -body { + variable BASE + set r [archscan [file join $BASE no_such_vfs] win32-x86_64] + lappend result [dict get $r ok] [string match "no such directory:*" [dict get $r reason]] + }\ + -result {0 1} + + test binaryarch_scan_does_not_write {the scan writes nothing into the tree it inspects}\ + -setup $common -body { + set root [mktree readonlycheck] + writebytes [file join $root lib thing.dll] [pebytes [expr {0x8664}]] + set before [lsort [glob -nocomplain -directory $root -tails *]] + archscan $root win32-x86_64 + set after [lsort [glob -nocomplain -directory $root -tails *]] + lappend result [expr {$before eq $after}] $before + }\ + -result {1 lib} + + # -- --- --- the trees a bake actually scans --- --- -- + + test binaryarch_scan_real_built_vfs_trees {assembled kit vfs trees in src/_build produce no findings beyond the KNOWN real ones - the advisory must not cry wolf on payloads that work today}\ + -constraints {builtvfsavailable} -setup $common -body { + variable buildfolder + #KNOWN REAL FINDINGS the scan is EXPECTED to report (true positives, not noise): + # zint.dll 2.13.0 in src/vfs/punk8win.vfs/lib_tcl8/ is a 32-bit (i386) PE + # ('file' confirms: PE32 Intel 80386) carried by the x64 tcl8 kits, where its + # unconditional pkgIndex load can never succeed - found by this scan's first + # real sweep 2026-07-27. Filtered here (not required to exist, so fixing the + # payload keeps this test green); anything OUTSIDE this list is a false + # positive or a new payload defect and fails the test. + set known_findings { + {*.exe.vfs:lib_tcl8/zint-2.13.0/zint.dll=i386} + } + set failures [list] + set checked 0 + foreach d [lsort [glob -nocomplain -type d -directory $buildfolder *.exe.vfs]] { + switch -glob -- [file tail $d] { + punkluck86.* - punk91ix86.* {set target win32-ix86} + default {set target win32-x86_64} + } + incr checked + set r [archscan $d $target] + if {![dict get $r ok]} { + foreach m [dict get $r mismatches] { + set finding "[file tail $d]:[dict get $m file]=[dict get $m arch]" + set is_known 0 + foreach pat $known_findings { + if {[string match $pat $finding]} { + set is_known 1 + break + } + } + if {!$is_known} { + lappend failures $finding + } + } + } + } + lappend result [expr {$checked > 0}] $failures + }\ + -result {1 {}} + + cleanupTests +} diff --git a/src/tests/shell/testsuites/punkexe/maketclpayloadcheck.test b/src/tests/shell/testsuites/punkexe/maketclpayloadcheck.test new file mode 100644 index 00000000..751cd6a4 --- /dev/null +++ b/src/tests/shell/testsuites/punkexe/maketclpayloadcheck.test @@ -0,0 +1,146 @@ +package require tcltest + +#Piped characterization of the make.tcl side of the G-133 payload/target consistency +#checks: the advisory binary-arch scan at the G-125 gate seam and the post-build +#smoke-require probe. Runs the WORKING TREE's src/make.tcl under the built punk +#executable's 'script' subcommand with output captured through a pipe, and pins: +# - 'check' reports the arch scan with one of the two documented statuses (ACTIVE with +# a current bootsupport punkboot::utils carrying vfs_binary_arch_report; UNAVAILABLE +# is the documented guarded-require degradation), states its advisory nature, the +# platform-discriminated-subdir exemption and the recapped BUILD-WARNING reporting +# - 'check' describes the smoke-require probe contract (host-runnable kits only, +# cross-target kits skip with a stated reason, undeclared kits run nothing new) and +# lists the kits declaring smoke-require packages. The declared list is +# characterization of the current src/runtime/mapvfs.config - a deliberate config +# change legitimately updates it (same stance as maketclbakelist.test row pins) +# - the report stays ESC-free under the G-113 piped colour policy +#The scan's VERDICT - what classifies and what mismatches - is characterized against +#generated header fixtures in modules/punkboot/utils/testsuites/utils/binaryarch.test; +#the classifier lives in punkboot::utils precisely so it is testable without a bake. +# +#Target executable resolved from env(PUNK_SHELL_TEST_EXE), else /bin/punk902z.exe +#then /bin/punkshell902. Skipped (constraint punkexeavailable) if none found. + +namespace eval ::testspace { + namespace import ::tcltest::* + + variable testdir [file dirname [file normalize [info script]]] + #/src/tests/shell/testsuites/punkexe -> 5 levels up to + variable projectroot [file normalize [file join $testdir .. .. .. .. ..]] + variable maketcl [file join $projectroot src make.tcl] + + variable punkexe "" + if {[info exists ::env(PUNK_SHELL_TEST_EXE)] && $::env(PUNK_SHELL_TEST_EXE) ne ""} { + set punkexe [file normalize $::env(PUNK_SHELL_TEST_EXE)] + } else { + foreach candidate [list [file join $projectroot bin punk902z.exe] [file join $projectroot bin punkshell902]] { + if {[file exists $candidate]} { + set punkexe $candidate + break + } + } + } + testConstraint punkexeavailable [expr {$punkexe ne "" && [file exists $punkexe]}] + + variable maketcl_run_timeout_ms 60000 + + variable runstate + array set runstate {} + + proc maketcl_run_read {chan} { + variable runstate + append runstate(output) [read $chan] + if {[chan eof $chan]} { + chan event $chan readable {} + set runstate(done) eof + } + } + + #Run script src/make.tcl with output captured through a + #pipe (stdin half-closed for immediate EOF - make.tcl must never wait on stdin for + #these subcommands). Returns dict: timedout 0|1, exitcode , output + #. + proc maketcl_run {cmdargs} { + variable runstate + variable maketcl_run_timeout_ms + variable punkexe + variable maketcl + array unset runstate + set runstate(output) "" + set runstate(done) "" + + set chan [open |[list $punkexe script $maketcl {*}$cmdargs 2>@1] r+] + chan configure $chan -blocking 0 -translation binary + catch {chan close $chan write} ;#no stdin for the child - immediate EOF + set timerid [after $maketcl_run_timeout_ms [list set [namespace current]::runstate(done) timeout]] + chan event $chan readable [list [namespace current]::maketcl_run_read $chan] + while {$runstate(done) eq ""} { + vwait [namespace current]::runstate(done) + } + after cancel $timerid + set timedout [expr {$runstate(done) eq "timeout"}] + set exitcode "" + if {$timedout} { + catch {exec {*}[auto_execok taskkill] /F /T /PID [lindex [pid $chan] 0]} + catch {chan close $chan} + } else { + chan configure $chan -blocking 1 + if {[catch {chan close $chan} errdata errdict]} { + set exitcode [lindex [dict get $errdict -errorcode] end] + } else { + set exitcode 0 + } + } + return [dict create timedout $timedout exitcode $exitcode output $runstate(output)] + } + + #added 2026-07-27 (agent, G-133) + test maketcl_check_payloadchecks_status {make.tcl check reports the payload/target arch scan with one of the two documented statuses, ESC-free}\ + -constraints {punkexeavailable} -body { + set r [maketcl_run {check}] + set out [dict get $r output] + set result [list] + lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode] + lappend result esc [regexp -all {\x1b} $out] + lappend result scanline [regexp -line {^payload/target arch scan \(G-133\): (ACTIVE \(advisory\)|UNAVAILABLE)} $out] + lappend result smokeline [regexp -line {^smoke-require probe \(G-133, advisory\):} $out] + set result + }\ + -result {timedout 0 exitcode 0 esc 0 scanline 1 smokeline 1} + + #added 2026-07-27 (agent, G-133) + test maketcl_check_payloadchecks_active_contract {with a current bootsupport snapshot the arch scan is ACTIVE and check states the advisory contract: subdir exemption, recapped warnings, cross-target coverage and skip, undeclared-kits-unchanged}\ + -constraints {punkexeavailable} -body { + set r [maketcl_run {check}] + set out [dict get $r output] + set result [list] + lappend result active [regexp -line {^payload/target arch scan \(G-133\): ACTIVE \(advisory\)\s*$} $out] + #structural coverage claim: nothing executed, cross-target kits included + lappend result crosstarget [regexp {nothing executed, cross-target kits included} $out] + #advisory reporting channel (appears for both the scan and the smoke probe) + lappend result recapped [expr {[regexp -all {recapped\s+BUILD-WARNINGs} $out] >= 1}] + #the exemption that keeps multi-arch payloads legitimate + lappend result exemption [regexp {platform-discriminated subdirs} $out] + #smoke probe contract lines + lappend result seesresolution [regexp {resolution-order defects} $out] + lappend result skipstated [regexp {cross-target kits skip with a stated reason} $out] + lappend result undeclared [regexp {undeclared kits run nothing new} $out] + set result + }\ + -result {active 1 crosstarget 1 recapped 1 exemption 1 seesresolution 1 skipstated 1 undeclared 1} + + #added 2026-07-27 (agent, G-133) - the declared list characterizes the current + #src/runtime/mapvfs.config smoke-require matrix (deliberate config changes update it) + test maketcl_check_payloadchecks_declared_kits {check lists the kits currently declaring smoke-require packages}\ + -constraints {punkexeavailable} -body { + set r [maketcl_run {check}] + set out [dict get $r output] + set result [list] + lappend result declared [regexp -line {^\s+declared: punkluck86 \(Thread\); punkshell902 \(Thread\); punk91ix86 \(Thread, iocp\)\s*$} $out] + set result + }\ + -result {declared 1} + + cleanupTests +} +namespace delete ::testspace