diff --git a/AGENTS.md b/AGENTS.md index 0c2854f0..07913555 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -185,7 +185,7 @@ The project version is fully independent of module versions. A module bump (even - Directories agents should not directly modify (no child DOX needed): - `callbacks/` — Experimental shellspy features, user-only - `scriptlib/` — Shared utilities + manual tests, user-only. EXCEPTION: `scriptlib/_punktest/` is test-owned (fixtures for `src/tests/shell/testsuites/punkexe/scriptexec.test`, resolved via `lib:_punktest/`); agents may manage that subfolder as part of test work. The rest of `scriptlib/` stays user-only. - - `bin/` — Built binaries and helpers, build output target + - `bin/` — Built binaries and helpers, build output target. This includes the polyglot `.cmd` launcher/utility scripts (e.g `bin/runtime.cmd`): they are GENERATED by the punk::mix scriptwrap machinery from sources under `src/scriptapps/` — a request to "fix bin/.cmd" means editing `src/scriptapps/.*` + `_wrap.toml` and re-wrapping (see bin/AGENTS.md), never editing the output - `modules/` (root) — Build output target for `tclsh src/make.tcl modules` - `lib/` (root) — Build output target for `tclsh src/make.tcl libs` - `modules_tcl8/`, `modules_tcl9/` — Tcl version-specific build output targets diff --git a/bin/AGENTS.md b/bin/AGENTS.md index 2bb98e38..511c7026 100644 --- a/bin/AGENTS.md +++ b/bin/AGENTS.md @@ -4,6 +4,27 @@ Built punk shell executables (kits with the punk boot layer), assorted build/experiment tooling, and plain runtime kits under `runtime/`. Executables here are build outputs - agents do not hand-edit binaries. +### Generated polyglot .cmd scripts - never edit in place + +The `.cmd` scripts here (e.g `runtime.cmd`) are punk MULTISHELL polyglots GENERATED by +`punk::mix::commandset::scriptwrap::multishell` from sources under `src/scriptapps/`: +payload scripts (`.ps1`, `.bash`, ...) plus a `_wrap.toml` config, +spliced into the `punk.multishell.cmd` template. The polyglot structure is deliberately +fragile (mutual shell-hiding tricks, LF-only endings, cmd.exe's 512-byte label-scanner +constraints) - a hand-edit can silently break one of the participating shells. + +When asked to "fix bin/.cmd": +1. Edit the payload/config sources under `src/scriptapps/` (never the output file). +2. Re-wrap from `src/scriptapps/`: `punk::mix::commandset::scriptwrap::multishell -askme 0` + (output defaults to `/bin`; requires Thread + the punk::mix modules, e.g + a punk shell or a tclsh with project module paths). +3. The wrapper runs `checkfile` (the 512-byte label/boundary validator) automatically - + heed ERROR output; "possibly bogus target" warnings are normal for polyglots. +4. Commit source AND regenerated output together: the scriptwrap test suite pins that a + re-wrap of the runtime scriptset reproduces `bin/runtime.cmd` byte for byte + (`src/tests/modules/punk/mix/testsuites/scriptwrap/multishell.test` + scriptwrap_runtime_cmd_roundtrip_no_drift), so drift in either direction fails tests. + ## Local Contracts ### Launch package modes (built punk shells) diff --git a/src/scriptapps/AGENTS.md b/src/scriptapps/AGENTS.md index 02577f05..1749e1ab 100644 --- a/src/scriptapps/AGENTS.md +++ b/src/scriptapps/AGENTS.md @@ -13,6 +13,7 @@ Standalone Tcl scripts that serve as entry points for Punk applications and util - Scripts here are invoked directly by the user or by build tools, not loaded as packages. - Wrapper configs (`.toml` files) define how scripts are packaged as platform executables. +- `_wrap.toml` scriptsets are the SOURCE of the generated polyglot `.cmd` scripts in `/bin` (e.g `runtime.ps1` + `runtime.bash` + `runtime_wrap.toml` -> `bin/runtime.cmd` via `punk::mix::commandset::scriptwrap::multishell runtime -askme 0`, run from this folder). Fixes to a `bin/*.cmd` polyglot are made HERE and re-wrapped - never in the output file. Regenerate and commit the bin output together with payload changes: `src/tests/modules/punk/mix/testsuites/scriptwrap/multishell.test` pins the runtime scriptset round-trip byte-identical against `bin/runtime.cmd`. - The `spud/` directory holds the spud build tool's app scripts. - The `tools/` directory holds miscellaneous build and deployment utilities. diff --git a/src/tests/modules/AGENTS.md b/src/tests/modules/AGENTS.md index d34364dc..5572b45e 100644 --- a/src/tests/modules/AGENTS.md +++ b/src/tests/modules/AGENTS.md @@ -41,7 +41,7 @@ Unit tests for editable source modules under `src/modules/`, `src/modules_tcl8/` - `punk/ansi/` — punk::ansi tests (`testsuites/ansi/`): ansistrip/ansimerge, plus characterization of the ANSI-at-position mechanisms (`ansistring.test`: INDEX/INDEXCODE/INDEXCHAR/RANGE/INSERT grapheme indexing with SGR-prefix merging, INDEXCOLUMNS/COLUMNINDEX double-wide column mapping, trim/VIEW), code splitting invariants (`ta.test`: detect/detectcode distinction, split_codes/split_codes_single/split_at_codes shapes and round-trip) and single-code/effective-state semantics (`codetype.test`: is_sgr_reset/has_sgr_leadingreset, has_any/all_effective, sgr_merge, sequence_type classify). ANSI codes in these tests are literal escape strings so results are colour-state independent - `punk/args/` — punk::args tests (`testsuites/args/`): parsing, choices/choicegroups, forms, rendering/indentation characterization, usage-marking characterization (`usagemarking.test`: -parsedargs/-badarg/-parsestatus/-scheme marking primitives plus goodchoice highlighting of selected/default-in-effect choice words, asserted by SGR-parameter subset against the live colour arrays; the G-049 nocolour/colour-leak GAP pins flipped 2026-07-10 to scheme-statelessness assertions), and the G-049 parse-status structure (`parsestatus.test`: punk::args::parse_status overall/per-argument statuses, badarg for type/allocation failures, -caller attribution, errorcode -argspecs stripping) - `punk/ns/` — punk::ns tests (`testsuites/ns/`): cmdwhich/cmdinfo/cmd_traverse doc-lookup flow (`cmdflow.test`, G-040 parity) and cmdhelp usage-rendering integration (`cmdhelp.test`: scheme selection, goodarg/badarg marking incl type/allocation failures, goodchoice highlighting of supplied/default choice words, alias path, cmdinfo result shape, queried-command failure attribution, and `-return dict` parse-status returns (G-049 - its GAP pins flipped 2026-07-10); remaining GAP pins for pseudo-command cmdtype + space-form docid prefixes (G-051, real `string is` pins behind the have_tclcoredocs constraint), TclOO undocumented-method fallback (G-052), and synopsis marking absence (G-050)) -- `punk/mix/` — punk::mix::cli tests (prune helpers, punkcheck virtual sources) and punk::mix::commandset::repo fossil move/rename characterization tests (`testsuites/repo/`, FOSSIL_HOME-isolated; GAP-marked tests pin behaviour G-022 will change) +- `punk/mix/` — punk::mix::cli tests (prune helpers, punkcheck virtual sources), punk::mix::commandset::repo fossil move/rename characterization tests (`testsuites/repo/`, FOSSIL_HOME-isolated; GAP-marked tests pin behaviour G-022 will change), and the MULTISHELL polyglot build machinery (`testsuites/scriptwrap/multishell.test`: scriptset wrap via the punk.multishell.cmd template - structure/LF-only/determinism, checkfile 512-byte label validation of fresh wraps AND the committed bin/runtime.cmd, the runtime scriptset round-trip byte-identity pin, and platform-gated execution smoke: cmd.exe→powershell payload on windows, sh payload on unix) - `punk/lib/` — punk::lib tests (`testsuites/lib/`): range/index/parse/compat/interp_sync utilities, and G-058 static-baseline seeding (`staticseed.test`: interp_sync_package_paths/snapshot_package_paths propagate a simulated ::punkboot static baseline and seed `load {} ` ifneeded mappings; no-op without a baseline) - `punk/packagepreference/` — punk::packagepreference tests (`testsuites/packagepreference/`): G-058 static-vs-bundled policy (`staticpolicy.test`: require of a baseline package triggers the index scan before resolution so a newer bundled copy wins, static beats older bundled, exact requires of bundled versions stay reachable, missing static mappings get seeded) - `punk/libunknown/` — .tm same-version shadowing pin-tests (`testsuites/shadowing/`): tcl::tm::add prepend rule, head-of-tm-list wins exact-version ties, version beats order, punk::libunknown parity — shipped behaviour depends on these (runtests tm ordering, punk_main package-mode precedence, G-033); mixed .tm/pkgIndex.tcl characterization is goal G-035 diff --git a/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell.test b/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell.test new file mode 100644 index 00000000..f2c82f8d --- /dev/null +++ b/src/tests/modules/punk/mix/testsuites/scriptwrap/multishell.test @@ -0,0 +1,236 @@ +package require tcltest + +package require Thread ;#punk::fileline textinfo (used by scriptwrap checkfile) calls thread::id +package require punk::mix::commandset::scriptwrap + +#Characterization of the punk MULTISHELL polyglot build machinery +#(punk::mix::commandset::scriptwrap::multishell + checkfile) - added 2026-07-10 at the +#user's direction: the polyglot technique is deliberately maintained despite its +#fragility, so the machinery that generates bin/*.cmd (e.g bin/runtime.cmd from +#src/scriptapps/runtime.ps1 + runtime.bash + runtime_wrap.toml) needs pins. +# +#Covered: +# - wrapping a scriptset (payload scripts + _wrap.toml) via the module-provided +# punk.multishell.cmd template produces the polyglot with payloads embedded verbatim +# - the output is LF-only (the cmd.exe label scanner reads 512-byte chunks; the whole +# mechanism depends on unix line endings - see checkfile comments in the module) +# - wrapping is deterministic (byte-identical on re-wrap) +# - checkfile (the 512-byte label/boundary validator) reports no label location errors +# for a fresh wrap AND for the committed bin/runtime.cmd - guarding against hand-edits +# or line-ending damage to the shipped artifact +# - execution smoke: the same file runs under cmd.exe on windows (dispatching to the +# configured win32 nextshell - powershell) and under sh/bash on unix. On windows the +# shell layer deliberately relaunches via the win32 nextshell, so the sh payload is +# execution-tested only on unix (constraint-gated); its embedding is asserted +# structurally everywhere. +# +#NOTE for agents: bin/*.cmd files are GENERATED - fix the payloads under src/scriptapps/ +#(.ps1/.bash/.tcl + _wrap.toml) and re-wrap; do not edit the polyglot output +#(see bin/AGENTS.md and src/scriptapps/AGENTS.md). + +namespace eval ::testspace { + namespace import ::tcltest::* + variable common { + set result "" + } + + variable testdir [file dirname [file normalize [info script]]] + #/src/tests/modules/punk/mix/testsuites/scriptwrap -> 7 levels up + variable projectroot [file normalize [file join $testdir .. .. .. .. .. .. ..]] + + variable ps_payload {Write-Host "MARKER_POWERSHELL_OK"} + variable sh_payload {echo "MARKER_SHELL_OK"} + + #write with explicit lf translation - payload/toml files are consumed byte-wise + proc writefile_lf {path content} { + set fd [open $path w] + fconfigure $fd -translation lf + puts $fd $content + close $fd + } + variable fixdir [makeDirectory scriptwrap_fixture] + writefile_lf $fixdir/wrapset.ps1 $ps_payload + writefile_lf $fixdir/wrapset.sh $sh_payload + writefile_lf $fixdir/wrapset_wrap.toml { +[application] + template="punk.multishell.cmd" + scripts=[ + "wrapset.ps1", + "wrapset.sh" + ] + default_outputfile="wrapset.cmd" + default_nextshellpath="/usr/bin/env bash" + default_nextshelltype="bash" + win32.nextshellpath="cmd.exe /c powershell -nop -nol -ExecutionPolicy bypass -File" + win32.nextshelltype="powershell" + win32.outputfile="wrapset.cmd" +} + + #wrap once at load - most tests examine this output (multishell chats on stdout; that + #output is not part of test comparisons) + variable wrapresult "" + variable outfile "" + variable wrap_ok 0 + variable startdir [pwd] + cd $fixdir + if {![catch {punk::mix::commandset::scriptwrap::multishell wrapset -outputfolder $fixdir -askme 0 -force 1} wrapresult]} { + if {[dict exists $wrapresult filename]} { + set outfile [dict get $wrapresult filename] + set wrap_ok [file exists $outfile] + } + } + cd $startdir + testConstraint wrapok $wrap_ok + testConstraint iswindows [expr {$::tcl_platform(platform) eq "windows"}] + testConstraint isunix [expr {$::tcl_platform(platform) eq "unix"}] + + proc readbytes {path} { + set fd [open $path r] + fconfigure $fd -translation binary + set data [read $fd] + close $fd + return $data + } + + test scriptwrap_multishell_wrap_produces_output {wrapping a scriptset with payloads + _wrap.toml produces the polyglot output file}\ + -setup $common -body { + variable wrapresult + variable outfile + lappend result [dict exists $wrapresult filename] + lappend result [file exists $outfile] + lappend result [file tail $outfile] + }\ + -cleanup { + }\ + -result [list 1 1 wrapset.cmd] + + test scriptwrap_multishell_output_structure {output starts with the MULTISHELL magic line, embeds both payloads verbatim between shell-payload markers, and contains the configured win32 nextshell}\ + -constraints wrapok\ + -setup $common -body { + variable outfile + variable ps_payload + variable sh_payload + set data [readbytes $outfile] + lappend result [string match {: "punk MULTISHELL*} $data] + lappend result [expr {[string first $ps_payload $data] >= 0}] + lappend result [expr {[string first $sh_payload $data] >= 0}] + lappend result [expr {[string first "#" $data] >= 0}] + lappend result [expr {[string first "powershell -nop -nol -ExecutionPolicy bypass -File" $data] >= 0}] + }\ + -cleanup { + }\ + -result [list 1 1 1 1 1] + + test scriptwrap_multishell_output_lf_only {the generated polyglot uses LF line endings exclusively (cmd label scanning depends on it)}\ + -constraints wrapok\ + -setup $common -body { + variable outfile + set data [readbytes $outfile] + lappend result [expr {[string first \r $data] == -1}] + }\ + -cleanup { + }\ + -result [list 1] + + test scriptwrap_multishell_wrap_deterministic {re-wrapping the same scriptset produces byte-identical output}\ + -constraints wrapok\ + -setup $common -body { + variable fixdir + variable outfile + variable startdir + set outdir2 [makeDirectory scriptwrap_fixture_rewrap] + cd $fixdir + set res2 [punk::mix::commandset::scriptwrap::multishell wrapset -outputfolder $outdir2 -askme 0 -force 1] + cd $startdir + set outfile2 [dict get $res2 filename] + lappend result [file exists $outfile2] + lappend result [expr {[readbytes $outfile] eq [readbytes $outfile2]}] + }\ + -cleanup { + cd $startdir + }\ + -result [list 1 1] + + test scriptwrap_checkfile_generated_no_label_errors {checkfile reports no 512-byte label location errors for a fresh wrap}\ + -constraints wrapok\ + -setup $common -body { + variable outfile + set summary [punk::mix::commandset::scriptwrap::checkfile $outfile] + lappend result [string match "*ERROR: label location errors*" $summary] + #sanity: the analysis actually ran (labels enumerated) + lappend result [string match "*call-labels-found:*" $summary] + }\ + -cleanup { + }\ + -result [list 0 1] + + test scriptwrap_checkfile_committed_runtime_cmd {the committed bin/runtime.cmd passes checkfile - no label/boundary damage from hand-edits or line-ending conversion}\ + -setup $common -body { + variable projectroot + set target [file join $projectroot bin runtime.cmd] + if {![file exists $target]} { + #not built/present in this checkout - treat as vacuous pass with marker + lappend result no_runtime_cmd + } else { + set data [readbytes $target] + #LF-only is part of the artifact's contract + lappend result [expr {[string first \r $data] == -1}] + set summary [punk::mix::commandset::scriptwrap::checkfile $target] + lappend result [string match "*ERROR: label location errors*" $summary] + } + }\ + -cleanup { + }\ + -result [list 1 0] + + test scriptwrap_runtime_cmd_roundtrip_no_drift {re-wrapping the runtime scriptset from src/scriptapps reproduces the committed bin/runtime.cmd byte for byte - the artifact is in sync with its sources and un-hand-edited}\ + -setup $common -body { + variable projectroot + variable startdir + set target [file join $projectroot bin runtime.cmd] + set scriptapps [file join $projectroot src scriptapps] + if {![file exists $target] || ![file exists $scriptapps/runtime_wrap.toml]} { + lappend result no_runtime_sources + } else { + set outdir [makeDirectory scriptwrap_runtime_roundtrip] + cd $scriptapps + set res [punk::mix::commandset::scriptwrap::multishell runtime -outputfolder $outdir -askme 0 -force 1] + cd $startdir + set fresh [dict get $res filename] + lappend result [expr {[readbytes $fresh] eq [readbytes $target]}] + #a failure here means either bin/runtime.cmd was edited directly (fix the + #src/scriptapps payloads instead and re-wrap), or the payloads/template + #changed without regenerating bin/runtime.cmd (re-wrap and commit it) + } + }\ + -cleanup { + cd $startdir + }\ + -result [list 1] + + test scriptwrap_multishell_exec_windows_cmd {on windows, cmd.exe runs the polyglot and dispatches to the configured win32 nextshell (powershell payload executes)}\ + -constraints {wrapok iswindows}\ + -setup $common -body { + variable outfile + set comspec [expr {[info exists ::env(ComSpec)] ? $::env(ComSpec) : "cmd.exe"}] + set out "" + catch {set out [exec -ignorestderr -- $comspec /c [file nativename $outfile] << ""]} out + lappend result [string match "*MARKER_POWERSHELL_OK*" $out] + }\ + -cleanup { + }\ + -result [list 1] + + test scriptwrap_multishell_exec_unix_sh {on unix, sh/bash runs the polyglot's shell payload}\ + -constraints {wrapok isunix}\ + -setup $common -body { + variable outfile + set out "" + catch {set out [exec -ignorestderr -- sh $outfile << ""]} out + lappend result [string match "*MARKER_SHELL_OK*" $out] + }\ + -cleanup { + }\ + -result [list 1] +} +tcltest::cleanupTests ;#needed to produce test summary line.