From ac7dcdb02a27e12cb141f8a203e9d3caf9eab1c0 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Sat, 25 Jul 2026 23:38:33 +1000 Subject: [PATCH] colour policy: never wrap utf-16 console channels; 8.6 windows console auto-detected by encoding signature Defect: the G-113 ansistrip transform was pushed onto Tcl 8.6 windows CONSOLE channels, which encode utf-16 ('unicode') - at byte level ESC arrives as 1B 00 so sequences passed through unstripped and the hold-back carry deferred/ dropped real content, seriously mangling interactive punk86/punksys 'src/make.tcl help' layout (user-reported). Chunk-boundary handling itself is sound: a rig proved strip output invariant across chunk sizes 1..4096 over byte-transparent encodings; the failing dimension was encoding, not cutting. (Comparison recorded: shellfilter::chan::ansistrip is the complement - it decodes per the channel encoding and buffers undecodable tails, but assumes line-buffering for sequence splits.) Fix is detection, not utf-16 stripping: only 8.6 windows console channels report -encoding unicode (pipes/files get the system encoding; hidden-console probes confirmed across bawt tclsh86 8.6.13, punk86 8.6.17, punksys 8.6.13; tcl9 consoles report utf-16 but are caught by -winsize first). The policy now treats that signature as tty: mode=tty, colour ON, no transform - interactive 8.6 windows colour with no env var and no twapi. The transform push is now PER CHANNEL and skips utf-16-class channels ('> file' from an 8.6 console wraps stdout only - stderr console keeps its ANSI); ansistrip_pushed is the wrapped-channel list, shell pops exactly those, policy/check lines print ansistrip=stdout+stderr|stdout|0. Verified: hidden-console 8.6 x3 mode=tty ansistrip=0; half-redirected punk86 console run ansistrip=stdout; piped 8.6/9 batteries unchanged (zero ESC, force re-colours); punkexe suites 54/0 (maketclcolour pins updated to the channel-list format). punkshell 0.23.2 + CHANGELOG; src/AGENTS bullet updated. 8.6 unix/WSL stays plain-by-default pending the probe-ladder goal. Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- CHANGELOG.md | 17 +++++ punkproject.toml | 2 +- src/AGENTS.md | 2 +- src/make.tcl | 74 +++++++++++++------ .../testsuites/punkexe/maketclcolour.test | 4 +- 5 files changed, 72 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 42e4da89..7d0b319e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,23 @@ The latest `## [X.Y.Z]` header must match the `version` field in `punkproject.to Entries are newest-first; one bullet per notable change. See the root `AGENTS.md` "Project Versioning" section for the bump policy. +## [0.23.2] - 2026-07-25 + +- make.tcl colour policy defect fix + 8.6 windows console auto-detection: the G-113 + ansistrip transform was being pushed onto Tcl 8.6 windows CONSOLE channels, which + encode utf-16 ("unicode") - at byte level ESC arrives as 1B 00, so sequences passed + unstripped and the hold-back logic deferred/dropped real content, seriously mangling + interactive `punk86`/`punksys src/make.tcl help` layout. The 8.6 windows console is + now detected by that very encoding signature (only console channels report + `-encoding unicode`; pipes/files get the system encoding): mode=tty, colour ON, no + transform - interactive 8.6 windows colour works with no env var and no twapi. + The transform push is now PER CHANNEL and never wraps a utf-16-class channel (e.g + `make.tcl ... > file` from an 8.6 console wraps stdout only; the console stderr + keeps its ANSI). Policy/check lines report the wrapped channels + (`ansistrip=stdout+stderr|stdout|0`). Chunk-boundary handling verified sound over + byte-transparent encodings (invariant across chunk sizes 1..4096); 8.6 unix/WSL + remains plain-by-default pending the probe-ladder goal. + ## [0.23.1] - 2026-07-25 - punk-runtime `use` accepts runtime/artifact names with or without the `.exe` suffix diff --git a/punkproject.toml b/punkproject.toml index 03164c5f..c8f60bd1 100644 --- a/punkproject.toml +++ b/punkproject.toml @@ -1,4 +1,4 @@ [project] name = "punkshell" -version = "0.23.1" +version = "0.23.2" license = "BSD-2-Clause" diff --git a/src/AGENTS.md b/src/AGENTS.md index 0784278b..14546ba5 100644 --- a/src/AGENTS.md +++ b/src/AGENTS.md @@ -63,7 +63,7 @@ Recovery after a wrong path guess: - Use `tclsh src/make.tcl modules` to build just the module packages. - Use `tclsh src/make.tcl libs` to build just the library packages. - Use `tclsh src/make.tcl packages` to build both modules and libraries. -- make.tcl colour is terminal-aware (G-113): piped/redirected runs (agent harnesses, CI, log capture) automatically produce fully ESC-free output with no caller action required - an ansistrip channel transform on stdout+stderr guarantees zero ESC bytes for every emitter, including module-side ones (punkcheck summaries, punk::args tables). Interactive terminal runs keep colour (stdout tty probe via the `-winsize` channel option, Tcl 8.7+/9). Precedence: `NO_COLOR` (any value) always suppresses colour; `PUNK_FORCE_COLOR`/`FORCE_COLOR` (value other than 0/false/no/off) re-enables ANSI on piped output; otherwise the probe decides. Tcl 8.6 has no boot-phase probe - colour fails OFF there (use the force vars for 8.6 interactive colour). stderr follows the stdout decision; `make.tcl shell` pops the transform (repl colour is the shell's own concern). `tclsh src/make.tcl check` reports the active policy (`colour policy (G-113): mode=...` line; modes tty|forced|nocolor|piped-plain|tcl86-plain). Pinned by `src/tests/shell/testsuites/punkexe/maketclcolour.test`. Agents no longer need to set NO_COLOR for captured runs (setting it remains harmless). +- make.tcl colour is terminal-aware (G-113): piped/redirected runs (agent harnesses, CI, log capture) automatically produce fully ESC-free output with no caller action required - an ansistrip channel transform on stdout+stderr guarantees zero ESC bytes for every emitter, including module-side ones (punkcheck summaries, punk::args tables). Interactive terminal runs keep colour (stdout tty probe via the `-winsize` channel option, Tcl 8.7+/9). Precedence: `NO_COLOR` (any value) always suppresses colour; `PUNK_FORCE_COLOR`/`FORCE_COLOR` (value other than 0/false/no/off) re-enables ANSI on piped output; otherwise the probe decides. Tcl 8.6 windows consoles are auto-detected via the console channel's utf-16 encoding signature (`-encoding unicode` - only 8.6 console channels report it), so interactive 8.6 windows runs keep colour; the byte-level strip transform is never pushed onto a utf-16-class channel (it would corrupt the stream - per-channel push, so `> file` from an 8.6 console wraps stdout only). 8.6 on unix/WSL has no probe - colour fails OFF there (use the force vars). stderr follows the stdout decision; `make.tcl shell` pops the transform (repl colour is the shell's own concern). `tclsh src/make.tcl check` reports the active policy (`colour policy (G-113): mode=...` line; modes tty|forced|nocolor|piped-plain|tcl86-plain). Pinned by `src/tests/shell/testsuites/punkexe/maketclcolour.test`. Agents no longer need to set NO_COLOR for captured runs (setting it remains harmless). - Use `tclsh src/make.tcl vendorupdate` to refresh vendormodules from config. It warns (non-fatal) for each source project whose fossil/git checkout is dirty - vendored artifacts built from a dirty tree have no committed provenance; commit in the source project to clear the warning (enforcement policy tracked by goal G-026). - All build/promotion commands (`bakehouse`, `packages`, `modules`, `libs`, `bake`, `vfslibs`, `bin`, `bootsupport`, `vfscommonupdate`) warn similarly when this project's own `src/` has uncommitted fossil/git changes (dirt outside `src/` is ignored). Warn-only by default EXCEPT `bakehouse` (aborts by default; `-dirty-abort 0` overrides); pass `-dirty-abort` elsewhere to make the check aborting. For evaluating uncommitted source without a build, prefer ` src` / ` src shell`. The check is `punkboot::utils::vcs_dirty_warnings` (optional scope argument) loaded guardedly from bootsupport - if the snapshot is stale/missing the check degrades to a skip notice (but `-dirty-abort` then aborts rather than silently losing the requested strictness). - Provenance warnings (dirty-src gate + vendorupdate source-project check) print with a plain column-0 `PROVENANCE-WARNING:` token (grep for it in captured build output) and are recapped at the end of the run via make.tcl's wrapped `::exit`. Interactive terminal runs get a 3-second ctrl-c grace countdown before a dirty build proceeds; piped/agent runs skip it. `tclsh src/make.tcl check` reports the current src provenance status and what the build commands would do. diff --git a/src/make.tcl b/src/make.tcl index adf24e92..5fed0a2a 100644 --- a/src/make.tcl +++ b/src/make.tcl @@ -11,18 +11,23 @@ # 2. PUNK_FORCE_COLOR / FORCE_COLOR set -> ANSI ON even when piped # (a value of 0/false/no/off is treated as not-set, per ecosystem convention) # 3. stdout tty probe -> colour ON only when stdout is a -# terminal channel (-winsize option present - Tcl 8.7+/9 terminal channels -# only, mirroring the stdin_is_interactive -inputmode precedent) +# terminal channel: -winsize option present (Tcl 8.7+/9 terminal channels, +# mirroring the stdin_is_interactive -inputmode precedent), or - Tcl 8.6 on +# windows - stdout -encoding reporting 'unicode' (utf-16), which only the +# 8.6 windows console channel does (pipes/files get the system encoding). # Piped/redirected stdout (probe fails, no force) additionally pushes a strip -# transform on stdout+stderr so the output is fully ESC-free - machine -# consumers get plain text no matter which emitter produced it (punk::ansi's -# colour_disabled contract deliberately retains non-colour SGR effects, and -# module-side emitters like punkcheck summaries follow that contract - the -# transform is the single choke point that guarantees zero ESC bytes). -# Tcl 8.6 has no boot-phase tty probe (no -winsize/-inputmode anywhere): the -# fail direction is plain/OFF (safe for piped/captured agent runs - unlike -# stdin's assume-interactive; use the force env vars for 8.6 interactive colour). -# stderr follows the same single stdout-keyed decision. +# transform so the output is fully ESC-free - machine consumers get plain text +# no matter which emitter produced it (punk::ansi's colour_disabled contract +# deliberately retains non-colour SGR effects, and module-side emitters like +# punkcheck summaries follow that contract - the transform is the single choke +# point that guarantees zero ESC bytes). The push is PER CHANNEL and skips any +# utf-16-class channel (a byte-level strip corrupts utf-16 - ESC arrives as +# 1B 00; such a channel is an 8.6 console, i.e. a real terminal, and keeps its +# ANSI) - e.g 'make.tcl ... > file' from an 8.6 console wraps stdout only. +# Tcl 8.6 on unix has no boot-phase tty probe at all: the fail direction is +# plain/OFF (safe for piped/captured agent runs - unlike stdin's +# assume-interactive; use the force env vars for 8.6 unix interactive colour). +# stderr colour follows the same single stdout-keyed decision. # The colour switch is ::punk::console::colour_disabled; when punk::ansi is # already loaded (punk-exe-hosted runs) its sgr cache is cleared per that # module's contract ("whatever function disables or re-enables colour should @@ -91,6 +96,20 @@ apply {{} { } } set stdout_is_tty [expr {![catch {chan configure stdout -winsize}]}] + if {!$stdout_is_tty && $::tcl_platform(platform) eq "windows" + && ![package vsatisfies [package provide Tcl] 8.7-]} { + #Tcl 8.6 windows: no -winsize/-inputmode probe exists, but 8.6's windows + #CONSOLE channels are the only ones that report -encoding unicode (utf-16; + #pipes/files get the system encoding) - a dependency-free console signature. + #This both restores interactive colour on 8.6 consoles and is load-bearing + #for the ansistrip transform below: the byte-level strip must never sit on + #a utf-16 channel (ESC arrives as 1B 00 - sequences pass unstripped and the + #hold-back logic defers/drops real content; field defect 2026-07-25 mangling + #punk86/punksys console help output). + if {[chan configure stdout -encoding] eq "unicode"} { + set stdout_is_tty 1 + } + } if {[info exists ::env(NO_COLOR)]} { namespace eval ::punk::console {variable colour_disabled 1} set ::punkboot::colour_mode nocolor @@ -111,12 +130,19 @@ apply {{} { #zero-ESC guarantee for piped/unknown output unless the caller forced ANSI on. #(NO_COLOR keeps punk::ansi's documented colour-only-stripping semantics on a #terminal; a NO_COLOR run with piped stdout still gets the full strip.) + #Per-channel: a byte-level strip must never sit on a utf-16-class channel (the + #8.6 windows console encoding) - e.g 'make.tcl ... > file' from an 8.6 console + #leaves stderr ON the console; that channel is a real terminal, so it keeps its + #ANSI rather than getting corrupted. ::punkboot::ansistrip_pushed records the + #channels actually wrapped (the shell subcommand pops exactly those). + set ::punkboot::ansistrip_pushed [list] if {!$force && !$stdout_is_tty} { - chan push stdout ::punkboot::ansistrip::transchan - chan push stderr ::punkboot::ansistrip::transchan - set ::punkboot::ansistrip_pushed 1 - } else { - set ::punkboot::ansistrip_pushed 0 + foreach _ch {stdout stderr} { + if {[chan configure $_ch -encoding] ni {unicode utf-16 utf-16le utf-16be}} { + chan push $_ch ::punkboot::ansistrip::transchan + lappend ::punkboot::ansistrip_pushed $_ch + } + } } #punk-exe-hosted runs arrive with punk::ansi pre-loaded and a warm sgr cache #generated under the kit's own colour state - honour the module's contract. @@ -124,7 +150,7 @@ apply {{} { catch {punk::ansi::sgr_cache -action clear} } if {[info exists ::env(PUNKBOOT_COLOUR_DEBUG)]} { - puts stderr "punkboot colour policy: mode=$::punkboot::colour_mode colour_disabled=$::punk::console::colour_disabled ansistrip=$::punkboot::ansistrip_pushed" + puts stderr "punkboot colour policy: mode=$::punkboot::colour_mode colour_disabled=$::punk::console::colour_disabled ansistrip=[expr {[llength $::punkboot::ansistrip_pushed] ? [join $::punkboot::ansistrip_pushed +] : 0}]" } }} # Colour-policy gate for raw-SGR emission sites that can run before @@ -2569,8 +2595,8 @@ if {$::punkboot::command in {bakehouse packages modules libs bake vfslibs bin bo if {$::punkboot::command eq "check"} { set sep [string repeat - 75] puts stdout $sep - puts stdout "colour policy (G-113): mode=$::punkboot::colour_mode colour_disabled=$::punk::console::colour_disabled ansistrip=$::punkboot::ansistrip_pushed" - puts stdout " (precedence: NO_COLOR > PUNK_FORCE_COLOR/FORCE_COLOR > stdout tty probe; piped/8.6-unknown output is fully ESC-stripped unless forced)" + puts stdout "colour policy (G-113): mode=$::punkboot::colour_mode colour_disabled=$::punk::console::colour_disabled ansistrip=[expr {[llength $::punkboot::ansistrip_pushed] ? [join $::punkboot::ansistrip_pushed +] : 0}]" + puts stdout " (precedence: NO_COLOR > PUNK_FORCE_COLOR/FORCE_COLOR > stdout tty probe incl. the 8.6 windows console-encoding signature; piped/unknown output is fully ESC-stripped per channel unless forced - utf-16 console channels are never wrapped)" puts stdout $sep puts stdout "module/library checks - paths from bootsupport only" puts stdout $sep @@ -3075,10 +3101,12 @@ if {$::punkboot::command eq "buildsuite"} { if {$::punkboot::command eq "shell"} { #G-113: the ansistrip transform serves make.tcl's own output only - the repl's #colour behaviour is the shell's own concern (colour on/off + NO_COLOR there). - if {[info exists ::punkboot::ansistrip_pushed] && $::punkboot::ansistrip_pushed} { - catch {chan pop stdout} - catch {chan pop stderr} - set ::punkboot::ansistrip_pushed 0 + #Pop exactly the channels the policy block wrapped (per-channel list). + if {[info exists ::punkboot::ansistrip_pushed]} { + foreach _ch $::punkboot::ansistrip_pushed { + catch {chan pop $_ch} + } + set ::punkboot::ansistrip_pushed [list] } package require struct::list package require punk diff --git a/src/tests/shell/testsuites/punkexe/maketclcolour.test b/src/tests/shell/testsuites/punkexe/maketclcolour.test index 7623b099..2a257f51 100644 --- a/src/tests/shell/testsuites/punkexe/maketclcolour.test +++ b/src/tests/shell/testsuites/punkexe/maketclcolour.test @@ -120,7 +120,7 @@ namespace eval ::testspace { set result [list] lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode] lappend result esc [esc_count [dict get $r output]] - lappend result policyline [regexp {colour policy \(G-113\): mode=piped-plain colour_disabled=1 ansistrip=1} [dict get $r output]] + lappend result policyline [regexp {colour policy \(G-113\): mode=piped-plain colour_disabled=1 ansistrip=stdout\+stderr} [dict get $r output]] set result } -result {timedout 0 exitcode 0 esc 0 policyline 1} @@ -141,7 +141,7 @@ namespace eval ::testspace { set result [list] lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode] lappend result esc [esc_count [dict get $r output]] - lappend result policyline [regexp {colour policy \(G-113\): mode=nocolor colour_disabled=1 ansistrip=1} [dict get $r output]] + lappend result policyline [regexp {colour policy \(G-113\): mode=nocolor colour_disabled=1 ansistrip=stdout\+stderr} [dict get $r output]] set result } -result {timedout 0 exitcode 0 esc 0 policyline 1}