jn
/
punkshell
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Browse Source

colour policy: never wrap utf-16 console channels; 8.6 windows console auto-detected by encoding signature 

Defect: the G-113 ansistrip transform was pushed onto Tcl 8.6 windows CONSOLE
channels, which encode utf-16 ('unicode') - at byte level ESC arrives as 1B 00
so sequences passed through unstripped and the hold-back carry deferred/
dropped real content, seriously mangling interactive punk86/punksys
'src/make.tcl help' layout (user-reported). Chunk-boundary handling itself is
sound: a rig proved strip output invariant across chunk sizes 1..4096 over
byte-transparent encodings; the failing dimension was encoding, not cutting.
(Comparison recorded: shellfilter::chan::ansistrip is the complement - it
decodes per the channel encoding and buffers undecodable tails, but assumes
line-buffering for sequence splits.)

Fix is detection, not utf-16 stripping: only 8.6 windows console channels
report -encoding unicode (pipes/files get the system encoding; hidden-console
probes confirmed across bawt tclsh86 8.6.13, punk86 8.6.17, punksys 8.6.13;
tcl9 consoles report utf-16 but are caught by -winsize first). The policy now
treats that signature as tty: mode=tty, colour ON, no transform - interactive
8.6 windows colour with no env var and no twapi. The transform push is now
PER CHANNEL and skips utf-16-class channels ('> file' from an 8.6 console
wraps stdout only - stderr console keeps its ANSI); ansistrip_pushed is the
wrapped-channel list, shell pops exactly those, policy/check lines print
ansistrip=stdout+stderr|stdout|0.

Verified: hidden-console 8.6 x3 mode=tty ansistrip=0; half-redirected punk86
console run ansistrip=stdout; piped 8.6/9 batteries unchanged (zero ESC,
force re-colours); punkexe suites 54/0 (maketclcolour pins updated to the
channel-list format). punkshell 0.23.2 + CHANGELOG; src/AGENTS bullet updated.
8.6 unix/WSL stays plain-by-default pending the probe-ladder goal.

Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
master
Julian Noble 5 days ago
parent
a646f1089f
commit
ac7dcdb02a
5 changed files with 72 additions and 27 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 17
      CHANGELOG.md
  2. 2
      punkproject.toml
  3. 2
      src/AGENTS.md
  4. 74
      src/make.tcl
  5. 4
      src/tests/shell/testsuites/punkexe/maketclcolour.test

17
CHANGELOG.md
Unescape View File

@ -5,6 +5,23 @@ The latest `## [X.Y.Z]` header must match the `version` field in `punkproject.to
Entries are newest-first; one bullet per notable change. See the root `AGENTS.md` Entries are newest-first; one bullet per notable change. See the root `AGENTS.md`
"Project Versioning" section for the bump policy. "Project Versioning" section for the bump policy.
## [0.23.2] - 2026-07-25
- make.tcl colour policy defect fix + 8.6 windows console auto-detection: the G-113
ansistrip transform was being pushed onto Tcl 8.6 windows CONSOLE channels, which
encode utf-16 ("unicode") - at byte level ESC arrives as 1B 00, so sequences passed
unstripped and the hold-back logic deferred/dropped real content, seriously mangling
interactive `punk86`/`punksys src/make.tcl help` layout. The 8.6 windows console is
now detected by that very encoding signature (only console channels report
`-encoding unicode`; pipes/files get the system encoding): mode=tty, colour ON, no
transform - interactive 8.6 windows colour works with no env var and no twapi.
The transform push is now PER CHANNEL and never wraps a utf-16-class channel (e.g
`make.tcl ... > file` from an 8.6 console wraps stdout only; the console stderr
keeps its ANSI). Policy/check lines report the wrapped channels
(`ansistrip=stdout+stderr|stdout|0`). Chunk-boundary handling verified sound over
byte-transparent encodings (invariant across chunk sizes 1..4096); 8.6 unix/WSL
remains plain-by-default pending the probe-ladder goal.
## [0.23.1] - 2026-07-25 ## [0.23.1] - 2026-07-25
- punk-runtime `use` accepts runtime/artifact names with or without the `.exe` suffix - punk-runtime `use` accepts runtime/artifact names with or without the `.exe` suffix

2
punkproject.toml
Unescape View File

@ -1,4 +1,4 @@
[project] [project]
name = "punkshell" name = "punkshell"
version = "0.23.1" version = "0.23.2"
license = "BSD-2-Clause" license = "BSD-2-Clause"

2
src/AGENTS.md
Unescape View File

@ -63,7 +63,7 @@ Recovery after a wrong path guess:
- Use `tclsh src/make.tcl modules` to build just the module packages. - Use `tclsh src/make.tcl modules` to build just the module packages.
- Use `tclsh src/make.tcl libs` to build just the library packages. - Use `tclsh src/make.tcl libs` to build just the library packages.
- Use `tclsh src/make.tcl packages` to build both modules and libraries. - Use `tclsh src/make.tcl packages` to build both modules and libraries.
- make.tcl colour is terminal-aware (G-113): piped/redirected runs (agent harnesses, CI, log capture) automatically produce fully ESC-free output with no caller action required - an ansistrip channel transform on stdout+stderr guarantees zero ESC bytes for every emitter, including module-side ones (punkcheck summaries, punk::args tables). Interactive terminal runs keep colour (stdout tty probe via the `-winsize` channel option, Tcl 8.7+/9). Precedence: `NO_COLOR` (any value) always suppresses colour; `PUNK_FORCE_COLOR`/`FORCE_COLOR` (value other than 0/false/no/off) re-enables ANSI on piped output; otherwise the probe decides. Tcl 8.6 has no boot-phase probe - colour fails OFF there (use the force vars for 8.6 interactive colour). stderr follows the stdout decision; `make.tcl shell` pops the transform (repl colour is the shell's own concern). `tclsh src/make.tcl check` reports the active policy (`colour policy (G-113): mode=...` line; modes tty|forced|nocolor|piped-plain|tcl86-plain). Pinned by `src/tests/shell/testsuites/punkexe/maketclcolour.test`. Agents no longer need to set NO_COLOR for captured runs (setting it remains harmless). - make.tcl colour is terminal-aware (G-113): piped/redirected runs (agent harnesses, CI, log capture) automatically produce fully ESC-free output with no caller action required - an ansistrip channel transform on stdout+stderr guarantees zero ESC bytes for every emitter, including module-side ones (punkcheck summaries, punk::args tables). Interactive terminal runs keep colour (stdout tty probe via the `-winsize` channel option, Tcl 8.7+/9). Precedence: `NO_COLOR` (any value) always suppresses colour; `PUNK_FORCE_COLOR`/`FORCE_COLOR` (value other than 0/false/no/off) re-enables ANSI on piped output; otherwise the probe decides. Tcl 8.6 windows consoles are auto-detected via the console channel's utf-16 encoding signature (`-encoding unicode` - only 8.6 console channels report it), so interactive 8.6 windows runs keep colour; the byte-level strip transform is never pushed onto a utf-16-class channel (it would corrupt the stream - per-channel push, so `> file` from an 8.6 console wraps stdout only). 8.6 on unix/WSL has no probe - colour fails OFF there (use the force vars). stderr follows the stdout decision; `make.tcl shell` pops the transform (repl colour is the shell's own concern). `tclsh src/make.tcl check` reports the active policy (`colour policy (G-113): mode=...` line; modes tty|forced|nocolor|piped-plain|tcl86-plain). Pinned by `src/tests/shell/testsuites/punkexe/maketclcolour.test`. Agents no longer need to set NO_COLOR for captured runs (setting it remains harmless).
- Use `tclsh src/make.tcl vendorupdate` to refresh vendormodules from config. It warns (non-fatal) for each source project whose fossil/git checkout is dirty - vendored artifacts built from a dirty tree have no committed provenance; commit in the source project to clear the warning (enforcement policy tracked by goal G-026). - Use `tclsh src/make.tcl vendorupdate` to refresh vendormodules from config. It warns (non-fatal) for each source project whose fossil/git checkout is dirty - vendored artifacts built from a dirty tree have no committed provenance; commit in the source project to clear the warning (enforcement policy tracked by goal G-026).
- All build/promotion commands (`bakehouse`, `packages`, `modules`, `libs`, `bake`, `vfslibs`, `bin`, `bootsupport`, `vfscommonupdate`) warn similarly when this project's own `src/` has uncommitted fossil/git changes (dirt outside `src/` is ignored). Warn-only by default EXCEPT `bakehouse` (aborts by default; `-dirty-abort 0` overrides); pass `-dirty-abort` elsewhere to make the check aborting. For evaluating uncommitted source without a build, prefer `<builtexe> src` / `<builtexe> src shell`. The check is `punkboot::utils::vcs_dirty_warnings` (optional scope argument) loaded guardedly from bootsupport - if the snapshot is stale/missing the check degrades to a skip notice (but `-dirty-abort` then aborts rather than silently losing the requested strictness). - All build/promotion commands (`bakehouse`, `packages`, `modules`, `libs`, `bake`, `vfslibs`, `bin`, `bootsupport`, `vfscommonupdate`) warn similarly when this project's own `src/` has uncommitted fossil/git changes (dirt outside `src/` is ignored). Warn-only by default EXCEPT `bakehouse` (aborts by default; `-dirty-abort 0` overrides); pass `-dirty-abort` elsewhere to make the check aborting. For evaluating uncommitted source without a build, prefer `<builtexe> src` / `<builtexe> src shell`. The check is `punkboot::utils::vcs_dirty_warnings` (optional scope argument) loaded guardedly from bootsupport - if the snapshot is stale/missing the check degrades to a skip notice (but `-dirty-abort` then aborts rather than silently losing the requested strictness).
- Provenance warnings (dirty-src gate + vendorupdate source-project check) print with a plain column-0 `PROVENANCE-WARNING:` token (grep for it in captured build output) and are recapped at the end of the run via make.tcl's wrapped `::exit`. Interactive terminal runs get a 3-second ctrl-c grace countdown before a dirty build proceeds; piped/agent runs skip it. `tclsh src/make.tcl check` reports the current src provenance status and what the build commands would do. - Provenance warnings (dirty-src gate + vendorupdate source-project check) print with a plain column-0 `PROVENANCE-WARNING:` token (grep for it in captured build output) and are recapped at the end of the run via make.tcl's wrapped `::exit`. Interactive terminal runs get a 3-second ctrl-c grace countdown before a dirty build proceeds; piped/agent runs skip it. `tclsh src/make.tcl check` reports the current src provenance status and what the build commands would do.

74
src/make.tcl
Unescape View File

@ -11,18 +11,23 @@
# 2. PUNK_FORCE_COLOR / FORCE_COLOR set -> ANSI ON even when piped # 2. PUNK_FORCE_COLOR / FORCE_COLOR set -> ANSI ON even when piped
# (a value of 0/false/no/off is treated as not-set, per ecosystem convention) # (a value of 0/false/no/off is treated as not-set, per ecosystem convention)
# 3. stdout tty probe -> colour ON only when stdout is a # 3. stdout tty probe -> colour ON only when stdout is a
# terminal channel (-winsize option present - Tcl 8.7+/9 terminal channels # terminal channel: -winsize option present (Tcl 8.7+/9 terminal channels,
# only, mirroring the stdin_is_interactive -inputmode precedent) # mirroring the stdin_is_interactive -inputmode precedent), or - Tcl 8.6 on
# windows - stdout -encoding reporting 'unicode' (utf-16), which only the
# 8.6 windows console channel does (pipes/files get the system encoding).
# Piped/redirected stdout (probe fails, no force) additionally pushes a strip # Piped/redirected stdout (probe fails, no force) additionally pushes a strip
# transform on stdout+stderr so the output is fully ESC-free - machine # transform so the output is fully ESC-free - machine consumers get plain text
# consumers get plain text no matter which emitter produced it (punk::ansi's # no matter which emitter produced it (punk::ansi's colour_disabled contract
# colour_disabled contract deliberately retains non-colour SGR effects, and # deliberately retains non-colour SGR effects, and module-side emitters like
# module-side emitters like punkcheck summaries follow that contract - the # punkcheck summaries follow that contract - the transform is the single choke
# transform is the single choke point that guarantees zero ESC bytes). # point that guarantees zero ESC bytes). The push is PER CHANNEL and skips any
# Tcl 8.6 has no boot-phase tty probe (no -winsize/-inputmode anywhere): the # utf-16-class channel (a byte-level strip corrupts utf-16 - ESC arrives as
# fail direction is plain/OFF (safe for piped/captured agent runs - unlike # 1B 00; such a channel is an 8.6 console, i.e. a real terminal, and keeps its
# stdin's assume-interactive; use the force env vars for 8.6 interactive colour). # ANSI) - e.g 'make.tcl ... > file' from an 8.6 console wraps stdout only.
# stderr follows the same single stdout-keyed decision. # Tcl 8.6 on unix has no boot-phase tty probe at all: the fail direction is
# plain/OFF (safe for piped/captured agent runs - unlike stdin's
# assume-interactive; use the force env vars for 8.6 unix interactive colour).
# stderr colour follows the same single stdout-keyed decision.
# The colour switch is ::punk::console::colour_disabled; when punk::ansi is # The colour switch is ::punk::console::colour_disabled; when punk::ansi is
# already loaded (punk-exe-hosted runs) its sgr cache is cleared per that # already loaded (punk-exe-hosted runs) its sgr cache is cleared per that
# module's contract ("whatever function disables or re-enables colour should # module's contract ("whatever function disables or re-enables colour should
@ -91,6 +96,20 @@ apply {{} {
} }
} }
set stdout_is_tty [expr {![catch {chan configure stdout -winsize}]}] set stdout_is_tty [expr {![catch {chan configure stdout -winsize}]}]
if {!$stdout_is_tty && $::tcl_platform(platform) eq "windows"
&& ![package vsatisfies [package provide Tcl] 8.7-]} {
#Tcl 8.6 windows: no -winsize/-inputmode probe exists, but 8.6's windows
#CONSOLE channels are the only ones that report -encoding unicode (utf-16;
#pipes/files get the system encoding) - a dependency-free console signature.
#This both restores interactive colour on 8.6 consoles and is load-bearing
#for the ansistrip transform below: the byte-level strip must never sit on
#a utf-16 channel (ESC arrives as 1B 00 - sequences pass unstripped and the
#hold-back logic defers/drops real content; field defect 2026-07-25 mangling
#punk86/punksys console help output).
if {[chan configure stdout -encoding] eq "unicode"} {
set stdout_is_tty 1
}
}
if {[info exists ::env(NO_COLOR)]} { if {[info exists ::env(NO_COLOR)]} {
namespace eval ::punk::console {variable colour_disabled 1} namespace eval ::punk::console {variable colour_disabled 1}
set ::punkboot::colour_mode nocolor set ::punkboot::colour_mode nocolor
@ -111,12 +130,19 @@ apply {{} {
#zero-ESC guarantee for piped/unknown output unless the caller forced ANSI on. #zero-ESC guarantee for piped/unknown output unless the caller forced ANSI on.
#(NO_COLOR keeps punk::ansi's documented colour-only-stripping semantics on a #(NO_COLOR keeps punk::ansi's documented colour-only-stripping semantics on a
#terminal; a NO_COLOR run with piped stdout still gets the full strip.) #terminal; a NO_COLOR run with piped stdout still gets the full strip.)
#Per-channel: a byte-level strip must never sit on a utf-16-class channel (the
#8.6 windows console encoding) - e.g 'make.tcl ... > file' from an 8.6 console
#leaves stderr ON the console; that channel is a real terminal, so it keeps its
#ANSI rather than getting corrupted. ::punkboot::ansistrip_pushed records the
#channels actually wrapped (the shell subcommand pops exactly those).
set ::punkboot::ansistrip_pushed [list]
if {!$force && !$stdout_is_tty} { if {!$force && !$stdout_is_tty} {
chan push stdout ::punkboot::ansistrip::transchan foreach _ch {stdout stderr} {
chan push stderr ::punkboot::ansistrip::transchan if {[chan configure $_ch -encoding] ni {unicode utf-16 utf-16le utf-16be}} {
set ::punkboot::ansistrip_pushed 1 chan push $_ch ::punkboot::ansistrip::transchan
} else { lappend ::punkboot::ansistrip_pushed $_ch
set ::punkboot::ansistrip_pushed 0 }
}
} }
#punk-exe-hosted runs arrive with punk::ansi pre-loaded and a warm sgr cache #punk-exe-hosted runs arrive with punk::ansi pre-loaded and a warm sgr cache
#generated under the kit's own colour state - honour the module's contract. #generated under the kit's own colour state - honour the module's contract.
@ -124,7 +150,7 @@ apply {{} {
catch {punk::ansi::sgr_cache -action clear} catch {punk::ansi::sgr_cache -action clear}
} }
if {[info exists ::env(PUNKBOOT_COLOUR_DEBUG)]} { if {[info exists ::env(PUNKBOOT_COLOUR_DEBUG)]} {
puts stderr "punkboot colour policy: mode=$::punkboot::colour_mode colour_disabled=$::punk::console::colour_disabled ansistrip=$::punkboot::ansistrip_pushed" puts stderr "punkboot colour policy: mode=$::punkboot::colour_mode colour_disabled=$::punk::console::colour_disabled ansistrip=[expr {[llength $::punkboot::ansistrip_pushed] ? [join $::punkboot::ansistrip_pushed +] : 0}]"
} }
}} }}
# Colour-policy gate for raw-SGR emission sites that can run before # Colour-policy gate for raw-SGR emission sites that can run before
@ -2569,8 +2595,8 @@ if {$::punkboot::command in {bakehouse packages modules libs bake vfslibs bin bo
if {$::punkboot::command eq "check"} { if {$::punkboot::command eq "check"} {
set sep [string repeat - 75] set sep [string repeat - 75]
puts stdout $sep puts stdout $sep
puts stdout "colour policy (G-113): mode=$::punkboot::colour_mode colour_disabled=$::punk::console::colour_disabled ansistrip=$::punkboot::ansistrip_pushed" puts stdout "colour policy (G-113): mode=$::punkboot::colour_mode colour_disabled=$::punk::console::colour_disabled ansistrip=[expr {[llength $::punkboot::ansistrip_pushed] ? [join $::punkboot::ansistrip_pushed +] : 0}]"
puts stdout " (precedence: NO_COLOR > PUNK_FORCE_COLOR/FORCE_COLOR > stdout tty probe; piped/8.6-unknown output is fully ESC-stripped unless forced)" puts stdout " (precedence: NO_COLOR > PUNK_FORCE_COLOR/FORCE_COLOR > stdout tty probe incl. the 8.6 windows console-encoding signature; piped/unknown output is fully ESC-stripped per channel unless forced - utf-16 console channels are never wrapped)"
puts stdout $sep puts stdout $sep
puts stdout "module/library checks - paths from bootsupport only" puts stdout "module/library checks - paths from bootsupport only"
puts stdout $sep puts stdout $sep
@ -3075,10 +3101,12 @@ if {$::punkboot::command eq "buildsuite"} {
if {$::punkboot::command eq "shell"} { if {$::punkboot::command eq "shell"} {
#G-113: the ansistrip transform serves make.tcl's own output only - the repl's #G-113: the ansistrip transform serves make.tcl's own output only - the repl's
#colour behaviour is the shell's own concern (colour on/off + NO_COLOR there). #colour behaviour is the shell's own concern (colour on/off + NO_COLOR there).
if {[info exists ::punkboot::ansistrip_pushed] && $::punkboot::ansistrip_pushed} { #Pop exactly the channels the policy block wrapped (per-channel list).
catch {chan pop stdout} if {[info exists ::punkboot::ansistrip_pushed]} {
catch {chan pop stderr} foreach _ch $::punkboot::ansistrip_pushed {
set ::punkboot::ansistrip_pushed 0 catch {chan pop $_ch}
}
set ::punkboot::ansistrip_pushed [list]
} }
package require struct::list package require struct::list
package require punk package require punk

4
src/tests/shell/testsuites/punkexe/maketclcolour.test
Unescape View File

@ -120,7 +120,7 @@ namespace eval ::testspace {
set result [list] set result [list]
lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode] lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode]
lappend result esc [esc_count [dict get $r output]] lappend result esc [esc_count [dict get $r output]]
lappend result policyline [regexp {colour policy \(G-113\): mode=piped-plain colour_disabled=1 ansistrip=1} [dict get $r output]] lappend result policyline [regexp {colour policy \(G-113\): mode=piped-plain colour_disabled=1 ansistrip=stdout\+stderr} [dict get $r output]]
set result set result
} -result {timedout 0 exitcode 0 esc 0 policyline 1} } -result {timedout 0 exitcode 0 esc 0 policyline 1}
@ -141,7 +141,7 @@ namespace eval ::testspace {
set result [list] set result [list]
lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode] lappend result timedout [dict get $r timedout] exitcode [dict get $r exitcode]
lappend result esc [esc_count [dict get $r output]] lappend result esc [esc_count [dict get $r output]]
lappend result policyline [regexp {colour policy \(G-113\): mode=nocolor colour_disabled=1 ansistrip=1} [dict get $r output]] lappend result policyline [regexp {colour policy \(G-113\): mode=nocolor colour_disabled=1 ansistrip=stdout\+stderr} [dict get $r output]]
set result set result
} -result {timedout 0 exitcode 0 esc 0 policyline 1} } -result {timedout 0 exitcode 0 esc 0 policyline 1}

Write Preview
Loading…
Cancel
Save