punkshell/goals/archive/G-145-piped-usage-ansi-remnants.md

G-145 Remnant-free piped usage tables: root-cause split ANSI fragments

Status: achieved 2026-08-01 Scope: src/make.tcl (::punkboot::ansistrip transform); src/modules/textblock-999999.0a1.0.tm + src/modules/punk/ansi-999999.0a1.0.tm (ANSI-aware cell/width handling as implicated); src/tests/shell/testsuites/punkexe/maketclcolour.test (remnant pin) Goal: Piped make.tcl usage/help output is free of ANSI remnant text: the orphan CSI fragments observed in piped help tables ('[0;1m', '[0m', lone 'm' beside cell borders) are root-caused (renderer-side sequence splitting vs a strip-transform edge case) and eliminated at the producing layer, extending the G-113 zero-ESC guarantee to zero orphan sequence-fragment text while leaving interactive colour output unchanged. Acceptance: piped 'tclsh src/make.tcl help ' for every declared subject plus bare 'make.tcl -help' yields output with no ESC bytes and no CSI-remnant substrings (regex {[[0-9;:]*m}; the help corpus carries no such literal text), pinned in maketclcolour.test; the root cause and chosen fix layer are recorded in this file; existing colour-policy pins (tty/forced/nocolor modes) pass unchanged.

Context

Observed 2026-08-01 while diffing before/after help captures for the argdoc restyle: piped help output intermittently carries ESC-less ANSI tails inside rendered tables (before the restyle: _toplevel x2, help, modules, vendorupdate; after: _toplevel x4, bakelist, bootsupport, help, info, libs - positions shift with table geometry, so the restyle did not introduce the class). Examples: '[0;1m' at a line end before a cell border, '[0m' at a row start, a lone 'm' beside a border.

Approach

• Evidence so far: the G-113 write-side strip (::punkboot::ansistrip) already carries a trailing incomplete sequence across write chunks (per-channel carry; flush preserves it, finalize drops it), so a clean ESC-boundary split at the channel would be handled. First suspect is therefore upstream: styled content split by ANSI-aware width/cell handling before write, leaving ESC-less fragment text a byte-level strip correctly ignores. (First suspect REFUTED 2026-08-01 - root cause is transform-side, see Notes.)
• Piped runs do legitimately receive ANSI from module emitters (the strip is load-bearing by design - G-113); punk::args resolve caching also carries a colour-state staleness todo (cached specs can embed colour from resolve time), a possible fragment source to check.
• Diagnosis route: capture the pre-transform stream for an affected subject (transform popped, or PUNK_FORCE_COLOR run) and determine whether fragments exist pre-strip; fix in textblock/punk::ansi width handling if so, else harden the transform.

Notes

• ROOT CAUSE (determined 2026-08-01): transform-lifecycle defect in ::punkboot::ansistrip (src/make.tcl), NOT renderer-side splitting. The Tcl core delivers a clear op to the transform stack before EVERY write op (output-buffer flush-down; observed on tclsh 8.7 and 9.0.3, on the real stdout stack and on a bare file channel with no make.tcl code involved). ansistrip::transchan's clear handler runs dict unset carry $chanid, discarding a split sequence's held tail at every chunk boundary: write #k ends mid-sequence (e.g ESC[ held in carry), clear drops it, write #k+1 begins with 0;1m which passes through unstripped, and the held ESC[ bytes are never emitted. The dropped ESC bytes are why the G-113 zero-ESC pin (maketclcolour.test) stays green while fragments leak. Evidence chain: (a) pre-transform stream captured via NO_COLOR=1 + PUNK_FORCE_COLOR=1 (NO_COLOR wins colour, force skips the transform push) is fully well-formed - every ESC[...m complete; textblock and punk::ansi cell/width handling are exonerated, and G-056 is not the mechanism. (b) the verbatim strip proc fed the captured stream under every possible 2-chunk split and 512..8192 block sizes yields zero fragments - the carry logic itself is sound for arbitrary chunking. (c) an identity logging transform pushed on stdout before sourcing make.tcl (ansistrip stacks on top) recorded clear before every write; the output chunk boundary landed exactly at the observed 0;1m remnant (prior chunk ended with the Default-cell spaces, the following ESC[ held then dropped). (d) minimal isolated repro (verbatim transform on a file channel, repetitive styled payload, no make.tcl): op log shows clear per flush-down, output carries a fragment (;1m). Determinism: fragments are deterministic per table geometry (buffer-boundary positions are fixed; a remnant appears only where a boundary lands inside a sequence), matching the observed intermittency and position shifts across subjects/restyles.
• Upstream documentation status (2026-08-01, TEMP_REFERENCE/tcl9 survey): the per-write clear is INTENTIONAL Tcl core behavior carrying an upstream documentation gap. generic/tclIORTrans.c ReflectOutput (~line 1280) invokes the transform's clear before every write when METH_CLEAR is declared, with the comment "Discard partial data in the input buffers, i.e. on the read side. Like we do when explicitly seeking as well." The core test suite pins it: tests/ioTrans.test iortrans-7.1 "chan write, write clears read buffers" expects {clear rt*} {write rt* snarf} from a plain puts+flush (7.2 seek, 7.3 result-ignored, 7.4 lifecycle bug 2921116). But doc/transchan.n (lines 47-54) documents clear as "called when a chan seek is performed on the channel being transformed", and TIP 230's text scopes it to "internal input buffers" that "happens only when the user seeks the channel the transformation is attached to" - both silent on the write-path call. Verdict: upstream DOCUMENTATION bug (behavior is deliberate and test-pinned; man page and TIP text are stale), not an implementation bug; no existing Tcl ticket found covering the discrepancy (one tangential refchan write-buffering ticket, de232b49f2, unrelated to transforms). Two consequences for the fix: (a) clear only fires at all when the transform DECLARES it (the core gates on METH_CLEAR), so dropping it from the supported-methods list eliminates the per-write calls entirely; (b) clear's documented scope is INPUT/read-side buffers - a write-only transform like ::punkboot::ansistrip has no read-side state, so its contract-conformant clear is a no-op; dropping the write-side carry was wrong under both the documented and the actual contract.
• Fix layer: the transform's clear handling in src/make.tcl (::punkboot::ansistrip::transchan) - clear must not discard the per-channel carry (drop the op from the supported-methods list or make it a no-op); finalize dropping the carry remains correct (incomplete sequence at end of stream). No change needed in textblock/punk::ansi.
• Encoding note (raised during investigation): the byte-level transform is encoding-safe for its approach - 0x1B never occurs inside UTF-8 multibyte chars (a box char split e2 94 | 81 across writes was observed and is harmless), and utf-16-class channels are guarded at push time (ESC would arrive as 1B 00). Compare shellfilter::chan::ansistrip (shellfilter-999999.0a1.0.tm): it decodes via encoding convertfrom and carries o_encbuf for split multibyte chars ("bytes can break at arbitrary points making encoding conversions invalid"), but explicitly assumes line buffering and does not handle split ANSI codes at all. The punkboot defect is lifecycle (clear), not encoding.
• Repro (2026-08-01, current geometry, tclsh 9.0.3, all deterministic): bare tclsh src/make.tcl help piped -> 1 fragment (0;1m); help libs / help info -> ;1m x1 each; help bootsupport -> 1m x1. Detector regex: (?<![\x1b\[])[0-9;]+m (bracket is eaten with the ESC, so the G-113-era {\[[0-9;:]*m} pattern alone would miss these). In-shell: set out [runout -n tclsh src/make.tcl help] then regexp -all -inline -- {(?:^|[^\x1b\[])[0-9;]+m} $out (exec/runout pipe stdout, engaging the transform).
• G-113 achieved 2026-07-25 (goals/archive/G-113-maketcl-tty-aware-colour.md) - the colour policy and zero-ESC pin surface (maketclcolour.test) this extends.
• Related: G-056 (proposed) - ANSI-aware wrap/width machinery ('styling in effect carries across the wrap'); if the root cause is renderer-side splitting, the fix layer coincides.
• Overlap survey 2026-08-01: goals_xref paths punk/ansi module -> G-078/G-079/G-080 (HTML rendering family, unrelated); textblock -> G-048 (table option parse), G-056 (named above), G-088 (footer rendering) - only G-056 shares the mechanism; punkexe tests -> G-077/G-131/G-141 (unrelated).

Progress

• 2026-08-01 (activated and achieved same day; user instruction): fix landed in src/make.tcl - ::punkboot::ansistrip::transchan no longer declares clear in its supported-methods list, so the core's per-write clear ops never fire and the per-channel split-sequence carry survives every write-chunk boundary; finalize still drops the carry (incomplete sequence at end of stream). No change in textblock/punk::ansi (exonerated, note (a)).
• Pin: maketclcolour.test gains maketcl_colour_help_tables_zero_remnants - piped tclsh src/make.tcl help for all 20 declared SUMMARIES subjects plus bare help and -help, asserting exitcode 0, zero ESC bytes, and zero orphan fragments. Two-step detector: strip complete sequences (the transform's own re_complete), then scan the remainder for (?:^|[^\x1b\[])[0-9;]+m - scanning raw output false-positives on multi-parameter SGR (';' inside a complete ESC[0;1m); when esc==0 holds the strip is a no-op. The pin failed pre-fix naming exactly the 5 then-fragmenting invocations (help, -help, help libs, help info, help bootsupport), proving detection; the NO_COLOR+PUNK_FORCE_COLOR pre-transform corpus (all 22 invocations, complete sequences stripped) carries no legit text matching the detector.
• Verified 2026-08-01 (tclsh 8.7a6, win32-x86_64): all 22 invocations piped clean (0 ESC bytes, 0 remnants); PUNK_FORCE_COLOR=1 piped help carries 1177 complete CSI sequences and zero orphan fragments (interactive/forced-colour path unchanged); the 3 existing G-113 colour-policy pins (tty-probe piped-plain, forced, nocolor modes) pass unchanged; full shell/testsuites/punkexe family green (15 files, 102 tests, 0 failures) and the shell/*** subtree green (127 tests, 0 failures - the single dtplite.test missing-cleanupTests warning is pre-existing, unrelated).