punkshell/goals/G-070-pure-tcl-tclparser.md

G-070 Pure-Tcl tclparser: parse-command API fallback with behavioural parity against the C library

Status: active Scope: src/modules/punk/lib-999999.0a1.0.tm (tclparser_tcl stub + dispatch; new module if size warrants - decided in the work); src/tests/modules/punk/lib/ (parity + fallback suites); external fork repo c:/repo/jn/tclparser_punk (read-only reference snapshots + the zig-built parser dll oracle - G-154, achieved - see goals/archive/G-154-tclparser-punk-zig-build.md) Goal: punk::lib's script-analysis machinery (tclword_to_scriptlist and the dependent analysis paths) runs on runtimes without the tclparser C binary, via a pure-Tcl implementation of the tclparser 'parse' command API covering at least the subcommands and token shapes punk::lib consumes - with behavioural parity verified against the C library rather than assumed, and the C parser still preferred where present (performance). Acceptance: the punk::lib tclparser_tcl error stub is replaced by a working pure-Tcl implementation of the 'parse' subcommands punk::lib currently uses ('parse command' at minimum; the full covered set enumerated in this file during the work), returning the same parse-tree shapes (ranges, token types, expansion handling) the C library returns for those calls; a parity testsuite compares pure-Tcl output against the C tclparser across a recorded probe corpus (representative punkshell module scripts plus edge cases: {*} expansion, comments, backslash-newline continuation, nested command substitution, braces/quotes in words) - parity tests capability-gated on the binary, pure-Tcl-only tests running everywhere; a documented punk::lib analysis entry point (e.g. tclword_to_scriptlist) demonstrably works under a plain tclsh with no parser binary on the package path; dispatch prefers the C parser when available with the fallback engaging automatically otherwise; the reference source consulted (the tclparser_punk fork repo - snapshot and checkin/commit identity per its src/reference provenance table) is recorded in this file.

Context

punk::lib uses the tclparser C library's 'parse command' for script analysis (word -> scriptlist conversion, argument/expansion classification - the G-019 dependency-scan direction). The binary ships only in the punk9win kit's lib_tcl9, so the analysis paths are unavailable on plain tclsh, other kits and other platforms - and lib-999999.0a1.0.tm carries an explicit placeholder: tclparser_tcl (currently an error stub telling the user to install the C library) plus a recorded deliberation at tclscript_to_toplevelinfo noting "we don't have a pure-tcl implementation of 'parse' available as a fallback". A tested pure-Tcl implementation removes the binary constraint from every current and future consumer and aligns with the G-004 no-committed-binaries direction. It also widens where the G-069 splitter cross-check lint can run.

The behavioural reference is the punkshell-maintained tclparser fork at c:/repo/jn/tclparser_punk (seeded 2026-08-02; G-154, achieved - see goals/archive/G-154-tclparser-punk-zig-build.md): byte-frozen snapshots of both upstream lineages (the aspect chiselapp fossil mirror at 1.4.1 and the tcltk-depot parser 1.9 Tcl 9 port - the candidate upstreams originally noted in the tclparser_tcl stub comments, surveyed and pinned) with a provenance table under src/reference/, the ported 114-case upstream testsuite (a ready parity-corpus seed), and zig-built parser dlls for both Tcl generations - the C oracle the parity suite compares against, no longer limited to the punk9win kit's shipped binary. punkshell also carries a moduledoc for the parse command at src/modules/punk/args/moduledoc/parser-999999.0a1.0.tm.

Approach

• Scope the API by consumption, not by the full tclparser surface: enumerate the subcommands and result fields punk::lib actually reads (parse command ranges, token lists, restRange handling, simple-expansion trees), implement those first, and record the covered/uncovered set here. Other subcommands (expr, varname, list) follow only if a consumer needs them.
• Parity-first workflow (G-055 spirit): build the probe corpus and the compare-against-C harness before/alongside the implementation, so every implemented construct lands with a parity pin. Corpus draws on real punkshell module sources (rich in expansions, multi-line commands and comments) plus targeted edge cases.
• Dispatch: a thin front (in punk::lib or the new module) that package-requires the C parser and falls back to the pure-Tcl engine; consumers call one entry point. Performance note recorded rather than chased - the C library stays preferred where installed.
• Placement: the stub lives in punk::lib today; a parser is a sizeable, separately-testable unit, so a dedicated module (name TBD, e.g. under punk::) with punk::lib delegating is the likely shape - decided in the work.

Progress

Increment 1 (2026-08-02): consumption audit - the covered set

Sweep basis: src/modules (canonical sources; bootsupport/_vfscommon copies are promotion artifacts of the same code, and punk9magicsplat.vfs/lib/tcldebugger is third-party consuming its own shipped C dll - both out of scope).

Consumers found:

1. punk::lib tclscript_info (lib ~2521): bare package require parser - NO fallback dispatch today (errors on binary-free runtimes before any parsing). Calls: parse command <str> {} per whole-cmdline (~2643 - currently write-only, only a commented-out debug read) and per-argument with a "bogus1 " prefix (~2665) and bare (~3027); parse getstring for all range extraction; parse countnewline <str> {} for line tracking.
2. punk::lib tclscript_info::tclword_to_scriptlist (~3674): the existing dispatch site (C preferred, tclparser_tcl stub fallback); parse command $boguscmdline {0 end}; parse getstring on command/variable subnode ranges (comment notes byte-safety as the reason over string range).
3. textblock height (~5112): parse countnewline <str> {} purely as a C fast path; a pure-Tcl height fallback (_height_tcl, string-map counting) already exists and is FASTER than any pure-Tcl countnewline would be - textblock needs nothing from this goal (see naming hazard below).
4. punk::args::moduledoc::parser documents ::parse command and ::parse countnewline only - doc site matching the consumed set.

Covered set (to implement): parse command (range args {} and {0 end}), parse getstring, parse countnewline. Uncovered (no consumer; follow only on demand): expr, varname, list, charindex, charlength.

Result-shape contract the consumers actually rely on (parity surface):

• parse command returns {commentRange commandRange restRange tree}; all four lassign'd; restRange ELEMENT 1 (length) == 0 asserted as full-consumption check (lib 2668, 3030). Correct restRange for multi-command strings (newline/semicolon) is part of parse-command semantics and comes free from a correct scanner terminator.
• Ranges are BYTE offsets {start len} into the parsed string; extraction must go through getstring (multibyte).
• Tree nodes are {type {start len} subnodes} triplets. Toplevel word types consumed: simple, word, expand. Simple-{} auto-expansion semantics relied on: tree llength 0 (empty literal expansion), >1 (one node per expanded word), 1 with type expand (non-literal expansion). Redundant {}simpleword parses as simpleword.
• simple node: subnodes exactly one text triplet (asserted, lib 3071-3073, 3110-3111).
• word node subnode types consumed: text, command, variable, backslash (backslash via membership tests + getstring/subst path, lib 3134-3149; a backslash token covers the backslash plus the following character).
• variable subnodes recursed for nested command/variable (array subscripts, lib 3772-3803).
• Quote-state inference (lib 3079, 3115, 3151): outer range length minus inner text/parts length == 2 means quoted - so range arithmetic must match C exactly (word/simple range includes quotes/braces, text subnode excludes).
• bogus1-prefix pattern: consumers verify tree[0] is simple with range {0 6}.

Design notes recorded for the implementation increments:

• Package-naming hazard: the pure-Tcl engine must NOT satisfy package require parser - textblock (5152) and tclword_to_scriptlist (3700) use exactly that probe to mean "the fast C library is present". The engine lives under its own name; dispatch shims stay namespace-local (a first-call-resolving local parse alias in the consuming namespaces - the textblock oneshot-rename pattern - lets tclscript_info's bare parse ... call sites fall back with zero body edits while a loaded C ::parse wins).
• Byte-range strategy candidate: scan the utf-8 byte view of the string (all Tcl syntax chars are ASCII; continuation bytes cannot alias them), so emitted offsets are natively byte offsets; getstring = string range on the byte view + convertfrom. Edge cases to pin in the corpus: backslash followed by a multibyte char (consume the full sequence), NUL bytes (C parses Tcl's modified utf-8 where NUL is 0xC0 0x80 - offsets may differ from strict utf-8), high-plane chars.

Increment 2 (2026-08-02): pure-Tcl engine + parity harness, both generations clean

New module src/modules/punk/tclparser-999999.0a1.0.tm (punk::tclparser 0.1.0): punk::tclparser::parse implementing the covered set (command, getstring, countnewline) with the C library's calling convention and result shapes. Zero package dependencies by design (loads on plain tclsh; PUNKARGS is inert documentation via the register mechanism). Engine scans a byte view of the string (modified utf-8: NUL as 0xC0 0x80) so all emitted ranges are byte offsets by construction; getstring/countnewline resolve ranges over the same view. Uncovered subcommands error advising the C library. The module does NOT provide package 'parser' (capability-probe hazard per increment 1).

Parity evidence (dev harnesses in the session scratchpad - parity_check.tcl edge corpus 117 cases x2 range forms + result-tree-walk getstring/countnewline comparisons; parity_sweep.tcl recursive organic sweep: module sources parsed command-by-command, recursing into command substitutions and braced bodies):

• Tcl 8.7a6 + tcl86-gen oracle dll: corpus 234 comparisons clean, ~990 getstring/countnewline clean; organic sweep punk::lib + punk::args + punk::ansi + textblock + punk::tclparser = 50005 command-parses, 0 fails, 332 error-parses ALL with byte-identical error messages.
• Tcl 9.0.3 + tcl9-gen oracle dll: corpus 234 comparisons clean; organic sweep same five files = 49928 command-parses, 0 fails, 330 error-parses message-identical.

Generation differences discovered via the oracle and modelled version- conditionally in the engine (verify on real 8.6 at the kit-verification increment - 8.7 was the Tcl 8 lane here):

• \xHH consumes at most 2 hex digits on 8.7+/9, unlimited on 8.6 (TIP 388).
• Tcl 9 rejects a bare '(' at array-index token level ("invalid character in array index"); 8.x treats it as index text ending at the first ')'.
• ${name}: Tcl 9 balances nested braces with backslash-escaped braces not counted; 8.x cuts at the FIRST '}' with no backslash handling.

Oracle-pinned shape facts baked into the engine (beyond increment 1's list): leading newlines are whitespace before a command starts; a bare '$' or a trailing backslash becomes its own text token (word, not simple); an empty array index still carries one empty text token; literal {} expansion is abandoned (expand node) when any bare/quoted list element contains a backslash (braced elements only for backslash-newline); {} followed by backslash-newline is a plain braced word; empty braced/quoted words carry one empty text token.

Remaining for the goal: dispatch wiring (punk::lib namespace-local parse shim + tclparser_tcl replacement), capability-gated tcltest parity suite + pure-Tcl fallback suite (port of the harness corpus), plain-tclsh tclword_to_scriptlist demonstration, real-8.6 verification, reference identity recording. [all resolved in increment 3 below]

Increment 3 (2026-08-02): dispatch wiring, test suites, three-runtime verification

punk::lib 0.6.0:

• tclparser_tcl error stub REPLACED by a working delegation to punk::tclparser::parse (same {subcmd string range} signature).
• New punk::lib::tclparser_prefer (PUNKARGS-documented): automatic memoized dispatch - 'package require parser' succeeding means the c library serves the global ::parse the analysis procs call (preferred, performance); otherwise namespace-local 'parse' aliases (::punk::lib::parse and ::punk::lib::tclscript_info::parse) route every existing bare 'parse ...' call site to the pure-Tcl engine with zero body edits. Optional argument forces c|tcl (testing/benchmarks); forcing c removes the aliases.
• tclscript_info's hard 'package require parser' and tclword_to_scriptlist's dispatch now go through tclparser_prefer; the 'may be less accurate' fallback warning removed (no longer true). tclword_to_scriptlist gained a PUNKARGS argdoc (the acceptance's documented entry point) and its three active debug puts were commented out - two used the repl-only a+ alias and crashed any plain tclsh exercising the variable-subnode path (found by the acceptance demo, which exists to catch exactly this).

Test suites (src/tests/modules/punk/lib/testsuites/lib/):

• tclparser_engine.test (20 tests, runs everywhere, no c library needed): engine result-shape pins - all values oracle-verified during increments 2-3 - covering the 4-tuple, byte ranges (multibyte), simple/word/expand nodes, literal-expansion, restRange walking, nested array-variable subnodes, getstring/countnewline, error messages, the uncovered-subcommand error, the no-package-'parser' guarantee, plus the punk::lib dispatch: forced-tcl tclword_to_scriptlist results and tclparser_tcl delegation.
• tclparser_parity.test (capability-gated on 'package require parser', with env PUNK_TCLPARSER_ORACLE_DIR appended to auto_path first so the fork's built dlls can supply the oracle on binary-less runners): the RECORDED PROBE CORPUS (77 labelled cases: quoting, {*} expansion incl literal/ complex/separator forms, comments incl backslash-continuation, backslash escapes incl word-separator backslash-newline, nested command substitution, variables incl generation-sensitive forms, multibyte, terminators, error cases, plus representative punkshell-style source snippets) compared c-vs-pure over both range forms; a getstring/countnewline walk over every range in every corpus parse tree; and entry-point parity (tclword_to_scriptlist forced-c vs forced-tcl).

Verification (all green):

• runtests modules/punk/lib/*** : Tcl 9.0.3 66/67 pass (1 unrelated skip), Tcl 8.7a6 66/67 with the single compat.test lpop-matrix failure PROVEN pre-existing (fails identically with the pristine punk::lib under 8.7a6 - this box's known-baseline class, not a G-070 regression). New suites 23/23 under both, parity constraint satisfied natively (magicsplat tclsh90 ships tclparser 1.9 - an INDEPENDENT c build also passing parity) and via PUNK_TCLPARSER_ORACLE_DIR under 8.7.
• Real Tcl 8.6.13 (punksys kit, scratch copy, 'script' subcommand): dev corpus harness PARITY CLEAN (234 comparisons + 989 getstring/countnewline) against the tcl86-gen fork oracle dll. Model correction from this run: \xHH consumes at most 2 hex digits on 8.6 TOO (TIP 388 is in since 8.6; unlimited \x is Tcl 8.5) - the increment-2 version-conditional was removed, the cap is now unconditional.
• Plain-tclsh acceptance demo (auto_path cleared so no parser binary is on the package path; punk modules via tm path): Tcl 9.0.3 and 8.7a6 - 'parser package available: 0', tclparser_prefer -> tcl, tclword_to_scriptlist returns correct scriptlists including nested array-index command substitutions; identical output with the c library enabled (prefer -> c).
• make.tcl modules clean.

Reference source consulted (per Acceptance): the tclparser_punk fork repo (gitea jn/tclparser, working copy c:/repo/jn/tclparser_punk) at commit pin 16d2f460b89dfd819aa08ceb97dfebf37b9fa55a - its zig-built parser dlls (zig-out/lib//win32-x86_64/parser1.9/) are the c oracle all parity evidence compares against, and its src/reference provenance table pins the upstream snapshots consulted: aspect chiselapp fossil checkin 53f5b44b14bdbdf4b7a3a1b02ba40425e145d76b (tclparser 1.4.1 lineage, 2015-06-18) and tcltk-depot git commit c6c841d6c4c03fe19b45279dd29266db7861fdb3 (parser 1.9 Tcl 9 port baseline, 2025-07-29).

Covered set (final): parse command (ranges {} and {0 end} plus general {first len|end}), parse getstring, parse countnewline. Uncovered (error advising the c library): expr, varname, list, charindex, charlength.

Notes

• Related: G-019 (dependency-scan module trimming - the main analysis consumer), G-004 (binary-free direction), G-069 (lint gains a binary-free substrate), G-055 (parity-verification workflow precedent).
• Related: G-154 (achieved 2026-08-02 - see goals/archive/G-154-tclparser-punk-zig-build.md) - the tclparser_punk fork/build lane this goal's reference snapshots and dll oracle come from; its testsuite gate + kit-shell transcripts and upstream-landscape survey live in that archived detail file and the fork's src/reference/AGENTS.md.
• The tclparser_tcl stub's partial char-loop sketch (in_dq/in_cb/escape state) is a starting point only; the reference source's Tcl_ParseCommand semantics are the contract.