punkshell/goals/G-070-pure-tcl-tclparser.md

G-070 Pure-Tcl tclparser: parse-command API fallback with behavioural parity against the C library

Status: proposed Scope: src/modules/punk/lib-999999.0a1.0.tm (tclparser_tcl stub + dispatch; new module if size warrants - decided in the work); src/tests/modules/punk/lib/ (parity + fallback suites); external fork repo c:/repo/jn/tclparser_punk (read-only reference snapshots + the zig-built parser dll oracle - G-154, achieved - see goals/archive/G-154-tclparser-punk-zig-build.md) Goal: punk::lib's script-analysis machinery (tclword_to_scriptlist and the dependent analysis paths) runs on runtimes without the tclparser C binary, via a pure-Tcl implementation of the tclparser 'parse' command API covering at least the subcommands and token shapes punk::lib consumes - with behavioural parity verified against the C library rather than assumed, and the C parser still preferred where present (performance). Acceptance: the punk::lib tclparser_tcl error stub is replaced by a working pure-Tcl implementation of the 'parse' subcommands punk::lib currently uses ('parse command' at minimum; the full covered set enumerated in this file during the work), returning the same parse-tree shapes (ranges, token types, expansion handling) the C library returns for those calls; a parity testsuite compares pure-Tcl output against the C tclparser across a recorded probe corpus (representative punkshell module scripts plus edge cases: {*} expansion, comments, backslash-newline continuation, nested command substitution, braces/quotes in words) - parity tests capability-gated on the binary, pure-Tcl-only tests running everywhere; a documented punk::lib analysis entry point (e.g. tclword_to_scriptlist) demonstrably works under a plain tclsh with no parser binary on the package path; dispatch prefers the C parser when available with the fallback engaging automatically otherwise; the reference source consulted (the tclparser_punk fork repo - snapshot and checkin/commit identity per its src/reference provenance table) is recorded in this file.

Context

punk::lib uses the tclparser C library's 'parse command' for script analysis (word -> scriptlist conversion, argument/expansion classification - the G-019 dependency-scan direction). The binary ships only in the punk9win kit's lib_tcl9, so the analysis paths are unavailable on plain tclsh, other kits and other platforms - and lib-999999.0a1.0.tm carries an explicit placeholder: tclparser_tcl (currently an error stub telling the user to install the C library) plus a recorded deliberation at tclscript_to_toplevelinfo noting "we don't have a pure-tcl implementation of 'parse' available as a fallback". A tested pure-Tcl implementation removes the binary constraint from every current and future consumer and aligns with the G-004 no-committed-binaries direction. It also widens where the G-069 splitter cross-check lint can run.

The behavioural reference is the punkshell-maintained tclparser fork at c:/repo/jn/tclparser_punk (seeded 2026-08-02; G-154, achieved - see goals/archive/G-154-tclparser-punk-zig-build.md): byte-frozen snapshots of both upstream lineages (the aspect chiselapp fossil mirror at 1.4.1 and the tcltk-depot parser 1.9 Tcl 9 port - the candidate upstreams originally noted in the tclparser_tcl stub comments, surveyed and pinned) with a provenance table under src/reference/, the ported 114-case upstream testsuite (a ready parity-corpus seed), and zig-built parser dlls for both Tcl generations - the C oracle the parity suite compares against, no longer limited to the punk9win kit's shipped binary. punkshell also carries a moduledoc for the parse command at src/modules/punk/args/moduledoc/parser-999999.0a1.0.tm.

Approach

• Scope the API by consumption, not by the full tclparser surface: enumerate the subcommands and result fields punk::lib actually reads (parse command ranges, token lists, restRange handling, simple-expansion trees), implement those first, and record the covered/uncovered set here. Other subcommands (expr, varname, list) follow only if a consumer needs them.
• Parity-first workflow (G-055 spirit): build the probe corpus and the compare-against-C harness before/alongside the implementation, so every implemented construct lands with a parity pin. Corpus draws on real punkshell module sources (rich in expansions, multi-line commands and comments) plus targeted edge cases.
• Dispatch: a thin front (in punk::lib or the new module) that package-requires the C parser and falls back to the pure-Tcl engine; consumers call one entry point. Performance note recorded rather than chased - the C library stays preferred where installed.
• Placement: the stub lives in punk::lib today; a parser is a sizeable, separately-testable unit, so a dedicated module (name TBD, e.g. under punk::) with punk::lib delegating is the likely shape - decided in the work.

Notes

• Related: G-019 (dependency-scan module trimming - the main analysis consumer), G-004 (binary-free direction), G-069 (lint gains a binary-free substrate), G-055 (parity-verification workflow precedent).
• Related: G-154 (achieved 2026-08-02 - see goals/archive/G-154-tclparser-punk-zig-build.md) - the tclparser_punk fork/build lane this goal's reference snapshots and dll oracle come from; its testsuite gate + kit-shell transcripts and upstream-landscape survey live in that archived detail file and the fork's src/reference/AGENTS.md.
• The tclparser_tcl stub's partial char-loop sketch (in_dq/in_cb/escape state) is a starting point only; the reference source's Tcl_ParseCommand semantics are the contract.