punkshell/goals/G-070-pure-tcl-tclparser.md

G-070 Pure-Tcl tclparser: parse-command API fallback with behavioural parity against the C library

Status: proposed Scope: src/modules/punk/lib-999999.0a1.0.tm (tclparser_tcl stub + dispatch; new module if size warrants - decided in the work), src/tests/modules/punk/lib/ (parity + fallback suites), TEMP_REFERENCE/ (tclparser reference source, user-provided, read-only) Goal: punk::lib's script-analysis machinery (tclword_to_scriptlist and the dependent analysis paths) runs on runtimes without the tclparser C binary, via a pure-Tcl implementation of the tclparser 'parse' command API covering at least the subcommands and token shapes punk::lib consumes - with behavioural parity verified against the C library rather than assumed, and the C parser still preferred where present (performance). Acceptance: the punk::lib tclparser_tcl error stub is replaced by a working pure-Tcl implementation of the 'parse' subcommands punk::lib currently uses ('parse command' at minimum; the full covered set enumerated in this file during the work), returning the same parse-tree shapes (ranges, token types, expansion handling) the C library returns for those calls; a parity testsuite compares pure-Tcl output against the C tclparser across a recorded probe corpus (representative punkshell module scripts plus edge cases: {*} expansion, comments, backslash-newline continuation, nested command substitution, braces/quotes in words) - parity tests capability-gated on the binary, pure-Tcl-only tests running everywhere; a documented punk::lib analysis entry point (e.g. tclword_to_scriptlist) demonstrably works under a plain tclsh with no parser binary on the package path; dispatch prefers the C parser when available with the fallback engaging automatically otherwise; the reference source's location under TEMP_REFERENCE and its provenance (origin URL, version/checkin) are recorded in this file.

Context

punk::lib uses the tclparser C library's 'parse command' for script analysis (word -> scriptlist conversion, argument/expansion classification - the G-019 dependency-scan direction). The binary ships only in the punk9win kit's lib_tcl9, so the analysis paths are unavailable on plain tclsh, other kits and other platforms - and lib-999999.0a1.0.tm carries an explicit placeholder: tclparser_tcl (currently an error stub telling the user to install the C library) plus a recorded deliberation at tclscript_to_toplevelinfo noting "we don't have a pure-tcl implementation of 'parse' available as a fallback". A tested pure-Tcl implementation removes the binary constraint from every current and future consumer and aligns with the G-004 no-committed-binaries direction. It also widens where the G-069 splitter cross-check lint can run.

The user will place the latest obtainable tclparser source (C implementation and any docs/tests) under TEMP_REFERENCE as the read-only behavioural reference (candidate upstreams: the aspect fossil fork at chiselapp and the ActiveState teapot tree, both already noted in the tclparser_tcl stub comments; punkshell also carries a moduledoc for the parse command at src/modules/punk/args/moduledoc/parser-999999.0a1.0.tm).

Approach

• Scope the API by consumption, not by the full tclparser surface: enumerate the subcommands and result fields punk::lib actually reads (parse command ranges, token lists, restRange handling, simple-expansion trees), implement those first, and record the covered/uncovered set here. Other subcommands (expr, varname, list) follow only if a consumer needs them.
• Parity-first workflow (G-055 spirit): build the probe corpus and the compare-against-C harness before/alongside the implementation, so every implemented construct lands with a parity pin. Corpus draws on real punkshell module sources (rich in expansions, multi-line commands and comments) plus targeted edge cases.
• Dispatch: a thin front (in punk::lib or the new module) that package-requires the C parser and falls back to the pure-Tcl engine; consumers call one entry point. Performance note recorded rather than chased - the C library stays preferred where installed.
• Placement: the stub lives in punk::lib today; a parser is a sizeable, separately-testable unit, so a dedicated module (name TBD, e.g. under punk::) with punk::lib delegating is the likely shape - decided in the work.

Notes

• Related: G-019 (dependency-scan module trimming - the main analysis consumer), G-004 (binary-free direction), G-069 (lint gains a binary-free substrate), G-055 (parity-verification workflow precedent).
• The tclparser_tcl stub's partial char-loop sketch (in_dq/in_cb/escape state) is a starting point only; the reference source's Tcl_ParseCommand semantics are the contract.