From b3b99924571aa0e122a667aaf8e5e9de043fa376 Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Sun, 12 Jul 2026 21:36:07 +1000 Subject: [PATCH] goals: add G-069 (splitter tclparser cross-check lint) and G-070 (pure-Tcl tclparser) Both user-approved drafts arising from the G-045 tclparser assessment: - G-069: dev-time lint cross-checking punk::args record-splitter boundaries against tclparser parse command where the binary is available - dialect-aware comparison (-& pre-join, ANSI strip), capability-gated, zero new runtime dependencies for punk::args. - G-070: pure-Tcl implementation of the tclparser parse-command API scoped to what punk::lib consumes, parity-verified against the C library over a probe corpus; replaces the tclparser_tcl error stub with prefer-C/fallback-pure dispatch. Reference source to be user-provided under TEMP_REFERENCE with recorded provenance. Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- GOALS.md | 8 ++++ goals/G-069-splitter-tclparser-lint.md | 50 +++++++++++++++++++++++ goals/G-070-pure-tcl-tclparser.md | 56 ++++++++++++++++++++++++++ 3 files changed, 114 insertions(+) create mode 100644 goals/G-069-splitter-tclparser-lint.md create mode 100644 goals/G-070-pure-tcl-tclparser.md diff --git a/GOALS.md b/GOALS.md index 6090dce6..66e29f87 100644 --- a/GOALS.md +++ b/GOALS.md @@ -277,3 +277,11 @@ Detail: goals/G-067-module-artifact-channel.md ### G-068 [proposed] Agent-assisted moduledoc generation workflow for vendored third-party libraries Scope: goals/G-068-vendored-moduledoc-workflow.md (workflow doc), src/modules/punk/args/moduledoc/ (generated companion modules), src/tests/modules/punk/args/ (probe verification where feasible) Detail: goals/G-068-vendored-moduledoc-workflow.md + +### G-069 [proposed] Dev-time lint: cross-check punk::args record splitting against tclparser where the binary is available +Scope: lint surface (location settled in the work: punk::args dev helper, dev commandset, or scriptlib/developer script), src/modules/punk/args-999999.0a1.0.tm (split_definition_records as consumed, no new dependency), src/tests/modules/punk/args/ (capability-gated suite) +Detail: goals/G-069-splitter-tclparser-lint.md + +### G-070 [proposed] Pure-Tcl tclparser: parse-command API fallback with behavioural parity against the C library +Scope: src/modules/punk/lib-999999.0a1.0.tm (tclparser_tcl stub + dispatch; new module if size warrants - decided in the work), src/tests/modules/punk/lib/ (parity + fallback suites), TEMP_REFERENCE/ (tclparser reference source, user-provided, read-only) +Detail: goals/G-070-pure-tcl-tclparser.md diff --git a/goals/G-069-splitter-tclparser-lint.md b/goals/G-069-splitter-tclparser-lint.md new file mode 100644 index 00000000..bdd5b5db --- /dev/null +++ b/goals/G-069-splitter-tclparser-lint.md @@ -0,0 +1,50 @@ +# G-069 Dev-time lint: cross-check punk::args record splitting against tclparser where the binary is available + +Status: proposed +Scope: lint surface (location settled in the work: punk::args dev helper, dev commandset, or scriptlib/developer script), src/modules/punk/args-999999.0a1.0.tm (split_definition_records as consumed, no new dependency), src/tests/modules/punk/args/ (capability-gated suite) +Goal: a dev-time lint cross-checks the record boundaries produced by punk::args' definition record splitter (private::split_definition_records) against the tclparser C library's 'parse command' tokenization of the same (ANSI-stripped, dialect-adjusted) text, so drift between the definition dialect and real Tcl parsing rules is caught as the dialect grows (-&, @normalize, future continuation/normalization directives) - without punk::args itself gaining any dependency on the tclparser binary. +Acceptance: a documented lint command (location decided in the work) accepts definition text and/or registered definition ids and reports record-boundary divergence between split_definition_records and tclparser 'parse command'; the comparison accounts for documented dialect features rather than flagging them (at minimum: ANSI escapes stripped before parse, -& record continuation pre-joined or classified as expected divergence); the lint is capability-gated - a clean, explicit skip/notice when the tclparser package is unavailable, never an error - and `package require punk::args` continues to succeed with no tclparser present; a run across all definitions registered in a punk9win kit shell (where the tclparser binary ships) yields zero unexplained divergences, with any real divergences found either fixed or recorded in this file; exercised by a capability-gated test or a documented manual run recorded here. + +## Context + +While implementing the -& record-continuation token (G-045, archived), tclparser's +'parse' command was considered and rejected as the splitter's parsing substrate: the +definition dialect legitimately deviates from Tcl (unbalanced-bracket ANSI escapes as +data inside quoted values, line-based base-indent semantics, the -& token), and a +binary dependency in punk::args - the module that must load everywhere, including +bootsupport contexts and plain tclsh - inverts the G-004 direction. The full analysis +is recorded in goals/archive/G-045-punkargs-authoring-ergonomics.md (Progress, +increment 3) and in the punk::args 0.7.0 changelog. + +The rejected option still has value as a diagnostic: records are shaped like Tcl +commands (newline-delimited, quote/brace/backslash continuation), so where the binary +is available, parse-command boundaries are an independent oracle for the splitter. +As the dialect grows, an automated cross-check catches unintended divergence early - +the splitter's own testsuites (recordcontinuation.test, normalize.test, defquoting.test, +rendering.test) pin known behaviour but cannot flag novel drift against Tcl semantics. + +## Approach + +- Comparison pipeline: take the definition text as the splitter receives it, produce + splitter records; separately ansistrip the text, pre-join -& continuations (or + classify them), then iterate tclparser 'parse command' to produce boundary ranges; + compare record counts and boundary line positions, reporting divergences with + context. +- Corpus: registered definitions via punk::args introspection (update_definitions + + raw defs), plus ad-hoc text input for testing definition snippets during dialect + work. +- Home: this is developer tooling, not runtime behaviour - candidates are a + punk::args dev/diagnostic namespace proc (loaded lazily), a dev commandset command, + or a scriptlib/developer script following goals_lint.tcl precedent. Decide in the + work; the constraint is only that punk::args' runtime footprint is unchanged. +- The tclparser binary currently ships only in the punk9win kit (lib_tcl9); the lint + is expected to run there (or any environment with the package installed). If G-070 + (pure-Tcl tclparser) lands, the gate widens automatically if the lint requires + 'parser' via ordinary package resolution. + +## Notes + +- Related: G-045 (archived - origin of the rejected-then-repurposed idea and the + dialect features the comparison must understand), G-070 (pure-Tcl tclparser would + let the lint run without the binary), G-004 (why the runtime must stay + dependency-free). diff --git a/goals/G-070-pure-tcl-tclparser.md b/goals/G-070-pure-tcl-tclparser.md new file mode 100644 index 00000000..beed06ad --- /dev/null +++ b/goals/G-070-pure-tcl-tclparser.md @@ -0,0 +1,56 @@ +# G-070 Pure-Tcl tclparser: parse-command API fallback with behavioural parity against the C library + +Status: proposed +Scope: src/modules/punk/lib-999999.0a1.0.tm (tclparser_tcl stub + dispatch; new module if size warrants - decided in the work), src/tests/modules/punk/lib/ (parity + fallback suites), TEMP_REFERENCE/ (tclparser reference source, user-provided, read-only) +Goal: punk::lib's script-analysis machinery (tclword_to_scriptlist and the dependent analysis paths) runs on runtimes without the tclparser C binary, via a pure-Tcl implementation of the tclparser 'parse' command API covering at least the subcommands and token shapes punk::lib consumes - with behavioural parity verified against the C library rather than assumed, and the C parser still preferred where present (performance). +Acceptance: the punk::lib tclparser_tcl error stub is replaced by a working pure-Tcl implementation of the 'parse' subcommands punk::lib currently uses ('parse command' at minimum; the full covered set enumerated in this file during the work), returning the same parse-tree shapes (ranges, token types, expansion handling) the C library returns for those calls; a parity testsuite compares pure-Tcl output against the C tclparser across a recorded probe corpus (representative punkshell module scripts plus edge cases: {*} expansion, comments, backslash-newline continuation, nested command substitution, braces/quotes in words) - parity tests capability-gated on the binary, pure-Tcl-only tests running everywhere; a documented punk::lib analysis entry point (e.g. tclword_to_scriptlist) demonstrably works under a plain tclsh with no parser binary on the package path; dispatch prefers the C parser when available with the fallback engaging automatically otherwise; the reference source's location under TEMP_REFERENCE and its provenance (origin URL, version/checkin) are recorded in this file. + +## Context + +punk::lib uses the tclparser C library's 'parse command' for script analysis (word +-> scriptlist conversion, argument/expansion classification - the G-019 +dependency-scan direction). The binary ships only in the punk9win kit's lib_tcl9, +so the analysis paths are unavailable on plain tclsh, other kits and other +platforms - and lib-999999.0a1.0.tm carries an explicit placeholder: +tclparser_tcl (currently an error stub telling the user to install the C library) +plus a recorded deliberation at tclscript_to_toplevelinfo noting "we don't have a +pure-tcl implementation of 'parse' available as a fallback". A tested pure-Tcl +implementation removes the binary constraint from every current and future consumer +and aligns with the G-004 no-committed-binaries direction. It also widens where the +G-069 splitter cross-check lint can run. + +The user will place the latest obtainable tclparser source (C implementation and +any docs/tests) under TEMP_REFERENCE as the read-only behavioural reference +(candidate upstreams: the aspect fossil fork at chiselapp and the ActiveState +teapot tree, both already noted in the tclparser_tcl stub comments; punkshell also +carries a moduledoc for the parse command at +src/modules/punk/args/moduledoc/parser-999999.0a1.0.tm). + +## Approach + +- Scope the API by consumption, not by the full tclparser surface: enumerate the + subcommands and result fields punk::lib actually reads (parse command ranges, + token lists, restRange handling, simple-expansion trees), implement those first, + and record the covered/uncovered set here. Other subcommands (expr, varname, + list) follow only if a consumer needs them. +- Parity-first workflow (G-055 spirit): build the probe corpus and the + compare-against-C harness before/alongside the implementation, so every + implemented construct lands with a parity pin. Corpus draws on real punkshell + module sources (rich in expansions, multi-line commands and comments) plus + targeted edge cases. +- Dispatch: a thin front (in punk::lib or the new module) that package-requires the + C parser and falls back to the pure-Tcl engine; consumers call one entry point. + Performance note recorded rather than chased - the C library stays preferred + where installed. +- Placement: the stub lives in punk::lib today; a parser is a sizeable, + separately-testable unit, so a dedicated module (name TBD, e.g. under punk::) + with punk::lib delegating is the likely shape - decided in the work. + +## Notes + +- Related: G-019 (dependency-scan module trimming - the main analysis consumer), + G-004 (binary-free direction), G-069 (lint gains a binary-free substrate), + G-055 (parity-verification workflow precedent). +- The tclparser_tcl stub's partial char-loop sketch (in_dq/in_cb/escape state) is a + starting point only; the reference source's Tcl_ParseCommand semantics are the + contract.