diff --git a/GOALS.md b/GOALS.md index 01cb059c..52cd76b8 100644 --- a/GOALS.md +++ b/GOALS.md @@ -270,7 +270,7 @@ Scope: lint surface (location settled in the work: punk::args dev helper, dev co Detail: goals/G-069-splitter-tclparser-lint.md ### G-070 [proposed] Pure-Tcl tclparser: parse-command API fallback with behavioural parity against the C library -Scope: src/modules/punk/lib-999999.0a1.0.tm (tclparser_tcl stub + dispatch; new module if size warrants - decided in the work); src/tests/modules/punk/lib/ (parity + fallback suites); TEMP_REFERENCE/ (tclparser reference source, user-provided, read-only) +Scope: src/modules/punk/lib-999999.0a1.0.tm (tclparser_tcl stub + dispatch; new module if size warrants - decided in the work); src/tests/modules/punk/lib/ (parity + fallback suites); external fork repo c:/repo/jn/tclparser_punk (read-only reference snapshots + the zig-built parser dll oracle - see G-154) Detail: goals/G-070-pure-tcl-tclparser.md ### G-072 [proposed] punk::args compound clause types: named alternates with per-element typing and per-alternate arity (try-class handlers) @@ -418,3 +418,7 @@ Detail: goals/G-151-punkargs-annotated-success-render.md Scope: external fork repo c:/repo/jn/metakit_punk (gitea jn/metakit - the port itself lands there); src/buildsuites/suite_tcl86/sources.config and src/buildsuites/suite_tcl86/build.zig.zon (ready-to-land metakit records drafted in the detail file; landing them is G-101-era consumption); TEMP_REFERENCE/{metakit_jnorthrup,metakit_pooryorick,KitCreator} (read-only patch/boot-chain reference - machine-local); goals/G-101-tcl86-kit-container-strategy.md (consumer decision record) Detail: goals/G-153-metakit-punk-zig-port.md +### G-154 [proposed] tclparser_punk zig build: fork-as-upstream parser dll for kits and the G-070 oracle +Scope: external fork repo c:/repo/jn/tclparser_punk (gitea jn/tclparser - the build lane lands there); src/buildsuites/suite_tcl90/sources.config and src/buildsuites/suite_tcl90/build.zig.zon (ready-to-land tclparser records drafted in the detail file; landing them is suite-consumption era); src/buildsuites/suite_tcl86/sources.config and src/buildsuites/suite_tcl86/build.zig.zon (same - the 8.6 records); src/vendorlib_tcl8/win32-x86_64/tclparser1.8 and src/vfs/*/lib_tcl9/tclparser1.9 (vendored third-party parser packages - replacement targets at consumption, not touched by this goal); goals/G-070-pure-tcl-tclparser.md (consumer - parity oracle and corpus seed) +Detail: goals/G-154-tclparser-punk-zig-build.md + diff --git a/goals/G-070-pure-tcl-tclparser.md b/goals/G-070-pure-tcl-tclparser.md index 8bd9a1ef..e911df54 100644 --- a/goals/G-070-pure-tcl-tclparser.md +++ b/goals/G-070-pure-tcl-tclparser.md @@ -1,9 +1,9 @@ # G-070 Pure-Tcl tclparser: parse-command API fallback with behavioural parity against the C library Status: proposed -Scope: src/modules/punk/lib-999999.0a1.0.tm (tclparser_tcl stub + dispatch; new module if size warrants - decided in the work); src/tests/modules/punk/lib/ (parity + fallback suites); TEMP_REFERENCE/ (tclparser reference source, user-provided, read-only) +Scope: src/modules/punk/lib-999999.0a1.0.tm (tclparser_tcl stub + dispatch; new module if size warrants - decided in the work); src/tests/modules/punk/lib/ (parity + fallback suites); external fork repo c:/repo/jn/tclparser_punk (read-only reference snapshots + the zig-built parser dll oracle - see G-154) Goal: punk::lib's script-analysis machinery (tclword_to_scriptlist and the dependent analysis paths) runs on runtimes without the tclparser C binary, via a pure-Tcl implementation of the tclparser 'parse' command API covering at least the subcommands and token shapes punk::lib consumes - with behavioural parity verified against the C library rather than assumed, and the C parser still preferred where present (performance). -Acceptance: the punk::lib tclparser_tcl error stub is replaced by a working pure-Tcl implementation of the 'parse' subcommands punk::lib currently uses ('parse command' at minimum; the full covered set enumerated in this file during the work), returning the same parse-tree shapes (ranges, token types, expansion handling) the C library returns for those calls; a parity testsuite compares pure-Tcl output against the C tclparser across a recorded probe corpus (representative punkshell module scripts plus edge cases: {*} expansion, comments, backslash-newline continuation, nested command substitution, braces/quotes in words) - parity tests capability-gated on the binary, pure-Tcl-only tests running everywhere; a documented punk::lib analysis entry point (e.g. tclword_to_scriptlist) demonstrably works under a plain tclsh with no parser binary on the package path; dispatch prefers the C parser when available with the fallback engaging automatically otherwise; the reference source's location under TEMP_REFERENCE and its provenance (origin URL, version/checkin) are recorded in this file. +Acceptance: the punk::lib tclparser_tcl error stub is replaced by a working pure-Tcl implementation of the 'parse' subcommands punk::lib currently uses ('parse command' at minimum; the full covered set enumerated in this file during the work), returning the same parse-tree shapes (ranges, token types, expansion handling) the C library returns for those calls; a parity testsuite compares pure-Tcl output against the C tclparser across a recorded probe corpus (representative punkshell module scripts plus edge cases: {*} expansion, comments, backslash-newline continuation, nested command substitution, braces/quotes in words) - parity tests capability-gated on the binary, pure-Tcl-only tests running everywhere; a documented punk::lib analysis entry point (e.g. tclword_to_scriptlist) demonstrably works under a plain tclsh with no parser binary on the package path; dispatch prefers the C parser when available with the fallback engaging automatically otherwise; the reference source consulted (the tclparser_punk fork repo - snapshot and checkin/commit identity per its src/reference provenance table) is recorded in this file. ## Context @@ -19,12 +19,16 @@ implementation removes the binary constraint from every current and future consu and aligns with the G-004 no-committed-binaries direction. It also widens where the G-069 splitter cross-check lint can run. -The user will place the latest obtainable tclparser source (C implementation and -any docs/tests) under TEMP_REFERENCE as the read-only behavioural reference -(candidate upstreams: the aspect fossil fork at chiselapp and the ActiveState -teapot tree, both already noted in the tclparser_tcl stub comments; punkshell also -carries a moduledoc for the parse command at -src/modules/punk/args/moduledoc/parser-999999.0a1.0.tm). +The behavioural reference is the punkshell-maintained tclparser fork at +c:/repo/jn/tclparser_punk (seeded 2026-08-02 - G-154): byte-frozen snapshots of +both upstream lineages (the aspect chiselapp fossil mirror at 1.4.1 and the +tcltk-depot parser 1.9 Tcl 9 port - the candidate upstreams originally noted in +the tclparser_tcl stub comments, surveyed and pinned) with a provenance table +under src/reference/, the ported 114-case upstream testsuite (a ready +parity-corpus seed), and zig-built parser dlls for both Tcl generations - the C +oracle the parity suite compares against, no longer limited to the punk9win +kit's shipped binary. punkshell also carries a moduledoc for the parse command +at src/modules/punk/args/moduledoc/parser-999999.0a1.0.tm. ## Approach @@ -51,6 +55,10 @@ src/modules/punk/args/moduledoc/parser-999999.0a1.0.tm). - Related: G-019 (dependency-scan module trimming - the main analysis consumer), G-004 (binary-free direction), G-069 (lint gains a binary-free substrate), G-055 (parity-verification workflow precedent). +- Related: G-154 (the tclparser_punk fork/build lane this goal's reference + snapshots and dll oracle come from; its testsuite gate transcript and + upstream-landscape survey live in that goal's detail file and the fork's + src/reference/AGENTS.md). - The tclparser_tcl stub's partial char-loop sketch (in_dq/in_cb/escape state) is a starting point only; the reference source's Tcl_ParseCommand semantics are the contract. diff --git a/goals/G-154-tclparser-punk-zig-build.md b/goals/G-154-tclparser-punk-zig-build.md new file mode 100644 index 00000000..322bff70 --- /dev/null +++ b/goals/G-154-tclparser-punk-zig-build.md @@ -0,0 +1,160 @@ +# G-154 tclparser_punk zig build: fork-as-upstream parser dll for kits and the G-070 oracle + +Status: proposed +Scope: external fork repo c:/repo/jn/tclparser_punk (gitea jn/tclparser - the build lane lands there); src/buildsuites/suite_tcl90/sources.config and src/buildsuites/suite_tcl90/build.zig.zon (ready-to-land tclparser records drafted in the detail file; landing them is suite-consumption era); src/buildsuites/suite_tcl86/sources.config and src/buildsuites/suite_tcl86/build.zig.zon (same - the 8.6 records); src/vendorlib_tcl8/win32-x86_64/tclparser1.8 and src/vfs/*/lib_tcl9/tclparser1.9 (vendored third-party parser packages - replacement targets at consumption, not touched by this goal); goals/G-070-pure-tcl-tclparser.md (consumer - parity oracle and corpus seed) +Goal: The punkshell-maintained tclparser fork (c:/repo/jn/tclparser_punk - gitea jn/tclparser - tcltk-depot parser 1.9 porting baseline with the aspect fossil lineage preserved as an in-repo reference snapshot) builds standalone under the pinned zig toolchain as stubs-linked parser dlls for Tcl 8.6 and Tcl 9, for win32-x86_64 and win32-ix86 alike, gated by the ported upstream testsuite - so kit parser packages (today unprovenanced third-party dlls) and the G-070 parity oracle rest on a first-party, commit-pinned, provenance-carrying source referenced by the buildsuites the punkzip/metakit way (commit pin in sources.config plus zon tarball record) rather than on dormant third-party upstreams or opaque vendored binaries. +Acceptance: In c:/repo/jn/tclparser_punk under the pinned zig 0.16.0, 'zig build' (lazy zon tcl pins, no local source trees) and 'zig build dual' pass, producing tcl9tclparser19.dll and tclparser19t.dll for both windows arches; 'zig build test -Dtclsh9= -Dtclsh86=' passes the ported 114-case upstream suite for both generations under suite-built shells with the transcript recorded in this file; both upstream snapshots are landed byte-frozen with the provenance table and root LICENSE (license.terms verbatim); a punk kit-class shell for each generation loads the built dll and completes a parse-command round trip recorded in this file; and the fork commit is pinned with ready-to-land sources.config and build.zig.zon record text for both suites captured in this file. + +## Context + +punkshell kits already ship the tclparser C extension (Tcl package `parser`: the +TclPro parser exposing Tcl_ParseCommand and friends as the `parse` command) as +third-party binaries: tclparser1.9 dirs (tcl9tclparser19.dll, magicsplat-shaped) +in several vfs lib_tcl9 trees and tclparser1.8 (tclparser18.dll) in +src/vendorlib_tcl8 and bawt-era vfs trees. Those dlls arrived in bulk commits +(ae837492, 332d6732) with no provenance records - exactly the gap the +provenance/trust direction closes for runtimes (G-123, achieved - see +goals/archive/G-123-thirdparty-runtime-tiers.md). punk::lib's script-analysis +machinery consumes the package (G-019 direction), G-069 wants it as a lint +oracle, and G-070 needs it as the behavioural reference for a pure-Tcl +implementation. + +Upstream landscape (survey 2026-08-02, detail in the fork's +src/reference/AGENTS.md): the aspect chiselapp fossil repo is a TclPro CVS +mirror with conservative fixes, version 1.4.1, dormant since 2015-06-18 (trunk +53f5b44b14); tcltk-depot/tcl-parser on github is parser 1.9 - the same TclPro +component extracted from the tclprodebug lineage and ported to Tcl 9 (July +2025, 4 commits, tip c6c841d6). The two lineages converged: the whole +tclParser.c delta is prototype/type modernisation including a Tcl_Size compat +shim that builds 8.6 AND 9 from one source; parse logic (including the +TclFindElement-based `parse list` body) is identical. Both upstreams are +dormant-to-minimal, so the buildsuites' live-upstream referencing pattern does +not fit; the applicable precedent is punkzip (G-126, achieved) / metakit_punk +(G-153, active): a punkshell-maintained fork that IS the upstream, self-hosted +on gitea, commit-pinned by the suites, with its own zig build system. + +The fork repo was seeded 2026-08-02 (user-created git repo, agent-built +skeleton per the metakit_punk shape): DOX/AGENTS.md governance, byte-frozen +snapshots of BOTH upstreams under src/reference/ with a provenance table and +fork-difference analysis, port baseline (tcltk-depot tclParser.c + 114-case +testsuite + parse docs, verbatim), root build.zig/build.zig.zon, and the +testsuite gate. This goal is the plan of record for the lane; the port is +driven from punkshell sessions with access to this goals tree. + +## Approach + +- Source-identity pins: the fork's lazy zon tcl dependencies are the SAME + checkouts the suites pin (core-9-0-branch 1a9c3b9d96 for tcl9, + core-8-6-branch 1d1d5cbd91 for tcl86), with -Dtcl9-src/-Dtcl86-src escape + hatches, so a bare clone builds with no punkshell present and source + identity matches the suites. +- Single source, per-generation compile: the depot baseline's Tcl_Size shim + means one tclParser.c compiled twice - against tcl9 headers producing + tcl9tclparser19.dll and against tcl86 headers producing tclparser19t.dll + (drop-in artifact names matching the packages kits already carry; install + layout lib///parser1.9/ with generated pkgIndex.tcl, explicit + Tclparser load prefix; platform dirs per punkshell canon win32-x86_64 / + win32-ix86 - G-105). +- Stub library compiled from generic/tclStubLib.c of the pinned tree, never a + prebuilt archive: a foreign-toolchain libtclstub .a was measured to leave + Tcl_InitStubs unresolved under zig's lld (2026-08-02, suite out/lib + archives) - compiling the one-file stub lib sidesteps archive-format + coupling entirely. +- Testsuite as the gate: tools/testsuite_check.zig materialises a scratch + dir, wraps the dll load + `source all.tcl` under a caller-provided tclsh, + and gates on the tcltest totals line (Failed 0, Total > 0). Dual-generation + green is the port-correctness signal; the same 114-case suite seeds G-070's + parity corpus. +- win32-ix86: the `dual` step builds both arches for both generations + (compile/link proof). The ix86 RUNTIME check needs a 32-bit shell and lands + when G-130's runtimes exist; risk direction is benign (the extension is a + thin wrapper over the core parser API). +- Suite consumption (later, not this goal): sources.config commit pin + zon + tarball record per suite, replacing the vendored parser package dirs at the + same time; delivery-tier shape (punkbin lib-tier precedent: G-138, achieved + - see goals/archive/G-138-punkbin-library-artifact-tier.md) decided at + consumption. Draft records below become concrete once the fork is pushed + and pinned. + +## Notes + +- Related: G-070 (proposed) - primary consumer: the fork is its behavioural + reference (snapshots + provenance) and the zig-built dll its parity oracle; + the ported testsuite is its corpus seed. G-070's Scope/Acceptance were + reworded 2026-08-02 to point here (formerly TEMP_REFERENCE, which never + received a tclparser tree). +- Related: G-069 (proposed) - consumer: the splitter cross-check lint runs + where the parser binary is available; first-party dlls widen that gate + (and G-070 widens it further). +- Related: G-153 (active) - the pattern precedent (fork-as-upstream, dual-arch + acceptance, suite pin-consumption drafts); this goal shares the + sources.config/build.zig.zon record surfaces with its metakit records. +- Related: G-101 (proposed) - owns the 8.6 suite consumption era in which the + suite_tcl86 records land (as G-101-era consumption, like metakit's). +- Related: G-105 (proposed) - platform-name canon the install layout follows. +- Related: G-130 (proposed) - the win32-ix86 runtime lane; the ix86 dll + runtime check runs under its 32-bit shells when they exist. +- Related: G-004/G-005/G-006 (proposed) - binary-free direction: consumption + replaces the unprovenanced vendored parser dlls (bulk commits ae837492, + 332d6732 carry no provenance records) with first-party builds. +- Related: G-019 (proposed) - downstream analysis consumer via punk::lib. +- Suite-dir neighbours G-131/G-108/G-110/G-116 have no parser coupling beyond + the shared suite record surfaces. +- Overlap survey 2026-08-02: goals_xref paths src/buildsuites/suite_tcl86, + src/buildsuites/suite_tcl90, src/vendorlib_tcl8, + src/modules/punk/lib-999999.0a1.0.tm -> G-101 G-153 G-131 G-105 G-130 + G-108 G-110 G-116 G-019 G-070 (dispositions above; src/vendorlib_tcl8 + intersects no live goal). +- Draft sources.config record (both suites; name/dir chosen to match the + gitea repo name; the ssh origin ssh://git@pcm-gitea1.corp.intx.com.au:2222/jn/tclparser + is the push remote, the https web URL is the fetch/pin form): + `source tclparser git https://www.gitea1.intx.com.au/jn/tclparser tclparser` +- Draft build.zig.zon record (both suites; hash computed with the pinned zig + at landing time via 'zig fetch' from the suite dir per that file's TRAP + note): + `.tclparser = .{ .url = "https://www.gitea1.intx.com.au/jn/tclparser/archive/.tar.gz", .hash = "", .lazy = true },` +- Package identity facts: the dll provides `parser 1.9` for BOTH generations + (PACKAGE_VERSION define; upstream-shaped for drop-in compatibility). The + fork repo's own version is its zon .version (0.1.0), independent - + punkzip/metakit precedent. +- Testsuite quirk worth knowing for G-070 corpus work: tests/all.tcl checks + `info commands parser` (not `parse`) before `package require parser`, so + the package-require path always runs unless a command named `parser` + exists; a loaded dll satisfies it via Tcl_PkgProvide. The 2 skips per + generation are the OTHER generation's constraint-gated cases (tcl8/tcl9 + tcltest constraints). +- Build accommodation facts (rationale in the fork's build.zig header): + -fno-sanitize=undefined/-fno-sanitize-trap=undefined (2000-era C under + zig's default trapping UBSan on Debug-class builds; suite tcl/tk/zlib + precedent), -Wno-inconsistent-dllimport (EXTERN storage-class quirk after + tclInt.h resets TCL_STORAGE_CLASS; symbol still exports). The single + tclInt.h dependency is TclFindElement (`parse list`), reached through the + internal stub table; going public-API-only is a recorded fork option, not + assumed. + +## Progress + +Seed increment landed 2026-08-02 (fork commits 7c18555 governance+snapshots, +16d2f46 port+build, on local master; push to the gitea origin pending - user +decision): DOX framework (root/src/src-reference AGENTS.md), byte-frozen +snapshots aspect@53f5b44b14 (21 files) + tcltk-depot@c6c841d6 (28 files) with +provenance table, port baseline verbatim (tclParser.c byte-identical to the +depot snapshot; tests; parse.man/parse.n), root LICENSE, build.zig/zon (lazy +suite-matching tcl pins + -D*-src escapes), tools/testsuite_check.zig, dual +step, generated pkgIndex per package dir. + +VERIFIED 2026-08-02 under the pinned zig 0.16.0 (punkshell +bin/tools/zig-x86_64-windows-0.16.0/zig.exe): +- 'zig build' green via BOTH source paths: -D*-src suite checkouts AND the + lazy zon fetches standalone (no local tcl trees). +- 'zig build dual' green: 4 dlls (tcl9+tcl86 x win32-x86_64+win32-ix86). +- 'zig build test' green both generations; totals transcript: + tcl9 (suite tclsh90szip, Tcl 9.0.5): all.tcl: Total 114 Passed 112 Skipped 2 Failed 0 (skipped constraint: 2 tcl8) + tcl86 (suite tclsh86t, Tcl 8.6.18): all.tcl: Total 114 Passed 112 Skipped 2 Failed 0 (skipped constraint: 2 tcl9) + +Remaining for acceptance: +- Push the fork, pin the commit, and make the sources.config/zon draft + records above concrete (zig-fetch hash from a suite dir). +- Kit-class shell load + parse-command round trip per generation, recorded + here (suite-shell load is proven via the gate; the kit-shell transcript is + the remaining acceptance evidence).