From 4189de7d3593a592988013a92c72b9da5e4fdebb Mon Sep 17 00:00:00 2001 From: Julian Noble Date: Tue, 21 Jul 2026 11:33:26 +1000 Subject: [PATCH] G-107 proposed: buildsuite library test runs (record/gate external testsuites) New goal from the 2026-07-21 discussion: the libraries the buildsuites compile and ship (thread, tclvfs, tcllib+tcllibc, tklib, tk) get their own testsuites run under the suite-built shell as named recipe steps with per-library policy - gate (core test-gate mechanism, dispositioned baselines), record (report artifacts under out/testreports/), or opt-in (tk). Context carries the source survey, test_gate.tcl generalization notes, G-102 cost knowns, and the G-103 metadata / 8.6-arc relationships. Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com --- GOALS.md | 4 ++ goals/G-107-buildsuite-library-tests.md | 84 +++++++++++++++++++++++++ 2 files changed, 88 insertions(+) create mode 100644 goals/G-107-buildsuite-library-tests.md diff --git a/GOALS.md b/GOALS.md index 85c8bb72..5775fa57 100644 --- a/GOALS.md +++ b/GOALS.md @@ -361,3 +361,7 @@ Detail: goals/G-105-buildsuite-cross-target.md ### G-106 [proposed] powershell console-mode fallback: maintained raw-mode path for twapi-less runtimes Scope: src/modules/punk/console-999999.0a1.0.tm (enableRaw_powershell/disableRaw_powershell + the persistent server lifecycle), scriptlib/utils/pwsh/ (consolemode_server_async.ps1 canonical + experiment-variant reconciliation), ps-script resolution/packaging across launch contexts (argv0-derived pstooldir; kits via the G-089 scriptlib-in-kits interplay), src/make.tcl shell (the verified launch context) Detail: goals/G-106-powershell-consolemode-fallback.md + +### G-107 [proposed] Buildsuite library test runs: record/gate external library testsuites +Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86) +Detail: goals/G-107-buildsuite-library-tests.md diff --git a/goals/G-107-buildsuite-library-tests.md b/goals/G-107-buildsuite-library-tests.md new file mode 100644 index 00000000..c7727130 --- /dev/null +++ b/goals/G-107-buildsuite-library-tests.md @@ -0,0 +1,84 @@ +# G-107 buildsuite library test runs: record/gate external library testsuites + +Status: proposed +Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86) +Goal: the external libraries the buildsuites compile and ship (thread, tclvfs, tcllib+tcllibc, tklib, tk; future bi members as they join) have their own testsuites run under the suite-built shell as named recipe steps with per-library policy - gate (parsed totals vs tracked dispositioned baseline, the core test-gate mechanism), record (complete + emit report artifacts the suite summary points to), or opt-in - so suite artifacts carry test evidence for the exact zig-built combinations nobody upstream tests. +Acceptance: thread and tclvfs run gated with tracked dispositioned baselines, deterministic across two consecutive full runs; tcllib (with tcllibc accelerators engaged) and tklib run in record mode emitting report artifacts whose paths are recorded in this file; a Tk step exists opt-in with a recorded first census; per-library policy is overridable at invocation without recipe edits; a testsuite that fails to complete (no totals) fails its step in every mode; the evidence-summary shape G-103 metadata consumes is documented here. + +## Context + +Drafted 2026-07-21 from user question: we run and gate Tcl's core testsuite +(G-098 mechanism, G-102 step shape) but never run or record the testsuites of +the libraries we build and ship. Survey of the pinned sources +(_build/suite_tcl90 checkouts, zon pins as of G-102): + +- thread: tests/ with all.tcl + 5 test files (thread, tpool, tsv, ttrace, one + tkt regression). Standard tcltest; timing-sensitive entries expected -> + dispositions. +- tclvfs: tests/ with all.tcl + 6 files; vfsFtp/vfsUrl are network-reaching - + an offline census will skip or disposition those. +- tcllib: no top-level tests dir - 305 .test files across modules/*/, driven + by sak.tcl (its own runner and log format, NOT all.tcl totals). +- tklib: same sak shape, 8 test files, Tk-dependent. +- tk: 128-file tests/all.tcl suite - needs an interactive desktop, steals + focus, maps real windows, runs tens of minutes, own per-platform census. +- critcl: has test/ but is a build tool - tcllibc building + its accelerators + passing tcllib's tests validates it transitively; its own suite is out of + scope here. + +MACHINERY: tools/test_gate.tcl is already the generic engine (-testsdir +-baseline -logfile -testargs; parses all.tcl totals rather than trusting exit +codes; extracts FAILED names; diffs vs dispositioned baseline; reports stale +entries). thread/tclvfs should run nearly as-is (verify each all.tcl vintage +emits standard runAllTests totals; supply a thin standard driver where not). +tcllib/tklib need a wrapper - drive sak and parse its log, or bypass sak with +our own tcltest runner emitting standard totals; the latter also forces the +mode that matters most: tcllib WITH tcllibc accelerators engaged. That is the +single highest-value run in this space - zig-compiled critcl accelerators on +windows against a static 9.0.5 tclsh is a combination nobody upstream tests. +Same argument, milder, for thread/tclvfs: load-bearing for every kit; a +miscompile surfaces as heisenbugs later. + +KNOWN COSTS (all characterized during G-102): initial failure census per +library per platform (one-time, then stable - zon pins mean baselines move +only on deliberate repin, not with a moving trunk); base-env needs (cmd.exe/ +COMSPEC/PATHEXT/SystemDrive for process-spawning tests, TCL_LIBRARY for +cached-exe runs, captureStdOut on cached Run steps); firewall consent when +socket-using tests first listen (tcllib comm will trigger it); wall-clock +(thread+tclvfs trivial next to the ~70k-test core gate; tcllib adds real +minutes; tk is the outlier). + +POLICY TIERS (per-library defaults in the recipe, -D overridable at +invocation; no new config file until suite_tcl86 forces a shared shape): + +- gate (after census): thread, tclvfs - small, load-bearing; fail only on + UNEXPLAINED failures vs the tracked dispositioned baseline (core-gate + semantics, including stale-entry reporting). +- record: tcllib+tcllibc, tklib - step completes regardless, emits parsed + summary + full log under _build//out/testreports/, and the suite + summary points at them; promote to gate per-library once the census is + dispositioned and two-run deterministic. +- opt-in: tk full suite - named step outside the default composite; intended + as mandatory before PUBLISHING a bi artifact, not on every build. + +Every mode: a run that does not complete (no totals line) FAILS the step - +infrastructure breakage must never masquerade as recorded results. + +Gate runs stay under the plain built shell (as the core gate today); +piperepl-variant divergence remains the behaviour-matrix territory of G-096 +(achieved 2026-07-20; its residual patch-characterization item). + +Relationships: G-098 (achieved 2026-07-20) - gate mechanism precedent. G-102 +(achieved 2026-07-21) - the post-tclsh step shape these runs slot into. +G-103 - artifact metadata is the consumer of the evidence summaries (a family +runtime published to punkbin carries per-component test evidence; aligns with +G-103's note that punkbin publication follows its artifacts). G-099/G-100/ +G-101 - the 8.6 arc inherits the mechanism (thread 2.8 / tclvfs baselines on +8.6); same born-onto-the-mechanism fork-timing logic as G-102-before-G-099. + +## Notes + +- 2026-07-21: drafted and approved from the library-test discussion (survey + + policy tiers above). Sequencing recommendation (agent, unconfirmed): + activation fits alongside or just after G-103, before the suite_tcl86 fork; + the user has not yet confirmed a slot in the G-103 -> 8.6-arc order.