# G-107 buildsuite library test runs: record/gate external library testsuites Status: achieved 2026-07-21 Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86) Goal: the external libraries the buildsuites compile and ship (thread, tclvfs, tcllib+tcllibc, tklib, tk; future bi members as they join) have their own testsuites run under the suite-built shell as named recipe steps with per-library policy - gate (parsed totals vs tracked dispositioned baseline, the core test-gate mechanism), record (complete + emit report artifacts the suite summary points to), or opt-in - so suite artifacts carry test evidence for the exact zig-built combinations nobody upstream tests. Acceptance: thread and tclvfs run gated with tracked dispositioned baselines, deterministic across two consecutive full runs; tcllib (with tcllibc accelerators engaged) and tklib run in record mode emitting report artifacts whose paths are recorded in this file; a Tk step exists opt-in with a recorded first census; per-library policy is overridable at invocation without recipe edits; a testsuite that fails to complete (no totals) fails its step in every mode; the evidence-summary shape G-103 metadata consumes is documented here. ## Context Drafted 2026-07-21 from user question: we run and gate Tcl's core testsuite (G-098 mechanism, G-102 step shape) but never run or record the testsuites of the libraries we build and ship. Survey of the pinned sources (_build/suite_tcl90 checkouts, zon pins as of G-102): - thread: tests/ with all.tcl + 5 test files (thread, tpool, tsv, ttrace, one tkt regression). Standard tcltest; timing-sensitive entries expected -> dispositions. - tclvfs: tests/ with all.tcl + 6 files; vfsFtp/vfsUrl are network-reaching - an offline census will skip or disposition those. - tcllib: no top-level tests dir - 305 .test files across modules/*/, driven by sak.tcl (its own runner and log format, NOT all.tcl totals). - tklib: same sak shape, 8 test files, Tk-dependent. - tk: 128-file tests/all.tcl suite - needs an interactive desktop, steals focus, maps real windows, runs tens of minutes, own per-platform census. - critcl: has test/ but is a build tool - tcllibc building + its accelerators passing tcllib's tests validates it transitively; its own suite is out of scope here. MACHINERY: tools/test_gate.tcl is already the generic engine (-testsdir -baseline -logfile -testargs; parses all.tcl totals rather than trusting exit codes; extracts FAILED names; diffs vs dispositioned baseline; reports stale entries). thread/tclvfs should run nearly as-is (verify each all.tcl vintage emits standard runAllTests totals; supply a thin standard driver where not). tcllib/tklib need a wrapper - drive sak and parse its log, or bypass sak with our own tcltest runner emitting standard totals; the latter also forces the mode that matters most: tcllib WITH tcllibc accelerators engaged. That is the single highest-value run in this space - zig-compiled critcl accelerators on windows against a static 9.0.5 tclsh is a combination nobody upstream tests. Same argument, milder, for thread/tclvfs: load-bearing for every kit; a miscompile surfaces as heisenbugs later. KNOWN COSTS (all characterized during G-102): initial failure census per library per platform (one-time, then stable - zon pins mean baselines move only on deliberate repin, not with a moving trunk); base-env needs (cmd.exe/ COMSPEC/PATHEXT/SystemDrive for process-spawning tests, TCL_LIBRARY for cached-exe runs, captureStdOut on cached Run steps); firewall consent when socket-using tests first listen (tcllib comm will trigger it); wall-clock (thread+tclvfs trivial next to the ~70k-test core gate; tcllib adds real minutes; tk is the outlier). POLICY TIERS (per-library defaults in the recipe, -D overridable at invocation; no new config file until suite_tcl86 forces a shared shape): - gate (after census): thread, tclvfs - small, load-bearing; fail only on UNEXPLAINED failures vs the tracked dispositioned baseline (core-gate semantics, including stale-entry reporting). - record: tcllib+tcllibc, tklib - step completes regardless, emits parsed summary + full log under _build//out/testreports/, and the suite summary points at them; promote to gate per-library once the census is dispositioned and two-run deterministic. - opt-in: tk full suite - named step outside the default composite; intended as mandatory before PUBLISHING a bi artifact, not on every build. Every mode: a run that does not complete (no totals line) FAILS the step - infrastructure breakage must never masquerade as recorded results. Gate runs stay under the plain built shell (as the core gate today); piperepl-variant divergence remains the behaviour-matrix territory of G-096 (achieved 2026-07-20; its residual patch-characterization item). Relationships: G-098 (achieved 2026-07-20) - gate mechanism precedent. G-102 (achieved 2026-07-21) - the post-tclsh step shape these runs slot into. G-103 - artifact metadata is the consumer of the evidence summaries (a family runtime published to punkbin carries per-component test evidence; aligns with G-103's note that punkbin publication follows its artifacts). G-099/G-100/ G-101 - the 8.6 arc inherits the mechanism (thread 2.8 / tclvfs baselines on 8.6); same born-onto-the-mechanism fork-timing logic as G-102-before-G-099. ## Evidence summary shape (G-103 consumer contract) Every test step (the per-library steps AND the core test-gate) writes a line-record evidence summary to `_build//out/testreports/.summary` (`tclcore.summary` for the core gate) alongside the full run log `.log`. The format is the suite's plain-Tcl line-record convention (sources.config precedent): `#` comment lines and blank lines ignored; every other line is a two-element Tcl list ` ` where `` may itself be a list. Parse: `foreach` non-comment line, `lindex $line 0` = key, `lindex $line 1` = value. Keys (write order as listed; `unexplained`/`stalebaseline` are gate-mode-only, totals keys are absent when the run produced no totals - otherwise every key is present after any attempted run): - `library` step label (`thread`, `tclvfs`, `tcllib`, `tklib`, `tk`, `tclcore`) - `mode` `gate` | `record` - `shell` absolute path of the shell that ran the suite ([info nameofexecutable]) - `driver` absolute path of the driver script run under the shell - `testsdir` absolute path of the run cwd - `testargs` extra driver/tcltest args the run was invoked with (list, may be empty) - `notfiles` test-file glob patterns excluded from the run (driver -notfile; list, may be empty) - `baseline` absolute path of the dispositioned baseline ("" in record mode) - `logfile` absolute path of the full run log - `total` `passed` `skipped` `failed` parsed aggregate totals (absent only when the run produced no totals - see `failreason`) - `failednames` distinct failed test names (parse of `==== FAILED`) - `errored_files` count of tcllib-family driver `@+` file-error blocks - `errorexit_files` file names from a runAllTests `Test files exiting with errors:` section - `unexplained` gate mode: failed names not in the baseline (gate-fail set) - `stalebaseline` gate mode, full runs: baseline entries that did not fail - `result` `pass` | `fail` - `failreason` present when result=fail: `no-totals` | `unexplained` Semantics: `result pass` in record mode asserts only completeness (totals parsed); consumers wanting "green" must also check `failed`/`failednames`. A missing summary file after a test step means the step failed before the run could be attempted (config error) - the step exit code is authoritative. G-103 artifact metadata should carry (or point at) these summaries per shipped component; publication tooling can require `result pass` + gate mode for the gate-tier libraries. ## Progress - 2026-07-21 implementation landed (suite_tcl90): - `tools/test_gate.tcl` generalized into the shared engine: `-mode gate|record`, `-driver` (default `all.tcl` in `-testsdir`; tcllib-family trees driven via their own `support/devel/all.tcl` - sak's underlying driver, standard aggregate totals, per-file child interps), `-summaryfile`/`-library` evidence output, file-error evidence capture (`@+` blocks, runAllTests error-exit sections). No-totals fails the step in every mode. Core gate behaviour unchanged (totals + baseline diff + stale notes); the core gate step now also writes `tclcore.summary`. - `build905.zig`: per-library steps `test-thread test-tclvfs test-tcllib test-tklib test-tk` + composite `test-libraries` (tk excluded - opt-in); per-library `-Dtestpolicy-=gate|record|skip` and `-Dtestargs-` invocation overrides (no recipe edits needed); baselines staged in both flows (suite.tcl item list + bootstrap recipe_items). - `suite.tcl`: `test` action takes `-steps` (default `test-gate`) and `-zigargs` passthrough; passes `-j1` so combined test invocations serialize (zig runs independent steps concurrently; thread's suite is timing-sensitive). - thread version-stamp defect FOUND AND FIXED by the first census attempt: `build_tclthread.zig` hardcoded 3.0.1, compiling thread trunk 3.0.7 sources with a stale PACKAGE_VERSION - the dll self-reported 3.0.1, the pkgIndex declared only capital-T `Thread`, and the thread testsuite's `package require -exact thread 3.0.7` could never load it (nothing previously loaded-tested thread at all: exactly this goal's gap). Now version-derived from the checkout's AC_INIT (`build_common.acInitVersion`), dll `tcl9thread.dll`, install dir `thread`, pkgIndex declares BOTH `Thread` and `thread`, rc resource fields derived. Stale 3.0.1 artifacts removed from the stage prefix. NOTE for G-103/kit work: the punk905 kit payload (`src/vfs/punk9wintk905.vfs/lib_tcl9/thread3.0.1/`) still carries the old self-consistent 3.0.1-stamped dll copy - refresh to 3.0.7 naming at the next kit/vfs update. - 2026-07-21 census + gates (suite-built tclsh90s.exe 9.0.5, zig 0.16.0, machine superbee; artifacts under `src/buildsuites/_build/suite_tcl90/out/testreports/`): - thread (trunk 3.0.7): 143 run / 117 passed / 26 skipped / 0 failed - EMPTY baseline (`expected_test_failures_thread.txt`); skips constraint- driven (chanTransfer, have_gdbm, have_lmdb). GATE PASS on consecutive full runs with identical totals (plus a third identical run in a combined -j1 invocation). - tclvfs (trunk, vfs 1.4.2): 51 run / 39 passed / 5 skipped / 7 failed, all dispositioned in `expected_test_failures_tclvfs.txt` (3x tcl9 glob-nomatch semantics through vfs handlers: vfs-4.2 vfsZip-1.6 vfsZip-1.7; 4x network-reaching: vfsFtp-1.1..1.3 vfsUrl-2.1). GATE PASS on two consecutive full runs with identical totals. - tklib (trunk 0.9): RECORDED 796 run / 688 passed / 90 skipped / 18 failed (widgetPlus entryPlus/spinboxPlus behaviour drift under Tk 9: entry-1.x, spinbox-1.x). Artifacts: `out/testreports/tklib.log` + `tklib.summary`. - tcllib (trunk 2.0) WITH tcllibc accelerators engaged: RECORDED 62328 run / 59240 passed / 3034 skipped / 54 failed / 0 errored files, ~45 min wall. Engagement evidence: `E tcllibc 2.0` markers (external tcllibc loaded from the suite's out/lib/tcllibc) and per-implementation test variants running (e.g. crc32-critcl-* alongside crc32-tcl-*). Failure classes (record-only, no dispositions required; candidates for later gate promotion work): picoirc 22x (network/socket behaviour), crc16/crc32 9x (both tcl and critcl variants fail identically - implementation-independent), httpd proxy/cgi 4x, fileutil find/multiop family 6x, fhome-1.1.1/2.3.1 (the machine's OS-username != profile-dir trait, same as the core fCmd dispositions), install-2.1/2.2, matrix-16.7/16.8, tar-add, uri-4.1, errorcommand-1.2/1.3, example-1.1/1.4/2.1, profiler-11.0-tkt, zipfile-encode-6.4/6.5. Artifacts: `out/testreports/tcllib.log` (6.5MB full output) + `tcllib.summary`. - no-totals failure path VERIFIED in both modes (acceptance clause): tclvfs driven with an aborting testarg (-badflag) fails the step (exit 1) in gate AND record mode, summary records `result fail` / `failreason no-totals`; clean gate rerun afterwards restored artifacts (third identical-totals PASS). - core gate parity under the generalized engine: subset run (`suite.tcl test -testargs {-file assocd.test}`) GATE PASS, 11 run / 11 skipped (assocd tests are C-test-command constrained - skip in release builds), `tclcore.summary` written. - tk (core-9-0-branch 9.0.2, opt-in step): RECORDED 9865 run / 8127 passed / 1711 skipped / 27 failed / 0 errored files. NATIVE-DIALOG FILES EXCLUDED by default (`notfiles {winDialog.test winMsgbox.test}` - 91 dialog-popping cases whose message-injection automation does not drive the real OS dialogs under the batch/suite-built shell; the first attempt hung on winDialog and the engine correctly reported no-totals/fail, which is what motivated the -notfiles exclusion). Failures are live-desktop/geometry-sensitive classes expected for a windowed suite run under automation: scrollbar-3.x (12), grid-16/18 (geometry), imgListFormat-3.x, winWm/wm-geometry (window-manager placement), clipboard-4.x, frame, bind-34.2 (1px pointer-warp), safe-1.2, textImage, pack-19.2, image-6.2. Artifacts: `out/testreports/tk.log` + `tk.summary`. This is the recorded FIRST CENSUS; promotion to any gated subset is future work (a desktop-stable Tk baseline needs its own characterization, out of this goal's scope). - The -notfiles exclusion mechanism (added when the Tk census surfaced the blocking-dialog problem): a dedicated engine option + `-Dtestnotfiles-` recipe knob for excluding whole test FILES that cannot run in a context, as a Tcl list value the driver receives as `-notfile` (-testargs cannot carry a multi-file list through the option chain). Default empty for every library except tk. Recorded in the evidence summary (`notfiles` key). - All six acceptance clauses satisfied: thread+tclvfs gated with tracked baselines deterministic across consecutive runs; tcllib (tcllibc engaged) and tklib recorded with artifact paths here; tk opt-in step with a recorded first census; per-library policy/testargs/notfiles overridable at invocation (no recipe edits); no-totals fails the step in gate AND record mode (verified); the evidence-summary shape documented above. ## Notes - 2026-07-21: drafted and approved from the library-test discussion (survey + policy tiers above). - 2026-07-21: activated at user request ("Activate G-107 and implement") - taken ahead of G-103 in the previously confirmed order.