punkshell/goals/archive/G-107-buildsuite-library-tests.md

G-107 buildsuite library test runs: record/gate external library testsuites

Status: achieved 2026-07-21 Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86) Goal: the external libraries the buildsuites compile and ship (thread, tclvfs, tcllib+tcllibc, tklib, tk; future bi members as they join) have their own testsuites run under the suite-built shell as named recipe steps with per-library policy - gate (parsed totals vs tracked dispositioned baseline, the core test-gate mechanism), record (complete + emit report artifacts the suite summary points to), or opt-in - so suite artifacts carry test evidence for the exact zig-built combinations nobody upstream tests. Acceptance: thread and tclvfs run gated with tracked dispositioned baselines, deterministic across two consecutive full runs; tcllib (with tcllibc accelerators engaged) and tklib run in record mode emitting report artifacts whose paths are recorded in this file; a Tk step exists opt-in with a recorded first census; per-library policy is overridable at invocation without recipe edits; a testsuite that fails to complete (no totals) fails its step in every mode; the evidence-summary shape G-103 metadata consumes is documented here.

Context

Drafted 2026-07-21 from user question: we run and gate Tcl's core testsuite (G-098 mechanism, G-102 step shape) but never run or record the testsuites of the libraries we build and ship. Survey of the pinned sources (_build/suite_tcl90 checkouts, zon pins as of G-102):

• thread: tests/ with all.tcl + 5 test files (thread, tpool, tsv, ttrace, one tkt regression). Standard tcltest; timing-sensitive entries expected -> dispositions.
• tclvfs: tests/ with all.tcl + 6 files; vfsFtp/vfsUrl are network-reaching - an offline census will skip or disposition those.
• tcllib: no top-level tests dir - 305 .test files across modules/*/, driven by sak.tcl (its own runner and log format, NOT all.tcl totals).
• tklib: same sak shape, 8 test files, Tk-dependent.
• tk: 128-file tests/all.tcl suite - needs an interactive desktop, steals focus, maps real windows, runs tens of minutes, own per-platform census.
• critcl: has test/ but is a build tool - tcllibc building + its accelerators passing tcllib's tests validates it transitively; its own suite is out of scope here.

MACHINERY: tools/test_gate.tcl is already the generic engine (-testsdir -baseline -logfile -testargs; parses all.tcl totals rather than trusting exit codes; extracts FAILED names; diffs vs dispositioned baseline; reports stale entries). thread/tclvfs should run nearly as-is (verify each all.tcl vintage emits standard runAllTests totals; supply a thin standard driver where not). tcllib/tklib need a wrapper - drive sak and parse its log, or bypass sak with our own tcltest runner emitting standard totals; the latter also forces the mode that matters most: tcllib WITH tcllibc accelerators engaged. That is the single highest-value run in this space - zig-compiled critcl accelerators on windows against a static 9.0.5 tclsh is a combination nobody upstream tests. Same argument, milder, for thread/tclvfs: load-bearing for every kit; a miscompile surfaces as heisenbugs later.

KNOWN COSTS (all characterized during G-102): initial failure census per library per platform (one-time, then stable - zon pins mean baselines move only on deliberate repin, not with a moving trunk); base-env needs (cmd.exe/ COMSPEC/PATHEXT/SystemDrive for process-spawning tests, TCL_LIBRARY for cached-exe runs, captureStdOut on cached Run steps); firewall consent when socket-using tests first listen (tcllib comm will trigger it); wall-clock (thread+tclvfs trivial next to the ~70k-test core gate; tcllib adds real minutes; tk is the outlier).

POLICY TIERS (per-library defaults in the recipe, -D overridable at invocation; no new config file until suite_tcl86 forces a shared shape):

• gate (after census): thread, tclvfs - small, load-bearing; fail only on UNEXPLAINED failures vs the tracked dispositioned baseline (core-gate semantics, including stale-entry reporting).
• record: tcllib+tcllibc, tklib - step completes regardless, emits parsed summary + full log under _build//out/testreports/, and the suite summary points at them; promote to gate per-library once the census is dispositioned and two-run deterministic.
• opt-in: tk full suite - named step outside the default composite; intended as mandatory before PUBLISHING a bi artifact, not on every build.

Every mode: a run that does not complete (no totals line) FAILS the step - infrastructure breakage must never masquerade as recorded results.

Gate runs stay under the plain built shell (as the core gate today); piperepl-variant divergence remains the behaviour-matrix territory of G-096 (achieved 2026-07-20; its residual patch-characterization item).

Relationships: G-098 (achieved 2026-07-20) - gate mechanism precedent. G-102 (achieved 2026-07-21) - the post-tclsh step shape these runs slot into. G-103 - artifact metadata is the consumer of the evidence summaries (a family runtime published to punkbin carries per-component test evidence; aligns with G-103's note that punkbin publication follows its artifacts). G-099/G-100/ G-101 - the 8.6 arc inherits the mechanism (thread 2.8 / tclvfs baselines on 8.6); same born-onto-the-mechanism fork-timing logic as G-102-before-G-099.

Evidence summary shape (G-103 consumer contract)

Every test step (the per-library steps AND the core test-gate) writes a line-record evidence summary to _build/<suite>/out/testreports/<library>.summary (tclcore.summary for the core gate) alongside the full run log <library>.log. The format is the suite's plain-Tcl line-record convention (sources.config precedent): # comment lines and blank lines ignored; every other line is a two-element Tcl list <key> <value> where <value> may itself be a list. Parse: foreach non-comment line, lindex $line 0 = key, lindex $line 1 = value.

Keys (write order as listed; unexplained/stalebaseline are gate-mode-only, totals keys are absent when the run produced no totals - otherwise every key is present after any attempted run):

• library step label (thread, tclvfs, tcllib, tklib, tk, tclcore)
• mode gate | record
• shell absolute path of the shell that ran the suite ([info nameofexecutable])
• driver absolute path of the driver script run under the shell
• testsdir absolute path of the run cwd
• testargs extra driver/tcltest args the run was invoked with (list, may be empty)
• notfiles test-file glob patterns excluded from the run (driver -notfile; list, may be empty)
• baseline absolute path of the dispositioned baseline ("" in record mode)
• logfile absolute path of the full run log
• total passed skipped failed parsed aggregate totals (absent only when the run produced no totals - see failreason)
• failednames distinct failed test names (parse of ==== <name> FAILED)
• errored_files count of tcllib-family driver @+ file-error blocks
• errorexit_files file names from a runAllTests Test files exiting with errors: section
• unexplained gate mode: failed names not in the baseline (gate-fail set)
• stalebaseline gate mode, full runs: baseline entries that did not fail
• result pass | fail
• failreason present when result=fail: no-totals | unexplained

Semantics: result pass in record mode asserts only completeness (totals parsed); consumers wanting "green" must also check failed/failednames. A missing summary file after a test step means the step failed before the run could be attempted (config error) - the step exit code is authoritative. G-103 artifact metadata should carry (or point at) these summaries per shipped component; publication tooling can require result pass + gate mode for the gate-tier libraries.

Progress

• 2026-07-21 implementation landed (suite_tcl90):
  • tools/test_gate.tcl generalized into the shared engine: -mode gate|record, -driver (default all.tcl in -testsdir; tcllib-family trees driven via their own support/devel/all.tcl - sak's underlying driver, standard aggregate totals, per-file child interps), -summaryfile/-library evidence output, file-error evidence capture (@+ blocks, runAllTests error-exit sections). No-totals fails the step in every mode. Core gate behaviour unchanged (totals + baseline diff + stale notes); the core gate step now also writes tclcore.summary.
  • build905.zig: per-library steps test-thread test-tclvfs test-tcllib test-tklib test-tk + composite test-libraries (tk excluded - opt-in); per-library -Dtestpolicy-<lib>=gate|record|skip and -Dtestargs-<lib> invocation overrides (no recipe edits needed); baselines staged in both flows (suite.tcl item list + bootstrap recipe_items).
  • suite.tcl: test action takes -steps (default test-gate) and -zigargs passthrough; passes -j1 so combined test invocations serialize (zig runs independent steps concurrently; thread's suite is timing-sensitive).
  • thread version-stamp defect FOUND AND FIXED by the first census attempt: build_tclthread.zig hardcoded 3.0.1, compiling thread trunk 3.0.7 sources with a stale PACKAGE_VERSION - the dll self-reported 3.0.1, the pkgIndex declared only capital-T Thread, and the thread testsuite's package require -exact thread 3.0.7 could never load it (nothing previously loaded-tested thread at all: exactly this goal's gap). Now version-derived from the checkout's AC_INIT (build_common.acInitVersion), dll tcl9thread<nodots>.dll, install dir thread<version>, pkgIndex declares BOTH Thread and thread, rc resource fields derived. Stale 3.0.1 artifacts removed from the stage prefix. NOTE for G-103/kit work: the punk905 kit payload (src/vfs/punk9wintk905.vfs/lib_tcl9/thread3.0.1/) still carries the old self-consistent 3.0.1-stamped dll copy - refresh to 3.0.7 naming at the next kit/vfs update.
• 2026-07-21 census + gates (suite-built tclsh90s.exe 9.0.5, zig 0.16.0, machine superbee; artifacts under src/buildsuites/_build/suite_tcl90/out/testreports/):
  • thread (trunk 3.0.7): 143 run / 117 passed / 26 skipped / 0 failed - EMPTY baseline (expected_test_failures_thread.txt); skips constraint- driven (chanTransfer, have_gdbm, have_lmdb). GATE PASS on consecutive full runs with identical totals (plus a third identical run in a combined -j1 invocation).
  • tclvfs (trunk, vfs 1.4.2): 51 run / 39 passed / 5 skipped / 7 failed, all dispositioned in expected_test_failures_tclvfs.txt (3x tcl9 glob-nomatch semantics through vfs handlers: vfs-4.2 vfsZip-1.6 vfsZip-1.7; 4x network-reaching: vfsFtp-1.1..1.3 vfsUrl-2.1). GATE PASS on two consecutive full runs with identical totals.
  • tklib (trunk 0.9): RECORDED 796 run / 688 passed / 90 skipped / 18 failed (widgetPlus entryPlus/spinboxPlus behaviour drift under Tk 9: entry-1.x, spinbox-1.x). Artifacts: out/testreports/tklib.log + tklib.summary.
  • tcllib (trunk 2.0) WITH tcllibc accelerators engaged: RECORDED 62328 run / 59240 passed / 3034 skipped / 54 failed / 0 errored files, ~45 min wall. Engagement evidence: E tcllibc 2.0 markers (external tcllibc loaded from the suite's out/lib/tcllibc) and per-implementation test variants running (e.g. crc32-critcl-* alongside crc32-tcl-*). Failure classes (record-only, no dispositions required; candidates for later gate promotion work): picoirc 22x (network/socket behaviour), crc16/crc32 9x (both tcl and critcl variants fail identically - implementation-independent), httpd proxy/cgi 4x, fileutil find/multiop family 6x, fhome-1.1.1/2.3.1 (the machine's OS-username != profile-dir trait, same as the core fCmd dispositions), install-2.1/2.2, matrix-16.7/16.8, tar-add, uri-4.1, errorcommand-1.2/1.3, example-1.1/1.4/2.1, profiler-11.0-tkt, zipfile-encode-6.4/6.5. Artifacts: out/testreports/tcllib.log (6.5MB full output) + tcllib.summary.
  • no-totals failure path VERIFIED in both modes (acceptance clause): tclvfs driven with an aborting testarg (-badflag) fails the step (exit 1) in gate AND record mode, summary records result fail / failreason no-totals; clean gate rerun afterwards restored artifacts (third identical-totals PASS).
  • core gate parity under the generalized engine: subset run (suite.tcl test -testargs {-file assocd.test}) GATE PASS, 11 run / 11 skipped (assocd tests are C-test-command constrained - skip in release builds), tclcore.summary written.
  • tk (core-9-0-branch 9.0.2, opt-in step): RECORDED 9865 run / 8127 passed / 1711 skipped / 27 failed / 0 errored files. NATIVE-DIALOG FILES EXCLUDED by default (notfiles {winDialog.test winMsgbox.test} - 91 dialog-popping cases whose message-injection automation does not drive the real OS dialogs under the batch/suite-built shell; the first attempt hung on winDialog and the engine correctly reported no-totals/fail, which is what motivated the -notfiles exclusion). Failures are live-desktop/geometry-sensitive classes expected for a windowed suite run under automation: scrollbar-3.x (12), grid-16/18 (geometry), imgListFormat-3.x, winWm/wm-geometry (window-manager placement), clipboard-4.x, frame, bind-34.2 (1px pointer-warp), safe-1.2, textImage, pack-19.2, image-6.2. Artifacts: out/testreports/tk.log + tk.summary. This is the recorded FIRST CENSUS; promotion to any gated subset is future work (a desktop-stable Tk baseline needs its own characterization, out of this goal's scope).
• The -notfiles exclusion mechanism (added when the Tk census surfaced the blocking-dialog problem): a dedicated engine option + -Dtestnotfiles-<lib> recipe knob for excluding whole test FILES that cannot run in a context, as a Tcl list value the driver receives as -notfile (-testargs cannot carry a multi-file list through the option chain). Default empty for every library except tk. Recorded in the evidence summary (notfiles key).
• All six acceptance clauses satisfied: thread+tclvfs gated with tracked baselines deterministic across consecutive runs; tcllib (tcllibc engaged) and tklib recorded with artifact paths here; tk opt-in step with a recorded first census; per-library policy/testargs/notfiles overridable at invocation (no recipe edits); no-totals fails the step in gate AND record mode (verified); the evidence-summary shape documented above.

Notes

• 2026-07-21: drafted and approved from the library-test discussion (survey + policy tiers above).
• 2026-07-21: activated at user request ("Activate G-107 and implement") - taken ahead of G-103 in the previously confirmed order.