Browse Source

G-107 achieved: buildsuite library test runs (record/gate external testsuites)

All six acceptance clauses satisfied (suite tclsh90s.exe 9.0.5, zig 0.16.0,
machine superbee): thread 143/0 + tclvfs 51/7 GATED with tracked dispositioned
baselines, deterministic across consecutive full runs; tcllib 62328/54 WITH
tcllibc accelerators engaged + tklib 796/18 RECORDED with artifact paths in the
detail file; tk opt-in step with a recorded first census (9865/27, native-
dialog files excluded by default); per-library policy/testargs/notfiles
overridable at invocation; no-totals fails the step in gate AND record mode
(verified); evidence-summary shape documented for the G-103 metadata consumer.

Flips G-107 active -> achieved and archives it (index entry to GOALS-archive.md,
detail file to goals/archive/). Reference sweep: G-103 Notes gain the thread-
payload-refresh forward-pointer (the suite now derives thread naming from the
checkout; the punk905 kit payload still carries the old 3.0.1 copy) and the
evidence-summary-shape pointer; G-108 Notes record that its constrained-test
runs reuse G-107's test_gate generalization + -notfiles knob.

Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
master
Julian Noble 6 days ago
parent
commit
d090f1da7c
  1. 4
      GOALS-archive.md
  2. 4
      GOALS.md
  3. 11
      goals/G-103-runtime-kit-family.md
  4. 9
      goals/G-108-buildsuite-debug-tier.md
  5. 149
      goals/archive/G-107-buildsuite-library-tests.md

4
GOALS-archive.md

@ -140,3 +140,7 @@ Acceptance: from a clean checkout plus the pinned zig (acquired via the consent-
### G-097 [achieved 2026-07-21] bin utility naming sweep: punk- prefix for remaining punkshell-own tools → detail: goals/archive/G-097-bin-utility-naming-sweep.md
Scope: src/scriptapps/ + src/scriptapps/bin/ (bits, runtime, tclargs scriptset renames; selfsign experiment scripts), bin/ (regenerated wrapper twins + renamed experiment scripts, old names retired), src/tests/modules/punk/mix/testsuites/scriptwrap/ + src/tests/shell/testsuites/binscripts/ (roundtrip pins), bin/AGENTS.md + referencing docs, src/project_layouts/vendor/punk/project-0.1/bin + the modpod-templates layout equivalent (layout-shipped utility copies - folded in by user 2026-07-21)
Acceptance: bin/ presents punk-bits, punk-runtime and punk-tclargs (wrapper twins where the polyglot system emits them) and punk- prefixed selfsign experiment scripts, with the old names removed; scriptset sources and _wrap.toml configs are renamed to match; the byte-roundtrip pins (scriptwrap multishell runtime roundtrip, binscripts suites as affected) and every doc/script reference to the old names are updated with the affected test suites passing; no behaviour changes ride along with the renames.
### G-107 [achieved 2026-07-21] Buildsuite library test runs: record/gate external library testsuites → detail: goals/archive/G-107-buildsuite-library-tests.md
Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86)
Acceptance: thread and tclvfs run gated with tracked dispositioned baselines, deterministic across two consecutive full runs; tcllib (with tcllibc accelerators engaged) and tklib run in record mode emitting report artifacts whose paths are recorded in this file; a Tk step exists opt-in with a recorded first census; per-library policy is overridable at invocation without recipe edits; a testsuite that fails to complete (no totals) fails its step in every mode; the evidence-summary shape G-103 metadata consumes is documented here.

4
GOALS.md

@ -362,10 +362,6 @@ Detail: goals/G-105-buildsuite-cross-target.md
Scope: src/modules/punk/console-999999.0a1.0.tm (enableRaw_powershell/disableRaw_powershell + the persistent server lifecycle), scriptlib/utils/pwsh/ (consolemode_server_async.ps1 canonical + experiment-variant reconciliation), ps-script resolution/packaging across launch contexts (argv0-derived pstooldir; kits via the G-089 scriptlib-in-kits interplay), src/make.tcl shell (the verified launch context)
Detail: goals/G-106-powershell-consolemode-fallback.md
### G-107 [proposed] Buildsuite library test runs: record/gate external library testsuites
Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86)
Detail: goals/G-107-buildsuite-library-tests.md
### G-108 [proposed] Debug buildsuite tier: tcl::test-enabled runtimes (-dbg<n>) with recorded constrained-test runs
Scope: src/buildsuites/suite_tcl90/ (build905.zig debug knobs + tcl::test wiring, suite.tcl debug variant steps, tools/test_gate.tcl record runs); kit-family debug members coordinate with G-103
Detail: goals/G-108-buildsuite-debug-tier.md

11
goals/G-103-runtime-kit-family.md

@ -152,3 +152,14 @@ contract the sequencing note promises G-101.
members carry Thread etc in the attached zip and the pr (punk-named) variants
exist - i.e. punkbin publication follows this goal's artifacts, not the current
interim tclsh905/punk905 shapes.
- G-107 (achieved - see goals/archive/G-107-buildsuite-library-tests.md) fixed
the suite's thread version stamping: the dll/install-dir/pkgIndex now derive
from the checkout (tcl9thread<nodots>.dll, thread<version>, both Thread+thread
provided) instead of a hardcoded 3.0.1. The existing punk905 kit payload
(src/vfs/punk9wintk905.vfs/lib_tcl9/thread3.0.1/) still carries the OLD
3.0.1-named copy - refresh it to the derived naming when this goal rebuilds
kit payloads from suite artifacts.
- G-107 (achieved) also defines the per-component test-evidence summaries this
goal's artifact metadata consumes: out/testreports/<lib>.summary line-records
(result/totals/failednames/baseline/... ), shape documented in
goals/archive/G-107-buildsuite-library-tests.md ("Evidence summary shape").

9
goals/G-108-buildsuite-debug-tier.md

@ -64,3 +64,12 @@ TCL_COMPILE_STATS similarly must be absent (not =0) for release performance.
is expected to exceed the release gate's.
- 2026-07-21: drafted and approved as-is from the build-info + constraint-skip
analysis discussion.
- G-107 (achieved - see goals/archive/G-107-buildsuite-library-tests.md)
landed the tools/test_gate.tcl generalization this goal's constrained-test
runs reuse: -mode record, -summaryfile/-library evidence summaries, and the
no-totals-fails-in-every-mode semantics. It also added the -notfiles engine
option + -Dtestnotfiles-<lib> recipe knob (excluding whole test FILES that
cannot run in a context) - directly reusable if a debug-tier run needs to
exclude problem files. The recipe's LibSuite pattern (per-library step +
policy/testargs/notfiles options) is the shape a tclcore-dbg<n> record step
slots into.

149
goals/archive/G-107-buildsuite-library-tests.md

@ -1,6 +1,6 @@
# G-107 buildsuite library test runs: record/gate external library testsuites
Status: proposed
Status: achieved 2026-07-21
Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86)
Goal: the external libraries the buildsuites compile and ship (thread, tclvfs, tcllib+tcllibc, tklib, tk; future bi members as they join) have their own testsuites run under the suite-built shell as named recipe steps with per-library policy - gate (parsed totals vs tracked dispositioned baseline, the core test-gate mechanism), record (complete + emit report artifacts the suite summary points to), or opt-in - so suite artifacts carry test evidence for the exact zig-built combinations nobody upstream tests.
Acceptance: thread and tclvfs run gated with tracked dispositioned baselines, deterministic across two consecutive full runs; tcllib (with tcllibc accelerators engaged) and tklib run in record mode emitting report artifacts whose paths are recorded in this file; a Tk step exists opt-in with a recorded first census; per-library policy is overridable at invocation without recipe edits; a testsuite that fails to complete (no totals) fails its step in every mode; the evidence-summary shape G-103 metadata consumes is documented here.
@ -76,9 +76,150 @@ G-103's note that punkbin publication follows its artifacts). G-099/G-100/
G-101 - the 8.6 arc inherits the mechanism (thread 2.8 / tclvfs baselines on
8.6); same born-onto-the-mechanism fork-timing logic as G-102-before-G-099.
## Evidence summary shape (G-103 consumer contract)
Every test step (the per-library steps AND the core test-gate) writes a
line-record evidence summary to `_build/<suite>/out/testreports/<library>.summary`
(`tclcore.summary` for the core gate) alongside the full run log
`<library>.log`. The format is the suite's plain-Tcl line-record convention
(sources.config precedent): `#` comment lines and blank lines ignored; every
other line is a two-element Tcl list `<key> <value>` where `<value>` may itself
be a list. Parse: `foreach` non-comment line, `lindex $line 0` = key,
`lindex $line 1` = value.
Keys (write order as listed; `unexplained`/`stalebaseline` are gate-mode-only,
totals keys are absent when the run produced no totals - otherwise every key
is present after any attempted run):
- `library` step label (`thread`, `tclvfs`, `tcllib`, `tklib`, `tk`, `tclcore`)
- `mode` `gate` | `record`
- `shell` absolute path of the shell that ran the suite ([info nameofexecutable])
- `driver` absolute path of the driver script run under the shell
- `testsdir` absolute path of the run cwd
- `testargs` extra driver/tcltest args the run was invoked with (list, may be empty)
- `notfiles` test-file glob patterns excluded from the run (driver -notfile; list, may be empty)
- `baseline` absolute path of the dispositioned baseline ("" in record mode)
- `logfile` absolute path of the full run log
- `total` `passed` `skipped` `failed` parsed aggregate totals (absent only when
the run produced no totals - see `failreason`)
- `failednames` distinct failed test names (parse of `==== <name> FAILED`)
- `errored_files` count of tcllib-family driver `@+` file-error blocks
- `errorexit_files` file names from a runAllTests `Test files exiting with errors:` section
- `unexplained` gate mode: failed names not in the baseline (gate-fail set)
- `stalebaseline` gate mode, full runs: baseline entries that did not fail
- `result` `pass` | `fail`
- `failreason` present when result=fail: `no-totals` | `unexplained`
Semantics: `result pass` in record mode asserts only completeness (totals
parsed); consumers wanting "green" must also check `failed`/`failednames`.
A missing summary file after a test step means the step failed before the run
could be attempted (config error) - the step exit code is authoritative.
G-103 artifact metadata should carry (or point at) these summaries per shipped
component; publication tooling can require `result pass` + gate mode for the
gate-tier libraries.
## Progress
- 2026-07-21 implementation landed (suite_tcl90):
- `tools/test_gate.tcl` generalized into the shared engine: `-mode gate|record`,
`-driver` (default `all.tcl` in `-testsdir`; tcllib-family trees driven via
their own `support/devel/all.tcl` - sak's underlying driver, standard
aggregate totals, per-file child interps), `-summaryfile`/`-library`
evidence output, file-error evidence capture (`@+` blocks, runAllTests
error-exit sections). No-totals fails the step in every mode. Core gate
behaviour unchanged (totals + baseline diff + stale notes); the core gate
step now also writes `tclcore.summary`.
- `build905.zig`: per-library steps `test-thread test-tclvfs test-tcllib
test-tklib test-tk` + composite `test-libraries` (tk excluded - opt-in);
per-library `-Dtestpolicy-<lib>=gate|record|skip` and `-Dtestargs-<lib>`
invocation overrides (no recipe edits needed); baselines staged in both
flows (suite.tcl item list + bootstrap recipe_items).
- `suite.tcl`: `test` action takes `-steps` (default `test-gate`) and
`-zigargs` passthrough; passes `-j1` so combined test invocations
serialize (zig runs independent steps concurrently; thread's suite is
timing-sensitive).
- thread version-stamp defect FOUND AND FIXED by the first census attempt:
`build_tclthread.zig` hardcoded 3.0.1, compiling thread trunk 3.0.7
sources with a stale PACKAGE_VERSION - the dll self-reported 3.0.1, the
pkgIndex declared only capital-T `Thread`, and the thread testsuite's
`package require -exact thread 3.0.7` could never load it (nothing
previously loaded-tested thread at all: exactly this goal's gap). Now
version-derived from the checkout's AC_INIT (`build_common.acInitVersion`),
dll `tcl9thread<nodots>.dll`, install dir `thread<version>`, pkgIndex
declares BOTH `Thread` and `thread`, rc resource fields derived. Stale
3.0.1 artifacts removed from the stage prefix. NOTE for G-103/kit work:
the punk905 kit payload (`src/vfs/punk9wintk905.vfs/lib_tcl9/thread3.0.1/`)
still carries the old self-consistent 3.0.1-stamped dll copy - refresh to
3.0.7 naming at the next kit/vfs update.
- 2026-07-21 census + gates (suite-built tclsh90s.exe 9.0.5, zig 0.16.0,
machine superbee; artifacts under
`src/buildsuites/_build/suite_tcl90/out/testreports/`):
- thread (trunk 3.0.7): 143 run / 117 passed / 26 skipped / 0 failed -
EMPTY baseline (`expected_test_failures_thread.txt`); skips constraint-
driven (chanTransfer, have_gdbm, have_lmdb). GATE PASS on consecutive
full runs with identical totals (plus a third identical run in a combined
-j1 invocation).
- tclvfs (trunk, vfs 1.4.2): 51 run / 39 passed / 5 skipped / 7 failed, all
dispositioned in `expected_test_failures_tclvfs.txt` (3x tcl9
glob-nomatch semantics through vfs handlers: vfs-4.2 vfsZip-1.6
vfsZip-1.7; 4x network-reaching: vfsFtp-1.1..1.3 vfsUrl-2.1). GATE PASS
on two consecutive full runs with identical totals.
- tklib (trunk 0.9): RECORDED 796 run / 688 passed / 90 skipped / 18 failed
(widgetPlus entryPlus/spinboxPlus behaviour drift under Tk 9: entry-1.x,
spinbox-1.x). Artifacts: `out/testreports/tklib.log` + `tklib.summary`.
- tcllib (trunk 2.0) WITH tcllibc accelerators engaged: RECORDED 62328 run /
59240 passed / 3034 skipped / 54 failed / 0 errored files, ~45 min wall.
Engagement evidence: `E tcllibc 2.0` markers (external tcllibc loaded from
the suite's out/lib/tcllibc) and per-implementation test variants running
(e.g. crc32-critcl-* alongside crc32-tcl-*). Failure classes (record-only,
no dispositions required; candidates for later gate promotion work):
picoirc 22x (network/socket behaviour), crc16/crc32 9x (both tcl and
critcl variants fail identically - implementation-independent), httpd
proxy/cgi 4x, fileutil find/multiop family 6x, fhome-1.1.1/2.3.1 (the
machine's OS-username != profile-dir trait, same as the core fCmd
dispositions), install-2.1/2.2, matrix-16.7/16.8, tar-add, uri-4.1,
errorcommand-1.2/1.3, example-1.1/1.4/2.1, profiler-11.0-tkt,
zipfile-encode-6.4/6.5. Artifacts: `out/testreports/tcllib.log` (6.5MB
full output) + `tcllib.summary`.
- no-totals failure path VERIFIED in both modes (acceptance clause): tclvfs
driven with an aborting testarg (-badflag) fails the step (exit 1) in gate
AND record mode, summary records `result fail` / `failreason no-totals`;
clean gate rerun afterwards restored artifacts (third identical-totals
PASS).
- core gate parity under the generalized engine: subset run
(`suite.tcl test -testargs {-file assocd.test}`) GATE PASS, 11 run / 11
skipped (assocd tests are C-test-command constrained - skip in release
builds), `tclcore.summary` written.
- tk (core-9-0-branch 9.0.2, opt-in step): RECORDED 9865 run / 8127 passed /
1711 skipped / 27 failed / 0 errored files. NATIVE-DIALOG FILES EXCLUDED by
default (`notfiles {winDialog.test winMsgbox.test}` - 91 dialog-popping
cases whose message-injection automation does not drive the real OS dialogs
under the batch/suite-built shell; the first attempt hung on winDialog and
the engine correctly reported no-totals/fail, which is what motivated the
-notfiles exclusion). Failures are live-desktop/geometry-sensitive classes
expected for a windowed suite run under automation: scrollbar-3.x (12),
grid-16/18 (geometry), imgListFormat-3.x, winWm/wm-geometry (window-manager
placement), clipboard-4.x, frame, bind-34.2 (1px pointer-warp), safe-1.2,
textImage, pack-19.2, image-6.2. Artifacts: `out/testreports/tk.log` +
`tk.summary`. This is the recorded FIRST CENSUS; promotion to any gated
subset is future work (a desktop-stable Tk baseline needs its own
characterization, out of this goal's scope).
- The -notfiles exclusion mechanism (added when the Tk census surfaced the
blocking-dialog problem): a dedicated engine option + `-Dtestnotfiles-<lib>`
recipe knob for excluding whole test FILES that cannot run in a context, as
a Tcl list value the driver receives as `-notfile` (-testargs cannot carry a
multi-file list through the option chain). Default empty for every library
except tk. Recorded in the evidence summary (`notfiles` key).
- All six acceptance clauses satisfied: thread+tclvfs gated with tracked
baselines deterministic across consecutive runs; tcllib (tcllibc engaged)
and tklib recorded with artifact paths here; tk opt-in step with a recorded
first census; per-library policy/testargs/notfiles overridable at invocation
(no recipe edits); no-totals fails the step in gate AND record mode
(verified); the evidence-summary shape documented above.
## Notes
- 2026-07-21: drafted and approved from the library-test discussion (survey +
policy tiers above). Sequencing recommendation (agent, unconfirmed):
activation fits alongside or just after G-103, before the suite_tcl86 fork;
the user has not yet confirmed a slot in the G-103 -> 8.6-arc order.
policy tiers above).
- 2026-07-21: activated at user request ("Activate G-107 and implement") -
taken ahead of G-103 in the previously confirmed order.

Loading…
Cancel
Save