diff --git a/GOALS-archive.md b/GOALS-archive.md index 8f2da52f..fbbdfdb6 100644 --- a/GOALS-archive.md +++ b/GOALS-archive.md @@ -140,3 +140,7 @@ Acceptance: from a clean checkout plus the pinned zig (acquired via the consent- ### G-097 [achieved 2026-07-21] bin utility naming sweep: punk- prefix for remaining punkshell-own tools → detail: goals/archive/G-097-bin-utility-naming-sweep.md Scope: src/scriptapps/ + src/scriptapps/bin/ (bits, runtime, tclargs scriptset renames; selfsign experiment scripts), bin/ (regenerated wrapper twins + renamed experiment scripts, old names retired), src/tests/modules/punk/mix/testsuites/scriptwrap/ + src/tests/shell/testsuites/binscripts/ (roundtrip pins), bin/AGENTS.md + referencing docs, src/project_layouts/vendor/punk/project-0.1/bin + the modpod-templates layout equivalent (layout-shipped utility copies - folded in by user 2026-07-21) Acceptance: bin/ presents punk-bits, punk-runtime and punk-tclargs (wrapper twins where the polyglot system emits them) and punk- prefixed selfsign experiment scripts, with the old names removed; scriptset sources and _wrap.toml configs are renamed to match; the byte-roundtrip pins (scriptwrap multishell runtime roundtrip, binscripts suites as affected) and every doc/script reference to the old names are updated with the affected test suites passing; no behaviour changes ride along with the renames. + +### G-107 [achieved 2026-07-21] Buildsuite library test runs: record/gate external library testsuites → detail: goals/archive/G-107-buildsuite-library-tests.md +Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86) +Acceptance: thread and tclvfs run gated with tracked dispositioned baselines, deterministic across two consecutive full runs; tcllib (with tcllibc accelerators engaged) and tklib run in record mode emitting report artifacts whose paths are recorded in this file; a Tk step exists opt-in with a recorded first census; per-library policy is overridable at invocation without recipe edits; a testsuite that fails to complete (no totals) fails its step in every mode; the evidence-summary shape G-103 metadata consumes is documented here. diff --git a/GOALS.md b/GOALS.md index db12783b..f5d03052 100644 --- a/GOALS.md +++ b/GOALS.md @@ -362,10 +362,6 @@ Detail: goals/G-105-buildsuite-cross-target.md Scope: src/modules/punk/console-999999.0a1.0.tm (enableRaw_powershell/disableRaw_powershell + the persistent server lifecycle), scriptlib/utils/pwsh/ (consolemode_server_async.ps1 canonical + experiment-variant reconciliation), ps-script resolution/packaging across launch contexts (argv0-derived pstooldir; kits via the G-089 scriptlib-in-kits interplay), src/make.tcl shell (the verified launch context) Detail: goals/G-106-powershell-consolemode-fallback.md -### G-107 [proposed] Buildsuite library test runs: record/gate external library testsuites -Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86) -Detail: goals/G-107-buildsuite-library-tests.md - ### G-108 [proposed] Debug buildsuite tier: tcl::test-enabled runtimes (-dbg) with recorded constrained-test runs Scope: src/buildsuites/suite_tcl90/ (build905.zig debug knobs + tcl::test wiring, suite.tcl debug variant steps, tools/test_gate.tcl record runs); kit-family debug members coordinate with G-103 Detail: goals/G-108-buildsuite-debug-tier.md diff --git a/goals/G-103-runtime-kit-family.md b/goals/G-103-runtime-kit-family.md index 6c0b2a67..6c9fd007 100644 --- a/goals/G-103-runtime-kit-family.md +++ b/goals/G-103-runtime-kit-family.md @@ -152,3 +152,14 @@ contract the sequencing note promises G-101. members carry Thread etc in the attached zip and the pr (punk-named) variants exist - i.e. punkbin publication follows this goal's artifacts, not the current interim tclsh905/punk905 shapes. +- G-107 (achieved - see goals/archive/G-107-buildsuite-library-tests.md) fixed + the suite's thread version stamping: the dll/install-dir/pkgIndex now derive + from the checkout (tcl9thread.dll, thread, both Thread+thread + provided) instead of a hardcoded 3.0.1. The existing punk905 kit payload + (src/vfs/punk9wintk905.vfs/lib_tcl9/thread3.0.1/) still carries the OLD + 3.0.1-named copy - refresh it to the derived naming when this goal rebuilds + kit payloads from suite artifacts. +- G-107 (achieved) also defines the per-component test-evidence summaries this + goal's artifact metadata consumes: out/testreports/.summary line-records + (result/totals/failednames/baseline/... ), shape documented in + goals/archive/G-107-buildsuite-library-tests.md ("Evidence summary shape"). diff --git a/goals/G-108-buildsuite-debug-tier.md b/goals/G-108-buildsuite-debug-tier.md index e8982deb..b7ae39c6 100644 --- a/goals/G-108-buildsuite-debug-tier.md +++ b/goals/G-108-buildsuite-debug-tier.md @@ -64,3 +64,12 @@ TCL_COMPILE_STATS similarly must be absent (not =0) for release performance. is expected to exceed the release gate's. - 2026-07-21: drafted and approved as-is from the build-info + constraint-skip analysis discussion. +- G-107 (achieved - see goals/archive/G-107-buildsuite-library-tests.md) + landed the tools/test_gate.tcl generalization this goal's constrained-test + runs reuse: -mode record, -summaryfile/-library evidence summaries, and the + no-totals-fails-in-every-mode semantics. It also added the -notfiles engine + option + -Dtestnotfiles- recipe knob (excluding whole test FILES that + cannot run in a context) - directly reusable if a debug-tier run needs to + exclude problem files. The recipe's LibSuite pattern (per-library step + + policy/testargs/notfiles options) is the shape a tclcore-dbg record step + slots into. diff --git a/goals/archive/G-107-buildsuite-library-tests.md b/goals/archive/G-107-buildsuite-library-tests.md index c7727130..197ea95f 100644 --- a/goals/archive/G-107-buildsuite-library-tests.md +++ b/goals/archive/G-107-buildsuite-library-tests.md @@ -1,6 +1,6 @@ # G-107 buildsuite library test runs: record/gate external library testsuites -Status: proposed +Status: achieved 2026-07-21 Scope: src/buildsuites/suite_tcl90/ (per-library test steps in recipe, tools/ runners, tracked dispositioned baselines), src/buildsuites/_build/ (report outputs via existing globs); pattern inherited by later suites (suite_tcl86) Goal: the external libraries the buildsuites compile and ship (thread, tclvfs, tcllib+tcllibc, tklib, tk; future bi members as they join) have their own testsuites run under the suite-built shell as named recipe steps with per-library policy - gate (parsed totals vs tracked dispositioned baseline, the core test-gate mechanism), record (complete + emit report artifacts the suite summary points to), or opt-in - so suite artifacts carry test evidence for the exact zig-built combinations nobody upstream tests. Acceptance: thread and tclvfs run gated with tracked dispositioned baselines, deterministic across two consecutive full runs; tcllib (with tcllibc accelerators engaged) and tklib run in record mode emitting report artifacts whose paths are recorded in this file; a Tk step exists opt-in with a recorded first census; per-library policy is overridable at invocation without recipe edits; a testsuite that fails to complete (no totals) fails its step in every mode; the evidence-summary shape G-103 metadata consumes is documented here. @@ -76,9 +76,150 @@ G-103's note that punkbin publication follows its artifacts). G-099/G-100/ G-101 - the 8.6 arc inherits the mechanism (thread 2.8 / tclvfs baselines on 8.6); same born-onto-the-mechanism fork-timing logic as G-102-before-G-099. +## Evidence summary shape (G-103 consumer contract) + +Every test step (the per-library steps AND the core test-gate) writes a +line-record evidence summary to `_build//out/testreports/.summary` +(`tclcore.summary` for the core gate) alongside the full run log +`.log`. The format is the suite's plain-Tcl line-record convention +(sources.config precedent): `#` comment lines and blank lines ignored; every +other line is a two-element Tcl list ` ` where `` may itself +be a list. Parse: `foreach` non-comment line, `lindex $line 0` = key, +`lindex $line 1` = value. + +Keys (write order as listed; `unexplained`/`stalebaseline` are gate-mode-only, +totals keys are absent when the run produced no totals - otherwise every key +is present after any attempted run): + +- `library` step label (`thread`, `tclvfs`, `tcllib`, `tklib`, `tk`, `tclcore`) +- `mode` `gate` | `record` +- `shell` absolute path of the shell that ran the suite ([info nameofexecutable]) +- `driver` absolute path of the driver script run under the shell +- `testsdir` absolute path of the run cwd +- `testargs` extra driver/tcltest args the run was invoked with (list, may be empty) +- `notfiles` test-file glob patterns excluded from the run (driver -notfile; list, may be empty) +- `baseline` absolute path of the dispositioned baseline ("" in record mode) +- `logfile` absolute path of the full run log +- `total` `passed` `skipped` `failed` parsed aggregate totals (absent only when + the run produced no totals - see `failreason`) +- `failednames` distinct failed test names (parse of `==== FAILED`) +- `errored_files` count of tcllib-family driver `@+` file-error blocks +- `errorexit_files` file names from a runAllTests `Test files exiting with errors:` section +- `unexplained` gate mode: failed names not in the baseline (gate-fail set) +- `stalebaseline` gate mode, full runs: baseline entries that did not fail +- `result` `pass` | `fail` +- `failreason` present when result=fail: `no-totals` | `unexplained` + +Semantics: `result pass` in record mode asserts only completeness (totals +parsed); consumers wanting "green" must also check `failed`/`failednames`. +A missing summary file after a test step means the step failed before the run +could be attempted (config error) - the step exit code is authoritative. +G-103 artifact metadata should carry (or point at) these summaries per shipped +component; publication tooling can require `result pass` + gate mode for the +gate-tier libraries. + +## Progress + +- 2026-07-21 implementation landed (suite_tcl90): + - `tools/test_gate.tcl` generalized into the shared engine: `-mode gate|record`, + `-driver` (default `all.tcl` in `-testsdir`; tcllib-family trees driven via + their own `support/devel/all.tcl` - sak's underlying driver, standard + aggregate totals, per-file child interps), `-summaryfile`/`-library` + evidence output, file-error evidence capture (`@+` blocks, runAllTests + error-exit sections). No-totals fails the step in every mode. Core gate + behaviour unchanged (totals + baseline diff + stale notes); the core gate + step now also writes `tclcore.summary`. + - `build905.zig`: per-library steps `test-thread test-tclvfs test-tcllib + test-tklib test-tk` + composite `test-libraries` (tk excluded - opt-in); + per-library `-Dtestpolicy-=gate|record|skip` and `-Dtestargs-` + invocation overrides (no recipe edits needed); baselines staged in both + flows (suite.tcl item list + bootstrap recipe_items). + - `suite.tcl`: `test` action takes `-steps` (default `test-gate`) and + `-zigargs` passthrough; passes `-j1` so combined test invocations + serialize (zig runs independent steps concurrently; thread's suite is + timing-sensitive). + - thread version-stamp defect FOUND AND FIXED by the first census attempt: + `build_tclthread.zig` hardcoded 3.0.1, compiling thread trunk 3.0.7 + sources with a stale PACKAGE_VERSION - the dll self-reported 3.0.1, the + pkgIndex declared only capital-T `Thread`, and the thread testsuite's + `package require -exact thread 3.0.7` could never load it (nothing + previously loaded-tested thread at all: exactly this goal's gap). Now + version-derived from the checkout's AC_INIT (`build_common.acInitVersion`), + dll `tcl9thread.dll`, install dir `thread`, pkgIndex + declares BOTH `Thread` and `thread`, rc resource fields derived. Stale + 3.0.1 artifacts removed from the stage prefix. NOTE for G-103/kit work: + the punk905 kit payload (`src/vfs/punk9wintk905.vfs/lib_tcl9/thread3.0.1/`) + still carries the old self-consistent 3.0.1-stamped dll copy - refresh to + 3.0.7 naming at the next kit/vfs update. +- 2026-07-21 census + gates (suite-built tclsh90s.exe 9.0.5, zig 0.16.0, + machine superbee; artifacts under + `src/buildsuites/_build/suite_tcl90/out/testreports/`): + - thread (trunk 3.0.7): 143 run / 117 passed / 26 skipped / 0 failed - + EMPTY baseline (`expected_test_failures_thread.txt`); skips constraint- + driven (chanTransfer, have_gdbm, have_lmdb). GATE PASS on consecutive + full runs with identical totals (plus a third identical run in a combined + -j1 invocation). + - tclvfs (trunk, vfs 1.4.2): 51 run / 39 passed / 5 skipped / 7 failed, all + dispositioned in `expected_test_failures_tclvfs.txt` (3x tcl9 + glob-nomatch semantics through vfs handlers: vfs-4.2 vfsZip-1.6 + vfsZip-1.7; 4x network-reaching: vfsFtp-1.1..1.3 vfsUrl-2.1). GATE PASS + on two consecutive full runs with identical totals. + - tklib (trunk 0.9): RECORDED 796 run / 688 passed / 90 skipped / 18 failed + (widgetPlus entryPlus/spinboxPlus behaviour drift under Tk 9: entry-1.x, + spinbox-1.x). Artifacts: `out/testreports/tklib.log` + `tklib.summary`. + - tcllib (trunk 2.0) WITH tcllibc accelerators engaged: RECORDED 62328 run / + 59240 passed / 3034 skipped / 54 failed / 0 errored files, ~45 min wall. + Engagement evidence: `E tcllibc 2.0` markers (external tcllibc loaded from + the suite's out/lib/tcllibc) and per-implementation test variants running + (e.g. crc32-critcl-* alongside crc32-tcl-*). Failure classes (record-only, + no dispositions required; candidates for later gate promotion work): + picoirc 22x (network/socket behaviour), crc16/crc32 9x (both tcl and + critcl variants fail identically - implementation-independent), httpd + proxy/cgi 4x, fileutil find/multiop family 6x, fhome-1.1.1/2.3.1 (the + machine's OS-username != profile-dir trait, same as the core fCmd + dispositions), install-2.1/2.2, matrix-16.7/16.8, tar-add, uri-4.1, + errorcommand-1.2/1.3, example-1.1/1.4/2.1, profiler-11.0-tkt, + zipfile-encode-6.4/6.5. Artifacts: `out/testreports/tcllib.log` (6.5MB + full output) + `tcllib.summary`. + - no-totals failure path VERIFIED in both modes (acceptance clause): tclvfs + driven with an aborting testarg (-badflag) fails the step (exit 1) in gate + AND record mode, summary records `result fail` / `failreason no-totals`; + clean gate rerun afterwards restored artifacts (third identical-totals + PASS). + - core gate parity under the generalized engine: subset run + (`suite.tcl test -testargs {-file assocd.test}`) GATE PASS, 11 run / 11 + skipped (assocd tests are C-test-command constrained - skip in release + builds), `tclcore.summary` written. + - tk (core-9-0-branch 9.0.2, opt-in step): RECORDED 9865 run / 8127 passed / + 1711 skipped / 27 failed / 0 errored files. NATIVE-DIALOG FILES EXCLUDED by + default (`notfiles {winDialog.test winMsgbox.test}` - 91 dialog-popping + cases whose message-injection automation does not drive the real OS dialogs + under the batch/suite-built shell; the first attempt hung on winDialog and + the engine correctly reported no-totals/fail, which is what motivated the + -notfiles exclusion). Failures are live-desktop/geometry-sensitive classes + expected for a windowed suite run under automation: scrollbar-3.x (12), + grid-16/18 (geometry), imgListFormat-3.x, winWm/wm-geometry (window-manager + placement), clipboard-4.x, frame, bind-34.2 (1px pointer-warp), safe-1.2, + textImage, pack-19.2, image-6.2. Artifacts: `out/testreports/tk.log` + + `tk.summary`. This is the recorded FIRST CENSUS; promotion to any gated + subset is future work (a desktop-stable Tk baseline needs its own + characterization, out of this goal's scope). +- The -notfiles exclusion mechanism (added when the Tk census surfaced the + blocking-dialog problem): a dedicated engine option + `-Dtestnotfiles-` + recipe knob for excluding whole test FILES that cannot run in a context, as + a Tcl list value the driver receives as `-notfile` (-testargs cannot carry a + multi-file list through the option chain). Default empty for every library + except tk. Recorded in the evidence summary (`notfiles` key). +- All six acceptance clauses satisfied: thread+tclvfs gated with tracked + baselines deterministic across consecutive runs; tcllib (tcllibc engaged) + and tklib recorded with artifact paths here; tk opt-in step with a recorded + first census; per-library policy/testargs/notfiles overridable at invocation + (no recipe edits); no-totals fails the step in gate AND record mode + (verified); the evidence-summary shape documented above. + ## Notes - 2026-07-21: drafted and approved from the library-test discussion (survey + - policy tiers above). Sequencing recommendation (agent, unconfirmed): - activation fits alongside or just after G-103, before the suite_tcl86 fork; - the user has not yet confirmed a slot in the G-103 -> 8.6-arc order. + policy tiers above). +- 2026-07-21: activated at user request ("Activate G-107 and implement") - + taken ahead of G-103 in the previously confirmed order.