jn
/
punkshell
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Browse Source

suite_tcl90: -notfiles exclusion knob; Tk census excludes native-dialog files 

The Tk opt-in census surfaced that the full suite cannot run unattended in
the batch/suite-built shell: winDialog.test (72 cases) and winMsgbox.test (19)
pop real OS dialogs whose message-injection automation does not drive them
here, so each blocks waiting on a human. The first attempt hung there and the
engine correctly reported no-totals/fail.

Adds a dedicated file-exclusion mechanism: tools/test_gate.tcl gains -notfiles
(a Tcl list of test-file globs passed to the driver as one -notfile list
value - a flat -testargs cannot carry a multi-file list through the option
chain, since -notfile takes a single list arg), and build905.zig gains a
per-library -Dtestnotfiles-<lib> knob defaulting to each LibSuite's
default_notfiles. tk defaults to excluding winDialog.test + winMsgbox.test so
its census completes hands-off; -Dtestnotfiles-tk="" re-includes them for a
manual interactive run. The exclusion is recorded in the evidence summary
(notfiles key). Empty for every other library (no behaviour change).

Verified: tk census now completes unattended (9865 run / 8127 passed / 1711
skipped / 27 failed, result pass; log shows winDialog/winMsgbox in the driver
skip list); thread gate still PASS with notfiles {} recorded.

Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
master
Julian Noble 1 week ago
parent
f5f2fe7d83
commit
b5206fa7c4
4 changed files with 46 additions and 12 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 0
      goals/archive/G-107-buildsuite-library-tests.md
  2. 14
      src/buildsuites/suite_tcl90/README.md
  3. 21
      src/buildsuites/suite_tcl90/build905.zig
  4. 23
      src/buildsuites/suite_tcl90/tools/test_gate.tcl

0
goals/G-107-buildsuite-library-tests.md → goals/archive/G-107-buildsuite-library-tests.md
Unescape View File

14
src/buildsuites/suite_tcl90/README.md
Unescape View File

@ -76,13 +76,21 @@ overridable at invocation without recipe edits):
EVERY mode fails the step when the run produces no parseable totals line - EVERY mode fails the step when the run produces no parseable totals line -
infrastructure breakage must never masquerade as recorded results. Per-library infrastructure breakage must never masquerade as recorded results. Per-library
driver args: `-Dtestargs-<lib>="..."` (e.g `-notfile x.test` to disposition a invocation overrides (recorded in the evidence summary):
problem file at invocation level; recorded in the evidence summary). `-Dtestargs-<lib>="..."` (extra tcltest/driver args) and
`-Dtestnotfiles-<lib>="pat ..."` (test-FILE glob patterns excluded from the
run; a dedicated knob because `-notfile` needs a single list value that a flat
`-testargs` cannot carry through the option chain).
`test-libraries` runs thread/tclvfs/tcllib/tklib. The full Tk suite is `test-libraries` runs thread/tclvfs/tcllib/tklib. The full Tk suite is
deliberately OUTSIDE it: **`test-tk` is opt-in** (needs an interactive desktop, deliberately OUTSIDE it: **`test-tk` is opt-in** (needs an interactive desktop,
maps real windows, steals focus, runs tens of minutes) - intended before maps real windows, steals focus, runs tens of minutes) - intended before
PUBLISHING bi-family artifacts, not on every build. PUBLISHING bi-family artifacts, not on every build. Tk excludes the
**native-dialog suites** `winDialog.test` (72 cases) and `winMsgbox.test` (19)
by default (`default_notfiles`): their message-injection automation does not
drive the real OS dialogs under the batch/suite-built shell, so each case
blocks waiting on a human and the census cannot complete unattended. To run
them, `-Dtestnotfiles-tk=""` and dismiss each dialog by hand.
Drivers: thread/tclvfs/tk use their trees' own `tests/all.tcl`. tcllib/tklib Drivers: thread/tclvfs/tk use their trees' own `tests/all.tcl`. tcllib/tklib
are driven via their own `support/devel/all.tcl` (sak's underlying driver, are driven via their own `support/devel/all.tcl` (sak's underlying driver,

21
src/buildsuites/suite_tcl90/build905.zig
Unescape View File

@ -2377,6 +2377,7 @@ pub fn build(b: *std.Build) !void {
driver: []const u8, //driver script, relative to testsdir driver: []const u8, //driver script, relative to testsdir
default_policy: []const u8, default_policy: []const u8,
baseline: []const u8, //tracked baseline filename ("" = none tracked yet) baseline: []const u8, //tracked baseline filename ("" = none tracked yet)
default_notfiles: []const u8, //default driver -notfile exclusions ("" = none)
deps: []const *std.Build.Step, //beyond the top-level install step deps: []const *std.Build.Step, //beyond the top-level install step
}; };
const thread_test_deps = [_]*std.Build.Step{install_libraries}; const thread_test_deps = [_]*std.Build.Step{install_libraries};
@ -2387,22 +2388,29 @@ pub fn build(b: *std.Build) !void {
const lib_suites = [_]LibSuite{ const lib_suites = [_]LibSuite{
//thread's testsuite requires the tree's exact version - the recipe derives //thread's testsuite requires the tree's exact version - the recipe derives
//the installed package version from the same tree (build_tclthread). //the installed package version from the same tree (build_tclthread).
.{ .name = "thread", .testsdir = "../tclthread/tests", .driver = "all.tcl", .default_policy = "gate", .baseline = "expected_test_failures_thread.txt", .deps = &thread_test_deps }, .{ .name = "thread", .testsdir = "../tclthread/tests", .driver = "all.tcl", .default_policy = "gate", .baseline = "expected_test_failures_thread.txt", .default_notfiles = "", .deps = &thread_test_deps },
.{ .name = "tclvfs", .testsdir = "../tclvfs/tests", .driver = "all.tcl", .default_policy = "gate", .baseline = "expected_test_failures_tclvfs.txt", .deps = &tclvfs_test_deps }, .{ .name = "tclvfs", .testsdir = "../tclvfs/tests", .driver = "all.tcl", .default_policy = "gate", .baseline = "expected_test_failures_tclvfs.txt", .default_notfiles = "", .deps = &tclvfs_test_deps },
//tcllib via its own support/devel/all.tcl (sak's underlying driver - //tcllib via its own support/devel/all.tcl (sak's underlying driver -
//standard aggregate totals, per-file child interps). The installed //standard aggregate totals, per-file child interps). The installed
//tcllibc is on the shell's package path, so accelerator-aware suites //tcllibc is on the shell's package path, so accelerator-aware suites
//engage the critcl accelerators ('E tcllibc <v>' markers in the log) - //engage the critcl accelerators ('E tcllibc <v>' markers in the log) -
//the single highest-value run here: zig-built accelerators against a //the single highest-value run here: zig-built accelerators against a
//static tclsh is a combination nobody upstream tests. //static tclsh is a combination nobody upstream tests.
.{ .name = "tcllib", .testsdir = "../tcllib", .driver = "support/devel/all.tcl", .default_policy = "record", .baseline = "", .deps = &tcllib_test_deps }, .{ .name = "tcllib", .testsdir = "../tcllib", .driver = "support/devel/all.tcl", .default_policy = "record", .baseline = "", .default_notfiles = "", .deps = &tcllib_test_deps },
.{ .name = "tklib", .testsdir = "../tklib", .driver = "support/devel/all.tcl", .default_policy = "record", .baseline = "", .deps = &tklib_test_deps }, .{ .name = "tklib", .testsdir = "../tklib", .driver = "support/devel/all.tcl", .default_policy = "record", .baseline = "", .default_notfiles = "", .deps = &tklib_test_deps },
.{ .name = "tk", .testsdir = "../tk9/tests", .driver = "all.tcl", .default_policy = "record", .baseline = "", .deps = &tk_test_deps }, //tk: the native-dialog suites (winDialog 72 cases, winMsgbox 19) are
//excluded by default - their message-injection automation does not
//drive the real OS dialogs under the batch/suite-built shell, so each
//case blocks waiting on a human (the census cannot complete unattended
//with them in). Re-include for a manual interactive run with
//-Dtestnotfiles-tk="" (then a human dismisses each dialog).
.{ .name = "tk", .testsdir = "../tk9/tests", .driver = "all.tcl", .default_policy = "record", .baseline = "", .default_notfiles = "winDialog.test winMsgbox.test", .deps = &tk_test_deps },
}; };
const libtests_step = b.step("test-libraries", "library testsuites under the built shell per policy tier (default: thread+tclvfs gate, tcllib+tklib record; tk excluded - see test-tk)"); const libtests_step = b.step("test-libraries", "library testsuites under the built shell per policy tier (default: thread+tclvfs gate, tcllib+tklib record; tk excluded - see test-tk)");
for (lib_suites) |ls| { for (lib_suites) |ls| {
const policy = b.option([]const u8, b.fmt("testpolicy-{s}", .{ls.name}), b.fmt("{s} testsuite policy: gate|record|skip (default {s})", .{ ls.name, ls.default_policy })) orelse ls.default_policy; const policy = b.option([]const u8, b.fmt("testpolicy-{s}", .{ls.name}), b.fmt("{s} testsuite policy: gate|record|skip (default {s})", .{ ls.name, ls.default_policy })) orelse ls.default_policy;
const lib_testargs = b.option([]const u8, b.fmt("testargs-{s}", .{ls.name}), b.fmt("extra tcltest/driver args for the {s} testsuite", .{ls.name})) orelse ""; const lib_testargs = b.option([]const u8, b.fmt("testargs-{s}", .{ls.name}), b.fmt("extra tcltest/driver args for the {s} testsuite", .{ls.name})) orelse "";
const lib_notfiles = b.option([]const u8, b.fmt("testnotfiles-{s}", .{ls.name}), b.fmt("test-file glob patterns excluded from the {s} run (default \"{s}\")", .{ ls.name, ls.default_notfiles })) orelse ls.default_notfiles;
const is_tk = std.mem.eql(u8, ls.name, "tk"); const is_tk = std.mem.eql(u8, ls.name, "tk");
const step = b.step(b.fmt("test-{s}", .{ls.name}), b.fmt("{s} testsuite under the built shell (policy: {s})", .{ ls.name, policy })); const step = b.step(b.fmt("test-{s}", .{ls.name}), b.fmt("{s} testsuite under the built shell (policy: {s})", .{ ls.name, policy }));
if (!is_tk) libtests_step.dependOn(step); if (!is_tk) libtests_step.dependOn(step);
@ -2436,6 +2444,9 @@ pub fn build(b: *std.Build) !void {
if (lib_testargs.len != 0) { if (lib_testargs.len != 0) {
run.addArgs(&.{ "-testargs", lib_testargs }); run.addArgs(&.{ "-testargs", lib_testargs });
} }
if (lib_notfiles.len != 0) {
run.addArgs(&.{ "-notfiles", lib_notfiles });
}
run.step.dependOn(b.getInstallStep()); run.step.dependOn(b.getInstallStep());
for (ls.deps) |dep| run.step.dependOn(dep); for (ls.deps) |dep| run.step.dependOn(dep);
step.dependOn(&run.step); step.dependOn(&run.step);

23
src/buildsuites/suite_tcl90/tools/test_gate.tcl
Unescape View File

@ -33,12 +33,22 @@
# ?-baseline <file>? (required in gate mode) # ?-baseline <file>? (required in gate mode)
# ?-driver <file>? (default all.tcl; relative paths resolve against -testsdir) # ?-driver <file>? (default all.tcl; relative paths resolve against -testsdir)
# ?-summaryfile <file>? ?-library <name>? ?-testargs {tcltest args}? # ?-summaryfile <file>? ?-library <name>? ?-testargs {tcltest args}?
# ?-notfiles {file-tail-glob ...}?
#
#-notfiles is a Tcl list of test-file glob patterns excluded from the run
#(driver -notfile). It exists because -testargs cannot carry a multi-file
#exclusion through the option chain (-notfile takes one list value; a flat
#-testargs re-split on whitespace would break it). Use it to disposition
#test FILES that cannot run in this context - e.g. Tk's native-dialog suites
#(winDialog.test, winMsgbox.test) whose message-injection automation does not
#drive the real OS dialogs under the batch/suite-built shell, so each of their
#~90 cases blocks waiting on a human. Recorded in the evidence summary.
proc fail {msg} {puts stderr "test_gate ERROR: $msg"; flush stderr; exit 1} proc fail {msg} {puts stderr "test_gate ERROR: $msg"; flush stderr; exit 1}
array set opt {-testsdir {} -baseline {} -logfile {} -testargs {} -mode gate -driver all.tcl -summaryfile {} -library {}} array set opt {-testsdir {} -baseline {} -logfile {} -testargs {} -mode gate -driver all.tcl -summaryfile {} -library {} -notfiles {}}
foreach {k v} $argv { foreach {k v} $argv {
if {![info exists opt($k)]} {fail "unknown option '$k' (expected -testsdir -baseline -logfile -mode -driver -summaryfile -library -testargs)"} if {![info exists opt($k)]} {fail "unknown option '$k' (expected -testsdir -baseline -logfile -mode -driver -summaryfile -library -testargs -notfiles)"}
set opt($k) $v set opt($k) $v
} }
foreach req {-testsdir -logfile} { foreach req {-testsdir -logfile} {
@ -101,14 +111,19 @@ sumset shell $exe
sumset driver $driver sumset driver $driver
sumset testsdir [file normalize $opt(-testsdir)] sumset testsdir [file normalize $opt(-testsdir)]
sumset testargs $opt(-testargs) sumset testargs $opt(-testargs)
sumset notfiles $opt(-notfiles)
sumset baseline [expr {$opt(-baseline) ne "" ? [file normalize $opt(-baseline)] : ""}] sumset baseline [expr {$opt(-baseline) ne "" ? [file normalize $opt(-baseline)] : ""}]
sumset logfile $logfile sumset logfile $logfile
puts "test_gate: $library testsuite under $exe [expr {[llength $opt(-testargs)] ? $opt(-testargs) : "(full suite)"}] (mode $opt(-mode)) -> $logfile" #-notfiles is passed as ONE driver arg (a Tcl list value tcltest reads as a
#list of -notfile patterns) - it must NOT be whitespace-split like -testargs.
set exclargs {}
if {[llength $opt(-notfiles)]} {lappend exclargs -notfile $opt(-notfiles)}
puts "test_gate: $library testsuite under $exe [expr {[llength $opt(-testargs)] ? $opt(-testargs) : "(full suite)"}] (mode $opt(-mode))[expr {[llength $opt(-notfiles)] ? " excluding files: $opt(-notfiles)" : ""}] -> $logfile"
flush stdout flush stdout
set savedpwd [pwd] set savedpwd [pwd]
cd $opt(-testsdir) cd $opt(-testsdir)
catch {exec $exe $driver {*}$opt(-testargs) > $logfile 2>@1} catch {exec $exe $driver {*}$opt(-testargs) {*}$exclargs > $logfile 2>@1}
cd $savedpwd cd $savedpwd
set f [open $logfile r]; set testout [read $f]; close $f set f [open $logfile r]; set testout [read $f]; close $f

Write Preview
Loading…
Cancel
Save