punkshell/goals/archive/G-098-suite-tk-tcllib-critcl.md

G-098 suite_tcl90 full punkshell dependency set: Tk+tklib, tcllib with critcl accelerators, and testsuite gating

Status: achieved 2026-07-20 Scope: src/buildsuites/suite_tcl90/ (recipe extensions, driver steps, testsuite gate), src/buildsuites/_build/ (tk/tklib/tcllib/critcl sources and outputs), bin/tools/ (pinned zig as consumed), src/tests/ (punkshell suites exercised against the suite-built runtime) Goal: suite_tcl90 builds/installs the remaining runtime pieces punkshell testing depends on - Tk (fossil core.tcl-lang.org/tk) loadable by the suite's Tcl 9.0 windows runtime together with a matching tklib (fossil core.tcl-lang.org/tklib; pure-tcl, installed via its own installer.tcl where that does more than a straight copy), and tcllib (fossil core.tcl-lang.org/tcllib) installed via its documented mechanism (installer.tcl / sak.tcl checked for installation behaviour that is not just a straight copy) WITH its critcl-built C accelerators (tcllibc; critcl from github.com/andreas-kupries/critcl driving zig cc as the compiler, bootstrapped by the suite-built tclsh - no MS toolchain) - and the suite's test surface becomes a gate: suite.tcl test parses the all.tcl totals rather than trusting the exit code and fails on failures absent from a recorded expected-failure baseline. Acceptance: from the pinned zig and fetched sources the suite produces a Tk the suite tclsh loads (package require Tk with a create/destroy-window verification recorded in this file) and a matching tklib whose packages resolve from the install (a representative tklib package require recorded); a tcllib install performed via its own installer machinery (installer.tcl/sak.tcl behaviour checked and recorded - module selection, pkgIndex/tm generation, critcl module handling) whose accelerators load (package require tcllibc) with a punkcheck-relevant hashing comparison showing the accelerated path engaged versus pure-tcl; critcl is version-pinned and driven with zig cc; suite.tcl test gates on parsed totals against a recorded expected-failure baseline and the current 72 core-test failures are dispositioned (fixed or baselined with reasons); punkshell's runtests can run against the suite-built runtime with Tk-dependent suites no longer skipped for lack of Tk.

Context

Follow-on from G-096 (tracked zig buildsuite; zig 0.16.0 pin adopted 2026-07-20), per user direction 2026-07-20:

• Tk must be built to run ALL punkshell tests (Tk-dependent suites currently skip).
• A matching tklib may be required for Tk to function properly - believed pure-tcl, so a fossil pull plus possibly its installer.tcl "depending on whether it does anything special" (user wording).
• tcllib is mostly pure-tcl but its critcl-built accelerators make a huge performance difference - without tcllibc, punkcheck hash checking is very slow. tcllib has an installer.tcl and sak.tcl which should be checked for installation behaviour that is not just a straight copy.
• critcl (github.com/andreas-kupries/critcl) may be required for the accelerators; it needs a tclsh (the suite-built shell) and a C compiler (zig cc - keeps the no-MS-toolchain policy; the mechanism is zig-version-agnostic).
• Testsuite state at goal creation: full Tcl core suite on the suite-built shell runs 66350 tests, 72 FAILED (triage pending; native-runtime baseline on this machine is exec-14.3 only), and all.tcl's exit code does not reflect failures - hence the totals-parsing gate requirement.

Notes

• Related goals: G-096 (the buildsuite this extends; its remaining items - piperepl rebase/matrix and punk-getzig per-version fetch - stay with G-096), G-005 (zig build infrastructure direction), G-060 (qemu cross-platform matrix - eventual multi-platform consumer), G-021/G-020 (agent visual verification - a working Tk underpins those).
• The 72-failure triage is tracked HERE (gate clause), not in G-096, to avoid double tracking.

Progress

2026-07-20 increment 1: activated; hermetic clone store; Tk builds and smokes

• Activated by the user. Clone-store design (two user course-checks): fossil clone dbs are CACHES, not build output - they now live in the punkshell-owned machine-level store ~/.punkshell/fossils (user preference: ~/... on all platforms for cross-platform doc simplicity, a deliberate exception to windows convention), independent of the personal ~/.fossils AND outside deletable _build-class dirs (git clean -fdx safe). Missing clones seed READ-ONLY by file copy from -seedfossils (default ~/.fossils; the seed is never written - the copy is pulled current), else network-clone. Override: -repofolder / PUNK_FOSSIL_STORE, the latter registered in punk::config's known env vars ('help env' surfacing; punk::config pending-0.2.0 entry). -refresh now pulls store clones before updating checkouts; clean wipes the stage only. Verified: full seed->open->build->smoke PASS from an empty store (6 clones seeded, ~1.2GB), then store migrated to ~/.punkshell/fossils and re-verified (checkouts re-opened from the new store, PASS, zero re-seeds).
• Sources wired: tk (core-9-0-branch -> stage/tk9; clone was stale at 2023/8.7a6 and pulled current - TK 9.0.2), tklib (trunk, current), tcllib (trunk, current).
• Tk recipe (build_tk/build_tk.zig, 0.16 API): tcl9tk90.dll from stock sources - object inventory from tk/win/makefile.vc TKOBJS+TTK_OBJS (generic/unix-shared/win/ ttk splits corrected against the tree; xlib emulation sources x{colors,draw,gc, image,util}.c were absent from the name-pattern extraction and restored; tkStubLib/ttkStubLib excluded as stub-archive objects), tkMain compiled ansi+UNICODE with the forced -U ansi guard (G-096 finding), CFG_* pkgconfig defines generated at build time, rc from win/rc/tk.rc with explicit include_paths (wish.exe.manifest and tcl headers), system libs incl oleaut32 (tkWinSend OLE). tk script library installs to out/lib/tk9.0; the driver generates tkUuid.h + wish.exe.manifest (TK_WIN_VERSION 9.0.2.902 by the tree's own formula) and writes the pkgIndex.tcl (tk + Tk ifneeded -> ../../bin/tcl9tk90.dll, makefile.vc install convention).
• SMOKE PASS on the suite shell: package require Tk -> 9.0.2; winfo screen 3840x2160 depth 32; label pack + update + destroy clean (real window station exercised, not just a load).

2026-07-20 increment 2: tklib installed via its own installer; tooltip smoke

• Installer behaviour checked (per the user's caveat) before use: NOT a straight copy
  • module exclusion list (unless +excluded), app-script generation, optional nroff/html doc install, and generation of the aggregating top-level pkgIndex.tcl (gen_main_index sources each module's index - required for package resolution since module indexes sit two levels below auto_path). critcl DSL procs are stubbed to no-ops (pure-tcl confirmed). Batch surface: -no-gui -no-wait with -pkgs/-apps/-html/-nroff/-examples toggles and *-path overrides; default paths derive from the RUNNING tclsh.
• Driver step: run installer.tcl with the SUITE-BUILT shell, batch, packages only (-no-gui -no-wait -pkgs -no-apps -no-html -no-nroff -no-examples) - defaults land it at out/lib/tklib0.9 with the generated main pkgIndex. Idempotent (skipped when a tklib* install exists).
• SMOKE PASS: in the suite shell, package require Tk + package require tooltip -> tooltip 2.0.4 (a Tk-dependent tklib package resolving through the aggregate index).

2026-07-20 increment 3: tcllib installed; tcllibc built via critcl 3.3.1 + zig cc; 248x hashing evidence

• tcllib pure-tcl side installs via its own installer (same family as tklib: critcl DSL stubbed, exclusions, aggregating pkgIndex; batch packages-only, suite shell). INCIDENT: the installer's default libdir derives from the running shell's tcl_pkgPath, which included the user's machine-global C:/TCLPKGS via inherited TCLLIBPATH - the first run installed tcllib2.0 THERE. Verified pure additive (all content fresh-dated, no pre-existing tcllib at that path) and removed. Fixes: the driver now UNSETS TCLLIBPATH/TCL_LIBRARY/TK_LIBRARY for all child shells (hermetic; also removes false-positive smoke risk via machine packages), and both tklib/tcllib installs pass explicit -pkg-path (version parsed from each project's support/installation/version.tcl). Smoke: md5 2.0.9, sha1 2.0.5, cmdline 1.5.3 from out/lib/tcllib2.0.
• install-libraries recipe fix surfaced by critcl's package interception: tcl's installed library/pkgIndex.tcl references subdir packages (msgcat/ opt/ cookiejar/ ...) the piecemeal top-level install never provided - phantom ifneeded entries. The recipe now installs the whole library tree to lib/tcl9.0 (stock-install parity) alongside the tm installs.
• tcllibc: critcl git-cloned and pinned at tag 3.3.1 under _build (hermetic git-mirror treatment analogous to the fossil store = noted refinement); tracked critcl_zig.config overrides critcl's gcc defaults with zig cc -target x86_64-windows-gnu compile/link (driver prepends the pinned zig dir to PATH). sak.tcl's windows path wants a tclkitsh+starkit combo, so the driver replicates its underlying invocation directly: critcl main.tcl -config ... -cache ... -force -libdir out/lib -pkg tcllibc + the 21 critcl files parsed from tcllib's own support/installation/version.tcl declarations (critcl_main first; brace-grouped lists handled). RESULT: tcllibc.dll built and placed at out/lib/tcllibc (provide tcllibc 2.0).
• Evidence: package require tcllibc -> 2.0; tcllib md5 auto-engages the critcl accelerator (accel(critcl)=1). Hashing comparison (punkcheck-relevant): sha1 -hex over ~5MB in the suite shell - critcl 12.3ms vs pure-tcl 3.06s (pure measured with tcllibc moved OUTSIDE lib/ - child pkgIndex scanning ignores dir names, so an in-place rename does not hide a package) = ~248x speedup, confirming the user-reported punkcheck hashing pain and its fix.

2026-07-20 increment 4: test gate live (72 -> 11 dispositioned, GATE PASS); thread/registry/vfs runtime gaps closed; runtests on the suite runtime

• suite.tcl test rewritten as a GATE: full output to /tcltest.log, totals PARSED (all.tcl's exit code does not reflect failures), failed test names collected and compared against the tracked expected_test_failures.txt baseline - unexplained failures fail the action listing them; stale baseline entries are noted on full runs. Verified failing correctly (11 listed pre-baseline) and passing after baselining: totals 67281 run / 55835 passed / 11435 skipped / 11 failed, GATE PASS, deterministic across two full runs.
• The 72 initial failures collapsed to 11 via earlier recipe fixes (http tm filename/provide mismatch -> 44; missing library subdirs -> 8 msgcat; library-tree install also cleared clock-vm/winTime/cmdAH). The 11 are dispositioned in the baseline with reasons: 6 fCmd (machine trait: OS user jnoble vs profile dir C:/Users/sleek - 'file home ' assumptions), 2 filename (windows admin-share self-globbing //hostname/c), 2 safe-stock (install-layout-sensitive safe-interp accessPath tokens - candidate deeper look), 1 console-bug-f10d91c2 (upstream regression test credited to the punkshell author; expects the FIXED behaviour via a powershell conhost-kill driver but yields an empty result in batch context - execution-policy interaction suspected; semantics verified independently by G-076's kill procedure; candidate upstream feedback).
• Runtime package gaps closed (all same class - built artifacts without installed package plumbing): thread pkgIndex generated (lib/thread3.0.1 -> bin dll; version derived from the dll name), registry dll install artifact existed but was hooked to nothing (now under install-libraries; registry 1.3.7 + dde 1.4.6 load), tclvfs script package installed with its two configure-products generated (vfs.tcl + pkgIndex.tcl from .in templates, version from AC_INIT; dll alongside) - vfs 1.4.2 + vfs::zip 1.0.4 load.
• punkshell runtests UNDER the suite runtime (TCLLIBPATH cleared): checkbugs 5/5; opunk console backends.test PASS - the Tk-widget console suite runs unskipped on the suite-built Tk; repl consolebackends: testconsole + sshconsole backends PASS, repl_console_backend_tkconsole FAILED (child repl reaches the prompt but repl_process_data errors reading ::codeinterp::console_pending_out - the foreign-console var-junction init in the code interp did not run; DRIVER-DONE timeout-failsafe). The vfs install did not change it. OPEN ITEM: diagnose the TkConsole child's code-interp init under the plain suite runtime (candidates: init-order/timing vs kit environments, twapi absence side-path).

2026-07-20 increment 5: beta kit for interactive trial - punk905_beta.exe wraps the suite runtime

• Suite runtime copied to bin/runtime/win32-x86_64/tclsh905_beta.exe (source _build/suite_tcl90/out/bin/tclsh90szip.exe - static 9.0.5 tclsh + attached zip carrying the tcl library; verified self-contained: info library = //zipfs:/app/tcl_library). bin/runtime is gitignored - local artifact, manual copy.
• New src/vfs/punk9wintk905_beta.vfs (tracked, modelled on punk9wintk903.vfs): root punk_main.tcl fauxlink; bin/ = suite-built tcl9tk90.dll + tcl9thread301.dll (pkgIndexes use the ../../bin convention); lib_tcl9/ = suite-built tk9.0, tklib0.9, tcllib2.0, tcllibc (sibling placement as in the suite out/lib layout), thread3.0.1, vfs1.4.2 (its dll rides inside - vfs.tcl loads from $::vfs::self); plus the two common-trial binaries carried over from the 903 vfs: twapi5.0.2, sqlite3.51.0. Deliberately minimal - tcludp NOT included (add a punk9wintk905_beta.vfs/lib_tcl9 target line to src/runtime/vendorlib_vfs.toml if -teelog workflows are wanted).
• mapvfs.config: tclsh905_beta.exe {punk9wintk905_beta.vfs punk905_beta zip} - the zip wrap mounts the runtime's attached zip and merge_overs it into the target vfs, so the runtime's tcl_library survives (same mechanism as the tclsh902z->punk902z line). Built via 'make.tcl vfs -confirm 0': punk905_beta.exe INSTALLED; smoke test under the kit: Thread/vfs/vfs::zip/sqlite3/twapi/tcllibc/md5(C-accelerated)/tk all load, sqlite + twapi + thread functional pokes pass, punk packagepreference boot machinery active. (punk91.exe deploy skipped this run - locked by a running shell, known stale-on-lock pattern.)
• POLICY (recorded in mapvfs.config too): wrap freshly suite-built runtimes as *_beta kits even BEFORE they pass the punk test suites, so behaviour can be trialled interactively by a human early. Drop the _beta suffix on acceptance.
• FUTURE (make.tcl): 'make.tcl vfs' should support building a single configured kit by name (e.g 'make.tcl vfs punk905_beta') and a listing mode showing the kits configured for building from mapvfs.config. Also planned: move at least tclvfs + thread dlls into the zip attached to the runtime exe itself rather than the .vfs.

2026-07-20 increment 6: tkconsole root-caused and fixed (punk::libunknown epoch-copy unsoundness); full-sweep PARITY with Tcl903

• repl_console_backend_tkconsole failure root cause (NOT a suite recipe defect - a latent punkshell bug the suite runtime's epoch phasing exposed deterministically): copying ::punk::libunknown::epoch VERBATIM into a fresh interp/thread carries the per-epoch per-searchpath 'added' maps that drive zipfs_tclPkgUnknown's can_skip_sourcing short-circuit. Those maps are only sound in the interp that did the scans: packages the donor obtained at an EARLIER epoch or with no unknown call at all (Thread baked into thread::create'd threads) are absent from the donor's CURRENT epoch map, so the recipient skips their pkgIndex sourcing and the require fails where stock tclPkgUnknown would succeed. Trigger chain in tk mode: parent 'package require Tk' bumps the epoch (both runtimes do this) -> codethread pastes the post-bump sparse dict -> code interp inherits it -> vfs/Thread fail -> punk::console absent -> 'registry' require inside punk's auto_execok chain errors the code-interp init BEFORE the G-001 conredirect junction vars are created -> first prompt's pending-collect reads a missing var -> repl reader left inactive -> driver timeout-failsafe. Tcl903 escaped by luck: its code interp got an extra epoch bump to a fresh epoch (no inherited added map -> full first-scan). Suite test/ssh modes escaped by inheriting the pre-bump comprehensive map.
• FIX: new punk::lib::libunknown_epoch_export (PUNKARGS-documented) returns the epoch record with every per-epoch 'added' map emptied - counters, untracked, scripts and index-glob caches preserved; recipients do their own first sourcing pass per searchpath. Applied at all four propagation sites: punk::lib interp_sync_package_paths + snapshot_package_paths, punk::repl init_script %lib_epoch% + the safe-arm direct epoch copy.
• HARDENING: repl_process_data's foreign-console pending-collect now guards for the junction vars' existence (init aborted before junction install => treat as no pending output) so a code-interp init failure can no longer wedge the repl reader.
• Windows firewall popups during punk91/punk905_beta test runs (user datapoint): test socketpairs listened on all interfaces - each newly built exe prompts. Pinned to -myaddr 127.0.0.1 (loopback binds don't prompt) in repl_console_driver.tcl + opunk backends.test socketpair. Residual: tcludp-based PUNK_TEST_UDPTEE teelog binds remain INADDR_ANY (tcludp has no myaddr control).
• VERIFICATION: consolebackends 3/3 on suite runtime (42s->14s, failsafe gone) and 3/3 on Tcl903; FULL runtests sweeps IDENTICAL on both runtimes: Total 1037 / Passed 1021 / Skipped 15 / Failed 1 = exec-14.3 (documented machine baseline), same skip census. Tk-dependent child-process suites run unskipped on the suite runtime. ACCEPTANCE SWEEP MET.
• Noted pre-existing quirk (both runtimes, out of scope): PUNK_TEST_TK=1 opt-in (backends.test Tk-in-testinterp widget test) silently ends the runtests run right after the file header - exit 0, no tally. Candidate small follow-up.
• Deployed punk905_beta.exe still carries the pre-fix repl/lib modules (kit payload built from _vfscommon before this increment) - unaffected for default-console interactive use (collect path is foreign-console only); picks the fix up at the next vfscommonupdate + project cycle.

2026-07-20 increment 7: kit-hosted runtests fixed ('src' package_mode children) + tally line surfaces file-level failures

• User datapoint: 'runtests.tcl -include-paths modules/punk/console/*** -jobs 16' FAILED under punk91/punk902z/punk905_beta while the tally said 'Failed 0' beside a red FAIL. Reproduced: ALL 7 console files failed at file level with ZERO tests run. Cause: child test processes ('kitexe child_test_runner.tcl ...') boot via plain kit script dispatch, which preloads kit-stamped punk packages (punk::console 0.7.2) before the payload's src paths apply - the suites' pinned 'require punk::console 999999.0a1.0-' then version-conflicts and the child exits 1. runtests already self-warned about exactly this ('prefer a native tclsh').
• FIX: runtests detects a PUNK kit host (kit container heuristics AND ::punkboot namespace - the punk_main.tcl boot marker) and launches children with the kit's 'src' package_mode ('kitexe src child_test_runner.tcl ...'), so children boot the project's src dev modules (probe: punk::console 999999.0a1.0 provided, dev require satisfied, argv/argv0 forwarding intact). Non-punk kit containers (e.g the suite's tclsh90szip.exe - zipfs app mount, no punk boot) keep the old warning and get NO extra arg (a plain tclsh would treat 'src' as the script to run). runtests_run_child_process gains a required exemodeargs param (tpool replication keeps working via info args/body).
• TALLY: file-level failures (load/setup errors, child process failures) are not test results, so the human tally could show 'Failed 0' on a FAIL run. The tally line now appends 'Testfiles-failed N' (fail-coloured) whenever files failed: 'Total 0 Passed 0 Skipped 0 Failed 0 Testfiles-failed 1'. RUNTESTS_RESULT already carried files_failed - unchanged.
• VERIFICATION: console suites 71/71 PASS under punk905_beta, punk91, punk902z (all previously 7 files failed / 0 run), native Tcl903 and suite tclsh90s unchanged 71/71 (no modeargs, no notes), tclsh90szip takes the non-punk warning path and passes; tally demo via throwaway error-at-load file shows Testfiles-failed 1.
• (superseded note kept for history below: the clone store subsequently moved from the suite-owned _build location to the machine-level ~/.punkshell/fossils - see increment 1 and the suite README for current truth)
• Fetch conventions follow suite_tcl90, with the hermetic clone store adopted at this goal's activation (user decision 2026-07-20): clones live in the suite-owned _build/suite_tcl90/fossils store (seeded READ-ONLY from ~/.fossils when present, else network-cloned; -repofolder/PUNK_FOSSIL_STORE override; clean keeps the store, clean -all removes it), checkouts under _build with FOSSIL_HOME isolation and --nested. critcl is git, not fossil - clone/pin convention to be settled in the work (git clone under _build with a pinned tag/commit is the expected shape).