Browse Source
Both user-approved 2026-07-18 from the runx -teelog / runtests watch-mode work (mastera65f9e5b) and the -singleproc 0 completion (218fbb1c). G-090 records the probe evidence deliberately not addressed in that work: float-placement starvation of a tee under the float-locked capture junction, insert_transform re-instantiation churn, parked-worker adoption never re-applying settings (silent cross-target log misrouting), the per-file teehandle worker-thread leak in unwatched singleproc runs, and the tcludp Tcl 9 receive-fileevent first-datagram-only behaviour. G-091 is the phase-2 parallel test plan over the phase-1 child spawn: tpool -jobs N, per-child tmpdirs, serial group for console-sensitive suites, parity + >=2.5x acceptance (floor ~90s, multishell.test bound). Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
3 changed files with 123 additions and 0 deletions
@ -0,0 +1,65 @@ |
|||||||
|
# G-090 shellfilter/shellthread stack and log-worker lifecycle audit |
||||||
|
|
||||||
|
Status: proposed |
||||||
|
Scope: src/modules/shellfilter-999999.0a1.0.tm (stack placement, insert_transform, junctions), src/modules/shellthread-999999.0a1.0.tm (worker park/adoption/termination), src/tests/modules/ (new characterization suites), README.md ('very unripe parts' note) |
||||||
|
Goal: the shellfilter transform-stack placement logic (float/sink/junction interactions, insert/remove re-instantiation) and the shellthread log-worker lifecycle (park/adoption/settings application, termination responsibility) behave by documented, test-pinned rules with no silent data loss or target misrouting, and the tcludp Tcl 9 receive-event limitation has a recorded resolution or workaround. |
||||||
|
Acceptance: characterization suites under src/tests/modules/ pin the probed hazard cases (a tee transform combined with a float-locked capture junction observes the stream in a supported configuration regardless of add order; stack add/remove churn does not silently drop a transform's log output or rebind its worker to stale targets; a new tag adopting a parked worker either receives its requested targets or errors - the 2026-07-18 misroute scenario writes to the correct file); the parked-worker termination contract (who runs shutdown_free_threads, when) and the tcludp receive caveat are documented in the owning module docs/AGENTS chain; goals_lint and the full test suite stay green. |
||||||
|
|
||||||
|
## Context |
||||||
|
|
||||||
|
The shellfilter stack system is production plumbing (e.g ansible-on-freebsd runs producing an |
||||||
|
ANSI-faithful log, an ansistripped log, and streaming console output; punkshell app logging to |
||||||
|
local UDP syslog targets), but README.md's 'very unripe parts' note ("api is clumsy") is |
||||||
|
largely about the float-placement logic: never audited for sanity/gotchas, not fully tested, |
||||||
|
yet used in working applications (user statement 2026-07-18). |
||||||
|
|
||||||
|
Probe evidence recorded 2026-07-18 during the runx -teelog work (session probes, scratch only - |
||||||
|
none of this is currently pinned by committed tests): |
||||||
|
|
||||||
|
1. Float placement: a float added while floats already exist is inserted BELOW the existing |
||||||
|
floats. Combined with runx's float-locked capture junction (a junction blocks flow to |
||||||
|
everything below it), a tee_to_log float is starved whichever order it is added in: |
||||||
|
added second it lands under the junction directly; added first, the junction's insertion |
||||||
|
churn re-instantiates it (finding 2) and it ends up non-functional. |
||||||
|
2. insert_transform re-instantiation churn: inserting/removing stack members pops and |
||||||
|
re-pushes the transforms above the insertion point as NEW transform objects. For |
||||||
|
tee_to_log this finalizes the old instance (closing/parking its log worker) and |
||||||
|
constructs a new one mid-flight, which re-opens the tag and hits the adoption paths of |
||||||
|
finding 3. |
||||||
|
3. shellthread parked-worker adoption never re-applies settings: both the placeholder-share |
||||||
|
branch and the free_threads reuse branch of shellthread::manager::new_worker return the |
||||||
|
existing worker thread without sending it the new settings, and the park-time reset |
||||||
|
(unsubscribe) resets only the worker's settings dict - NOT the operative |
||||||
|
sysloghost_port/logfile variables its log proc actually reads. Probe result: after |
||||||
|
closing tag 'regrow' (targeting file A), opening tag 'tagB' (targeting file B) adopted |
||||||
|
the parked worker and tagB's messages were written to file A - silent cross-target |
||||||
|
misrouting. The code's own comment asks "review - where/when do we override the default |
||||||
|
settings?". |
||||||
|
4. Unwatched singleproc test runs leak two teehandle worker threads (punkshout/punksherr) |
||||||
|
per test file: each per-file testinterp's shellrun load spawns them and interp deletion |
||||||
|
orphans them (manager state dies with the interp; the threads persist). Benign in |
||||||
|
practice (process exits at run end; an unconditional per-file terminate was measured at |
||||||
|
+59s per full suite run and deliberately not adopted) - runtests terminates them only in |
||||||
|
watch mode. See src/tests/AGENTS.md and commit a65f9e5b. |
||||||
|
5. tcludp receive events under Tcl 9.0.3/Windows: a udp_open listener's readable fileevent |
||||||
|
delivered only the FIRST datagram (canonical one-read-per-event pattern, blocking and |
||||||
|
non-blocking both tried); the send side is unaffected (a .NET UdpClient listener received |
||||||
|
all datagrams from the same tcludp sender). Bites any tcludp-based viewer (e.g the |
||||||
|
PUNK_TEST_UDPTEE tk watch app) - workaround until resolved: timer-driven non-blocking |
||||||
|
reads or a non-tcludp receiver. |
||||||
|
|
||||||
|
## Notes |
||||||
|
|
||||||
|
- The shipped workaround that motivated the probes: shellrun 0.2.0 runx -teelog implements |
||||||
|
its live tee as write traces on the capture variables specifically to avoid findings 1-3 |
||||||
|
(commit a65f9e5b); if this audit makes stacked tees safe alongside junctions, -teelog |
||||||
|
could optionally move back to a transform-based tee. |
||||||
|
- The exit-wedge history (tcludp 1.0.12 UDP_ExitProc, Tcl 9 console+udp worker wedge on |
||||||
|
exit/quit, async unsubscribe hardening; G-036, achieved - see |
||||||
|
goals/archive/G-036-tcl9-udp-console-worker-wedge.md) is adjacent prior work in the same |
||||||
|
lifecycle territory - see shellthread's unsubscribe/shutdown_free_threads comments. |
||||||
|
- shellthread::manager::close_worker is noted in-code as no longer called since the |
||||||
|
unsubscribe mechanism landed, with a free-thread cap TODO - part of the same lifecycle |
||||||
|
contract question. |
||||||
|
- Related: PUNK_TEST_UDPTEE watch mode contracts in src/tests/AGENTS.md; user registered the |
||||||
|
env var in punk::config's known-env-vars table 2026-07-18. |
||||||
@ -0,0 +1,50 @@ |
|||||||
|
# G-091 runtests multiprocess parallelism (-jobs N) |
||||||
|
|
||||||
|
Status: proposed |
||||||
|
Scope: src/tests/runtests.tcl, src/tests/testsupport/, src/tests/AGENTS.md |
||||||
|
Goal: src/tests/runtests.tcl runs multi-process mode's per-file child processes through a -jobs N worker pool (tpool; per-child tmpdirs and output files; longest-first scheduling; a serial group for console-sensitive suites; per-file report sections emitted in discovery order) so the full source-tree suite completes in a fraction of its sequential wall time with unchanged results. |
||||||
|
Acceptance: -jobs 8 produces aggregate results identical to -jobs 1 (scriptlib/developer/runtests_parity.tcl reports PARITY ok) across repeated full-suite runs on a native tclsh 9.0.x, reduces full-suite wall time by at least 2.5x versus the sequential baseline on the reference machine, leaves the repo clean after runs (git status unchanged), and src/tests/AGENTS.md documents the -jobs flag and the serial-group policy. |
||||||
|
|
||||||
|
## Context |
||||||
|
|
||||||
|
Groundwork shipped 2026-07-18 (commit 218fbb1c): -singleproc 0 multi-process mode with a |
||||||
|
per-file child bootstrap (testsupport/child_test_runner.tcl + generated environment payload), |
||||||
|
plain-exec file-captured child spawn designed for pool reuse, honest nonzero-exit |
||||||
|
classification, and the parity harness (scriptlib/developer/runtests_parity.tcl). Full-suite |
||||||
|
parity between singleproc and multiproc modes verified same day (87 files / 989 tests; |
||||||
|
exec-14.3 the sole expected failure on native tclsh 9.0.3). |
||||||
|
|
||||||
|
Why processes are the parallel unit: test files mutate process-global state (cd in |
||||||
|
multishell/dtplite/path suites, ::env save/mutate/restore in console suites, thread creation |
||||||
|
and tsv use, shellrun/shellfilter global channel transforms) - safe across processes, unsafe |
||||||
|
within one. Suites are already well-behaved across processes (FOSSIL_HOME isolation in |
||||||
|
fossilmove.test, port-0 ephemeral sockets, tmpdir-scoped fixtures). |
||||||
|
|
||||||
|
Measured shape (native tclsh90, piped context, 2026-07-18): ~350s sequential wall; slowest |
||||||
|
files multishell.test ~85s, argparsingtest ~27s, dtplite ~25s, scriptexec ~10s. Parallel |
||||||
|
floor is the slowest single file (~90s incl. overhead) -> expect roughly 3-3.5x at 8 jobs |
||||||
|
with diminishing returns beyond. |
||||||
|
|
||||||
|
## Notes |
||||||
|
|
||||||
|
- Per-child tmpdir subdirs are mandatory under parallelism (shared tcltest -tmpdir would |
||||||
|
collide on same-named makeFile helpers across concurrent files); phase-1's shared default |
||||||
|
matches singleproc and must become per-child in pool mode. Same for the per-child |
||||||
|
stdout/stderr capture file names. |
||||||
|
- Serial-group default candidates (relaxable after parity evidence under a real console): |
||||||
|
modules/punk/console/**, modules/opunk/console/**, modules/punk/repl/**, shell/**. In |
||||||
|
piped contexts these mostly skip anyway. |
||||||
|
- Kit-executable caveat carries over from phase 1: children of a kit exe boot with |
||||||
|
kit-stamped punk packages preloaded (runner warns; native tclsh preferred). |
||||||
|
- Candidate independent win (either order): make the 17 preload-dependent files' punk::* |
||||||
|
package requires explicit so children can drop the shellrun preload + runx warmup and go |
||||||
|
leaner/faster - the parity tool gates the change. See child_test_runner.tcl comments for |
||||||
|
the two-stage preload detail (shellrun require + runx execution pulling punk::lib). |
||||||
|
- Raising the ceiling beyond the multishell bound means splitting multishell.test (two 30s+ |
||||||
|
tests) and dtplite.test - suite restructure, optional, floor then ~40s (slowest single |
||||||
|
test ~32s). |
||||||
|
- PUNK_TEST_UDPTEE watch mode (commit a65f9e5b): under a pool, the parent's per-file relay |
||||||
|
and lifecycle events still work; the cooked log format's source column plus file-start/ |
||||||
|
file-end events let one viewer demux interleaved children. |
||||||
|
- Timing noise: -slowest per-test microseconds get noisier under CPU oversubscription; |
||||||
|
document rather than fix. |
||||||
Loading…
Reference in new issue