Browse Source
Both user-approved 2026-07-18 from the runx -teelog / runtests watch-mode work (mastera65f9e5b) and the -singleproc 0 completion (218fbb1c). G-090 records the probe evidence deliberately not addressed in that work: float-placement starvation of a tee under the float-locked capture junction, insert_transform re-instantiation churn, parked-worker adoption never re-applying settings (silent cross-target log misrouting), the per-file teehandle worker-thread leak in unwatched singleproc runs, and the tcludp Tcl 9 receive-fileevent first-datagram-only behaviour. G-091 is the phase-2 parallel test plan over the phase-1 child spawn: tpool -jobs N, per-child tmpdirs, serial group for console-sensitive suites, parity + >=2.5x acceptance (floor ~90s, multishell.test bound). Assisted-by: harness=claude; primary-model=claude-fable-5; api-location=anthropic.com
3 changed files with 123 additions and 0 deletions
@ -0,0 +1,65 @@
|
||||
# G-090 shellfilter/shellthread stack and log-worker lifecycle audit |
||||
|
||||
Status: proposed |
||||
Scope: src/modules/shellfilter-999999.0a1.0.tm (stack placement, insert_transform, junctions), src/modules/shellthread-999999.0a1.0.tm (worker park/adoption/termination), src/tests/modules/ (new characterization suites), README.md ('very unripe parts' note) |
||||
Goal: the shellfilter transform-stack placement logic (float/sink/junction interactions, insert/remove re-instantiation) and the shellthread log-worker lifecycle (park/adoption/settings application, termination responsibility) behave by documented, test-pinned rules with no silent data loss or target misrouting, and the tcludp Tcl 9 receive-event limitation has a recorded resolution or workaround. |
||||
Acceptance: characterization suites under src/tests/modules/ pin the probed hazard cases (a tee transform combined with a float-locked capture junction observes the stream in a supported configuration regardless of add order; stack add/remove churn does not silently drop a transform's log output or rebind its worker to stale targets; a new tag adopting a parked worker either receives its requested targets or errors - the 2026-07-18 misroute scenario writes to the correct file); the parked-worker termination contract (who runs shutdown_free_threads, when) and the tcludp receive caveat are documented in the owning module docs/AGENTS chain; goals_lint and the full test suite stay green. |
||||
|
||||
## Context |
||||
|
||||
The shellfilter stack system is production plumbing (e.g ansible-on-freebsd runs producing an |
||||
ANSI-faithful log, an ansistripped log, and streaming console output; punkshell app logging to |
||||
local UDP syslog targets), but README.md's 'very unripe parts' note ("api is clumsy") is |
||||
largely about the float-placement logic: never audited for sanity/gotchas, not fully tested, |
||||
yet used in working applications (user statement 2026-07-18). |
||||
|
||||
Probe evidence recorded 2026-07-18 during the runx -teelog work (session probes, scratch only - |
||||
none of this is currently pinned by committed tests): |
||||
|
||||
1. Float placement: a float added while floats already exist is inserted BELOW the existing |
||||
floats. Combined with runx's float-locked capture junction (a junction blocks flow to |
||||
everything below it), a tee_to_log float is starved whichever order it is added in: |
||||
added second it lands under the junction directly; added first, the junction's insertion |
||||
churn re-instantiates it (finding 2) and it ends up non-functional. |
||||
2. insert_transform re-instantiation churn: inserting/removing stack members pops and |
||||
re-pushes the transforms above the insertion point as NEW transform objects. For |
||||
tee_to_log this finalizes the old instance (closing/parking its log worker) and |
||||
constructs a new one mid-flight, which re-opens the tag and hits the adoption paths of |
||||
finding 3. |
||||
3. shellthread parked-worker adoption never re-applies settings: both the placeholder-share |
||||
branch and the free_threads reuse branch of shellthread::manager::new_worker return the |
||||
existing worker thread without sending it the new settings, and the park-time reset |
||||
(unsubscribe) resets only the worker's settings dict - NOT the operative |
||||
sysloghost_port/logfile variables its log proc actually reads. Probe result: after |
||||
closing tag 'regrow' (targeting file A), opening tag 'tagB' (targeting file B) adopted |
||||
the parked worker and tagB's messages were written to file A - silent cross-target |
||||
misrouting. The code's own comment asks "review - where/when do we override the default |
||||
settings?". |
||||
4. Unwatched singleproc test runs leak two teehandle worker threads (punkshout/punksherr) |
||||
per test file: each per-file testinterp's shellrun load spawns them and interp deletion |
||||
orphans them (manager state dies with the interp; the threads persist). Benign in |
||||
practice (process exits at run end; an unconditional per-file terminate was measured at |
||||
+59s per full suite run and deliberately not adopted) - runtests terminates them only in |
||||
watch mode. See src/tests/AGENTS.md and commit a65f9e5b. |
||||
5. tcludp receive events under Tcl 9.0.3/Windows: a udp_open listener's readable fileevent |
||||
delivered only the FIRST datagram (canonical one-read-per-event pattern, blocking and |
||||
non-blocking both tried); the send side is unaffected (a .NET UdpClient listener received |
||||
all datagrams from the same tcludp sender). Bites any tcludp-based viewer (e.g the |
||||
PUNK_TEST_UDPTEE tk watch app) - workaround until resolved: timer-driven non-blocking |
||||
reads or a non-tcludp receiver. |
||||
|
||||
## Notes |
||||
|
||||
- The shipped workaround that motivated the probes: shellrun 0.2.0 runx -teelog implements |
||||
its live tee as write traces on the capture variables specifically to avoid findings 1-3 |
||||
(commit a65f9e5b); if this audit makes stacked tees safe alongside junctions, -teelog |
||||
could optionally move back to a transform-based tee. |
||||
- The exit-wedge history (tcludp 1.0.12 UDP_ExitProc, Tcl 9 console+udp worker wedge on |
||||
exit/quit, async unsubscribe hardening; G-036, achieved - see |
||||
goals/archive/G-036-tcl9-udp-console-worker-wedge.md) is adjacent prior work in the same |
||||
lifecycle territory - see shellthread's unsubscribe/shutdown_free_threads comments. |
||||
- shellthread::manager::close_worker is noted in-code as no longer called since the |
||||
unsubscribe mechanism landed, with a free-thread cap TODO - part of the same lifecycle |
||||
contract question. |
||||
- Related: PUNK_TEST_UDPTEE watch mode contracts in src/tests/AGENTS.md; user registered the |
||||
env var in punk::config's known-env-vars table 2026-07-18. |
||||
@ -0,0 +1,50 @@
|
||||
# G-091 runtests multiprocess parallelism (-jobs N) |
||||
|
||||
Status: proposed |
||||
Scope: src/tests/runtests.tcl, src/tests/testsupport/, src/tests/AGENTS.md |
||||
Goal: src/tests/runtests.tcl runs multi-process mode's per-file child processes through a -jobs N worker pool (tpool; per-child tmpdirs and output files; longest-first scheduling; a serial group for console-sensitive suites; per-file report sections emitted in discovery order) so the full source-tree suite completes in a fraction of its sequential wall time with unchanged results. |
||||
Acceptance: -jobs 8 produces aggregate results identical to -jobs 1 (scriptlib/developer/runtests_parity.tcl reports PARITY ok) across repeated full-suite runs on a native tclsh 9.0.x, reduces full-suite wall time by at least 2.5x versus the sequential baseline on the reference machine, leaves the repo clean after runs (git status unchanged), and src/tests/AGENTS.md documents the -jobs flag and the serial-group policy. |
||||
|
||||
## Context |
||||
|
||||
Groundwork shipped 2026-07-18 (commit 218fbb1c): -singleproc 0 multi-process mode with a |
||||
per-file child bootstrap (testsupport/child_test_runner.tcl + generated environment payload), |
||||
plain-exec file-captured child spawn designed for pool reuse, honest nonzero-exit |
||||
classification, and the parity harness (scriptlib/developer/runtests_parity.tcl). Full-suite |
||||
parity between singleproc and multiproc modes verified same day (87 files / 989 tests; |
||||
exec-14.3 the sole expected failure on native tclsh 9.0.3). |
||||
|
||||
Why processes are the parallel unit: test files mutate process-global state (cd in |
||||
multishell/dtplite/path suites, ::env save/mutate/restore in console suites, thread creation |
||||
and tsv use, shellrun/shellfilter global channel transforms) - safe across processes, unsafe |
||||
within one. Suites are already well-behaved across processes (FOSSIL_HOME isolation in |
||||
fossilmove.test, port-0 ephemeral sockets, tmpdir-scoped fixtures). |
||||
|
||||
Measured shape (native tclsh90, piped context, 2026-07-18): ~350s sequential wall; slowest |
||||
files multishell.test ~85s, argparsingtest ~27s, dtplite ~25s, scriptexec ~10s. Parallel |
||||
floor is the slowest single file (~90s incl. overhead) -> expect roughly 3-3.5x at 8 jobs |
||||
with diminishing returns beyond. |
||||
|
||||
## Notes |
||||
|
||||
- Per-child tmpdir subdirs are mandatory under parallelism (shared tcltest -tmpdir would |
||||
collide on same-named makeFile helpers across concurrent files); phase-1's shared default |
||||
matches singleproc and must become per-child in pool mode. Same for the per-child |
||||
stdout/stderr capture file names. |
||||
- Serial-group default candidates (relaxable after parity evidence under a real console): |
||||
modules/punk/console/**, modules/opunk/console/**, modules/punk/repl/**, shell/**. In |
||||
piped contexts these mostly skip anyway. |
||||
- Kit-executable caveat carries over from phase 1: children of a kit exe boot with |
||||
kit-stamped punk packages preloaded (runner warns; native tclsh preferred). |
||||
- Candidate independent win (either order): make the 17 preload-dependent files' punk::* |
||||
package requires explicit so children can drop the shellrun preload + runx warmup and go |
||||
leaner/faster - the parity tool gates the change. See child_test_runner.tcl comments for |
||||
the two-stage preload detail (shellrun require + runx execution pulling punk::lib). |
||||
- Raising the ceiling beyond the multishell bound means splitting multishell.test (two 30s+ |
||||
tests) and dtplite.test - suite restructure, optional, floor then ~40s (slowest single |
||||
test ~32s). |
||||
- PUNK_TEST_UDPTEE watch mode (commit a65f9e5b): under a pool, the parent's per-file relay |
||||
and lifecycle events still work; the cooked log format's source column plus file-start/ |
||||
file-end events let one viewer demux interleaved children. |
||||
- Timing noise: -slowest per-test microseconds get noisier under CPU oversubscription; |
||||
document rather than fix. |
||||
Loading…
Reference in new issue