punkshell/goals/G-002-non-nested-subshell.md

G-002 Non-nested subshell with console targeting and inter-subshell comms

Status: proposed Scope: src/modules/punk/repl-999999.0a1.0.tm, src/modules/punk/repl/codethread-999999.0a1.0.tm Goal: a subshell can target a named console (default or non-default) and run without blocking the parent, replacing the synchronous nested interp-eval model. Acceptance: a parent REPL launches a subshell against a named console and continues processing its own input while the subshell runs; the parent can signal/query the running subshell; thread::send -async dispatched from within the subshell's code interp arrives at that interp (so packages like promise work when thread features aren't disabled); the "first subshell asymmetry" TODO at repl-999999.0a1.0.tm:3130 is resolved; existing synchronous subshell punk/safe/safebase/punksafe behaviour is preserved as a default mode.

Context

The ::subshell command (src/modules/punk/repl-999999.0a1.0.tm:4147-4155) is an alias to ::repl::interphelpers::subshell_ensemble. The ensemble has four procs (punk, safe, safebase, punksafe, lines 3514-3560) and every one of them launches a subshell the same way:

set replresult [interp eval code {
    package require punk::repl
    repl::init -type punk
    repl::start stdin
}]

Three properties follow from this shape:

1. Always nested. The subshell runs in a child interpreter named code created earlier in repl::init (lines 3575-3620). The parent's call to interp eval code { ... } is synchronous and blocks until the child REPL exits.
2. Always stdin. repl::start stdin hardcodes the parent's stdin as the subshell's input. There is no way to point a subshell at a different console.
3. First-subshell asymmetry. The TODO at lines 3130-3132 notes the first subshell runs differently from subsequent nested ones, and that resolving this matters for "control aspects of the code interp such as cpu/memory resource limits and sandboxing" and for "consistency for how thread calls are routed to the parent interp vs a child interp."

A second TODO at lines 3135-3139 is directly on-point for non-blocking communication:

"investigate whether Tcl code or thread extension code is responsible for routing thread::send -async calls. We want to be able to control which interp receives the call. If we can't do this in pure tcl, then first investigate if we can do it purely in an alternative thread extension. Such an extension should be based closely on the existing thread extension, with minimal changes to allow us to control which interp receives the call, the intention being to see if the changes can be made unobtrusive and backwards compatible so that it has a chance of being accepted into the mainline thread extension."

The primitives for non-blocking inter-thread state already exist in codethread-999999.0a1.0.tm: tsv::set/tsv::get for shared state and thread::cond notify/thread::wait for signalling. They are used between the codethread and the repl thread but are not wired to the subshell command. The subshell is a purely nested, synchronous structure today.

This goal depends on G-001's launch-time console selection: "target a named console" requires the REPL to accept a non-default console spec, which G-001 adds.

Approach

Split the subshell launch into layers. The canonical launch API (section 0) is a prerequisite for the others — the asymmetry fix (section 4) and the non-blocking launch (section 2) both depend on a single dispatch owning code-interp creation.

0. Canonical launch API. Make subshell the single public entry point for starting a REPL; demote repl::init+repl::start to internal implementation that only subshell's dispatch calls. The interp alias mechanism — already in use at repl-999999.0a1.0.tm:4155 (code alias subshell ::repl::interphelpers::subshell_ensemble) — becomes the context signal, not caller introspection:

   • At root: subshell is a real proc (or aliased to the same dispatch in the root interp). The dispatch creates the first code interp, installs the subshell alias in it pointing back to the dispatch, and runs the REPL in code.
   • Inside a code interp: subshell is the alias installed at creation time. Calling it runs the dispatch in the parent, which creates a nested code interp (sub-interp of the calling code), installs the same alias in the nested interp, and runs the REPL there.
   • The caller's code is identical in both cases (subshell punk, subshell safe, etc.). The caller never introspects "am I in a code interp?" — the alias is the answer. Safe interps simply don't get the alias (or get a restricted one permitting only configured launch types), so they can't launch unrestricted subshells and can't ascertain their nesting level.
   • Pre-configuration (-console, -limits, -sandbox) passes through the same API at every level; the dispatch applies it at code-interp creation time (see G-003). The existing subshell punk/safe/safebase/punksafe ensemble procs become type presets that set -type and default -sandbox; option keywords (-console, -limits, -sandbox, later -async) layer on top.
   • Design constraint: the dispatch runs in the parent interp (via alias), so the parent must be responsive enough to service the alias call. In the synchronous-nested model today the parent is blocked in interp eval anyway. For the non-blocking model (section 2), the parent's event loop must reach the alias dispatch — this ties into the thread::send -async routing concern and must not be made unreachable by the non-blocking design.

1. Console targeting. repl::init accepts a -console spec (channel pair, anchored instance name, or ::opunk::Console object value) resolved via punk::console::console_spec_resolve. The subshell ensemble procs pass through a -console argument instead of hardcoding stdin. This is the G-001 seam extended into the subshell command.

2. Non-blocking launch. Replace the synchronous interp eval code { repl::start stdin } with an asynchronous dispatch that returns control to the parent REPL. Two implementation directions to investigate (the user's TODO at 3135-3139 already frames this as an investigation):

   • Pure-Tcl: keep the child interp in a separate thread and use thread::send -async with a -callback to the parent interp. The parent's REPL loop must yield to its event loop between inputs so the callback can be delivered. This requires the parent to control which interp receives the async send, which the TODO flags as the open question.
   • Thread-extension variant: if pure-Tcl routing of thread::send -async to a chosen interp isn't possible, build a minimally-modified thread extension (per the TODO's mainline-acceptance criterion) and gate it behind a package require with fallback to the synchronous mode.

   Hard constraint — standard thread semantics must be preserved. Any non-blocking launch solution must keep thread::send -async working as it does in a plain tclsh session for code running inside the subshell's code interp. The standard Tcl Thread package (written in C, maintained by the Tcl core team) routes thread::send -async messages to the root interpreter of the target thread, not to a named sub-interp; this is documented in-code at codethread-999999.0a1.0.tm:112-113 ("expecting to be called from a thread::send in parent repl - ie in the toplevel interp so that the sub-interp 'code' is available"). Packages like promise (vendored at src/vendormodules/promise-1.2.0.tm) rely on thread::send -async callbacks arriving at the interp that dispatched them; if the message lands at the root interp instead of the code interp, promise and similar packages break. The investigation must therefore answer: can thread::send -async be routed to a chosen sub-interp in pure Tcl, or does the Thread package (and potentially Tcl itself) need modification?

   • If a customised Thread package is required: the modification is written in C, based closely on the upstream Thread extension, with minimal changes to allow controlling which interp receives the call. The existing TODO at repl:3135-3139 sets a mainline-acceptance criterion — the changes should be unobtrusive and backwards-compatible enough to have a chance of being accepted upstream. If this path is taken, create a follow-up goal for the fork's maintenance and upstreaming; do not let it grow inside G-002.
   • If Tcl itself needs patching: same criterion — minimal, upstreamable. A separate follow-up goal should be created for the Tcl patch's maintenance if this path is taken.

3. Inter-subshell communication. Build on the existing tsv::/thread::cond primitives already in codethread-999999.0a1.0.tm. A running subshell registers a well-known tsv:: key (keyed by subshell id); the parent signals/queries via tsv:: and thread::cond. The exact message vocabulary is deferred to implementation — this goal's acceptance is only that the parent can signal/query a running subshell, not a full RPC protocol.

4. First-subshell asymmetry. Resolve TODO at repl:3130 by making the first subshell use the same code-interp launch path as subsequent ones. The asymmetry today is that the first subshell's code interp is created lazily/implicitly; subsequent ones reuse the pattern. The fix is a consequence of the canonical launch API (section 0): root and nested both go through the same dispatch, so there is no longer a "first subshell is special" path — the only branch in the dispatch is whether to create a top-level code or a nested one, which is one branch, not a separate code path.

5. Default mode preserved. The existing synchronous subshell punk/safe/safebase/punksafe behaviour stays available as a default (no -console/-async flags) so existing scripts and muscle memory don't break. The new behaviour is opt-in via flags.

Alternatives considered

• Always-async, drop synchronous mode. Rejected: backward compatibility for existing scripts and the existing subshell punk UX matters. The acceptance criterion explicitly preserves synchronous as the default.
• Coroutines instead of threads for non-blocking. Rejected for now: the subshell runs in a child interp, and coroutines don't cross interp boundaries. Threads are already in use (codethread, %replthread%), so extending the thread model is consistent. Coroutines could be revisited if the thread-routing TODO finds no pure-Tcl answer.
• Separate goal for inter-subshell comms. Considered (the split analysis proposed this as a possible G-003). Deferred: the user framed non-blocking launch and inter-subshell comms together, and the comms vocabulary depends on the launch model chosen. Splitting now risks an orphan goal whose acceptance can't be written until the launch model is fixed. Revisit after G-002's approach firms up.
• Resolve the thread-routing TODO before writing this goal. Rejected: the TODO is an investigation question, not a prerequisite. The goal's acceptance is satisfied by whichever implementation direction the investigation settles on; the goal itself is stable either way.

Notes

• Depends on G-001's launch-time console selection - satisfied: G-001 achieved 2026-07-11 (see goals/archive/G-001-pluggable-console-backends.md), the prerequisite is in place.
• The thread::send -async routing problem is the hardest open question in G-002. The standard Thread package targets the root interp of the receiving thread; the code sub-interp is where subshell code actually runs, so async messages miss it. promise (vendored at src/vendormodules/promise-1.2.0.tm) is the concrete canary — if promise works inside a subshell, the routing is correct.
• The TODO at repl:3135-3139 frames the thread-extension investigation with a mainline-acceptance criterion. If that investigation concludes a fork is needed (customised Thread package and/or Tcl patch), create a follow-up goal for the fork's maintenance and upstreaming; do not let it grow inside G-002. The fork itself is out of scope for G-002's acceptance — G-002 is satisfied by whichever routing solution is found, not by the fork existing.
• tsv:: and thread::cond are already used in codethread-999999.0a1.0.tm (lines 203-218) for cross-thread result/status handoff. The inter-subshell comms layer should reuse the same primitives rather than introducing a new message bus.
• The "first subshell asymmetry" fix touches the same code interp creation path used by the synchronous mode, so it should be done first (it's the smallest piece and unblocks the others).
• No persisted prior chat on this topic was found in project sessions; the motivation comes from the existing TODOs at repl:3130-3139 and the user's stated intent.