punkshell/ARCHITECTURE.md

ARCHITECTURE.md - Punkshell System Mechanics

About this document

This file is the descriptive, current-state map of how punkshell fits together. It occupies its own lane alongside the project's other documentation:

• AGENTS.md files (root + children) are binding work contracts: what you must do when working here.
• GOALS.md, GOALS-archive.md and goals/ track intended and landed changes with rationale: what changed and why.
• This file says how the system works now.

Rules that keep it useful instead of rotten:

1. Pointer-first. Sections state durable mechanics briefly and route to canonical sources (module files, owning AGENTS.md, goal detail files). Never transcribe content that lives elsewhere - point at it. If a section wants to grow past a screenful, the detail belongs in the module or a child doc, with this section pointing at it.
2. Current state only. History, rejected alternatives and verification records stay in goal detail files; release-level history stays in CHANGELOG.md. Do not narrate change history here.
3. Staleness. When this file conflicts with the code, the code wins and this file is stale. Update or delete the stale section in the same change that breaks it (root AGENTS.md Closeout). Delete stale sections rather than annotating them.
4. In-flux areas. Subsystems under active redesign are documented as one-liners with goal pointers, not detailed prose (see "In-flux areas").
5. Verification. tclsh scriptlib/developer/architecture_lint.tcl - backtick-quoted repo paths must exist, G-<id> references must resolve against the goal indexes, size budget enforced. Run after editing this file.

The ten-thousand-foot view

Punkshell is an alternative Tcl shell distributed as self-contained executables ("kits"): a zipfs/metakit-aware Tcl runtime wrapped with a virtual filesystem (VFS) payload carrying the punk module set. A kit boots through a single entry script, which selects a package mode (which module sources to trust) and a subcommand (which application to run: interactive shell, non-interactive script runner, stock tclsh emulation, and others). The interactive shell is a REPL in the main interp that evaluates user code in a separate code thread and talks to the terminal through a two-layer console abstraction. Everything is built, tested and released by a Tcl-driven build system (src/make.tcl) with file-level provenance tracking (punkcheck).

bin/<kit>.exe  =  Tcl runtime (zipfs/metakit)  +  <name>.vfs payload
       | boots
       v
src/vfs/_config/punk_main.tcl
  - package mode:  dev / os / src / internal   (which module sources to trust)
  - subcommand:    shell / script / punk / tclsh / shellspy
       |
       v
app package from src/lib/  (app-punkshell, app-punkscript, app-repl, app-shellspy)
       |
       v
+------------------- interactive shell --------------------+
|  REPL interp (punk::repl)  <-->  console abstraction     |
|    + codethread (user code)       (punk::console over    |
|    + shellthread workers          opunk::Console)        |
+----------------------------------------------------------+

Boot and launch chain

• Kit anatomy. Each src/vfs/*.vfs folder is a runtime payload; src/runtime/mapvfs.config maps payload folders to platform runtimes under bin/runtime/<platform>/; src/vfs/_vfscommon.vfs/ is a generated merge of common libraries (never hand-edited); src/vfs/_config/ holds the entry scripts. Sources: src/vfs/AGENTS.md, src/runtime/AGENTS.md.
• Entry point. src/vfs/_config/punk_main.tcl. An optional first argument is a dash-delimited package mode composed of tokens from dev, os, src, internal (internal is always appended when absent), optionally scoped with the proj: prefix (G-033: dev/src resolve against the project containing the cwd - walk-up to the nearest git/fossil repo root with a punkshell-style src tree - instead of the executable's own project; discovery is always reported, never a silent rebind). The next argument is the subcommand: shell, script, punk, tclsh, shellspy. Any other first argument is treated as a script invocation; no arguments at all means interactive shell. Subcommands map to app packages under src/lib/ (app-punkshell, app-punkscript, app-repl, app-shellspy).
• Package modes pick module provenance. internal uses module snapshots baked into the kit at build time; dev loads build outputs from repo-root modules/ and lib/; src loads the unbuilt working tree (src/modules, src/lib, src/bootsupport, src/vendormodules) with package prefer latest so magic-version dev modules (999999.0a1.0) beat stamped snapshots. src mode is the standard way to verify working-tree changes without a rebuild. Source: bin/AGENTS.md "Launch package modes".
• Payload mount derivation (G-129, achieved). The boot keys zipfs presence on tcl::zipfs::mount and derives where the executable's attached archive actually mounted from the mount table itself (the entry whose archive file is the executable; fallback: the mount containing the boot script) instead of assuming tcl::zipfs::root + //zipfs:/app. Modern runtimes still answer //zipfs:/app; the androwish/undroidwish 8.6 backport has no root command, mounts at the executable's own path, and boots the same payload through its app/main.tcl in-archive hook (src/vfs/punk8win.vfs carries fauxlinks for both conventions). A mounted archive the boot cannot attribute is reported on stderr rather than silently yielding no internal paths. The derivation procs stay defined post-boot (::punkboot::zipfs_kit_mountbase). Sources: src/vfs/AGENTS.md, bin/AGENTS.md "Kit-wrappable runtime requirements", src/tests/shell/testsuites/punkexe/kitmountpoint.test.
• Static package capture (G-058, achieved). Boot probe-loads the runtime's statically linked packages in a throwaway interp and records what each provides; every interp the shell fabricates (boot, codethread, shellthread workers) seeds package ifneeded <name> <ver> {load {} <prefix>} from that record, and punk::packagepreference resolves static-vs-bundled version-aware. Sources: src/vfs/AGENTS.md, goals/archive/G-058-static-runtime-packages.md.
• script subcommand (G-015, achieved). The lean non-interactive path: default punk shell module/alias environment, no shellfilter stacks/transforms, honest exit codes, launch plumbing emits nothing on stdout/stderr (exec-style callers see only the script's own output). Supports lib:<name> scriptlib resolution. App package: src/lib/app-punkscript/.
• tclsh subcommand (G-118, achieved). Plain-tclsh emulation with NO punk modules loaded: stock argument-form parity (-encoding name file; any other leading - argument means no script, everything stays in ::argv; no -e one-liner - deliberate stock parity, see G-077 for the punk-level one-liner direction), lib: refused with a pointer at script, missing script file is a clean exit 1. On piperepl-patched runtimes (the punk family runtimes - src/buildsuites/suite_tcl90/patches/) the C-level machinery publishes ::tclsh launch state (istty, dorepl, evalinput, inputbuffer, reopened) allowing piped input to hand over to a live console repl at eof; runtimes without the machinery evaluate piped stdin whole-buffer with honest exit codes and fail fast on console stdin. The subcommand-family contract (tclsh/script/shell/punk/shellspy) is documented as punk::args definitions in src/modules/punk/args/moduledoc/punkexe-999999.0a1.0.tm (ids (script)::punkexe + per-subcommand; view via punk::args::usage), consumable by the future launcher help wiring (G-032). Pipe-testable behaviour pinned by src/tests/shell/testsuites/punkexe/tclshcmd.test.
• bin/ launch surface. Built kits plus polyglot .cmd utilities (punk-runtime, punk-getzig, punk-tclargs, dtplite) generated from src/scriptapps/ sources by the punk::mix scriptwrap machinery - outputs are never hand-edited. bin/punk-runtime.cmd fetches/lists/selects plain runtimes under bin/runtime/<platform>/ from the punkbin artifact server with sha1 verification. Source: bin/AGENTS.md.

Interp and thread topology

• REPL interp vs codethread. punk::repl runs the reader/prompt loop in the main interp; user commands evaluate in a separate thread's interp (src/modules/punk/repl/codethread-999999.0a1.0.tm) so the reader stays responsive and user-code state is isolated from shell state.
• shellfilter stacks. Channel transform/logging layers. Among other roles, var JUNCTION stacks divert the code interp's stdout/stderr into per-run variables that the repl collects and emits to the console after each run (G-001). The API is known-clumsy; audit is proposed as G-090. Source: src/modules/shellfilter-999999.0a1.0.tm.
• shellthread workers. Pooled worker threads (logging, tee handlers). tcludp 1.0.12's Windows per-thread exit handler closed process-global events and wedged worker event loops (G-036, achieved); kits now bundle tcludp >= 1.0.13 and a runtime has_bug-style check surfaces the vulnerable combination. Worker lifecycle audit is also G-090. Source: src/modules/shellthread-999999.0a1.0.tm.
• Shared state. Infrastructure tsv arrays are punk_-prefixed by contract (legacy unprefixed arrays predate the rule); console facts, ownership and mode caches live in tsv so every thread sees one truth (G-007). Source: src/modules/AGENTS.md.
• Subshells. Named secondary repls exist; the substantial subshell roadmap is in-flux - see "In-flux areas".

Process exit and shutdown

• App-level ordered teardown. The shell does not leave thread/channel cleanup to Tcl's process finalization: the app package runs an explicit shutdown sequence while interps are fully alive - shellthread::manager::shutdown_free_threads terminates free workers under a bounded timeout (a wedged worker must not hang shutdown), remaining threads are released, the repl cancels its deferred stdin reader registration and drops input arriving after codethread teardown, and the codethread flushes stdout/stderr before returning results. Sources: src/lib/app-punkshell/punkshell.tcl, src/modules/shellthread-999999.0a1.0.tm, src/modules/punk/repl-999999.0a1.0.tm, src/tests/shell/testsuites/punkexe/shellexit.test.
• Piped-stdin exit policy. PUNK_PIPE_EOF selects terminate vs console-reopen at piped EOF (G-015, achieved) so scripted termination is deterministic rather than dependent on console-availability heuristics. Source: src/lib/app-punkshell/punkshell.tcl.
• Upstream constraint: Tcl finalization order is not a documented contract. Tcl_Finalize runs a fixed-but-undocumented subsystem order in which extension exit handlers can fire while other threads are still alive and while channels, allocators and mutexes are mid-teardown - the shape behind the G-036 tcludp worker wedge ("Interp and thread topology") and the dead-console class ("Console abstraction"). Current upstream state: trunk carries a partial fix for the busy-thread finalize crash (tcl ticket e55a589d2b, 2026-05) explicitly NOT backported to 9.0, so 9.0.x runtimes (buildsuite target 9.0.5) still carry it. Work guidance derived from this constraint: src/AGENTS.md "Work Guidance". Surveyed core-source detail: TEMP_REFERENCE/AGENTS.md surveyed-areas entry for tcl9/ finalize path.

Console abstraction

Two layers with a deliberate dependency direction (the class never depends on the integrating layer):

• opunk::Console (src/modules/opunk/) - a voo value-OO base class modelling a console as an in/out pair with settled capability facts. Objects are plain Tcl list values; -virtual methods dispatch on a class tag so subclasses can override everything channel-related. Cooperative seams are pluggable: the probe-byte store (waiting_chunks_arrayvar), the size-query provider (size_query_provider), and an anchor lifecycle callback. Source: src/modules/opunk/AGENTS.md.
• punk::console (src/modules/punk/console-999999.0a1.0.tm) - the integrating layer: console-spec resolution, per-console terminal-property facts, raw mode, ANSI emit/query helpers, and registration of the providers the base class consumes.
• Spec forms and selection. A console spec is an {in out} channel pair, an anchored instance name, or an ::opunk::Console object value, normalised by punk::console::console_spec_resolve. repl::init -console <spec> selects the console per repl (G-001, achieved); the default is the process console {stdin stdout}. Source: src/modules/punk/AGENTS.md.
• Backends (G-001, achieved). Reference subclasses opunk::console::test, ::ssh, ::tk (deterministic test double, socket-carried terminal, Tk text widget) prove non-detectable terminal-like devices can host a repl with no base-class or punk::console changes. Detail: goals/archive/G-001-pluggable-console-backends.md.
• Location transparency (G-007, achieved). Facts are tsv-backed and terminal queries are owner-routed: a query from any thread forwards to the console-owning thread through a thread::send choke point, so code-interp code sees the same console the repl sees without repl eval round-trips. Ownership lives in a tsv registry keyed by canonical channel pair.
• Raw mode. twapi on Windows when available; otherwise a persistent powershell console-mode named-pipe server serves raw transitions on twapi-less runtimes (G-106, achieved) - quiet, lazy-started, exits with the owning process.
• Dead-console handling. The Tcl 9 Windows console driver never reported a killed host terminal to the script, leaving orphaned shells spinning a core (G-039, achieved; upstream ticket filed and fixed). The repl arms a close-on-probe-failure watchdog only on runtimes a punk::lib::check classifier reports as affected (G-076, achieved); runtimes at or past 9.0.5 deliver the dead console as a readable event and use the normal eof teardown.

Module ecosystem and package loading

• Editable modules live in src/modules/ (+ src/modules_tcl8/, src/modules_tcl9/ siblings); filenames carry the magic dev version 999999.0a1.0 with the real semver in a <name>-buildversion.txt sidecar, and namespaces mirror directory layout (punk/ansi/ = punk::ansi). punk::libunknown is the manually-versioned exception. Source: src/modules/AGENTS.md.
• punk::args is both parser and documentation system. PUNKARGS/argdoc blocks travel with each proc and power inline usage tables (i <command> at the repl); even manually-parsing procs carry documentation-only definitions. Much of the goals-era work has been hardening this system - see the punkargs entries across GOALS.md and GOALS-archive.md.
• Discovery and modpods. punk::libunknown provides module discovery/registration beyond tcl::tm defaults; #modpod-* source directories pack into zip-based multi-file .tm modules at build time (src mode registers them via an inline boot scanner written in builtins).
• Bootsupport. src/bootsupport/ holds snapshot copies of build-time-critical modules (manifest: include_modules.config), analogous to devDependencies. make.tcl classifies snapshot staleness for five runtime-critical packages (punkcheck, punk::repo, punk::mix, punk::tdl, punk::args) as abort/prompt/warn from the version delta. Source: src/bootsupport/AGENTS.md.
• opunk layer. src/modules/opunk/ explores voo-based value-OO reimplementations (objects as plain Tcl values); the punk::* modules remain the production implementations.
• Vendored code. Third-party packages live in src/vendormodules/ and src/vendorlib/ (+ _tcl8/_tcl9 siblings), refreshed by make.tcl vendorupdate from config; never hand-edited (enforcement policy is proposed as G-026).

Build, provenance and kits

• src/make.tcl is the single build entry. Key subcommands: modules/libs/packages (repo-root build outputs), vfscommonupdate (regenerate _vfscommon.vfs), bake (kit assembly into bin/; optional kit names confine the run to those kits - G-121), bakelist (report the configured kit matrix with runtime/vfs presence and deployed state), bin, bakehouse (packages + bake for a clean checkout; project/vfs remain as deprecated aliases - G-112 rename, achieved), bootsupport (snapshot refresh), vendorupdate, check, workflow, projectversion. Dispatch and help dogfood punk::args (G-030, achieved) and degrade to plain scan/help when bootsupport punk::args is stale (PUNKBOOT_PLAIN=1 forces the degraded mode). Source: src/AGENTS.md.
• Host vs target platform (G-122). The kit surfaces separate what the driving tclsh IS from what a bake is FOR. Target-keyed: the bin/runtime/<tier> store a runtime is read from, .exe suffixing of runtimes and kit outputs, presence checks, and the pre-deploy process sweep's tooling (tasklist/taskkill vs ps/kill, skipped when the target's processes cannot exist on this host). Host-keyed: copy commands, path handling, filesystem case rules, prompts. The default target is the host canon except for cygwin-family hosts (msys2/cygwin-runtime tclsh - tcl_platform(platform) unix on windows), which target win32-x86_64; a mapvfs.config entry may declare its own target and is then addressed in that platform's tier. make.tcl check prints the derivation. Zip-type kits assemble without zipfs in the driving tcl (raw-runtime split + punk::zip::mkzip + concatenation), and since G-124 they EXTRACT without it too: punk::zip reads a zip - plain, or attached to an executable under either offset convention - with stock Tcl only, so a bake needs neither zipfs nor tcllib to carry a runtime's tcl_library into the kit. Sources: src/AGENTS.md, src/runtime/AGENTS.md, src/modules/punk/platform-999999.0a1.0.tm, src/modules/punk/zip-999999.0a1.0.tm.
• punkcheck records every install/delete as events in per-folder .punkcheck directories - the basis for skip/copy change detection, superseded-module pruning and provenance. Single OO record lifecycle (G-094, achieved) with atomic saves and advisory event-scoped locking for concurrent writers (G-095, achieved).
• Provenance gates. Build/promotion commands warn on uncommitted src/ changes (column-0 PROVENANCE-WARNING: token, -dirty-abort for strict mode); vendorupdate warns for dirty source-project checkouts.
• Boot-precondition gate (G-125). A bake refuses a kit whose merged .vfs supplies no tcl library rather than deploying an artifact that cannot initialise: the kit lands in FAILED KITS and nothing is written, so the previously deployed bin/<kit> survives. Structural and non-executing (so cross-target kits are covered), reading the merged tree rather than the extraction outcome. The predicate is punkboot::utils::vfs_boot_library_report, called through the same guarded require as the provenance check so a stale bootsupport snapshot degrades it to a notice; make.tcl check reports ACTIVE/UNAVAILABLE. Sources: src/AGENTS.md, src/modules/punkboot/utils-999999.0a1.0.tm.
• Payload/target consistency checks (G-133 + G-134), all advisory. At the same post-merge seam, a bake classifies each binary library in the merged tree by header (punkboot::utils::binary_arch_classify - PE/ELF/Mach-O, honest unknowns) against the kit's target platform; wrong-arch libraries outside platform-discriminated subdirs (canonical <os>-<cpu> names, vendor spellings like win-x64) earn recapped BUILD-WARNINGs. After assembly, a kit declaring smoke-require packages (mapvfs.config 5th entry element) has each one plain-package required inside the freshly built artifact via its tclsh subcommand - the only check that sees resolution-order defects (wrong-arch version shadowing); cross-target kits skip with a stated reason. The assembled image is also probed for its zip offset convention (G-134, punkboot::utils::kit_offsetstyle_report over punk::zip::archive_info): a FILE-relative attached payload warns (the pipeline emits archive-relative; the G-128 stamper refuses file-relative by default), no-zip/plain results stay silent. Same guarded-require degradation; make.tcl check reports all three. None guarantees statics, pure-tcl packages with binary deps, or version preference beyond the declared smoke set. Sources: src/AGENTS.md, src/runtime/AGENTS.md, src/modules/punkboot/utils-999999.0a1.0.tm.
• Kit icon step (G-057 + G-128). Every kit a bake builds gets a <kitname>.resources.toml sidecar (deployed beside bin/<kit>) recording the build-time icon choice - default src/runtime/punkshell.ico, overridable by a root punkshell.ico in the kit's own custom .vfs folder (pre-merge) - with sha256 identity and provenance lifted from the icon's G-135 assetorigin record. win32-target kits get the icon embedded as PE RT_ICON/RT_GROUP_ICON behind a single seam (::punkboot::kit_icon_process): the vendored punkres portable stamper (src/tools/punkres, built to bin/punkres(.exe) by the make.tcl tool step) is selected when present and works from any build host; otherwise the twapi arm (tcl-sfe mechanism) serves windows hosts with nothing built. Both arms share one semantic (delete all icon/group entries, write ids 1..N; the group is NAMED from the icon file's uppercased rootname - PUNKSHELL - with language adopted from the replaced group) and stamp a per-kit copy of the payload-free raw runtime prefix BEFORE payload attach; punkres can also stamp a FINISHED kit post-hoc, preserving an appended zip payload (archive-relative moved verbatim; file-relative refused by default, shifted with a consent flag). Non-PE targets skip as not applicable; a host where neither mechanism serves skips with a combined notice naming the punkres build remedy; the sidecar is written in every case. Sources: src/AGENTS.md, bin/AGENTS.md, src/vfs/AGENTS.md, src/tools/AGENTS.md.
• Buildsuites and the kit family. src/buildsuites/suite_tcl90/ builds Tcl/Tk/tcllib from source with a pinned zig toolchain and produces the runtime kit family (plain / punk / bi) plus artifact metadata (G-096-G-117 era: see archived goals G-096, G-098, G-102, G-103, G-107); artifacts publish to the punkbin repo that bin/punk-runtime.cmd fetches from. src/buildsuites/suite_tcl86/ is the 8.6 sibling (G-099 + G-100, both achieved): a Tcl 8.6 windows runtime (static + dynamic shells, tcl86t.dll, on-disk lib tree - no zipfs, so no self-contained kit) with thread 2.8 + tclvfs + Tk 8.6 + tklib + tcllib(+tcllibc critcl accelerators) companions, gated against the 8.6 core/thread/tclvfs testsuites with tcllib/tklib on a record tier and an opt-in test-tk; punkshell's own runtests is censused on that runtime at parity with a same-day native-8.6 baseline. Suite child shells scrub TCL<major>_<minor>_TM_PATH as well as TCLLIBPATH/TCL_LIBRARY/TK_LIBRARY - without it a census measures the machine's module trees. The 8.6 kit container strategy remains in-flux - see "In-flux areas".
• Project generation. dev project.new composes thin layouts from src/project_layouts/ (overlay chain with .anti deletion markers and name@base derivation - G-087, achieved) and injects bootsupport modules from the generating shell at generation time.
• Workflow overview. tclsh src/make.tcl workflow prints the embedded ASCII data-flow overview of the build/release pipeline, with its own update contract in src/AGENTS.md. This section deliberately summarises rather than copies it.

Test harness

• src/tests/runtests.tcl is the entry point. Path-glob discovery over testsuites/ with the X/***-style lattice (-include-paths/-exclude-paths, G-093), single-process or multi-process child modes, a -jobs N worker pool (G-091), report modes (compact/markdown/json) and an optional UDP watch stream. Source: src/tests/AGENTS.md.
• Characterization style. Suites pin observed current behaviour, including _GAP_-named tests that pin known bugs and flip when a goal fixes them - goal acceptance criteria routinely name the pins they flip.
• Child-process suites by design. A repl cannot run inside the shared testinterp (codethread exit callbacks target the thread's main interp), so console/repl verification drives child processes (src/tests/testsupport/repl_console_driver.tcl); kit and launcher behaviour is likewise exec-driven under src/tests/shell/.
• Mode parity. scriptlib/developer/runtests_parity.tcl compares -report json outputs across runner modes.

In-flux areas

Mechanics here are documented as pointers only - detailed prose would be obsolete on arrival. Read the goal detail files for intended end states and the code for current state.

• Subshell roadmap: G-002, G-003, G-008, G-009, G-010 (non-nested subshells, resource limits, scoped console state, themed profiles, tree navigation).
• Input mode and completion: G-013 (raw-mode default), G-044 (punk::args-driven completion).
• Stored configuration and help: G-014 (punk::config toml), G-042 (subshell help topics), G-043 (subshell definition plugins).
• Console err channel: G-011.
• Launch surface: G-023 (version-named binaries), G-077 (-e one-liners), G-089 (scriptlib in kits).
• 8.6 buildsuite family: G-101 (kit container) - the core runtime (G-099) and the Tk/tklib/tcllib dependency set (G-100) are achieved 2026-07-26.

Doc map - which lane holds what

• How it works now (durable mechanics): this file.
• What you must do when working here: root AGENTS.md + the child AGENTS.md chain (binding).
• What changed and why: GOALS.md, GOALS-archive.md, goals/ detail files; CHANGELOG.md for release-level history.
• Per-command and per-proc usage: PUNKARGS blocks in the modules (i <command> at a repl); generated docs under src/doc/.
• User-facing features: README.md.