# ARCHITECTURE.md - Punkshell System Mechanics ## About this document This file is the descriptive, current-state map of how punkshell fits together. It occupies its own lane alongside the project's other documentation: - `AGENTS.md` files (root + children) are binding work contracts: what you must do when working here. - `GOALS.md`, `GOALS-archive.md` and `goals/` track intended and landed changes with rationale: what changed and why. - This file says how the system works now. Rules that keep it useful instead of rotten: 1. **Pointer-first.** Sections state durable mechanics briefly and route to canonical sources (module files, owning AGENTS.md, goal detail files). Never transcribe content that lives elsewhere - point at it. If a section wants to grow past a screenful, the detail belongs in the module or a child doc, with this section pointing at it. 2. **Current state only.** History, rejected alternatives and verification records stay in goal detail files; release-level history stays in `CHANGELOG.md`. Do not narrate change history here. 3. **Staleness.** When this file conflicts with the code, the code wins and this file is stale. Update or delete the stale section in the same change that breaks it (root `AGENTS.md` Closeout). Delete stale sections rather than annotating them. 4. **In-flux areas.** Subsystems under active redesign are documented as one-liners with goal pointers, not detailed prose (see "In-flux areas"). 5. **Verification.** `tclsh scriptlib/developer/architecture_lint.tcl` - backtick-quoted repo paths must exist, `G-` references must resolve against the goal indexes, size budget enforced. Run after editing this file. ## The ten-thousand-foot view Punkshell is an alternative Tcl shell distributed as self-contained executables ("kits"): a zipfs/metakit-aware Tcl runtime wrapped with a virtual filesystem (VFS) payload carrying the punk module set. A kit boots through a single entry script, which selects a package mode (which module sources to trust) and a subcommand (which application to run: interactive shell, non-interactive script runner, stock tclsh emulation, and others). The interactive shell is a REPL in the main interp that evaluates user code in a separate code thread and talks to the terminal through a two-layer console abstraction. Everything is built, tested and released by a Tcl-driven build system (`src/make.tcl`) with file-level provenance tracking (`punkcheck`). ``` bin/.exe = Tcl runtime (zipfs/metakit) + .vfs payload | boots v src/vfs/_config/punk_main.tcl - package mode: dev / os / src / internal (which module sources to trust) - subcommand: shell / script / punk / tclsh / shellspy | v app package from src/lib/ (app-punkshell, app-punkscript, app-repl, app-shellspy) | v +------------------- interactive shell --------------------+ | REPL interp (punk::repl) <--> console abstraction | | + codethread (user code) (punk::console over | | + shellthread workers opunk::Console) | +----------------------------------------------------------+ ``` ## Boot and launch chain - **Kit anatomy.** Each `src/vfs/*.vfs` folder is a runtime payload; `src/runtime/mapvfs.config` maps payload folders to platform runtimes under `bin/runtime//`; `src/vfs/_vfscommon.vfs/` is a generated merge of common libraries (never hand-edited); `src/vfs/_config/` holds the entry scripts. Sources: `src/vfs/AGENTS.md`, `src/runtime/AGENTS.md`. - **Entry point.** `src/vfs/_config/punk_main.tcl`. An optional first argument is a dash-delimited package mode composed of tokens from `dev`, `os`, `src`, `internal` (`internal` is always appended when absent), optionally scoped with the `proj:` prefix (G-033: `dev`/`src` resolve against the project containing the cwd - walk-up to the nearest git/fossil repo root with a punkshell-style src tree - instead of the executable's own project; discovery is always reported, never a silent rebind). The next argument is the subcommand: `shell`, `script`, `punk`, `tclsh`, `shellspy`. Any other first argument is treated as a script invocation; no arguments at all means interactive shell. Subcommands map to app packages under `src/lib/` (`app-punkshell`, `app-punkscript`, `app-repl`, `app-shellspy`). - **Package modes pick module provenance.** `internal` uses module snapshots baked into the kit at build time; `dev` loads build outputs from repo-root `modules/` and `lib/`; `src` loads the unbuilt working tree (`src/modules`, `src/lib`, `src/bootsupport`, `src/vendormodules`) with `package prefer latest` so magic-version dev modules (`999999.0a1.0`) beat stamped snapshots. `src` mode is the standard way to verify working-tree changes without a rebuild. Source: `bin/AGENTS.md` "Launch package modes". - **Payload mount derivation (G-129, achieved).** The boot keys zipfs presence on `tcl::zipfs::mount` and derives where the executable's attached archive actually mounted from the mount table itself (the entry whose archive file is the executable; fallback: the mount containing the boot script) instead of assuming `tcl::zipfs::root` + `//zipfs:/app`. Modern runtimes still answer `//zipfs:/app`; the androwish/undroidwish 8.6 backport has no `root` command, mounts at the executable's own path, and boots the same payload through its `app/main.tcl` in-archive hook (`src/vfs/punk8win.vfs` carries fauxlinks for both conventions). A mounted archive the boot cannot attribute is reported on stderr rather than silently yielding no internal paths. The derivation procs stay defined post-boot (`::punkboot::zipfs_kit_mountbase`). Sources: `src/vfs/AGENTS.md`, `bin/AGENTS.md` "Kit-wrappable runtime requirements", `src/tests/shell/testsuites/punkexe/kitmountpoint.test`. - **Static package capture (G-058, achieved).** Boot probe-loads the runtime's statically linked packages in a throwaway interp and records what each provides; every interp the shell fabricates (boot, codethread, shellthread workers) seeds `package ifneeded {load {} }` from that record, and `punk::packagepreference` resolves static-vs-bundled version-aware. Sources: `src/vfs/AGENTS.md`, `goals/archive/G-058-static-runtime-packages.md`. - **`script` subcommand (G-015, achieved).** The lean non-interactive path: default punk shell module/alias environment, no shellfilter stacks/transforms, honest exit codes, launch plumbing emits nothing on stdout/stderr (exec-style callers see only the script's own output). Supports `lib:` scriptlib resolution. App package: `src/lib/app-punkscript/`. - **`tclsh` subcommand (G-118, achieved).** Plain-tclsh emulation with NO punk modules loaded: stock argument-form parity (`-encoding name file`; any other leading `-` argument means no script, everything stays in `::argv`; no `-e` one-liner - deliberate stock parity, see G-077 for the punk-level one-liner direction), `lib:` refused with a pointer at `script`, missing script file is a clean exit 1. On piperepl-patched runtimes (the punk family runtimes - `src/buildsuites/suite_tcl90/patches/`) the C-level machinery publishes `::tclsh` launch state (`istty`, `dorepl`, `evalinput`, `inputbuffer`, `reopened`) allowing piped input to hand over to a live console repl at eof; runtimes without the machinery evaluate piped stdin whole-buffer with honest exit codes and fail fast on console stdin. The subcommand-family contract (tclsh/script/shell/punk/shellspy) is documented as punk::args definitions in `src/modules/punk/args/moduledoc/punkexe-999999.0a1.0.tm` (ids `(script)::punkexe` + per-subcommand; view via `punk::args::usage`), consumable by the future launcher help wiring (G-032). Pipe-testable behaviour pinned by `src/tests/shell/testsuites/punkexe/tclshcmd.test`. - **`bin/` launch surface.** Built kits plus polyglot `.cmd` utilities (`punk-runtime`, `punk-getzig`, `punk-tclargs`, `dtplite`) generated from `src/scriptapps/` sources by the `punk::mix` scriptwrap machinery - outputs are never hand-edited. `bin/punk-runtime.cmd` fetches/lists/selects plain runtimes under `bin/runtime//` from the punkbin artifact server with sha1 verification. Source: `bin/AGENTS.md`. ## Interp and thread topology - **REPL interp vs codethread.** `punk::repl` runs the reader/prompt loop in the main interp; user commands evaluate in a separate thread's interp (`src/modules/punk/repl/codethread-999999.0a1.0.tm`) so the reader stays responsive and user-code state is isolated from shell state. - **shellfilter stacks.** Channel transform/logging layers. Among other roles, `var` JUNCTION stacks divert the code interp's stdout/stderr into per-run variables that the repl collects and emits to the console after each run (G-001). The API is known-clumsy; audit is proposed as G-090. Source: `src/modules/shellfilter-999999.0a1.0.tm`. - **shellthread workers.** Pooled worker threads (logging, tee handlers). tcludp 1.0.12's Windows per-thread exit handler closed process-global events and wedged worker event loops (G-036, achieved); kits now bundle tcludp >= 1.0.13 and a runtime `has_bug`-style check surfaces the vulnerable combination. Worker lifecycle audit is also G-090. Source: `src/modules/shellthread-999999.0a1.0.tm`. - **Shared state.** Infrastructure `tsv` arrays are `punk_`-prefixed by contract (legacy unprefixed arrays predate the rule); console facts, ownership and mode caches live in tsv so every thread sees one truth (G-007). Source: `src/modules/AGENTS.md`. - **Subshells.** Named secondary repls exist; the substantial subshell roadmap is in-flux - see "In-flux areas". ## Process exit and shutdown - **App-level ordered teardown.** The shell does not leave thread/channel cleanup to Tcl's process finalization: the app package runs an explicit shutdown sequence while interps are fully alive - `shellthread::manager::shutdown_free_threads` terminates free workers under a bounded timeout (a wedged worker must not hang shutdown), remaining threads are released, the repl cancels its deferred stdin reader registration and drops input arriving after codethread teardown, and the codethread flushes stdout/stderr before returning results. Sources: `src/lib/app-punkshell/punkshell.tcl`, `src/modules/shellthread-999999.0a1.0.tm`, `src/modules/punk/repl-999999.0a1.0.tm`, `src/tests/shell/testsuites/punkexe/shellexit.test`. - **Piped-stdin exit policy.** PUNK_PIPE_EOF selects terminate vs console-reopen at piped EOF (G-015, achieved) so scripted termination is deterministic rather than dependent on console-availability heuristics. Source: `src/lib/app-punkshell/punkshell.tcl`. - **Upstream constraint: Tcl finalization order is not a documented contract.** Tcl_Finalize runs a fixed-but-undocumented subsystem order in which extension exit handlers can fire while other threads are still alive and while channels, allocators and mutexes are mid-teardown - the shape behind the G-036 tcludp worker wedge ("Interp and thread topology") and the dead-console class ("Console abstraction"). Current upstream state: trunk carries a partial fix for the busy-thread finalize crash (tcl ticket e55a589d2b, 2026-05) explicitly NOT backported to 9.0, so 9.0.x runtimes (buildsuite target 9.0.5) still carry it. Work guidance derived from this constraint: `src/AGENTS.md` "Work Guidance". Surveyed core-source detail: `TEMP_REFERENCE/AGENTS.md` surveyed-areas entry for `tcl9/` finalize path. ## Console abstraction Two layers with a deliberate dependency direction (the class never depends on the integrating layer): - **`opunk::Console`** (`src/modules/opunk/`) - a `voo` value-OO base class modelling a console as an in/out pair with settled capability facts. Objects are plain Tcl list values; `-virtual` methods dispatch on a class tag so subclasses can override everything channel-related. Cooperative seams are pluggable: the probe-byte store (`waiting_chunks_arrayvar`), the size-query provider (`size_query_provider`), and an anchor lifecycle callback. Source: `src/modules/opunk/AGENTS.md`. - **`punk::console`** (`src/modules/punk/console-999999.0a1.0.tm`) - the integrating layer: console-spec resolution, per-console terminal-property facts, raw mode, ANSI emit/query helpers, and registration of the providers the base class consumes. - **Spec forms and selection.** A console spec is an `{in out}` channel pair, an anchored instance name, or an `::opunk::Console` object value, normalised by `punk::console::console_spec_resolve`. `repl::init -console ` selects the console per repl (G-001, achieved); the default is the process console `{stdin stdout}`. Source: `src/modules/punk/AGENTS.md`. - **Backends (G-001, achieved).** Reference subclasses `opunk::console::test`, `::ssh`, `::tk` (deterministic test double, socket-carried terminal, Tk text widget) prove non-detectable terminal-like devices can host a repl with no base-class or `punk::console` changes. Detail: `goals/archive/G-001-pluggable-console-backends.md`. - **Location transparency (G-007, achieved).** Facts are tsv-backed and terminal queries are owner-routed: a query from any thread forwards to the console-owning thread through a `thread::send` choke point, so code-interp code sees the same console the repl sees without `repl eval` round-trips. Ownership lives in a tsv registry keyed by canonical channel pair. - **Raw mode.** twapi on Windows when available; otherwise a persistent powershell console-mode named-pipe server serves raw transitions on twapi-less runtimes (G-106, achieved) - quiet, lazy-started, exits with the owning process. - **Dead-console handling.** The Tcl 9 Windows console driver never reported a killed host terminal to the script, leaving orphaned shells spinning a core (G-039, achieved; upstream ticket filed and fixed). The repl arms a close-on-probe-failure watchdog only on runtimes a `punk::lib::check` classifier reports as affected (G-076, achieved); runtimes at or past 9.0.5 deliver the dead console as a readable event and use the normal eof teardown. ## Module ecosystem and package loading - **Editable modules** live in `src/modules/` (+ `src/modules_tcl8/`, `src/modules_tcl9/` siblings); filenames carry the magic dev version `999999.0a1.0` with the real semver in a `-buildversion.txt` sidecar, and namespaces mirror directory layout (`punk/ansi/` = `punk::ansi`). `punk::libunknown` is the manually-versioned exception. Source: `src/modules/AGENTS.md`. - **`punk::args` is both parser and documentation system.** `PUNKARGS`/`argdoc` blocks travel with each proc and power inline usage tables (`i ` at the repl); even manually-parsing procs carry documentation-only definitions. Much of the goals-era work has been hardening this system - see the `punkargs` entries across `GOALS.md` and `GOALS-archive.md`. - **Discovery and modpods.** `punk::libunknown` provides module discovery/registration beyond `tcl::tm` defaults; `#modpod-*` source directories pack into zip-based multi-file `.tm` modules at build time (src mode registers them via an inline boot scanner written in builtins). - **Bootsupport.** `src/bootsupport/` holds snapshot copies of build-time-critical modules (manifest: `include_modules.config`), analogous to devDependencies. `make.tcl` classifies snapshot staleness for five runtime-critical packages (`punkcheck`, `punk::repo`, `punk::mix`, `punk::tdl`, `punk::args`) as abort/prompt/warn from the version delta. Source: `src/bootsupport/AGENTS.md`. - **opunk layer.** `src/modules/opunk/` explores `voo`-based value-OO reimplementations (objects as plain Tcl values); the `punk::*` modules remain the production implementations. - **Vendored code.** Third-party packages live in `src/vendormodules/` and `src/vendorlib/` (+ `_tcl8`/`_tcl9` siblings), refreshed by `make.tcl vendorupdate` from config; never hand-edited (enforcement policy is proposed as G-026). ## Build, provenance and kits - **`src/make.tcl` is the single build entry.** Key subcommands: `modules`/`libs`/`packages` (repo-root build outputs), `vfscommonupdate` (regenerate `_vfscommon.vfs`), `bake` (kit assembly into `bin/`; optional kit names confine the run to those kits - G-121), `bakelist` (report the configured kit matrix with runtime/vfs presence and deployed state), `bin`, `bakehouse` (packages + bake for a clean checkout; `project`/`vfs` remain as deprecated aliases - G-112 rename, achieved), `bootsupport` (snapshot refresh), `vendorupdate`, `check`, `workflow`, `projectversion`. Dispatch and help dogfood `punk::args` (G-030, achieved) and degrade to plain scan/help when bootsupport `punk::args` is stale (`PUNKBOOT_PLAIN=1` forces the degraded mode). Source: `src/AGENTS.md`. - **Host vs target platform (G-122).** The kit surfaces separate what the driving tclsh IS from what a bake is FOR. Target-keyed: the `bin/runtime/` store a runtime is read from, `.exe` suffixing of runtimes and kit outputs, presence checks, and the pre-deploy process sweep's tooling (`tasklist`/`taskkill` vs `ps`/`kill`, skipped when the target's processes cannot exist on this host). Host-keyed: copy commands, path handling, filesystem case rules, prompts. The default target is the host canon except for cygwin-family hosts (msys2/cygwin-runtime tclsh - `tcl_platform(platform)` `unix` on windows), which target `win32-x86_64`; a `mapvfs.config` entry may declare its own target and is then addressed in that platform's tier. `make.tcl check` prints the derivation. Zip-type kits assemble without zipfs in the driving tcl (raw-runtime split + `punk::zip::mkzip` + concatenation), and since G-124 they EXTRACT without it too: `punk::zip` reads a zip - plain, or attached to an executable under either offset convention - with stock Tcl only, so a bake needs neither zipfs nor tcllib to carry a runtime's `tcl_library` into the kit. Sources: `src/AGENTS.md`, `src/runtime/AGENTS.md`, `src/modules/punk/platform-999999.0a1.0.tm`, `src/modules/punk/zip-999999.0a1.0.tm`. - **punkcheck** records every install/delete as events in per-folder `.punkcheck` directories - the basis for skip/copy change detection, superseded-module pruning and provenance. Single OO record lifecycle (G-094, achieved) with atomic saves and advisory event-scoped locking for concurrent writers (G-095, achieved). - **Provenance gates.** Build/promotion commands warn on uncommitted `src/` changes (column-0 `PROVENANCE-WARNING:` token, `-dirty-abort` for strict mode); `vendorupdate` warns for dirty source-project checkouts. - **Boot-precondition gate (G-125).** A bake refuses a kit whose merged `.vfs` supplies no tcl library rather than deploying an artifact that cannot initialise: the kit lands in `FAILED KITS` and nothing is written, so the previously deployed `bin/` survives. Structural and non-executing (so cross-target kits are covered), reading the merged tree rather than the extraction outcome. The predicate is `punkboot::utils::vfs_boot_library_report`, called through the same guarded require as the provenance check so a stale bootsupport snapshot degrades it to a notice; `make.tcl check` reports ACTIVE/UNAVAILABLE. Sources: `src/AGENTS.md`, `src/modules/punkboot/utils-999999.0a1.0.tm`. - **Payload/target consistency checks (G-133 + G-134), all advisory.** At the same post-merge seam, a bake classifies each binary library in the merged tree by header (`punkboot::utils::binary_arch_classify` - PE/ELF/Mach-O, honest unknowns) against the kit's target platform; wrong-arch libraries outside platform-discriminated subdirs (canonical `-` names, vendor spellings like `win-x64`) earn recapped `BUILD-WARNING`s. After assembly, a kit declaring smoke-require packages (`mapvfs.config` 5th entry element) has each one plain-`package require`d inside the freshly built artifact via its tclsh subcommand - the only check that sees resolution-order defects (wrong-arch version shadowing); cross-target kits skip with a stated reason. The assembled image is also probed for its zip offset convention (G-134, `punkboot::utils::kit_offsetstyle_report` over `punk::zip::archive_info`): a FILE-relative attached payload warns (the pipeline emits archive-relative; the G-128 stamper refuses file-relative by default), no-zip/plain results stay silent. Same guarded-require degradation; `make.tcl check` reports all three. None guarantees statics, pure-tcl packages with binary deps, or version preference beyond the declared smoke set. Sources: `src/AGENTS.md`, `src/runtime/AGENTS.md`, `src/modules/punkboot/utils-999999.0a1.0.tm`. - **Kit icon step (G-057 + G-128).** Every kit a bake builds gets a `.resources.toml` sidecar (deployed beside `bin/`) recording the build-time icon choice - default `src/runtime/punkshell.ico`, overridable by a root `punkshell.ico` in the kit's own custom `.vfs` folder (pre-merge) - with sha256 identity and provenance lifted from the icon's G-135 assetorigin record. win32-target kits get the icon embedded as PE `RT_ICON`/`RT_GROUP_ICON` behind a single seam (`::punkboot::kit_icon_process`): the vendored punkres portable stamper (`src/tools/punkres`, built to `bin/punkres(.exe)` by the make.tcl tool step) is selected when present and works from any build host; otherwise the twapi arm (tcl-sfe mechanism) serves windows hosts with nothing built. Both arms share one semantic (delete all icon/group entries, write ids 1..N; the group is NAMED from the icon file's uppercased rootname - `PUNKSHELL` - with language adopted from the replaced group) and stamp a per-kit copy of the payload-free raw runtime prefix BEFORE payload attach; punkres can also stamp a FINISHED kit post-hoc, preserving an appended zip payload (archive-relative moved verbatim; file-relative refused by default, shifted with a consent flag). Non-PE targets skip as not applicable; a host where neither mechanism serves skips with a combined notice naming the punkres build remedy; the sidecar is written in every case. Sources: `src/AGENTS.md`, `bin/AGENTS.md`, `src/vfs/AGENTS.md`, `src/tools/AGENTS.md`. - **Buildsuites and the kit family.** `src/buildsuites/suite_tcl90/` builds Tcl/Tk/tcllib from source with a pinned zig toolchain and produces the runtime kit family (plain / punk / bi) plus artifact metadata (G-096-G-117 era: see archived goals G-096, G-098, G-102, G-103, G-107); artifacts publish to the punkbin repo that `bin/punk-runtime.cmd` fetches from. `src/buildsuites/suite_tcl86/` is the 8.6 sibling (G-099 + G-100, both achieved): a Tcl 8.6 windows runtime (static + dynamic shells, `tcl86t.dll`, on-disk lib tree - no zipfs, so no self-contained kit) with thread 2.8 + tclvfs + Tk 8.6 + tklib + tcllib(+tcllibc critcl accelerators) companions, gated against the 8.6 core/thread/tclvfs testsuites with tcllib/tklib on a record tier and an opt-in `test-tk`; punkshell's own runtests is censused on that runtime at parity with a same-day native-8.6 baseline. Suite child shells scrub `TCL__TM_PATH` as well as TCLLIBPATH/TCL_LIBRARY/TK_LIBRARY - without it a census measures the machine's module trees. The 8.6 kit container strategy remains in-flux - see "In-flux areas". - **Project generation.** `dev project.new` composes thin layouts from `src/project_layouts/` (overlay chain with `.anti` deletion markers and `name@base` derivation - G-087, achieved) and injects bootsupport modules from the generating shell at generation time. - **Workflow overview.** `tclsh src/make.tcl workflow` prints the embedded ASCII data-flow overview of the build/release pipeline, with its own update contract in `src/AGENTS.md`. This section deliberately summarises rather than copies it. ## Test harness - **`src/tests/runtests.tcl` is the entry point.** Path-glob discovery over `testsuites/` with the `X/***`-style lattice (`-include-paths`/`-exclude-paths`, G-093), single-process or multi-process child modes, a `-jobs N` worker pool (G-091), report modes (compact/markdown/json) and an optional UDP watch stream. Source: `src/tests/AGENTS.md`. - **Characterization style.** Suites pin observed current behaviour, including `_GAP_`-named tests that pin known bugs and flip when a goal fixes them - goal acceptance criteria routinely name the pins they flip. - **Child-process suites by design.** A repl cannot run inside the shared testinterp (codethread exit callbacks target the thread's main interp), so console/repl verification drives child processes (`src/tests/testsupport/repl_console_driver.tcl`); kit and launcher behaviour is likewise exec-driven under `src/tests/shell/`. - **Mode parity.** `scriptlib/developer/runtests_parity.tcl` compares `-report json` outputs across runner modes. ## In-flux areas Mechanics here are documented as pointers only - detailed prose would be obsolete on arrival. Read the goal detail files for intended end states and the code for current state. - Subshell roadmap: G-002, G-003, G-008, G-009, G-010 (non-nested subshells, resource limits, scoped console state, themed profiles, tree navigation). - Input mode and completion: G-013 (raw-mode default), G-044 (punk::args-driven completion). - Stored configuration and help: G-014 (punk::config toml), G-042 (subshell help topics), G-043 (subshell definition plugins). - Console err channel: G-011. - Launch surface: G-023 (version-named binaries), G-077 (`-e` one-liners), G-089 (scriptlib in kits). - 8.6 buildsuite family: G-101 (kit container) - the core runtime (G-099) and the Tk/tklib/tcllib dependency set (G-100) are achieved 2026-07-26. ## Doc map - which lane holds what - How it works now (durable mechanics): this file. - What you must do when working here: root `AGENTS.md` + the child AGENTS.md chain (binding). - What changed and why: `GOALS.md`, `GOALS-archive.md`, `goals/` detail files; `CHANGELOG.md` for release-level history. - Per-command and per-proc usage: PUNKARGS blocks in the modules (`i ` at a repl); generated docs under `src/doc/`. - User-facing features: `README.md`.