# ARCHITECTURE.md - Punkshell System Mechanics ## About this document This file is the descriptive, current-state map of how punkshell fits together. It occupies its own lane alongside the project's other documentation: - `AGENTS.md` files (root + children) are binding work contracts: what you must do when working here. - `GOALS.md`, `GOALS-archive.md` and `goals/` track intended and landed changes with rationale: what changed and why. - This file says how the system works now. Rules that keep it useful instead of rotten: 1. **Pointer-first.** Sections state durable mechanics briefly and route to canonical sources (module files, owning AGENTS.md, goal detail files). Never transcribe content that lives elsewhere - point at it. If a section wants to grow past a screenful, the detail belongs in the module or a child doc, with this section pointing at it. 2. **Current state only.** History, rejected alternatives and verification records stay in goal detail files; release-level history stays in `CHANGELOG.md`. Do not narrate change history here. 3. **Staleness.** When this file conflicts with the code, the code wins and this file is stale. Update or delete the stale section in the same change that breaks it (root `AGENTS.md` Closeout). Delete stale sections rather than annotating them. 4. **In-flux areas.** Subsystems under active redesign are documented as one-liners with goal pointers, not detailed prose (see "In-flux areas"). 5. **Verification.** `tclsh scriptlib/developer/architecture_lint.tcl` - backtick-quoted repo paths must exist, `G-` references must resolve against the goal indexes, size budget enforced. Run after editing this file. ## The ten-thousand-foot view Punkshell is an alternative Tcl shell distributed as self-contained executables ("kits"): a zipfs/metakit-aware Tcl runtime wrapped with a virtual filesystem (VFS) payload carrying the punk module set. A kit boots through a single entry script, which selects a package mode (which module sources to trust) and a subcommand (which application to run: interactive shell, non-interactive script runner, stock tclsh emulation, and others). The interactive shell is a REPL in the main interp that evaluates user code in a separate code thread and talks to the terminal through a two-layer console abstraction. Everything is built, tested and released by a Tcl-driven build system (`src/make.tcl`) with file-level provenance tracking (`punkcheck`). ``` bin/.exe = Tcl runtime (zipfs/metakit) + .vfs payload | boots v src/vfs/_config/punk_main.tcl - package mode: dev / os / src / internal (which module sources to trust) - subcommand: shell / script / punk / tclsh / shellspy | v app package from src/lib/ (app-punkshell, app-punkscript, app-repl, app-shellspy) | v +------------------- interactive shell --------------------+ | REPL interp (punk::repl) <--> console abstraction | | + codethread (user code) (punk::console over | | + shellthread workers opunk::Console) | +----------------------------------------------------------+ ``` ## Boot and launch chain - **Kit anatomy.** Each `src/vfs/*.vfs` folder is a runtime payload; `src/runtime/mapvfs.config` maps payload folders to platform runtimes under `bin/runtime//`; `src/vfs/_vfscommon.vfs/` is a generated merge of common libraries (never hand-edited); `src/vfs/_config/` holds the entry scripts. Sources: `src/vfs/AGENTS.md`, `src/runtime/AGENTS.md`. - **Entry point.** `src/vfs/_config/punk_main.tcl`. An optional first argument is a dash-delimited package mode composed of tokens from `dev`, `os`, `src`, `internal` (`internal` is always appended when absent), optionally scoped with the `proj:` prefix (G-033: `dev`/`src` resolve against the project containing the cwd - walk-up to the nearest git/fossil repo root with a punkshell-style src tree - instead of the executable's own project; discovery is always reported, never a silent rebind). The next argument is the subcommand: `shell`, `script`, `punk`, `tclsh`, `shellspy`. Any other first argument is treated as a script invocation; no arguments at all means interactive shell. Subcommands map to app packages under `src/lib/` (`app-punkshell`, `app-punkscript`, `app-repl`, `app-shellspy`). - **Package modes pick module provenance.** `internal` uses module snapshots baked into the kit at build time; `dev` loads build outputs from repo-root `modules/` and `lib/`; `src` loads the unbuilt working tree (`src/modules`, `src/lib`, `src/bootsupport`, `src/vendormodules`) with `package prefer latest` so magic-version dev modules (`999999.0a1.0`) beat stamped snapshots. `src` mode is the standard way to verify working-tree changes without a rebuild. Source: `bin/AGENTS.md` "Launch package modes". - **Payload mount derivation (G-129, achieved).** The boot keys zipfs presence on `tcl::zipfs::mount` and derives where the executable's attached archive actually mounted from the mount table itself (the entry whose archive file is the executable; fallback: the mount containing the boot script) instead of assuming `tcl::zipfs::root` + `//zipfs:/app`. Modern runtimes still answer `//zipfs:/app`; the androwish/undroidwish 8.6 backport has no `root` command, mounts at the executable's own path, and boots the same payload through its `app/main.tcl` in-archive hook (`src/vfs/punk8win.vfs` carries fauxlinks for both conventions). A mounted archive the boot cannot attribute is reported on stderr rather than silently yielding no internal paths. The derivation procs stay defined post-boot (`::punkboot::zipfs_kit_mountbase`). Sources: `src/vfs/AGENTS.md`, `bin/AGENTS.md` "Kit-wrappable runtime requirements", `src/tests/shell/testsuites/punkexe/kitmountpoint.test`. - **Static package capture (G-058, achieved).** Boot probe-loads the runtime's statically linked packages in a throwaway interp and records what each provides; every interp the shell fabricates (boot, codethread, shellthread workers) seeds `package ifneeded {load {} }` from that record, and `punk::packagepreference` resolves static-vs-bundled version-aware. Sources: `src/vfs/AGENTS.md`, `goals/archive/G-058-static-runtime-packages.md`. - **`script` subcommand (G-015, achieved).** The lean non-interactive path: default punk shell module/alias environment, no shellfilter stacks/transforms, honest exit codes, launch plumbing emits nothing on stdout/stderr (exec-style callers see only the script's own output). Supports `lib:` scriptlib resolution. App package: `src/lib/app-punkscript/`. - **`tclsh` subcommand (G-118, achieved).** Plain-tclsh emulation with NO punk modules loaded: stock argument-form parity (`-encoding name file`; any other leading `-` argument means no script, everything stays in `::argv`; no `-e` one-liner - deliberate stock parity, see G-077 for the punk-level one-liner direction), `lib:` refused with a pointer at `script`, missing script file is a clean exit 1. On piperepl-patched runtimes (the punk family runtimes - `src/buildsuites/suite_tcl90/patches/`) the C-level machinery publishes `::tclsh` launch state (`istty`, `dorepl`, `evalinput`, `inputbuffer`, `reopened`) allowing piped input to hand over to a live console repl at eof; runtimes without the machinery evaluate piped stdin whole-buffer with honest exit codes and fail fast on console stdin. The subcommand-family contract (tclsh/script/shell/punk/shellspy) is documented as punk::args definitions in `src/modules/punk/args/moduledoc/punkexe-999999.0a1.0.tm` (ids `(script)::punkexe` + per-subcommand; view via `punk::args::usage`), consumable by the future launcher help wiring (G-032). Pipe-testable behaviour pinned by `src/tests/shell/testsuites/punkexe/tclshcmd.test`. - **`bin/` launch surface.** Built kits plus polyglot `.cmd` utilities (`punk-runtime`, `punk-getzig`, `punk-tclargs`, `dtplite`) generated from `src/scriptapps/` sources by the `punk::mix` scriptwrap machinery - outputs are never hand-edited. `bin/punk-runtime.cmd` fetches/lists/selects plain runtimes under `bin/runtime//` from the punkbin artifact server with sha1 verification. Source: `bin/AGENTS.md`. ## Interp and thread topology - **REPL interp vs codethread.** `punk::repl` runs the reader/prompt loop in the main interp; user commands evaluate in a separate thread's interp (`src/modules/punk/repl/codethread-999999.0a1.0.tm`) so the reader stays responsive and user-code state is isolated from shell state. - **shellfilter stacks.** Channel transform/logging layers. Among other roles, `var` JUNCTION stacks divert the code interp's stdout/stderr into per-run variables that the repl collects and emits to the console after each run (G-001). The API is known-clumsy; audit is proposed as G-090. Source: `src/modules/shellfilter-999999.0a1.0.tm`. - **shellthread workers.** Pooled worker threads (logging, tee handlers). tcludp 1.0.12's Windows per-thread exit handler closed process-global events and wedged worker event loops (G-036, achieved); kits now bundle tcludp >= 1.0.13 and a runtime `has_bug`-style check surfaces the vulnerable combination. Worker lifecycle audit is also G-090. Source: `src/modules/shellthread-999999.0a1.0.tm`. - **Shared state.** Infrastructure `tsv` arrays are `punk_`-prefixed by contract (legacy unprefixed arrays predate the rule); console facts, ownership and mode caches live in tsv so every thread sees one truth (G-007). Source: `src/modules/AGENTS.md`. - **Subshells.** Named secondary repls exist; the substantial subshell roadmap is in-flux - see "In-flux areas". ## Console abstraction Two layers with a deliberate dependency direction (the class never depends on the integrating layer): - **`opunk::Console`** (`src/modules/opunk/`) - a `voo` value-OO base class modelling a console as an in/out pair with settled capability facts. Objects are plain Tcl list values; `-virtual` methods dispatch on a class tag so subclasses can override everything channel-related. Cooperative seams are pluggable: the probe-byte store (`waiting_chunks_arrayvar`), the size-query provider (`size_query_provider`), and an anchor lifecycle callback. Source: `src/modules/opunk/AGENTS.md`. - **`punk::console`** (`src/modules/punk/console-999999.0a1.0.tm`) - the integrating layer: console-spec resolution, per-console terminal-property facts, raw mode, ANSI emit/query helpers, and registration of the providers the base class consumes. - **Spec forms and selection.** A console spec is an `{in out}` channel pair, an anchored instance name, or an `::opunk::Console` object value, normalised by `punk::console::console_spec_resolve`. `repl::init -console ` selects the console per repl (G-001, achieved); the default is the process console `{stdin stdout}`. Source: `src/modules/punk/AGENTS.md`. - **Backends (G-001, achieved).** Reference subclasses `opunk::console::test`, `::ssh`, `::tk` (deterministic test double, socket-carried terminal, Tk text widget) prove non-detectable terminal-like devices can host a repl with no base-class or `punk::console` changes. Detail: `goals/archive/G-001-pluggable-console-backends.md`. - **Location transparency (G-007, achieved).** Facts are tsv-backed and terminal queries are owner-routed: a query from any thread forwards to the console-owning thread through a `thread::send` choke point, so code-interp code sees the same console the repl sees without `repl eval` round-trips. Ownership lives in a tsv registry keyed by canonical channel pair. - **Raw mode.** twapi on Windows when available; otherwise a persistent powershell console-mode named-pipe server serves raw transitions on twapi-less runtimes (G-106, achieved) - quiet, lazy-started, exits with the owning process. - **Dead-console handling.** The Tcl 9 Windows console driver never reported a killed host terminal to the script, leaving orphaned shells spinning a core (G-039, achieved; upstream ticket filed and fixed). The repl arms a close-on-probe-failure watchdog only on runtimes a `punk::lib::check` classifier reports as affected (G-076, achieved); runtimes at or past 9.0.5 deliver the dead console as a readable event and use the normal eof teardown. ## Module ecosystem and package loading - **Editable modules** live in `src/modules/` (+ `src/modules_tcl8/`, `src/modules_tcl9/` siblings); filenames carry the magic dev version `999999.0a1.0` with the real semver in a `-buildversion.txt` sidecar, and namespaces mirror directory layout (`punk/ansi/` = `punk::ansi`). `punk::libunknown` is the manually-versioned exception. Source: `src/modules/AGENTS.md`. - **`punk::args` is both parser and documentation system.** `PUNKARGS`/`argdoc` blocks travel with each proc and power inline usage tables (`i ` at the repl); even manually-parsing procs carry documentation-only definitions. Much of the goals-era work has been hardening this system - see the `punkargs` entries across `GOALS.md` and `GOALS-archive.md`. - **Discovery and modpods.** `punk::libunknown` provides module discovery/registration beyond `tcl::tm` defaults; `#modpod-*` source directories pack into zip-based multi-file `.tm` modules at build time (src mode registers them via an inline boot scanner written in builtins). - **Bootsupport.** `src/bootsupport/` holds snapshot copies of build-time-critical modules (manifest: `include_modules.config`), analogous to devDependencies. `make.tcl` classifies snapshot staleness for five runtime-critical packages (`punkcheck`, `punk::repo`, `punk::mix`, `punk::tdl`, `punk::args`) as abort/prompt/warn from the version delta. Source: `src/bootsupport/AGENTS.md`. - **opunk layer.** `src/modules/opunk/` explores `voo`-based value-OO reimplementations (objects as plain Tcl values); the `punk::*` modules remain the production implementations. - **Vendored code.** Third-party packages live in `src/vendormodules/` and `src/vendorlib/` (+ `_tcl8`/`_tcl9` siblings), refreshed by `make.tcl vendorupdate` from config; never hand-edited (enforcement policy is proposed as G-026). ## Build, provenance and kits - **`src/make.tcl` is the single build entry.** Key subcommands: `modules`/`libs`/`packages` (repo-root build outputs), `vfscommonupdate` (regenerate `_vfscommon.vfs`), `bake` (kit assembly into `bin/`; optional kit names confine the run to those kits - G-121), `bakelist` (report the configured kit matrix with runtime/vfs presence and deployed state), `bin`, `bakehouse` (packages + bake for a clean checkout; `project`/`vfs` remain as deprecated aliases - G-112 rename, achieved), `bootsupport` (snapshot refresh), `vendorupdate`, `check`, `workflow`, `projectversion`. Dispatch and help dogfood `punk::args` (G-030, achieved) and degrade to plain scan/help when bootsupport `punk::args` is stale (`PUNKBOOT_PLAIN=1` forces the degraded mode). Source: `src/AGENTS.md`. - **Host vs target platform (G-122).** The kit surfaces separate what the driving tclsh IS from what a bake is FOR. Target-keyed: the `bin/runtime/` store a runtime is read from, `.exe` suffixing of runtimes and kit outputs, presence checks, and the pre-deploy process sweep's tooling (`tasklist`/`taskkill` vs `ps`/`kill`, skipped when the target's processes cannot exist on this host). Host-keyed: copy commands, path handling, filesystem case rules, prompts. The default target is the host canon except for cygwin-family hosts (msys2/cygwin-runtime tclsh - `tcl_platform(platform)` `unix` on windows), which target `win32-x86_64`; a `mapvfs.config` entry may declare its own target and is then addressed in that platform's tier. `make.tcl check` prints the derivation. Zip-type kits assemble without zipfs in the driving tcl (raw-runtime split + `punk::zip::mkzip` + concatenation), and since G-124 they EXTRACT without it too: `punk::zip` reads a zip - plain, or attached to an executable under either offset convention - with stock Tcl only, so a bake needs neither zipfs nor tcllib to carry a runtime's `tcl_library` into the kit. Sources: `src/AGENTS.md`, `src/runtime/AGENTS.md`, `src/modules/punk/platform-999999.0a1.0.tm`, `src/modules/punk/zip-999999.0a1.0.tm`. - **punkcheck** records every install/delete as events in per-folder `.punkcheck` directories - the basis for skip/copy change detection, superseded-module pruning and provenance. Single OO record lifecycle (G-094, achieved) with atomic saves and advisory event-scoped locking for concurrent writers (G-095, achieved). - **Provenance gates.** Build/promotion commands warn on uncommitted `src/` changes (column-0 `PROVENANCE-WARNING:` token, `-dirty-abort` for strict mode); `vendorupdate` warns for dirty source-project checkouts. - **Boot-precondition gate (G-125).** A bake refuses a kit whose merged `.vfs` supplies no tcl library rather than deploying an artifact that cannot initialise: the kit lands in `FAILED KITS` and nothing is written, so the previously deployed `bin/` survives. Structural and non-executing (so cross-target kits are covered), reading the merged tree rather than the extraction outcome. The predicate is `punkboot::utils::vfs_boot_library_report`, called through the same guarded require as the provenance check so a stale bootsupport snapshot degrades it to a notice; `make.tcl check` reports ACTIVE/UNAVAILABLE. Sources: `src/AGENTS.md`, `src/modules/punkboot/utils-999999.0a1.0.tm`. - **Payload/target consistency checks (G-133 + G-134), all advisory.** At the same post-merge seam, a bake classifies each binary library in the merged tree by header (`punkboot::utils::binary_arch_classify` - PE/ELF/Mach-O, honest unknowns) against the kit's target platform; wrong-arch libraries outside platform-discriminated subdirs (canonical `-` names, vendor spellings like `win-x64`) earn recapped `BUILD-WARNING`s. After assembly, a kit declaring smoke-require packages (`mapvfs.config` 5th entry element) has each one plain-`package require`d inside the freshly built artifact via its tclsh subcommand - the only check that sees resolution-order defects (wrong-arch version shadowing); cross-target kits skip with a stated reason. The assembled image is also probed for its zip offset convention (G-134, `punkboot::utils::kit_offsetstyle_report` over `punk::zip::archive_info`): a FILE-relative attached payload warns (the pipeline emits archive-relative; the G-128 stamper refuses file-relative by default), no-zip/plain results stay silent. Same guarded-require degradation; `make.tcl check` reports all three. None guarantees statics, pure-tcl packages with binary deps, or version preference beyond the declared smoke set. Sources: `src/AGENTS.md`, `src/runtime/AGENTS.md`, `src/modules/punkboot/utils-999999.0a1.0.tm`. - **Kit icon step (G-057 + G-128).** Every kit a bake builds gets a `.resources.toml` sidecar (deployed beside `bin/`) recording the build-time icon choice - default `src/runtime/punkshell.ico`, overridable by a root `punkshell.ico` in the kit's own custom `.vfs` folder (pre-merge) - with sha256 identity and provenance lifted from the icon's G-135 assetorigin record. win32-target kits get the icon embedded as PE `RT_ICON`/`RT_GROUP_ICON` behind a single seam (`::punkboot::kit_icon_process`): the vendored punkres portable stamper (`src/tools/punkres`, built to `bin/punkres(.exe)` by the make.tcl tool step) is selected when present and works from any build host; otherwise the twapi arm (tcl-sfe mechanism) serves windows hosts with nothing built. Both arms share one semantic (delete all icon/group entries, write ids 1..N; the group is NAMED from the icon file's uppercased rootname - `PUNKSHELL` - with language adopted from the replaced group) and stamp a per-kit copy of the payload-free raw runtime prefix BEFORE payload attach; punkres can also stamp a FINISHED kit post-hoc, preserving an appended zip payload (archive-relative moved verbatim; file-relative refused by default, shifted with a consent flag). Non-PE targets skip as not applicable; a host where neither mechanism serves skips with a combined notice naming the punkres build remedy; the sidecar is written in every case. Sources: `src/AGENTS.md`, `bin/AGENTS.md`, `src/vfs/AGENTS.md`, `src/tools/AGENTS.md`. - **Buildsuites and the kit family.** `src/buildsuites/suite_tcl90/` builds Tcl/Tk/tcllib from source with a pinned zig toolchain and produces the runtime kit family (plain / punk / bi) plus artifact metadata (G-096-G-117 era: see archived goals G-096, G-098, G-102, G-103, G-107); artifacts publish to the punkbin repo that `bin/punk-runtime.cmd` fetches from. `src/buildsuites/suite_tcl86/` is the 8.6 sibling (G-099 + G-100, both achieved): a Tcl 8.6 windows runtime (static + dynamic shells, `tcl86t.dll`, on-disk lib tree - no zipfs, so no self-contained kit) with thread 2.8 + tclvfs + Tk 8.6 + tklib + tcllib(+tcllibc critcl accelerators) companions, gated against the 8.6 core/thread/tclvfs testsuites with tcllib/tklib on a record tier and an opt-in `test-tk`; punkshell's own runtests is censused on that runtime at parity with a same-day native-8.6 baseline. Suite child shells scrub `TCL__TM_PATH` as well as TCLLIBPATH/TCL_LIBRARY/TK_LIBRARY - without it a census measures the machine's module trees. The 8.6 kit container strategy remains in-flux - see "In-flux areas". - **Project generation.** `dev project.new` composes thin layouts from `src/project_layouts/` (overlay chain with `.anti` deletion markers and `name@base` derivation - G-087, achieved) and injects bootsupport modules from the generating shell at generation time. - **Workflow overview.** `tclsh src/make.tcl workflow` prints the embedded ASCII data-flow overview of the build/release pipeline, with its own update contract in `src/AGENTS.md`. This section deliberately summarises rather than copies it. ## Test harness - **`src/tests/runtests.tcl` is the entry point.** Path-glob discovery over `testsuites/` with the `X/***`-style lattice (`-include-paths`/`-exclude-paths`, G-093), single-process or multi-process child modes, a `-jobs N` worker pool (G-091), report modes (compact/markdown/json) and an optional UDP watch stream. Source: `src/tests/AGENTS.md`. - **Characterization style.** Suites pin observed current behaviour, including `_GAP_`-named tests that pin known bugs and flip when a goal fixes them - goal acceptance criteria routinely name the pins they flip. - **Child-process suites by design.** A repl cannot run inside the shared testinterp (codethread exit callbacks target the thread's main interp), so console/repl verification drives child processes (`src/tests/testsupport/repl_console_driver.tcl`); kit and launcher behaviour is likewise exec-driven under `src/tests/shell/`. - **Mode parity.** `scriptlib/developer/runtests_parity.tcl` compares `-report json` outputs across runner modes. ## In-flux areas Mechanics here are documented as pointers only - detailed prose would be obsolete on arrival. Read the goal detail files for intended end states and the code for current state. - Subshell roadmap: G-002, G-003, G-008, G-009, G-010 (non-nested subshells, resource limits, scoped console state, themed profiles, tree navigation). - Input mode and completion: G-013 (raw-mode default), G-044 (punk::args-driven completion). - Stored configuration and help: G-014 (punk::config toml), G-042 (subshell help topics), G-043 (subshell definition plugins). - Console err channel: G-011. - Launch surface: G-023 (version-named binaries), G-077 (`-e` one-liners), G-089 (scriptlib in kits). - 8.6 buildsuite family: G-101 (kit container) - the core runtime (G-099) and the Tk/tklib/tcllib dependency set (G-100) are achieved 2026-07-26. ## Doc map - which lane holds what - How it works now (durable mechanics): this file. - What you must do when working here: root `AGENTS.md` + the child AGENTS.md chain (binding). - What changed and why: `GOALS.md`, `GOALS-archive.md`, `goals/` detail files; `CHANGELOG.md` for release-level history. - Per-command and per-proc usage: PUNKARGS blocks in the modules (`i ` at a repl); generated docs under `src/doc/`. - User-facing features: `README.md`.