Browse Source
New root ARCHITECTURE.md: descriptive, pointer-first map of boot/launch, interp/thread topology, console abstraction, module loading, build/provenance/kits and the test harness - own lane beside DOX (what you must do) and the goals system (what changed and why). In-flux subsystems documented as goal pointers only. scriptlib/developer/architecture_lint.tcl validates the doc's reference contract: backticked repo paths exist, G-<id> references resolve against both goal indexes, 400-line size budget. Plain tclsh, no deps, mutation-tested. Root AGENTS.md: Child DOX Index entry, Closeout item 5 (update-or-prune described mechanisms + run the linter), scriptlib/developer inventory. Assisted-by: harness=opencode; primary-model=openrouter/moonshotai/kimi-k3; api-location=openrouter.aimaster
3 changed files with 256 additions and 4 deletions
@ -0,0 +1,113 @@
|
||||
# ARCHITECTURE.md - Punkshell System Mechanics |
||||
|
||||
## About this document |
||||
|
||||
This file is the descriptive, current-state map of how punkshell fits together. It occupies its own lane alongside the project's other documentation: |
||||
|
||||
- `AGENTS.md` files (root + children) are binding work contracts: what you must do when working here. |
||||
- `GOALS.md`, `GOALS-archive.md` and `goals/` track intended and landed changes with rationale: what changed and why. |
||||
- This file says how the system works now. |
||||
|
||||
Rules that keep it useful instead of rotten: |
||||
|
||||
1. **Pointer-first.** Sections state durable mechanics briefly and route to canonical sources (module files, owning AGENTS.md, goal detail files). Never transcribe content that lives elsewhere - point at it. If a section wants to grow past a screenful, the detail belongs in the module or a child doc, with this section pointing at it. |
||||
2. **Current state only.** History, rejected alternatives and verification records stay in goal detail files; release-level history stays in `CHANGELOG.md`. Do not narrate change history here. |
||||
3. **Staleness.** When this file conflicts with the code, the code wins and this file is stale. Update or delete the stale section in the same change that breaks it (root `AGENTS.md` Closeout). Delete stale sections rather than annotating them. |
||||
4. **In-flux areas.** Subsystems under active redesign are documented as one-liners with goal pointers, not detailed prose (see "In-flux areas"). |
||||
5. **Verification.** `tclsh scriptlib/developer/architecture_lint.tcl` - backtick-quoted repo paths must exist, `G-<id>` references must resolve against the goal indexes, size budget enforced. Run after editing this file. |
||||
|
||||
## The ten-thousand-foot view |
||||
|
||||
Punkshell is an alternative Tcl shell distributed as self-contained executables ("kits"): a zipfs/metakit-aware Tcl runtime wrapped with a virtual filesystem (VFS) payload carrying the punk module set. A kit boots through a single entry script, which selects a package mode (which module sources to trust) and a subcommand (which application to run: interactive shell, non-interactive script runner, stock tclsh emulation, and others). The interactive shell is a REPL in the main interp that evaluates user code in a separate code thread and talks to the terminal through a two-layer console abstraction. Everything is built, tested and released by a Tcl-driven build system (`src/make.tcl`) with file-level provenance tracking (`punkcheck`). |
||||
|
||||
``` |
||||
bin/<kit>.exe = Tcl runtime (zipfs/metakit) + <name>.vfs payload |
||||
| boots |
||||
v |
||||
src/vfs/_config/punk_main.tcl |
||||
- package mode: dev / os / src / internal (which module sources to trust) |
||||
- subcommand: shell / script / punk / tclsh / shellspy |
||||
| |
||||
v |
||||
app package from src/lib/ (app-punkshell, app-punkscript, app-repl, app-shellspy) |
||||
| |
||||
v |
||||
+------------------- interactive shell --------------------+ |
||||
| REPL interp (punk::repl) <--> console abstraction | |
||||
| + codethread (user code) (punk::console over | |
||||
| + shellthread workers opunk::Console) | |
||||
+----------------------------------------------------------+ |
||||
``` |
||||
|
||||
## Boot and launch chain |
||||
|
||||
- **Kit anatomy.** Each `src/vfs/*.vfs` folder is a runtime payload; `src/runtime/mapvfs.config` maps payload folders to platform runtimes under `bin/runtime/<platform>/`; `src/vfs/_vfscommon.vfs/` is a generated merge of common libraries (never hand-edited); `src/vfs/_config/` holds the entry scripts. Sources: `src/vfs/AGENTS.md`, `src/runtime/AGENTS.md`. |
||||
- **Entry point.** `src/vfs/_config/punk_main.tcl`. An optional first argument is a dash-delimited package mode composed of tokens from `dev`, `os`, `src`, `internal` (`internal` is always appended when absent). The next argument is the subcommand: `shell`, `script`, `punk`, `tclsh`, `shellspy`. Any other first argument is treated as a script invocation; no arguments at all means interactive shell. Subcommands map to app packages under `src/lib/` (`app-punkshell`, `app-punkscript`, `app-repl`, `app-shellspy`). |
||||
- **Package modes pick module provenance.** `internal` uses module snapshots baked into the kit at build time; `dev` loads build outputs from repo-root `modules/` and `lib/`; `src` loads the unbuilt working tree (`src/modules`, `src/lib`, `src/bootsupport`, `src/vendormodules`) with `package prefer latest` so magic-version dev modules (`999999.0a1.0`) beat stamped snapshots. `src` mode is the standard way to verify working-tree changes without a rebuild. Source: `bin/AGENTS.md` "Launch package modes". |
||||
- **Static package capture (G-058, achieved).** Boot probe-loads the runtime's statically linked packages in a throwaway interp and records what each provides; every interp the shell fabricates (boot, codethread, shellthread workers) seeds `package ifneeded <name> <ver> {load {} <prefix>}` from that record, and `punk::packagepreference` resolves static-vs-bundled version-aware. Sources: `src/vfs/AGENTS.md`, `goals/archive/G-058-static-runtime-packages.md`. |
||||
- **`script` subcommand (G-015, achieved).** The lean non-interactive path: default punk shell module/alias environment, no shellfilter stacks/transforms, honest exit codes, launch plumbing emits nothing on stdout/stderr (exec-style callers see only the script's own output). Supports `lib:<name>` scriptlib resolution. App package: `src/lib/app-punkscript/`. |
||||
- **`bin/` launch surface.** Built kits plus polyglot `.cmd` utilities (`punk-runtime`, `punk-getzig`, `punk-tclargs`, `dtplite`) generated from `src/scriptapps/` sources by the `punk::mix` scriptwrap machinery - outputs are never hand-edited. `bin/punk-runtime.cmd` fetches/lists/selects plain runtimes under `bin/runtime/<platform>/` from the punkbin artifact server with sha1 verification. Source: `bin/AGENTS.md`. |
||||
|
||||
## Interp and thread topology |
||||
|
||||
- **REPL interp vs codethread.** `punk::repl` runs the reader/prompt loop in the main interp; user commands evaluate in a separate thread's interp (`src/modules/punk/repl/codethread-999999.0a1.0.tm`) so the reader stays responsive and user-code state is isolated from shell state. |
||||
- **shellfilter stacks.** Channel transform/logging layers. Among other roles, `var` JUNCTION stacks divert the code interp's stdout/stderr into per-run variables that the repl collects and emits to the console after each run (G-001). The API is known-clumsy; audit is proposed as G-090. Source: `src/modules/shellfilter-999999.0a1.0.tm`. |
||||
- **shellthread workers.** Pooled worker threads (logging, tee handlers). tcludp 1.0.12's Windows per-thread exit handler closed process-global events and wedged worker event loops (G-036, achieved); kits now bundle tcludp >= 1.0.13 and a runtime `has_bug`-style check surfaces the vulnerable combination. Worker lifecycle audit is also G-090. Source: `src/modules/shellthread-999999.0a1.0.tm`. |
||||
- **Shared state.** Infrastructure `tsv` arrays are `punk_`-prefixed by contract (legacy unprefixed arrays predate the rule); console facts, ownership and mode caches live in tsv so every thread sees one truth (G-007). Source: `src/modules/AGENTS.md`. |
||||
- **Subshells.** Named secondary repls exist; the substantial subshell roadmap is in-flux - see "In-flux areas". |
||||
|
||||
## Console abstraction |
||||
|
||||
Two layers with a deliberate dependency direction (the class never depends on the integrating layer): |
||||
|
||||
- **`opunk::Console`** (`src/modules/opunk/`) - a `voo` value-OO base class modelling a console as an in/out pair with settled capability facts. Objects are plain Tcl list values; `-virtual` methods dispatch on a class tag so subclasses can override everything channel-related. Cooperative seams are pluggable: the probe-byte store (`waiting_chunks_arrayvar`), the size-query provider (`size_query_provider`), and an anchor lifecycle callback. Source: `src/modules/opunk/AGENTS.md`. |
||||
- **`punk::console`** (`src/modules/punk/console-999999.0a1.0.tm`) - the integrating layer: console-spec resolution, per-console terminal-property facts, raw mode, ANSI emit/query helpers, and registration of the providers the base class consumes. |
||||
- **Spec forms and selection.** A console spec is an `{in out}` channel pair, an anchored instance name, or an `::opunk::Console` object value, normalised by `punk::console::console_spec_resolve`. `repl::init -console <spec>` selects the console per repl (G-001, achieved); the default is the process console `{stdin stdout}`. Source: `src/modules/punk/AGENTS.md`. |
||||
- **Backends (G-001, achieved).** Reference subclasses `opunk::console::test`, `::ssh`, `::tk` (deterministic test double, socket-carried terminal, Tk text widget) prove non-detectable terminal-like devices can host a repl with no base-class or `punk::console` changes. Detail: `goals/archive/G-001-pluggable-console-backends.md`. |
||||
- **Location transparency (G-007, achieved).** Facts are tsv-backed and terminal queries are owner-routed: a query from any thread forwards to the console-owning thread through a `thread::send` choke point, so code-interp code sees the same console the repl sees without `repl eval` round-trips. Ownership lives in a tsv registry keyed by canonical channel pair. |
||||
- **Raw mode.** twapi on Windows when available; otherwise a persistent powershell console-mode named-pipe server serves raw transitions on twapi-less runtimes (G-106, achieved) - quiet, lazy-started, exits with the owning process. |
||||
- **Dead-console handling.** The Tcl 9 Windows console driver never reported a killed host terminal to the script, leaving orphaned shells spinning a core (G-039, achieved; upstream ticket filed and fixed). The repl arms a close-on-probe-failure watchdog only on runtimes a `punk::lib::check` classifier reports as affected (G-076, achieved); runtimes at or past 9.0.5 deliver the dead console as a readable event and use the normal eof teardown. |
||||
|
||||
## Module ecosystem and package loading |
||||
|
||||
- **Editable modules** live in `src/modules/` (+ `src/modules_tcl8/`, `src/modules_tcl9/` siblings); filenames carry the magic dev version `999999.0a1.0` with the real semver in a `<name>-buildversion.txt` sidecar, and namespaces mirror directory layout (`punk/ansi/` = `punk::ansi`). `punk::libunknown` is the manually-versioned exception. Source: `src/modules/AGENTS.md`. |
||||
- **`punk::args` is both parser and documentation system.** `PUNKARGS`/`argdoc` blocks travel with each proc and power inline usage tables (`i <command>` at the repl); even manually-parsing procs carry documentation-only definitions. Much of the goals-era work has been hardening this system - see the `punkargs` entries across `GOALS.md` and `GOALS-archive.md`. |
||||
- **Discovery and modpods.** `punk::libunknown` provides module discovery/registration beyond `tcl::tm` defaults; `#modpod-*` source directories pack into zip-based multi-file `.tm` modules at build time (src mode registers them via an inline boot scanner written in builtins). |
||||
- **Bootsupport.** `src/bootsupport/` holds snapshot copies of build-time-critical modules (manifest: `include_modules.config`), analogous to devDependencies. `make.tcl` classifies snapshot staleness for five runtime-critical packages (`punkcheck`, `punk::repo`, `punk::mix`, `punk::tdl`, `punk::args`) as abort/prompt/warn from the version delta. Source: `src/bootsupport/AGENTS.md`. |
||||
- **opunk layer.** `src/modules/opunk/` explores `voo`-based value-OO reimplementations (objects as plain Tcl values); the `punk::*` modules remain the production implementations. |
||||
- **Vendored code.** Third-party packages live in `src/vendormodules/` and `src/vendorlib/` (+ `_tcl8`/`_tcl9` siblings), refreshed by `make.tcl vendorupdate` from config; never hand-edited (enforcement policy is proposed as G-026). |
||||
|
||||
## Build, provenance and kits |
||||
|
||||
- **`src/make.tcl` is the single build entry.** Key subcommands: `modules`/`libs`/`packages` (repo-root build outputs), `vfscommonupdate` (regenerate `_vfscommon.vfs`), `vfs`, `bin`, `project` (full kit build into `bin/`), `bootsupport` (snapshot refresh), `vendorupdate`, `check`, `workflow`, `projectversion`. Dispatch and help dogfood `punk::args` (G-030, achieved) and degrade to plain scan/help when bootsupport `punk::args` is stale (`PUNKBOOT_PLAIN=1` forces the degraded mode). Source: `src/AGENTS.md`. |
||||
- **punkcheck** records every install/delete as events in per-folder `.punkcheck` directories - the basis for skip/copy change detection, superseded-module pruning and provenance. Single OO record lifecycle (G-094, achieved) with atomic saves and advisory event-scoped locking for concurrent writers (G-095, achieved). |
||||
- **Provenance gates.** Build/promotion commands warn on uncommitted `src/` changes (column-0 `PROVENANCE-WARNING:` token, `-dirty-abort` for strict mode); `vendorupdate` warns for dirty source-project checkouts. |
||||
- **Buildsuites and the kit family.** `src/buildsuites/suite_tcl90/` builds Tcl/Tk/tcllib from source with a pinned zig toolchain and produces the runtime kit family (plain / punk / bi) plus artifact metadata (G-096-G-117 era: see archived goals G-096, G-098, G-102, G-103, G-107); artifacts publish to the punkbin repo that `bin/punk-runtime.cmd` fetches from. The 8.6 family and kit container strategy are in-flux - see "In-flux areas". |
||||
- **Project generation.** `dev project.new` composes thin layouts from `src/project_layouts/` (overlay chain with `.anti` deletion markers and `name@base` derivation - G-087, achieved) and injects bootsupport modules from the generating shell at generation time. |
||||
- **Workflow overview.** `tclsh src/make.tcl workflow` prints the embedded ASCII data-flow overview of the build/release pipeline, with its own update contract in `src/AGENTS.md`. This section deliberately summarises rather than copies it. |
||||
|
||||
## Test harness |
||||
|
||||
- **`src/tests/runtests.tcl` is the entry point.** Path-glob discovery over `testsuites/` with the `X/***`-style lattice (`-include-paths`/`-exclude-paths`, G-093), single-process or multi-process child modes, a `-jobs N` worker pool (G-091), report modes (compact/markdown/json) and an optional UDP watch stream. Source: `src/tests/AGENTS.md`. |
||||
- **Characterization style.** Suites pin observed current behaviour, including `_GAP_`-named tests that pin known bugs and flip when a goal fixes them - goal acceptance criteria routinely name the pins they flip. |
||||
- **Child-process suites by design.** A repl cannot run inside the shared testinterp (codethread exit callbacks target the thread's main interp), so console/repl verification drives child processes (`src/tests/testsupport/repl_console_driver.tcl`); kit and launcher behaviour is likewise exec-driven under `src/tests/shell/`. |
||||
- **Mode parity.** `scriptlib/developer/runtests_parity.tcl` compares `-report json` outputs across runner modes. |
||||
|
||||
## In-flux areas |
||||
|
||||
Mechanics here are documented as pointers only - detailed prose would be obsolete on arrival. Read the goal detail files for intended end states and the code for current state. |
||||
|
||||
- Subshell roadmap: G-002, G-003, G-008, G-009, G-010 (non-nested subshells, resource limits, scoped console state, themed profiles, tree navigation). |
||||
- Input mode and completion: G-013 (raw-mode default), G-044 (punk::args-driven completion). |
||||
- Stored configuration and help: G-014 (punk::config toml), G-042 (subshell help topics), G-043 (subshell definition plugins). |
||||
- Console err channel: G-011. |
||||
- Launch surface: G-023 (version-named binaries), G-033 (`proj:` package mode), G-077 (`-e` one-liners), G-089 (scriptlib in kits), G-112 (make.tcl subcommand rename), G-118 (tclsh subcommand review). |
||||
- 8.6 buildsuite family: G-099, G-100, G-101. |
||||
|
||||
## Doc map - which lane holds what |
||||
|
||||
- How it works now (durable mechanics): this file. |
||||
- What you must do when working here: root `AGENTS.md` + the child AGENTS.md chain (binding). |
||||
- What changed and why: `GOALS.md`, `GOALS-archive.md`, `goals/` detail files; `CHANGELOG.md` for release-level history. |
||||
- Per-command and per-proc usage: PUNKARGS blocks in the modules (`i <command>` at a repl); generated docs under `src/doc/`. |
||||
- User-facing features: `README.md`. |
||||
@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env tclsh |
||||
# architecture_lint.tcl - lint the root ARCHITECTURE.md descriptive map |
||||
# |
||||
# Validates the front-matter contract declared in ARCHITECTURE.md: |
||||
# - every backtick-quoted repo-relative path reference exists |
||||
# (anchored tokens only - see $path_anchors / $root_files below; |
||||
# tokens with glob/placeholder characters are patterns, not paths) |
||||
# - every G-<id> reference resolves to a goal in GOALS.md or |
||||
# GOALS-archive.md |
||||
# - size budget: the file stays a cheap full read (<= $max_lines lines) |
||||
# - line 1 is a '# ' title |
||||
# |
||||
# Plain tclsh (8.6+ or 9), no package dependencies - runnable from any checkout: |
||||
# tclsh scriptlib/developer/architecture_lint.tcl [repo-root] |
||||
# Exit 0 when clean; exit 1 with one 'path: problem' line per finding. |
||||
|
||||
set root [expr {$argc >= 1 ? [lindex $argv 0] : [file dirname [file dirname [file dirname [file normalize [info script]]]]]}] |
||||
set archfile [file join $root ARCHITECTURE.md] |
||||
if {![file isfile $archfile]} { |
||||
puts stderr "architecture_lint: '$root' does not look like the punkshell repo root (need ARCHITECTURE.md)" |
||||
exit 2 |
||||
} |
||||
|
||||
set findings {} |
||||
proc flag {path msg} { |
||||
global findings |
||||
lappend findings "$path: $msg" |
||||
} |
||||
|
||||
proc read_lines {path} { |
||||
set f [open $path r] |
||||
fconfigure $f -encoding utf-8 |
||||
set lines [split [read $f] \n] |
||||
close $f |
||||
return $lines |
||||
} |
||||
|
||||
set max_lines 400 |
||||
set path_anchors {src/ bin/ goals/ scriptlib/ .fossil-settings/} |
||||
set root_files {AGENTS.md ARCHITECTURE.md GOALS.md GOALS-archive.md CHANGELOG.md README.md LICENSE.txt punkproject.toml CLAUDE.md tclint.toml} |
||||
|
||||
# --- collect goal ids from both indexes -------------------------------------- |
||||
set goalids [dict create] |
||||
foreach indexname {GOALS.md GOALS-archive.md} { |
||||
set indexpath [file join $root $indexname] |
||||
if {![file isfile $indexpath]} { |
||||
flag $indexname "goal index missing - cannot resolve G-<id> references" |
||||
continue |
||||
} |
||||
foreach line [read_lines $indexpath] { |
||||
if {[regexp {^### G-(\d+) \[} $line -> id]} { |
||||
dict set goalids $id 1 |
||||
} |
||||
} |
||||
} |
||||
|
||||
# --- classify one backtick token ---------------------------------------------- |
||||
# returns: skip | file | dir |
||||
proc classify_token {tok} { |
||||
global path_anchors root_files |
||||
# patterns and placeholders are not literal paths |
||||
if {[regexp {[*<>$%\\ ()`;:'",]} $tok]} {return skip} |
||||
if {[string index $tok 0] eq "-"} {return skip} |
||||
set anchored 0 |
||||
foreach anchor $path_anchors { |
||||
if {[string match "$anchor*" $tok]} { |
||||
set anchored 1 |
||||
break |
||||
} |
||||
} |
||||
if {!$anchored && $tok ni $root_files} {return skip} |
||||
if {[string index $tok end] eq "/"} {return dir} |
||||
return file |
||||
} |
||||
|
||||
# --- run ---------------------------------------------------------------------- |
||||
set lines [read_lines $archfile] |
||||
set nlines [llength $lines] |
||||
if {$nlines > $max_lines} { |
||||
flag ARCHITECTURE.md "$nlines lines exceeds the $max_lines-line budget - the file must stay a cheap full read; move detail into owning docs and point at it" |
||||
} |
||||
if {![regexp {^# } [lindex $lines 0]]} { |
||||
flag ARCHITECTURE.md "line 1 is not a '# ' title" |
||||
} |
||||
|
||||
# path references (unique; first-mention line reported) |
||||
set seen_paths [dict create] |
||||
# goal references (unique; first-mention line reported) |
||||
set seen_goals [dict create] |
||||
set lineno 0 |
||||
foreach line $lines { |
||||
incr lineno |
||||
foreach {- tok} [regexp -all -inline {`([^`\n]+)`} $line] { |
||||
set kind [classify_token $tok] |
||||
if {$kind eq "skip"} continue |
||||
if {![dict exists $seen_paths $tok]} { |
||||
dict set seen_paths $tok [list $kind $lineno] |
||||
} |
||||
} |
||||
foreach gid [regexp -all -inline {G-\d+} $line] { |
||||
if {![dict exists $seen_goals $gid]} { |
||||
dict set seen_goals $gid $lineno |
||||
} |
||||
} |
||||
} |
||||
|
||||
dict for {tok info} $seen_paths { |
||||
lassign $info kind lineno |
||||
if {$kind eq "dir"} { |
||||
set tok [string range $tok 0 end-1] |
||||
} |
||||
set full [file join $root $tok] |
||||
if {$kind eq "dir"} { |
||||
if {![file isdirectory $full]} { |
||||
flag ARCHITECTURE.md "line $lineno: referenced directory '$tok/' does not exist" |
||||
} |
||||
} else { |
||||
if {![file exists $full]} { |
||||
flag ARCHITECTURE.md "line $lineno: referenced path '$tok' does not exist" |
||||
} |
||||
} |
||||
} |
||||
|
||||
dict for {gid lineno} $seen_goals { |
||||
set num [string range $gid 2 end] |
||||
if {![dict exists $goalids $num]} { |
||||
flag ARCHITECTURE.md "line $lineno: $gid does not resolve to a goal in GOALS.md or GOALS-archive.md" |
||||
} |
||||
} |
||||
|
||||
if {[llength $findings]} { |
||||
puts [join $findings \n] |
||||
puts stderr "architecture_lint: [llength $findings] finding(s)" |
||||
exit 1 |
||||
} |
||||
puts "architecture_lint: clean ([dict size $seen_paths] path refs, [dict size $seen_goals] goal refs, $nlines lines)" |
||||
exit 0 |
||||
Loading…
Reference in new issue