# src/modules — Main Editable Module Source

## Purpose

Source of truth for all editable Punk project modules. This is where agents should make generic module source edits. Each `.tm` file is a Tcl module belonging to a named package.

## Ownership

- Agents should edit files here as the primary development area for generic modules.
- The top-level modules (non-punk namespace) include shellfilter, shellrun, overtype, textblock, patternpunk, and related packages.
- The `punk/` subdirectory holds all `punk::*` namespace modules.
- Tcl-major-specific sibling trees under `src/modules_tcl8/` and `src/modules_tcl9/` mirror these module conventions where applicable, but their local AGENTS.md files control source selection for version-specific behavior.

## Local Contracts

- Module filenames use the literal suffix `-999999.0a1.0.tm`.
- Corresponding `<modulename>-buildversion.txt` files hold the real version number.
- The exception is `punk::libunknown`, which is manually versioned: its real `major.minor.patch` version lives in the filename (`libunknown-<version>.tm`) and there is no buildversion.txt. The same bump rules apply as for buildversion-tracked modules (see "Versioning And Releases"); the mechanics differ — see the manual-versioning bullet there.
- `#modpod-*` directories contain internal files packed into `.tm` archives during build; do not flatten or edit them without understanding the modpod format.
- `_build/` directory holds build intermediates and should not be manually edited.
- Always declare dependencies explicitly using `package require <name>` near file tops.
- Prefer fully qualified namespaces when referencing external packages, such as `package require tcl::zlib` or `package require TclOO`.
- Organize custom modules as namespaces mirroring directory structure, such as `namespace eval punk::<modulename>` or deeper paths like `punk::lib::util::<somename>`.
- Use semantic versions that `package vcompare` can interpret; strip leading zeros according to tcl package version rules ie in each segment including either side of a or b.
- For optional features, probe with `if {[catch {package require foo}]} { ... }` and degrade gracefully.
- Vendormodules, bootsupport snapshots, VFS modules, and built output modules do not use the magic version scheme unless their own local docs explicitly say otherwise.
- tsv shared-array names are process-global and subshell application code may use tsv for its own purposes: infrastructure tsv arrays must be `punk_`-prefixed (e.g `punk_console`, `punk_console_facts`). Legacy unprefixed arrays still in use (`repl`, `codethread_*`, `zzzload_pkg*`) predate this rule; rename them only as a coordinated change across all referencing modules.

### Module Subsystems

- Terminal/display modules include `punk::ansi` for ANSI codes, cp437, and Sixel support; `punk::console`; `textblock` for text composition and framing; and `overtype` for text overlay.
- Shell modules include `punk::repl`, `punk::shellrun` for `run`/`runout`/`runerr`/`runx`, and `punk::nav::ns` for namespace browsing.
- Functional and pattern modules include `pattern` and `patterncmd`; treat them as experimental and use them only with user instruction or confirmation.
- Argument handling uses `punk::args` for processing, validation, and documentation generation; prefer it for public APIs and user-facing utilities unless the code path is performance critical.
- Performance-critical procs may parse arguments manually, but must still carry an accurate `punk::args` (PUNKARGS) definition for documentation, with a comment in the proc referencing the definition id; use `punk::args::parse` whenever speed isn't critical or manual parsing would be complex.
- Networking modules include `punk::imap4`, `punk::netbox`, `punk::basictelnet`, and `punk::net::vxlan`.
- Plugin and capability provider/handler support uses `punk::cap`.

## Work Guidance

- Treat `src/modules/` as the source of truth for generic editable Punk modules.
- Expect the literal suffix `-999999.0a1.0.tm` and do not shorten or normalize it.
- Prefer Unix-style LF line endings for `.tm` source files in this tree.
- If the same proc exists in both `src/modules/` and `src/modules_tcl<major>/`, prefer `src/modules/` unless version-specific behavior is relevant or only the version-specific file is active.
- Use `deck module.new <name>` or `punk::mix::commandset::module::new` to scaffold new modules.
- Run `tclsh src/make.tcl modules` to build modules, or `tclsh src/make.tcl project` for a full build.
- When adding a new proc to a `.tm` file, add a PUNKARGS `argdoc` block immediately before it — even if the proc parses arguments manually and the PUNKARGS is documentation-only. "User-facing" includes any proc in an exported namespace, not just shell commands. Developer-facing utility procs (e.g `punk::lib::*`) are not exempt.

### Formatting And Layout

- Opening braces stay on the same line for procs.
- Multiline control structures may span lines for readability if Tcl syntax allows without line-continuation backslashes, such as braced `expr` contents or `if` expressions.
- Line-continuation backslashes should only be used in documentation blocks for `punk::args`, such as `punk::args::define` arguments or `lappend PUNKARGS` elements.
- In `PUNKARGS` and `punk::args::define` text blocks, preserve the embedded documentation/parser indentation already used inside the block; do not indent added help text further just to match surrounding Tcl nesting.
- Proc bodies should not use line-continuation backslashes because they make debug line-number matching harder; use Tcl expand syntax where practical. Do not write hybrid forms such as `dict create {*}{} \`.
- For literal-only Tcl values, prefer one braced literal expanded into the command; do not add per-line empty-expansion continuation markers inside the literal.
- These expand-style patterns apply to dictionaries, lists, and other commands with many arguments. Preserve argument order whenever command semantics, list order, duplicate-key behavior, or display order matters.
- For mixed literal and substituted values where element order does not matter, group reorderable literal values into one expanded braced literal and put substituted values on aligned continuation lines.
- When substituted values or nested commands require Tcl expand-style continuation inside one command argument list, vertically align each trailing `{*}{` marker in that list.
- When entries in a many-argument command need inline documentation, an empty command substitution marker such as `{*}[ ... ]` may be used between arguments so comments can live inside the command. Comments may be on their own lines or at the end of marker lines. Keep only whitespace and comments inside those square brackets; do not put executable Tcl there. Align trailing `{*}[` markers the same way as trailing `{*}{` markers.
- Before closeout for `.tcl` or `.tm` edits that add or alter proc bodies, search modified files for trailing `\` and confirm every remaining hit is inside allowed `punk::args` documentation text.
- For test-specific continuation rules, use `src/tests/AGENTS.md`.
- Keep pipelines readable by aligning `% var = ...` and `pipecase` segments when practical.
- Keep inline comments concise and describe intent, not mechanics.
- Document non-trivial procedures and exports with the standard `argdoc`/`PUNKARGS` template below.
- For documentation-only `lappend PUNKARGS` metadata that must not add a `package require punk::args` dependency or use `punk::args::define`, include the top-level `::punk::args::register` block shown in the module template so `punk::args` can discover the metadata later.

Example expand-style dictionary literal; no per-line continuation markers are needed for literal values:

```tcl
dict create {*}{
    a A
    b B
    c C
}
```

Example expand-style dictionary command with mixed literal and substituted values where key order does not matter:

```tcl
dict create             {*}{
    literal_a A
    literal_c C
}   dynamic_b $BVAL     {*}{
}   dynamic_d [get_d]   {*}{
}
```

Example documented dictionary command using empty command substitutions for comments:

```tcl
dict create                   {*}[
    # Padding character.
]   PAD [list \x80 $pad]      {*}[
    # Start of string.
]   SOS [list \x98 $sos]      {*}[
    # Control sequence introducer.
]   CSI [list \x9b $csi]      {*}[
]
```

### Naming Conventions

- Procedures: `lowercase_with_underscores` for internals; `camelCase` is allowed for public APIs where existing patterns fit.
- Variables: `lowercase_with_underscores`; avoid single-letter names except for loop indices.
- Namespaces: mirror directory structure; nested modules should reflect filesystem hierarchy.
- Private helpers: prefix with `_`, such as `_resolve_stream`; do not export them.
- Constants: `UPPER_CASE_WITH_UNDERSCORES` declared through `namespace eval { variable CONSTANT value }` when practical.
- Keep `namespace export` lists alphabetized for clarity.

### Procedure Documentation Template Guide

```tcl
    #- comments such as this with a leading dash are descriptions for the agent of how-to/what-to to implement
    #- whereas comments without the dash are either literal comments, or if angle-bracketed, descriptions of what to put in the comment.
    namespace eval argdoc {
        lappend PUNKARGS [list {
            @id     -id     ::full::namespace::path::procedure_name
            @cmd    -name   "full::namespace::path::procedure_name"\
                -summary\
                    "Short single line procedure description"\
                -help\
                    "Procedure summary without too much overlap with documented
                    arguments below.

                    Existing
                    - include description of return value"
            @leaders
                arg1        -type any -optional 0 -multiple 0 -help\
                    "description of mandatory leading argument arg1"
            @opts
                -force      -type boolean
                -flag1      -type none -help\
                    "description of non-argument accepting flag -flag1"
                --          -type none -help\
                    "End of opts marker"
            @values -min 1 -max -1
                arg2        -type list      -optional 0 -help\
                    {Description of mandatory arg arg2
                      possibly spanning multiple lines, but each line should
                      generally be wrapped manually to keep sensibly short
                      for terminal display.
                      Help blocks may be wrapped in curly braces or double quotes.
                      See the punk::args module for details of acceptable structure.
                      }
                glob        -type string    -optional 1 -multiple 1 -help\
                    "Description of optional argument glob which can be repeated.
                      Indenting of argument -help contents should follow this example.
                      Note that we indent subsequent lines by 2 additional spaces,
                      whereas the @cmd -help does no such indenting. This provides
                      a more presentable display when calling `i procedure_name`."
        }]
    }
    proc procedure_name {args} {
        # <optional summary line to aid debug during viewing of proc body using tools like `cmdtrace` or `corp procedure_name`>
        #- agent: these comments are not literal contents of the proc - they are to be read as a guide.
        #- argument parsing implementation:
        #-  If not highly performance sensitive use punk::args to parse,
        #-    make use of `punk::args::parse $args withid ::full::namespace::path::procedurename`
        #-  namespace eval ::punk::args::register {
        #-      #use fully qualified so 8.6 doesn't find existing var in global namespace
        #-      lappend ::punk::args::register::NAMESPACES ::punk::ansi ::punk::ansi::ansistring ::punk::ansi::colour ::punk::ansi::argdoc
        #-  }
        #-  If performance sensitive or not part of a broad-concept of 'API' for the module and useful utilities,
        #-    use fast `switch` based code to parse the arguments.

        #- procedure functionality implementation
    }
```

Update the template with concrete details whenever functions are user-facing or complex.

### Argument Order In PUNKARGS

The `@leaders`, `@opts`, and `@values` sections are not interchangeable — they determine the position of arguments in the synopsis and the expected parsing order:

- **`@leaders`** — arguments that appear before opts (e.g `interp` in `interp_sync_package_paths interp -libunknown 1`). Use these when the proc signature has a fixed leading argument before optional flags, even if the proc parses args manually with `{interp args}`.
- **`@opts`** — flags that appear after leaders (and before values).
- **`@values`** — trailing arguments that appear after opts.

Putting a leading argument in `@values` instead of `@leaders` produces an incorrect synopsis (e.g `[-libunknown <bool>] interp` instead of `interp [-libunknown <bool>]`). Match the PUNKARGS section to the actual call site order, not just to which arguments are "required" vs "optional".

### Error Handling And Logging

- Prefer `try { ... } on error {result options} { ... }` for structured Tcl 8.6+ error handling.
- Fallback pattern:

```tcl
if {[catch {some_command} result]} {
    puts stderr "Error: $result"
    return -code error $result
}
```

- For Punk pipelines, wrap risky commands inside `pipecase` blocks and emit descriptive messages via `puts stderr` or Punk logging helpers.
- Never swallow errors silently; propagate with context so shell users see actionable details.

### Pipeline And Functional Style Notes

- Punk Pipeline functionality is experimental and should usually be avoided in agent-generated code unless the user specifically requests it.
- Pipeline syntax includes forms such as `var.= command1 |> command2 |> command3`.
- Pattern-match assignment syntax includes forms such as `x@0,y@1,z@2.= list a b c`, where `@` targets an index and `/` is range-safe.
- Destructuring examples include `x@end/@@k1.= list {k1 aaa} {k1 bb}`.
- `.=` means following args form a command; `=` accepts a single value.
- Use `% var = ...` bindings to capture intermediate values; keep names meaningful.
- `pipecase` should list specific patterns before catch-alls to avoid hidden matches.
- `fun name pattern { ... }` definitions should remain side-effect light; treat them as pure functions unless otherwise documented.
- Keep pipelines short and composable; extract helper procs or `fun` definitions when they exceed about 10 logical steps.

### Module Structure Expectations

```tcl
# <Module description>
package require <dependencies>
#- comments such as this with a leading dash are descriptions for the agent of how-to/what-to to implement
#- whereas comments without the dash are either literal comments, or if angle-bracketed, descriptions of what to put in the comment.

namespace eval <module_namespace> {
    variable version <semver>
    namespace export public_proc1 public_proc2

    namespace eval argdoc {
        lappend PUNKARGS [list {
            @id     -id     <module_namespace>::public_proc1
            ... elided ...
            #- punk::args textblock similar to example given in Procedure Documentation Template
            #- This should exist immediately prior to each proc (or at least very close),
            #- even if it is only being used for documentation and not for parsing.
            #- If the public_proc1 proc does not use the PUNKARGS definition for parsing, then the parsing within the proc
            #- and the punk::args definitions of proc args and behaviour must be kept synchronized.
        }]
    }
    proc public_proc1 {args} {
        #- argument parsing code, using `punk::args::parse` if part of API and not performance critical
        #- if argument parsing is done in code here, the behaviour must match the arguments as defined in the PUNKARGS list created above.

        #- Implementation
    }

    namespace eval argdoc {
        lappend PUNKARGS [list {
            @id     -id     <module_namespace>::some_other_public_proc
            ... elided ...
        }]
    }
    proc some_other_public_proc {args} {
        ... elided ...
    }

    namespace eval private {
        # <basic description of namespace - optional but recommended>
        #- use of same PUNKARGS mechanism within a further nested argdoc namespace is recommended for clarity
        #- even within private namespaces.
        #- Generally privately namespaced procs will use the PUNKARGS for documentation only,
        #- unless there is no concern about the usually slight overhead of punk::args::parse.

        proc private_helper {arg1 arg2} {
            #- Private implementation
        }

        proc private_helper2 {args} {
            #- argument parsing code, using `switch`, `regexp`, etc if needed.

            #- Private implementation
        }
    }

    namespace eval utils {
        # <basic description of namespace - optional but recommended>
        #- other appropriately named sub namespaces such as 'utils' or 'lib' can be used to structure the codebase.
        #- If these form part of what can reasonably be considered the API of the module, they should
        #- be documented using the punk::args mechanism described above.

        #- more procs
    }
}

namespace eval ::punk::args::register {
    #use fully qualified so 8.6 doesn't find existing var in global namespace
    lappend ::punk::args::register::NAMESPACES <module_namespace> <module_namespace>::utils
}

# Required when PUNKARGS metadata is inert documentation and the module does not
# directly require punk::args or call punk::args::define. Register the module/API
# namespaces that punk::args should scan for PUNKARGS or PUNKARGS_aliases data.

##  Ready
package provide <module_namespace> [tcl::namespace::eval <module_namespace> {
    variable version
    #- this version number, exactly 999999.0a1.0, is a literal used in src module folders
    #- we refer to this sometimes as the magic version number
    set version 999999.0a1.0
}]
return
```

### Type And Data Handling

- Tcl is dynamically typed; emulate structural typing through argument validation at proc boundaries.
- Validate user inputs with `punk::args::parse`, `switch -exact`, `regexp`, or Punk pipeline predicates before mutation.
- Use dictionaries for structured data, or parallel lists if it is likely to be a performance win.
- Tcl arrays can be used, but limit them to simple flat structures or cases where they are specifically warranted.
- When bridging to binary data, such as ANSI/xbin parsing, document expected encodings and conversions.

### Versioning And Releases

- Stick to semantic versioning: `major.minor.patch`.
- Use the magic version `999999.0a1.0` for source modules and add or update the actual version number in the associated `<modulename>-buildversion.txt` file.
- Bump `<modulename>-buildversion.txt` whenever the module's API or behavior changes:
  - **Patch** (e.g 0.1.0 → 0.1.1): bug fixes, internal refactors with no API change, or documentation-only updates to PUNKARGS.
  - **Minor** (e.g 0.1.0 → 0.2.0): new flags, procs, or backward-compatible API additions.
  - **Major** (e.g 1.0.0 → 2.0.0): breaking API changes (removed flags, changed return shapes, renamed procs without aliases).
- When bumping, append changelog comment lines below the version documenting each change, prefixed with the new version number. e.g:
  ```
  0.2.0
  #First line must be a semantic version number
  #all other lines are ignored.
  #0.2.0 - merged -exclude-dirsegments + -antiglob_paths into unified -exclude-paths (backward-compat aliases retained)
  #0.1.2 - fixed path_relative delegation edge case on Windows
  ```
  This keeps the changelog discoverable alongside the version number without requiring a separate CHANGES file.
- **Bootstrap-tracked files only** — `punkcheck-buildversion.txt`, `punk/repo-buildversion.txt`, `punk/mix-buildversion.txt` (the top-level `punk::mix` file only; `punk::mix::util`, `punk::mix::cli`, and `punk/mix/commandset/*` version independently and are not covered by this check), and `punk/tdl-buildversion.txt`: bump at least **minor** whenever a call site inside that file is updated to use a new API, even if the file's own interface is unchanged. `src/make.tcl` reads only these four files to classify bootsupport staleness (major=abort, minor=prompt, patch=silent-proceed); under-bumping one of them to patch when the call-site change is more significant causes `make.tcl` to mis-classify and either wrongly proceed without prompting or wrongly abort. See `src/bootsupport/AGENTS.md` "Bootsupport Staleness Handling" for the full contract.
- **All other modules** (including `punk::mix` submodules): call-site updates follow ordinary judgement from the Patch/Minor/Major rules above — a behavior-preserving call-site change is a patch (or no bump if genuinely a no-op); reserve minor for changes that add capability to the module's *own* API. `src/make.tcl` does not read these versions, so there is no automated consequence, but keep the changelog accurate since it is the only record of the module's real semantic version.
- **Manually versioned modules** (currently only `punk::libunknown`): the real `major.minor.patch` version is the filename suffix and the `package provide` block value; there is no `<name>-buildversion.txt` and the build does not stamp a version. The Patch/Minor/Major bump rules above apply identically — an agent changing such a module bumps by (1) renaming the file (`git mv`) to the new version, (2) updating the `# Application <name> <version>` Meta line, the doctools `manpage_begin` version and the provide-block `set version`, and (3) appending a changelog comment line to the version-history block in the module header (it substitutes for the buildversion.txt changelog). Before the first bump of such a module, verify nothing requires it by exact version or hardcoded filename (for punk::libunknown: punk_main.tcl and punk::repl glob `libunknown-*.tm` and pick the highest by vcompare, bootsupport's include_modules.config lists it by name only, and all requires are unversioned - verified 2026-07-11). New modules should use the magic version mechanism instead.
- Modules with the magic version number must not appear in output paths such as `<projectdir>/modules`.
- When referencing ranges, use bounded specs such as `1.2.3-2.0.0`.
- Convert loose versions to bounded form in module metadata; helper utilities exist in boot modules for this purpose.

### Documentation And Comments

- Primary documentation for procs should use the `punk::args` system, preferably within an `argdoc` sub-namespace using `lappend PUNKARGS <list of textblocks>`.
- "User-facing" in the Procedure Documentation Template Guide means any proc in an exported namespace or any proc a developer might call — not just shell commands. Developer-facing utility procs are not exempt.
- If a public proc parses arguments manually, keep the implementation behavior synchronized with its PUNKARGS definition.
- Update relevant docs or usage notes when behavior changes.
- Mention environment variables or flags required to run new features.

## Verification

- VS Code Tcl lint diagnostics are clean for modified `.tm` files when available.
- New procs in modified `.tm` files have PUNKARGS `argdoc` blocks (documentation-only is acceptable if the proc parses args manually).
- `tclsh src/make.tcl modules` completes without errors for module changes.
- Module tests pass: `tclsh src/tests/modules/<path>/tests/all.tcl`.

## Child DOX Index

- `punk/` — Core punk namespace modules (see punk/AGENTS.md)
- `opunk/` — Alternative punk namespace, voo-based classes (opunk::str, opunk::console) (see opunk/AGENTS.md)
- `punkcheck/` — Build/check system (punkcheck + punkcheck::cli)
- `test/` — Installed-module test packages (see test/AGENTS.md)