punkshell/goals/G-055-tclcore-regen-workflow.md

G-055 Agent-driven tclcore moduledoc regeneration workflow with behavioural parity verification

Status: proposed Scope: goals/G-055-tclcore-regen-workflow.md (workflow doc), src/modules/punk/args/moduledoc/tclcore-999999.0a1.0.tm (+ buildversion), src/tests/modules/punk/args/testsuites/args/ (parity pins), TEMP_REFERENCE/tcl9 (read-only source input; source retrieval mechanism deferred to the buildsuites toml configs / G-005 era) Goal: refreshing punk::args::moduledoc::tclcore is a documented, repeatable agent-driven workflow taking a Tcl source tree + version as input - man-page/source text carried verbatim (rearranged into @cmd -help/@examples/choicelabels, not reimagined; exceptions: synopsis notation is translated into punkshell's own more generic synopsis syntax rather than copied from the source docs, and text may be re-folded/line-wrapped to keep help display-width friendly while punk::args lacks word wrapping), one adaptive package with version-conditional definitions where released behaviour differs (8.6 vs 9.x), each regenerated command verified by a real-vs-model behavioural probe before acceptance, and source-checkin provenance recorded - designed so the same workflow extends later to tkcore (which loads on 'package require tk' and relies heavily on shared documentation sections, e.g the (default)::punk::args::moduledoc::tkcore::tk_standardoptions id) and to moduledocs for other core.tcl-lang.org projects (e.g tcludp). Acceptance: the workflow is documented in the detail file (inputs, verbatim-text fidelity policy, probe-verification gate, provenance recording, shared-section reuse guidance for the tkcore pattern); the whole ::string ensemble plus a small selection of other commands including the multi-form ::after have been regenerated or verified through the workflow against the reference Tcl 9 sources, with parity pins added and 8.6 released-behaviour differences honoured (G-054's harvest/version-conditional techniques); an initial scan across tclcore commands identifying constructs not adequately modelable in the current punk::args system is recorded in the detail file, with each gap flagged as a candidate goal rather than worked around silently; tkcore and other-project moduledocs remain out of scope beyond the workflow being demonstrably reusable for them.

Context

punk::args::moduledoc::tclcore was produced manually from the Tcl man pages. A behavioural audit of 'string is' (2026-07-10, driven by the G-049 parse_status machinery) showed the hand modelling to be surprisingly accurate - arity, the "last word is always str" C-parser semantics, option/class prefix acceptance and ambiguity rejection all agree with the real interpreter - but exposed the failure mode manual transcription cannot avoid: version drift in enumerated facts. The static class list is the Tcl 9.0 set, so 'string is dict' is wrongly accepted when running under 8.6 (dict is 8.7+/9.x). Note 8.6, not 8.7, is the real Tcl 8 target: 8.7 is unreleased and unmaintained (its 'unicode' class never shipped in any released Tcl; 9.0 removed it).

The audit method generalizes and is the maintainability answer: doc authoring needs judgement (the man pages are not consistently machine-parsable; some text belongs in @examples, some in choicelabels - rearrangement yes, reimagining no), but doc verification is mechanical - drive the same argument shapes through the real command (safe, deliberately-invalid invocations) and through punk::args::parse_status against the definition, and require error-vs-ok agreement. G-054 applies this tactically to 'string is'; this goal makes it a repeatable workflow.

Approach

Workflow inputs: a Tcl source tree (doc/.n man pages, generic/.c for ground truth) plus its version/checkin id. For now the tree is TEMP_REFERENCE/tcl9 (reference-only fossil checkout); how source trees are fetched (git/fossil, to a settled location) is a buildsuites/G-005-era decision - the workflow only takes "path + version" and must stay indifferent to how the tree arrived.

Per command:

1. Read the man page section and, where behaviour is load-bearing (arity, option windows, enum tables), the C implementation.
2. Author/refresh the punk::args definition block - verbatim text, rearranged into @cmd -help/@examples/argument -help/choicelabels as fits the punk::args idiom. Exception to verbatim: synopsis notation. punkshell deliberately uses its own, more generic synopsis syntax (as rendered by punk::ns::synopsis / @form -synopsis), so synopsis lines and usage examples embedded in the source docs (man-page SYNOPSIS sections, wrong-#-args strings like "class ?-strict? ?-failindex var? str") are TRANSLATED into that syntax, never copied in Tcl's notation. Prose and semantic descriptions stay verbatim. Second exception: line folding. punk::args help rendering currently has no word wrapping, so source text is re-folded manually to keep help blocks terminal-width friendly (the existing convention of short hand-wrapped lines in -help/choicelabel blocks). Folding changes whitespace/line breaks only - words and their order stay verbatim. If punk::args later gains word wrapping, this exception lapses for new work. Enumerations the runtime can answer (class lists etc.) are harvested at define time rather than hard-coded (G-054 technique); released-version differences (8.6 vs 9.x) handled by version-conditional definition in the single adaptive package - no per-major package forks.
3. Verification gate: a probe matrix (valid, incomplete, invalid, prefix, flag-lookalike shapes) run against the real command and parse_status, asserting error-vs-ok agreement; committed as a parity test deriving expectations from the live interpreter, not version arithmetic. Additionally (G-074 achieved 2026-07-13 - goals/archive/G-074-punkargs-multiform-ambiguity-lint.md): run punk::args::formcheck on every regenerated MULTIFORM command - new UNSANCTIONED findings block acceptance (a type_weakness finding means a discriminator faces a permissive type and the forms need tightening or the ambiguity is real; a genuine documented overlap is sanctioned with @form -overlapallowed, as adopted for ::after cancel). The result dict's 'unsanctioned' key is the gate value (must be empty). Known pre-existing exception: ::lseq's range/start_count and range/count findings stay unsanctioned pending an expr syntax-validating type (G-069/G-070).
4. Record provenance: the source checkin id the text was read from, noted with the definition (comment) and in the module changelog.

Modelability scan: an initial pass across tclcore commands to identify constructs the current punk::args system cannot model adequately (e.g exotic clause grammars, context-dependent option windows). Each gap is recorded here and flagged as a candidate goal - not silently approximated in doc blocks.

Proving set for acceptance: the whole ::string ensemble, plus a small selection of other commands including the multi-form ::after.

Notes

• tkcore: a started moduledoc loads on 'package require tk'. It is to be treated with the same workflow later, but relies heavily on shared documentation sections as currently modeled by the (default)::punk::args::moduledoc::tkcore::tk_standardoptions id - the workflow's text-placement guidance must cover shared-section reuse, so common option text is referenced once rather than duplicated per widget.
• Other core.tcl-lang.org projects (e.g core.tcl-lang.org/tcludp) get moduledocs following the same pattern further down the track; out of scope here beyond the workflow being reusable.
• The 'string is' audit artifacts (probe matrices and real-vs-model comparisons on 9.0.3 / 8.7a6 / 8.6.13) that motivated this goal are summarized in the G-049 session work of 2026-07-10; the C ground truth for the option-window semantics is StringIsCmd in generic/tclCmdMZ.c (objv[2..objc-2] option loop - the final word is unconditionally str).
• Probing policy: define-time/verification probing is limited to safe, side-effect-free invocations of pure builtins (deliberately-invalid calls harvesting error metadata). This does not relax cmdhelp's rule of never calling arbitrary commands to elicit usage at query time.
• 2026-07-12 modelability findings from if/switch/try/lseq real-vs-model probing (G-041 prework; all in the over-acceptance class - model reports valid where the real command errors - so parity probes per this workflow's verification gate would catch them):
  • try handler compound types: the current handler clause {literal(on)|literal(trap) string list string} cannot type the second element per-alternate (on wants code choices ok/error/return/break/continue/int; trap wants a pattern list), so 'try {} on bogus {e} {s}' over-accepts. The moduledoc's own embedded notes at ::try sketch candidate mechanisms (named compound types with arity indicators, @typeimport, RPN type expressions).
  • reserved-word awareness in clause allocation: 'if {0} {b} elseif' over-accepts (bare trailing word is a valid else-script EXCEPT when it is a reserved keyword); clause members have no deny-list concept (cf -choiceprefixdenylist for choices).
  • '-' fallthrough constraints (try handler script, switch body): a literal '-' is invalid in the LAST handler/pattern position - a cross-element constraint outside per-element typing.
  • switch two-argument special case: leading dash words are options only when more than two arguments are present ('switch -exact {p b}' means string="-exact") - argument-count-conditional option parsing.
  • -type expr performs no validation (syntactically invalid '3+' accepted, as is the non-expression 'to'); syntax-level checking is the most that is possible (expressions may reference runtime vars/functions) - decide and record whether to add it. Consequence under G-041 form candidacy (found 2026-07-13): 'lseq 1 count 5' reports as ambiguous (range + start_count) because the range form's expr-typed end accepts the word 'count' - real lseq resolves it as the start_count form. Tightening the expr operands (or making them version-conditional per TIP 746 below) removes the false candidate. Probed 2026-07-13 (user suggestion: -type indexexpression instead): a poor lexical fit for lseq operands on 9.0.3 - indexexpression (lindex-style int/end/end+-N/M+-N) UNDER-accepts real-valid operands 1e2, 2*3, 6/2, sqrt(9) and the man page's own {[llength $l]-1} idiom, and OVER-accepts end/end-1 (real lseq rejects); number|indexexpression repairs only the number literals. The precise discriminator is expr SYNTAX validation: the problem words (count/to/end) all fail Tcl's expr PARSER as invalid barewords while every real-valid shape parses - but a validator must parse without evaluating (operand exprs may contain command substitution), i.e tclparser 'parse expr' (G-069/G-070) or equivalent, not catch{expr}. Also noted: count-position operands reject doubles where start/end accept them ('lseq 1.5' errors, 'lseq 1.5 3.5' works) - a value-dependent constraint no lexical type expresses.
  • ::after cancelid/cancelscript discrimination (user suggestion 2026-07-13: per-version probe of the id shape + an id type "starting with after#"). Probed on 9.0.3 and 8.6.11: the id shape is after#N on both (hardcoded "after#%d" in tclTimer.c; harvest at define time anyway via the safe create+cancel probe 'set id [after 999999 {}]; after cancel $id' - G-054 technique, no event loop needed, no output). NO new type needed: the existing stringstartswith(after#) type-alternate is validate-live (not just synopsis display) - typing the cancelid form's id (and the info form's id) with it makes 'after cancel someid' resolve cleanly to cancelscript, matching real behaviour (real 'after cancel ' is a silent script-match no-op; 'after info ' errors at runtime - model-reject vs real-reject stays parity-true). RESIDUAL (truthful): 'after cancel after#12' remains ambiguous in the model because a script can also be after#-shaped - real Tcl resolves by id LIVENESS at runtime (tries id first, falls back to script match), which no static type expresses; the G-074 sanction (shipped 2026-07-13 as @form -overlapallowed, adopted on the cancelid form in tclcore 0.3.4) acknowledges the cancelid/cancelscript pair for formcheck reporting - thanks to the id typing, the only witnesses the pair still shares ARE the id-shaped words. APPLIED 2026-07-13 (tclcore moduledoc 0.3.0, user-directed): both ids typed stringstartswith() via a build-time %AFTERIDPREFIX% string map. CORRECTED note for the workflow (the first record of this wrinkle misattributed it): parse-field tstr ${...} IS expanded for plain registered PUNKARGS definitions - but in the DEFSPACE, which is the argdoc subnamespace whenever one exists, even when the PUNKARGS list lives in the parent namespace (update_definitions evalns rule); a variable set in the parent is unresolvable there and the param is left SILENTLY literal (diagnostic gap - a literal ${...} reaching a parse field is always an authoring error and could warn). Load-time-computed values work via plain tstr placeholders with the variable set in the argdoc namespace (the ::after harvest showcases this since tclcore 0.3.1, user-preferred style), or via build-time substitution (explicit tstr wrap as ::tcl::string::is; string-map token reserved for when build-time substitution is genuinely necessary). Parity pins in tclcoreparity.test including the accepted dead-id divergence.
  • TIP 746 (user-flagged 2026-07-12, https://core.tcl-lang.org/tips/doc/trunk/tip/746.md) removes the expr behaviour from lseq operands in Tcl 9.1: the lseq model's number|expr operand types are correct for 9.0 but must become version-conditional (number-only on 9.1+) - the same adaptive-definition technique as the 'string is' class list (G-054). punkshell's own code already adapted (punk::args 0.3.2 zero_based_posns). Parity probes for lseq must therefore derive expectations from the live interpreter, not from the man page vintage.
• Doc-direction guidance (user, 2026-07-12) for this workflow: punkshell synopses are deliberately richer than the underlying man pages - where the source docs simplify (a single synopsis line hiding genuinely orthogonal usages), the definition SHOULD split into multiple @form entries even though the man page does not present them as forms (lseq range/start_count/count and switch separate/block are the in-tree precedents). This extends the existing synopsis-translation exception: form structure, like synopsis notation, follows punkshell's modelling needs rather than the source document's presentation.
• Archived-goal references in this file: G-049 achieved 2026-07-10 (goals/archive/G-049-punkargs-parse-status-model.md);G-054 achieved 2026-07-11 (goals/archive/G-054-tclcore-stringis-harvest.md); G-041 achieved 2026-07-13 (goals/archive/G-041-punkargs-form-matching.md).