punkshell/goals/archive/G-045-punkargs-authoring-ergonomics.md

G-045 punk::args definition authoring ergonomics: record continuation, @cmd unindented fields, constructed-definition normalization

Status: achieved 2026-07-12 Scope: src/modules/punk/args-999999.0a1.0.tm (record parsing in resolve, tstr interplay, arg_error @cmd rendering), src/tests/modules/punk/args/testsuites/ (rendering.test/defquoting.test as the safety net), src/modules/punk-999999.0a1.0.tm (::punk::helptopic::define_docs de-hacked as the consumer proof) Goal: authoring punk::args definitions no longer requires backslash line-continuations or ad-hoc workarounds for multi-line records and constructed definitions - a parser-recognised record-continuation mechanism (candidate: an unquoted trailing -& token, with the detail file recording the collision analysis and the element-count disambiguation alternative), -unindentedfields honoured for @cmd fields, and constructed (string-built) definitions able to opt into the same whole-block indent normalization file-style definitions get - with the container quoting rules (braced=literal, quoted=Tcl backslash semantics, ${ escape) promoted from defquoting.test into the define documentation. Acceptance: a definition using the chosen record-continuation mechanism parses identically to its backslash-continuation equivalent (existing definitions unchanged - continuation is additive), with the token's collision rules documented and an escape/rejection story for values that legitimately match it; @cmd -help/-summary honour -unindentedfields (the rendering.test GAP rendering_unindentedfields_cmd_help_GAP flips to aligned); a constructed definition can request whole-block normalization so embedded continuation indentation behaves as in file-style definitions (the rendering_constructed_def_indent_characterization expectations updated to the chosen semantics), and ::punk::helptopic::define_docs drops its manual pre-normalization to prove it; the quoting rules from defquoting.test appear in the punk::args::define -help documentation; the full punk::args suite (128 tests incl. the rendering invariants: nesting independence, relative-indent preservation) passes with GAP tests flipped, none weakened.

Context

Drafted 2026-07-09 after the tests-first characterization pass (user direction: implement tests before changes in this area). The pain points, in the user's framing:

• The define-block syntax deliberately avoids Tcl dict syntax for human readability, which led to backslash line-continuations and a complex interplay between record parsing in punk::args::resolve and tstr (especially ${...} expansions keeping whitespace aligned during multiline substitutions).
• Constructed (string-built) definitions - ::punk::helptopic::define_docs and punk::args::ensemble_subcommands_definition are the in-tree examples - have poor ergonomics for getting indentation right: unlike file-style definitions they receive no whole-block indent normalization (proven: rendering.test rendering_constructed_def_indent_characterization - continuations at exactly 4 spaces align, other depths leak), so builders pre-normalize by hand (undent/trim hacks, \n-relative authoring).
• A key property of normal definitions that must be preserved: they do not depend on the indentation of the code block they appear in (proven and pinned: rendering_nesting_independence_plain / _tstr).
• The user's experiment used an ad-hoc -- flag followed by a braced line-spanning empty string as a de-facto record continuation - functional but not designed-for, and unsatisfying.

The -& record-continuation candidate (implemented 2026-07-12 - see Progress increment 3)

The user's proposal: a special token the record parser recognises as record continuation, e.g. an unquoted -& at end of line. Design considerations recorded for implementation:

• Collision risk is low but not zero (a value or flag legitimately spelled -&). Candidate rules: only an UNQUOTED trailing -& at end of a record line counts; and/or use the element count of the assembled record to disambiguate (an odd element count where pairs are expected implies the trailing token is a continuation marker, not a value). The detail decision must include an escape story for a literal trailing -& value (bracing it suffices if 'unquoted' is part of the rule).
• Detection must happen in the record parser (resolve) before/alongside line splitting - the same place backslash-continuation assembly effectively happens via Tcl's own parse of braced blocks today. Unlike backslash-newline, -& survives inside braced blocks of CONSTRUCTED definitions too - which is precisely why it helps: string-built defs cannot use backslash-newline ergonomically (it is consumed by the building code's own quoting).

Decision as implemented (punk::args 0.7.0):

• Rule: unquoted trailing -& only - the token must be a bare word preceded by whitespace (or be the whole line) and be the last element on the line; trailing whitespace after it is tolerated (deliberately more forgiving than raw backslash-newline, whose invisible-trailing-whitespace failure is a classic trap).
• Escape story: brace or double-quote a literal trailing -& value ({-&}) - the raw line then ends with the closing delimiter and never matches. A -& mid-line, as a word suffix (abc-&), or on a line inside a still-open braced/quoted value is data.
• Element-count disambiguation: NOT implemented. The positional rule is deterministic and locally decidable per line; element counting would require assembling and list-parsing the record first (fragile against in-progress records) and gives confusing action-at-a-distance when a distant key/value slips the count. Rejected rather than deferred - the escape story covers the residual collision (a literal trailing -default -& must be braced).
• Assembly semantics: byte-identity with the backslash equivalent. Key finding: for braced (file-style) definitions Tcl itself collapses backslash-newline plus following whitespace to a single space BEFORE the text reaches the record splitter - so -& is implemented the same way (token dropped, single-space join, next line's leading whitespace collapsed), making the assembled record byte-identical to its backslash-continued twin. Proven by parse-result and rendered-table equality in recordcontinuation.test.
• Reserved-word consequence: an argument cannot be NAMED -& via a record line ending in the bare token (e.g. a lone '-& ' line would read as a continuation). Any such need is met by bracing. Considered acceptable and documented here rather than guarded in code.

@cmd -unindentedfields

Accepted but ignored today (arg_error's cmd_info rendering carries an '#unindentedfields ?' todo; pinned by rendering_unindentedfields_cmd_help_GAP: left-margin-authored @cmd help renders its first line +4 vs continuations). Honouring it makes left-margin authoring available for @cmd -help/-summary as it already is for argument -help and -choicelabels.

Constructed-definition normalization

Options to decide at implementation (record the choice):

• an explicit define option/directive requesting whole-block undent of the supplied text
• or automatic common-prefix normalization when a definition arrives as a single text block (risk: changing existing constructed defs' rendering - the characterization test documents current expectations to update deliberately) ::punk::helptopic::define_docs currently pre-normalizes by hand (undent/trim + \n-relative help strings + -unindentedfields) - it converts to the new mechanism as the proof, and the user's uncommitted experimental hack in punk.tm is superseded.

Quoting documentation

defquoting.test pinned the container rules (braced values fully literal - $, [], two-char \n, bare backslashes; quoted values get Tcl backslash semantics - \n newline, \ -> \ - with $/[] still literal; ${...} renders a literal ${...} in tstr-processed blocks). Promote these rules into the punk::args::define -help documentation so documenters (e.g. punk::imap4's {\Deleted}/{$MDNSent} choice values) have them stated, not just tested.

Alternatives considered

• Keeping backslash continuations + manual pre-normalization as the documented way - rejected: the friction is proven by two in-tree constructed-def workarounds and the user's punk.tm experiments.
• Tcl dict syntax for definitions - rejected long ago by design (readability).
• The ad-hoc -- + braced-gap trick as the blessed continuation - rejected: exploits parser behaviour that was never a contract.

Notes

• Safety net: src/tests/modules/punk/args/testsuites/args/rendering.test (15 tests: nesting independence, relative-indent preservation, unindentedfields, constructed-def characterization, multiline tstr insertions, @dynamic) and defquoting.test (container quoting rules). GAPs to flip here: rendering_unindentedfields_cmd_help_GAP, rendering_constructed_def_indent_characterization (expectations updated to chosen semantics).
• Related: G-046 (deferred -help resolution and rendering fixes - same code neighbourhood; sequencing between the two goals is free but they touch resolve together), G-044 (completion consumes the documentation corpus these ergonomics grow).
• Session records 2026-07-09: probe scripts renderprobe*.tcl (scratchpad), the user's experimental define_docs hack (uncommitted punk.tm working-tree diff at the time of drafting - superseded by this goal when implemented).
• Archived-goal references in this file: G-046 achieved 2026-07-10 (goals/archive/G-046-punkargs-deferred-help-and-fixes.md).

Progress

2026-07-12 increment 1: @cmd -unindentedfields honoured (punk::args 0.6.1)

• arg_error's cmd-help display transform (undent " "+help, max 4 - the '#unindentedfields ?' todo site) is now gated by "-help" membership in the @cmd line's -unindentedfields list, mirroring the existing per-argument gate. The single transform site feeds both the table and string renderers.
• rendering_unindentedfields_cmd_help_GAP flipped to rendering_unindentedfields_cmd_help (expects aligned; both renderers measured).
• @cmd -summary: verified no renderer applies an indent transform to -summary (it is only used inline in synopsis "# ..." lines), so -unindentedfields membership for -summary is accepted and vacuously honoured - nothing to gate. This satisfies the acceptance's "-help/-summary honour" clause for -summary.
• Blast radius checked before the change: no in-tree definition sets -unindentedfields on a @cmd line, so no existing rendering changed.
• define doc for -unindentedfields now states where the option is valid (argument lines and the @cmd directive's -help).
• Verified (tclsh 9.0.3): full punk::args suite 175 pass / 1 pre-existing skip / 0 fail; punk::ns suite 53/53 (arg_error consumer); make.tcl modules builds clean.

2026-07-12 increment 2: 'i help' alignment via -unindentedfields (punk 0.2.4)

• ::punk::helptopic::define_docs now authors its help text at the left margin and declares -unindentedfields {-help} on the generated @cmd line (using increment 1's gate) and on the topic argument line (a gate that existed all along but was never applied here). Verified in punk902z src: the Description block's +12 continuation leak and the topic Help first-line +4 are both gone; 'i help' renders flush end-to-end. punk::ns suite 53/53; make.tcl modules clean.
• Decision (user, 2026-07-12): when the constructed-def normalization mechanism lands, define_docs converts from its interim left-margin authoring to indented-plus-normalized authoring - it remains the acceptance's consumer proof as written. Left-margin authoring via -unindentedfields stays a supported style proven by tests (rendering_unindentedfields_arg_help, rendering_unindentedfields_cmd_help), but it is not the preferred style for define_docs itself.

2026-07-12 increment 3: -& record continuation (punk::args 0.7.0)

• Implemented in private::split_definition_records per the decision recorded in "The -& record-continuation candidate" section above (unquoted trailing token, brace escape, element-count alternative rejected, byte-identical assembly to the backslash equivalent).
• First implementation attempt kept a literal backslash-newline in the assembled record and failed: braced definitions never contain backslash-newline by the time they reach the splitter (Tcl pre-collapses it), and downstream record handling assumes the collapsed shape. The collapse-to-single-space rewrite is the correct equivalence and is what landed.
• New testsuite recordcontinuation.test (6 tests): backslash-twin parse+render equality (same-length ids so table geometry matches), constructed-def chaining incl. into a multi-line quoted value, braced {-&} escape, mid-line -&, word-suffix abc-&, -& inside still-open braces.
• define doc documents -& alongside the backslash continuation paragraph.
• Verified (tclsh 9.0.3): full punk::args suite 181 pass / 1 pre-existing skip / 0 fail; full source-tree suite (splitter regression sweep) 815 total, 801 pass, 13 skip, 1 fail = exec-14.3 only (the known pre-existing core-test baseline) - zero regressions.
• tclparser considered and rejected for the splitter (user question 2026-07-12): parse command mis-tokenizes the dialect's legitimate unbalanced-bracket ANSI data inside quoted values (info complete gets a throwaway ansistripped copy; parse would need strip+range-remapping), the punk semantics (base-indent line trimming, -&) sit outside Tcl's grammar either way, and a binary dependency in punk::args inverts G-004 for the module that must load everywhere. tclparser remains right for real-script analysis (punk::lib/G-019). Candidate flagged: dev-time lint cross-checking splitter boundaries against parse where the binary is available.

2026-07-12 increment 4: @normalize constructed-def normalization (punk::args 0.8.0, punk 0.2.5)

• Mechanism (user-approved 2026-07-12): bare @normalize directive (like @dynamic; options are an error), detected as a resolve pre-pass over the split records. Semantics user-confirmed as re-base-to-+4; narrowed during implementation to BLOCK-FORM values only (first line whitespace-only): the structural first newline and a whitespace-only trailing line are dropped (the user's leading-newline ergonomics question - resolved as automatic), the content lines' common leading whitespace is the base, first content line unindented fully, subsequent lines re-based to 4 spaces with deeper relative indents preserved. -unindentedfields fields exempt.
• Why block-form only: head-form values have an unknowable base - continuations uniformly at 6 may be base-4 with the deliberate +2 relative convention (P2) or base-6 flush; common-prefix re-basing flattens the former. The idempotence test caught exactly this on a file-style def, forcing the narrowing. Block form is unambiguous (the first content line carries the full base), and head-form untouchedness makes @normalize a proven no-op on conforming file-style definitions.
• Consumer proof: define_docs converted from interim left-margin authoring to indented block-form values + @normalize (per the increment 2 decision); -unindentedfields declarations dropped; 'i help' and 'i help_chunks' verified aligned in punk902z src (including the blank-line separator in the combined basehelp+extra block).
• rendering_constructed_def_indent_characterization stays pinned as the deliberate unopted default, its description/header updated to reference @normalize as the opt-in remedy (the acceptance's "expectations updated to the chosen semantics" - the chosen semantics being opt-in, unopted behaviour is contract, not gap).
• New testsuite normalize.test (5 tests): block-form re-base, block-form left-margin, head-form untouched boundary, -unindentedfields exemption (structural newline kept byte-exact), idempotence on file-style defs.
• Verified (tclsh 9.0.3): punk::args suite 186 pass / 1 pre-existing skip / 0 fail; punk::ns suite 53/53; full source-tree suite 820 total, 806 pass, 13 skip, 1 fail = exec-14.3 only (known pre-existing core-test baseline) - zero regressions; make.tcl modules builds clean; 'i help'/'i help_chunks' verified aligned in punk902z src.

2026-07-12 increment 5: define quoting documentation (punk::args 0.8.1) - acceptance complete

• The container quoting rules pinned by defquoting.test now appear in the punk::args::define -help documentation (-help key section): braced values fully literal; double-quoted values with Tcl backslash semantics at record parse ($ and [] literal, no substitution outside tstr placeholders); the backslash-escaped placeholder idiom. Rendered output of the new section verified in punk902z src - the meta-escapes display as intended (\n as two characters, \Deleted, doubled backslash, literal dollar-brace placeholders).
• punk::args suite 186 pass / 1 pre-existing skip / 0 fail (tclsh 9.0.3).

Acceptance review at flip (2026-07-12): all criteria satisfied.

• -& record continuation parses identically to its backslash equivalent (recordcontinuation.test byte-equality; additive - existing definitions unchanged); collision rules and the {-&} escape documented (increment 3).
• @cmd -help/-summary honour -unindentedfields; rendering_unindentedfields_cmd_help_GAP flipped to aligned (increment 1; -summary vacuously honoured - no renderer transforms it, verified).
• Constructed definitions can request whole-block normalization (@normalize, block-form values); rendering_constructed_def_indent_characterization updated to the chosen opt-in semantics (unopted behaviour pinned as contract); ::punk::helptopic::define_docs dropped its manual pre-normalization - first to interim left-margin authoring (increment 2), then to indented blocks + @normalize as the consumer proof (increment 4).
• Quoting rules from defquoting.test appear in the define -help documentation (increment 5).
• Full punk::args suite passes with GAP tests flipped, none weakened: 186 pass / 1 pre-existing skip / 0 fail (suite grown from 128 to 187 tests via the new recordcontinuation.test and normalize.test); full source-tree suite showed zero regressions at increments 3 and 4 (exec-14.3 known baseline only). Verified on native tclsh 9.0.3 plus punk902z src for rendered output; punk 0.2.5 / punk::args 0.6.1-0.8.1 shipped in project versions 0.12.4-0.12.8.