diff --git a/GOALS.md b/GOALS.md index d52a254b..001b8202 100644 --- a/GOALS.md +++ b/GOALS.md @@ -314,3 +314,15 @@ Scope: src/modules/punk/repl-999999.0a1.0.tm (editbuf/reader integration, provid Detail: goals/G-044-repl-command-completion.md Goal: the interactive repl (raw mode primary) offers command completion and hinting driven by punk::args documentation - the command word resolved through the same doc-lookup flow as 'i' (ensembles, subcommands, ensemble parameters) and the argument position through the definition (options, choices with parse-consistent matching, literals, form awareness per G-041) - via an activation scheme that preserves the editbuf's literal-tab support (not exclusively plain Tab), behind a per-subshell completion-provider seam so alternative-language subshells (e.g. an interactive xtal session) can replace, augment or cleanly disable the Tcl-centric completer. Acceptance: in a raw-mode interactive session a documented trigger (recorded in the detail file; a literal tab remains enterable into the editbuf) presents completions for command words, ensemble subcommands (resolved via the punk::ns doc-lookup flow without executing candidate commands), option flags and choice values (matching consistent with parse semantics: prefix/deny/reserve, and aliases once G-040 lands); a hint display shows the synopsis of the form(s) matching the partial input (consumes G-041's candidacy API when available - until then all-forms or form-0 display with the limitation noted is acceptable); no per-keystroke terminal queries are added when no completion display is active (G-013 approach note); the completer is a provider interface with the punk::args-driven implementation as the default - a subshell can declare an alternative provider or none, proven at minimum by a subshell with completion disabled showing no Tcl-centric interference (the xtal minimum bar; full xtal completion out of scope but not precluded by the interface); line-mode behaviour is unchanged unless a documented subset is added; G-013's raw editor essentials are a prerequisite and this goal does not weaken G-013's acceptance. + +### G-045 [proposed] punk::args definition authoring ergonomics: record continuation, @cmd unindented fields, constructed-definition normalization +Scope: src/modules/punk/args-999999.0a1.0.tm (record parsing in resolve, tstr interplay, arg_error @cmd rendering), src/tests/modules/punk/args/testsuites/ (rendering.test/defquoting.test as the safety net), src/modules/punk-999999.0a1.0.tm (::punk::helptopic::define_docs de-hacked as the consumer proof) +Detail: goals/G-045-punkargs-authoring-ergonomics.md +Goal: authoring punk::args definitions no longer requires backslash line-continuations or ad-hoc workarounds for multi-line records and constructed definitions - a parser-recognised record-continuation mechanism (candidate: an unquoted trailing -& token, with the detail file recording the collision analysis and the element-count disambiguation alternative), -unindentedfields honoured for @cmd fields, and constructed (string-built) definitions able to opt into the same whole-block indent normalization file-style definitions get - with the container quoting rules (braced=literal, quoted=Tcl backslash semantics, \$\{ escape) promoted from defquoting.test into the define documentation. +Acceptance: a definition using the chosen record-continuation mechanism parses identically to its backslash-continuation equivalent (existing definitions unchanged - continuation is additive), with the token's collision rules documented and an escape/rejection story for values that legitimately match it; @cmd -help/-summary honour -unindentedfields (the rendering.test GAP rendering_unindentedfields_cmd_help_GAP flips to aligned); a constructed definition can request whole-block normalization so embedded continuation indentation behaves as in file-style definitions (the rendering_constructed_def_indent_characterization expectations updated to the chosen semantics), and ::punk::helptopic::define_docs drops its manual pre-normalization to prove it; the quoting rules from defquoting.test appear in the punk::args::define -help documentation; the full punk::args suite (128 tests incl. the rendering invariants: nesting independence, relative-indent preservation) passes with GAP tests flipped, none weakened. + +### G-046 [proposed] punk::args deferred -help resolution (parse-time performance + reentrancy) and rendering/value-shape fixes +Scope: src/modules/punk/args-999999.0a1.0.tm (resolve/get_dict: display-field deferral, dynamic-cache subst path, prefix writeback, string renderer, cmdhelp-facing messages), src/modules/punk/ansi-999999.0a1.0.tm (mark_columns argdoc as the reentrancy/perf testbed), src/tests/modules/punk/args/testsuites/ (GAP tests flip; perf verification) +Detail: goals/G-046-punkargs-deferred-help-and-fixes.md +Goal: argument resolution no longer processes -help and other display-only fields - their tstr expansion is deferred to display time (separately cached, per the existing in-source review notes) - so first parse of heavily documented commands gets measurably faster and definitions whose -help calls punk::args-parsing commands (the punk::ansi::mark_columns class) neither loop nor stall; alongside, the mechanical defects pinned by the characterization suites are fixed: @dynamic double-substituted multiline values align at their insertion column, prefix-normalized choice values keep the same shape as exact input, the -return string renderer aligns cmd-help continuations under the first line, and the misleading goodargs parse-error prefix in 'i ' output is fixed. +Acceptance: parsing/argument resolution provably skips -help expansion (a definition whose -help contains a ${[...]} that would error or record its invocation shows the substitution did NOT run during a parse-only path, only for help display); first parse of punk::ansi::mark_columns drops from ~4s to well under a second with 'i punk::ansi::mark_columns' still rendering the embedded example, and a -help that parses its OWN definition id resolves or errors cleanly rather than looping; first-parse timing improves for at least one other heavily documented command (recorded in the detail file); rendering_atdynamic_multiline_help_insertion_GAP flips to all-aligned; choicegroups_imap_prefix_listwrap_GAP flips to shape-identical (prefix input yields the same plain string as exact input); the -return string renderer's cmd-help continuations align under the first line with relative indents preserved (rendering_string_renderer_characterization updated); the 'Bad number of leading values...' prefix shown by goodargs parsing in 'i string is'-style output is reworded or suppressed for the usage-display path; full punk::args and punk::ns suites pass with no non-GAP expectations weakened. diff --git a/goals/G-045-punkargs-authoring-ergonomics.md b/goals/G-045-punkargs-authoring-ergonomics.md new file mode 100644 index 00000000..a7a6b99b --- /dev/null +++ b/goals/G-045-punkargs-authoring-ergonomics.md @@ -0,0 +1,95 @@ +# G-045 punk::args definition authoring ergonomics: record continuation, @cmd unindented fields, constructed-definition normalization + +Status: proposed +Scope: src/modules/punk/args-999999.0a1.0.tm (record parsing in resolve, tstr interplay, arg_error @cmd rendering), src/tests/modules/punk/args/testsuites/ (rendering.test/defquoting.test as the safety net), src/modules/punk-999999.0a1.0.tm (::punk::helptopic::define_docs de-hacked as the consumer proof) +Acceptance: a definition using the chosen record-continuation mechanism parses identically to its backslash-continuation equivalent (existing definitions unchanged - continuation is additive), with the token's collision rules documented and an escape/rejection story for values that legitimately match it; @cmd -help/-summary honour -unindentedfields (the rendering.test GAP rendering_unindentedfields_cmd_help_GAP flips to aligned); a constructed definition can request whole-block normalization so embedded continuation indentation behaves as in file-style definitions (the rendering_constructed_def_indent_characterization expectations updated to the chosen semantics), and ::punk::helptopic::define_docs drops its manual pre-normalization to prove it; the quoting rules from defquoting.test appear in the punk::args::define -help documentation; the full punk::args suite (128 tests incl. the rendering invariants: nesting independence, relative-indent preservation) passes with GAP tests flipped, none weakened. + +## Context + +Drafted 2026-07-09 after the tests-first characterization pass (user direction: implement +tests before changes in this area). The pain points, in the user's framing: + +- The define-block syntax deliberately avoids Tcl dict syntax for human readability, which + led to backslash line-continuations and a complex interplay between record parsing in + punk::args::resolve and tstr (especially ${...} expansions keeping whitespace aligned + during multiline substitutions). +- Constructed (string-built) definitions - ::punk::helptopic::define_docs and + punk::args::ensemble_subcommands_definition are the in-tree examples - have poor + ergonomics for getting indentation right: unlike file-style definitions they receive no + whole-block indent normalization (proven: rendering.test + rendering_constructed_def_indent_characterization - continuations at exactly 4 spaces + align, other depths leak), so builders pre-normalize by hand (undent/trim hacks, + \n-relative authoring). +- A key property of normal definitions that must be preserved: they do not depend on the + indentation of the code block they appear in (proven and pinned: + rendering_nesting_independence_plain / _tstr). +- The user's experiment used an ad-hoc `--` flag followed by a braced line-spanning empty + string as a de-facto record continuation - functional but not designed-for, and + unsatisfying. + +## The -& record-continuation candidate + +The user's proposal: a special token the record parser recognises as record continuation, +e.g. an unquoted `-&` at end of line. Design considerations recorded for implementation: + +- Collision risk is low but not zero (a value or flag legitimately spelled -&). Candidate + rules: only an UNQUOTED trailing `-&` at end of a record line counts; and/or use the + element count of the assembled record to disambiguate (an odd element count where pairs + are expected implies the trailing token is a continuation marker, not a value). The + detail decision must include an escape story for a literal trailing -& value (bracing it + suffices if 'unquoted' is part of the rule). +- Detection must happen in the record parser (resolve) before/alongside line splitting - + the same place backslash-continuation assembly effectively happens via Tcl's own parse + of braced blocks today. Unlike backslash-newline, -& survives inside braced blocks of + CONSTRUCTED definitions too - which is precisely why it helps: string-built defs cannot + use backslash-newline ergonomically (it is consumed by the building code's own quoting). + +## @cmd -unindentedfields + +Accepted but ignored today (arg_error's cmd_info rendering carries an '#unindentedfields ?' +todo; pinned by rendering_unindentedfields_cmd_help_GAP: left-margin-authored @cmd help +renders its first line +4 vs continuations). Honouring it makes left-margin authoring +available for @cmd -help/-summary as it already is for argument -help and -choicelabels. + +## Constructed-definition normalization + +Options to decide at implementation (record the choice): +- an explicit define option/directive requesting whole-block undent of the supplied text +- or automatic common-prefix normalization when a definition arrives as a single text + block (risk: changing existing constructed defs' rendering - the characterization test + documents current expectations to update deliberately) +::punk::helptopic::define_docs currently pre-normalizes by hand (undent/trim + \n-relative +help strings + -unindentedfields) - it converts to the new mechanism as the proof, and the +user's uncommitted experimental hack in punk.tm is superseded. + +## Quoting documentation + +defquoting.test pinned the container rules (braced values fully literal - $, [], two-char +\n, bare backslashes; quoted values get Tcl backslash semantics - \n newline, \\ -> \ - +with $/[] still literal; \$\{...\} renders a literal ${...} in tstr-processed blocks). +Promote these rules into the punk::args::define -help documentation so documenters (e.g. +punk::imap4's {\Deleted}/{$MDNSent} choice values) have them stated, not just tested. + +## Alternatives considered + +- Keeping backslash continuations + manual pre-normalization as the documented way - + rejected: the friction is proven by two in-tree constructed-def workarounds and the + user's punk.tm experiments. +- Tcl dict syntax for definitions - rejected long ago by design (readability). +- The ad-hoc `--` + braced-gap trick as the blessed continuation - rejected: exploits + parser behaviour that was never a contract. + +## Notes + +- Safety net: src/tests/modules/punk/args/testsuites/args/rendering.test (15 tests: + nesting independence, relative-indent preservation, unindentedfields, constructed-def + characterization, multiline tstr insertions, @dynamic) and defquoting.test (container + quoting rules). GAPs to flip here: rendering_unindentedfields_cmd_help_GAP, + rendering_constructed_def_indent_characterization (expectations updated to chosen + semantics). +- Related: G-046 (deferred -help resolution and rendering fixes - same code + neighbourhood; sequencing between the two goals is free but they touch resolve + together), G-044 (completion consumes the documentation corpus these ergonomics grow). +- Session records 2026-07-09: probe scripts renderprobe*.tcl (scratchpad), the user's + experimental define_docs hack (uncommitted punk.tm working-tree diff at the time of + drafting - superseded by this goal when implemented). diff --git a/goals/G-046-punkargs-deferred-help-and-fixes.md b/goals/G-046-punkargs-deferred-help-and-fixes.md new file mode 100644 index 00000000..628ba516 --- /dev/null +++ b/goals/G-046-punkargs-deferred-help-and-fixes.md @@ -0,0 +1,98 @@ +# G-046 punk::args deferred -help resolution (parse-time performance + reentrancy) and rendering/value-shape fixes + +Status: proposed +Scope: src/modules/punk/args-999999.0a1.0.tm (resolve/get_dict: display-field deferral, dynamic-cache subst path, prefix writeback, string renderer, cmdhelp-facing messages), src/modules/punk/ansi-999999.0a1.0.tm (mark_columns argdoc as the reentrancy/perf testbed), src/tests/modules/punk/args/testsuites/ (GAP tests flip; perf verification) +Acceptance: parsing/argument resolution provably skips -help expansion (a definition whose -help contains a ${[...]} that would error or record its invocation shows the substitution did NOT run during a parse-only path, only for help display); first parse of punk::ansi::mark_columns drops from ~4s to well under a second with 'i punk::ansi::mark_columns' still rendering the embedded example, and a -help that parses its OWN definition id resolves or errors cleanly rather than looping; first-parse timing improves for at least one other heavily documented command (recorded in the detail file); rendering_atdynamic_multiline_help_insertion_GAP flips to all-aligned; choicegroups_imap_prefix_listwrap_GAP flips to shape-identical (prefix input yields the same plain string as exact input); the -return string renderer's cmd-help continuations align under the first line with relative indents preserved (rendering_string_renderer_characterization updated); the 'Bad number of leading values...' prefix shown by goodargs parsing in 'i string is'-style output is reworded or suppressed for the usage-display path; full punk::args and punk::ns suites pass with no non-GAP expectations weakened. + +## Context + +Drafted 2026-07-09 from defects found (and pinned as GAP tests) during the tests-first +characterization pass. User emphasis at approval: "Any performance gain we can gain from +deferring -help processing out of argument resolution would be worthwhile" - deferral is +the headline item, not merely a reentrancy patch. + +### 1. Deferred -help resolution (performance + reentrancy) + +punk::args::resolve currently tstr-processes entire definition blocks, including -help +sections. The in-source review note (resolve, near the block-processing loop) already +observes: "-help section processing is sometimes expensive ... isn't required for parsing +of arguments ... ideally we would delay expansion of -help sections until needed for +display, and use a different cache key for the parsing vs display versions of the resolved +definition." + +The reentrancy class this also fixes, reproduced 2026-07-09 with punk::ansi::mark_columns: +its argdoc -help contains ${[punk::ansi::argdoc::num_columns_example]}, and that example +proc itself calls mark_columns, whose punk::args::parse resolves the same still-resolving +definition. First call measured at ~3.94 SECONDS (60us thereafter, once cached). The +mark_columns source (punk/ansi ~6443-6470) documents the hazard - including the observed +"too many nested evaluations (infinite loop?)" when [a red] etc. are used in that -help - +and lists the candidate fixes: (a) avoid parsing functions in argdocs (limiting, not DRY), +(b) reentrancy guard, (c) don't process -help during argument resolution. This goal takes +(c) as primary (with the perf benefit), with a reentrancy guard as defence-in-depth for +the display path (a -help that renders help for its OWN id must error cleanly or resolve, +never loop). + +Design notes: +- separate cache keys for parse-spec vs display-spec (per the in-source note); the + @dynamic unresolved-cache (ptlist/paramlist subst mechanism) splits accordingly. +- fields in display-only scope: -help on @cmd/@examples/args, -choicelabels content, + @formdisplay bodies - anything not consulted by get_dict for matching/validation. + (-choices/-choicealiases/-typesynopsis ARE parse-relevant or synopsis-relevant - + classify explicitly in implementation.) + +### 2. @dynamic multiline paramindent gap + +rendering_atdynamic_multiline_help_insertion_GAP: a ${$DYN_X} double substitution whose +second-round result is multiline lands lines 2+ at insertion-column minus 4 (internal +relative indents preserved), because the cached-path plain `subst` has no paramindent +re-alignment - contrast single-level ${[cmd]} insertion which aligns fully +(rendering_tstr_multiline_command_insertion). Fix: apply the same paramindent treatment +to second-round substitution results. Note interaction with item 1: if display fields +defer, the fix belongs in the display-time expansion path. + +### 3. Prefix-normalization value shape (list-wrap) + +choicegroups_imap_prefix_listwrap_GAP: for choices containing backslashes (punk::imap4's +{\Deleted} etc. - spaces would behave likewise), exact input stores the plain string but +prefix-normalized input stores a one-element LIST ({\Deleted}) - the get_dict overwrite +writeback lsets into the stored value treating a scalar as a list. Consumers get a +different value shape depending on how the user typed the choice. Fix the writeback for +the single-value case (the -multiple/clause paths already index deliberately). + +### 4. -return string renderer cmd-help alignment + +rendering_string_renderer_characterization: the string renderer joins the -help first +line onto the "Description: " label but drops continuations flush-left (relative indents +preserved among themselves). The table renderer aligns properly. Align continuations +under the first line. (The characterization test's pinned flush-left expectations get +updated to the aligned form.) + +### 5. cmdhelp goodargs message accuracy + +'i string is' (and any command whose definition requires leaders) prefixes its usage +output with "Bad number of leading values for punk::args::parse $args_remaining ..." - +an internal-looking, inaccurate message produced by cmdhelp's goodargs-marking parse of +the (empty) trailing args. Reword for the usage-display path or suppress (the parse is +only advisory for highlighting supplied args). User: "the message is not accurate and +could do with a fix up at some stage." + +## Alternatives considered + +- Reentrancy guard alone (without deferral) for item 1 - rejected as primary: it fixes + the loop but forfeits the first-parse performance win the user explicitly wants, and + heavy -help processing would still run on every first parse. +- Documenting "don't call parsing functions from argdocs" - rejected (the mark_columns + source itself calls this limiting and non-DRY). + +## Notes + +- GAP tests to flip: rendering_atdynamic_multiline_help_insertion_GAP, + choicegroups_imap_prefix_listwrap_GAP; characterization tests to update: + rendering_string_renderer_characterization. Safety net: the full 128-test punk/args + suite (esp. rendering.test invariants and dev/dynamic-cache.test) and punk/ns cmdflow. +- Perf evidence baseline (2026-07-09): mark_columns first parse ~3.94s / subsequent 60us + (punk902z dev, this machine). tclcore moduledoc commands (e.g. ::lseq with + helpers::example blocks) are candidates for the second timing datapoint. +- Related: G-045 (authoring ergonomics - same resolve neighbourhood; free sequencing), + G-041 (form matching will exercise resolve paths more heavily - deferral lightens its + per-form attempts), G-044 (completion wants fast parse-spec resolution).