punkshell/goals/archive/G-174-cmdtraverse-word-accounting.md

G-174 cmd_traverse word accounting - the doc-lookup walk never drops argument words

Status: achieved 2026-08-09 Scope: src/modules/punk/ns-999999.0a1.0.tm (cmd_traverse/cmdinfo doc-lookup walk, cmdhelp advisory parse + synopsis consumers); src/tests/modules/punk/ns/testsuites/ns/{cmdflow,cmdhelp}.test (conservation characterisation + consumer pins); src/modules/punk/ansi-999999.0a1.0.tm (punk::ansi::a? argdoc - real-world witness, edits only if the sample-form tightening decision lands) Goal: the doc-lookup walk behind punk::ns::cmdinfo has a stated, characterised word-accounting contract - every supplied trailing word lands in exactly one of args_resolved or args_remaining - so args_remaining-sensitive consumers (cmdhelp's advisory parse and G-041 form narrowing, synopsis form marking) always operate on the full unconsumed tail, and a command whose definition's first form is zero-arg (the punk::ansi::a? class) no longer has its literal discriminator words silently dropped before form narrowing runs. Acceptance: characterisation tests in cmdflow.test assert the conservation invariant (args_resolved plus args_remaining account for every supplied trailing word, each word in exactly one) across the walk's return lanes - including an a?-shaped fixture (plain proc, multiform definition, zero-arg first form) with multi-word tails, plus witnesses for the previously unwitnessed code-7 and code-10 return lanes; punk::ns::cmdhelp for 'a? web green' runs its advisory parse on the full tail and presents the web/sample multipleformmatches ambiguity (web's argument table rendered first, both forms marked in the synopsis) rather than a clean sample-only render, and punk::ns::synopsis marks the same forms for the same words; the first-form-only inspection (fid = first of form_names) is either extended or pinned as a recorded limit with its own witness, with the decision recorded in this file; the punk::ns suite passes under the canonical runtests interpreter with no regressions in cmdflow/cmdhelp.

Context

Root-caused 2026-08-07 from the user observation that 'i a? web green' presents the help for the loose 'sample' form (ordinal 5, first value typed string|stringstartswith(...)) instead of the 'web' form (ordinal 3, first value literal(web)) - "why don't the literals win out?".

The form-narrowing machinery is NOT at fault. Parsed directly, punk::args::parse_status {web green} withid ::punk::ansi::a? correctly reports BOTH 'web' and 'sample' as clean matches (multipleformmatches, no silent preference - the G-041 decision; achieved 2026-07-13, see goals/archive/G-041-punkargs-form-matching.md), web first in declaration order, so cmdhelp would render web's argument table and mark both forms. The literal-affinity ranking (form_literal_affinity) only orders candidates in the no-form-matches failure display and plays no role here.

The defect is one layer up: punk::ns::cmd_traverse (sole caller: cmdinfo's coroutine spawn) drops all but the LAST trailing word for commands whose definition's first form is zero-arg. Mechanism (line anchors are point-in-time, ns-999999.0a1.0.tm as at 2026-08-07):

1. Each outer-loop iteration re-snapshots queryargs/queryargs_untested to [lrange $args $i end] (~ns:5013) - a fresh tail per word.
2. The walk inspects only the spec's FIRST form - set fid [lindex $form_names 0] (~ns:5036); the multiform-discrimination block beside it is commented out ("multiform discrimination not yet supported").
3. a?'s first form is sgr_overview (zero-arg overview form): no opts, no values, so the ensemble-shape test ![llength $optnames] && ![llength $valnames] (~ns:5063) routes into the subcommand walk - but the form has no leaders either, so leadernames_matched is empty and the foreach immediately hits if {$lname eq ""} break (~ns:5073, annotated #todo - return? - the author already suspected the spot). Nothing is consumed, returned, or restored.
4. Control falls back to the outer for, which just advances i and re-snapshots. At loop exhaustion the code-10 fallthrough return (~ns:5309, annotated #REVIEW!!!) reports the LAST snapshot: with N trailing words, N-1 are silently discarded - present in NEITHER args_resolved nor args_remaining.

Probe evidence (punk91 src, 2026-08-07):

cmdinfo punk::ansi::a? web        -> args_remaining={web}
cmdinfo punk::ansi::a? web green  -> args_remaining={green}   ("web" vanished)
cmdinfo punk::ansi::a? term pastel-> args_remaining={pastel}  ("term" vanished)

cmdhelp's advisory parse then legitimately single-matches 'sample' on the remaining word ('green' fails web's required literal(web) first value, matches sample's string) - a CLEAN ok=1 narrow, so the mispresentation carries no failure signal anywhere.

Consumer sensitivity split (all cmdinfo consumers in the dev tree):

• docid-only, immune to the drop: punk::ns::forms, punk::ns::eg, punk::args::eg, punk::lib::tclscript_info, punk::args::ensemble_subcommands_definition.
• args_remaining-sensitive, inherit the corruption: punk::ns::cmdhelp (the 'i' advisory parse / G-041 narrowing - the observed defect), punk::ns::synopsis and synopsis_raw ('s' form marking from unresolved_args), cmdtrace/cmdtracebasic, and cmdinfo's own curried-notfound recursion.

Coverage determination (2026-08-07 review of both suites): cmdflow.test (26 tests) pins the choice-walk arcs - parse-parity per G-040 (achieved 2026-07-08, goals/archive/G-040-punkargs-choicealiases.md) covering denylist/reservelist/aliases/prefix/nocase, space-form docid jumps per G-051 (achieved 2026-07-13, goals/archive/G-051-cmdinfo-pseudo-and-prefix.md), unavailable attribution per G-166 (achieved 2026-08-06, goals/archive/G-166-cmdinfo-unavailable-surfacing.md), ensemble -parameters traversal. cmdhelp.test (~50 tests) pins rendering and POST-resolution multiform narrowing (the G-041/G-143/G-164 arcs; G-143 achieved 2026-08-01, goals/archive/G-143-maketcl-multiform-help.md; G-164 achieved 2026-08-04, goals/archive/G-164-viability-alternative-allocation.md), doconly/'string is' real-world pins, oo methods. Many tests assert args_remaining values, but always through subcommand-walk or single-trailing-word scenarios. Nothing asserts the conservation invariant; no fixture has the zero-arg-first-form shape; the code-7 lane's own source comment names its test ('i namespace which -v x') but no such test exists; code-10 has none; the first-form-only inspection is unpinned. The defect lives precisely in the uncovered lanes.

Approach

1. Characterise first, fix second (defect witnesses included): add cmdflow.test coverage asserting the conservation invariant across the walk's return lanes, an a?-shaped fixture (plain proc, multiform definition, zero-arg first form) exercised with multi-word tails, and witnesses for the code-7 and code-10 lanes. Pin the CURRENT dropping behaviour as the defect witness, flipped by the fix commit.
2. Fix minimally at the annotated sites: return the remaining tail from the empty-leader break (code-8-like return honouring #todo - return?), and/or tighten subcommand-walk eligibility so a level is only walkable when the inspected form actually has a choices-bearing leader. Choose the smallest change that restores conservation without disturbing the characterised parity/ensemble/unavailable lanes.
3. Consumer verification: cmdhelp ('i a? web green' ambiguity presentation, -return text parity) and synopsis form marking on the same words.
4. Decision item - first-form-only inspection (fid = form_names[0]): default is pin-as-recorded-limit with a witness; cross-form discrimination during traversal is its own follow-on goal if wanted.
5. Decision item (optional, ansi-side): whether a?'s 'sample' form stays string-loose so discriminator words genuinely ambiguate (default: accept - the marked-ambiguity presentation IS the G-041 contract), or the argdoc is tightened. No ansi edit unless this lands the other way.

Notes

• The a? argdoc (ansi-999999.0a1.0.tm ~3213-3279) is the trigger and real-world witness, not an edit target: forms sgr_overview/term/tk/web/x11/sample, with term/tk carrying their literals as @leaders while web/x11 carry them as @values - both shapes should appear in the fixture.
• Single-word tails survive today only because the last word is always the final snapshot - 'i a? web' working is luck, not contract.
• args-side channels are already adequate: parse_status reports multipleformmatches with per-form formstatus (all-valid keys) and noformmatch with ranked -formerrors; cmdhelp consumes them correctly. No punk::args change is expected under this goal.
• Related (activation overlap survey 2026-08-09, goals_xref score G-174): G-050 (punk::ns::synopsis argument-validity marking) - the synopsis marking consumer this goal feeds the full tail; G-174 pins WHICH forms are marked, G-050 owns richer validity marking. G-175 (@dynamic resolve multiplicity) - the same cmdhelp render pipeline and by-id fetches the advisory parse exercises. G-052 (TclOO method autodef) - sibling doc-walk/cmdhelp fallback surface pinned in the same suites.
• DECISION (fix arm, 2026-08-09): the minimal empty-leader-return landed - the annotated #todo - return? break now returns the unconsumed tail as new lane code 11, remaining shape [list {*}$eparams {*}$queryargs_untested] mirroring the sibling code-3 lane. Subcommand-walk eligibility tightening (requiring a choices-bearing leader) was NOT needed: record-level probing showed every other walk exit already conserved, and tightening would have re-routed the characterised code-3 lane through a different return. cmdinfo's driving loop tests only final == 0, so the new nonzero lane code is transparent to it.
• DECISION (first-form-only inspection, 2026-08-09): pinned as a recorded limit. Witness: cmdflow.test cmdtraverse_firstformonly_recorded_limit on the flowmultilead fixture (zero-arg first form plus a LATER form carrying a choices-bearing subcommand leader with a subhelp mapping) - the walk never consults the later form, the word stays conserved in args_remaining at the parent docid. Cross-form traversal discrimination remains a possible follow-on goal; flipping that pin is the signal it landed. The limit is also stated in the word-accounting contract comment above cmd_traverse.
• DECISION (ansi-side, 2026-08-09): a?'s 'sample' form stays string-loose - the marked-ambiguity presentation IS the G-041 multipleformmatches contract. No punk::ansi edit.
• The code-7 lane's stale source comment ('test with: i namespace which -v x') was corrected at the site: that invocation actually terminates in the code-8 lane (::tcl::namespace::which's autodef first form carries values, so the subcommand walk is never entered at that level) - verified by record-level probing 2026-08-09. The code-7 witness is the purpose-built flownomap fixture (a choice the resolver matches but no choiceinfo mapping or space-form id serves).
• Observed corner, deliberately not addressed here (recorded as an open follow-on): with nonempty eparams, the code-3/code-11 remaining shape prepends the already-consumed -ensembleparameter words while resolvedargs also holds them - double-listing beyond the sanctioned descent re-presentation. Reachable only by a hand-authored definition placing a choice-less (or absent) leader after -ensembleparameter leaders in a no-opts-no-values first form; no shipped definition has the shape.

Progress

2026-08-09 activated (user direction) and landed in one increment (punk::ns 0.11.1, project 0.66.3):

• Fix: cmd_traverse's empty-leader position returns lane 11 with the unconsumed tail; the word-accounting contract is stated at the record-shape comment above cmd_traverse (conservation, canonical spellings in resolvedargs, verbatim tail in remainingargs, the two sanctioned exceptions - ensemble-parameter re-presentation and alias/script currying - and the first-form-only limit).
• Characterisation, cmdflow.test G-174 section: traverse_final (drives the coroutine exactly as cmdinfo's reducer loop, exposing the final record's lane code) and word_accounting (cmdinfo-level counts + verbatim-suffix check) helpers; fixtures flowmultiform (a?-shaped: zero-arg first form, literal-leader form, literal-first-value form, loose string form), flowmultilead (first-form-only limit), flownomap (code-7), flowplainlead (code-3); the lane-code + conservation battery across return lanes 0a,2,X,3,4,6,7,8,9,10,11 - code-7 and code-10 witnessed for the first time; multi-word no-drop pins (2- and 3-word tails, both discriminator shapes); single-word by-contract pin (lane 11, no longer last-snapshot luck); the real-world a? witness ('web green'/'term pastel' whole-tail args_remaining); the first-form-only recorded-limit pin.
• Characterisation, cmdhelp.test G-174 section: 'a? web green' -return dict (args_remaining={web green}, ok=0, failureclass=multipleformmatches, formstatus keys {web sample}); -return text (web's --ARGUMENTS section renders before sample's, no other form sections); punk::ns::synopsis underline-marks exactly {web sample} of the six forms.
• Defect witness (pre-fix runs of the new tests, canonical tclsh90s): 8 of the 9 new tests fail against the unfixed walk (5 cmdflow + 3 cmdhelp); the code-7 witness passes both sides - that lane was conserved all along, merely unwitnessed. Pre-fix signature: cmdinfo a? web green -> args_remaining={green} ('web' in neither list, final record code=10 consumed={} remaining={green}); post-fix: {web green} via lane 11.
• Verification: full punk/ns subtree green under the canonical zig-built tclsh90s (140 tests: 138 pass, 2 pre-existing gated skips, 0 fail - no regressions in cmdflow/cmdhelp); cmdflow+cmdhelp green under tclsh86ts (86/86, the 8.6 leg); testbody_lint clean over the ns testsuite dir (7 files, 140 tests, 0 findings).

Acceptance satisfied in full; flipped achieved 2026-08-09.

Follow-ons

Follow-on: cross-form traversal discrimination - extend cmd_traverse's first-form-only inspection (fid = first of form_names) so choices-bearing subcommand leaders on later forms are consulted during the walk; flip cmdflow.test cmdtraverse_firstformonly_recorded_limit when it lands => open Follow-on: tighten punk::ansi::a?'s loose 'sample' form so discriminator words do not ambiguate => goal G-179 Follow-on: eparam double-listing corner - with nonempty eparams the code-3/code-11 remaining shape prepends already-consumed -ensembleparameter words beyond the sanctioned descent re-presentation (hand-authored -ensembleparameter-then-choiceless-leader first forms only; no shipped definition has the shape) => open