punkshell/goals/G-177-punkargs-scale-independence.md

G-177 punk::args scale independence: cost decoupled from documentation size and registry size

Status: proposed Scope: src/modules/punk/args-999999.0a1.0.tm (resolve/define/undefine cache keying, update_definitions registry structures and scan phasing, parse/get_dict by-id entry path, parse_cache bounding); src/modules/punk/ns-999999.0a1.0.tm (per-subcommand update_definitions call sites in the cmdhelp and get_ns_dicts walks); src/tests/modules/punk/args/testsuites/args/ (new scaling suite + corpus render characterization) Goal: punk::args' per-call and per-query costs are decoupled from the two quantities a universally-documented corpus makes large - the byte size of the documentation a command carries, and the number of registered definition namespaces - so that a richly documented command parses no slower than a sparsely documented one of the same argument shape, a single-namespace documentation query costs the same whether 50 or 5000 namespaces are registered, and the raw definition text of the corpus is resident once rather than twice, with every existing render byte-identical across the change. Acceptance: measured in one process on one machine with the pre-change baseline recorded in this file - (1) two definitions of identical argument shape whose -help text differs by 40x or more in bytes parse within 1.15x of each other, and punk::args::resolve on a warm cache likewise within 1.15x (today 1.97x and ~20x); (2) update_definitions for one namespace against a fully-loaded registry costs no more than 2x at 4000 registered namespaces what it costs at 100, and an unknown-id real_id miss holds the same bound (today ~270x at 2000); (3) a pin proves the no-work early-out actually fires in a stock session, and a second pin proves a genuinely unloaded namespace still loads when registered/loaded counts coincide - the false-positive the current length-equality test permits; (4) a counter pin shows the namespaces scanned to answer a single-namespace query is O(1) rather than O(registry); (5) raw definition text resident per definition drops by 40% or more against the recorded baseline, and parse_cache has a documented bound with an eviction pin; (6) usage, synopsis and arg_error output over the whole existing definition corpus is byte-identical before and after; full punk/args and punk/ns suites pass under the canonical runtests interpreter.

Context

Drafted 2026-08-08 from a scalability review asking what breaks if every command - including third-party libraries - carries punk::args documentation. Baseline measured via bin/punk91.exe src script, corpus of 58 registered namespaces / 693 definitions (point-in-time; line anchors likewise):

parse withid, 220 B definition                        28.8 us
parse withid, 9776 B definition (same arg shape)      56.8 us
resolve warm, same pair                          1.5 -> 29.6 us
dict exists on the text key, same pair          0.27 -> 5.80 us
update_definitions one ns, R=100 / 500 / 2000    9.3 us / 218 us / 2515 us
punklib_ldiff, n=58 / 250 / 1000 / 4000          8.9 us / 85 us / 914 us / 12.0 ms
cold update_definitions *                        10.3 ms / 693 defs = 15 us per definition
memory                                           1625 B raw text/def held TWICE
                                                 + 7721 B resolved spec/def
                                                 ~= 11 KB per documented command, per interp

Three root causes:

1. Definition text is the cache key. resolve keys rawdef_cache_about and rawdef_cache_argdata on the whole raw definition (~args:2152, where the author already flagged it: "for now we will just key using the whole string... memory usage probably not ideal"); parse -cache 1 keys parse_cache on [list $parseargs $deflist $form] (~args:7234). That is ~4 full-text hashes per parse call. The isolation probe shows the ENTIRE per-call growth with documentation size sits inside resolve - in the keys, not in the parsing work. A documentation tax levied at runtime, on every invocation, permanently.

2. Registry membership is lists. punklib_ldiff (~args:15595) is a nested-loop ni scan, and the early-out is [llength $loaded_packages] == [llength $registered] (~args:5106). In a stock session registered=58 loaded=60 - ::punk::auto_exec::argdoc enters loaded_packages without ever being in NAMESPACES, and registration is not deduped (appending ::punk::lib twice leaves two entries) - so the fast path never fires and every call pays the quadratic. punk::ns calls this per ensemble subcommand: at R=2000 a 30-subcommand help render burns ~75 ms in bookkeeping that returns nothing. The same length test can also coincidentally hold while a namespace is genuinely unloaded - silently missing documentation, and likelier as the registrant count grows.

3. The scan phase is eager and global. The first call scans EVERY registered namespace regardless of the requested nslist (~args:5113-5157) at 15 us per definition - ~300 ms at 20000 definitions, paid on the first help keystroke to answer a question about one namespace.

Approach

1. Key the resolved-spec caches on the definition id (or a cheap digest), keeping text -> id only for the anonymous withdef path. The seam already exists: parse withid holds the id, calls raw_def to get the text, then hands the text to get_dict, which re-derives the id from it.
2. Dict-as-set for registered/scanned/loaded; dedup at registration; replace the length-equality early-out with an explicit pending-set emptiness test.
3. Scan per-namespace on demand rather than globally on first call.
4. Drop the double residency of raw text; give parse_cache a bound and eviction.
5. Ratio pins rather than absolute times, so the suite pins the complexity class and survives hardware changes.

Notes

• id_cache_spec (~args:614) is declared and initialised but never read - dead; remove in step.
• Related: G-175 (@dynamic resolve multiplicity) - same resolve path, disjoint question. G-175 owns HOW MANY TIMES one render resolves; G-177 owns WHAT ONE RESOLVE COSTS. Neither subsumes the other, and G-177's re-keying does not help @dynamic definitions, which bypass the resolved cache entirely (if {!$is_dynamic} ~args:2175).
• Related: G-044 (repl command completion and hinting) - the latency consumer; high-frequency id lookups during typing are what these ratios make viable.
• Related: G-050 (punk::ns::synopsis validity marking) - consumer of synopsis cost (627 us cold / 48 us warm per command measured at the baseline above).
• Related: G-075 (punk::args (package) ids) - shares the id-lookup and update_definitions prefix-handling path.
• Related: G-178 (multi-provider safety) - the registration dedup lands here as a performance requirement and there as a correctness one; whichever lands first satisfies the other's share.
• Related: G-167 (version-delta availability) and G-056 (display-time word wrapping) - both deliberately CHANGE render output, which acceptance (6) pins byte-identical across G-177. No conflict, but whichever lands second re-baselines the corpus render characterization; G-177's pin is "unchanged by the re-keying", never "unchanged forever".
• Predecessor: G-046 (achieved - see goals/archive/G-046-punkargs-deferred-help-and-fixes.md) deferred expensive -help processing to display time. That deferral is why the residual doc-size tax is PURELY the cache keys; a re-keying must compose with it, not duplicate it.
• Candidate deliverable, deliberately NOT part of acceptance: a scriptlib/developer/punkargs_bench.tcl sibling to goals_lint.tcl so regressions are diagnosable rather than merely detectable.