diff --git a/GOALS.md b/GOALS.md index b0329494..1c54eaf8 100644 --- a/GOALS.md +++ b/GOALS.md @@ -345,3 +345,11 @@ Detail: goals/G-090-shellfilter-shellthread-audit.md ### G-093 [proposed] runtests path-glob discovery: fix -include-paths multi-pattern/repeated-flag matching + expose -exclude-paths Scope: src/tests/runtests.tcl, src/modules/punk/path-999999.0a1.0.tm (treefilenames/globmatchpath as implicated), src/modules/punk/args-999999.0a1.0.tm (only if -multiple implicated), src/tests/ (runner-targeting regression suite), src/tests/AGENTS.md Detail: goals/G-093-runtests-includepaths-discovery.md + +### G-094 [proposed] punkcheck single record lifecycle: OO installtrack as sole implementation with a persistence policy +Scope: src/modules/punkcheck-999999.0a1.0.tm (installtrack/installevent lifecycle unification, installfile_* retirement, punkcheck::install as OO consumer), src/modules/punk/mix/cli-999999.0a1.0.tm + src/make.tcl (consumers verified unchanged), src/tests/modules/punkcheck/testsuites/punkcheck/ (characterization + new equivalence/staleness coverage) +Detail: goals/G-094-punkcheck-single-lifecycle.md + +### G-095 [proposed] punkcheck concurrent-writer safety: atomic saves and advisory event-scoped locking +Scope: src/modules/punkcheck-999999.0a1.0.tm (save/load chokepoint: atomic rename, lockfile protocol, merge-on-flush, non-interactive recovery), src/make.tcl + src/modules/punk/mix/cli-999999.0a1.0.tm (event-scoped lock consumers as verified), src/tests/modules/punkcheck/testsuites/punkcheck/ (new concurrency suite) +Detail: goals/G-095-punkcheck-concurrent-writer-safety.md diff --git a/goals/G-027-derived-project-pull-updates.md b/goals/G-027-derived-project-pull-updates.md index dbd607a7..413872cd 100644 --- a/goals/G-027-derived-project-pull-updates.md +++ b/goals/G-027-derived-project-pull-updates.md @@ -3,7 +3,7 @@ Status: proposed Scope: src/modules/punk/mix/commandset/project-999999.0a1.0.tm (project.new push path), src/modules/punkcheck-999999.0a1.0.tm (install provenance records), src/make.tcl (derived-project pull entrypoint) Goal: a project generated from a punkshell layout can pull infrastructure updates (make.tcl/build.tcl, bootsupport modules and libs, layout template payloads) from its originating punkshell project by running one command inside the derived project - replacing the current push model (`dev project.new -force 1 -update 1` run from punkshell) - with install provenance robust to derived-project workdir moves (not local relative paths alone), VCS-state awareness on both ends, and a recorded decision on pulling from remote sources. -Acceptance: one documented command run inside a derived project updates its punkshell-derived infrastructure from the source punkshell project; the update still works after the derived project's working directory has been moved (proven by moving a scratch derived project and pulling); VCS integration on both ends: the pull applies the G-026 clean-checkout policy to the punkshell source, and reports (or refuses per option) when target files it would overwrite carry uncommitted local modifications in the derived project's git/fossil checkout; .punkcheck records remain the provenance basis (updates are recorded and unchanged targets skipped, as with existing punkcheck-tracked installs); the push flow keeps working until explicitly retired; the remote-pull question (updating from a remote punkshell repository rather than a local checkout) has a recorded design decision - implemented, or deferred with rationale in the detail file. +Acceptance: one documented command run inside a derived project updates its punkshell-derived infrastructure from the source punkshell project; the update still works after the derived project's working directory has been moved (proven by moving a scratch derived project and pulling); VCS integration on both ends: the pull applies the G-026 clean-checkout policy to the punkshell source, and reports (or refuses per option) when target files it would overwrite carry uncommitted local modifications in the derived project's git/fossil checkout; .punkcheck records remain the acceleration layer (updates recorded, unchanged targets skipped) but the safe-overwrite classification survives their loss: the pull maintains a committed baseline record (origin identity + pulled version) from which pristine content is re-derivable via the origin's VCS, and a derived project with absent or deleted .punkcheck records still classifies every target conservatively (report/confirm - never silent overwrite, never a wedged pull), proven by deleting the scratch project's .punkcheck files and re-pulling; the push flow keeps working until explicitly retired; the remote-pull question (updating from a remote punkshell repository rather than a local checkout) has a recorded design decision - implemented, or deferred with rationale in the detail file. ## Context @@ -54,6 +54,13 @@ work from a stale project, or delivered as its first step). - Target side: before overwriting, classify each target file - unmodified since last punkcheck install (safe), locally modified but committed (report/confirm), uncommitted local changes (refuse or explicit override). + No punkcheck record -> unverifiable -> report/confirm, never overwrite. + With the committed baseline record (origin identity + pulled version), + pristine content is reconstructed from the origin's VCS at the recorded + version and classification runs against that - .punkcheck records + accelerate the answer but are not its sole basis (they are lossable: + deletion is a documented recovery, and G-095's concurrency work treats + them as a rebuildable cache). - Remote pulls: decide whether updates may come directly from a remote punkshell repository (fossil clone/sync or git fetch of a punkshell repo, or the G-006 artifact-download channel) rather than a local checkout. @@ -121,3 +128,9 @@ What a pull may touch splits by ownership, not by file type: - G-062 (archived) recorded a follow-on relevant here: seeding LICENSE.txt into generated project layouts and pulling it into derived projects (deliberately excluded from G-062's acceptance) - see goals/archive/G-062-project-license-file.md. +- punkcheck internals this goal leans on are now goal-tracked (2026-07-19): G-094 + unifies the record lifecycle on the OO installtrack layer - the pull entrypoint + should drive that surface (or the batch punkcheck::install consumer), not the + legacy installfile_* pipeline G-094 retires; G-095 adds concurrent-writer + safety - a pull writing a derived project's .punkcheck while builds run there + is a future instance of exactly its scenario. diff --git a/goals/G-094-punkcheck-single-lifecycle.md b/goals/G-094-punkcheck-single-lifecycle.md new file mode 100644 index 00000000..a404cd7b --- /dev/null +++ b/goals/G-094-punkcheck-single-lifecycle.md @@ -0,0 +1,88 @@ +# G-094 punkcheck single record lifecycle: OO installtrack as sole implementation with a persistence policy + +Status: proposed +Scope: src/modules/punkcheck-999999.0a1.0.tm (installtrack/installevent lifecycle unification, installfile_* retirement, punkcheck::install as OO consumer), src/modules/punk/mix/cli-999999.0a1.0.tm + src/make.tcl (consumers verified unchanged), src/tests/modules/punkcheck/testsuites/punkcheck/ (characterization + new equivalence/staleness coverage) +Goal: The installtrack/installevent OO layer is the only implementation of the .punkcheck record lifecycle (targetset init / add-source / started / end), with an explicit per-event persistence policy - eager per-operation read-modify-write (current OO behaviour) and deferred event-scoped flush (current batch behaviour, QUERY's no-save being the in-object precedent) - punkcheck::install rebuilt as a tree-walking consumer of that layer in deferred mode, the zero-caller legacy pipeline (installfile_begin / installfile_finished_install / installfile_skipped_install) retired to error-with-pointer shims, and every .punkcheck save flowing through a single chokepoint. +Acceptance: characterization suites install.test and installtrack.test pass (pins flipped only where this goal's contract deliberately changes behaviour, each flip recorded); a new test pins field equivalence between batch-installed and OO-installed records (including -targets_cksums, which the legacy proc pipeline never stored); a new regression test covers o_record_list coherence - two sequential events on one installtrack instance, with targetset writes in the first event, lose nothing at the second start_event; grep of src/modules, src/make.tcl and src/project_layouts finds no live caller of the retired trio outside shims and tests; a full make.tcl build completes with skip/copy decisions unchanged against pre-refactor .punkcheck state (no spurious recopies of unchanged sources). + +## Context + +2026-07-19 module review found three parallel implementations of the record +lifecycle, with drift: + +- **Legacy proc pipeline** (installfile_begin/started/finished/skipped + + start_installer_event): frozen at an earlier design - INSTALL-only (no + MODIFY/DELETE/VIRTUAL/QUERY), no FAILED status, no -note, and + installfile_finished_install stores no post-install target cksums, so its + records can never serve `-overwrite synced-targets`. Zero live callers of + begin/finished/skipped anywhere (punk::mix::cli's calls are commented out + next to their OO replacements); they are kept alive only by installfile_help + and the characterization tests. +- **OO layer** (installtrack/installevent): targetset_started, + targetset_addsource and targetset_addsource_virtual delegate down to the + legacy procs, while targetset_init and targetset_end are parallel rewrites + of installfile_begin and finished/skipped with the extensions above - a + half-inversion of the desired layering (procs should be the facade, the OO + lifecycle the substrate). +- **Batch punkcheck::install**: bypasses both - uses start_installer_event and + the metadata fetcher but inlines its own begin and finalize so it can defer + all persistence to one end-of-run save, independently re-growing + -targets_cksums. Its in-memory recordset handling is the deferred + persistence mode this goal makes first-class. + +Latent single-process clobber: start_event saves the whole constructor-era +o_record_list, which targetset operations never refresh (they re-load from +file into locals) - a second start_event on one instance would discard every +FILEINFO update the first event's targetsets wrote. Unexercised-path evidence: +installtrack get_recordlist reads a misspelled variable ($o_recordlist) and +targetset_dict calls a nonexistent punk::records_as_target_dict - both would +error if ever called. + +Performance shape that motivates the policy knob rather than "just save +eagerly everywhere": eager OO costs three full parse+prettyprint rewrites of a +MB-scale file per installed target (modules/.punkcheck is ~1.9MB), which is +why the batch path defers; deferring must be a first-class mode of the one +lifecycle, not an ad-hoc bypass with its own record code. + +## Approach + +- Persistence policy selected per event (eager | deferred); in deferred mode + the installtrack's in-memory recordset is authoritative between flushes, so + the o_record_list coherence fix is a precondition, not a side quest. +- installfile_add_source_and_fetch_metadata stays as the shared pure engine - + it is the one correctly-layered piece (record transform + filesystem + metadata fetch, no saves), used by all three current paths. +- start_installer_event's role is subsumed by installtrack construction + + start_event; the surviving convenience surface for callers is + punkcheck::install / install_tm_files / install_non_tm_files as thin OO + consumers. +- Single save chokepoint: all writes to a .punkcheck file (event start, eager + targetset ops, deferred flush) route through one proc/method - this is the + attachment point G-095 locks onto. + +## Alternatives considered + +- Making the legacy procs true facades that construct transient event objects + while preserving their dict-passing contract - rejected: zero live callers + make the reconstitution ceremony pointless; error-with-pointer shims are + honest and cheaper. +- Locking/concurrency first, refactor later - rejected: the safety measures + would have to be implemented and kept in sync across three lifecycle + implementations; sequencing the unification first means G-095 lands once. + +## Notes + +- Prerequisite for G-095 (concurrent-writer safety) - that goal's lock, + atomic-rename and merge-on-flush measures assume this goal's single save + chokepoint. +- Version-skew during rollout: bootsupport punkcheck (0.4.0) and the dev + module can both write the same tree's .punkcheck files (make.tcl runs + bootsupport's copy; a dev shell runs the built one) - re-vendor bootsupport + promptly after landing, and sequence this refactor before any on-disk + protocol change from G-095. +- G-087 recorded a punkcheck MAX_PATH candidate (deep relative source chains + exceed Windows path limits, unnormalized) - natural rider on this goal's + load/save consolidation. +- The characterization suites were written expressly to lock behaviour "before + any refactoring" - this is that refactoring. +- punkcheck::cli (status/paths) is a read-only consumer and stays untouched. diff --git a/goals/G-095-punkcheck-concurrent-writer-safety.md b/goals/G-095-punkcheck-concurrent-writer-safety.md new file mode 100644 index 00000000..f40faa3f --- /dev/null +++ b/goals/G-095-punkcheck-concurrent-writer-safety.md @@ -0,0 +1,96 @@ +# G-095 punkcheck concurrent-writer safety: atomic saves and advisory event-scoped locking + +Status: proposed +Scope: src/modules/punkcheck-999999.0a1.0.tm (save/load chokepoint: atomic rename, lockfile protocol, merge-on-flush, non-interactive recovery), src/make.tcl + src/modules/punk/mix/cli-999999.0a1.0.tm (event-scoped lock consumers as verified), src/tests/modules/punkcheck/testsuites/punkcheck/ (new concurrency suite) +Goal: Concurrent writers to the same .punkcheck file cannot corrupt it or silently erase each other's records, and readers never observe torn content - every save is write-temp-then-atomic-rename (bounded retry for Windows sharing violations); an advisory cross-platform lockfile (open {WRONLY CREAT EXCL}, contents naming holder pid/host/installer/timestamp, bounded-retry acquisition, age-based stale-break with warning) serializes whole installer events per punkcheck root; deferred-mode flushes merge own records (INSTALLER by -name, FILEINFO by -targets) into freshly-loaded file state instead of wholesale overwrite; and the duplicate-INSTALLER sanity recovery degrades to a clean actionable error instead of an askuser prompt when stdin is non-interactive. +Acceptance: a concurrency test drives two child tclsh installer processes against one .punkcheck folder - the file parses cleanly throughout (a polling load_records_from_file reader hits no TDL parse error), both installers' records are present afterwards, and no duplicate INSTALLER records exist; lock behaviour verified - the second writer observably waits, or times out with a message naming the holder from lockfile contents; stale-break verified against a planted orphan lockfile; suite passes on the project's Windows Tcl 8.6 + Tcl 9 test matrix (capability-gated where child-process facilities require it), with a unix run when an environment is available. + +## Context + +2026-07-19 risk review: every punkcheck writer is an unguarded +read-modify-write cycle against a shared file, and the batch/deferred path +holds the entire recordset in memory for the whole tree walk +(seconds-to-minutes of checksumming) before a truncate-in-place rewrite of a +MB-scale file. Failure modes ranked by operational pain: + +1. Duplicate INSTALLER/FILEINFO records from interleaved insert cycles - the + next run either aborts (records_as_target_dict errors in targetset_init) + or demands interactive confirmation to delete the file (installtrack's + duplicate-INSTALLER sanity check via punk::lib::askuser) - which blocks or + errors unattended `-confirm 0` builds. +2. Torn reads - save truncates in place, so a concurrent reader (another + installer's load, or punkcheck::cli status) can parse a half-written file + and abort. +3. Silent lost updates - mildest: .punkcheck is an expendable + skip-optimization cache, so losses mean redundant recopies next run. + +The lock's larger value: two processes racing one .punkcheck are also racing +`file copy` into the same target tree - serializing the installer event +serializes the real hazard, not just the metadata. + +Current avoidance is convention only: per-target-root file partitioning, +strictly sequential make.tcl stages that re-load at stage boundaries, and +single-user build habits. Nothing technical prevents overlap (two builds, or +user + agent). + +## Approach + +- Depends on G-094's single save chokepoint - lock, atomic rename and + merge-on-flush are implemented once there, inherited by every path. +- Lock scope: the whole installer event (start_event through final + flush/end; batch install = the whole depth-0 call), sibling lockfile + `.punkcheck.lock` in the punkcheck folder. +- Acquisition: `open ... {WRONLY CREAT EXCL}` (atomic create-new on Windows + and POSIX, Tcl 8.6 and 9, no extra packages); hold the channel open for the + duration (on Windows an open handle also resists deletion); on contention + retry with backoff to a timeout, then fail with a message quoting the + holder identity from the lockfile. +- Stale locks: age-based break with warning as the portable baseline; + optional liveness hardening behind capability checks (twapi process_exists + on Windows, kill -0 probe on unix). +- Optimistic tripwire independent of the lock: capture a file signature + (mtime/size) at load, verify at flush; mismatch means a writer bypassed the + protocol (e.g. an older vendored punkcheck during the transition window) - + reload-and-merge or warn loudly rather than silently overwrite. + +## Alternatives considered + +- SQLite storage - rejected for now (user decision 2026-07-19): the TDL text + format's easy human readability/debugability is an overriding factor. + Reconsideration requires all three: performance becomes an actual problem; + sqlite is determined to provide an overwhelming advantage in that + situation; and good debugging/viewing tools are built alongside the + migration. (sqlite3 is also only conditionally available across the target + interps.) +- mkdir-as-mutex - not implementable in Tcl: `file mkdir` succeeds silently + on an existing directory, so there is no atomic test-and-set. +- TclX flock / OS byte-range locks / twapi-exclusive CreateFile - not + cross-platform across the 8.6+9 win/unix envelope; mandatory locking would + also punish read-only consumers (punkcheck::cli status). + +## Notes + +- Expendability is tiered, not absolute (2026-07-19 review): in build trees + (modules/, lib/) deletion costs only redundant recopies - overwrite modes + there are source-driven and targets are clobber-by-design; for + synced-targets consumers (the bin/ deploy) deletion wedges existing targets + into skip-until-manual-action - fail-safe but stuck, and not self-healing, + since skipped records never store target cksums; in derived projects the + G-027 committed-baseline direction keeps safe-overwrite classification + working after record loss. The invariant to preserve across all tiers: + absent records degrade to skip/report-confirm, never to overwrite. Within + those consequence bounds, deletion stays the documented last-resort + recovery; locking does not make the file precious. +- The installtrack constructor's deletion prompt ("It is safe to delete + .punkcheck files...") oversells for synced-targets consumers - add the + wedge caveat to that message when it is touched under G-094/G-095. +- Advisory protocol is only effective once all writers carry it - re-vendor + bootsupport immediately after landing; the signature tripwire covers the + transition window. +- Readers (punkcheck::cli status/paths) gain torn-read immunity from atomic + renames without any reader-side change. +- Related: G-028 (Restart Manager locker naming for OS-mandatory locks on + deploy targets) - same "name the blocker" diagnostic principle; this goal's + timeout message naming the lock holder is the advisory-lock sibling. +- Related: G-027 - a derived-project pull writing .punkcheck files while + builds run in that project is a future instance of exactly this scenario.