diff --git a/CHANGELOG.md b/CHANGELOG.md index 54454aac..a207bc22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,10 @@ The latest `## [X.Y.Z]` header must match the `version` field in `punkproject.to Entries are newest-first; one bullet per notable change. See the root `AGENTS.md` "Project Versioning" section for the bump policy. +## [0.17.3] - 2026-07-21 + +- punkcheck 0.6.0 (G-095, goal achieved): concurrent writers to one `.punkcheck` file can no longer corrupt it or silently erase each other's records, and readers never see torn content. Saves are write-temp-then-atomic-rename (with reader-side retry for the Windows replace-window open denial); whole installer events hold an advisory sibling `.punkcheck.lock` (new `punkcheck::lock` namespace - holder pid/host/installer/timestamp in the lockfile, backoff-retry acquisition to a timeout that errors naming the holder, stale-break on provably-dead holder pid or age, in-process reference-counted re-acquisition; env overrides `PUNKCHECK_LOCK_TIMEOUT`/`PUNKCHECK_LOCK_STALEAGE`); deferred flushes merge own records into freshly-loaded file state; an mtime/size tripwire warns on protocol-bypassing writers; the duplicate-INSTALLER recovery raises a clean actionable error instead of prompting when stdin is non-interactive. Verified by a new concurrency suite (real child installer processes) on Windows Tcl 9.0.3 + 8.6.17 and linux (WSL tclsh 8.6.14). + ## [0.17.2] - 2026-07-21 - punkcheck 0.5.0 (G-094, goal achieved): the installtrack/installevent object layer is the sole implementation of the .punkcheck record lifecycle, with a per-event persistence policy (`start_event ?-persistence eager|deferred?`; deferred updates flush via `$installer flush`, QUERY is never persisted) and a single save chokepoint (installtrack `save_working_recordset` - the G-095 attachment point). `punkcheck::install` is now a tree-walking OO consumer in deferred mode and its records carry the full field set including `-targets_cksums`. The zero-caller legacy proc pipeline (`start_installer_event`, `installfile_begin/started_install/finished_install/skipped_install`) is retired to error shims pointing at `punkcheck::installfile_help` (rewritten for the OO lifecycle). Sequential events on one installtrack no longer clobber prior targetset writes (ending an event now clears the active slot), and several latent bugs are fixed (installer-record `-name` trailing space on flush, event-history timestamp rewrites on reconstruction, `get_recordlist`, `targetset_dict`). Build skip/copy decisions are unchanged against pre-refactor `.punkcheck` state. diff --git a/GOALS-archive.md b/GOALS-archive.md index d135810b..8278b29b 100644 --- a/GOALS-archive.md +++ b/GOALS-archive.md @@ -129,6 +129,10 @@ Acceptance: from a clean checkout plus a punk-getzig-fetched pinned zig, the sui Scope: src/modules/punkcheck-999999.0a1.0.tm (installtrack/installevent lifecycle unification, installfile_* retirement, punkcheck::install as OO consumer), src/modules/punk/mix/cli-999999.0a1.0.tm + src/make.tcl (consumers verified unchanged), src/tests/modules/punkcheck/testsuites/punkcheck/ (characterization + new equivalence/staleness coverage) Acceptance: characterization suites install.test and installtrack.test pass (pins flipped only where this goal's contract deliberately changes behaviour, each flip recorded); a new test pins field equivalence between batch-installed and OO-installed records (including -targets_cksums, which the legacy proc pipeline never stored); a new regression test covers o_record_list coherence - two sequential events on one installtrack instance, with targetset writes in the first event, lose nothing at the second start_event; grep of src/modules, src/make.tcl and src/project_layouts finds no live caller of the retired trio outside shims and tests; a full make.tcl build completes with skip/copy decisions unchanged against pre-refactor .punkcheck state (no spurious recopies of unchanged sources). +### G-095 [achieved 2026-07-21] punkcheck concurrent-writer safety: atomic saves and advisory event-scoped locking → detail: goals/archive/G-095-punkcheck-concurrent-writer-safety.md +Scope: src/modules/punkcheck-999999.0a1.0.tm (save/load chokepoint: atomic rename, lockfile protocol, merge-on-flush, non-interactive recovery), src/make.tcl + src/modules/punk/mix/cli-999999.0a1.0.tm (event-scoped lock consumers as verified), src/tests/modules/punkcheck/testsuites/punkcheck/ (new concurrency suite) +Acceptance: a concurrency test drives two child tclsh installer processes against one .punkcheck folder - the file parses cleanly throughout (a polling load_records_from_file reader hits no TDL parse error), both installers' records are present afterwards, and no duplicate INSTALLER records exist; lock behaviour verified - the second writer observably waits, or times out with a message naming the holder from lockfile contents; stale-break verified against a planted orphan lockfile; suite passes on the project's Windows Tcl 8.6 + Tcl 9 test matrix (capability-gated where child-process facilities require it), with a unix run when an environment is available. + ### G-102 [achieved 2026-07-21] buildsuite bootstrap without a pre-existing tclsh: zig-driver hybrid → detail: goals/archive/G-102-buildsuite-zig-bootstrap.md Scope: src/buildsuites/suite_tcl90/ (driver split: zig-layer fetch/generate/compile + suite-tclsh-run Tcl phases; sources.config -> zon-shaped manifest migration), src/buildsuites/ (the pattern future suites fork - interplay with G-099), bin/tools/ (pinned zig acquisition per the consent-gated mechanism), src/scriptapps/bin/ punk-getzig (acquisition layer, unchanged in role) Acceptance: from a clean checkout plus the pinned zig (acquired via the consent-gated download mechanism or self-built), a documented zig invocation performs the whole pipeline in an environment where no pre-existing tclsh is resolvable (PATH-scrubbed demonstration recorded in this file): fetch/stage sources per the declarative manifest (content-integrity pins for zig-fetched trees; the fossil-checkout flow retained as a documented dev alternative for live-branch work), generate the configure-products, build the runtime, then run the post-tclsh phases under the suite-built shell - tcllib install with tcllibc accelerators via critcl, tklib, tk - and the expected-failure-baselined test gate, with results matching the suite.tcl-driven equivalents; the manifest is the documented tweak surface and copied suite trees remain isolated (G-096 copy-and-tweak property preserved); the recipe pins the zig version it is written against (API-churn mitigation - the 0.14->0.16 migration cost is the recorded precedent); the bash+powershell polyglot alternative for this layer is recorded as considered and rejected, with polyglot craft remaining the ACQUISITION layer's mechanism (punk-getzig/getpunk first-contact story). diff --git a/GOALS.md b/GOALS.md index 9b2a3121..293231da 100644 --- a/GOALS.md +++ b/GOALS.md @@ -334,10 +334,6 @@ Detail: goals/G-089-scriptlib-kits-and-modes.md Scope: src/modules/shellfilter-999999.0a1.0.tm (stack placement, insert_transform, junctions), src/modules/shellthread-999999.0a1.0.tm (worker park/adoption/termination), src/tests/modules/ (new characterization suites), README.md ('very unripe parts' note) Detail: goals/G-090-shellfilter-shellthread-audit.md -### G-095 [active] punkcheck concurrent-writer safety: atomic saves and advisory event-scoped locking -Scope: src/modules/punkcheck-999999.0a1.0.tm (save/load chokepoint: atomic rename, lockfile protocol, merge-on-flush, non-interactive recovery), src/make.tcl + src/modules/punk/mix/cli-999999.0a1.0.tm (event-scoped lock consumers as verified), src/tests/modules/punkcheck/testsuites/punkcheck/ (new concurrency suite) -Detail: goals/G-095-punkcheck-concurrent-writer-safety.md - ### G-097 [proposed] bin utility naming sweep: punk- prefix for remaining punkshell-own tools Scope: src/scriptapps/ + src/scriptapps/bin/ (bits, runtime, tclargs scriptset renames; selfsign experiment scripts), bin/ (regenerated wrapper twins + renamed experiment scripts, old names retired), src/tests/modules/punk/mix/testsuites/scriptwrap/ + src/tests/shell/testsuites/binscripts/ (roundtrip pins), bin/AGENTS.md + referencing docs Detail: goals/G-097-bin-utility-naming-sweep.md diff --git a/goals/G-027-derived-project-pull-updates.md b/goals/G-027-derived-project-pull-updates.md index 350f9576..cf780579 100644 --- a/goals/G-027-derived-project-pull-updates.md +++ b/goals/G-027-derived-project-pull-updates.md @@ -133,9 +133,11 @@ What a pull may touch splits by ownership, not by file type: lifecycle on the OO installtrack layer with a per-event persistence policy - the pull entrypoint should drive that surface (or the batch punkcheck::install consumer); the legacy installfile_* pipeline is retired - to error shims. G-095 (proposed) adds concurrent-writer safety - a pull - writing a derived project's .punkcheck while builds run there is a future - instance of exactly its scenario. + to error shims. G-095 (achieved 2026-07-21 - see + goals/archive/G-095-punkcheck-concurrent-writer-safety.md) added + concurrent-writer safety - a pull writing a derived project's .punkcheck + while builds run there is covered by its advisory event lock, atomic saves + and merge-on-flush, provided the pull drives the current punkcheck (>= 0.6.0). - G-087 (archived - goals/archive/G-087-thin-project-layouts.md) seeded the interim provenance carrier this goal's baseline record needs: generated projects now carry a root punkorigin.toml (format = 1; [generation] utc, diff --git a/goals/G-028-file-locker-identification.md b/goals/G-028-file-locker-identification.md index e92eff7b..4c908d1b 100644 --- a/goals/G-028-file-locker-identification.md +++ b/goals/G-028-file-locker-identification.md @@ -66,6 +66,10 @@ the kind of file this fails on. - Unix analogues (lsof/fuser) could slot behind the same helper interface later; out of scope for acceptance (Windows is where the pain is and where the API answer is non-obvious). +- G-095 (archived) implemented the advisory-lock sibling of this goal's + name-the-blocker principle: punkcheck's lock-timeout error quotes the holder + identity (pid/host/installer/timestamp) read from .punkcheck.lock contents - + see goals/archive/G-095-punkcheck-concurrent-writer-safety.md. - Sibling candidate parked here 2026-07-14 (G-030 follow-up): make.tcl's pre-deploy process sweep matches processes by executable NAME only (tasklist gives no path), so an unrelated same-named executable elsewhere on the system diff --git a/goals/G-095-punkcheck-concurrent-writer-safety.md b/goals/G-095-punkcheck-concurrent-writer-safety.md index 35a41e04..ccc2fdb2 100644 --- a/goals/G-095-punkcheck-concurrent-writer-safety.md +++ b/goals/G-095-punkcheck-concurrent-writer-safety.md @@ -1,6 +1,6 @@ # G-095 punkcheck concurrent-writer safety: atomic saves and advisory event-scoped locking -Status: active +Status: achieved 2026-07-21 Scope: src/modules/punkcheck-999999.0a1.0.tm (save/load chokepoint: atomic rename, lockfile protocol, merge-on-flush, non-interactive recovery), src/make.tcl + src/modules/punk/mix/cli-999999.0a1.0.tm (event-scoped lock consumers as verified), src/tests/modules/punkcheck/testsuites/punkcheck/ (new concurrency suite) Goal: Concurrent writers to the same .punkcheck file cannot corrupt it or silently erase each other's records, and readers never observe torn content - every save is write-temp-then-atomic-rename (bounded retry for Windows sharing violations); an advisory cross-platform lockfile (open {WRONLY CREAT EXCL}, contents naming holder pid/host/installer/timestamp, bounded-retry acquisition, age-based stale-break with warning) serializes whole installer events per punkcheck root; deferred-mode flushes merge own records (INSTALLER by -name, FILEINFO by -targets) into freshly-loaded file state instead of wholesale overwrite; and the duplicate-INSTALLER sanity recovery degrades to a clean actionable error instead of an askuser prompt when stdin is non-interactive. Acceptance: a concurrency test drives two child tclsh installer processes against one .punkcheck folder - the file parses cleanly throughout (a polling load_records_from_file reader hits no TDL parse error), both installers' records are present afterwards, and no duplicate INSTALLER records exist; lock behaviour verified - the second writer observably waits, or times out with a message naming the holder from lockfile contents; stale-break verified against a planted orphan lockfile; suite passes on the project's Windows Tcl 8.6 + Tcl 9 test matrix (capability-gated where child-process facilities require it), with a unix run when an environment is available. @@ -109,3 +109,67 @@ user + agent). timeout message naming the lock holder is the advisory-lock sibling. - Related: G-027 - a derived-project pull writing .punkcheck files while builds run in that project is a future instance of exactly this scenario. + +## Progress + +- 2026-07-21 landed in full (punkcheck 0.6.0; bootsupport re-vendored the same day per the + advisory-protocol transition note; no consumer code changes were needed - see verification): + - Atomic saves: save_records_to_file is write-temp-then-atomic-rename (same-directory temp, + bounded-backoff retry for Windows sharing violations, temp cleanup on failure plus age-gated + sweep of abandoned temps). Complement found during testing: a reader's open can transiently + be denied on Windows while a writer's MoveFileEx replace is in flight, so + load_records_from_file retries transient read failures (bounded ~2s) while leaving TDL parse + errors strict - they now always indicate real corruption. + - Advisory lock: new punkcheck::lock namespace (acquire/release PUNKARGS-documented). + Sibling .lock via open {WRONLY CREAT EXCL}; contents name holder + pid/host/installer/timestamp; the channel is held open for the duration; in-process + re-acquisition is reference-counted (sequential/nested installers over one root in one + process - the make.tcl pattern - cannot self-deadlock). start_event acquires; event end and + installtrack destroy release; flush and chokepoint saves outside an event take a transient + lock. Contention: backoff retry to -timeout (default 15000ms, env PUNKCHECK_LOCK_TIMEOUT) + then an error with errorcode {PUNKCHECK LOCK TIMEOUT } quoting the holder. + Stale-break: provably-dead holder pid on this host (twapi process_exists / kill -0, + capability-gated) breaks immediately; age beyond -staleage (default 300s, env + PUNKCHECK_LOCK_STALEAGE) is the portable baseline. Warnings go to -warnchannel. + - Merge-on-flush: flush re-loads the file under the lock and carries over only this + installer's own records - the INSTALLER record by -name and touched FILEINFO records by + -targets (tracked per instance as events merge finalized records) - preserving other + writers' records instead of wholesale overwrite. + - Tripwire: mtime/size signature captured at load, checked at the save chokepoint; mismatch + warns loudly (protocol-bypassing writer, e.g. an older vendored punkcheck) and proceeds. + - Non-interactive recovery: the duplicate-INSTALLER sanity check errors cleanly (actionable + message naming the duplicate and the recovery path) instead of prompting, when + punkcheck::lib::stdin_is_interactive reports non-interactive - strict twapi GetConsoleMode + on the real stdin handle on Windows (twapi::get_console_handle is unsuitable: it succeeds + for piped children of console shells), -inputmode channel-option probe then test -t 0 on + unix, defaulting to non-interactive when no probe exists. The interactive prompt gains the + synced-targets deletion-wedge caveat (per this goal's note). + - default_excludefiletail_core adds .punkcheck.* so lock/temp protocol siblings are never + treated as installable content (pinned). +- Acceptance verification (2026-07-21): + - concurrency.test (new, 6 tests): two child installer processes over one .punkcheck folder + with a polling load_records_from_file reader - zero parse errors, both INSTALLER records + present, no duplicates, all 12 targetset records present, no lock/temp leftovers; lock + contention - a short-timeout second writer errors naming HOLDER-CHILD from the lockfile + then acquires successfully after release; stale-break against planted orphan lockfiles - + fresh lockfile naming a provably-dead pid (liveness break, warning names the ghost) and + hour-old lockfile from another host (age break at -staleage 5); duplicate-INSTALLER + recovery in a pipe-stdin child errors cleanly with no hang; protocol-sibling exclusion. + - Matrix: Windows Tcl 9.0.3 - all four punkcheck suites 84/84 via runtests; Windows Tcl + 8.6.17 - 84/84 via a direct tcltest drive (the runtests harness itself cannot boot under + native 8.6: its two-stage boot hits missing lpop before compat shims load, and the punk86 + kit runtests route fails on kit/dev punk::args-vs-punk::path skew - both pre-existing, + outside this goal; the punkcheck module and suites are fully 8.6-clean); unix - linux WSL + tclsh 8.6.14, all four suites 84/84 from a native-filesystem staged run (kill -0 liveness + and test -t 0 paths exercised for real). + - Consumers verified without edits: punk::mix::cli and five make.tcl installer sites end + their events (releasing the lock); make.tcl's vfs/bin kit flows destroy their installers + on every path including FAILED branches (destructor releases). punk/mix subtree suites + 150 pass / 1 pre-existing constraint-skip. + - Real-build exercise under re-vendored 0.6.0: make.tcl modules and a full make.tcl project + complete exit 0 with zero lock timeouts, zero tripwire warnings, unchanged skip/copy + decisions, all kit installs skipping 'no change detected', and no .punkcheck.lock or + .punkcheck.tmp.* leftovers in the tree. +- Not addressed (explicitly out of this goal's acceptance): the punkcheck MAX_PATH load/save + normalization rider inherited from G-087/G-094 - with this goal archived it has no live + carrier and was flagged to the user as a candidate goal. diff --git a/punkproject.toml b/punkproject.toml index 06eb3d1a..fb909703 100644 --- a/punkproject.toml +++ b/punkproject.toml @@ -1,4 +1,4 @@ [project] name = "punkshell" -version = "0.17.2" +version = "0.17.3" license = "BSD-2-Clause"