punkshell/goals/G-117-self-describing-runtimes.md

G-117 Self-describing family runtimes: embedded artifact record + metadata schema v1

Status: active Scope: src/buildsuites/suite_tcl90/ (kit-family staging embeds the record; family_artifacts.tcl schema v1 fields + emission ordering; -Doriginurl/-Dpackager options), tools/family_check.tcl (embedded-record verification), src/scriptapps/bin/punk-runtime.* + bin/punk-runtime.cmd via rewrap ('info' action; schema-tolerant parsing), punkbin AGENTS.md (record relationship + schema/field documentation) Goal: every family runtime carries a copy of its artifact metadata record INSIDE the attached image (e.g /app/punkbin-artifact.toml, written at kit-family staging time - necessarily WITHOUT the final sha1, which is computed over the finished binary and lives only in the sidecar toml + sha1sums), so a runtime separated from its sidecar remains identifiable via a punk-runtime 'info' action (embedded record and sidecar shown side by side, disagreements flagged); AND the metadata record is hardened as SCHEMA v1: schema (format version), build_id (uuid stamped in embedded AND sidecar - the offline correlation key for renamed copies), origin (canonical artifact repo the artifact was built FOR - mirrors preserve it; -Doriginurl, default punkbin), packager (declared identity; -Dpackager > env > git identity > unrecorded), project/project_url, license summary, build_host_platform - so records circulating on punkbin-compatible repos from different packagers stay interpretable and attributable. Acceptance: kit-family embeds the record in all three members with the embed-then-hash ordering documented (embedded copy carries the v1 fields but no self-sha1; the sidecar + sha1sums remain the integrity authority); family_check verifies the embedded record exists and its variant/patchlevel/battery fields MATCH the probed facts for each member; 'punk-runtime info ' reports identity for (a) a fetched artifact with its sidecar and (b) a bare RENAMED copy with no sidecar - the embedded-read mechanism is decided and documented in the work (zip-central-directory read of the exe-appended archive by the host payload where available vs a cooperative probe that executes the target with a generated script, execution caveat stated) - and flags embedded-vs-sidecar disagreement; the v1 field set is documented in punkbin AGENTS.md with each field's semantics (notably origin's built-for-not-published-on meaning and packager's declarative-not-verified status, with minisign sidecars noted as the verification complement); family_artifacts emits all v1 fields with build_id present and IDENTICAL in embedded and sidecar copies, and the emitted records carry brief '#' comment lines documenting the non-obvious fields (toml-spec comments, ignored by the existing line-based consumers); punk-runtime metadata parsing tolerates unknown fields and absent schema (pre-v1 r1 records keep working); both payloads in parity with the roundtrip pin green; the next family emission after this goal lands is r2 carrying the full v1 record (r1 stays immutable as published).

Context

Drafted 2026-07-22 (user-approved wording) from two exchanges during the G-103 publication session:

MOTIVATION (embedding): the published family artifacts are metadata-carrying via SIDECAR tomls only - nothing is stamped inside the executables (a deliberate G-103 naming-decision property: the metadata record is authoritative, the filename identifies, the binary stays unstamped). The implication: an exe separated from its toml (out-of-band copy, rename) is identifiable only by executing probes against it or sha1-matching it back to a repo's sha1sums. Embedding a COPY of the record in the attached image makes runtimes self-describing without changing the authority model - the sidecar + sha1sums remain the integrity authority; the embedded copy is a convenience duplicate that cannot contain its own final sha1 (embed at staging -> wrap -> hash -> sidecar; no circularity).

SCHEMA v1 (user-floated fields, analysed 2026-07-22):

• origin: the user's "repository url it was first published on" idea, with a semantic reframe dissolving the timing problem they spotted (emission precedes publication; publish-time sidecar mutation would permanently diverge embedded vs sidecar - the exact disagreement 'info' flags): origin = the canonical artifact repo the artifact was BUILT FOR, known at emission (-Doriginurl, default punkbin), truthful regardless of publish timing, and preserved by mirrors - an artifact found on a third-party repo still declares its home. Where-it-actually-lives is the hosting repo's own business (a repo-level identity file beside platforms.txt/defaults.txt is a separate later idea).
• packager: declared identity for multi-packager punkbin-compatible ecosystems (-Dpackager > env > the building checkout's git identity > "unrecorded"). DECLARATIVE, not proof - the verifiable layer is signing, with punkbin's existing minisign practice (zig tool archives) as precedent; per-artifact .minisig sidecars are the natural future complement, deliberately outside the metadata file (a signature cannot live inside what it signs).
• schema: format-version field - the most important future-proofing addition; nearly free now, impossible to retrofit onto records already circulating.
• build_id: uuid stamped in BOTH embedded and sidecar copies - the offline correlation key that re-joins a renamed stray exe to its record without content hashing.
• license summary + project/project_url: distribution-platform eligibility (the G-004 motivation) asks licensing first; the project pointer makes a stray artifact self-explaining beyond its origin repo. Component licenses ride inside the image (tcl_library etc).
• build_host_platform: near-free now, genuinely interesting once G-105 cross-builds exist (built ON win32-x86_64 FOR linux-x86_64).
• REJECTED: lifecycle fields (superseded_by, expires) - repo-level curation (defaults.txt and successors), not artifact facts; artifacts stay immutable statements of what they are.

Comment lines: emitted records gain brief '#' comments documenting non-obvious fields (origin/packager semantics). Toml-spec comments; the existing line-based consumers (punk-runtime ps1 regex / bash sed per-line matching, the summary parsers from G-107, achieved - see goals/archive/G-107-buildsuite-library-tests.md) ignore them by construction - family_artifacts already emits a '#' header line today.

Read-mechanism design fork (the main in-goal decision): exe-appended zips are readable from the central directory by generic zip readers (.NET System.IO.Compression for the ps1 payload, unzip for bash) WITHOUT executing the target - preferred, since identifying an untrusted stray binary by running it is what an identity mechanism should avoid; the cooperative probe (execute the target with a generated script) is the documented fallback for hosts without a zip reader.

Progress

• 2026-07-24 activated (user). Confirmed at activation: -Doriginurl default punkbin and the -Dpackager fallback chain (flag > env > building checkout's git identity > "unrecorded"); read-mechanism preference stands (zip central-directory read, no execution of the target; cooperative probe documented as fallback). Sequencing (user-confirmed): ONE suite rebuild cycle shared with G-118's patch-rev items (reopen-noise removal + TCLSH_PIPEREPL_DEBUG gate, publish ::tclsh(reopened)) - patch rev + v1 embedding -> fresh family emission re-issuing r2 WITH v1 records (current local r2 is pre-v1 and unpublished; published r1 stays immutable) -> family_check -> only then punkbin r2 publication + defaults.txt flip (G-103 follow-through), with G-119 fixture-server verdict tests after that.
• 2026-07-24 implementation design (recorded before staging work; from reading tools/family_artifacts.tcl): family_artifacts.tcl runs AFTER kits are built (copies finished exes, hashes them), so the embed happens at the kit-family staging step and emission READS IT BACK: family_artifacts mounts each kit exe (tcl::zipfs::mount on the file) and loads the embedded record, deriving the sidecar's shared fields (build_id, origin, packager, schema...) from it - single source of truth, identical build_id by construction, and every emission run inherently verifies the embed exists (fail if absent). Staging generates build_id per member and writes the embedded record WITHOUT sha1/size (finished-binary facts stay sidecar-only). Embedded record filename namespaced per the G-025 reconciliation note.
• 2026-07-24 staging anatomy (build905.zig ~2337-2466; build.zig is the G-102 entry shim): two shared addWriteFiles payload trees (0 = core for plain+punk, 1 = bi) are wrapped per-kit by tools/zipfs_mkimg.tcl (-infile prefix exe, -indir tree). The embedded record is PER-MEMBER (variant/working_name/build_id) so it cannot live in the shared trees: plan is a per-kit record file (small addWriteFiles or generated arg) handed to zipfs_mkimg.tcl via a new optional embed argument merged into the image at wrap time. -Doriginurl/-Dpackager become b.option()s beside familyrev (~2459) and flow to both the record generation and family_artifacts invocation. BUILD_ID DETERMINISM FORK to decide at implementation: (i) random uuid per configure - simple but busts the zig cache every run; (ii) uuid-shaped digest derived from stable identity inputs (working_name | familyrev | source checkout uuids) - reproducible, cache-friendly, still unique per artifact identity + rev + source state. Leaning (ii); record the choice + rationale here when implemented.
• Remaining for acceptance: extend the kit-family staging step per the anatomy above, family_artifacts v1 fields + mount-read derivation + '#' field comments + -Doriginurl/-Dpackager plumbing, family_check embedded-record verification, punk-runtime 'info' action (ps1 + bash + cmd rewrap; zip central-directory read, cooperative-probe fallback documented), punkbin AGENTS.md schema v1 docs, suite rebuild + r2 re-emission + wrap trial (also verifies the G-118 items 1+7 patch rev behaviour on rebuilt kits).

Notes

G-025 relationship (recorded 2026-07-24 after overlap review; neither file previously referenced the other): G-025 is the KIT-layer counterpart - make.tcl stamps punkshell kits with project version + runtime name + vfs folder, reported by a subcommand and an in-shell command. Two implementation considerations from the review: (a) make.tcl's kit wrap overlays the kit vfs onto the runtime's attached image without clearing it, so this goal's embedded record is expected to SURVIVE into wrapped kits - state the intended behaviour when implementing (a kit inheriting its runtime's record gives G-025's report a second provenance layer for free and closes the kit->-rN traceability gap); (b) namespace the embedded record's filename so a future G-025 kit-stamp file can coexist beside it in the same image. G-025's "input runtime binary name" stamp field should cite this record's build_id/artifact name when present (exact -rN linkage vs mutable working name).

Relationships: archived G-103 (metadata shape + family staging this extends - see goals/archive/G-103-runtime-kit-family.md), G-067 (library-class artifacts should adopt the same schema + embedding pattern), G-105 (embedded target field aids cross-platform staging hygiene alongside the sidecar !TARGET-MISMATCH check), G-006/G-067 consent gates (unchanged - metadata travels with artifacts through the existing channels).

• G-116 relationship (recorded 2026-07-24 after overlap review): its tls battery metadata fields (tls version, backend name+version) extend THIS schema - v1's comment-documented field set is the extension point; no parallel record.