diff --git a/CHANGELOG.md b/CHANGELOG.md index 680919da..2662139f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,21 @@ The latest `## [X.Y.Z]` header must match the `version` field in `punkproject.to Entries are newest-first; one bullet per notable change. See the root `AGENTS.md` "Project Versioning" section for the bump policy. +## [0.31.0] - 2026-07-30 + +- New `make.tcl libfetch` subcommand (G-139 phase 1): fetches the punkbin + lib-tier library artifacts declared in `src/runtime/libpackages.toml` + (suite-built tcllib/tcllibc zips, both Tcl generations - G-138) into the + untracked `bin/packages//` input tier with sha1 verification + against the server's per-target sha1sums.txt, and materializes the + installed-shape package trees under `bin/packages//tcl/` + (self-describing via their embedded punkbin-artifact.toml records). + Consent is keyed to server trust: the canonical punkbin origin needs no + flag; any other origin (`-serverurl` / env `PUNKBIN_URL`) requires the + explicit `-trust-server` flag. `file://` origins serve local mirrors and + fixtures. Idempotent: present artifacts that verify are skipped and + current trees are left untouched (`-force` overrides). + ## [0.30.7] - 2026-07-29 - Baked kits pinned archive-relative (G-134): every assembled kit image is diff --git a/bin/AGENTS.md b/bin/AGENTS.md index f6de5aa9..ba5f04ba 100644 --- a/bin/AGENTS.md +++ b/bin/AGENTS.md @@ -361,6 +361,26 @@ punkshell platform-dir names defined by the `punk::platform` module and surfaced aarch64->arm64; the bash payload's local-platform prongs emit canonical names - the FreeBSD dir was realigned from `freebsd-amd64` to `freebsd-x86_64` accordingly). +### Library package input tier (`bin/packages/`, G-139) + +`bin/packages//` is the untracked input tier for punkbin LIB-TIER library +artifacts (tcllib/tcllibc as immutable generation-tagged zips - shellspy G-138), +mirroring the `bin/runtime//` store idiom: per-target folders holding +the sha1-verified `-r` zips, their `.toml` sidecars and a local copy of the +server's `sha1sums.txt`, plus the materialized installed-shape package trees +under `bin/packages//tcl//` (generation subfolders because +both Tcl generations install identically-named folders). Materialized trees are +self-describing via their embedded `punkbin-artifact.toml` records (the G-138 +schema) - `make.tcl libfetch` uses that record to skip re-materializing a +current tree. Populated by `tclsh src/make.tcl libfetch` from the declarations +in `src/runtime/libpackages.toml` (consent keyed to server trust: the canonical +punkbin origin needs no flag, any other origin - `-serverurl` or env +`PUNKBIN_URL` - requires the explicit `-trust-server` flag; `file://` origins +serve local mirrors/fixtures). Everything under `bin/packages/` is +fetch-regenerable and ignored by both VCS (`/bin/*` rules) - never commit from +it; downstream consumption into `lib_tcl/` and kit vfs trees is the +remainder of G-139. + ### Interactive verification shells - Interactive console/repl verification should cover both Tcl generations - behaviour can differ materially (e.g. the Tcl 8.6 windows console channel driver vs the Tcl 9 rewrite). Use a Tcl 8.6-based punk shell (`punksys.exe`) and a current Tcl 9-based punk shell (named for the Tcl release it embeds, e.g. `punk902z.exe` at the time of writing - ask the user which is current rather than assuming). `info patchlevel` in-session confirms the runtime. diff --git a/goals/G-139-punkbin-library-consumption.md b/goals/G-139-punkbin-library-consumption.md index b179ff7e..455b2a42 100644 --- a/goals/G-139-punkbin-library-consumption.md +++ b/goals/G-139-punkbin-library-consumption.md @@ -119,3 +119,63 @@ State at drafting worth recording: - Activation survey 2026-07-30: goals_xref score G-139 re-run at activation (user-directed, same day as drafting - no goals drafted in the interval); no overlaps beyond the drafting survey. + +## Progress + +- 2026-07-30 phase 1 landed: the fetch/materialize input tier and its + make.tcl entrypoint (project 0.31.0). + - `make.tcl libfetch` (argdoc'd subcommand, workflow-text + plain-help + updated): reads `src/runtime/libpackages.toml` declarations + ([artifact.] name/target), fetches from /lib// + into `bin/packages//` - zip + .toml sidecar, each sha1-verified + against the server's fetched per-target sha1sums.txt (kept as the local + tier copy, the punk-runtime store idiom) - then materializes each + verified zip via punk::zip::unzip (defaults: crc-verified, mtimes + restored, punkzip-accelerator-eligible) into + `bin/packages//tcl//` (generation subdirs: both + generations install identically-named folders). Idempotent: present + artifacts re-verify and are skipped; a materialized tree is skipped + while its embedded record's name matches the declared artifact (the + G-138 self-description payoff); -force re-does both. + - Consent (per this goal's acceptance + the G-123 posture): canonical + punkbin origin (the punk-runtime raw-base convention) prompts nothing; + ANY other origin - -serverurl flag or PUNKBIN_URL env - refuses before + network access without the explicit -trust-server flag. Never an + interactive prompt. file:// origins supported natively (local mirrors, + fixtures). Transport for http(s): Tcl http (+tls for https) when + loadable, else curl, else PowerShell. + - Verified 2026-07-30 against a file:// mirror assembled from BOTH + suites' out/library/punkbin staging (the shape publication will + produce): trust-gate refusal without the flag; consented fetch of all + 4 declared r1 artifacts with sha1 verification; materialization of all + 4 trees; full idempotent re-run (verified-present + tree-current); + package-load proof from the tier (md5 2.0.9 + tcllibc from + bin/packages/*/tcl9 under the plain family kit, accelerator engaged, + ifneeded source path in the tier). + - Emission-side amendment (both suites' library_artifacts.tcl): sidecar + tomls now ride the per-tier staging sha1sums.txt (the punkbin repo's + build_sha1sums.tcl already sums them; the staging mirror now matches), + so sidecars are verifiable from a staging-derived mirror too. + - bin/packages is ignored by both VCS via the existing /bin/* rules (no + ignore edits needed). Docs: bin/AGENTS.md "Library package input tier", + src/runtime/AGENTS.md libpackages.toml bullet, workflow diagram block. + - Session note: the suite_tcl86 staging r1 zips changed bytes when its + installer/critcl Run steps re-ran under this session's different PATH + (zig Run cache keys include the env) - immaterial pre-publication, and + exactly the deliberate-publish revision discipline's point; the tcl9 + zips stayed byte-identical across three independent emissions. +- Remaining for acceptance: libs-step materialization of + lib_tcl/allplatforms + lib_tcl/ from the tier with + punkcheck provenance; tcllib/tcllibc vendorlib_vfs.toml (or successor) + declarations with supersedes removing tcllib1.21 (incl. nested tcllibc + 0.4 + md5c) from 8.6-family kits; punk86-family tcllib 2.0 upgrade + validated by kit-hosted runtests vs a same-day baseline (differences + dispositioned); vendored src/vendorlib_tcl8|9 tcllib-tree removal in its + own commit after a bake fed from the artifact path reproduces working + kits; a bake on a checkout without those trees shipping kits with + verifiable embedded provenance; the derived-project decision recorded + here (fetch default; in-project suite build per G-104 as the documented + alternative - the stale samplesuite1 download_and_build.config stub to be + replaced/removed when recording it). Also pending: publication of the r1 + artifacts to the real punkbin origin (user step) so libfetch's canonical + path goes live end-to-end. diff --git a/punkproject.toml b/punkproject.toml index b07bd4f1..87a588c2 100644 --- a/punkproject.toml +++ b/punkproject.toml @@ -1,4 +1,4 @@ [project] name = "punkshell" -version = "0.30.7" +version = "0.31.0" license = "BSD-2-Clause" diff --git a/src/buildsuites/suite_tcl86/tools/library_artifacts.tcl b/src/buildsuites/suite_tcl86/tools/library_artifacts.tcl index 46dd26da..4929fc2d 100644 --- a/src/buildsuites/suite_tcl86/tools/library_artifacts.tcl +++ b/src/buildsuites/suite_tcl86/tools/library_artifacts.tcl @@ -383,6 +383,9 @@ foreach {folder tier license testlibs pkgpath} $opt(-packages) { fconfigure $f -translation lf puts $f [compose_record $ident [dict create sha1 $sha1 size $size built $built] $tests] close $f + #sidecars ride sha1sums too (punkbin repo convention: build_sha1sums.tcl sums + #every tracked file) - so a staging mirror serves verifiable sidecars as well + dict lappend sha1lines $tier "[sha1::sha1 -hex -file $mf] *[file tail $mf]" lappend emitted "$folder -> lib/$tier/$artifact" note "emitted lib/$tier/$artifact (sha1 $sha1, [expr {$size/1024}] KB, $members members) + [file tail $mf]" } diff --git a/src/buildsuites/suite_tcl90/tools/library_artifacts.tcl b/src/buildsuites/suite_tcl90/tools/library_artifacts.tcl index 46dd26da..4929fc2d 100644 --- a/src/buildsuites/suite_tcl90/tools/library_artifacts.tcl +++ b/src/buildsuites/suite_tcl90/tools/library_artifacts.tcl @@ -383,6 +383,9 @@ foreach {folder tier license testlibs pkgpath} $opt(-packages) { fconfigure $f -translation lf puts $f [compose_record $ident [dict create sha1 $sha1 size $size built $built] $tests] close $f + #sidecars ride sha1sums too (punkbin repo convention: build_sha1sums.tcl sums + #every tracked file) - so a staging mirror serves verifiable sidecars as well + dict lappend sha1lines $tier "[sha1::sha1 -hex -file $mf] *[file tail $mf]" lappend emitted "$folder -> lib/$tier/$artifact" note "emitted lib/$tier/$artifact (sha1 $sha1, [expr {$size/1024}] KB, $members members) + [file tail $mf]" } diff --git a/src/make.tcl b/src/make.tcl index 1a4aafe3..61ce8b36 100644 --- a/src/make.tcl +++ b/src/make.tcl @@ -187,7 +187,7 @@ namespace eval ::punkboot { variable foldername [file tail $scriptfolder] variable pkg_requirements [list]; variable pkg_missing [list];variable pkg_loaded [list] variable help_flags [list -help --help /? -h] - variable known_commands [list bakehouse packages modules libs bake bakelist vfslibs bin info check shell vendorupdate bootsupport vfscommonupdate projectversion workflow buildsuite tool project vfs] + variable known_commands [list bakehouse packages modules libs bake bakelist vfslibs bin info check shell vendorupdate libfetch bootsupport vfscommonupdate projectversion workflow buildsuite tool project vfs] } @@ -2135,6 +2135,10 @@ proc ::punkboot::punkboot_gethelp {args} { append h " - propagate declared vendored platform-library packages into kit vfs lib_tcl trees" \n append h " - declarations (per-package per-kit, with superseded-version removal) live in src/runtime/vendorlib_vfs.toml" \n append h " - also runs automatically as a phase of '$scriptname bake' (and therefore of '$scriptname bakehouse')" \n \n + append h " $scriptname libfetch ?-serverurl ? ?-trust-server? ?-force?" \n + append h " - fetch declared punkbin lib-tier library artifacts (src/runtime/libpackages.toml) into bin/packages/" \n + append h " - sha1-verified against the server's per-target sha1sums.txt; materializes package trees under bin/packages//tcl" \n + append h " - canonical origin trusted by default; other servers (incl. PUNKBIN_URL overrides) require -trust-server" \n \n append h " $scriptname vfscommonupdate" \n append h " - promotion gate: update the src/vfs/_vfscommon.vfs from compiled src/modules and src/lib etc" \n append h " - before calling this (followed by '$scriptname bake') - you can test using '(.exe) dev'" \n @@ -2323,6 +2327,26 @@ DIAGRAM 2 - KIT ASSEMBLY DETAIL (the 'make.tcl bake' stage; incl. vfslibs phase) running shell still holds the old exe [K5] +LIBRARY-ARTIFACT FETCH (G-139; input tier so far - lib_tcl/kit consumption follows) +---------------------------------------------------------------------------- + + src/runtime/libpackages.toml which punkbin LIB-TIER artifacts this checkout + consumes (declared artifact names + targets) + | + | 'make.tcl libfetch' - sha1-verified download of the declared + | --r.zip artifacts + .toml sidecars from + | /lib// (canonical punkbin origin trusted by + | default; -serverurl/PUNKBIN_URL override needs -trust-server; + | file:// origins serve local mirrors/fixtures), then + | materialization of each verified zip's package tree + v + bin/packages// verified -r zips + toml sidecars (input tier, + untracked - mirrors bin/runtime/) + bin/packages//tcl// + materialized installed-shape trees, self-describing + via their embedded punkbin-artifact.toml records + + KEY / NOTES ---------------------------------------------------------------------------- [K1] Source modules carry the literal magic version 999999.0a1.0 in the filename. @@ -2527,6 +2551,7 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO vfslibs "Propagate declared vendored platform-library packages into kit vfs lib_tcl trees" bin "Install executables from src/bin into /bin, then build kits as for bake" vendorupdate "Update src/vendormodules based on src/vendormodules/include_modules.config" + libfetch "Fetch declared punkbin lib-tier library artifacts into bin/packages/ (sha1-verified) and materialize the package trees" bootsupport "Promotion gate: update src/bootsupport modules from built project modules" vfscommonupdate "Promotion gate: replace src/vfs/_vfscommon.vfs modules and libs with the project's built set" info "Show the name and base folder of the project to be built" @@ -2630,6 +2655,24 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO Also runs automatically as a phase of 'make.tcl bake' (and therefore of 'make.tcl bakehouse'); this standalone narrowing serves quick iteration on the declarations." + libfetch + " + Fetch the punkbin LIB-TIER library artifacts declared in + src/runtime/libpackages.toml into the untracked input tier + bin/packages// - immutable --r.zip + artifacts plus their .toml sidecars, each verified against the + server's per-target sha1sums.txt - then materialize each verified + zip's installed-shape package tree under + bin/packages//tcl/ (self-describing via the embedded + punkbin-artifact.toml record; G-138/G-139). + Present artifacts that already verify are skipped (-force + re-downloads). The canonical punkbin origin is trusted by + default; -serverurl (or env PUNKBIN_URL) overrides it, and any + NON-canonical origin requires the explicit -trust-server flag + (never an interactive prompt - unattended runs stay unattended). + A file:// origin is supported for local mirrors/fixtures. + Transport for http(s): Tcl http+tls when loadable, else curl, + else PowerShell (windows)." bin " Install executables from src/bin to /bin, then check vfs @@ -2799,6 +2842,30 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO (packages -> test -> bootsupport/vfscommonupdate -> commit -> bake). Pass -dirty-abort 0 to build anyway with PROVENANCE-WARNINGs." } + #libfetch fragments (G-139): consent is keyed to SERVER TRUST (the G-123 + #posture) - the canonical punkbin origin needs no acknowledgement, any + #other origin needs the explicit flag. Never an interactive prompt. + variable OPT_SERVERURL { + -serverurl -type string -default "" -help\ + "Artifact server base url override (default: the canonical punkbin + origin; env PUNKBIN_URL overrides the default, this flag overrides + both). The lib tier is addressed as /lib//. + file:// urls are supported for local mirrors and fixtures. + A NON-canonical origin additionally requires -trust-server." + } + variable OPT_TRUSTSERVER { + -trust-server -type none -default 0 -help\ + "Acknowledge fetching from a NON-canonical artifact server + (consent keyed to server trust - the interim G-123/G-006 + mechanism). Without this flag, a non-canonical -serverurl or + PUNKBIN_URL aborts before any network access. Fetches from the + canonical origin never require it." + } + variable OPT_FORCE { + -force -type none -default 0 -help\ + "Re-download and re-materialize even when a present artifact + already verifies against the server sha1sums." + } #which shared option fragments each subcommand declares #(shell and help have bespoke definitions below: shell passes trailing args # through to the repl untouched, help takes a subject value) @@ -2812,6 +2879,7 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO vfslibs {OPT_DIRTYABORT OPT_CONFIRM} bin {OPT_FORCEKILL OPT_DIRTYABORT OPT_CONFIRM} vendorupdate {OPT_CONFIRM} + libfetch {OPT_SERVERURL OPT_TRUSTSERVER OPT_FORCE} bootsupport {OPT_DIRTYABORT OPT_CONFIRM} vfscommonupdate {OPT_DIRTYABORT OPT_CONFIRM} info {OPT_CONFIRM} @@ -2829,6 +2897,9 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO OPT_DIRTYABORT "?-dirty-abort?" OPT_DIRTYABORT_ON "?-dirty-abort 1|0?" OPT_CONFIRM "?-confirm 0|1?" + OPT_SERVERURL "?-serverurl ?" + OPT_TRUSTSERVER "?-trust-server?" + OPT_FORCE "?-force?" } #per-subcommand positional values (default: none). bake (and its deprecated #alias vfs) and bakelist take optional kit names (G-121 selective bake). @@ -2940,7 +3011,7 @@ if {[info exists ::env(PUNKBOOT_PLAIN)] && [string is true -strict $::env(PUNKBO variable SUBGROUPS { "build & bake" {bakehouse packages modules libs bake vfslibs bin} "promotion gates" {bootsupport vfscommonupdate} - "source maintenance" {vendorupdate} + "source maintenance" {vendorupdate libfetch} "buildsuites" {buildsuite} "vendored tools" {tool} "informational" {info check bakelist projectversion workflow help} @@ -2996,6 +3067,9 @@ set help_exitcode 0 set ::punkboot::opt_forcekill 0 set ::punkboot::opt_dirty_abort 0 set ::punkboot::opt_confirm 1 +set ::punkboot::opt_serverurl "" +set ::punkboot::opt_trust_server 0 +set ::punkboot::opt_force 0 set ::punkboot::opt_kitnames [list] ;#G-121: requested kit names for bake/bakelist (empty = all configured) if {$::punkboot::punkargs_ok} { @@ -3046,7 +3120,7 @@ if {$::punkboot::punkargs_ok} { } set ::punkboot::command $subcommand set _opts [dict get $argd opts] - foreach {_optname _var} {-k opt_forcekill -dirty-abort opt_dirty_abort -confirm opt_confirm} { + foreach {_optname _var} {-k opt_forcekill -dirty-abort opt_dirty_abort -confirm opt_confirm -serverurl opt_serverurl -trust-server opt_trust_server -force opt_force} { if {[dict exists $_opts $_optname]} { set ::punkboot::$_var [dict get $_opts $_optname] } @@ -3083,6 +3157,12 @@ if {$::punkboot::punkargs_ok} { switch -- $a { -k {set ::punkboot::opt_forcekill 1} -dirty-abort {set ::punkboot::opt_dirty_abort 1} + -trust-server {set ::punkboot::opt_trust_server 1} + -force {set ::punkboot::opt_force 1} + -serverurl { + set ::punkboot::opt_serverurl [lindex $argsleft 0] + set argsleft [lrange $argsleft 1 end] + } -confirm { set v [lindex $argsleft 0] set argsleft [lrange $argsleft 1 end] @@ -5121,6 +5201,303 @@ if {$::punkboot::command eq "bootsupport"} { +if {$::punkboot::command eq "libfetch"} { + #G-139 phase 1: fetch the punkbin LIB-TIER artifacts declared in + #src/runtime/libpackages.toml into the untracked bin/packages/ input + #tier (mirroring bin/runtime/) with sha1 verification against the + #server's per-target sha1sums.txt, then materialize each verified zip's + #installed-shape package tree under bin/packages//tcl/ (the tree + #is self-describing via its embedded punkbin-artifact.toml - the G-138 + #record). Consumption into lib_tcl/ and kit vfs trees is the remainder + #of G-139 - this entrypoint owns fetch + verify + materialize only. + + proc ::punkboot::libfetch_transfer {url outfile} { + #one url -> local file. file:// is served natively (local mirrors, + #fixtures - no network, no external tools). http(s) transports in + #order: Tcl http (+tls for https) when loadable, curl (ubiquitous, + #including stock windows 10+), powershell Invoke-WebRequest (windows). + #Returns the transport used; errors when all fail. + file mkdir [file dirname $outfile] + if {[string match -nocase "file://*" $url]} { + set path [string range $url 7 end] + if {[regexp {^/+([A-Za-z]:/.*)$} $path -> winpath]} { + set path $winpath ;#file:///C:/x -> C:/x + } + if {![file isfile $path]} { + error "file origin has no such file: $path" + } + file copy -force $path $outfile + return "file-copy" + } + set is_https [string match -nocase "https://*" $url] + set tcl_http_ok 0 + if {![catch {package require http}]} { + if {!$is_https} { + set tcl_http_ok 1 + } elseif {![catch {package require tls}]} { + catch {http::register https 443 [list ::tls::socket -autoservername true]} + set tcl_http_ok 1 + } + } + if {$tcl_http_ok} { + set fd [open $outfile wb] + set ok 0 + if {![catch {http::geturl $url -channel $fd -binary 1} tok]} { + if {[http::status $tok] eq "ok" && [http::ncode $tok] == 200} {set ok 1} + http::cleanup $tok + } + close $fd + if {$ok} {return "tcl-http"} + file delete -force $outfile + } + set curl [auto_execok curl] + if {$curl ne ""} { + if {![catch {exec {*}$curl --fail -sSL -o $outfile $url 2>@1} curlerr]} { + return "curl" + } + file delete -force $outfile + } + if {$::tcl_platform(platform) eq "windows"} { + set ps [auto_execok powershell] + if {$ps ne ""} { + set pscmd "\$ProgressPreference='SilentlyContinue'; Invoke-WebRequest -Uri '$url' -OutFile '$outfile' -ErrorAction Stop" + if {![catch {exec {*}$ps -NoProfile -NonInteractive -Command $pscmd 2>@1} pserr]} { + return "powershell" + } + file delete -force $outfile + } + } + set detail "" + if {[info exists curlerr]} {set detail " (curl: [string range $curlerr 0 200])"} + error "all transports failed for $url$detail" + } + + set libfetch_config $sourcefolder/runtime/libpackages.toml + #Canonical origin: the punkbin raw-file base (the punk-runtime.cmd + #convention; lib tier addressed as /lib//...). + #Server-trust consent (G-139, the G-123 posture): the canonical origin + #never gates; ANY other origin - flag or PUNKBIN_URL env - requires the + #explicit -trust-server flag. Never an interactive prompt: unattended + #runs stay unattended, and refusal happens before any network access. + set libfetch_canonical "https://www.gitea1.intx.com.au/jn/punkbin/raw/branch/master" + set libfetch_origin $libfetch_canonical + if {[info exists ::env(PUNKBIN_URL)] && $::env(PUNKBIN_URL) ne ""} { + set libfetch_origin $::env(PUNKBIN_URL) + } + if {$::punkboot::opt_serverurl ne ""} { + set libfetch_origin $::punkboot::opt_serverurl + } + set libfetch_origin [string trimright $libfetch_origin /] + if {$libfetch_origin ne $libfetch_canonical && !$::punkboot::opt_trust_server} { + puts stderr "$A(BAD)LIBFETCH: origin '$libfetch_origin' is not the canonical punkbin origin$A(RST)" + puts stderr "LIBFETCH: canonical: $libfetch_canonical" + puts stderr "LIBFETCH: fetching from a non-canonical server requires the explicit -trust-server flag (consent keyed to server trust - G-139/G-123). Nothing was fetched." + exit 3 + } + if {![file exists $libfetch_config]} { + puts stdout "LIBFETCH: no config at $libfetch_config - nothing declared, nothing to fetch" + } else { + if {[catch {package require tomlish} tomlish_err]} { + puts stderr "$A(BAD)LIBFETCH: cannot process $libfetch_config - tomlish package unavailable ($tomlish_err)$A(RST)" + exit 3 + } + if {[catch {package require sha1} sha1_err]} { + puts stderr "$A(BAD)LIBFETCH: sha1 package unavailable ($sha1_err) - cannot verify artifacts$A(RST)" + exit 3 + } + if {[catch {package require punk::zip} punkzip_err]} { + puts stderr "$A(BAD)LIBFETCH: punk::zip package unavailable ($punkzip_err) - cannot materialize package trees$A(RST)" + exit 3 + } + set fd [open $libfetch_config r] + fconfigure $fd -encoding utf-8 + set libfetch_tomldata [read $fd] + close $fd + if {[catch {tomlish::to_dict [tomlish::from_toml $libfetch_tomldata]} libfetch_dict]} { + puts stderr "$A(BAD)LIBFETCH: failed to parse $libfetch_config\n$libfetch_dict$A(RST)" + exit 3 + } + if {![llength [info commands ::punkboot::lib::toml_untag]]} { + #tomlish::to_dict values are type-tagged (also defined by the vfslibs + #phase - that block does not run for the libfetch command) + proc ::punkboot::lib::toml_untag {tagged} { + if {[dict get $tagged type] eq "ARRAY"} { + set plain [list] + foreach el [dict get $tagged value] { + lappend plain [::punkboot::lib::toml_untag $el] + } + return $plain + } + return [dict get $tagged value] + } + } + set libfetch_artifacts [dict create] + if {[dict exists $libfetch_dict artifact]} { + set libfetch_artifacts [dict get $libfetch_dict artifact] + } + if {![dict size $libfetch_artifacts]} { + puts stdout "LIBFETCH: no \[artifact.\] entries found in $libfetch_config" + } + puts stdout "LIBFETCH: origin $libfetch_origin ([dict size $libfetch_artifacts] declared artifact(s))" + set libfetch_sums [dict create] ;#target -> dict(filename -> sha1) from the server's sha1sums.txt + set libfetch_failed 0 + set libfetch_actions [list] + dict for {aname aentry} $libfetch_artifacts { + foreach reqkey {name target} { + if {![dict exists $aentry $reqkey]} { + puts stderr "$A(BAD)LIBFETCH: entry artifact.$aname in $libfetch_config is missing required key '$reqkey'$A(RST)" + exit 3 + } + } + set artifact [punkboot::lib::toml_untag [dict get $aentry name]] + set target [punkboot::lib::toml_untag [dict get $aentry target]] + if {![regexp {^(.+)-(tcl[0-9]+)-r([0-9]+)\.zip$} $artifact -> pkgfolder generation rev]} { + puts stderr "$A(BAD)LIBFETCH: artifact.$aname name '$artifact' does not match -tcl-r.zip$A(RST)" + exit 3 + } + set tierdir $projectroot/bin/packages/$target + file mkdir $tierdir + #server truth per target: lib//sha1sums.txt (fetched once per + #run per target; the local copy in the tier mirrors the punk-runtime + #store idiom) + if {![dict exists $libfetch_sums $target]} { + set sumsurl "$libfetch_origin/lib/$target/sha1sums.txt" + set sumslocal $tierdir/sha1sums.txt + if {[catch {::punkboot::libfetch_transfer $sumsurl $sumslocal} terr]} { + puts stderr "$A(BAD)LIBFETCH: cannot fetch $sumsurl - $terr$A(RST)" + exit 3 + } + set sfd [open $sumslocal r] + set sumsdata [read $sfd] + close $sfd + set sums [dict create] + foreach ln [split $sumsdata \n] { + set ln [string trim $ln] + if {$ln eq ""} {continue} + set sp [string first " " $ln] + if {$sp < 1} {continue} + set h [string range $ln 0 $sp-1] + set tag [string range $ln $sp+1 end] + dict set sums [string range $tag 1 end] [string tolower $h] + } + dict set libfetch_sums $target $sums + } + set sums [dict get $libfetch_sums $target] + if {![dict exists $sums $artifact]} { + puts stderr "$A(BAD)LIBFETCH: artifact.$aname: '$artifact' is not listed in the server's lib/$target/sha1sums.txt$A(RST)" + set libfetch_failed 1 + continue + } + set want_sha1 [dict get $sums $artifact] + set ziplocal $tierdir/$artifact + set have_ok 0 + if {[file isfile $ziplocal] && !$::punkboot::opt_force} { + if {[string tolower [sha1::sha1 -hex -file $ziplocal]] eq $want_sha1} { + set have_ok 1 + } + } + if {!$have_ok} { + set zipurl "$libfetch_origin/lib/$target/$artifact" + set partfile $ziplocal.part + if {[catch {::punkboot::libfetch_transfer $zipurl $partfile} terr]} { + puts stderr "$A(BAD)LIBFETCH: artifact.$aname: fetch failed - $terr$A(RST)" + set libfetch_failed 1 + continue + } + set got_sha1 [string tolower [sha1::sha1 -hex -file $partfile]] + if {$got_sha1 ne $want_sha1} { + file delete -force $partfile + puts stderr "$A(BAD)LIBFETCH: artifact.$aname: sha1 MISMATCH for $artifact (got $got_sha1, server sha1sums says $want_sha1) - deleted$A(RST)" + set libfetch_failed 1 + continue + } + file rename -force $partfile $ziplocal + lappend libfetch_actions "fetched lib/$target/$artifact" + } else { + lappend libfetch_actions "verified-present lib/$target/$artifact" + } + #beside-toml sidecar (also sha1sums-listed by emission) + set sidecar "[string range $artifact 0 end-4].toml" + set sidecarlocal $tierdir/$sidecar + set sidecar_ok 0 + if {[dict exists $sums $sidecar]} { + set want_side [dict get $sums $sidecar] + if {[file isfile $sidecarlocal] && !$::punkboot::opt_force} { + if {[string tolower [sha1::sha1 -hex -file $sidecarlocal]] eq $want_side} { + set sidecar_ok 1 + } + } + if {!$sidecar_ok} { + if {[catch {::punkboot::libfetch_transfer "$libfetch_origin/lib/$target/$sidecar" $sidecarlocal.part} terr]} { + puts stderr "$A(BAD)LIBFETCH: artifact.$aname: sidecar fetch failed - $terr$A(RST)" + set libfetch_failed 1 + continue + } + if {[string tolower [sha1::sha1 -hex -file $sidecarlocal.part]] ne $want_side} { + file delete -force $sidecarlocal.part + puts stderr "$A(BAD)LIBFETCH: artifact.$aname: sidecar sha1 MISMATCH for $sidecar - deleted$A(RST)" + set libfetch_failed 1 + continue + } + file rename -force $sidecarlocal.part $sidecarlocal + } + } else { + puts stderr "LIBFETCH: note - $sidecar not listed in server sha1sums (artifact accepted; sidecar skipped)" + } + #materialize: installed-shape tree under /tcl/ + #(generation subdir because both generations install identically-named + #folders). Fresh tree per materialization; -mtime restores member + #mtimes so the tree mirrors what was zipped. + set gendir $tierdir/$generation + file mkdir $gendir + set treedir $gendir/$pkgfolder + if {[file isdirectory $treedir] && !$::punkboot::opt_force} { + #re-materialize only when the tree's embedded record no longer + #matches this artifact name (self-describing tier - G-138) + set embrec $treedir/punkbin-artifact.toml + set tree_current 0 + if {[file isfile $embrec]} { + set efd [open $embrec r] + set edata [read $efd] + close $efd + if {[regexp -line "^name = \"$artifact\"$" $edata]} { + set tree_current 1 + } + } + if {$tree_current} { + lappend libfetch_actions "tree-current $target/$generation/$pkgfolder" + continue + } + } + file delete -force $treedir + #defaults: whole archive, -overwrite 1 -mtime 1 (member mtimes restored) + #-verify 1 (per-member crc32) - and accelerator-eligible (punkzip exe) + if {[catch {punk::zip::unzip $ziplocal $gendir} unzip_err]} { + puts stderr "$A(BAD)LIBFETCH: artifact.$aname: materialization failed - $unzip_err$A(RST)" + file delete -force $treedir + set libfetch_failed 1 + continue + } + if {![file isfile $treedir/pkgIndex.tcl] || ![file isfile $treedir/punkbin-artifact.toml]} { + puts stderr "$A(BAD)LIBFETCH: artifact.$aname: materialized tree at $treedir lacks pkgIndex.tcl and/or punkbin-artifact.toml$A(RST)" + set libfetch_failed 1 + continue + } + lappend libfetch_actions "materialized $target/$generation/$pkgfolder" + } + foreach act $libfetch_actions { + puts stdout "LIBFETCH: $act" + } + if {$libfetch_failed} { + puts stderr "$A(BAD)LIBFETCH: FAILED - one or more declared artifacts could not be fetched/verified/materialized (see above)$A(RST)" + exit 3 + } + puts stdout "LIBFETCH: done ([llength $libfetch_actions] action(s)) -> $projectroot/bin/packages" + } + flush stdout + ::exit 0 +} + if {$::punkboot::command ni {bakehouse packages modules libs bake vfslibs bin}} { puts stderr "Command $::punkboot::command not implemented - aborting." flush stderr @@ -5341,6 +5718,7 @@ if {$::punkboot::command in {bakehouse bake vfslibs}} { } } + if {$::punkboot::command in {bakehouse packages libs}} { ######################################################## lappend projectlibfolders lib diff --git a/src/runtime/AGENTS.md b/src/runtime/AGENTS.md index 0a4bf68a..815cadce 100644 --- a/src/runtime/AGENTS.md +++ b/src/runtime/AGENTS.md @@ -17,6 +17,7 @@ Houses the `mapvfs.config` that maps VFS payloads to platform runtimes, plus the - **Target platform (4th element, G-122).** A kit config entry may name the canonical punkshell platform its runtime is for (`help platforms` lists the names). It decides which `bin/runtime//` tier the runtime is read from (macosx-* collapsing to the universal `macosx` folder), whether the runtime file and the built kit carry `.exe`, and whether the pre-deploy process sweep applies. It is a property of the RUNTIME: entries on one runtime line may repeat it but must not disagree (a conflict, or a name that is not an `-` platform-dir name, is a fatal config error). Omitted means the build host's default target - which for an msys2/cygwin-runtime tclsh is `win32-x86_64`, not that host's own canon. The live example is the `tclkit-902-Linux64-intel-dyn` entry (target `linux-x86_64`, suffixless `punkshell902` artifact). The schema's eventual home is the G-024 toml conversion; consumers read the parsed model (`::punkboot::lib::mapvfs_*`), never the file format. - Runtime executables are placed in `bin/runtime//` by the `bin/punk-runtime.cmd` helper, manually, or (G-103 family runtimes) copied from the suite_tcl90 `kit-family` build products under `src/buildsuites/_build/suite_tcl90/out/family/`. - `punkshell.ico` here is the project-DEFAULT kit icon: the bake icon step (G-057) records it in every kit's `.resources.toml` sidecar and embeds it into win32-target kits (per-kit override + format + skip semantics: bin/AGENTS.md). It is a derived copy of `src/assets/logo/punk-mark.ico` - its `.assetorigin.toml` sidecar records the derivation; regenerate via the logo pipeline, never edit the `.ico` here. +- `libpackages.toml` (G-139) declares the punkbin LIB-TIER library artifacts this checkout consumes: `[artifact.]` entries with `name` (immutable `--r.zip` artifact filename) and `target` (`allplatforms` or a platform-dir name). Processed by `tclsh src/make.tcl libfetch` into the untracked `bin/packages//` input tier (sha1-verified against the server's per-target sha1sums.txt; trees materialized under `bin/packages//tcl/`). Consuming a new revision = editing the declared name here (deliberate, reviewable - mirroring the emission side's deliberate-publish discipline). The file's own header comment is the user-facing spec; keep the two in step. - The `_build/` subdirectory holds build intermediates; it can be safely deleted. ## Work Guidance diff --git a/src/runtime/libpackages.toml b/src/runtime/libpackages.toml new file mode 100644 index 00000000..5dc5c3aa --- /dev/null +++ b/src/runtime/libpackages.toml @@ -0,0 +1,47 @@ +# punkbin lib-tier library artifacts this checkout consumes (goal G-139) +# +# Processed by 'tclsh src/make.tcl libfetch': each declared artifact is fetched +# from /lib// into the untracked input tier +# bin/packages// (zip + .toml sidecar, sha1-verified against the +# server's per-target sha1sums.txt) and its installed-shape package tree is +# materialized under bin/packages//tcl/ - self-describing via the +# embedded punkbin-artifact.toml record (G-138). The origin defaults to the +# canonical punkbin raw base; PUNKBIN_URL / -serverurl override it (any +# non-canonical origin requires the explicit -trust-server flag). +# +# Artifact names are punkbin-immutable: --r.zip. +# The generation tag is identity (both generations install identically-named +# folders); consuming a new revision means editing the declared name here - +# a deliberate, reviewable act, mirroring the deliberate-publish discipline on +# the emission side (zig-built dlls are not bit-reproducible). +# +# Keys per [artifact.] entry: +# name - artifact filename in the punkbin lib tier +# target - lib-tier subfolder: 'allplatforms' (platform-neutral, pure-tcl) +# or a canonical platform-dir name (binary packages) +# +# NOTE: the r1 artifacts below are emitted by the suite_tcl90/suite_tcl86 +# 'library-artifacts' steps (G-138, achieved 2026-07-30) but PUBLICATION to the +# canonical punkbin origin is a deliberate user step - until it happens, +# libfetch against the canonical origin reports them missing from the server +# sha1sums. A local mirror (file:// origin + -trust-server) or a pre-populated +# bin/packages tier serves in the interim. +# +# Downstream consumption (lib_tcl deployment, kit vfs declarations) is the +# remainder of G-139 - see goals/G-139-punkbin-library-consumption.md. + +[artifact.tcllib-tcl9] +name = "tcllib2.0-tcl9-r1.zip" +target = "allplatforms" + +[artifact.tcllibc-tcl9] +name = "tcllibc-tcl9-r1.zip" +target = "win32-x86_64" + +[artifact.tcllib-tcl8] +name = "tcllib2.0-tcl8-r1.zip" +target = "allplatforms" + +[artifact.tcllibc-tcl8] +name = "tcllibc-tcl8-r1.zip" +target = "win32-x86_64"