15 Commits

Include renames

Author	SHA1	Message	Date
4 people	8297afc449	feat(tools): per-platform VST3 slicing for rig content packs (#1025) ... Rebased onto merged main (was stacked on #1023/#1024, whose venue work is now in main) so it no longer carries a stale content_packs.py that would revert 1023's build_pack fixes. - build_vst_pack: slice a fat .vst3 tree to one platform (keep its binary dir + shared bundle files, drop the two foreign platform dirs and src/ build trees). Pins create_system=3 like build_pack — without it the same tree hashes differently on a Windows runner (native .vst3 are built there), breaking the precomputable-hash guarantee exactly where it matters. - Publish wiring: 'python tools/content_packs.py <vst-root> --vst --version N --publish' builds+uploads vst-<plat>-vN releases for mac/win/linux and emits a platform-keyed {url,sha256,bytes} manifest — the shape rig_builder's data/vst_packs.json consumes. publish() refactored onto a shared _publish_release helper (venue behaviour unchanged). - Tests: slice keeps target+shared/drops foreign, per-platform binary, reproducibility, unknown-platform reject, and a simulated-win32 guard that fails if the create_system pin is dropped. selfcheck covers the VST path. Original build_vst_pack by Matthew Harris Glover; reworked for the create_system fix, publish wiring, and rebase. Signed-off-by: byrongamatos <xasiklas@gmail.com> Co-authored-by: byrongamatos <xasiklas@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 01:01:51 +02:00
4 people	59bcf338a3	feat(career): host higher venues as opt-in content packs (#1023) ... * feat(career): host higher venues as opt-in content packs Move the club and arena venue packs (~678 MB of crowd MP4s) out of the bundle and download them on demand, keeping the bar starter bundled so career still works offline. Leans on career's existing pack pipeline (_download_pack: stream -> sha256 -> extract -> validate -> swap), which already degrades gracefully when a pack is absent. - venues.json: club/arena gain `pack` URLs pointing at per-pack, versioned, immutable releases (venue-<id>-v<N>, matching the existing venue-arena-v1). Arena's sha256/bytes are the real published asset (verified end-to-end); club is a placeholder until its release is published. - tools/content_packs.py: reusable, reproducible pack build/publish/manifest tool. Byte-identical output for identical media (fixed order/mtime/perms, STORED) so a pack's hash can be known before upload. --local (file://) for offline tests, --publish for the per-pack release. Has a --selfcheck. - .github/workflows/content-packs.yml: workflow_dispatch automation that builds/publishes packs and opens the venues.json manifest-bump PR, so publishing is never a manual checklist. - test: round-trips a tool-built pack through career's real _download_pack. Part of the nightly-slimming effort (feedBack-desktop#122). The desktop bundle change (stop shipping club/arena) is a companion PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(career): don't offer a venue pack until its release is published A committed venues.json entry carries a 0-byte placeholder (and all-zero sha) until its release exists. Previously has_pack was true as soon as a `pack` object was present, so the UI showed a "Download" button that could only fail (the placeholder URL 404s). Gate on a real, publish-stamped size via _pack_published(): the card shows "coming soon" and the download endpoint 404s until the pack is actually published. Caught by a real bundle+runtime smoke. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(content-packs): address CodeRabbit review on #1023 - workflow: stop interpolating dispatch inputs into Bash (template injection flagged by zizmor). Pass venues/version via env, validate formats, use an argument array. - content_packs: reject top-level files the career downloader would refuse (PACK_FILENAME_RE) before publishing — a stray .DS_Store would otherwise ship and fail _validate_pack_dir for every client. + test. - content_packs: pin ZipInfo.create_system=3 so packs hash identically across Windows/Unix runners (was the documented reproducibility caveat). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(changelog): note opt-in career venue packs (#122) Signed-off-by: Matthew Harris Glover <matthew@harrisglover.com> * docs(content_packs): correct --publish usage in module docstring --publish is a flag (no tag arg) and publish() deliberately omits --clobber; the docstring said otherwise. Signed-off-by: byrongamatos <xasiklas@gmail.com> --------- Signed-off-by: Matthew Harris Glover <matthew@harrisglover.com> Signed-off-by: byrongamatos <xasiklas@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: byrongamatos <xasiklas@gmail.com>	2026-07-23 00:09:46 +02:00
3 people	329cc86315	fix(sloppak): the full mix is a stem — drop the invented original_audio key (#946) ... * fix(sloppak): the full mix is a stem — drop the invented `original_audio` key (#933) Core read, served, and depended on `original_audio:` — a top-level manifest key this repo invented in #583 that the feedpak spec never defined. The format already had a home for the pre-separation mixdown: it is a stem. feedpak 1.15.0 (feedpak-spec#53) RESERVES the id `full` for it, so read it from there. The key existed to work around a bug in our own reader. The packer's comment said so plainly: "we must NOT list the full mix as a playable stem — the player sums every entry in `stems` and does not gate playback on `default`, so a listed full mix plays on top of the stems". Faced with a reader that would double the song, the packer put the mixdown outside `stems` and invented a key to point at it. The fix belongs in the reader, and that is what this is. load_song() now partitions the stem list: `full` comes out as LoadedSloppak.full_mix, the instruments stay in .stems. Nothing that sums stems or draws one fader per stem can see the mixdown, so retaining it is safe — which is what lets the packer put it where the format says it goes. - ws_highway: `song_info` gains full_mix_url / has_full_mix. The old original_audio_url / has_original_audio remain as deprecated aliases for one release so an older stems plugin keeps working (#945). - `stems` on the wire, and stem_ids / stem_count in the library index, are now INSTRUMENT stems only — a separated pack that retains its mixdown no longer advertises a bogus "full" chip or an inflated stem count. - enrichment: fingerprint against the mixdown wherever it lives. This widens coverage — _song_audio_file() previously returned None for any pack without the invented key, so fingerprinting silently did nothing for nearly every pack. - sloppak: `original_audio:` is still READ as a deprecated fallback, because every pack in the wild carries it and would otherwise lose its pristine mix. tools/migrate_full_mix_stem.py rewrites those packs into the spec shape (original/full.ogg -> stems/full.ogg, add the `full` stem at default:off, drop the key); the fallback and the aliases die with #945. The spec gate keeps the debt honest: the grandfather entry now tracks #945, and the gate fails if it goes stale. Verified: spec gate OK (4/4, incl. ingesting the spec's new example pack that retains `full`); 2493 python tests, 995 js tests; migrator round-tripped over real packs from the library and the results pass the spec's reference validator. * fix(migrate): discover directory-form packs instead of silently skipping them iter_packs() searched only files, so a directory-form pack (`song.sloppak/`, the authoring shape) was walked INTO and never yielded — silently missed by a run that's meant to be exhaustive. Discover suffix-named directories too (yielded whole, not descended into), and route packs through migrate_pack/verify_pack. Directory packs are REPORTED as `dir-form-unsupported`, not rewritten in place: a single-file pack is replaced atomically (a fully-built temp archive swapped in with one os.replace), but a populated directory can't be swapped that way, so an interrupted in-place rewrite could leave an authoring pack half-migrated. The status is a problem status, so it counts against the run's exit code and shows in the summary — the operator re-packs or migrates it as a `.feedpak` instead of it vanishing from the report. Addresses a CodeRabbit review finding. Signed-off-by: Kris Anderson <topkoa@gmail.com> * fix(migrate): verify requires an explicit `off` on a retained full mix verify_zip accepted any non-truthy `default` on a multi-stem `full` (missing, empty, boolean, `false`/`no`/`0`, malformed) as "ok". But core defaults an ABSENT `default` to True — ON (lib/sloppak.py: `s.get("default", True)`) — and treats an empty/unrecognized string as ON too, so a migrated-shape pack whose `full` stem has a missing or blank default beside instrument stems would actually play the mixdown on open and double the song. verify was certifying that as safe. Require an explicit normalized `off` beside instrument stems: `on`-ish values are reported `full-stem-default-on` (actively plays), everything that is not a normalized `off` is reported `full-stem-default-not-off`. The migrator already writes the literal `off`, so its own output is unaffected; this also certifies the pack is in the tool's canonical, most-portable shape. The len>1 gate is kept, so a sole `full` stem (which IS the audio) is not policed. Adds parametrized coverage for missing / empty / boolean / off-ish / malformed defaults, and a sole-full-stem case. Addresses a CodeRabbit review finding. Signed-off-by: Kris Anderson <topkoa@gmail.com> --------- Signed-off-by: Kris Anderson <topkoa@gmail.com> Co-authored-by: Kris Anderson <topkoa@gmail.com>	2026-07-13 12:22:42 -04:00
byrongamatos	ac5c5ad20d	ci: cover gap-fill manifest key scans ... Signed-off-by: byrongamatos <xasiklas@gmail.com>	2026-07-13 13:25:44 +02:00
topkoa	a60dcd10c2	ci: legible errors for a missing baseline and an unparseable reader ... Two review nits: check_allowlist_closed() raised a traceback when --baseline-exceptions pointed at a missing file (the error now says CI derives it from the base branch and local runs should omit the flag), and check_key_coverage() would traceback on a reader with a SyntaxError (now a ::error:: naming the module — belt-and-braces, since such a module can't pass pytest either, but this job may run first). Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-13 01:23:59 -04:00
topkoa	c485f02211	ci: track the spec's HEAD — the app conforms to the living spec ... Design change, at the maintainer's direction: the gate now checks out feedpak-spec at HEAD instead of a pinned SHA. .feedpak-spec-ref, its 40-char validation step, and the pin-bump machinery are gone. Rationale: it is vital that the app conforms to the spec — the current spec, not a snapshot. The pin bought determinism at the cost of a maintenance loop (bump PRs, a PAT, weekly latency) and a window where the gate verified against a stale spec. Tracking HEAD makes the dev flow fully self-serve with zero upkeep: gated PR -> FEP -> spec merge -> re-run checks -> green. Nothing to bump. The trade-off is accepted with eyes open, and the docs state it: the normal FEP is additive and can only loosen the gate, so it cannot redden anyone's PR. Only a breaking spec change (rare, deliberate, MAJOR per the spec's compatibility policy) turns PRs red repo-wide — which is the correct org-wide signal that the app is out of conformance. The CI job logs the spec SHA each run verified against, so any red run is reproducible. Failure messages now also say why it matters beyond the one PR (also at the maintainer's direction): non-conformance that lands shows up as red CI on every teammate's PR until it is resolved, and only its author can clear it — the FEP route keeps everyone else unblocked. Tone softened throughout (the exceptions-file header now explains rather than shouts). Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-13 01:07:12 -04:00
topkoa	203f82b6fe	ci: legible failures for malformed exceptions file; docs catch up ... _parse_exceptions() now validates the document shape — top level must be a mapping, 'exceptions' must be a list, each entry a mapping, and YAML parse errors are caught — each failing with a ::error:: instead of an AttributeError traceback. CI output must say what to fix. Parametrised tests cover all four malformed shapes. docs/feedpak-spec-gate.md: the Limitations section still described the pre-flow-aware scanner (KEY_OPS, name-list-only receivers). Now states the actual residual gaps: function-parameter manifests are recognised by name only, and helper-mediated literal keys (song.py's _gap_fill_manifest_absent(manifest, "album")) are unseen by both the scan and the readers-complete guard, since they share one detector. Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-13 00:53:09 -04:00
topkoa	0158286d06	ci: flow-aware manifest discovery — a name list missed real readers ... Review found lib/routers/chart.py binding `m = load_manifest(p) or {}` and reading eight manifest keys through it. `m` was not in MANIFEST_VARS and `m.get` did not match the readers-complete regex, so the module was invisible to BOTH halves of the gate — unlisted and unscanned. Same for lib/routers/song.py (binds `manifest` from load_manifest for enrichment gap-fill). Both are now in READERS. The structural fix, not the name-list patch: keys_touched() now discovers receivers flow-aware — any local assigned from load_manifest(...) is a manifest dict, whatever it is called. MANIFEST_VARS remains only as the fallback for manifests that arrive as function parameters (ws_highway). A plain `m = {}` is not a receiver; test pins that. readers-complete now reuses keys_touched() itself instead of a parallel KEY_OPS regex — the two detectors diverged once already (that is exactly how chart.py slipped through), so now there is one detector and one truth. check_reverse() gets a 300s subprocess timeout: the validator executes at a pinned SHA, but a pathological pack or validator bug should fail the job, not hang the runner to the Actions-level timeout. Tests: flow-aware receiver under an arbitrary name (read + write), and the negative — a plain dict named `m` stays out of the scan. 17 pass. Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-13 00:45:38 -04:00
topkoa	b54b65d35c	test: give the spec gate its own regression suite ... Self-review finding: the gate is what keeps the app from drifting off the feedpak spec, but the gate itself had zero pytest coverage — a refactor could quietly weaken keys_touched or the allowlist logic and nothing would notice. The protector needs protecting. tests/test_spec_gate.py pins the load-bearing behaviours: - read/write classification: get() reads; subscript Store and setdefault() write (the two forms that were blind spots in review); the load_manifest-wrapped get; unrelated dicts and non-literal keys ignored. - exceptions file: duplicate keys and issue-less entries rejected. - the closed allowlist: growth fails, shrink and steady state pass, bootstrap skips. - live-tree checks, same as CI: READERS matches the codebase, and the only non-spec key core touches is the grandfathered original_audio. Also fixes stale "Layer 2/3" docstrings on check_forward/check_reverse (they are layers 3/4 since allowlist-closed landed) — the same docs-lag-the-code class this PR's review kept catching; now the numbering is asserted by the printed [n/4] headers next to them. Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-13 00:39:36 -04:00
topkoa	ab2e68a638	ci: resolve the allowlist baseline against the real base branch ... The allowlist-closed diff hardcoded `origin main`, but ship-ci.yml also runs this workflow for PRs into release/** and for pushes to release/**, where a main baseline diffs against the wrong branch and can fail changes that have nothing to do with the allowlist. It now resolves the base: PR -> github.event.pull_request.base.ref (the branch it merges into) push -> github.ref_name (the branch itself; its tip already contains the change, so the diff is a no-op — enforcement happens at PR time) Also from review, all documentation drift introduced by my own earlier commits: - The layer count said "three" in the module docstring, the workflow comment, the docs, and the changelog. There are four (allowlist-closed was added). - The changelog listed three scanned modules; there are five. - The docs and changelog stated the rule for keys core *reads*, omitting writes — which are equally gated, and land in every pack we emit. - The CI summary line labelled grandfathered keys "pending spec", implying adoption is the only resolution. For original_audio it is not: the fix is removal. Relabelled "grandfathered (tracked debt)". - feedpak-spec-exceptions.yml said an entry clears when core "stops reading" the key; the rule is "no longer reads or writes". - Replaced a bitwise `&` over two bools with two named results and an explicit `and` — both checks must run (a stale READERS list and an undeclared key are separate failures; short-circuiting would hide one), and `&` reads like a typo. Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-13 00:32:24 -04:00
topkoa	d806d12c22	ci: close two blind spots in the key scan ... Review found the gate was scanning less than it claimed. READERS missed two modules that genuinely touch feedpak manifests: lib/routers/ws_highway.py (reads `authors`) and lib/gp2notation.py (loads manifest.yaml, stamps feedpak_version, writes the file back). Keys touched there were going entirely unchecked. The scan also only recognised writes done via subscript, so `manifest.setdefault("k", v)` — exactly how gp2notation.py stamps feedpak_version — was invisible. setdefault with a literal key now counts as a write. The deeper problem is that READERS is hand-maintained, and a hand-maintained list rots; that is how both modules went unnoticed. check_readers_complete() now re-derives the set: any module under lib/ (or server.py) that both touches manifest keys and shows a feedpak signal must be listed, or the build fails. It is a guard on the gate itself. The list stays explicit rather than becoming a glob, because `manifest` is overloaded here: lib/loosefolder.py (the loose-folder manifest.json) and lib/diagnostics_bundle.py (the diagnostics bundle manifest) have their own unrelated manifests, and scanning those would flag *their* keys as feedpak drift. Both score zero on the feedpak signals, which is what keeps them out. Now scanning 5 modules: 20 reads, 2 writes, all spec-declared. Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-13 00:23:36 -04:00
topkoa	32d723b774	ci: close the escape hatches — the FEP process is the only route ... The gate's purpose is to make a non-conforming change *not merge*, so the person merging must stop and decide whether to take it through the format process. The escape hatches defeated exactly that: a developer who did not want to write a FEP could name their key `x-whatever`, or append an entry to feedpak-spec-exceptions.yml with any issue link, and merge. Both were self-serve and in-repo. That is a speed bump with a signed excuse note, not a gate. The relief valve is the FEP process itself, not something in this repo. The spec's governance already says so: "A change is not part of the format until it lands here." Removed the `x-` prefix bypass. It was invented here, not in the spec — the spec reserves no experimental namespace. Its "unknown keys are reserved for forward-compatibility" rule is about *tolerating* other implementations' keys, not a licence for core to mint its own. feedpak-spec-exceptions.yml is now a CLOSED grandfather list. A new check (allowlist-closed) diffs it against the base branch and fails any PR that ADDS an entry; removal stays allowed, so the list can only shrink. Deleting an entry does not by itself pass the gate — key-coverage still fails while core reads the key, so the entry goes when the code goes. Every failure message now points at the FEP process and at bumping .feedpak-spec-ref to the merged spec SHA, which is the one supported way a new manifest key reaches core. CI fetches the base branch to diff the allowlist; the bootstrap flag covers the one case with no baseline — the PR introducing the gate. Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-13 00:14:57 -04:00
topkoa	ceb1e143cd	ci: second review pass — recurse examples, reject duplicate exceptions ... Six more findings from CodeRabbit and Copilot on #934. All valid; four were my own docs lagging the write-checking change in d0626f5. check_forward() now discovers example packs recursively, so a pack nested under examples/<group>/ can't slip past the "every example pack" contract. Taken WITHOUT the suggested is_file() filter, which would have broken it: a feedpak is dual-form — a zip (foo.feedpak) or a directory (foo.feedpak/) — and the spec's own examples ship as directories, so is_file() would have matched zero packs. Suffix matching covers both forms. load_exceptions() rejects duplicate keys instead of silently keeping the last one, which would quietly retarget the tracking issue for a piece of debt this file exists to track. The sloppak import is wrapped so a missing dependency produces a CI-legible ::error:: rather than a bare traceback. Docs caught up with the code: the exceptions file header, its stale-entry rule, and the changelog all said "reads" when the gate checks reads AND writes. Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-12 23:54:07 -04:00
topkoa	d0626f5618	ci: address review — check writes too, pin deps, harden the spec pin ... Review feedback from CodeRabbit and Copilot on #934. All six findings were valid; one is fixed the other way round from how it was suggested. Key-coverage now checks manifest WRITES as well as reads. Copilot correctly spotted that `ast.walk` ignored subscript context, so `manifest["year"] = ...` (lib/songmeta.py) scored as a read — but the fix is not to drop writes. A key core *writes* is spec surface pointed outward: it lands in every pack we emit, so an undeclared one seeds the ecosystem with non-spec data. Subscripts are now classified by ctx (Store = write, Load = read) and both sets are checked, with distinct error messages. Today: 19 reads, 2 writes, all declared. Workflow: - persist-credentials: false on the repo checkout — the job runs repository code and never pushes (CodeRabbit / zizmor artipacked). - .feedpak-spec-ref must be a full 40-char SHA. actions/checkout resolves branches and tags in `ref` too, so a non-SHA there would silently un-pin the spec — precisely what the file exists to prevent. - Pin jsonschema==4.26.0, for the same reason the spec SHA is pinned: an upstream release must not redden this job on a PR that changed neither this repo nor the spec. Script: - check_forward() guards a missing examples/ dir instead of raising an unhandled FileNotFoundError. - TemporaryDirectory() instead of mkdtemp(), so a local run doesn't leak. Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-12 23:44:15 -04:00
topkoa	22332bef22	ci: gate core against the feedpak spec ... feedpak is published as an open format with its own repo, normative spec, JSON Schemas, and reference validator. That makes the spec a contract with everyone outside this repo: third-party packers, converters, and players build against it, and it is meant to be the complete description of a pack. Nothing enforced that. #583 added a manifest key (`original_audio`) that core, lib/enrichment.py, and the stems plugin all now depend on, but which was never added to the spec — so a spec-compliant pack stopped being a fully-working pack, the reference validator could not warn authors about a key it had never heard of, and third-party tooling began emitting an `original/` directory reverse-engineered from an example in a code comment. See #933. We cannot mechanically prove core interprets a key the way the spec means. We can prove three surface properties, and they cover the drift that actually happens: 1. key-coverage — every manifest key core reads is declared in the spec's manifest.schema.json (AST scan of lib/sloppak.py, lib/enrichment.py, lib/songmeta.py). 2. forward — core's load_song() ingests every example pack the spec ships. 3. reverse — every pack committed here passes the spec's own tools/validate.py (7/7 pass today). The spec is pinned by SHA in .feedpak-spec-ref rather than tracked from its default branch, so a change over there cannot redden an unrelated PR here; bump it in its own PR, where a red result is precisely the signal that core does not satisfy the new spec. A gate with no legitimate way to say "yes, deliberately, not yet" gets switched off the first time it blocks a release, so there are two escape hatches: the reserved `x-` key prefix (always permitted, and it tells every third-party packer the key is not stable surface), and feedpak-spec-exceptions.yml, which requires a tracking issue per entry. An exception that goes stale — the spec caught up, or core stopped reading the key — fails the build, so the allowlist cannot become somewhere drift quietly accumulates. `original_audio` is seeded there against #933 so the gate lands green and starts blocking the next instance immediately, rather than requiring #933 to be resolved first. Dev/CI tooling only; never on the serve or Docker path (constitution Principle I). jsonschema is installed in the CI job, not added to requirements.txt. Signed-off-by: topkoa <topkoa@gmail.com>	2026-07-12 23:30:45 -04:00