* fix(sloppak): the full mix is a stem — drop the invented `original_audio` key (#933)
Core read, served, and depended on `original_audio:` — a top-level manifest key
this repo invented in #583 that the feedpak spec never defined. The format
already had a home for the pre-separation mixdown: it is a stem. feedpak 1.15.0
(feedpak-spec#53) RESERVES the id `full` for it, so read it from there.
The key existed to work around a bug in our own reader. The packer's comment
said so plainly: "we must NOT list the full mix as a playable stem — the player
sums every entry in `stems` and does not gate playback on `default`, so a listed
full mix plays on top of the stems". Faced with a reader that would double the
song, the packer put the mixdown outside `stems` and invented a key to point at
it. The fix belongs in the reader, and that is what this is.
load_song() now partitions the stem list: `full` comes out as
LoadedSloppak.full_mix, the instruments stay in .stems. Nothing that sums stems
or draws one fader per stem can see the mixdown, so retaining it is safe — which
is what lets the packer put it where the format says it goes.
- ws_highway: `song_info` gains full_mix_url / has_full_mix. The old
original_audio_url / has_original_audio remain as deprecated aliases for one
release so an older stems plugin keeps working (#945).
- `stems` on the wire, and stem_ids / stem_count in the library index, are now
INSTRUMENT stems only — a separated pack that retains its mixdown no longer
advertises a bogus "full" chip or an inflated stem count.
- enrichment: fingerprint against the mixdown wherever it lives. This widens
coverage — _song_audio_file() previously returned None for any pack without
the invented key, so fingerprinting silently did nothing for nearly every pack.
- sloppak: `original_audio:` is still READ as a deprecated fallback, because
every pack in the wild carries it and would otherwise lose its pristine mix.
tools/migrate_full_mix_stem.py rewrites those packs into the spec shape
(original/full.ogg -> stems/full.ogg, add the `full` stem at default:off, drop
the key); the fallback and the aliases die with #945.
The spec gate keeps the debt honest: the grandfather entry now tracks #945, and
the gate fails if it goes stale.
Verified: spec gate OK (4/4, incl. ingesting the spec's new example pack that
retains `full`); 2493 python tests, 995 js tests; migrator round-tripped over
real packs from the library and the results pass the spec's reference validator.
* fix(migrate): discover directory-form packs instead of silently skipping them
iter_packs() searched only files, so a directory-form pack (`song.sloppak/`, the
authoring shape) was walked INTO and never yielded — silently missed by a run
that's meant to be exhaustive. Discover suffix-named directories too (yielded
whole, not descended into), and route packs through migrate_pack/verify_pack.
Directory packs are REPORTED as `dir-form-unsupported`, not rewritten in place:
a single-file pack is replaced atomically (a fully-built temp archive swapped in
with one os.replace), but a populated directory can't be swapped that way, so an
interrupted in-place rewrite could leave an authoring pack half-migrated. The
status is a problem status, so it counts against the run's exit code and shows
in the summary — the operator re-packs or migrates it as a `.feedpak` instead of
it vanishing from the report. Addresses a CodeRabbit review finding.
Signed-off-by: Kris Anderson <topkoa@gmail.com>
* fix(migrate): verify requires an explicit `off` on a retained full mix
verify_zip accepted any non-truthy `default` on a multi-stem `full` (missing,
empty, boolean, `false`/`no`/`0`, malformed) as "ok". But core defaults an ABSENT
`default` to True — ON (lib/sloppak.py: `s.get("default", True)`) — and treats an
empty/unrecognized string as ON too, so a migrated-shape pack whose `full` stem
has a missing or blank default beside instrument stems would actually play the
mixdown on open and double the song. verify was certifying that as safe.
Require an explicit normalized `off` beside instrument stems: `on`-ish values are
reported `full-stem-default-on` (actively plays), everything that is not a
normalized `off` is reported `full-stem-default-not-off`. The migrator already
writes the literal `off`, so its own output is unaffected; this also certifies
the pack is in the tool's canonical, most-portable shape. The len>1 gate is kept,
so a sole `full` stem (which IS the audio) is not policed.
Adds parametrized coverage for missing / empty / boolean / off-ish / malformed
defaults, and a sole-full-stem case. Addresses a CodeRabbit review finding.
Signed-off-by: Kris Anderson <topkoa@gmail.com>
---------
Signed-off-by: Kris Anderson <topkoa@gmail.com>
Co-authored-by: Kris Anderson <topkoa@gmail.com>
Two review nits: check_allowlist_closed() raised a traceback when
--baseline-exceptions pointed at a missing file (the error now says CI
derives it from the base branch and local runs should omit the flag), and
check_key_coverage() would traceback on a reader with a SyntaxError (now a
::error:: naming the module — belt-and-braces, since such a module can't
pass pytest either, but this job may run first).
Signed-off-by: topkoa <topkoa@gmail.com>
Design change, at the maintainer's direction: the gate now checks out
feedpak-spec at HEAD instead of a pinned SHA. .feedpak-spec-ref, its
40-char validation step, and the pin-bump machinery are gone.
Rationale: it is vital that the app conforms to the spec — the current
spec, not a snapshot. The pin bought determinism at the cost of a
maintenance loop (bump PRs, a PAT, weekly latency) and a window where the
gate verified against a stale spec. Tracking HEAD makes the dev flow fully
self-serve with zero upkeep: gated PR -> FEP -> spec merge -> re-run
checks -> green. Nothing to bump.
The trade-off is accepted with eyes open, and the docs state it: the
normal FEP is additive and can only loosen the gate, so it cannot redden
anyone's PR. Only a breaking spec change (rare, deliberate, MAJOR per the
spec's compatibility policy) turns PRs red repo-wide — which is the
correct org-wide signal that the app is out of conformance. The CI job
logs the spec SHA each run verified against, so any red run is
reproducible.
Failure messages now also say why it matters beyond the one PR (also at
the maintainer's direction): non-conformance that lands shows up as red CI
on every teammate's PR until it is resolved, and only its author can clear
it — the FEP route keeps everyone else unblocked. Tone softened throughout
(the exceptions-file header now explains rather than shouts).
Signed-off-by: topkoa <topkoa@gmail.com>
_parse_exceptions() now validates the document shape — top level must be a
mapping, 'exceptions' must be a list, each entry a mapping, and YAML parse
errors are caught — each failing with a ::error:: instead of an
AttributeError traceback. CI output must say what to fix. Parametrised
tests cover all four malformed shapes.
docs/feedpak-spec-gate.md: the Limitations section still described the
pre-flow-aware scanner (KEY_OPS, name-list-only receivers). Now states the
actual residual gaps: function-parameter manifests are recognised by name
only, and helper-mediated literal keys (song.py's
_gap_fill_manifest_absent(manifest, "album")) are unseen by both the scan
and the readers-complete guard, since they share one detector.
Signed-off-by: topkoa <topkoa@gmail.com>
Review found lib/routers/chart.py binding `m = load_manifest(p) or {}` and
reading eight manifest keys through it. `m` was not in MANIFEST_VARS and
`m.get` did not match the readers-complete regex, so the module was
invisible to BOTH halves of the gate — unlisted and unscanned. Same for
lib/routers/song.py (binds `manifest` from load_manifest for enrichment
gap-fill). Both are now in READERS.
The structural fix, not the name-list patch: keys_touched() now discovers
receivers flow-aware — any local assigned from load_manifest(...) is a
manifest dict, whatever it is called. MANIFEST_VARS remains only as the
fallback for manifests that arrive as function parameters (ws_highway).
A plain `m = {}` is not a receiver; test pins that.
readers-complete now reuses keys_touched() itself instead of a parallel
KEY_OPS regex — the two detectors diverged once already (that is exactly
how chart.py slipped through), so now there is one detector and one truth.
check_reverse() gets a 300s subprocess timeout: the validator executes at a
pinned SHA, but a pathological pack or validator bug should fail the job,
not hang the runner to the Actions-level timeout.
Tests: flow-aware receiver under an arbitrary name (read + write), and the
negative — a plain dict named `m` stays out of the scan. 17 pass.
Signed-off-by: topkoa <topkoa@gmail.com>
Self-review finding: the gate is what keeps the app from drifting off the
feedpak spec, but the gate itself had zero pytest coverage — a refactor
could quietly weaken keys_touched or the allowlist logic and nothing would
notice. The protector needs protecting.
tests/test_spec_gate.py pins the load-bearing behaviours:
- read/write classification: get() reads; subscript Store and setdefault()
write (the two forms that were blind spots in review); the
load_manifest-wrapped get; unrelated dicts and non-literal keys ignored.
- exceptions file: duplicate keys and issue-less entries rejected.
- the closed allowlist: growth fails, shrink and steady state pass,
bootstrap skips.
- live-tree checks, same as CI: READERS matches the codebase, and the only
non-spec key core touches is the grandfathered original_audio.
Also fixes stale "Layer 2/3" docstrings on check_forward/check_reverse
(they are layers 3/4 since allowlist-closed landed) — the same
docs-lag-the-code class this PR's review kept catching; now the numbering
is asserted by the printed [n/4] headers next to them.
Signed-off-by: topkoa <topkoa@gmail.com>
The allowlist-closed diff hardcoded `origin main`, but ship-ci.yml also runs
this workflow for PRs into release/** and for pushes to release/**, where a
main baseline diffs against the wrong branch and can fail changes that have
nothing to do with the allowlist. It now resolves the base:
PR -> github.event.pull_request.base.ref (the branch it merges into)
push -> github.ref_name (the branch itself; its tip already contains the
change, so the diff is a no-op — enforcement happens at PR time)
Also from review, all documentation drift introduced by my own earlier
commits:
- The layer count said "three" in the module docstring, the workflow comment,
the docs, and the changelog. There are four (allowlist-closed was added).
- The changelog listed three scanned modules; there are five.
- The docs and changelog stated the rule for keys core *reads*, omitting
writes — which are equally gated, and land in every pack we emit.
- The CI summary line labelled grandfathered keys "pending spec", implying
adoption is the only resolution. For original_audio it is not: the fix is
removal. Relabelled "grandfathered (tracked debt)".
- feedpak-spec-exceptions.yml said an entry clears when core "stops reading"
the key; the rule is "no longer reads or writes".
- Replaced a bitwise `&` over two bools with two named results and an
explicit `and` — both checks must run (a stale READERS list and an
undeclared key are separate failures; short-circuiting would hide one), and
`&` reads like a typo.
Signed-off-by: topkoa <topkoa@gmail.com>
Review found the gate was scanning less than it claimed.
READERS missed two modules that genuinely touch feedpak manifests:
lib/routers/ws_highway.py (reads `authors`) and lib/gp2notation.py (loads
manifest.yaml, stamps feedpak_version, writes the file back). Keys touched
there were going entirely unchecked.
The scan also only recognised writes done via subscript, so
`manifest.setdefault("k", v)` — exactly how gp2notation.py stamps
feedpak_version — was invisible. setdefault with a literal key now counts as
a write.
The deeper problem is that READERS is hand-maintained, and a hand-maintained
list rots; that is how both modules went unnoticed. check_readers_complete()
now re-derives the set: any module under lib/ (or server.py) that both
touches manifest keys and shows a feedpak signal must be listed, or the build
fails. It is a guard on the gate itself.
The list stays explicit rather than becoming a glob, because `manifest` is
overloaded here: lib/loosefolder.py (the loose-folder manifest.json) and
lib/diagnostics_bundle.py (the diagnostics bundle manifest) have their own
unrelated manifests, and scanning those would flag *their* keys as feedpak
drift. Both score zero on the feedpak signals, which is what keeps them out.
Now scanning 5 modules: 20 reads, 2 writes, all spec-declared.
Signed-off-by: topkoa <topkoa@gmail.com>
The gate's purpose is to make a non-conforming change *not merge*, so the
person merging must stop and decide whether to take it through the format
process. The escape hatches defeated exactly that: a developer who did not
want to write a FEP could name their key `x-whatever`, or append an entry to
feedpak-spec-exceptions.yml with any issue link, and merge. Both were
self-serve and in-repo. That is a speed bump with a signed excuse note, not a
gate.
The relief valve is the FEP process itself, not something in this repo. The
spec's governance already says so: "A change is not part of the format until
it lands here."
Removed the `x-` prefix bypass. It was invented here, not in the spec — the
spec reserves no experimental namespace. Its "unknown keys are reserved for
forward-compatibility" rule is about *tolerating* other implementations'
keys, not a licence for core to mint its own.
feedpak-spec-exceptions.yml is now a CLOSED grandfather list. A new check
(allowlist-closed) diffs it against the base branch and fails any PR that
ADDS an entry; removal stays allowed, so the list can only shrink. Deleting
an entry does not by itself pass the gate — key-coverage still fails while
core reads the key, so the entry goes when the code goes.
Every failure message now points at the FEP process and at bumping
.feedpak-spec-ref to the merged spec SHA, which is the one supported way a
new manifest key reaches core.
CI fetches the base branch to diff the allowlist; the bootstrap flag covers
the one case with no baseline — the PR introducing the gate.
Signed-off-by: topkoa <topkoa@gmail.com>
Six more findings from CodeRabbit and Copilot on #934. All valid; four were
my own docs lagging the write-checking change in d0626f5.
check_forward() now discovers example packs recursively, so a pack nested
under examples/<group>/ can't slip past the "every example pack" contract.
Taken WITHOUT the suggested is_file() filter, which would have broken it: a
feedpak is dual-form — a zip (foo.feedpak) or a directory (foo.feedpak/) —
and the spec's own examples ship as directories, so is_file() would have
matched zero packs. Suffix matching covers both forms.
load_exceptions() rejects duplicate keys instead of silently keeping the
last one, which would quietly retarget the tracking issue for a piece of
debt this file exists to track.
The sloppak import is wrapped so a missing dependency produces a CI-legible
::error:: rather than a bare traceback.
Docs caught up with the code: the exceptions file header, its stale-entry
rule, and the changelog all said "reads" when the gate checks reads AND
writes.
Signed-off-by: topkoa <topkoa@gmail.com>
Review feedback from CodeRabbit and Copilot on #934. All six findings were
valid; one is fixed the other way round from how it was suggested.
Key-coverage now checks manifest WRITES as well as reads. Copilot correctly
spotted that `ast.walk` ignored subscript context, so
`manifest["year"] = ...` (lib/songmeta.py) scored as a read — but the fix is
not to drop writes. A key core *writes* is spec surface pointed outward: it
lands in every pack we emit, so an undeclared one seeds the ecosystem with
non-spec data. Subscripts are now classified by ctx (Store = write, Load =
read) and both sets are checked, with distinct error messages. Today: 19
reads, 2 writes, all declared.
Workflow:
- persist-credentials: false on the repo checkout — the job runs repository
code and never pushes (CodeRabbit / zizmor artipacked).
- .feedpak-spec-ref must be a full 40-char SHA. actions/checkout resolves
branches and tags in `ref` too, so a non-SHA there would silently un-pin
the spec — precisely what the file exists to prevent.
- Pin jsonschema==4.26.0, for the same reason the spec SHA is pinned: an
upstream release must not redden this job on a PR that changed neither
this repo nor the spec.
Script:
- check_forward() guards a missing examples/ dir instead of raising an
unhandled FileNotFoundError.
- TemporaryDirectory() instead of mkdtemp(), so a local run doesn't leak.
Signed-off-by: topkoa <topkoa@gmail.com>
feedpak is published as an open format with its own repo, normative spec,
JSON Schemas, and reference validator. That makes the spec a contract with
everyone outside this repo: third-party packers, converters, and players
build against it, and it is meant to be the complete description of a pack.
Nothing enforced that. #583 added a manifest key (`original_audio`) that
core, lib/enrichment.py, and the stems plugin all now depend on, but which
was never added to the spec — so a spec-compliant pack stopped being a
fully-working pack, the reference validator could not warn authors about a
key it had never heard of, and third-party tooling began emitting an
`original/` directory reverse-engineered from an example in a code comment.
See #933.
We cannot mechanically prove core interprets a key the way the spec means.
We can prove three surface properties, and they cover the drift that
actually happens:
1. key-coverage — every manifest key core reads is declared in the spec's
manifest.schema.json (AST scan of lib/sloppak.py, lib/enrichment.py,
lib/songmeta.py).
2. forward — core's load_song() ingests every example pack the spec ships.
3. reverse — every pack committed here passes the spec's own
tools/validate.py (7/7 pass today).
The spec is pinned by SHA in .feedpak-spec-ref rather than tracked from its
default branch, so a change over there cannot redden an unrelated PR here;
bump it in its own PR, where a red result is precisely the signal that core
does not satisfy the new spec.
A gate with no legitimate way to say "yes, deliberately, not yet" gets
switched off the first time it blocks a release, so there are two escape
hatches: the reserved `x-` key prefix (always permitted, and it tells every
third-party packer the key is not stable surface), and
feedpak-spec-exceptions.yml, which requires a tracking issue per entry. An
exception that goes stale — the spec caught up, or core stopped reading the
key — fails the build, so the allowlist cannot become somewhere drift
quietly accumulates. `original_audio` is seeded there against #933 so the
gate lands green and starts blocking the next instance immediately, rather
than requiring #933 to be resolved first.
Dev/CI tooling only; never on the serve or Docker path (constitution
Principle I). jsonschema is installed in the CI job, not added to
requirements.txt.
Signed-off-by: topkoa <topkoa@gmail.com>
