ci: gate core against the feedpak spec

feedpak is published as an open format with its own repo, normative spec,
JSON Schemas, and reference validator. That makes the spec a contract with
everyone outside this repo: third-party packers, converters, and players
build against it, and it is meant to be the complete description of a pack.

Nothing enforced that. #583 added a manifest key (`original_audio`) that
core, lib/enrichment.py, and the stems plugin all now depend on, but which
was never added to the spec — so a spec-compliant pack stopped being a
fully-working pack, the reference validator could not warn authors about a
key it had never heard of, and third-party tooling began emitting an
`original/` directory reverse-engineered from an example in a code comment.
See #933.

We cannot mechanically prove core interprets a key the way the spec means.
We can prove three surface properties, and they cover the drift that
actually happens:

  1. key-coverage — every manifest key core reads is declared in the spec's
     manifest.schema.json (AST scan of lib/sloppak.py, lib/enrichment.py,
     lib/songmeta.py).
  2. forward — core's load_song() ingests every example pack the spec ships.
  3. reverse — every pack committed here passes the spec's own
     tools/validate.py (7/7 pass today).

The spec is pinned by SHA in .feedpak-spec-ref rather than tracked from its
default branch, so a change over there cannot redden an unrelated PR here;
bump it in its own PR, where a red result is precisely the signal that core
does not satisfy the new spec.

A gate with no legitimate way to say "yes, deliberately, not yet" gets
switched off the first time it blocks a release, so there are two escape
hatches: the reserved `x-` key prefix (always permitted, and it tells every
third-party packer the key is not stable surface), and
feedpak-spec-exceptions.yml, which requires a tracking issue per entry. An
exception that goes stale — the spec caught up, or core stopped reading the
key — fails the build, so the allowlist cannot become somewhere drift
quietly accumulates. `original_audio` is seeded there against #933 so the
gate lands green and starts blocking the next instance immediately, rather
than requiring #933 to be resolved first.

Dev/CI tooling only; never on the serve or Docker path (constitution
Principle I). jsonschema is installed in the CI job, not added to
requirements.txt.

Signed-off-by: topkoa <topkoa@gmail.com>

docs/feedpak-spec-gate.md

@@ -0,0 +1,78 @@
+# The feedpak spec-conformance gate
+
+`tools/check_spec_conformance.py`, run in CI as the `feedpak-spec` job.
+
+## Why
+
+feedpak is published as an **open format**: its own repo
+([got-feedback/feedpak-spec](https://github.com/got-feedback/feedpak-spec)), a normative spec, JSON
+Schemas, and a reference validator. That is a promise to everyone outside this codebase — third-party
+packers, converters, and players build against the spec, and the spec is meant to be the complete and
+authoritative description of a pack.
+
+The moment core reads a manifest key the spec doesn't define, that promise breaks silently:
+
+- A spec-compliant pack is no longer guaranteed to be a fully-working pack.
+- The reference validator can't warn authors about a key it has never heard of — it will happily green-light
+  the key, and every misspelling of it.
+- The format's real definition drifts into our source tree. In the case that motivated this gate
+  ([#933](https://github.com/got-feedback/feedback/issues/933)), third-party tooling started emitting an
+  `original/` directory that no code anywhere requires — the convention was reverse-engineered from an
+  example in a *code comment*.
+
+The rule this gate enforces: **any manifest key core reads must be in the spec before core ships code that
+depends on it.** Spec first, implementation second.
+
+## What it checks
+
+We can't mechanically prove core *interprets* a key the way the spec means. We can prove three surface
+properties, and they cover the drift that actually occurs.
+
+| Layer | Check | Catches |
+|---|---|---|
+| 1. key-coverage | Every manifest key core reads is declared in the spec's `manifest.schema.json`. | Core growing a key the spec never defined — the #933 class. |
+| 2. forward | Core's `load_song()` ingests every example pack the spec ships. | The spec adding or tightening something core ignores or breaks on. |
+| 3. reverse | Every pack committed to this repo passes the spec's `tools/validate.py`. | Core (or a contributor) committing a pack the spec would reject. |
+
+Layer 1 works by walking the AST of the modules listed in `READERS` and collecting every literal key read
+off a manifest dict (`manifest.get("x")`, `manifest["x"]`, and the wrapped
+`(load_manifest(p) or {}).get("x")` form used in `lib/enrichment.py`).
+
+## When it fails
+
+You added a manifest key. Three ways forward, in order of preference:
+
+1. **Land it in the spec first.** Open a PR against `feedpak-spec` adding the key to
+   `schemas/manifest.schema.json` and `spec/feedpak-v1.md`, bump `.feedpak-spec-ref` here to the merged
+   SHA, and your key passes. This is the intended path.
+2. **Mark it experimental.** Prefix the key `x-` (e.g. `x-my_new_key`). The gate permits `x-`-prefixed keys
+   unconditionally, and the prefix signals to every third-party packer that the key is not stable surface.
+3. **Record an exception.** Add it to `feedpak-spec-exceptions.yml` with a tracking issue. This is debt, and
+   the gate treats it as such: an exception goes stale (and fails the build) the moment the spec catches up
+   or core stops reading the key, so the allowlist can't become somewhere drift quietly accumulates.
+
+## Pinning
+
+`.feedpak-spec-ref` holds the SHA of the `feedpak-spec` commit this repo is verified against. Pinned rather
+than tracking the spec's default branch on purpose — a change over there must never turn CI red on an
+unrelated PR here.
+
+When the spec moves, bump the SHA in its own PR. If that PR is red, the spec changed in a way core doesn't
+satisfy — exactly the signal we want, delivered as a reviewable PR rather than a surprise on someone else's
+branch.
+
+## Limitations
+
+Known, and worth fixing in follow-ups rather than blocking on:
+
+- **Layer 1 is name-heuristic.** It recognises manifest dicts bound to locals named in `MANIFEST_VARS`
+  (`manifest`, `mf`) plus the `load_manifest(...)` call form. This works because the loaders use a uniform
+  idiom, but it is fragile against a refactor that renames the local. The hardening step is to route all
+  manifest access through a single declared `KNOWN_MANIFEST_KEYS` registry in `lib/sloppak.py`; the gate
+  then compares registry against schema exactly instead of inferring.
+- **Layer 1 covers top-level keys only.** Nested structure (`arrangements[].file`, `.id`, `.notation`) isn't
+  checked. Extending to it means walking the schema's `$ref` subschemas.
+- **Layer 3 can't catch unknown keys**, because `manifest.schema.json` sets `additionalProperties: true` and
+  the reference validator deliberately "treats unknown keys/files as forward-compatible". Fixing this
+  properly belongs in the spec (tighten the schema, or give the validator a `--strict` mode). Until then,
+  layer 1 is the only thing standing between us and the next `original_audio`.