Compare commits

...
Author SHA1 Message Date
byrongamatosandClaude Fable 5 72212e7bfd fix(career): gig backfill offsets by qualifying taken, not picks length
CodeRabbit on #954: after the stakes loop appends near-bar songs,
qualifying[len(picks):] overshoots and skips eligible qualifying songs
— a stocked passport could still get a short set.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 01:26:04 +02:00
byrongamatosandClaude Fable 5 c59883aac9 feat(career): gigs backend — propose a setlist, log the completed set
Career v3, WS3 (backend half). A gig is career's verb:

- POST /gigs/propose {instrument, genre, size}: setlist from the
  passport's own stubs — qualifying songs (per the genre's badge bar,
  family-aware) shuffled for a free re-roll, topped with the
  highest-accuracy near-bar songs as stakes, and filled from UNPLAYED
  genre songs when the passport is young (the first gig is how stubs
  start). Names the highest venue the current stars can book.
- POST /gigs: logs a COMPLETED set only (abandoned sets never log — no
  fail state). Per-song accuracy = MAX(last_accuracy) from song_stats,
  freshly written by the set's own plays; encore = avg ≥ the data-driven
  bar (passports.json gig.encore_accuracy, 0.75). Appends to the career
  state file (same atomic _save_json pattern).
- Passports view: per-passport gigs (newest first, capped 20) and
  per-instrument gig_count — the profile wall's gig line lights up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 01:26:04 +02:00
6272af8d33 feat(career): profile passport wall, home career card, shareable PNG card (#955)
* feat(career): profile passport wall, home career card, shareable PNG card

Career v3, WS2. The identity artifact leaves the plugin tab:

- Profile: #v3-profile-passports-mount (core, one div) filled by career
  on v3:profile-rendered — per-instrument shelves of earned covers,
  hours, gig count, open-career link. Absent-not-empty.
- Home: the plugin-count stat tile becomes #v3-dash-career-slot with the
  old stat as fallback content; career replaces it with a trading-card
  tile (leather + foil shine, badge count, hours, closest-stamp ask) on
  the existing v3:dashboard-rendered event.
- Shareable card: static/js/blob-io.js (downloadBlob lifts the idiom
  duplicated verbatim in settings-io/diagnostics-export — both
  refactored; copyImageBlob wraps ClipboardItem, returns false to signal
  the download fallback). Earned passports get Save/Copy card: a
  natively-drawn 480×640 canvas (leather, stamp ring, stubs+hours line);
  copy falls back to download with a notice when the clipboard refuses.
- Mount-point convention documented in docs/plugin-v3-ui.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(career): external surfaces stay absent until a passport exists

CodeRabbit on #955: a bare commitment produced a zero-passport wall and
replaced the dashboard fallback. Docs also now say mounts may hold
fallback content and plugins REPLACE, never append.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 01:25:39 +02:00
831117fb96 feat(career): practice invitations — closest stamps + bring-these-up (#953)
* feat(career): practice invitations — closest stamps + bring-these-up

Career v3, WS1. The passport now points at the practice that pays:

- Stubs carry next_star_at (the same primitive _stars() uses) and each
  passport exposes `nearest`: the top 3 non-qualifying songs by distance
  to their next star, in the worklist order.
- "Closest stamps" strip above the shelf: the in-progress graded
  passports nearest to minting, each row naming the ask — N more songs
  (with the nearest title + best %) or the blocking Virtuoso drill.
  Rows open the passport.
- "Bring these up" list on the stubs page of in-progress passports;
  earned pages stay memorabilia (no homework on a won badge).
- Invitation-voiced throughout: no meters, no completion pressure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(test): comment says qualifying-bar ranking, matching the assertion

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 01:10:55 +02:00
6cc0312661 feat(career): genre families — sub-genres inherit the family drill (#951)
The enrichment fallback made the passport rack real (hundreds of MB
sub-genres) but only the five exact umbrella keys carried Virtuoso
drills. Genres now resolve to a family by keyword substring (MB's
vocabulary is open — 'metalcore' must hit metal without an alias),
first-match-wins in list order ('blues rock' → blues), and inherit the
family's requirement from the same genres map. Exact entries still win;
per-instrument scoping unchanged; unmatched genres stay songs-only.

Data: families for metal (incl. djent/grindcore/thrash/doom), blues,
jazz (bebop/swing/bossa), funk (disco), rock (punk/grunge/shoegaze).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 20:05:05 +02:00
329cc86315 fix(sloppak): the full mix is a stem — drop the invented original_audio key (#946)
Nightly / build-docker (push) Has been cancelled
ship-ci / ci (push) Waiting to run
* fix(sloppak): the full mix is a stem — drop the invented `original_audio` key (#933)

Core read, served, and depended on `original_audio:` — a top-level manifest key
this repo invented in #583 that the feedpak spec never defined. The format
already had a home for the pre-separation mixdown: it is a stem. feedpak 1.15.0
(feedpak-spec#53) RESERVES the id `full` for it, so read it from there.

The key existed to work around a bug in our own reader. The packer's comment
said so plainly: "we must NOT list the full mix as a playable stem — the player
sums every entry in `stems` and does not gate playback on `default`, so a listed
full mix plays on top of the stems". Faced with a reader that would double the
song, the packer put the mixdown outside `stems` and invented a key to point at
it. The fix belongs in the reader, and that is what this is.

load_song() now partitions the stem list: `full` comes out as
LoadedSloppak.full_mix, the instruments stay in .stems. Nothing that sums stems
or draws one fader per stem can see the mixdown, so retaining it is safe — which
is what lets the packer put it where the format says it goes.

- ws_highway: `song_info` gains full_mix_url / has_full_mix. The old
  original_audio_url / has_original_audio remain as deprecated aliases for one
  release so an older stems plugin keeps working (#945).
- `stems` on the wire, and stem_ids / stem_count in the library index, are now
  INSTRUMENT stems only — a separated pack that retains its mixdown no longer
  advertises a bogus "full" chip or an inflated stem count.
- enrichment: fingerprint against the mixdown wherever it lives. This widens
  coverage — _song_audio_file() previously returned None for any pack without
  the invented key, so fingerprinting silently did nothing for nearly every pack.
- sloppak: `original_audio:` is still READ as a deprecated fallback, because
  every pack in the wild carries it and would otherwise lose its pristine mix.
  tools/migrate_full_mix_stem.py rewrites those packs into the spec shape
  (original/full.ogg -> stems/full.ogg, add the `full` stem at default:off, drop
  the key); the fallback and the aliases die with #945.

The spec gate keeps the debt honest: the grandfather entry now tracks #945, and
the gate fails if it goes stale.

Verified: spec gate OK (4/4, incl. ingesting the spec's new example pack that
retains `full`); 2493 python tests, 995 js tests; migrator round-tripped over
real packs from the library and the results pass the spec's reference validator.

* fix(migrate): discover directory-form packs instead of silently skipping them

iter_packs() searched only files, so a directory-form pack (`song.sloppak/`, the
authoring shape) was walked INTO and never yielded — silently missed by a run
that's meant to be exhaustive. Discover suffix-named directories too (yielded
whole, not descended into), and route packs through migrate_pack/verify_pack.

Directory packs are REPORTED as `dir-form-unsupported`, not rewritten in place:
a single-file pack is replaced atomically (a fully-built temp archive swapped in
with one os.replace), but a populated directory can't be swapped that way, so an
interrupted in-place rewrite could leave an authoring pack half-migrated. The
status is a problem status, so it counts against the run's exit code and shows
in the summary — the operator re-packs or migrates it as a `.feedpak` instead of
it vanishing from the report. Addresses a CodeRabbit review finding.

Signed-off-by: Kris Anderson <topkoa@gmail.com>

* fix(migrate): verify requires an explicit `off` on a retained full mix

verify_zip accepted any non-truthy `default` on a multi-stem `full` (missing,
empty, boolean, `false`/`no`/`0`, malformed) as "ok". But core defaults an ABSENT
`default` to True — ON (lib/sloppak.py: `s.get("default", True)`) — and treats an
empty/unrecognized string as ON too, so a migrated-shape pack whose `full` stem
has a missing or blank default beside instrument stems would actually play the
mixdown on open and double the song. verify was certifying that as safe.

Require an explicit normalized `off` beside instrument stems: `on`-ish values are
reported `full-stem-default-on` (actively plays), everything that is not a
normalized `off` is reported `full-stem-default-not-off`. The migrator already
writes the literal `off`, so its own output is unaffected; this also certifies
the pack is in the tool's canonical, most-portable shape. The len>1 gate is kept,
so a sole `full` stem (which IS the audio) is not policed.

Adds parametrized coverage for missing / empty / boolean / off-ish / malformed
defaults, and a sole-full-stem case. Addresses a CodeRabbit review finding.

Signed-off-by: Kris Anderson <topkoa@gmail.com>

---------

Signed-off-by: Kris Anderson <topkoa@gmail.com>
Co-authored-by: Kris Anderson <topkoa@gmail.com>
2026-07-13 12:22:42 -04:00
Byron GamatosandGitHub d876ded00f fix(sloppak): bound the unpack cache; add read_member_bytes() so callers stop unpacking whole songs (#950)
* fix(sloppak): bound the unpack cache, and add a way to read a song without unpacking it

A tester's sloppak_cache reached 60 GB from an 1800-song library — his entire
library, unpacked, none of it played. Stems are already-compressed audio, so an
unpacked pack is ~1.1x its zip: the cache is a second, DECOMPRESSED copy of every
song it touches. It had no size cap, no LRU, and no cleanup of any kind — not even
when the song itself was deleted.

Two halves:

1. resolve_source_dir() now evicts least-recently-used songs to stay under a cap
   (FEEDBACK_SLOPPAK_CACHE_MAX_MB, default 4 GB ≈ 130 songs of recency; 0 disables).
   The sweep runs on unpack — the only moment the cache grows — so it can't drift.
   An evicted song is dropped from _source_cache too: get_cached_source_dir() is
   the only thing media.py consults before falling back, so a stale path there
   would 404 every stem for the rest of the process instead of re-unpacking.
   get_cached_source_dir() now also verifies the dir still exists, which makes
   "just delete sloppak_cache/ to reclaim disk" safe advice.

2. read_member_bytes() reads ONE file out of a pack without unpacking it — the
   same trick read_cover_bytes() uses so the library grid doesn't explode every
   pack to show a cover. Unpacking a whole song to read a few KB of JSON is ~45x
   write amplification; doing it in a loop over the library is what produced the
   60 GB. rig_builder's library-wide tone batch is the caller that did exactly
   that (fixed separately); this gives it, and everyone else, the right primitive.

Eviction is concurrency-safe: unpacks run 2-at-a-time, so a dir being written is
marked in-flight and the sweep skips it — checked and rmtree'd under one hold of
the guard, and the marker is released even if the unpack raises (a leaked marker
would make that dir permanently un-evictable).

read_member_bytes normalizes both the requested path AND the archive's stored
member names through safe_join, taking the last match — so './arrangements/x.json',
backslash members from Windows tooling, and duplicate members that normalize to
the same path all read back exactly as unpack-then-read did. Zip-slip is rejected
before anything is opened.

Tests: tests/test_sloppak_unpack_cache.py. All bite-tested (reverted each fix,
watched it fail) — including one that was passing vacuously: a freshly-unpacked
dir is the most-recently-used, so the LRU never reaches it and the in-flight race
test proved nothing until the packs were sized to force the sweep that far.

* test: split semicolon-joined statements (E702)

CodeRabbit on #950. Style only; no behaviour change.
2026-07-13 17:13:08 +02:00
18d77d2d41 feat(library): effective genre falls back to MusicBrainz enrichment (#949)
* feat(library): effective genre falls back to MusicBrainz enrichment

Converted packs rarely carry a genres manifest key — on Byron's real
library 1188/1190 songs had no genre, starving the genre facet and the
career passport rack (2 usable genres) while song_enrichment already
held MB genres for 636 matched songs.

The effective-genre expression now resolves: per-song override → pack
genre → json_extract(enrichment.genres, '$[0]') for MATCHED rows only
(review/failed candidates could carry the wrong recording's genres).
Same fast-path gating as before: the plain indexed column is used
unless overrides or enrichment genres actually exist; stand-in DBs
without the table degrade via the OperationalError guard. Career
passports pick this up automatically through _effective_genre_expr().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: changelog names manual rows in the enrichment fallback

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 16:53:06 +02:00
5921157f35 test(stats): prove seconds-only recency on a fresh row (#948)
CodeRabbit on #947: the prior lastPlayPosition POST already stamped
last_played_at, so the assertion passed even if the seconds-only path
left it unchanged — a guard that cannot fail. Assert on a fresh row.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 15:36:05 +02:00
3e57ba0345 fix(career): hours polish — recency stamp + non-2xx POST is a failure (#947)
CodeRabbit follow-up on #942:

- add_play_seconds() now stamps last_played_at (like touch_position):
  an unscored play that ran to the natural end WAS played — recent /
  Continue ordering must see it. Resume position stays untouched.
- stats-recorder post() treats non-2xx as failure: a 4xx/5xx JSON error
  body parsed as an object read as success, silently dropping the
  accrued seconds instead of re-queuing them.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 15:31:12 +02:00
29 changed files with 3089 additions and 281 deletions
+63
View File
@@ -8,6 +8,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **Career on the Profile and Home pages** — the Profile gains a passport
wall (earned-badge covers per instrument, hours, gig count; absent until a
passport exists), injected through the same mount-point + rendered-event
seam the achievements plugin uses (now documented in docs/plugin-v3-ui.md).
The home page's plugin-count stat tile becomes a career trading card
(badges, hours, the closest stamp ask, foil shine) with the old stat as the
built-in fallback when career has no state. Earned passports gain **Save
card / Copy card** — a natively-drawn PNG passport card, downloadable or
copied straight to the clipboard for pasting outside the app (shared
`blob-io` helpers replace the download idiom previously duplicated in
settings-io and diagnostics-export).
- **Gigs (backend)** — career mode gains its verb: `POST
/api/plugins/career/gigs/propose` builds a playable setlist for an
instrument+genre (your qualifying songs plus a couple of stakes songs near
the bar; a young passport fills from unplayed genre songs — the first gig
is how stubs start; re-roll by calling again), naming the room your stars
can book. `POST /gigs` logs a **completed** set — per-song accuracies read
from the set's own freshly-recorded stats, an encore flag at the
data-driven bar (avg ≥ 75%) — into the career state; abandoned sets never
log (no fail state: the gig you finished is the gig you played). Passports
carry their gig log; instruments their gig count.
### Changed
- **The full mix is a stem** (#933) — core no longer depends on `original_audio:`, a
top-level manifest key this repo invented (#583) that the feedpak spec never had.
The format already carried the pre-separation mixdown as a stem; feedpak 1.15.0
(feedpak-spec#53) reserves the id **`full`** for it, so that is where core reads it
from now.
`full` is a mixdown, not a layer — it already contains every instrument — so
`load_song()` lifts it OUT of `LoadedSloppak.stems` onto `LoadedSloppak.full_mix`.
Nothing that sums stems or renders one fader per stem can see it, which is what
makes retaining it safe; leaving it in the list would double the whole song and
leave "guitar" audible with the guitar fader muted. That trap is exactly why the
packer invented the key instead of putting the mixdown where the format says it
goes — the bug was in the reader, and this fixes the reader.
Consequences worth knowing:
- The highway WS `song_info` frame gains `full_mix_url` / `has_full_mix`.
`original_audio_url` / `has_original_audio` remain as **deprecated aliases**
(same values) for one release so a client built against the old frame keeps
working; they go with the fallback below (#945).
- `stems` on `song_info`, and `stem_ids` / `stem_count` in the library index, now
describe *instrument* stems only — a separated pack that retains its mixdown no
longer advertises a bogus "full" stem chip or an inflated stem count.
- Audio fingerprinting (`lib/enrichment.py`) now resolves the mixdown the same
way, which **widens** its coverage: it previously returned `None` for any pack
without the invented key, so fingerprinting silently did nothing for the
overwhelming majority of packs.
- Core still **reads** `original_audio:` as a deprecated fallback, because every
pack written before the spec caught up carries it and would otherwise lose its
pristine mix. `tools/migrate_full_mix_stem.py` rewrites those packs into the
spec shape (moves `original/full.ogg` → `stems/full.ogg`, adds the `full` stem
at `default: off`, drops the key); the fallback and the aliases are removed once
they are migrated (#945).
### Added
- **Genres fall back to MusicBrainz enrichment** — the effective genre now
resolves override → pack genre → the enrichment match's primary genre
(matched or user-pinned manual rows only). Converted packs rarely carry a `genres` manifest key,
which starved the library genre facet and career passports on real
libraries; with the fallback, every enriched song's genre is browsable and
passport-able immediately, and coverage grows as enrichment runs.
- **Badge ceremony in the venue** — earning a genre badge now stages a moment:
the crowd layer erupts (new public `v3VenueCrowd.celebrate()` — instant
ecstatic loop bypassing the stability/dwell hysteresis, plus a cheer stinger;
+20
View File
@@ -189,3 +189,23 @@ out of the capability graph.
- [ ] `#player` overlays keep `z-index` ≤ the chrome layers (transport/HUD 20,
rail 30, popovers 40).
- [ ] Verify at `/` — it and `/v3` serve the same (and only) v3 shell.
## Injecting into core shells (profile, dashboard)
Core screens that accept plugin sections render **mount points** — usually
empty, sometimes holding core's own **fallback content** (the Dashboard's
career slot ships the plugin-count stat) — and announce each (re)build with a
DOM event, because their `innerHTML` swap wipes anything previously injected.
A plugin listens for the event and **replaces the mount's content** (never
append — a fallback may be present) by id — the same seam every time:
| Shell | Event | Mounts |
| --- | --- | --- |
| Profile | `v3:profile-rendered` | `#v3-profile-passports-mount` (career wall), `#v3-profile-feats-slot`, `#v3-profile-achievements-mount` |
| Dashboard | `v3:dashboard-rendered` | `#v3-dash-career-slot` (career card; core's plugin-count stat is the fallback content a plugin may replace) |
| Settings | `v3:settings-rendered` | per-plugin `settings.html` panels |
Rules: inject on every event (the mount is fresh), keep the section
**absent-not-empty** (no state → leave the mount alone / empty), and guard
re-wired listeners with a `dataset` flag when your own refresh path can run
against an unwiped mount.
+15 -9
View File
@@ -34,17 +34,23 @@
exceptions:
- key: original_audio
issue: https://github.com/got-feedback/feedback/issues/933
issue: https://github.com/got-feedback/feedback/issues/945
reason: >-
Added by #583 (the full mix played while every stem fader sits at unity,
since demucs recombination is lossy). Core, lib/enrichment.py, and the
stems plugin all depend on it, but it never went through a FEP and the
since demucs recombination is lossy). It never went through a FEP and the
spec does not define it — the drift this gate exists to prevent.
The resolution is REMOVAL, not a FEP: the spec already carries the mixdown
as a stem ({id: full, file: stems/full.ogg}), so this key added a second,
redundant location for audio to a format that already had one. See #933.
#933 fixed the drift: feedpak 1.15.0 RESERVES the stem id `full` for the
complete mixdown (feedpak-spec#53), and core now reads the full mix from
that stem. Nothing depends on this key any more — not the loader, not
lib/enrichment.py, not the stems plugin, and the packer no longer writes it.
Grandfathered so the gate can land green and start blocking the *next*
instance immediately, rather than blocking on #933. This entry goes away
when core no longer reads or writes the key.
What remains is a READ-ONLY deprecated fallback in lib/sloppak.py
(_legacy_full_mix), kept for one release because every pack produced before
the spec caught up carries `original_audio: original/full.ogg` and would
otherwise silently lose its pristine mix. tools/migrate_full_mix_stem.py
rewrites those packs into the spec shape.
This entry disappears with that fallback — tracked by #945, which cannot be
forgotten: the gate fails if the entry goes stale, and deleting the read is
what makes it stale.
+20 -5
View File
@@ -368,10 +368,12 @@ def _acoustid_gate() -> "JSONResponse | None":
def _song_audio_file(filename: str) -> "str | None":
"""Resolve a LIBRARY song (by filename/id) to a local master-audio file for
fingerprinting: the full-mix `original_audio` extracted from a sloppak, or a
loose folder's audio. None when the song can't be found or ships no full-mix
audio (some packs carry only stems). Mirrors serve_sloppak_file's containment
guards so a crafted filename can't read outside DLC_DIR / the pack."""
fingerprinting: a sloppak's complete mixdown, or a loose folder's audio. None
when the song can't be found or carries no mixdown (a pack that kept only its
separated stems — an acoustic fingerprint of one re-summed from them would not
match the recording, so we decline rather than submit a lossy reconstruction).
Mirrors serve_sloppak_file's containment guards so a crafted filename can't
read outside DLC_DIR / the pack."""
dlc = _get_dlc_dir()
if not dlc:
return None
@@ -383,7 +385,20 @@ def _song_audio_file(filename: str) -> "str | None":
canon = resolved.relative_to(dlc.resolve()).as_posix()
except ValueError:
return None
rel = (sloppak_mod.load_manifest(resolved) or {}).get("original_audio")
manifest = sloppak_mod.load_manifest(resolved) or {}
# The mixdown is the RESERVED `full` stem (spec §5.3). Unlike playback,
# fingerprinting wants it even when it is the pack's ONLY stem — a
# single-mix pack is exactly the master audio we want to fingerprint —
# so this asks find_full_mix() rather than partition_stems().
stems = manifest.get("stems") or []
full = sloppak_mod.find_full_mix(
[s for s in stems if isinstance(s, dict)]
)
rel = full.get("file") if full else None
# DEPRECATED fallback: packs written before the spec reserved `full` put
# the mixdown behind a top-level `original_audio:` key instead (#933).
if not isinstance(rel, str) or not rel.strip():
rel = manifest.get("original_audio")
if not isinstance(rel, str) or not rel.strip():
return None
src = sloppak_mod.get_cached_source_dir(canon)
+53 -13
View File
@@ -1085,26 +1085,57 @@ class MetadataDB:
vals["artist"], vals["title"] = self._romaji_display(filename, vals["artist"], vals["title"])
return vals
# Effective genre = a per-song genre OVERRIDE (Fix-metadata popup) else the
# scanned pack genre. Applied at FILTER/FACET time (like the P4 artist alias)
# so a corrected genre is browsable — the correlated subquery is used ONLY
# when genre overrides actually exist; the common case stays on the plain
# indexed `genre` column. Genre stays a library-only overlay (it isn't a
# write-to-file field), so it never touches the pack.
_EFFECTIVE_GENRE_SQL = (
# Effective genre precedence: per-song OVERRIDE (Fix-metadata popup)
# scanned pack genre → MusicBrainz enrichment primary genre (matched/manual rows
# only — a 'review'/'failed' candidate's genres could belong to the wrong
# recording). Applied at FILTER/FACET time (like the P4 artist alias) so a
# corrected or enriched genre is browsable. The vast majority of converted
# packs carry no `genres` manifest key, so without the enrichment leg the
# genre facet (and career passports) starve on real libraries. The
# correlated subqueries are used ONLY when overrides/enrichment genres
# actually exist; the common case stays on the plain indexed `genre`
# column. Genre stays a library-only overlay (it isn't a write-to-file
# field), so it never touches the pack.
_EFFECTIVE_GENRE_OVERRIDE_SQL = (
"COALESCE((SELECT o.value FROM song_field_override o "
"WHERE o.filename = songs.filename AND o.field = 'genre' "
"AND o.value IS NOT NULL AND o.value != ''), genre)"
)
_EFFECTIVE_GENRE_SQL = (
"COALESCE((SELECT o.value FROM song_field_override o "
"WHERE o.filename = songs.filename AND o.field = 'genre' "
"AND o.value IS NOT NULL AND o.value != ''), "
"NULLIF(genre, ''), "
"(SELECT json_extract(e.genres, '$[0]') FROM song_enrichment e "
"WHERE e.filename = songs.filename AND e.match_state IN ('matched', 'manual') "
"AND e.genres IS NOT NULL AND e.genres NOT IN ('', '[]')), "
"'')"
)
def _has_genre_overrides(self) -> bool:
return self.conn.execute(
"SELECT 1 FROM song_field_override WHERE field = 'genre' "
"AND value IS NOT NULL AND value != '' LIMIT 1").fetchone() is not None
def _has_enrichment_genres(self) -> bool:
try:
return self.conn.execute(
"SELECT 1 FROM song_enrichment WHERE match_state IN ('matched', 'manual') "
"AND genres IS NOT NULL AND genres NOT IN ('', '[]') "
"LIMIT 1").fetchone() is not None
except sqlite3.OperationalError:
return False # stand-ins / DBs without the enrichment table
def _effective_genre_expr(self) -> str:
"""`genre` normally; the override-aware COALESCE only when overrides exist."""
return self._EFFECTIVE_GENRE_SQL if self._has_genre_overrides() else "genre"
"""`genre` normally; the enrichment-aware COALESCE only when trusted
enrichment genres exist (which also proves the table exists — a
stand-in DB without song_enrichment must never receive SQL that
references it); the override-only form when just overrides exist."""
if self._has_enrichment_genres():
return self._EFFECTIVE_GENRE_SQL
if self._has_genre_overrides():
return self._EFFECTIVE_GENRE_OVERRIDE_SQL
return "genre"
def set_song_tags(self, filename: str, tags) -> list:
"""Replace ALL of a song's tags with the given set (each normalized;
@@ -2130,16 +2161,25 @@ class MetadataDB:
return self._stats_row(filename, int(arrangement))
def add_play_seconds(self, filename: str, arrangement: int, seconds: float) -> dict:
"""Accrue wall-clock play time only (no plays/score/position change) —
"""Accrue wall-clock play time (no plays/score/position change) —
the recorder's seconds-only flush for unscored plays that ran to the
song's natural end (no resume position to touch there: `song:ended`
must not overwrite Continue with the end-of-song offset)."""
must not overwrite Continue with the end-of-song offset). Stamps
last_played_at like touch_position does: the song WAS played, so
/api/stats/recent and Continue ordering must see it. Accepted skew:
the recorder retries FAILED flushes later, which stamps recency at
retry time — rare (offline corner), self-healing on the next play,
and preferable to the alternative (keep-existing would leave repeat
plays looking stale, the common case)."""
with self._lock:
self.conn.execute(
"""INSERT INTO song_stats (filename, arrangement, seconds_total, updated_at)
VALUES (?, ?, ?, strftime('%Y-%m-%d %H:%M:%f','now'))
"""INSERT INTO song_stats (filename, arrangement, seconds_total,
last_played_at, updated_at)
VALUES (?, ?, ?, strftime('%Y-%m-%d %H:%M:%f','now'),
strftime('%Y-%m-%d %H:%M:%f','now'))
ON CONFLICT(filename, arrangement) DO UPDATE SET
seconds_total = song_stats.seconds_total + excluded.seconds_total,
last_played_at = excluded.last_played_at,
updated_at = excluded.updated_at""",
(filename, int(arrangement), float(seconds)),
)
+43 -19
View File
@@ -321,11 +321,16 @@ async def highway_ws(websocket: WebSocket, filename: str, arrangement: int = -1,
audio_url = None
audio_error: str | None = None # Surfaced in song_info when audio_url is None
stems_payload: list[dict] = []
# URL of the single full-mix audio (sloppak `original_audio:`), when the
# pack ships one. The stems plugin uses this to play the untouched mix
# while every stem slider is at unity; None otherwise (separate stems
# only, loose folder, or archive).
original_audio_url: str | None = None
# URL of the pack's complete mixdown — the RESERVED `full` stem (spec
# §5.3), which sloppak.load_song() lifts out of `stems` because it is a
# mixdown, not a layer. The stems plugin plays it while every stem slider
# is at unity (separation is lossy, so it beats re-summing the stems) and
# crosses to the separated stems as soon as one is attenuated.
#
# None when the pack has no mixdown to offer separately from its stems:
# a single-mix pack (its one stem IS the mixdown), a loose folder, or an
# archive.
full_mix_url: str | None = None
if is_loose:
# Loose folder filenames are relative paths (artist/album/song).
# Hash the *canonical* dlc-relative path (so two URL spellings
@@ -365,21 +370,25 @@ async def highway_ws(websocket: WebSocket, filename: str, arrangement: int = -1,
url = f"/api/sloppak/{q_fn}/file/{quote(s['file'])}"
stems_payload.append({"id": s["id"], "url": url, "default": s["default"]})
# Full-mix URL (served by the same /api/sloppak/.../file/ endpoint).
if loaded_slop is not None and loaded_slop.original_audio:
original_audio_url = (
f"/api/sloppak/{q_fn}/file/{quote(loaded_slop.original_audio)}"
if loaded_slop is not None and loaded_slop.full_mix:
full_mix_url = (
f"/api/sloppak/{q_fn}/file/{quote(loaded_slop.full_mix)}"
)
if stems_payload:
# Stems present: keep the core <audio> pointed at stem[0]. This
# URL is only ever heard in the degraded path (stems plugin
# refuses takeover / decode fails); the full-mix↔stems switch is
# driven client-side by `original_audio_url`, not `audio_url`.
# driven client-side by `full_mix_url`, not `audio_url`.
audio_url = stems_payload[0]["url"]
elif original_audio_url:
elif full_mix_url:
# Stem-less full-mix pack: nothing to separate, so play the full
# mix natively through the core <audio>. The stems plugin's
# onSongReady returns early on an empty stems list (no graph).
audio_url = original_audio_url
# Reachable only via the deprecated `original_audio:` key, whose
# packs put the mixdown outside `stems` — a pack that carries its
# mixdown as the `full` stem has it IN `stems`, so it lands in the
# branch above with stems_payload == [full].
audio_url = full_mix_url
else:
audio_error = "This sloppak has no playable stems."
else:
@@ -521,16 +530,31 @@ async def highway_ws(websocket: WebSocket, filename: str, arrangement: int = -1,
# for the credits overlay, so minigames / synthetic highway uses
# (no manifest) never trigger it.
"authors": _sanitize_authors(loaded_slop.manifest) if (is_slop and loaded_slop is not None) else [],
# Instrument stems ONLY. The pack's complete mixdown (the RESERVED
# `full` stem, spec §5.3) is deliberately NOT in this list: consumers
# sum `stems` into one mix and render one fader per entry, and the
# mixdown is neither a layer nor an instrument — summing it would
# double the whole song. It is surfaced separately, below.
"stems": stems_payload,
# Full-mix audio (sloppak `original_audio:`) served alongside the
# separate `stems`. The stems plugin plays this single file while
# every stem slider is at unity and switches to the separate stems
# the moment one drops below 100%. None when the pack ships stems
# only. `has_*` flags mirror the has_drum_tab/has_keys convention so
# a client can branch without re-deriving from the URLs.
"original_audio_url": original_audio_url,
"has_original_audio": bool(original_audio_url),
# The complete mixdown, served by the same /api/sloppak/.../file/
# endpoint as the stems. The stems plugin plays this single file
# while every stem slider is at unity and crosses to the separated
# stems the moment one drops below 100% — separation is lossy, so the
# mixdown is strictly better audio when nothing is muted. None when
# the pack has no mixdown apart from its stems. The `has_*` flags
# mirror the has_drum_tab/has_keys convention so a client can branch
# without re-deriving from the URLs.
"full_mix_url": full_mix_url,
"has_full_mix": bool(full_mix_url),
"has_stems": bool(stems_payload),
# DEPRECATED aliases of the two keys above, kept so a client built
# against the old frame keeps working across one release. They were
# named after `original_audio:` — a manifest key this repo invented
# and the feedpak spec never had (#933). The key is gone; the mixdown
# is a stem. Remove these once the shipped stems plugin reads
# `full_mix_url` (#945).
"original_audio_url": full_mix_url,
"has_original_audio": bool(full_mix_url),
# Surface a drum_tab presence flag so the visualization picker
# can auto-activate the drums plugin even when the chosen
# arrangement isn't named "Drums" (drum_tab.json lives next
+409 -61
View File
@@ -15,6 +15,7 @@ from __future__ import annotations
import logging
import math
import os
import shutil
import threading
import zipfile
@@ -34,6 +35,21 @@ FEEDPAK_EXT = ".feedpak"
SLOPPAK_EXT = ".sloppak"
SONG_EXTS = (FEEDPAK_EXT, SLOPPAK_EXT) # accepted on read/discovery
# ── The full mix ──────────────────────────────────────────────────────────────
#
# Spec §5.3 RESERVES the stem id `full` for the song's complete mixdown: the
# whole song in one file, as heard before source separation. It is a stem — it
# lives in `stems` like every other audio file in a pack — but it is a *mixdown,
# not a layer*. A reader that sums stems must never include it in the sum: it
# already contains every instrument, so summing it doubles the whole song and
# muting `guitar` still leaves guitar audible inside it.
#
# Keeping it matters because separation is lossy: re-summing guitar+bass+drums+
# vocals does NOT reproduce the file they came from. The mixdown is the only
# faithful rendering of the song a pack can carry, so we play it whenever every
# stem sits at unity and nothing is muted.
FULL_MIX_STEM_ID = "full"
import yaml
from jsonc import load_json
@@ -51,6 +67,97 @@ import drums as drums_mod
import notation as notation_mod
def find_full_mix(stems: list[dict]) -> dict | None:
"""The RESERVED `full` stem (spec §5.3) — the pack's complete mixdown — or None.
Answers "what is this pack's master audio", which is what fingerprinting
wants. For playback use partition_stems() instead: a pack whose *only* stem
is `full` has no mixdown to play *separately from* its stems, and this
function still returns it.
"""
return next(
(s for s in stems if str(s.get("id", "")) == FULL_MIX_STEM_ID), None
)
def partition_stems(stems: list[dict]) -> tuple[dict | None, list[dict]]:
"""Split stem descriptors into (mixdown, instrument_stems) for PLAYBACK.
The mixdown is lifted OUT of the stem list because every consumer of `stems`
treats that list as layers to sum or to show as mixer channels, and `full` is
neither (spec §5.3). Leaving it in is precisely the bug that made the packer
invent `original_audio` in the first place: a listed full mix plays on top of
the stems.
A pack whose only stem is `full` is a single-mix pack, not a separated one:
there are no instruments to be pristine *against*, so `full` stays the sole
playable stem and no mixdown is surfaced. That keeps the freshly-converted
single-stem pack much the most common shape behaving exactly as before.
EVERY entry with the reserved id is removed, not just the one we surface. A
malformed pack that lists `full` twice would otherwise leave a copy of the
whole song behind in the stem list, to be summed with the instruments the
precise failure this function exists to prevent, reintroduced by a duplicate.
"""
if len(stems) < 2:
return None, stems
full = find_full_mix(stems)
if full is None:
return None, stems
return full, [s for s in stems if str(s.get("id", "")) != FULL_MIX_STEM_ID]
def _legacy_full_mix(manifest: dict, source_dir: Path) -> str | None:
"""Full mix from the DEPRECATED `original_audio:` manifest key, or None.
Before feedpak 1.15.0 reserved `full`, §5.3 said the mixdown was "commonly
replaced" by the per-instrument stems on splitting — so it had nowhere to
live, and this repo invented a top-level key pointing at a parallel
`original/` directory (#583) to hold it. That key was never in the spec, and
#933 removed our dependence on it: the mixdown is a stem.
We still READ it, because every pack written before the spec caught up
carries `original_audio: original/full.ogg` and would otherwise lose its full
mix. We never write it. Delete this once those packs are migrated (#945);
`tools/migrate_full_mix_stem.py` is the migration.
NOTE the string literal below. tools/check_spec_conformance.py AST-scans for
`manifest.get("<literal>")` to prove every manifest key core reads is one the
spec declares. Hoisting "original_audio" into a named constant would hide
this read from that scan the gate would conclude core no longer touches the
key, and the grandfather entry that documents this debt would go stale. The
literal is what keeps the deprecation honest and visible to CI. Leave it.
Same permissive, path-traversal-guarded posture as the optional side-files: a
missing / escaping / unreadable file leaves the pack without a full mix (the
player falls back to the separated stems) rather than aborting the load.
Returns the manifest-relative string, so callers build its URL exactly as
they build a stem's.
"""
rel_raw = manifest.get("original_audio")
if not isinstance(rel_raw, str) or not rel_raw.strip():
return None
rel = rel_raw.strip()
try:
target = (source_dir / rel).resolve()
target.relative_to(source_dir.resolve())
except ValueError:
log.warning("sloppak: original_audio path %r escapes source_dir — skipped", rel)
return None
except OSError as e:
log.warning("sloppak: original_audio path resolution failed (%s) — skipped", e)
return None
if not target.is_file():
return None
log.info(
"sloppak: pack uses the deprecated `original_audio:` key (%r) — the full mix "
"is a stem (id `full`, feedpak spec §5.3). Re-pack with "
"tools/migrate_full_mix_stem.py; support for this key will be removed.",
rel,
)
return rel
# ── Format detection ──────────────────────────────────────────────────────────
def is_sloppak(path: Path) -> bool:
@@ -81,6 +188,116 @@ _unpack_semaphore = threading.BoundedSemaphore(_UNPACK_MAX_CONCURRENCY)
_unpack_locks: dict[str, threading.Lock] = {}
_unpack_locks_guard = threading.Lock()
# Destinations with an unpack in flight right now. Eviction MUST skip these: two
# unpacks run concurrently, so one finishing could otherwise rmtree the other's
# half-written directory and leave that resolver caching an incomplete song.
_unpacking: set[Path] = set()
_unpacking_guard = threading.Lock()
# Cap the unpack cache. Stems are already-compressed audio, so an unpacked song
# is ~1.1x its zip — the cache is effectively a second, DECOMPRESSED copy of
# every song it holds, and it used to grow without any bound at all. A tester
# reached 60 GB from a 1800-song library: their whole library, unpacked, because
# one caller looped the library calling load_song(). Nothing ever deleted any of
# it — not even when the song itself was deleted.
#
# Default 4 GB ≈ 130 average songs of recency, which is far more than the "the
# song I'm playing, and the last few I played" that this cache actually exists
# to serve. Override with FEEDBACK_SLOPPAK_CACHE_MAX_MB (0 disables eviction).
def _unpack_cache_cap_bytes() -> int:
raw = os.environ.get("FEEDBACK_SLOPPAK_CACHE_MAX_MB", "").strip()
try:
mb = int(raw) if raw else 4096
except ValueError:
mb = 4096
return max(0, mb) * 1024 * 1024
def _dir_size(path: Path) -> int:
total = 0
for f in path.rglob("*"):
try:
if f.is_file():
total += f.stat().st_size
except OSError:
continue
return total
def _touch(path: Path) -> None:
"""Bump mtime so the LRU sweep below treats this song as recently used.
Reading files out of an unpacked dir doesn't change the DIRECTORY's mtime,
so without this the song you are actively playing looks as stale as one you
unpacked days ago and a burst of unpacks could evict it mid-song.
"""
try:
os.utime(path, None)
except OSError:
pass
def _evict_unpack_cache(root: Path, keep: Path | None = None) -> None:
"""Bound the unpack cache: drop least-recently-used songs until under the cap.
`keep` is never evicted it's the song the caller just resolved, i.e. almost
certainly the one about to be played.
Evicting a directory MUST also drop its `_source_cache` entry. Otherwise
get_cached_source_dir() keeps handing out a path that no longer exists and
the media route 404s on every stem instead of re-unpacking (it only falls
back to resolve_source_dir when the cache returns None).
"""
cap = _unpack_cache_cap_bytes()
if cap <= 0:
return
try:
entries = []
total = 0
for d in root.iterdir():
if not d.is_dir():
continue
try:
size = _dir_size(d)
mtime = d.stat().st_mtime
except OSError:
continue
entries.append((mtime, size, d))
total += size
if total <= cap:
return
keep_resolved = keep.resolve() if keep else None
entries.sort(key=lambda e: e[0]) # oldest first
for _mtime, size, d in entries:
if total <= cap:
break
try:
if keep_resolved and d.resolve() == keep_resolved:
continue
except OSError:
continue
# Check-and-delete under ONE hold of the guard. Releasing between the
# two would let a resolver mark this dest in-flight and start writing
# into it in the gap, and we'd rmtree a song mid-unpack. A resolver
# that blocks here simply proceeds afterwards — _unpack_zip recreates
# the directory anyway.
with _unpacking_guard:
if d in _unpacking:
continue # another thread is writing this
shutil.rmtree(d, ignore_errors=True)
if d.exists():
continue # couldn't remove — don't claim the bytes back
total -= size
with _source_lock:
for fn, (cached_dir, _m, _s) in list(_source_cache.items()):
if cached_dir == d:
_source_cache.pop(fn, None)
log.info("sloppak: evicted %s from the unpack cache (%.0f MB)",
d.name, size / 1e6)
except OSError:
log.warning("sloppak: unpack-cache eviction failed", exc_info=True)
def _unpack_lock_for(filename: str) -> threading.Lock:
"""Return a stable per-file lock so concurrent unpacks of the same sloppak
@@ -145,10 +362,17 @@ def resolve_source_dir(
re-unpacks if mtime/size changed, then returns that dir.
Caches the resolution so subsequent calls are ~free.
NOTE: this writes the WHOLE pack every stem to disk. Only call it for a
song you are about to play. To read a *part* of a song (an arrangement, the
lyrics, a tone blob), use read_member_bytes(): unpacking a pack to read a few
KB of JSON is ~45x write amplification, and doing it in a loop over the
library fills the disk with a decompressed copy of every song.
"""
path = dlc_root / filename
stat = path.stat()
mtime, size = stat.st_mtime, stat.st_size
guarded: Path | None = None # a dir WE unpacked, shielded from eviction
with _source_lock:
cached = _source_cache.get(filename)
@@ -159,42 +383,76 @@ def resolve_source_dir(
and cached_size == size
and cached_dir.exists()
):
# Mark it recently-used before returning — see _touch().
if cached_dir != path:
_touch(cached_dir)
return cached_dir
if path.is_dir():
resolved = path
else:
# Zip form — unpack to the cache. Serialize per-file (so concurrent
# callers don't rmtree + re-extract the same dest at once) and cap
# global unpack concurrency (so a burst can't saturate disk/CPU).
dest = unpack_cache_root / _safe_id(filename)
with _unpack_lock_for(filename):
# Re-check the cache inside the per-file lock — a prior holder may
# have just finished unpacking this exact (mtime, size).
with _source_lock:
cached = _source_cache.get(filename)
if (
cached
and cached[1] == mtime
and cached[2] == size
and cached[0].exists()
):
resolved = cached[0]
else:
with _unpack_semaphore:
_unpack_zip(path, dest)
resolved = dest
try:
if path.is_dir():
resolved = path
else:
# Zip form — unpack to the cache. Serialize per-file (so concurrent
# callers don't rmtree + re-extract the same dest at once) and cap
# global unpack concurrency (so a burst can't saturate disk/CPU).
dest = unpack_cache_root / _safe_id(filename)
with _unpack_lock_for(filename):
# Re-check the cache inside the per-file lock — a prior holder may
# have just finished unpacking this exact (mtime, size).
with _source_lock:
cached = _source_cache.get(filename)
if (
cached
and cached[1] == mtime
and cached[2] == size
and cached[0].exists()
):
resolved = cached[0]
else:
# Shield `dest` from eviction from the moment we start writing
# until it is safely in _source_cache. `keep` only shields it
# from OUR OWN sweep — a concurrent resolver sweeping with a
# different `keep` would delete it, and we would then cache and
# return a path that no longer exists. The `finally` below
# releases it on EVERY exit, including a failed unpack: leaving
# a dest marked in-flight would make it un-evictable forever.
with _unpacking_guard:
_unpacking.add(dest)
guarded = dest
with _unpack_semaphore:
_unpack_zip(path, dest)
resolved = dest
# The only moment this cache grows. Sweep here rather than on a
# timer so it can never drift far past the cap.
_evict_unpack_cache(unpack_cache_root, keep=dest)
with _source_lock:
_source_cache[filename] = (resolved, mtime, size)
return resolved
with _source_lock:
_source_cache[filename] = (resolved, mtime, size)
return resolved
finally:
if guarded is not None:
with _unpacking_guard:
_unpacking.discard(guarded)
def get_cached_source_dir(filename: str) -> Path | None:
"""Return the cached source dir for a sloppak if one is known."""
"""Return the cached source dir for a sloppak if one is known AND still there.
The existence check is load-bearing: callers (media.py) only fall back to
resolve_source_dir() when this returns None, so handing back a path that has
been evicted or that the user deleted by hand to reclaim disk would 404
every stem for the rest of the process instead of re-unpacking.
"""
with _source_lock:
cached = _source_cache.get(filename)
return cached[0] if cached else None
if not cached:
return None
src = cached[0]
if not src.is_dir():
_source_cache.pop(filename, None)
return None
_touch(src)
return src
# ── Manifest + song loading ───────────────────────────────────────────────────
@@ -233,6 +491,82 @@ def load_manifest(path: Path) -> dict:
return _read_manifest_from_zip(path)
_ZIP_ROOT = Path("/_root").resolve()
def _zip_member_key(name: str) -> str | None:
"""Canonical lookup key for a zip member name, or None if it escapes the root.
Collapses './', 'a/../b' and backslash separators the same normalization
_unpack_zip()/safe_join() apply when extracting. Both the name the caller asks
for AND the names the archive actually stores must go through this, or a pack
that stores './arrangements/lead.json' unpacks fine but reads back as missing.
"""
safe = safe_join(_ZIP_ROOT, name or "")
# None → escapes the root; == root → a degenerate name like "." or "a/..".
if safe is None or safe == _ZIP_ROOT:
return None
return safe.relative_to(_ZIP_ROOT).as_posix()
def read_member_bytes(path: Path, rel: str) -> bytes | None:
"""Return the bytes of ONE file inside a sloppak, or None if it isn't there.
For a zipped sloppak this opens that single member instead of unpacking the
archive the same trick read_cover_bytes() uses to keep the library grid
from exploding every pack just to show a cover.
Reach for this whenever you want a *part* of a song (an arrangement's JSON,
the lyrics, a tone blob) rather than a song you're about to play. The
alternative, load_song(), calls resolve_source_dir() and writes the WHOLE
pack every stem into the unpack cache. That is a ~45x write amplification
when all you wanted was a few KB of JSON, and looping the library on it
unpacks the entire library (got-feedBack/feedBack: a tester hit 60 GB that
way). Stems are already-compressed audio, so an unpacked song is ~1.1x its
zip: the cache becomes a second, decompressed copy of everything it touches.
"""
rel = (rel or "").strip()
if not rel:
return None
if path.is_dir():
target = safe_join(path.resolve(), rel)
if target is None or not target.is_file():
return None
try:
return target.read_bytes()
except OSError:
return None
# Zip form — read just that member, no unpack. Zip-slip is rejected before we
# open anything, and both sides of the comparison are normalized, so a
# non-canonical-but-valid name ('./arrangements/lead.json') resolves the same
# way it did when we unpacked first.
member = _zip_member_key(rel)
if member is None:
log.warning("sloppak: rejected unsafe member name %r in %r", rel, path)
return None
try:
with zipfile.ZipFile(str(path), "r") as zf:
# Match on the NORMALIZED stored name, and take the LAST match — the
# archive may store './x' or a backslash path (Windows tooling), and
# if it stores two names that normalize to the same file, _unpack_zip
# writes them in order so the last one wins. Reading the raw member by
# exact name would miss the first case and return the wrong bytes in
# the second. A pack has a handful of members; the scan is free.
info = None
for cand in zf.infolist():
if _zip_member_key(cand.filename) == member:
info = cand
if info is None or info.is_dir():
return None
with zf.open(info) as f:
return f.read()
except (zipfile.BadZipFile, OSError, RuntimeError) as e:
log.warning("sloppak: failed to read %r from %s: %s", rel, path.name, e)
return None
_COVER_MEDIA_TYPES = {
".jpg": "image/jpeg", ".jpeg": "image/jpeg",
".png": "image/png", ".webp": "image/webp",
@@ -367,14 +701,21 @@ class LoadedSloppak:
# song.arrangements (not to manifest["arrangements"]) — skipped entries are
# absent so indexing by song.arrangements index is safe.
arrangement_ids: list[str | None] = field(default_factory=list)
# Manifest-relative path to the single full-mix audio file, taken from the
# manifest `original_audio:` key (e.g. "original/full.ogg"). This is the
# pre-separation mixdown that exists alongside the per-instrument `stems`.
# None when the key is absent, points outside source_dir, or the file is
# missing on disk. Served to the front-end via the highway WS as
# `original_audio_url`; the stems plugin uses it to play the untouched mix
# when every stem slider is at unity (and the separate stems otherwise).
original_audio: str | None = None
# Manifest-relative path to the pack's complete mixdown — the whole song in
# one file, as heard before source separation. This is the RESERVED `full`
# stem (spec §5.3), lifted out of `stems` above precisely because it is NOT
# an instrument layer: summing it with the per-instrument stems it was split
# into would double the entire song. See partition_stems().
#
# None when the pack has no mixdown to offer *separately* from its stems —
# which includes the common single-mix pack, whose only stem IS the mixdown
# (there is nothing to be pristine against, so it stays in `stems`).
#
# Served to the front-end via the highway WS as `full_mix_url`; the stems
# plugin plays it while every stem slider sits at unity and crosses to the
# separated stems the moment one drops below 100% — demucs recombination is
# lossy, so the mixdown is strictly the better audio when nothing is muted.
full_mix: str | None = None
def load_song(
@@ -766,6 +1107,13 @@ def load_song(
default_on = bool(default_val)
stems.append({"id": sid, "file": sfile, "default": default_on})
# The complete mixdown is a stem (spec §5.3), but it is not a *layer*: lift
# it out so that no consumer of `stems` — the mixer, the library's stem
# chips, the WS payload — sums it with, or lists it beside, the instruments
# it was separated into. `full_mix_stem` is None for a single-mix pack,
# whose only stem IS the mixdown and stays in the list.
full_mix_stem, stems = partition_stems(stems)
# Optional keys.json — song-level, instrument-independent key/scale track
# (manifest `keys:` key, spec §7.7). Permissive like the other side-files:
# missing / unreadable / malformed -> None, never fatal. Stored as a
@@ -828,28 +1176,22 @@ def load_song(
}
_fpv = manifest.get("feedpak_version")
# Optional full-mix audio — manifest `original_audio:` key. The single
# pre-separation mixdown that ships alongside the per-instrument stems.
# Same permissive, path-traversal-guarded posture as drum_tab above: a
# missing/escaping/absent file simply leaves the full mix unavailable (the
# player falls back to the separate stems) rather than aborting the load.
# We store the manifest-relative string so server.py can build its URL the
# same way it builds stem URLs (via the /api/sloppak/.../file/ endpoint).
original_audio_data: str | None = None
original_audio_rel = manifest.get("original_audio")
if isinstance(original_audio_rel, str) and original_audio_rel.strip():
rel = original_audio_rel.strip()
try:
oa_path = (source_dir / rel).resolve()
oa_path.relative_to(source_dir.resolve())
except ValueError:
log.warning("sloppak: original_audio path %r escapes source_dir — skipped", rel)
oa_path = None
except OSError as e:
log.warning("sloppak: original_audio path resolution failed (%s) — skipped", e)
oa_path = None
if oa_path is not None and oa_path.is_file():
original_audio_data = rel
# The pack's full mix. Normally the RESERVED `full` stem partitioned out
# above (spec §5.3) — no path work needed, it was validated with the other
# stems and its URL is built the same way. Only when the pack has no `full`
# stem do we fall back to the DEPRECATED `original_audio:` key, which is the
# shape every pack written before feedpak 1.15.0 uses.
if full_mix_stem is not None:
full_mix_data: str | None = full_mix_stem["file"]
elif find_full_mix(stems) is not None:
# Single-mix pack: its ONE stem is the mixdown, so there is no mixdown to
# offer *apart from* the stems. Never fall through to the legacy key here
# — a pack that both carries a `full` stem and names the old key would
# otherwise surface the mixdown twice (once as the stem the player is
# already playing, once as a "pristine" track to cross to).
full_mix_data = None
else:
full_mix_data = _legacy_full_mix(manifest, source_dir)
return LoadedSloppak(
song=song,
@@ -864,7 +1206,7 @@ def load_song(
keys=keys_data,
notation_by_id=notation_by_id_data,
arrangement_ids=arrangement_ids_acc,
original_audio=original_audio_data,
full_mix=full_mix_data,
)
@@ -909,7 +1251,7 @@ def extract_meta(path: Path) -> dict:
tuning_offsets = _tuning_for_meta(arr_list)
stems_list = manifest.get("stems", []) or []
stem_ids: list[str] = []
valid_stems: list[dict] = []
for s in stems_list:
if not isinstance(s, dict):
continue
@@ -923,7 +1265,13 @@ def extract_meta(path: Path) -> dict:
isinstance(sid, str) and sid
and isinstance(sfile, str) and sfile
):
stem_ids.append(sid)
valid_stems.append({"id": sid, "file": sfile})
# Partition exactly as load_song() does, for the same reason the library
# filter must not lie: `full` is the mixdown, not an instrument (spec §5.3).
# A separated pack that retains it would otherwise offer the user a "full"
# stem chip alongside guitar/bass/drums and count it as a seventh stem.
_full, instrument_stems = partition_stems(valid_stems)
stem_ids = [s["id"] for s in instrument_stems]
stem_count = len(stem_ids)
return {
+156 -1
View File
@@ -99,7 +99,8 @@
.pp-inst-plus { color: #6b7280; }
/* Leather covers per-instrument hue, embossed with layered shadows and a
subtle grain gradient (no image assets). */
subtle grain gradient (no image assets). Keep the hex pairs in sync with
PP_LEATHER_HEX in screen.js (the canvas card draws the same leather). */
.pp-leather-guitar { background: linear-gradient(160deg, #5c2321, #401412); }
.pp-leather-bass { background: linear-gradient(160deg, #1f3252, #131f36); }
.pp-leather-keys { background: linear-gradient(160deg, #1e4034, #122a21); }
@@ -559,3 +560,157 @@
/* The hover glint is motion theatrics too — not just the JS tilt. */
.pp-tilt::after { display: none; }
}
/* Practice invitations — closest stamps + bring-these-up */
.pp-closest {
border: 1px solid rgba(75, 85, 99, 0.45);
border-radius: 0.6rem;
background: linear-gradient(165deg, rgba(45, 55, 72, 0.4), rgba(31, 41, 55, 0.4));
padding: 0.6rem 0.75rem;
display: flex;
flex-direction: column;
gap: 0.3rem;
}
.pp-closest-head {
font-size: 0.62rem;
letter-spacing: 0.22em;
text-transform: uppercase;
color: #9ca3af;
}
.pp-closest-row {
display: flex;
align-items: baseline;
gap: 0.75rem;
text-align: left;
font-size: 0.8rem;
padding: 0.15rem 0.25rem;
border-radius: 0.35rem;
}
.pp-closest-row:hover { background: rgba(55, 65, 81, 0.5); }
.pp-closest-genre { color: #e5e7eb; font-weight: 600; white-space: nowrap; }
.pp-closest-ask { color: #9ca3af; font-size: 0.72rem; }
.pp-closest-ask em { color: #cbd5e1; font-style: italic; }
.pp-nearest { margin-top: 0.6rem; border-top: 1px dashed rgba(138, 122, 94, 0.4); padding-top: 0.5rem; }
.pp-nearest-head {
font-size: 0.58rem;
letter-spacing: 0.22em;
text-transform: uppercase;
color: #8a7a5e;
margin-bottom: 0.25rem;
}
.pp-nearest-row { font-size: 0.7rem; color: #6d5d40; padding: 0.1rem 0; }
.pp-nearest-row em { color: #3f3428; }
/* ── Career surfaces outside the plugin: profile wall + home card ───────── */
.pp-wall { display: flex; flex-direction: column; gap: 0.6rem; }
.pp-wall-head {
display: flex;
align-items: baseline;
justify-content: space-between;
font-weight: 600;
color: #e5e7eb;
font-size: 0.9rem;
}
.pp-wall-meta { color: #9ca3af; font-size: 0.7rem; font-weight: 400; }
.pp-wall-shelf {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 0.5rem;
padding: 0.35rem 0;
border-bottom: 1px solid rgba(75, 85, 99, 0.25);
}
.pp-wall-inst {
font-size: 0.62rem;
letter-spacing: 0.18em;
text-transform: uppercase;
color: #6b7280;
min-width: 3.6rem;
}
.pp-wall-cover {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: 0.1rem;
width: 4.2rem;
height: 5.6rem;
border-radius: 0.3rem 0.45rem 0.45rem 0.3rem;
box-shadow: inset 0 0 0 1px rgba(255, 255, 255, 0.07),
inset 0.25rem 0 0.4rem -0.25rem rgba(0, 0, 0, 0.8),
0 3px 8px rgba(0, 0, 0, 0.4);
padding: 0.3rem;
transition: transform 0.15s ease;
}
.pp-wall-cover:hover { transform: translateY(-3px); }
.pp-wall-cover span {
font-size: 0.5rem;
font-weight: 700;
letter-spacing: 0.1em;
color: rgba(240, 226, 195, 0.9);
overflow-wrap: anywhere;
text-align: center;
}
.pp-wall-cover em {
font-size: 0.42rem;
letter-spacing: 0.22em;
font-style: normal;
color: #d9a253;
}
.pp-wall-none { font-size: 0.7rem; color: #6b7280; font-style: italic; }
.pp-wall-link {
align-self: flex-end;
font-size: 0.72rem;
color: #22d3ee;
padding: 0.15rem 0.3rem;
}
.pp-wall-link:hover { text-decoration: underline; }
/* The home-page career card — a trading card among stat tiles. */
.pp-dash-card {
position: relative;
overflow: hidden;
width: 100%;
height: 100%;
display: flex;
flex-direction: column;
justify-content: center;
gap: 0.2rem;
text-align: left;
padding: 1rem;
border-radius: 0.5rem;
border: 1px solid rgba(217, 162, 83, 0.35);
background:
linear-gradient(135deg, rgba(92, 35, 33, 0.85), rgba(30, 27, 34, 0.92)),
linear-gradient(160deg, #2b1414, #17111c);
box-shadow: inset 0 0 0 1px rgba(255, 255, 255, 0.05), 0 4px 14px rgba(0, 0, 0, 0.35);
transition: transform 0.15s ease, box-shadow 0.15s ease;
}
.pp-dash-card:hover { transform: translateY(-2px); box-shadow: 0 8px 20px rgba(0, 0, 0, 0.5); }
.pp-dash-shine {
position: absolute;
inset: 0;
background: linear-gradient(105deg, transparent 42%, rgba(255, 223, 128, 0.18) 50%, transparent 58%);
transform: translateX(-130%);
pointer-events: none;
}
.pp-dash-card:hover .pp-dash-shine { animation: pp-foil 1.4s ease-out; }
.pp-dash-head {
font-size: 0.58rem;
letter-spacing: 0.3em;
text-transform: uppercase;
color: #d9a253;
}
.pp-dash-badges { color: #f3ead2; font-size: 1.05rem; }
.pp-dash-badges b { font-weight: 700; margin: 0 0.25rem 0 0.35rem; }
.pp-dash-meta { color: #b5a488; font-size: 0.72rem; }
.pp-dash-ask { color: #8d9aa8; font-size: 0.66rem; }
.pp-dash-ask em { color: #cbd5e1; }
.pp-card-actions { display: flex; gap: 0.5rem; margin-top: 0.9rem; }
@media (prefers-reduced-motion: reduce) {
.pp-dash-card:hover .pp-dash-shine { animation: none; }
.pp-wall-cover, .pp-dash-card { transition: none; }
}
+13
View File
@@ -3,6 +3,19 @@
"songs": 5,
"min_stars": 2
},
"gig": {
"min_songs": 3,
"max_songs": 5,
"stakes_songs": 2,
"encore_accuracy": 0.75
},
"families": [
{ "key": "metal", "match": ["metal", "djent", "grindcore", "thrash", "doom"] },
{ "key": "blues", "match": ["blues"] },
{ "key": "jazz", "match": ["jazz", "bebop", "swing", "bossa"] },
{ "key": "funk", "match": ["funk", "disco"] },
{ "key": "rock", "match": ["rock", "punk", "grunge", "shoegaze"] }
],
"genres": {
"blues": { "virtuoso_nodes": { "guitar": ["blues_shuffle"] } },
"rock": { "virtuoso_nodes": { "guitar": ["rock_power_backbeat"] } },
+228 -5
View File
@@ -26,11 +26,14 @@ Endpoints (all under /api/plugins/career/):
POST /passports/commit commit to an instrument (the wax seal, Stage 0)
POST /passports/open open a genre passport for an instrument
POST /drill-state relayed virtuoso.progress snapshot (drill intake)
POST /gigs/propose build a playable setlist for a genre gig
POST /gigs log a COMPLETED gig (abandoned sets never log)
"""
import hashlib
import json
import logging
import random
import re
import shutil
import tempfile
@@ -114,10 +117,9 @@ def _stars():
detail = []
for filename, acc, title, artist in rows:
acc = acc or 0.0
stars = sum(1 for t in thresholds if acc >= t)
stars, next_at = _star_progress(acc, thresholds)
if stars:
per_song[filename] = stars
next_at = next((t for t in thresholds if acc < t), None)
detail.append({
"filename": filename,
"title": title or filename,
@@ -249,10 +251,18 @@ def _played_by_instrument_genre():
for stub in stubs.values():
acc = stub["best_accuracy"]
stub["best_accuracy"] = round(acc, 4)
stub["stars"] = sum(1 for t in thresholds if acc >= t)
stub["stars"], stub["next_star_at"] = _star_progress(acc, thresholds)
return out, seconds
def _star_progress(acc, thresholds):
"""(stars, next_star_at) — the one place the ascending-thresholds
assumption lives; _stars() and the passport stubs both use it."""
stars = sum(1 for t in thresholds if acc >= t)
next_at = next((t for t in thresholds if acc < t), None)
return stars, next_at
def _library_genres():
"""Distinct effective genres across the live library (the brochure rack)."""
db = _state["meta_db"]
@@ -276,12 +286,33 @@ def _library_genres():
key=lambda r: (-r["songs_in_library"], r["genre_key"]))
def _genre_family(gkey):
"""First family whose keyword appears in the genre key (substring — MB's
vocabulary is open: 'metalcore' must hit the 'metal' family without an
exact alias). List order decides ambiguity: families are checked top to
bottom, so 'blues rock' lands on whichever of blues/rock is listed first."""
for fam in _state["passports_content"].get("families") or []:
if not isinstance(fam, dict):
continue
for kw in fam.get("match") or []:
if isinstance(kw, str) and kw and kw in gkey:
return fam.get("key")
return None
def _badge_requirement(gkey, instrument="guitar"):
cfg = _state["passports_content"]
req = dict(cfg.get("badge_requirement") or {})
req.setdefault("songs", 5)
req.setdefault("min_stars", 2)
override = (cfg.get("genres") or {}).get(gkey)
# Exact per-genre override wins; otherwise the genre inherits its FAMILY's
# requirement — so 'death metal' / 'metalcore' passports carry the metal
# drill without curating every MB sub-genre by hand.
genres_cfg = cfg.get("genres") or {}
override = genres_cfg.get(gkey)
if not isinstance(override, dict):
family = _genre_family(gkey)
override = genres_cfg.get(family) if family else None
if isinstance(override, dict):
req.update(override)
# virtuoso_nodes: {instrument: [node_ids]} — a passport only carries its
@@ -355,6 +386,7 @@ def _passports_view():
cfg = _state["passports_content"]
graded = set(cfg.get("graded_instruments") or [])
st = _career_state()
all_gigs = st.get("gigs") if isinstance(st.get("gigs"), list) else []
played, played_seconds = _played_by_instrument_genre()
received_at, by_node = _drill_by_node()
instruments = {}
@@ -385,6 +417,17 @@ def _passports_view():
badge = "earned"
else:
badge = "in_progress"
# Practice invitation: the non-qualifying songs closest to the
# QUALIFYING bar (the badge ask), nearest first — invitation
# data, the UI voices it without meters.
thresholds = _state["content"]["star_accuracy_thresholds"]
bar = (thresholds[req["min_stars"] - 1]
if 0 < req["min_stars"] <= len(thresholds) else None)
nearest = [] if bar is None else sorted(
(s for s in songs if not s["qualifies"]),
key=lambda s: bar - s["best_accuracy"])[:3]
for s in nearest:
s["bar_at"] = bar
passports.append({
"genre_key": gkey,
"genre": meta.get("genre") or gkey,
@@ -393,13 +436,18 @@ def _passports_view():
"graded": is_graded,
"songs": songs,
"qualifying_count": qualifying,
"nearest": nearest,
# Honest hours odometer (Stage 5 post-cap): a true fact that
# only grows — never a target, never a meter.
"seconds_total": round(played_seconds.get((inst, gkey), 0.0), 1),
"drills": {"required": required, "cleared": cleared},
"badge": badge,
})
instruments[inst] = {"committed_at": committed_at, "passports": passports}
inst_gigs = [g for g in all_gigs if g.get("instrument") == inst]
for p in passports:
p["gigs"] = [g for g in inst_gigs if g.get("genre_key") == p["genre_key"]][-20:][::-1]
instruments[inst] = {"committed_at": committed_at, "passports": passports,
"gig_count": len(inst_gigs)}
return {
"config": {
"badge_requirement": cfg.get("badge_requirement") or {},
@@ -414,6 +462,60 @@ def _passports_view():
}
def _gig_config():
cfg = _state["passports_content"].get("gig")
cfg = cfg if isinstance(cfg, dict) else {}
def _num(key, default, cast):
# Tuning data, not code: junk falls back instead of 500ing both gig
# endpoints, and a legitimate 0 (stakes_songs: 0) is respected.
val = cfg.get(key)
if isinstance(val, bool) or not isinstance(val, (int, float)):
return default
return cast(val)
return {
"min_songs": max(1, _num("min_songs", 3, int)),
"max_songs": max(1, _num("max_songs", 5, int)),
"stakes_songs": max(0, _num("stakes_songs", 2, int)),
"encore_accuracy": _num("encore_accuracy", 0.75, float),
}
def _current_venue():
"""Highest unlocked venue (the room you can book today)."""
stars_total, _, _ = _stars()
best = None
for v in _state["content"]["venues"]:
if stars_total >= v["star_threshold"]:
if best is None or v["star_threshold"] >= best["star_threshold"]:
best = v
return best
def _unplayed_genre_songs(gkey, exclude, limit):
"""Library songs of a genre with no stats yet — a young passport's gig
still gets a full set (playing them is how stubs start).
ponytail: full stat-less scan + python-side genre match (a few ms at 7k
songs, single-user); push the match into SQL if propose ever feels slow."""
db = _state["meta_db"]
if db is None:
return []
rows = db.conn.execute(
f"SELECT filename, title, artist, {_genre_expr(db)} AS g FROM songs "
"WHERE filename NOT IN (SELECT filename FROM song_stats)"
).fetchall()
out = []
for filename, title, artist, genre in rows:
if _genre_key(genre) != gkey or filename in exclude:
continue
out.append({"filename": filename, "title": title or filename,
"artist": artist or ""})
if len(out) >= limit:
break
return out
def _validate_pack_dir(pack_dir: Path):
"""Raise ValueError unless pack_dir holds a complete venue pack."""
manifest_path = pack_dir / "manifest.json"
@@ -595,6 +697,127 @@ def setup(app, context):
"snapshot": snapshot})
return {"ok": True}
@app.post(f"/api/plugins/{PLUGIN_ID}/gigs/propose")
def propose_gig(body: dict = Body(...)):
inst = str((body or {}).get("instrument") or "")
genre = _genre_display((body or {}).get("genre"))
gkey = genre.lower()
if inst not in (_state["passports_content"].get("instruments") or []):
raise HTTPException(400, "Unknown instrument.")
if not gkey or len(genre) > GENRE_MAX_LEN:
raise HTTPException(400, "Provide a genre.")
cfg = _gig_config()
try:
size = int((body or {}).get("size") or 4)
except (TypeError, ValueError):
raise HTTPException(400, "size must be a number.")
size = max(cfg["min_songs"], min(cfg["max_songs"], size))
played, _seconds = _played_by_instrument_genre()
stubs = list(played.get((inst, gkey), {}).values())
req = _badge_requirement(gkey, inst)
qualifying = [s for s in stubs if s["stars"] >= req["min_stars"]]
rest = [s for s in stubs if s["stars"] < req["min_stars"]]
# The set: mostly songs you own, plus a couple of stakes songs near
# the bar; a young passport fills from unplayed genre songs so the
# first gig is how stubs start. random per call = free re-roll.
random.shuffle(qualifying)
rest.sort(key=lambda s: -s["best_accuracy"])
qtaken = max(1, size - cfg["stakes_songs"])
picks = qualifying[:qtaken]
for s in rest:
if len(picks) >= size:
break
picks.append(s)
# Surplus qualifying songs backfill a short set — a mature passport
# with no near-bar songs left must still fill the bill. Offset by how
# many QUALIFYING songs were taken, not len(picks): rest's stakes
# additions would otherwise skip eligible qualifying songs entirely.
for s in qualifying[qtaken:]:
if len(picks) >= size:
break
picks.append(s)
if len(picks) < size:
exclude = {s["filename"] for s in picks}
picks.extend(_unplayed_genre_songs(gkey, exclude, size - len(picks)))
if not picks:
raise HTTPException(404, "No songs of this genre in the library.")
venue = _current_venue()
return {
"instrument": inst,
"genre": genre,
"genre_key": gkey,
"venue_id": venue["id"] if venue else None,
"venue_name": venue["name"] if venue else "",
"songs": [{"filename": s["filename"], "title": s.get("title") or s["filename"],
"artist": s.get("artist") or ""} for s in picks[:size]],
}
@app.post(f"/api/plugins/{PLUGIN_ID}/gigs")
def log_gig(body: dict = Body(...)):
# Called by the runner ONLY when the set completed — an abandoned set
# never logs (no fail state; the gig you finished is the gig you
# played). Accuracies come from song_stats, freshly written by the
# set's own plays.
inst = str((body or {}).get("instrument") or "")
genre = _genre_display((body or {}).get("genre"))
gkey = genre.lower()
venue_id = str((body or {}).get("venue_id") or "")
songs = (body or {}).get("songs")
if inst not in (_state["passports_content"].get("instruments") or []):
raise HTTPException(400, "Unknown instrument.")
if not gkey or len(genre) > GENRE_MAX_LEN:
raise HTTPException(400, "Provide a genre.")
if venue_id and (not VENUE_ID_RE.fullmatch(venue_id) or _venue(venue_id) is None):
raise HTTPException(400, "Unknown venue.")
if (not isinstance(songs, list) or not songs or len(songs) > 8
or not all(isinstance(f, str) and f.strip() for f in songs)):
raise HTTPException(400, "songs must be 1-8 filenames.")
db = _state["meta_db"]
entries = []
accuracies = []
for filename in songs:
title = filename
accuracy = None
if db is not None:
# The NEWEST row is the set's own just-recorded play — a
# MAX(last_accuracy) across arrangements would happily log a
# stale higher score from another instrument's old session.
row = db.conn.execute(
"SELECT last_accuracy FROM song_stats WHERE filename = ? "
"ORDER BY last_played_at DESC LIMIT 1",
(filename,)).fetchone()
if row and row[0] is not None:
accuracy = round(float(row[0]), 4)
accuracies.append(accuracy)
trow = db.conn.execute(
"SELECT title FROM songs WHERE filename = ?", (filename,)).fetchone()
if trow and trow[0]:
title = trow[0]
entries.append({"filename": filename, "title": title, "accuracy": accuracy})
# Encore needs the WHOLE set scored at the bar — one scored song must
# not earn an encore for a set that was 4/5 unheard.
encore = (len(accuracies) == len(songs) and
sum(accuracies) / len(accuracies) >= _gig_config()["encore_accuracy"])
gig = {
"at": _now_iso(),
"venue_id": venue_id or None,
"instrument": inst,
"genre": genre,
"genre_key": gkey,
"songs": entries,
"encore": encore,
}
with _lock:
st = _career_state()
if not isinstance(st.get("gigs"), list):
st["gigs"] = []
st["gigs"].append(gig)
# ponytail: hard cap — nothing reads past the last 20 per
# passport; the state file must not grow (and export) forever.
st["gigs"] = st["gigs"][-500:]
_save_json(_state_file(), st)
return {"ok": True, "gig": gig}
@app.post(f"/api/plugins/{PLUGIN_ID}/packs/{{venue_id}}/download")
def start_download(venue_id: str):
venue = _venue(venue_id) if VENUE_ID_RE.fullmatch(venue_id) else None
+1
View File
@@ -29,6 +29,7 @@
<div id="career-tab-passports" class="hidden" role="tabpanel" aria-labelledby="career-tab-btn-passports">
<p class="text-sm text-gray-400 mb-4">Commit to an instrument, pick a genre, and stamp your way to its badge — five ★★ songs mint a Bronze. Your passport wall is who you are as a musician.</p>
<div id="pp-instruments" class="pp-instruments"></div>
<div id="pp-closest" class="mt-4"></div>
<div id="pp-shelf-wrap" class="mt-5">
<div id="pp-shelf" class="pp-shelf"></div>
</div>
+295 -16
View File
@@ -458,6 +458,8 @@
_pp = view;
detectNewBadges(view);
renderPassports();
renderProfileWall();
renderDashCard();
if (!_ppBootstrapped) {
_ppBootstrapped = true;
// Sync the local drill snapshot once per session — drill progress
@@ -497,6 +499,56 @@
</button>`;
}
// Practice invitations: which stamps are closest, and what would bring
// them home. Invitations only — no meters, no obligations.
// Floor, never round: 74.9% must not display as the already-met "75%".
function pct(frac) { return Math.floor((Number(frac) || 0) * 100); }
function ppNeed(p) {
return Math.max(0, ((p.requirement || {}).songs || 0) - (p.qualifying_count || 0));
}
// The one blocker phrase — shared by the Closest-stamps strip and the
// passport book's invite line so they can never contradict each other.
function ppAskHTML(p, withHint) {
const req = p.requirement || {};
const need = ppNeed(p);
const starGl = '★'.repeat(req.min_stars || 0);
if (need > 0) {
const near = withHint ? (p.nearest || [])[0] : null;
const hint = near
? ` · nearest: <em>${esc(near.title)}</em> at ${pct(near.best_accuracy)}%`
: '';
return `${need === 1 ? `one more ${starGl} song` : `${need} more ${starGl} songs`}${hint}`;
}
const labels = ((_pp && _pp.config) || {}).drill_labels || {};
const drills = p.drills || {};
const pending = (drills.required || []).filter((n) => !(drills.cleared || []).includes(n));
return `clear ${pending.map((n) => esc(labels[n] || n)).join(', ') || 'the genre drill'} in Virtuoso`;
}
function closestLineHTML(p) {
return `<button class="pp-closest-row" data-pp-open="${esc(p.genre_key)}">
<span class="pp-closest-genre">${esc(p.genre)}</span>
<span class="pp-closest-ask">${ppAskHTML(p, true)}</span>
</button>`;
}
function renderClosest(inst, data) {
const host = $('pp-closest');
if (!host) return;
const candidates = (data.passports || [])
.filter((p) => p.badge === 'in_progress')
.sort((a, b) => ppNeed(a) - ppNeed(b))
.slice(0, 3);
if (!candidates.length) { host.innerHTML = ''; return; }
host.innerHTML = `<div class="pp-closest">
<div class="pp-closest-head">Closest stamps</div>
${candidates.map(closestLineHTML).join('')}
</div>`;
}
function renderShelf(inst, data) {
const shelf = $('pp-shelf');
if (!shelf) return;
@@ -551,6 +603,7 @@
${esc(ppLabel(i))}${earned ? ` <span class="pp-inst-badges">⚡${earned}</span>` : ''}${committed ? '' : ' <span class="pp-inst-plus">+</span>'}
</button>`;
}).join('');
renderClosest(inst, data);
renderShelf(inst, data);
renderRack(inst, data);
}
@@ -576,22 +629,13 @@
function ppBookHTML(inst, p, pendingSlam) {
const req = p.requirement || {};
const need = Math.max(0, (req.songs || 0) - p.qualifying_count);
const starGl = '★'.repeat(req.min_stars || 0);
const reqNodes = (p.drills || {}).required || [];
const clearedNodes = new Set((p.drills || {}).cleared || []);
const labels = ((_pp && _pp.config) || {}).drill_labels || {};
const pendingDrills = reqNodes.filter((n) => !clearedNodes.has(n));
// The invite names what actually blocks the stamp: songs first, then
// the genre drill once the song bar is met.
let invite;
if (need > 0) {
invite = need === 1 ? `One more ${starGl} song mints this stamp.`
: `${need} more ${starGl} songs mint this stamp.`;
} else {
const names = pendingDrills.map((n) => labels[n] || n).join(', ');
invite = `Clear ${names || 'the genre drill'} in Virtuoso to mint this stamp.`;
}
// The invite names what actually blocks the stamp — same shared
// phrase as the Closest-stamps strip, so they can't contradict.
const invite = `${ppAskHTML(p, false)} mints this stamp.`;
let badgeArea = '';
if (p.badge === 'shown_not_judged') {
badgeArea = `<div class="pp-snj">Shown, not judged — your ${esc(ppLabel(inst).toLowerCase())} repertoire speaks for itself.</div>`;
@@ -601,14 +645,18 @@
<span class="pp-stamp-tier">BRONZE</span>
</div>
<div class="pp-gold-foil" aria-hidden="true">GOLD</div>
<div class="pp-gold-note">Gold rung coming improvise it, verified.</div>`;
<div class="pp-gold-note">Gold rung coming improvise it, verified.</div>
<div class="pp-card-actions">
<button class="career-btn career-btn-ghost" data-pp-card="save">Save card</button>
<button class="career-btn career-btn-ghost" data-pp-card="copy">Copy card</button>
</div>`;
} else {
const fill = (ppFillFraction(p) * 100).toFixed(0);
badgeArea = `<div class="pp-stamp pp-stamp-page pp-stamp-ghost" style="--pp-rot:${ppJitter(p.genre_key, 7).toFixed(1)}deg; --pp-fill:${fill}%">
<span class="pp-stamp-genre">${esc(p.genre.toUpperCase())}</span>
<span class="pp-stamp-tier">BRONZE</span>
</div>
<div class="pp-invite">${esc(invite)}</div>`;
<div class="pp-invite">${invite.charAt(0).toUpperCase()}${invite.slice(1)}</div>`;
}
const hours = fmtHours(p.seconds_total);
const odometer = hours
@@ -628,6 +676,16 @@
: `Play ${esc(p.genre)} songs at ${starGl} to collect ticket stubs.`;
const stubsHTML = stubs.length ? stubs.map(ppStubHTML).join('')
: `<div class="pp-stub-empty">${emptyLine}</div>`;
// Bring-these-up: nearest-to-the-bar songs (graded, unearned only —
// an earned page is memorabilia, not homework).
let nearest = '';
if (p.badge === 'in_progress' && (p.nearest || []).length) {
nearest = `<div class="pp-nearest">
<div class="pp-nearest-head">Bring these up</div>
${p.nearest.map((s) =>
`<div class="pp-nearest-row"><em>${esc(s.title)}</em> — best ${pct(s.best_accuracy)}%, ${starGl} at ${pct(s.bar_at)}%</div>`).join('')}
</div>`;
}
return `<div class="pp-book-wrap" data-pp-close-bg="1" role="dialog" aria-modal="true" aria-label="${esc(p.genre)} ${esc(ppLabel(inst))} passport">
<div class="pp-book">
<div class="pp-page pp-page-left">
@@ -636,7 +694,7 @@
</div>
<div class="pp-page pp-page-right">
<div class="pp-page-head">Ticket stubs</div>
<div class="pp-stubs">${stubsHTML}</div>
<div class="pp-stubs">${stubsHTML}${nearest}</div>
</div>
<div class="pp-book-cover pp-leather-${esc(inst)}">
<span class="pp-cover-title">${esc(p.genre.toUpperCase())}</span>
@@ -771,6 +829,217 @@
if (_tiltEl) { resetTilt(_tiltEl); _tiltEl = null; }
}
// ── Shareable passport card (canvas → PNG, save or clipboard) ─────────
// Keep in sync with the .pp-leather-* gradients in assets/career.css —
// canvas can't consume a CSS class, so the pairs live twice on purpose.
const PP_LEATHER_HEX = {
guitar: ['#5c2321', '#401412'],
bass: ['#1f3252', '#131f36'],
keys: ['#1e4034', '#122a21'],
drums: ['#3f3f46', '#26262b'],
};
function drawPassportCard(inst, p) {
const W = 480;
const H = 640;
const canvas = document.createElement('canvas');
canvas.width = W;
canvas.height = H;
const ctx = canvas.getContext('2d');
const [c1, c2] = PP_LEATHER_HEX[inst] || PP_LEATHER_HEX.guitar;
const bg = ctx.createLinearGradient(0, 0, W, H);
bg.addColorStop(0, c1);
bg.addColorStop(1, c2);
ctx.fillStyle = bg;
ctx.fillRect(0, 0, W, H);
// Emboss frame
ctx.strokeStyle = 'rgba(240,226,195,0.35)';
ctx.lineWidth = 3;
ctx.strokeRect(18, 18, W - 36, H - 36);
// Genre title
ctx.fillStyle = 'rgba(240,226,195,0.95)';
ctx.textAlign = 'center';
ctx.font = '700 34px Georgia, serif';
ctx.fillText(p.genre.toUpperCase(), W / 2, 92, W - 80);
ctx.font = '400 15px Georgia, serif';
ctx.fillStyle = 'rgba(240,226,195,0.55)';
ctx.fillText(`${ppLabel(inst).toUpperCase()} PASSPORT`, W / 2, 122);
// Stamp ring
const gold = p.badge === 'gold';
const ink = gold ? '#d9a253' : '#b06a2a';
const cy = 330;
ctx.strokeStyle = ink;
ctx.lineWidth = 6;
ctx.beginPath();
ctx.arc(W / 2, cy, 118, 0, Math.PI * 2);
ctx.stroke();
ctx.lineWidth = 2;
ctx.beginPath();
ctx.arc(W / 2, cy, 106, 0, Math.PI * 2);
ctx.stroke();
ctx.fillStyle = ink;
ctx.font = '800 26px Georgia, serif';
ctx.fillText(p.genre.toUpperCase(), W / 2, cy - 6, 190);
ctx.font = '600 16px Georgia, serif';
ctx.fillText(gold ? 'G O L D' : 'B R O N Z E', W / 2, cy + 28);
// Facts
const stubCount = (p.songs || []).filter((sng) => sng.qualifies).length;
const hours = fmtHours(p.seconds_total);
ctx.fillStyle = 'rgba(240,226,195,0.75)';
ctx.font = '400 17px Georgia, serif';
ctx.fillText(`${stubCount} ticket stub${stubCount === 1 ? '' : 's'}${hours ? ` · ${hours} played` : ''}`, W / 2, 512);
ctx.fillStyle = 'rgba(240,226,195,0.4)';
ctx.font = '400 13px Georgia, serif';
ctx.fillText('fee[dB]ack · career passport', W / 2, H - 44);
return canvas;
}
function exportPassportCard(mode) {
if (!_ppBook || !_pp) return;
const { inst, gkey } = _ppBook;
const p = (((_pp.instruments || {})[inst] || {}).passports || [])
.find((x) => x.genre_key === gkey);
if (!p) return;
const canvas = drawPassportCard(inst, p);
canvas.toBlob(async (blob) => {
const fail = (why) => {
if (window.fbNotify) window.fbNotify.show({ icon: '⚠️', title: 'Card export failed', message: why });
};
if (!blob) { fail('The canvas produced no image.'); return; }
const filename = `passport-${inst}-${gkey.replace(/[^a-z0-9-]+/g, '-')}.png`;
try {
const io = await import('/static/js/blob-io.js');
if (mode === 'copy') {
const ok = await io.copyImageBlob(blob);
if (ok) {
if (window.fbNotify) window.fbNotify.show({ icon: '📋', title: 'Card copied', message: 'Paste it anywhere.' });
return;
}
if (window.fbNotify) window.fbNotify.show({ icon: '💾', title: 'Clipboard unavailable', message: 'Saved the card instead.' });
}
io.downloadBlob(blob, filename);
} catch (e) { fail('Export helper unavailable.'); }
}, 'image/png');
}
// ── Career surfaces outside the plugin screen ─────────────────────────
// Profile passport wall + the home-page career card. Both inject into
// core-owned mounts announced by v3:profile-rendered /
// v3:dashboard-rendered (the achievements seam). Absent-not-empty: with
// no committed instrument they render nothing and the dashboard keeps
// its built-in fallback stat.
function careerTotals() {
if (!_pp) return null;
let badges = 0;
let seconds = 0;
let gigs = 0;
const walls = [];
for (const inst of (_pp.config || {}).instruments || []) {
const d = (_pp.instruments || {})[inst];
// A commitment with no opened passport isn't a wall yet — the
// external surfaces (profile, home card) stay ABSENT until a
// passport exists (absent-not-empty).
if (!d || !d.committed_at || !(d.passports || []).length) continue;
const earned = (d.passports || []).filter((p) => p.badge === 'earned' || p.badge === 'gold');
badges += earned.length;
seconds += (d.passports || []).reduce((t, p) => t + (p.seconds_total || 0), 0);
gigs += d.gig_count || 0;
walls.push({ inst, earned, opened: d.passports.length });
}
if (!walls.length) return null;
return { badges, seconds, gigs, walls };
}
function closestAskHTML() {
if (!_pp) return '';
let best = null;
for (const inst of (_pp.config || {}).instruments || []) {
for (const p of (((_pp.instruments || {})[inst] || {}).passports || [])) {
if (p.badge !== 'in_progress') continue;
const need = Math.max(0, ((p.requirement || {}).songs || 0) - p.qualifying_count);
if (!best || need < best.need) best = { p, need };
}
}
if (!best) return '';
const starGl = '★'.repeat((best.p.requirement || {}).min_stars || 0);
if (best.need > 0) {
return `${esc(best.p.genre)}${best.need === 1 ? `one more ${starGl} song` : `${best.need} more ${starGl} songs`}`;
}
return `${esc(best.p.genre)} — one drill away`;
}
function renderProfileWall() {
const mount = document.getElementById('v3-profile-passports-mount');
if (!mount) return;
const totals = careerTotals();
if (!totals) { mount.innerHTML = ''; return; }
const shelves = totals.walls.map(({ inst, earned, opened }) => {
const covers = earned.map((p) =>
`<button class="pp-wall-cover pp-leather-${esc(inst)}" data-pp-wall-inst="${esc(inst)}" data-pp-wall-gkey="${esc(p.genre_key)}" title="${esc(p.genre)}">
<span>${esc(p.genre.toUpperCase())}</span>
<em>${p.badge === 'gold' ? 'GOLD' : 'BRONZE'}</em>
</button>`).join('');
const line = earned.length
? covers
: `<span class="pp-wall-none">${opened} passport${opened === 1 ? '' : 's'} open — first stamp pending</span>`;
return `<div class="pp-wall-shelf"><span class="pp-wall-inst">${esc(ppLabel(inst))}</span>${line}</div>`;
}).join('');
const hours = fmtHours(totals.seconds);
mount.innerHTML = `<div class="bg-fb-card/80 backdrop-blur rounded-lg p-4 border border-fb-border/50 pp-wall">
<div class="pp-wall-head">
<span>Passport wall</span>
<span class="pp-wall-meta">${totals.badges} badge${totals.badges === 1 ? '' : 's'}${hours ? ` · ${hours} played` : ''}${totals.gigs ? ` · ${totals.gigs} gig${totals.gigs === 1 ? '' : 's'}` : ''}</span>
</div>
${shelves}
<button class="pp-wall-link" data-pp-wall-career="1">Open career </button>
</div>`;
if (!mount.dataset.ppWired) {
mount.dataset.ppWired = '1';
mount.addEventListener('click', onWallClick);
}
}
function onWallClick(e) {
const open = e.target.closest('[data-pp-wall-inst]');
if (open) {
// Two attributes, not a '/'-joined pair: a genre key may itself
// contain '/' ("drum/bass") and must round-trip intact.
const inst = open.dataset.ppWallInst;
const gkey = open.dataset.ppWallGkey;
lsSet(PP_INST_KEY, inst);
if (window.showScreen) window.showScreen('plugin-career');
showCareerTab('passports');
renderPassports();
openBook(inst, gkey);
return;
}
if (e.target.closest('[data-pp-wall-career]')) {
if (window.showScreen) window.showScreen('plugin-career');
showCareerTab('passports');
}
}
function renderDashCard() {
const slot = document.getElementById('v3-dash-career-slot');
if (!slot) return;
const totals = careerTotals();
if (!totals) return; // keep core's fallback stat card
const hours = fmtHours(totals.seconds);
const ask = closestAskHTML();
slot.innerHTML = `<button class="pp-dash-card" data-pp-wall-career="1">
<span class="pp-dash-shine" aria-hidden="true"></span>
<span class="pp-dash-head">Career</span>
<span class="pp-dash-badges">${'⚡'.repeat(Math.min(totals.badges, 5))}<b>${totals.badges}</b> badge${totals.badges === 1 ? '' : 's'}</span>
<span class="pp-dash-meta">${hours ? `${hours} played` : 'the stage is set'}${totals.gigs ? ` · ${totals.gigs} gig${totals.gigs === 1 ? '' : 's'}` : ''}</span>
${ask ? `<span class="pp-dash-ask">closest: ${ask}</span>` : ''}
</button>`;
if (!slot.dataset.ppWired) {
slot.dataset.ppWired = '1';
slot.addEventListener('click', onWallClick);
}
}
function openGenre(inst, genre) {
fetch(`${API}/passports/open`, {
method: 'POST',
@@ -820,6 +1089,11 @@
closeBook();
return;
}
const cardBtn = e.target.closest('[data-pp-card]');
if (cardBtn) {
exportPassportCard(cardBtn.dataset.ppCard);
return;
}
const dlBtn = e.target.closest('[data-career-download]');
const delBtn = e.target.closest('[data-career-delete]');
const playBtn = e.target.closest('[data-career-play]');
@@ -879,6 +1153,10 @@
document.addEventListener('keydown', (e) => {
if (e.key === 'Escape' && _ppBook) closeBook();
});
// Core re-renders profile/dashboard shells (innerHTML wipe) and
// announces the fresh mount points — same seam achievements uses.
document.addEventListener('v3:profile-rendered', renderProfileWall);
document.addEventListener('v3:dashboard-rendered', renderDashCard);
refresh();
}
@@ -886,7 +1164,8 @@
// the badge-diff logic; nothing here touches the DOM.
window.__careerPassportTest = {
ppKey, ppJitter, ppLabel, detectNewBadges, seenBadges, markBadgeSeen,
fmtHours, ppFillFraction,
fmtHours, ppFillFraction, careerTotals, closestAskHTML,
setView(v) { _pp = v; },
};
if (document.readyState === 'loading') {
+27
View File
@@ -150,3 +150,30 @@ test('ppFillFraction: song progress toward the bar, in-progress only', () => {
assert.equal(ppFillFraction(p('in_progress', 3, 0)), 0); // no bar → no fill
assert.equal(ppFillFraction(null), 0);
});
test('careerTotals / wall + dash card stay absent without commitment', () => {
const w = load();
const t = w.__careerPassportTest;
// No _pp at all → null; committed-less view → null (absent-not-empty).
assert.equal(t.careerTotals(), null);
t.setView({ config: { instruments: ['guitar'] },
instruments: { guitar: { committed_at: null, passports: [] } } });
assert.equal(t.careerTotals(), null);
// Committed but zero passports opened: still absent (no zero-wall).
t.setView({ config: { instruments: ['guitar'] },
instruments: { guitar: { committed_at: 'x', passports: [] } } });
assert.equal(t.careerTotals(), null);
// Committed with an earned badge + hours → totals aggregate.
t.setView({ config: { instruments: ['guitar', 'bass'] },
instruments: {
guitar: { committed_at: 'x', passports: [
{ badge: 'earned', seconds_total: 3600, genre: 'Blues', genre_key: 'blues' },
{ badge: 'in_progress', seconds_total: 120, genre: 'Funk', genre_key: 'funk',
qualifying_count: 4, requirement: { songs: 5, min_stars: 2 } }] },
bass: { committed_at: null, passports: [] },
} });
const totals = t.careerTotals();
assert.equal(totals.badges, 1);
assert.equal(totals.seconds, 3720);
assert.equal(totals.walls.length, 1);
});
+14 -10
View File
@@ -1908,17 +1908,21 @@ function createHighway() {
// never routable.
const isAudioUrl = msg.audio_url.startsWith('/audio/');
// "Full mix" covers BOTH single-mix pack shapes:
// - stem-less packs (original_audio: in the manifest,
// audio_url == original_audio_url), and
// - single-stem packs (stems: [full.ogg] only) — the server
// puts the full mix in the stems list, has_original_audio
// is false, and audio_url points at the one stem. With one
// stem there is no per-stem mix to preserve, so routing it
// natively loses nothing. Real multi-stem (>1) stays out
// until Phase 2.
// - single-stem packs (stems: [full.ogg] only) — the pack's
// one stem IS its mixdown, so the server leaves it in the
// stems list, has_full_mix is false, and audio_url points
// at that one stem; and
// - legacy stem-less packs, whose mixdown sits outside stems
// behind the deprecated original_audio: key, so has_stems
// is false and audio_url == full_mix_url.
// Either way there is one audible source and no per-stem mix
// to preserve, so routing it natively loses nothing. A pack
// that retains its `full` stem ALONGSIDE separated stems is
// multi-stem (has_full_mix && has_stems) and stays out until
// Phase 2 — routing it natively would drop the mixer.
const isFeedpakFullMix = !isAudioUrl
&& msg.audio_url.startsWith('/api/sloppak/')
&& ((!!msg.has_original_audio && !msg.has_stems)
&& ((!!msg.has_full_mix && !msg.has_stems)
|| (msg.stems || []).length === 1);
// Record the loaded song's audio so app.js can re-route it
// between the HTML5 and JUCE paths if the audio engine is
@@ -1943,7 +1947,7 @@ function createHighway() {
'isFeedpakFullMix=', isFeedpakFullMix,
'has_stems=', !!msg.has_stems,
'stems=', (msg.stems || []).length,
'has_original_audio=', !!msg.has_original_audio,
'has_full_mix=', !!msg.has_full_mix,
'format=', msg.format,
'alreadyLoaded=', alreadyLoaded,
'juceApi=', !!window.feedBackDesktop?.audio);
+28
View File
@@ -0,0 +1,28 @@
// Blob export helpers — the download idiom that used to be duplicated in
// settings-io.js and diagnostics-export.js, plus image-to-clipboard for
// shareable cards/posters. A LEAF module: imports nothing. Classic-script
// plugins reach it via dynamic import('/static/js/blob-io.js').
export function downloadBlob(blob, filename) {
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = filename;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
}
// Copy an image blob to the system clipboard. Returns true on success, false
// when the Clipboard API is unavailable or refuses (insecure context, no user
// gesture, permission denied) — callers fall back to downloadBlob and say so.
export async function copyImageBlob(blob) {
try {
if (!navigator.clipboard || typeof ClipboardItem === 'undefined') return false;
await navigator.clipboard.write([new ClipboardItem({ [blob.type || 'image/png']: blob })]);
return true;
} catch (_) {
return false;
}
}
+3 -8
View File
@@ -21,6 +21,8 @@
// redact toggles.
// 3. Stream the returned zip to disk.
import { downloadBlob } from './blob-io.js';
function _diagIncludeFromUI() {
const v = (id) => document.getElementById(id)?.checked !== false;
return {
@@ -265,14 +267,7 @@ export async function exportDiagnostics() {
}
try {
const blob = await resp.blob();
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = filename;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
downloadBlob(blob, filename);
status.textContent = `Exported ${filename}`;
} catch (e) {
status.textContent = `Export failed during download: ${e.message}`;
+4 -9
View File
@@ -1,6 +1,6 @@
// Settings backup — the export / import bundle.
//
// Carved verbatim out of static/app.js (R3a). A LEAF module: imports nothing.
// Carved verbatim out of static/app.js (R3a). Imports only the blob-io leaf.
//
// Two entry points, both inline handlers on the Settings screen, so app.js keeps
// re-exposing them on window. The import is two-phase (server first, atomic; then
@@ -29,6 +29,8 @@
// phase 2; the localStorage side is best-effort merge after server
// success. Failures are reported, never silenced.
import { downloadBlob } from './blob-io.js';
export async function exportSettings() {
const status = document.getElementById('backup-status');
status.textContent = 'Exporting...';
@@ -66,14 +68,7 @@ export async function exportSettings() {
if (match) filename = match[1];
}
const blob = new Blob([JSON.stringify(bundle, null, 2)], { type: 'application/json' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = filename;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
downloadBlob(blob, filename);
status.textContent = `Exported ${filename}`;
} catch (e) {
status.textContent = `Export failed: ${e.message}`;
+6 -1
View File
@@ -208,12 +208,17 @@
'</div></div></div>' +
continueCard +
'</div>' +
// Stats row
// Stats row. The third slot belongs to the career plugin (it
// replaces the slot's content on v3:dashboard-rendered); the
// plugin-count stat is the built-in fallback when career is
// absent or has no state yet.
'<div class="grid md:grid-cols-3 gap-6 mt-6">' +
audioRoutingCard() +
statCard(String(songCount), 'songs', 'text-fb-gold') +
'<div id="v3-dash-career-slot" class="grid">' +
statCard(String(pluginCount), 'active', 'text-fb-good') +
'</div>' +
'</div>' +
recentSection +
'</div>';
+4
View File
@@ -191,6 +191,10 @@
'<div class="space-y-6">' +
headerCard +
bestsCard +
// Passport wall — rendered by the career plugin on
// v3:profile-rendered (absent-not-empty: nothing shows until a
// passport exists).
'<div id="v3-profile-passports-mount"></div>' +
// Feats of Power trophy shelf — rendered by the achievements plugin
// (earned Feats only; hidden-until-earned, so empty when none).
'<div id="v3-profile-feats-slot"></div>' +
+4
View File
@@ -101,6 +101,10 @@
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
// A 4xx/5xx JSON error body must read as FAILURE — callers
// re-queue accrued seconds on null, and a parsed error object
// would silently drop them.
if (!r.ok) return null;
try { return await r.json(); } catch (e) { return null; }
} catch (e) { return null; /* offline / endpoint absent — non-fatal */ }
}
+7 -5
View File
@@ -26,7 +26,7 @@ class FakeMetaDb:
self.conn.execute(
"""CREATE TABLE song_stats (
filename TEXT, arrangement TEXT, best_accuracy REAL,
last_played_at TEXT,
last_accuracy REAL, last_played_at TEXT,
seconds_total REAL NOT NULL DEFAULT 0
)"""
)
@@ -38,10 +38,12 @@ class FakeMetaDb:
)
def add(self, filename, arrangement, best_accuracy, in_library=True,
genre="", arrangements=None, last_played_at=None, seconds_total=0):
self.conn.execute("INSERT INTO song_stats VALUES (?, ?, ?, ?, ?)",
(filename, arrangement, best_accuracy, last_played_at,
seconds_total))
genre="", arrangements=None, last_played_at=None, seconds_total=0,
last_accuracy=None):
self.conn.execute("INSERT INTO song_stats VALUES (?, ?, ?, ?, ?, ?)",
(filename, arrangement, best_accuracy,
last_accuracy if last_accuracy is not None else best_accuracy,
last_played_at, seconds_total))
if in_library:
self.conn.execute(
"INSERT INTO songs SELECT ?, ?, ?, ?, ? WHERE NOT EXISTS "
+163
View File
@@ -208,3 +208,166 @@ def test_drill_state_merge_is_gained_only(client, meta_db):
p = _passport(client)
assert p["drills"]["cleared"] == ["blues_shuffle"]
assert p["badge"] == "earned"
def test_genre_families_inherit_drills(client, meta_db):
# 'death metal' has no exact entry — it inherits the metal family's drill.
for i in range(5):
meta_db.add(f"dm{i}.feedpak", 0, 0.9, genre="Death Metal", arrangements=LEAD)
_open(client, "guitar", "Death Metal")
p = _passport(client, "guitar", "death metal")
assert p["drills"]["required"] == ["melodic_metal_gallop"]
assert p["badge"] == "in_progress"
# 'metalcore' (single word) matches by substring, no alias needed.
_open(client, "guitar", "Metalcore")
assert _passport(client, "guitar", "metalcore")["drills"]["required"] == \
["melodic_metal_gallop"]
# 'blues rock' resolves by family LIST ORDER: blues comes before rock.
_open(client, "guitar", "Blues Rock")
assert _passport(client, "guitar", "blues rock")["drills"]["required"] == \
["blues_shuffle"]
# A genre outside every family stays songs-only.
_open(client, "guitar", "Reggae")
assert _passport(client, "guitar", "reggae")["drills"]["required"] == []
# Exact per-genre entries still beat the family (the shipped 'metal' entry
# IS the exact entry for genre key 'metal').
_open(client, "guitar", "Metal")
assert _passport(client, "guitar", "metal")["drills"]["required"] == \
["melodic_metal_gallop"]
def test_family_drills_stay_per_instrument(client, meta_db):
# Family inheritance must not leak guitar drills onto other instruments.
keys_arr = [{"type": "lead", "name": "Keys"}]
for i in range(5):
meta_db.add(f"kdm{i}.feedpak", 0, 0.9, genre="Death Metal", arrangements=keys_arr)
_open(client, "keys", "Death Metal")
p = _passport(client, "keys", "death metal")
assert p["drills"]["required"] == []
assert p["badge"] == "earned"
def test_nearest_invitations_order_and_exclusions(client, meta_db):
# Non-qualifying songs sorted by distance to the QUALIFYING bar;
# qualifying songs never appear; capped at 3.
meta_db.add("q.feedpak", 0, 0.80, genre="Soul", arrangements=LEAD) # qualifies
meta_db.add("close.feedpak", 0, 0.74, genre="Soul", arrangements=LEAD) # 1% to 2★
meta_db.add("mid.feedpak", 0, 0.70, genre="Soul", arrangements=LEAD) # 5% to 2★
meta_db.add("far.feedpak", 0, 0.30, genre="Soul", arrangements=LEAD) # 30% to 1★
meta_db.add("far2.feedpak", 0, 0.25, genre="Soul", arrangements=LEAD)
_open(client, "guitar", "Soul")
p = _passport(client, "guitar", "soul")
names = [s["filename"] for s in p["nearest"]]
assert names == ["close.feedpak", "mid.feedpak", "far.feedpak"]
assert all(s["next_star_at"] is not None for s in p["nearest"])
assert "q.feedpak" not in names
def test_nearest_targets_the_qualifying_bar_not_next_star(client, meta_db):
# A 0★ song 1% from its NEXT star is farther from the ★★ badge bar than
# a 1★ song 5% from it — nearest must rank by the badge bar.
meta_db.add("one_star.feedpak", 0, 0.70, genre="Soul", arrangements=LEAD) # 5% to bar
meta_db.add("zero_star.feedpak", 0, 0.59, genre="Soul", arrangements=LEAD) # 1% to next ★, 16% to bar
_open(client, "guitar", "Soul")
p = _passport(client, "guitar", "soul")
assert [s["filename"] for s in p["nearest"]] == ["one_star.feedpak", "zero_star.feedpak"]
assert all(s["bar_at"] == 0.75 for s in p["nearest"])
# ── Gigs ──────────────────────────────────────────────────────────────────────
def test_gig_propose_mixes_owned_and_stakes(client, meta_db):
for i in range(4):
meta_db.add(f"own{i}.feedpak", 0, 0.85, genre="Soul", arrangements=LEAD)
meta_db.add("stake.feedpak", 0, 0.70, genre="Soul", arrangements=LEAD)
meta_db.add_song_only("fresh.feedpak", genre="Soul")
res = client.post("/api/plugins/career/gigs/propose",
json={"instrument": "guitar", "genre": "Soul", "size": 4})
assert res.status_code == 200
gig = res.json()
files = [s["filename"] for s in gig["songs"]]
assert len(files) == 4
assert "stake.feedpak" in files # a near-bar song gives the set stakes
assert gig["venue_id"] == "bar" # 9 stars < 50: the dive bar
# A young passport (nothing played) still gets a playable set from the
# library's unplayed genre songs.
res2 = client.post("/api/plugins/career/gigs/propose",
json={"instrument": "guitar", "genre": "Ska"})
assert res2.status_code == 404 # no ska in the library at all
meta_db.add_song_only("ska1.feedpak", genre="Ska")
res3 = client.post("/api/plugins/career/gigs/propose",
json={"instrument": "guitar", "genre": "Ska"})
assert [s["filename"] for s in res3.json()["songs"]] == ["ska1.feedpak"]
def test_gig_log_computes_encore_and_surfaces_in_passports(client, meta_db):
for i in range(2):
meta_db.add(f"s{i}.feedpak", 0, 0.9, genre="Soul", arrangements=LEAD,
last_accuracy=0.9)
_open(client, "guitar", "Soul")
res = client.post("/api/plugins/career/gigs", json={
"instrument": "guitar", "genre": "Soul", "venue_id": "bar",
"songs": ["s0.feedpak", "s1.feedpak"]})
assert res.status_code == 200
gig = res.json()["gig"]
assert gig["encore"] is True # avg 0.9 ≥ 0.75
assert gig["songs"][0]["accuracy"] == 0.9
view = client.get("/api/plugins/career/passports").json()
assert view["instruments"]["guitar"]["gig_count"] == 1
p = _passport(client, "guitar", "soul")
assert len(p["gigs"]) == 1 and p["gigs"][0]["encore"] is True
def test_gig_log_validation_and_no_fail_state(client):
# Unknown venue / bad songs shapes are rejected; nothing is ever logged
# as a failed gig — the endpoint only appends completed sets.
assert client.post("/api/plugins/career/gigs", json={
"instrument": "guitar", "genre": "Soul", "venue_id": "nope",
"songs": ["x"]}).status_code == 400
assert client.post("/api/plugins/career/gigs", json={
"instrument": "guitar", "genre": "Soul", "songs": []}).status_code == 400
assert client.post("/api/plugins/career/gigs", json={
"instrument": "guitar", "genre": "Soul",
"songs": ["f"] * 9}).status_code == 400
def test_gig_accuracy_reads_newest_row_and_encore_needs_full_set(client, meta_db):
# Newest row wins: a stale higher accuracy on another arrangement must
# not inflate the gig log.
meta_db.add("dual.feedpak", 1, 0.95, genre="Soul", arrangements=BASS,
last_accuracy=0.95, last_played_at="2026-06-01T00:00:00")
meta_db.add("dual.feedpak", 0, 0.60, genre="Soul", arrangements=LEAD,
last_accuracy=0.60, last_played_at="2026-07-14T00:00:00")
res = client.post("/api/plugins/career/gigs", json={
"instrument": "guitar", "genre": "Soul", "songs": ["dual.feedpak"]})
assert res.json()["gig"]["songs"][0]["accuracy"] == 0.6
# A set with an unscored song never earns the encore off one good song.
meta_db.add("scored.feedpak", 0, 0.9, genre="Soul", arrangements=LEAD,
last_accuracy=0.9, last_played_at="2026-07-14T00:01:00")
res2 = client.post("/api/plugins/career/gigs", json={
"instrument": "guitar", "genre": "Soul",
"songs": ["scored.feedpak", "ghost.feedpak"]})
assert res2.json()["gig"]["encore"] is False
def test_gig_propose_backfills_from_surplus_qualifying(client, meta_db):
# Mature passport: plenty of qualifying songs, nothing near the bar,
# nothing unplayed — the set still fills to size.
for i in range(8):
meta_db.add(f"own{i}.feedpak", 0, 0.9, genre="Ska", arrangements=LEAD)
res = client.post("/api/plugins/career/gigs/propose",
json={"instrument": "guitar", "genre": "Ska", "size": 5})
assert len(res.json()["songs"]) == 5
def test_gig_propose_backfill_offset_survives_stakes(client, meta_db):
# 4 qualifying + 1 near-bar stake, size 5: the stake must not shift the
# qualifying backfill window past eligible songs.
for i in range(4):
meta_db.add(f"q{i}.feedpak", 0, 0.9, genre="Reggae", arrangements=LEAD)
meta_db.add("near.feedpak", 0, 0.7, genre="Reggae", arrangements=LEAD)
res = client.post("/api/plugins/career/gigs/propose",
json={"instrument": "guitar", "genre": "Reggae", "size": 5})
files = [s["filename"] for s in res.json()["songs"]]
assert len(files) == 5 and len(set(files)) == 5
assert "near.feedpak" in files
+48
View File
@@ -273,3 +273,51 @@ def test_title_keyset_paging_is_complete_with_overrides(client, server):
if not cursor:
break
assert sorted(seen) == ["a.archive", "b.archive", "c.archive"] # each exactly once
def test_enrichment_genre_fallback_precedence(client, server):
# Precedence: override → pack genre → MusicBrainz enrichment (matched only).
_put(server, "a.archive", title="A", genre="Rock") # pack wins over enrichment
_put(server, "b.archive", title="B", genre="") # falls back to enrichment
_put(server, "c.archive", title="C", genre="") # override beats enrichment
_put(server, "d.archive", title="D", genre="") # unmatched candidate: ignored
ins = "INSERT INTO song_enrichment (filename, match_state, genres) VALUES (?, ?, ?)"
server.meta_db.conn.execute(ins, ("a.archive", "matched", '["metal"]'))
server.meta_db.conn.execute(ins, ("b.archive", "matched", '["progressive rock", "rock"]'))
server.meta_db.conn.execute(ins, ("c.archive", "matched", '["jazz"]'))
server.meta_db.conn.execute(ins, ("d.archive", "review", '["country"]'))
_put(server, "e.archive", title="E", genre="") # manual pin is trusted too
server.meta_db.conn.execute(ins, ("e.archive", "manual", '["ska"]'))
server.meta_db.conn.commit()
server.meta_db.set_song_override("c.archive", "genre", value="City Pop")
genres = client.get("/api/library/genres").json()["genres"]
assert "Rock" in genres # pack value kept for a
assert "metal" not in genres # enrichment never overrides a pack genre
assert "progressive rock" in genres # b: enrichment primary ([0]) surfaces
assert "City Pop" in genres and "jazz" not in genres # override beats enrichment
assert "country" not in genres # review/failed candidates never leak
assert "ska" in genres # user-pinned (manual) matches count
# Filtering by the enriched genre finds the song.
r = client.get("/api/library", params={"genre": "progressive rock"}).json()
assert [s["filename"] for s in r["songs"]] == ["b.archive"]
def test_no_enrichment_and_no_overrides_uses_plain_column(server):
_put(server, "a.archive", title="A", genre="Rock")
assert server.meta_db._effective_genre_expr() == "genre"
def test_overrides_without_enrichment_table_stay_safe(server):
# A stand-in scenario: overrides exist but song_enrichment is gone — the
# expression must not reference the missing table.
server.meta_db.conn.execute("DROP TABLE song_enrichment")
_put(server, "a.archive", title="A", genre="")
server.meta_db.set_song_override("a.archive", "genre", value="City Pop")
expr = server.meta_db._effective_genre_expr()
assert "song_enrichment" not in expr
# And it still evaluates: the override surfaces through the facet query.
row = server.meta_db.conn.execute(
f"SELECT {expr} FROM songs WHERE filename = 'a.archive'").fetchone()
assert row[0] == "City Pop"
+429
View File
@@ -0,0 +1,429 @@
"""tools/migrate_full_mix_stem.py — packs off the deprecated `original_audio:` key.
The migration moves real audio inside tens of thousands of archives, so the
interesting cases are the ones where it must NOT act: a pack it would corrupt, a
pack it has already done, a pack whose mixdown isn't where the key claims.
"""
from __future__ import annotations
import importlib.util
import zipfile
from pathlib import Path
import pytest
import yaml
_SPEC = importlib.util.spec_from_file_location(
"migrate_full_mix_stem",
Path(__file__).resolve().parent.parent / "tools" / "migrate_full_mix_stem.py",
)
mig = importlib.util.module_from_spec(_SPEC)
_SPEC.loader.exec_module(mig)
def _manifest(**extra) -> dict:
m = {
"feedpak_version": "1.13.0",
"title": "T",
"artist": "A",
"duration": 1.0,
"arrangements": [{"id": "lead", "file": "arrangements/lead.json"}],
"stems": [
{"id": "guitar", "file": "stems/guitar.ogg", "default": "on"},
{"id": "drums", "file": "stems/drums.ogg", "default": "on"},
],
"original_audio": "original/full.ogg",
}
m.update(extra)
return m
def _write_pack(path: Path, manifest: dict, files: dict[str, bytes] | None = None) -> Path:
files = files or {
"original/full.ogg": b"MIXDOWN",
"stems/guitar.ogg": b"g",
"stems/drums.ogg": b"d",
"arrangements/lead.json": b"{}",
}
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zf:
zf.writestr("manifest.yaml", yaml.safe_dump(manifest, sort_keys=False))
for name, data in files.items():
zf.writestr(name, data)
return path
def _read(path: Path) -> tuple[dict, set[str]]:
with zipfile.ZipFile(path) as zf:
return yaml.safe_load(zf.read("manifest.yaml")), set(zf.namelist())
# ── plan_manifest: the decisions, without the archives ──────────────────────
def test_plan_adds_the_full_stem_and_drops_the_key():
new, move = mig.plan_manifest(_manifest())
assert move == "original/full.ogg"
assert "original_audio" not in new
assert new["stems"][0] == {
"id": "full",
"file": "stems/full.ogg",
"default": "off",
}
# The separated stems survive, in order, untouched.
assert [s["id"] for s in new["stems"]] == ["full", "guitar", "drums"]
assert new["feedpak_version"] == "1.15.0"
def test_plan_marks_the_retained_mixdown_default_off():
"""The one line that keeps a pre-1.15.0 reader from doubling the song: a
reader that sums every stem still won't play `full` on open if it honours
`default`, which has been normative since 1.0.0."""
new, _ = mig.plan_manifest(_manifest())
assert new["stems"][0]["default"] == "off"
def test_plan_marks_a_sole_mixdown_default_on():
"""With no separated stems the mixdown IS the audio — off would mute the pack."""
new, _ = mig.plan_manifest(_manifest(stems=[]))
assert new["stems"] == [{"id": "full", "file": "stems/full.ogg", "default": "on"}]
def test_plan_preserves_unknown_keys_verbatim():
"""Spec §3: a writer that re-emits a pack SHOULD preserve unknown keys."""
new, _ = mig.plan_manifest(_manifest(source_tool="ExampleTool v1.2.3", rigs="rigs.json"))
assert new["source_tool"] == "ExampleTool v1.2.3"
assert new["rigs"] == "rigs.json"
def test_plan_skips_an_already_migrated_pack():
m = _manifest(
stems=[{"id": "full", "file": "stems/full.ogg", "default": "off"}],
)
del m["original_audio"]
with pytest.raises(mig.Skip):
mig.plan_manifest(m)
def test_plan_skips_a_pack_that_never_had_the_key():
m = _manifest()
del m["original_audio"]
with pytest.raises(mig.Skip):
mig.plan_manifest(m)
def test_plan_drops_a_stale_key_without_moving_anything():
"""Mixdown already a stem, dead key lingering beside it."""
new, move = mig.plan_manifest(
_manifest(stems=[{"id": "full", "file": "stems/full.ogg", "default": "off"}])
)
assert move == ""
assert "original_audio" not in new
assert [s["id"] for s in new["stems"]] == ["full"]
def test_plan_forces_an_existing_full_stem_off_beside_instrument_stems():
"""Dropping the stale key is not enough if the mixdown it duplicated is left
ENABLED: a reader that honours `default` would then play the whole song on top
of the stems on open. The migration must not hand back a pack in the exact
state it exists to remove."""
new, move = mig.plan_manifest(
_manifest(
stems=[
{"id": "full", "file": "stems/full.ogg", "default": "on"},
{"id": "guitar", "file": "stems/guitar.ogg", "default": "on"},
]
)
)
assert move == ""
assert new["stems"][0] == {"id": "full", "file": "stems/full.ogg", "default": "off"}
assert new["stems"][1]["default"] == "on" # instruments untouched
def test_plan_leaves_a_sole_full_stem_enabled_when_dropping_a_stale_key():
"""No instruments beside it — the mixdown IS the audio. Forcing it off here
would mute the pack."""
new, _ = mig.plan_manifest(
_manifest(stems=[{"id": "full", "file": "stems/full.ogg", "default": "on"}])
)
assert new["stems"] == [{"id": "full", "file": "stems/full.ogg", "default": "on"}]
def test_plan_needs_no_move_when_the_key_already_points_at_the_canonical_path():
new, move = mig.plan_manifest(_manifest(original_audio="stems/full.ogg"))
assert move == ""
assert new["stems"][0]["file"] == "stems/full.ogg"
# ── migrate_zip: the archive rewrite ────────────────────────────────────────
def test_migrate_moves_the_audio_and_rewrites_the_manifest(tmp_path: Path):
pak = _write_pack(tmp_path / "song.feedpak", _manifest())
assert mig.migrate_zip(pak, dry_run=False) == "migrated"
manifest, names = _read(pak)
assert "original/full.ogg" not in names # the invented directory is gone
assert "stems/full.ogg" in names # audio lives where the format says
assert "original_audio" not in manifest
assert manifest["stems"][0]["id"] == "full"
assert mig.verify_zip(pak) == "ok"
def test_migrate_preserves_the_mixdown_bytes(tmp_path: Path):
"""It is a rename, not a re-encode. Losing a byte here loses the master audio."""
pak = _write_pack(tmp_path / "song.feedpak", _manifest())
mig.migrate_zip(pak, dry_run=False)
with zipfile.ZipFile(pak) as zf:
assert zf.read("stems/full.ogg") == b"MIXDOWN"
assert zf.read("stems/guitar.ogg") == b"g"
def test_migrate_is_idempotent(tmp_path: Path):
pak = _write_pack(tmp_path / "song.feedpak", _manifest())
assert mig.migrate_zip(pak, dry_run=False) == "migrated"
before = pak.read_bytes()
assert mig.migrate_zip(pak, dry_run=False) == "skip"
assert pak.read_bytes() == before # a re-run touches nothing
def test_dry_run_changes_nothing(tmp_path: Path):
pak = _write_pack(tmp_path / "song.feedpak", _manifest())
before = pak.read_bytes()
assert mig.migrate_zip(pak, dry_run=True) == "would-migrate"
assert pak.read_bytes() == before
def test_migrate_refuses_when_the_mixdown_is_absent(tmp_path: Path):
"""The key points at audio the archive doesn't contain. Fabricating a stem
entry for a missing file would break every reader refuse, don't guess."""
pak = _write_pack(
tmp_path / "song.feedpak",
_manifest(),
files={"stems/guitar.ogg": b"g", "arrangements/lead.json": b"{}"},
)
before = pak.read_bytes()
assert mig.migrate_zip(pak, dry_run=False) == "missing-audio"
assert pak.read_bytes() == before
def test_migrate_refuses_when_the_target_path_is_taken(tmp_path: Path):
"""A `stems/full.ogg` that is NOT the mixdown already occupies the target.
Overwriting it would destroy a stem."""
pak = _write_pack(
tmp_path / "song.feedpak",
_manifest(),
files={
"original/full.ogg": b"MIXDOWN",
"stems/full.ogg": b"SOMETHING-ELSE",
"arrangements/lead.json": b"{}",
},
)
assert mig.migrate_zip(pak, dry_run=False) == "target-occupied"
with zipfile.ZipFile(pak) as zf:
assert zf.read("stems/full.ogg") == b"SOMETHING-ELSE"
def test_migrate_drops_a_stale_key_beside_a_non_canonical_full_stem(tmp_path: Path):
"""The mixdown is already a stem, but at a path of the pack's own choosing —
which is legal (§2.2: readers resolve through the manifest, never by
filename). Only the dead key needs removing. Demanding `stems/full.ogg` here
would reject a perfectly valid pack as `missing-audio`."""
m = _manifest(stems=[{"id": "full", "file": "audio/mixdown.ogg", "default": "off"}])
pak = _write_pack(
tmp_path / "song.feedpak",
m,
files={"audio/mixdown.ogg": b"MIXDOWN", "arrangements/lead.json": b"{}"},
)
assert mig.migrate_zip(pak, dry_run=False) == "migrated"
manifest, names = _read(pak)
assert "original_audio" not in manifest
assert manifest["stems"] == [
{"id": "full", "file": "audio/mixdown.ogg", "default": "off"}
]
assert "audio/mixdown.ogg" in names # the audio never moved
assert mig.verify_zip(pak) == "ok"
# ── verify_zip ──────────────────────────────────────────────────────────────
def test_verify_rejects_a_retained_mixdown_that_plays_on_open(tmp_path: Path):
"""The hazard the migration must never create: `full` alongside instrument
stems AND default-on means a summing reader plays the whole song twice."""
m = _manifest(
stems=[
{"id": "full", "file": "stems/full.ogg", "default": "on"},
{"id": "guitar", "file": "stems/guitar.ogg", "default": "on"},
]
)
del m["original_audio"]
pak = _write_pack(
tmp_path / "song.feedpak",
m,
files={"stems/full.ogg": b"M", "stems/guitar.ogg": b"g"},
)
assert mig.verify_zip(pak) == "full-stem-default-on"
@pytest.mark.parametrize(
"default, expected",
[
({"default": "off"}, "ok"), # the one safe, canonical shape
({"default": "OFF"}, "ok"), # case-insensitive
({"default": " off "}, "ok"), # surrounding whitespace tolerated
({}, "full-stem-default-not-off"), # MISSING — core defaults to True (ON)
({"default": ""}, "full-stem-default-not-off"), # empty → ON in core
({"default": False}, "full-stem-default-not-off"), # boolean, not the string
({"default": True}, "full-stem-default-on"), # boolean truthy → plays
({"default": "false"}, "full-stem-default-not-off"), # off-ish but non-canonical
({"default": "0"}, "full-stem-default-not-off"),
({"default": "no"}, "full-stem-default-not-off"),
({"default": "maybe"}, "full-stem-default-not-off"), # malformed
({"default": "on"}, "full-stem-default-on"),
({"default": "yes"}, "full-stem-default-on"),
({"default": "1"}, "full-stem-default-on"),
],
)
def test_verify_requires_an_explicit_off_on_a_retained_mixdown(tmp_path, default, expected):
"""Beside instrument stems, `full` is safe only with an explicit normalized
`off`. Core defaults an ABSENT `default` to ON and treats empty/unknown as
ON, so a missing or blank default is the double-audio hazard itself, not a
lesser one `verify` must not certify it."""
m = _manifest(
stems=[
{"id": "full", "file": "stems/full.ogg", **default},
{"id": "guitar", "file": "stems/guitar.ogg", "default": "on"},
]
)
del m["original_audio"]
pak = _write_pack(
tmp_path / f"{tmp_path.name}.feedpak",
m,
files={"stems/full.ogg": b"M", "stems/guitar.ogg": b"g"},
)
assert mig.verify_zip(pak) == expected
def test_verify_ignores_default_on_a_sole_full_stem(tmp_path: Path):
"""A single `full` stem IS the audio — the len>1 gate means its default is
not policed, so an on/absent default is fine (off would mute the pack)."""
for default in ({"default": "on"}, {}, {"default": ""}):
m = _manifest(stems=[{"id": "full", "file": "stems/full.ogg", **default}])
del m["original_audio"]
pak = _write_pack(
tmp_path / f"{tmp_path.name}-{len(default)}.feedpak",
m,
files={"stems/full.ogg": b"M"},
)
assert mig.verify_zip(pak) == "ok"
def test_verify_rejects_an_unmigrated_pack(tmp_path: Path):
pak = _write_pack(tmp_path / "song.feedpak", _manifest())
assert mig.verify_zip(pak) == "still-has-key"
# ── Unsafe manifest paths must not be laundered into playable audio ─────────
@pytest.mark.parametrize(
"rel", ["../outside.ogg", "/etc/passwd", "a/../../x.ogg", "C:/x.ogg", "a\\b.ogg"]
)
def test_migrate_refuses_an_unsafe_full_mix_path(tmp_path: Path, rel: str):
"""Core's loader REFUSES a full-mix path that escapes the pack — such a pack
simply has no full mix, and the audio is inert. Migrating it into
`stems/full.ogg` would take content the reader deliberately rejected and hand
it back as a valid, playable stem. Report it; never promote it."""
pak = _write_pack(
tmp_path / "song.feedpak",
_manifest(original_audio=rel),
files={rel: b"EVIL", "stems/guitar.ogg": b"g"},
)
before = pak.read_bytes()
assert mig.migrate_zip(pak, dry_run=False) == "unsafe-path"
assert pak.read_bytes() == before
def test_safe_relpath_accepts_ordinary_pack_paths():
assert mig.is_safe_relpath("stems/full.ogg")
assert mig.is_safe_relpath("original/full.ogg")
assert not mig.is_safe_relpath("")
assert not mig.is_safe_relpath("a//b.ogg")
# ── Damaged packs must not abort the run ────────────────────────────────────
def test_a_corrupt_archive_is_reported_not_fatal(tmp_path: Path, capsys):
"""A real library has damage in it — a truncated download, an archive left
half-written by an interrupted converter. One of those must not kill a
50,000-pack run and throw away the summary: the pack is reported, skipped,
and everything else still migrates."""
good = _write_pack(tmp_path / "good.feedpak", _manifest())
bad = tmp_path / "bad.feedpak"
bad.write_bytes(b"this is not a zip file at all")
rc = mig.main([str(tmp_path)])
out = capsys.readouterr().out
assert rc == 1 # a problem pack fails the run's exit code
assert "corrupt-zip" in out
assert "migrated" in out
assert mig.verify_zip(good) == "ok" # the healthy pack still got migrated
assert bad.read_bytes() == b"this is not a zip file at all" # untouched
# ── Directory-form (authoring) packs are discovered, not silently skipped ────
def _write_dir_pack(path: Path, manifest: dict, files: dict[str, bytes] | None = None) -> Path:
"""Build a directory-form pack (`song.sloppak/`), the authoring shape."""
files = files or {
"original/full.ogg": b"MIXDOWN",
"stems/guitar.ogg": b"g",
"stems/drums.ogg": b"d",
"arrangements/lead.json": b"{}",
}
path.mkdir()
(path / "manifest.yaml").write_text(yaml.safe_dump(manifest, sort_keys=False))
for name, data in files.items():
p = path / name
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(data)
return path
def test_iter_packs_discovers_directory_form_packs(tmp_path: Path):
"""A `song.sloppak/` directory is a pack; os.walk must yield it whole and
NOT descend into it (its stems/ are contents, not packs)."""
d = _write_dir_pack(tmp_path / "song.sloppak", _manifest())
z = _write_pack(tmp_path / "other.feedpak", _manifest())
found = set(mig.iter_packs(tmp_path))
assert d in found and z in found
# Nothing inside the directory pack was yielded as its own pack.
assert not any(d in p.parents for p in found)
def test_iter_packs_yields_a_directly_passed_dir_pack(tmp_path: Path):
d = _write_dir_pack(tmp_path / "song.sloppak", _manifest())
assert list(mig.iter_packs(d)) == [d]
def test_directory_form_pack_is_reported_not_silently_skipped(tmp_path: Path, capsys):
"""The migrator rewrites single-file packs atomically; a directory can't be
swapped that way, so it is surfaced as a problem rather than vanishing from
the run (the silent-skip this guards against) or being rewritten unsafely."""
d = _write_dir_pack(tmp_path / "song.sloppak", _manifest())
good = _write_pack(tmp_path / "good.feedpak", _manifest())
rc = mig.main([str(tmp_path)])
out = capsys.readouterr().out
assert rc == 1 # a reported problem fails the exit code
assert "dir-form-unsupported" in out
assert mig.verify_zip(good) == "ok" # the zip pack still migrated
# The directory pack is untouched: legacy key intact, mixdown not moved.
manifest = yaml.safe_load((d / "manifest.yaml").read_text())
assert manifest.get("original_audio") == "original/full.ogg"
assert (d / "original" / "full.ogg").read_bytes() == b"MIXDOWN"
def test_verify_reports_directory_form_packs(tmp_path: Path):
d = _write_dir_pack(tmp_path / "song.sloppak", _manifest())
assert mig.verify_pack(d) == "dir-form-unsupported"
+277
View File
@@ -0,0 +1,277 @@
"""The sloppak loader's handling of a pack's complete mixdown (#933).
The mixdown is a stem: feedpak spec §5.3 RESERVES the id `full` for it. It is a
mixdown, not a layer it already contains every instrument, so a reader that
sums `stems` must never include it in that sum, and `load_song()` therefore
lifts it OUT of `LoadedSloppak.stems` and onto `LoadedSloppak.full_mix`.
Also covers the DEPRECATED `original_audio:` manifest key a key this repo
invented (#583) before the spec reserved `full`, which every pack in the wild
still carries. We read it as a fallback so those packs keep their full mix; we
never write it. Those tests are the deprecation contract: they go when the key
does (#945).
"""
from __future__ import annotations
import json
from pathlib import Path
import yaml
import sloppak as sloppak_mod
def _write_dir_sloppak(
root: Path,
manifest_extras: dict,
*,
write_legacy_full_mix: bool = False,
stems: list[dict] | None = None,
) -> Path:
"""Build a minimal directory-form sloppak that load_song will accept.
Uses the tmp_path leaf name to make the sloppak filename unique per test,
avoiding the module-level ``resolve_source_dir`` cache being poisoned by a
previous test that happened to share the same "song.sloppak" filename.
"""
pak = root / f"{root.name}.sloppak"
pak.mkdir()
arr_dir = pak / "arrangements"
arr_dir.mkdir()
arr = {
"name": "Lead",
"tuning": [0, 0, 0, 0, 0, 0],
"capo": 0,
"notes": [],
"chords": [],
"anchors": [],
"handshapes": [],
"templates": [],
"beats": [],
"sections": [],
}
(arr_dir / "lead.json").write_text(json.dumps(arr))
manifest = {
"title": "Test",
"artist": "Tester",
"album": "",
"year": 2026,
"duration": 10.0,
"arrangements": [{"id": "lead", "name": "Lead", "file": "arrangements/lead.json"}],
"stems": (
stems
if stems is not None
else [{"id": "guitar", "file": "stems/guitar.ogg", "default": True}]
),
}
manifest.update(manifest_extras)
(pak / "manifest.yaml").write_text(yaml.safe_dump(manifest, sort_keys=False))
if write_legacy_full_mix:
orig_dir = pak / "original"
orig_dir.mkdir()
# The loader only checks presence (is_file); contents are irrelevant.
(orig_dir / "full.ogg").write_bytes(b"OggS-not-real")
return pak
def _load(pak_path: Path, tmp_path: Path):
dlc_root = pak_path.parent
cache = tmp_path / "cache"
cache.mkdir()
return sloppak_mod.load_song(pak_path.name, dlc_root, cache)
def _separated(**extra) -> list[dict]:
"""A separated pack that RETAINS its mixdown, as spec §5.3 asks writers to."""
return [
{"id": "full", "file": "stems/full.ogg", "default": False, **extra},
{"id": "guitar", "file": "stems/guitar.ogg", "default": True},
{"id": "drums", "file": "stems/drums.ogg", "default": True},
]
# ── The `full` stem is the mixdown (spec §5.3) ───────────────────────────────
def test_full_stem_is_surfaced_as_the_mixdown(tmp_path: Path):
pak = _write_dir_sloppak(tmp_path, {}, stems=_separated())
loaded = _load(pak, tmp_path)
# Manifest-relative, so the WS builds its URL exactly as it builds a stem's.
assert loaded.full_mix == "stems/full.ogg"
def test_full_stem_is_removed_from_the_stem_list(tmp_path: Path):
"""The regression this whole change exists to prevent.
Every consumer sums `stems` into one mix and renders one fader per entry. The
mixdown already contains every instrument, so leaving it in the list doubles
the entire song and muting `guitar` would still leave guitar audible inside
it. That exact trap is why the packer invented `original_audio` rather than
putting the mixdown where the format says it goes.
"""
pak = _write_dir_sloppak(tmp_path, {}, stems=_separated())
loaded = _load(pak, tmp_path)
assert [s["id"] for s in loaded.stems] == ["guitar", "drums"]
def test_single_mix_pack_keeps_full_as_its_only_stem(tmp_path: Path):
"""A pack whose ONLY stem is `full` is a single-mix pack, not a separated one.
There are no instruments to be pristine against, so the mixdown stays the sole
playable stem and nothing is surfaced separately. Anything else would strip the
stem list of the most common pack shape in the library and leave it silent.
"""
pak = _write_dir_sloppak(
tmp_path, {}, stems=[{"id": "full", "file": "stems/full.ogg", "default": True}]
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix is None
assert [s["id"] for s in loaded.stems] == ["full"]
def test_every_full_entry_is_removed_not_just_the_first(tmp_path: Path):
"""A malformed pack listing `full` twice must not leave one behind.
Removing the mixdown by object identity would drop only the entry we surface
and leave its duplicate in the stem list a whole copy of the song, summed
with the instruments. That is the exact bug this partition prevents, so a
duplicate must not smuggle it back in.
"""
pak = _write_dir_sloppak(
tmp_path,
{},
stems=[
{"id": "full", "file": "stems/full.ogg", "default": False},
{"id": "guitar", "file": "stems/guitar.ogg", "default": True},
{"id": "full", "file": "original/full.ogg", "default": True},
],
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix == "stems/full.ogg"
assert [s["id"] for s in loaded.stems] == ["guitar"]
def test_separated_pack_without_a_full_stem_has_no_mixdown(tmp_path: Path):
"""Stems only, mixdown discarded — the pre-1.15.0 shape. Nothing to surface."""
pak = _write_dir_sloppak(
tmp_path,
{},
stems=[
{"id": "guitar", "file": "stems/guitar.ogg", "default": True},
{"id": "drums", "file": "stems/drums.ogg", "default": True},
],
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix is None
assert [s["id"] for s in loaded.stems] == ["guitar", "drums"]
def test_single_mix_pack_ignores_a_lingering_deprecated_key(tmp_path: Path):
"""`full` is the pack's only stem AND the old key is still there.
The stem wins, and it stays the sole playable stem falling back to the key
would surface the mixdown twice: once as the stem the player is already
playing, and once as a "pristine" track for it to cross over to.
"""
pak = _write_dir_sloppak(
tmp_path,
{"original_audio": "original/full.ogg"},
stems=[{"id": "full", "file": "stems/full.ogg", "default": True}],
write_legacy_full_mix=True,
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix is None
assert [s["id"] for s in loaded.stems] == ["full"]
def test_full_stem_wins_over_the_deprecated_key(tmp_path: Path):
"""A migrated pack that still carries the old key must use the stem."""
pak = _write_dir_sloppak(
tmp_path,
{"original_audio": "original/full.ogg"},
stems=_separated(),
write_legacy_full_mix=True,
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix == "stems/full.ogg"
# ── The library index must not advertise the mixdown as an instrument ────────
def test_extract_meta_excludes_the_mixdown_from_stem_ids(tmp_path: Path):
"""The library's stem chips / stem_count come from here, and must agree with
load_song() otherwise the filter offers a "full" chip beside guitar+drums
and counts a third stem that no mixer will ever show."""
pak = _write_dir_sloppak(tmp_path, {}, stems=_separated())
meta = sloppak_mod.extract_meta(pak)
assert meta["stem_ids"] == ["guitar", "drums"]
assert meta["stem_count"] == 2
def test_extract_meta_keeps_full_for_a_single_mix_pack(tmp_path: Path):
pak = _write_dir_sloppak(
tmp_path, {}, stems=[{"id": "full", "file": "stems/full.ogg", "default": True}]
)
meta = sloppak_mod.extract_meta(pak)
assert meta["stem_ids"] == ["full"]
assert meta["stem_count"] == 1
# ── DEPRECATED `original_audio:` fallback — delete with the key (#945) ───────
def test_legacy_key_still_provides_the_full_mix(tmp_path: Path):
"""Every pack written before the spec reserved `full` looks like this. Dropping
the read would silently take the pristine mix away from all of them."""
pak = _write_dir_sloppak(
tmp_path, {"original_audio": "original/full.ogg"}, write_legacy_full_mix=True
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix == "original/full.ogg"
# The legacy mixdown lives OUTSIDE `stems`, so the stem list is untouched.
assert [s["id"] for s in loaded.stems] == ["guitar"]
def test_legacy_key_absent_means_no_full_mix(tmp_path: Path):
pak = _write_dir_sloppak(tmp_path, {}, write_legacy_full_mix=True)
loaded = _load(pak, tmp_path)
assert loaded.full_mix is None
def test_legacy_key_none_when_file_missing(tmp_path: Path):
# Manifest points at a full mix that isn't on disk — disabled silently.
pak = _write_dir_sloppak(
tmp_path, {"original_audio": "original/full.ogg"}, write_legacy_full_mix=False
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix is None
def test_legacy_key_none_when_value_blank(tmp_path: Path):
pak = _write_dir_sloppak(
tmp_path, {"original_audio": " "}, write_legacy_full_mix=True
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix is None
# ── Security / path-traversal branches (legacy key only — a stem `file` is
# resolved through the same /api/sloppak/.../file/ guard as every other stem)
def test_legacy_key_none_when_path_escapes_sloppak(tmp_path: Path):
pak = _write_dir_sloppak(
tmp_path, {"original_audio": "../outside.ogg"}, write_legacy_full_mix=True
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix is None
def test_legacy_key_none_when_path_is_absolute(tmp_path: Path):
pak = _write_dir_sloppak(
tmp_path, {"original_audio": "/etc/passwd"}, write_legacy_full_mix=True
)
loaded = _load(pak, tmp_path)
assert loaded.full_mix is None
-119
View File
@@ -1,119 +0,0 @@
"""End-to-end test for the sloppak loader recognising an `original_audio:`
manifest key (the single full-mix file shipped alongside the separate stems)
and surfacing the manifest-relative path on the LoadedSloppak."""
from __future__ import annotations
import json
from pathlib import Path
import yaml
import sloppak as sloppak_mod
def _write_dir_sloppak(root: Path, manifest_extras: dict, *, write_full_mix: bool) -> Path:
"""Build a minimal directory-form sloppak that load_song will accept.
Uses the tmp_path leaf name to make the sloppak filename unique per test,
avoiding the module-level ``resolve_source_dir`` cache being poisoned by a
previous test that happened to share the same "song.sloppak" filename.
"""
pak = root / f"{root.name}.sloppak"
pak.mkdir()
arr_dir = pak / "arrangements"
arr_dir.mkdir()
arr = {
"name": "Lead",
"tuning": [0, 0, 0, 0, 0, 0],
"capo": 0,
"notes": [],
"chords": [],
"anchors": [],
"handshapes": [],
"templates": [],
"beats": [],
"sections": [],
}
(arr_dir / "lead.json").write_text(json.dumps(arr))
manifest = {
"title": "Test",
"artist": "Tester",
"album": "",
"year": 2026,
"duration": 10.0,
"arrangements": [{"id": "lead", "name": "Lead", "file": "arrangements/lead.json"}],
"stems": [{"id": "guitar", "file": "stems/guitar.ogg", "default": True}],
}
manifest.update(manifest_extras)
(pak / "manifest.yaml").write_text(yaml.safe_dump(manifest, sort_keys=False))
if write_full_mix:
orig_dir = pak / "original"
orig_dir.mkdir()
# The loader only checks presence (is_file); contents are irrelevant.
(orig_dir / "full.ogg").write_bytes(b"OggS-not-real")
return pak
def _load(pak_path: Path, tmp_path: Path):
dlc_root = pak_path.parent
cache = tmp_path / "cache"
cache.mkdir()
return sloppak_mod.load_song(pak_path.name, dlc_root, cache)
# ── Happy path ───────────────────────────────────────────────────────────────
def test_load_song_attaches_original_audio_when_manifest_opts_in(tmp_path: Path):
pak = _write_dir_sloppak(
tmp_path, {"original_audio": "original/full.ogg"}, write_full_mix=True
)
loaded = _load(pak, tmp_path)
# Stored as the manifest-relative string so server.py can build the URL the
# same way it builds stem URLs.
assert loaded.original_audio == "original/full.ogg"
# ── Absent / degraded branches ───────────────────────────────────────────────
def test_load_song_original_audio_none_when_manifest_silent(tmp_path: Path):
pak = _write_dir_sloppak(tmp_path, {}, write_full_mix=True)
loaded = _load(pak, tmp_path)
assert loaded.original_audio is None
def test_load_song_original_audio_none_when_file_missing(tmp_path: Path):
# Manifest points at a full mix that isn't on disk — disabled silently.
pak = _write_dir_sloppak(
tmp_path, {"original_audio": "original/full.ogg"}, write_full_mix=False
)
loaded = _load(pak, tmp_path)
assert loaded.original_audio is None
def test_load_song_original_audio_none_when_value_blank(tmp_path: Path):
pak = _write_dir_sloppak(tmp_path, {"original_audio": " "}, write_full_mix=True)
loaded = _load(pak, tmp_path)
assert loaded.original_audio is None
# ── Security / path-traversal branches ──────────────────────────────────────
def test_load_song_original_audio_none_when_path_escapes_sloppak(tmp_path: Path):
pak = _write_dir_sloppak(
tmp_path, {"original_audio": "../outside.ogg"}, write_full_mix=True
)
loaded = _load(pak, tmp_path)
assert loaded.original_audio is None
def test_load_song_original_audio_none_when_path_is_absolute(tmp_path: Path):
pak = _write_dir_sloppak(
tmp_path, {"original_audio": "/etc/passwd"}, write_full_mix=True
)
loaded = _load(pak, tmp_path)
assert loaded.original_audio is None
+309
View File
@@ -0,0 +1,309 @@
"""The unpack cache is bounded, and reading part of a song doesn't explode it.
`sloppak_cache/` holds every song ever unpacked, fully decompressed. Stems are
already-compressed audio, so an unpacked song is ~1.1x its zip the cache is a
second copy of the library. It used to have no cap, no LRU, and no cleanup at
all: a tester reached 60 GB from an 1800-song library because one caller looped
the library calling load_song() (rig_builder's library-wide tone batch), which
unpacks the WHOLE pack stems included to read a few KB of tone JSON.
Pins, so neither half can silently come back:
- resolve_source_dir() evicts LRU songs to stay under the cap,
- it never evicts the song the caller just asked for,
- an evicted song is dropped from _source_cache too (otherwise the media route
keeps serving a path that no longer exists and 404s every stem instead of
re-unpacking),
- get_cached_source_dir() self-heals if the cache dir is deleted by hand,
- read_member_bytes() reads one file WITHOUT unpacking anything.
"""
import importlib
import zipfile
import pytest
import yaml
import sloppak as sloppak_mod
STEM = b"\x00" * (400 * 1024) # 400 KB of "audio" — the bulk of a real pack
ARR = b'{"tones": {"definitions": [{"Key": "clean"}]}}'
def _zip_pack(path, stem_bytes=STEM):
with zipfile.ZipFile(path, "w") as zf:
zf.writestr("manifest.yaml", yaml.safe_dump({
"title": path.stem,
"arrangements": [{"file": "arrangements/lead.json", "name": "Lead"}],
"stems": [{"id": "full", "file": "stems/audio.ogg"}],
}))
zf.writestr("arrangements/lead.json", ARR)
zf.writestr("stems/audio.ogg", stem_bytes)
return path
@pytest.fixture(autouse=True)
def _fresh_module_state():
# _source_cache is module state and would leak across tests.
importlib.reload(sloppak_mod)
yield
importlib.reload(sloppak_mod)
def _cap_mb(monkeypatch, mb):
monkeypatch.setenv("FEEDBACK_SLOPPAK_CACHE_MAX_MB", str(mb))
def test_read_member_bytes_does_not_unpack(tmp_path, monkeypatch):
dlc = tmp_path / "dlc"
dlc.mkdir()
cache = tmp_path / "cache"
cache.mkdir()
pack = _zip_pack(dlc / "song.feedpak")
data = sloppak_mod.read_member_bytes(pack, "arrangements/lead.json")
assert data == ARR
assert list(cache.iterdir()) == [], (
"reading one member must not unpack the pack — this is the whole point: "
"load_song() would have written the 400 KB stem to disk to get 45 bytes of JSON"
)
def test_read_member_bytes_missing_member_is_none(tmp_path):
dlc = tmp_path / "dlc"
dlc.mkdir()
pack = _zip_pack(dlc / "song.feedpak")
assert sloppak_mod.read_member_bytes(pack, "arrangements/nope.json") is None
assert sloppak_mod.read_member_bytes(pack, "") is None
def test_unpack_cache_evicts_lru_to_stay_under_cap(tmp_path, monkeypatch):
dlc = tmp_path / "dlc"
dlc.mkdir()
cache = tmp_path / "cache"
cache.mkdir()
_cap_mb(monkeypatch, 1) # 1 MB — holds ~2 of our 400 KB packs
for i in range(6):
_zip_pack(dlc / f"song{i}.feedpak")
for i in range(6):
sloppak_mod.resolve_source_dir(f"song{i}.feedpak", dlc, cache)
total = sum(f.stat().st_size for f in cache.rglob("*") if f.is_file())
assert total <= 1 * 1024 * 1024, (
f"unpack cache ran to {total/1e6:.1f} MB against a 1 MB cap — this is the "
"unbounded growth that reached 60 GB in the field"
)
# The most recent song must survive; the oldest must not.
names = {d.name for d in cache.iterdir()}
assert "song5.feedpak" in names, "the song just resolved must never be evicted"
assert "song0.feedpak" not in names, "the least-recently-used song should go first"
def test_eviction_drops_the_source_cache_entry(tmp_path, monkeypatch):
"""An evicted song must not keep being handed out by get_cached_source_dir().
media.py only falls back to resolve_source_dir() when this returns None. If a
stale path survives, every stem 404s for the rest of the process instead of
re-unpacking a silently broken song, not a slow one.
"""
dlc = tmp_path / "dlc"
dlc.mkdir()
cache = tmp_path / "cache"
cache.mkdir()
_cap_mb(monkeypatch, 1)
for i in range(6):
_zip_pack(dlc / f"song{i}.feedpak")
for i in range(6):
sloppak_mod.resolve_source_dir(f"song{i}.feedpak", dlc, cache)
evicted = sloppak_mod.get_cached_source_dir("song0.feedpak")
assert evicted is None, "an evicted song must be dropped from _source_cache"
# ...and asking for it again just re-unpacks it. Self-healing, not broken.
again = sloppak_mod.resolve_source_dir("song0.feedpak", dlc, cache)
assert (again / "stems" / "audio.ogg").is_file()
def test_get_cached_source_dir_self_heals_after_manual_delete(tmp_path, monkeypatch):
"""Telling a user to delete sloppak_cache/ to reclaim disk must be safe."""
import shutil
dlc = tmp_path / "dlc"
dlc.mkdir()
cache = tmp_path / "cache"
cache.mkdir()
_cap_mb(monkeypatch, 0) # eviction off — isolate the delete
_zip_pack(dlc / "song.feedpak")
src = sloppak_mod.resolve_source_dir("song.feedpak", dlc, cache)
assert sloppak_mod.get_cached_source_dir("song.feedpak") == src
shutil.rmtree(src) # the user clears the folder
assert sloppak_mod.get_cached_source_dir("song.feedpak") is None, (
"a path that no longer exists must not be served — the caller would 404 "
"every stem instead of re-unpacking"
)
assert (sloppak_mod.resolve_source_dir("song.feedpak", dlc, cache)
/ "stems" / "audio.ogg").is_file()
def test_cap_of_zero_disables_eviction(tmp_path, monkeypatch):
dlc = tmp_path / "dlc"
dlc.mkdir()
cache = tmp_path / "cache"
cache.mkdir()
_cap_mb(monkeypatch, 0)
for i in range(4):
_zip_pack(dlc / f"song{i}.feedpak")
for i in range(4):
sloppak_mod.resolve_source_dir(f"song{i}.feedpak", dlc, cache)
assert len(list(cache.iterdir())) == 4, "cap 0 must mean 'never evict'"
def test_read_member_bytes_normalizes_non_canonical_names(tmp_path):
"""A manifest may name a member './arrangements/lead.json' — valid, and it
resolved fine once unpacked. Reading the zip member by the raw string would
KeyError and silently report no tones. Same trap read_cover_bytes already hit."""
dlc = tmp_path / "dlc"
dlc.mkdir()
pack = _zip_pack(dlc / "song.feedpak")
assert sloppak_mod.read_member_bytes(pack, "./arrangements/lead.json") == ARR
assert sloppak_mod.read_member_bytes(pack, "stems/../arrangements/lead.json") == ARR
assert sloppak_mod.read_member_bytes(pack, "arrangements\\lead.json") == ARR
def test_read_member_bytes_rejects_zip_slip(tmp_path):
dlc = tmp_path / "dlc"
dlc.mkdir()
pack = _zip_pack(dlc / "song.feedpak")
assert sloppak_mod.read_member_bytes(pack, "../../etc/passwd") is None
assert sloppak_mod.read_member_bytes(pack, "/etc/passwd") is None
assert sloppak_mod.read_member_bytes(pack, ".") is None
def test_eviction_never_deletes_an_in_flight_unpack(tmp_path, monkeypatch):
"""Two unpacks run concurrently (_UNPACK_MAX_CONCURRENCY = 2). One finishing
must not rmtree the other's half-written dir — that resolver would then cache
an incomplete song and serve a broken pack.
Sized so the sweep genuinely has to reach the in-flight directory: each pack
is ~700 KB against a 1 MB cap, so once `keep` is protected the sweep must
delete EVERY other dir to get under the cap including the one being written.
(A naive version of this test passes even without the guard, because a
freshly-created dir is the most-recently-used and the sweep never gets to it.)
"""
import threading
big = b"\x00" * (700 * 1024)
dlc = tmp_path / "dlc"
dlc.mkdir()
cache = tmp_path / "cache"
cache.mkdir()
_cap_mb(monkeypatch, 1)
for i in range(3):
_zip_pack(dlc / f"song{i}.feedpak", stem_bytes=big)
victim = cache / "song2.feedpak"
started = threading.Event()
release = threading.Event()
real_unpack = sloppak_mod._unpack_zip
def slow_unpack(zip_path, dest):
real_unpack(zip_path, dest) # dir now exists — "half written"
if dest == victim:
started.set()
release.wait(5) # hold it open while the other sweeps
monkeypatch.setattr(sloppak_mod, "_unpack_zip", slow_unpack)
t = threading.Thread(target=sloppak_mod.resolve_source_dir,
args=("song2.feedpak", dlc, cache))
t.start()
assert started.wait(5), "victim unpack did not start"
# song0 lands and sweeps: keep=song0, cache holds song0+song2 = 1.4 MB > 1 MB,
# so the sweep MUST try to delete song2 — which is still being written.
sloppak_mod.resolve_source_dir("song0.feedpak", dlc, cache)
in_flight_survived = victim.is_dir()
release.set()
t.join(5)
assert in_flight_survived, (
"eviction deleted a directory another thread was still unpacking into — "
"that resolver caches an incomplete song and serves a broken pack"
)
def test_read_member_bytes_finds_backslash_members(tmp_path):
"""Windows-authored packs store members as 'arrangements\\lead.json'.
_unpack_zip() normalizes those on extract, so unpack-then-read found them.
An exact getinfo() would not and we'd silently report the song has no tones."""
dlc = tmp_path / "dlc"
dlc.mkdir()
pack = dlc / "win.feedpak"
with zipfile.ZipFile(pack, "w") as zf:
zf.writestr("manifest.yaml", yaml.safe_dump({"title": "w"}))
zf.writestr("arrangements\\lead.json", ARR) # backslash member name
assert sloppak_mod.read_member_bytes(pack, "arrangements/lead.json") == ARR
def test_read_member_bytes_finds_non_canonical_STORED_names(tmp_path):
"""The archive itself may store './arrangements/lead.json'. _unpack_zip()
normalizes stored names on extract, so unpack-then-read resolved it. Both the
requested path and the stored name must be normalized, or the tones vanish."""
dlc = tmp_path / "dlc"
dlc.mkdir()
pack = dlc / "odd.feedpak"
with zipfile.ZipFile(pack, "w") as zf:
zf.writestr("manifest.yaml", yaml.safe_dump({"title": "o"}))
zf.writestr("./arrangements/lead.json", ARR) # stored non-canonically
assert sloppak_mod.read_member_bytes(pack, "arrangements/lead.json") == ARR
def test_read_member_bytes_matches_unpack_last_write_wins(tmp_path):
"""If a pack stores two names that normalize to the same file, _unpack_zip
writes them in order and the LAST one is what ends up on disk. Reading the
raw member by exact name would hand back the first stale arrangement data
that no unpacked read would ever have produced."""
dlc = tmp_path / "dlc"
dlc.mkdir()
pack = dlc / "dupe.feedpak"
with zipfile.ZipFile(pack, "w") as zf:
zf.writestr("manifest.yaml", yaml.safe_dump({"title": "d"}))
zf.writestr("arrangements/lead.json", b'{"tones": {"definitions": [{"Key": "STALE"}]}}')
zf.writestr("./arrangements/lead.json", ARR) # normalizes to the same path
assert sloppak_mod.read_member_bytes(pack, "arrangements/lead.json") == ARR
def test_failed_unpack_does_not_leave_the_dir_un_evictable(tmp_path, monkeypatch):
"""A dir marked in-flight is skipped by eviction. If a failed unpack leaves the
marker behind, that dir becomes permanently un-evictable a slow leak of
exactly the thing this cap exists to prevent."""
dlc = tmp_path / "dlc"
dlc.mkdir()
cache = tmp_path / "cache"
cache.mkdir()
_cap_mb(monkeypatch, 1)
_zip_pack(dlc / "boom.feedpak")
def blow_up(zip_path, dest):
dest.mkdir(parents=True, exist_ok=True)
raise OSError("disk full")
monkeypatch.setattr(sloppak_mod, "_unpack_zip", blow_up)
with pytest.raises(OSError):
sloppak_mod.resolve_source_dir("boom.feedpak", dlc, cache)
assert not sloppak_mod._unpacking, (
"a failed unpack left its destination marked in-flight — eviction will "
"skip it forever"
)
+5
View File
@@ -480,6 +480,11 @@ def test_seconds_only_post_accrues_without_touching_position(client):
# overwrite Continue with the end-of-song offset).
assert row["plays"] == 0
assert row["last_position"] == pytest.approx(42.0)
# Recency must come from the seconds-only POST itself — prove it on a
# FRESH row (the position touch above already stamps last_played_at,
# which would make an assertion here vacuous).
r2 = client.post("/api/stats", json={"filename": "fresh.archive", "seconds": 30})
assert r2.json()["stats"]["last_played_at"]
# Still counts as playing today for the streak.
assert r.json()["progress"]["current_streak"] == 1
+445
View File
@@ -0,0 +1,445 @@
#!/usr/bin/env python3
"""Migrate packs off the deprecated `original_audio:` key — the full mix is a stem.
Before feedpak 1.15.0 reserved the stem id `full`, spec §5.3 said the mixdown was
"commonly replaced" by the per-instrument stems when a pack was split so after
separation it had nowhere to live. This repo worked around that by inventing a
top-level `original_audio:` manifest key pointing at a parallel `original/`
directory (#583). The key was never in the spec, and #933 removed core's
dependence on it: the mixdown is a stem, and its id is `full`.
This rewrites a pack into the shape the spec now defines:
original/full.ogg -> stems/full.ogg (entry moved)
original_audio: original/full.ogg -> stems: [{id: full, file: stems/full.ogg,
default: 'off'}, ...]
`default: 'off'` is what makes the retained mixdown safe: a reader that honours
`default` (normative since feedpak 1.0.0) will not play it on open, so it never
doubles the mix even in a reader that predates the reserved id.
Nothing else in the pack is touched every other key, file and stem is preserved
verbatim, and `feedpak_version` is stamped to the version the result conforms to.
The rewrite is atomic per pack: a new archive is built beside the original and
renamed over it only on success, so an interrupted run leaves every pack either
fully migrated or untouched never truncated.
Idempotent: a pack that already carries a `full` stem and no `original_audio:` is
reported as `skip` and left alone, so a partial run can simply be re-run.
Usage:
python tools/migrate_full_mix_stem.py --dry-run /path/to/packs # report only
python tools/migrate_full_mix_stem.py /path/to/packs # migrate
python tools/migrate_full_mix_stem.py --verify /path/to/packs # check results
Exit status is 0 only when every pack ended up in the migrated shape (or was
already there).
"""
from __future__ import annotations
import argparse
import os
import shutil
import sys
import tempfile
import threading
import time
import zipfile
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import yaml
# The version this migration brings a pack up to: the one that reserved `full`.
TARGET_FEEDPAK_VERSION = "1.15.0"
FULL_MIX_STEM_ID = "full"
LEGACY_KEY = "original_audio"
# Where the mixdown lands. §2.1's conventional layout — readers resolve through
# the manifest and never care about the path, but a pack that says `stems/` and
# means it is the one a human can read.
CANONICAL_FULL_MIX_PATH = "stems/full.ogg"
PACK_EXTS = (".feedpak", ".sloppak")
class Skip(Exception):
"""Pack needs no migration."""
def is_safe_relpath(rel: str) -> bool:
"""True when `rel` is a manifest path the spec allows (§2.2 rule 2).
POSIX-style relative: forward slashes, no leading `/`, no `..` segments, no
empty segments, no colon (which excludes drive letters and NTFS alternate
data streams), no backslashes.
This is a TRUST BOUNDARY, not a tidiness check. Core's loader refuses a
full-mix path that escapes the pack and reports the pack as having no full
mix the audio is inert. A migration that moved such an entry into
`stems/full.ogg` would take content the reader deliberately rejected and
hand it back as a valid, playable stem. So a pack like this is reported, not
migrated.
"""
if not rel or rel.startswith("/") or "\\" in rel or ":" in rel:
return False
parts = rel.split("/")
return all(p and p != ".." for p in parts)
def plan_manifest(manifest: dict) -> tuple[dict, str]:
"""Return (new_manifest, relpath_of_audio_to_move); "" = no file needs moving.
Raises Skip when the pack needs no migration. Pure no I/O so the part
with the decisions in it is testable without building archives.
"""
raw_stems = manifest.get("stems")
stems: list = raw_stems if isinstance(raw_stems, list) else []
has_full_stem = any(
isinstance(s, dict) and str(s.get("id", "")) == FULL_MIX_STEM_ID for s in stems
)
legacy_rel = manifest.get(LEGACY_KEY)
legacy_rel = legacy_rel.strip() if isinstance(legacy_rel, str) else ""
if not legacy_rel:
# Nothing invented to undo: either the pack already keeps its mixdown as
# the `full` stem, or it never carried one.
raise Skip("already migrated" if has_full_stem else "no original_audio key")
if has_full_stem:
# The mixdown is already a stem and the dead key merely lingers beside it.
# Drop the key; move nothing. But do NOT trust its `default`: a mixdown
# left enabled beside instrument stems is the double-audio hazard this
# migration exists to remove, and a reader that honours `default` would
# play the whole song on top of the stems on open. Force it off — unless
# `full` is the only stem, in which case it IS the audio.
others = [
s
for s in stems
if isinstance(s, dict) and str(s.get("id", "")) != FULL_MIX_STEM_ID
]
new_stems = []
for s in stems:
if isinstance(s, dict) and str(s.get("id", "")) == FULL_MIX_STEM_ID and others:
s = {**s, "default": "off"}
new_stems.append(s)
to_move = ""
else:
# `default` decides whether a reader plays this on open, and that is the
# whole safety margin: alongside per-instrument stems the mixdown must be
# OFF (a reader that sums the list would otherwise double the song), but
# when it is the pack's only stem it IS the audio and must be ON.
entry = {
"id": FULL_MIX_STEM_ID,
"file": CANONICAL_FULL_MIX_PATH,
"default": "off" if stems else "on",
}
# First in the list, matching the spec's §5.3 example.
new_stems = [entry, *stems]
# If the key already pointed at the canonical path, only the manifest is wrong.
to_move = "" if legacy_rel == CANONICAL_FULL_MIX_PATH else legacy_rel
out: dict = {}
for k, v in manifest.items():
if k == LEGACY_KEY:
continue # the invented key disappears
out[k] = new_stems if k == "stems" else v
out.setdefault("stems", new_stems) # a pack that had no stems list gets one
out["feedpak_version"] = TARGET_FEEDPAK_VERSION
return out, to_move
def migrate_zip(path: Path, dry_run: bool) -> str:
"""Rewrite one zipped pack in place. Returns a one-word status."""
with zipfile.ZipFile(path) as zf:
try:
raw = zf.read("manifest.yaml")
except KeyError:
return "no-manifest"
manifest = yaml.safe_load(raw) or {}
try:
new_manifest, old_rel = plan_manifest(manifest)
except Skip:
return "skip"
names = set(zf.namelist())
if old_rel:
if not is_safe_relpath(old_rel):
# Core refuses this path and plays no full mix for the pack. Do
# not launder it into a valid stem — see is_safe_relpath().
return "unsafe-path"
if old_rel not in names:
# The key points at audio that isn't in the archive. Core already
# treats that as "no full mix"; migrating would fabricate a stem
# entry for a file that does not exist and break every reader.
return "missing-audio"
if CANONICAL_FULL_MIX_PATH in names:
return "target-occupied"
else:
# Manifest-only rewrite (a stale key beside a mixdown that is already
# a stem, or a key that already pointed at the canonical path). Check
# the file the resulting `full` stem will actually NAME — not the
# canonical path, which an already-migrated pack is free not to use:
# §2.2 says readers resolve through the manifest, so a valid pack may
# keep its mixdown anywhere.
full_file = next(
(
s.get("file")
for s in new_manifest.get("stems", [])
if isinstance(s, dict) and str(s.get("id", "")) == FULL_MIX_STEM_ID
),
None,
)
if full_file not in names:
return "missing-audio"
if dry_run:
return "would-migrate"
# Build the replacement beside the original, on the same filesystem, so
# the final rename is atomic and an interrupted run can't truncate a pack.
tmp_fd, tmp_name = tempfile.mkstemp(
dir=str(path.parent), prefix=f".{path.name}.", suffix=".tmp"
)
os.close(tmp_fd)
tmp_path = Path(tmp_name)
try:
with zipfile.ZipFile(tmp_path, "w", zipfile.ZIP_DEFLATED) as out:
for item in zf.infolist():
if item.filename == "manifest.yaml":
out.writestr(
item,
yaml.safe_dump(
new_manifest, sort_keys=False, allow_unicode=True
),
)
continue
data = zf.read(item.filename)
if old_rel and item.filename == old_rel:
# Same bytes, same compression, new name: the mixdown moves
# from original/ into stems/ where the format says audio goes.
moved = zipfile.ZipInfo(
CANONICAL_FULL_MIX_PATH, date_time=item.date_time
)
moved.compress_type = item.compress_type
moved.external_attr = item.external_attr
out.writestr(moved, data)
continue
out.writestr(item, data)
except BaseException:
tmp_path.unlink(missing_ok=True)
raise
shutil.copystat(path, tmp_path)
os.replace(tmp_path, path) # atomic
return "migrated"
def verify_zip(path: Path) -> str:
"""Confirm a pack is in the migrated shape and its mixdown is really there."""
with zipfile.ZipFile(path) as zf:
try:
manifest = yaml.safe_load(zf.read("manifest.yaml")) or {}
except KeyError:
return "no-manifest"
if LEGACY_KEY in manifest:
return "still-has-key"
stems = manifest.get("stems") or []
full = next(
(
s
for s in stems
if isinstance(s, dict) and str(s.get("id", "")) == FULL_MIX_STEM_ID
),
None,
)
if full is None:
return "no-full-stem"
if full.get("file") not in set(zf.namelist()):
return "full-stem-missing-file"
# A retained mixdown that plays on open would double the mix in any reader
# that sums the stem list — the whole hazard this migration must not create.
# Beside instrument stems, `full` MUST carry an explicit, normalized "off":
# - core (lib/sloppak.py) defaults an ABSENT `default` to True (ON) and
# treats an empty / unrecognized string as ON, so a missing or blank
# default is not merely non-canonical — core would play the mixdown on
# open, doubling the song. It is the exact hazard, not a lesser one.
# - the migrator always writes the literal "off", so requiring it also
# certifies the pack is in the shape this tool produces — the most
# portable spelling, understood even by a reader that only knows
# "on"/"off" and would choke on a boolean or `false`/`0`/`no`.
# So: `on`-ish values are reported as actively-playing; everything that is
# not a normalized "off" (missing, empty, boolean, `false`/`no`/`0`,
# malformed) is reported as an unsafe/non-canonical default.
if len(stems) > 1:
default = str(full.get("default", "")).strip().lower()
if default in ("true", "on", "yes", "1"):
return "full-stem-default-on"
if default != "off":
return "full-stem-default-not-off"
return "ok"
def migrate_pack(path: Path, dry_run: bool) -> str:
"""Dispatch by pack form. ZIP-file packs are rewritten in place; directory
(authoring) packs are REPORTED, not rewritten.
A single-file pack is replaced atomically a fully-built temp archive
swapped in with one os.replace(), so an interrupted run leaves it either
fully migrated or untouched. A directory can't be swapped that way (no
atomic replace of a populated directory), so an in-place rewrite could leave
an authoring pack half-migrated. Rather than risk that, directory packs are
surfaced as `dir-form-unsupported` (a problem status, so the run's exit code
and summary flag them) for the operator to re-pack or migrate as a `.feedpak`.
"""
if path.is_dir():
return "dir-form-unsupported"
return migrate_zip(path, dry_run)
def verify_pack(path: Path) -> str:
"""Verify a pack; directory (authoring) packs are reported, see migrate_pack."""
if path.is_dir():
return "dir-form-unsupported"
return verify_zip(path)
def iter_packs(root: Path):
"""Yield every pack under `root`. A pack is a suffix-named ZIP FILE or a
suffix-named DIRECTORY (the authoring form) both are discovered so a
directory-form pack is never silently walked past. A directory pack is
yielded whole, not descended into: its `stems/` and `arrangements/` are pack
contents, not packs. (migrate/verify then report directory packs rather than
rewriting them in place see migrate_pack.)"""
if root.is_file():
yield root
return
# A directory whose OWN name is a pack suffix is a single directory-form
# pack passed directly, not a tree of packs to search.
if root.name.endswith(PACK_EXTS):
yield root
return
for dirpath, dirnames, filenames in os.walk(root):
for fn in sorted(filenames):
if fn.endswith(PACK_EXTS):
yield Path(dirpath) / fn
for dn in sorted(dn for dn in dirnames if dn.endswith(PACK_EXTS)):
yield Path(dirpath) / dn
# Don't descend INTO a pack directory — its contents aren't packs.
dirnames[:] = [dn for dn in dirnames if not dn.endswith(PACK_EXTS)]
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("root", type=Path, help="pack, or directory of packs")
ap.add_argument("--dry-run", action="store_true", help="report, change nothing")
ap.add_argument("--verify", action="store_true", help="check the migrated shape")
ap.add_argument("--jobs", type=int, default=8, help="parallel packs (default 8)")
args = ap.parse_args(argv)
if not args.root.exists():
print(f"error: {args.root} does not exist", file=sys.stderr)
return 2
packs = list(iter_packs(args.root))
if not packs:
print(f"no packs found under {args.root}", file=sys.stderr)
return 2
action = verify_pack if args.verify else (lambda p: migrate_pack(p, args.dry_run))
def work(pack: Path) -> str:
"""Never raise. One unreadable pack must not kill a 50,000-pack run.
A library this size has damage in it a truncated download, an archive
left half-written by an interrupted converter. Letting that propagate
aborts the whole job partway through and throws away the summary, which
is exactly when you most need to know what happened. Report it as a
problem status instead: the pack is untouched, the run continues, and the
final report names it.
"""
try:
return action(pack)
except zipfile.BadZipFile:
return "corrupt-zip"
except OSError as e:
return f"io-error ({e.__class__.__name__})"
except Exception as e: # malformed YAML, unexpected manifest shape, …
return f"error ({e.__class__.__name__})"
counts: dict[str, int] = {}
problems: list[tuple[str, Path]] = []
# A real run rewrites every archive under `root` — tens of thousands of packs
# and hundreds of gigabytes. Printing only a final summary means hours of
# silence, in which a stall and steady progress look identical. Emit a
# heartbeat instead: rate and ETA come from the packs actually finished, so
# it stays honest when the disk slows down. stderr, so `> report.txt` keeps
# the summary clean.
total = len(packs)
started = time.monotonic()
# The heartbeat runs on its OWN CLOCK, in its own thread.
#
# Two weaker designs were tried and both go quiet exactly when you need them
# to speak. Ticking every N packs ties the cadence to how slow a pack is: 500
# packs is a blink in a --dry-run and many minutes in a real migration, so the
# run that most needs watching says the least. Ticking on time but only when a
# pack *finishes* is no better: if every worker is grinding on a huge archive,
# nothing completes, so nothing prints — and a stall becomes indistinguishable
# from progress, which is the one thing a progress meter must never allow.
#
# A daemon thread on a fixed interval reports regardless. If the count stops
# advancing between beats, you are looking at a stall, and you can see it.
HEARTBEAT_SECONDS = 10.0
done = 0 # only the main loop writes it; the beat thread only reads
stop_beat = threading.Event()
def heartbeat() -> None:
while not stop_beat.wait(HEARTBEAT_SECONDS):
elapsed = time.monotonic() - started
rate = done / elapsed if elapsed > 0 else 0.0
eta = (total - done) / rate if rate > 0 else 0.0
print(
f" {done}/{total} ({100 * done / total:.1f}%) "
f"{rate:.1f} packs/s eta {eta / 60:.0f}m "
f"[{len(problems)} problem(s)]",
file=sys.stderr,
flush=True,
)
print(
f"{total} pack(s) under {args.root}"
f"{'verifying' if args.verify else 'dry run' if args.dry_run else 'migrating'} "
f"with {max(1, args.jobs)} job(s)",
file=sys.stderr,
flush=True,
)
beat = threading.Thread(target=heartbeat, daemon=True)
beat.start()
# as_completed, not pool.map: map yields in SUBMISSION order, so the counter
# would stall behind one slow pack while later ones were already done — a
# progress meter that lies about progress. Count each pack as it finishes.
try:
with ThreadPoolExecutor(max_workers=max(1, args.jobs)) as pool:
futures = {pool.submit(work, p): p for p in packs}
for fut in as_completed(futures):
pack = futures[fut]
status = fut.result()
counts[status] = counts.get(status, 0) + 1
if status not in ("migrated", "skip", "would-migrate", "ok"):
problems.append((status, pack))
done += 1
finally:
stop_beat.set()
beat.join(timeout=1)
print(f"\n{len(packs)} pack(s) under {args.root}")
for status, n in sorted(counts.items(), key=lambda kv: -kv[1]):
print(f" {n:>7} {status}")
if problems:
print(f"\n{len(problems)} pack(s) need a look:", file=sys.stderr)
for status, pack in problems[:20]:
print(f" {status:<22} {pack}", file=sys.stderr)
if len(problems) > 20:
print(f" … and {len(problems) - 20} more", file=sys.stderr)
return 1 if problems else 0
if __name__ == "__main__":
raise SystemExit(main())