mirror of
https://github.com/got-feedBack/feedBack.git
synced 2026-08-11 03:09:57 +00:00
4a1b055afdba03140bd8e7bab84a812e659d22f2
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
165475d115 |
refactor(server): move the metadata-enrichment subsystem into lib/enrichment.py (R3) (#861)
* refactor(server): move the metadata-enrichment subsystem into lib/enrichment.py (R3)
MusicBrainz / Cover-Art-Archive / AcoustID transport, the match-scorer glue, and
the background enrichment worker (~930 lines, 61 defs) leave server.py as one
cohesive unit. Bodies are verbatim; the only changes are seam reads:
meta_db / config_dir / sloppak_cache_dir / art_cache_dir -> appstate.<slot>
_song_pack_art_exists / _art_override_paths (stay in server.py for the art +
delete routes) -> appstate.<callable> (new seam slots, injected by reference)
_env_flag -> env_compat.env_flag_compat (the existing identical helper)
_artist_title_from_filename -> imported from metadata_db (its home)
the User-Agent VERSION lookup: Path(__file__).parent ->
Path(__file__).resolve().parents[1] (lib/enrichment.py -> app root)
server.py drives the worker through the module (import enrichment; the routes +
scan lifecycle call enrichment.X). Tests that faked the network on `server`
(_mb_http_get, _enrich_network_enabled, _caa_http_get, ...) now patch the same
names on `enrichment` — the module attribute is resolved at call time, so one
setattr reaches both the routes and the worker's internal callers. Acyclic:
enrichment imports appstate/appconfig/dlc_paths/metadata_db/mb_match/
acoustid_match/sloppak/loosefolder, never server.
server.py: 6,917 -> 5,988 (-929).
Verified: pyflakes clean (bar the pre-existing File/safe_join/tuning_name/ET);
route table IDENTICAL (143); full pytest 2400 passed (140 enrichment/art cases
incl the offline-safety + transport-error-pauses-pass contracts that fake the
network); test_packaging 44 passed (enrichment.py resolves under lib/); eslint 0.
Boot smoke: /api/enrichment/status + POST /kick serve from the new module.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: reset enrichment worker state between tests (CodeRabbit)
lib/enrichment.py now owns the worker, and it stays imported for the whole
session while the `server` fixtures pop-and-reimport `server` — so the cancel
Event / status dict / caches would leak across tests, and a stale `_enrich_cancel`
could short-circuit a later direct `_background_enrich()`. An autouse conftest
fixture clears that process-global state before each test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: tighten the enrichment-reset fixture (CodeRabbit)
Narrow the import guard to ImportError (not blind Exception, BLE001), and stop
clearing _caa_index_locks — it's guarded by _caa_index_locks_guard, so an
unlocked clear() would race a still-alive worker, and its per-release mutexes
carry no test state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
af1170cec3 |
feat(v3 library): "Fix metadata" popup — per-song override + lock, cover picker, MusicBrainz + AcoustID (#777)
* feat(library): metadata override + lock store, enforced by enrichment (popup slices 1–2)
Backend foundation for the Fix-metadata popup. Not yet surfaced in the UI (the
display + 3-tab popup are the next slices); no PR until it's user-visible.
Slice 1 — the store:
- `song_field_override(filename, field, value, locked)` table: a reversible
DISPLAY overlay (never written to the pack), filename-keyed so it survives a
rescan (never purged by delete_missing) and is dropped only with the song.
- DB methods (partial upsert that drops empty+unlocked rows; batch map) +
`GET`/`PUT /api/song/{fn}/overrides` (field allowlist title/artist/album/
year/genre; clearing rides PUT since DELETE /api/song/{path} shadows sub-
routes; PUT demo-blocked).
Slice 2 — locks respected by enrichment:
- The auto-matcher composes a per-song `_compose_lock_filter` onto the global
apply-filter, so a match still applies IDENTITY (mbid/release → art) but never
re-canonicalizes a LOCKED display field.
- Gap-fill (write-to-file) skips locked album/year/genre — writing the matched
value would be exactly the clobber the lock exists to prevent.
- Review/manual picks bypass the filter (an explicit confirm overrides a lock).
Tests: store semantics + rescan-survival + API; the lock filter + reader; an
auto-match leaving a locked field un-canonicalized; gap-fill excluding locked
keys.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(library): show per-song overrides in the grid (popup slice 3)
The grid now displays the user's per-song title/artist/album/year override
in place of the pack value ("grid shows only overrides") — a matched
MusicBrainz canon never silently re-titles a card; canon stays in the
Details drawer + art. Overlaid in Python over the visible window, keyset-safe
like the P4 artist-alias re-label: the seek still runs on the raw column, and
the one overridable keyset column (title) stashes its raw value for the cursor
so paging never skips/dupes. The private stash is dropped from the payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(library): 3-tab Fix-metadata popup — Details / Cover art / Match (slice 4)
Turns the thin single-song fix-match modal into the Plex-style metadata
editor reached from a card's "Fix metadata…" menu:
- Details tab: type + lock the displayed title/artist/album/year. Values
ride the reversible override store (GET/PUT /api/song/{fn}/overrides); each
field sits on its pack value (Yours/Pack provenance + revert-to-pack), a lock
pins it against auto-match, and Save repaints the grid via library:changed
(slice-3 overlay). This is the real tool for the blank-artist city-pop pile
MusicBrainz can't surface — you just type the right title.
- Cover art tab: hands off to the shared image picker (its own modal); the
pick refreshes the thumbnail everywhere.
- Match tab: the existing MusicBrainz search + candidate/pick flow, refactored
into shared body/footer helpers (the queue-review flow is untouched).
Backend: GET /overrides now also returns the pack baseline so the Details tab
can pre-fill + show provenance. tailwind.min.css regenerated (build-tailwind.sh)
for the popup's new utility classes.
Identify-by-audio (AcoustID) is deferred: it lives in unmerged PR #759, off
main — the Match tab gains the button once #759 lands and this branch rebases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(library): wire "Identify by audio" in the tabbed popup's Match tab
The AcoustID Identify button (#759) merged in referencing an out-of-scope
`panel` in the wiring — a leftover from the pre-popup fix-match modal that my
tab refactor renamed to `root`. Under strict mode that threw, so the handler
never attached and the button did nothing. Scope it to `root` (the tab body),
which is where the search-results area it renders into lives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(library): make "Identify by audio" outcomes unmistakable
An empty AcoustID result read the same as a broken button. Each state now says
plainly which outcome it is — ✓ fingerprinted-but-no-match vs no-audio vs off vs
unavailable — and, in the popup, points at the manual fallback (Search, or set
the album in Details + cover in Cover art by hand). A ✓ marks the states that
actually ran, so "worked, found nothing" no longer looks like a failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
74cff4e0d6 |
feat(enrichment): alias-aware scoring — auto-confirm non-Latin-primary artists (#772)
ship-ci / ci (push) Waiting to run
* feat(enrichment): loose MusicBrainz search fallback (find aliased artists)
The MB text search used a strict field-phrase query
(`recording:"<title>" AND artist:"<artist>"`). A field phrase only matches
MusicBrainz's *primary* artist/title — it never searches ALIASES — so a
recording stored under a non-Latin primary name (大橋純子) whose romanized
form ("Junko Ohashi") is only an alias returns ZERO results, even though MB
has it. Whole swaths of a community library (e.g. romanized J-pop / city-pop
charts) were unsearchable.
- `build_recording_query(..., loose=True)` drops the field scoping + phrases
for plain AND-ed term groups (`(telephone number) AND (junko ohashi)`),
which searches the whole document incl. aliases.
- `_mb_search_recordings` runs the strict query first (unchanged, high
precision) and only on an EMPTY result retries once with the loose query —
so mainstream matches are untouched and the extra throttled request is spent
only on a miss. Results are re-scored by rank_candidates, so recall goes up
without lowering match quality (auto-accept still needs the per-field floors).
Verified live: "Junko Ohashi / Telephone Number" and "Anri / Windy Summer"
(both 0 under the strict query) now surface the real records; "AC/DC /
Highway to Hell" still hits strict at score 1.0 with no loose retry.
Follow-up (separate): alias-aware SCORING so these can auto-confirm, not just
appear as manual candidates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(enrichment): alias-aware scoring (auto-confirm non-Latin-primary artists)
Builds on the loose-search fallback: that surfaces a recording stored under a
Japanese primary name (大橋純子) via its romanized alias, but the SCORER still
compared the reference ("Junko Ohashi") against the primary only → artist
similarity 0 → below the auto floor, so it could only ever be a manual
candidate, never an auto-fill.
- mb_match: `cand_artist_sim` takes the best similarity over the candidate's
primary name AND its `artist_aliases`; score_candidate + classify use it.
- server: `_mb_artist_aliases(id)` fetches an artist's aliases (one throttled
lookup, process-cached — a one-artist discography costs ONE request) and
`_alias_enrich` attaches them ONLY to promising near-misses (title agrees,
primary artist doesn't) so a normal pass spends zero extra requests. Wired
into both the auto-matcher (_enrich_one) and the manual search proxy.
Verified live: "Junko Ohashi / Telephone Number" → 大橋純子 candidate goes from
score 0.5 (loose-only) to 1.0 (auto-confirmable), ranked #1; "AC/DC / Highway
to Hell" unchanged at 1.0 with no alias lookup.
Stacks on #771 (feat/mb-loose-search-fallback).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(enrichment): keep live exclusion in the loose search fallback
The loose fallback dropped the strict path's -secondarytype:Live filter, so a
studio chart whose strict query missed could fall back to — and, since
score_candidate doesn't penalize live takes, auto-confirm — a live-only
recording. Apply the same live gate to the loose query (skipped only when the
source title is itself a live take).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: byrongamatos <xasiklas@gmail.com>
|
||
|
|
18c4e229e1 |
feat(enrichment): loose MusicBrainz search fallback (find aliased/romanized artists) (#771)
* feat(enrichment): loose MusicBrainz search fallback (find aliased artists)
The MB text search used a strict field-phrase query
(`recording:"<title>" AND artist:"<artist>"`). A field phrase only matches
MusicBrainz's *primary* artist/title — it never searches ALIASES — so a
recording stored under a non-Latin primary name (大橋純子) whose romanized
form ("Junko Ohashi") is only an alias returns ZERO results, even though MB
has it. Whole swaths of a community library (e.g. romanized J-pop / city-pop
charts) were unsearchable.
- `build_recording_query(..., loose=True)` drops the field scoping + phrases
for plain AND-ed term groups (`(telephone number) AND (junko ohashi)`),
which searches the whole document incl. aliases.
- `_mb_search_recordings` runs the strict query first (unchanged, high
precision) and only on an EMPTY result retries once with the loose query —
so mainstream matches are untouched and the extra throttled request is spent
only on a miss. Results are re-scored by rank_candidates, so recall goes up
without lowering match quality (auto-accept still needs the per-field floors).
Verified live: "Junko Ohashi / Telephone Number" and "Anri / Windy Summer"
(both 0 under the strict query) now surface the real records; "AC/DC /
Highway to Hell" still hits strict at score 1.0 with no loose retry.
Follow-up (separate): alias-aware SCORING so these can auto-confirm, not just
appear as manual candidates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(enrichment): keep live exclusion in the loose search fallback
The loose fallback dropped the strict path's -secondarytype:Live filter, so a
studio chart whose strict query missed could fall back to — and, since
score_candidate doesn't penalize live takes, auto-confirm — a live-only
recording. Apply the same live gate to the loose query (skipped only when the
source title is itself a live take).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: byrongamatos <xasiklas@gmail.com>
|
||
|
|
97a941c45d |
fix(tests): join background scan/enrich workers before closing the DB (#735)
Root cause of the flaky pytest segfault (exit 139): the background scan and enrichment daemon threads (_scan_runner/_enrich_runner) use the shared MetadataDB connection, but test fixtures closed that connection in teardown without stopping them. A daemon thread mid-query on a freed SQLite conn is a native use-after-free → SIGSEGV. The app's startup kicks a scan, so almost any app-booting fixture was vulnerable. It only surfaced now because got-feedback/feedBack#728 added a push trigger, so ci/test runs on every push to main. Fix: server.py retains the scan/enrich thread handles and adds _join_background_db_threads(); every test fixture now joins the workers before conn.close(). Verified: the full suite runs to completion (no segfault) where it previously crashed at ~25%. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
8e953e8bc4 |
library: opt-in gap-fill — write confirmed missing metadata into the pack (R4a) (#724)
* library: opt-in gap-fill — write confirmed missing metadata into the pack (R4a)
The write-back contract agreed with the spec chair (alignment doc §7),
made executable, now that feedpak-spec 1.14.0 (mbid/isrc) is merged:
opt-in + user-initiated, adds ABSENT keys only, spec'd-keys allowlist,
values only from a CONFIRMED identity, atomic write + .bak. Single-song
only — batch write-back stays an open question with the chair.
- songmeta.gap_fill_sloppak: append-only manifest writer. Every added
key is absent by definition, so the new lines are APPENDED — the
author's existing bytes (key order, comments, formatting) survive
verbatim, unlike the metadata editor's full re-serialize. Directory
form gets a one-time manifest.yaml.bak + temp + atomic replace; zip
form reuses the editor's backup/temp/replace rewriter. Raises on any
already-present key: the never-clobber rule lives in the writer, not
just the callers.
- GET /api/song/{fn}/gap-fill: read-only preview — which of
album/year/genres/mbid/isrc are missing from the file (absent or
empty; year 0 = empty), with the values the enrichment match
supplies. Only a CONFIRMED identity is eligible (matched or a user
pin); review-tier rows are refused until a human confirms —
wrong-match > fast, same as everywhere else in the enrichment layer.
- POST /api/song/{fn}/gap-fill {keys}: writes the user-confirmed
subset. Proposals are RECOMPUTED under _song_io_lock, so a key that
gained an author value between preview and confirm is skipped, never
replaced. mbid/isrc written in canonical form only (validated).
DB stays scanner-consistent (album/year/genre columns + mtime/size
re-stat, cache invalidation + scan kick — the metadata editor's
contract). Demo mode blocks the write.
- Details drawer (Identity section): "Write missing info to file…" →
per-key checkbox confirm ("Only adds what's missing — nothing already
in the file is changed. A backup (.bak) is kept.") → written
confirmation; not-eligible states explain themselves. v3 only; no
new tailwind classes.
- Rides along: _manifest_exact_ids now strips ISRC display separators
(spec 1.14.0's strip rule) — a hand-authored "AU-AP0-90-00045" hits
the exact-match tier instead of silently falling back to text.
Tests: tests/test_gap_fill.py (10) — preview eligibility incl.
review-refusal + empty-as-gap, author-bytes-preserved-verbatim on dir
AND zip (with .bak content pinned), skip-not-replace on the mixed
request, the writer's ValueError guard, key validation, demo block,
DB sync; +1 hyphenated-ISRC test in test_mb_enrichment.py. 46 targeted
green; full-suite failure set A/B-identical to the main base (39
env/pre-existing).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nm7tHs1Yvjjtnnu4nzJgdN
* gap-fill: align preview with append-only writer (no cleared-value 500)
The R4a preview offered present-but-empty manifest values (album: '',
genres: [], year: 0) as gaps, but the append-only writer's never-clobber
guard raises on ANY key already present — so a user-confirmed POST for
those keys turned into a 500 "write failed" instead of filling the gap.
Appending can't fill an empty-but-present key anyway (it would duplicate
the YAML key).
Fix: _gap_fill_manifest_absent now treats only genuinely-MISSING keys as
gaps; a present-but-empty value is left to the metadata editor (which
re-serializes and can replace in place). This closes the preview→POST
mismatch — the preview never offers what the writer would refuse.
Tests: test_preview_treats_empty_values_as_gaps replaced by
test_preview_excludes_present_but_empty_keys (present-but-empty not
offered; genuinely-absent still offered) + test_write_present_but_empty_
key_is_refused_not_500 (POST → clean 409, file untouched, no .bak; a
genuinely-absent key alongside still writes). Closes the write-path blind
spot in the original empty-value test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: byrongamatos <xasiklas@gmail.com>
|
||
|
|
74cd08f765 |
library: MusicBrainz text matching + Match-Review UI (P8) (#710)
* library: MusicBrainz text matching + Match-Review UI — P8
Replaces the enrichment plumbing's no-op matcher (P7) with the real
pipeline, per the library-metadata design: a wrong match is worse than a
slow one, so medium confidence goes to a human review queue and never
straight to canonical values.
- lib/mb_match.py (new, pure — no network/DB/server imports): denoise
(author credits, (440Hz)/(Live)/(No Lead)/(v2) parentheticals,
diacritics/punctuation, ACDC / AC DC / AC/DC folding via compacted
token equality), token-set similarity, scoring with year/duration
corroboration bonuses, tier classification (auto needs combined
>= 0.95 AND per-field floors — a perfect-title cover by the wrong
artist, or a chart with no artist, can never auto-match), Lucene
query building, MusicBrainz response normalization.
- Matcher precedence in _enrich_one: content-hash cache copy (another
chart of the same recording matches with no network) -> manifest
mbid (tier 0) / isrc (tier 1) exact keys, feature-detected and
strictly shape-validated, read-only -> text search tiers
(auto / review / failed).
- Lifecycle: review rows store their ranked candidate list (JSON) and
write NO canonical fields until a human accepts; failed rows retry on
an exponential backoff (1 h doubling, 7 d cap) via the attempts
column; user-rejected rows never auto-retry; an identity edit
re-queues anything and resets the backoff; never-overwrite-manual is
enforced inside the single writer (apply_enrichment_match) so no call
path can forget it.
- Network: _mb_http_get is the one transport seam — throttled to
<= 1 req/s through P7's _enrich_throttle, identified with a real
User-Agent from VERSION, and a 503 pauses the whole pass without
burning attempts. Offline guard: no sockets under
FEEDBACK_ENRICH_OFFLINE or FEEDBACK_SKIP_STARTUP_TASKS, so pytest can
never reach MusicBrainz; the pass still stamps identity hashes
(two-phase), which is why every P7 test passes unchanged.
- Routes: GET /api/enrichment/review, POST
/api/enrichment/review/{filename}/accept|reject|pick, GET
/api/enrichment/search (throttled manual-search proxy). All four are
demo-mode blocked.
- Match facet: match= CSV accepted by /api/library AND
/api/library/stats (the A-Z rail's letter counts stay lockstep with
the grid) — review / matched (incl. manual) / unmatched / pending,
the same EXISTS idiom as the mastery facet.
- UI: static/v3/match-review.js (new, self-contained) — an ambient
"N to review" chip beside the song count (rendered only when
non-zero; silent on success, no toasts), and a review drawer on the
filter-drawer slide idiom (Escape + focus trap; row click accepts,
"Not a match" rejects, "Search instead" is the fix-match escape
hatch). songs.js gets the chip mount, a Match filter section, and
session-only match state; also fixes the latent applySavedPrefs bug
where restored filters dropped the mastery key, which made the
filter drawer throw for anyone with saved prefs.
- static/tailwind.min.css regenerated (scripts/build-tailwind.sh) for
the new utility classes; conflicts with sibling PRs resolve by
re-running the script.
Nothing is ever written to pack files — canonical values live only in
the song_enrichment display cache. Cover art caching and acoustic
fingerprinting are follow-up slices.
22 pure unit tests + 19 server tests (fake transport injected over the
_mb_http_get seam) + demo-mode route cases; full-suite failure set
A/B-identical with the change stashed vs applied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nm7tHs1Yvjjtnnu4nzJgdN
* library: match-review modal + configurable auto-apply confidence (P8 R0)
Follow-up to the initial P8 commit, folding in the first round of tester
feedback on the review surface and the matcher's knobs:
- Review GUI is a centred MODAL now, not a sidebar — one chart at a
time (the scraper-review model from media-server / emulation-frontend
apps): the chart's current metadata with explicit amber
"Missing: album / year / cover art" chips (art detected via the art
request failing), candidates each carrying "Adds: year - genres -
ISRC" / "Shows as: ACDC -> AC/DC" per-field chips, and Skip /
Not a match / Search instead / Use selected with prev-next + arrow-key
navigation. Chip + window API surface unchanged, so songs.js needed no
edits for the rework.
- Auto-apply confidence is a SETTING: default drops 0.95 -> 0.90
(mb_match.AUTO_MIN; classify() takes an auto_min override). The
per-field floors are untouched and threshold-independent — a
perfect-title cover by the wrong artist still can't auto-match at any
setting. New validated settings keys: enrich_enabled (bool) +
enrich_auto_threshold (0.5–1.01; >1.0 = "Always review", since a
capped score can equal exactly 1.0). Read once per pass; disabling
gates only the BACKGROUND matcher — manual search/fix stays available.
- Settings -> Library -> "Metadata matching" card: enable toggle,
confidence select (85 / 90 / 95 / Always review), a Match Now button
(new POST /api/enrichment/kick, single-flight like every other kick,
demo-mode blocked), and a live status line fed by the same fetch as
the review chip. Markup in index.html per the v3 settings pattern,
wired by match-review.js, null-guarded so v2 no-ops.
- Review queue orders missing-data charts first — confirming those has
the most to gain; complete charts only stand to be re-labelled.
Tests: threshold moves the auto/review boundary via settings; the
enable toggle gates matching but not the manual proxy; settings
validation; kick route; queue ordering; classify(auto_min=...) floors.
Full-suite failure set byte-identical to the pre-change baseline.
tailwind.min.css regenerated for the modal's utility classes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nm7tHs1Yvjjtnnu4nzJgdN
* fix(library): lock MusicBrainz throttle across sleep + de-dup enrich queue (PR #710 review)
Hold a module-level lock across _enrich_throttle's read/sleep/write so the
background daemon and threadpooled sync search route serialize outbound MB
requests instead of bursting past the 1 req/s limit. De-dup the enrich queue
by filename so a changed-hash failed row isn't processed twice per pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: byrongamatos <xasiklas@gmail.com>
|