mirror of
https://github.com/got-feedBack/feedBack.git
synced 2026-07-22 21:01:40 +00:00
main
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Byron Gamatos
|
165475d115
|
refactor(server): move the metadata-enrichment subsystem into lib/enrichment.py (R3) (#861)
* refactor(server): move the metadata-enrichment subsystem into lib/enrichment.py (R3)
MusicBrainz / Cover-Art-Archive / AcoustID transport, the match-scorer glue, and
the background enrichment worker (~930 lines, 61 defs) leave server.py as one
cohesive unit. Bodies are verbatim; the only changes are seam reads:
meta_db / config_dir / sloppak_cache_dir / art_cache_dir -> appstate.<slot>
_song_pack_art_exists / _art_override_paths (stay in server.py for the art +
delete routes) -> appstate.<callable> (new seam slots, injected by reference)
_env_flag -> env_compat.env_flag_compat (the existing identical helper)
_artist_title_from_filename -> imported from metadata_db (its home)
the User-Agent VERSION lookup: Path(__file__).parent ->
Path(__file__).resolve().parents[1] (lib/enrichment.py -> app root)
server.py drives the worker through the module (import enrichment; the routes +
scan lifecycle call enrichment.X). Tests that faked the network on `server`
(_mb_http_get, _enrich_network_enabled, _caa_http_get, ...) now patch the same
names on `enrichment` — the module attribute is resolved at call time, so one
setattr reaches both the routes and the worker's internal callers. Acyclic:
enrichment imports appstate/appconfig/dlc_paths/metadata_db/mb_match/
acoustid_match/sloppak/loosefolder, never server.
server.py: 6,917 -> 5,988 (-929).
Verified: pyflakes clean (bar the pre-existing File/safe_join/tuning_name/ET);
route table IDENTICAL (143); full pytest 2400 passed (140 enrichment/art cases
incl the offline-safety + transport-error-pauses-pass contracts that fake the
network); test_packaging 44 passed (enrichment.py resolves under lib/); eslint 0.
Boot smoke: /api/enrichment/status + POST /kick serve from the new module.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: reset enrichment worker state between tests (CodeRabbit)
lib/enrichment.py now owns the worker, and it stays imported for the whole
session while the `server` fixtures pop-and-reimport `server` — so the cancel
Event / status dict / caches would leak across tests, and a stale `_enrich_cancel`
could short-circuit a later direct `_background_enrich()`. An autouse conftest
fixture clears that process-global state before each test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: tighten the enrichment-reset fixture (CodeRabbit)
Narrow the import guard to ImportError (not blind Exception, BLE001), and stop
clearing _caa_index_locks — it's guarded by _caa_index_locks_guard, so an
unlocked clear() would race a still-alive worker, and its per-release mutexes
carry no test state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Byron Gamatos
|
97a941c45d
|
fix(tests): join background scan/enrich workers before closing the DB (#735)
Root cause of the flaky pytest segfault (exit 139): the background scan and enrichment daemon threads (_scan_runner/_enrich_runner) use the shared MetadataDB connection, but test fixtures closed that connection in teardown without stopping them. A daemon thread mid-query on a freed SQLite conn is a native use-after-free → SIGSEGV. The app's startup kicks a scan, so almost any app-booting fixture was vulnerable. It only surfaced now because got-feedback/feedBack#728 added a push trigger, so ci/test runs on every push to main. Fix: server.py retains the scan/enrich thread handles and adds _join_background_db_threads(); every test fixture now joins the workers before conn.close(). Verified: the full suite runs to completion (no segfault) where it previously crashed at ~25%. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Byron Gamatos
|
f5c9c34291
|
library: scraper options — per-source + per-field auto-apply, review-queue order (R1) (#726)
* library: scraper options — per-source + per-field auto-apply toggles, review-queue order (R1) Grows the Settings→Library "Metadata matching" card into the full scraper-options panel — not everybody needs the same things out of a scraper: - Sources: enrich_src_musicbrainz gates the background matcher (phase 2; identity hashes still stamp, and manual Fix-match/search stays available — same contract as the master toggle). enrich_src_caa gates the Cover Art Archive fetch (phase 3). Rows skipped by an off toggle stay unevaluated, so re-enabling picks them up on the next pass — nothing is permanently forfeited. - Auto-apply fields: enrich_apply_names/year/genres filter what an AUTOMATIC match may canonicalize (_enrich_field_filter, applied on all three automatic paths: cache copy, mbid/isrc exact keys, text auto). MusicBrainz ids always stamp — they're identity, not display; the art fetch and future re-matching need them. A match the user confirms in the review modal applies in full. enrich_apply_art gates the art fetch alongside the CAA source toggle (two axes, one behaviour today — future art sources slot in without re-teaching the panel). - Review queue order: enrich_review_order = missing_first (default, today's behaviour) | artist | recent, read by GET /api/enrichment/review; unknown stored values degrade to the default. - Settings card: Sources / Auto-apply / Review-queue-order groups wired in match-review.js; the master toggle is relabelled "Match songs automatically" so it doesn't read the same as the new MusicBrainz source toggle. No tailwind rebuild needed — every class was already scanned from core source. Tests: tests/test_scraper_options.py (9) — settings validation, MB-source-off stamps-without-matching + re-enable, per-field stripping on auto matches with ids preserved, review-accept full-apply despite toggles, CAA gating on both axes, review-order modes incl. the unknown-value fallback. Full-suite failure set A/B-identical to the base (39 env/pre-existing). Stacked on feat/enrichment-art (#715) — merge that first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nm7tHs1Yvjjtnnu4nzJgdN * library: per-field auto-apply honours "nothing forfeited" (backfill + no partial seeding) The R1 per-FIELD auto-apply toggles settled a `matched` row with the disabled fields stripped, but enrichment_pending() never revisits an unchanged-hash matched row — so re-enabling a field never backfilled it, and enrichment_cache_lookup() (which gates only on mb_recording_id) could seed sibling charts with the stripped blanks. This broke the same "nothing is permanently forfeited" contract the source (mb_on) and art (art_on) toggles already keep. Fix: persist an `apply_mask` marker (sorted blocked apply-keys) on every AUTOMATIC match: - migration: additive `apply_mask TEXT` column (idempotent ALTER). - enrichment_pending(allowed_keys=...): re-queues a `matched` row whose apply_mask names a field that is now re-enabled → backfill on re-enable, converges (a fully-applied row is never re-queued). - enrichment_cache_lookup: only fully-applied donors (apply_mask empty/NULL) may seed siblings; a partial row is skipped and the sibling falls through to its own re-filtered match. - _enrich_apply_mask()/_enrich_blocked_apply_keys() helpers; threaded through _enrich_one → apply_enrichment_match. Review/manual writers leave it NULL (a confirmed pick applies in full). Tests: re-enable-backfills-and-converges; partial row is not a cache donor (fully-applied one is). 13 scraper-options tests pass; 170 enrichment/ settings tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: ChrisBeWithYou <christian.a.cowan@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |