library: opt-in gap-fill — write confirmed missing metadata into the pack (R4a) (#724)

* library: opt-in gap-fill — write confirmed missing metadata into the pack (R4a)

The write-back contract agreed with the spec chair (alignment doc §7),
made executable, now that feedpak-spec 1.14.0 (mbid/isrc) is merged:
opt-in + user-initiated, adds ABSENT keys only, spec'd-keys allowlist,
values only from a CONFIRMED identity, atomic write + .bak. Single-song
only — batch write-back stays an open question with the chair.

- songmeta.gap_fill_sloppak: append-only manifest writer. Every added
  key is absent by definition, so the new lines are APPENDED — the
  author's existing bytes (key order, comments, formatting) survive
  verbatim, unlike the metadata editor's full re-serialize. Directory
  form gets a one-time manifest.yaml.bak + temp + atomic replace; zip
  form reuses the editor's backup/temp/replace rewriter. Raises on any
  already-present key: the never-clobber rule lives in the writer, not
  just the callers.
- GET /api/song/{fn}/gap-fill: read-only preview — which of
  album/year/genres/mbid/isrc are missing from the file (absent or
  empty; year 0 = empty), with the values the enrichment match
  supplies. Only a CONFIRMED identity is eligible (matched or a user
  pin); review-tier rows are refused until a human confirms —
  wrong-match > fast, same as everywhere else in the enrichment layer.
- POST /api/song/{fn}/gap-fill {keys}: writes the user-confirmed
  subset. Proposals are RECOMPUTED under _song_io_lock, so a key that
  gained an author value between preview and confirm is skipped, never
  replaced. mbid/isrc written in canonical form only (validated).
  DB stays scanner-consistent (album/year/genre columns + mtime/size
  re-stat, cache invalidation + scan kick — the metadata editor's
  contract). Demo mode blocks the write.
- Details drawer (Identity section): "Write missing info to file…" →
  per-key checkbox confirm ("Only adds what's missing — nothing already
  in the file is changed. A backup (.bak) is kept.") → written
  confirmation; not-eligible states explain themselves. v3 only; no
  new tailwind classes.
- Rides along: _manifest_exact_ids now strips ISRC display separators
  (spec 1.14.0's strip rule) — a hand-authored "AU-AP0-90-00045" hits
  the exact-match tier instead of silently falling back to text.

Tests: tests/test_gap_fill.py (10) — preview eligibility incl.
review-refusal + empty-as-gap, author-bytes-preserved-verbatim on dir
AND zip (with .bak content pinned), skip-not-replace on the mixed
request, the writer's ValueError guard, key validation, demo block,
DB sync; +1 hyphenated-ISRC test in test_mb_enrichment.py. 46 targeted
green; full-suite failure set A/B-identical to the main base (39
env/pre-existing).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nm7tHs1Yvjjtnnu4nzJgdN

* gap-fill: align preview with append-only writer (no cleared-value 500)

The R4a preview offered present-but-empty manifest values (album: '',
genres: [], year: 0) as gaps, but the append-only writer's never-clobber
guard raises on ANY key already present — so a user-confirmed POST for
those keys turned into a 500 "write failed" instead of filling the gap.
Appending can't fill an empty-but-present key anyway (it would duplicate
the YAML key).

Fix: _gap_fill_manifest_absent now treats only genuinely-MISSING keys as
gaps; a present-but-empty value is left to the metadata editor (which
re-serializes and can replace in place). This closes the preview→POST
mismatch — the preview never offers what the writer would refuse.

Tests: test_preview_treats_empty_values_as_gaps replaced by
test_preview_excludes_present_but_empty_keys (present-but-empty not
offered; genuinely-absent still offered) + test_write_present_but_empty_
key_is_refused_not_500 (POST → clean 409, file untouched, no .bak; a
genuinely-absent key alongside still writes). Closes the write-path blind
spot in the original empty-value test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: byrongamatos <xasiklas@gmail.com>

server.py

@@ -244,6 +244,8 @@ _DEMO_BLOCKED: list[tuple[str, re.Pattern]] = [
     # rate limit; Get-info exposes filesystem paths.
     ("POST",   re.compile(r"^/api/enrichment/refresh/.+$")),
     ("GET",    re.compile(r"^/api/chart/.+/fileinfo$")),
+    # Gap-fill (R4a) rewrites pack files on disk — never for demo visitors.
+    ("POST",   re.compile(r"^/api/song/.+/gap-fill$")),
 ]
 
 
@@ -5711,6 +5713,11 @@ def _manifest_exact_ids(filename: str) -> dict:
     if _MBID_RE.match(mbid):
         out["mbid"] = mbid
     isrc = str(manifest.get("isrc", "") or "").strip().upper()
+    # Spec 1.14.0: the stored form is the bare 12-char code, but ISRCs
+    # circulate hyphenated in the wild (US-ABC-24-00001) — the separators
+    # are presentation, not part of the code, so a hand-authored display
+    # form still matches (consumers SHOULD strip before comparing).
+    isrc = isrc.replace("-", "").replace(" ", "")
     if _ISRC_RE.match(isrc):
         out["isrc"] = isrc
     return out
@@ -10231,6 +10238,158 @@ def update_song_meta(filename: str, data: dict):
     return {"ok": True, "persisted": persisted}
 
 
+# ── Gap-fill: write CONFIRMED missing metadata into the pack (R4a) ────────────
+# The agreed write-back contract (spec-alignment §7): opt-in + user-initiated
+# (nothing here runs in the background), adds ABSENT keys only (never replaces
+# an author-set value — the writer refuses, and existing manifest bytes are
+# preserved verbatim by appending), spec'd-keys allowlist, values only from a
+# CONFIRMED identity (an auto/exact match or a user pin — review-tier rows are
+# not eligible until a human confirms), atomic write + .bak. Single-song only;
+# batch write-back stays an open question with the spec chair.
+_GAP_FILL_KEYS = ("album", "year", "genres", "mbid", "isrc")
+
+
+def _gap_fill_manifest_absent(manifest: dict, key: str) -> bool:
+    """A key is a GAP only when it's genuinely MISSING from the manifest.
+
+    Gap-fill is append-only: the writer's never-clobber guard raises on ANY
+    key already present, and appending a second `album:` line to a manifest
+    that already carries `album: ''` would just create a duplicate YAML key.
+    So a present-but-empty value (None / '' / [] / year 0) is NOT a gap the
+    append-only writer can fill — offering it in the preview would only lead
+    to a POST the writer refuses. Present-but-empty keys are therefore left
+    to the metadata editor (which re-serializes and can replace in place)."""
+    return key not in manifest
+
+
+def _gap_fill_proposals(cache_key: str, resolved) -> tuple[dict, str]:
+    """What gap-fill could add for this song: (proposals, reason). Empty
+    proposals explain themselves via reason — 'not-sloppak', 'no-match'
+    (nothing confirmed yet), 'review' (a human hasn't confirmed the match),
+    or 'nothing-missing'."""
+    if resolved is None or not resolved.exists() or not sloppak_mod.is_sloppak(resolved):
+        return {}, "not-sloppak"
+    row = meta_db.get_enrichment(cache_key)
+    if not row or row.get("match_state") not in ("matched", "manual"):
+        state = (row or {}).get("match_state")
+        return {}, ("review" if state == "review" else "no-match")
+    try:
+        manifest = sloppak_mod.load_manifest(resolved) or {}
+    except Exception:
+        return {}, "not-sloppak"
+    out = {}
+    album = (row.get("canon_album") or "").strip()
+    if album and _gap_fill_manifest_absent(manifest, "album"):
+        out["album"] = album
+    year = (row.get("canon_year") or "").strip()
+    if year.isdigit() and int(year) and _gap_fill_manifest_absent(manifest, "year"):
+        out["year"] = int(year)
+    genres = [str(g) for g in (row.get("genres") or []) if isinstance(g, str) and g.strip()]
+    if genres and _gap_fill_manifest_absent(manifest, "genres"):
+        out["genres"] = genres
+    # Identity keys (feedpak spec 1.14.0) — written in canonical form only.
+    mbid = (row.get("mb_recording_id") or "").strip().lower()
+    if _MBID_RE.match(mbid) and _gap_fill_manifest_absent(manifest, "mbid"):
+        out["mbid"] = mbid
+    isrc = (row.get("isrc") or "").strip().upper().replace("-", "").replace(" ", "")
+    if _ISRC_RE.match(isrc) and _gap_fill_manifest_absent(manifest, "isrc"):
+        out["isrc"] = isrc
+    return out, ("" if out else "nothing-missing")
+
+
+@app.get("/api/song/{filename:path}/gap-fill")
+def get_song_gap_fill(filename: str):
+    """Preview what "Write missing info to file" would add — the Details
+    drawer renders its confirm list straight from this. Read-only."""
+    dlc = _get_dlc_dir()
+    cache_key, resolved = filename, None
+    if dlc:
+        resolved = _resolve_dlc_path(dlc, filename)
+        if resolved is None:
+            return JSONResponse({"error": "forbidden"}, 403)
+        try:
+            cache_key = resolved.relative_to(dlc.resolve()).as_posix()
+        except ValueError:
+            pass
+    proposals, reason = _gap_fill_proposals(cache_key, resolved)
+    row = meta_db.get_enrichment(cache_key) or {}
+    return {
+        "eligible": bool(proposals),
+        "reason": reason,
+        "match_state": row.get("match_state"),
+        "missing": [{"key": k, "value": v} for k, v in proposals.items()],
+    }
+
+
+@app.post("/api/song/{filename:path}/gap-fill")
+def post_song_gap_fill(filename: str, data: dict):
+    """Write the user-confirmed subset of the preview into the pack file.
+    Proposals are recomputed under the io lock, so a key that gained an
+    author value between preview and confirm is skipped, never replaced."""
+    keys = (data or {}).get("keys")
+    if not isinstance(keys, list) or not keys:
+        return JSONResponse({"error": "keys must be a non-empty list"}, 400)
+    bad = [k for k in keys if k not in _GAP_FILL_KEYS]
+    if bad:
+        return JSONResponse(
+            {"error": "unknown key(s): " + ", ".join(sorted(set(map(str, bad))))}, 400)
+
+    dlc = _get_dlc_dir()
+    cache_key, resolved = filename, None
+    if dlc:
+        resolved = _resolve_dlc_path(dlc, filename)
+        if resolved is None:
+            return JSONResponse({"error": "forbidden"}, 403)
+        try:
+            cache_key = resolved.relative_to(dlc.resolve()).as_posix()
+        except ValueError:
+            pass
+
+    with _song_io_lock:
+        proposals, reason = _gap_fill_proposals(cache_key, resolved)
+        additions = {k: proposals[k] for k in _GAP_FILL_KEYS if k in keys and k in proposals}
+        skipped = sorted(set(keys) - set(additions))
+        if not additions:
+            return JSONResponse({"error": "nothing to write", "reason": reason,
+                                 "skipped": skipped}, 409)
+        try:
+            import songmeta
+            songmeta.gap_fill_sloppak(resolved, additions)
+        except Exception:
+            log.warning("gap-fill write failed for %s", cache_key, exc_info=True)
+            return JSONResponse({"error": "write failed"}, 500)
+
+        # Keep the cache row consistent with what the scanner would now derive
+        # (same contract as the metadata editor above): sync the columns the
+        # scan reads from the keys we appended, then re-stat so the row stays
+        # cache-fresh.
+        fields = {}
+        if "album" in additions:
+            fields["album"] = additions["album"]
+        if "year" in additions:
+            fields["year"] = str(additions["year"])
+        if "genres" in additions:
+            fields["genre"] = additions["genres"][0]
+        with meta_db._lock:
+            updates = [f"{field} = ?" for field in fields]
+            params = list(fields.values())
+            try:
+                mtime, size = _stat_for_cache(resolved)
+                updates += ["mtime = ?", "size = ?"]
+                params += [mtime, size]
+            except OSError:
+                pass
+            if updates:
+                params.append(cache_key)
+                meta_db.conn.execute(
+                    f"UPDATE songs SET {', '.join(updates)} WHERE filename = ?", params)
+                meta_db.conn.commit()
+
+    _invalidate_song_caches(cache_key)
+    _kick_scan()
+    return {"ok": True, "written": additions, "skipped": skipped}
+
+
 @app.post("/api/song/{filename:path}/art/upload")
 async def upload_song_art_b64(filename: str, data: dict):
     """Upload custom album art as base64 PNG/JPG."""