diff --git a/CHANGELOG.md b/CHANGELOG.md
index 780ae48..a72eb9d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- **Keyset (cursor) pagination for the library grid — the data layer for an upcoming virtualized grid, and a latent paging bug fixed along the way.** Every library sort now carries a unique `filename` tiebreak, making the order **total** — which fixes a latent bug where rows sharing a sort key (e.g. two songs by the same artist) could be skipped or duplicated across `OFFSET` pages. `GET /api/library` gains an opaque `after` cursor + a `next_cursor` in the response: passing the cursor back fetches the next page with a **WHERE-seek** instead of `OFFSET`, so deep paging is O(page) regardless of depth. The seek is NULL-aware and exactly `OFFSET`-equivalent (verified across artist/title/recent, ascending + descending, including the legacy `dir=desc` shape and NULL sort keys); unknown/compound sorts and bad cursors fall back to `OFFSET`, and only the local provider is handed a cursor (collections/remote page by `OFFSET`). New composite `(artist NOCASE, filename)` / `(title NOCASE, filename)` / `(mtime, filename)` indexes cover the order. This is stage 1 of the virtualized-grid project (got-feedback/feedBack#636 item 3); the DOM-recycling render window builds on it next. Tests: `tests/test_library_keyset.py` (keyset==OFFSET parity, stable tiebreak, dir=desc, NULL keys, cursor fallback).
- **The settings backup now includes your library database + custom art — your scores, favorites, playlists, and play history are no longer the one thing a backup can't save.** `GET /api/settings/export` gains an additive `core_server_files` section carrying a **consistent snapshot of `web_library.db`** (taken via the SQLite online-backup API, so it's a complete single file even while the server is running) plus any custom **playlist covers** and **avatar** (`CONFIG_DIR/playlist_covers/`, `CONFIG_DIR/avatars/`). On `POST /api/settings/import` the database is **staged** to `web_library.db.restore` rather than written over the live, open DB; it's swapped in at the next startup (`_apply_pending_db_restore`, before the connection opens), which also clears the old WAL sidecars so a stale `-wal` can't be replayed onto the restored file — the import response sets `restart_required: true` and warns accordingly. Custom art is written immediately. The bundle stays backward-compatible (older servers ignore the new section). Came out of the library design charrette (dev-ops lens's top "protect irreplaceable data" pick, got-feedback/feedBack#636). _Known gap:_ custom uploaded **song** art is still commingled with the rebuildable thumbnail cache in `art_cache/`, so it isn't bundled yet (a tracked follow-up). Tests: `tests/test_settings_export_library_db.py` (snapshot consistency, staged-not-live restore, sidecar clearing, traversal rejection, full round-trip).
- **A persisted wishlist — keep a list of songs you want but don't own yet.** New `wanted` table + `GET`/`POST`/`DELETE /api/wanted` give FeedBack the *arr-style "Wanted/Monitored" primitive it was missing: an entry is a *not-owned* song (artist/title/source/source_ref/note), so it lives in its own table rather than the playlist subsystem (which references owned local files). The API is idempotent on identity (case-insensitive artist+title, plus source+source_ref), so a producer — the `find_more` ownership-diff, or a manual add — can re-post without duplicating. Newest-first. Backend primitive for the charrette's wishlist finding (got-feedback/feedBack#636 item 4); the consuming UI lives in the producing plugin. Tests: `tests/test_wanted_api.py`.
- **Practice-aware library home — a "Repertoire" meter + a "Keep practicing" shelf on the v3 Songs page.** The library opened cold into a flat sorted grid; now the unfiltered grid front door leads with two practice-aware surfaces built entirely from data already on hand (no new endpoints or stored state). A **Repertoire meter** shows how much of your library you can actually play — *"Repertoire: 12 of 80 songs · 7 in progress"* with a progress bar — counting songs at or above the same mastery threshold the green accuracy badge uses (≥ 90% best accuracy) over the unfiltered library total. A **"Keep practicing" shelf** is a horizontal row of your recently-played-but-not-yet-mastered songs (newest first, click to play) — the practice-accuracy-driven "continue" rail a media server can't do. Both reuse `/api/stats/best` (already loaded for the card badges) + `/api/stats/recent`; they show **only** on the grid view when you aren't searching/filtering/selecting, refresh after a song is scored, and collapse to nothing on an empty library. Soft-gamification only — descriptive encouragement (goal-gradient / endowed-progress), never content-gating, decay, or nagging. Frontend-only: `static/v3/songs.js` (`renderLibraryHome`/`_repertoireCounts`), `static/v3/v3.css`. Came out of the library design charrette (the UX + gamification lenses' top pick). Tests: `tests/js/v3_keep_practicing.test.js`.
diff --git a/server.py b/server.py
index e14f9e6..c546291 100644
--- a/server.py
+++ b/server.py
@@ -427,6 +427,80 @@ def _apply_pending_db_restore(config_dir: Path) -> None:
log.info("applied pending library DB restore from settings import")
+# ── Keyset (cursor) pagination for the library grid (feedBack#636 item 3) ─────
+# Forward-only, O(page) deep paging that doesn't grow with OFFSET. Only simple
+# single-column sorts can keyset cleanly (the compound tuning/year sorts fall
+# back to OFFSET). Every sort gets a unique `filename` tiebreak so the order is
+# TOTAL — which also fixes a latent OFFSET skip/dupe across equal-key rows.
+# (column, collate-clause, primary-direction) — tiebreak is always `filename` ASC.
+_KEYSET_SORTS = {
+ "artist": ("artist", "COLLATE NOCASE", "ASC"),
+ "artist-desc": ("artist", "COLLATE NOCASE", "DESC"),
+ "title": ("title", "COLLATE NOCASE", "ASC"),
+ "title-desc": ("title", "COLLATE NOCASE", "DESC"),
+ "recent": ("mtime", "", "DESC"),
+}
+# Index into a query_page row tuple for each keyset column (see the SELECT in
+# query_page: filename, title, artist, ... mtime at 9).
+_KEYSET_ROW_IDX = {"artist": 2, "title": 1, "mtime": 9}
+
+
+def _encode_cursor(values: list) -> str:
+ import base64
+ return base64.urlsafe_b64encode(json.dumps(values).encode("utf-8")).decode("ascii")
+
+
+def _decode_cursor(cursor: str):
+ """Decode an opaque keyset cursor to [sort_value, filename], or None if it's
+ malformed (a bad cursor degrades to the first page, never 500s)."""
+ import base64
+ try:
+ out = json.loads(base64.urlsafe_b64decode(cursor.encode("ascii")).decode("utf-8"))
+ except (ValueError, TypeError):
+ return None
+ return out if isinstance(out, list) and len(out) == 2 else None
+
+
+def _effective_keyset_sort(sort: str, direction: str) -> str:
+ """Fold the legacy `dir=desc` toggle into the canonical keyset sort key, so
+ the seek/cursor direction matches the ORDER BY that same toggle produces
+ (without this, `sort=artist&dir=desc` would seek with `>` against a DESC
+ order → gaps/dupes)."""
+ if direction == "desc" and sort in ("artist", "title"):
+ return sort + "-desc"
+ return sort
+
+
+def _keyset_seek(col: str, collate: str, primary_dir: str, cv, fn: str):
+ """(sql, params) for 'rows strictly after (cv, fn)' in the total order
+ `
, filename ASC`, matching SQLite's NULL placement
+ (NULLs sort first in ASC, last in DESC) so keyset is exactly OFFSET-
+ equivalent even for NULL sort keys."""
+ ce = f"{col} {collate}".strip()
+ if primary_dir == "ASC": # NULLs first
+ if cv is None:
+ return (f"(({col} IS NULL AND filename > ?) OR {col} IS NOT NULL)", [fn])
+ return (f"({col} IS NOT NULL AND ({ce} > ? OR ({ce} = ? AND filename > ?)))",
+ [cv, cv, fn])
+ # DESC — NULLs last
+ if cv is None:
+ return (f"({col} IS NULL AND filename > ?)", [fn])
+ return (f"({col} IS NULL OR ({col} IS NOT NULL AND "
+ f"({ce} < ? OR ({ce} = ? AND filename > ?))))", [cv, cv, fn])
+
+
+def next_library_cursor(sort: str, last_song: dict | None) -> str | None:
+ """The cursor for the last row of a page, so the next request resumes after
+ it. None when the sort can't keyset or the page was empty."""
+ if sort not in _KEYSET_SORTS or not last_song:
+ return None
+ col = _KEYSET_SORTS[sort][0]
+ key = "mtime" if col == "mtime" else col
+ if key not in last_song or "filename" not in last_song:
+ return None
+ return _encode_cursor([last_song[key], last_song["filename"]])
+
+
class MetadataDB:
def __init__(self):
CONFIG_DIR.mkdir(parents=True, exist_ok=True)
@@ -478,6 +552,13 @@ class MetadataDB:
pass
self.conn.execute("CREATE INDEX IF NOT EXISTS idx_songs_artist ON songs(artist COLLATE NOCASE)")
self.conn.execute("CREATE INDEX IF NOT EXISTS idx_songs_title ON songs(title COLLATE NOCASE)")
+ # Composite (sort col, filename) indexes cover the grid's ORDER BY +
+ # its unique filename tiebreak — for both the OFFSET scan and keyset
+ # seek (feedBack#636 item 3). idx_songs_artist/title above stay for the
+ # distinct-artist / letter-bar aggregates.
+ self.conn.execute("CREATE INDEX IF NOT EXISTS idx_songs_artist_fn ON songs(artist COLLATE NOCASE, filename)")
+ self.conn.execute("CREATE INDEX IF NOT EXISTS idx_songs_title_fn ON songs(title COLLATE NOCASE, filename)")
+ self.conn.execute("CREATE INDEX IF NOT EXISTS idx_songs_mtime_fn ON songs(mtime, filename)")
self.conn.execute("CREATE INDEX IF NOT EXISTS idx_songs_tuning_name ON songs(tuning_name COLLATE NOCASE)")
self.conn.execute("CREATE INDEX IF NOT EXISTS idx_songs_tuning_sort_key ON songs(tuning_sort_key)")
self.conn.execute("CREATE INDEX IF NOT EXISTS idx_songs_year ON songs(year)")
@@ -1943,8 +2024,14 @@ class MetadataDB:
stems_lacks: list[str] | None = None,
has_lyrics: int | None = None,
tunings: list[str] | None = None,
+ after: str | None = None,
naming_mode: str = "legacy") -> tuple[list[dict], int]:
- """Server-side paginated search. Returns (songs, total_count)."""
+ """Server-side paginated search. Returns (songs, total_count).
+
+ `after` is an opaque keyset cursor (the last row of the previous page).
+ When supplied and the sort can keyset, the page is fetched with a
+ WHERE-seek instead of OFFSET — O(page), independent of depth. Unknown
+ sorts / bad cursors fall back to OFFSET, so it's always safe."""
where, params = self._build_where(
q=q, favorites_only=favorites_only, format_filter=format_filter,
artist_filter=artist_filter, album_filter=album_filter,
@@ -2002,14 +2089,33 @@ class MetadataDB:
# those sorts use the explicit `-desc` sort key instead.
if direction == "desc" and " ASC" not in order and " DESC" not in order:
order += " DESC"
+ # Unique, deterministic tiebreak → a TOTAL order. Without it, rows with
+ # an equal sort key can reshuffle between OFFSET pages (skip/dupe); it's
+ # also what makes keyset seeking correct.
+ order += ", filename"
total = self.conn.execute(f"SELECT COUNT(*) FROM songs {where}", params).fetchone()[0]
- rows = self.conn.execute(
- f"SELECT filename, title, artist, album, year, duration, tuning, arrangements, has_lyrics, mtime, "
- f"format, stem_count, stem_ids, tuning_name, tuning_offsets "
- f"FROM songs {where} ORDER BY {order} LIMIT ? OFFSET ?",
- params + [size, page * size]
- ).fetchall()
+
+ cols = ("SELECT filename, title, artist, album, year, duration, tuning, "
+ "arrangements, has_lyrics, mtime, format, stem_count, stem_ids, "
+ "tuning_name, tuning_offsets FROM songs ")
+ cursor = _decode_cursor(after) if after else None
+ eff_sort = _effective_keyset_sort(sort, direction)
+ if cursor and eff_sort in _KEYSET_SORTS:
+ # Keyset seek: rows strictly after the cursor in the total order
+ # ` , filename ASC` (NULL-aware, so == OFFSET exactly).
+ col, collate, primary_dir = _KEYSET_SORTS[eff_sort]
+ seek, seek_params = _keyset_seek(col, collate, primary_dir, cursor[0], cursor[1])
+ seek_where = where + (" AND " if where else " WHERE ") + seek
+ rows = self.conn.execute(
+ f"{cols}{seek_where} ORDER BY {order} LIMIT ?",
+ params + seek_params + [size],
+ ).fetchall()
+ else:
+ rows = self.conn.execute(
+ f"{cols}{where} ORDER BY {order} LIMIT ? OFFSET ?",
+ params + [size, page * size],
+ ).fetchall()
estd = self._estd_set()
favs = self.favorite_set()
@@ -2773,7 +2879,7 @@ def _require_library_provider_capability(provider: object, capability: str) -> N
)
-_OPTIONAL_NEW_PROVIDER_KWARGS = ("naming_mode", "sort", "want_sort_letters")
+_OPTIONAL_NEW_PROVIDER_KWARGS = ("naming_mode", "sort", "want_sort_letters", "after")
def _filter_provider_kwargs(method: object, kwargs: dict) -> dict:
@@ -4650,11 +4756,21 @@ async def list_library(q: str = "", page: int = 0, size: int = 24, sort: str = "
arrangements_has: str = "", arrangements_lacks: str = "",
stems_has: str = "", stems_lacks: str = "",
has_lyrics: str = "", tunings: str = "", provider: str = "local",
- naming_mode: str = "legacy"):
- """Paginated library search through the selected library provider."""
+ after: str = "", naming_mode: str = "legacy"):
+ """Paginated library search through the selected library provider.
+
+ `after` is an opaque keyset cursor (feedBack#636 item 3): pass back the
+ `next_cursor` from the previous response to fetch the next page with a
+ WHERE-seek instead of OFFSET. Providers that don't support it ignore it and
+ page by OFFSET, so the client can always fall back."""
size = min(size, 100)
library_provider = _get_library_provider(provider)
_require_library_provider_capability(library_provider, "library.read")
+ # Only the true local provider keysets: it's the one whose effective sort is
+ # exactly the request `sort`. A smart collection may pin its own sort and
+ # remote providers don't keyset — both must page by OFFSET, so never hand
+ # them a cursor (a mismatched one would mis-seek).
+ is_local = getattr(library_provider, "id", "") == "local"
songs, total = await _call_library_provider_async(
library_provider,
"query_page",
@@ -4662,6 +4778,7 @@ async def list_library(q: str = "", page: int = 0, size: int = 24, sort: str = "
size=size,
sort=sort,
direction=dir,
+ after=((after or None) if is_local else None),
naming_mode=naming_mode,
**_library_filter_args(
q=q, favorites=favorites, format=format,
@@ -4671,7 +4788,11 @@ async def list_library(q: str = "", page: int = 0, size: int = 24, sort: str = "
has_lyrics=has_lyrics, tunings=tunings,
),
)
- return {"songs": songs, "total": total, "page": page, "size": size}
+ # The cursor to resume after this page (effective sort folds in dir=desc).
+ next_cursor = (next_library_cursor(_effective_keyset_sort(sort, dir), songs[-1])
+ if (is_local and songs) else None)
+ return {"songs": songs, "total": total, "page": page, "size": size,
+ "next_cursor": next_cursor}
@app.get("/api/library/artists")
diff --git a/tests/test_library_keyset.py b/tests/test_library_keyset.py
new file mode 100644
index 0000000..47fdf4b
--- /dev/null
+++ b/tests/test_library_keyset.py
@@ -0,0 +1,154 @@
+"""Keyset (cursor) pagination for the library grid (feedBack#636 item 3, stage 1).
+
+Pins the data layer the virtualized grid builds on:
+ - every sort gets a unique `filename` tiebreak → a TOTAL order (fixes the
+ latent OFFSET skip/dupe across equal-key rows);
+ - `/api/library?after=` walks the SAME total order with a WHERE-seek,
+ returning exactly the OFFSET page would, with no gaps or dupes;
+ - bad cursors / non-keyset sorts fall back to OFFSET safely.
+"""
+
+import importlib
+import sys
+
+import pytest
+from fastapi.testclient import TestClient
+
+
+@pytest.fixture()
+def server_mod(tmp_path, monkeypatch):
+ monkeypatch.setenv("CONFIG_DIR", str(tmp_path))
+ sys.modules.pop("server", None)
+ mod = importlib.import_module("server")
+ yield mod
+ conn = getattr(getattr(mod, "meta_db", None), "conn", None)
+ if conn is not None:
+ conn.close()
+
+
+@pytest.fixture()
+def client(server_mod):
+ c = TestClient(server_mod.app)
+ try:
+ yield c
+ finally:
+ c.close()
+
+
+def _seed(server_mod, n=25, *, shared_artist=False):
+ for i in range(n):
+ artist = "SameArtist" if shared_artist else f"Artist{i:02d}"
+ server_mod.meta_db.put(f"song{i:02d}.archive", float(i), 1, {
+ "title": f"Title{i:02d}", "artist": artist, "album": "LP", "year": "",
+ "duration": 1.0, "tuning": "E Standard", "arrangements": [], "has_lyrics": False,
+ "format": "archive", "stem_count": 0, "stem_ids": [], "tuning_name": "E Standard",
+ "tuning_sort_key": 0, "tuning_offsets": "",
+ })
+
+
+def _walk_keyset(client, sort, size, total):
+ """Page the whole library via the cursor and return the filename order."""
+ seen, cursor, guard = [], "", 0
+ while len(seen) < total and guard < total + 5:
+ guard += 1
+ params = {"sort": sort, "size": size}
+ if cursor:
+ params["after"] = cursor
+ body = client.get("/api/library", params=params).json()
+ seen.extend(s["filename"] for s in body["songs"])
+ cursor = body.get("next_cursor")
+ if not body["songs"] or not cursor:
+ break
+ return seen
+
+
+def _walk_offset(client, sort, size, total):
+ seen, page = [], 0
+ while len(seen) < total:
+ body = client.get("/api/library", params={"sort": sort, "size": size, "page": page}).json()
+ if not body["songs"]:
+ break
+ seen.extend(s["filename"] for s in body["songs"])
+ page += 1
+ return seen
+
+
+@pytest.mark.parametrize("sort", ["artist", "artist-desc", "title", "title-desc", "recent"])
+def test_keyset_matches_offset_exactly(client, server_mod, sort):
+ _seed(server_mod, 25)
+ offset_order = _walk_offset(client, sort, 7, 25)
+ keyset_order = _walk_keyset(client, sort, 7, 25)
+ assert keyset_order == offset_order # same order...
+ assert len(keyset_order) == 25
+ assert len(set(keyset_order)) == 25 # ...no gaps, no dupes
+
+
+def test_stable_tiebreak_on_equal_keys(client, server_mod):
+ # 25 songs, all the SAME artist → the artist sort is decided entirely by the
+ # filename tiebreak. Both pagers must still cover all 25 with no dupe.
+ _seed(server_mod, 25, shared_artist=True)
+ keyset_order = _walk_keyset(client, "artist", 6, 25)
+ assert len(keyset_order) == 25 and len(set(keyset_order)) == 25
+ assert keyset_order == sorted(keyset_order) # tiebreak is filename ASC
+
+
+def test_first_page_has_cursor_and_no_after_is_offset(client, server_mod):
+ _seed(server_mod, 5)
+ body = client.get("/api/library", params={"sort": "artist", "size": 2}).json()
+ assert body["next_cursor"] # cursor offered
+ assert [s["filename"] for s in body["songs"]] == ["song00.archive", "song01.archive"]
+
+
+def test_bad_cursor_falls_back_to_first_page(client, server_mod):
+ _seed(server_mod, 5)
+ body = client.get("/api/library", params={"sort": "artist", "size": 3, "after": "not-a-cursor"}).json()
+ assert [s["filename"] for s in body["songs"]] == ["song00.archive", "song01.archive", "song02.archive"]
+
+
+def test_legacy_dir_desc_keysets_correctly(client, server_mod):
+ # The legacy `sort=artist&dir=desc` shape must keyset against a DESC order
+ # (canonicalized to artist-desc), not seek `>` against it → no gaps/dupes.
+ _seed(server_mod, 20)
+ offset_order, page = [], 0
+ while True:
+ body = client.get("/api/library", params={"sort": "artist", "dir": "desc", "size": 6, "page": page}).json()
+ if not body["songs"]:
+ break
+ offset_order.extend(s["filename"] for s in body["songs"])
+ page += 1
+ keyset, cursor, guard = [], "", 0
+ while len(keyset) < 20 and guard < 25:
+ guard += 1
+ params = {"sort": "artist", "dir": "desc", "size": 6}
+ if cursor:
+ params["after"] = cursor
+ body = client.get("/api/library", params=params).json()
+ keyset.extend(s["filename"] for s in body["songs"])
+ cursor = body.get("next_cursor")
+ if not body["songs"] or not cursor:
+ break
+ assert keyset == offset_order
+ assert len(set(keyset)) == 20
+
+
+@pytest.mark.parametrize("sort", ["artist", "artist-desc", "recent"])
+def test_keyset_handles_null_sort_keys(client, server_mod, sort):
+ # NULL artist/mtime (corrupt/legacy rows past put()'s '' defaults) sort
+ # first in ASC / last in DESC; keyset must cover them exactly like OFFSET.
+ _seed(server_mod, 10)
+ server_mod.meta_db.conn.executemany(
+ "INSERT INTO songs (filename, mtime, size, title, artist) VALUES (?, NULL, 1, ?, NULL)",
+ [("zznull1.archive", "ZZ1"), ("zznull2.archive", "ZZ2")],
+ )
+ server_mod.meta_db.conn.commit()
+ offset_order = _walk_offset(client, sort, 4, 12)
+ keyset_order = _walk_keyset(client, sort, 4, 12)
+ assert keyset_order == offset_order
+ assert len(keyset_order) == 12 and len(set(keyset_order)) == 12
+
+
+def test_non_keyset_sort_offers_no_cursor(client, server_mod):
+ _seed(server_mod, 5)
+ body = client.get("/api/library", params={"sort": "tuning", "size": 2}).json()
+ assert body["next_cursor"] is None # compound sort → OFFSET only
+ assert len(body["songs"]) == 2