From 514461167eafe34ba5f7b7a6ff1a4d058a3529b5 Mon Sep 17 00:00:00 2001 From: Byron Gamatos Date: Fri, 10 Jul 2026 21:12:54 +0200 Subject: [PATCH] refactor(server): extract the highway WebSocket into routers/ws_highway.py (R3) (#844) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The single largest handler in server.py — the 902-line /ws/highway/{filename} chart streamer — plus its 3 exclusive helpers (_pick_smart_arrangement, _sanitize_authors, _sanitized_song_offset). server.py: 9,008 -> 8,003 (-1,005, the biggest single R3 cut). Clean move despite the size: the handler's only server-module deps are the 4 path constants + 3 exclusive helpers + app + log. Everything else it uses is either NESTED inside the handler (_evict_audio_cache, _fill_scale_degree, _manifest_entries, _tone_names, _xml_rank, _send_keepalives) or imported from the shared lib modules (song/audio/sloppak/drums/notation/dlc_paths/metadata_db). - Path constants read through the appstate seam: added static_dir / sloppak_cache_dir / audio_cache_dir slots (config_dir already there); server.py configures them. Bodies otherwise verbatim (@app.websocket -> @router.websocket, PATHS -> appstate.*, log -> module logger). - sloppak_cache_dir IS setattr-patched, so the 3 test_highway_ws_* suites now also `setattr(appstate, "sloppak_cache_dir", ...)` next to their existing server patch. _sanitize_authors unit tests import it from routers.ws_highway (it moved). No other test churn. - Removed 23 now-dead imports from server.py (song/audio/drums/notation/ bisect/contextvars/structlog/WebSocket*/_arr_smart_sort_key) — diffed against the origin/main unused-import baseline so only NEWLY-dead ones went. owns_tmp (assigned, never read) moved verbatim — it's pre-existing dead on origin/main too; left as-is to keep the move faithful. Verified: route table identical to origin/main (143, paths/methods/order); handler body verbatim spot-checked; pyflakes clean (server has no new undefined/dead); pytest 2400 passed; packaging guard 53; eslint 0. Boot smoke: the highway WS streams the full chart (song_info/beats/sections/notes/chords/ notation/anchors/drum_tab -> ready) BYTE-for-byte the same message sequence as origin/main across arrangements 0/1/2, zero tracebacks. Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 2 +- docs/size-exemptions.md | 4 +- lib/appstate.py | 14 +- lib/routers/ws_highway.py | 1048 +++++++++++++++++++ server.py | 1023 +----------------- tests/test_highway_ws_authors.py | 11 +- tests/test_highway_ws_instrument_routing.py | 3 + tests/test_highway_ws_notation.py | 3 + 8 files changed, 1087 insertions(+), 1021 deletions(-) create mode 100644 lib/routers/ws_highway.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 319d4a6..1e6ddf7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,7 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 next root-level module can't ship broken. ### Added -- **`routers/` — extracting `server.py`'s route layer, cheapest-first (R3).** Each PR moves a cohesive route group into a `fastapi.APIRouter` under `lib/routers/`, mounted with `app.include_router(...)` at its original site (FastAPI matches in registration order; the full route table stays byte-identical). Bodies are verbatim — only the decorator receiver (`@app` → `@router`) and singleton reads (`meta_db` → `appstate.meta_db`, resolved at call time) change. So far: `audio_effects` (5), `artist_aliases` (5), `loops` (3), `playlists` (12 + custom covers). The DLC library-path resolution (`_get_dlc_dir`, pure `_resolve_dlc_path`) moved to `lib/dlc_paths.py`, reading paths through the seam; `config_dir`/`dlc_dir`/`dlc_dir_env` now ride the `appstate` seam (env-derived, so the pop-and-reimport fixtures reconfigure it for free), and the shared request-field sanitizer `_clean_str` moved to `lib/reqfields.py`. The next cut is picked by a dependency-closure scan that ranks groups by how many `monkeypatch.setattr(server, …)` targets they'd drag along. +- **`routers/` — extracting `server.py`'s route layer, cheapest-first (R3).** Each PR moves a cohesive route group into a `fastapi.APIRouter` under `lib/routers/`, mounted with `app.include_router(...)` at its original site (FastAPI matches in registration order; the full route table stays byte-identical). Bodies are verbatim — only the decorator receiver (`@app` → `@router`) and singleton reads (`meta_db` → `appstate.meta_db`, resolved at call time) change. So far: `audio_effects` (5), `artist_aliases` (5), `loops` (3), `playlists` (12 + covers), `ws_highway` (the 902-line highway chart WebSocket — the single largest handler). The DLC library-path resolution (`_get_dlc_dir`, pure `_resolve_dlc_path`) moved to `lib/dlc_paths.py`, reading paths through the seam; `config_dir`/`dlc_dir`/`dlc_dir_env` now ride the `appstate` seam (env-derived, so the pop-and-reimport fixtures reconfigure it for free), and the shared request-field sanitizer `_clean_str` moved to `lib/reqfields.py`. The next cut is picked by a dependency-closure scan that ranks groups by how many `monkeypatch.setattr(server, …)` targets they'd drag along. - **`routers/` — the first extracted route module (R3).** The five audio-effects mapping endpoints move out of `server.py` into `lib/routers/audio_effects.py` as a `fastapi.APIRouter`, mounted with `app.include_router(...)` **at the point in the file diff --git a/docs/size-exemptions.md b/docs/size-exemptions.md index 89d46f5..e774c75 100644 --- a/docs/size-exemptions.md +++ b/docs/size-exemptions.md @@ -55,8 +55,8 @@ without a *signed* exemption" is unenforceable. ## Planned, NOT exempt (owned by split plans — listed so nothing falls between states) core `static/app.js` (11,852) · `static/highway.js` (4,168, whole file) · `server.py` -(9,008 — was 14,037; ratcheted by the R3 `MetadataDB` + `AudioEffectsMappingDB` -extractions and four `routers/` modules) · +(8,003 — was 14,037; ratcheted by the R3 `MetadataDB` + `AudioEffectsMappingDB` +extractions and five `routers/` modules) · `lib/metadata_db.py` (4,373 — new in R3; the `MetadataDB` class alone is 4,018 lines and is a monolith in its own right, to be split per-table once the router train lands) · `static/v3/songs.js` (4,134) · `static/capabilities/audio-session.js` diff --git a/lib/appstate.py b/lib/appstate.py index 43e8b19..2b599db 100644 --- a/lib/appstate.py +++ b/lib/appstate.py @@ -73,8 +73,20 @@ config_dir = None dlc_dir = None # the DLC_DIR env value as a Path (Path("") if unset) dlc_dir_env = None # the raw DLC_DIR env string, "" if unset — distinguishes # "unset" from Path("")→"." (see dlc_paths._get_dlc_dir) +# Cache/asset dirs. static_dir + sloppak_cache_dir are patched via +# `setattr(server, …)` in a few tests, so a router reading them here needs those +# setattr sites retargeted to `setattr(appstate, …)` in the same PR (ws_highway +# retargets the 3 test_highway_ws_* SLOPPAK sites). config_dir-derived dirs are +# reconfigured for free on a setenv+reimport. +static_dir = None +sloppak_cache_dir = None +audio_cache_dir = None -_SLOTS = frozenset({"meta_db", "audio_effect_mappings", "config_dir", "dlc_dir", "dlc_dir_env"}) +_SLOTS = frozenset({ + "meta_db", "audio_effect_mappings", + "config_dir", "dlc_dir", "dlc_dir_env", + "static_dir", "sloppak_cache_dir", "audio_cache_dir", +}) def configure(**kwargs) -> None: diff --git a/lib/routers/ws_highway.py b/lib/routers/ws_highway.py new file mode 100644 index 0000000..c420780 --- /dev/null +++ b/lib/routers/ws_highway.py @@ -0,0 +1,1048 @@ +"""The highway chart WebSocket — /ws/highway/{filename}. + +The single largest handler in server.py, extracted to its own router (R3). Path +constants are read through the appstate seam (appstate.static_dir / +sloppak_cache_dir / audio_cache_dir / config_dir); the smart-arrangement / +author / offset helpers it exclusively uses move with it. Everything else is +nested inside the handler or imported from the shared lib modules. Logs through +the same `feedBack.server` logger name, so existing filters resolve unchanged. +""" + +import asyncio +import bisect +import contextvars +import hashlib +import json +import logging +import math +import os +import shutil +import uuid +from pathlib import Path + +import structlog + +from fastapi import APIRouter, WebSocket, WebSocketDisconnect + +from song import ( + anchor_to_wire, + arrangement_string_count, + base_open_string_midis, + chord_template_to_wire, + chord_to_wire, + compute_smart_names, + hand_shape_to_wire, + key_to_tonic_pc, + load_song, + note_to_wire, + phrase_to_wire, + pitch_from_base, + scale_degree_for_pitch, +) +from audio import find_wem_files, convert_wem +import sloppak as sloppak_mod +import drums as drums_mod +import notation as notation_mod +import loosefolder as loosefolder_mod +from metadata_db import _arr_smart_sort_key +from dlc_paths import _get_dlc_dir, _resolve_dlc_path + +import appstate + +log = logging.getLogger("feedBack.server") + +router = APIRouter() + + +def _pick_smart_arrangement( + arrangements: list, + smart_names: list, + pref: str, +) -> int: + """Return the best arrangement index for `pref` using smart-name priority. + + Priority order: + 1. Exact match — smart_name == pref (e.g. "Lead") + 2. Alt. variants — "Alt. Lead", "Alt. Lead 1", ... + 3. Bonus variants — "Bonus Lead", "Bonus Lead 1", ... + 4. First arrangement in smart sort order (Lead > Rhythm > Bass > ...) + + Returns -1 when `pref` is empty / "Auto" or `arrangements` is empty + (caller falls through to the existing most-notes fallback). + """ + pref = (pref or "").strip() + if not pref or pref.lower() == "auto" or not arrangements: + return -1 + + sorted_pairs = sorted( + enumerate(smart_names), + key=lambda x: _arr_smart_sort_key({"smart_name": x[1]}), + ) + + alt_prefix = f"Alt. {pref}" + bonus_prefix = f"Bonus {pref}" + + for i, sn in sorted_pairs: + if sn == pref: + return i + + for i, sn in sorted_pairs: + if sn and (sn == alt_prefix or sn.startswith(alt_prefix + " ")): + return i + + for i, sn in sorted_pairs: + if sn and (sn == bonus_prefix or sn.startswith(bonus_prefix + " ")): + return i + + if sorted_pairs: + return sorted_pairs[0][0] + return 0 + + +def _sanitized_song_offset(song) -> float: + """Return song.offset coerced to a finite float, or 0.0. + + Malformed loose-folder XMLs can put `NaN`/`Infinity` into ; + Python's `float()` happily accepts those, but Starlette's JSON + encoder then emits the literal `NaN` token which is invalid JSON + and breaks the frontend's song_info parsing. + """ + try: + v = float(getattr(song, "offset", 0.0)) + except (TypeError, ValueError): + return 0.0 + return v if math.isfinite(v) else 0.0 + + +def _sanitize_authors(manifest: dict | None) -> list[dict]: + """Extract a display-safe contributor list from a feedpak manifest. + + The feedpak spec (§5.4) defines an OPTIONAL top-level `authors` list of + objects `{name (required), role?, email?, url?}`. We surface only `name` + and `role` to the highway — contact fields (email/url) are intentionally + dropped from the on-screen credits. Malformed entries (non-dict, missing / + blank name) are skipped; absent / non-list `authors` yields `[]`. + """ + if not isinstance(manifest, dict): + return [] + raw = manifest.get("authors") + if not isinstance(raw, list): + return [] + out: list[dict] = [] + for entry in raw: + if not isinstance(entry, dict): + continue + name = entry.get("name") + if not isinstance(name, str) or not name.strip(): + continue + role = entry.get("role") + out.append({ + "name": name.strip(), + "role": role.strip() if isinstance(role, str) and role.strip() else None, + }) + return out + + +@router.websocket("/ws/highway/{filename:path}") +async def highway_ws(websocket: WebSocket, filename: str, arrangement: int = -1, naming_mode: str = "legacy"): + """Stream song data for the highway renderer over WebSocket.""" + await websocket.accept() + structlog.contextvars.bind_contextvars(ws_conn_id=uuid.uuid4().hex[:8]) + + dlc = _get_dlc_dir() + if not dlc: + await websocket.send_json({"error": "DLC folder not configured"}) + await websocket.close() + return + + song_path = _resolve_dlc_path(dlc, filename) + if song_path is None: + await websocket.send_json({"error": "forbidden"}) + await websocket.close() + return + if not song_path.exists(): + await websocket.send_json({"error": "File not found"}) + await websocket.close() + return + + is_slop = sloppak_mod.is_sloppak(song_path) + # Sloppak wins precedence: `_extract_meta_for_file()` and the + # background scanner both treat a `.sloppak` directory as sloppak + # even if it happens to contain WEM/XML. Gate is_loose on that + # so the loose-only branches (audio_id, offset, audio conversion) + # don't fire for sloppak bundles. + is_loose = (not is_slop) and loosefolder_mod.is_loose_song(song_path) + tmp = None + owns_tmp = False + loaded_slop = None # LoadedSloppak when is_slop + _keepalive_active = True + + async def _send_keepalives(): + while _keepalive_active: + try: + await asyncio.sleep(3) + if _keepalive_active: + await websocket.send_json({"type": "loading", "stage": "Loading..."}) + except Exception: + break + + try: + await websocket.send_json({"type": "loading", "stage": "Extracting..."}) + keepalive_task = asyncio.create_task(_send_keepalives()) + + try: + loop = asyncio.get_running_loop() + _ctx = contextvars.copy_context() + if is_slop: + appstate.sloppak_cache_dir.mkdir(parents=True, exist_ok=True) + loaded_slop = await loop.run_in_executor( + None, + lambda: _ctx.run(sloppak_mod.load_song, filename, dlc, appstate.sloppak_cache_dir), + ) + song = loaded_slop.song + tmp = str(loaded_slop.source_dir) + owns_tmp = False + elif is_loose: + # Loose folders need no extraction — load_song reads the + # arrangement XMLs directly from the flat directory. + # song_path is already DLC-containment-validated by + # _resolve_dlc_path, so audio conversion below can use + # it directly. + song = await loop.run_in_executor(None, lambda: load_song(str(song_path))) + tmp = str(song_path) + owns_tmp = False + else: + # Only open formats (.sloppak bundles and loose folders) are + # servable. There is no fallback container extraction path. + raise ValueError("Unsupported song format") + finally: + _keepalive_active = False + keepalive_task.cancel() + + if not song.arrangements: + await websocket.send_json({"error": "No arrangements found"}) + await websocket.close() + return + + # Smart names are needed for smart-mode arrangement selection. + smart_names = compute_smart_names(song.arrangements) + + # Pick arrangement: explicit request > user preference > most notes + best = -1 + if 0 <= arrangement < len(song.arrangements): + best = arrangement + else: + # Read the user's config once: their selected instrument (route the chart + # to the matching part) and their default-arrangement preference. + pref = "" + sel_instrument = "" + config_file = appstate.config_dir / "config.json" + if config_file.exists(): + try: + _cfg = json.loads(config_file.read_text(encoding="utf-8")) + pref = _cfg.get("default_arrangement", "") + sel_instrument = (_cfg.get("instrument", "") or "") + except Exception: + pass + # Instrument routing: load the part that matches the selected instrument so + # "your instrument" and "the chart you play" line up. The default ordering + # is Lead/guitar-first, so without this a bass player gets handed a guitar + # chart (and any tune-check then compares a 4-string bass against a 6-string + # part). Currently routes bass -> a Bass arrangement; guitar — and any + # unknown/future instrument (drums, keys) — falls through to the + # preference/most-notes logic below, which already lands on a guitar part. + # Drums/keys get their own match when those arrangement types + selector + # entries land. Only applies when no explicit arrangement was requested, so + # a manual arrangement switch is always respected. + if sel_instrument.lower() == "bass": + # Candidate bass parts, preferring the structured pathBass flag; the + # normalized smart name (itself pathBass-derived) and raw name are + # fallbacks for sources without the flag. + bass_idxs = [ + i + for i, a in enumerate(song.arrangements) + if getattr(a, "path_bass", False) + or (smart_names[i] or "").lower().startswith("bass") + or "bass" in (getattr(a, "name", "") or "").lower() + ] + if bass_idxs: + # Among the bass parts: (1) honor the saved default-arrangement + # preference if it names one of them (so a bass player who prefers + # "Bass 2"/"Alt. Bass" keeps it), (2) else the canonical main "Bass", + # (3) else the first bass part in order. + pref_bass = -1 + if pref: + for i in bass_idxs: + nm = (smart_names[i] if naming_mode == "smart" and i < len(smart_names) + else getattr(song.arrangements[i], "name", "")) + if nm == pref: + pref_bass = i + break + if pref_bass >= 0: + best = pref_bass + else: + best = next( + (i for i in bass_idxs + if (smart_names[i] if i < len(smart_names) else "") == "Bass"), + bass_idxs[0], + ) + # User's default arrangement preference (only when instrument routing did not + # already resolve a part — i.e. guitar, or a bass player with no bass part). + if best < 0 and pref: + if naming_mode == "smart": + best = _pick_smart_arrangement(song.arrangements, smart_names, pref) + else: + for i, a in enumerate(song.arrangements): + if a.name == pref: + best = i + break + if best < 0: + # Fallback: most notes + best = 0 + best_count = 0 + for i, a in enumerate(song.arrangements): + c = len(a.notes) + sum(len(ch.notes) for ch in a.chords) + if c > best_count: + best_count = c + best = i + arr = song.arrangements[best] + + # Resolve the manifest arrangement id for notation lookup (Option B loader). + # Use the parallel arrangement_ids list (indexed by compacted position, + # i.e. song.arrangements index) so skipped manifest entries can't shift + # the index and serve the wrong arrangement's notation. + _notation_arr_id: str | None = None + if is_slop and loaded_slop is not None: + _ids = loaded_slop.arrangement_ids + if best < len(_ids): + _notation_arr_id = _ids[best] + + # Convert audio with unique filename (check cache first) + audio_url = None + audio_error: str | None = None # Surfaced in song_info when audio_url is None + stems_payload: list[dict] = [] + # URL of the single full-mix audio (sloppak `original_audio:`), when the + # pack ships one. The stems plugin uses this to play the untouched mix + # while every stem slider is at unity; None otherwise (separate stems + # only, loose folder, or archive). + original_audio_url: str | None = None + if is_loose: + # Loose folder filenames are relative paths (artist/album/song). + # Hash the *canonical* dlc-relative path (so two URL spellings + # of the same physical folder share a cache key) PLUS the + # source WEM's mtime+size so: + # - different songs with the same leaf folder name can't + # collide (a `/`→`__` escape would collapse `a/b__c` and + # `a__b/c`); + # - editing audio.wem in place invalidates the cached + # converted file (without this, in-place custom song iteration + # keeps serving the stale mp3/ogg from the cache). + try: + canonical = song_path.relative_to(dlc.resolve()).as_posix() + except ValueError: + canonical = filename + wem_for_id = loosefolder_mod.find_audio(song_path) + try: + wem_stat = wem_for_id.stat() if wem_for_id else None + except OSError: + wem_stat = None + stamp = f"{wem_stat.st_mtime_ns}-{wem_stat.st_size}" if wem_stat else "" + digest = hashlib.sha256( + (canonical + "|" + stamp).encode("utf-8") + ).hexdigest()[:12] + leaf = Path(canonical.rstrip("/\\")).stem.replace(" ", "_")[:40] or "song" + audio_id = f"{leaf}_{digest}" + else: + audio_id = Path(filename).stem.replace(" ", "_") + + if is_slop: + # Stems are served via the sloppak file endpoint; the first stem + # (or explicit default) is the core