mirror of
https://github.com/got-feedBack/feedBack.git
synced 2026-07-26 23:01:22 +00:00
20de8a4a93
A tester's sloppak_cache reached 60 GB from an 1800-song library — his entire library, unpacked, none of it played. Stems are already-compressed audio, so an unpacked pack is ~1.1x its zip: the cache is a second, DECOMPRESSED copy of every song it touches. It had no size cap, no LRU, and no cleanup of any kind — not even when the song itself was deleted. Two halves: 1. resolve_source_dir() now evicts least-recently-used songs to stay under a cap (FEEDBACK_SLOPPAK_CACHE_MAX_MB, default 4 GB ≈ 130 songs of recency; 0 disables). The sweep runs on unpack — the only moment the cache grows — so it can't drift. An evicted song is dropped from _source_cache too: get_cached_source_dir() is the only thing media.py consults before falling back, so a stale path there would 404 every stem for the rest of the process instead of re-unpacking. get_cached_source_dir() now also verifies the dir still exists, which makes "just delete sloppak_cache/ to reclaim disk" safe advice. 2. read_member_bytes() reads ONE file out of a pack without unpacking it — the same trick read_cover_bytes() uses so the library grid doesn't explode every pack to show a cover. Unpacking a whole song to read a few KB of JSON is ~45x write amplification; doing it in a loop over the library is what produced the 60 GB. rig_builder's library-wide tone batch is the caller that did exactly that (fixed separately); this gives it, and everyone else, the right primitive. Eviction is concurrency-safe: unpacks run 2-at-a-time, so a dir being written is marked in-flight and the sweep skips it — checked and rmtree'd under one hold of the guard, and the marker is released even if the unpack raises (a leaked marker would make that dir permanently un-evictable). read_member_bytes normalizes both the requested path AND the archive's stored member names through safe_join, taking the last match — so './arrangements/x.json', backslash members from Windows tooling, and duplicate members that normalize to the same path all read back exactly as unpack-then-read did. Zip-slip is rejected before anything is opened. Tests: tests/test_sloppak_unpack_cache.py. All bite-tested (reverted each fix, watched it fail) — including one that was passing vacuously: a freshly-unpacked dir is the most-recently-used, so the LRU never reaches it and the in-flight race test proved nothing until the packs were sized to force the sweep that far. |
||
|---|---|---|
| .. | ||
| routers | ||
| acoustid_match.py | ||
| appconfig.py | ||
| appstate.py | ||
| audio_effects_db.py | ||
| audio.py | ||
| builtin_content.py | ||
| demo_mode.py | ||
| diagnostics_bundle.py | ||
| diagnostics_hardware.py | ||
| diagnostics_redact.py | ||
| dlc_paths.py | ||
| drums.py | ||
| enrichment.py | ||
| env_compat.py | ||
| gp2midi.py | ||
| gp2notation.py | ||
| gp2rs_gpx.py | ||
| gp2rs.py | ||
| gp8_audio_sync.py | ||
| gp_autosync.py | ||
| jsonc.py | ||
| library_registry.py | ||
| logging_setup.py | ||
| loosefolder.py | ||
| lyrics_transcribe.py | ||
| mb_match.py | ||
| metadata_db.py | ||
| midi_import.py | ||
| notation_lift.py | ||
| notation.py | ||
| progression.py | ||
| reqfields.py | ||
| safepath.py | ||
| scan_worker.py | ||
| scan.py | ||
| sloppak.py | ||
| song_score.py | ||
| song.py | ||
| songmeta.py | ||
| tailwind_rebuild.py | ||
| tones.py | ||
| tunings.py | ||
| vocal_pitch.py | ||
| wem_decode.py | ||
| xp.py | ||