perf(paths): resolve the library root once, not on every path check (#966)

`Path.resolve()` is a filesystem call — it lstats every component of the path.
`_resolve_dlc_path` and `safe_join` both re-resolved their ROOT on every single
call, and those run once per song, per art fetch, per scanned row.

Found while profiling a 2-fps report: on a real 50,944-song library the server
was issuing ~23,500 stat/lstat calls per second, re-walking the same three
parent directories over and over, and burning ~50% of a core doing it. It is
worst exactly where big libraries live — the library was on an NTFS-3G (FUSE)
mount, where every stat is a userspace round trip through mount.ntfs-3g (itself
visible in top). The cost was the constant re-resolution, not the work.

A root is fixed for the life of the process, so resolve it once
(safepath.resolved_root, lru_cache). Measured, 5,000 lookups against a real
library path:

    before:  15,264 stat syscalls   (54.2 ms)
    after:       277 stat syscalls   ( 0.8 ms)     55x fewer

Containment is unchanged, which is the part that matters:
  - safe_join still resolves the CANDIDATE on every call — following its
    symlinks IS the zip-slip / traversal defence, so it is never cached. Only
    the server-owned root is.
  - _resolve_dlc_path keeps its lexical containment check (deliberately does not
    follow symlinks, so junction-mounted libraries keep working).

Tradeoff, documented on resolved_root: if a root's symlink is re-pointed at a
NEW target while the server runs, the old target stays in effect until restart.
Fine for a library path fixed at startup; the cache is keyed on the Path, so
switching library dir is a different key.

Tests: root resolved once across 500 lookups (the regression), a different root
is a different entry, and the containment contract re-pinned — traversal,
Windows drive-absolute, backslash, NUL, empty, and a symlink escaping the root
must still be refused. Full suite green.
This commit is contained in:
Byron Gamatos
2026-07-14 19:48:39 +02:00
committed by GitHub
parent 2991612531
commit e729c44d5b
3 changed files with 143 additions and 4 deletions
+9 -1
View File
@@ -14,6 +14,7 @@ import os
from pathlib import Path
import appstate
from safepath import resolved_root
def _get_dlc_dir(cfg: dict | None = None) -> Path | None:
@@ -86,7 +87,14 @@ def _resolve_dlc_path(dlc: Path, filename: str) -> Path | None:
or PureWindowsPath(safe).drive):
return None
try:
root = dlc.resolve()
# The library root is fixed for the life of the process, but this
# function runs once per song / art fetch / scanned row — and
# `.resolve()` lstats every path component. Re-resolving here was
# ~23,500 stat calls/sec on a 50,944-song library, which pins a core
# when the library sits on a FUSE mount (NTFS-3G, SMB, sshfs) where each
# stat is a userspace round trip. Resolve the root once; see
# safepath.resolved_root for the caching contract.
root = resolved_root(dlc)
# normpath collapses `.`/`..`/duplicate separators purely lexically —
# it never touches the filesystem, so an in-library junction component
# is preserved (allowed) while `..`/absolute segments still escape and
+31 -3
View File
@@ -4,9 +4,34 @@ under a server-owned root.
from __future__ import annotations
from functools import lru_cache
from pathlib import Path
@lru_cache(maxsize=16)
def resolved_root(root: Path) -> Path:
"""Canonical (link-resolved) form of a server-owned root directory.
``Path.resolve()`` is a filesystem call: it lstats every component of the
path. The roots we join against — the DLC library, a plugin's asset dir —
are fixed for the life of the process, but the containment helpers below
(and ``dlc_paths._resolve_dlc_path``) were re-resolving them on EVERY call,
and those are called once per song, per art fetch, per scanned row.
On a real 50,944-song library that cost ~23,500 stat/lstat calls per second,
pinning a core. It is brutal when the library lives on a FUSE mount
(NTFS-3G, SMB, sshfs), where every stat is a userspace round trip: the same
three parent directories were being walked over and over.
Cached because a root is a constant here, not because resolution is cheap.
Consequence: if a root's symlink/junction is re-pointed at a NEW target
while the server is running, the old target stays in effect until restart.
That is fine for a library path fixed at startup, and the cache is keyed on
the Path, so switching to a different library dir is a different key.
"""
return root.resolve()
def safe_join(root: Path, name: str) -> Path | None:
"""Resolve ``name`` under ``root`` and return the resolved Path, or
``None`` if it would escape ``root`` or is unrepresentable.
@@ -35,9 +60,12 @@ def safe_join(root: Path, name: str) -> Path | None:
return None
safe = name.replace("\\", "/")
try:
root_resolved = root.resolve()
candidate = (root_resolved / safe).resolve()
if not candidate.is_relative_to(root_resolved):
# The ROOT is a constant — resolve it once (see resolved_root). The
# CANDIDATE must still be resolved on every call: following its symlinks
# is exactly the zip-slip / traversal defence, so it is never cached.
root_res = resolved_root(root)
candidate = (root_res / safe).resolve()
if not candidate.is_relative_to(root_res):
return None
except (ValueError, OSError):
return None