mirror of
https://github.com/got-feedBack/feedBack.git
synced 2026-07-21 20:31:21 +00:00
lib/scan.py (326). server.py 2,098 -> 1,870.
The background scan, its spawn ProcessPoolExecutor, and the kick/runner plumbing that
serialises passes. Bodies VERBATIM except the seam reads.
Everything shared is read LATE off appstate — the same contract every module in
lib/routers/ uses, and it is not cosmetic: tests monkeypatch CONFIG_DIR and swap meta_db,
so a value captured at import time pins the wrong one for the life of the process.
CONFIG_DIR -> appstate.config_dir
meta_db -> appstate.meta_db
_default_settings -> appstate.default_settings()
_stat_for_cache -> appstate.stat_for_cache()
━━━ THE SCAN STATUS IS REBOUND, NOT MUTATED ━━━
_background_scan does `global _scan_status; _scan_status = {**INIT, ...}` at every stage
transition. It REPLACES the dict; it never updates it in place. So nothing may hold that
dict by value — a reference captured once goes permanently stale at the first stage change
and would report "listing" forever while the scan ran to completion.
Hence `scan.status()`, a getter, and hence appstate publishes scan_status as a CALLABLE.
appstate.py already said so in a comment; this is the code that makes it true. (Same for
the plugin_context entry, which was already `lambda: dict(_scan_status)` — late-bound, so
it survives the move unchanged. The contract test from #898 covers it.)
━━━ appstate.server_root: A TRAP CLOSED PERMANENTLY ━━━
_background_scan seeds the builtin content, which needs the directory holding server.py.
`Path(__file__).resolve().parent` is correct in server.py and silently WRONG anywhere under
lib/ — it yields lib/, which holds no docs/ or data/ — and it fails by finding NOTHING
rather than by raising, so the seeds would just quietly never run.
lib/builtin_content.py (#900) closed that by taking the root as a parameter. This adds the
other half: server.py publishes it ONCE as appstate.server_root, so no module under lib/
ever has a reason to derive it. Documented at the slot.
pyflakes caught two more missing imports on the way in (loosefolder_mod, enrichment) —
each a NameError on a live scan path, and the suite would have handed them over one failure
at a time. It stays part of every server.py slice.
TESTS. The two scan fixtures (test_settings_api::scan_module,
test_feedpak_extension::scan_server) patched server._make_scan_executor to swap the spawn
pool for an in-process ThreadPool; they now patch it on lib/scan.py. Worth noting WHY that
still works: the fixtures re-import `server` per test, but `scan` stays cached in
sys.modules — and it picks up the fresh CONFIG_DIR anyway, because the appstate reads are
late-bound. The seam is doing exactly the job it was built for.
━━━ TEST ISOLATION: A REGRESSION THE CARVE ITSELF CREATED (Codex [P2]) ━━━
background_scan() deliberately NEVER sets running=False — ownership of that flag lives in
_scan_runner, so a kick_scan() racing the terminal write cannot observe a stale False and
start a second runner. Correct in production.
But the scan fixtures call background_scan() DIRECTLY, skipping the runner. That was
harmless while the state lived on `server`, which the fixtures RE-IMPORT per test. It is
NOT harmless now: `scan` stays cached in sys.modules across sys.modules.pop("server"), so
the status dict OUTLIVES the test. One direct call leaves the shared scanner marked
"running" forever, and every later scan or rescan returns "already in progress" and quietly
does nothing.
Verified: after a direct call, kick_scan() returns False and starts no scan at all.
The suite passed anyway, on ordering luck — which is exactly how this class of bug ships.
tests/conftest.py::reset_scan_state now snapshots and restores lib/scan.py's module state
around the two fixtures that drive it directly.
pytest 2398, pyflakes 0, Codex 0.
Refs #48
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
170 lines
6.9 KiB
Python
170 lines
6.9 KiB
Python
"""Shared pytest fixtures for the feedBack test suite."""
|
|
|
|
import importlib
|
|
import logging
|
|
import sys
|
|
|
|
import pytest
|
|
import structlog
|
|
|
|
|
|
_LOGGING_NAMES = ("feedBack", "uvicorn", "uvicorn.error", "uvicorn.access")
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _reset_enrichment_state():
|
|
"""Reset the enrichment worker's process-global state between tests.
|
|
|
|
The `server` fixtures pop-and-reimport `server`, but `lib/enrichment.py`
|
|
(which now owns the worker) stays imported for the whole session, so its
|
|
module globals — the cancel Event, the status dict, the caches — would
|
|
otherwise leak across tests. A test that set `_enrich_cancel` (or a stale
|
|
`running` status) could silently short-circuit a later direct
|
|
`_background_enrich()` call. Clear it up front so each test starts clean.
|
|
"""
|
|
try:
|
|
import enrichment
|
|
except ImportError:
|
|
yield
|
|
return
|
|
enrichment._enrich_cancel.clear()
|
|
enrichment._enrich_pending_pass = False
|
|
enrichment._enrich_status.update(
|
|
{"running": False, "processed": 0, "last_pass_at": None,
|
|
"total": 0, "matched": 0, "current": None})
|
|
enrichment._enrich_last_fetch = 0.0
|
|
enrichment._artist_alias_cache.clear()
|
|
# _caa_index_locks is deliberately left alone: it's guarded by
|
|
# _caa_index_locks_guard, so clearing it here (unlocked) would race a
|
|
# still-alive worker thread, and its entries are stateless per-release
|
|
# mutexes that don't leak test state anyway.
|
|
yield
|
|
|
|
|
|
@pytest.fixture()
|
|
def isolate_logging():
|
|
"""Restore feedBack / uvicorn logger state after each test.
|
|
|
|
Saves handlers, level, and propagate flag before the test runs and
|
|
restores all three on teardown. Import into any test module that calls
|
|
configure_logging() so mutations don't bleed across tests.
|
|
"""
|
|
saved = {}
|
|
for name in _LOGGING_NAMES:
|
|
lg = logging.getLogger(name)
|
|
saved[name] = (
|
|
list(lg.handlers), # snapshot the handler list
|
|
lg.level,
|
|
lg.propagate,
|
|
)
|
|
yield
|
|
for name in _LOGGING_NAMES:
|
|
lg = logging.getLogger(name)
|
|
original_handlers, original_level, original_propagate = saved[name]
|
|
|
|
# Close and remove any handlers that were added during the test.
|
|
for h in list(lg.handlers):
|
|
if h not in original_handlers:
|
|
lg.removeHandler(h)
|
|
h.close()
|
|
# Remove any original handlers that may have been removed during the test
|
|
# so we can add them back cleanly.
|
|
for h in list(lg.handlers):
|
|
lg.removeHandler(h)
|
|
# Reattach the original handlers.
|
|
for h in original_handlers:
|
|
lg.addHandler(h)
|
|
|
|
lg.setLevel(original_level)
|
|
lg.propagate = original_propagate
|
|
structlog.reset_defaults()
|
|
|
|
|
|
# ── Plugin-loader isolation ─────────────────────────────────────────────────────
|
|
#
|
|
# Lifted verbatim out of tests/test_plugins.py so more than one test module can drive
|
|
# the real plugins.load_plugins(). It has to be ONE fixture, not a copy per file:
|
|
# load_plugins() mutates sys.path, sys.modules, PENDING_PLUGINS and LOADED_PLUGINS, and a
|
|
# partial restore makes the suite order- and environment-dependent (Codex [P2] on
|
|
# test_plugin_context_contract.py — it was right).
|
|
|
|
# Bare module names that this test module pre-populates into
|
|
# sys.modules to simulate the bare-import path. Saved/restored by
|
|
# the reset_plugin_state fixture so they don't leak to other test
|
|
# files. Codex / Copilot review on PR for feedBack#33.
|
|
_BARE_NAMES_USED = ("util", "extractor")
|
|
|
|
|
|
@pytest.fixture()
|
|
def reset_plugin_state(monkeypatch):
|
|
"""Clear loader module-level state and restore on teardown.
|
|
|
|
Saves and restores:
|
|
* `plugins.LOADED_PLUGINS`
|
|
* any `plugin_*` keys we add to `sys.modules`
|
|
* the bare names this module simulates (`util`, `extractor`)
|
|
* `sys.path` — `plugins.load_plugins()` mutates it
|
|
Also unsets `FEEDBACK_PLUGINS_DIR` for the test's duration
|
|
(via monkeypatch) so a CI env that pre-sets it can't leak
|
|
real user plugins into a tmp_path-driven test. Per-module
|
|
locks are owned by the standard import system
|
|
(`importlib._bootstrap._module_locks`) and are not our
|
|
responsibility to reset.
|
|
"""
|
|
monkeypatch.delenv("FEEDBACK_PLUGINS_DIR", raising=False)
|
|
plugins = importlib.import_module("plugins")
|
|
saved_loaded = list(plugins.LOADED_PLUGINS)
|
|
saved_pending = dict(plugins.PENDING_PLUGINS)
|
|
saved_modules = {k: v for k, v in sys.modules.items() if k.startswith("plugin_")}
|
|
saved_bare = {k: sys.modules[k] for k in _BARE_NAMES_USED if k in sys.modules}
|
|
saved_path = list(sys.path)
|
|
plugins.LOADED_PLUGINS.clear()
|
|
plugins.PENDING_PLUGINS.clear()
|
|
for k in list(sys.modules):
|
|
if k.startswith("plugin_") or k in _BARE_NAMES_USED:
|
|
del sys.modules[k]
|
|
try:
|
|
yield plugins
|
|
finally:
|
|
plugins.LOADED_PLUGINS.clear()
|
|
plugins.LOADED_PLUGINS.extend(saved_loaded)
|
|
plugins.PENDING_PLUGINS.clear()
|
|
plugins.PENDING_PLUGINS.update(saved_pending)
|
|
for k in list(sys.modules):
|
|
if k.startswith("plugin_") or k in _BARE_NAMES_USED:
|
|
del sys.modules[k]
|
|
sys.modules.update(saved_modules)
|
|
sys.modules.update(saved_bare)
|
|
sys.path[:] = saved_path
|
|
|
|
|
|
# ── Scanner isolation ───────────────────────────────────────────────────────────
|
|
#
|
|
# lib/scan.py holds MODULE-LEVEL state (_scan_status, and the kick/runner bookkeeping),
|
|
# and `scan` is NOT re-imported by the fixtures that re-import `server` — so unlike the
|
|
# old server-globals arrangement, that state now outlives a test.
|
|
#
|
|
# It matters because of a deliberate asymmetry in the scanner: background_scan() never
|
|
# sets `running` back to False. Ownership of that flag lives in _scan_runner, so that a
|
|
# kick_scan() racing the terminal write cannot see a stale False and start a second runner.
|
|
# Correct in production — but a test that calls background_scan() DIRECTLY skips the runner
|
|
# entirely and therefore leaves the scanner marked "running" forever. Every later scan or
|
|
# rescan then returns "already in progress" and quietly does nothing.
|
|
#
|
|
# The suite passed anyway, on ordering luck. Codex [P2] caught it. So: snapshot and restore.
|
|
@pytest.fixture()
|
|
def reset_scan_state():
|
|
"""Restore lib/scan.py's module-level state around a test that drives it directly."""
|
|
import scan
|
|
|
|
saved_status = scan._scan_status
|
|
saved_thread = scan._scan_thread
|
|
saved_pending = scan._scan_rescan_pending
|
|
scan._scan_status = dict(scan._SCAN_STATUS_INIT)
|
|
try:
|
|
yield scan
|
|
finally:
|
|
scan._scan_status = saved_status
|
|
scan._scan_thread = saved_thread
|
|
scan._scan_rescan_pending = saved_pending
|