R0: module-migration rails (src/ serving, live-edit cache, scriptType loading, governance) (#812)
Some checks failed
ship-ci / ci (push) Has been cancelled

Host enablement for the plugin ES-module migration: sandboxed /api/plugins/{id}/src/ serving, no-cache+weak-ETag/304 live-edit caching on src/+screen.js+assets, scriptType:module loader injection + scriptType/minHost manifest passthrough; constitution v1.2.0 + module playbook + signed size-exemptions register + maintainer/CI-only ESLint gate; rerunnable perf-baseline harness. Reviewed by Codex (local), Copilot, and CodeRabbit.
This commit is contained in:
Byron Gamatos 2026-07-08 10:14:40 +02:00 committed by GitHub
parent a18a818e8b
commit 950e348357
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
15 changed files with 2520 additions and 19 deletions

View File

@ -123,3 +123,28 @@ jobs:
sys.exit(1)
print(f"Validated {len(manifests)} manifest(s) — OK")
EOF
lint:
# Maintainer/CI-only size + module-hygiene gate (constitution Principle I:
# dev tooling, never on the serve/Docker path — same category as
# scripts/build-tailwind.sh). max-lines WARNS (the 1,500-line size ratchet;
# non-blocking), while import-x/no-unresolved + no-cycle HARD-ERROR on the
# ES-module graphs the refactor produces. Exemptions: docs/size-exemptions.md.
name: lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# npm ci runs third-party postinstall scripts; don't leave the token in
# git config for them (this job never pushes).
with:
persist-credentials: false
- uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: ESLint (size norm + module hygiene)
run: npm run lint

View File

@ -48,11 +48,30 @@ output is committed, so Docker / desktop / end users never build. New
features extend `app.js` and the existing globals (`window.playSong`,
`window.showScreen`, `window.createHighway`, `window.feedBack`).
Native ES modules are a first-class, build-free extension mechanism.
Because `<script type="module">` and `import` are browser features — not
a bundler — a large source file MAY be split into an `import`-ed module
graph of plain source files, with **no build step and no framework**. A
plugin opts in with `"scriptType": "module"` in `plugin.json`: its
`screen.js` becomes a one-line `import './src/main.js'`, and the host
serves the `src/` subtree from the sandboxed `/api/plugins/<id>/src/…`
route and injects the entry as `<script type="module">`. The classic
global-scope `screen.js` path remains fully supported; both coexist, and
module scripts are still source-served — the no-bundler, no-transpiler,
build-free-at-serve rule is unchanged. Core's own `static/` tree may
migrate to the same module-graph shape (`static/js/…`) over time under
this rule.
**Non-negotiable rules**
- Do not introduce a frontend framework, JSX, or a JS build pipeline in
core. Plugins MAY ship their own bundled assets but core MUST remain
source-served.
- ES-module plugins remain source-served: no bundler or transpiler, and
their own asset URLs (worklets, WASM, images) resolve via
`import.meta.url` — never `document.currentScript`, which is `null`
inside a module. `scriptType:"module"` and the optional `minHost`
version floor are the only new `plugin.json` keys the module path adds.
- Because the core Tailwind stylesheet is prebuilt, it contains only the
classes present in core source at build time. Core's committed
`static/tailwind.min.css` MUST stay in sync with source — CI enforces
@ -214,6 +233,15 @@ no `..`, no absolute paths).
runs first). Plugins MUST tolerate dependent globals being absent
at load time and check at runtime
(`typeof window.X === 'function'`).
- **Module load contract**: a `scriptType:"module"` plugin is injected
as `<script type="module">`, whose load event fires only after its
whole static-import graph fetches and evaluates — so the loader's
completion-by-`onload` guarantee (and the `playSong` wrapper-chain
order above) is preserved exactly. The host loads `screen.js` once per
version and `showScreen` re-injects nothing, so a plugin's per-visit
re-initialization comes from its `screen:changed` handler, not from
screen.js re-running; ES-module plugins inherit this unchanged (module
top-level code does not re-execute on same-version re-mount).
## Development Workflow
@ -256,4 +284,4 @@ no `..`, no absolute paths).
higher-numbered principle's escape hatch is to live in a plugin
with its own bundled assets.
**Version**: 1.1.0 | **Ratified**: 2026-05-09 | **Last Amended**: 2026-06-01
**Version**: 1.2.0 | **Ratified**: 2026-05-09 | **Last Amended**: 2026-07-08

View File

@ -8,6 +8,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **Plugins can ship an ES-module `src/` tree (module-migration rails, R0).** The host gains three things so a plugin can move off a single global-scope `screen.js` IIFE onto native ES modules with **no build step**: (1) a new sandboxed `GET /api/plugins/{id}/src/{path}` route that serves a plugin's `src/` source subtree, containment-checked by the same `safe_join` guard as `assets/` (traversal/absolute/NUL → 404); (2) the live-edit cache contract — `Cache-Control: no-cache` + a weak mtime/size `ETag` + `If-None-Match`→`304` — applied to `src/`, `screen.js`, and `assets/` (previously `screen.js` sent no cache headers and `assets/` emitted an ETag but never revalidated), so an edited module reloads on refresh while unchanged ones `304`; and (3) `scriptType`/`minHost` passthrough from `plugin.json` to `/api/plugins`, with the loader injecting a plugin that declares `"scriptType":"module"` as `<script type="module">` (its screen.js becomes `import './src/main.js'`). A `<script type=module>` fires its load event only after its whole static-import graph evaluates, preserving the loader's completion-by-`onload` + `_loadingPluginId` contract. Classic plugins are unaffected; `minHost` is passthrough-only for now (enforcement deferred). Tests: `tests/test_plugin_src_route.py` (serve/media-type/traversal/304/no-stale-304/screen.js+assets conditional), `tests/js/plugin_loader_script_type.test.js` (guarded module injection).
- **Module-migration governance & rails (R0).** Constitution amended to **v1.2.0**: Principle II now names native ES modules as a first-class, *build-free* extension mechanism (the `scriptType:"module"` load path, both plugins and — over time — core's `static/js/`), keeping the no-bundler/no-transpiler/source-served rule intact; Operating Constraints gains a "Module load contract" clause (a `<script type=module>` load event awaits the whole static-import graph, so completion-by-`onload` is preserved; per-visit re-init comes from the `screen:changed` event, not screen.js re-execution). Mirrored into `CLAUDE.md`. New `docs/plugin-modules.md` (the migration playbook — layering, import-time purity, `import.meta.url` assets, the ETag live-edit loop) and `docs/size-exemptions.md` (the signed 1,500-line size-norm register; Byron signs core/bundled rows, Christian the authored virtuoso row). Adds a **maintainer/CI-only** ESLint gate (`eslint.config.js` + a `lint` CI job): `max-lines` warns at 1,500 as a non-blocking ratchet (ceilings for exempt files mirror the register), and `import-x/no-unresolved` + `import-x/no-cycle` hard-error on ES-module graphs — dormant until module code lands, never on the serve/Docker path.
- **Perf-baseline harness (R0).** `scripts/perf-baseline.mjs` (maintainer-only, Playwright-driven) captures server p50/p95/p99 latency, cold boot-to-interactive, JS-heap after an idle soak, and the injected plugin-script shape (how many are `type=module`), so every refactor phase can be checked for "screen-entry and frame-time no worse." Methodology + the R0 baseline live in `docs/perf-baseline.md`; playback frame-time and chart-loaded screen-entry rows need a seeded library and are re-taken per environment.
- **Sort and filter the library by your personal difficulty rating — now visible at a glance, not just in the edit drawer.** `song_user_meta.user_difficulty` (the 15 planning rating, settable manually or seeded by a plugin like the community `difficulty_tagger`) already existed but was only readable by opening a song's per-song details drawer. The library API gains `sort=difficulty` / `sort=difficulty-desc` — a correlated subquery over `song_user_meta`, following the same unrated-rows-sort-to-the-bottom-in-both-directions pattern as the existing `mastery` sort — and library cards now show the rating as a `◆N` badge (v2 grid/tree views and the v3 grid alike), next to the tuning and lyrics badges. The classic tree view's `query_artists` batch-attaches `user_difficulty` the same way `query_page` already did for the grid, so the badge actually renders there too instead of staying dark. Tests: `tests/test_library_filters.py::test_difficulty_sort_pushes_unrated_to_bottom`, `tests/test_library_filters.py::test_tree_view_songs_carry_user_difficulty`.
- **`lib/midi_import.py`: `convert_midi_tempo_map` — MIDI imports can finally carry
their bars.** The keys/drums note converters always computed a tempo-aware

View File

@ -117,6 +117,8 @@ Notes:
**Frontend scripts** — `screen.js` runs in the global scope via a `<script>` tag. It can access `window.playSong`, `window.showScreen`, `window.createHighway`, the `<audio>` element, and the `window.feedBack` event emitter.
**ES-module plugins (`scriptType:"module"`)** — a plugin may instead ship a native ES-module graph with **no build step**: set `"scriptType": "module"` in `plugin.json`, make `screen.js` a one-line `import './src/main.js'`, and put the module tree under `src/` (served by the sandboxed `/api/plugins/<id>/src/{path}` route). The host injects it as `<script type="module">`, whose `onload` fires only after the whole static-import graph evaluates — so the loader's completion-by-`onload` + `_loadingPluginId` + `playSong` wrapper-chain ordering all hold. Resolve your own asset URLs (worklets, WASM) with `import.meta.url``document.currentScript` is `null` in a module. Module top-level code does **not** re-run when the user re-enters the screen at the same version (the host loads screen.js once and `showScreen` re-injects nothing), so keep per-visit re-init in a `screen:changed` handler, exactly as classic plugins do. Classic global-scope `screen.js` remains fully supported. See `docs/plugin-modules.md`.
**The playSong wrapper chain** — Plugins commonly wrap `window.playSong` to hook into song playback. Plugins load alphabetically, so the last-loaded (alphabetically later) wrapper runs first, while the alphabetically first plugin runs closest to the original. Be aware that `await` calls in inner wrappers yield to the event loop — WebSocket messages can arrive before outer wrappers finish setup.
## Plugin Best Practices

67
docs/perf-baseline.md Normal file
View File

@ -0,0 +1,67 @@
# Perf baseline — module-migration refactor
The refactor promises "measured runtime wins, no hand-waved perf claims" and
"screen-entry and frame-time no worse." This is the baseline to hold it to.
Rerun the harness after every phase (R0 → R3c) and compare.
## Running it
```
# 1. start core against a library with real charts (see caveat below)
CONFIG_DIR=… DLC_DIR=/path/to/songs PYTHONPATH=lib \
python3 -m uvicorn server:app --host 127.0.0.1 --port 8000
# 2. capture (maintainer/CI-only; uses the committed Playwright chromium)
node scripts/perf-baseline.mjs --base http://127.0.0.1:8000 --n 60 --soak 30
```
The script prints a markdown block; paste it under "Results" below with the date
and the commit it was taken at.
## What it measures
- **Server latency** — p50/p95/p99 over N requests for `/api/version`,
`/api/plugins`, `/api/library`, `/api/library/artists`.
- **Cold boot → interactive** — full page load to `networkidle`.
- **JS heap**`performance.memory.usedJSHeapSize` after load and after an idle
soak (a leak signal across a session).
- **Plugin-script shape** — how many plugin `<script>`s the loader injected (a
"the app booted with its plugins" sanity signal).
**Not yet captured — needs a seeded library with charts** (fill in when run
against a real environment): playback **frame-time p95** on the 2D and 3D
highway, and **screen-entry** (plugin inject → interactive) for
editor / notedetect / highway_3d with a chart loaded. These are the
perf-sensitive numbers that gate the `highway.js` split (R3c); the harness has
the hooks, they just need real songs in `DLC_DIR`.
## Results
### R0 baseline — 2026-07-08 (branch `feat/r0-plugin-module-rails`)
> ⚠️ A quick capture (`--n 50 --soak 8`) against an **empty** library (no charts
> in `DLC_DIR`), so the `/api/library*` and boot numbers are floor values —
> re-take on a seeded environment with the recommended `--n 60 --soak 30` for the
> real R0 baseline before comparing R1+ against it. Recorded here to prove the
> harness and lock the methodology.
Server latency (ms), n=50:
| Endpoint | status | p50 | p95 | p99 |
|---|---|---|---|---|
| `/api/version` | 200 | 0.9 | 1.8 | 22.3 |
| `/api/plugins` | 200 | 1.6 | 2.1 | 3.4 |
| `/api/library?limit=60` | 200 | 1.4 | 1.7 | 2.9 |
| `/api/library/artists` | 200 | 1.3 | 1.8 | 2.7 |
Client:
| Metric | Value |
|---|---|
| Cold boot → networkidle | 1268 ms |
| JS heap after load | 10.1 MB |
| JS heap after idle soak | 10.1 MB (no idle growth) |
| Plugin scripts injected | 12 |
No plugin has migrated yet, so all 12 are classic. When the R1 pilot (stems)
lands, cold-boot / heap should not regress.

103
docs/plugin-modules.md Normal file
View File

@ -0,0 +1,103 @@
# Plugin ES-module migration playbook
How to move a plugin off a single global-scope `screen.js` IIFE onto a native
ES-module graph — **no build step, no framework, no bundler**. This is the
mechanism the monolith-killing refactor uses; the host rails for it shipped in
R0 (see `.specify/memory/constitution.md` Principle II + the "Module load
contract" in Operating Constraints).
## The shape
```
my-plugin/
plugin.json + "scriptType": "module" ← opt in
screen.js import './src/main.js'; ← the entire file
src/
state.js (0) module state + accessors
util/… (1) pure helpers — real-import testable
…/… (2..4) model → render/audio/io → input
globals.js (5) THE ONLY file that writes window.*
main.js (5) boot: wire modules, register screen:changed
assets/… worklets / WASM / images (unchanged, served as today)
```
`screen.js` becomes a one-line static `import`. The host injects it as
`<script type="module">`, whose load event fires **only after the whole
static-import graph fetches and evaluates** — so the loader's
completion-by-`onload` + `_loadingPluginId` window + `playSong` wrapper-chain
order are all preserved. (A classic IIFE that fired a fire-and-forget
`import()` would break that contract — don't do that; use `scriptType:"module"`.)
## Non-negotiable rules
1. **Source-served, no build.** Modules are plain source files fetched from
`/api/plugins/<id>/src/<path>`. No bundler, transpiler, or TypeScript.
2. **Layering points downward** — `state → util → commands/model →
render/audio/io → input → globals/main`. A lint check (`import-x/no-cycle`)
enforces acyclicity; extract bottom-up so each move only imports
already-extracted layers.
3. **`globals.js` is the only writer of `window.*`.** The deliberate global
surface shrinks to one auditable file; everything else is module-scoped.
4. **Import-time purity.** `node --test` runs a module's top-level code on
import, so a module you want to unit-test must be side-effect-free at import:
no `document` / `window` / `localStorage` at module top level — lift init
into an exported `init()` called by `main.js`. (Constitution Principle V's
"no implicit IO at import time", applied to the frontend.) Tests are `.mjs`
and use real `import`, retiring the regex/`extractFunction` harness.
5. **Assets resolve via `import.meta.url`.** `document.currentScript` is `null`
inside a module. `assets/` lives at the plugin root, so a `src/` module must
climb out of `src/`: from `src/main.js`, `new URL('../assets/x.js',
import.meta.url)` (deeper modules need more `../`). Simpler and
depth-independent: the absolute route `/api/plugins/<id>/assets/x.js`.
Worklets run in a *separate* module graph (`AudioWorkletGlobalScope`) and
cannot share modules with `src/`.
6. **Re-init comes from `screen:changed`, not re-execution.** The host loads
`screen.js` once per version and `showScreen` re-injects nothing, so module
top-level code does **not** re-run when the user re-enters the screen at the
same version. Keep per-visit setup/teardown in a `window.feedBack.on(
'screen:changed', …)` handler — exactly as classic plugins (tuner,
minigames) already do. Do not rely on the IIFE re-running.
7. **Inline `onclick=` keeps working** during migration via `globals.js` (which
keeps every referenced symbol on `window`); retire inline handlers to
module-side `addEventListener` opportunistically, never as a blocking step.
## The live-edit loop
The host serves `screen.js`, `src/**`, and `assets/**` with
`Cache-Control: no-cache` + a weak `ETag` and honors `If-None-Match``304`.
So: edit a `src/` file → **refresh the browser** → the edited module returns
`200` and reloads while every unchanged module `304`s. There is no hot-reload;
the loop is edit → refresh → see change, exactly as before. The `?v=<version>`
query on `screen.js` is the legacy version buster; it does **not** propagate
into the `src/` graph and does not need to — ETag/mtime is the correctness
authority for the whole graph.
## Host-version floor (`minHost`)
A migrated plugin *requires* a host new enough to serve `src/` and inject
`type=module`. Declare the floor with `"minHost": "X.Y.Z"` in `plugin.json`.
(R0 plumbs the field through `/api/plugins`; enforcement — refuse-with-message
on an older host — is deferred, so bundled plugins are unaffected. Community
plugins should state the floor and not migrate below it.)
## Migration mechanics
- **Move-only PRs.** One slice extracts one module: cut code, add
imports/exports, update `globals.js` — zero behavior change. Behavior fixes
are separate PRs. (Init-lifts for import purity are the one non-pure move —
budget them.)
- **Bottom-up, layer by layer.** Within a layer, independent modules are
independent PRs (a DAG, not a chain); use a git worktree per branch.
- Tests move with their subject and convert to real `.mjs` imports in the same
PR (assertions unchanged).
- Size norm: no source file over **1,500 lines**; legitimate exceptions
(hot renderers, etc.) go in the signed register at `docs/size-exemptions.md`.
## Verifying a migration
`node --test <plugin>/tests/*.mjs`; load the plugin on the `:8000` testbed and
confirm it boots (`<script type=module>` in DevTools, the `src/` graph in
Network); edit a `src/` file → refresh → change visible (`200` on the edited
file, `304` on the rest); leave and re-enter the screen at the same version →
it re-inits via `screen:changed`. The R1 pilots (stems, then studio) certify
this end-to-end before the flagship repos migrate.

62
docs/size-exemptions.md Normal file
View File

@ -0,0 +1,62 @@
# Size-exemption register
The working norm (constitution Principle II; enforced by the `max-lines` lint
gate) is **no source file over 1,500 lines**. A few files are allowed to exceed
it because splitting them would do more harm than good — hot per-frame
renderers, C++, offline generators, cohesive registries. This register is the
list of those exceptions: each row is a **deliberate, signed** decision with a
ceiling, a rationale, and a review trigger. Without it, "no file over 1,500
without a *signed* exemption" is unenforceable.
**Rules**
- One row per file: a ceiling, a rationale, a signer, a review trigger.
- The `max-lines` per-file ceilings in `eslint.config.js` mirror this table —
keep them in sync (this register is canonical).
- Files with a scheduled split **plan** are *not* exempt — they live in
"Planned, not exempt" at the bottom so nothing falls between the two states.
- **Signers** (decided 2026-07-08): **Byron** signs core + bundled rows;
**Christian** signs the authored-plugin row (virtuoso, its own repo/track).
## Permanent exemptions (structural rationale)
| Repo / file | Lines (7-07) | Ceiling | Rationale | Signer | Review |
|---|---|---|---|---|---|
| core `static/highway.js` → residual `renderer-2d.js` (post-split) | ~2,4002,900 est. | **3,000** | 60 fps hot path; no module boundary inside the per-frame loop | Byron | after the highway.js split |
| core `plugins/highway_3d/` → residual renderer | sized at split; likely **>3,000** | set at split, flagged now | same hot-path rule; the draw core can't be cut without behavior risk | Byron | after the highway_3d split |
| core `static/capabilities.js` | 1,538 | 1,600 | cohesive registry + `window.feedBack` bus, 38 lines over; a split spends credibility for nothing | Byron | R4 |
| tutorials `builtin/reading-the-highway/generate.py` | 1,818 | 2,000 | offline content generator, never imported at runtime, deps not in runtime requirements | Byron | if a 3rd builtin pack appears |
| desktop `src/audio/NodeAddon.cpp` | 3,542 | as-is | C++, outside the ESM/routes playbooks; under active use-after-free crash work — do not churn | Byron | after crash-class work settles |
| desktop `src/audio/AudioEngine.cpp` | 2,977 | as-is | same | Byron | same |
| desktop `src/vst-host/main.cpp` | 1,928 | as-is | same | Byron | same |
| virtuoso `screen.js` (authored, own track) | 25,741 | as-is until its own split | authored plugin on a separate roadmap; migrates on its own schedule | Christian | virtuoso split kickoff |
## Split-when-touched (no scheduled train; row retires when split)
| Repo / file | Lines | Ceiling | Rationale | Signer | Review |
|---|---|---|---|---|---|
| core `lib/gp2rs_gpx.py` | 2,540 | as-is | import converter, off the serve-path hot loop | Byron | when next touched |
| core `lib/gp2rs.py` | 2,055 | as-is | same | Byron | when next touched |
| core `lib/song.py` | 1,689 | as-is | data models + wire format; cohesive | Byron | when next touched |
| core `lib/gp_autosync.py` | 1,572 | as-is | under active dev (#787/#791) — don't collide | Byron | after in-flight work lands |
| core `plugins/capability_inspector/screen.js` | 1,752 | as-is | bundled diagnostics plugin, low churn | Byron | when next touched |
| core `plugins/folder_library/screen.js` | 1,672 | as-is | bundled plugin, low churn | Byron | when next touched |
## Temporary rows (cleared by a scheduled PR)
| Repo / file | Lines | Cleared by |
|---|---|---|
| core `plugins/__init__.py` | ~2,470 (grew under R0) | the `plugins/_routes.py` + `plugins/_registry.py` split (rides the server.py router work) |
## Watch list (under the norm — no row needed, re-census each phase)
`musicxml-import/mxml2notation.py` (1,456) · core `static/capabilities/audio-effects.js`
(1,436) · `studio routes.py` (1,399) · `update-manager screen.js` (1,492 — zero headroom).
## Planned, NOT exempt (owned by split plans — listed so nothing falls between states)
core `static/app.js` (11,821) · `static/highway.js` (4,154, whole file) · `server.py`
(13,948) · `static/v3/songs.js` (4,134) · `static/capabilities/audio-session.js`
(2,974) · `plugins/highway_3d/screen.js` (15,656) · `plugins/keys_highway_3d/screen.js`
(3,780) · `plugins/drum_highway_3d/screen.js` (3,597) — and every monolith with a PR
train in the refactor plan. Test files (e.g. `tests/test_plugins.py`) are out of scope
by policy — the norm governs source files.

66
eslint.config.js Normal file
View File

@ -0,0 +1,66 @@
// Flat ESLint config — MAINTAINER / CI ONLY. Never runs on the serve or Docker
// path (constitution Principle I: dev-only tooling is exempt, same category as
// scripts/build-tailwind.sh). It enforces the module-migration guardrails:
//
// * max-lines — the 1,500-line size norm, as a WARNING ratchet. Legacy
// monoliths warn (the "this is over the norm, split it" signal) and shrink
// as the refactor lands; warnings do not fail CI. Genuinely-large files are
// exempted below, mirroring the signed register in docs/size-exemptions.md.
// * import-x/no-unresolved + no-cycle — module hygiene, scoped to the real
// ES-module graphs the refactor produces (a plugin's src/ tree, .mjs
// tests). no-unresolved (a HARD error) catches broken import paths;
// no-cycle enforces the downward-only layering rule. Core's classic scripts
// have no import graph, so both are dormant today and become live gates the
// moment module code appears — validated against the first real module
// plugin (R1 pilot).
const importX = require('eslint-plugin-import-x');
// Per-file size ceilings — a mirror of docs/size-exemptions.md (canonical).
// Keep in sync; each entry corresponds to a signed row in the register.
const SIZE_EXEMPTIONS = [
{ files: ['**/static/capabilities.js'], max: 1600 },
{ files: ['**/plugins/capability_inspector/screen.js'], max: 100000 },
{ files: ['**/plugins/folder_library/screen.js'], max: 100000 },
];
const sizeRule = (max) => ['warn', { max, skipBlankLines: false, skipComments: false }];
module.exports = [
{
ignores: [
'node_modules/**',
'static/vendor/**',
'plugins/**/assets/vendor/**',
'**/*.min.js',
'static/tailwind.min.css',
],
},
// Size norm across all first-party JS. Classic scripts are parsed as
// scripts (no import/export); module files get their own block below.
{
files: ['**/*.js', '**/*.cjs'],
languageOptions: { ecmaVersion: 'latest', sourceType: 'script' },
rules: { 'max-lines': sizeRule(1500) },
},
// ES-module graphs (a plugin's src/ tree, .mjs tests): module parsing + the
// acyclic-imports hard gate + the size norm. A migrated bundled plugin's
// entry `import './src/main.js'` screen.js must parse as a module — add its
// glob here in that plugin's migration PR (classic screen.js stays a script).
{
files: ['**/src/**/*.js', '**/*.mjs'],
languageOptions: { ecmaVersion: 'latest', sourceType: 'module' },
plugins: { 'import-x': importX },
// v4 flat-config resolver (resolver-next + createNodeResolver). Without
// it the import rules silently skip imports they can't resolve.
settings: { 'import-x/resolver-next': [importX.createNodeResolver()] },
rules: {
'max-lines': sizeRule(1500),
'import-x/no-unresolved': 'error',
'import-x/no-cycle': 'error',
},
},
// Signed size exemptions (docs/size-exemptions.md) — raise the ceiling so
// registered files don't warn below it.
...SIZE_EXEMPTIONS.map(({ files, max }) => ({ files, rules: { 'max-lines': sizeRule(max) } })),
];

1759
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -8,9 +8,12 @@
"test:headed": "playwright test --headed",
"test:debug": "playwright test --debug",
"test:js": "node --test tests/js/*.test.js 'tests/plugins/*/js/*.test.js'",
"install:playwright": "playwright install chromium"
"install:playwright": "playwright install chromium",
"lint": "eslint ."
},
"devDependencies": {
"@playwright/test": "^1.59.1"
"@playwright/test": "^1.59.1",
"eslint": "^9.39.4",
"eslint-plugin-import-x": "^4.17.1"
}
}

View File

@ -18,6 +18,54 @@ from safepath import safe_join
log = logging.getLogger("feedBack.plugins")
def _plugin_media_type(path: Path) -> str:
"""Best-effort Content-Type for a served plugin file. `.js`/`.css` must come
back as JavaScript/CSS so `<script type=module>` / `addModule()` / a `<link>`
accept them; `mimetypes.guess_type` can miss these on a stripped platform
registry, so fall back explicitly (mirrors the assets/ route)."""
media_type = mimetypes.guess_type(path.name)[0]
if media_type is None and path.suffix == ".js":
return "application/javascript"
if media_type is None and path.suffix == ".css":
return "text/css"
return media_type or "application/octet-stream"
def _plugin_file_etag(path: Path) -> str | None:
"""Weak ETag from mtime+size — cheap, stable across reads, changes on edit.
This is what makes the live-edit loop work for module graphs: a conditional
GET revalidates and 304s unchanged files on refresh instead of re-downloading
the whole `src/` tree. Returns None if the file can't be stat'd."""
try:
st = path.stat()
except OSError:
return None
return f'W/"{st.st_mtime_ns:x}-{st.st_size:x}"'
def _if_none_match(request: Request, etag: str) -> bool:
"""True when the client's If-None-Match already holds `etag`."""
# ponytail: we serve one weak ETag; the browser echoes it back verbatim, so
# a direct compare is enough (comma-split tolerates a proxy concatenation).
return etag in [t.strip() for t in request.headers.get("if-none-match", "").split(",")]
def _plugin_file_response(request: Request, path: Path, media_type: str) -> Response:
"""Serve a plugin source/asset file with the live-edit cache contract:
`Cache-Control: no-cache` (browser may store but MUST revalidate) + a weak
ETag, and a bodyless 304 when the client's If-None-Match already matches.
Starlette's `FileResponse` emits an ETag but never evaluates If-None-Match
itself, so the conditional handling has to live here."""
headers = {"Cache-Control": "no-cache"}
etag = _plugin_file_etag(path)
if etag:
headers["ETag"] = etag
if _if_none_match(request, etag):
return Response(status_code=304, headers=headers)
# FileResponse sets etag/last-modified via setdefault, so the ETag above wins.
return FileResponse(path, media_type=media_type, headers=headers)
PLUGINS_DIR = Path(__file__).parent
# Holds only *ready* (loaded) plugins — those whose dependencies installed
# and whose routes registered. A plugin GRADUATES from PENDING_PLUGINS into
@ -1373,6 +1421,12 @@ def load_plugins(app: FastAPI, context: dict, progress_cb=None, route_setup_fn=N
"version": manifest.get("version"),
"has_screen": bool(manifest.get("screen")),
"has_script": bool(manifest.get("script")),
# Module-migration (R0): `scriptType:"module"` tells the loader to
# inject screen.js as <script type="module">; `minHost` is the
# min core version a migrated plugin needs (passthrough only in R0 —
# enforcement is deferred to R4, master §4b). None when unset.
"script_type": manifest.get("scriptType"),
"min_host": manifest.get("minHost"),
"has_settings": bool(manifest.get("settings")),
"settings_category": _settings_category,
# Drives the v3 shell's immersive (full-screen) mode for this
@ -2089,6 +2143,11 @@ def register_plugin_api(app: FastAPI):
"fallback": p.get("fallback", False),
"has_screen": p["has_screen"],
"has_script": p["has_script"],
# Module-migration passthrough (R0). Re-read from the manifest
# like `version` above so stubbed test entries (built without
# _nav_entry) don't need the key.
"script_type": (p.get("_manifest") or {}).get("scriptType"),
"min_host": (p.get("_manifest") or {}).get("minHost"),
"has_settings": p["has_settings"],
# v3 immersive screen opt-in (full-screen plugin UI).
"fullscreen": p.get("fullscreen", False),
@ -2142,6 +2201,9 @@ def register_plugin_api(app: FastAPI):
"fallback": False,
"has_screen": e.get("has_screen", False),
"has_script": e.get("has_script", False),
# Pending entries come from _nav_entry, so they carry these.
"script_type": e.get("script_type"),
"min_host": e.get("min_host"),
"has_settings": e.get("has_settings", False),
"settings_category": e.get("settings_category"),
"fullscreen": e.get("fullscreen", False),
@ -2307,7 +2369,7 @@ def register_plugin_api(app: FastAPI):
return HTMLResponse("", status_code=404)
@app.get("/api/plugins/{plugin_id}/screen.js")
def plugin_screen_js(plugin_id: str):
def plugin_screen_js(request: Request, plugin_id: str):
with PLUGINS_LOCK:
snapshot = list(LOADED_PLUGINS)
for p in snapshot:
@ -2315,8 +2377,11 @@ def register_plugin_api(app: FastAPI):
if p.get("status", "ready") != "ready":
break
script_file = p["_dir"] / p["_manifest"].get("script", "screen.js")
if script_file.exists():
return Response(script_file.read_text(encoding="utf-8"), media_type="application/javascript")
if script_file.is_file():
# no-cache + ETag/304 so an edited screen.js reloads on
# refresh while an unchanged one revalidates cheaply — the
# same live-edit contract the src/ module graph relies on.
return _plugin_file_response(request, script_file, "application/javascript")
return Response("", status_code=404)
@app.get("/api/plugins/{plugin_id}/settings.html")
@ -2377,7 +2442,7 @@ def register_plugin_api(app: FastAPI):
return Response("{}", status_code=404, media_type="application/json")
@app.get("/api/plugins/{plugin_id}/assets/{asset_path:path}")
def plugin_asset(plugin_id: str, asset_path: str):
def plugin_asset(request: Request, plugin_id: str, asset_path: str):
"""Serve a static file a plugin bundles under its own ``assets/``
directory (e.g. an AudioWorklet module, WASM, or image). Unlike the
fixed screen.js/settings.html handlers above, this is a generic
@ -2399,16 +2464,35 @@ def register_plugin_api(app: FastAPI):
log.warning("Plugin %r: asset path rejected: %r", plugin_id, asset_path)
break
if target.is_file():
media_type = mimetypes.guess_type(target.name)[0]
# .js must come back as JavaScript so addModule() / <script>
# accept it; guess_type can miss this on some platforms.
if media_type is None and target.suffix == ".js":
media_type = "application/javascript"
# .css must come back as text/css so a <link rel=stylesheet>
# (the styles capability) is honoured; guess_type can miss it
# on a stripped platform mimetypes registry, same as .js.
elif media_type is None and target.suffix == ".css":
media_type = "text/css"
return FileResponse(target, media_type=media_type or "application/octet-stream")
# no-cache + ETag/304 so a live-edited worklet/asset reloads
# on refresh (bare FileResponse emits an ETag but never 304s).
return _plugin_file_response(request, target, _plugin_media_type(target))
break
return Response("", status_code=404)
@app.get("/api/plugins/{plugin_id}/src/{src_path:path}")
def plugin_src(request: Request, plugin_id: str, src_path: str):
"""Serve a file from a plugin's ES-module source tree under ``src/``.
This is the R0 host capability that lets a migrated plugin's
``screen.js`` (a one-line ``import './src/main.js'``) load its whole
module graph. Containment mirrors the assets/ route exactly
``safe_join`` against ``<plugin>/src`` rejects ``..``, absolute paths,
and NUL bytes and the live-edit cache contract (no-cache + ETag/304)
makes an edited module reload on refresh while unchanged ones 304.
Read-only; the src/ tree is source files, never executed server-side.
"""
with PLUGINS_LOCK:
snapshot = list(LOADED_PLUGINS)
for p in snapshot:
if p["id"] == plugin_id:
if p.get("status", "ready") != "ready":
break
target = safe_join(p["_dir"] / "src", src_path)
if target is None:
log.warning("Plugin %r: src path rejected: %r", plugin_id, src_path)
break
if target.is_file():
return _plugin_file_response(request, target, _plugin_media_type(target))
break
return Response("", status_code=404)

94
scripts/perf-baseline.mjs Normal file
View File

@ -0,0 +1,94 @@
// Perf-baseline harness for the module-migration refactor (R0).
//
// Rerun this after every phase (R0 → R3c) to prove the split does not regress
// screen-entry, frame-time, memory, or server latency. It writes a markdown
// results block to stdout; paste it into docs/perf-baseline.md (or redirect).
//
// Usage:
// node scripts/perf-baseline.mjs --base http://127.0.0.1:8000 [--n 60] [--soak 30]
//
// Maintainer/CI-only dev tooling (uses the committed @playwright/test browser);
// never part of the serve or Docker path. Metrics that need a seeded library
// with charts (playback frame-time, screen-entry into a live highway) are
// clearly labelled — run those against an environment with real songs.
import { createRequire } from 'module';
const require = createRequire(import.meta.url);
const { chromium } = require('@playwright/test');
const args = new Map();
for (let i = 2; i < process.argv.length; i += 2) args.set(process.argv[i].replace(/^--/, ''), process.argv[i + 1]);
const BASE = args.get('base') || 'http://127.0.0.1:8000';
const N = parseInt(args.get('n') || '60', 10);
const SOAK_S = parseInt(args.get('soak') || '30', 10);
const pct = (xs, p) => {
if (!xs.length) return null;
const s = [...xs].sort((a, b) => a - b);
return s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))];
};
const ms = (x) => (x == null ? '—' : `${x.toFixed(1)}`);
// ── Server latency: p50/p95/p99 over N requests per endpoint ──────────────────
async function serverLatency(paths) {
const rows = [];
for (const path of paths) {
const t = [];
let status = 0;
for (let i = 0; i < N; i++) {
const t0 = performance.now();
try {
const r = await fetch(BASE + path);
status = r.status;
await r.arrayBuffer();
} catch { status = -1; }
t.push(performance.now() - t0);
}
rows.push({ path, status, p50: pct(t, 50), p95: pct(t, 95), p99: pct(t, 99) });
}
return rows;
}
// ── Client: cold boot-to-interactive + idle memory after a soak ───────────────
async function clientMetrics() {
const browser = await chromium.launch();
const page = await browser.newPage();
const t0 = Date.now();
await page.goto(BASE, { waitUntil: 'networkidle', timeout: 60000 });
const bootMs = Date.now() - t0;
// performance.memory is Chromium-only; JS heap after settle.
const mem0 = await page.evaluate(() => (performance.memory ? performance.memory.usedJSHeapSize : null));
await page.waitForTimeout(SOAK_S * 1000);
const mem1 = await page.evaluate(() => (performance.memory ? performance.memory.usedJSHeapSize : null));
const scripts = await page.evaluate(() =>
document.querySelectorAll('script[data-plugin-id]').length);
await browser.close();
return { bootMs, memStartMB: mem0 && mem0 / 1048576, memSoakMB: mem1 && mem1 / 1048576, scripts };
}
const server = await serverLatency([
'/api/version',
'/api/plugins',
'/api/library?limit=60',
'/api/library/artists',
]);
const client = await clientMetrics();
const now = new Date().toISOString();
let out = `\n<!-- generated by scripts/perf-baseline.mjs @ ${now} against ${BASE} (n=${N}, soak=${SOAK_S}s) -->\n\n`;
out += `### Server latency (ms)\n\n| Endpoint | status | p50 | p95 | p99 |\n|---|---|---|---|---|\n`;
for (const r of server) out += `| \`${r.path}\` | ${r.status} | ${ms(r.p50)} | ${ms(r.p95)} | ${ms(r.p99)} |\n`;
out += `\n### Client\n\n| Metric | Value |\n|---|---|\n`;
out += `| Cold boot → networkidle | ${client.bootMs} ms |\n`;
out += `| JS heap after load | ${client.memStartMB ? client.memStartMB.toFixed(1) + ' MB' : '—'} |\n`;
out += `| JS heap after ${SOAK_S}s idle soak | ${client.memSoakMB ? client.memSoakMB.toFixed(1) + ' MB' : '—'} |\n`;
out += `| Plugin scripts injected | ${client.scripts} |\n`;
out += `\n> **Requires a seeded library** (not captured by this run): playback frame-time p95\n`;
out += `> on the 2D + 3D highway, and screen-entry (plugin inject → interactive) for\n`;
out += `> editor / notedetect / highway_3d with a real chart loaded. Run this harness\n`;
out += `> against an environment with charts in \`DLC_DIR\` to fill those in.\n`;
console.log(out);

View File

@ -11567,6 +11567,14 @@ async function loadPlugins() {
// URL ?v=mtime convention elsewhere in this file).
const v = encodeURIComponent(wantedVersion);
script.src = `/api/plugins/${plugin.id}/screen.js${v ? `?v=${v}` : ''}`;
// Module-migration (R0): a migrated plugin declares
// scriptType:"module" and its screen.js is `import
// './src/main.js'`. A <script type="module"> fires load
// only after its whole static-import graph evaluates, so
// the await-onload completion + _loadingPluginId contract
// below is preserved (a classic-IIFE dynamic import()
// would not). Classic plugins are unaffected.
if (plugin.script_type === 'module') script.type = 'module';
script.dataset.pluginId = plugin.id;
script.dataset.pluginVersion = wantedVersion;
window.feedBack._loadingPluginId = plugin.id;

View File

@ -0,0 +1,59 @@
// Guards the R0 module-migration loader change in static/app.js: a migrated
// plugin (manifest scriptType:"module", surfaced as plugin.script_type) must be
// injected as <script type="module"> so its screen.js `import './src/main.js'`
// graph loads, while classic plugins stay untouched.
//
// The injection is a single line inside the large async loadPlugins() closure
// (it depends on loadedScripts, _removePluginScriptTags, and the
// _loadingPluginId completion window), so a faithful behavioural harness would
// need to stub the whole loader. Instead this asserts the *structural*
// contract in source — the guard exists, is gated (not unconditional), and sits
// inside the screen.js injection block before appendChild. The behavioural proof
// is the R0 end-to-end live-edit check (a real module plugin booting in-browser).
const { test } = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const APP_JS = path.join(__dirname, '..', '..', 'static', 'app.js');
const src = fs.readFileSync(APP_JS, 'utf8');
// Isolate the screen.js <script> injection block: from where its src is built
// to where the element is appended.
function injectionBlock() {
const start = src.indexOf('/api/plugins/${plugin.id}/screen.js');
assert.ok(start !== -1, 'screen.js injection src not found — loader moved?');
const end = src.indexOf('document.body.appendChild(script)', start);
assert.ok(end !== -1, 'appendChild(script) not found after screen.js src');
return src.slice(start, end);
}
test('module plugins are injected as <script type="module">', () => {
const block = injectionBlock();
assert.match(
block,
/if\s*\(\s*plugin\.script_type\s*===\s*['"]module['"]\s*\)\s*script\.type\s*=\s*['"]module['"]\s*;/,
'expected a guarded `script.type = "module"` keyed on plugin.script_type === "module"',
);
});
test('the module type is gated, never set unconditionally', () => {
const block = injectionBlock();
// Every assignment of script.type in the block must be on the same line as
// the plugin.script_type guard (i.e. no bare `script.type = 'module'`).
for (const line of block.split('\n')) {
if (/script\.type\s*=/.test(line)) {
assert.match(line, /plugin\.script_type\s*===\s*['"]module['"]/,
`unguarded script.type assignment: ${line.trim()}`);
}
}
});
test('the module guard sits before appendChild, after the src assignment', () => {
const guardAt = src.indexOf('script.type = \'module\'');
const srcAt = src.indexOf('/api/plugins/${plugin.id}/screen.js');
const appendAt = src.indexOf('document.body.appendChild(script)', srcAt);
assert.ok(guardAt > srcAt && guardAt < appendAt,
'the module guard must live inside the screen.js injection block');
});

View File

@ -0,0 +1,140 @@
"""Tests for the plugin `src/` module-serving route and the live-edit cache
contract added in R0 (module-migration rails).
Covers:
* GET /api/plugins/{id}/src/{path} serves a plugin's ES-module source tree
with the right Content-Type, including nested paths.
* Path containment: `..`, absolute, and NUL are rejected (404) the same
`safe_join` guard the assets/ route uses.
* The live-edit cache contract: no-cache + a weak ETag, a bodyless 304 on
matching If-None-Match, and no stale 304 after an in-place edit.
* screen.js and assets/ now also emit an ETag and honor If-None-Match
(previously screen.js sent no headers and assets/ never returned 304).
The routes read the module-global `plugins.LOADED_PLUGINS`, so each test
registers a fake ready plugin directly (save/restore that global) and drives
`register_plugin_api` on a fresh FastAPI app no full server import needed.
"""
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
import plugins
PLUGIN_ID = "srctest"
@pytest.fixture()
def client(tmp_path):
"""A TestClient with `register_plugin_api` wired and a single fake ready
plugin whose dir (`tmp_path`) holds a src/ tree, an asset, and a screen.js.
Restores LOADED_PLUGINS afterward."""
(tmp_path / "src").mkdir()
(tmp_path / "src" / "main.js").write_text("import './util/x.js';\nexport const boot = 1;\n")
(tmp_path / "src" / "util").mkdir()
(tmp_path / "src" / "util" / "x.js").write_text("export const x = 42;\n")
(tmp_path / "src" / "theme.css").write_text(".a{color:red}\n")
(tmp_path / "assets").mkdir()
(tmp_path / "assets" / "worklet.js").write_text("// worklet\n")
(tmp_path / "screen.js").write_text("import './src/main.js';\n")
saved = list(plugins.LOADED_PLUGINS)
plugins.LOADED_PLUGINS.clear()
plugins.LOADED_PLUGINS.append({
"id": PLUGIN_ID,
"status": "ready",
"_dir": tmp_path,
"_manifest": {"script": "screen.js", "scriptType": "module"},
})
app = FastAPI()
plugins.register_plugin_api(app)
c = TestClient(app, raise_server_exceptions=True)
try:
yield c, tmp_path
finally:
c.close()
plugins.LOADED_PLUGINS.clear()
plugins.LOADED_PLUGINS.extend(saved)
def test_src_file_served_with_js_media_type(client):
c, _ = client
r = c.get(f"/api/plugins/{PLUGIN_ID}/src/main.js")
assert r.status_code == 200
# Either application/javascript or text/javascript is a valid module-script
# MIME (guess_type returns text/javascript on newer platforms); browsers
# accept both for <script type=module>.
assert "javascript" in r.headers["content-type"]
assert "export const boot" in r.text
assert r.headers["cache-control"] == "no-cache"
assert r.headers.get("etag")
def test_src_nested_path_and_css_media_type(client):
c, _ = client
assert c.get(f"/api/plugins/{PLUGIN_ID}/src/util/x.js").status_code == 200
r = c.get(f"/api/plugins/{PLUGIN_ID}/src/theme.css")
assert r.status_code == 200
assert r.headers["content-type"].startswith("text/css")
@pytest.mark.parametrize("bad", [
"..%2f..%2fplugin.json", # escape the src/ dir
"..%2f..%2f..%2fetc%2fpasswd",
"%2fetc%2fpasswd", # absolute
"util%2f..%2f..%2fscreen.js",
])
def test_src_traversal_rejected(client, bad):
c, _ = client
assert c.get(f"/api/plugins/{PLUGIN_ID}/src/{bad}").status_code == 404
def test_src_missing_is_404(client):
c, _ = client
assert c.get(f"/api/plugins/{PLUGIN_ID}/src/nope.js").status_code == 404
def test_src_conditional_304(client):
c, _ = client
r1 = c.get(f"/api/plugins/{PLUGIN_ID}/src/main.js")
etag = r1.headers["etag"]
r2 = c.get(f"/api/plugins/{PLUGIN_ID}/src/main.js", headers={"If-None-Match": etag})
assert r2.status_code == 304
assert r2.content == b""
def test_src_no_stale_304_after_edit(client):
c, root = client
etag = c.get(f"/api/plugins/{PLUGIN_ID}/src/main.js").headers["etag"]
(root / "src" / "main.js").write_text("export const boot = 2; // edited, longer body\n")
r = c.get(f"/api/plugins/{PLUGIN_ID}/src/main.js", headers={"If-None-Match": etag})
assert r.status_code == 200
assert "boot = 2" in r.text
assert r.headers["etag"] != etag
def test_screen_js_now_conditional(client):
c, _ = client
r1 = c.get(f"/api/plugins/{PLUGIN_ID}/screen.js")
assert r1.status_code == 200
assert r1.headers["cache-control"] == "no-cache"
etag = r1.headers["etag"]
r2 = c.get(f"/api/plugins/{PLUGIN_ID}/screen.js", headers={"If-None-Match": etag})
assert r2.status_code == 304
def test_asset_now_conditional(client):
c, _ = client
r1 = c.get(f"/api/plugins/{PLUGIN_ID}/assets/worklet.js")
assert r1.status_code == 200
etag = r1.headers["etag"]
r2 = c.get(f"/api/plugins/{PLUGIN_ID}/assets/worklet.js", headers={"If-None-Match": etag})
assert r2.status_code == 304
def test_unready_plugin_src_is_404(client):
c, _ = client
plugins.LOADED_PLUGINS[0]["status"] = "installing"
assert c.get(f"/api/plugins/{PLUGIN_ID}/src/main.js").status_code == 404