got-feedBack/feedBack
1
0
Fork 0
mirror of https://github.com/got-feedBack/feedBack.git synced 2026-08-11 03:09:57 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Clean release snapshot

Browse Source
This commit is contained in:
byrongamatos
2026-06-16 18:47:13 +02:00
commit 6c110398b4
574 changed files with 162566 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/3dhighway-open-string-double-sustain.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

docs/NOTE_FAILURE_PLAN.md
+243
View File
@@ -0,0 +1,243 @@
# Note Failure Feedback — Implementation Plan
Depends on: `docs/NOTE_FAILURE_SPEC.md` (read that first)
---
## Phase 0: Detection Plugin Foundation
**Goal:** Working note detection plugin streaming detected notes via WebSocket.
This phase was previously tracked in a separate NOTE_DETECTION_PLUGIN_PLAN
document (in the `slopsmith-plugin-notedetect` repository). The relevant scope
is summarized here to avoid relying on an internal git-only reference:
- [ ] Plugin skeleton: `slopsmith-plugin-notedetect/` with plugin.json, routes.py, screen.js
- [ ] Port TonalRecall YIN detection (aubio + sounddevice) to routes.py
- [ ] WebSocket at `/api/plugins/note_detect/stream` streaming `{ note, freq, confidence, time }`
- [ ] Device selection UI in screen.html
- [ ] requirements.txt: aubio, sounddevice, numpy
**Exit criterion:** With plugin active and guitar plugged in, playing a note
causes a JSON event to appear in the browser console.
---
## Phase 1: Note Matching Core
**Goal:** Client-side matching of detected notes to chart notes. No rendering yet —
console logging only.
**Files:**
- `screen.js` in the notedetect plugin
**Tasks:**
- [ ] Implement `expectedFreq(string, fret, tuningOffsets, capo, stringCount, arrangementName)` using
base open-string frequencies, semitone offsets from `highway.getSongInfo().tuning`,
and semitone math (`2^(semitones/12)`) rather than assuming 6-string standard
tuning; use `highway.getStringCount()` as the authoritative string count because
tuning may be padded to length 6 for RS XML sources even for bass/extended-range
arrangements; include `highway.getSongInfo().capo` as additional semitones if
the intent is expected sounding pitch; pass `highway.getSongInfo().arrangement` as
`arrangementName` to disambiguate 5-string bass vs 5-string guitar (matching the
spec's `getBaseTuning` helper)
- [ ] Implement `NoteJudgmentTracker` class with:
- `addDetection(detected)` — correlate with nearest unmatched chart note
- `update(currentTime)` — expire pending notes whose match window has passed
- `getJudgmentsInRange(tStart, tEnd)` — return judgments in time range
- `reset()` — clear all state
- [ ] Connect to detection WebSocket, feed events into tracker
- [ ] Initialize tracker with `highway.getNotes()` and `highway.getChords()` on song ready
- [ ] Console.log each judgment as it resolves (HIT/MISSED/EARLY/LATE/SHARP/FLAT)
- [ ] Re-initialize tracker on `song:ready` (fires on every new song **and** on every
arrangement switch — no need to hook `highway.reconnect` or other internals);
do **not** use `song:loaded` — note/chord arrays are still empty at that point (data
arrives incrementally and only completes at `song:ready`)
**Exit criterion:** Playing along with a song, console shows correct HIT/MISSED
judgments with timing and pitch error values.
**Estimated scope:** ~200 lines JS
---
## Phase 2: Hit/Miss Highway Overlay
**Goal:** Visual feedback on the highway — green glow for hits, red X for misses.
**Files:**
- `screen.js` in the notedetect plugin (draw hook)
**Tasks:**
- [ ] Register `highway.addDrawHook()` that reads judgments from the tracker
- [ ] **Hit rendering:** Green glow ring behind notes at the now-line, fading over
`hitGlowDuration` seconds. Use `highway.project()` and `highway.fretX()` for
positioning. Additive blend via `ctx.globalCompositeOperation = 'lighter'`.
- [ ] **Miss rendering:** Red `✕` marker at the note's string/fret position, drawn
in the "past" region below the now-line. Do **not** rely on
`highway.project(negative_offset)` for long-lived placement — the current
renderer returns `null` for offsets more than ~50ms into the past. Instead,
anchor at the now-line (`highway.project(0)`) and map elapsed time since the
miss to a linear below-now-line Y position (configurable pixels/second), fading
the marker after `missMarkerDuration` seconds.
- [ ] **String pulse:** Brief red tint on the missed note's string (200ms fade on
the string line segment near the now-line).
- [ ] Handle lefty mode: use `highway.fillTextUnmirrored()` for text markers.
- [ ] Cleanup: `highway.removeDrawHook()` on plugin destroy.
**Exit criterion:** Playing a song, you see green flashes on hit notes and red X
markers scrolling past on missed notes.
**Estimated scope:** ~150 lines JS
---
## Phase 3: Diagnostic Labels (Timing + Pitch)
**Goal:** Show *why* a note was missed — too early, too late, sharp, flat.
**Files:**
- `screen.js` in the notedetect plugin
**Tasks:**
- [ ] Extend draw hook to render timing indicators:
- EARLY: orange `↑` + "-XXms" label above the miss marker
- LATE: orange `↓` + "+XXms" label below the miss marker
- Only shown when timing error exceeds `timingThresholdMs`
- [ ] Extend draw hook to render pitch indicators:
- SHARP: blue `♯` + "+XX¢" label
- FLAT: blue `♭` + "-XX¢" label
- Only shown when pitch error exceeds `pitchThresholdCents`
- [ ] Compound states: stack timing label on top, pitch label below
- [ ] Add settings UI in plugin settings panel for threshold configuration
- [ ] Ensure labels don't overlap — offset vertically when multiple notes
miss at close timestamps
**Exit criterion:** Playing intentionally early/late or bending sharp/flat
shows the correct diagnostic labels.
**Estimated scope:** ~100 lines JS, ~30 lines settings HTML
---
## Phase 4: Loop Iteration Tracking
**Goal:** Track performance across loop iterations, show summary on each wrap.
**Files:**
- `screen.js` in the notedetect plugin
**Tasks:**
- [ ] Detect loop wrap: `currentTime < previousTime - 0.5` in the frame update
- [ ] On wrap: snapshot `{ hits, misses, total, percentage }` to `loopHistory[]`
- [ ] Reset judgments for notes in `[loopA, loopB]` range (keep tracker alive
for notes outside the loop)
- [ ] Render loop summary overlay (top-center, semi-transparent background):
```
Loop N | X/Y notes (Z%) | Best: W%
```
Displayed for 1.5s, then fades.
- [ ] Track `bestIteration` across all iterations for "Best" display
- [ ] Emit `loop:complete` event via `window.slopsmith.emit()` so other plugins
(practice journal) can record the data
- [ ] Reset loop history when loop boundaries change or loop is cleared
**Exit criterion:** Looping a 4-bar phrase, you see iteration count and accuracy
flash briefly at each loop wrap. Best score persists across iterations.
**Estimated scope:** ~120 lines JS
---
## Phase 5: Section Grading
**Goal:** Grade each song section (intro, verse, chorus, solo) and surface weak spots.
**Files:**
- `screen.js` in the notedetect plugin
**Tasks:**
- [ ] Use `highway.getSections()` to identify section boundaries
- [ ] Track hits/misses per section as notes are judged
- [ ] At section boundaries (when `currentTime` crosses a section end),
briefly flash the section grade:
- A: 90%+, B: 75%+, C: 60%+, D: 40%+, F: below 40%
- Color: green (A/B), yellow (C), red (D/F)
- [ ] After song completes (or at any point via a hotkey), show a section
summary panel listing all sections with grades
- [ ] Highlight lowest-scoring section with a "Loop this section" button
that sets A-B points to that section's boundaries
- [ ] Emit `note:sectionGrade` event for other plugins
**Exit criterion:** Playing through a song, section grades flash at each
transition. Lowest section is highlighted for targeted practice.
**Estimated scope:** ~150 lines JS, ~40 lines HTML
---
## Phase 6: Polish + Settings
**Goal:** Configurable thresholds, visual polish, performance.
**Tasks:**
- [ ] Full settings panel in plugin settings HTML:
- Match window slider (100-500ms)
- Pitch tolerance slider (20-100 cents)
- Toggle timing/pitch labels
- Toggle loop summary
- Miss marker duration slider
- [ ] Performance: ensure draw hook stays under 1ms per frame
- Pre-compute judgment positions, don't recalculate in draw loop
- Binary search over judgments by time (same pattern as `drawNotes`)
- [ ] Smooth animations: glow/fade using eased alpha, not linear
- [ ] Color-blind accessible palette option (use shapes not just colors)
- [ ] Persist settings in plugin-local storage (e.g. `localStorage` prefixed with
plugin id) — do **not** use `/api/settings` for this; the current server only
persists a fixed set of known keys and will silently discard `notedetect_feedback`
**Estimated scope:** ~100 lines JS, ~60 lines HTML
---
## Dependency Graph
```
Phase 0 (detection plugin)
↓
Phase 1 (matching core)
↓
Phase 2 (hit/miss overlay) ← Minimum viable feature
↓
Phase 3 (diagnostic labels)
↓
Phase 4 (loop tracking) ← Core practice value
↓
Phase 5 (section grading)
↓
Phase 6 (polish)
```
Phases 3-5 are independent of each other and can be done in any order after Phase 2.
Phase 6 should be last.
---
## Risk / Open Questions
1. **Latency budget:** Detection → WebSocket → matching → render adds latency.
If total pipeline > 100ms, the match window needs to compensate with asymmetric
tolerance (more lenient for "late" detections). Measure in Phase 1.
2. **Chord matching granularity:** Current plan matches chord notes individually.
Should a chord be "missed" if 4/6 notes hit? Propose: grade chords as
percentage, treat as HIT if ≥50% of notes matched. Revisit after Phase 2 testing.
3. **Tempo-scaled thresholds:** At 200 BPM, a 200ms match window covers almost
an entire beat. Should thresholds scale with tempo? Propose: don't over-engineer
this initially. Fixed thresholds work for most tempos. Revisit if users report
issues at extreme tempos.
4. **Detection plugin availability:** Everything in Phases 1-6 degrades gracefully
if the detection WebSocket isn't connected — the draw hook simply has no
judgments to render, and the highway looks exactly as it does today.
docs/NOTE_FAILURE_SPEC.md
+390
View File
@@ -0,0 +1,390 @@
# Note Failure Feedback — Technical Spec
## Goal
When a user loops over a lick, **show note misses on the highway** with diagnostic
detail: which note was missed, and *how* it was missed (timing vs pitch).
Many rhythm-practice tools show a `!` marker at the missed note position after it
passes. We improve on this by showing *why* the note was missed — too early, too
late, wrong pitch, or not played at all.
---
## Prerequisites
This feature depends on the **note detection plugin** (`slopsmith-plugin-notedetect`),
which provides real-time pitch detection via server-side aubio/YIN over WebSocket.
The detection plugin streams `DetectedNote` events; this spec describes the
**matching, judgment, and rendering** layer that consumes those events.
Without the detection plugin active, no miss/hit feedback is shown — the highway
renders exactly as it does today.
---
## Architecture
```
Guitar → USB Adapter → sounddevice (server)
↓
aubio YIN detection
↓
WebSocket: detected notes
↓
┌───────────────────────┐
│ Note Matcher │ ← THIS SPEC
│ (client-side JS) │
│ │
│ Chart notes (highway) │
│ × Detected notes (WS) │
│ = Match/Miss/Extra │
└───────────────────────┘
↓
Highway draw hook overlay
(hit glow, miss markers, diagnostics)
```
### Data Flow
1. **Chart notes** arrive via existing highway WebSocket (`/ws/highway/{filename}`).
Wire format: `{ t, s, f, sus, bn, ho, po, ... }` (see `lib/song.py:note_to_wire`)
2. **Detected notes** arrive via detection plugin WebSocket
(`/api/plugins/note_detect/stream`).
Wire format: `{ note: "A2", freq: 110.0, confidence: 0.92, time: 1.234 }`
> **Plugin naming note:** The detection plugin's repository is named
> `slopsmith-plugin-notedetect`, but the plugin registers with the id
> `note_detect` (snake_case). Its HTTP/WebSocket routes therefore appear
> under `/api/plugins/note_detect/…`. There is no `window.slopsmithPlugin_*`
> global pattern in Slopsmith — to check whether the detection plugin is
> available at runtime, attempt a fetch to `/api/plugins/note_detect/status`
> (or similar) or consult the `/api/plugins` list. Use the repo name only
> in documentation links.
3. **Note Matcher** (new, client-side) correlates these two streams in real-time.
4. **Draw hook** renders results on the highway via `highway.addDrawHook()`.
> **⚠ Limitation:** `addDrawHook()` is only invoked by the **default 2D renderer**.
> If the user has switched to a custom renderer (e.g., a WebGL 3D highway plugin),
> draw hooks are not called and this overlay will be invisible. Implementers should
> note this in the plugin's UI (e.g., a warning banner when a non-default renderer
> is detected) and may want to explore a renderer-agnostic overlay approach
> (own canvas + own rAF loop, reading public highway state via getters) as a
> future improvement.
---
## Note Matching Algorithm
### Match Window
A detected note matches a chart note when:
| Criterion | Threshold | Notes |
|----------------|------------------------|---------------------------------------------|
| **Time** | ±200ms (configurable) | Centered on chart note time |
| **Pitch** | ±50 cents | Accounts for imperfect intonation |
| **String** | Pitch-only (for now) | `DetectedNote` carries no string field; exact-string matching requires the detection plugin to be extended to emit a string estimate. Treat string as always-unknown until that extension lands. |
### Expected Frequency Calculation
```javascript
// Open-string base frequencies (Hz), string index 0 = lowest string.
// Select by highway.getStringCount() + arrangement name from highway.getSongInfo():
// 4-string → BASS_TUNING (E1 A1 D2 G2)
// 5-string bass → BASS5_TUNING (B0 E1 A1 D2 G2)
// 6-string (default)→ GUITAR_TUNING (E2 A2 D3 G3 B3 E4)
// 7-string → GUITAR7_TUNING (B1 E2 A2 D3 G3 B3 E4)
const GUITAR_TUNING = [82.41, 110.00, 146.83, 196.00, 246.94, 329.63];
const BASS_TUNING = [41.20, 55.00, 73.42, 98.00];
const BASS5_TUNING = [30.87, 41.20, 55.00, 73.42, 98.00];
const GUITAR7_TUNING = [61.74, 82.41, 110.00, 146.83, 196.00, 246.94, 329.63];
function getBaseTuning(stringCount, arrangementName) {
const isBass = /bass/i.test(arrangementName || '');
if (stringCount === 4) return BASS_TUNING;
if (stringCount === 5 && isBass) return BASS5_TUNING;
if (stringCount === 7) return GUITAR7_TUNING;
return GUITAR_TUNING; // 6-string or unknown
}
function expectedFreq(string, fret, tuningOffsets, capo = 0, stringCount, arrangementName = '') {
// tuningOffsets: per-string semitone offsets from standard (from song metadata).
// stringCount: REQUIRED — pass highway.getStringCount(). Do NOT default to
// tuningOffsets.length: RS XML sources pad the tuning array to length 6 even for
// 4-string bass, which would cause incorrect base-tuning selection.
// arrangementName: pass highway.getSongInfo().arrangement to resolve ambiguous
// 5-string cases (5-string bass vs 5-string extended guitar).
const BASE = getBaseTuning(stringCount, arrangementName);
if (string < 0 || string >= stringCount || string >= BASE.length) {
return null;
}
const semitones = tuningOffsets[string] + capo;
const base = BASE[string] * Math.pow(2, semitones / 12);
return base * Math.pow(2, fret / 12);
}
```
### Match States
Each chart note resolves to a **judgment** with two independent axes:
- **Timing axis** (`timingState`): `'OK'` if within `timingThresholdMs`, `'EARLY'` if matched
more than `timingThresholdMs` before chart time, `'LATE'` if more than `timingThresholdMs`
after. `null` if the note was never matched (MISSED).
- **Pitch axis** (`pitchState`): `'OK'` if detected pitch is within `pitchThresholdCents` of
expected, `'SHARP'` if above by more than `pitchThresholdCents`, `'FLAT'` if below. `null`
if unmatched.
- **`hit`**: `true` when both axes are `'OK'`; `false` for MISSED or any off-axis result.
The axes combine independently (e.g., `LATE + FLAT`, `EARLY + SHARP`). The state diagram
below shows possible terminal values per axis:
```
PENDING → hit=true, timingState='OK', pitchState='OK'
→ hit=false, timingState=null, pitchState=null (MISSED — window expired)
→ hit=false, timingState='EARLY', pitchState=... (too early)
→ hit=false, timingState='LATE', pitchState=... (too late)
→ hit=false, timingState='OK', pitchState='SHARP'
→ hit=false, timingState='OK', pitchState='FLAT'
```
Timing and pitch thresholds are read from configuration (`timingThresholdMs`,
`pitchThresholdCents`) — never hard-coded.
### Judgment Data Structure
```javascript
// Per-note judgment, attached after the note passes the now-line.
// Compound judgments (e.g. LATE + FLAT) are expressed as separate
// timingState / pitchState fields; never concatenate them into `state`.
{
chartNote: { t, s, f, ... }, // Original chart note
// Overall outcome — top-level quick check
hit: false, // true iff timing AND pitch are both clean
// Timing axis: null if no detection arrived (pure MISSED)
timingState: 'EARLY' | 'LATE' | 'OK' | null,
timingError: -120, // Milliseconds (negative = early); null if no detection
// Pitch axis: null if no detection arrived (same condition as timingState).
// Pitch is evaluated independently for any matched detection — a LATE note
// can also be FLAT (both axes are set even when timingState ≠ 'OK').
pitchState: 'SHARP' | 'FLAT' | 'OK' | null,
pitchError: +15, // Cents (positive = sharp); null if no detection
// Raw detection data (null if no detection arrived)
detectedFreq: 112.3, // What was actually played
expectedFreq: 110.0, // What should have been played
detectedAt: 1.354, // When the detection arrived
}
```
**Precedence / rendering rules:**
- `hit: true` → green glow; both timing/pitch states will be `'OK'`. A judgment is a hit when `|timingError| ≤ timingThresholdMs` **and** `|pitchError| ≤ pitchThresholdCents` (see §Configuration).
- `timingState: null` (no detection) → pure miss (`✕`); skip pitch display.
- Non-null `timingState` + non-null `pitchState` → compound: render timing
indicator on top, pitch indicator below.
- Emitted `note:hit` / `note:miss` events carry the full judgment object so
subscribers can inspect either axis independently.
---
## Highway Rendering
### Hit Feedback
Notes matched within `timingThresholdMs` (default 100 ms) **and**
`pitchThresholdCents` (default 20 ¢) get a **green glow ring** that fades over
`hitGlowDuration` (default 0.5 s). This is the combination that sets `hit: true`
on the judgment object. The existing note rendering is unchanged — the glow is
drawn *behind* the note at the now-line position as it passes.
```
[existing note bubble]
└── green glow ring (additive blend, fades)
```
### Miss Markers
Missed notes get a persistent marker that continues downward past the now-line
and remains visible for 2 seconds (configurable). The marker stays at the
note's string/fret position on the "past" portion of the highway (below
now-line).
**Positioning rule:** do **not** rely on `highway.project(tOffset)` for the full
miss-marker lifetime below the now-line. That helper returns `null` for offsets
more than ~50ms into the past, so it cannot place markers that persist for
seconds after the note passes. Instead, define a dedicated mapping anchored at
the now-line:
- `tOffset = 0` starts at the now-line (use `highway.project(0)` to get this Y).
- For the past region (`tOffset < 0` up to `-missMarkerDuration`), place the
marker below the now-line using a configurable linear pixels-per-second mapping,
clamped to the visible past area.
- The existing `highway.project()` may still be used for positions at/above the
now-line (approaching notes in the last ~50ms), but once a marker has crossed
into the past region its Y is governed by this below-now-line mapping.
| State | Visual |
|--------|-------------------------------------------------------------|
| MISSED | Red `✕` at note position + red tint on string segment |
| EARLY | Orange `↑` (up arrow) + timing offset label (e.g., "-120ms")|
| LATE | Orange `↓` (down arrow) + timing offset label ("+85ms") |
| SHARP | Blue `♯` + cents label ("+35¢") |
| FLAT | Blue `♭` + cents label ("-42¢") |
Compound states stack vertically: timing indicator on top, pitch indicator below.
### Miss markers on the string area
Below the now-line, all strings for the active arrangement are always visible
(use `highway.getStringCount()` — 4 for bass, 6 for guitar, 7+ for extended-range).
For a missed note,
the relevant string segment between the now-line and ~20px below it gets a brief
red pulse (200ms fade).
### Loop Iteration Summary
When A-B looping is active, at the end of each loop iteration (when playback
wraps from B back to A), show a brief overlay:
```
┌─────────────────────┐
│ Loop 3/∞ │
│ 5/7 notes hit (71%)│
│ Best: 6/7 (86%) │
└─────────────────────┘
```
Displayed for 1.5s, then fades. Does not block the highway.
---
## State Management
### NoteJudgmentTracker
Client-side class that manages the correlation between chart notes and detections.
```javascript
class NoteJudgmentTracker {
constructor(chartNotes, chartChords, tuning) { ... }
// Called when a detected note arrives from the detection WebSocket
addDetection(detected) { ... }
// Called each frame; checks for expired match windows
update(currentTime) { ... }
// Returns judgments for notes in the visible time range
getJudgmentsInRange(tStart, tEnd) { ... }
// Reset (on song change, loop restart, arrangement switch)
reset() { ... }
// Stats for the current loop iteration
getLoopStats() { ... }
}
```
### Memory Management
- Judgments older than 10 seconds behind current time are pruned each frame.
- Detection buffer holds last 5 seconds of raw detections.
- On loop wrap (B→A), archive current iteration stats, reset judgments for
the loop range, keep detections flowing.
### Loop-Aware Behavior
The tracker must handle A-B looping:
1. Detect loop wrap: `currentTime < previousTime - 0.5` (jumped backward).
2. On wrap: snapshot current stats to `loopHistory[]`, reset judgments
for notes in `[loopA, loopB]` range.
3. `getLoopStats()` returns current iteration + best historical iteration.
---
## Integration Points
### Existing Highway API Used
| API | Purpose |
|------------------------------|------------------------------------------|
| `highway.addDrawHook(fn)` | Register the overlay renderer |
| `highway.removeDrawHook(fn)` | Cleanup on plugin unload |
| `highway.getTime()` | Current chart time (audio-aligned) |
| `highway.getAvOffset()` | A/V offset in ms; visual render clock = `getTime() + getAvOffset()/1000` — use this when computing `tOffset` for `project()` calls inside draw hooks, otherwise markers appear shifted when the user has calibrated A/V latency |
| `highway.getNotes()` | All chart notes (for matching) |
| `highway.getChords()` | All chart chords (match individual notes)|
| `highway.getSections()` | Section boundaries (for section grading) |
| `highway.getSongInfo()` | Tuning offsets for frequency calculation |
| `highway.project(tOffset)` | Convert time offset to Y position |
| `highway.fretX(fret, scale, w)` | Convert fret to X position using `scale` from `highway.project(tOffset)` |
| `highway.fillTextUnmirrored` | Text that stays readable in lefty mode |
### Existing App.js Used
| Global | Purpose |
|------------------------------|------------------------------------------|
| `loopA`, `loopB` | Current A-B loop boundaries |
| `audio.currentTime` | Actual audio playback position |
### New Events Emitted (via `window.slopsmith.emit`)
| Event | Payload |
|------------------------------|------------------------------------------|
| `note:hit` | full `Judgment` object (see §Judgment Data Structure) |
| `note:miss` | full `Judgment` object |
| `loop:complete` | `{ iteration, stats }` |
| `note:sectionGrade` | `{ section, grade, hits, total }` |
`note:hit` and `note:miss` always carry the complete `Judgment` object so
subscribers can inspect `timingState`, `pitchState`, timing/pitch errors, and
raw detection data independently, without the emitter having to pre-select fields.
---
## Configuration (plugin settings)
There are three distinct threshold tiers — keep them conceptually separate:
| Tier | Setting(s) | Role |
|------|-----------|------|
| **Match window** | `matchWindowMs`, `pitchToleranceCents` | Outer gate: a detection is only correlated to a chart note if it falls within these limits. Outside → ignored (extra note, not an attempt). |
| **Hit threshold** | `timingThresholdMs`, `pitchThresholdCents` | Sets `hit: true`. A matched note is a clean hit when `|timingError| ≤ timingThresholdMs` **and** `|pitchError| ≤ pitchThresholdCents`. Also triggers the green glow. Must be ≤ match window values. |
| **Label threshold** | (same keys) | Same values double as the boundary at which EARLY/LATE/SHARP/FLAT labels appear. Within the hit threshold = `'OK'` state; outside = labeled state. |
| Setting | Default | Description |
|------------------------|---------|----------------------------------------------------------------|
| `matchWindowMs` | 200 | Outer time tolerance for correlating a detection to a chart note (ms) |
| `pitchToleranceCents` | 50 | Outer pitch tolerance for correlation (cents) |
| `timingThresholdMs` | 100 | `|timingError| ≤ this` → `timingState: 'OK'` and `hit` eligible; also defines the EARLY/LATE label boundary |
| `pitchThresholdCents` | 20 | `|pitchError| ≤ this` → `pitchState: 'OK'` and `hit` eligible; also defines the SHARP/FLAT label boundary |
| `showTimingErrors` | true | Show EARLY/LATE labels when `timingState` is non-OK |
| `showPitchErrors` | true | Show SHARP/FLAT labels when `pitchState` is non-OK |
| `missMarkerDuration` | 2.0 | How long miss markers stay visible (sec) |
| `showLoopSummary` | true | Show stats on loop wrap |
| `hitGlowDuration` | 0.5 | Green glow fade time (sec) |
Persist these settings in plugin-local storage (e.g. `localStorage` prefixed
with the plugin id). Do **not** assume they can be saved through Slopsmith's
`/api/settings` endpoint under a `notedetect_feedback` key — the current server
only persists a fixed set of known settings keys. If backend support for a
dedicated persisted key is added later, this plugin may migrate to `/api/settings`.
---
## What This Does NOT Cover
- **Audio input / pitch detection** — handled by the detection plugin
- **Device selection UI** — handled by the detection plugin
- **Score persistence / history** — future work (practice journal plugin)
- **Difficulty scaling** — automatic dynamic-difficulty scaling is not implemented
- **Chord grading** — chords are graded per-note (each note in the chord
is independently matched), not as a single unit
docs/benchmarks/note_detect_bass_v1/BENCHMARK.md
+34
View File
@@ -0,0 +1,34 @@
# Slopsmith Note Detect Bass Benchmark — v1
A bass-focused companion to the guitar benchmarks
(note_detect_v1 + note_detect_v2). Tests `note_detect` against bass-
specific idioms: walking lines, octave jumps, root+fifth patterns,
double-stops, and long low-E holds that stress YIN's accumulator at
~41 Hz.
- **Tempo**: 90 BPM
- **Tuning**: E standard 4-string (E1 A1 D2 G2, no capo)
- **Audio**: metronome click track only — play *over* the click.
- **Duration**: 181 s
## Sections
| Section | Tests |
|---|---|
| A. Open strings (slow walk) | Mono detection on each open string, low → high → low |
| B. 5th-fret (slow walk) | Fretted-note detection across the 4 strings |
| C. Sustained notes | 3 × 4-second held roots |
| D. Octave walks | Root ↔ octave alternation, 2 strings + 2 frets up |
| E. Walking bassline | A minor pentatonic ascending + descending |
| F. Root + fifth pattern | Classic rock bass pattern (8 events) |
| G. Double-stops | 2-string voicings — the chord-scorer test for bass |
| H. Long low-E holds | 3 × 5-second E1 holds, stresses YIN under-buffer regime |
## Reporting
Diagnostic JSON schema is `note_detect.diagnostic.v1`. Filter
`benchmark_hint` to bucket bass vs guitar runs.
## Source
Built by `docs/benchmarks/note_detect_bass_v1/build_benchmark.py`.
docs/benchmarks/note_detect_bass_v1/build_benchmark.py
+503
View File
@@ -0,0 +1,503 @@
"""Builds the Note Detect Bass Benchmark sloppak (v1).
A bass-focused companion to the guitar benchmarks (note_detect_v1 +
note_detect_v2). Same 90 BPM click, similar half-note pacing as v2,
but the sections are built around what bass actually plays: single-
note lines, octave jumps, walking patterns, sustained roots, and
two-string double-stops (the closest bass gets to "chords").
Why a separate bass benchmark instead of toggling string count on
the guitar one:
- Tuning is different — 4-string bass open MIDI is [28, 33, 38, 43]
(E1, A1, D2, G2) vs the guitar's [40, 45, 50, 55, 59, 64]. The
benchmark needs to produce notes the player can actually play on
the instrument they have plugged in.
- Bass idioms are different from guitar idioms. Strumming sections
don't apply; walking bass + octave patterns do.
- Low-frequency detection is materially harder for YIN — E1 at
~41 Hz needs more accumulated samples for confident detection
than guitar E2 at ~82 Hz. The benchmark should exercise that
regime explicitly so we can spot regressions there.
How to run inside the slopsmith container:
docker cp docs/benchmarks/note_detect_bass_v1/build_benchmark.py \\
slopsmith-web-1:/tmp/build_benchmark_bass.py
docker exec slopsmith-web-1 python /tmp/build_benchmark_bass.py \\
/app/static/sloppak_cache/note_detect_benchmark_bass_v1.sloppak
After regenerating, copy the zip output to the tracked path with the
`.sloppak` (not `.sloppak.zip`) suffix.
"""
import json
import math
import shutil
import struct
import subprocess
import sys
import wave
from pathlib import Path
import yaml
# ── Benchmark parameters ────────────────────────────────────────────────
BPM = 90.0
SECONDS_PER_BEAT = 60.0 / BPM
BEATS_PER_BAR = 4
BAR_S = BEATS_PER_BAR * SECONDS_PER_BEAT
INTRO_BARS = 2
OUTRO_BARS = 2
EXERCISE_BARS = 8
# 4-string bass open MIDI per string, low → high.
# Matches lib/tunings convention used by note_detect when the
# arrangement is 'bass' and stringCount is 4.
OPEN_MIDI = [28, 33, 38, 43] # E1 A1 D2 G2
N_STRINGS = 4
SR = 44100
# ── Click-track audio generator ────────────────────────────────────────
def _sine_burst(freq_hz, duration_s, amplitude):
n = int(SR * duration_s)
out = []
fade = max(1, int(0.004 * SR))
for i in range(n):
env = 1.0
if i < fade:
env = i / fade
elif i >= n - fade:
env = (n - 1 - i) / fade
s = math.sin(2 * math.pi * freq_hz * (i / SR)) * amplitude * env
out.append(max(-1.0, min(1.0, s)))
return out
def write_click_wav(path: Path, duration_s: float):
total_samples = int(SR * duration_s)
pcm = [0] * total_samples
beat = 0
t = 0.0
while t < duration_s:
is_downbeat = (beat % BEATS_PER_BAR == 0)
freq = 1200 if is_downbeat else 800
amp = 0.6 if is_downbeat else 0.35
burst = _sine_burst(freq, 0.040, amp)
start = int(t * SR)
for i, s in enumerate(burst):
j = start + i
if 0 <= j < total_samples:
pcm[j] = int(max(-1.0, min(1.0, pcm[j] / 32767 + s)) * 32767)
t += SECONDS_PER_BEAT
beat += 1
pcm = [struct.pack('<h', v) for v in pcm]
path.parent.mkdir(parents=True, exist_ok=True)
with wave.open(str(path), 'wb') as w:
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(SR)
w.writeframes(bytes(b''.join(pcm)))
# ── Chart helpers ─────────────────────────────────────────────────────
def note(t, s, f, sus=0.0, **flags):
return {
't': round(t, 3),
's': s,
'f': f,
'sus': round(sus, 3),
'sl': flags.get('sl', -1),
'slu': flags.get('slu', -1),
'bn': flags.get('bn', 0.0),
'ho': flags.get('ho', False),
'po': flags.get('po', False),
'hm': flags.get('hm', False),
'hp': flags.get('hp', False),
'pm': flags.get('pm', False),
'mt': flags.get('mt', False),
'vb': flags.get('vb', False),
'tr': flags.get('tr', False),
'ac': flags.get('ac', False),
'tp': flags.get('tp', False),
}
def chord(t, id_, notes):
return {
't': round(t, 3),
'id': id_,
'hd': False,
'notes': notes,
}
def chord_note(s, f, sus=0.0, **flags):
n = note(0.0, s, f, sus, **flags)
n.pop('t')
return n
# ── Exercises ─────────────────────────────────────────────────────────
# Bass idioms: single notes dominate, occasional double-stops (root +
# fifth on adjacent higher string two frets up, or root + octave two
# strings + two frets up), long sustains. Half-note pacing throughout
# for the same "give the player time to land cleanly" reasoning as
# guitar v2.
def exercise_open_strings_slow(t0):
"""All 4 open strings, low → high → low. Tests the lowest end of
YIN's range (E1 = 41 Hz) where the under-buffering threshold
kicks in."""
seq = [0, 1, 2, 3, 3, 2, 1, 0]
notes_out = []
for i, s in enumerate(seq):
notes_out.append(note(t0 + i * 2 * SECONDS_PER_BEAT, s, 0,
sus=SECONDS_PER_BEAT * 1.6))
notes_out[-1]['sus'] = round(SECONDS_PER_BEAT * 4, 3)
return notes_out, [], 'Open strings (slow walk)'
def exercise_fretted_5th_slow(t0):
"""5th fret on each string, ascending. Maps to A1 / D2 / G2 / C3
— comfortable register for hand position, no stretch."""
seq = [(s, 5) for s in range(N_STRINGS)]
notes_out = []
for i, (s, f) in enumerate(seq):
notes_out.append(note(t0 + i * 2 * SECONDS_PER_BEAT, s, f,
sus=SECONDS_PER_BEAT * 1.6))
notes_out[-1]['sus'] = round(SECONDS_PER_BEAT * 4, 3)
return notes_out, [], 'Fretted positions (5th fret, slow walk)'
def exercise_sustained(t0):
"""Three 4-second sustained roots across the range. Tests the
`_sustainStillHeld` active-glow path on bass tonalities."""
sus = 4.0
targets = [(0, 5), (1, 7), (2, 5)] # A1, E2, G2 — spread across mid-range
notes_out = []
for i, (s, f) in enumerate(targets):
notes_out.append(note(t0 + i * (sus + 1.0), s, f, sus=sus))
return notes_out, [], 'Sustained notes (3 holds, 4 s each)'
def exercise_octave_walk(t0):
"""Octave jumps — common bass pattern (root note + octave on the
string two above). Pairs: (0,0)↔(2,2) = E1↔E2 octave. Plays root,
octave, root, octave at half-note pacing."""
pairs = [
(0, 0, 2, 2), # E1 ↔ E2
(1, 0, 3, 2), # A1 ↔ A2
]
notes_out = []
t = 0.0
for (sa, fa, sb, fb) in pairs:
notes_out.append(note(t0 + t, sa, fa, sus=SECONDS_PER_BEAT * 1.6))
t += 2 * SECONDS_PER_BEAT
notes_out.append(note(t0 + t, sb, fb, sus=SECONDS_PER_BEAT * 1.6))
t += 2 * SECONDS_PER_BEAT
notes_out.append(note(t0 + t, sa, fa, sus=SECONDS_PER_BEAT * 1.6))
t += 2 * SECONDS_PER_BEAT
notes_out.append(note(t0 + t, sb, fb, sus=SECONDS_PER_BEAT * 1.6))
t += 2 * SECONDS_PER_BEAT
notes_out[-1]['sus'] = round(SECONDS_PER_BEAT * 2, 3)
return notes_out, [], 'Octave walks (root ↔ octave)'
def exercise_walking_line(t0):
"""Walking bassline — root, third, fifth, sixth ascending, then
descending. Classic 4-bar walking pattern in A minor pentatonic
starting on A string open. Tests detection across a fretted run."""
# A1, C2, D2, E2 (ascend), E2, D2, C2, A1 (descend)
pattern = [
(1, 0), # A1
(1, 3), # C2
(1, 5), # D2
(1, 7), # E2
(1, 7), # E2
(1, 5), # D2
(1, 3), # C2
(1, 0), # A1
]
notes_out = []
for i, (s, f) in enumerate(pattern):
notes_out.append(note(t0 + i * 2 * SECONDS_PER_BEAT, s, f,
sus=SECONDS_PER_BEAT * 1.6))
notes_out[-1]['sus'] = round(SECONDS_PER_BEAT * 4, 3)
return notes_out, [], 'Walking bassline (A minor pentatonic)'
def exercise_root_fifth_pattern(t0):
"""Root + fifth alternation — single most common bass pattern in
rock / country. Plays (root, fifth, root, fifth) on each of two
voicings. The fifth sits on the next-higher string, 2 frets up
from the root — a one-finger reach with no string skip."""
# Root on (0, 0) = E1, fifth = (1, 2) = B1 (A string fret 2)
# Then root on (1, 0) = A1, fifth = (2, 2) = E2 (D string fret 2)
pattern = [
(0, 0), (1, 2), (0, 0), (1, 2),
(1, 0), (2, 2), (1, 0), (2, 2),
]
notes_out = []
for i, (s, f) in enumerate(pattern):
notes_out.append(note(t0 + i * 2 * SECONDS_PER_BEAT, s, f,
sus=SECONDS_PER_BEAT * 1.6))
notes_out[-1]['sus'] = round(SECONDS_PER_BEAT * 4, 3)
return notes_out, [], 'Root + fifth pattern'
def exercise_double_stops(t0):
"""Two-string "chord" events — closest bass gets to chords.
Root + fifth simultaneously on adjacent strings, repeated 8
times at half-note pacing. Lets the chord scorer exercise the
2-string code path with bass-range frequencies."""
# Voicing: E1 + B1 (root + fifth on E + A strings)
voicing = [(0, 0), (1, 2)]
strums = 8
chords_out = []
sus = SECONDS_PER_BEAT * 1.6
# Sloppak wire spec keeps chord-template fingers/frets in six-slot
# arrays even for bass (docs/sloppak-spec.md §chord-template), so we
# pad the unused two slots with -1; the chord notes themselves stay
# on strings 0–1.
template = {
'name': 'E5 (bass)', 'displayName': 'E5', 'arp': False,
'fingers': [-1, -1, -1, -1, -1, -1],
'frets': [ 0, 2, -1, -1, -1, -1],
}
for i in range(strums):
chord_notes = [chord_note(s, f, sus=sus) for (s, f) in voicing]
chords_out.append(chord(t0 + i * 2 * SECONDS_PER_BEAT, 0, chord_notes))
return [], (chords_out, [template]), 'Double-stops (root + fifth, 8 strums)'
def exercise_low_e_long_holds(t0):
"""Three long-held low E (open string, lowest note on the
instrument). Specifically targets YIN's under-buffering regime
— E1 at 41 Hz needs roughly 4096 samples for a confident lock
at 44.1 kHz, so the detector should spend ~95 ms accumulating
before it can report. Holds of 5 s each give the scorer huge
runway; if the detector can't lock here it can't lock anywhere."""
sus = 5.0
notes_out = []
for i in range(3):
notes_out.append(note(t0 + i * (sus + 0.5), 0, 0, sus=sus))
return notes_out, [], 'Long low-E holds (5 s each)'
EXERCISES = [
('A. Open strings (slow)', exercise_open_strings_slow),
('B. 5th-fret (slow)', exercise_fretted_5th_slow),
('C. Sustained notes', exercise_sustained),
('D. Octave walks', exercise_octave_walk),
('E. Walking bassline', exercise_walking_line),
('F. Root + fifth pattern', exercise_root_fifth_pattern),
('G. Double-stops (root + 5)', exercise_double_stops),
('H. Long low-E holds', exercise_low_e_long_holds),
]
# ── Driver ─────────────────────────────────────────────────────────────
def build(out_dir: Path):
notes_all = []
chords_all = []
templates_all = []
sections = []
beats = []
t = INTRO_BARS * BAR_S
for label, fn in EXERCISES:
sections.append({'name': label, 'number': len(sections) + 1, 'time': round(t, 3)})
result = fn(t)
ns, ch_or_tuple, _desc = result
notes_all.extend(ns)
if isinstance(ch_or_tuple, tuple):
cs, tmpls = ch_or_tuple
# Rebase section-local chord template ids — see v1/v2
# builders for the full explanation. Bass v1 only has one
# chord exercise today (double-stops), but applying the
# same offset pattern future-proofs the driver against the
# day someone adds a second chord exercise that also uses
# local-zero-based ids.
offset = len(templates_all)
for c in cs:
c['id'] = c.get('id', 0) + offset
chords_all.extend(cs)
templates_all.extend(tmpls)
else:
chords_all.extend(ch_or_tuple)
t += EXERCISE_BARS * BAR_S
end_t = t + OUTRO_BARS * BAR_S
bar_count = 0
bt = 0.0
while bt < end_t:
is_downbeat = abs(bt % BAR_S) < 1e-3
if is_downbeat:
bar_count += 1
beats.append({'time': round(bt, 3), 'measure': bar_count})
else:
beats.append({'time': round(bt, 3), 'measure': -1})
bt += SECONDS_PER_BEAT
anchors = [{'time': 0.0, 'fret': 1, 'width': 12}]
for sec in sections:
anchors.append({'time': sec['time'], 'fret': 1, 'width': 12})
arrangement = {
'name': 'Bass',
# Pad to 6 slots even on bass — slopsmith's `tuning_name()` only
# recognises named tunings (E Standard, Drop D, etc.) on 6-element
# arrays, so a 4-element array shows up in the library card as the
# raw numeric form ("0 0 0 0") instead of "E Standard". The
# arrangement name ("Bass") + note positions still drive the
# detector's bass-specific behaviour; this just makes the library
# display friendly.
'tuning': [0] * 6,
'capo': 0,
'notes': sorted(notes_all, key=lambda n: n['t']),
'chords': sorted(chords_all, key=lambda c: c['t']),
'anchors': anchors,
'handshapes': [],
'templates': templates_all,
'beats': beats,
'sections': sections,
}
manifest = {
'title': 'Note Detect Bass Benchmark v1',
'artist': 'Slopsmith',
'album': 'Note Detection Benchmark',
'year': 2026,
'duration': round(end_t, 3),
'arrangements': [
{
'id': 'bass',
'name': 'Bass',
'file': 'arrangements/bass.json',
# Pad to 6 slots — see arrangement-level comment.
'tuning': [0] * 6,
'capo': 0,
},
],
'stems': [
{'id': 'full', 'file': 'stems/full.ogg', 'default': True},
],
'benchmark': {
'id': 'slopsmith-note-detect-benchmark-bass',
'version': 1,
},
}
out_dir = Path(out_dir)
if out_dir.exists():
# Defensive — see v1 builder. Only rmtree something that looks
# like a sloppak so a typo on the CLI doesn't nuke an unrelated
# directory.
if not (out_dir.suffix == '.sloppak'
or (out_dir / 'manifest.yaml').exists()):
raise RuntimeError(
f"refusing to rmtree {out_dir!r}: does not look like a sloppak "
f"(no .sloppak suffix, no manifest.yaml)."
)
shutil.rmtree(out_dir)
out_dir.mkdir(parents=True)
(out_dir / 'arrangements').mkdir()
(out_dir / 'stems').mkdir()
(out_dir / 'manifest.yaml').write_text(
yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True),
encoding='utf-8',
)
(out_dir / 'arrangements' / 'bass.json').write_text(
json.dumps(arrangement, separators=(',', ':')),
encoding='utf-8',
)
wav_path = out_dir / 'stems' / 'full.wav'
write_click_wav(wav_path, end_t)
ogg_path = out_dir / 'stems' / 'full.ogg'
subprocess.run(
['ffmpeg', '-y', '-loglevel', 'error',
'-i', str(wav_path),
'-c:a', 'libvorbis', '-q:a', '5',
str(ogg_path)],
check=True,
)
wav_path.unlink()
(out_dir / 'BENCHMARK.md').write_text(_benchmark_readme(end_t), encoding='utf-8')
_build_zip(out_dir)
print(f'Built {out_dir}')
print(f' {out_dir}.zip')
print(f' Duration: {end_t:.1f} s')
print(f' Notes: {len(arrangement["notes"])}')
print(f' Chords: {len(arrangement["chords"])}')
print(f' Templates:{len(arrangement["templates"])}')
def _build_zip(src_dir: Path):
"""Pack with fixed dates / attrs for zip-metadata reproducibility.
See v1 guitar builder docstring for full caveats."""
import zipfile
zip_path = src_dir.with_suffix(src_dir.suffix + '.zip')
if zip_path.exists():
zip_path.unlink()
with zipfile.ZipFile(zip_path, 'w', compression=zipfile.ZIP_DEFLATED) as zf:
for p in sorted(src_dir.rglob('*')):
if p.is_file():
rel = p.relative_to(src_dir).as_posix()
info = zipfile.ZipInfo(filename=rel, date_time=(1980, 1, 1, 0, 0, 0))
info.compress_type = zipfile.ZIP_DEFLATED
info.external_attr = (0o644 & 0xFFFF) << 16
info.create_system = 3 # POSIX — see v1 builder for why
zf.writestr(info, p.read_bytes())
def _benchmark_readme(duration_s):
return f"""# Slopsmith Note Detect Bass Benchmark — v1
A bass-focused companion to the guitar benchmarks
(note_detect_v1 + note_detect_v2). Tests `note_detect` against bass-
specific idioms: walking lines, octave jumps, root+fifth patterns,
double-stops, and long low-E holds that stress YIN's accumulator at
~41 Hz.
- **Tempo**: {BPM:g} BPM
- **Tuning**: E standard 4-string (E1 A1 D2 G2, no capo)
- **Audio**: metronome click track only — play *over* the click.
- **Duration**: {duration_s:.0f} s
## Sections
| Section | Tests |
|---|---|
| A. Open strings (slow walk) | Mono detection on each open string, low → high → low |
| B. 5th-fret (slow walk) | Fretted-note detection across the 4 strings |
| C. Sustained notes | 3 × 4-second held roots |
| D. Octave walks | Root ↔ octave alternation, 2 strings + 2 frets up |
| E. Walking bassline | A minor pentatonic ascending + descending |
| F. Root + fifth pattern | Classic rock bass pattern (8 events) |
| G. Double-stops | 2-string voicings — the chord-scorer test for bass |
| H. Long low-E holds | 3 × 5-second E1 holds, stresses YIN under-buffer regime |
## Reporting
Diagnostic JSON schema is `note_detect.diagnostic.v1`. Filter
`benchmark_hint` to bucket bass vs guitar runs.
## Source
Built by `docs/benchmarks/note_detect_bass_v1/build_benchmark.py`.
"""
if __name__ == '__main__':
out = Path(sys.argv[1]) if len(sys.argv) > 1 else Path('./note_detect_benchmark_bass_v1.sloppak')
build(out)
docs/benchmarks/note_detect_bass_v1/note_detect_benchmark_bass_v1.sloppak
BIN
View File
Binary file not shown.
docs/benchmarks/note_detect_v1/BENCHMARK.md
+46
View File
@@ -0,0 +1,46 @@
# Slopsmith Note Detect Benchmark — v1
A short test piece for tuning Slopsmith's `note_detect` plugin. Eight
exercises, each isolating a specific detection failure mode. Run with
**Detect** enabled, play through, then export the diagnostic JSON
(Settings → Plugins → Note Detection → Download Diagnostic JSON, or
the button on the end-of-session summary modal).
- **Tempo**: 90 BPM
- **Tuning**: E standard (no capo)
- **Audio**: metronome click track only (downbeat = louder + higher
tone). Play *over* the click — `note_detect` listens to your guitar
signal, not the audio in this file.
- **Duration**: 139 s
## Sections
| Section | Tests | Watch in the diagnostic |
|---|---|---|
| A. Open strings (low→high→low) | Basic mono detection on each open string | `pure` (mic/audio chain), per-string accuracy |
| B. 5th-fret positions | Fretted-note detection across all 6 strings | per-string variance |
| C. 12th-fret octaves | Higher-frequency detection — YIN's octave-up risk | `sharp` bin spiking |
| D. Sustained notes (4 s) | The `active` held-on-pitch glow | `sharp`/`flat` drift while held |
| E. Hammer-on / pull-off | Transient detection without a fresh pick attack | `pure` (no transient registered) |
| F. Power chords (2-string) | Chord leniency on sparse voicings | `chordPartial` |
| G. Open major chords | Chord leniency on dense voicings (E, A, D, G) | `chordPartial` |
| H. Bends (half- + whole-step) | Single-note pitch tolerance with pitch in motion | `sharp` bin |
## Reporting
Share the JSON (schema `note_detect.diagnostic.v1`). It includes:
- Hit/miss totals split single-note vs chord
- Primary-cause bin per miss (pure / chord-partial / early / late / sharp / flat)
- Per-string hit rate
- Signed timing- and pitch-error percentiles (p10 / median / p90)
- Detection settings snapshot (method, tolerances, leniency)
- Per-judgment event log (capped at 2000 events) with the chart note's
technique flags so each miss can be re-binned by `SUS`/`B`/`H`/etc. offline
- `benchmark_hint`: `{title, artist, arrangement}` — filter on these
to bucket reports from different runs of this benchmark.
## Source
Built by `docs/benchmarks/note_detect_v1/build_benchmark.py` in the
slopsmith repo. Tweak the exercise list there and regenerate.
docs/benchmarks/note_detect_v1/build_benchmark.py
+601
View File
@@ -0,0 +1,601 @@
"""Builds the Note Detect Benchmark sloppak (v1).
A reproducible, distributable test piece for the note_detect plugin: 8
short exercises designed to isolate specific failure modes (open-string
mono, fretted positions, octaves, sustained held notes, hammer-on /
pull-off, sparse power chords, dense open chords, bends).
How to run inside the slopsmith container (recommended — has ffmpeg +
pyyaml already):
docker cp docs/benchmarks/note_detect_v1/build_benchmark.py \
slopsmith-web-1:/tmp/build_benchmark.py
docker exec slopsmith-web-1 python /tmp/build_benchmark.py \
/app/static/sloppak_cache/note_detect_benchmark_v1.sloppak
The output sloppak lands under `static/sloppak_cache/` on the host
(bind-mounted into the container). Copy / zip it from there.
"""
import json
import math
import shutil
import struct
import subprocess
import sys
import wave
from pathlib import Path
import yaml # bundled with the slopsmith image
# ── Benchmark parameters ────────────────────────────────────────────────
BPM = 90.0
SECONDS_PER_BEAT = 60.0 / BPM # 0.6667
BEATS_PER_BAR = 4
BAR_S = BEATS_PER_BAR * SECONDS_PER_BEAT # 2.667
INTRO_BARS = 2 # silence before the first event
OUTRO_BARS = 2 # tail after the last
EXERCISE_BARS = 6 # length of each exercise
# Standard E-tuning open MIDI per string, low → high (matches lib/tunings
# convention used by note_detect when arrangement is 'guitar').
OPEN_MIDI = [40, 45, 50, 55, 59, 64] # E2 A2 D3 G3 B3 E4
SR = 44100 # sample rate for the click WAV
# ── Click-track audio generator ────────────────────────────────────────
def _sine_burst(freq_hz, duration_s, amplitude):
"""Short sine burst with a linear attack/release envelope so the
click reads as a tick, not a pop."""
n = int(SR * duration_s)
out = []
fade = max(1, int(0.004 * SR)) # 4 ms fade in + out
for i in range(n):
env = 1.0
if i < fade:
env = i / fade
elif i >= n - fade:
env = (n - 1 - i) / fade
s = math.sin(2 * math.pi * freq_hz * (i / SR)) * amplitude * env
out.append(s)
return out
def write_click_wav(path: Path, total_duration_s: float):
"""A click on every beat; the downbeat (beat 0 of each bar) is louder
and a tone higher. Steady reference for the player; the chart's
event times sit on the same beat grid."""
n_total = int(math.ceil(total_duration_s * SR))
buf = [0.0] * n_total
click_dur = 0.045
downbeat_tone = 1500
upbeat_tone = 1000
downbeat_amp = 0.22
upbeat_amp = 0.12
beat_idx = 0
t = 0.0
while t < total_duration_s - click_dur:
is_downbeat = (beat_idx % BEATS_PER_BAR) == 0
click = _sine_burst(
downbeat_tone if is_downbeat else upbeat_tone,
click_dur,
downbeat_amp if is_downbeat else upbeat_amp,
)
i0 = int(t * SR)
for j, v in enumerate(click):
if i0 + j < n_total:
buf[i0 + j] += v
t += SECONDS_PER_BEAT
beat_idx += 1
# Soft clip to keep within 16-bit headroom even if a future tweak
# piles bursts up.
pcm = bytearray()
for v in buf:
s = max(-1.0, min(1.0, v))
pcm.extend(struct.pack('<h', int(s * 32700)))
path.parent.mkdir(parents=True, exist_ok=True)
with wave.open(str(path), 'wb') as w:
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(SR)
w.writeframes(bytes(pcm))
# ── Chart helpers ─────────────────────────────────────────────────────
def note(t, s, f, sus=0.0, **flags):
"""Build a single-note dict in the sloppak wire format. Defaults
match the wire-format defaults from docs/sloppak-spec.md §3.2."""
return {
't': round(t, 3),
's': s,
'f': f,
'sus': round(sus, 3),
'sl': flags.get('sl', -1),
'slu': flags.get('slu', -1),
'bn': flags.get('bn', 0.0),
'ho': flags.get('ho', False),
'po': flags.get('po', False),
'hm': flags.get('hm', False),
'hp': flags.get('hp', False),
'pm': flags.get('pm', False),
'mt': flags.get('mt', False),
'vb': flags.get('vb', False),
'tr': flags.get('tr', False),
'ac': flags.get('ac', False),
'tp': flags.get('tp', False),
}
def chord(t, id_, notes):
return {
't': round(t, 3),
'id': id_,
'hd': False,
'notes': notes,
}
def chord_note(s, f, sus=0.0, **flags):
n = note(0.0, s, f, sus, **flags)
n.pop('t') # chord notes inherit the chord's time
return n
# ── Exercises ─────────────────────────────────────────────────────────
# Each returns a 3-tuple `(notes, chords_or_with_templates, description)`.
# The middle slot is overloaded so single-note exercises don't have to
# carry a useless empty `templates` list:
# • Single-note exercises return `(notes, [], desc)` — second slot is
# just the (empty) chords list.
# • Chord exercises return `(notes, (chords, templates), desc)` — the
# driver unpacks the tuple when it sees one (see `build()`).
# Exercise start times are computed by the driver; helpers use `t0` as
# the exercise's bar-aligned start time, then place events relative to it.
def exercise_open_strings(t0):
"""Single notes — open strings, low → high → low, quarter notes."""
seq = [0, 1, 2, 3, 4, 5, 5, 4, 3, 2, 1, 0] # 12 notes = 3 bars at q-note
notes = []
for i, s in enumerate(seq):
notes.append(note(t0 + i * SECONDS_PER_BEAT, s, 0, sus=SECONDS_PER_BEAT * 0.9))
# Cap the last note's sustain into the trailing bar so it rings out
notes[-1]['sus'] = round(SECONDS_PER_BEAT * 3, 3)
return notes, [], 'Open strings (low→high→low)'
def exercise_fretted_positions(t0):
"""Each string's 5th fret, ascending then descending. Tests basic
fretted-note detection across the range."""
seq = [(s, 5) for s in range(6)] + [(s, 5) for s in range(5, -1, -1)]
notes = []
for i, (s, f) in enumerate(seq):
notes.append(note(t0 + i * SECONDS_PER_BEAT, s, f, sus=SECONDS_PER_BEAT * 0.9))
notes[-1]['sus'] = round(SECONDS_PER_BEAT * 3, 3)
return notes, [], 'Fretted positions (5th fret on each string)'
def exercise_octaves(t0):
"""12th-fret octaves on each string. Tests detection at higher
frequencies where YIN can lock onto the second harmonic."""
notes = []
# 6 notes, half-note each (2 beats), so the player has time to land
# cleanly. 6 × 2 = 12 beats = 3 bars.
for i, s in enumerate(range(6)):
notes.append(note(t0 + i * 2 * SECONDS_PER_BEAT, s, 12,
sus=SECONDS_PER_BEAT * 1.6))
notes[-1]['sus'] = round(SECONDS_PER_BEAT * 3, 3)
return notes, [], '12th-fret octaves'
def exercise_sustained(t0):
"""Four-second sustained notes. The renderer's `active` glow
requires the provider to keep returning state — exercises the
on-pitch hold check (`_sustainStillHeld`)."""
sus = 4.0
# Three targets spread across the range (low / mid / high). Held at
# 3 to keep the whole exercise within the section's 16 s slot —
# 4 events with a 4-s sustain at a 5-s cadence would end at t0+19
# and bleed 3 s into the next section's note-detect window, which
# contaminates the bin attribution we promise section-by-section.
targets = [(0, 5), (2, 7), (5, 5)]
notes = []
# One every 5 seconds (4-sec sustain + 1-sec gap). 3 events × 5 s
# = 14 s of music, comfortably inside EXERCISE_BARS * BAR_S = 16 s.
for i, (s, f) in enumerate(targets):
notes.append(note(t0 + i * (sus + 1.0), s, f, sus=sus))
return notes, [], 'Sustained notes (4 s each, on-pitch hold)'
def exercise_hammer_pull(t0):
"""Open → hammer-on → pull-off. Hammer-ons and pull-offs have no
fresh pick attack, so transient detection is what's tested."""
notes = []
# Pattern per bar: D3 (s=1, f=5 — A-string fretted at 5) picked, HO
# to f=7 (E3), PO back to f=5 (D3). HO/PO flags ride the destination
# note, not the source — that's where the technique is performed.
# Use 4 bars.
for bar in range(4):
bt = t0 + bar * BAR_S
notes.append(note(bt + 0 * SECONDS_PER_BEAT, 1, 5, sus=0.4)) # pluck D3
notes.append(note(bt + 1 * SECONDS_PER_BEAT, 1, 7, sus=0.4, ho=True))
notes.append(note(bt + 2 * SECONDS_PER_BEAT, 1, 5, sus=0.4, po=True))
# rest on beat 4
return notes, [], 'Hammer-on / pull-off (no pick attack)'
def exercise_power_chords(t0):
"""Two-string power chords. Sparse voicing tests whether the chord
leniency threshold is appropriate for 2-string chord events."""
# Wire format: s=0 is the lowest-pitched string (low E on guitar),
# s=5 the highest (high E). Two-string power-chord voicings, each
# rooted on the lower of the two strings:
# E5 — low E open + A fret 2 (E2 + B2)
# A5 — A open + D fret 2 (A2 + E3)
# D5 — D open + G fret 2 (D3 + A3)
# G5 — G open + B fret 3 (G3 + D4)
voicings = [
('E5', [(0, 0), (1, 2)]),
('A5', [(1, 0), (2, 2)]),
('D5', [(2, 0), (3, 2)]),
('G5', [(3, 0), (4, 3)]),
]
templates = []
chords_out = []
sus = SECONDS_PER_BEAT * 1.6
# 8 chord events over 8 half-note slots (4 bars at half notes).
pattern = list(range(4)) + list(range(4)) # play each voicing twice
for slot, idx in enumerate(pattern):
name, sf = voicings[idx]
tmpl_id = idx
if slot < len(voicings): # only add each template once
frets = [-1] * 6
for (s, f) in sf:
frets[s] = f
templates.append({
'name': name,
'displayName': name,
'arp': False,
'fingers': [-1] * 6,
'frets': frets,
})
chord_notes = [chord_note(s, f, sus=sus) for (s, f) in sf]
chords_out.append(chord(t0 + slot * 2 * SECONDS_PER_BEAT, tmpl_id, chord_notes))
return [], (chords_out, templates), 'Power chords (2-string sparse voicings)'
def exercise_open_chords(t0):
"""Open major chords. Dense voicings test whether the leniency
threshold is too strict when the player can't reliably ring every
string."""
# Standard open-chord voicings, low → high string. Strings with `-1`
# in the template's frets list aren't part of the chord.
# E open: E0 A2 D2 G1 B0 e0 (all 6 strings)
# A open: — A0 D2 G2 B2 e0 (skip low E)
# D open: — — D0 G2 B3 e2 (skip low E + A)
# G open: E3 A2 D0 G0 B0 e3 (all 6 strings; common 6-string fingering)
voicings = [
('E', [(0, 0), (1, 2), (2, 2), (3, 1), (4, 0), (5, 0)]),
('A', [(1, 0), (2, 2), (3, 2), (4, 2), (5, 0)]),
('D', [(2, 0), (3, 2), (4, 3), (5, 2)]),
('G', [(0, 3), (1, 2), (2, 0), (3, 0), (4, 0), (5, 3)]),
]
templates = []
chords_out = []
sus = SECONDS_PER_BEAT * 1.6
pattern = list(range(4)) + list(range(4))
for slot, idx in enumerate(pattern):
name, sf = voicings[idx]
# Local-zero-based template id. The driver in `build()` rebases
# these onto the global `templates_all` index before emitting
# the arrangement, so we don't need to pre-offset here — and
# in fact mustn't, since double-offsetting would point at
# template ids past the end of the list.
tmpl_id = idx
if slot < len(voicings):
frets = [-1] * 6
for (s, f) in sf:
frets[s] = f
templates.append({
'name': name,
'displayName': name,
'arp': False,
'fingers': [-1] * 6,
'frets': frets,
})
chord_notes = [chord_note(s, f, sus=sus) for (s, f) in sf]
chords_out.append(chord(t0 + slot * 2 * SECONDS_PER_BEAT, tmpl_id, chord_notes))
return [], (chords_out, templates), 'Open major chords (E A D G — dense)'
def exercise_bends(t0):
"""Half-step and whole-step bends. Bends shift pitch mid-note —
tests whether the single-note pitch tolerance is wide enough."""
notes = []
# Whole-step bend on G string fret 7 (D4 → E4): bn=2.0 semitones.
# Half-step bend on B string fret 8 (G4 → G#4): bn=1.0 semitone.
pattern = [
(3, 7, 2.0), # whole-step on G string
(4, 8, 1.0), # half-step on B string
(3, 7, 2.0),
(4, 8, 1.0),
]
for i, (s, f, bn) in enumerate(pattern):
# 4 bends, half-note each (2 beats), 4 × 2 = 8 beats = 2 bars.
notes.append(note(t0 + i * 2 * SECONDS_PER_BEAT, s, f,
sus=SECONDS_PER_BEAT * 1.6, bn=bn))
notes[-1]['sus'] = round(SECONDS_PER_BEAT * 3, 3)
return notes, [], 'Bends (half-step + whole-step)'
EXERCISES = [
('A. Open strings', exercise_open_strings),
('B. 5th-fret positions', exercise_fretted_positions),
('C. 12th-fret octaves', exercise_octaves),
('D. Sustained notes', exercise_sustained),
('E. Hammer / pull', exercise_hammer_pull),
('F. Power chords', exercise_power_chords),
('G. Open chords', exercise_open_chords),
('H. Bends', exercise_bends),
]
# ── Driver ─────────────────────────────────────────────────────────────
def build(out_dir: Path):
notes_all = []
chords_all = []
templates_all = []
sections = []
beats = []
t = INTRO_BARS * BAR_S
for label, fn in EXERCISES:
sections.append({'name': label, 'number': len(sections) + 1, 'time': round(t, 3)})
result = fn(t)
ns, ch_or_tuple, _desc = result
notes_all.extend(ns)
if isinstance(ch_or_tuple, tuple):
cs, tmpls = ch_or_tuple
# Rebase section-local chord template ids onto the global
# `templates_all` list — see v2 builder for the full
# explanation. Multiple chord exercises in this benchmark
# (power, open) each use ids 0..N locally; without
# offsetting, open-chord events would silently point at
# power-chord templates.
offset = len(templates_all)
for c in cs:
c['id'] = c.get('id', 0) + offset
chords_all.extend(cs)
templates_all.extend(tmpls)
else:
chords_all.extend(ch_or_tuple)
t += EXERCISE_BARS * BAR_S
end_t = t + OUTRO_BARS * BAR_S
# Beats array — one entry per beat, measure markers on downbeats.
bar_count = 0
bt = 0.0
while bt < end_t:
is_downbeat = abs(bt % BAR_S) < 1e-3
if is_downbeat:
bar_count += 1
beats.append({'time': round(bt, 3), 'measure': bar_count})
else:
beats.append({'time': round(bt, 3), 'measure': -1})
bt += SECONDS_PER_BEAT
# Anchors — keep the highway zoom wide enough for everything on
# screen. One anchor at start, then per-exercise re-anchors so the
# camera doesn't drift to the wrong neighbourhood between sections.
anchors = [{'time': 0.0, 'fret': 1, 'width': 12}]
for sec in sections:
anchors.append({'time': sec['time'], 'fret': 1, 'width': 12})
arrangement = {
'name': 'Lead',
'tuning': [0, 0, 0, 0, 0, 0],
'capo': 0,
'notes': sorted(notes_all, key=lambda n: n['t']),
'chords': sorted(chords_all, key=lambda c: c['t']),
'anchors': anchors,
'handshapes': [],
'templates': templates_all,
'beats': beats,
'sections': sections,
}
manifest = {
'title': 'Note Detect Benchmark v1',
'artist': 'Slopsmith',
'album': 'Note Detection Benchmark',
'year': 2026,
'duration': round(end_t, 3),
'arrangements': [
{
'id': 'lead',
'name': 'Lead',
'file': 'arrangements/lead.json',
'tuning': [0, 0, 0, 0, 0, 0],
'capo': 0,
},
],
'stems': [
{'id': 'full', 'file': 'stems/full.ogg', 'default': True},
],
# Non-standard key — picked up by future tooling that wants to
# detect "this is the benchmark, schema v1". The loader ignores it.
'benchmark': {
'id': 'slopsmith-note-detect-benchmark',
'version': 1,
},
}
# ── Write files ──
out_dir = Path(out_dir)
if out_dir.exists():
# Defensive: only blow away a directory that LOOKS like a
# sloppak (has a manifest.yaml at its root, or matches the
# `.sloppak` suffix this builder generates). A user who
# passes e.g. `python build_benchmark.py /tmp` by accident
# otherwise loses `/tmp` to a recursive delete.
if not (out_dir.suffix == '.sloppak'
or (out_dir / 'manifest.yaml').exists()):
raise RuntimeError(
f"refusing to rmtree {out_dir!r}: does not look like a sloppak "
f"(no .sloppak suffix, no manifest.yaml). Pass a path ending in "
f".sloppak or pointing at an existing sloppak directory."
)
shutil.rmtree(out_dir)
out_dir.mkdir(parents=True)
(out_dir / 'arrangements').mkdir()
(out_dir / 'stems').mkdir()
(out_dir / 'manifest.yaml').write_text(
yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True),
encoding='utf-8',
)
(out_dir / 'arrangements' / 'lead.json').write_text(
json.dumps(arrangement, separators=(',', ':')),
encoding='utf-8',
)
# Click track. Write WAV first, then transcode to OGG via ffmpeg —
# the loader expects `stems/full.ogg`.
wav_path = out_dir / 'stems' / 'full.wav'
write_click_wav(wav_path, end_t)
ogg_path = out_dir / 'stems' / 'full.ogg'
subprocess.run(
['ffmpeg', '-y', '-loglevel', 'error',
'-i', str(wav_path),
'-c:a', 'libvorbis', '-q:a', '5',
str(ogg_path)],
check=True,
)
wav_path.unlink() # ogg is canonical; wav was scaffolding
# Distribution README — ships inside the sloppak so other devs can
# follow the exercises without external docs. The loader ignores
# files it doesn't know about, so this travels with the package.
(out_dir / 'BENCHMARK.md').write_text(_benchmark_readme(end_t), encoding='utf-8')
# Zip-archive distribution form alongside the directory. Built with
# the stdlib zipfile module so paths use forward slashes regardless
# of the host OS — PowerShell's Compress-Archive on Windows produces
# backslash paths inside the zip, which the loader (running on
# Linux) then reads as literal filenames instead of directory
# separators and quietly drops every arrangement.
_build_zip(out_dir)
print(f'Built {out_dir}')
print(f' {out_dir}.zip')
print(f' Duration: {end_t:.1f} s')
print(f' Notes: {len(arrangement["notes"])}')
print(f' Chords: {len(arrangement["chords"])}')
print(f' Templates:{len(arrangement["templates"])}')
def _build_zip(src_dir: Path):
"""Pack `src_dir` into `<src_dir>.zip` with forward-slash paths.
Zip-level reproducibility: every entry uses a fixed `date_time` (the
zip spec's earliest legal value, 1980-01-01 00:00:00), a fixed
`external_attr` (rw-r--r--), and an explicit `ZipInfo` so the
archive metadata depends only on contents, not on when the build
ran. JSON / YAML / MD entries are byte-identical across rebuilds.
Caveat: the bundled `stems/full.ogg` is still non-deterministic
across rebuilds because libvorbis writes a random bitstream serial
number to every Ogg page (~1% of the file's bytes are container
framing, not audio). The audio PCM that the detector listens to is
deterministic; only the container headers differ. So a diff of the
tracked sloppak will always show OGG churn after `_build_zip`, but
the chart, manifest, and audible signal are stable. If a future PR
needs full byte-stability, it can either cache a hand-built OGG or
switch the stem to FLAC.
"""
import zipfile
zip_path = src_dir.with_suffix(src_dir.suffix + '.zip')
if zip_path.exists():
zip_path.unlink()
with zipfile.ZipFile(zip_path, 'w', compression=zipfile.ZIP_DEFLATED) as zf:
for p in sorted(src_dir.rglob('*')):
if p.is_file():
# Force POSIX-style arcname so a Windows build still
# emits a Linux-loadable archive.
rel = p.relative_to(src_dir).as_posix()
info = zipfile.ZipInfo(filename=rel, date_time=(1980, 1, 1, 0, 0, 0))
info.compress_type = zipfile.ZIP_DEFLATED
# rw-r--r-- in the upper 16 bits where ZIP stores
# external attrs on POSIX. Avoids "executable" / weird
# permission bits leaking from the host filesystem.
info.external_attr = (0o644 & 0xFFFF) << 16
# Force POSIX (3) for the create-system byte so the
# zip's central-directory metadata doesn't drift when
# the same builder runs on Windows vs Linux. Python's
# default is host-dependent (3 on POSIX, 0 on Windows)
# and was the last source of zip-level non-determinism
# after the date_time + external_attr fixes.
info.create_system = 3
zf.writestr(info, p.read_bytes())
def _benchmark_readme(duration_s):
return f"""# Slopsmith Note Detect Benchmark — v1
A short test piece for tuning Slopsmith's `note_detect` plugin. Eight
exercises, each isolating a specific detection failure mode. Run with
**Detect** enabled, play through, then export the diagnostic JSON
(Settings → Plugins → Note Detection → Download Diagnostic JSON, or
the button on the end-of-session summary modal).
- **Tempo**: {BPM:g} BPM
- **Tuning**: E standard (no capo)
- **Audio**: metronome click track only (downbeat = louder + higher
tone). Play *over* the click — `note_detect` listens to your guitar
signal, not the audio in this file.
- **Duration**: {duration_s:.0f} s
## Sections
| Section | Tests | Watch in the diagnostic |
|---|---|---|
| A. Open strings (low→high→low) | Basic mono detection on each open string | `pure` (mic/audio chain), per-string accuracy |
| B. 5th-fret positions | Fretted-note detection across all 6 strings | per-string variance |
| C. 12th-fret octaves | Higher-frequency detection — YIN's octave-up risk | `sharp` bin spiking |
| D. Sustained notes (4 s) | The `active` held-on-pitch glow | `sharp`/`flat` drift while held |
| E. Hammer-on / pull-off | Transient detection without a fresh pick attack | `pure` (no transient registered) |
| F. Power chords (2-string) | Chord leniency on sparse voicings | `chordPartial` |
| G. Open major chords | Chord leniency on dense voicings (E, A, D, G) | `chordPartial` |
| H. Bends (half- + whole-step) | Single-note pitch tolerance with pitch in motion | `sharp` bin |
## Reporting
Share the JSON (schema `note_detect.diagnostic.v1`). It includes:
- Hit/miss totals split single-note vs chord
- Primary-cause bin per miss (pure / chord-partial / early / late / sharp / flat)
- Per-string hit rate
- Signed timing- and pitch-error percentiles (p10 / median / p90)
- Detection settings snapshot (method, tolerances, leniency)
- Per-judgment event log (capped at 2000 events) with the chart note's
technique flags so each miss can be re-binned by `SUS`/`B`/`H`/etc. offline
- `benchmark_hint`: `{{title, artist, arrangement}}` — filter on these
to bucket reports from different runs of this benchmark.
## Source
Built by `docs/benchmarks/note_detect_v1/build_benchmark.py` in the
slopsmith repo. Tweak the exercise list there and regenerate.
"""
if __name__ == '__main__':
if len(sys.argv) != 2:
print('usage: build_benchmark.py <output-sloppak-dir>', file=sys.stderr)
sys.exit(2)
build(Path(sys.argv[1]))
docs/benchmarks/note_detect_v1/note_detect_benchmark_v1.sloppak
BIN
View File
Binary file not shown.
docs/benchmarks/note_detect_v2/BENCHMARK.md
+37
View File
@@ -0,0 +1,37 @@
# Slopsmith Note Detect Benchmark — v2
A slower-paced companion to v1, focused on what players can actually
land cleanly. Half-note spacing throughout (~1.33 s between events at
90 BPM), with multiple **strumming** sections — single chord voicings
repeated at half-note cadence — to exercise the chord scorer's
consistency across a sequence of strikes.
- **Tempo**: 90 BPM
- **Tuning**: E standard (no capo)
- **Audio**: metronome click track only — play *over* the click.
- **Duration**: 181 s
## Sections
| Section | Tests |
|---|---|
| A. Open strings (slow walk) | Basic mono detection, low → high → low at half-note pacing |
| B. 5th-fret (slow walk) | Fretted-note detection, ascending half-notes |
| C. Sustained notes | Long-hold pitch detection, 4 s each |
| D. E5 power chord strum | Chord scorer on a 2-string voicing, 8 strums |
| E. A5 / E5 alternating | Chord scorer on a voicing change, 8 strums total |
| F. E major strum | 6-string dense voicing, 8 strums |
| G. A major strum | 5-string voicing (skips low E), 8 strums |
| H. D major strum | 4-string voicing (skips low E + A), 8 strums |
No hammer/pull, no bends — those are next on the algorithm-tuning
list and aren't useful as benchmarks until that work lands.
## Reporting
Share the diagnostic JSON (schema `note_detect.diagnostic.v1`).
Filter `benchmark_hint` to bucket v1 vs v2 runs.
## Source
Built by `docs/benchmarks/note_detect_v2/build_benchmark.py`.
docs/benchmarks/note_detect_v2/build_benchmark.py
+512
View File
@@ -0,0 +1,512 @@
"""Builds the Note Detect Benchmark sloppak (v2).
A relaxed-pace test piece tuned for the player's strengths: half-note
spacing throughout, no hammer-on / pull-off section, no bend section,
no fast staccato. Adds explicit strumming sections (single chord
repeated at half-note cadence) so the chord scorer is exercised
across a sequence of strums on the same voicing — closer to how
chords actually appear in real songs than v1's single-stroke
voicings.
Goals vs v1:
- More breathing room between every event (half-notes, ~1.33 s at
90 BPM, instead of v1's quarter notes at ~0.667 s).
- More chord events overall, with strumming patterns.
- Drop the technique sections (HO/PO/bends) — the detector's
technique handling is the next algorithm focus, separate from
measuring "do basic single notes + chords score correctly?"
How to run inside the slopsmith container:
docker cp docs/benchmarks/note_detect_v2/build_benchmark.py \\
slopsmith-web-1:/tmp/build_benchmark_v2.py
docker exec slopsmith-web-1 python /tmp/build_benchmark_v2.py \\
/app/static/sloppak_cache/note_detect_benchmark_v2.sloppak
After regenerating, copy the zip output to the tracked path with the
`.sloppak` (not `.sloppak.zip`) suffix — same gotcha as v1:
cp static/sloppak_cache/note_detect_benchmark_v2.sloppak.zip \\
docs/benchmarks/note_detect_v2/note_detect_benchmark_v2.sloppak
"""
import json
import math
import shutil
import struct
import subprocess
import sys
import wave
from pathlib import Path
import yaml
# ── Benchmark parameters ────────────────────────────────────────────────
BPM = 90.0
SECONDS_PER_BEAT = 60.0 / BPM
BEATS_PER_BAR = 4
BAR_S = BEATS_PER_BAR * SECONDS_PER_BEAT
INTRO_BARS = 2
OUTRO_BARS = 2
EXERCISE_BARS = 8 # v2 uses 8-bar sections (was 6 in v1) for extra breathing room.
# Standard E-tuning open MIDI per string, low → high.
OPEN_MIDI = [40, 45, 50, 55, 59, 64] # E2 A2 D3 G3 B3 E4
SR = 44100
# ── Click-track audio generator ────────────────────────────────────────
def _sine_burst(freq_hz, duration_s, amplitude):
n = int(SR * duration_s)
out = []
fade = max(1, int(0.004 * SR))
for i in range(n):
env = 1.0
if i < fade:
env = i / fade
elif i >= n - fade:
env = (n - 1 - i) / fade
s = math.sin(2 * math.pi * freq_hz * (i / SR)) * amplitude * env
out.append(max(-1.0, min(1.0, s)))
return out
def write_click_wav(path: Path, duration_s: float):
"""Per-beat click track. Downbeats louder + higher pitch."""
total_samples = int(SR * duration_s)
pcm = [0] * total_samples
beat = 0
t = 0.0
while t < duration_s:
is_downbeat = (beat % BEATS_PER_BAR == 0)
freq = 1200 if is_downbeat else 800
amp = 0.6 if is_downbeat else 0.35
burst = _sine_burst(freq, 0.040, amp)
start = int(t * SR)
for i, s in enumerate(burst):
j = start + i
if 0 <= j < total_samples:
pcm[j] = int(max(-1.0, min(1.0, pcm[j] / 32767 + s)) * 32767)
t += SECONDS_PER_BEAT
beat += 1
pcm = [struct.pack('<h', v) for v in pcm]
path.parent.mkdir(parents=True, exist_ok=True)
with wave.open(str(path), 'wb') as w:
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(SR)
w.writeframes(bytes(b''.join(pcm)))
# ── Chart helpers ─────────────────────────────────────────────────────
def note(t, s, f, sus=0.0, **flags):
return {
't': round(t, 3),
's': s,
'f': f,
'sus': round(sus, 3),
'sl': flags.get('sl', -1),
'slu': flags.get('slu', -1),
'bn': flags.get('bn', 0.0),
'ho': flags.get('ho', False),
'po': flags.get('po', False),
'hm': flags.get('hm', False),
'hp': flags.get('hp', False),
'pm': flags.get('pm', False),
'mt': flags.get('mt', False),
'vb': flags.get('vb', False),
'tr': flags.get('tr', False),
'ac': flags.get('ac', False),
'tp': flags.get('tp', False),
}
def chord(t, id_, notes):
return {
't': round(t, 3),
'id': id_,
'hd': False,
'notes': notes,
}
def chord_note(s, f, sus=0.0, **flags):
n = note(0.0, s, f, sus, **flags)
n.pop('t')
return n
# ── Exercises ─────────────────────────────────────────────────────────
# v2 single-note exercises: HALF-NOTE pacing (2 beats / 1.33 s between
# events). That's roughly half the density of v1's quarter-note pacing,
# giving the player time to mute, reset, and re-pluck cleanly.
#
# v2 chord exercises: each chord voicing is STRUMMED multiple times at
# the same half-note cadence. Two reasons:
# 1. Real songs strum chords; single-stroke voicings are an
# artificial test that doesn't exercise the chord scorer's
# consistency across repeated strikes.
# 2. Multiple strums per voicing give the user a forgiving runway —
# if they nail 3 of 4 strums of an E5 power chord, that's still
# mostly hits.
def exercise_open_strings_slow(t0):
"""Open strings, half-note pacing, low → high → low. Wide spacing
lets each string ring out before the next is plucked, so the
detector has clean steady-state pitch to lock onto."""
seq = [0, 1, 2, 3, 4, 5, 5, 4, 3, 2, 1, 0] # 12 strings, half-notes = 24 beats = 6 bars
notes_out = []
for i, s in enumerate(seq):
notes_out.append(note(t0 + i * 2 * SECONDS_PER_BEAT, s, 0,
sus=SECONDS_PER_BEAT * 1.6))
# Final note rings into the 2-bar tail of the section.
notes_out[-1]['sus'] = round(SECONDS_PER_BEAT * 4, 3)
return notes_out, [], 'Open strings (slow walk)'
def exercise_fretted_positions_slow(t0):
"""5th-fret on each string, half-note pacing, ascending. One
direction (no descent) so total runtime fits comfortably in 8 bars
with plenty of tail room."""
seq = [(s, 5) for s in range(6)] # 6 notes × 2 beats = 12 beats = 3 bars
notes_out = []
for i, (s, f) in enumerate(seq):
notes_out.append(note(t0 + i * 2 * SECONDS_PER_BEAT, s, f,
sus=SECONDS_PER_BEAT * 1.6))
notes_out[-1]['sus'] = round(SECONDS_PER_BEAT * 4, 3)
return notes_out, [], 'Fretted positions (slow walk, 5th fret)'
def exercise_sustained(t0):
"""Three 4-second sustained notes (low E, D, high E — spread across
the range). 4 s ring + 1 s gap = 5 s per event × 3 events = 15 s,
comfortably inside an 8-bar (≈ 21.3 s) section."""
sus = 4.0
targets = [(0, 5), (2, 7), (5, 5)]
notes_out = []
for i, (s, f) in enumerate(targets):
notes_out.append(note(t0 + i * (sus + 1.0), s, f, sus=sus))
return notes_out, [], 'Sustained notes (3 holds, 4 s each)'
def exercise_e5_strum(t0):
"""E5 power chord strummed at half-note cadence. 8 strums × 2
beats = 16 beats = 4 bars of strumming, plus 4 bars of tail."""
voicing = [(0, 0), (1, 2)] # low E open + A fret 2 = E5
strums = 8
chords_out = []
sus = SECONDS_PER_BEAT * 1.6 # ring through the next strum, not past it
template = {
'name': 'E5', 'displayName': 'E5', 'arp': False,
'fingers': [-1] * 6,
'frets': [0 if s == 0 else (2 if s == 1 else -1) for s in range(6)],
}
for i in range(strums):
chord_notes = [chord_note(s, f, sus=sus) for (s, f) in voicing]
chords_out.append(chord(t0 + i * 2 * SECONDS_PER_BEAT, 0, chord_notes))
return [], (chords_out, [template]), 'E5 power chord — slow strum (8×)'
def exercise_a5_e5_alternating(t0):
"""A5 / E5 alternating, half-note strums. 8 strums total (4 of
each), gives a "1 5 1 5" feel that's the simplest chord progression
a player can land — minimal hand movement between voicings."""
voicings = [
('A5', [(1, 0), (2, 2)]), # A open + D fret 2 = A5
('E5', [(0, 0), (1, 2)]), # E open + A fret 2 = E5
]
templates = []
for i, (name, sf) in enumerate(voicings):
frets = [-1] * 6
for (s, f) in sf:
frets[s] = f
templates.append({
'name': name, 'displayName': name, 'arp': False,
'fingers': [-1] * 6, 'frets': frets,
})
chords_out = []
sus = SECONDS_PER_BEAT * 1.6
strums = 8
for i in range(strums):
idx = i % 2 # alternate A5 / E5
_, sf = voicings[idx]
chord_notes = [chord_note(s, f, sus=sus) for (s, f) in sf]
chords_out.append(chord(t0 + i * 2 * SECONDS_PER_BEAT, idx, chord_notes))
return [], (chords_out, templates), 'A5 / E5 alternating strums (8×)'
def exercise_e_open_strum(t0):
"""E major open chord, half-note strums. All 6 strings ringing —
the densest voicing in the benchmark, tests the chord scorer's
per-string differentiation on the full set."""
voicing = [(0, 0), (1, 2), (2, 2), (3, 1), (4, 0), (5, 0)]
strums = 8
chords_out = []
sus = SECONDS_PER_BEAT * 1.6
template = {
'name': 'E', 'displayName': 'E', 'arp': False,
'fingers': [-1] * 6,
'frets': [0, 2, 2, 1, 0, 0],
}
for i in range(strums):
chord_notes = [chord_note(s, f, sus=sus) for (s, f) in voicing]
chords_out.append(chord(t0 + i * 2 * SECONDS_PER_BEAT, 0, chord_notes))
return [], (chords_out, [template]), 'E major open chord — slow strum (8×)'
def exercise_a_open_strum(t0):
"""A major open chord, half-note strums. 5 strings (skips low E).
Slightly easier than E for the player (less stretch) and tests
the scorer's behaviour on a missing-low-string voicing."""
voicing = [(1, 0), (2, 2), (3, 2), (4, 2), (5, 0)]
strums = 8
chords_out = []
sus = SECONDS_PER_BEAT * 1.6
template = {
'name': 'A', 'displayName': 'A', 'arp': False,
'fingers': [-1] * 6,
'frets': [-1, 0, 2, 2, 2, 0],
}
for i in range(strums):
chord_notes = [chord_note(s, f, sus=sus) for (s, f) in voicing]
chords_out.append(chord(t0 + i * 2 * SECONDS_PER_BEAT, 0, chord_notes))
return [], (chords_out, [template]), 'A major open chord — slow strum (8×)'
def exercise_d_open_strum(t0):
"""D major open chord, half-note strums. 4 strings (skips low E
and A). Tests the chord scorer on partial voicings — common in
real songs and an easy stretch for new players."""
voicing = [(2, 0), (3, 2), (4, 3), (5, 2)]
strums = 8
chords_out = []
sus = SECONDS_PER_BEAT * 1.6
template = {
'name': 'D', 'displayName': 'D', 'arp': False,
'fingers': [-1] * 6,
'frets': [-1, -1, 0, 2, 3, 2],
}
for i in range(strums):
chord_notes = [chord_note(s, f, sus=sus) for (s, f) in voicing]
chords_out.append(chord(t0 + i * 2 * SECONDS_PER_BEAT, 0, chord_notes))
return [], (chords_out, [template]), 'D major open chord — slow strum (8×)'
EXERCISES = [
('A. Open strings (slow)', exercise_open_strings_slow),
('B. 5th-fret (slow)', exercise_fretted_positions_slow),
('C. Sustained notes', exercise_sustained),
('D. E5 power chord strum', exercise_e5_strum),
('E. A5 / E5 alternating', exercise_a5_e5_alternating),
('F. E major strum', exercise_e_open_strum),
('G. A major strum', exercise_a_open_strum),
('H. D major strum', exercise_d_open_strum),
]
# ── Driver ─────────────────────────────────────────────────────────────
def build(out_dir: Path):
notes_all = []
chords_all = []
templates_all = []
sections = []
beats = []
t = INTRO_BARS * BAR_S
for label, fn in EXERCISES:
sections.append({'name': label, 'number': len(sections) + 1, 'time': round(t, 3)})
result = fn(t)
ns, ch_or_tuple, _desc = result
notes_all.extend(ns)
if isinstance(ch_or_tuple, tuple):
cs, tmpls = ch_or_tuple
# Rebase section-local chord template ids onto the global
# `templates_all` list. Each exercise emits its chords
# with `tmpl_id` numbered from 0 within the exercise; if
# we naively appended both chords and templates without
# offsetting, later sections' chords would silently
# reference earlier sections' templates (e.g. an open
# chord pointing at a power-chord shape). Apply the
# offset to each chord's `id` field before extending the
# global lists.
offset = len(templates_all)
for c in cs:
c['id'] = c.get('id', 0) + offset
chords_all.extend(cs)
templates_all.extend(tmpls)
else:
chords_all.extend(ch_or_tuple)
t += EXERCISE_BARS * BAR_S
end_t = t + OUTRO_BARS * BAR_S
# Beats — measure markers on downbeats.
bar_count = 0
bt = 0.0
while bt < end_t:
is_downbeat = abs(bt % BAR_S) < 1e-3
if is_downbeat:
bar_count += 1
beats.append({'time': round(bt, 3), 'measure': bar_count})
else:
beats.append({'time': round(bt, 3), 'measure': -1})
bt += SECONDS_PER_BEAT
# Anchors — re-anchor on each section so the camera doesn't drift.
anchors = [{'time': 0.0, 'fret': 1, 'width': 12}]
for sec in sections:
anchors.append({'time': sec['time'], 'fret': 1, 'width': 12})
arrangement = {
'name': 'Lead',
'tuning': [0, 0, 0, 0, 0, 0],
'capo': 0,
'notes': sorted(notes_all, key=lambda n: n['t']),
'chords': sorted(chords_all, key=lambda c: c['t']),
'anchors': anchors,
'handshapes': [],
'templates': templates_all,
'beats': beats,
'sections': sections,
}
manifest = {
'title': 'Note Detect Benchmark v2',
'artist': 'Slopsmith',
'album': 'Note Detection Benchmark',
'year': 2026,
'duration': round(end_t, 3),
'arrangements': [
{
'id': 'lead',
'name': 'Lead',
'file': 'arrangements/lead.json',
'tuning': [0, 0, 0, 0, 0, 0],
'capo': 0,
},
],
'stems': [
{'id': 'full', 'file': 'stems/full.ogg', 'default': True},
],
'benchmark': {
'id': 'slopsmith-note-detect-benchmark',
'version': 2,
},
}
# ── Write files ──
out_dir = Path(out_dir)
if out_dir.exists():
# Defensive — see v1 builder. Only rmtree something that looks
# like a sloppak so a typo on the CLI doesn't nuke an unrelated
# directory.
if not (out_dir.suffix == '.sloppak'
or (out_dir / 'manifest.yaml').exists()):
raise RuntimeError(
f"refusing to rmtree {out_dir!r}: does not look like a sloppak "
f"(no .sloppak suffix, no manifest.yaml)."
)
shutil.rmtree(out_dir)
out_dir.mkdir(parents=True)
(out_dir / 'arrangements').mkdir()
(out_dir / 'stems').mkdir()
(out_dir / 'manifest.yaml').write_text(
yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True),
encoding='utf-8',
)
(out_dir / 'arrangements' / 'lead.json').write_text(
json.dumps(arrangement, separators=(',', ':')),
encoding='utf-8',
)
wav_path = out_dir / 'stems' / 'full.wav'
write_click_wav(wav_path, end_t)
ogg_path = out_dir / 'stems' / 'full.ogg'
subprocess.run(
['ffmpeg', '-y', '-loglevel', 'error',
'-i', str(wav_path),
'-c:a', 'libvorbis', '-q:a', '5',
str(ogg_path)],
check=True,
)
wav_path.unlink()
(out_dir / 'BENCHMARK.md').write_text(_benchmark_readme(end_t), encoding='utf-8')
_build_zip(out_dir)
print(f'Built {out_dir}')
print(f' {out_dir}.zip')
print(f' Duration: {end_t:.1f} s')
print(f' Notes: {len(arrangement["notes"])}')
print(f' Chords: {len(arrangement["chords"])}')
print(f' Templates:{len(arrangement["templates"])}')
def _build_zip(src_dir: Path):
"""Pack with fixed dates / attrs for zip-metadata reproducibility.
See v1 builder docstring for full caveats (OGG framing has its own
non-determinism we don't try to fix here)."""
import zipfile
zip_path = src_dir.with_suffix(src_dir.suffix + '.zip')
if zip_path.exists():
zip_path.unlink()
with zipfile.ZipFile(zip_path, 'w', compression=zipfile.ZIP_DEFLATED) as zf:
for p in sorted(src_dir.rglob('*')):
if p.is_file():
rel = p.relative_to(src_dir).as_posix()
info = zipfile.ZipInfo(filename=rel, date_time=(1980, 1, 1, 0, 0, 0))
info.compress_type = zipfile.ZIP_DEFLATED
info.external_attr = (0o644 & 0xFFFF) << 16
info.create_system = 3 # POSIX — see v1 builder for why
zf.writestr(info, p.read_bytes())
def _benchmark_readme(duration_s):
return f"""# Slopsmith Note Detect Benchmark — v2
A slower-paced companion to v1, focused on what players can actually
land cleanly. Half-note spacing throughout (~1.33 s between events at
90 BPM), with multiple **strumming** sections — single chord voicings
repeated at half-note cadence — to exercise the chord scorer's
consistency across a sequence of strikes.
- **Tempo**: {BPM:g} BPM
- **Tuning**: E standard (no capo)
- **Audio**: metronome click track only — play *over* the click.
- **Duration**: {duration_s:.0f} s
## Sections
| Section | Tests |
|---|---|
| A. Open strings (slow walk) | Basic mono detection, low → high → low at half-note pacing |
| B. 5th-fret (slow walk) | Fretted-note detection, ascending half-notes |
| C. Sustained notes | Long-hold pitch detection, 4 s each |
| D. E5 power chord strum | Chord scorer on a 2-string voicing, 8 strums |
| E. A5 / E5 alternating | Chord scorer on a voicing change, 8 strums total |
| F. E major strum | 6-string dense voicing, 8 strums |
| G. A major strum | 5-string voicing (skips low E), 8 strums |
| H. D major strum | 4-string voicing (skips low E + A), 8 strums |
No hammer/pull, no bends — those are next on the algorithm-tuning
list and aren't useful as benchmarks until that work lands.
## Reporting
Share the diagnostic JSON (schema `note_detect.diagnostic.v1`).
Filter `benchmark_hint` to bucket v1 vs v2 runs.
## Source
Built by `docs/benchmarks/note_detect_v2/build_benchmark.py`.
"""
# ── CLI ───────────────────────────────────────────────────────────────
if __name__ == '__main__':
out = Path(sys.argv[1]) if len(sys.argv) > 1 else Path('./note_detect_benchmark_v2.sloppak')
build(out)
docs/benchmarks/note_detect_v2/note_detect_benchmark_v2.sloppak
BIN
View File
Binary file not shown.
docs/capability-domains.md
+290
View File
@@ -0,0 +1,290 @@
# Capability Domains
Capability domains are Slopsmith-wide coordination surfaces for core, bundled first-party plugins, external plugins, and future adapters. Plugins declare the runtime surfaces they use in `plugin.json`; core declares and owns host workflows directly in the runtime. These declarations let diagnostics and support tools reason about behavior without relying on private globals.
## Standards
Migrated plugins should declare standards explicitly:
```json
{
"standards": ["capability-pipelines.v1", "plugin-runtime-idempotent.v1"]
}
```
Only declare `plugin-runtime-idempotent.v1` when repeated script hydration cannot duplicate wrappers, listeners, timers, DOM roots, diagnostics contributors, jobs, media nodes, or capability participants.
## UI Contributions
Legacy `nav`, `screen`, and `settings` fields still work through the existing plugin loader. PR1 keeps UI capability domains out of the runtime graph, so migrated plugins should not treat `ui.navigation`, `ui.plugin-screens`, or `settings` as active capability contracts yet. Their candidate manifest shape is reserved for a future UI-host PR:
```json
{
"ui": {
"ui.navigation": [{ "id": "my-plugin-nav", "region": "plugins", "label": "My Plugin" }],
"ui.plugin-screens": [{ "id": "my-plugin-screen", "region": "plugin-screens", "label": "My Plugin" }],
"settings": [{ "id": "my-plugin-settings", "region": "plugin-settings", "label": "My Plugin" }]
}
}
```
Core continues to load legacy UI fields normally. It does not emit PR1 compatibility shim entries for UI placement or visualization `type`; the PR that promotes those domains will own their shim accounting and tests.
## Runtime Domains
Declare non-UI runtime surfaces under `domains` or `runtime_domains`:
```json
{
"domains": {
"library": { "role": "provider" }
}
}
```
If a plugin still uses `routes`, the backend loader continues to load `routes.py` normally. PR1 does not expose that legacy surface as `backend.routes`; the backend route domain is deferred until a future PR has a concrete route/provider workflow and privilege review.
Plugins that call `context["register_library_provider"](...)` are attributed to the loading plugin id in `/api/library/providers` as `owner_plugin_id`. The browser library capability module at [static/capabilities/library.js](../static/capabilities/library.js) owns the `library` domain as a `provider-coordinator`: it refreshes `/api/library/providers`, registers the built-in `local` provider as `core.library.local`, and registers plugin-backed providers under their `owner_plugin_id` when one is known. Provider manifests should still declare the `library` capability so diagnostics and the bundled inspector can show intended relationships before the backend route code runs.
Route-only external plugins that participate in library workflows without registering a browsable provider should declare requester/observer intent instead of provider ownership when they adopt this contract in their own repositories. This PR documents the generic shape only: such plugins use `library` requester/observer `requests` and `observes` declarations and do not appear as providers, owners, or separate `backend.routes` domains.
```json
{
"capabilities": {
"library": {
"roles": ["provider"],
"operations": ["query-page", "query-artists", "query-stats", "tuning-names", "get-art", "sync-song"],
"description": "Adds a browsable library source and optional song sync.",
"mode": "active",
"compatibility": "none",
"safety": "safe"
}
}
}
```
The frontend exposes the current source list through `window.slopsmith.capabilities.command('library', 'list-providers')`. Public owner commands (`list-providers`, `refresh-providers`, `get-current`, `select-provider`, `sync-song`, `inspect`) are distinct from provider operations (`query-page`, `query-artists`, `query-stats`, `tuning-names`, `get-art`, `sync-song`). The app-owned handler delegates to the existing provider registry and source selector, so plugins should not scrape the `#lib-provider` dropdown.
Capability declarations may include a short `description`. The bundled Capability Inspector shows that text on expanded domain owner cards; when it is omitted, the inspector falls back to a compact generated owner summary.
## Audio Graph/Session Domains
The audio graph/session slice promotes four player-audio domains into the runtime graph: `audio-mix`, `audio-input`, `audio-monitoring`, and `stems`. The browser module at [static/capabilities/audio-session.js](../static/capabilities/audio-session.js) owns the active session boundary, contributes diagnostics under `slopsmith.audio_session.diagnostics.v1`, and records compatibility bridge hits for legacy audio surfaces.
`audio-mix`, `audio-input`, and `audio-monitoring` are core-owned provider-coordinator domains. They expose bounded inspect/register/start/stop style commands, redaction-safe diagnostics, and bridge accounting for legacy fader, analyser, input, and monitoring handshakes.
For `audio-mix`, native fader providers register mix participants with stable `participantId`, `kind`, `sourceMode`, optional `logicalFaderKey`, and `fader` metadata. The public command surface is `inspect`, `list-faders`, `get-fader-value`, `set-fader-value`, `inspect-route`, `inspect-analyser`, `register-participant`, and `unregister-participant`; provider operations are `fader.get-value`, `fader.set-value`, `route.get-current`, and `analyser.get-summary`. Providers own persistence for plugin faders and must return committed values from set operations so the player mixer can display the value that actually applied.
Legacy `window.slopsmith.audio.registerFader(...)` remains supported as an audio-mix compatibility bridge. The bridge registers a compatibility-backed participant, wraps legacy `getValue`/`setValue` callbacks as provider operations, and preserves `window.slopsmith.audio.getFaders()` for external callers. If a native participant and a legacy fader share the same logical fader key, the native participant owns the visible control; the legacy participant is retained for diagnostics with `supersededBy` and an `overshadowed` bridge hit. Removal gates for the bridge are: native providers cover bundled mixer integrations, diagnostics show no unexpected legacy hits in normal playback, and repeated plugin hydration does not create duplicate faders.
Audio-mix diagnostics live under `slopsmith.audio_session.diagnostics.v1`. The `audio-mix` domain snapshot includes session state, participants, visible fader summaries, required participant-kind coverage, route summary, analyser summary, bridge hits, and bounded recent outcomes. Fader outcomes include operation name, participant id, fader id, status such as `committed`, `normalized`, `unavailable`, or `timeout`, and a bounded reason. Diagnostics must not include raw audio buffers, FFT arrays, device labels, stable hardware identifiers, secrets, or unredacted local paths; route/analyser payloads are summaries only.
For `audio-input`, native providers register source summaries with `sourceId`, `providerId`, `logicalSourceKey`, `kind`, redaction-safe label/pseudonym, `availability`, `channelSummary`, `sourceMode`, and provider operations. The public command surface is `inspect`, `list-sources`, `register-source`, `unregister-source`, `select-source`, `open-source`, and `close-source`; provider operations are `source.enumerate`, `source.describe`, `source.open`, and `source.close`. `inspect`, `list-sources`, and `select-source` are prompt-free and never open live input or call enumeration. `source.enumerate` runs only when explicitly requested by provider/user discovery. `open-source` is the permission boundary: it routes to `source.open`, attributes the requester, checks the selected source and requested channel shape, and records `handled`, `denied`, `degraded`, `failed`, `no-owner`, `no-handler`, `unsupported-command`, or `incompatible-version` outcomes.
Selected input is persisted by `logicalSourceKey` when browser storage is available. If storage is unavailable, the in-memory selection remains usable for the current session and diagnostics report the storage status. Start/stop/song switches preserve selected input independently of playback transport while clearing live open sessions. Compatible requesters share one open session per logical source and channel shape; requester references are released via `close-source`, and the provider receives `source.close` only after the last requester releases.
Compatibility-backed input sources should record `sourceMode: "compatibility"` plus `compatibilitySource` and, when applicable, an `audio-input.legacy-source` bridge hit. If a native provider and a compatibility-backed source share the same logical source key, the native source owns the visible source list and the compatibility source is retained in diagnostics with `supersededBy`. Removal gates for input bridges are: native providers cover bundled source discovery/open flows, diagnostics show no unexpected compatibility hits in normal playback, denied/unavailable/failure outcomes are distinguishable, repeated hydration does not create duplicate sources, and support snapshots contain no raw device labels, hardware ids, paths, secrets, live handles, buffers, samples, or waveform data.
For `audio-monitoring`, native providers register monitoring summaries with `providerId`, `logicalMonitoringKey`, redaction-safe label/pseudonym, `availability`, `sourceMode`, provider operations, `directMonitor`, and `latencySummary`. The public command surface is `inspect`, `list-providers`, `register-provider`, `unregister-provider`, `select-provider`, `start`, `stop`, and `set-direct-monitor`; provider operations are `monitoring.start`, `monitoring.stop`, `monitoring.status`, and `monitoring.set-direct-monitor`. `inspect`, `list-providers`, `select-provider`, and `monitoring.status` are prompt-free and must not open audio input or start monitoring.
Fresh monitoring start is a user-action boundary. A requester that calls `start` without `authorization: "user-action"` receives `user-action-required` unless it can attach to an already-active compatible monitoring session. Start dispatch opens the selected audio-input source through the `audio-input` domain, checks the requested channel shape, and then calls the provider's `monitoring.start` with a redaction-safe `sourceRef`, requester id, required channel shape, direct-monitor preference, and optional requester requirement. Outcomes distinguish `handled`, `degraded`, `denied`, `unavailable`, `failed`, `no-owner`, `no-handler`, `unsupported-command`, `incompatible`, `incompatible-version`, `provider-selection-required`, and `user-action-required`.
Monitoring sessions are keyed by provider, selected source, required channel shape, and direct-monitor policy. Compatible requesters share an active session without re-calling `monitoring.start`; each requester later calls `stop`, and the provider receives `monitoring.stop` only after the final requester releases it. Song switches and playback stops preserve active monitoring sessions for the current browser runtime, while page reload restores only the selected provider and direct-monitor preference; live monitoring stays stopped until a new explicit start.
Direct-monitor state is user-authoritative. `set-direct-monitor` updates the user's/default preference and applies provider control to active sessions only when the provider supports it. Requester `directMonitorRequirement` values are advisory constraints: when they conflict with the user's preference or provider support, the requester/session is marked degraded or unsupported, but the stored user/default preference is not changed.
Compatibility-backed monitoring providers should record `sourceMode: "compatibility"` plus `compatibilitySource` (which becomes the bridge id, defaulting to `audio-monitoring.legacy-provider` when unset) and, when applicable, the `audio-monitoring.audio-barrier` startup-barrier bridge hit. If a native provider and compatibility-backed provider share a logical monitoring key, the native provider owns the visible provider list and the compatibility provider is retained in diagnostics with `supersededBy`. Removal gates for monitoring bridges are: native providers cover bundled start/stop/status/direct-monitor flows, normal playback shows no unexpected legacy hits, background requesters cannot silently start live monitoring, repeated hydration does not duplicate providers or sessions, and support snapshots contain no raw device labels, hardware ids, paths, secrets, live handles, buffers, samples, waveform data, recordings, or provider-private payloads.
`stems` is different: `core.audio.session` is a coordinator, not the semantic owner of stem playback. The Stems plugin, or another active stem provider, remains the provider/owner of actual stem state, mute/restore mechanics, and per-song availability. The session coordinator records the active provider via `registerStemOwner(...)`, brokers claim/override/orphan diagnostics, and returns `no-owner` when no stem provider is available.
New bundled audio code should use the session host or native capability dispatch instead of adding new globals, private stem-state reads, direct analyser ownership, or plugin-specific handshakes. Existing legacy paths remain supported through named compatibility bridges until their migration notes and removal gates are satisfied.
## Audio Effects Domain
The audio-effects slice promotes `audio-effects` as a core-owned provider-coordinator domain implemented by [static/capabilities/audio-effects.js](../static/capabilities/audio-effects.js). The host owns provider selection, compatible executor selection, route state, fallback accounting, redaction-safe diagnostics, and the constrained chain-plan schema. Providers do not call executors. Provider code proposes plans and, for execution requests, returns a provider-private trusted asset map to the host; the host immediately hands that private request to a compatible executor such as trusted Desktop native audio or NAM Tone's browser/WASM executor.
The public command surface is `inspect`, `list-providers`, `list-executors`, `register-provider`, `unregister-provider`, `register-executor`, `unregister-executor`, `select-chain`, `resolve-plan`, `load-plan`, `inspect-route`, `list-mappings`, `upsert-mapping`, `delete-mapping`, `activate-mapping`, `clear-active-mapping`, `bypass`, `restore`, `fallback`, `activate-segment`, `set-stage-bypass`, `set-stage-parameter`, and `record-bridge-hit`. Provider operations are `chain.resolve`, `chain.inspect`, `mapping.list`, `mapping.upsert`, `mapping.delete`, `mapping.activate`, `mapping.clear-active`, `segment.activate`, `stage.set-bypass`, `stage.set-parameter`, `route.bypass`, and `route.restore`; executor operations are `loadChainPlan`, `activateSegment`, `setStageBypass`, and `setStageParameter`. Fresh chain selection and route bypass/restore require `authorization: "user-action"` or `authorization: "restore-selection"`; physical loading through `load-plan` requires `authorization: "user-action"`, `authorization: "restore-selection"`, or `authorization: "playback-session"`. Background requesters may inspect the current route and resolve an already selected compatible provider.
Core also owns the durable public mapping index at `/api/audio-effects/mappings`. A mapping answers "for this song/tone, this provider has an addressable effect plan"; it does not contain the provider's preset or chain data. Rows are keyed by `song_key + tone_key + provider_id`, carry an opaque `provider_ref`, and may be marked as the active mapping for that song/tone. Providers CRUD their own rows through the audio-effects host and resolve `provider_ref` inside their own storage when `chain.resolve` runs. This lets NAM Tone and Rig Builder coexist for the same song/tone while core owns arbitration and fallback order. `song_key` should be the playback domain's redaction-safe settings key when available; `filename` is optional legacy/debug context for migration and display.
Providers register stable `providerId`, `pluginId`, `routeKey`, priority, availability, source mode, operations, and operation handlers. Executors register stable `executorId`, `pluginId`, `routeKey`, priority, availability, source mode, supported provider ids, supported stage kinds, optional maximum stage count, operations, and handlers. The host chooses the highest-priority enabled provider for a route unless the caller requests a specific provider, then chooses the highest-priority compatible executor for that provider and resolved plan. Compatibility means both provider-compatible and plan-compatible: a browser/WASM NAM executor can advertise `providerIds: ["nam-tone"]`, `supportedKinds: ["nam", "ir"]`, and `maxStages: 2`, so it will not be asked to execute a Rig Builder VST/full-chain plan. If a selected provider has no compatible executor and the caller supplies a fallback provider, the host may fall back to that provider; if the caller explicitly requested the original provider, the host reports `unavailable` instead of silently changing providers. The initial default route is `desktop-main`, matching the desktop native executor path planned for full-chain NAM/IR/VST playback while still allowing browser executors for non-Desktop runtimes.
`chain.resolve` returns schema `slopsmith.audio_effects.chain_plan.v1`. A valid plan includes `planId`, `routeKey`, `providerId`, `stages`, optional `segments`, and optional redaction-safe summaries. Each stage exposes only stable opaque `stageId`, `kind` (`nam`, `ir`, `vst`, `utility`, or `bypass`), `role` (`pre-pedal`, `amp`, `cab`, `rack`, `master-pre`, etc.), opaque `assetRef`, optional opaque `stateRef`, bypass state, gain summary, and safe summary metadata. Raw file paths, URLs, model filenames, IR filenames, VST state blobs, native preset JSON, callbacks, handles, DOM nodes, audio buffers, samples, and waveform data are rejected or omitted.
Diagnostics live under `slopsmith.audio_effects.diagnostics.v1`. The snapshot includes provider summaries, executor summaries, route summaries, bridge hits, bounded recent outcomes, limits, and redaction notes. It intentionally omits full chain plans, stage asset references, provider-private mapping payloads, raw filenames, and song keys; diagnostics should explain which provider/executor/route failed without leaking local library structure or licensed asset names. Legacy NAM Tone/Rig Builder fetch interception, direct Desktop `loadPreset` calls, legacy tone controls, old `nam_tone.db` `tone_mappings` access, and MIDI/external effect handoffs are attributed through `audio-effects.legacy-nam-routing`, `audio-effects.legacy-native-load`, `audio-effects.legacy-tone-controls`, `audio-effects.legacy-tone-db`, and `audio-effects.legacy-midi-amp` bridge records while providers migrate.
## Playback Control Plane
The playback slice promotes `playback` as a core-owned command domain implemented by [static/capabilities/playback.js](../static/capabilities/playback.js). The public command surface is `inspect`, `start`, `pause`, `resume`, `stop`, `seek`, `set-loop`, `clear-loop`, `register-requester`, and `register-observer`. The domain emits `playback:*` lifecycle events for requests, loading, ready/start/pause/resume/seek/stop/end, route transitions, loop changes, failures/degraded states, superseded sessions, and compatibility bridge hits.
`static/app.js` remains the transport data plane. It registers a private playback adapter that can start songs, pause/resume/stop, seek, and manage loops, but the capability snapshot never exposes the `<audio>` element, JUCE player object, raw audio buffers, native route handles, samples, waveforms, recordings, local file paths, or URL payloads. Exported diagnostics use pseudonymous `target-*` ids for arrangement-scoped identity and hashed `settings-*` keys for per-song plugin settings; the local Capability Inspector may show visible title, artist, and arrangement labels for the active song.
Legacy playback surfaces remain supported during migration and are attributed through bridges such as `playback.window-play-song`, `playback.song-events`, `playback.window-slopsmith-transport`, `playback.loop-api`, and native route handoff records. Fresh audible `start` commands require `authorization: "user-action"`; background requesters may inspect or control an existing session, but user-priority pause/stop decisions block lower-priority automation until a user action resumes or starts a new session.
## Progression Domain
The progression slice (spec 010) promotes `progression` as a core-owned command domain implemented by [static/v3/progression-core.js](../static/v3/progression-core.js). Core owns the player's mastery rank (onboarding calibration + instrument-path levels), the challenge/quest engine, the Decibels wallet, and the cosmetics shop; all definitions are bundled content under `data/progression/` so new paths, levels, challenges, quests, and shop items are JSON edits, never code.
The public command surface is `inspect`, `record-event`, `list-shop`, `buy-item`, and `equip-item`. `record-event` accepts only whitelisted externally-postable event types (`minigame_run` in v1); `song_completed` is server-derived inside `/api/stats` so scored-session authority stays in one place and is denied at this surface. `buy-item` and `equip-item` require `authorization: "user-action"`. Backend plugins use the symmetric plugin-context hook `record_progression_event` (the minigames hub reports runs through it), which trusts backend code and skips the HTTP whitelist.
The domain emits `challenge-completed`, `quest-completed`, `path-level-up`, `rank-changed`, `db-changed`, `calibration-completed`, and `cosmetic-equipped` on the capability surface, mirrored as `progression:*` events on `window.slopsmith` for non-capability consumers. Diagnostics live under `slopsmith.progression.diag.v1` and contain content-load warnings, rank/path-level/quest counts, and wallet totals only — no song filenames or display names.
Decibels are earned exclusively by playing (songs, minigame runs, quest rewards); there is no real-money acquisition path and none may be added. The wallet tracks spend separately from the monotonic lifetime-earned total, so per-source XP resets and `db_earned` goals stay correct. A deferred release slice adds a `contributor` role so plugins can ship their own challenge/quest content (e.g. a drums plugin contributing drums challenges); content stays core-bundled until then.
## Visualization Domain
The visualization slice (cap:6) promotes `visualization` as a core-owned provider-coordinator implemented by [static/capabilities/visualization.js](../static/capabilities/visualization.js). Viz plugins are providers of the highway renderer surface; the core picker/auto-match machinery in `static/app.js` stays the selection workflow and attributes every renderer change into the domain.
The public command surface is `inspect`, `list-providers`, `select-renderer`, and `clear-renderer`. Selection delegates to the existing picker (`setViz`) so localStorage persistence, WebGL2 gating, and fallback semantics have exactly one implementation. The domain emits `providers-refreshed`, `renderer-changed` (with a `source` of `auto-match`, `user-select`, `fallback`, or `command:<requester>`), `renderer-ready`, and `renderer-failed`.
Provider discovery is still the legacy surface — `type: "visualization"` manifests populate the picker and `window.slopsmithViz_*` factory globals carry the renderer contract — and both are registered as compatibility shims (`visualization:type-visualization-manifest`, `visualization:window.slopsmithViz_*`) with hit accounting, so the Inspector shows exactly how much of the domain still rides the bridge. Plugins migrate by declaring a `visualization` provider capability in their manifests; the renderer factory contract (`init`/`draw`/`resize`/`destroy`, `contextType`, `matchesArrangement`) is unchanged.
**Per-instance provider settings (#849).** A provider may declare a `settings` array on its `visualization` capability — generic control descriptors (`{ key, label, type: "toggle" | "range" | "select", default, min/max/step, options }`) the capability-pipelines schema validates on *any* domain (the field lives on the shared `capabilityDeclaration`, not a visualization-only spot — see `docs/plugin-manifest.schema.json`). Descriptors flow through the **generic participant model**: the backend validates them for `/api/plugins` (`plugins/__init__.py`), `static/capabilities.js` normalizes + preserves them on the registered participant (so generic `inspect('visualization')` carries them), and the visualization owner reads them back from the participant by id — no app.js/picker side channel. They surface in the `list-providers` snapshot (each provider's `settings`, deep-frozen) plus a `provider_policy.hasSettings` flag in diagnostics, so a consuming host — splitscreen's per-panel control popover — can render the controls generically without per-plugin hardcoding. The visualization domain's **apply contract**: a provider that declares `settings` MUST implement `applySetting(key, value)` on its renderer instance (the host calls it on the specific per-panel instance, which is inherently per-panel — no canvas→panel resolution, no shared global keys); `getSetting(key)` is optional (the host falls back to the declared `default`). The host owns persistence. **This core slice lands the declarative surface + participant plumbing only** — no bundled provider declares `settings` yet (`highway_3d` still ships the legacy `factory.panelControls` static). The `highway_3d` migration and the splitscreen generic consumer are the remaining #849 follow-ups.
Diagnostics live under `slopsmith.visualization_capability.v1` and contain provider ids/labels/context types, the active renderer id and its selection source, the last auto-match outcome (resolved id + whether any predicate claimed the song), and the last failure (provider id + reason) — never song filenames, titles, or arrangement names. Per-panel selection (splitscreen #90) and per-panel provider settings (#849) are tracked follow-ups; the domain currently models the primary highway surface.
## Note-Detection Domain
The note-detection slice (spec 009, issues #727/#728) promotes `note-detection` as a core-owned provider-coordinator implemented by [static/capabilities/note-detection.js](../static/capabilities/note-detection.js). Doctrine per [specs/009-note-detection-domain/spec.md](../specs/009-note-detection-domain/spec.md): the domain exposes detection PRIMITIVES through requester-owned, context-scoped bindings — a monophonic pitch estimate and a polyphonic "is this note set ringing now?" verification — and consumers own all judgment semantics (hit windows, streaks, accuracy, tiers). Hit/miss/verdict results flow through the domain as observability events, never as domain-owned scoring.
The public command surface is `inspect`, `register-provider`, `unregister-provider`, `open-binding`, `close-binding`, `set-target`, and `clear-target`. Providers declare a kind — `midi` (a digital instrument producing exact verdicts, e.g. the keys highway's Web-MIDI input), `engine` (the desktop JUCE verifier), or `js` (the browser harmonic-comb / YIN fallback) — and the primitives they serve (`pitch.estimate`, `verify.target`). Each binding carries its requester's own redacted context summary (arrangement kind, string count, capo, MIDI range), independent of whatever song the host highway has loaded; concurrent bindings never perturb one another (spec 009 FR-003). With no provider registered, `open-binding` reports `unavailable` — consumers degrade, never block.
The legacy chart-coupled surface — `highway.setNoteStateProvider(fn)`, the single-global-detector path spec 009 retires — keeps working unchanged and is wrapped for compatibility-shim hit accounting (`note-detection:highway.setNoteStateProvider`). Migrating the chart `note_detect` consumer, Step Mode verify, minigames YIN scoring, and the `setVerifyTarget` bridge onto real bindings — and wiring per-binding tuning contexts into the engine verifier — is the remainder of the spec-009 slice and lands behind the Spec 003 migration gate.
Diagnostics live under `slopsmith.note_detection_capability.v1` and contain provider ids/labels/kinds, binding summaries (requester, provider, redacted context, target size), availability, and the last bounded outcome (event, binding, provider, MIDI number, hit flag) — never raw audio buffers, sample data, device labels, or song identity.
## Capability Roles
Use capability declarations for provider/requester/observer relationships:
```json
{
"capabilities": {
"library": {
"roles": ["requester", "observer"],
"requests": ["list-providers", "get-current", "inspect"],
"observes": ["providers-refreshed", "source-changed"],
"mode": "active",
"compatibility": "none"
}
}
}
```
Future app-level workflows can then express intent through capability domains instead of hard-coding plugin-private implementation details.
Core registers manifest capability declarations from `/api/plugins` before plugin scripts hydrate. Runtime owners can then re-register the same participant with command handlers, event handlers, and current availability state. The merged participant view is visible through `window.slopsmith.capabilities.snapshotDiagnostics()` and `getDiagnostics()`.
Core domains include review metadata in diagnostics:
- `active`: wired to current Slopsmith behavior and expected to work as an integration point.
- `diagnostic`: support/inspection-only runtime surfaces.
PR1 includes only the delivered domains listed in [capability-roadmap.md](capability-roadmap.md): `pipeline`, `diagnostics`, and `library`. The follow-up audio graph/session slice promotes `audio-mix`, `audio-input`, `audio-monitoring`, and a coordinated `stems` surface. The playback slice promotes `playback` as an active transport control plane. The audio-effects slice promotes provider-selected effect-chain planning while leaving physical processor loading to compatible executors such as trusted Desktop native audio or a browser/WASM executor. The visualization slice promotes `visualization` as the highway renderer provider-coordinator, and the note-detection slice (spec 009) promotes `note-detection` as the detection-binding control plane. Backend routes, app UI, settings, and other hardware-facing domains remain documented in the roadmap and safety matrix until their own host workflow/provider slice exists.
Capability metadata is versioned by the `capability-pipelines.v1` standard. Invalid roles, commands, operations, requests, observes, emits, events, owner kinds, compatibility modes, ownership policies, safety classes, or version fields are excluded from the capability graph and surfaced through `capability_validation_warnings`; legacy plugin fields continue to load through their existing app paths. Plugins that declare a future `capability-pipelines` version are reported through `capability_unsupported_versions` and their runtime handlers are marked incompatible.
`diagnostics` and `pipeline` are adjacent support domains. `diagnostics` is the read-only snapshot/export surface: `snapshot` returns the redaction-safe state used by support bundles and the Capability Inspector. `pipeline` is the graph operations surface: `inspect`, `validate`, and `participant.set-enabled` operate on the capability graph itself and emit graph lifecycle events such as `resolved`, `runtime.validated`, and `participant.state-changed`.
Requesters should use the public claim/dispatch/release flow instead of mutating another plugin's globals:
```js
const api = window.slopsmith.capabilities;
const releaseClaim = api.claim({ capability: 'example.plugin-domain', claimId: 'example.automation-active', requester: 'example_requester' });
await api.dispatch({
capability: 'example.plugin-domain',
command: 'apply',
source: 'example_requester',
claim: { claimId: 'example.automation-active' },
args: { target: { kind: 'example-target' } },
});
releaseClaim();
```
The claim owner is inferred from the active owner participant for the capability, so requesters should identify themselves with `requester` or `source` instead of passing an `owner` field. `release` only needs the `claimId` and, when useful for disambiguation, the `capability`.
Manual user actions win over matching automation claims. When an owner records a user override for the same capability target, the registry reports the command as `overridden` and skips re-applying automation for that target. Owners keep restore snapshots for their own surfaces so requesters do not need to read private state.
When a requester disappears, the registry releases its active claims and clears restore snapshot references. When an owner or live handler disappears, matching claims become `orphaned` and non-dispatchable until the user or owning plugin resolves them. Runtime enable/disable state is lifecycle metadata, not a manual override.
## Owner Kinds And Dispatch Outcomes
Owner participants use a `kind` that describes how the domain is coordinated:
- `command`: one active owner handles public commands.
- `provider-coordinator`: one owner coordinates provider participants through provider operations.
- `event`: the owner primarily emits or coordinates events.
- `diagnostic`: read-only support and inspector surfaces.
- `privileged`: command execution needs an explicit enforcement plan before shipping.
Legacy `ownership` remains accepted in manifests for compatibility and diagnostics, but new domains should prefer `kind` plus participant roles. Ownership is derived for core owners where possible: `provider-coordinator` behaves like a multi-provider domain, diagnostics are diagnostic-only, privileged owners are privileged, and command/event owners are exclusive by default.
The compatibility ownership vocabulary remains:
- `exclusive-owner`: at most one active owner; duplicate owners produce a conflict and dispatch degrades.
- `multi-provider`: multiple providers may participate, but ordering must be deterministic through fixed priority or `before`/`after` constraints.
- `observer-only`: participants listen for events and should not handle commands.
- `requester-only`: participants request commands from another owner.
- `privileged`: command execution needs an explicit enforcement plan before shipping.
- `diagnostic-only`: read-only support and inspector surfaces.
Dispatch results use explicit outcomes: `handled`, `transformed`, `denied`, `failed`, `degraded`, `short-circuited`, `overridden`, `no-owner`, `no-handler`, `no-target`, `unsupported-command`, `incompatible`, `incompatible-version`, `unavailable`, `provider-selection-required`, `user-action-required`, `stale`, `cancelled`, and `stopped`. No-owner, no-handler, no-target, unsupported-command, incompatible, incompatible-version, provider-selection-required, user-action-required, stale, and cancelled decisions are recorded in diagnostics so support bundles explain why nothing happened.
## Deferred Core Adapters
UI placement and settings contributions are real Slopsmith surfaces, but they are not PR1 capability contracts (visualization is active as of the cap:6 slice; note-detection as of the spec-009 slice). Audio mixer/session domains are active as of the audio graph/session slice, playback is active as of the playback control-plane slice, and audio-effects is active as a provider/route/chain-plan coordinator; plugins should keep using current documented APIs for remaining areas until the corresponding domain PR ships the host workflow, command/event contract, compatibility shims, diagnostics fields, and tests.
The library provider workflow is the PR1 core adapter and is implemented natively as the `library` capability module. Provider refresh, selection, and sync run through `library` owner commands; backend provider registration remains the way providers enter the library registry, and the browser module turns that registry into provider participants. The app event bus continues to dispatch local `window.slopsmith` events for legacy listeners; playback now mirrors song transport, route, seek, and loop lifecycle into `playback`, and visualization attributes renderer selection/failure, while navigation, note, and route-only surfaces remain outside capability domains until their own slices land.
The direct `window.highway` object remains the renderer data plane. Per-frame reads such as notes, chords, beats, and renderer hooks should not be moved behind asynchronous capability commands until there is a dedicated chart/render facade.
## First-Party Management Plugins
Large management surfaces should prefer plugin-owned UI over crowding normal Settings. First-party management plugins can contribute screens and settings panels while core keeps shared services and diagnostics contracts centralized.
The bundled Capability Inspector plugin is the support surface for the current graph. It reads `window.slopsmith.capabilities.snapshotDiagnostics()`, filters by domain, and summarizes manifest participants, runtime participants, conflicts, unsupported versions, safety classes, expected legacy event surfaces, and compatibility shim hits without rendering raw runtime objects. Domains are grouped in review order: application/library, player/audio runtime, plugin-defined surfaces, then capability runtime. In the all-domains view, each domain starts collapsed with a domain-specific icon plus compact summary badges for participant-lane count, endpoint count, observed links, shimmed links, and status; badge labels live in tooltips/ARIA labels so the header stays scannable. Clicking the domain label expands or collapses the domain, opening the same graph view used by the single-domain filter. The graph places owner details and right-aligned command/event groups on the left, with short owner descriptions bottom-aligned as the final part of that pane. Participant usage is grouped the same way on the right, with observed or shimmed links between border-aligned endpoint ports. In multi-provider domains, links to provider participants use provider-family colors: purple for owner-to-provider command delegation and a lighter violet for provider events. Provider participants, including `library` sources, stay on the right lane and show a provider icon in their header. Headers show role-aware core/non-core origin badges such as Core owner, Core provider, or Non-core participant; owner headers place the origin badge directly after the owner icon, and the built-in local library provider is marked as core-origin. Observer and requester roles are implied by the command/event links rather than separate header badges. Participant cards are shown only when the plugin or runtime source has visible command or event usage for the current graph filter; domains with no such usage show zero participants, and attribution-only shims with no matching endpoint stay out of the lane. Command and event groups can collapse; when collapsed, all links for that side and group converge on the single group port. Hovering a participant, endpoint, or command/event group emphasizes the matching links and dims unrelated links; owner-side labels outside the current focus de-emphasize so the active source endpoints are easy to track. Expanded domain graphs progressively enhance to Cytoscape.js overlays that route bezier links between measured DOM endpoint ports, while keeping the HTML lanes as the fallback and readable data surface. Its Plugins-menu entry is hidden by default; enable **Capability Inspector → Show in Plugins menu** from Settings when reviewing or debugging capability behavior.
## Diagnostics Contract
Capability diagnostics use schema `slopsmith.capabilities.diagnostics.v1`. Snapshots are redaction-safe and capped at 64 KB by trimming older `recentDecisions` first while preserving current participants, active or orphaned claims, conflicts, domain review metadata, shim summaries, safety notes, and unsupported-version reports. Server diagnostics bundles include plugin manifest capability metadata, validation warnings, unsupported-version metadata, and compatibility shim summaries.
Compatibility shim entries include `shimId`, `source`, `capability`, `legacySurface`, `status`, `reason`, and optional hit fields. A shim with `hitCount > 0` means legacy behavior was observed, not merely declared. The `library` domain no longer uses compatibility shims for provider registration or source selection; provider attribution comes from `owner_plugin_id` and runtime provider participants. Future domains should add expected shim entries only in the PR that implements their actual legacy bridge.
## Expected Future Domains
Expected future domains live in [capability-roadmap.md](capability-roadmap.md) and [capability-safety-matrix.md](capability-safety-matrix.md) instead of the runtime graph. They are reserved names and candidate command shapes for future PRs, not current contracts. A future-domain PR should add the real host workflow, runtime registration, diagnostics redaction rules, tests, and compatibility shims in the same slice that makes the domain visible to plugins.
## Incremental Roadmap
Release slices should stay reviewable. The domain-level roadmap, PR1 domain set, deferred domains, shim policy, and future-domain PR checklist live in [capability-roadmap.md](capability-roadmap.md).
Future privileged domains must state user value, included and excluded commands, safety class, diagnostics fields, failure recovery, and tests proving disabled or incompatible participants cannot execute handlers before implementation begins.
## Rehydration Pattern
Plugins that wrap shared functions such as `window.playSong` or `window.showScreen` should store wrapper state on a stable `window.__slopsmith...Hooks` object. Re-running the script should replace the implementation object and return before installing another wrapper.
```js
const hookState = window.__slopsmithMyPluginHooks || (window.__slopsmithMyPluginHooks = {});
hookState.impl = { afterPlaySong(filename) { /* current implementation */ } };
if (hookState.installed) return;
hookState.installed = true;
hookState.basePlaySong = window.playSong;
window.playSong = async function(filename, arrangement) {
await hookState.basePlaySong.call(this, filename, arrangement);
hookState.impl?.afterPlaySong?.(filename, arrangement);
};
```
## Validation Commands
From the `slopsmith/` directory:
```bash
node --check static/app.js
node --check static/capabilities.js
node --check static/diagnostics.js
node --check plugins/capability_inspector/screen.js
node --test tests/js/*.test.js
pytest tests/test_plugin_runtime_idempotence.py tests/test_plugins.py tests/test_diagnostics_bundle.py -q
```
docs/capability-recipes.md
+560
View File
@@ -0,0 +1,560 @@
# Capability Authoring Recipes
Use these examples as small manifest fragments when migrating plugin-facing integrations to capability pipelines. The capability model is system-wide; these recipes focus on plugin manifests because core-owned domains are registered by Slopsmith itself. Each example is intentionally complete enough to pass the loader contract in [plugin-manifest.schema.json](plugin-manifest.schema.json).
> **Self-hosted CSS?** If your plugin uses Tailwind classes core doesn't ship (notably arbitrary values like `text-[11px]`), declare a `styles` key and bundle your own preflight-off stylesheet — see [plugin-styles.md](plugin-styles.md). That is separate from the capability-pipeline recipes below.
## Owner And Provider
A plugin that owns a domain and handles commands declares `owner` and `provider`. Use this for a single canonical implementation in a plugin-owned domain, such as a future stem-control capability.
```json
{
"id": "stems",
"name": "Stems",
"standards": ["capability-pipelines.v1", "plugin-runtime-idempotent.v1"],
"capabilities": {
"stems": {
"roles": ["owner", "provider"],
"commands": ["mute", "restore", "inspect"],
"events": ["claim:created", "claim:released", "stems.ready"],
"mode": "active",
"compatibility": "none",
"ownership": "exclusive-owner",
"safety": "safe",
"version": 1
}
}
}
```
## Requester And Observer
A plugin that requests work from another domain and listens for lifecycle events declares `requester` and `observer`.
```json
{
"id": "example_requester",
"name": "Example Requester",
"standards": ["capability-pipelines.v1", "plugin-runtime-idempotent.v1"],
"capabilities": {
"example.plugin-domain": {
"roles": ["requester", "observer"],
"commands": ["apply", "restore", "inspect"],
"events": ["claim:created", "claim:released", "example.manual-override"],
"mode": "active",
"compatibility": "none",
"ownership": "requester-only",
"safety": "safe",
"version": 1
}
}
}
```
## Observer Only
A plugin that only reads public events should declare `observer` and no command handlers. In PR1, this is most useful for participants that observe the delivered `library` workflow; future plugin-owned domains can use the same pattern once promoted.
```json
{
"id": "practice_hud",
"name": "Practice HUD",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"example.plugin-domain": {
"roles": ["observer"],
"commands": [],
"events": ["claim:created", "claim:released", "example.manual-override"],
"mode": "active",
"compatibility": "degrade-noop",
"ownership": "observer-only",
"safety": "safe",
"version": 1
}
}
}
```
## Library Provider
A plugin that registers a remote client or generated library source declares itself as a `library` provider. The backend registration call is still made from `routes.py` with `context["register_library_provider"](...)`; the native browser library capability turns the provider registry into runtime provider participants. A thin server wrapper that only exposes the local library over HTTP should not declare `library` as a provider unless it also registers a provider in the library registry.
```json
{
"id": "remote_library_client",
"name": "Remote Library Client",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"library": {
"roles": ["provider"],
"operations": ["query-page", "query-artists", "query-stats", "tuning-names", "get-art", "sync-song"],
"mode": "active",
"compatibility": "none",
"safety": "safe",
"version": 1
}
}
}
```
## Library Requester And Observer
A route-only wrapper that uses the library capability without registering a browsable provider should declare requester/observer intent instead of provider ownership. This is a generic manifest shape for external plugins to adopt in their own repositories; it does not make the wrapper part of this PR's delivered domain set.
```json
{
"id": "library_route_wrapper",
"name": "Library Route Wrapper",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"library": {
"roles": ["requester", "observer"],
"requests": ["list-providers", "get-current", "inspect"],
"observes": ["providers-refreshed", "source-changed"],
"description": "Uses the library source list through its own route surface without registering a provider.",
"mode": "active",
"compatibility": "none",
"safety": "safe",
"version": 1
}
}
}
```
## Audio Mix Fader Provider
Existing plugins can keep using `window.slopsmith.audio.registerFader(spec)` while migrating. The compatibility bridge records the fader as an `audio-mix` participant. New bundled code should prefer a native participant declaration plus the audio-session helper once available in its integration point.
```json
{
"id": "delay_fx",
"name": "Delay FX",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"audio-mix": {
"roles": ["provider"],
"operations": ["fader.get-value", "fader.set-value"],
"events": ["fader-value-changed", "fader-unavailable"],
"mode": "active",
"compatibility": "none",
"safety": "safe",
"version": 1
}
}
}
```
Native audio-mix fader providers should register a stable participant id and fader id, return the committed value from every set operation, and settle get/set operations within two seconds. The player mixer displays the committed value rather than the raw requested value. If the fader is temporarily unavailable, keep the participant registered with unavailable/disabled state so the mixer can render a disabled control and diagnostics can explain why it cannot be changed.
During migration, a plugin may still call `window.slopsmith.audio.registerFader(spec)`. Core maps that legacy fader into a compatibility-backed audio-mix participant and records bridge hits. If a native participant and a legacy fader represent the same logical source, the native participant owns the visible control and the legacy path is reported as compatibility-backed/overshadowed.
## Audio Effects Provider
Plugins that can provide guitar/bass processing chains should declare `audio-effects` as a provider and register at runtime with `window.slopsmith.audioEffects.registerProvider(...)`. The provider returns opaque chain plans; it must not expose local filenames, URLs, native preset JSON, VST state blobs, or raw handles through diagnostics or public route state.
```json
{
"id": "rig_builder",
"name": "Rig Builder",
"standards": ["capability-pipelines.v1", "plugin-runtime-idempotent.v1"],
"capabilities": {
"audio-effects": {
"roles": ["provider"],
"operations": ["chain.resolve", "chain.inspect", "segment.activate", "stage.set-bypass", "stage.set-parameter"],
"events": ["provider-registered", "route-selected", "plan-resolved", "changed", "fallback", "bridge-hit"],
"mode": "active",
"compatibility": "shim-allowed",
"safety": "sensitive",
"version": 1
}
}
}
```
```js
const effects = window.slopsmith && window.slopsmith.audioEffects;
effects.registerProvider({
providerId: 'rig-builder',
pluginId: 'rig_builder',
routeKey: 'desktop-main',
priority: 40,
operations: ['chain.resolve', 'segment.activate', 'stage.set-bypass', 'stage.set-parameter'],
operationHandlers: {
'chain.resolve': request => ({
outcome: 'handled',
plan: {
schema: 'slopsmith.audio_effects.chain_plan.v1',
planId: 'song-tone-plan',
routeKey: request.routeKey,
providerId: 'rig-builder',
stages: [
{ stageId: 'amp', kind: 'nam', role: 'amp', assetRef: 'rig-builder:asset:amp-main' },
{ stageId: 'cab', kind: 'ir', role: 'cab', assetRef: 'rig-builder:asset:cab-main' }
],
segments: [{ segmentId: 'base', stageIds: ['amp', 'cab'] }],
summary: { stageCount: 2 }
}
})
}
});
```
User-facing controls should dispatch through the domain instead of mutating another plugin's private state:
```js
await window.slopsmith.capabilities.dispatch({
capability: 'audio-effects',
command: 'select-chain',
source: 'rig_builder',
payload: { routeKey: 'desktop-main', providerId: 'rig-builder', authorization: 'user-action' }
});
const resolved = await window.slopsmith.capabilities.dispatch({
capability: 'audio-effects',
command: 'resolve-plan',
source: 'nam_tone',
payload: { routeKey: 'desktop-main', target: { settingsKey: 'settings-v1-...' } }
});
```
Providers should store public song/tone routing through the host-owned mapping index and keep their own preset or chain rows private. The mapping's `provider_ref` is opaque to core: NAM Tone can use a preset id, Rig Builder can use a chain/preset id, and each provider resolves that reference in `chain.resolve`.
```js
await window.slopsmith.audioEffects.upsertMapping({
song_key: playbackTarget.settingsKey,
filename: playbackTarget.filename, // optional migration/debug context
tone_key: 'Dist',
provider_id: 'rig-builder',
provider_ref: 'chain:99',
label: 'Full Rig Builder chain',
source: 'manual',
active: true
});
const mappings = await window.slopsmith.audioEffects.listMappings({
song_key: playbackTarget.settingsKey,
tone_key: 'Dist'
});
```
Only one mapping is active for a `song_key + tone_key` at a time, but multiple providers may have rows for the same song/tone. The active row decides which provider core asks first; provider fallback remains explicit through provider priority and `fallbackProviderId` during `loadPlan(...)`.
Browser or native executors should declare both provider scope and plan scope. A NAM-only browser executor should not claim Rig Builder plans just because it can load NAM files:
```js
window.slopsmith.audioEffects.registerExecutor({
executorId: 'nam-tone-browser-wasm',
pluginId: 'nam_tone',
routeKey: 'desktop-main',
providerIds: ['nam-tone'],
supportedKinds: ['nam', 'ir'],
maxStages: 2,
sourceMode: 'browser',
loadChainPlan: request => loadNamToneWasmPlan(request)
});
```
Trusted Desktop can advertise broader support, while provider-specific browser executors should keep their `providerIds`, `supportedKinds`, and `maxStages` as narrow as the runtime actually supports.
The desktop executor is the trust boundary for physical loading. It should treat `assetRef` and `stateRef` values as opaque provider references, validate them through provider-owned lookup code, enforce local policy, then load or reject processor stages. Browser diagnostics should report route/provider/outcome summaries only.
## Audio Input And Monitoring Requester
Plugins that need live instrument input should declare requester/observer intent and let the host expose redaction-safe source identity. Diagnostics must not contain raw device labels, stable hardware ids, or audio buffers.
```json
{
"id": "note_detect",
"name": "Note Detect",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"audio-input": {
"roles": ["requester", "observer"],
"requests": ["inspect", "list-sources", "select-source", "open-source", "close-source"],
"observes": ["source-registered", "source-selected", "source-opened", "source-open-degraded", "source-closed", "permission-denied"],
"mode": "active",
"compatibility": "shim-allowed",
"ownership": "requester-only",
"safety": "sensitive",
"version": 1
},
"audio-monitoring": {
"roles": ["requester", "observer"],
"requests": ["inspect", "list-providers", "select-provider", "start", "stop", "set-direct-monitor"],
"observes": ["provider-registered", "provider-selected", "provider-selection-required", "monitoring-started", "monitoring-degraded", "monitoring-unavailable", "monitoring-failed", "monitoring-denied", "monitoring-stopped", "direct-monitor-changed"],
"mode": "active",
"compatibility": "shim-allowed",
"ownership": "requester-only",
"safety": "sensitive",
"version": 1
}
}
}
```
Requesters should list or inspect sources before opening them. `inspect`, `list-sources`, and `select-source` are prompt-free and must not call provider enumeration or open live input. When a requester needs audio, it dispatches `open-source` with a purpose and required channel shape. The requester identity is taken from the dispatch `source` (the authenticated caller) — a payload-supplied `requesterId` is ignored, so a requester cannot spoof another's identity or release a shared session it does not own. Compatible requesters share one open session; each requester later dispatches `close-source`, and the provider is closed only after the last requester releases it.
```js
const api = window.slopsmith.capabilities;
await api.dispatch({ capability: 'audio-input', command: 'select-source', source: 'user', payload: { logicalSourceKey: 'browser:instrument:primary' } });
const opened = await api.dispatch({
capability: 'audio-input',
command: 'open-source',
source: 'note_detect', // identity for the open session; payload requesterId is ignored
payload: { purpose: 'note-detection', requiredChannelShape: 'mono' },
});
// Keep provider-owned streams/nodes private. Diagnostics receive only opened.payload summaries.
await api.dispatch({ capability: 'audio-input', command: 'close-source', source: 'note_detect', payload: { openSessionId: opened.payload.openSessionId } });
```
Monitoring is a separate lifecycle layered on top of input readiness. A fresh live monitoring start must come from an explicit user action; background requesters can attach only when an already-active compatible session exists.
```js
const monitoring = await api.dispatch({
capability: 'audio-monitoring',
command: 'start',
source: 'note_detect',
payload: {
authorization: 'user-action',
requiredChannelShape: 'mono',
directMonitorRequirement: 'muted'
},
});
if (monitoring.outcome === 'user-action-required') {
// Show your own UI affordance; do not trigger a device prompt in the background.
}
await api.dispatch({
capability: 'audio-monitoring',
command: 'stop',
source: 'note_detect',
payload: { monitoringId: monitoring.payload && monitoring.payload.monitoringId },
});
```
## Audio Input Provider
Native input providers register redaction-safe source summaries with stable logical keys. Use `source.enumerate` only for an explicit user/provider discovery action; normal list/inspect/select flows should use already-registered summaries.
```json
{
"id": "desktop_audio",
"name": "Desktop Audio",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"audio-input": {
"roles": ["provider", "observer"],
"operations": ["source.enumerate", "source.open", "source.close"],
"events": ["source-registered", "source-opened", "source-closed", "source-open-degraded", "permission-denied"],
"mode": "active",
"compatibility": "none",
"safety": "sensitive",
"version": 1
}
}
}
```
Provider source records should include `sourceId`, `providerId`, `logicalSourceKey`, `kind`, safe label or diagnostics pseudonym, availability, `channelSummary`, and supported operations/handlers. Do not put browser `MediaStream`, `AudioNode`, native handles, buffers, samples, waveform data, raw device labels, stable hardware ids, paths, or secrets in returned payloads; keep those in provider-private state.
## Audio Monitoring Provider
Native monitoring providers register a stable `logicalMonitoringKey` and keep actual audio streams, native handles, and device labels provider-private. The core host coordinates selected provider, requester sharing, direct-monitor policy, and diagnostics, but the provider owns the actual live monitor graph.
```json
{
"id": "desktop_audio",
"name": "Desktop Audio",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"audio-monitoring": {
"roles": ["provider", "observer"],
"operations": ["monitoring.start", "monitoring.stop", "monitoring.status", "monitoring.set-direct-monitor"],
"events": ["provider-registered", "monitoring-started", "monitoring-stopped", "direct-monitor-changed"],
"mode": "active",
"compatibility": "none",
"safety": "sensitive",
"version": 1
}
}
}
```
Provider records should include `providerId`, `logicalMonitoringKey`, safe label or diagnostics pseudonym, `availability`, `sourceMode`, supported operations, `directMonitor` summary, and `latencySummary`. `monitoring.start` receives a redaction-safe `sourceRef`, `requesterId`, `requiredChannelShape`, `directMonitorPreference`, and optional `directMonitorRequirement`; it should return only status summaries such as active/degraded/denied/unavailable/failed. `monitoring.status` must be prompt-free and must not open audio input. `monitoring.set-direct-monitor` may apply the user's preference for active sessions; requester requirements must never mutate the user's stored preference.
## Stems Provider Behind Audio Session Coordination
The Stems plugin remains the provider/owner of actual stem playback state. `core.audio.session` coordinates dispatch, claims, overrides, orphan detection, and diagnostics, but it does not replace the provider.
```json
{
"id": "stems",
"name": "Stems",
"standards": ["capability-pipelines.v1", "plugin-runtime-idempotent.v1"],
"capabilities": {
"stems": {
"roles": ["owner", "provider"],
"commands": ["mute", "restore", "inspect"],
"operations": ["stem.get-state", "stem.apply-automation", "stem.restore-automation"],
"events": ["owner-available", "automation-applied", "automation-restored", "automation-overridden", "claim-orphaned"],
"mode": "active",
"compatibility": "shim-allowed",
"ownership": "exclusive-owner",
"safety": "safe",
"version": 1
}
}
}
```
## Playback Requester And Observer
Plugins that need to inspect or coordinate song transport should declare `playback` requester/observer intent and use the capability dispatch surface instead of wrapping `window.playSong` or scraping the `<audio>` element. Raw media handles stay private to core; diagnostics expose only pseudonymous targets, sanitized timing, route, loop, requester, observer, and recent outcome summaries.
```json
{
"id": "practice_hud",
"name": "Practice HUD",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"playback": {
"roles": ["requester", "observer"],
"requests": ["inspect", "pause", "resume", "seek", "set-loop", "clear-loop"],
"observes": ["ready", "started", "paused", "resumed", "seeking", "seeked", "stopped", "loop-set", "loop-cleared"],
"mode": "active",
"compatibility": "shim-allowed",
"ownership": "requester-only",
"safety": "safe",
"version": 1
}
}
}
```
Fresh audible starts require a user action. Background plugins should call `inspect` first and attach to an existing compatible session; if a plugin needs to offer a play/start action, wire it to a visible user gesture and pass `authorization: "user-action"`.
```js
const api = window.slopsmith.capabilities;
const state = await api.dispatch({
capability: 'playback',
command: 'inspect',
source: 'practice_hud',
args: {},
});
if (state.status !== 'idle') {
await api.dispatch({
capability: 'playback',
command: 'seek',
source: 'practice_hud',
args: { time: 42.0, reason: 'practice segment jump' },
});
}
```
During migration, legacy uses of `window.playSong`, `song:*` events, `window.slopsmith.seek`, and loop helpers remain available and are recorded as playback bridge hits. Treat bridge hits as migration telemetry: native capability requests should eventually cover normal plugin workflows so unexpected legacy hits disappear from diagnostics.
## Progression Requester And Observer
Plugins that report gameplay outcomes or react to player progression (spec 010) should declare `progression` requester/observer intent and use capability dispatch instead of private fetches. Externally postable event types are whitelisted (`minigame_run` in v1); `song_completed` is server-derived inside `/api/stats` and is denied at this surface. Backend plugin code can use the plugin-context hook `record_progression_event` instead (the minigames hub does).
```json
{
"id": "my_minigame",
"name": "My Minigame",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"progression": {
"roles": ["requester", "observer"],
"requests": ["inspect", "record-event"],
"observes": ["challenge-completed", "quest-completed", "path-level-up", "rank-changed", "db-changed"],
"mode": "active",
"compatibility": "none",
"ownership": "requester-only",
"safety": "safe",
"version": 1
}
}
}
```
`buy-item` and `equip-item` require a visible user gesture (`authorization: "user-action"`). Decibels are play-earned only; plugins must not present any purchase path.
```js
const api = window.slopsmith.capabilities;
const result = await api.dispatch({
capability: 'progression',
command: 'record-event',
source: 'my_minigame',
payload: { type: 'minigame_run', payload: { game_id: 'my-minigame', score: 420 } },
});
// result.payload lists challenges/quests completed by this event (toast UX).
window.slopsmith.on('progression:quest-completed', (e) => {
console.log('quest done:', e.detail.title, '+' + e.detail.reward_db + ' dB');
});
```
## Future Expansion Domains
Some domain names are reserved for expected future contracts, but they are not registered in the runtime graph yet. For example, `ui.player-panels` is documented as a likely panel-host surface, but Slopsmith does not currently expose a capability command for panel contributions. See [capability-roadmap.md](capability-roadmap.md) for the PR1 domain set and deferred-domain checklist.
Plugins should not declare future expansion domains until the corresponding host workflow ships. For current integrations, prefer active domains such as `library`, `playback`, `audio-mix`, `audio-input`, `audio-monitoring`, or `stems` intent matching the recipes above.
Invalid capability metadata is excluded from the capability graph, but legacy manifest fields still load through their existing app paths. The `library` workflow is native in PR1 and does not use compatibility shim metadata. Unsupported `capability-pipelines` versions are reported as incompatible and their runtime handlers must not execute.
## Library Card Action (`ui.library-card-injection`)
Delivered in fee[dB]ack v0.3.0 (frontend host). Plugins add per-song actions to
the library cards by REGISTERING them instead of DOM-injecting onto
`.song-card`. The library renders applicable actions in each card's action
menu, dispatches the handler on click, and emits `action-result` events;
the owner is visible in the Capability Inspector.
```json
{
"id": "my_card_action",
"name": "My Card Action",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"ui.library-card-injection": {
"roles": ["provider"],
"operations": ["action.run"],
"events": ["action-registered", "action-result"],
"mode": "active",
"compatibility": "none",
"safety": "safe",
"version": 1
}
}
}
```
Register the action from the plugin's `screen.js`:
```js
window.slopsmith.libraryCardActions.register({
id: 'my_card_action.run',
pluginId: 'my_card_action',
label: 'Do the thing',
placement: 'menu', // 'menu' | 'inline' | 'overlay'
order: 50,
applies: (song) => song.format === 'sloppak', // shown only when relevant
enabled: (song) => true,
run: async (song, ctx) => { // ctx.source identifies the surface
await fetch('/api/plugins/my_card_action/run', {
method: 'POST', headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ filename: song.filename }),
});
},
});
```
`register(spec)` returns an `unregister()` fn. The host owns rendering,
applicability, enabled state, and `action-result` events — plugins do not touch
library DOM. Legacy `.song-card` DOM injection still works in the 0.2.x UI;
migrate to this for the v0.3.0 native library.
docs/capability-review-preflight.md
+81
View File
@@ -0,0 +1,81 @@
# Capability Slice Review Preflight
Use this preflight before opening the next capability-slice PR. It captures the review patterns found while landing the audio/input/monitoring/playback slices, so a future slice should only need to analyze this document plus the immediately previous PR for newly discovered review themes.
## How To Use This
1. Read this checklist before creating the PR.
2. Scan the new slice for each pattern below.
3. Add focused regression tests for every pattern that applies to the new domain.
4. Then inspect only the last merged capability PR for new reviewer feedback that is not already covered here.
## Identity And Authority
- Command attribution must come from the capability dispatch caller (`requester` / `source`), not from payload fields such as `requesterId`.
- Payload identity is allowed only on explicit registration commands, such as `register-requester`, `register-observer`, or provider/source registration surfaces.
- Docs and examples must not tell callers to pass ignored identity fields in command payloads.
- Shared-session release paths must verify the releasing requester owns that attachment. A spoofed payload must not release another requester.
- User-action boundaries must be explicit. Fresh audible or live-input starts from background code should return `user-action-required`; background requesters may attach only to already-active compatible sessions when the contract allows it.
## Redaction And Diagnostics
- Treat all local storage, provider replies, adapter replies, bridge payloads, event details, command payloads, and dispatch caller strings as untrusted.
- Exported diagnostics must not contain raw filenames, titles, artists, paths, URLs, secrets, API keys, tokens, device labels, hardware ids, media/native handles, buffers, samples, waveforms, recordings, route-private objects, or provider-private objects.
- Redact before normalizing ids. Charset-only sanitizers can preserve path or token fragments such as `Users-me-plugin-token-abc`.
- Filter raw keys after normalizing camelCase to snake_case, so names like `accessToken`, `apiKey`, `nativeHandleRef`, `rawDeviceId`, and `mediaStream` are caught.
- Allow safe display fields only through an explicit allowlist. Do not let a general `label` exemption reintroduce raw device labels.
- Stored ids or persisted selections must be accepted only if they are already redaction-safe; otherwise ignore and clear them when possible.
- Bridge entries are exported verbatim enough to deserve the same redaction/bounding as command outcomes.
## Outcomes And Provider Results
- Propagate explicit provider/adapter outcomes when they are part of the domain contract. Do not collapse `no-handler`, `no-owner`, `unsupported-command`, `incompatible-version`, `overridden`, or similar actionable statuses into generic `degraded` or empty success.
- A provider list/enumerate command with providers but no matching handler should return `no-handler`, not `handled` with an empty list.
- Malformed provider or adapter results should be `failed` or another explicit contract outcome. Void/missing fields must not be treated as successful active/handled state.
- Command return values, recent outcomes, lifecycle events, and inspector display should agree on the same outcome/status names.
- If a command records an outcome on an early return, diagnostics must be refreshed immediately. In current hosts this usually means routing through the central outcome helper that calls the diagnostic touch/contribution path.
## Identifier Semantics
- Document whether each id is a public round-trip handle, an internal generated id, or a per-snapshot pseudonym.
- Do not compare raw caller ids against pseudonyms from diagnostics snapshots.
- If duplicate providers can share a logical key, disambiguate with all fields that define the selected winner, such as provider id, source mode, route kind, or channel shape.
- Generated internal ids that need intra-snapshot correlation may remain stable within that snapshot. Untrusted caller-supplied ids should be bounded, redacted, hashed, or pseudonymized before export.
- Use one pseudonymizer for a batch when returning multiple related records, so distinct raw ids do not all become the same `source-01`/`route-01` style value.
## Lifecycle And Teardown
- Session replacement must close or finalize provider-owned live resources before discarding the old session state.
- Stop paths and route/session switches should emit the same redaction-safe summary shape as normal close/ended paths.
- Prompt-free commands such as inspect, list, and status must not trigger provider enumeration, permission prompts, device opening, or route activation.
- Optional fields need precise fallback semantics. If an optional disambiguator is omitted, fallback only when the match is unambiguous; if the caller supplied an explicit but wrong disambiguator, fail instead of touching a different session.
## Schema And Docs
- Every emitted event name should have one payload shape. Avoid ad-hoc degraded/denied payloads that omit fields present in the success shape.
- Contract request shapes must match fields read by the runtime. Remove fields the runtime ignores, and document separate fields such as `storageStatus` instead of inventing extra enum values.
- Data-model docs must match actual implementation semantics, especially global-vs-provider-scoped ids, `supersededBy` meanings, restore statuses, and outcome enums.
- Recipes should use realistic dispatch caller values: `source: 'user'` for user preference changes, plugin ids for plugin work, and no payload identity when attribution comes from dispatch.
## Tests To Add Per Slice
- Anti-spoofing: payload `requesterId` must not override the dispatch caller for control commands.
- Redaction: unsafe caller ids, persisted values, bridge payloads, provider results, adapter results, nested payloads, and camelCase raw keys must not appear in exported diagnostics or events.
- Outcome preservation: provider/adapter `denied`, `failed`, `degraded`, `no-owner`, `no-handler`, `unsupported-command`, `incompatible-version`, and domain-specific outcomes should round-trip exactly when supported.
- Diagnostics freshness: early denied/no-owner/no-handler/failed returns must update the exported diagnostics contribution immediately.
- Schema consistency: denied/degraded/unavailable events should have the same summary shape as handled events for the same command family.
- Duplicate identity: native-vs-compatibility or multi-provider duplicates should resolve to the selected/canonical winner and reject non-selected hints.
- Teardown: stop/session-switch paths should close live provider resources and preserve diagnosable final state.
- Documentation grep: scan for stale `requesterId` payload examples, removed enum values, unsupported request fields, and bridge ids that do not exist at runtime.
## Validation Baseline
For each slice, run the focused suite for the domain plus the cross-domain regression suite it touches. Also run:
```bash
git diff --check origin/main..HEAD
node --check static/capabilities.js
python3 -m py_compile tests/test_plugin_runtime_idempotence.py
```
Add domain-specific `node --check` and `node --test` commands to the slice quickstart, including the core capability host and any inspector or adapter files changed by the slice.
docs/capability-roadmap.md
+160
View File
@@ -0,0 +1,160 @@
# Capability Roadmap
This roadmap keeps the first capability PR reviewable while making the future domain plan explicit. PR1 ships the substrate and only the domains whose current behavior is implemented, diagnosed, and tested. Future domains stay planned or reserved until the PR that implements their host workflow also adds runtime registration, compatibility shims when needed, diagnostics, and tests.
[plugin-capability-inventory.md](plugin-capability-inventory.md) is the current plugin evidence pass for this roadmap. It inventories 41 included plugins, confirms that no included manifest currently declares `capabilities`, and maps legacy behavior to recommended future domains. Roadmap entries below should be read together with that inventory: domains listed here are planned names and migration targets, while the inventory explains which real plugins are likely to use or declare them.
## PR1 Domain Set
PR1 should include these delivered domains:
| Domain | Scope | Owner Kind | Safety | Why It Is In PR1 |
|--------|-------|------------|--------|------------------|
| `pipeline` | Core diagnostic surface | diagnostic | diagnostic-only | Exposes capability graph inspection, validation, and participant enablement diagnostics. |
| `diagnostics` | Core diagnostic surface | diagnostic | diagnostic-only | Lets support bundles explain capability state safely. |
| `library` | Core app workflow | provider-coordinator | safe | Models current local and plugin-provided library sources, source selection, and song sync. |
The core/runtime domains in PR1 are intentionally small: diagnostics snapshots, pipeline graph operations, and one concrete app workflow (`library`). Runtime claim and override mechanics are covered by focused behavior tests, but no plugin-owned proving domain is promoted into the runtime graph in PR1.
`diagnostics` and `pipeline` are support domains, not feature workflow domains. `diagnostics` is the read-only snapshot/export facade consumed by support bundles and the Capability Inspector. `pipeline` is the graph operations facade: resolve, inspect, validate, and enable or disable participants.
## PR1 Compatibility Shims
PR1 does not expose expected compatibility shims for `library`. Library is implemented as a native provider-coordinator domain; provider attribution comes from backend `owner_plugin_id` metadata and browser runtime provider participants.
Future domains should not add expected shim entries until their own implementation PR. A domain PR owns its compatibility story.
## Audio Graph/Session And Effects Slices
The audio graph/session and effects slices promote these domains after PR1:
| Domain | Scope | Owner Kind | Safety | Compatibility Bridges |
|--------|-------|------------|--------|-----------------------|
| `audio-mix` | Song volume, fader participants, route summary, analyser bridge accounting | provider-coordinator | safe | `audio-mix.fader-registry`, `audio-mix.song-volume`, `audio-mix.analyser` |
| `audio-input` | Redaction-safe input source registration, selection, open-session lifecycle, channel compatibility, and provider migration diagnostics | provider-coordinator | sensitive | `audio-input.legacy-source` |
| `audio-monitoring` | Monitoring provider selection, explicit user-action start/stop, direct-monitor policy, requester sharing, and audio startup barrier readiness | provider-coordinator | sensitive | `audio-monitoring.audio-barrier`, `audio-monitoring.legacy-provider` |
| `stems` | Stem automation claims and active provider status | coordinator plus plugin provider | safe | `stems.master-volume`, `stems.private-state` |
| `audio-effects` | Provider-selected effect routes, constrained chain plans, route/stage controls, fallback, and native-executor bridge accounting | provider-coordinator | sensitive | `audio-effects.legacy-tone-controls`, `audio-effects.legacy-nam-routing`, `audio-effects.legacy-midi-amp` |
`core.audio.session` is the runtime coordinator for all four domains. It owns `audio-mix`, `audio-input`, and `audio-monitoring`; for `stems`, it coordinates the active Stems provider without replacing the Stems plugin as the owner of actual stem playback/state.
The focused audio-mix control-plane slice promotes fader discovery, read/write operations, committed-value events, native-over-legacy duplicate handling, route/analyser inspection, and compatibility removal gates into `audio-mix`. During migration, `window.slopsmith.audio.registerFader(...)` remains available as a compatibility adapter, but the player mixer consumes the audio-mix control plane as its source of truth.
The focused audio-input control-plane slice promotes source listing, prompt-free selection/inspection, explicit provider enumeration, open/close dispatch, channel-shape compatibility, selected-source persistence, shared requester sessions, and redaction-safe failure diagnostics into `audio-input`. During migration, legacy browser, desktop, or plugin-specific input handoffs should be recorded as `audio-input.legacy-source` bridge hits. Native providers own the visible source when they share a logical source key with a compatibility-backed source; the compatibility source remains diagnostics-only until normal playback shows no unexpected legacy hits.
The focused audio-monitoring control-plane slice promotes monitoring provider registration, selected-provider persistence, explicit user-action live monitoring start, shared requester attachment, final-requester stop, provider disappearance/orphan diagnostics, direct-monitor preference/control summaries, prompt-free status inspection, and redaction-safe failure outcomes into `audio-monitoring`. Fresh monitoring starts from plugins/background code return `user-action-required`; background requesters may attach only to an already-active compatible monitoring session. Native providers own a logical monitoring path when they share it with compatibility-backed startup barrier or legacy monitoring surfaces, and the legacy path remains diagnostics-only until normal playback shows no unexpected bridge hits.
The focused audio-effects control-plane slice promotes provider registration, user-authorized chain selection, constrained chain-plan resolution, route bypass/restore, segment activation, stage bypass/parameter routing, fallback accounting, and redaction-safe diagnostics into `audio-effects`. Providers propose opaque chain plans; trusted desktop/native code validates the references and physically loads NAM, IR, VST, or utility stages. During migration, legacy tone controls, NAM Tone/Rig Builder native-preset route interception, and MIDI/external effect handoffs should be recorded as audio-effects bridge hits. Bridge removal gates are: NAM Tone and Rig Builder register native providers, normal playback resolves through `audio-effects` without fetch interception, failure paths degrade to NAM Tone or bypass cleanly, repeated hydration does not duplicate providers/routes, and support snapshots contain no local paths, filenames, model/IR names, URLs, raw native preset JSON, VST state blobs, handles, callbacks, DOM nodes, audio buffers, samples, or waveforms.
## Playback Control Plane Slice
The playback slice promotes `playback` from a deferred domain to an active exclusive-owner core domain. It owns transport commands (`start`, `pause`, `resume`, `stop`, `seek`, `set-loop`, `clear-loop`, `inspect`), lifecycle events (`playback:requested`, `playback:loading`, `playback:ready`, `playback:started`, `playback:paused`, `playback:resumed`, `playback:seeking`, `playback:seeked`, `playback:ended`, `playback:stopped`, route events, bridge hits, and loop events), and redaction-safe diagnostics for session, target, timing, route, loop, requester, observer, bridge, and recent outcome state.
The implementation deliberately keeps raw transport handles in `static/app.js`: the domain host registers a private adapter and receives sanitized snapshots instead of exposing the `<audio>` element, JUCE player, decoded audio buffers, waveform data, or native route handles. Playback targets expose a pseudonymous arrangement-scoped `targetId` plus a hashed per-song `settingsKey` so observers can store local per-song settings without reading raw filenames or paths. Compatibility bridges currently account for `window.playSong`, `window.slopsmith` transport helpers, legacy song events, loop helpers, media snapshots, route switching, and native-route handoff. Fresh audible starts require `authorization: "user-action"`; background requesters can inspect or control only an existing compatible session according to the command conflict policy.
Playback bridge removal gates are: bundled and first-party plugins use native playback dispatch for normal requester/observer workflows; normal play/pause/seek/loop/route smoke runs show no unexpected bridge hits beyond compatibility-only listeners; playback diagnostics distinguish denied, no-target, stale, cancelled, degraded, unavailable, failed, and stopped outcomes; repeated plugin hydration does not duplicate requesters, observers, wrappers, or bridge entries; and exported support snapshots contain no raw song filenames, paths, URLs, media handles, buffers, waveforms, samples, or recordings.
## Progression Domain Slice
The progression slice (spec 010) promotes `progression` as an active exclusive-owner core domain: mastery rank (onboarding calibration + per-path levels), the data-driven challenge/quest engine under `data/progression/`, the Decibels wallet (frontend rename of the unified XP store; spend tracked separately from lifetime earnings), and the cosmetics shop. Commands are `inspect`, `record-event` (whitelisted types; `minigame_run` in v1), `list-shop`, `buy-item`, and `equip-item` (buy/equip require explicit user action). Lifecycle events mirror as `progression:*` window events.
Deferred follow-up slices: a `contributor` role so plugins ship their own challenge/quest content (drums challenges from a drums-scoring plugin, quest-pool entries from minigame plugins), and drums scoring wiring so `song_completed {instrument: "drums"}` goals become satisfiable.
## Recommended Next Slices
The plugin inventory suggests this migration order after the audio graph/session and playback slices:
1. `jobs`: coordinate conversion, import, update, preview, and studio work with progress, cancellation, retry, and terminal failure semantics.
2. `note-detection` (control-plane slice done — see spec-009): continue with chart-path/engine migration onto bindings; audio-input coupling and calibration diagnostics are follow-up slices.
3. UI contribution host: migrate navigation, plugin screens, player controls, player panels, overlays, shortcuts, and guided tours under placement/lifecycle policy.
4. Backend and privileged capability cleanup: migrate routes, plugin lifecycle, media import/export, recording, external services, and subprocess-backed workflows with explicit user confirmation and diagnostics redaction.
## UI/UX Migration Path
This is the recommended order for UI/UX capability work only. It excludes audio semantics, backend route execution, media jobs, and plugin install/update behavior except where those systems need a visible contribution point.
| Order | Slice | Domains | Legacy Surfaces To Bridge | Migration Target | Removal Gate |
|-------|-------|---------|---------------------------|------------------|--------------|
| 1 | UI contribution substrate | `ui.navigation`, `ui.plugin-screens`, `settings`, `ui.player-controls`, `ui.player-overlays`, `ui.player-panels` | Manifest `nav`, `screen`, `settings`, direct DOM insertion, screen-specific globals | Shared contribution registry with regions, ordering, mount/unmount, visibility, focus, teardown, diagnostics, and compatibility shim accounting | Every legacy UI field is represented as a contribution in diagnostics, with no duplicate mounts after script rehydration. |
| 2 | App navigation and plugin screens | `ui.navigation`, `ui.plugin-screens` | `window.showScreen` wrappers, plugin nav entries, ad hoc screen initialization | Central screen host that owns navigation events, plugin screen lifecycle, current-screen state, and back/restore behavior | Plugins can register screens without wrapping `showScreen`; legacy wrappers are observed only as compatibility hits. |
| 3 | Settings contribution host | `settings` | Manifest `settings.html`, plugin settings panels, settings backup hints | Settings registry with panel metadata, redaction class, backup/import allowlist summary, visibility policy, and diagnostics | Settings UI can render from registered contributions; settings values remain plugin-owned and redacted. |
| 4 | Keyboard and command UX | `keyboard-shortcuts` plus UI host regions | `window.registerShortcut`, panel-scoped shortcut helpers, help panel entries | Shortcut contribution registry with scope, priority, conflict reporting, enable/disable state, and help metadata | Shortcut conflicts are diagnosable and panel-scoped shortcuts do not require private registries. |
| 5 | Player controls | `ui.player-controls` | Direct player control DOM edits, control popovers, button/slider globals | Ordered player-control regions with stable command buttons, popovers, sliders, disabled states, and contribution teardown | Player controls can be added/removed/reordered without plugins mutating the control bar directly. |
| 6 | Player overlays | `ui.player-overlays`, `tours` | Overlay canvases, tour overlays, highway visibility listeners, direct z-index management | Overlay host with anchors, z-order, hit-testing, renderer compatibility flags, visibility events, and cleanup | Fretboard, section map, tours, transpose, step mode, and similar overlays can coexist without private layering rules. |
| 7 | Player panels | `ui.player-panels` | Splitscreen panel DOM, panel-local highway instances, panel-local shortcuts | Panel host with layout slots, active-panel focus, per-panel renderer selection, per-panel shortcuts, visibility, and teardown | Splitscreen-style panels can be composed through host APIs instead of wrapping playback/screen globals. |
| 8 | Visualization UX | `visualization` | `type: "visualization"`, `window.slopsmithViz_*`, viz picker state, auto-match hooks | Renderer provider registry with picker integration, auto-match ordering, context-type metadata, fallback/revert events, and per-panel selection | Renderer selection and failure recovery are fully attributed in diagnostics; picker options no longer depend on global scans. |
| 9 | Library and guided UX extensions | `ui.library-card-injection`, `tours` | Library card buttons, tour registration globals, target selectors | Contribution APIs for library card actions and guided-tour steps with applicability, target resolution, and action-result events | Library actions and tours can be inspected, disabled, and tested independently of plugin-private DOM injection. |
| 10 | Theme and polish surfaces | `settings` or candidate `ui.theme` | Global theme settings, direct stylesheet/class mutation | Theme contribution metadata for tokens, selected theme, preview/apply/restore lifecycle, and diagnostics without user secrets | Themes are reversible and attributable, and visual changes do not depend on hidden global state. |
The UI contribution substrate should land first because every later UI/UX slice needs the same basic primitives: contribution identity, stable regions, deterministic ordering, mount/unmount, visibility, focus, teardown, diagnostics, and compatibility shim hit accounting. Specialized UI domains should stay small and should only add behavior that the shared substrate cannot express cleanly.
UI/UX migration should preserve current plugin fields during the transition. Core can translate manifest `nav`, `screen`, and `settings` into contribution records before plugin scripts hydrate, then let runtime plugins re-register richer metadata when their scripts load. The removal gate for each legacy UI API is not just a new command name; it is proof that repeated script hydration, screen switching, player navigation, and plugin disable/enable cycles do not duplicate DOM nodes, wrappers, listeners, shortcuts, canvases, or tours.
## Deferred Domains
These domains are planned but should stay out of the runtime graph until a host workflow exists:
| Domain | Expected Ownership | Expected Safety | Candidate Scope | Implementation Trigger |
|--------|--------------------|-----------------|-----------------|------------------------|
| `stems` | coordinated plugin provider | safe | Stem mute/restore, ownership claims, manual override events, and requester/observer coordination. | Promoted by the audio graph/session slice as a coordinated provider domain. |
| `ui.navigation` | exclusive-owner | safe | Navigation contributions and screen-change events. | A UI host PR that owns contribution placement and route/screen semantics. |
| `ui.plugin-screens` | exclusive-owner | safe | Plugin screen registration and lifecycle. | A screen host PR with mount/unmount and visibility policy. |
| `settings` | exclusive-owner | sensitive | Plugin settings contribution metadata without settings values. | A settings contribution PR with redaction rules and migration story. |
| `audio-mix` | multi-provider | safe | Mixer fader registration and current fader inspection. | Promoted by the audio graph/session slice. |
| `audio-monitoring` | multi-provider | sensitive | Monitoring provider selection, live start/stop lifecycle, shared requester sessions, direct-monitor state, and redacted failure diagnostics. | Promoted by the audio graph/session slice and implemented by the audio-monitoring control-plane slice. |
| `backend.routes` | multi-provider | privileged | Server route/provider participation and route inspection. | A backend domain PR with concrete core/provider workflow, privilege review, and route diagnostics. |
| `ui.player-controls` | exclusive-owner | safe | Player-control contributions and ordering. | A first-party player-control host and layout policy. |
| `ui.player-panels` | exclusive-owner | safe | Player panel contributions, mount/unmount, visibility, ordering. | A panel host with layout and focus rules. |
| `ui.player-overlays` | exclusive-owner | safe | Overlay contributions layered over player or highway surfaces. | Overlay placement and z-order rules that coexist with legacy overlays. |
| `plugins` | exclusive-owner | privileged | Plugin enable/disable/install/update workflows. | Visible user confirmation, rollback, and disabled-handler enforcement. |
| `jobs` | multi-provider | privileged | Long-running jobs, cancellation, status, failures. | Scheduling limits, cancellation semantics, and user-visible failures. |
| `midi-control` | multi-provider | sensitive | MIDI device providers and control mappings. | Device consent and redacted diagnostics. |
| `audio-input` | multi-provider | sensitive | Audio input device providers, source selection, open/close lifecycle, shared sessions, and redacted failure diagnostics. | Promoted by the audio graph/session slice and implemented by the audio-input control-plane slice. |
| `tempo-clock` | multi-provider | safe | Tempo/clock provider registration and consumers. | A concrete tempo source and consumer workflow. |
Deferred domains may remain documented or reserved, but they should not produce expected shims, inspector links, or runtime handlers before their implementation slice.
## Candidate Domains From Plugin Inventory
These candidate domains were surfaced by the included plugin inventory but are not yet part of the core deferred-domain table. They should be promoted only if a focused spec proves that the boundary is clearer than folding the behavior into an existing domain.
| Candidate Domain | Expected Ownership | Expected Safety | Candidate Scope | Initial Evidence |
|------------------|--------------------|-----------------|-----------------|------------------|
| `ui.library-card-injection` | exclusive-owner | safe | Library card actions, placement, applicability, enabled/disabled state, and action-result events. | Find More and Sloppak Converter add library-card actions that are separate from browsable library providers. |
| `tours` | exclusive-owner | safe | Guided tour registration, eligibility, target resolution, step lifecycle, and screen/navigation dependencies. | Library/settings tours, tutorials, and guided plugin walkthroughs use tour-specific lifecycle behavior. |
| `keyboard-shortcuts` | exclusive-owner | safe | Shortcut contribution registration, scope, conflict resolution, enable/disable state, and help-panel metadata. | Splitscreen, Step Mode, and practice-style plugins use or imply scoped shortcuts beyond normal UI placement. |
| `media-import-export` | multi-provider | privileged | Upload/import/export/conversion requests, accepted file types, generated artifacts, cleanup, and failure semantics. | Editor, Tab Import, Profile Import, Sloppak Converter, Studio, and Rig Builder all move user files through backend workflows. |
| `recording` | multi-provider | sensitive | Arm/start/stop capture, take upload/import, capture-source binding, latency metadata, and storage cleanup. | Studio and karaoke workflows need capture/session semantics distinct from raw audio input. |
| `practice-session` | multi-provider | safe | Practice session lifecycle, goals, score/progress events, chart segment focus, and journal persistence boundaries. | Practice Journal, Minigames, Guitar Theory, Flappy Bend, and Note Detect imply practice/progression state. |
| `collaboration` | multi-provider | sensitive | Room/session lifecycle, participant identity redaction, shared playback sync, conflict policy, and disconnect recovery. | Multiplayer is a distinct real-time coordination surface. |
| `external-services` | diagnostic or privileged metadata | privileged | Network/download/subprocess integration inventory, endpoint attribution, confirmation policy, and failure diagnostics. | Update Manager, Find More, Sloppak Converter, and media jobs reach outside local Slopsmith state. |
Candidate domains can also remain as safety metadata on existing domains. For example, `external-services` may be more useful as a cross-cutting review tag than as a dispatchable runtime capability.
## Domain Versioning
PR1 does not add per-domain versioning. The `capability-pipelines.v1` standard versions the overall manifest/runtime/diagnostics contract. Domain evolution follows compatibility rules:
- Adding optional commands, events, diagnostics fields, or participant metadata is non-breaking.
- Removing or renaming commands/events is breaking.
- Changing ownership semantics is breaking.
- Changing command payloads, return payloads, or dispatch outcomes incompatibly is breaking.
- A breaking change requires either a future `capability-pipelines` version or a clearly new domain name if parallel support is needed.
Per-domain versions should wait until Slopsmith has a concrete need for multiple incompatible versions of the same domain to coexist.
## Future Domain PR Checklist
A PR that promotes a deferred domain into the runtime graph should include:
1. User value and included/excluded command scope.
2. Host workflow or provider implementation.
3. Runtime domain review metadata.
4. Manifest and runtime registration path.
5. Compatibility shims only for legacy behavior the PR actually bridges.
6. Diagnostics fields and redaction rules.
7. Inspector behavior and meaningful labels/tooltips.
8. Tests for valid metadata, invalid metadata, unsupported versions, disabled participants, command outcomes, and shim hit accounting.
9. Documentation updates in the safety matrix and capability docs.
Before opening the PR, run the reusable review checklist in [capability-review-preflight.md](capability-review-preflight.md). It records the cross-cutting findings from previous capability reviews so each new slice can focus additional review research on the most recently merged PR.
docs/capability-safety-matrix.md
+46
View File
@@ -0,0 +1,46 @@
# Capability Safety Matrix
Capability declarations include a safety class so reviewers can decide whether a domain can ship as a normal plugin contract or needs extra enforcement first.
Core domains also have a review scope. **Active contract** domains are wired to current Slopsmith behavior and should be tested as working integration points. Expected future domains are documented below, but are intentionally not registered in the runtime graph until Slopsmith ships the corresponding host UI or provider workflow.
| Domain | Owner Kind | Safety Class | Stable Commands | Provider Operations | Notes |
|--------|------------|--------------|-----------------|---------------------|-------|
| pipeline | diagnostic | diagnostic-only | resolve, inspect, validate, participant.set-enabled | none | Graph inspection, validation, and participant lifecycle diagnostics. |
| diagnostics | diagnostic | diagnostic-only | snapshot | none | Redaction-safe snapshot/export surface for support bundles and the Capability Inspector. |
| library | provider-coordinator | safe | list-providers, refresh-providers, select-provider, get-current, sync-song, inspect | query-page, query-artists, query-stats, tuning-names, get-art, sync-song | Library source selection and provider-owned song sync; provider ids are public UI labels, while provider internals stay backend-owned. |
| audio-mix | provider-coordinator | safe | inspect, list-faders, get-fader-value, set-fader-value, inspect-route, inspect-analyser, register-participant, unregister-participant | fader.get-value, fader.set-value, analyser.get-summary, route.get-current | Song route/fader/analyser summaries, committed fader values, native-over-legacy duplicate handling, and bridge accounting; no raw audio data is exposed. |
| audio-input | provider-coordinator | sensitive | inspect, list-sources, register-source, unregister-source, select-source, open-source, close-source | source.enumerate, source.describe, source.open, source.close | Source/device identity is redacted or pseudonymized per diagnostics snapshot. Inspect/list/select are prompt-free; `source.enumerate` runs only when explicitly requested; `open-source` is the permission boundary and records denied/unavailable/failed/incompatible/no-owner/no-handler outcomes without exposing live handles, buffers, samples, or raw device labels. |
| audio-monitoring | provider-coordinator | sensitive | inspect, list-providers, register-provider, unregister-provider, select-provider, start, stop, set-direct-monitor | monitoring.start, monitoring.stop, monitoring.status, monitoring.set-direct-monitor | Inspect/list/select/status are prompt-free. Fresh monitoring start requires explicit user action; background requesters may only attach to an active compatible session. Outcomes distinguish handled, stopped, denied, unavailable, degraded, failed, no-owner, no-handler, unsupported-command, incompatible, incompatible-version, provider-selection-required, and user-action-required. Diagnostics redact raw device labels, hardware ids, paths, secrets, live handles, buffers, samples, waveforms, and recordings. |
| stems | coordinator plus plugin provider | safe | inspect, mute, restore | stem.get-state, stem.apply-automation, stem.restore-automation | Core coordinates claims/overrides; the active Stems provider owns actual stem state/playback. |
| playback | exclusive-owner | safe | inspect, start, pause, resume, stop, seek, set-loop, clear-loop, register-requester, register-observer | none | Core owns the transport control plane while `app.js` keeps raw media handles private. Fresh audible starts require explicit user action. Diagnostics expose pseudonymous targets, sanitized route/timing/loop state, requester/observer summaries, bridge hits, bounded recent outcomes, and no audio elements, native handles, decoded buffers, samples, waveforms, or recordings. |
| progression | exclusive-owner | safe | inspect, record-event, list-shop, buy-item, equip-item | none | Core owns mastery rank, the challenge/quest engine, the Decibels wallet, and the cosmetics shop (spec 010). `record-event` accepts whitelisted types only (`minigame_run`); `song_completed` is server-derived in `/api/stats` and denied here. `buy-item`/`equip-item` require explicit user action. Decibels are play-earned only — no real-money path exists or may be added. Diagnostics (`slopsmith.progression.diag.v1`) carry content warnings, rank/level/quest counts, and wallet totals; no song filenames or display names. |
| audio-effects | provider-coordinator | sensitive | inspect, list-providers, register-provider, unregister-provider, select-chain, resolve-plan, inspect-route, bypass, restore, fallback, activate-segment, set-stage-bypass, set-stage-parameter, record-bridge-hit | chain.resolve, chain.inspect, segment.activate, stage.set-bypass, stage.set-parameter, route.bypass, route.restore | Core owns provider selection, route state, chain-plan schema validation, fallback accounting, and diagnostics. Providers propose opaque NAM/IR/VST/utility chain plans; trusted desktop/native code validates and loads processors. Chain selection and route bypass/restore require explicit user action or restored selection. Diagnostics omit raw paths, filenames, URLs, model/IR names, native preset JSON, VST state blobs, handles, callbacks, DOM nodes, audio buffers, samples, and waveforms. |
| visualization | provider-coordinator | safe | inspect, list-providers, select-renderer, clear-renderer | renderer.create, renderer.destroy | Highway renderer provider registry, picker-delegated selection, auto-match attribution, and failure fallback. `renderer.create` maps to the legacy `window.slopsmithViz_*` factory `init(canvas, ctx)` call; `renderer.destroy` maps to the factory `destroy()` teardown. Legacy `type: "visualization"` manifests and `window.slopsmithViz_*` globals are accounted compatibility shims. Diagnostics carry provider ids/labels, selection source, last auto-match outcome, and last failure — no song filenames, titles, or arrangement names. |
| note-detection | provider-coordinator | sensitive | inspect, register-provider, unregister-provider, open-binding, close-binding, set-target, clear-target | pitch.estimate, verify.target | Detection-binding control plane (spec 009): providers (midi/engine/js) serve primitives; each requester binds its own redacted tuning context; consumers own judgment, hit/miss flow as observability events. Legacy `highway.setNoteStateProvider` is an accounted shim. Diagnostics carry provider/binding summaries and bounded outcomes — no raw audio, sample data, device labels, or song identity. |
Privileged commands are roadmap-only until they have: a visible user confirmation path, diagnostics redaction rules, failure recovery, and tests that prove disabled or incompatible participants cannot execute handlers.
## Expected Future Domains
These domains are expected future capability contracts, not current runtime graph entries. They should stay documentation-only until a PR adds the corresponding host workflow and tests.
| Domain | Expected Ownership Policy | Expected Safety Class | Candidate Commands | Review Gate |
|--------|---------------------------|-----------------------|--------------------|-------------|
| ui.navigation | exclusive-owner | safe | register-contribution, mount, unmount, set-visible, reorder-by-policy, navigate, inspect | Needs a UI host PR with contribution placement and route/screen semantics. |
| ui.plugin-screens | exclusive-owner | safe | register-contribution, mount, unmount, set-visible, reorder-by-policy, inspect | Needs a screen host PR with mount/unmount and visibility policy. |
| settings | exclusive-owner | sensitive | register-contribution, mount, unmount, set-visible, reorder-by-policy, inspect | Needs redaction rules and a migration story for settings metadata. |
| backend.routes | multi-provider | privileged | register, inspect | Needs a concrete backend route/provider workflow, privilege review, and route diagnostics. |
| ui.player-controls | exclusive-owner | safe | register-contribution, mount, unmount, set-visible, reorder-by-policy, inspect | Needs a first-party player-control host. |
| ui.player-panels | exclusive-owner | safe | register-contribution, mount, unmount, set-visible, reorder-by-policy, inspect | Needs a first-party panel host and layout policy. |
| ui.player-overlays | exclusive-owner | safe | register-contribution, mount, unmount, set-visible, reorder-by-policy, inspect | Needs overlay placement rules that coexist with legacy highway overlays. |
| plugins | exclusive-owner | privileged | enable, disable, install-missing, update, inspect | Needs explicit user confirmation for writes/install/update. |
| jobs | multi-provider | privileged | register, inspect, cancel | Needs scheduling limits, cancellation semantics, and user-visible failures. |
| midi-control | multi-provider | sensitive | register, inspect | Needs device consent and redacted diagnostics. |
| tempo-clock | multi-provider | safe | register, inspect | Needs a concrete provider and consumer workflow. |
Planned domains should also stay out of the runtime graph until Slopsmith ships the corresponding user-facing workflows.
When promoting a planned domain, use [capability-review-preflight.md](capability-review-preflight.md) before opening the PR. The preflight captures recurring review requirements for identity, redaction, outcome propagation, diagnostics freshness, schema consistency, and teardown.
docs/diagnostics-bundle-spec.md
+533
View File
@@ -0,0 +1,533 @@
# Slopsmith Diagnostics Bundle — Format Specification
This document is the authoritative reference for the `slopsmith-diag-*.zip`
file produced by Settings → Export Diagnostics (slopsmith#166).
The bundle is consumed by humans (maintainers reading bug reports) **and**
AI agents (auto-triage, code-aware assistants). Every JSON file inside
the zip carries an explicit `schema` field so consumers can dispatch by
version without guessing.
---
## Overview
A diagnostic bundle is a plain ZIP archive. The default filename is:
```
slopsmith-diag-<slopsmith-version>-<YYYYMMDD-HHMMSS>.zip
```
Top-level layout:
```
slopsmith-diag-0.2.4-20260503-143022.zip
├── manifest.json AI-friendly index, schema 1
├── README.txt Human-friendly: what's in here, how to read
├── system/
│ ├── version.json slopsmith + python + OS
│ ├── env.json allowlisted env vars only (no secrets)
│ ├── hardware.json backend hardware (container-limited if Docker)
│ └── plugins.json loaded + orphan plugins, with git info
├── logs/
│ ├── server.log tail of LOG_FILE (last ~5 MB), redacted if requested
│ ├── server.pretty.log human-readable companion when LOG_FORMAT=json (auto-detected)
│ └── server.log.meta.json
├── client/
│ ├── console.json all console levels + window errors + rejections
│ ├── hardware.json browser-visible hardware: WebGL/WebGPU, host OS
│ ├── local_storage.json filtered
│ └── ua.json browser, screen, page URL on export
└── plugins/<plugin_id>/ per-plugin contributed diagnostics
```
Sections are conditional on the user's include toggles (system, hardware,
logs, console, plugins). Missing sections are not represented in
`manifest.json`'s `files` array.
---
## `manifest.json` (bundle-level, schema `1`)
```jsonc
{
"schema": 1, // bundle schema; bump = breaking change
"exported_at": "2026-05-03T14:30:22Z",
"slopsmith_version": "0.2.4",
"runtime": "docker", // "docker" | "electron" | "bare"
"redacted": true, // were redactions applied?
"files": [
{ "path": "system/version.json", "kind": "json", "schema": "system.version.v1", "size": 312 },
{ "path": "logs/server.log", "kind": "text", "lines": 41203, "size": 5242880 }
],
"redactions": { // present when redacted=true
"paths_replaced": 142,
"ips_replaced": 3,
"song_names_replaced": 27,
"secrets_replaced": 1
},
"notes": [
"container masks host CPU/RAM in system/hardware.json — real host info lives in client/hardware.json"
]
}
```
Field semantics:
- `schema: 1` — top-level bundle schema. Increment only on breaking changes
to the layout (file moves, mandatory new sections). New optional fields
are NOT a schema bump; consumers must ignore unknown keys.
- `runtime` — single source of truth for "where was this bundle produced"
so an agent can pick the right interpretation rules. See
[Runtime kinds](#runtime-kinds).
- `files[].schema` — present only when the file's first-level JSON object
carries a string `schema` field (e.g. `"system.hardware.v1"`).
- `files[].kind` — `"json"` | `"text"` | `"binary"`.
- `notes` — human-readable callouts. Always present; may be empty.
---
## Per-file schemas
### `system.version.v1` — `system/version.json`
```jsonc
{
"schema": "system.version.v1",
"slopsmith_version": "0.2.4",
"python": { "version": "3.12.4", "implementation": "CPython", "executable": "/usr/bin/python" },
"os": { "system": "Linux", "release": "6.5.0", "machine": "x86_64" },
"exported_at": "2026-05-03T14:30:22Z"
}
```
### `system.env.v1` — `system/env.json`
```jsonc
{
"schema": "system.env.v1",
"vars": {
"LOG_LEVEL": "INFO",
"LOG_FORMAT": "json",
"SLOPSMITH_RUNTIME": "electron"
}
}
```
Allowlisted env var keys only (see `ENV_ALLOWLIST` in `lib/diagnostics_bundle.py`):
`LOG_LEVEL`, `LOG_FORMAT`, `LOG_FILE`, `SLOPSMITH_RUNTIME`, `PORT`, `HOST`,
`TZ`, `PYTHONUNBUFFERED`, `DEMUCS_SERVER_URL`. New entries require an
allowlist edit; secrets must never be added.
### `system.hardware.v1` — `system/hardware.json`
```jsonc
{
"schema": "system.hardware.v1",
"runtime": { "kind": "docker", "in_docker": true, "in_kubernetes": false },
"os": { "system": "Linux", "release": "6.5.0", "version": "...", "machine": "x86_64" },
"cpu": {
"brand": "AMD Ryzen 9 7950X 16-Core Processor",
"arch": "x86_64",
"cores_logical": 32,
"cores_physical": 16,
"freq_mhz_current": 4500,
"freq_mhz_max": 5700
},
"memory": { "total_bytes": 67108864000, "available_bytes": 42000000000 },
"gpu": [
{
"source": "nvidia-smi",
"name": "NVIDIA GeForce RTX 4070",
"driver": "550.54.14",
"memory_total_mb": 12282
}
],
"notes": ["container masks host CPU/RAM"]
}
```
`gpu` is a list (zero, one, or many entries). Source values in the wild:
`"nvidia-smi"`, `"rocm-smi"`, `"system_profiler"`. Container deployments
without NVIDIA Container Toolkit will have an empty list and a `notes`
entry explaining why.
### `system.plugins.v1` — `system/plugins.json`
```jsonc
{
"schema": "system.plugins.v1",
"plugins": [
{
"id": "stems",
"name": "Stems",
"version": "1.2.0",
"type": null,
"loaded": true,
"has_screen": true,
"has_script": true,
"has_settings": false,
"has_routes": true,
"diagnostics_declared": true,
"dir": "stems",
"standards": ["capability-pipelines.v1"],
"capabilities": {
"stems": { "roles": ["owner", "provider"], "commands": ["mute", "restore"] }
},
"capability_validation_warnings": [],
"capability_unsupported_versions": [],
"compatibility_shims": [],
"git": { "sha": "abc123d", "remote": "https://github.com/topkoa/slopsmith-plugin-stems.git" }
}
],
"orphans": [
{
"id": "broken",
"name": "Broken Plugin",
"version": "0.1.0",
"loaded": false,
"dir": "broken",
"path": "/home/user/.config/slopsmith/plugins/broken"
}
]
}
```
`orphans` covers plugin directories that contain a `plugin.json` but are
NOT in `LOADED_PLUGINS`. Two sub-cases:
- **Failed-to-load** (no `evicted` field): the plugin id is not loaded at
all — usually requirements install failure or manifest error. A plugin
appearing only in `orphans` without `evicted` is the single best
diagnostic signal for "user installed plugin X but it's not working".
- **Evicted/superseded** (`"evicted": true`): the plugin id IS loaded, but
from a *different* directory. Typical cause: bundled-wins logic discarded
an old user-installed clone in favour of the in-tree copy. Also covers
bundled plugin directories whose routes failed and whose server fell back
to a user copy (the bundled dir then has a different path from the loaded
entry). Check the server startup log for the specific failure reason.
`dir` is the bare directory name. `path` is the full resolved absolute path
to the orphan directory — the key disambiguator when the bundled copy and a
user-installed copy share the same directory name (e.g. both `highway_3d`).
In a redacted bundle `path` has home-dir and config-dir prefixes replaced
with placeholder tokens (e.g. `<HOME>/...`, `<CONFIG_DIR>/...`) so
filesystem paths and usernames do not leak.
Capability fields are redaction-safe manifest metadata. Invalid capability
declarations are excluded from `capabilities` and explained in
`capability_validation_warnings`; legacy surfaces still appear as
`compatibility_shims` so maintainers can see which old fields were bridged
into the capability model. Unsupported future `capability-pipelines` versions
appear in `capability_unsupported_versions` and should be treated as
non-executable runtime intent.
Client-side capability snapshots contributed under `plugins/capabilities/client.json`
use schema `slopsmith.capabilities.diagnostics.v1`. They include current
pipelines, participants, conflicts, missing providers, user overrides, active
or orphaned claims, claim lifecycle records, compatibility shim hit counts,
unsupported-version reports, and recent decisions. The runtime caps this
snapshot at 64 KB by trimming older `recentDecisions` first while preserving
current graph state.
### `logs.server.v1` — `logs/server.log.meta.json`
```jsonc
{
"schema": "logs.server.v1",
"log_file": "/data/log/slopsmith.log",
"exists": true,
"size_bytes": 8388608,
"tail_bytes": 5242880,
"truncated": true
}
```
The companion `logs/server.log` is the raw text tail (UTF-8). When
`LOG_FORMAT=json`, every line is independently parseable as JSON.
When the file exceeds 5 MB, the partial first line is dropped before
serialization so log parsers don't choke.
When the tail is JSON-per-line (auto-detected by content, not by env
var), an additional `logs/server.pretty.log` companion is written:
human-readable lines of the form `<timestamp> [<LEVEL>] <event> k=v
k=v`. Mixed-format tails (a config flip mid-run) preserve non-JSON
lines verbatim. The original `server.log` is still emitted unchanged
for machine consumers. `server.log.meta.json:pretty_companion` is set
to `true` whenever `server.pretty.log` is present.
### `client.console.v1` — `client/console.json`
```jsonc
{
"schema": "client.console.v1",
"entries": [
{
"t": 1714752622123,
"kind": "console", // "console" | "error" | "rejection"
"level": "warn", // "log" | "info" | "warn" | "error" | "debug"
"msg": "WebSocket disconnected: 1006",
"args": ["WebSocket disconnected: 1006"],
"ua": "Mozilla/5.0 ...",
"screen": { "width": 2560, "height": 1440, "devicePixelRatio": 1, "colorDepth": 24 }
},
{
"t": 1714752623456,
"kind": "rejection",
"level": "error",
"msg": "fetch failed",
"stack": "Error: ...\n at ...",
"ua": "...",
"screen": { ... }
}
]
}
```
Bounded ring buffer: 500 entries, ~250 KB cap. Each entry's `args` may
contain truncated stringifications of non-string console arguments —
depth limit 4, key cap 30, string truncation at 1024 chars, circular refs
serialized as `"[circular]"`.
### `client.hardware.v1` — `client/hardware.json`
```jsonc
{
"schema": "client.hardware.v1",
"runtime": {
"kind": "electron",
"electron": "28.1.0",
"chrome": "120.0.6099.109",
"node": "18.18.2",
"v8": "12.0.267.8",
"app_version": "0.2.4"
},
"navigator": {
"userAgent": "Mozilla/5.0 ...",
"platform": "Win32",
"hardwareConcurrency": 16,
"deviceMemory": 8,
"languages": ["en-US"]
},
"userAgentData": {
"platform": "Windows",
"platformVersion": "15.0.0",
"architecture": "x86",
"model": "",
"bitness": "64"
},
"screen": { "width": 2560, "height": 1440, "devicePixelRatio": 1, "colorDepth": 24 },
"webgl": {
"available": true,
"vendor": "Google Inc. (NVIDIA)",
"renderer": "ANGLE (NVIDIA, NVIDIA GeForce RTX 4070 Direct3D11 vs_5_0 ps_5_0)",
"version": "WebGL 2.0 (OpenGL ES 3.0 Chromium)",
"shading_language_version": "WebGL GLSL ES 3.00 (OpenGL ES GLSL ES 3.0 Chromium)",
"max_texture_size": 16384,
"redacted": false
},
"webgpu": {
"available": true,
"adapter_info": {
"vendor": "nvidia",
"architecture": "ada",
"device": "",
"description": ""
}
}
}
```
`runtime.kind` rules:
- `"electron"` if `navigator.userAgent` contains `Electron/`. Versions
populated when the desktop launcher exposes `window.slopsmithElectron`
via a preload `contextBridge`.
- `"browser"` otherwise.
`webgl.redacted: true` indicates the browser refused to expose the real
renderer string (Firefox privacy mode, Safari ≥17). Treat the `vendor`
and `renderer` fields as advisory in that case.
### `client.local_storage.v1` — `client/local_storage.json`
```jsonc
{
"schema": "client.local_storage.v1",
"data": { "<key>": "<value as string>" }
}
```
Every key/value in browser `localStorage` at export time. Plugins
typically prefix their keys with their `plugin_id`.
### `client.ua.v1` — `client/ua.json`
```jsonc
{
"schema": "client.ua.v1",
"userAgent": "...",
"url": "https://slopsmith.local/",
"screen": { ... }
}
```
### Plugin diagnostics — `plugins/<plugin_id>/...`
Per-plugin directory. Two ways to populate it:
1. `diagnostics.server_files` — relpaths under `config_dir`, copied
verbatim. Same allowlist semantics as `settings.server_files`.
2. `diagnostics.callable` — `<module>:<function>`; called with
`({"plugin_id", "config_dir"})`. Return values:
- `dict` / `list` → written to `plugins/<id>/callable.json`
- `bytes` → written to `plugins/<id>/callable.bin`
- `str` → written to `plugins/<id>/callable.txt`
- other types → discarded with a warning
Exceptions are caught and logged to the bundle's `manifest.notes`
— a buggy plugin never crashes the export.
Plugins are encouraged to embed their own `schema` field
(`"<plugin_id>.diag.v1"`) in any JSON they emit so future tooling can
dispatch by plugin schema.
---
## Runtime kinds
`manifest.runtime` and `system/hardware.json:runtime.kind` and
`client/hardware.json:runtime.kind` may take these values:
| Kind | Backend sees… | Frontend sees… | Cross-correlate? |
|------------|-------------------------|----------------|------------------|
| `docker` | container-limited | host | NO — different machines |
| `electron` | host (Python is child) | host | YES — same machine |
| `bare` | host | host | YES — same machine |
Detection precedence (backend):
1. `SLOPSMITH_RUNTIME` env var (`"electron"`/`"docker"`/`"bare"`)
2. `/.dockerenv` exists OR `/proc/1/cgroup` mentions `docker`/
`containerd`/`kubepods` → `docker`
3. Parent process name matches `electron` or `Slopsmith` → `electron`
4. Default: `bare`
Detection (frontend): `Electron/` in user agent → `electron`, else
`browser`.
---
## Redaction
Applied to `logs/server.log` text and `client/console.json` entry
messages when `redact: true` (default). The bundle's
`manifest.json:redactions` reports per-token-class counts.
Token grammar (stable within a single bundle, salted differently
between bundles):
| Token | Source |
|--------------------|-----------------------------------------------------|
| `<DLC_DIR>` | configured DLC root path |
| `<HOME>` | user's home directory |
| `<CONFIG_DIR>` | slopsmith config directory |
| `<song:HASH8>` | song filename / basename (8-char salted SHA-256) |
| `<ip:HASH6>` | IPv4 / IPv6 address |
| `<redacted>` | bearer token, `key=`/`token=`/`api_key=` query strings |
`hardware.json` and `plugins.json` are NOT redacted (no PII).
`local_storage.json` always has values for keys matching secret-name
patterns (`api_key`, `token`, `secret`, `password`, `auth`, `bearer`,
etc.) replaced with `"<redacted>"` — this happens unconditionally,
regardless of the main redaction toggle, because plugin authors
commonly store tokens in localStorage.
---
## Versioning policy
- **Bundle schema (`manifest.schema`)**: integer. Bump on breaking
layout changes (file relocations, removed required sections,
incompatible structural changes to existing schemas). Today: `1`.
- **Per-file schemas (`<area>.<name>.v<n>`)**: bumped independently.
A bundle MAY mix old and new file schemas during transitions.
- **Adding optional fields** to an existing schema is NOT a bump.
Consumers MUST ignore unknown keys.
- **Removing a field** is a bump.
Bundles older than the consumer's known schemas should be processed on
a best-effort basis (display what's recognized, warn about the rest).
---
## AI agent reading guide
Start at `manifest.json`. It lists every file with its schema id —
dispatch on schema, never on path or filename heuristics.
Common symptom → file map:
| Symptom | Files to inspect |
|----------------------------------|-----------------------------------------------------------------------------------------|
| Audio not playing | `system/plugins.json` (stems plugin loaded?), grep `logs/server.log` for `ffmpeg`/`vgmstream`, `client/console.json` for fetch errors |
| 3D highway slow / black | `client/hardware.json` (`webgl.renderer`, `webgpu.adapter_info`); `client/console.json` for WebGL warnings |
| Plugin error on load | grep `logs/server.log` for `Plugin %r`, check `system/plugins.json:orphans` for failed-to-load |
| WebSocket disconnects | `client/console.json` (`level: "warn"` / `"error"`) |
| "Works on my machine" | Diff `system/version.json` + `system/env.json` + `system/hardware.json` between bundles |
| Song-specific bug | grep `logs/server.log` for the song's `<song:HASH>` token (stable across the bundle) |
| Cross-platform crash | `manifest.runtime` + `system/hardware.json:runtime` + `client/hardware.json:runtime` |
| Cache / disk issue | `system/env.json:LOG_FILE`, `logs/server.log.meta.json:exists` |
When the bundle was redacted, the redaction token map is documented
above. Two log lines mentioning `<song:a3f1c2>` are about the same song
— but a bundle exported separately with the same song will use a
different token.
When `manifest.runtime == "docker"`, the backend `system/hardware.json`
reports container-limited values. Real host CPU / RAM / GPU live in
`client/hardware.json` only. Don't cross-correlate.
When `manifest.runtime == "electron"`, both halves describe the same
machine.
---
## Plugin contribution contract
```jsonc
// plugin.json
{
"id": "nam_tone",
"name": "NAM Tone",
"version": "1.0.0",
"diagnostics": {
"server_files": ["nam_tone.db.diag.json"],
"callable": "diagnostics:collect"
}
}
```
Frontend plugins push diagnostics by calling
`window.slopsmith.diagnostics.contribute(plugin_id, payload)` before the
user clicks Export. The payload is written to `plugins/<id>/client.json`
(gated on the same "Plugin diagnostics" toggle as backend plugin files).
Backend callable signature:
```python
# plugins/nam_tone/diagnostics.py
def collect(ctx: dict) -> dict | bytes | str:
"""ctx: {'plugin_id': 'nam_tone', 'config_dir': Path(...)}"""
return {
"schema": "nam_tone.diag.v1",
"models": [...],
}
```
Best practices:
- Return small payloads (< 100 KB). Diagnostics are not a backup channel.
- Embed your own `schema` field in returned dicts.
- Never raise — but if you do, the export keeps going and notes the
failure.
- Don't include user secrets, API keys, or session tokens.
docs/diagnostics/README.md
+73
View File
@@ -0,0 +1,73 @@
# Slopsmith diagnostic sloppaks
Generated, non-copyrighted mini-songs for technique-assessment style
checks. Report-only — they do not change gameplay settings or detection
thresholds.
## Basic Guitar (POC)
**Artifact:** `slopsmith-diagnostic-basic-guitar.sloppak`
**Contents (~55 s):**
- 3 s count-in (quiet click)
- Open thickest string, open next string
- Thickest string, 5th fret
- Repeated E5 power chords (thickest open + next string fret 2)
- Repeat pass: open, fretted, power chords again
Sections: Intro, Open Strings, Fretted Note, Power Chords, Repeat Check.
Manifest includes a custom `diagnostic:` tag (ignored by the loader today;
for future Technique Assessment integration).
## Rebuild
From the slopsmith repo root (requires `ffmpeg`; the slopsmith Docker image
has `libvorbis`, Homebrew ffmpeg may use the built-in `vorbis` encoder):
```bash
python3 docs/diagnostics/build_diagnostic_basic_guitar.py
```
## Builtin seeding
On library scan startup (and periodic rescans), the server copies bundled
diagnostic sloppaks into the user DLC folder when missing or when the
bundled source is newer:
`DLC_DIR/diagnostics-builtin/slopsmith-diagnostic-basic-guitar.sloppak`
Source: `docs/diagnostics/slopsmith-diagnostic-basic-guitar.sloppak` (next to
`server.py` in dev; must be included in the desktop bundle — see
`slopsmith-desktop/scripts/bundle-slopsmith.sh`).
Unlike `tutorials-builtin/`, `diagnostics-builtin/` **is** included in the
library scan. Tracks appear under **Slopsmith** /
**Technique Assessment Diagnostics**.
Existing destination files are not overwritten unless the bundled source
has a newer modification time. User files elsewhere (e.g. `diagnostics-test/`)
are never touched.
## Manual install / test
Normally seeding is automatic once a DLC folder is configured. To test a
custom copy or an unreleased build:
1. Copy `slopsmith-diagnostic-basic-guitar.sloppak` into your Slopsmith
DLC folder (e.g. `diagnostics-test/` or any scanned path).
2. Restart Slopsmith or trigger a library rescan if the song does not appear.
3. Load **Slopsmith Diagnostic — Basic Guitar**.
4. Play the **Diagnostic Guitar** arrangement.
5. Confirm the 3D highway shows open notes and power-chord gems.
6. Turn **Detect** on — note_detect should push the chart to the desktop
verifier on `song:ready` like any other sloppak.
## Future
- Bass diagnostic sloppak
- 7/8-string guitar variants
- Drums (`drum_tab.json`)
- Piano/keys (separate wire model)
- Detection Health “Run Basic Guitar Diagnostic” launch button (note_detect)
docs/diagnostics/build_diagnostic_basic_guitar.py
+441
View File
@@ -0,0 +1,441 @@
"""Build the Slopsmith Diagnostic — Basic Guitar sloppak (POC).
A short, generated, non-copyrighted mini-song for technique-assessment
style checks: open strings, one fretted note, and repeated E5 power chords.
Click-track backing only — no external audio.
Run from the slopsmith repo root:
python3 docs/diagnostics/build_diagnostic_basic_guitar.py
Output (zip archive):
docs/diagnostics/slopsmith-diagnostic-basic-guitar.sloppak
Pattern matches docs/benchmarks/note_detect_v1/build_benchmark.py.
"""
from __future__ import annotations
import json
import math
import shutil
import struct
import subprocess
import sys
import wave
import zipfile
from pathlib import Path
try:
import yaml
except ImportError:
yaml = None
def _yaml_scalar(v):
if isinstance(v, bool):
return 'true' if v else 'false'
if isinstance(v, int):
return str(v)
if isinstance(v, float):
return repr(v)
if v is None:
return 'null'
s = str(v)
if any(c in s for c in ':{}[]&*#?|-<>=!%@`"') or s.strip() != s:
return json.dumps(s, ensure_ascii=False)
return s
def _yaml_lines(obj, indent=0):
prefix = ' ' * indent
lines = []
if isinstance(obj, dict):
for k, v in obj.items():
if isinstance(v, dict):
lines.append(f'{prefix}{k}:')
lines.extend(_yaml_lines(v, indent + 1))
elif isinstance(v, list):
if not v:
lines.append(f'{prefix}{k}: []')
elif all(isinstance(x, dict) for x in v):
lines.append(f'{prefix}{k}:')
for item in v:
lines.append(f'{prefix} -')
for ik, iv in item.items():
if isinstance(iv, (dict, list)):
lines.append(f'{prefix} {ik}:')
lines.extend(_yaml_lines(iv, indent + 3))
else:
lines.append(f'{prefix} {ik}: {_yaml_scalar(iv)}')
else:
lines.append(f'{prefix}{k}:')
for item in v:
lines.append(f'{prefix} - {_yaml_scalar(item)}')
else:
lines.append(f'{prefix}{k}: {_yaml_scalar(v)}')
elif isinstance(obj, list):
for item in obj:
if isinstance(item, dict):
lines.append(f'{prefix}-')
for k, v in item.items():
if isinstance(v, (dict, list)):
lines.append(f'{prefix} {k}:')
lines.extend(_yaml_lines(v, indent + 2))
else:
lines.append(f'{prefix} {k}: {_yaml_scalar(v)}')
else:
lines.append(f'{prefix}- {_yaml_scalar(item)}')
return lines
def dump_manifest_yaml(manifest: dict) -> str:
if yaml is not None:
return yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True)
return '\n'.join(_yaml_lines(manifest)) + '\n'
# ── Chart timing ────────────────────────────────────────────────────────
BPM = 90.0
SECONDS_PER_BEAT = 60.0 / BPM
BEATS_PER_BAR = 4
BAR_S = BEATS_PER_BAR * SECONDS_PER_BEAT
COUNT_IN_S = 3.0 # silence / quiet count-in before first note
NOTE_SUS = 2.8 # single-note ring time
CHORD_SUS = 3.0 # power-chord ring time
OUTRO_S = 4.0 # tail after last event
SR = 44100
def note(t, s, f, sus=0.0, **flags):
return {
't': round(t, 3),
's': s,
'f': f,
'sus': round(sus, 3),
'sl': flags.get('sl', -1),
'slu': flags.get('slu', -1),
'bn': flags.get('bn', 0.0),
'ho': flags.get('ho', False),
'po': flags.get('po', False),
'hm': flags.get('hm', False),
'hp': flags.get('hp', False),
'pm': flags.get('pm', False),
'mt': flags.get('mt', False),
'vb': flags.get('vb', False),
'tr': flags.get('tr', False),
'ac': flags.get('ac', False),
'tp': flags.get('tp', False),
}
def chord(t, id_, notes):
return {
't': round(t, 3),
'id': id_,
'hd': False,
'notes': notes,
}
def chord_note(s, f, sus=0.0, **flags):
n = note(0.0, s, f, sus, **flags)
n.pop('t')
return n
def _sine_burst(freq_hz, duration_s, amplitude):
n = int(SR * duration_s)
out = []
fade = max(1, int(0.004 * SR))
for i in range(n):
env = 1.0
if i < fade:
env = i / fade
elif i >= n - fade:
env = (n - 1 - i) / fade
s = math.sin(2 * math.pi * freq_hz * (i / SR)) * amplitude * env
out.append(s)
return out
def write_click_wav(path: Path, total_duration_s: float, count_in_s: float):
"""Metronome click on every beat; quieter during count-in."""
n_total = int(math.ceil(total_duration_s * SR))
buf = [0.0] * n_total
click_dur = 0.045
downbeat_tone = 1500
upbeat_tone = 1000
downbeat_amp = 0.22
upbeat_amp = 0.12
count_in_amp_scale = 0.35
beat_idx = 0
t = 0.0
while t < total_duration_s - click_dur:
is_downbeat = (beat_idx % BEATS_PER_BAR) == 0
amp = downbeat_amp if is_downbeat else upbeat_amp
if t < count_in_s:
amp *= count_in_amp_scale
click = _sine_burst(
downbeat_tone if is_downbeat else upbeat_tone,
click_dur,
amp,
)
i0 = int(t * SR)
for j, v in enumerate(click):
if i0 + j < n_total:
buf[i0 + j] += v
t += SECONDS_PER_BEAT
beat_idx += 1
pcm = bytearray()
for v in buf:
s = max(-1.0, min(1.0, v))
pcm.extend(struct.pack('<h', int(s * 32700)))
path.parent.mkdir(parents=True, exist_ok=True)
with wave.open(str(path), 'wb') as w:
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(SR)
w.writeframes(bytes(pcm))
def _power_chord_e5(t):
"""E5: thickest string open + next string fret 2."""
return chord(
t,
0,
[
chord_note(0, 0, sus=CHORD_SUS),
chord_note(1, 2, sus=CHORD_SUS),
],
)
def build_chart():
notes = []
chords = []
sections = []
# ── Event times (seconds) ──
t_open_low = 4.0
t_open_next = 8.0
t_fret5 = 12.0
power_times_1 = [16.0, 20.0, 24.0, 28.0]
t_open_repeat = 32.0
t_fret5_repeat = 36.0
power_times_2 = [40.0, 44.0, 48.0]
notes.append(note(t_open_low, 0, 0, sus=NOTE_SUS))
notes.append(note(t_open_next, 1, 0, sus=NOTE_SUS))
notes.append(note(t_fret5, 0, 5, sus=NOTE_SUS))
notes.append(note(t_open_repeat, 0, 0, sus=NOTE_SUS))
notes.append(note(t_fret5_repeat, 0, 5, sus=NOTE_SUS))
for t in power_times_1 + power_times_2:
chords.append(_power_chord_e5(t))
last_event_t = max(power_times_2)
end_t = last_event_t + CHORD_SUS + OUTRO_S
sections = [
{'name': 'Intro', 'number': 1, 'time': 0.0},
{'name': 'Open Strings', 'number': 2, 'time': round(t_open_low, 3)},
{'name': 'Fretted Note', 'number': 3, 'time': round(t_fret5, 3)},
{'name': 'Power Chords', 'number': 4, 'time': round(power_times_1[0], 3)},
{'name': 'Repeat Check', 'number': 5, 'time': round(t_open_repeat, 3)},
]
beats = []
bar_count = 0
bt = 0.0
while bt < end_t:
if abs(bt % BAR_S) < 1e-3:
bar_count += 1
beats.append({'time': round(bt, 3), 'measure': bar_count})
else:
beats.append({'time': round(bt, 3), 'measure': -1})
bt += SECONDS_PER_BEAT
anchors = [{'time': 0.0, 'fret': 0, 'width': 6}]
for sec in sections:
anchors.append({'time': sec['time'], 'fret': 0, 'width': 6})
templates = [{
'name': 'E5',
'displayName': 'E5',
'arp': False,
'fingers': [-1, -1, -1, -1, -1, -1],
'frets': [0, 2, -1, -1, -1, -1],
}]
arrangement = {
'name': 'Diagnostic Guitar',
'tuning': [0, 0, 0, 0, 0, 0],
'capo': 0,
'notes': sorted(notes, key=lambda n: n['t']),
'chords': sorted(chords, key=lambda c: c['t']),
'anchors': anchors,
'handshapes': [],
'templates': templates,
'beats': beats,
'sections': sections,
}
manifest = {
'title': 'Slopsmith Diagnostic — Basic Guitar',
'artist': 'Slopsmith',
'album': 'Technique Assessment Diagnostics',
'year': 2026,
'duration': round(end_t, 3),
'arrangements': [{
'id': 'lead',
'name': 'Diagnostic Guitar',
'file': 'arrangements/lead.json',
'tuning': [0, 0, 0, 0, 0, 0],
'capo': 0,
}],
'stems': [{
'id': 'full',
'file': 'stems/full.ogg',
'default': True,
}],
'diagnostic': {
'kind': 'technique-assessment-basic',
'instrument': 'guitar',
'string_count': 6,
'version': 1,
},
}
return manifest, arrangement, end_t
def _build_zip(src_dir: Path, zip_path: Path):
if zip_path.exists():
zip_path.unlink()
with zipfile.ZipFile(zip_path, 'w', compression=zipfile.ZIP_DEFLATED) as zf:
for p in sorted(src_dir.rglob('*')):
if p.is_file():
rel = p.relative_to(src_dir).as_posix()
info = zipfile.ZipInfo(filename=rel, date_time=(1980, 1, 1, 0, 0, 0))
info.compress_type = zipfile.ZIP_DEFLATED
info.external_attr = (0o644 & 0xFFFF) << 16
info.create_system = 3
zf.writestr(info, p.read_bytes())
def build(output_zip: Path) -> dict:
manifest, arrangement, end_t = build_chart()
staging = output_zip.parent / '_diag_basic_guitar_staging'
if staging.exists():
shutil.rmtree(staging)
staging.mkdir(parents=True)
(staging / 'arrangements').mkdir()
(staging / 'stems').mkdir()
(staging / 'manifest.yaml').write_text(
dump_manifest_yaml(manifest),
encoding='utf-8',
)
(staging / 'arrangements' / 'lead.json').write_text(
json.dumps(arrangement, separators=(',', ':')),
encoding='utf-8',
)
wav_path = staging / 'stems' / 'full.wav'
write_click_wav(wav_path, end_t, COUNT_IN_S)
ogg_path = staging / 'stems' / 'full.ogg'
encoder_cmds = [
['-c:a', 'libvorbis', '-q:a', '5'],
# FFmpeg 8's built-in vorbis encoder requires stereo input.
['-strict', '-2', '-ac', '2', '-c:a', 'vorbis', '-q:a', '5'],
]
last_err = None
for enc_args in encoder_cmds:
try:
subprocess.run(
['ffmpeg', '-y', '-loglevel', 'error',
'-i', str(wav_path),
*enc_args,
str(ogg_path)],
check=True,
stderr=subprocess.DEVNULL if enc_args != encoder_cmds[-1] else None,
)
last_err = None
break
except FileNotFoundError as e:
shutil.rmtree(staging, ignore_errors=True)
raise RuntimeError(
'ffmpeg not found — install ffmpeg to build the OGG stem.'
) from e
except subprocess.CalledProcessError as e:
last_err = e
if last_err is not None:
shutil.rmtree(staging, ignore_errors=True)
raise RuntimeError(
'ffmpeg failed to encode stems/full.ogg — tried libvorbis and vorbis encoders.'
) from last_err
wav_path.unlink()
(staging / 'DIAGNOSTIC.md').write_text(
_diagnostic_readme(end_t),
encoding='utf-8',
)
_build_zip(staging, output_zip)
shutil.rmtree(staging, ignore_errors=True)
return {
'output': output_zip,
'duration_s': end_t,
'notes': len(arrangement['notes']),
'chords': len(arrangement['chords']),
'sections': len(arrangement['sections']),
'stem': 'stems/full.ogg',
'size_bytes': output_zip.stat().st_size,
}
def _diagnostic_readme(duration_s: float) -> str:
return f"""# Slopsmith Diagnostic — Basic Guitar
Short generated diagnostic track for technique-assessment style checks.
Non-copyrighted click-track backing only.
- Duration: {duration_s:.0f} s
- Tuning: E standard (6-string), capo 0
- Sections: Intro, Open Strings, Fretted Note, Power Chords, Repeat Check
Report-only — does not change gameplay settings.
Built by docs/diagnostics/build_diagnostic_basic_guitar.py
"""
def main():
repo_root = Path(__file__).resolve().parents[2]
default_out = Path(__file__).resolve().parent / 'slopsmith-diagnostic-basic-guitar.sloppak'
out = Path(sys.argv[1]) if len(sys.argv) > 1 else default_out
if not out.is_absolute():
out = repo_root / out
stats = build(out)
print(f'Built {stats["output"]}')
print(f' Duration: {stats["duration_s"]:.1f} s')
print(f' Notes: {stats["notes"]}')
print(f' Chords: {stats["chords"]}')
print(f' Sections: {stats["sections"]}')
print(f' Stem: {stats["stem"]}')
print(f' Size: {stats["size_bytes"]} bytes')
if __name__ == '__main__':
main()
docs/diagnostics/slopsmith-diagnostic-basic-guitar.sloppak
BIN
View File
Binary file not shown.
docs/library.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 MiB

docs/note-detect-tuning.md
+231
View File
@@ -0,0 +1,231 @@
# Note Detection Tuning Workflow
How to iterate on the `note_detect` plugin's detection quality with objective, repeatable measurements instead of "feels worse / feels better" guesswork. The same workflow works for tuning the user's environment (A/V offset, latency comp, channel selection) and for tuning the detector code itself (frame size, confidence thresholds, chord-scoring algorithm).
## Why this exists
Detection quality varies by guitar pickup, audio interface, monitor latency, the user's playing style, and the chart's note density. Eyeballing the player UI tells you whether something feels right, not whether a change improved or regressed scoring. The pieces below let you record once and replay many times against arbitrary parameter combinations:
- **Reference recording** — captures the exact PCM frames the live detector saw, so a single take can be re-scored against any settings.
- **Benchmark sloppak** — a known, distributable chart with isolated failure-mode sections.
- **Headless harness** — runs the same `processFrame` / `matchNotes` / `checkMisses` code path the browser uses, off Node, in seconds per run.
- **Diagnostic JSON** — both live (in-browser) and harness output share the `note_detect.diagnostic.v1` schema, so cross-comparison is trivial.
## The benchmark sloppak
The distributable sloppak ships in-tree at [docs/benchmarks/note_detect_v1/note_detect_benchmark_v1.sloppak](benchmarks/note_detect_v1/note_detect_benchmark_v1.sloppak) — drop it directly in your library folder (e.g. `<your-library>/sloppak/`) and it shows up in the library. The file is a zip under the hood but slopsmith's loader (`is_sloppak`) keys off the `.sloppak` suffix, so don't rename. After playing it once it ends up extracted under `static/sloppak_cache/note_detect_benchmark_v1.sloppak/`, which is where the harness reads its `arrangements/lead.json` from. 90 BPM, 8 numbered sections, ~2:20 total:
| Section | Notes | Isolates |
|---|---|---|
| A. Open strings | 12 single notes | low-frequency YIN behaviour (E2=82 Hz) |
| B. 5th-fret positions | 12 single notes | mid-range pitch accuracy |
| C. 12th-fret octaves | sparse single notes | high-frequency YIN behaviour |
| D. Sustained notes | long-hold single notes | sustain matching / pure-miss vs detected |
| E. Hammer / pull | legato pairs | technique-flag handling, attack ambiguity |
| F. Power chords | 8 chord events | 2-string chord scorer |
| G. Open chords | 8 chord events | dense chord scorer (5+ strings ringing) |
| H. Bends | bend pairs | pitch-tolerance edge behaviour |
Every chart note has `sus > 0` — so anything you tune against this benchmark exercises the sustain path, not staccato detection. (If we add a staccato section later, the cleanest split is by section name; don't categorize by `sus` value on the event log — see the "Common pitfalls" section.)
To rebuild after edits to the exercise list, follow the docstring at the top of `build_benchmark.py`. The script writes both an unzipped directory (`.sloppak/`) and a zipped archive (`.sloppak.zip`). The slopsmith library scanner (`lib/sloppak.py::is_sloppak()`) matches on the `.sloppak` suffix, **not** on `.sloppak.zip` — the directory form is usable as-is, but the zip output needs its suffix swapped before it'll be discovered. After regenerating, copy the zip output to the tracked path with the `.sloppak` suffix so it stays a drop-in install. Run from the slopsmith repo root so the relative paths resolve:
```bash
# From the slopsmith repo root.
cp static/sloppak_cache/note_detect_benchmark_v1.sloppak.zip \
docs/benchmarks/note_detect_v1/note_detect_benchmark_v1.sloppak
```
Also update `docs/benchmarks/note_detect_v1/BENCHMARK.md` if you changed sections — it's the user-facing description that ships inside the sloppak, kept alongside the tracked file so contributors can see the section list without having to unzip.
## End-to-end iteration loop
The typical cycle for one tuning hypothesis:
1. **Enable tuning mode** (Settings → Note Detection → "Detection tuning (advanced)"). Off by default; turns on the dev surfaces (Reference Recording, Diagnostic JSON, miss-category breakdown).
2. **Arm a recording** from the gear popover next to the Detect button on the player. Arm before pressing Play.
3. **Play through the benchmark** (or any song) at **1.0× playback speed**. Half-speed playback breaks audio↔chart alignment and produces all-miss garbage — see Pitfalls.
4. **Auto-save fires on song end.** The WAV lands in `static/note_detect_recordings/note_detect_<slug>_<timestamp>.wav` (bind-mounted, so it's reachable from the host without a copy step).
5. **Run the headless harness** with a known config. Paths below assume the note_detect plugin is cloned into `plugins/note_detect/` (see the slopsmith README for the plugin-install flow — note_detect ships as a separate repo):
```bash
node plugins/note_detect/tools/harness.js \
--audio static/note_detect_recordings/note_detect_<…>.wav \
--chart static/sloppak_cache/note_detect_benchmark_v1.sloppak/arrangements/lead.json \
--out /tmp/run.json
```
Prints a one-liner: `<hits>/<total> hits (<%>) — breakdown {pure, chordPartial, early, late, sharp, flat}`.
6. **Sweep parameters** by re-running the harness with different flags (see "Harness flags" below). Compare bins side-by-side. The same recording can drive dozens of runs in seconds.
7. **Form a hypothesis, change code or settings, repeat.** Each PR or settings tweak should move at least one bin in the right direction. If you can't show that, you don't have evidence to ship it.
## Harness flags
All flags map 1:1 to a runtime setting; defaults mirror what a fresh plugin install ships with:
| Flag | Default | Notes |
|---|---|---|
| `--audio <path>` | — | WAV/OGG/MP3 input. WAV is parsed natively; other formats need ffmpeg on PATH. |
| `--chart <path>` | — | The arrangement JSON (e.g. `arrangements/lead.json` from a sloppak directory). |
| `--out <path>` | — | Diagnostic JSON destination. |
| `--method yin\|hps` | `yin` | CREPE is not exercised by the harness (needs WebGL). |
| `--pitch-tolerance <cents>` | `50` | Outer match window for pitch. |
| `--pitch-hit-threshold <cents>` | `20` | Tighter band that counts as "clean" pitch. |
| `--timing-tolerance <s>` | `0.150` | Outer match window for timing. |
| `--timing-hit-threshold <s>` | `0.100` | Tighter band that counts as "clean" timing. |
| `--chord-hit-ratio <r>` | `0.6` | Fraction of strings that must ring for a chord hit (per-string energy bands). |
| `--latency <s>` | `0.080` | Detector pipeline latency compensation. |
| `--frame-size <n>` | `1024` | YIN buffer size in samples. Bigger = better low-freq detection, more latency. |
| `--sample-rate <hz>` | `44100` | Decode target. The WAV reader resamples if the file is different. |
| `--arrangement guitar\|bass` | `guitar` | Picks the open-string MIDI table. |
| `--string-count <n>` | `6` | Used by the string-fret → MIDI math. |
| `--av-offset-ms <ms>` | `0` | Same semantics as `setAvOffsetMs` — pass the user's main-Settings value when replaying their take. **Use `=` for negatives**: `--av-offset-ms=-100`. |
| `--verbose` | off | Logs progress to stderr. |
## Diagnostic JSON — the bits that matter for iteration
Schema `note_detect.diagnostic.v1`. Identical output from live (Settings → Download Diagnostic JSON) and harness. Key fields when comparing runs:
- `summary.hits / misses / accuracy` — top-line score.
- `miss_breakdown` — per-category miss bins:
- `pure` — detector never reported a confident matching pitch in the note's time window. Usually a detector or buffer issue.
- `chordPartial` — chord saw some strings but missed the per-string ratio.
- `early / late` — pitch was right but timing landed outside the inner hit threshold.
- `sharp / flat` — pitch was outside the pitch hit threshold (but inside the outer tolerance, otherwise it'd be `pure`).
- `timing_error_ms` — distribution over **all matched judgments**. Pinned near a constant when av-offset is wrong (matcher snaps to nearest chart note); use for diagnostics only, *not* as a calibration signal.
- `timing_error_ms_hits` — distribution over **only hits**. Responds linearly to av-offset. The A/V auto-calibrate feature keys off this.
- `pitch_error_cents` — same shape as timing but for pitch.
- `events[]` — per-judgment log (capped). Each entry: `{t, at, s, f, sus, hit, chord, ts, ps, te, pe, ex, dx, cnf, tf}`. The `cnf` field is the pitch-detection confidence at match time; `dx` is the detected MIDI; `ex` is the expected MIDI.
## A/V auto-calibrate — the iterative pattern
Settings → Note Detection → "A/V Sync — Auto-Calibrate" surfaces a button that reads `timing_error_ms_hits.median` and applies `setAvOffsetMs(currentOffset − median)`. Expected workflow:
1. If your current A/V offset is wildly off and you're getting almost no hits, **reset the main Settings A/V slider to 0 first**. The matcher snaps to wrong chart notes when offset is far off, which makes `te-hits` an unreliable signal.
2. Play a section with Detect on until you see at least 5 hits on the counter.
3. Click **Apply** — it sets the new offset and clears the timing samples so the next reading reflects only the new regime.
4. Play another section. Apply again. Usually converges in 2–3 rounds; the button greys out as "Already within 20 ms" when there's nothing useful left to suggest.
Crucially: **don't trust the suggestion at low hit counts.** Hits at a far-off offset come from coincidental near-matches to wrong chart notes, and their median is noise. The button gates on `n ≥ 5` but for noisy players a higher manual threshold is wise.
## Common pitfalls
- **Playback speed must be 1.0× during recording.** The recording captures audio at whatever pace it actually played, but the chart times are absolute. A half-speed take produces all-miss output because every chart event fires its match window before the audio has reached that note. Always confirm the speed slider before pressing Play.
- **Don't categorize event-log entries by `event.sus`.** `checkMisses` historically passed only `{s, f}` into miss judgments, so every pure-missed sustained note showed up as `sus=0` in the event log. The bug is fixed (full chart-note flows through now) but old recordings on older builds will mislead you. The reliable answer is to join event entries back against the source chart by `(t, s, f)` and read `sus` from there.
- **All-matched `timing_error_ms.median` is *not* a calibration signal.** When A/V offset is wrong, the matcher matches the user's pluck against whatever chart note is closest in time, not the intended one. The resulting te median is pinned near a constant regardless of the offset value. Always use `timing_error_ms_hits.median` for calibration math.
- **At a very wrong A/V offset, the auto-calibrate suggestion can point further wrong.** When few hits land, their te median is a property of which wrong chart notes happened to be reachable, not of the user's real skew. Start near zero or near a known reasonable value if you suspect the offset is far off.
- **Sweeping parameter X won't fix a problem that lives outside X.** If pure misses dominate at the default config and stay pinned across a 4× range of frame sizes or pitch tolerances, the bottleneck is not those parameters — likely the detector algorithm, the chord scorer, or the matching window logic. Recognise the ceiling and pivot to code changes.
- **Match the recording's sample rate when scoring chords.** The chord scorer is fully self-contained (its own FFT, not `AnalyserNode`) and runs in the harness identically to the browser path. But the harness defaults to `--sample-rate 44100` while most modern USB interfaces capture at 48000 — passing the WAV at the wrong rate resamples it linearly, which smears the FFT bins enough to swing chord-hit counts by 1–2 per take. Cross-validated against one contributor's 48 kHz recording, harness at `--sample-rate 48000 --frame-size 2048` reproduces his live chord-hit count within ±1 (9/16 vs his 10/16). Single-note scoring is less sensitive to this and the default sample rate is usually fine.
- **Bumping the latency-offset default doesn't generalise.** The right latency comp is heavily audio-chain-dependent (USB interface vs. on-board, ScriptProcessor buffering, OS audio path). A value that's perfect for one user over-corrects for another — bumping the default to match the best-tuned user we had data for regressed two of four fixtures. Leave latency at the conservative default and rely on the A/V auto-calibrate panel + the user-facing slider to dial it in per-chain.
## Recipes
### Live judgment streaming — watching a session in flight
When tuning mode is on, the plugin POSTs each judgment to `POST /api/plugins/note_detect/live-judgment` as it's produced. Backend appends one JSON line to `static/note_detect_recordings/live_<sessionId>.jsonl`. A fresh session id is minted on every `song:play`, so each take produces its own file paired with the recorded WAV (when arming) by timestamp.
The file is human-readable and updates while the song plays. Tail it with `Get-Content -Wait` on Windows or `tail -f` on macOS/Linux:
```jsonl
{"t":5.333,"s":0,"f":0,"hit":true,"ts":"OK","te":12,"pe":3,"cnf":0.94}
{"t":6.000,"s":1,"f":0,"hit":false,"ts":"EARLY","te":-180,"cnf":0.71}
{"t":6.667,"s":2,"f":0,"hit":true,"ts":"OK","te":-20,"pe":8}
```
This is the lowest-friction way to share a session with a collaborator: they don't need to wait for the song to end, you don't need to upload anything — the file lives in the bind-mounted `static/` tree, so any host-side process can read it during play.
Limitations:
- Streaming is fire-and-forget; the POSTs don't block detection. A request failure is silently swallowed so the in-memory diagnostic stays the source of truth.
- File cap is 8 MB per session (a 3-minute song produces ~60 KB, so this is 100× headroom). Beyond the cap the route returns 413 and the in-memory log keeps growing.
- Disabled outside tuning mode — normal users pay no overhead.
### "Did my detector change improve things?" — the regression suite
For a single fixture, two ad-hoc harness runs work (see below). For real iteration where you want **all** your fixtures measured against a stored baseline, use the regression driver in the plugin:
```bash
cd plugins/note_detect
# One-time: copy the example, edit paths to point at your recordings.
cp tools/regression-fixtures.example.json tools/regression-fixtures.json
# Capture a baseline (do this BEFORE making any code changes).
npm run regression:update
# ...make detector changes...
# Re-measure against the baseline. Exit code 1 if any fixture regresses.
npm run regression:vs-baseline
```
The driver iterates each fixture, runs `harness.js`, and prints a table of `hits/total · pure · chordPartial · Δhits-vs-baseline`. Both the fixtures file and the baseline are gitignored — they reference your local recordings, which aren't portable across contributors. Commit them in your fork if you want CI, otherwise treat them as local state.
The same workflow works on any tuning change — A/V offset sweep, frame-size sweep, algorithm experiments. Just make sure the baseline was captured *before* the change you want to measure.
### "Did my detector change improve things?" — ad hoc
Same recording, same chart, two harness runs. Recipe assumes you're at the slopsmith repo root *and* that the Note Detection plugin is cloned at `plugins/note_detect/` per the README. The detector source lives in that nested plugin repo, which slopsmith's `.gitignore` excludes via `plugins/*/`, so the stash dance has to run **inside** the plugin repo — `git stash` from the slopsmith root would either bail out or, worse, stash unrelated slopsmith edits.
The stash dance below uses **`git stash push -u -m "..."`** to give the stash a known name *and* include untracked files. `-u` matters: if your detector change added a new module or fixture, an untracked-file-blind stash would leave it on disk during the "before" run and contaminate the baseline. The script then asserts a stash was actually created before popping (so a clean worktree doesn't silently pop someone else's WIP), wraps each step in **`set -euo pipefail`** so a failed `git stash pop` (e.g., conflict) aborts before the "after" harness records an invalid result, and uses `trap` to surface any failure with a clear message.
```bash
set -euo pipefail
PLUGIN_DIR=plugins/note_detect
HARNESS=$PLUGIN_DIR/tools/harness.js
STASH_MSG="harness-before-$$"
trap 'echo "harness recipe aborted — stash may still be in $PLUGIN_DIR (\"git -C $PLUGIN_DIR stash list\")" >&2' ERR
# Stash the detector edits inside the plugin repo, not the slopsmith root.
# -u also stashes untracked files (new modules, fixtures) so they don't
# leak into the "before" baseline. `|| true` only swallows the
# clean-worktree case, which the next line catches explicitly.
git -C "$PLUGIN_DIR" stash push -u -m "$STASH_MSG" || true
# Bail out cleanly if nothing was stashed — running the "before" against
# the same code as "after" would just produce identical numbers.
git -C "$PLUGIN_DIR" stash list | grep -q "$STASH_MSG" || { echo "no detector changes to stash in $PLUGIN_DIR — try again with edits in place"; exit 1; }
node $HARNESS --audio <wav> --chart <json> --out /tmp/before.json
# `stash pop` failures (e.g., conflicts that auto-merge can't resolve)
# now abort via set -e instead of silently rolling into the "after" run
# with a half-restored tree.
git -C "$PLUGIN_DIR" stash pop "$(git -C "$PLUGIN_DIR" stash list | grep "$STASH_MSG" | head -1 | cut -d: -f1)"
node $HARNESS --audio <wav> --chart <json> --out /tmp/after.json
node -e "
const fs = require('fs');
for (const [n, p] of [['before','/tmp/before.json'],['after','/tmp/after.json']]) {
const d = JSON.parse(fs.readFileSync(p,'utf8'));
console.log(n, d.summary, d.miss_breakdown);
}
"
```
If `summary.hits` went up *and* no miss-bin went up by more than ~1, ship it. If hits went up but `sharp/flat` went up too, you traded pure misses for pitch misses — investigate whether the tolerance shift makes sense.
### "Find the optimal A/V offset for this take"
Sweep:
```bash
HARNESS=plugins/note_detect/tools/harness.js
for AV in -100 -50 0 50 100 150 200; do
echo "=== av=$AV ==="
node $HARNESS --audio <wav> --chart <json> --av-offset-ms=$AV --out /tmp/sw_$AV.json | tail -1
done
```
Pick the highest hit count, then narrow in with finer steps. Cross-reference with `timing_error_ms_hits.median` — at the optimum it'll be close to zero.
### "Categorize misses by chart section"
Join the event log against the chart's `sections[]` to bin per-section hit rate. Useful for finding which exercises in the benchmark sloppak a tuning change improves or regresses.
### "Why is this specific note pure-missed?"
Find the note's `t` in the chart, then grep the event log for entries near that time. If `cnf` is 0 for every nearby event, the detector never fired confidently — likely a YIN buffer / confidence issue. If `cnf > 0` but `dx` doesn't match `ex`, pitch detection is firing on a different note (octave error, harmonic, neighbour string).
## Reference
The Note Detection plugin lives in its own repository — these links go to the canonical source at github.com. If you've cloned the plugin into a local `plugins/note_detect/` next to this repo, the same files are at the equivalent path on disk.
- Plugin source: [`screen.js`](https://github.com/byrongamatos/slopsmith-plugin-notedetect/blob/main/screen.js) — `matchNotes`, `checkMisses`, `_diagTimingErrors` / `_diagTimingErrorsHits`, `getDiagnostic`.
- Routes: [`routes.py`](https://github.com/byrongamatos/slopsmith-plugin-notedetect/blob/main/routes.py) — the `/api/plugins/note_detect/recording` and `/api/plugins/note_detect/live-judgment` endpoints.
- Harness: [`tools/harness.js`](https://github.com/byrongamatos/slopsmith-plugin-notedetect/blob/main/tools/harness.js).
- Regression driver: [`tools/regression.js`](https://github.com/byrongamatos/slopsmith-plugin-notedetect/blob/main/tools/regression.js).
- Benchmark builder: [docs/benchmarks/note_detect_v1/build_benchmark.py](benchmarks/note_detect_v1/build_benchmark.py).
- Settings UI: [`settings.html`](https://github.com/byrongamatos/slopsmith-plugin-notedetect/blob/main/settings.html) — A/V auto-calibrate panel, tuning-mode toggle, diagnostic block.
docs/player-2d.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 75 KiB

docs/player-3d.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 212 KiB

docs/player-3d.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.9 MiB

docs/plugin-capability-inventory.md
+257
View File
@@ -0,0 +1,257 @@
# Plugin Capability Inventory
This report inventories the currently included plugins staged in `plugins/` and maps their observed behavior to Slopsmith capability domains. It is intended to inform the capability roadmap and the next migration specs now that PR1, the audio graph/session slice, playback, and audio-effects are active capability surfaces.
## Scope And Method
- Inventory source: 41 `plugin.json` manifests under `plugins/`, plus the first-party plugin repos that were migrated during the 008/playback and audio-effects work (`nam_tone`, `rig_builder`, Remote Library Client, and Remote Library Server).
- Verification pass: the original bundled-plugin scan found 25 plugins with backend `routes.py` and 14 plugins with `settings.html`. First-party plugin repos outside `plugins/` were checked separately from their current manifests and handoff docs.
- Most bundled plugin entries below are still inferred/recommended declarations. Current first-party manifests now declare active capability intent for `diagnostics`, `pipeline`, `library`, `audio-mix`, `audio-input`, `audio-monitoring`, `stems`, `playback`, `audio-effects`, `jobs`, and privileged capability inventory surfaces where their repos have already migrated.
- Manifest fields such as `nav`, `screen`, `settings`, `routes`, and `type: "visualization"` were treated as high-confidence evidence.
- Code patterns such as `window.slopsmithViz_*`, `window.playSong` wrappers, `window.showScreen` wrappers, `window.registerShortcut`, `window.slopsmithTour.register`, `window.slopsmith.audio.registerFader`, `highway.setNoteStateProvider`, and route/WebSocket handlers were treated as behavior evidence.
## Roadmap Baseline
The current roadmap already covers several surfaces implied by the plugin set:
| Roadmap State | Domains |
|---------------|---------|
| Active PR1 domains | `pipeline`, `diagnostics`, `library` |
| Active audio graph/session slice | `audio-mix`, `audio-input`, `audio-monitoring`, `stems` |
| Active playback slice | `playback` |
| Active audio-effects slice | `audio-effects` |
| Planned UI domains | `ui.navigation`, `ui.plugin-screens`, `ui.player-controls`, `ui.player-panels`, `ui.player-overlays`, `settings` |
| Planned player/runtime domains | `midi-control`, `tempo-clock` (`visualization` active as of the cap:6 slice; `note-detection` active as of the spec-009 slice) |
| Planned privileged domains | `backend.routes`, `plugins`, `jobs` |
The plugin inventory confirms these planned domains are directionally right. The main gaps are additional domain names or command scopes for library card injection, guided tours, keyboard shortcuts, media import/export, recording/capture, practice/session scoring, external services, and collaboration. `audio-effects` is no longer a missing candidate: it is an active provider-coordinator domain, with remaining work concentrated in provider adoption, bridge removal, and executor coverage.
## Executive Summary
- The most common plugin surface is a plugin screen backed by optional routes and settings. `ui.plugin-screens`, `backend.routes`, and `settings` should be treated as first-class migration targets.
- Player integrations are still heavily legacy-global today. `playback` now provides the active transport lifecycle/control plane, and `visualization` now provides the renderer provider/selection contract (legacy globals ride accounted shims), while `ui.player-overlays`, `ui.player-controls`, and `ui.player-panels` still need lifecycle and ordering contracts before wrappers can be retired.
- The audio domains promoted by the audio graph/session slice match real plugin behavior. `audio-mix`, `audio-input`, `audio-monitoring`, and `stems` are active; remaining work is mostly native provider adoption, bridge-hit cleanup, and cross-domain smoke coverage.
- `audio-effects` is active. Core owns provider/executor selection, route state, mapping index, constrained chain-plan validation, route gain/release, segment activation, stage bypass/parameter dispatch, fallback accounting, and redaction-safe diagnostics. Desktop owns trusted native execution. NAM Tone provides the baseline NAM+IR provider/executor path, while Rig Builder is the high-priority full-chain provider for NAM/VST/IR routes.
- `playback` is active as of the 008 slice, with redaction-safe target/settings keys and transport lifecycle diagnostics. Legacy `window.playSong` and song-event wrappers remain compatibility paths while plugins migrate requester/observer workflows.
- Long-running work is spread across conversion, update, import, preview, and studio plugins. The `jobs` domain should include progress, cancellation, terminal failure, and provider attribution.
- Several plugins perform privileged or externally mediated work: subprocesses, downloads, native audio bridges, plugin updates, and media conversion. These should stay out of broad capability activation until each surface has user confirmation, diagnostics redaction, and failure recovery.
## Current Manifest Declarations Since The First Pass
| Plugin Or Runtime | Current Declarations | Notes |
|-------------------|----------------------|-------|
| Core capability runtime | `pipeline`, `diagnostics`, `library`, `audio-mix`, `audio-input`, `audio-monitoring`, `stems`, `playback`, `audio-effects` | Active runtime owners/coordinators are registered by core, not by plugin manifests. |
| `capability_inspector` | `diagnostics`, `pipeline` | Bundled support surface for the graph and diagnostics snapshots. |
| `remote_library_client` | `library` provider | Declares provider operations for query, art, stats, tuning names, and sync. |
| `remote_library_server` | `library` requester/observer | Wraps the local provider for direct remote clients without claiming provider ownership. |
| `nam_tone` | `stems`, `audio-mix`, `audio-input`, `audio-monitoring`, `audio-effects`, `playback` | Baseline NAM+IR provider/executor and live guitar requester/observer path. |
| `rig_builder` | `library`, `audio-effects`, `playback`, `jobs`, privileged inventory, UI contributions | High-priority full-chain provider; library migration is complete for the Songs tab, while privileged routes and long-running work still need host-backed execution slices. |
| Desktop native executor | `audio-effects` executor | Trusted native execution for the `desktop-main` route: load/clear chain, activate segments, set stage bypass/parameters, set route gain, start audio, and release/mute routes. It does not own provider semantics. |
## Per-Plugin Mapping
| Plugin | Recommended Domains | Expected Roles | Roadmap Status | Confidence | Evidence |
|--------|---------------------|----------------|----------------|------------|----------|
| `app_tour_library` | `tours`, `ui.player-overlays` | provider, observer | Missing `tours`; overlay planned | High | Tour registration and screen-change observation. |
| `app_tour_settings` | `tours`, `ui.player-overlays` | provider, observer | Missing `tours`; overlay planned | High | Tour registration and screen-change observation. |
| `audio_engine` | `audio-monitoring`, `audio-effects`, `ui.plugin-screens`, `settings` | provider, requester | Audio monitoring/effects active; plugin adoption pending | Medium | Screen/settings surfaces and native/VST audio engine intent. |
| `drum_highway_3d` | `visualization`, `ui.player-overlays`, `settings`, `midi-control` | visualization provider, observer | Planned | High | `type: "visualization"`, WebGL renderer, settings surface, drum/MIDI use case. |
| `drums` | `visualization` | visualization provider | Planned | High | `type: "visualization"` and renderer script. |
| `editor` | `ui.plugin-screens`, `backend.routes`, `media-import-export`, `jobs` | screen provider, route provider, job provider | UI/routes/jobs planned; media domain missing | High | Screen plus backend routes for editing/import/export workflows. |
| `find_more` | `library`, `ui.plugin-screens`, `backend.routes`, `ui.library-card-injection`, `external-services` | requester/provider, route provider | Library active; card injection/external services missing | High | Screen/routes plus library discovery and card injection behavior. |
| `flappy_bend` | `ui.plugin-screens`, `backend.routes`, `practice-session` | screen provider, route provider | UI/routes planned; practice-session missing | Medium | Game screen and backend route surface. |
| `fretboard` | `ui.player-overlays` | overlay provider | Planned | High | Highway-state overlay pattern. |
| `guitar_theory` | `ui.plugin-screens`, `settings`, `practice-session` | screen provider, settings provider | UI/settings planned; practice-session missing | Medium | Screen/settings manifest surfaces for theory tools. |
| `highway_3d` | `visualization`, `ui.player-overlays`, `backend.routes`, `settings`, `audio-monitoring` | visualization provider, route provider, observer | Audio-monitoring active; visualization/UI/routes planned | High | `type: "visualization"`, WebGL renderer, routes/settings, analyser monitoring bridge. |
| `invert_highway` | `ui.player-overlays`, `settings`, `visualization` | overlay provider, observer | Planned | High | Settings surface and highway/playback wrapper behavior. |
| `jumpingtab` | `visualization`, `ui.player-overlays` | visualization provider, observer | Planned | High | `type: "visualization"`, renderer factory, highway visibility behavior. |
| `lyrics_karaoke` | `ui.plugin-screens`, `backend.routes`, `playback`, `recording` | screen provider, route provider, observer | Playback active; UI/routes planned; recording missing | High | Screen/routes plus karaoke timing and lyric/audio workflows. |
| `metronome` | `ui.player-overlays`, `audio-mix`, `playback`, `tempo-clock` | overlay provider, audio participant, observer | Audio-mix/playback active; overlay/tempo planned | High | Player overlay behavior, metronome audio, playback coupling. |
| `midi_amp` | `midi-control`, `ui.plugin-screens`, `backend.routes`, `settings`, `audio-effects` | MIDI provider, screen provider, route provider | MIDI/UI/routes planned; audio-effects active bridge target | High | Manifest id `midi_amp`, routes, settings, and MIDI amp workflow. |
| `minigames` | `ui.plugin-screens`, `backend.routes`, `settings`, `diagnostics`, `practice-session` | screen provider, route provider, diagnostics provider | Mostly planned/active; practice-session missing | High | Routes/settings, diagnostics files, minigame state. |
| `multiplayer` | `collaboration`, `ui.plugin-screens`, `backend.routes`, `playback`, `audio-mix` | collaboration provider, route provider, observer | Collaboration missing | Medium | Screen/routes and real-time multiplayer/audio mix behavior. |
| `rig_builder` | `library`, `playback`, `audio-effects`, `jobs`, `privileged-capabilities`, `ui.plugin-screens`, `backend.routes`, `media-import-export` | effects provider/requester/observer, screen provider, route provider | First-party manifest active; backend/jobs/UI hosts still incomplete | High | Full-chain NAM/VST/IR provider, core mapping writes, library provider routing, mega-chain playback, tone3000/import/export routes. |
| `nam_tone` | `audio-mix`, `audio-input`, `audio-monitoring`, `stems`, `audio-effects`, `playback`, `ui.plugin-screens`, `backend.routes`, `settings` | audio provider/requester/observer, effects provider/executor, screen provider, route provider | First-party manifest active; UI/routes/settings planned | High | Fader registration, input/monitoring graph, stem ducking, baseline NAM+IR provider/executor, model/IR routes/settings. |
| `note_detect` | `note-detection`, `audio-input`, `audio-monitoring`, `ui.player-overlays`, `backend.routes`, `settings`, `diagnostics` | note provider, audio requester, overlay provider | Audio input/monitoring active; note/UI/routes planned | High | `highway.setNoteStateProvider`, calibration/settings/routes, diagnostic workflow. |
| `piano` | `visualization` | visualization provider | Planned | High | `type: "visualization"` and renderer script. |
| `plugin_manager` | `plugins`, `ui.plugin-screens` | plugin lifecycle provider, screen provider | Planned | High | Plugin management screen and desktop bridge integration. |
| `practice_journal` | `practice-session`, `ui.plugin-screens`, `backend.routes` | practice provider, screen provider, route provider | Practice-session missing | High | Practice journal screen/routes. |
| `profileimport` | `media-import-export`, `ui.plugin-screens`, `backend.routes`, `jobs` | import provider, screen provider, route provider | Media domain missing; jobs planned | High | Profile import screen/routes. |
| `section_map` | `ui.player-overlays`, `playback` | overlay provider, observer | Planned | High | Highway section overlay behavior. |
| `setlist` | `library`, `playback`, `ui.plugin-screens`, `backend.routes` | requester/provider, screen provider, route provider | Library/playback active; UI/routes planned | High | Setlist screen/routes and song selection/playback workflow. |
| `sloppak_converter` | `media-import-export`, `jobs`, `library`, `ui.plugin-screens`, `backend.routes`, `ui.library-card-injection` | conversion provider, job provider, route provider | Library active; jobs/UI/routes planned; media/card missing | High | Converter routes, queue UI, library card actions, conversion jobs. |
| `slopscale` | `ui.plugin-screens`, `backend.routes`, `settings`, `visualization` | screen provider, route provider, observer | Planned | High | Routes/settings and 3D highway visualization observation. |
| `song_preview` | `playback`, `audio-mix`, `ui.plugin-screens`, `backend.routes`, `settings` | preview provider, route provider, audio participant | Playback/audio-mix active; UI/routes planned | Medium | Preview screen/routes/settings and audio preview behavior. |
| `splitscreen` | `ui.player-panels`, `ui.player-overlays`, `visualization`, `playback`, `keyboard-shortcuts`, `settings` | panel provider, observer, shortcut provider | Playback active; UI/visualization planned; shortcuts missing | High | Multi-highway panels, playback/screen wrappers, panel shortcuts/settings. |
| `stem_mixer` | `stems`, `audio-mix`, `ui.plugin-screens`, `backend.routes`, `settings`, `jobs` | stem provider, mixer provider, route provider | Audio active; jobs planned | High | Stems mixer routes/settings and stem/audio mix ownership. |
| `step_mode` | `ui.player-overlays`, `playback`, `settings`, `keyboard-shortcuts` | overlay provider, observer, shortcut provider | Shortcuts missing | Medium | Player overlay/settings and step-practice behavior. |
| `studio` | `audio-mix`, `audio-input`, `audio-monitoring`, `recording`, `media-import-export`, `jobs`, `ui.plugin-screens`, `backend.routes` | DAW provider, route provider, job provider | Audio active; jobs/UI/routes planned; recording/media missing | Medium | Studio screen/routes, multitrack recording/mixing workflows. |
| `tab_import` | `media-import-export`, `ui.plugin-screens`, `backend.routes`, `jobs` | import provider, route provider, job provider | Media missing; jobs planned | High | Tab import screen/routes. |
| `tabview` | `visualization`, `backend.routes` | visualization provider, route provider | Planned | High | `type: "visualization"` and backend tab routes. |
| `themes` | `settings`, `ui.theme` | theme provider | Settings planned; theme domain missing | Medium | Global settings/routes for theming. |
| `tones` | `audio-effects`, `playback`, `ui.plugin-screens`, `backend.routes` | tone provider, playback observer, route provider | Audio-effects/playback active; provider adoption pending | High | Tone screen/routes and playback wrapper behavior. |
| `transpose-chords` | `ui.player-overlays`, `visualization`, `playback` | overlay provider, highway observer | Planned | High | Chord/highway reader and playback wrapper behavior. |
| `tutorials` | `tours`, `ui.plugin-screens`, `backend.routes`, `settings` | tutorial provider, route provider | Tours missing | Medium | Tutorial screen/routes/settings and guided content. |
| `update_manager` | `plugins`, `jobs`, `ui.plugin-screens`, `backend.routes`, `external-services` | update provider, job provider, route provider | Plugins/jobs planned; external services missing | Medium | Update screen/routes and desktop/network integration. |
## Domain Coverage Summary
| Domain | Approximate Plugin Count | Roadmap Fit | Notes |
|--------|--------------------------|-------------|-------|
| `ui.plugin-screens` | 24 | Planned | Main extension surface; should include screen lifecycle, visibility, focus, and teardown. |
| `backend.routes` | 25 route files | Planned privileged | Needs route diagnostics, plugin attribution, and privilege review. |
| `settings` | 14 | Planned sensitive | Should cover contribution metadata and backup/import allowlists without exposing values. |
| `visualization` | 6 declared providers plus observers | Planned | Existing renderer factory contract is mature enough to formalize. |
| `ui.player-overlays` | 14 | Planned | Needs overlay placement, visibility, z-order, and coexistence policy. |
| `audio-mix` | 6+ | Active | Runtime control plane exists; migration work is native participant coverage and legacy fader bridge removal gates. |
| `audio-input` | 4+ | Active | Needs broader provider coverage across browser, Desktop, and native paths. |
| `audio-monitoring` | 5+ | Active | Needs broader provider coverage and cross-domain failure smoke tests. |
| `stems` | 3 | Active coordinated provider | Current coordinator/provider split matches plugin ownership. |
| `library` | 3+ | Active | Needs to account for library card actions separately from browsable providers. |
| `jobs` | 7+ | Planned privileged | Conversion/import/update/studio work all need a common job model. |
| `playback` | 9+ | Active | Wrapper chains should migrate to transport commands, requester/observer declarations, and lifecycle events. |
| `note-detection` | 1 | Planned sensitive | Current provider is high-impact enough for a focused spec. |
| `midi-control` | 2 | Planned sensitive | Needs consent, device redaction, and mapping diagnostics. |
| `tempo-clock` | 1+ | Planned | Metronome and practice tools imply clock source/consumer semantics. |
| `plugins` | 2 | Planned privileged | Plugin manager/update manager require confirmation and rollback. |
| `diagnostics` | 2+ | Active | Existing diagnostics contributions should become easier to inspect by domain. |
| `ui.library-card-injection` | 2+ | Missing | Library card buttons/actions are distinct from library source providers. |
| `tours` | 4 | Missing | Guided tours behave like UI overlays with screen navigation coupling. |
| `keyboard-shortcuts` | 2+ | Missing | Existing global shortcut registry needs contribution and conflict policy. |
| `media-import-export` | 6+ | Missing | Import/export/conversion is broader than `jobs` and often uses privileged backend routes. |
| `recording` | 2+ | Missing | Studio and karaoke workflows need capture/session semantics. |
| `audio-effects` | 5+ | Active | Core host, mapping index, provider/executor registry, route gain/release, segment/stage controls, fallback, and diagnostics exist; remaining work is provider adoption and bridge removal. |
| `practice-session` | 4+ | Missing | Practice journal, minigames, theory, and note detection imply scoring/progression state. |
| `collaboration` | 1 | Missing | Multiplayer needs its own trust, identity, and sync model. |
| `external-services` | 3+ | Missing or safety inventory | Network/download/subprocess integrations may be better tracked as safety metadata than as one capability. |
## Operation And Event Gaps
### High Priority
| Domain | Missing Or Under-Specified Surface |
|--------|------------------------------------|
| `playback` | Migrate legacy wrapper users onto active `start`, `pause`, `resume`, `stop`, `seek`, `set-loop`, `clear-loop`, and `inspect` commands plus playback lifecycle events. Keep compatibility bridge hits visible until normal playback/loop/route smoke runs are clean. |
| `audio-effects` | Finish provider adoption: NAM Tone remains the fallback provider/executor and owns the legacy player `Chain` control; Rig Builder owns full-chain routes while enabled/pending/failed/active; Desktop owns trusted native execution only. Remaining gaps are host-backed active mapping read paths, replacement of old fetch/native-load bridges, additional provider declarations for `tones`/`midi_amp`/`audio_engine`, and zero-leak diagnostics around asset refs. |
| `jobs` | Add `register-provider`, `enqueue`, `list`, `inspect`, `cancel`, `pause`, `resume`, and `retry`. Emit `queued`, `started`, `progress`, `log`, `completed`, `failed`, `cancelled`, and `provider-unavailable`. |
| `note-detection` | Add provider registration, active provider selection, note-state provider lifecycle, input binding, hit/miss/state events, calibration diagnostics, and performance-data redaction. |
### Medium Priority
| Domain | Missing Or Under-Specified Surface |
|--------|------------------------------------|
| `ui.plugin-screens` | Define contribution registration, mount/unmount, visibility, focus, navigation, teardown, and rehydration policy. |
| `ui.player-overlays` | Define surface, anchor, z-order, visibility, teardown, hit-testing, and renderer compatibility flags. |
| `ui.player-panels` | Define panel registration, per-panel renderer state, focus, shortcut scope, layout constraints, and teardown. |
| `ui.player-controls` | Define ordered contribution regions, command buttons, popovers, sliders, disabled states, and conflict policy. |
| `visualization` | Formalize provider registration, `contextType`, `matchesArrangement`, `panelControls`, per-panel selection, fallback/revert events, and renderer failure diagnostics. |
| `media-import-export` | Add import/export job requests, accepted file types, source trust metadata, generated artifact paths, and cleanup/failure semantics. |
| `audio-input` | Broaden native/browser provider adoption and test denied/unavailable/degraded/failure outcomes without raw device labels or hardware ids. |
| `audio-monitoring` | Broaden native/browser provider adoption, direct-monitor policy coverage, startup barrier accounting, and safe input-level summaries. |
### Lower Priority Or Cross-Cutting
| Domain | Missing Or Under-Specified Surface |
|--------|------------------------------------|
| `settings` | Add settings contribution metadata, export/import participation, settings schema hints, redaction class, and per-plugin backup diagnostics. |
| `plugins` | Add install/enable/disable/update commands with user confirmation, rollback, disabled-handler enforcement, and desktop bridge failure recovery. |
| `midi-control` | Add device enumerate/open/close, message send/listen, mapping registration, consent, and redacted diagnostics. |
| `tempo-clock` | Add tempo provider registration, BPM/time-signature changes, beat events, metronome tick state, and consumer subscription. |
| `keyboard-shortcuts` | Add shortcut contribution registration, scope, conflict resolution, enable/disable, and help-panel metadata. |
| `tours` | Add tour registration, eligibility, start/stop, step lifecycle, target resolution, and screen navigation dependency declarations. |
| `ui.library-card-injection` | Add card action registration, placement, enabled/disabled state, per-provider applicability, and action-result events. |
| `recording` | Add arm/start/stop capture, take upload/import, latency metadata, capture-source binding, and storage cleanup. |
| `practice-session` | Add session start/stop, goal registration, score/progress events, chart segment focus, and journal persistence boundaries. |
| `collaboration` | Add room/session lifecycle, participant identity redaction, shared playback sync, conflict policy, and disconnect recovery. |
## Audio Domain Notes
The audio graph/session and audio-effects slices should stay scoped to coordination, dispatch, and redaction-safe diagnostics. This inventory reinforces four follow-up requirements:
1. `audio-mix` has a control plane, but native provider coverage and duplicate native-over-legacy cleanup still need release gates before the legacy fader registry can become compatibility-only.
2. `stems` should remain coordinated by core but owned by the active Stems provider. Stem playback, mute/restore semantics, availability, and per-stem state belong to the provider.
3. `audio-input` and `audio-monitoring` should cover both browser and Desktop/native paths without leaking raw device labels, source ids, or capture details in diagnostics.
4. `audio-effects` should keep provider semantics out of Desktop native code. Core selects providers/executors and coordinates route lifecycle; providers resolve opaque chain plans and private trusted asset maps; Desktop only executes the validated native requests and must release/mute routes cleanly. Provider-positive UI should follow route ownership: the legacy player `Chain` control belongs to `nam_tone` and should stay hidden while Rig Builder owns tones.
## Recommended Roadmap Updates
1. Finish audio-effects provider adoption and bridge removal gates: NAM Tone and Rig Builder should read/write the core mapping index first, route playback through the active provider/executor path, and reserve legacy fetch/native-load/DB bridges for fallback/import only.
2. Move long-running and privileged plugin work behind host-owned `jobs` and privileged-operation inventories before migrating converter/import/update/studio/Rig Builder routes. This gives backend work a shared cancellation/progress/error and approval model.
3. Promote `note-detection` as its own sensitive provider domain. It touches audio input, monitoring, visualization feedback, calibration, and diagnostics.
4. Create a UI contribution host spec that includes `ui.navigation`, `ui.plugin-screens`, `ui.player-controls`, `ui.player-overlays`, `ui.player-panels`, `keyboard-shortcuts`, and possibly `tours`.
5. Add missing candidate domains or safety inventories for `media-import-export`, `recording`, `practice-session`, `collaboration`, `ui.library-card-injection`, and `external-services`.
6. Revisit active playback consumers and retire wrapper-only tone/overlay/visualizer integrations once requester/observer declarations and compatibility bridge counts show clean normal playback.
## Suggested Manifest Direction
When these plugins migrate, manifests should describe intent even before runtime handlers hydrate. For example, a visualization plugin might declare:
```json
{
"standards": ["capability-pipelines.v1", "plugin-runtime-idempotent.v1"],
"capabilities": {
"visualization": {
"roles": ["provider"],
"operations": ["renderer.create", "renderer.destroy", "renderer.inspect"],
"emits": ["renderer-ready", "renderer-failed"],
"mode": "optional",
"compatibility": "legacy-window-shim",
"safety": "safe"
}
}
}
```
An audio plugin that participates in the active audio-session domains should declare requester/provider relationships more explicitly:
```json
{
"capabilities": {
"audio-mix": {
"roles": ["provider"],
"operations": ["fader.get-value", "fader.set-value"],
"emits": ["fader-value-changed"],
"mode": "active",
"compatibility": "legacy-window-shim",
"safety": "safe"
},
"stems": {
"roles": ["requester", "observer"],
"requests": ["mute", "restore", "inspect"],
"observes": ["owner-available", "automation-applied", "automation-restored", "automation-overridden", "claim-orphaned"],
"mode": "active",
"compatibility": "none",
"safety": "safe"
}
}
}
```
For active domains, command and operation names should follow [capability-domains.md](capability-domains.md) and the relevant host module. For deferred domains, these examples remain direction markers rather than current contracts.
## Highway String Colors (data-plane API)
User-customizable per-string highway colors (the "Highway String Colors" setting in the 3D Highway plugin's panel) are **not** a capability domain. Consistent with `capability-domains.md` keeping highway-rendering and `visualization` surfaces off the capability graph until a dedicated render-facade slice lands, they are exposed as a synchronous **data-plane** API on `window.slopsmith.highwayColors` plus a change event. Visualization/overlay plugins (custom highways, minigames, fretboard widgets) should read colors from here so their gems/strings match the user's theme.
Colors are keyed by **named string slot**, not raw index, so a string keeps its color across arrangements (Low E stays Low E's color on a 6-string guitar, 4-string bass, or 7/8-string, where the extra low strings use the `low7`/`low8` slots). Slots: `highE`, `B`, `G`, `D`, `A`, `lowE`, `low7` (7-string Low B), `low8` (8-string Low F#).
`window.slopsmith.highwayColors` (`version: 1`):
| Member | Returns | Purpose |
|--------|---------|---------|
| `slots` | `[{key,label,sub}]` | Ordered named slots (stable `key`s). |
| `get()` | `{slot:hex}` | User overrides only (`{}` = pure defaults). |
| `getDefaults()` | `{slot:hex}` | Canonical default color per slot. |
| `getResolved()` | `{slot:hex}` | Defaults overlaid with overrides — colors in effect by name. |
| `keysForChart(sc, isBass)` | `[slotKey…]` | Which slot each chart string index maps to (index 0 = lowest). |
| `toEffective(sc?, isBass?)` | `[hex…]` | Per-string-**index** colors for an arrangement (resolved). Omit args for the loaded chart. |
| `getCurrent()` | `[hex…]` | Per-index colors actually applied to the live 2D highway right now. |
| `apply(slotMap)` | — | Set colors (persists + applies to both highways); `null`/`{}` reverts. |
| `encodeShare(name, map)` / `decodeShare(code)` | `string` / `{name,colors}` | The `SLOPHWY2.` copy/paste share format. |
| `onChange(fn)` / `offChange(fn)` | unsubscribe fn | `fn(resolvedMap)` fires on any color change (also on song load when the slot→index mapping shifts). |
The underlying change event is `window.slopsmith.emit('highway:stringColors', …)`; `onChange` wraps it and hands back the resolved map. The raw `window.highway.getStringColors()` data-plane accessor (per-index) remains available for renderers that only need the current applied array. When a `visualization` capability slice eventually lands, this facade is the natural thing to fold into it.
## Validation Notes
This report should be revisited after more bundled plugins adopt manifest capability declarations. The document now separates current first-party declarations from inferred legacy behavior; the next revision should turn each active-domain row into a removal-gate checklist with Capability Inspector smoke output, bridge-hit expectations, and support-bundle redaction checks.
docs/plugin-manifest.schema.json
+64
View File
@@ -0,0 +1,64 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://slopsmith.local/contracts/plugin-manifest-capabilities.schema.json",
"title": "Slopsmith Plugin Manifest Capability Contract",
"type": "object",
"required": ["id", "name"],
"properties": {
"id": { "type": "string", "minLength": 1, "pattern": "^[A-Za-z0-9_.-]+$" },
"name": { "type": "string", "minLength": 1 },
"version": { "type": ["string", "null"] },
"private": { "type": "boolean" },
"standards": { "type": "array", "items": { "type": "string", "minLength": 1 }, "uniqueItems": true },
"capability_api": {
"type": "object",
"properties": { "standard": { "const": "capability-pipelines.v1" }, "version": { "const": 1 } },
"additionalProperties": false
},
"capabilities": {
"type": "object",
"propertyNames": { "$ref": "#/$defs/domainName" },
"additionalProperties": { "$ref": "#/$defs/capabilityDeclaration" }
},
"ui_contributions": { "type": "object", "propertyNames": { "$ref": "#/$defs/domainName" }, "additionalProperties": { "$ref": "#/$defs/contributionList" } },
"ui": { "type": "object", "propertyNames": { "$ref": "#/$defs/domainName" }, "additionalProperties": { "$ref": "#/$defs/contributionList" } },
"runtime_domains": { "type": "object", "propertyNames": { "$ref": "#/$defs/domainName" }, "additionalProperties": { "$ref": "#/$defs/domainDeclaration" } },
"domains": { "type": "object", "propertyNames": { "$ref": "#/$defs/domainName" }, "additionalProperties": { "$ref": "#/$defs/domainDeclaration" } },
"settings_schema": { "type": "object" },
"nav": {}, "screen": {}, "script": {}, "routes": {}, "settings": {}, "diagnostics": {}, "type": { "type": "string" }, "tour": {},
"description": { "type": "string", "description": "Short one-sentence summary of the plugin, surfaced on the v3 Pedalboard Plugins page (clamped to ~2 lines). Optional and additive." },
"category": { "type": "string", "description": "Which pedalboard the plugin sits on in the v3 Plugins page. Suggested values: 'audio', 'creation', 'practice', 'game', 'tools'. Free-form; unknown/absent values fall back to a curated default then 'other'. Optional and additive." },
"icon": { "type": "string", "minLength": 1, "pattern": "^assets/(?!.*\\.\\.)[^\\\\?#]+$", "description": "Plugin-root-relative path under assets/ (e.g. 'assets/thumb.png') to a thumbnail (~square, ~256x256 PNG/SVG) shown as the pedal graphic on the v3 Plugins page. Same containment rule as `styles`. If omitted, the loader auto-detects assets/thumb.png; failing that the UI shows a default pedal graphic. Optional and additive." },
"styles": { "type": "string", "minLength": 1, "pattern": "^assets/(?!.*\\.\\.)[^\\\\?#]+$", "description": "Plugin-root-relative path under assets/ (e.g. 'assets/plugin.css') to a compiled, preflight-off Tailwind stylesheet the frontend injects as a <link>. Must stay under assets/ with no '..', backslash, or query/fragment. See docs/plugin-styles.md." }
},
"additionalProperties": true,
"$defs": {
"domainName": { "type": "string", "minLength": 1, "pattern": "^[A-Za-z0-9_.:-]+$" },
"capabilityDeclaration": {
"type": "object",
"properties": {
"roles": { "type": "array", "items": { "enum": ["owner", "provider", "observer", "requester", "transformer", "handler", "validator", "short-circuiter", "contributor"] }, "uniqueItems": true },
"commands": { "type": "array", "items": { "type": "string", "minLength": 1 }, "uniqueItems": true },
"operations": { "type": "array", "items": { "type": "string", "minLength": 1 }, "uniqueItems": true },
"requests": { "type": "array", "items": { "type": "string", "minLength": 1 }, "uniqueItems": true },
"observes": { "type": "array", "items": { "type": "string", "minLength": 1 }, "uniqueItems": true },
"emits": { "type": "array", "items": { "type": "string", "minLength": 1 }, "uniqueItems": true },
"events": { "type": "array", "items": { "type": "string", "minLength": 1 }, "uniqueItems": true },
"kind": { "enum": ["command", "provider-coordinator", "event", "diagnostic", "privileged"] },
"mode": { "enum": ["active", "optional", "legacy-shim", "disabled"] },
"compatibility": { "enum": ["none", "shim-allowed", "degrade-noop", "required", "legacy-window-shim"] },
"ownership": { "enum": ["exclusive-owner", "multi-provider", "observer-only", "requester-only", "privileged", "diagnostic-only"] },
"safety": { "enum": ["safe", "privileged", "sensitive", "diagnostic-only"] },
"order": { "type": "object", "properties": { "fixed": { "type": "boolean" }, "before": { "type": "array", "items": { "type": "string", "minLength": 1 }, "uniqueItems": true }, "after": { "type": "array", "items": { "type": "string", "minLength": 1 }, "uniqueItems": true } }, "additionalProperties": false },
"provider_policy": { "type": "object" },
"settings": { "type": "array", "description": "Declarative per-instance control descriptors (toggle / range / select) a participant exposes for a consuming host to render generically. Domain-agnostic in shape; how a host applies a chosen value is defined by each capability domain's contract (the visualization domain requires an applySetting(key, value) method on the renderer instance).", "items": { "type": "object", "required": ["key", "type"], "properties": { "key": { "type": "string", "minLength": 1 }, "label": { "type": "string" }, "type": { "enum": ["toggle", "range", "select"] }, "default": {}, "min": { "type": "number" }, "max": { "type": "number" }, "step": { "type": "number" }, "options": { "type": "array", "items": { "type": "object", "required": ["id"], "properties": { "id": { "type": "string", "minLength": 1 }, "label": { "type": "string" } }, "additionalProperties": false } } }, "additionalProperties": false } },
"description": { "type": "string" },
"summary": { "type": "string" },
"version": { "const": 1 }
},
"additionalProperties": false
},
"domainDeclaration": { "oneOf": [{ "type": "object", "properties": { "role": { "type": "string", "minLength": 1 }, "roles": { "type": "array", "items": { "type": "string", "minLength": 1 } }, "ownership": { "enum": ["exclusive-owner", "multi-provider", "observer-only", "requester-only", "privileged", "diagnostic-only"] }, "safety": { "enum": ["safe", "privileged", "sensitive", "diagnostic-only"] }, "legacy_source": { "type": "string", "minLength": 1 } }, "additionalProperties": true }, { "type": "array" }] },
"contributionList": { "type": "array", "items": { "type": "object", "required": ["id"], "properties": { "id": { "type": "string", "minLength": 1 }, "region": { "type": "string", "minLength": 1 }, "label": { "type": "string" }, "order": { "type": ["number", "integer", "string"] } }, "additionalProperties": true } }
}
}
docs/plugin-styles.md
+142
View File
@@ -0,0 +1,142 @@
# Plugin styling — the `styles` capability
> Building for the redesigned **v3 UI** (`SLOPSMITH_UI=v3` / `/v3`)? v3 uses `fb-*`
> design tokens and a restructured player chrome with a dedicated plugin-control
> slot. See **[plugin-v3-ui.md](plugin-v3-ui.md)** for the player-chrome contract
> plugins must follow in v3.
Slopsmith serves Tailwind as a **prebuilt** stylesheet
(`static/tailwind.min.css`), never the runtime Play CDN. The CDN's on-the-fly
JIT rescanned the DOM on the main thread and dropped ~26% of frames with the 3D
highway running (slopsmith-desktop#110). See **constitution Principle II**.
A prebuilt stylesheet only contains the classes the build scanner saw in **core
source at core build time**. That has a consequence for plugins:
- Core's build scans bundled plugins on disk, but **a plugin installed at
runtime** (community / NAS) was never scanned. Its classes — especially
arbitrary values like `text-[11px]`, `grid-cols-[1fr_auto]`,
`shadow-[0_0_8px_rgba(0,0,0,.5)]` — are **absent** from the served CSS, so its
UI renders unstyled.
The `styles` capability fixes this: your plugin ships its **own** compiled
stylesheet and declares it in the manifest. The frontend injects one versioned
`<link rel="stylesheet">` into `<head>` when your plugin activates, covering both
your screen and your settings panel.
> You only need this if you use Tailwind classes that aren't guaranteed in core —
> in practice, **any arbitrary-value class** (`w-[37px]`), or a custom class core
> doesn't ship. If you use only common core utilities (`flex`, `p-4`,
> `text-gray-300`, `bg-dark-600`), you can omit `styles` and rely on core's CSS.
## 1. Declare it in `plugin.json`
```json
{
"id": "my_plugin",
"name": "My Plugin",
"version": "1.2.0",
"screen": "screen.html",
"script": "screen.js",
"styles": "assets/plugin.css"
}
```
`styles` is a **plugin-root-relative path that must live under `assets/`** (like
`screen`/`script`/`routes` are root-relative). It serves through the sandboxed
`/api/plugins/<id>/assets/...` route, so the file must be at
`<plugin>/assets/plugin.css`. The injected `<link>` is cache-busted with
`?v=<version>`, so **bump your manifest `version`** whenever you rebuild the CSS,
or browsers may serve a stale copy within a session.
## 2. Build the stylesheet — utilities only, `preflight: false`
Core already ships Tailwind's base reset (preflight) once. Your plugin must
**not** re-apply it, or it would double the reset and fight core's styles. Build
with `corePlugins: { preflight: false }` so your sheet emits **only the utility
classes your files use**.
`tailwind.config.js` (in your plugin repo):
```js
/** Plugin stylesheet build — utilities only, scanned from this plugin's files.
* Regenerate assets/plugin.css with: bash build-tailwind.sh */
module.exports = {
corePlugins: { preflight: false }, // core owns the single base reset
content: [
'./screen.js',
'./settings.html',
'./screen.html',
// add any other file that carries Tailwind classes (e.g. './tour.json')
],
theme: {
extend: {
// Re-declare any core theme tokens you reference so they compile here.
colors: {
dark: { 900: '#050508', 800: '#0a0a12', 700: '#10101e', 600: '#181830', 500: '#1e1e3a' },
accent: { DEFAULT: '#4080e0', light: '#60a0ff', dark: '#2060b0' },
gold: '#e8c040',
},
fontFamily: { display: ['"Inter"', 'system-ui', 'sans-serif'] },
},
},
// Mirror only the dynamically-built classes your code generates at runtime
// (Tailwind can't see them textually). Drop this if you have none.
safelist: [
{ pattern: /^(bg|text|border)-(dark|accent)(-.+)?$/ },
],
plugins: [],
};
```
Input CSS — **`@tailwind utilities;` only** (no `@tailwind base`, that's the
preflight you're disabling):
`_plugin.src.css`:
```css
@tailwind utilities;
```
Build script `build-tailwind.sh` (run at your plugin's release time — the output
is committed; end users never build):
```bash
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"
# Pin the same Tailwind 3.x core uses so output stays diff-stable.
exec npx -y tailwindcss@3.4.19 \
-c tailwind.config.js \
-i _plugin.src.css \
-o assets/plugin.css \
--minify
```
```bash
bash build-tailwind.sh # writes assets/plugin.css — commit it
```
## 3. Rules (inherited from the constitution)
- **No Play CDN, no runtime CSS JIT** — anywhere, ever. Same rule that binds core.
- **`preflight: false`** — utilities only; core ships the one base reset.
- **`styles` under `assets/`** — it serves through the sandboxed asset route;
`..`, absolute paths, and NUL bytes are rejected by `safe_join`.
- **Bump `version` on every CSS rebuild** so the `?v=` cache-buster fetches fresh.
- Plugins without `styles` are unaffected and inject no `<link>`.
## How it works (for reference)
- The loader derives a manifest-only `has_styles` boolean and passes the `styles`
path through to `/api/plugins` — no plugin code is imported
(`plugins/__init__.py::_nav_entry`).
- The frontend (`static/app.js::_injectPluginStyles`) injects one
`<link rel="stylesheet" data-plugin-id data-plugin-version
href="/api/plugins/<id>/assets/plugin.css?v=<version>">` (the `styles` value —
e.g. `assets/plugin.css` — appended to `/api/plugins/<id>/`) into `<head>`, **before** the
screen markup so styles are present on first paint. It's deduped by version: a
plugin upgrade swaps the old `<link>` for the new one; re-activation never piles
up duplicates.
- The stylesheet is served by the existing
`/api/plugins/<id>/assets/<path>` route as `text/css`.
docs/plugin-v3-ui.md
+191
View File
@@ -0,0 +1,191 @@
# Building plugins for the v3 UI (fee[dB]ack v0.3.0)
v0.3.0 ("fee[dB]ack") ships a redesigned UI **behind a flag** — `SLOPSMITH_UI=v3`
or the `/v3` route. The classic UI (v2) remains the default until 0.3.0 ships, so
plugins must work in **both**.
The good news: v3 **reuses the same engine** as v2 — same `server.py`, `app.js`,
`highway.js`, `playSong`, `showScreen`, capability registry, library providers,
and the `window.slopsmithViz_<id>` / `setRenderer` visualization contract. So your
plugin's **backend, capabilities, library providers, `nav`/`screen`, visualization
renderers, diagnostics, and settings export all work unchanged in v3.** v3 surfaces
your `nav` entry in the new sidebar (via `shell.js` `renderPluginNav`) and your
screen mounts exactly as before.
**The one thing that changed is the player chrome** — and only if your plugin
injects controls into it.
## What changed in the player
In v2, `#player-controls` was a wide, **always-visible** bottom bar. In v3 it
became a **minimal, auto-hiding centered transport** (it fades ~2.5 s after the
pointer goes still during playback), flanked by a **hover-reveal left icon rail**
with popovers.
So the legacy way of injecting a control breaks in v3 two ways:
1. **Auto-hide** — a button you append to `#player-controls` vanishes with the
transport after a couple seconds.
2. **Dead anchors** — legacy code commonly inserts before a `<span class="text-gray-700">`
separator or the `button:last-child` (the ✕ Close button). **Neither exists in
the v3 transport**, so your control lands at the wrong end or is unreachable.
## The contract: detect v3, mount into the plugin-control slot
The host exposes:
- `window.slopsmith.uiVersion === 'v3'` — detect v3 (absent / not `'v3'` in v2).
- `window.slopsmith.ui.playerControlSlot()` — returns a **stable, always-reachable
container** (the "Plugins" rail popover). In v3, append your control(s) here
instead of `#player-controls`.
Canonical pattern for any control you inject into the player:
```js
function playerSlot() {
return (window.slopsmith && window.slopsmith.uiVersion === 'v3'
&& window.slopsmith.ui && typeof window.slopsmith.ui.playerControlSlot === 'function')
? window.slopsmith.ui.playerControlSlot() : null;
}
function injectMyButton() {
const slot = playerSlot();
const controls = slot || document.getElementById('player-controls'); // v3 slot, else v2 bar
if (!controls) return;
if (myBtn && controls.contains(myBtn)) return; // guard the ACTUAL container
// Legacy inserts before a separator / the ✕ Close button; the v3 slot has no
// such anchor, so just append there.
const anchor = slot ? null : controls.querySelector('span.text-gray-700, button:last-child');
myBtn = document.createElement('button');
/* ... */
if (anchor) controls.insertBefore(myBtn, anchor); else controls.appendChild(myBtn);
}
```
Rules:
- **Gate v3 behavior on `uiVersion`** so v2 is byte-for-byte unchanged.
- **Never** `insertBefore` the legacy `span.text-gray-700` separator or
`button:last-child` — they don't exist in the v3 transport. Append instead.
- **Guard idempotency against the actual container** (`controls.contains(myBtn)`),
not a hard-coded `#player-controls` — otherwise re-injection logic breaks in v3.
- **Dropdowns/panels** your control opens: position them via the trigger's
`getBoundingClientRect()` (portal to `document.body` or `#player`), **not**
relative to `#player-controls` — the trigger now lives in the rail popover.
- **Overlays/HUDs/canvases** you attach to `#player` keep working; just keep their
`z-index` **under the chrome layers**: transport/HUD `z-20`, rail `z-30`,
popovers `z-40`.
## Pedalboard metadata (icon, description, category)
The v3 **Plugins page** renders each plugin as a guitar **pedal** grouped onto
category **pedalboards**. To make your pedal look good, declare three optional,
additive manifest fields (all surfaced in `/api/plugins`):
```json
{
"id": "my_plugin",
"name": "My Plugin",
"description": "One short sentence shown under the pedal name.",
"category": "audio",
"icon": "assets/thumb.png"
}
```
- **`description`** — one short sentence (clamped to ~2 lines on the pedal).
- **`category`** — which board the pedal sits on. Suggested:
`audio | creation | practice | game | tools`. Unknown/absent → a curated default
then `"other"`.
- **`icon`** — assets-relative thumbnail (~square, ~256×256 PNG/SVG), served via
the existing sandboxed `/api/plugins/<id>/assets/...` route (same containment
rule as `styles`). **Shortcut:** if you omit `icon` but ship
`assets/thumb.png`, the loader auto-detects it — no manifest edit needed.
Plugins with no thumbnail get a default pedal graphic.
All three are backward-compatible: omit them and the plugin still loads and shows
a default pedal.
## The compatibility shim (don't rely on it)
So un-updated plugins still function, the host runs a `MutationObserver` that
re-homes any non-native `#player-controls` child into the slot. It's a safety net
— but it **breaks plugins that guard re-injection with
`#player-controls.contains(myBtn)`** (once the host moves the node out, the check
goes false and the plugin re-injects every song). **Mount into the slot yourself**
(the pattern above) to be correct; treat the shim as a fallback only.
## Styling
v3 uses `fb-*` design tokens (`fb-card`, `fb-text`, `fb-textDim`, `fb-primary`,
`fb-border`); v2 uses `dark-*` / `accent`. Legacy classes still **render
acceptably** in v3's dark theme, so a plugin that only uses core-guaranteed
utilities is functional in both. For polish, ship your own stylesheet via the
`styles` capability ([plugin-styles.md](plugin-styles.md)) declaring the tokens you
use — but the host slot already provides a styled container, so simple controls
need nothing special.
## Enabling / disabling plugins (Pedalboard footswitch)
The v3 **Pedalboard** Plugins page renders each plugin as a guitar pedal whose
"footswitch" turns the plugin on or off. The backend contract:
- **`enabled` field on every `/api/plugins` entry** — a boolean, default `true`.
Absent (older entries, stubbed test rows) is treated as enabled. The frontend
hides the nav and shows the footswitch unlit when `enabled` is `false`.
- **`POST /api/plugins/{plugin_id}/enabled`** — body `{"enabled": <bool>}`,
returns `{"id": "<id>", "enabled": <bool>}`.
- `400` if the body is missing/invalid or `enabled` isn't a real boolean
(`0`/`1`/strings are rejected).
- `400` if you try to disable an always-on plugin — `capability_inspector`
and any `app_tour_*` may never be disabled (disabling would brick the app or
the capability-graph review surface). Bundled plugins are otherwise
disable-able.
- `404` for an unknown plugin id (not loaded and not pending).
### Persistence
The choice is persisted under `CONFIG_DIR/plugin_state.json` as
`{"<plugin_id>": {"enabled": false}, ...}`. **Only non-default (`enabled:false`)
entries are stored** — re-enabling drops the key entirely, so the file stays
small and "absent ⇒ enabled" is the invariant. A missing or corrupt state file
is tolerated (logged, falls back to `{}`) and never crashes startup.
### Restart semantics
- Toggling **persists immediately** and flips the **in-memory** `enabled` flag,
so the very next `/api/plugins` (and thus the nav, the Pedalboard, and the
capability pipeline) reflects the change at once — no restart needed for the
UI to update.
- A plugin **disabled at runtime keeps its already-mounted routes/screen** until
the next restart; full hot-unload is out of scope. The frontend treats
`enabled:false` as "off" regardless.
- At **startup**, the loader **skips disabled plugins entirely** — it does not
install requirements, run `routes.setup()`, or register their screen, nav, or
capabilities. They still appear in `/api/plugins` as a disabled entry
(`status: "disabled"`, `enabled: false`) so the UI can show an "off" pedal you
can switch back on. **Re-enabling** a plugin that was skipped at startup
updates the flag immediately but the plugin only actually mounts on the next
restart.
### Capability pipeline
A disabled plugin is **excluded from the capability pipeline**: its
`capabilities`, `standards`, `capability_validation_warnings`,
`capability_unsupported_versions`, and `compatibility_shims` are emptied in the
`/api/plugins` response whenever `enabled` is `false` (covering both
startup-skipped and runtime-toggled-off plugins). Because the browser capability
registry registers any entry that carries a capability declaration regardless of
status, suppressing the metadata here is what actually keeps a disabled plugin
out of the capability graph.
## Checklist
- [ ] Backend / capabilities / library provider / `nav` + `screen` /
visualization renderer — **no change needed** (they work in v3 as-is).
- [ ] If you inject a control into the player: detect v3 and mount into
`window.slopsmith.ui.playerControlSlot()`; drop the dead separator /
`button:last-child` anchor; guard `contains()` against the actual container.
- [ ] Dropdowns positioned via `getBoundingClientRect()`, not `#player-controls`.
- [ ] `#player` overlays keep `z-index` ≤ the chrome layers (transport/HUD 20,
rail 30, popovers 40).
- [ ] Verify in **both** `/` (v2) and `/v3`.
docs/skills/debugging-keyboard-shortcuts.md
+206
View File
@@ -0,0 +1,206 @@
# Debugging Keyboard Shortcuts
This skill helps you debug keyboard shortcut issues in Slopsmith.
## Quick Start
1. **Start Slopsmith:**
```bash
cd ~/path/to/slopsmith
LIBRARY_PATH=/path/to/your/library docker compose up -d
```
2. **Open browser:** http://localhost:8000
3. **Open DevTools:** Press `F12` or `Ctrl+Shift+I`
## Debugging Commands
Open the browser console and run these commands:
### Enable Debug Logging
```javascript
_setDebugShortcuts(true)
```
This will log every keypress and shortcut match attempt.
### List All Registered Shortcuts
```javascript
_listShortcuts()
```
Shows all shortcuts with their keys, scopes, and descriptions.
### Test a Specific Shortcut
```javascript
_testShortcut('Space')
```
Shows if a shortcut would be active in the current context.
## Common Issues
### 1. Shortcut Not Triggering
**Check:**
- Are you on the right screen? (Player shortcuts only work on player screen)
- Is focus in an input field? (Shortcuts are disabled when typing)
- Is the key registered? Run `_listShortcuts()` to see all registered shortcuts
**Debug:**
```javascript
_setDebugShortcuts(true)
// Now press your key and watch the console
```
### 2. Scope Issues
**Check current context:**
```javascript
// This shows which screen you're on
document.querySelector('.screen.active')?.id
```
**Common scopes:**
- `global` - Works on any screen
- `player` - Only on player screen
- `library` - On home, favorites, or settings screens
- `plugin-{id}` - Only on a specific plugin's screen
### 3. Key Matching Issues
The system matches on both `e.key` (character produced) and `e.code` (physical key):
- Use `e.key` for letters/symbols that depend on keyboard layout
- Use `e.code` for special keys (Space, ArrowLeft, Escape, etc.)
**Example:**
```javascript
// Good for special keys
registerShortcut({ key: 'Space', ... }) // or 'ArrowLeft', 'Escape'
// Good for layout-dependent keys
registerShortcut({ key: '?', ... }) // or '[', ']', 'k'
```
### 4. Condition Not Met
If your shortcut has a condition function, it must return `true`:
```javascript
registerShortcut({
key: 'k',
description: 'My action',
scope: 'player',
condition: () => _isMyViewActive, // Must be true
handler: () => _myAction()
})
```
**Test it:**
```javascript
_testShortcut('k')
// Check if `conditionMet` is true
```
## Testing Your Changes
1. Make changes to `static/app.js`
2. Refresh the browser (changes are live-reloaded via Docker volume mount)
3. Run `_listShortcuts()` to verify your shortcut is registered
4. Press `?` to open the shortcuts help panel
5. Test your shortcut
## Built-in Shortcuts
Press `?` to see all shortcuts in the UI. Built-in shortcuts:
| Key | Scope | Description |
|-----|-------|-------------|
| `?` | Global | Show keyboard shortcuts |
| `Space` | Player | Play/Pause |
| `ArrowLeft` | Player | Seek back 5 seconds |
| `ArrowRight` | Player | Seek forward 5 seconds |
| `Escape` | Player | Back to library |
| `[` | Player | Offset audio back (Shift: 50ms, else 10ms) |
| `]` | Player | Offset audio forward (Shift: 50ms, else 10ms) |
## Adding Your Own Shortcuts
```javascript
registerShortcut({
key: 'k', // Key to press
description: 'Toggle my view', // Shown in help panel
scope: 'player', // When it's active
condition: () => _isMyViewActive, // Optional guard
handler: (e) => _myAction() // What to do
});
```
## Panel-Scoped Shortcuts
For plugins that create multiple panels (e.g., splitscreen), shortcuts are automatically scoped to the active panel:
```javascript
// Create panels (must exist before setActiveShortcutPanel can target them)
const panel1 = window.createShortcutPanel('panel-1');
const panel2 = window.createShortcutPanel('panel-2');
// Set active panel and register shortcuts
window.setActiveShortcutPanel('panel-1');
registerShortcut({
key: 'd',
description: 'Dock panel',
scope: 'global',
handler: () => _dockPanel()
});
// Switch to another panel
window.setActiveShortcutPanel('panel-2');
registerShortcut({
key: 'f',
description: 'Toggle fullscreen',
scope: 'global',
handler: () => _toggleFullscreen()
});
// Clean up when done — clear every panel you created
panel1.clearShortcuts();
panel2.clearShortcuts();
```
**Important:** In splitscreen, `scope: 'player'` means "player screen in the current panel". Each panel can have its own player shortcuts without collisions.
**Truly global shortcuts:** Use `window.getGlobalShortcutContext()` for shortcuts that must work in all panels (exceptional case, logs warning).
## Network Issues
If shortcuts aren't working at all:
1. Check the **Network** tab in DevTools
2. Look for failed requests to `/api/plugins`
3. Check the **Console** tab for JavaScript errors
4. Verify the container is running:
```bash
docker compose ps
docker compose logs -f
```
## WebSocket Issues
Keyboard shortcuts don't require WebSocket, but if other features aren't working:
1. Check **Network** tab → "WS" filter
2. Look for WebSocket connections to `/ws/highway/...`
3. Should show status "101 Switching Protocols"
## Getting Help
If you're still stuck:
1. Enable debug mode: `_setDebugShortcuts(true)`
2. Reproduce the issue
3. Copy the console output
4. Share it along with:
- Which screen you're on
- What key you're pressing
- What you expect to happen
- What actually happens
docs/sloppak-hand-editing.md
+266
View File
@@ -0,0 +1,266 @@
# Sloppak Hand-Editing — User Guide
A `.sloppak` is just a zip of plain files: some YAML, some JSON, some OGG audio, maybe a JPEG. That means you can open one up and change it. Want to record your own rhythm guitar take and use that instead of the mix? Fix an artist typo? Swap the cover art? Replace a Demucs split that bled drums into the "other" stem? You don't need to rebuild the whole sloppak from its source — just edit the file.
This guide walks through the most common edits, aimed at musicians who are comfortable with a text editor and Audacity but don't live on the command line.
> For the format **schema** (what every field means, how the wire format works, how to extend the format with new data types), see [sloppak-spec.md](sloppak-spec.md). This document is the **how-do-I-actually-edit-mine** companion.
---
## 1. The two forms — directory and zip
A sloppak exists in two interchangeable forms:
| Form | What it is | When to use it |
|---|---|---|
| **Directory** | A folder named `something.sloppak/` with the files loose inside | **Authoring** — easy to edit, no zip/unzip cycle |
| **Zip** | A `something.sloppak` file (zip with the same files inside) | **Distributing** — single file to share |
Slopsmith reads both. You can drop either one straight into your DLC folder and it'll show up in the library.
### Unzipping for editing
Slopsmith's converter ships sloppaks in zip form. To edit one, unzip it:
- **Windows:** rename `mysong.sloppak` → `mysong.zip`, right-click → Extract All. Then rename the resulting folder back to `mysong.sloppak/` (with the trailing slash / folder form). Or use [7-Zip](https://www.7-zip.org/) and unzip without renaming.
- **macOS:** rename `.sloppak` → `.zip`, double-click. Or use The Unarchiver.
- **Linux:** `unzip mysong.sloppak -d mysong.sloppak/`.
Once you have the directory form, you can edit any file inside and Slopsmith will pick it up — no re-zipping required for your own use.
### Cache: when changes don't appear
The first time Slopsmith opens a zip-form sloppak, it extracts a working copy into its config directory's cache: `${CONFIG_DIR}/sloppak_cache/<safe-id>` (in the standard Docker setup that's inside the `slopsmith-config` volume, mounted at `/config` in the container). The `<safe-id>` is the sloppak filename with each path separator (`/` or `\`) replaced by `__` and each space replaced by `_`. So `My-Song.sloppak` stays `My-Song.sloppak`, and `Artist/My Song.sloppak` becomes `Artist__My_Song.sloppak`.
You almost never need to touch this cache directly. If you edit the **original zip** in your DLC folder, Slopsmith re-extracts automatically when the zip's modification time or size changes — just save your edits and reload.
If a change still isn't appearing, the simplest reset is to remove the matching cache folder so Slopsmith rebuilds it on the next song load. In a default Docker install that's `docker exec <container> rm -rf /config/sloppak_cache/<safe-id>` (or the equivalent for your setup).
If you'd rather skip the cache layer entirely, **drop the directory form straight into your DLC folder** — Slopsmith uses it in place and there's nothing to invalidate.
---
## 2. Record and add your own rhythm stem
The use case: the converted rhythm guitar sounds muddy (Demucs has a tough time isolating fingerpicked acoustic, hi-gain palm mutes, etc.), and you'd rather play and record your own take.
### Step 1 — Set up your reference
1. Unzip the sloppak (see §1).
2. Look in `stems/` and open the reference audio in [Audacity](https://www.audacityteam.org/) (File → Open). Two cases:
- **`stems/full.ogg` is present** (pre-Demucs-split sloppak, or one where you kept `full.ogg` as a fallback — see §3). Open it directly; it's the original mixed audio.
- **No `full.ogg`, only per-instrument stems** (`guitar.ogg`, `bass.ogg`, `drums.ogg`, …). This is the default after Demucs splitting, since the converter deletes `full.ogg` once split stems exist. Select **all** the per-instrument stems and open them together — Audacity loads each as its own track aligned at `t=0`, and playing all of them at once reconstructs the full mix.
3. Note the **sample rate** displayed in Audacity's status bar (typically `44100 Hz`). Your recording must match this.
### Step 2 — Record your take aligned to the mix
1. In Audacity, with the reference track(s) open, add a new audio track (Tracks → Add New → Mono/Stereo Track).
2. Set Audacity to play the reference through your headphones (so you can hear what you're playing along to) while recording your own input.
3. Hit Record and play your rhythm part along with the reference from `t=0`. Critical: **start recording at the very beginning of the song.** If you punch in late, alignment will be off when you drop it in.
4. Stop when the song ends. Trim any silence/click at the very start of your recorded track so its first sample lines up with `t=0` of the reference (zoom in tight and check visually against the kick or first guitar hit).
### Step 3 — Export as OGG
1. **Solo** your recorded track (mute every reference track).
2. File → Export → Export as OGG Vorbis.
3. Quality slider: **5** (matches what the converter uses). Save as `rhythm_custom.ogg`.
4. Confirm in the export dialog that the sample rate is the same `44100 Hz` you noted in Step 1.
### Step 4 — Drop it in and update the manifest
1. Copy `rhythm_custom.ogg` into the sloppak's `stems/` folder.
2. Open `manifest.yaml` in any text editor (Notepad++, VS Code, BBEdit, gedit — all fine; just **don't use Word**).
3. Find the `stems:` block. Two things matter here:
- **Order:** Slopsmith's base `<audio>` element always plays the **first** stem listed in `stems[]`, regardless of `default:` flags. So if you want your custom stem to be what the player plays out-of-the-box (and what users without the Stems plugin will hear), put it **first**.
- **`default:` flags:** consulted by the [Stems plugin](https://github.com/topkoa/slopsmith-plugin-stems) to decide which faders start un-muted. They do **not** affect what the base `<audio>` element plays — that's purely the first-stem rule above.
Example for a Demucs-split sloppak where you re-recorded the rhythm guitar:
```yaml
stems:
- id: rhythm_custom # listed first → base <audio> plays this
file: stems/rhythm_custom.ogg
default: true
- id: guitar
file: stems/guitar.ogg
default: false # Stems plugin starts this fader muted
- id: bass
file: stems/bass.ogg
default: true
- id: drums
file: stems/drums.ogg
default: true
# … other stems unchanged …
```
Example for a pre-split sloppak (only `full.ogg` exists):
```yaml
stems:
- id: rhythm_custom # listed first → base <audio> plays this
file: stems/rhythm_custom.ogg
default: true
- id: full
file: stems/full.ogg
default: false # Stems plugin starts the full mix muted
```
4. Save the file. **Mind the indentation** — two spaces, no tabs. YAML is fussy about this.
### Step 5 — Reload and verify
Reload the song in Slopsmith. The [Stems plugin](https://github.com/topkoa/slopsmith-plugin-stems) will show a fader for `rhythm_custom` next to the others. If you don't see it, check the cache notes in §1.
### Common gotchas
- **Sample-rate mismatch** → choppy/pitched-wrong playback. Re-export from Audacity at exactly the rate the other stems use.
- **Mono vs stereo mismatch** is fine for playback but levels can feel different — match what the other stems use if you want consistent behavior in the mixer.
- **Silence padding at the start** of your recording → your stem will play late. Trim it tight in Audacity before exporting.
- **Tabs in `manifest.yaml`** → Slopsmith will refuse to load the song. Use two spaces.
---
## 3. Replace a bad Demucs stem
Demucs is good but not perfect. `htdemucs_6s` will occasionally bleed snare into `other.ogg` or leave drum overtones in the bass track. Fixing it works the same way as adding a custom stem — you're just overwriting an existing one.
### Option A: overwrite in place
1. Source or record a clean replacement and export it as OGG with the same sample rate.
2. Save it directly over the bad file (e.g. `stems/other.ogg`).
3. Reload — no manifest change needed.
### Option B: keep the original, add a replacement
Useful if you want to A/B them:
1. Save your new file as `stems/other_v2.ogg`.
2. In `manifest.yaml`, change the `file:` path on that stem's entry:
```yaml
- id: other
file: stems/other_v2.ogg # was stems/other.ogg
default: true
```
3. The old `other.ogg` stays in the folder but is no longer referenced. Delete it later if you want.
### Removing a stem entirely
If you want to drop a stem (e.g. `piano.ogg` is empty for this song):
1. Delete the file from `stems/`.
2. **Also remove** its entry from `manifest.yaml stems[]`. Leaving an orphan manifest entry pointing at a missing file produces a 404 in the player.
### A word on `full.ogg`
A converted sloppak starts with just `stems/full.ogg`. After Demucs splits it, the converter rewrites the manifest to list the per-instrument stems and removes `full.ogg`. If you're hand-editing and want to *keep* `full.ogg` as a fallback (mixed audio in case all the individual stems are muted), that's fine — leave the file in place and add a manifest entry with `default: false`. Don't delete `full.ogg` unless the per-instrument stems sum cleanly to a full mix.
---
## 4. Edit metadata, cover art, lyrics, tuning
All of these are tweaks to either `manifest.yaml` or files it points at. Open `manifest.yaml` in a text editor for the next three sections.
### Title, artist, album, year
Top-level keys in `manifest.yaml`. Just edit the strings:
```yaml
title: "Black Hole Sun"
artist: "Soundgarden"
album: "Superunknown"
year: 1994
duration: 320.5
```
Keep the quotes if the value already has them (especially when there's an apostrophe or colon). Reload the song — the library card updates next time the library refreshes.
### Cover art
Drop a square JPEG or PNG (500–1500 px on a side is the sweet spot) into the sloppak root and point the manifest at it:
```yaml
cover: cover.jpg
```
If the manifest doesn't have a `cover:` line, add one. The converter normally produces `cover.jpg` already; this is mostly relevant if you want to replace it with a better image.
### Lyrics
`lyrics.json` is a flat JSON list of syllable objects:
```json
[
{"t": 12.34, "d": 0.18, "w": "Hel"},
{"t": 12.52, "d": 0.22, "w": "lo-"},
{"t": 13.10, "d": 0.30, "w": "world"}
]
```
| Field | Meaning |
|---|---|
| `t` | Time the syllable starts, in seconds (float) |
| `d` | Duration in seconds |
| `w` | The syllable text. A trailing `-` joins it to the next syllable as one word. A trailing `+` marks the last syllable of a line (the renderer wraps after it). Both are suffixes on a real syllable — not standalone entries |
Common hand-edits:
- **Karaoke timing is off** — bump `t` values up or down a few hundredths of a second.
- **Wrong word** — edit `w`.
- **Missing line break** — append `+` to the last syllable of the line that should end there (e.g. change `"w": "world"` to `"w": "world+"`). Don't insert a standalone `"+"` entry — that creates an empty syllable that still consumes word-spacing in the renderer.
It's plain JSON — edit in any text editor.
### Tuning
Per-arrangement, in `manifest.yaml`:
```yaml
arrangements:
- id: lead
name: Lead
file: arrangements/lead.json
tuning: [0, 0, 0, 0, 0, 0] # E standard
capo: 0
```
Each number is **semitones from E A D G B E**, lowest string first. Common tunings:
| Tuning | Offsets |
|---|---|
| E Standard | `[0, 0, 0, 0, 0, 0]` |
| Eb Standard | `[-1, -1, -1, -1, -1, -1]` |
| D Standard | `[-2, -2, -2, -2, -2, -2]` |
| Drop D | `[-2, 0, 0, 0, 0, 0]` |
| Drop C | `[-4, -2, -2, -2, -2, -2]` |
| DADGAD | `[-2, 0, 0, 0, -2, -2]` |
The manifest tuning overrides whatever's stored inside `arrangements/lead.json` — so fixing it here is enough; you don't need to touch the arrangement JSON.
For 4-string bass, only indices 0–3 are meaningful; leave 4 and 5 at `0`.
### What *not* to put in `manifest.yaml`
Don't add per-machine settings (audio device picks, MIDI port IDs), UI state, or your own play counts. The sloppak holds the song's authored data — anything that varies by user or machine lives in Slopsmith's config dir or the metadata DB. See [sloppak-spec.md §5.7](sloppak-spec.md#57-dont-break-the-manifest-contract) for the full list.
---
## 5. Re-zipping for distribution
If you want to share your modified sloppak with someone else, re-zip it:
1. Open the `mysong.sloppak/` directory.
2. Select **everything inside** — `manifest.yaml`, `arrangements/`, `stems/`, `lyrics.json`, `cover.jpg`.
3. Zip the **contents**, not the parent folder. (If you zip the folder, the zip will have a top-level `mysong.sloppak/` directory inside, which Slopsmith won't parse — the manifest must be at the zip root.)
4. Rename `mysong.zip` → `mysong.sloppak`.
For your own use, you can skip this entirely — Slopsmith reads the directory form straight from your DLC folder.
---
## Out of scope (for now)
- **Authoring a sloppak from scratch** (no Guitar Pro / MusicXML source file) — that's a developer task. Start at [sloppak-spec.md §4.2](sloppak-spec.md#42-writing-python-server-side).
- **Editing notes / chords in `arrangements/*.json`** — technically possible but extremely tedious by hand: hundreds of objects with short field names per song. The fields are documented in [sloppak-spec.md §3](sloppak-spec.md#3-arrangement-json--the-wire-format), but for any real chart edit you want the [Arrangement Editor plugin](https://github.com/byrongamatos/slopsmith-plugin-editor).
- **Loudness normalization / advanced stem processing** — out of scope here; standard Audacity or ffmpeg workflows apply to any OGG file before you drop it into `stems/`.
docs/sloppak-spec.md
+939
View File
@@ -0,0 +1,939 @@
# Sloppak Format — Developer Guide
Sloppak is Slopsmith's open, hand-editable song format. This guide is for developers who want to **read**, **write**, or **extend** the format — including adding new data types like drum tabs, vocal pitches, lighting cues, key/scale annotations, or anything else a future visualization plugin might need.
> If you're a **user** wanting to modify an existing sloppak — record your own rhythm stem, fix metadata, swap cover art, replace a Demucs split — see [sloppak-hand-editing.md](sloppak-hand-editing.md). That guide is the practical, step-by-step companion to this developer reference.
The authoritative format reference lives in code (`lib/sloppak.py`, `lib/song.py`); this doc explains the why, the how, and the conventions you should follow when adding to it.
---
## 1. Format at a glance
A sloppak exists in **two interchangeable forms**:
| Form | What it is | Used for |
|---|---|---|
| **Directory** | A folder named `*.sloppak/` containing the files below | Authoring, hand editing, plugin development |
| **Zip archive** | A `.sloppak` file (zip with the same files inside) | Distribution |
Both forms hold identical contents. Slopsmith resolves either transparently — zip files are unpacked to a cache the first time they're opened (see `resolve_source_dir()` in [lib/sloppak.py](../lib/sloppak.py)).
### Directory layout
```
my-song.sloppak/
├── manifest.yaml # Required — all metadata + file index
├── arrangements/
│ ├── lead.json # One JSON per playable arrangement
│ ├── rhythm.json
│ └── bass.json
├── stems/
│ ├── full.ogg # Mixed audio (initial single-stem output; may be absent after stem splitting)
│ ├── guitar.ogg # Optional individual stems
│ ├── bass.ogg
│ ├── drums.ogg
│ ├── vocals.ogg
│ └── other.ogg
├── lyrics.json # Optional — syllable-level lyrics
└── cover.jpg # Optional — album art
```
Three rules to remember:
1. **`manifest.yaml` is the index.** Nothing inside the sloppak is auto-discovered — every file path is listed in the manifest. This makes the format predictable: no scanning, no guessing. (One historical exception: the cover-art handler in `server.py` falls back to `cover.jpg` when `manifest.cover` is missing. New code should not add similar filename fallbacks.)
2. **Filenames in `manifest.yaml` are POSIX paths**, relative to the sloppak root (forward slashes, no leading `/`).
3. **YAML for the manifest, JSON for everything else.** YAML is hand-editable for users; JSON is fast-parsed and easy to round-trip in code.
---
## 2. `manifest.yaml` reference
Minimal valid manifest:
```yaml
title: "Black Hole Sun"
artist: "Soundgarden"
duration: 320.5
arrangements:
- id: lead
name: Lead
file: arrangements/lead.json
tuning: [0, 0, 0, 0, 0, 0]
capo: 0
stems:
- id: full
file: stems/full.ogg
default: true
```
Full set of currently-recognized top-level keys:
| Key | Type | Required | Description |
|---|---|---|---|
| `title` | string | yes | Song title |
| `artist` | string | yes | Artist name |
| `album` | string | no | Album |
| `year` | int | no | Release year |
| `duration` | float | yes | Song length in seconds |
| `arrangements` | list | yes | Playable arrangements (see §2.1) |
| `stems` | list | yes | Audio stems (see §2.2) |
| `stem_separation` | object | no | Structured metadata when stems were produced by an automated separation engine (currently `demucs`). Shape: `{engine, model, version}`. See §2.2 for fields + semver semantics per [slopsmith#357](https://github.com/byrongamatos/slopsmith/issues/357). Omitted for single-stem sloppaks (`stems: [{id: full, ...}]`) and for hand-edited / user-recorded stems |
| `lyrics` | string | no | Path to lyrics JSON |
| `lyrics_source` | string | no | Where the lyrics came from: `xml` (vocals XML from the chart source), `whisperx` (auto-transcribed), or `user` (hand-edited). Absent on legacy sloppaks — readers should treat missing as `xml` |
| `lyric_transcription` | object | no | Structured metadata when lyrics came from an automated engine (currently `whisperx`). Same shape as the parent `stem_separation` block defined by [slopsmith#357](https://github.com/byrongamatos/slopsmith/issues/357) — see §2.3 for fields and semver semantics. Omitted for authored lyrics (`xml`/`user`) |
| `vocal_pitch` | string | no | Path to per-syllable pitch JSON (`{"version": 1, "notes": [{t, d, midi}, ...]}`). Consumed by [slopsmith-plugin-lyrics-karaoke](https://github.com/byrongamatos/slopsmith-plugin-lyrics-karaoke) to render karaoke note bars. See §2.4 |
| `pitch_extraction` | object | no | Structured metadata when pitch was extracted by an automated engine (currently `crepe` via the demucs server's `/pitch` endpoint). Same shape as `stem_separation` / `lyric_transcription`. Omitted for hand-edited pitch tracks |
| `cover` | string | no | Path to cover image |
| `preview` | string | no | Path to a short preview audio clip (OGG) at the sloppak root. Populated when the source carries a separate short browser-preview clip (decoded to `preview.ogg`); absent otherwise. Consumed by [`slopsmith-plugin-song-preview`](https://github.com/byrongamatos/slopsmith-plugin-song-preview) for hover-to-listen previews in the library |
| `song_timeline` | string | no | Path to a `song_timeline.json` file carrying song-wide beats and sections (see §5.3). When present, its data takes priority over any beats/sections embedded in arrangement JSONs. Older readers ignore the key and fall back to reading beats/sections from the first arrangement JSON as before |
| `drum_tab` | string | no | Path to `drum_tab.json` — per-piece drum hits (see §5.3). Implemented end-to-end as of slopsmith#344 |
Unknown keys are **silently ignored** by the loader. This is deliberate — it's the extensibility hook (see §5).
### 2.1. `arrangements[]`
Each entry describes one playable arrangement and points at its JSON file:
```yaml
arrangements:
- id: lead # filesystem-safe stable ID, used for filenames
name: Lead # display name (Lead/Rhythm/Bass/Combo are sorted first)
file: arrangements/lead.json
tuning: [0, 0, 0, 0, 0, 0] # six semitone offsets from E A D G B E
capo: 0
centOffset: 0.0 # optional float, cents; default 0.0
```
- `tuning` is a list of semitone offsets from standard `E2 A2 D2 G3 B3 E4`. **Six elements is the standard six-string convention** and the only length `lib/tunings.py` produces friendly names for; 5- and 7-string content is accepted by the loader and falls through to a numeric label. For bass, the four bass strings are at indices 0–3; the other two slots are `0`. Consumers should not hard-code `len(tuning) == 6`.
- `name` controls the sort order in the UI: `Lead > Combo > Rhythm > Bass > everything else`.
- `centOffset` is a pitch-shift value in cents. Commonly `-1200.0` for extended-range bass arrangements tuned one octave down; small non-zero values for songs mastered at a non-A440 reference pitch (e.g. A443 ≈ +11.8 cents). Absent / `0.0` means no shift. Exposed to plugins via `getSongInfo().centOffset`.
- Manifest-level `tuning`, `capo`, and `centOffset` **override** anything embedded in the arrangement JSON. The arrangement JSON's own values are fallbacks.
- `notation` (optional string) — path to a `notation_<id>.json` file carrying standard musical notation data for this arrangement (see §5.3). When present, the loader surfaces it on `LoadedSloppak.notation_by_id[id]` and the highway WS streams `notation_info` + `notation_measures` messages. The `file:` key may be omitted when `notation:` is present — the loader creates a stub arrangement so the notation file can be the sole data source.
### 2.2. `stems[]`
```yaml
stems:
- id: full
file: stems/full.ogg
default: true # plays by default when the song opens
- id: guitar
file: stems/guitar.ogg
default: true
- id: drums
file: stems/drums.ogg
default: false
```
- `id` is referenced by the Stems plugin and any other consumer; keep it stable.
- `default` accepts `true`/`false`, or strings (`"on"`/`"off"`/`"true"`/etc.) for hand-edited manifests.
- A freshly converted sloppak from `lib/sloppak_convert.py` starts with a single `{id: full, file: stems/full.ogg, ...}` entry. After stem-splitting (Demucs), `full.ogg` is removed and the manifest is rewritten with per-instrument entries (`guitar`, `bass`, `drums`, `vocals`, `other`). The format requires only that `stems` is non-empty — there's no specific filename or id that must always be present.
When stems were produced by an automated separation engine (Demucs), an optional `stem_separation` block records which engine + model produced them. Per [slopsmith#357](https://github.com/byrongamatos/slopsmith/issues/357):
```yaml
stem_separation:
engine: demucs # stable engine id; only `demucs` today
model: htdemucs_6s # specific model name (htdemucs_6s / htdemucs_ft / htdemucs / mdx_extra / ...)
version: 1.0.0 # semver for slopsmith's stem-artifact contract
```
Fields:
- `engine` — stable identifier for the separation engine. Currently always `demucs`. New engines (e.g. a hypothetical `spleeter`) would get their own stable id.
- `model` — the engine-specific model id used for this split. For Demucs this is the `-n` flag value.
- `version` — semver for Slopsmith's stem-artifact contract (independent of upstream Demucs / model versions). Bump per the same semantics #357 defines: patch = metadata-only fixes, minor = backward-compatible additions, major = stem set / packing / post-processing changed and existing splits should be regenerated.
Omitted for single-stem sloppaks (`stems: [{id: full, ...}]` — no automated separation ran) and for hand-edited / user-recorded stems. The RFC reserves a separate `stem_authoring` sibling block for the hand-edit case; that's deferred to a follow-up.
A remote Demucs server can use this block as part of a cache key so that changing the model or major version naturally produces a cache miss. Local plugin jobs should preserve this metadata in job state and in any copied/downloaded manifests.
### 2.3. `lyrics`
If present, points at a JSON file containing a flat list of syllable objects:
```json
[
{"t": 12.34, "d": 0.18, "w": "Hel"},
{"t": 12.52, "d": 0.22, "w": "lo-"},
{"t": 13.10, "d": 0.30, "w": "world"}
]
```
| Field | Meaning |
|---|---|
| `t` | Time in seconds |
| `d` | Duration in seconds |
| `w` | Syllable text. Trailing `-` joins to the next syllable as one word; trailing `+` marks the last syllable of a line (renderer wraps after it). Both are suffixes on a real syllable — not standalone entries. See `static/highway.js` for the rendering: `raw.endsWith('+')` flags end-of-line, and `sylText` strips the trailing marker before drawing |
When lyrics are present, the optional top-level `lyrics_source` key records where they came from. The assembler sets it to `xml` when the lyrics were parsed from the source chart's vocals XML; the WhisperX auto-transcription fallback (`scripts/transcribe_lyrics.py`, or `--auto-lyrics` on the split scripts) sets it to `whisperx`. Hand-edited lyrics should bump it to `user` so UI consumers can render a different badge (or no badge) than for machine-generated lyrics. The key is absent on sloppaks produced before this field existed — readers should treat missing as `xml` for backward compatibility.
When `lyrics_source` is `whisperx` (or any future automated engine), an optional `lyric_transcription` block records which engine + model produced the file. Shape mirrors the parent `stem_separation` RFC ([slopsmith#357](https://github.com/byrongamatos/slopsmith/issues/357)):
```yaml
lyric_transcription:
engine: whisperx # stable engine id
model: medium # the WhisperX model size that ran (tiny/base/small/medium/large-v2/large-v3)
version: 1.0.0 # semver for slopsmith's lyric-transcription artifact contract
```
Fields:
- `engine` — stable identifier for the transcription engine; currently always `whisperx`.
- `model` — the engine-specific model id used for this transcription.
- `version` — semver for Slopsmith's lyric-transcription artifact contract (independent of upstream Whisper / WhisperX versions). Bump per the same semantics #357 defines for stems: patch = metadata-only fixes, minor = backward-compatible additions, major = output shape changed and existing transcriptions should be regenerated.
Omitted for authored lyrics (`xml` / `user`). A remote WhisperX server can use this block as part of a cache key the same way #357 envisions for stems — caches should miss whenever any of the three fields change, ensuring stale transcriptions don't get returned after a model bump.
### 2.4. `vocal_pitch`
If present, points at a JSON file holding per-syllable pitch data — the karaoke companion to `lyrics`. Consumed by [slopsmith-plugin-lyrics-karaoke](https://github.com/byrongamatos/slopsmith-plugin-lyrics-karaoke) to render karaoke-style note bars over the lyric text. Shape:
```json
{
"version": 1,
"notes": [
{"t": 12.34, "d": 0.40, "midi": 64},
{"t": 12.78, "d": 0.55, "midi": 67}
]
}
```
| Field | Meaning |
|---|---|
| `version` | Schema version of this `vocal_pitch.json` file (currently the integer `1`). Bump on a breaking change to the `notes` entry shape. This is *not* the same as the top-level `pitch_extraction.version` block below, which is a semver string used as a cache-key for the extractor engine |
| `notes` | List of pitch entries, one per syllable that the extractor could lock onto. `t` + `d` mirror the matching `lyrics.json` entry; `midi` is the MIDI note number (60 = middle C). Syllables the extractor couldn't pitch (silent / sub-confidence) are omitted from this list — it may be shorter than `lyrics.json` |
When pitch came from an automated engine (the demucs server's `/pitch` endpoint, which runs CREPE), the optional top-level `pitch_extraction` block records which engine + model produced the file. Same shape and semver-string semantics as `stem_separation` / `lyric_transcription` — distinct from the in-file integer `version` field above:
```yaml
pitch_extraction:
engine: crepe
model: v1
version: 1.0.0
```
Omitted for hand-edited pitch tracks. As with the other two automated-artifact blocks, a remote pitch server can use this for cache-key invalidation.
The sloppak assembler runs pitch extraction automatically when `pitch_extraction.enabled` is set in its config AND a server URL is configured (either `pitch_extraction.server_url` or the shared `demucs_server_url`) AND the sloppak has lyrics + a `stems/vocals.ogg` after the split pass — either because `_maybe_transcribe_lyrics` just produced them via WhisperX OR because they were already on disk (from the source chart's vocals XML, hand-authoring, or an earlier build). Pitch is *not* coupled to `whisperx.enabled` — setting `pitch_extraction.enabled=true` alone (with WhisperX off) is enough to retro-generate pitch over any existing on-disk lyrics. Sloppaks built before this field existed simply don't carry it — readers should treat missing `vocal_pitch` as "no pitch data, fall back to whatever the karaoke plugin's local-extraction path produces (if any)".
---
## 3. Arrangement JSON — the wire format
Arrangement JSON files use the **wire format** produced by `arrangement_to_wire()` — the on-disk representation of a complete arrangement. Slopsmith's `/ws/highway/{filename}` endpoint transports similar data as a sequence of typed messages (`notes`, `chords`, `anchors`, `chord_templates`, `phrases`, …) rather than as one identical top-level JSON object. In practice, the WebSocket stream reuses the same per-object field names where applicable, but it should not be treated as a byte-for-byte match for `arrangements/*.json`.
The authoritative serializer/deserializer is in [lib/song.py](../lib/song.py):
- `arrangement_to_wire(arr) → dict` — write
- `arrangement_from_wire(dict) → Arrangement` — read
### 3.1. Top-level shape
```json
{
"name": "Lead",
"tuning": [0, 0, 0, 0, 0, 0],
"capo": 0,
"centOffset": 0.0, /* optional, float cents, default 0.0 */
"notes": [ /* see 3.2 */ ],
"chords": [ /* see 3.3 */ ],
"anchors": [ /* see 3.4 */ ],
"handshapes": [ /* see 3.5 */ ],
"templates": [ /* see 3.6 */ ],
"phrases": [ /* optional, see 3.7 */ ],
"tones": { /* optional, see 3.9 */ },
"beats": [ /* see 3.8, only on first arrangement */ ],
"sections": [ /* see 3.8, only on first arrangement */ ]
}
```
`beats` and `sections` are **song-level** but live on the first arrangement's JSON for legacy reasons — `lib/sloppak.py` hoists them to the `Song` object on load. If you author multiple arrangements, only put them in one file. **New sloppaks should use `song_timeline.json` instead** (see §2 and §5.3) — when the manifest carries a `song_timeline:` key pointing at a schema-valid file, its beats/sections **replace** whatever the arrangement JSONs loaded (the override is applied after arrangement loading, so a valid `song_timeline.json` always wins). Arrangement-JSON beats/sections remain supported for backward compatibility with all existing sloppaks and are the fallback when the file is absent or invalid.
### 3.2. Notes
Field names are short on purpose — these get streamed thousands of times per song. Don't expand them.
```json
{
"t": 12.345, // time (s)
"s": 2, // string (0 = lowest)
"f": 7, // fret (0 = open, 24 = max)
"sus": 0.5, // sustain (s, 0 = none)
"sl": 9, // pitched slide-to fret (-1 = no slide)
"slu": -1, // unpitched slide-to fret (-1 = no slide)
"bn": 1.0, // bend amount in semitones
"ho": false, // hammer-on
"po": false, // pull-off
"hm": false, // natural harmonic
"hp": false, // pinch harmonic
"pm": false, // palm mute
"mt": false, // string mute
"vb": false, // vibrato
"tr": false, // tremolo
"ac": false, // accent
"tp": false, // tap
"ln": false, // link-next (chord linking metadata; renderers may ignore — runtime linking is derived from proximity)
"fhm": false, // fret-hand mute
"plk": false, // pluck (pop, bass)
"slp": false, // slap (bass)
"rh": -1, // right-hand fingering (-1 = unset)
"pkd": -1, // pick direction (-1 = unset, 0 = down, 1 = up)
"ig": false // ignore (chart-author flag — note is rendered but not scored / sequenced)
}
```
Default values: numbers → `0` or `-1` (slides / `rh` / `pkd`), bools → `false`. Omit fields equal to their default if you're authoring by hand — the parser fills them in. **Encoders should default-omit the newer technique keys** (`ln`, `fhm`, `plk`, `slp`, `rh`, `pkd`, `ig`) — the highway streams notes thousands of times per song, so trimming the common case keeps the WebSocket payload tight. The pre-existing keys are still emitted unconditionally to preserve the legacy wire contract.
### 3.3. Chords
A chord groups note-shaped objects under a single time:
```json
{
"t": 30.0,
"id": 12, // index into templates[]
"hd": false, // high-density flag
"notes": [
{"s": 0, "f": 3, "sus": 0.0, ...},
{"s": 1, "f": 5, "sus": 0.0, ...}
]
}
```
Chord notes use the same field set as standalone notes, **except `t` is omitted** (the chord carries the time). The fingering / shape lookup is `chord.id → templates[id]`.
### 3.4. Anchors
Where the fretting hand sits. Drives the highway zoom box.
```json
{"time": 12.0, "fret": 5, "width": 4}
```
### 3.5. Hand shapes
Spans during which a chord shape is held:
```json
{"chord_id": 12, "start_time": 30.0, "end_time": 31.5, "arp": false}
```
- `chord_id` (`int`, default `0`) — index into `templates[]`; identifies which chord template the span is holding.
- `start_time` (`float`, default `0.0`) — start of the span in seconds.
- `end_time` (`float`, default `0.0`) — end of the span in seconds.
- `arp` (`bool`, default `false`, allowed values `true`/`false`) — whether this hand shape should be treated as an arpeggio span rather than a fully-strummed chord hold.
### 3.6. Chord templates
Named shapes referenced by `chord.id` and `handshape.chord_id`:
```json
{
"name": "Em7",
"displayName": "Em7",
"arp": false,
"fingers": [-1, 2, 1, -1, -1, -1],
"frets": [ 0, 2, 2, 0, 0, 0]
}
```
- `name` (`string`, default `""`) — canonical template name used by the parser / authoring data.
- `displayName` (`string`, default `name`) — label shown in the UI; source XML may use this for display-specific variants such as `-arp`.
- `arp` (`bool`, default `false`, allowed values `true`/`false`) — whether the template is flagged as arpeggiated. Parsed from explicit XML attributes (`arpeggio` / `arp`, any common casing) or inferred from `displayName` markers such as `-arp`.
- `fingers` (`int[6]`, default `[-1, -1, -1, -1, -1, -1]`) — fretting-hand finger numbers, lowest string first. `-1` = unused string, `0` = open string / no fretting finger, `1..4` = index/middle/ring/pinky.
- `frets` (`int[6]`, default `[-1, -1, -1, -1, -1, -1]`) — fret numbers, lowest string first. `-1` = unused string, `0` = open string, positive values = fretted note.
### 3.7. Phrases (optional, multi-difficulty data)
Sources that carry per-phrase difficulty ladders (phrase-aware arrangement XML) include this. GP imports and legacy sloppaks omit it:
```json
"phrases": [
{
"start_time": 0.0,
"end_time": 12.5,
"max_difficulty": 4,
"levels": [
{ "difficulty": 0, "notes": [...], "chords": [...], "anchors": [...], "handshapes": [...] },
{ "difficulty": 1, "notes": [...], "chords": [...], "anchors": [...], "handshapes": [...] },
...
]
}
]
```
If you're writing a converter that doesn't have multi-difficulty data, **omit the `phrases` key entirely** (don't emit `"phrases": []`). A missing key signals "no ladder, disable the master-difficulty slider"; an empty list is the same in current code but reads ambiguously.
### 3.8. Beats and sections
```json
"beats": [{"time": 0.5, "measure": 1}, {"time": 1.0, "measure": -1}, ...],
"sections": [{"name": "verse", "number": 1, "time": 12.5}, ...]
```
`measure: -1` = sub-beat (not a downbeat). Section `name` follows the usual song-structure conventions (`intro`, `verse`, `chorus`, `bridge`, `solo`, `outro`, …).
### 3.9. Tones (optional)
`tones` carries the arrangement's guitar tones — the amp/pedal/cabinet gear and the in-song tone switches. It's populated when the source chart carries tone data (`lib/tones.py`); a sloppak authored from scratch may omit it entirely.
```json
"tones": {
"base": "Clean Rhythm",
"changes": [
{"t": 12.5, "name": "Lead Drive"},
{"t": 48.0, "name": "Clean Rhythm"}
],
"definitions": [
{
"Name": "Clean Rhythm",
"Key": "Tone_A",
"GearList": { /* raw gear blocks: Amp, PrePedal1-4, … */ }
}
]
}
```
- `base` (string) — the tone in effect before the first change.
- `changes` (list, time-sorted) — `{"t": seconds, "name": str}` tone switches. The highway draws a marker at each. Omit when the arrangement never switches tone.
- `definitions` (list) — the **raw tone objects** (`Name`, `Key`, `GearList`), copied verbatim from the source chart's tone manifest. The Tones plugin parses these into the rendered signal chain (it owns the gear-name/image map, so the data is stored unparsed here).
All three sub-keys are individually optional; an arrangement with none of them simply omits `tones`. Readers that don't know about tones ignore the key (the loader preserves it verbatim).
---
## 4. Reading and writing sloppaks programmatically
### 4.1. Reading (Python, server-side)
```python
from pathlib import Path
from sloppak import load_song, load_manifest
# Quick metadata only (parses manifest, skips arrangement JSONs)
manifest = load_manifest(Path("song.sloppak"))
# Full song load (manifest + all arrangements + lyrics)
loaded = load_song("song.sloppak", dlc_root=Path("/dlc"), unpack_cache_root=Path("/cache"))
print(loaded.song.title, len(loaded.song.arrangements))
print(loaded.stems) # [{"id": "full", "file": "stems/full.ogg", "default": True}]
print(loaded.manifest) # raw dict — read your custom keys here
```
### 4.2. Writing (Python, server-side)
There's no general-purpose writer in `lib/` yet. The current writer lives in [lib/sloppak_convert.py](../lib/sloppak_convert.py) inside the sloppak assembly function — it's the single source of truth for "how a sloppak gets built." If you need to write sloppaks from a new source, copy the structure of that function:
1. Build a `work_dir/` in temp.
2. Write `arrangements/{id}.json` per arrangement using `arrangement_to_wire()`.
3. Encode audio to OGG into `stems/`.
4. Optionally write `lyrics.json`, `cover.jpg`.
5. Compose the `manifest` dict and dump as YAML with `yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True)`.
6. Either `shutil.copytree(work_dir, out)` for directory form, or `_zip_dir(work_dir, out)` for zip form.
Always use `yaml.safe_dump` (not `yaml.dump`) and pass `sort_keys=False` so the human-readable order is preserved.
### 4.3. Reading (JavaScript, plugin-side)
Plugins typically don't read the sloppak file directly — they consume the `/ws/highway/{filename}` WebSocket stream (see `CLAUDE.md` for the message protocol), which produces the same shapes. If you specifically need raw manifest access from the browser, expose it through a custom backend route in your plugin's `routes.py` and fetch it.
---
## 5. Extending the format — adding new data
Sloppak is designed to be extended without breaking older readers. The conventions below come from how `lyrics`, `stems`, and the optional `phrases` ladder were each added.
### 5.1. The golden rule: **manifest opt-in, file off to the side**
New data types should follow this pattern:
1. **Drop a new file** alongside the standard ones (e.g., `drums.json`, `keys.json`, `lighting.json`).
2. **Add a manifest key** that *points at* that file (e.g., `drum_tab: drums.json`).
3. **Make consumers gate on the manifest key**: if the key is absent, do nothing. Never auto-discover by filename — that breaks the "manifest is the index" rule.
So a sloppak with drum tabs would look like:
```yaml
# manifest.yaml
title: "Song"
artist: "Band"
duration: 240.0
arrangements: [...]
stems: [...]
drum_tab: drum_tab.json # ← new key
```
```
my-song.sloppak/
├── manifest.yaml
├── arrangements/...
├── stems/...
└── drum_tab.json # ← new file
```
Older Slopsmith readers ignore the unknown `drum_tab` key (the loader uses `manifest.get("drum_tab")` / unknown keys pass through). Your plugin checks for it and renders accordingly. **Zero coordination needed with core.**
### 5.2. Naming conventions for new keys and files
- **Manifest keys**: `snake_case`, descriptive, singular when the value is one thing (`lyrics`, `cover`, `drum_tab`), plural when it's a list (`stems`, `arrangements`).
- **File names**: lowercase, hyphenated or underscored, JSON for structured data, OGG for audio, JPG/PNG for images.
- **Inside JSON**: short field names for hot-path data that gets streamed thousands of times (`t`, `s`, `f` — see §3.2). Long names are fine for one-off metadata.
- **Time fields**: always `t` or `time` (not `start`, not `timestamp`) — and always **seconds as floats**, not ms or ticks. Be consistent with the existing wire format.
- **Indexes / IDs**: stable, filesystem-safe, lowercase. Don't reuse a source format's internal numeric IDs unless you have to.
### 5.3. Worked examples for the kinds of additions you mentioned
#### Drum tab
`drum_tab.json` carries per-piece hits authored on top of the song's audio.
Implemented end-to-end as of slopsmith#344 (drums-from-scratch): the loader
in `lib/sloppak.py` parses it, `lib/drums.py` defines the canonical piece-id
vocabulary, and `/ws/highway/{filename}` streams it as `drum_tab` + chunked
`drum_hits` messages.
```json
{
"version": 1,
"name": "Drums",
"kit": [
{"id": "kick", "name": "Kick"},
{"id": "snare", "name": "Snare"},
{"id": "hh_closed", "name": "Hi-hat (closed)"},
{"id": "hh_open", "name": "Hi-hat (open)"},
{"id": "crash_r", "name": "Crash (right)"},
{"id": "ride", "name": "Ride"}
],
"hits": [
{"t": 0.500, "p": "kick", "v": 110},
{"t": 0.750, "p": "snare", "v": 92},
{"t": 0.750, "p": "hh_closed", "v": 70},
{"t": 1.000, "p": "snare", "v": 60, "g": true},
{"t": 1.250, "p": "snare", "v": 105, "f": true},
{"t": 4.000, "p": "crash_r", "v": 120, "k": 0.080}
]
}
```
Manifest:
```yaml
drum_tab: drum_tab.json
```
##### Hit fields
| key | type | meaning |
| --- | --- | --- |
| `t` | float seconds | hit time, required, monotonic in `hits[]` |
| `p` | string | piece-id from the closed list below; required |
| `v` | int 1-127 | velocity (default 100) |
| `g` | bool | ghost note (renders smaller / outline-only) |
| `f` | bool | flam (renders a small leading ghost glyph 30 ms early) |
| `k` | float seconds | cymbal-choke tail duration (renders a fade-out) |
##### Canonical piece-id vocabulary
A closed list lives in `lib/drums.py::PIECES`. Open/closed hi-hat are
**distinct piece-ids**, not articulation flags — hit detection must reject
a closed-hat strike on an open-hat note, which it can only do if the
articulation is part of the piece-id.
| piece-id | category | default GM MIDI | default shape |
| --- | --- | --- | --- |
| `kick` | kick | 35, 36 | bar (full-width across all non-kick lanes) |
| `snare` | drum | 38, 40 | rectangle |
| `snare_xstick` | drum | 37 | hatched rectangle |
| `tom_hi` | drum | 50, 48 | rectangle |
| `tom_mid` | drum | 47, 45 | rectangle |
| `tom_low` | drum | 43 | rectangle |
| `tom_floor` | drum | 41 | rectangle |
| `hh_closed` | cymbal | 42 | filled circle |
| `hh_open` | cymbal | 46 | ring (outline) circle |
| `hh_pedal` | cymbal | 44 | small circle with × |
| `stack` | cymbal | 30 | jagged circle (no GM standard — reuses 30 from extended-percussion range) |
| `crash_l` | cymbal | 49 | circle |
| `crash_r` | cymbal | 57 | circle |
| `splash` | cymbal | 55 | small circle |
| `china` | cymbal | 52 | jagged circle |
| `ride` | cymbal | 51, 59 | circle |
| `ride_bell` | cymbal | 53 | circle with centre dot |
| `bell` | cymbal | 80 | circle with centre dot (no GM standard — reuses "Mute Triangle") |
Unknown piece-ids round-trip through the loader (forward-compat); the
client just renders them as a default rectangle.
##### Wire format
Streamed as two highway-WS message types:
```json
{ "type": "drum_tab", "version": 1, "name": "Drums",
"kit": [{"id": "kick", "name": "Kick"}, ...], "total": 1234 }
```
…followed by one or more chunks of 500 hits:
```json
{ "type": "drum_hits", "data": [{"t": 0.5, "p": "kick", "v": 110}, ...],
"total": 1234 }
```
##### Design notes
- `kit[]` is the legend — fixed metadata, separated from hot-path data.
- `hits[]` uses short field names because this list can be thousands long.
- `v` defaults to 100; ghost / flam / choke flags are all optional.
- Older sloppaks whose drums are encoded as guitar notes (`midi = string*24 + fret`) still play — the drums plugin keeps a legacy decoder that reads the standard `notes` stream and synthesises `drum_hits` from it.
#### Song timeline (beats and sections as a top-level file)
`song_timeline.json` moves song-wide beats and sections out of the first
arrangement JSON and into a dedicated file. Implemented in `lib/sloppak.py`
alongside the notation format: the loader reads the manifest's optional
`song_timeline:` key, validates the file, and populates `Song.beats` /
`Song.sections` from it, taking priority over any beats/sections embedded
in arrangement JSONs.
```json
{
"version": 1,
"beats": [
{"time": 0.500, "measure": 1},
{"time": 1.000, "measure": -1},
{"time": 1.500, "measure": -1},
{"time": 2.000, "measure": 2}
],
"sections": [
{"name": "intro", "number": 1, "time": 0.0},
{"name": "verse", "number": 1, "time": 16.0},
{"name": "chorus", "number": 1, "time": 32.0}
]
}
```
Manifest:
```yaml
song_timeline: song_timeline.json
```
| Field in `beats[]` | Type | Notes |
|---|---|---|
| `time` | float seconds | Beat timestamp. Matches the existing arrangement-JSON wire convention |
| `measure` | int | 1-based downbeat number. `-1` = sub-beat (not a downbeat) |
| Field in `sections[]` | Type | Notes |
|---|---|---|
| `name` | string | song-structure convention: `intro`, `verse`, `chorus`, `bridge`, `solo`, `outro`, … |
| `number` | int | Section repeat number |
| `time` | float seconds | Section start |
**Backward compatibility.** Sloppaks without `song_timeline:` continue to
work — the loader falls through to reading beats/sections from the first
arrangement JSON exactly as before. No migration is needed.
**New sloppaks** should put beats/sections here and leave arrangement JSONs
free of timeline data. This is especially important for notation-only
arrangements (see below) where there may be no arrangement JSON at all.
---
#### Notation format (standard musical notation per arrangement)
The notation format promotes keys, piano, violin, and any other
staff-notation instrument to first-class status with their own data
structure, separate from the guitar wire format. Implemented in
`lib/sloppak.py` and `lib/notation.py`; the highway WS streams
`notation_info` + `notation_measures` messages when notation data is
present for the active arrangement.
**Architecture: per-arrangement, not song-wide.** Unlike `drum_tab`
(one drum track per song, top-level manifest key), notation is
per-instrument. A song could carry both `notation_keys.json` and
`notation_violin.json`. The manifest key lives on the **arrangement
entry**, not at the top level.
```yaml
arrangements:
- id: keys
name: Keys
type: piano
notation: notation_keys.json # per-arrangement sub-key
# file: is optional when notation: is present
```
```text
my-song.sloppak/
├── manifest.yaml
├── song_timeline.json
├── notation_keys.json
└── stems/
└── full.ogg
```
**`notation_<id>.json` — file schema:**
```json
{
"version": 1,
"instrument": "piano",
"staves": [
{"id": "rh", "clef": "G2", "label": "Right Hand"},
{"id": "lh", "clef": "F4", "label": "Left Hand"}
],
"measures": [
{
"idx": 1,
"t": 0.0,
"ts": [4, 4],
"ks": 0,
"tempo": 120.0,
"staves": {
"rh": {
"voices": [{"v": 1, "beats": [
{"t": 0.000, "dur": 4, "notes": [{"midi": 64}]},
{"t": 0.500, "dur": 4, "notes": [{"midi": 67}]}
]}]
},
"lh": {
"voices": [{"v": 1, "beats": [
{"t": 0.000, "dur": 1, "notes": [{"midi": 52}, {"midi": 60}]}
]}]
}
}
}
]
}
```
**Top-level fields:**
| Field | Type | Notes |
|---|---|---|
| `version` | int | Always `1`. Bump on breaking schema change |
| `instrument` | string | Mirrors arrangement `type`: `piano`, `violin`, `guitar`, etc. Makes the file self-describing |
| `rights` | string | Optional copyright / rights text (MusicXML `<rights>`). Omit when absent |
| `lyricist` | string | Optional lyricist credit (MusicXML `<creator type="lyricist">`). Omit when absent |
| `arranger` | string | Optional arranger credit (MusicXML `<creator type="arranger">`). Omit when absent |
| `staves` | list | Static staff definitions. Each has `id` (stable, referenced by `measures[].staves` keys), `clef` (see below), and optional `label` |
| `measures` | list | Ordered measure data — the hot path |
**Clef vocabulary** (defined in `lib/notation.py::CLEFS`):
| Value | Meaning |
|---|---|
| `G2` | Treble clef — guitar, violin, flute, piano RH |
| `F4` | Bass clef — bass guitar, cello, piano LH |
| `C3` | Alto clef — viola |
| `C4` | Tenor clef — cello upper register, trombone |
| `neutral` | Unpitched / percussion staff |
**Measure fields:**
| Field | Type | Notes |
|---|---|---|
| `idx` | int | 1-based measure number |
| `t` | float | Time in seconds at measure downbeat |
| `ts` | int[2] | Time signature `[numerator, denominator]`. Omit if unchanged |
| `beat_groups` | int[] | Beat grouping for compound and irregular meters, as a list of integers. Each integer is the count of time-signature denominator units in that primary beat group. The sum must equal the time-signature numerator. E.g. 6/8 → `[3, 3]`; 9/8 → `[3, 3, 3]`; 7/8 → `[2, 2, 3]`; 5/8 → `[2, 3]` or `[3, 2]`. Omit for simple meters (2/4, 3/4, 4/4) where grouping is unambiguous. Renderers translate this to their own beam-grouping API at render time — this field is renderer-agnostic. |
| `ks` | int | Key signature: semitones from C, −7 to +7 (negative = flats, positive = sharps). Omit if unchanged |
| `tempo` | float | BPM. Omit if unchanged |
| `pickup` | bool | `true` when this measure is an anacrusis (pickup / upbeat) shorter than the time signature implies (MusicXML `implicit="yes"`). Renderers suppress the measure number and start counting from the next full measure. Omit when false |
| `staves` | object | Keyed by staff `id`. Each staff has optional `clef` (omit if unchanged) and `voices` |
**Beat fields** (inside `staves → voices → beats`):
| Field | Default | Notes |
|---|---|---|
| `t` | required | Time in seconds |
| `dur` | required | Duration denominator: `1`=whole, `2`=half, `4`=quarter, `8`=eighth, `16`=sixteenth, `32`=thirty-second |
| `dot` | omit | Augmentation dots: `1`=dotted, `2`=double-dotted |
| `rest` | omit | `true` if this beat is a rest; `notes` is omitted |
| `tu` | omit | Tuplet: `[numerator, denominator]`, e.g. `[3, 2]` for triplet |
| `beat_pos` | omit | Exact position within the measure as a rational `[numerator, denominator]` pair, where the denominator is the time-signature denominator. E.g. beat 2 in 6/8 (the second dotted quarter) = `[3, 8]`. Avoids floating-point imprecision when deriving beat position from tempo and absolute time. Omit if not set by the importer. Renderers that do not recognise this field derive position from `t` and the tempo map as before. |
| `notes` | omit | List of note objects (omit for rests) |
| `dyn` | omit | Dynamic: `ppp`, `pp`, `p`, `mp`, `mf`, `f`, `ff`, `fff` |
| `slr` | omit | Slur start |
| `slre` | omit | Slur end |
| `grace` | omit | Grace-note beat, typed: `"a"` = acciaccatura (slashed, steals time from the previous note; MusicXML `<grace slash="yes">`), `"p"` = appoggiatura (unslashed, steals time from the following note; `<grace>`). The beat's `dur` is the grace note's written duration. Vocabulary in `lib/notation.py::GRACE_TYPES` |
| `arp` | omit | `true` when the beat's chord is arpeggiated (rolled; MusicXML `<arpeggiate>`) |
| `ferm` | omit | `true` when the beat carries a fermata (MusicXML `<fermata>`) |
| `spd` / `sph` / `spu` | omit | Sustain pedal: pedal **d**own / **h**old-through-this-beat / **u**p. This is the only pedal encoding — there is deliberately no separate `ped` field. MusicXML mapping: `<pedal type="start">` → `spd`, `<pedal type="change">` → `spu` + `spd` on the same beat (re-pedal), `<pedal type="stop">` → `spu`; beats inside an active pedal span carry `sph` |
| Additional beat effects | omit | `cre`, `dec`, `vib`, `vibw`, `fade`, `pm`, `lr`, `slap`, `pop`, `tap`, `su`, `sd`, `rasg`, `golpe`, `wah`, `txt`, `chrd` — all optional, omit when absent |
**Note fields** (inside `beats → notes`):
| Field | Default | Notes |
|---|---|---|
| `midi` | required | MIDI pitch 0–127. Unambiguous — no string/fret/tuning indirection |
| `tied` | omit | Tied from the previous beat |
| `acc` | omit | Accidental override: `null`/omit = derive from key sig; `0` = force natural (♮); `−2`/`−1`/`1`/`2` = double-flat/flat/sharp/double-sharp |
| `stem` | omit | Force stem direction: `"up"` or `"down"` (MusicXML `<stem>`). Omit to let the renderer decide. Vocabulary in `lib/notation.py::STEM_DIRECTIONS` |
| Additional note effects | omit | `stc`, `ten`, `ac`, `hac`, `vib`, `vibw`, `dead`, `ghost`, `fng`, `rfng`, `str`, `harm`, `bend`, `slide`, `trill`, `ho`, `po`, `tp`, `barre` — all optional |
**Wire format.** `song_info` carries `has_notation: bool`. Notation data
is streamed as two highway-WS message types after `sections`, before `anchors`:
```json
{"type": "notation_info", "version": 1, "instrument": "piano",
"staves": [...], "total": 64}
```
…followed by one or more chunks of 32 measures:
```json
{"type": "notation_measures", "data": [...], "total": 64}
```
`total` is the measure count across **all** chunks. Clients accumulate `data` arrays until the accumulated measure count reaches `total` (an individual chunk's `data.length` says nothing — every full chunk of a multi-chunk stream is shorter than `total`). The `anchors` frame that follows the notation block is a secondary end-of-block signal.
**`lib/notation.py`** is the vocabulary library: `SCHEMA_VERSION`, `CLEFS`, `DURATIONS`, `validate_notation()`, `measure_to_wire()`, `measures_to_wire()`.
**Legacy fallback.** Sloppaks that carry keys as guitar wire format (Clone Hero converted content) continue to work — the notation plugin checks for the `notation` key on the arrangement entry. When absent, it falls back to decoding guitar wire format notes via `midi = s * 24 + f`.
**v1 non-features (accepted limitations).** The following are deliberately
out of schema v1; they ship, if ever, as **additive v1.x patches** (new
optional fields old consumers ignore — the permissive validator passes
unknown fields through by design):
- Microtonal pitch (anything finer than the ±2 semitone `acc` vocabulary).
- Figured bass.
- Mid-measure key-signature, time-signature, or clef changes (all three are
measure-granular in v1).
- Ottava lines (`ott`), repeat/volta barline semantics (`barline`),
ornaments beyond trills (mordents, turns), tremolo (`trem`), and notated
glissando lines (`glis`).
Importers MUST drop these source features with a logged warning rather than
approximate them into wrong notation; renderers MUST NOT invent semantics
for field names from this list before a v1.x patch specifies them.
---
#### Key / scale annotations (for theory-aware visualizations)
`keys.json` mirroring the `sections[]` shape:
```json
{
"version": 1,
"events": [
{"t": 0.0, "key": "Em", "scale": "natural_minor"},
{"t": 64.5, "key": "G", "scale": "major"},
{"t": 142.0, "key": "Em", "scale": "natural_minor"}
]
}
```
Manifest:
```yaml
keys: keys.json
```
Each entry implicitly applies until the next event. Same model as `sections[]`.
#### Vocal pitch contour (a different shape, a different key)
The canonical `vocal_pitch` key + file (defined in §2.4) is the
per-syllable note format consumed by the karaoke plugin —
`{version: 1, notes: [{t, d, midi}]}`. If you want to ship a finer-
grained pitch *contour* (one sample every 20 ms, Hz instead of MIDI),
that's a different shape and should ride on its own manifest key so
the two don't collide:
```yaml
vocal_pitch_contour: vocal_pitch_contour.json
```
```json
{
"version": 1,
"samples": [
{"t": 0.000, "hz": 220.5},
{"t": 0.020, "hz": 222.1}
]
}
```
Per §5.1, manifest keys are cheap — reach for a new one when the
schema diverges, don't overload an existing key with a second shape.
### 5.4. `version` field — always include it
Every new file should have `"version": 1` at the top. It's free insurance: when you change the schema later, `version: 2` consumers can branch on it. Old consumers without that branch ignore the file (or fall back gracefully).
### 5.5. Stay backward-compatible
If you change a field that already shipped:
- **Adding fields** is always safe (older readers ignore them).
- **Removing fields** breaks older readers. Don't.
- **Repurposing fields** (changing meaning or units) is the worst — bump `version` and branch.
If you're tempted to remove or repurpose: leave the old field, add a new one, and sunset the old one over a release or two.
### 5.6. When to put data inside an arrangement vs. its own file
- **Inside arrangement JSON** (`arrangements/lead.json`):
- Data that is *per-arrangement* and *per-instrument* (notes, chords, anchors, hand-shapes — guitar specifics).
- Data that meaningfully differs between Lead and Rhythm versions of the same song.
- **Its own file** (and pointed-at via manifest key):
- Data that is *song-wide* (lyrics, beats, sections, tempo map, drum tab, lighting, key/scale changes).
- Data that may be authored or generated independently of the playable arrangement (a stem split, an AI-generated drum tab).
Beats and sections historically lived inside the first arrangement JSON (early arrangement XML put them there). The `song_timeline.json` file (see §5.3) is the correct home for new sloppaks — the loader reads it first and it takes priority. New song-wide data should always be its own file.
### 5.7. Don't break the manifest contract
A few things that should *not* end up in `manifest.yaml`:
- **Per-machine settings** (DMX universes, IPs, output device picks) — those go in `${CONFIG_DIR}/...json`, not the sloppak.
- **UI state** (last zoom level, panel sizes) — `localStorage` only.
- **User progress / play counts** — Slopsmith stores these in its metadata DB, not in the sloppak.
The sloppak holds **the song's authored data**. Anything that varies by user or by machine is out.
---
## 6. Quick reference — file types you'll touch
| File | Format | Schema lives in | Authority |
|---|---|---|---|
| `manifest.yaml` | YAML | `lib/sloppak.py` (`load_manifest`, `extract_meta`) | This doc + the loader |
| `arrangements/*.json` | JSON | `lib/song.py` (`arrangement_to_wire`, `arrangement_from_wire`) | The wire-format functions |
| `lyrics.json` | JSON (flat list) | `lib/sloppak.py` (passed through to `Song.lyrics`) | This doc §2.3 |
| `song_timeline.json` | JSON | `lib/sloppak.py` (loader) | This doc §5.3 |
| `notation_<id>.json` | JSON | `lib/notation.py` (`validate_notation`, `measures_to_wire`) | This doc §5.3 |
| `stems/*.ogg` | OGG Vorbis | — | Convention: `q:a 5` for size/quality balance |
| `cover.jpg` | JPEG | — | Convention: square, 500–1500 px on a side |
| Your new file | JSON (preferred) | Your plugin's spec doc | You |
---
## 7. Testing your extension
If you add a new file type or manifest key:
1. **Round-trip test**: write a sample, load it, write it back, compare. Add to `tests/test_sloppak.py`.
2. **Backward-compat test**: load a sloppak that *doesn't* have your new key — your code must not crash, and the song must still play.
3. **Hand-edit test**: open the directory form in a text editor, change a field by hand, reload Slopsmith. The format is meant to be hand-editable; your additions should preserve that.
4. **Both forms**: test with both the directory form and the zipped form. The unpack cache is invalidated based on mtime and size, so you can repackage and reload without restarting the server.
The full pytest suite (`pytest`) must stay green before any PR.
---
## 8. Where to look in the code
| For… | Read |
|---|---|
| Format detection, source resolution, zip unpacking | [lib/sloppak.py](../lib/sloppak.py) |
| Data classes (`Note`, `Chord`, `Arrangement`, `Song`, `Phrase`) | [lib/song.py](../lib/song.py) |
| Wire-format helpers (`*_to_wire` / `*_from_wire`) | [lib/song.py](../lib/song.py) |
| The reference sloppak writer | [lib/sloppak_convert.py](../lib/sloppak_convert.py) |
| Drum tab vocabulary and wire helpers | [lib/drums.py](../lib/drums.py) |
| Notation vocabulary and wire helpers | [lib/notation.py](../lib/notation.py) |
| Live streaming over WebSocket (consumes the same shapes) | `server.py` (`/ws/highway/{filename}`) |
| The plugin system (where new viz consumers go) | [CLAUDE.md](../CLAUDE.md) — Plugin System section |
| Tests | [tests/test_sloppak.py](../tests/test_sloppak.py), [tests/test_sloppak_convert.py](../tests/test_sloppak_convert.py) |