Clean release snapshot

2026-08-10 18:59:56 +00:00 · 2026-06-16 18:47:13 +02:00
commit 6c110398b4
574 changed files with 162566 additions and 0 deletions
@@ -0,0 +1,390 @@
+# Note Failure Feedback — Technical Spec
+
+## Goal
+
+When a user loops over a lick, **show note misses on the highway** with diagnostic
+detail: which note was missed, and *how* it was missed (timing vs pitch).
+
+Many rhythm-practice tools show a `!` marker at the missed note position after it
+passes. We improve on this by showing *why* the note was missed — too early, too
+late, wrong pitch, or not played at all.
+
+---
+
+## Prerequisites
+
+This feature depends on the **note detection plugin** (`slopsmith-plugin-notedetect`),
+which provides real-time pitch detection via server-side aubio/YIN over WebSocket.
+The detection plugin streams `DetectedNote` events; this spec describes the
+**matching, judgment, and rendering** layer that consumes those events.
+
+Without the detection plugin active, no miss/hit feedback is shown — the highway
+renders exactly as it does today.
+
+---
+
+## Architecture
+
+```
+Guitar → USB Adapter → sounddevice (server)
+                          ↓
+                   aubio YIN detection
+                          ↓
+              WebSocket: detected notes
+                          ↓
+              ┌───────────────────────┐
+              │   Note Matcher        │  ← THIS SPEC
+              │   (client-side JS)    │
+              │                       │
+              │ Chart notes (highway) │
+              │ × Detected notes (WS) │
+              │ = Match/Miss/Extra    │
+              └───────────────────────┘
+                          ↓
+              Highway draw hook overlay
+              (hit glow, miss markers, diagnostics)
+```
+
+### Data Flow
+
+1. **Chart notes** arrive via existing highway WebSocket (`/ws/highway/{filename}`).
+   Wire format: `{ t, s, f, sus, bn, ho, po, ... }` (see `lib/song.py:note_to_wire`)
+
+2. **Detected notes** arrive via detection plugin WebSocket
+   (`/api/plugins/note_detect/stream`).
+   Wire format: `{ note: "A2", freq: 110.0, confidence: 0.92, time: 1.234 }`
+
+   > **Plugin naming note:** The detection plugin's repository is named
+   > `slopsmith-plugin-notedetect`, but the plugin registers with the id
+   > `note_detect` (snake_case). Its HTTP/WebSocket routes therefore appear
+   > under `/api/plugins/note_detect/…`. There is no `window.slopsmithPlugin_*`
+   > global pattern in Slopsmith — to check whether the detection plugin is
+   > available at runtime, attempt a fetch to `/api/plugins/note_detect/status`
+   > (or similar) or consult the `/api/plugins` list. Use the repo name only
+   > in documentation links.
+
+3. **Note Matcher** (new, client-side) correlates these two streams in real-time.
+
+4. **Draw hook** renders results on the highway via `highway.addDrawHook()`.
+
+   > **⚠ Limitation:** `addDrawHook()` is only invoked by the **default 2D renderer**.
+   > If the user has switched to a custom renderer (e.g., a WebGL 3D highway plugin),
+   > draw hooks are not called and this overlay will be invisible. Implementers should
+   > note this in the plugin's UI (e.g., a warning banner when a non-default renderer
+   > is detected) and may want to explore a renderer-agnostic overlay approach
+   > (own canvas + own rAF loop, reading public highway state via getters) as a
+   > future improvement.
+
+---
+
+## Note Matching Algorithm
+
+### Match Window
+
+A detected note matches a chart note when:
+
+| Criterion      | Threshold              | Notes                                      |
+|----------------|------------------------|---------------------------------------------|
+| **Time**       | ±200ms (configurable)  | Centered on chart note time                 |
+| **Pitch**      | ±50 cents              | Accounts for imperfect intonation           |
+| **String**     | Pitch-only (for now)   | `DetectedNote` carries no string field; exact-string matching requires the detection plugin to be extended to emit a string estimate. Treat string as always-unknown until that extension lands. |
+
+### Expected Frequency Calculation
+
+```javascript
+// Open-string base frequencies (Hz), string index 0 = lowest string.
+// Select by highway.getStringCount() + arrangement name from highway.getSongInfo():
+//   4-string          → BASS_TUNING    (E1 A1 D2 G2)
+//   5-string bass     → BASS5_TUNING   (B0 E1 A1 D2 G2)
+//   6-string (default)→ GUITAR_TUNING  (E2 A2 D3 G3 B3 E4)
+//   7-string          → GUITAR7_TUNING (B1 E2 A2 D3 G3 B3 E4)
+const GUITAR_TUNING  = [82.41, 110.00, 146.83, 196.00, 246.94, 329.63];
+const BASS_TUNING    = [41.20,  55.00,  73.42,  98.00];
+const BASS5_TUNING   = [30.87,  41.20,  55.00,  73.42,  98.00];
+const GUITAR7_TUNING = [61.74,  82.41, 110.00, 146.83, 196.00, 246.94, 329.63];
+
+function getBaseTuning(stringCount, arrangementName) {
+    const isBass = /bass/i.test(arrangementName || '');
+    if (stringCount === 4) return BASS_TUNING;
+    if (stringCount === 5 && isBass) return BASS5_TUNING;
+    if (stringCount === 7) return GUITAR7_TUNING;
+    return GUITAR_TUNING;  // 6-string or unknown
+}
+
+function expectedFreq(string, fret, tuningOffsets, capo = 0, stringCount, arrangementName = '') {
+    // tuningOffsets: per-string semitone offsets from standard (from song metadata).
+    // stringCount: REQUIRED — pass highway.getStringCount(). Do NOT default to
+    // tuningOffsets.length: RS XML sources pad the tuning array to length 6 even for
+    // 4-string bass, which would cause incorrect base-tuning selection.
+    // arrangementName: pass highway.getSongInfo().arrangement to resolve ambiguous
+    // 5-string cases (5-string bass vs 5-string extended guitar).
+    const BASE = getBaseTuning(stringCount, arrangementName);
+    if (string < 0 || string >= stringCount || string >= BASE.length) {
+        return null;
+    }
+    const semitones = tuningOffsets[string] + capo;
+    const base = BASE[string] * Math.pow(2, semitones / 12);
+    return base * Math.pow(2, fret / 12);
+}
+```
+
+### Match States
+
+Each chart note resolves to a **judgment** with two independent axes:
+
+- **Timing axis** (`timingState`): `'OK'` if within `timingThresholdMs`, `'EARLY'` if matched
+  more than `timingThresholdMs` before chart time, `'LATE'` if more than `timingThresholdMs`
+  after. `null` if the note was never matched (MISSED).
+- **Pitch axis** (`pitchState`): `'OK'` if detected pitch is within `pitchThresholdCents` of
+  expected, `'SHARP'` if above by more than `pitchThresholdCents`, `'FLAT'` if below. `null`
+  if unmatched.
+- **`hit`**: `true` when both axes are `'OK'`; `false` for MISSED or any off-axis result.
+
+The axes combine independently (e.g., `LATE + FLAT`, `EARLY + SHARP`). The state diagram
+below shows possible terminal values per axis:
+
+```
+PENDING  →  hit=true,  timingState='OK',    pitchState='OK'
+         →  hit=false, timingState=null,    pitchState=null   (MISSED — window expired)
+         →  hit=false, timingState='EARLY', pitchState=...    (too early)
+         →  hit=false, timingState='LATE',  pitchState=...    (too late)
+         →  hit=false, timingState='OK',    pitchState='SHARP'
+         →  hit=false, timingState='OK',    pitchState='FLAT'
+```
+
+Timing and pitch thresholds are read from configuration (`timingThresholdMs`,
+`pitchThresholdCents`) — never hard-coded.
+
+### Judgment Data Structure
+
+```javascript
+// Per-note judgment, attached after the note passes the now-line.
+// Compound judgments (e.g. LATE + FLAT) are expressed as separate
+// timingState / pitchState fields; never concatenate them into `state`.
+{
+    chartNote: { t, s, f, ... },      // Original chart note
+
+    // Overall outcome — top-level quick check
+    hit: false,                        // true iff timing AND pitch are both clean
+
+    // Timing axis: null if no detection arrived (pure MISSED)
+    timingState: 'EARLY' | 'LATE' | 'OK' | null,
+    timingError: -120,                // Milliseconds (negative = early); null if no detection
+
+    // Pitch axis: null if no detection arrived (same condition as timingState).
+    // Pitch is evaluated independently for any matched detection — a LATE note
+    // can also be FLAT (both axes are set even when timingState ≠ 'OK').
+    pitchState: 'SHARP' | 'FLAT' | 'OK' | null,
+    pitchError: +15,                  // Cents (positive = sharp); null if no detection
+
+    // Raw detection data (null if no detection arrived)
+    detectedFreq: 112.3,             // What was actually played
+    expectedFreq: 110.0,             // What should have been played
+    detectedAt: 1.354,               // When the detection arrived
+}
+```
+
+**Precedence / rendering rules:**
+- `hit: true` → green glow; both timing/pitch states will be `'OK'`. A judgment is a hit when `|timingError| ≤ timingThresholdMs` **and** `|pitchError| ≤ pitchThresholdCents` (see §Configuration).
+- `timingState: null` (no detection) → pure miss (`✕`); skip pitch display.
+- Non-null `timingState` + non-null `pitchState` → compound: render timing
+  indicator on top, pitch indicator below.
+- Emitted `note:hit` / `note:miss` events carry the full judgment object so
+  subscribers can inspect either axis independently.
+
+---
+
+## Highway Rendering
+
+### Hit Feedback
+
+Notes matched within `timingThresholdMs` (default 100 ms) **and**
+`pitchThresholdCents` (default 20 ¢) get a **green glow ring** that fades over
+`hitGlowDuration` (default 0.5 s). This is the combination that sets `hit: true`
+on the judgment object. The existing note rendering is unchanged — the glow is
+drawn *behind* the note at the now-line position as it passes.
+
+```
+  [existing note bubble]
+  └── green glow ring (additive blend, fades)
+```
+
+### Miss Markers
+
+Missed notes get a persistent marker that continues downward past the now-line
+and remains visible for 2 seconds (configurable). The marker stays at the
+note's string/fret position on the "past" portion of the highway (below
+now-line).
+
+**Positioning rule:** do **not** rely on `highway.project(tOffset)` for the full
+miss-marker lifetime below the now-line. That helper returns `null` for offsets
+more than ~50ms into the past, so it cannot place markers that persist for
+seconds after the note passes. Instead, define a dedicated mapping anchored at
+the now-line:
+
+- `tOffset = 0` starts at the now-line (use `highway.project(0)` to get this Y).
+- For the past region (`tOffset < 0` up to `-missMarkerDuration`), place the
+  marker below the now-line using a configurable linear pixels-per-second mapping,
+  clamped to the visible past area.
+- The existing `highway.project()` may still be used for positions at/above the
+  now-line (approaching notes in the last ~50ms), but once a marker has crossed
+  into the past region its Y is governed by this below-now-line mapping.
+
+| State  | Visual                                                      |
+|--------|-------------------------------------------------------------|
+| MISSED | Red `✕` at note position + red tint on string segment       |
+| EARLY  | Orange `↑` (up arrow) + timing offset label (e.g., "-120ms")|
+| LATE   | Orange `↓` (down arrow) + timing offset label ("+85ms")     |
+| SHARP  | Blue `♯` + cents label ("+35¢")                             |
+| FLAT   | Blue `♭` + cents label ("-42¢")                             |
+
+Compound states stack vertically: timing indicator on top, pitch indicator below.
+
+### Miss markers on the string area
+
+Below the now-line, all strings for the active arrangement are always visible
+(use `highway.getStringCount()` — 4 for bass, 6 for guitar, 7+ for extended-range).
+For a missed note,
+the relevant string segment between the now-line and ~20px below it gets a brief
+red pulse (200ms fade).
+
+### Loop Iteration Summary
+
+When A-B looping is active, at the end of each loop iteration (when playback
+wraps from B back to A), show a brief overlay:
+
+```
+┌─────────────────────┐
+│  Loop 3/∞           │
+│  5/7 notes hit (71%)│
+│  Best: 6/7 (86%)    │
+└─────────────────────┘
+```
+
+Displayed for 1.5s, then fades. Does not block the highway.
+
+---
+
+## State Management
+
+### NoteJudgmentTracker
+
+Client-side class that manages the correlation between chart notes and detections.
+
+```javascript
+class NoteJudgmentTracker {
+    constructor(chartNotes, chartChords, tuning) { ... }
+
+    // Called when a detected note arrives from the detection WebSocket
+    addDetection(detected) { ... }
+
+    // Called each frame; checks for expired match windows
+    update(currentTime) { ... }
+
+    // Returns judgments for notes in the visible time range
+    getJudgmentsInRange(tStart, tEnd) { ... }
+
+    // Reset (on song change, loop restart, arrangement switch)
+    reset() { ... }
+
+    // Stats for the current loop iteration
+    getLoopStats() { ... }
+}
+```
+
+### Memory Management
+
+- Judgments older than 10 seconds behind current time are pruned each frame.
+- Detection buffer holds last 5 seconds of raw detections.
+- On loop wrap (B→A), archive current iteration stats, reset judgments for
+  the loop range, keep detections flowing.
+
+### Loop-Aware Behavior
+
+The tracker must handle A-B looping:
+
+1. Detect loop wrap: `currentTime < previousTime - 0.5` (jumped backward).
+2. On wrap: snapshot current stats to `loopHistory[]`, reset judgments
+   for notes in `[loopA, loopB]` range.
+3. `getLoopStats()` returns current iteration + best historical iteration.
+
+---
+
+## Integration Points
+
+### Existing Highway API Used
+
+| API                          | Purpose                                  |
+|------------------------------|------------------------------------------|
+| `highway.addDrawHook(fn)`    | Register the overlay renderer            |
+| `highway.removeDrawHook(fn)` | Cleanup on plugin unload                 |
+| `highway.getTime()`          | Current chart time (audio-aligned)       |
+| `highway.getAvOffset()`      | A/V offset in ms; visual render clock = `getTime() + getAvOffset()/1000` — use this when computing `tOffset` for `project()` calls inside draw hooks, otherwise markers appear shifted when the user has calibrated A/V latency |
+| `highway.getNotes()`         | All chart notes (for matching)           |
+| `highway.getChords()`        | All chart chords (match individual notes)|
+| `highway.getSections()`      | Section boundaries (for section grading) |
+| `highway.getSongInfo()`      | Tuning offsets for frequency calculation |
+| `highway.project(tOffset)`   | Convert time offset to Y position        |
+| `highway.fretX(fret, scale, w)` | Convert fret to X position using `scale` from `highway.project(tOffset)` |
+| `highway.fillTextUnmirrored` | Text that stays readable in lefty mode   |
+
+### Existing App.js Used
+
+| Global                       | Purpose                                  |
+|------------------------------|------------------------------------------|
+| `loopA`, `loopB`             | Current A-B loop boundaries              |
+| `audio.currentTime`          | Actual audio playback position           |
+
+### New Events Emitted (via `window.slopsmith.emit`)
+
+| Event                        | Payload                                  |
+|------------------------------|------------------------------------------|
+| `note:hit`                   | full `Judgment` object (see §Judgment Data Structure) |
+| `note:miss`                  | full `Judgment` object                   |
+| `loop:complete`              | `{ iteration, stats }`                   |
+| `note:sectionGrade`          | `{ section, grade, hits, total }`        |
+
+`note:hit` and `note:miss` always carry the complete `Judgment` object so
+subscribers can inspect `timingState`, `pitchState`, timing/pitch errors, and
+raw detection data independently, without the emitter having to pre-select fields.
+
+---
+
+## Configuration (plugin settings)
+
+There are three distinct threshold tiers — keep them conceptually separate:
+
+| Tier | Setting(s) | Role |
+|------|-----------|------|
+| **Match window** | `matchWindowMs`, `pitchToleranceCents` | Outer gate: a detection is only correlated to a chart note if it falls within these limits. Outside → ignored (extra note, not an attempt). |
+| **Hit threshold** | `timingThresholdMs`, `pitchThresholdCents` | Sets `hit: true`. A matched note is a clean hit when `|timingError| ≤ timingThresholdMs` **and** `|pitchError| ≤ pitchThresholdCents`. Also triggers the green glow. Must be ≤ match window values. |
+| **Label threshold** | (same keys) | Same values double as the boundary at which EARLY/LATE/SHARP/FLAT labels appear. Within the hit threshold = `'OK'` state; outside = labeled state. |
+
+| Setting                | Default | Description                                                    |
+|------------------------|---------|----------------------------------------------------------------|
+| `matchWindowMs`        | 200     | Outer time tolerance for correlating a detection to a chart note (ms) |
+| `pitchToleranceCents`  | 50      | Outer pitch tolerance for correlation (cents)                  |
+| `timingThresholdMs`    | 100     | `|timingError| ≤ this` → `timingState: 'OK'` and `hit` eligible; also defines the EARLY/LATE label boundary |
+| `pitchThresholdCents`  | 20      | `|pitchError| ≤ this` → `pitchState: 'OK'` and `hit` eligible; also defines the SHARP/FLAT label boundary |
+| `showTimingErrors`     | true    | Show EARLY/LATE labels when `timingState` is non-OK            |
+| `showPitchErrors`      | true    | Show SHARP/FLAT labels when `pitchState` is non-OK             |
+| `missMarkerDuration`   | 2.0     | How long miss markers stay visible (sec)                       |
+| `showLoopSummary`      | true    | Show stats on loop wrap                                        |
+| `hitGlowDuration`      | 0.5     | Green glow fade time (sec)                                     |
+
+Persist these settings in plugin-local storage (e.g. `localStorage` prefixed
+with the plugin id). Do **not** assume they can be saved through Slopsmith's
+`/api/settings` endpoint under a `notedetect_feedback` key — the current server
+only persists a fixed set of known settings keys. If backend support for a
+dedicated persisted key is added later, this plugin may migrate to `/api/settings`.
+
+---
+
+## What This Does NOT Cover
+
+- **Audio input / pitch detection** — handled by the detection plugin
+- **Device selection UI** — handled by the detection plugin
+- **Score persistence / history** — future work (practice journal plugin)
+- **Difficulty scaling** — automatic dynamic-difficulty scaling is not implemented
+- **Chord grading** — chords are graded per-note (each note in the chord
+  is independently matched), not as a single unit