feedBack/docs/NOTE_FAILURE_PLAN.md

# Note Failure Feedback — Implementation Plan

Depends on: `docs/NOTE_FAILURE_SPEC.md` (read that first)

---

## Phase 0: Detection Plugin Foundation

**Goal:** Working note detection plugin streaming detected notes via WebSocket.

This phase was previously tracked in a separate NOTE_DETECTION_PLUGIN_PLAN
document (in the `feedBack-plugin-notedetect` repository). The relevant scope
is summarized here to avoid relying on an internal git-only reference:

- [ ] Plugin skeleton: `feedBack-plugin-notedetect/` with plugin.json, routes.py, screen.js
- [ ] Port TonalRecall YIN detection (aubio + sounddevice) to routes.py
- [ ] WebSocket at `/api/plugins/note_detect/stream` streaming `{ note, freq, confidence, time }`
- [ ] Device selection UI in screen.html
- [ ] requirements.txt: aubio, sounddevice, numpy

**Exit criterion:** With plugin active and guitar plugged in, playing a note
causes a JSON event to appear in the browser console.

---

## Phase 1: Note Matching Core

**Goal:** Client-side matching of detected notes to chart notes. No rendering yet —
console logging only.

**Files:**
- `screen.js` in the notedetect plugin

**Tasks:**
- [ ] Implement `expectedFreq(string, fret, tuningOffsets, capo, stringCount, arrangementName)` using
      base open-string frequencies, semitone offsets from `highway.getSongInfo().tuning`,
      and semitone math (`2^(semitones/12)`) rather than assuming 6-string standard
      tuning; use `highway.getStringCount()` as the authoritative string count because
      tuning may be padded to length 6 for RS XML sources even for bass/extended-range
      arrangements; include `highway.getSongInfo().capo` as additional semitones if
      the intent is expected sounding pitch; pass `highway.getSongInfo().arrangement` as
      `arrangementName` to disambiguate 5-string bass vs 5-string guitar (matching the
      spec's `getBaseTuning` helper)
- [ ] Implement `NoteJudgmentTracker` class with:
  - `addDetection(detected)` — correlate with nearest unmatched chart note
  - `update(currentTime)` — expire pending notes whose match window has passed
  - `getJudgmentsInRange(tStart, tEnd)` — return judgments in time range
  - `reset()` — clear all state
- [ ] Connect to detection WebSocket, feed events into tracker
- [ ] Initialize tracker with `highway.getNotes()` and `highway.getChords()` on song ready
- [ ] Console.log each judgment as it resolves (HIT/MISSED/EARLY/LATE/SHARP/FLAT)
- [ ] Re-initialize tracker on `song:ready` (fires on every new song **and** on every
      arrangement switch — no need to hook `highway.reconnect` or other internals);
      do **not** use `song:loaded` — note/chord arrays are still empty at that point (data
      arrives incrementally and only completes at `song:ready`)

**Exit criterion:** Playing along with a song, console shows correct HIT/MISSED
judgments with timing and pitch error values.

**Estimated scope:** ~200 lines JS

---

## Phase 2: Hit/Miss Highway Overlay

**Goal:** Visual feedback on the highway — green glow for hits, red X for misses.

**Files:**
- `screen.js` in the notedetect plugin (draw hook)

**Tasks:**
- [ ] Register `highway.addDrawHook()` that reads judgments from the tracker
- [ ] **Hit rendering:** Green glow ring behind notes at the now-line, fading over
      `hitGlowDuration` seconds. Use `highway.project()` and `highway.fretX()` for
      positioning. Additive blend via `ctx.globalCompositeOperation = 'lighter'`.
- [ ] **Miss rendering:** Red `✕` marker at the note's string/fret position, drawn
      in the "past" region below the now-line. Do **not** rely on
      `highway.project(negative_offset)` for long-lived placement — the current
      renderer returns `null` for offsets more than ~50ms into the past. Instead,
      anchor at the now-line (`highway.project(0)`) and map elapsed time since the
      miss to a linear below-now-line Y position (configurable pixels/second), fading
      the marker after `missMarkerDuration` seconds.
- [ ] **String pulse:** Brief red tint on the missed note's string (200ms fade on
      the string line segment near the now-line).
- [ ] Handle lefty mode: use `highway.fillTextUnmirrored()` for text markers.
- [ ] Cleanup: `highway.removeDrawHook()` on plugin destroy.

**Exit criterion:** Playing a song, you see green flashes on hit notes and red X
markers scrolling past on missed notes.

**Estimated scope:** ~150 lines JS

---

## Phase 3: Diagnostic Labels (Timing + Pitch)

**Goal:** Show *why* a note was missed — too early, too late, sharp, flat.

**Files:**
- `screen.js` in the notedetect plugin

**Tasks:**
- [ ] Extend draw hook to render timing indicators:
  - EARLY: orange `↑` + "-XXms" label above the miss marker
  - LATE: orange `↓` + "+XXms" label below the miss marker
  - Only shown when timing error exceeds `timingThresholdMs`
- [ ] Extend draw hook to render pitch indicators:
  - SHARP: blue `♯` + "+XX¢" label
  - FLAT: blue `♭` + "-XX¢" label
  - Only shown when pitch error exceeds `pitchThresholdCents`
- [ ] Compound states: stack timing label on top, pitch label below
- [ ] Add settings UI in plugin settings panel for threshold configuration
- [ ] Ensure labels don't overlap — offset vertically when multiple notes
      miss at close timestamps

**Exit criterion:** Playing intentionally early/late or bending sharp/flat
shows the correct diagnostic labels.

**Estimated scope:** ~100 lines JS, ~30 lines settings HTML

---

## Phase 4: Loop Iteration Tracking

**Goal:** Track performance across loop iterations, show summary on each wrap.

**Files:**
- `screen.js` in the notedetect plugin

**Tasks:**
- [ ] Detect loop wrap: `currentTime < previousTime - 0.5` in the frame update
- [ ] On wrap: snapshot `{ hits, misses, total, percentage }` to `loopHistory[]`
- [ ] Reset judgments for notes in `[loopA, loopB]` range (keep tracker alive
      for notes outside the loop)
- [ ] Render loop summary overlay (top-center, semi-transparent background):
  ```
  Loop N  |  X/Y notes (Z%)  |  Best: W%
  ```
  Displayed for 1.5s, then fades.
- [ ] Track `bestIteration` across all iterations for "Best" display
- [ ] Emit `loop:complete` event via `window.feedBack.emit()` so other plugins
      (practice journal) can record the data
- [ ] Reset loop history when loop boundaries change or loop is cleared

**Exit criterion:** Looping a 4-bar phrase, you see iteration count and accuracy
flash briefly at each loop wrap. Best score persists across iterations.

**Estimated scope:** ~120 lines JS

---

## Phase 5: Section Grading

**Goal:** Grade each song section (intro, verse, chorus, solo) and surface weak spots.

**Files:**
- `screen.js` in the notedetect plugin

**Tasks:**
- [ ] Use `highway.getSections()` to identify section boundaries
- [ ] Track hits/misses per section as notes are judged
- [ ] At section boundaries (when `currentTime` crosses a section end),
      briefly flash the section grade:
  - A: 90%+, B: 75%+, C: 60%+, D: 40%+, F: below 40%
  - Color: green (A/B), yellow (C), red (D/F)
- [ ] After song completes (or at any point via a hotkey), show a section
      summary panel listing all sections with grades
- [ ] Highlight lowest-scoring section with a "Loop this section" button
      that sets A-B points to that section's boundaries
- [ ] Emit `note:sectionGrade` event for other plugins

**Exit criterion:** Playing through a song, section grades flash at each
transition. Lowest section is highlighted for targeted practice.

**Estimated scope:** ~150 lines JS, ~40 lines HTML

---

## Phase 6: Polish + Settings

**Goal:** Configurable thresholds, visual polish, performance.

**Tasks:**
- [ ] Full settings panel in plugin settings HTML:
  - Match window slider (100-500ms)
  - Pitch tolerance slider (20-100 cents)
  - Toggle timing/pitch labels
  - Toggle loop summary
  - Miss marker duration slider
- [ ] Performance: ensure draw hook stays under 1ms per frame
  - Pre-compute judgment positions, don't recalculate in draw loop
  - Binary search over judgments by time (same pattern as `drawNotes`)
- [ ] Smooth animations: glow/fade using eased alpha, not linear
- [ ] Color-blind accessible palette option (use shapes not just colors)
- [ ] Persist settings in plugin-local storage (e.g. `localStorage` prefixed with
      plugin id) — do **not** use `/api/settings` for this; the current server only
      persists a fixed set of known keys and will silently discard `notedetect_feedback`

**Estimated scope:** ~100 lines JS, ~60 lines HTML

---

## Dependency Graph

```
Phase 0 (detection plugin)
    ↓
Phase 1 (matching core)
    ↓
Phase 2 (hit/miss overlay)  ← Minimum viable feature
    ↓
Phase 3 (diagnostic labels)
    ↓
Phase 4 (loop tracking)     ← Core practice value
    ↓
Phase 5 (section grading)
    ↓
Phase 6 (polish)
```

Phases 3-5 are independent of each other and can be done in any order after Phase 2.
Phase 6 should be last.

---

## Risk / Open Questions

1. **Latency budget:** Detection → WebSocket → matching → render adds latency.
   If total pipeline > 100ms, the match window needs to compensate with asymmetric
   tolerance (more lenient for "late" detections). Measure in Phase 1.

2. **Chord matching granularity:** Current plan matches chord notes individually.
   Should a chord be "missed" if 4/6 notes hit? Propose: grade chords as
   percentage, treat as HIT if ≥50% of notes matched. Revisit after Phase 2 testing.

3. **Tempo-scaled thresholds:** At 200 BPM, a 200ms match window covers almost
   an entire beat. Should thresholds scale with tempo? Propose: don't over-engineer
   this initially. Fixed thresholds work for most tempos. Revisit if users report
   issues at extreme tempos.

4. **Detection plugin availability:** Everything in Phases 1-6 degrades gracefully
   if the detection WebSocket isn't connected — the draw hook simply has no
   judgments to render, and the highway looks exactly as it does today.