mirror of
https://github.com/got-feedBack/feedBack.git
synced 2026-09-11 13:04:29 +00:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Byron GamatosandBret Mogilefsky
|
cc843245e1 |
@@ -199,11 +199,6 @@ usually an unrelated plugin's per-frame DOM work.
|
||||
|
||||
Slopsmith supports two ways for a plugin to participate in the main player's visuals. They coexist; the setRenderer contract is the default for any viz that draws a highway-shaped surface, and overlays handle layered decorations on top.
|
||||
|
||||
**Deeper references** (in [`docs/`](docs/)):
|
||||
- [`realtime-scoring-pipeline.md`](docs/realtime-scoring-pipeline.md) — end-to-end trace from audio sample → lit gem, across native engine, IPC, `note_detect`, highway, renderer.
|
||||
- [`note-state-provider.md`](docs/note-state-provider.md) — full contract for `setNoteStateProvider` / `bundle.getNoteState` (key formats, sustain semantics, alpha ownership).
|
||||
- [`visualization-feedback-guide.md`](docs/visualization-feedback-guide.md) — practical walkthrough with a minimal 2D Canvas example + troubleshooting decision tree.
|
||||
|
||||
**Pick the right shape:**
|
||||
- Replacing the whole highway drawing on the existing highway canvas (your renderer owns its rendering context / resources; `createHighway()` still owns the canvas element and the rAF loop)? → **setRenderer** (section 1). Enters the viz picker. Works in both the main player and per-panel under splitscreen.
|
||||
- Adding a layer on top of whichever viz is active? → **Overlay** (section 2). Navbar toggle, not in the picker.
|
||||
@@ -367,8 +362,6 @@ A previous standalone-pane contract (`window.createMyVisualization({ container }
|
||||
|
||||
#### 3. Note-state provider — for scorers that want renderers to "light up" notes (slopsmith#254)
|
||||
|
||||
> **Full reference**: [`docs/note-state-provider.md`](docs/note-state-provider.md) covers the contract in depth (key formats, sustain `'active'` vs `'hit'` semantics, alpha-ownership, common pitfalls). The summary below is the API at a glance.
|
||||
|
||||
A scoring plugin (note_detect) can publish a per-note judgment so whichever renderer is active draws the **gem itself** lit on a correct hit, and keeps a sustain trail glowing while it's still being played correctly — instead of a separate overlay ring floating near the note.
|
||||
|
||||
```js
|
||||
|
||||
@@ -1,180 +0,0 @@
|
||||
# Note-state provider API
|
||||
|
||||
The contract that lets a scorer (today: `note_detect`) tell every renderer (today: built-in 2D, bundled 3D, anything you build) when a chart note has been **hit / actively held / missed**.
|
||||
|
||||
Introduced in [slopsmith#254](https://github.com/byrongamatos/slopsmith/issues/254). See [`realtime-scoring-pipeline.md`](realtime-scoring-pipeline.md) for how data flows into the provider; this doc focuses on the API itself.
|
||||
|
||||
**Audience**: plugin authors who want to either (a) **register** as the scorer (replacing `note_detect`), or (b) **read** per-note state from a custom visualization.
|
||||
|
||||
## The API in one minute
|
||||
|
||||
Two endpoints on the `highway` object, one bundle field exposed to renderers:
|
||||
|
||||
```js
|
||||
// Producer side (a scorer plugin)
|
||||
highway.setNoteStateProvider((note, chartTime) => {
|
||||
// return null / 'hit' / 'active' / 'miss' / { state, alpha, color }
|
||||
});
|
||||
highway.getNoteStateProvider(); // current provider, or null
|
||||
highway.setNoteStateProvider(null); // clear
|
||||
|
||||
// Consumer side (inside your renderer's draw)
|
||||
const st = bundle.getNoteState(note, chartTime); // same return shape
|
||||
```
|
||||
|
||||
**One provider at a time** — last writer wins. There's no event-emitter / multi-subscriber pattern; the contract is deliberately singular so the renderer's per-frame call is allocation-free and unambiguous. If two scorer plugins coexist, the second one's `setNoteStateProvider` call replaces the first.
|
||||
|
||||
The provider is called **per visible note, per frame, from the renderer's `draw()` loop**. Keep it cheap — a typical highway can have 30+ visible gems per frame across 60 fps. The bundled `note_detect` provider does one Map lookup + a few arithmetic operations.
|
||||
|
||||
## Return values
|
||||
|
||||
| Return | Meaning | When to use |
|
||||
|---|---|---|
|
||||
| `null` / `undefined` / `false` / `0` / `""` | No state for this note this frame. Renderer draws it normally. | The vast majority of returns. Default for unstruck notes, expired effects, notes the scorer doesn't know about. |
|
||||
| `'hit'` | Note was struck cleanly. | Right after a successful detection, while the brief post-strike glow is visible. |
|
||||
| `'active'` | A sustained note is *currently being held on-pitch*. | While a sustain trail should glow. Re-emit every frame the hold is still valid; stop returning state when the sustain ends. |
|
||||
| `'miss'` | Note expired without a clean detection. | After the scoring window closed and no hit was registered. |
|
||||
| `{ state, alpha, color }` | Full object form. Same `state`. Adds `alpha` (0..1 brightness multiplier) and optional `color` override. | Whenever you want a custom fade or to tint with something other than the renderer's default string colour. |
|
||||
|
||||
### `alpha`
|
||||
|
||||
A brightness multiplier in `[0, 1]`. **The provider owns the fade.** The renderer doesn't track timestamps or decay; it just renders whatever brightness you return.
|
||||
|
||||
- Returning a bare string (`'hit'`) is equivalent to `{ state: 'hit', alpha: 1, color: null }`.
|
||||
- Returning `{ state: 'hit', alpha: 0 }` is the same as returning `null` — the highway clamps it and drops the result before the renderer sees it. Use this as a clean way to signal "stop rendering" without changing your control flow.
|
||||
- Out-of-range numbers are clamped to `[0, 1]`. Non-finite (`NaN`, `Infinity`) falls back to `1`.
|
||||
|
||||
### `color`
|
||||
|
||||
Optional CSS-style colour string (e.g., `'#7ef'`, `'rgb(255, 90, 90)'`). When present, the renderer uses it instead of the default string colour for the lit gem / sustain trail. When absent or non-string, the renderer keeps its default palette.
|
||||
|
||||
## What the highway does to your return value
|
||||
|
||||
[`static/highway.js`](../static/highway.js) line 290:
|
||||
|
||||
```js
|
||||
function _noteState(note, chartTime) {
|
||||
if (!_noteStateProvider) return null;
|
||||
let raw;
|
||||
try { raw = _noteStateProvider(note, chartTime); } catch (e) { return null; }
|
||||
if (!raw) return null;
|
||||
const state = typeof raw === 'string' ? raw : raw.state;
|
||||
if (state !== 'hit' && state !== 'active' && state !== 'miss') return null;
|
||||
const alpha = (raw && typeof raw === 'object' && Number.isFinite(raw.alpha))
|
||||
? Math.max(0, Math.min(1, raw.alpha))
|
||||
: 1;
|
||||
if (alpha <= 0) return null;
|
||||
const color = (raw && typeof raw === 'object' && typeof raw.color === 'string') ? raw.color : null;
|
||||
return { state, alpha, color };
|
||||
}
|
||||
```
|
||||
|
||||
Practical implications:
|
||||
|
||||
- **Provider exceptions are swallowed.** If your provider throws, the renderer sees `null` for that note this frame. The next frame's call is independent. Cleanest for the renderer (no draw stall), trickiest for the provider author (no console signal that something's wrong). Log inside your provider if you need to diagnose.
|
||||
- **Invalid `state` strings return `null`.** Typos like `'Hit'` or `'hits'` are silently rejected. The renderer keeps drawing the note dim.
|
||||
- **`alpha === 0` returns `null`.** The dim-render path is what you want here anyway; this is the cleanest exit.
|
||||
- **Renderers receive the normalized object** (`{ state, alpha, color }`) or `null`. They never see your raw return value, so feel free to return shorthands.
|
||||
|
||||
## How to call `bundle.getNoteState` from a renderer
|
||||
|
||||
The bundle passed to `draw(bundle)` carries a stable reference (`getNoteState: _noteState` — same function every frame, never reallocated). You can cache it on the renderer instance:
|
||||
|
||||
```js
|
||||
init(canvas, bundle) {
|
||||
this.ctx = canvas.getContext('2d');
|
||||
this.getNoteState = bundle.getNoteState; // safe to cache the function ref
|
||||
},
|
||||
draw(bundle) {
|
||||
for (const n of bundle.notes) {
|
||||
const st = this.getNoteState(n, n.t);
|
||||
if (st && st.state === 'hit') /* paint bright */;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Two pitfalls when calling it
|
||||
|
||||
1. **Pass `chord.t`, not `cn.t`, for chord constituents.** Chord constituents are keyed in the producer by the chord's time, not the constituent's. (Most chart formats give the constituent the same `t` as the chord anyway, but the contract is explicit and you should follow it.)
|
||||
```js
|
||||
for (const ch of bundle.chords) {
|
||||
for (const cn of ch.notes) {
|
||||
const st = bundle.getNoteState(cn, ch.t); // ← ch.t
|
||||
}
|
||||
}
|
||||
```
|
||||
2. **Don't cache the return value across frames.** It encodes a fade. Re-call every frame.
|
||||
|
||||
## How to register as a provider
|
||||
|
||||
```js
|
||||
function myNoteStateFor(note, chartTime) {
|
||||
// note: { t, s, f, sus, … } — chart note object (or a chord constituent)
|
||||
// chartTime: number — `t` for single notes, the chord's `t` for chord constituents
|
||||
const verdict = myStore.lookup(`${chartTime}_${note.s}_${note.f}`);
|
||||
if (!verdict) return null;
|
||||
const age = performance.now() - verdict.timestamp;
|
||||
if (age > 800) return null; // owned fade — drop after 800 ms
|
||||
return {
|
||||
state: verdict.hit ? 'hit' : 'miss',
|
||||
alpha: Math.max(0, 1 - age / 800), // linear fade
|
||||
};
|
||||
}
|
||||
|
||||
// Mount on every highway in the session. In a typical plugin:
|
||||
// - call once on `song:ready`, after `createHighway()` has wired the panel
|
||||
// - call again whenever a new panel mounts (splitscreen)
|
||||
highway.setNoteStateProvider(myNoteStateFor);
|
||||
```
|
||||
|
||||
To play nicely with other plugins (or with `note_detect`'s default registration), check what's already there:
|
||||
|
||||
```js
|
||||
const existing = highway.getNoteStateProvider();
|
||||
if (existing == null) {
|
||||
highway.setNoteStateProvider(myNoteStateFor);
|
||||
} else {
|
||||
// Decide: chain (fallback to existing if you return null),
|
||||
// replace (existing.foo behaviour is lost), or abort.
|
||||
}
|
||||
```
|
||||
|
||||
## Keying conventions used by `note_detect`
|
||||
|
||||
If your provider is replacing or extending `note_detect`'s, match its key format:
|
||||
|
||||
| Note shape | Key in `noteResults` |
|
||||
|---|---|
|
||||
| Single note at `t = 12.34`, string `s`, fret `f` | `"12.34_{s}_{f}"` |
|
||||
| Chord constituent (chord at `t = 12.34`, constituent string `s`, fret `f`) | `"12.34_{s}_{f}"` — keyed by **chord time**, not constituent time |
|
||||
| Chord-frame verdict for that chord | `"12.34_chord"` |
|
||||
|
||||
The string interpolation uses the chart's `t` directly (no rounding, no string coercion beyond `${}`). Don't `.toFixed(2)` it — small floating-point drift between chord and constituent `t` values would create a key mismatch.
|
||||
|
||||
## Sustain state — `'hit'` vs `'active'`
|
||||
|
||||
Both are "lit" states; the difference is *who owns the fade*:
|
||||
|
||||
- `'hit'`: the gem was struck cleanly. The provider returns this once per frame, decaying `alpha` over a few hundred ms, then returns `null` when the glow should end.
|
||||
- `'active'`: a sustained note (one with `sus > 0`) is currently being held on-pitch. The provider keeps returning `{ state: 'active', alpha: 1 }` every frame as long as the hold is valid. When the hold ends (pitch slipped, sustain expired, song paused), the provider stops returning state.
|
||||
|
||||
A renderer typically treats both the same way visually — bright gem, bright sustain trail — but `'active'` differs in *duration*: it stays for as long as the hold lasts (potentially seconds), while `'hit'` is a short post-strike glow.
|
||||
|
||||
`note_detect` tracks the on-pitch state of held sustains in a `_susActiveUntil` Map (key → `performance.now()` when the grace window ends). A typical grace window is 30 ms (one or two pitch frames), so a brief audio gap during a sustained note doesn't kill the glow.
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
1. **Provider returns truthy forever → gem stays lit forever.** Always return `null` when the effect should end. The renderer has no other way to know.
|
||||
2. **Provider tracks fade in renderer state instead.** Don't. The provider is consulted by *every* renderer and you'd end up with mismatched fades on each. Owns-its-own-decay is the design.
|
||||
3. **Renderer passes a per-constituent `t` instead of the chord's.** Silent failure — the lookup misses, all chord gems render dim. See the second pitfall above.
|
||||
4. **Calling `setNoteStateProvider` once at module load.** The highway might not exist yet, or a later splitscreen panel might create another highway. Re-register on every `song:ready` and every new panel mount.
|
||||
5. **Renderer caches the bundle.** Bundle arrays are live references but the object itself is constructed per frame for some fields. Don't hold `let myBundle = bundle` across frames. Caching the function ref `bundle.getNoteState` is fine — it's stable.
|
||||
6. **Provider does heavy work per call.** It runs O(visible notes × FPS) — typically 1000+ calls/sec during dense passages. A Map lookup is fine; a regex / JSON parse / sort is not.
|
||||
|
||||
## Where to look next
|
||||
|
||||
- [`realtime-scoring-pipeline.md`](realtime-scoring-pipeline.md) — full data flow from audio → verdict → provider.
|
||||
- [`visualization-feedback-guide.md`](visualization-feedback-guide.md) — practical "how do I add this to my viz" walkthrough with a minimal 2D Canvas example.
|
||||
- [`../plugins/note_detect/`](../plugins/note_detect/) — the canonical producer. Search `screen.js` for `setNoteStateProvider` and `noteStateFor`.
|
||||
- [`../plugins/highway_3d/screen.js`](../plugins/highway_3d/screen.js) — canonical custom-renderer consumer. Search for `getNoteState`.
|
||||
- [`../CLAUDE.md`](../CLAUDE.md) § "Note-state provider" — the short version of this doc embedded in the plugin-system overview.
|
||||
@@ -1,227 +0,0 @@
|
||||
# Real-time scoring pipeline
|
||||
|
||||
This doc traces how a single audio sample becomes a lit-up gem on the highway. It exists because the path crosses five layers — native engine, IPC bridge, `note_detect` plugin, highway core, renderer — and the contract between them is currently spread across `slopsmith/CLAUDE.md`, `slopsmith-desktop/`, and four different plugin source files.
|
||||
|
||||
**Audience**: plugin authors building a visualization that wants to react to detected hits, and maintainers debugging why a gem isn't lighting up.
|
||||
|
||||
**Companion docs**
|
||||
- [`note-state-provider.md`](note-state-provider.md) — the `setNoteStateProvider` / `bundle.getNoteState` API contract, in isolation.
|
||||
- [`visualization-feedback-guide.md`](visualization-feedback-guide.md) — a practical "how do I add hit feedback to my custom viz" walkthrough.
|
||||
- `slopsmith-desktop/docs/audio-engine-architecture.md` (companion repo) — internals of the JUCE engine + ML detector that produce the verdicts this pipeline consumes.
|
||||
|
||||
## Pipeline at a glance
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ slopsmith-desktop (native, JUCE + ONNX + Basic Pitch) │
|
||||
│ │
|
||||
│ audio device → ring buffer → ┌─ PitchDetector (YIN) │
|
||||
│ ├─ MlNoteDetector (Basic Pitch / ONNX) │
|
||||
│ ├─ ChordScorer (FFT + harmonic comb) │
|
||||
│ └─ NoteVerifier (background thread) │
|
||||
│ │ │
|
||||
└───────────────────────────────────────┼──────────────────────────────┘
|
||||
│ IPC (Electron preload)
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ note_detect plugin (slopsmith/plugins/note_detect/screen.js) │
|
||||
│ │
|
||||
│ drain audio:getNoteVerdicts(songTime, playing) │
|
||||
│ (or fall back to browser matchNotes + audio:scoreChord) │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ noteResults Map keyed by `${time}_${string}_${fret}` │
|
||||
│ │ │
|
||||
│ ├─► window.slopsmith.emit('note:hit' | 'note:miss', judgment) │
|
||||
│ │ │
|
||||
│ └─► highway.setNoteStateProvider(noteStateFor) │
|
||||
└───────────────────────────────────────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ highway core (slopsmith/static/highway.js) │
|
||||
│ │
|
||||
│ bundle.getNoteState(note, chartTime) → null │
|
||||
│ │ 'hit' | 'active' | 'miss' │
|
||||
│ │ { state, alpha, color } │
|
||||
└───────────────────────────────────────┬──────────────────────────────┘
|
||||
│ per visible note, per frame
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ Renderer (built-in 2D, bundled 3D, or your custom viz) │
|
||||
│ │
|
||||
│ draw(bundle) { │
|
||||
│ for (const n of bundle.notes) { │
|
||||
│ const st = bundle.getNoteState(n, n.t); │
|
||||
│ // light gem, sustain, sparkle … │
|
||||
│ } │
|
||||
│ } │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Two facts shape every other detail:
|
||||
|
||||
1. **Verdicts are produced once and consumed many times.** A renderer must not call `audio:scoreChord` itself. Scoring runs once in the engine (or in `note_detect`'s browser fallback) and the resulting per-note state is published through a single, idempotent provider. Renderers read the provider in their `draw()` loop and that's it.
|
||||
2. **The chord-time semantic is load-bearing.** Chord constituents are keyed in `noteResults` as `${chordTime}_${string}_${fret}`. A renderer that passes a per-constituent timestamp instead of the chord's `t` will look up nothing and silently render dim gems. See [`note-state-provider.md`](note-state-provider.md) for the rule.
|
||||
|
||||
## Layer 1 — Native engine
|
||||
|
||||
In normal desktop sessions, scoring runs in C++ on a JUCE audio thread + a background verifier thread, not in JavaScript. The architecture doc in the companion repo covers internals; the surface area visible to JS is:
|
||||
|
||||
| IPC method | Purpose | Cadence on the JS side |
|
||||
|---|---|---|
|
||||
| `audio:setChart(chart)` | Push the active arrangement once per song / arrangement switch. | Once per `playSong()`. |
|
||||
| `audio:getNoteVerdicts(songTime, playing)` | Drain finalized per-note verdicts AND push the renderer's playhead. | ~20 Hz (the `detectInterval` tick). |
|
||||
| `audio:getPitchDetection()` | Monophonic dominant pitch (YIN or ML, whichever is ready). | ~20 Hz. |
|
||||
| `audio:scoreChord(ctx)` | One-shot polyphonic score against the engine's input ring. | Per chord event during matching (fallback path). |
|
||||
| `audio:detectNotes()` | Full ML active-pitch set: `{notes: [{midi, confidence, onsetMs, onsetSeq}], sampleRate}`. | Optional; only the diagnostic / ML-aware code paths use it. |
|
||||
| `audio:isMlNoteDetection()` | `true` when Basic Pitch is loaded and ready. | Once at startup. |
|
||||
|
||||
All of these are feature-detected (`typeof audio.getNoteVerdicts === 'function'`) so the renderer keeps working against a downlevel addon.
|
||||
|
||||
Handlers live in [`audio-bridge.ts`](../../slopsmith-desktop/src/main/audio-bridge.ts) (in the `slopsmith-desktop` repo). Look for the `ipcMain.handle('audio:setChart', ...)`, `'audio:getNoteVerdicts'`, etc. blocks around lines 500–560.
|
||||
|
||||
## Layer 2 — `note_detect` plugin
|
||||
|
||||
`note_detect` is the only consumer of those IPC methods today. Everything downstream consumes `note_detect`'s output, not the engine's output directly. There are **two paths** through this layer; the engine path is preferred on desktop, the browser path is the fallback.
|
||||
|
||||
### Engine-verifier path (modern, desktop only)
|
||||
|
||||
On `playSong()`, `note_detect` calls `audio:setChart(arrangementChart)` and then polls `audio:getNoteVerdicts(songTime, playing)` on the detect tick. The verifier in C++ has already scored each chart note against the audio ring (using harmonic-comb + onset detection); each call returns the verdicts that have finalized since the last drain.
|
||||
|
||||
Each verdict has the shape:
|
||||
|
||||
```js
|
||||
{
|
||||
id: string, // chart-note id assigned at setChart time
|
||||
detected: boolean, // hit or miss
|
||||
detectedSongTime: number, // when the engine saw the onset
|
||||
centsError: number, // cents off the expected pitch
|
||||
snr: number // confidence-ish signal:noise ratio
|
||||
}
|
||||
```
|
||||
|
||||
`note_detect` translates each verdict into its internal `judgment` shape and writes it into `noteResults`, keyed `${chartNote.t}_${chartNote.s}_${chartNote.f}` for single notes, and `${chord.t}_${cn.s}_${cn.f}` for each chord constituent. Chord-level verdicts also write a chord-time-only entry `${chord.t}_chord` used for chord-frame tint.
|
||||
|
||||
### Browser fallback path (legacy / non-desktop)
|
||||
|
||||
If the desktop bridge isn't available (web build, downlevel addon, audio engine stopped), the same plugin runs the entire pipeline in the browser:
|
||||
|
||||
1. `getUserMedia` → `AudioContext` → `ChannelSplitterNode` → `ScriptProcessor` (per-channel mono).
|
||||
2. Per-frame YIN pitch detect + onset gate.
|
||||
3. `matchNotes()` — windowed match between detected events and chart notes (timing tolerance + pitch tolerance from settings).
|
||||
4. For chord events: `_ndScoreChord()` runs a renderer-side FFT + band-energy scorer on the accumulated buffer.
|
||||
|
||||
Either way, the result lands in `noteResults` with the same keys, and the downstream consumers don't care which path produced it.
|
||||
|
||||
### Publishing
|
||||
|
||||
Two channels:
|
||||
|
||||
```js
|
||||
// 1. Global event (for stats panels, journaling plugins, etc.)
|
||||
window.slopsmith.emit(judgment.hit ? 'note:hit' : 'note:miss', judgment);
|
||||
|
||||
// 2. Per-note state provider (for the highway + renderers)
|
||||
highway.setNoteStateProvider(noteStateFor);
|
||||
```
|
||||
|
||||
The provider function `noteStateFor(note, chartTime)` reads `noteResults` and returns one of:
|
||||
|
||||
| Return | Meaning |
|
||||
|---|---|
|
||||
| `null` (or anything falsy) | No state — renderer draws this note normally. |
|
||||
| `'hit'` | Note was struck cleanly. Fade owned by the provider. |
|
||||
| `'active'` | Sustained note currently still being held on-pitch. |
|
||||
| `'miss'` | Note expired without a clean detection. |
|
||||
| `{ state, alpha, color }` | Object form — same `state` field, plus explicit `alpha` (0..1) and optional `color` override. |
|
||||
|
||||
The full contract — including the keying rules and sustain semantics — is in [`note-state-provider.md`](note-state-provider.md).
|
||||
|
||||
## Layer 3 — Highway core
|
||||
|
||||
The highway exposes the provider to renderers via the per-frame `bundle` passed to `draw(bundle)`. The relevant lines in [`static/highway.js`](../static/highway.js):
|
||||
|
||||
- The provider slot itself: `let _noteStateProvider = null;` at line 153.
|
||||
- The normalizer that the bundle exposes: `function _noteState(note, chartTime) { … }` at line 290. This catches exceptions thrown by the provider, validates the `state` is one of `'hit'/'active'/'miss'`, clamps `alpha` to `[0, 1]`, drops zero-alpha returns, and returns `null` otherwise.
|
||||
- The bundle field: `getNoteState: _noteState` (line 514) — a **stable reference**, no per-frame allocation. Custom renderers can safely cache it.
|
||||
- The API method: `setNoteStateProvider(fn)` on the highway's exported object (line 2974). Last writer wins; passing `null` clears the provider.
|
||||
|
||||
The built-in 2D renderer consumes the provider in three places — `drawNote` (line 1445), `drawSustains` (line 1364), and the chord-frame path (line 1655) — so the default visuals work as soon as `note_detect` is enabled, without any per-plugin wiring.
|
||||
|
||||
## Layer 4 — Renderer
|
||||
|
||||
In your renderer's `draw(bundle)`, the typical pattern is:
|
||||
|
||||
```js
|
||||
draw(bundle) {
|
||||
for (const n of bundle.notes) {
|
||||
const state = bundle.getNoteState(n, n.t); // n.t for single notes
|
||||
if (state) {
|
||||
// state.state is 'hit' | 'active' | 'miss'
|
||||
// state.alpha is 0..1 (provider-owned decay)
|
||||
// state.color is optional string override
|
||||
drawLitGem(n, state);
|
||||
} else {
|
||||
drawDimGem(n);
|
||||
}
|
||||
}
|
||||
for (const ch of bundle.chords) {
|
||||
for (const cn of ch.notes) {
|
||||
const state = bundle.getNoteState(cn, ch.t); // ch.t, NOT cn.t
|
||||
// …
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Two patterns the bundled renderers use that you can copy:
|
||||
|
||||
- **2D highway** ([`static/highway.js`](../static/highway.js) `drawNote` at line 1445): `'hit'`/`'active'` → bright string color + additive halo + sparkle. `'miss'` → faint red wash. Sustain trail brightens with `'active'`.
|
||||
- **3D highway** ([`plugins/highway_3d/screen.js`](../plugins/highway_3d/screen.js) — search for `getNoteState`): outline mesh + body switch from dim material (`mStr[s]`) to bright (`mGlow[s]`); sustained notes keep the glow while `'active'`; a contained sparkle is queued on the 2D overlay layer.
|
||||
|
||||
For a from-scratch viz, [`visualization-feedback-guide.md`](visualization-feedback-guide.md) has a minimal Canvas2D example.
|
||||
|
||||
## Concrete trace: strumming a C major at chart time 12.34 s
|
||||
|
||||
Chart event: a `chord` at `t = 12.34` with three constituents — strings 1 (B), 2 (G), 3 (D) at frets `(1, 0, 2)`.
|
||||
|
||||
1. **t ≈ 12.30 s**: the player strums. The audio thread captures samples into `inputFrameRing`. The Basic Pitch ML detector publishes `B3`, `G3`, `D3` as active pitches; the `NoteVerifier` thread, which has had the chart since `playSong()`, opens scoring windows around `t = 12.34` for each chord-note (`12.34 ± timingTolerance`).
|
||||
2. **t ≈ 12.34 s**: Each open-window note's harmonic comb confirms the expected pitch present. The verifier closes the windows shortly after `12.34 + timingTolerance` and pushes three verdicts onto the drain queue: `{detected: true, detectedSongTime: 12.31, centsError: +4, snr: 5.3}` for each.
|
||||
3. **t ≈ 12.39 s**: `note_detect`'s detect tick calls `audio:getNoteVerdicts(12.39, true)`. It receives the three verdicts, builds three `judgment` objects, writes:
|
||||
- `noteResults["12.34_1_1"] = { hit: true, … }`
|
||||
- `noteResults["12.34_2_0"] = { hit: true, … }`
|
||||
- `noteResults["12.34_3_2"] = { hit: true, … }`
|
||||
- `noteResults["12.34_chord"] = { hit: true, hitStrings: 3, totalStrings: 3, … }`
|
||||
- Emits `window.slopsmith.emit('note:hit', judgment)` for the chord.
|
||||
4. **Next rAF**: the renderer's `draw(bundle)` iterates `bundle.chords`. For the chord at `t = 12.34` it calls `bundle.getNoteState(cn, 12.34)` for each constituent. Each call resolves `noteStateFor(cn, 12.34)`, which looks up `noteResults["12.34_${cn.s}_${cn.f}"]`, finds a recent `hit`, returns `{ state: 'hit', alpha: ~1, color: null }`.
|
||||
5. **2D renderer**: each gem in the chord renders with bright string color + halo + sparkle. The chord-frame tint goes green.
|
||||
6. **3D renderer (if active)**: outline + body swap to `mGlow[s]`; a sparkle entry is queued for each gem on the 2D overlay layer; the chord frame box tints green.
|
||||
7. **Over the next ~300 ms**: `noteStateFor` returns a decaying `alpha` (provider-owned fade). When `alpha` drops below the floor, it returns `null` and the gems revert to dim.
|
||||
|
||||
If any string had been missed (say string 3 silent because of an over-aggressive mute), step 2 would have closed that window with `detected: false`. `noteResults["12.34_3_2"]` would be a `miss`, the chord-frame verdict would still be `hit` (any-constituent-hit rule per [PR #316](https://github.com/byrongamatos/slopsmith/pull/316)), and the renderer would paint string 3 with the miss wash while the others lit up.
|
||||
|
||||
## Where to look next
|
||||
|
||||
If you want to…
|
||||
|
||||
- **Build a custom viz that reacts to hits** → [`visualization-feedback-guide.md`](visualization-feedback-guide.md).
|
||||
- **Understand the provider contract in full** (semantics of `alpha`, key formats, sustain rules) → [`note-state-provider.md`](note-state-provider.md).
|
||||
- **Dig into the engine internals** (NoteVerifier thread, ML detector loading, ring buffer) → `slopsmith-desktop/docs/audio-engine-architecture.md` in the companion repo.
|
||||
- **Tune detection quality** (false misses, latency, A/V offset) → [`note-detect-tuning.md`](note-detect-tuning.md).
|
||||
- **Read the canonical setRenderer contract** → [`../CLAUDE.md`](../CLAUDE.md) § "Visualization plugins".
|
||||
|
||||
## Key files
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `slopsmith-desktop/src/audio/AudioEngine.cpp:1447` | Audio device callback — entry point for every input sample. |
|
||||
| `slopsmith-desktop/src/audio/AudioEngine.h:196` | `inputFrameRing` declaration (lock-free SPSC, 8192 samples). |
|
||||
| `slopsmith-desktop/src/audio/NoteVerifier.cpp:152` | Background verifier thread loop. |
|
||||
| `slopsmith-desktop/src/audio/NoteVerifier.cpp:52` | `setChart()` — push the active arrangement. |
|
||||
| `slopsmith-desktop/src/main/audio-bridge.ts` | IPC handlers (look for `audio:setChart`, `audio:getNoteVerdicts`, `audio:scoreChord`). |
|
||||
| `slopsmith/plugins/note_detect/screen.js` | `noteResults` map, `noteStateFor`, `setNoteStateProvider` registration, verdict drain loop. |
|
||||
| `slopsmith/static/highway.js:153` | `_noteStateProvider` slot. |
|
||||
| `slopsmith/static/highway.js:290` | `_noteState()` normalizer (catches exceptions, clamps alpha, validates state). |
|
||||
| `slopsmith/static/highway.js:514` | `getNoteState: _noteState` bundle field. |
|
||||
| `slopsmith/static/highway.js:2974` | `setNoteStateProvider` on the public highway API. |
|
||||
@@ -1,203 +0,0 @@
|
||||
# Adding hit feedback to a visualization
|
||||
|
||||
A practical guide to making your custom visualization light up when notes are hit, dim when they're missed, and stay glowing while sustained notes are held correctly.
|
||||
|
||||
**Audience**: you're building a visualization plugin (per [`../CLAUDE.md`](../CLAUDE.md) § "Visualization plugins") and you want it to react to detected hits the same way the built-in 2D and 3D highways do.
|
||||
|
||||
**Prerequisites**
|
||||
- You can already get a `setRenderer`-style plugin loading. (If not, start with [`../CLAUDE.md`](../CLAUDE.md) — that contract is the entry point.)
|
||||
- You've skimmed [`realtime-scoring-pipeline.md`](realtime-scoring-pipeline.md). You don't need to understand the engine internals, just the shape of `bundle.getNoteState`.
|
||||
- `note_detect` is enabled in your test setup. Without it, the provider stays unregistered and `bundle.getNoteState` returns `null` for everything — your viz will work but you won't see feedback.
|
||||
|
||||
## The whole contract, in 5 lines
|
||||
|
||||
```js
|
||||
draw(bundle) {
|
||||
for (const n of bundle.notes) {
|
||||
const st = bundle.getNoteState(n, n.t); // for single notes, pass n.t
|
||||
// st is null | { state: 'hit' | 'active' | 'miss', alpha: 0..1, color: string|null }
|
||||
}
|
||||
for (const ch of bundle.chords) {
|
||||
for (const cn of ch.notes) {
|
||||
const st = bundle.getNoteState(cn, ch.t); // for chord constituents, pass CHORD time
|
||||
// …
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
That's it. The full semantics — when the provider fires, who owns fade timing, what the keys are — are in [`note-state-provider.md`](note-state-provider.md). This doc is about how to use them.
|
||||
|
||||
## Minimal 2D Canvas example
|
||||
|
||||
A complete `setRenderer`-style plugin that draws each chart note as a circle, lit up when hit, with a green halo for active sustains and a red wash for misses. ~80 lines, no dependencies. Drop it in `plugins/my_viz/screen.js` along with a matching `plugin.json` (`"type": "visualization"`).
|
||||
|
||||
```js
|
||||
(function () {
|
||||
'use strict';
|
||||
|
||||
window.slopsmithViz_my_viz = function () {
|
||||
let ctx = null;
|
||||
let W = 0, H = 0;
|
||||
|
||||
return {
|
||||
contextType: '2d',
|
||||
|
||||
init(canvas, bundle) {
|
||||
ctx = canvas.getContext('2d');
|
||||
W = canvas.width;
|
||||
H = canvas.height;
|
||||
},
|
||||
|
||||
resize(w, h) {
|
||||
W = w; H = h;
|
||||
},
|
||||
|
||||
draw(bundle) {
|
||||
if (!ctx) return;
|
||||
ctx.clearRect(0, 0, W, H);
|
||||
|
||||
const now = bundle.currentTime;
|
||||
const stringCount = bundle.stringCount || 6;
|
||||
|
||||
// Project a chart time onto x. Notes ahead of `now` slide
|
||||
// in from the right; notes behind `now` slide off left.
|
||||
const xFor = (t) => W * 0.5 + (t - now) * 100;
|
||||
const yFor = (s) => H * (0.2 + 0.6 * (s / (stringCount - 1)));
|
||||
|
||||
// Single notes.
|
||||
for (const n of bundle.notes) {
|
||||
const x = xFor(n.t);
|
||||
if (x < -50 || x > W + 50) continue; // off-screen cull
|
||||
const y = yFor(n.s);
|
||||
const st = bundle.getNoteState(n, n.t);
|
||||
drawGem(ctx, x, y, n, st);
|
||||
}
|
||||
|
||||
// Chord constituents — pass CHORD time, not constituent time.
|
||||
for (const ch of bundle.chords) {
|
||||
const x = xFor(ch.t);
|
||||
if (x < -50 || x > W + 50) continue;
|
||||
for (const cn of ch.notes) {
|
||||
const y = yFor(cn.s);
|
||||
const st = bundle.getNoteState(cn, ch.t);
|
||||
drawGem(ctx, x, y, cn, st);
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
destroy() {
|
||||
ctx = null;
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
function drawGem(ctx, x, y, note, st) {
|
||||
const r = 12;
|
||||
ctx.beginPath();
|
||||
ctx.arc(x, y, r, 0, Math.PI * 2);
|
||||
|
||||
if (!st) {
|
||||
// Dim — no judgment yet (or expired).
|
||||
ctx.fillStyle = '#445';
|
||||
ctx.fill();
|
||||
return;
|
||||
}
|
||||
|
||||
// Lit. `alpha` is provider-owned fade.
|
||||
const a = st.alpha;
|
||||
if (st.state === 'hit' || st.state === 'active') {
|
||||
ctx.fillStyle = st.color || `rgba(120, 220, 140, ${a})`;
|
||||
ctx.fill();
|
||||
// Halo grows with brightness — purely cosmetic.
|
||||
ctx.beginPath();
|
||||
ctx.arc(x, y, r + 6 * a, 0, Math.PI * 2);
|
||||
ctx.strokeStyle = `rgba(120, 220, 140, ${a * 0.5})`;
|
||||
ctx.lineWidth = 2;
|
||||
ctx.stroke();
|
||||
} else if (st.state === 'miss') {
|
||||
ctx.fillStyle = `rgba(220, 90, 90, ${a})`;
|
||||
ctx.fill();
|
||||
}
|
||||
}
|
||||
})();
|
||||
```
|
||||
|
||||
Things this example deliberately skips:
|
||||
|
||||
- **Sustain trails.** Add a `ctx.fillRect` from `xFor(n.t)` to `xFor(n.t + n.sus)` and switch its colour based on the `'active'` state.
|
||||
- **Inverted / lefty.** Check `bundle.inverted` and `bundle.lefty` and mirror your y-axis / x-axis transforms.
|
||||
- **Difficulty filter awareness.** `bundle.notes` is already filtered — you get this for free.
|
||||
- **Capo / tuning offsets.** Not needed for visualization; the gems just show the chart positions.
|
||||
|
||||
## Case study — how the 3D highway does it
|
||||
|
||||
[`../plugins/highway_3d/screen.js`](../plugins/highway_3d/screen.js) is the canonical custom renderer. It uses the same `bundle.getNoteState(note, chartTime)` call, but applied to Three.js materials.
|
||||
|
||||
The cliff notes (search the file for `getNoteState` to find each call site):
|
||||
|
||||
- For each visible note, the 3D highway calls `bundle.getNoteState(n, n.t)` (or `ch.t` for chord constituents).
|
||||
- A non-null return switches the gem's outline mesh from `mStr[s]` (dim, dull string colour) to `mGlow[s]` (bright emissive). The body mesh is swapped the same way.
|
||||
- Sustain trails use the bright material while `'active'`, the dim one otherwise.
|
||||
- A separate sparkle layer (`drawNotedetectSizzle()` on the 2D overlay canvas) queues a few crackling arcs and dots projected through the camera, fading with `st.alpha`. The 3D highway runs this on its 2D *overlay* canvas, not in the 3D scene, so the sparkles don't blow out the bloom or get hidden behind world geometry.
|
||||
- Chord-frame tint: when the chord-level entry (`${chord.t}_chord`) is `hit`, the chord-frame box tints green; on miss, it tints red. Per-constituent verdicts still drive the individual gem visuals so a partial-hit chord shows the frame green with individual constituents lit or dim.
|
||||
|
||||
Two implementation details worth copying:
|
||||
|
||||
1. The 3D highway caches `bundle.getNoteState` once per frame — `const getNs = bundle.getNoteState;` — and uses the cached ref for the inner loops. The function reference is stable across frames, so this is safe and saves a property read per gem.
|
||||
2. It null-guards every consumption (`if (st && st.state === 'hit')`) rather than relying on `st.state` to be defined. `null` is the common case — a single visible chord with 30 unstruck notes ahead of the playhead produces 30 null returns and 0 lit returns.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "I see no lit gems at all"
|
||||
|
||||
In order of likelihood:
|
||||
|
||||
1. **`note_detect` isn't running.** Open DevTools console and run:
|
||||
```js
|
||||
window.highway && window.highway.getNoteStateProvider()
|
||||
```
|
||||
If it returns `null`, no scorer is registered. Click the "Detect" button in the player controls (or enable it via your test fixture).
|
||||
2. **Your renderer isn't calling `bundle.getNoteState`.** Add a `console.log(st)` inside the loop. If you never see anything other than `null`, the provider is registered but you might be on a song with no scoring events yet (try strumming and watch the console).
|
||||
3. **You're passing the wrong `chartTime`.** Chord constituents need `chord.t`, not `cn.t`. Easy to miss if your inner loop reuses a variable named `t`.
|
||||
4. **Your renderer's `draw()` isn't firing.** Make sure the highway has had a `ready` message — `bundle.isReady` should be `true`. The factory gates `draw()` calls behind the ready flag.
|
||||
|
||||
### "Single notes light up but chord notes don't"
|
||||
|
||||
You're almost certainly passing `cn.t` instead of `ch.t` to `getNoteState`. The lookup map is keyed by the chord's time. Most chord constituents have the same `t` as the chord, but the contract still requires the chord time explicitly — and there are chart formats where constituent `t` values drift slightly.
|
||||
|
||||
### "Notes light up but never fade — they stay green forever"
|
||||
|
||||
The provider — not your renderer — owns fade timing. If you've replaced `note_detect` with your own provider and it returns a truthy state forever, gems stay lit forever. Return `null` when the effect should end. See [`note-state-provider.md`](note-state-provider.md) § "`alpha`" for how to encode a fade.
|
||||
|
||||
### "Sustains don't glow while held"
|
||||
|
||||
The provider needs to return `'active'` (not `'hit'`) for as long as the held pitch is detected. Check whether the producer in use (`note_detect` on desktop) is tracking sustain state — it does, via `_susActiveUntil`, but only when monophonic pitch detection is enabled and on-pitch within the configured cents tolerance. If your audio is noisy or the cents tolerance is tight, sustains may decay to `null` between frames.
|
||||
|
||||
### "My splitscreen panels all light up the same notes at the same time"
|
||||
|
||||
Two things might be going on:
|
||||
|
||||
1. **Two panels are scoring the same audio input.** Each `note_detect` instance currently shares the engine's single-source detection. See [issue #375](https://github.com/byrongamatos/slopsmith/issues/375) for the RFC on per-source detection. Until that lands, the fact that both panels light identically is expected.
|
||||
2. **Each panel's highway has its own provider slot.** If you're building a plugin that replaces the provider, register it on each panel's highway instance, not on a global. Splitscreen creates one highway per panel — `setNoteStateProvider` is per-highway.
|
||||
|
||||
### "My renderer prints `TypeError: bundle.getNoteState is not a function`"
|
||||
|
||||
The bundle field has been part of the contract since slopsmith#254. If `bundle.getNoteState` is missing, the host slopsmith you're running against predates that issue — you're on an old build. Check the version with `cat VERSION` in the slopsmith repo; anything ≥ 0.2 should have it. The field is set to a stable reference, never undefined.
|
||||
|
||||
### "I want to draw something on hit that isn't a gem (a particle burst, a sound, anything)"
|
||||
|
||||
Two options:
|
||||
|
||||
1. **Subscribe to events from inside your renderer.** `window.slopsmith.on('note:hit', judgment => { … })`. The judgment object has the chord/note info, timing/pitch error, etc. Fires once per detection, not per frame. Good for one-shots (sounds, screen shakes, achievement triggers).
|
||||
2. **Watch for state transitions in your draw loop.** Cache the last-seen state per note key in your renderer; when this frame's state changes from `null` to truthy, you know an event just landed. Good for cumulative effects (particle bursts that need to know their start time).
|
||||
|
||||
The provider doesn't itself give you an event stream — it gives you a sample-per-frame view of current state. Pick the right tool for your effect.
|
||||
|
||||
## Where to look next
|
||||
|
||||
- [`realtime-scoring-pipeline.md`](realtime-scoring-pipeline.md) — full data flow, top to bottom.
|
||||
- [`note-state-provider.md`](note-state-provider.md) — the producer contract (write your own scorer).
|
||||
- [`../CLAUDE.md`](../CLAUDE.md) — the canonical `setRenderer` / overlay contract.
|
||||
- [`../plugins/highway_3d/screen.js`](../plugins/highway_3d/screen.js) — case-study renderer.
|
||||
- [`note-detect-tuning.md`](note-detect-tuning.md) — what to do when detection itself is flaky (your viz works fine but the underlying hits / misses look wrong).
|
||||
@@ -6,11 +6,6 @@ The whole renderer is **one file** — `screen.js`, wrapped in an IIFE, register
|
||||
|
||||
**Styling (slopsmith `styles` capability).** This plugin owns its Tailwind CSS: it ships `assets/plugin.css` and declares `"styles": "assets/plugin.css"` in `plugin.json`, so core's prebuilt `static/tailwind.min.css` no longer scans it (it's excluded from core's content globs). The frontend injects `assets/plugin.css` as a `<link>` when the renderer activates. This is the one maintainer-time build step: after you add/change a Tailwind class in `screen.js` or `settings.html`, run `bash build-tailwind.sh` (pinned `tailwindcss@3.4.19`, `corePlugins.preflight=false` — utilities only) and **bump the `version` in `plugin.json`** so the injected `<link>`'s `?v=` cache-buster fetches the fresh file. The generated `assets/plugin.css` is committed; end users never build. See [docs/plugin-styles.md](../../docs/plugin-styles.md).
|
||||
|
||||
**For the contracts this plugin implements** (setRenderer, the note-state provider, the real-time scoring pipeline), see the repo-root docs:
|
||||
- [`../../docs/realtime-scoring-pipeline.md`](../../docs/realtime-scoring-pipeline.md) — end-to-end audio → lit gem trace. This plugin is one of the renderers in the final step.
|
||||
- [`../../docs/note-state-provider.md`](../../docs/note-state-provider.md) — the `bundle.getNoteState(note, chartTime)` contract this plugin consumes for hit/active/miss feedback.
|
||||
- [`../../docs/visualization-feedback-guide.md`](../../docs/visualization-feedback-guide.md) — practical "how do I add hit feedback to a viz" walkthrough, using this plugin as a case study.
|
||||
|
||||
> **Navigation note:** This guide references functions by name and uses the existing banner comments (`/* ── Scene initialisation ─ */`, etc.) as section anchors. Line numbers are deliberately avoided so this stays correct as the file evolves. Use `Grep` for the function name or banner text to jump to a section.
|
||||
|
||||
## File structure at a glance
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# Implementation Plan: Note-Detection Capability Domain
|
||||
|
||||
**Status**: Draft stub. This plan records scope, dependencies, and the migration-gate posture so the slice can be scheduled. The full plan/data-model/contracts/tasks are generated when the slice starts.
|
||||
|
||||
## Scope
|
||||
|
||||
Introduce a `note-detection` capability domain: a per-binding, chart-decoupled, multi-consumer control plane over the existing detection DSP. It exposes two primitives — a monophonic pitch estimate and a polyphonic note-set verification verdict — each scored against a requester-supplied tuning context, and consolidates today's two fragmented surfaces (`slopsmithMinigames.scoring.createContinuous` and `window.noteDetect`) behind one contract. It does **not** implement DSP and does **not** own consumer judgment.
|
||||
|
||||
See `spec.md` for requirements, entities, and success criteria.
|
||||
|
||||
## Dependencies / ordering
|
||||
|
||||
- **Depends on** Spec 006 (audio-input domain — source identity, open-session state; consumed, not redefined) and Spec 007 (audio-monitoring domain). 007 is currently paused (PR #667); this slice should not start until 006/007 are settled enough to consume.
|
||||
- **Builds on** the capability-pipeline runtime (Spec 002, PR #245) and follows the migration standard (Spec 003).
|
||||
- **Interim bridge already shipped/in-flight**: notedetect `setVerifyTarget(notes, ctx)` (plugin PR #62). Its per-call tuning context is the forward-compatible seed of this domain's per-binding context; the SlopScale consumer adapter (fork PR) and Chord Sprint are the first non-chart requesters.
|
||||
|
||||
## Migration gate (per Spec 003)
|
||||
|
||||
This slice must pass the central + per-domain migration checklist:
|
||||
- Per-slice legacy inventory: the chart-coupled `note_detect` scoring/verify path, Step Mode verify consumption, minigames YIN scoring, and the `setVerifyTarget` bridge.
|
||||
- Staged deprecation gates + compatibility-bridge accounting (record legacy handoffs; native wins on overlap).
|
||||
- Diagnostics/Inspector expectations: bindings, provider attribution, per-binding context summary, outcomes — redaction-safe, no raw audio.
|
||||
- Removal gate: legacy detection handoffs removed only after consumers migrate, migration notes are published, and external usage review completes.
|
||||
|
||||
## Providers
|
||||
|
||||
- Desktop: JUCE engine verifier (harmonic-comb `scoreChord`, bass temporal-persistence floor) + monophonic pitch.
|
||||
- Web/dev: JS harmonic-comb / YIN fallback.
|
||||
Both sit behind one provider abstraction so DSP improvements land once and reach all consumers.
|
||||
|
||||
## Explicitly out of scope
|
||||
|
||||
Detection DSP/model accuracy, consumer judgment/scoring UX, audio-input source ownership (006), monitoring lifecycle (007), recording, playback transport (008), plugin installation, and tunings outside the provider's current tables (e.g. 6-string bass).
|
||||
@@ -0,0 +1,129 @@
|
||||
# Feature Specification: Note-Detection Capability Domain
|
||||
|
||||
**Status**: Draft stub — scheduling placeholder for the next capability-domain slice after audio-input (006) and audio-monitoring (007). Authored 2026-06-06 from a concrete non-chart consumer requirement (SlopScale, Chord Sprint). Full plan / data-model / contracts / tasks to be generated when the slice is scheduled, passing the Spec 003 migration gate.
|
||||
|
||||
## Why now
|
||||
|
||||
The capability-domain roadmap has been laying foundations *for* this slice from the start: Spec 002 names note detection on the roadmap; Spec 003 lists it among the domains that must pass the migration gate; Spec 004 defines `audio-input` named source identity explicitly so "a later note-detection domain needs per-source binding … without inheriting a single global detector assumption"; Specs 006/007 already carry `{ requesterId: 'note_detect', purpose: 'note-detection' }` requesters. This spec turns that anticipated slice into a concrete one, driven by a real consumer that the current surfaces cannot serve.
|
||||
|
||||
**Concrete trigger.** Detection capability is today fragmented across two surfaces and coupled to the host chart:
|
||||
- `slopsmithMinigames.scoring.createContinuous` — monophonic YIN, reachable from contained playback but weak (no chords, 70 Hz floor, distortion-unprobed).
|
||||
- `window.noteDetect` verify/scoring — the strong harmonic-comb verifier, but its tuning/arrangement state is mutated by the host's loaded song (`song:loaded`), so it is a **single global detector** that two consumers with different tuning needs fight over.
|
||||
|
||||
A contained-playback consumer (SlopScale runs its own transport and computes targets from the *player's real instrument*, not the chart's nominal tuning; Chord Sprint similarly) has no clean way to use the strong verifier against its own tuning. The interim bridge (notedetect `setVerifyTarget(notes, ctx)`, PR #62 on the plugin repo) proves the requirement and is forward-compatible with this domain's per-binding context, but it still relies on a single shared detector instance. This domain is the long-term home.
|
||||
|
||||
## Clarifications
|
||||
|
||||
### Session 2026-06-06
|
||||
- Q: Does this domain perform detection DSP itself? → A: No. It is a capability/control plane over existing detection providers (the desktop JUCE engine verifier and the JS harmonic-comb / YIN fallback). The DSP stays where it is; the domain gives it a per-binding, chart-decoupled, multi-consumer contract.
|
||||
- Q: Does this domain own scoring/judgment (hit windows, gems, tiers)? → A: No. Consumers own judgment semantics. The domain exposes detection PRIMITIVES only: a monophonic pitch estimate, and a "is this (string,fret) note-set ringing now?" verification verdict against caller-supplied tuning. (Doctrine: host owns detection DSP; consumers own judgment.)
|
||||
- Q: Is detection bound to the host highway's loaded song? → A: No. That single-global-detector coupling is the problem this slice removes. Each requester binds its own tuning context.
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
|
||||
### User Story 1 - Verify against the player's own tuning from contained playback (Priority: P1)
|
||||
|
||||
A contained-playback consumer (SlopScale) that runs its own transport and has no host song loaded asks "is the expected note/chord ringing now?" against the *player's* instrument tuning, and receives polyphonic, distortion-robust verdicts — without reading plugin-private globals and without the result being perturbed by whatever song the host highway has open.
|
||||
|
||||
**Why P1**: This is the requirement no current surface satisfies; it is the reason the slice exists.
|
||||
|
||||
**Acceptance**:
|
||||
- **Given** no host song is loaded, **When** a requester opens a detection binding with its own arrangement + tuning and registers a target note set, **Then** it receives verification verdicts scored against that tuning.
|
||||
- **Given** the host loads or switches a song underneath, **When** the requester's binding is active, **Then** its verdicts are unaffected (no shared-state perturbation).
|
||||
|
||||
### User Story 2 - Two consumers detect concurrently with different tunings (Priority: P2)
|
||||
|
||||
The highway (chart tuning) and a minigame (player tuning) request detection at the same time, each against its own context, without a single global detector's mutable arrangement/tuning state being clobbered by the other.
|
||||
|
||||
**Why P2**: Spec 004 deliberately avoided the single-global-detector assumption for exactly this; the domain realizes it.
|
||||
|
||||
**Acceptance**:
|
||||
- **Given** two active detection bindings with different arrangement/tuning, **When** both score concurrently, **Then** neither alters the other's tuning context or verdicts.
|
||||
|
||||
### User Story 3 - One detection capability, two primitives (Priority: P3)
|
||||
|
||||
A consumer that needs a live monophonic pitch (a tuner, a pitch strip) and a consumer that needs polyphonic note-set verification (a chord drill) use the **same** capability domain — not two unrelated surfaces (`createContinuous` vs `noteDetect`) — so DSP improvements (the bass temporal-persistence floor, distortion handling, future models) land once and reach every consumer.
|
||||
|
||||
**Acceptance**:
|
||||
- **Given** a single capability contract, **When** a consumer requests a monophonic pitch primitive or a polyphonic verify primitive, **Then** both are served by one provider over one input binding.
|
||||
- **Given** a DSP improvement lands in the provider, **When** any consumer requests detection, **Then** it benefits without consumer changes.
|
||||
|
||||
### User Story 4 - Migrate detection consumers and providers safely (Priority: P4)
|
||||
|
||||
The existing chart-coupled `note_detect` path, the minigames YIN scoring, Step Mode's verify consumption, and the bridge `setVerifyTarget(notes, ctx)` all migrate onto the domain behind the Spec 003 migration gate, with compatibility bridges and a removal gate, leaving each app area cleaner.
|
||||
|
||||
**Acceptance**:
|
||||
- **Given** the domain exists, **When** a legacy detection handoff occurs during migration, **Then** it is mapped into domain diagnostics and recorded as a compatibility bridge hit.
|
||||
- **Given** a new detection consumer is added after this slice, **When** it needs detection, **Then** it uses the domain rather than a new legacy-only handoff.
|
||||
|
||||
### Edge Cases
|
||||
|
||||
- No microphone / insecure context / no detection provider → detection bindings report unavailable; consumers degrade (scoring disables, never blocks).
|
||||
- A requester's declared tuning references strings the provider's tuning tables cannot represent (e.g. 6-string bass) → bounded `incompatible` outcome, not a silent NaN verdict.
|
||||
- Host song-switch while a player-tuning binding is active → the binding's context is unchanged.
|
||||
- Capo / drop tunings / re-tunings → the binding carries the real open-string pitches; no double transposition.
|
||||
- Polyphony the provider cannot resolve (heavy distortion, sub-floor strings) → verdict reports it honestly rather than guessing.
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
|
||||
### Functional Requirements
|
||||
|
||||
- **FR-001**: System MUST provide an authoritative note-detection control plane for opening a detection binding, registering/clearing a target, requesting verification verdicts, requesting a monophonic pitch estimate, and reporting provider/availability state.
|
||||
- **FR-002**: System MUST let each requester supply its own tuning context (arrangement, per-string tuning as absolute open MIDI or standard-tuning offsets, capo, string count) per binding, and MUST score that binding only against that context.
|
||||
- **FR-003**: System MUST NOT bind detection to the host highway's loaded song or any single global arrangement/tuning state; concurrent bindings with different contexts MUST NOT perturb one another.
|
||||
- **FR-004**: System MUST consume audio-input (Spec 006) source identity and open-session state for its capture rather than redefining input or assuming one global detector.
|
||||
- **FR-005**: System MUST expose detection as PRIMITIVES — a monophonic pitch estimate and a polyphonic note-set verification verdict — and MUST NOT perform consumer-side judgment (hit windows, streaks, gems, accuracy, tiers).
|
||||
- **FR-006**: System MUST provide a timing-free verification mode (score a registered target every frame independent of any playhead) so a frozen-playhead or self-transported consumer can ask "is this note-set ringing now?".
|
||||
- **FR-007**: System MUST surface per-binding verdict detail sufficient for consumer judgment (at minimum: overall hit, and per-string/per-note ring state for a multi-note target) without exposing raw audio buffers or sample data.
|
||||
- **FR-008**: System MUST report distinct outcomes for unavailable, denied, degraded, failed, no-provider, unsupported-context, and incompatible-version, rather than silently producing a verdict.
|
||||
- **FR-009**: System MUST reject or degrade tuning contexts the provider cannot represent (unsupported arrangement/string-count) with an `incompatible` outcome and MUST NOT emit NaN/garbage verdicts.
|
||||
- **FR-010**: System MUST route every detection request through one provider abstraction so DSP improvements reach all consumers at once (single source of truth), with the desktop engine verifier and a JS fallback as interchangeable providers.
|
||||
- **FR-011**: System MUST preserve the existing chart-coupled detection path during the compatibility period by mapping it onto the domain, recording compatibility bridge hits.
|
||||
- **FR-012**: System MUST document the migration path for detection providers and requesters (the chart `note_detect` consumer, Step Mode verify, minigames YIN scoring, and the `setVerifyTarget` bridge), including a removal gate, per Spec 003.
|
||||
- **FR-013**: System MUST emit observable events when bindings open/close, targets change, verdicts are produced, and availability changes.
|
||||
- **FR-014**: System MUST include active bindings, provider attribution, per-binding context summary, availability, and recent outcomes in diagnostics, redaction-safe.
|
||||
- **FR-015**: System MUST NOT expose raw audio buffers, sample/waveform data, or live capture handles through detection state, verdicts, diagnostics, or capability payloads.
|
||||
- **FR-016**: System MUST leave audio-input source ownership, monitoring lifecycle, recording, playback transport, and plugin installation outside this feature except as state it consumes.
|
||||
- **FR-017**: System MUST avoid creating new legacy-only detection integration points once the native domain exists.
|
||||
|
||||
### Key Entities
|
||||
|
||||
- **Detection Binding**: A requester-owned, context-scoped detection session over a selected audio-input source — carries the requester's tuning context and target, independent of any host song. Multiple bindings coexist.
|
||||
- **Tuning Context**: Arrangement + per-string tuning (absolute open MIDI or standard offsets) + capo + string count, supplied by the requester; the only tuning a binding's verdicts are scored against.
|
||||
- **Verify Target**: A registered note set (string/fret + technique flags) the binding scores against live audio every frame, independent of any playhead.
|
||||
- **Verification Verdict**: A bounded result — overall hit, per-note/per-string ring state, score, hit/total counts — with no raw audio.
|
||||
- **Pitch Estimate**: A monophonic frequency/MIDI + confidence primitive (the tuner/pitch-strip use case), the consolidation target for `createContinuous`.
|
||||
- **Detection Provider**: The participant performing DSP — the desktop JUCE engine verifier, or the JS harmonic-comb / YIN fallback — interchangeable behind the domain contract.
|
||||
- **Detection Requester**: A consumer (note_detect chart path, Step Mode, SlopScale, Chord Sprint, a tuner) that needs detection primitives but owns its own judgment.
|
||||
- **Compatibility Bridge Hit**: A record that a legacy chart-coupled or minigames-YIN detection handoff was used during migration.
|
||||
- **Detection Outcome**: A bounded diagnostic record (provider, binding, requester, status, outcome, safe reason) with no live handles or sample data.
|
||||
|
||||
## Success Criteria *(mandatory)*
|
||||
|
||||
### Measurable Outcomes
|
||||
|
||||
- **SC-001**: A contained-playback requester with no host song loaded receives verification verdicts scored against its own tuning in 100% of focused scenarios.
|
||||
- **SC-002**: A host song-switch while a player-tuning binding is active changes that binding's verdicts in 0% of focused scenarios.
|
||||
- **SC-003**: Two concurrent bindings with different arrangement/tuning never alter each other's context or verdicts in 100% of focused scenarios.
|
||||
- **SC-004**: A polyphonic (chord) target produces a real per-note/overall verdict — not an all-or-nothing exemption — in 100% of focused chord scenarios.
|
||||
- **SC-005**: A DSP improvement landed in the provider reaches every domain consumer with zero consumer code changes in representative cases.
|
||||
- **SC-006**: Unsupported tuning contexts produce an `incompatible` outcome and zero NaN/garbage verdicts in 100% of focused scenarios.
|
||||
- **SC-007**: 100% of detection verdicts, state snapshots, and diagnostics contain zero raw audio buffers, sample/waveform data, or live capture handles.
|
||||
- **SC-008**: New detection consumers added after this slice use the domain rather than a new legacy-only handoff in 100% of reviewed cases.
|
||||
- **SC-009**: A maintainer can identify provider, binding context, availability, and outcome for a representative detection failure in under 5 minutes from diagnostics/inspector.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Audio-input (006) and audio-monitoring (007) slices are available as foundation; this slice consumes their source identity and monitoring facts rather than redefining them.
|
||||
- The detection DSP (desktop JUCE engine verifier; JS harmonic-comb / YIN fallback) already exists and is correct; this slice gives it a per-binding, chart-decoupled, multi-consumer contract — it does not reimplement DSP.
|
||||
- Consumers retain ownership of judgment semantics (hit windows, streaks, gems, accuracy, tiers); the domain provides primitives only.
|
||||
- The notedetect `setVerifyTarget(notes, ctx)` bridge (plugin PR #62) is the interim, forward-compatible step; its per-call context maps onto this domain's per-binding context.
|
||||
- Detection is sensitive: raw audio must never cross the capability boundary.
|
||||
- Existing chart-coupled and minigames-YIN detection paths may coexist during migration behind the Spec 003 gate.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Detection DSP/model implementation or accuracy improvements (owned by the provider plugins).
|
||||
- Consumer judgment/scoring UX (gems, tiers, accuracy) — owned by each requester.
|
||||
- Audio-input source ownership/selection (006), monitoring lifecycle (007), recording, playback transport (008), and plugin installation.
|
||||
- 6-string bass and other tunings outside the provider's current tuning tables (tracked separately).
|
||||
Reference in New Issue
Block a user