mirror of
https://github.com/got-feedBack/feedBack.git
synced 2026-09-11 18:34:31 +00:00
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Kris AndersonandBret Mogilefsky
|
5db2c1a2d4 | ||
|
Kris AndersonandBret Mogilefsky
|
1246c290f1 |
@@ -199,6 +199,11 @@ usually an unrelated plugin's per-frame DOM work.
|
||||
|
||||
Slopsmith supports two ways for a plugin to participate in the main player's visuals. They coexist; the setRenderer contract is the default for any viz that draws a highway-shaped surface, and overlays handle layered decorations on top.
|
||||
|
||||
**Deeper references** (in [`docs/`](docs/)):
|
||||
- [`realtime-scoring-pipeline.md`](docs/realtime-scoring-pipeline.md) — end-to-end trace from audio sample → lit gem, across native engine, IPC, `note_detect`, highway, renderer.
|
||||
- [`note-state-provider.md`](docs/note-state-provider.md) — full contract for `setNoteStateProvider` / `bundle.getNoteState` (key formats, sustain semantics, alpha ownership).
|
||||
- [`visualization-feedback-guide.md`](docs/visualization-feedback-guide.md) — practical walkthrough with a minimal 2D Canvas example + troubleshooting decision tree.
|
||||
|
||||
**Pick the right shape:**
|
||||
- Replacing the whole highway drawing on the existing highway canvas (your renderer owns its rendering context / resources; `createHighway()` still owns the canvas element and the rAF loop)? → **setRenderer** (section 1). Enters the viz picker. Works in both the main player and per-panel under splitscreen.
|
||||
- Adding a layer on top of whichever viz is active? → **Overlay** (section 2). Navbar toggle, not in the picker.
|
||||
@@ -362,6 +367,8 @@ A previous standalone-pane contract (`window.createMyVisualization({ container }
|
||||
|
||||
#### 3. Note-state provider — for scorers that want renderers to "light up" notes (slopsmith#254)
|
||||
|
||||
> **Full reference**: [`docs/note-state-provider.md`](docs/note-state-provider.md) covers the contract in depth (key formats, sustain `'active'` vs `'hit'` semantics, alpha-ownership, common pitfalls). The summary below is the API at a glance.
|
||||
|
||||
A scoring plugin (note_detect) can publish a per-note judgment so whichever renderer is active draws the **gem itself** lit on a correct hit, and keeps a sustain trail glowing while it's still being played correctly — instead of a separate overlay ring floating near the note.
|
||||
|
||||
```js
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
# Note-state provider API
|
||||
|
||||
The contract that lets a scorer (today: `note_detect`) tell every renderer (today: built-in 2D, bundled 3D, anything you build) when a chart note has been **hit / actively held / missed**.
|
||||
|
||||
Introduced in [slopsmith#254](https://github.com/byrongamatos/slopsmith/issues/254). See [`realtime-scoring-pipeline.md`](realtime-scoring-pipeline.md) for how data flows into the provider; this doc focuses on the API itself.
|
||||
|
||||
**Audience**: plugin authors who want to either (a) **register** as the scorer (replacing `note_detect`), or (b) **read** per-note state from a custom visualization.
|
||||
|
||||
## The API in one minute
|
||||
|
||||
Two endpoints on the `highway` object, one bundle field exposed to renderers:
|
||||
|
||||
```js
|
||||
// Producer side (a scorer plugin)
|
||||
highway.setNoteStateProvider((note, chartTime) => {
|
||||
// return null / 'hit' / 'active' / 'miss' / { state, alpha, color }
|
||||
});
|
||||
highway.getNoteStateProvider(); // current provider, or null
|
||||
highway.setNoteStateProvider(null); // clear
|
||||
|
||||
// Consumer side (inside your renderer's draw)
|
||||
const st = bundle.getNoteState(note, chartTime); // same return shape
|
||||
```
|
||||
|
||||
**One provider at a time** — last writer wins. There's no event-emitter / multi-subscriber pattern; the contract is deliberately singular so the renderer's per-frame call is allocation-free and unambiguous. If two scorer plugins coexist, the second one's `setNoteStateProvider` call replaces the first.
|
||||
|
||||
The provider is called **per visible note, per frame, from the renderer's `draw()` loop**. Keep it cheap — a typical highway can have 30+ visible gems per frame across 60 fps. The bundled `note_detect` provider does one Map lookup + a few arithmetic operations.
|
||||
|
||||
## Return values
|
||||
|
||||
| Return | Meaning | When to use |
|
||||
|---|---|---|
|
||||
| `null` / `undefined` / `false` / `0` / `""` | No state for this note this frame. Renderer draws it normally. | The vast majority of returns. Default for unstruck notes, expired effects, notes the scorer doesn't know about. |
|
||||
| `'hit'` | Note was struck cleanly. | Right after a successful detection, while the brief post-strike glow is visible. |
|
||||
| `'active'` | A sustained note is *currently being held on-pitch*. | While a sustain trail should glow. Re-emit every frame the hold is still valid; stop returning state when the sustain ends. |
|
||||
| `'miss'` | Note expired without a clean detection. | After the scoring window closed and no hit was registered. |
|
||||
| `{ state, alpha, color }` | Full object form. Same `state`. Adds `alpha` (0..1 brightness multiplier) and optional `color` override. | Whenever you want a custom fade or to tint with something other than the renderer's default string colour. |
|
||||
|
||||
### `alpha`
|
||||
|
||||
A brightness multiplier in `[0, 1]`. **The provider owns the fade.** The renderer doesn't track timestamps or decay; it just renders whatever brightness you return.
|
||||
|
||||
- Returning a bare string (`'hit'`) is equivalent to `{ state: 'hit', alpha: 1, color: null }`.
|
||||
- Returning `{ state: 'hit', alpha: 0 }` is the same as returning `null` — the highway clamps it and drops the result before the renderer sees it. Use this as a clean way to signal "stop rendering" without changing your control flow.
|
||||
- Out-of-range numbers are clamped to `[0, 1]`. Non-finite (`NaN`, `Infinity`) falls back to `1`.
|
||||
|
||||
### `color`
|
||||
|
||||
Optional CSS-style colour string (e.g., `'#7ef'`, `'rgb(255, 90, 90)'`). When present, the renderer uses it instead of the default string colour for the lit gem / sustain trail. When absent or non-string, the renderer keeps its default palette.
|
||||
|
||||
## What the highway does to your return value
|
||||
|
||||
[`static/highway.js`](../static/highway.js) line 290:
|
||||
|
||||
```js
|
||||
function _noteState(note, chartTime) {
|
||||
if (!_noteStateProvider) return null;
|
||||
let raw;
|
||||
try { raw = _noteStateProvider(note, chartTime); } catch (e) { return null; }
|
||||
if (!raw) return null;
|
||||
const state = typeof raw === 'string' ? raw : raw.state;
|
||||
if (state !== 'hit' && state !== 'active' && state !== 'miss') return null;
|
||||
const alpha = (raw && typeof raw === 'object' && Number.isFinite(raw.alpha))
|
||||
? Math.max(0, Math.min(1, raw.alpha))
|
||||
: 1;
|
||||
if (alpha <= 0) return null;
|
||||
const color = (raw && typeof raw === 'object' && typeof raw.color === 'string') ? raw.color : null;
|
||||
return { state, alpha, color };
|
||||
}
|
||||
```
|
||||
|
||||
Practical implications:
|
||||
|
||||
- **Provider exceptions are swallowed.** If your provider throws, the renderer sees `null` for that note this frame. The next frame's call is independent. Cleanest for the renderer (no draw stall), trickiest for the provider author (no console signal that something's wrong). Log inside your provider if you need to diagnose.
|
||||
- **Invalid `state` strings return `null`.** Typos like `'Hit'` or `'hits'` are silently rejected. The renderer keeps drawing the note dim.
|
||||
- **`alpha === 0` returns `null`.** The dim-render path is what you want here anyway; this is the cleanest exit.
|
||||
- **Renderers receive the normalized object** (`{ state, alpha, color }`) or `null`. They never see your raw return value, so feel free to return shorthands.
|
||||
|
||||
## How to call `bundle.getNoteState` from a renderer
|
||||
|
||||
The bundle passed to `draw(bundle)` carries a stable reference (`getNoteState: _noteState` — same function every frame, never reallocated). You can cache it on the renderer instance:
|
||||
|
||||
```js
|
||||
init(canvas, bundle) {
|
||||
this.ctx = canvas.getContext('2d');
|
||||
this.getNoteState = bundle.getNoteState; // safe to cache the function ref
|
||||
},
|
||||
draw(bundle) {
|
||||
for (const n of bundle.notes) {
|
||||
const st = this.getNoteState(n, n.t);
|
||||
if (st && st.state === 'hit') /* paint bright */;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Two pitfalls when calling it
|
||||
|
||||
1. **Pass `chord.t`, not `cn.t`, for chord constituents.** Chord constituents are keyed in the producer by the chord's time, not the constituent's. (Most chart formats give the constituent the same `t` as the chord anyway, but the contract is explicit and you should follow it.)
|
||||
```js
|
||||
for (const ch of bundle.chords) {
|
||||
for (const cn of ch.notes) {
|
||||
const st = bundle.getNoteState(cn, ch.t); // ← ch.t
|
||||
}
|
||||
}
|
||||
```
|
||||
2. **Don't cache the return value across frames.** It encodes a fade. Re-call every frame.
|
||||
|
||||
## How to register as a provider
|
||||
|
||||
```js
|
||||
function myNoteStateFor(note, chartTime) {
|
||||
// note: { t, s, f, sus, … } — chart note object (or a chord constituent)
|
||||
// chartTime: number — `t` for single notes, the chord's `t` for chord constituents
|
||||
const verdict = myStore.lookup(`${chartTime}_${note.s}_${note.f}`);
|
||||
if (!verdict) return null;
|
||||
const age = performance.now() - verdict.timestamp;
|
||||
if (age > 800) return null; // owned fade — drop after 800 ms
|
||||
return {
|
||||
state: verdict.hit ? 'hit' : 'miss',
|
||||
alpha: Math.max(0, 1 - age / 800), // linear fade
|
||||
};
|
||||
}
|
||||
|
||||
// Mount on every highway in the session. In a typical plugin:
|
||||
// - call once on `song:ready`, after `createHighway()` has wired the panel
|
||||
// - call again whenever a new panel mounts (splitscreen)
|
||||
highway.setNoteStateProvider(myNoteStateFor);
|
||||
```
|
||||
|
||||
To play nicely with other plugins (or with `note_detect`'s default registration), check what's already there:
|
||||
|
||||
```js
|
||||
const existing = highway.getNoteStateProvider();
|
||||
if (existing == null) {
|
||||
highway.setNoteStateProvider(myNoteStateFor);
|
||||
} else {
|
||||
// Decide: chain (fallback to existing if you return null),
|
||||
// replace (existing.foo behaviour is lost), or abort.
|
||||
}
|
||||
```
|
||||
|
||||
## Keying conventions used by `note_detect`
|
||||
|
||||
If your provider is replacing or extending `note_detect`'s, match its key format:
|
||||
|
||||
| Note shape | Key in `noteResults` |
|
||||
|---|---|
|
||||
| Single note at `t = 12.34`, string `s`, fret `f` | `"12.34_{s}_{f}"` |
|
||||
| Chord constituent (chord at `t = 12.34`, constituent string `s`, fret `f`) | `"12.34_{s}_{f}"` — keyed by **chord time**, not constituent time |
|
||||
| Chord-frame verdict for that chord | `"12.34_chord"` |
|
||||
|
||||
The string interpolation uses the chart's `t` directly (no rounding, no string coercion beyond `${}`). Don't `.toFixed(2)` it — small floating-point drift between chord and constituent `t` values would create a key mismatch.
|
||||
|
||||
## Sustain state — `'hit'` vs `'active'`
|
||||
|
||||
Both are "lit" states; the difference is *who owns the fade*:
|
||||
|
||||
- `'hit'`: the gem was struck cleanly. The provider returns this once per frame, decaying `alpha` over a few hundred ms, then returns `null` when the glow should end.
|
||||
- `'active'`: a sustained note (one with `sus > 0`) is currently being held on-pitch. The provider keeps returning `{ state: 'active', alpha: 1 }` every frame as long as the hold is valid. When the hold ends (pitch slipped, sustain expired, song paused), the provider stops returning state.
|
||||
|
||||
A renderer typically treats both the same way visually — bright gem, bright sustain trail — but `'active'` differs in *duration*: it stays for as long as the hold lasts (potentially seconds), while `'hit'` is a short post-strike glow.
|
||||
|
||||
`note_detect` tracks the on-pitch state of held sustains in a `_susActiveUntil` Map (key → `performance.now()` when the grace window ends). A typical grace window is 30 ms (one or two pitch frames), so a brief audio gap during a sustained note doesn't kill the glow.
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
1. **Provider returns truthy forever → gem stays lit forever.** Always return `null` when the effect should end. The renderer has no other way to know.
|
||||
2. **Provider tracks fade in renderer state instead.** Don't. The provider is consulted by *every* renderer and you'd end up with mismatched fades on each. Owns-its-own-decay is the design.
|
||||
3. **Renderer passes a per-constituent `t` instead of the chord's.** Silent failure — the lookup misses, all chord gems render dim. See the second pitfall above.
|
||||
4. **Calling `setNoteStateProvider` once at module load.** The highway might not exist yet, or a later splitscreen panel might create another highway. Re-register on every `song:ready` and every new panel mount.
|
||||
5. **Renderer caches the bundle.** Bundle arrays are live references but the object itself is constructed per frame for some fields. Don't hold `let myBundle = bundle` across frames. Caching the function ref `bundle.getNoteState` is fine — it's stable.
|
||||
6. **Provider does heavy work per call.** It runs O(visible notes × FPS) — typically 1000+ calls/sec during dense passages. A Map lookup is fine; a regex / JSON parse / sort is not.
|
||||
|
||||
## Where to look next
|
||||
|
||||
- [`realtime-scoring-pipeline.md`](realtime-scoring-pipeline.md) — full data flow from audio → verdict → provider.
|
||||
- [`visualization-feedback-guide.md`](visualization-feedback-guide.md) — practical "how do I add this to my viz" walkthrough with a minimal 2D Canvas example.
|
||||
- [`../plugins/note_detect/`](../plugins/note_detect/) — the canonical producer. Search `screen.js` for `setNoteStateProvider` and `noteStateFor`.
|
||||
- [`../plugins/highway_3d/screen.js`](../plugins/highway_3d/screen.js) — canonical custom-renderer consumer. Search for `getNoteState`.
|
||||
- [`../CLAUDE.md`](../CLAUDE.md) § "Note-state provider" — the short version of this doc embedded in the plugin-system overview.
|
||||
@@ -0,0 +1,227 @@
|
||||
# Real-time scoring pipeline
|
||||
|
||||
This doc traces how a single audio sample becomes a lit-up gem on the highway. It exists because the path crosses five layers — native engine, IPC bridge, `note_detect` plugin, highway core, renderer — and the contract between them is currently spread across `slopsmith/CLAUDE.md`, `slopsmith-desktop/`, and four different plugin source files.
|
||||
|
||||
**Audience**: plugin authors building a visualization that wants to react to detected hits, and maintainers debugging why a gem isn't lighting up.
|
||||
|
||||
**Companion docs**
|
||||
- [`note-state-provider.md`](note-state-provider.md) — the `setNoteStateProvider` / `bundle.getNoteState` API contract, in isolation.
|
||||
- [`visualization-feedback-guide.md`](visualization-feedback-guide.md) — a practical "how do I add hit feedback to my custom viz" walkthrough.
|
||||
- `slopsmith-desktop/docs/audio-engine-architecture.md` (companion repo) — internals of the JUCE engine + ML detector that produce the verdicts this pipeline consumes.
|
||||
|
||||
## Pipeline at a glance
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ slopsmith-desktop (native, JUCE + ONNX + Basic Pitch) │
|
||||
│ │
|
||||
│ audio device → ring buffer → ┌─ PitchDetector (YIN) │
|
||||
│ ├─ MlNoteDetector (Basic Pitch / ONNX) │
|
||||
│ ├─ ChordScorer (FFT + harmonic comb) │
|
||||
│ └─ NoteVerifier (background thread) │
|
||||
│ │ │
|
||||
└───────────────────────────────────────┼──────────────────────────────┘
|
||||
│ IPC (Electron preload)
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ note_detect plugin (slopsmith/plugins/note_detect/screen.js) │
|
||||
│ │
|
||||
│ drain audio:getNoteVerdicts(songTime, playing) │
|
||||
│ (or fall back to browser matchNotes + audio:scoreChord) │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ noteResults Map keyed by `${time}_${string}_${fret}` │
|
||||
│ │ │
|
||||
│ ├─► window.slopsmith.emit('note:hit' | 'note:miss', judgment) │
|
||||
│ │ │
|
||||
│ └─► highway.setNoteStateProvider(noteStateFor) │
|
||||
└───────────────────────────────────────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ highway core (slopsmith/static/highway.js) │
|
||||
│ │
|
||||
│ bundle.getNoteState(note, chartTime) → null │
|
||||
│ │ 'hit' | 'active' | 'miss' │
|
||||
│ │ { state, alpha, color } │
|
||||
└───────────────────────────────────────┬──────────────────────────────┘
|
||||
│ per visible note, per frame
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ Renderer (built-in 2D, bundled 3D, or your custom viz) │
|
||||
│ │
|
||||
│ draw(bundle) { │
|
||||
│ for (const n of bundle.notes) { │
|
||||
│ const st = bundle.getNoteState(n, n.t); │
|
||||
│ // light gem, sustain, sparkle … │
|
||||
│ } │
|
||||
│ } │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Two facts shape every other detail:
|
||||
|
||||
1. **Verdicts are produced once and consumed many times.** A renderer must not call `audio:scoreChord` itself. Scoring runs once in the engine (or in `note_detect`'s browser fallback) and the resulting per-note state is published through a single, idempotent provider. Renderers read the provider in their `draw()` loop and that's it.
|
||||
2. **The chord-time semantic is load-bearing.** Chord constituents are keyed in `noteResults` as `${chordTime}_${string}_${fret}`. A renderer that passes a per-constituent timestamp instead of the chord's `t` will look up nothing and silently render dim gems. See [`note-state-provider.md`](note-state-provider.md) for the rule.
|
||||
|
||||
## Layer 1 — Native engine
|
||||
|
||||
In normal desktop sessions, scoring runs in C++ on a JUCE audio thread + a background verifier thread, not in JavaScript. The architecture doc in the companion repo covers internals; the surface area visible to JS is:
|
||||
|
||||
| IPC method | Purpose | Cadence on the JS side |
|
||||
|---|---|---|
|
||||
| `audio:setChart(chart)` | Push the active arrangement once per song / arrangement switch. | Once per `playSong()`. |
|
||||
| `audio:getNoteVerdicts(songTime, playing)` | Drain finalized per-note verdicts AND push the renderer's playhead. | ~20 Hz (the `detectInterval` tick). |
|
||||
| `audio:getPitchDetection()` | Monophonic dominant pitch (YIN or ML, whichever is ready). | ~20 Hz. |
|
||||
| `audio:scoreChord(ctx)` | One-shot polyphonic score against the engine's input ring. | Per chord event during matching (fallback path). |
|
||||
| `audio:detectNotes()` | Full ML active-pitch set: `{notes: [{midi, confidence, onsetMs, onsetSeq}], sampleRate}`. | Optional; only the diagnostic / ML-aware code paths use it. |
|
||||
| `audio:isMlNoteDetection()` | `true` when Basic Pitch is loaded and ready. | Once at startup. |
|
||||
|
||||
All of these are feature-detected (`typeof audio.getNoteVerdicts === 'function'`) so the renderer keeps working against a downlevel addon.
|
||||
|
||||
Handlers live in [`audio-bridge.ts`](../../slopsmith-desktop/src/main/audio-bridge.ts) (in the `slopsmith-desktop` repo). Look for the `ipcMain.handle('audio:setChart', ...)`, `'audio:getNoteVerdicts'`, etc. blocks around lines 500–560.
|
||||
|
||||
## Layer 2 — `note_detect` plugin
|
||||
|
||||
`note_detect` is the only consumer of those IPC methods today. Everything downstream consumes `note_detect`'s output, not the engine's output directly. There are **two paths** through this layer; the engine path is preferred on desktop, the browser path is the fallback.
|
||||
|
||||
### Engine-verifier path (modern, desktop only)
|
||||
|
||||
On `playSong()`, `note_detect` calls `audio:setChart(arrangementChart)` and then polls `audio:getNoteVerdicts(songTime, playing)` on the detect tick. The verifier in C++ has already scored each chart note against the audio ring (using harmonic-comb + onset detection); each call returns the verdicts that have finalized since the last drain.
|
||||
|
||||
Each verdict has the shape:
|
||||
|
||||
```js
|
||||
{
|
||||
id: string, // chart-note id assigned at setChart time
|
||||
detected: boolean, // hit or miss
|
||||
detectedSongTime: number, // when the engine saw the onset
|
||||
centsError: number, // cents off the expected pitch
|
||||
snr: number // confidence-ish signal:noise ratio
|
||||
}
|
||||
```
|
||||
|
||||
`note_detect` translates each verdict into its internal `judgment` shape and writes it into `noteResults`, keyed `${chartNote.t}_${chartNote.s}_${chartNote.f}` for single notes, and `${chord.t}_${cn.s}_${cn.f}` for each chord constituent. Chord-level verdicts also write a chord-time-only entry `${chord.t}_chord` used for chord-frame tint.
|
||||
|
||||
### Browser fallback path (legacy / non-desktop)
|
||||
|
||||
If the desktop bridge isn't available (web build, downlevel addon, audio engine stopped), the same plugin runs the entire pipeline in the browser:
|
||||
|
||||
1. `getUserMedia` → `AudioContext` → `ChannelSplitterNode` → `ScriptProcessor` (per-channel mono).
|
||||
2. Per-frame YIN pitch detect + onset gate.
|
||||
3. `matchNotes()` — windowed match between detected events and chart notes (timing tolerance + pitch tolerance from settings).
|
||||
4. For chord events: `_ndScoreChord()` runs a renderer-side FFT + band-energy scorer on the accumulated buffer.
|
||||
|
||||
Either way, the result lands in `noteResults` with the same keys, and the downstream consumers don't care which path produced it.
|
||||
|
||||
### Publishing
|
||||
|
||||
Two channels:
|
||||
|
||||
```js
|
||||
// 1. Global event (for stats panels, journaling plugins, etc.)
|
||||
window.slopsmith.emit(judgment.hit ? 'note:hit' : 'note:miss', judgment);
|
||||
|
||||
// 2. Per-note state provider (for the highway + renderers)
|
||||
highway.setNoteStateProvider(noteStateFor);
|
||||
```
|
||||
|
||||
The provider function `noteStateFor(note, chartTime)` reads `noteResults` and returns one of:
|
||||
|
||||
| Return | Meaning |
|
||||
|---|---|
|
||||
| `null` (or anything falsy) | No state — renderer draws this note normally. |
|
||||
| `'hit'` | Note was struck cleanly. Fade owned by the provider. |
|
||||
| `'active'` | Sustained note currently still being held on-pitch. |
|
||||
| `'miss'` | Note expired without a clean detection. |
|
||||
| `{ state, alpha, color }` | Object form — same `state` field, plus explicit `alpha` (0..1) and optional `color` override. |
|
||||
|
||||
The full contract — including the keying rules and sustain semantics — is in [`note-state-provider.md`](note-state-provider.md).
|
||||
|
||||
## Layer 3 — Highway core
|
||||
|
||||
The highway exposes the provider to renderers via the per-frame `bundle` passed to `draw(bundle)`. The relevant lines in [`static/highway.js`](../static/highway.js):
|
||||
|
||||
- The provider slot itself: `let _noteStateProvider = null;` at line 153.
|
||||
- The normalizer that the bundle exposes: `function _noteState(note, chartTime) { … }` at line 290. This catches exceptions thrown by the provider, validates the `state` is one of `'hit'/'active'/'miss'`, clamps `alpha` to `[0, 1]`, drops zero-alpha returns, and returns `null` otherwise.
|
||||
- The bundle field: `getNoteState: _noteState` (line 514) — a **stable reference**, no per-frame allocation. Custom renderers can safely cache it.
|
||||
- The API method: `setNoteStateProvider(fn)` on the highway's exported object (line 2974). Last writer wins; passing `null` clears the provider.
|
||||
|
||||
The built-in 2D renderer consumes the provider in three places — `drawNote` (line 1445), `drawSustains` (line 1364), and the chord-frame path (line 1655) — so the default visuals work as soon as `note_detect` is enabled, without any per-plugin wiring.
|
||||
|
||||
## Layer 4 — Renderer
|
||||
|
||||
In your renderer's `draw(bundle)`, the typical pattern is:
|
||||
|
||||
```js
|
||||
draw(bundle) {
|
||||
for (const n of bundle.notes) {
|
||||
const state = bundle.getNoteState(n, n.t); // n.t for single notes
|
||||
if (state) {
|
||||
// state.state is 'hit' | 'active' | 'miss'
|
||||
// state.alpha is 0..1 (provider-owned decay)
|
||||
// state.color is optional string override
|
||||
drawLitGem(n, state);
|
||||
} else {
|
||||
drawDimGem(n);
|
||||
}
|
||||
}
|
||||
for (const ch of bundle.chords) {
|
||||
for (const cn of ch.notes) {
|
||||
const state = bundle.getNoteState(cn, ch.t); // ch.t, NOT cn.t
|
||||
// …
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Two patterns the bundled renderers use that you can copy:
|
||||
|
||||
- **2D highway** ([`static/highway.js`](../static/highway.js) `drawNote` at line 1445): `'hit'`/`'active'` → bright string color + additive halo + sparkle. `'miss'` → faint red wash. Sustain trail brightens with `'active'`.
|
||||
- **3D highway** ([`plugins/highway_3d/screen.js`](../plugins/highway_3d/screen.js) — search for `getNoteState`): outline mesh + body switch from dim material (`mStr[s]`) to bright (`mGlow[s]`); sustained notes keep the glow while `'active'`; a contained sparkle is queued on the 2D overlay layer.
|
||||
|
||||
For a from-scratch viz, [`visualization-feedback-guide.md`](visualization-feedback-guide.md) has a minimal Canvas2D example.
|
||||
|
||||
## Concrete trace: strumming a C major at chart time 12.34 s
|
||||
|
||||
Chart event: a `chord` at `t = 12.34` with three constituents — strings 1 (B), 2 (G), 3 (D) at frets `(1, 0, 2)`.
|
||||
|
||||
1. **t ≈ 12.30 s**: the player strums. The audio thread captures samples into `inputFrameRing`. The Basic Pitch ML detector publishes `B3`, `G3`, `D3` as active pitches; the `NoteVerifier` thread, which has had the chart since `playSong()`, opens scoring windows around `t = 12.34` for each chord-note (`12.34 ± timingTolerance`).
|
||||
2. **t ≈ 12.34 s**: Each open-window note's harmonic comb confirms the expected pitch present. The verifier closes the windows shortly after `12.34 + timingTolerance` and pushes three verdicts onto the drain queue: `{detected: true, detectedSongTime: 12.31, centsError: +4, snr: 5.3}` for each.
|
||||
3. **t ≈ 12.39 s**: `note_detect`'s detect tick calls `audio:getNoteVerdicts(12.39, true)`. It receives the three verdicts, builds three `judgment` objects, writes:
|
||||
- `noteResults["12.34_1_1"] = { hit: true, … }`
|
||||
- `noteResults["12.34_2_0"] = { hit: true, … }`
|
||||
- `noteResults["12.34_3_2"] = { hit: true, … }`
|
||||
- `noteResults["12.34_chord"] = { hit: true, hitStrings: 3, totalStrings: 3, … }`
|
||||
- Emits `window.slopsmith.emit('note:hit', judgment)` for the chord.
|
||||
4. **Next rAF**: the renderer's `draw(bundle)` iterates `bundle.chords`. For the chord at `t = 12.34` it calls `bundle.getNoteState(cn, 12.34)` for each constituent. Each call resolves `noteStateFor(cn, 12.34)`, which looks up `noteResults["12.34_${cn.s}_${cn.f}"]`, finds a recent `hit`, returns `{ state: 'hit', alpha: ~1, color: null }`.
|
||||
5. **2D renderer**: each gem in the chord renders with bright string color + halo + sparkle. The chord-frame tint goes green.
|
||||
6. **3D renderer (if active)**: outline + body swap to `mGlow[s]`; a sparkle entry is queued for each gem on the 2D overlay layer; the chord frame box tints green.
|
||||
7. **Over the next ~300 ms**: `noteStateFor` returns a decaying `alpha` (provider-owned fade). When `alpha` drops below the floor, it returns `null` and the gems revert to dim.
|
||||
|
||||
If any string had been missed (say string 3 silent because of an over-aggressive mute), step 2 would have closed that window with `detected: false`. `noteResults["12.34_3_2"]` would be a `miss`, the chord-frame verdict would still be `hit` (any-constituent-hit rule per [PR #316](https://github.com/byrongamatos/slopsmith/pull/316)), and the renderer would paint string 3 with the miss wash while the others lit up.
|
||||
|
||||
## Where to look next
|
||||
|
||||
If you want to…
|
||||
|
||||
- **Build a custom viz that reacts to hits** → [`visualization-feedback-guide.md`](visualization-feedback-guide.md).
|
||||
- **Understand the provider contract in full** (semantics of `alpha`, key formats, sustain rules) → [`note-state-provider.md`](note-state-provider.md).
|
||||
- **Dig into the engine internals** (NoteVerifier thread, ML detector loading, ring buffer) → `slopsmith-desktop/docs/audio-engine-architecture.md` in the companion repo.
|
||||
- **Tune detection quality** (false misses, latency, A/V offset) → [`note-detect-tuning.md`](note-detect-tuning.md).
|
||||
- **Read the canonical setRenderer contract** → [`../CLAUDE.md`](../CLAUDE.md) § "Visualization plugins".
|
||||
|
||||
## Key files
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `slopsmith-desktop/src/audio/AudioEngine.cpp:1447` | Audio device callback — entry point for every input sample. |
|
||||
| `slopsmith-desktop/src/audio/AudioEngine.h:196` | `inputFrameRing` declaration (lock-free SPSC, 8192 samples). |
|
||||
| `slopsmith-desktop/src/audio/NoteVerifier.cpp:152` | Background verifier thread loop. |
|
||||
| `slopsmith-desktop/src/audio/NoteVerifier.cpp:52` | `setChart()` — push the active arrangement. |
|
||||
| `slopsmith-desktop/src/main/audio-bridge.ts` | IPC handlers (look for `audio:setChart`, `audio:getNoteVerdicts`, `audio:scoreChord`). |
|
||||
| `slopsmith/plugins/note_detect/screen.js` | `noteResults` map, `noteStateFor`, `setNoteStateProvider` registration, verdict drain loop. |
|
||||
| `slopsmith/static/highway.js:153` | `_noteStateProvider` slot. |
|
||||
| `slopsmith/static/highway.js:290` | `_noteState()` normalizer (catches exceptions, clamps alpha, validates state). |
|
||||
| `slopsmith/static/highway.js:514` | `getNoteState: _noteState` bundle field. |
|
||||
| `slopsmith/static/highway.js:2974` | `setNoteStateProvider` on the public highway API. |
|
||||
@@ -0,0 +1,203 @@
|
||||
# Adding hit feedback to a visualization
|
||||
|
||||
A practical guide to making your custom visualization light up when notes are hit, dim when they're missed, and stay glowing while sustained notes are held correctly.
|
||||
|
||||
**Audience**: you're building a visualization plugin (per [`../CLAUDE.md`](../CLAUDE.md) § "Visualization plugins") and you want it to react to detected hits the same way the built-in 2D and 3D highways do.
|
||||
|
||||
**Prerequisites**
|
||||
- You can already get a `setRenderer`-style plugin loading. (If not, start with [`../CLAUDE.md`](../CLAUDE.md) — that contract is the entry point.)
|
||||
- You've skimmed [`realtime-scoring-pipeline.md`](realtime-scoring-pipeline.md). You don't need to understand the engine internals, just the shape of `bundle.getNoteState`.
|
||||
- `note_detect` is enabled in your test setup. Without it, the provider stays unregistered and `bundle.getNoteState` returns `null` for everything — your viz will work but you won't see feedback.
|
||||
|
||||
## The whole contract, in 5 lines
|
||||
|
||||
```js
|
||||
draw(bundle) {
|
||||
for (const n of bundle.notes) {
|
||||
const st = bundle.getNoteState(n, n.t); // for single notes, pass n.t
|
||||
// st is null | { state: 'hit' | 'active' | 'miss', alpha: 0..1, color: string|null }
|
||||
}
|
||||
for (const ch of bundle.chords) {
|
||||
for (const cn of ch.notes) {
|
||||
const st = bundle.getNoteState(cn, ch.t); // for chord constituents, pass CHORD time
|
||||
// …
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
That's it. The full semantics — when the provider fires, who owns fade timing, what the keys are — are in [`note-state-provider.md`](note-state-provider.md). This doc is about how to use them.
|
||||
|
||||
## Minimal 2D Canvas example
|
||||
|
||||
A complete `setRenderer`-style plugin that draws each chart note as a circle, lit up when hit, with a green halo for active sustains and a red wash for misses. ~80 lines, no dependencies. Drop it in `plugins/my_viz/screen.js` along with a matching `plugin.json` (`"type": "visualization"`).
|
||||
|
||||
```js
|
||||
(function () {
|
||||
'use strict';
|
||||
|
||||
window.slopsmithViz_my_viz = function () {
|
||||
let ctx = null;
|
||||
let W = 0, H = 0;
|
||||
|
||||
return {
|
||||
contextType: '2d',
|
||||
|
||||
init(canvas, bundle) {
|
||||
ctx = canvas.getContext('2d');
|
||||
W = canvas.width;
|
||||
H = canvas.height;
|
||||
},
|
||||
|
||||
resize(w, h) {
|
||||
W = w; H = h;
|
||||
},
|
||||
|
||||
draw(bundle) {
|
||||
if (!ctx) return;
|
||||
ctx.clearRect(0, 0, W, H);
|
||||
|
||||
const now = bundle.currentTime;
|
||||
const stringCount = bundle.stringCount || 6;
|
||||
|
||||
// Project a chart time onto x. Notes ahead of `now` slide
|
||||
// in from the right; notes behind `now` slide off left.
|
||||
const xFor = (t) => W * 0.5 + (t - now) * 100;
|
||||
const yFor = (s) => H * (0.2 + 0.6 * (s / (stringCount - 1)));
|
||||
|
||||
// Single notes.
|
||||
for (const n of bundle.notes) {
|
||||
const x = xFor(n.t);
|
||||
if (x < -50 || x > W + 50) continue; // off-screen cull
|
||||
const y = yFor(n.s);
|
||||
const st = bundle.getNoteState(n, n.t);
|
||||
drawGem(ctx, x, y, n, st);
|
||||
}
|
||||
|
||||
// Chord constituents — pass CHORD time, not constituent time.
|
||||
for (const ch of bundle.chords) {
|
||||
const x = xFor(ch.t);
|
||||
if (x < -50 || x > W + 50) continue;
|
||||
for (const cn of ch.notes) {
|
||||
const y = yFor(cn.s);
|
||||
const st = bundle.getNoteState(cn, ch.t);
|
||||
drawGem(ctx, x, y, cn, st);
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
destroy() {
|
||||
ctx = null;
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
function drawGem(ctx, x, y, note, st) {
|
||||
const r = 12;
|
||||
ctx.beginPath();
|
||||
ctx.arc(x, y, r, 0, Math.PI * 2);
|
||||
|
||||
if (!st) {
|
||||
// Dim — no judgment yet (or expired).
|
||||
ctx.fillStyle = '#445';
|
||||
ctx.fill();
|
||||
return;
|
||||
}
|
||||
|
||||
// Lit. `alpha` is provider-owned fade.
|
||||
const a = st.alpha;
|
||||
if (st.state === 'hit' || st.state === 'active') {
|
||||
ctx.fillStyle = st.color || `rgba(120, 220, 140, ${a})`;
|
||||
ctx.fill();
|
||||
// Halo grows with brightness — purely cosmetic.
|
||||
ctx.beginPath();
|
||||
ctx.arc(x, y, r + 6 * a, 0, Math.PI * 2);
|
||||
ctx.strokeStyle = `rgba(120, 220, 140, ${a * 0.5})`;
|
||||
ctx.lineWidth = 2;
|
||||
ctx.stroke();
|
||||
} else if (st.state === 'miss') {
|
||||
ctx.fillStyle = `rgba(220, 90, 90, ${a})`;
|
||||
ctx.fill();
|
||||
}
|
||||
}
|
||||
})();
|
||||
```
|
||||
|
||||
Things this example deliberately skips:
|
||||
|
||||
- **Sustain trails.** Add a `ctx.fillRect` from `xFor(n.t)` to `xFor(n.t + n.sus)` and switch its colour based on the `'active'` state.
|
||||
- **Inverted / lefty.** Check `bundle.inverted` and `bundle.lefty` and mirror your y-axis / x-axis transforms.
|
||||
- **Difficulty filter awareness.** `bundle.notes` is already filtered — you get this for free.
|
||||
- **Capo / tuning offsets.** Not needed for visualization; the gems just show the chart positions.
|
||||
|
||||
## Case study — how the 3D highway does it
|
||||
|
||||
[`../plugins/highway_3d/screen.js`](../plugins/highway_3d/screen.js) is the canonical custom renderer. It uses the same `bundle.getNoteState(note, chartTime)` call, but applied to Three.js materials.
|
||||
|
||||
The cliff notes (search the file for `getNoteState` to find each call site):
|
||||
|
||||
- For each visible note, the 3D highway calls `bundle.getNoteState(n, n.t)` (or `ch.t` for chord constituents).
|
||||
- A non-null return switches the gem's outline mesh from `mStr[s]` (dim, dull string colour) to `mGlow[s]` (bright emissive). The body mesh is swapped the same way.
|
||||
- Sustain trails use the bright material while `'active'`, the dim one otherwise.
|
||||
- A separate sparkle layer (`drawNotedetectSizzle()` on the 2D overlay canvas) queues a few crackling arcs and dots projected through the camera, fading with `st.alpha`. The 3D highway runs this on its 2D *overlay* canvas, not in the 3D scene, so the sparkles don't blow out the bloom or get hidden behind world geometry.
|
||||
- Chord-frame tint: when the chord-level entry (`${chord.t}_chord`) is `hit`, the chord-frame box tints green; on miss, it tints red. Per-constituent verdicts still drive the individual gem visuals so a partial-hit chord shows the frame green with individual constituents lit or dim.
|
||||
|
||||
Two implementation details worth copying:
|
||||
|
||||
1. The 3D highway caches `bundle.getNoteState` once per frame — `const getNs = bundle.getNoteState;` — and uses the cached ref for the inner loops. The function reference is stable across frames, so this is safe and saves a property read per gem.
|
||||
2. It null-guards every consumption (`if (st && st.state === 'hit')`) rather than relying on `st.state` to be defined. `null` is the common case — a single visible chord with 30 unstruck notes ahead of the playhead produces 30 null returns and 0 lit returns.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "I see no lit gems at all"
|
||||
|
||||
In order of likelihood:
|
||||
|
||||
1. **`note_detect` isn't running.** Open DevTools console and run:
|
||||
```js
|
||||
window.highway && window.highway.getNoteStateProvider()
|
||||
```
|
||||
If it returns `null`, no scorer is registered. Click the "Detect" button in the player controls (or enable it via your test fixture).
|
||||
2. **Your renderer isn't calling `bundle.getNoteState`.** Add a `console.log(st)` inside the loop. If you never see anything other than `null`, the provider is registered but you might be on a song with no scoring events yet (try strumming and watch the console).
|
||||
3. **You're passing the wrong `chartTime`.** Chord constituents need `chord.t`, not `cn.t`. Easy to miss if your inner loop reuses a variable named `t`.
|
||||
4. **Your renderer's `draw()` isn't firing.** Make sure the highway has had a `ready` message — `bundle.isReady` should be `true`. The factory gates `draw()` calls behind the ready flag.
|
||||
|
||||
### "Single notes light up but chord notes don't"
|
||||
|
||||
You're almost certainly passing `cn.t` instead of `ch.t` to `getNoteState`. The lookup map is keyed by the chord's time. Most chord constituents have the same `t` as the chord, but the contract still requires the chord time explicitly — and there are chart formats where constituent `t` values drift slightly.
|
||||
|
||||
### "Notes light up but never fade — they stay green forever"
|
||||
|
||||
The provider — not your renderer — owns fade timing. If you've replaced `note_detect` with your own provider and it returns a truthy state forever, gems stay lit forever. Return `null` when the effect should end. See [`note-state-provider.md`](note-state-provider.md) § "`alpha`" for how to encode a fade.
|
||||
|
||||
### "Sustains don't glow while held"
|
||||
|
||||
The provider needs to return `'active'` (not `'hit'`) for as long as the held pitch is detected. Check whether the producer in use (`note_detect` on desktop) is tracking sustain state — it does, via `_susActiveUntil`, but only when monophonic pitch detection is enabled and on-pitch within the configured cents tolerance. If your audio is noisy or the cents tolerance is tight, sustains may decay to `null` between frames.
|
||||
|
||||
### "My splitscreen panels all light up the same notes at the same time"
|
||||
|
||||
Two things might be going on:
|
||||
|
||||
1. **Two panels are scoring the same audio input.** Each `note_detect` instance currently shares the engine's single-source detection. See [issue #375](https://github.com/byrongamatos/slopsmith/issues/375) for the RFC on per-source detection. Until that lands, the fact that both panels light identically is expected.
|
||||
2. **Each panel's highway has its own provider slot.** If you're building a plugin that replaces the provider, register it on each panel's highway instance, not on a global. Splitscreen creates one highway per panel — `setNoteStateProvider` is per-highway.
|
||||
|
||||
### "My renderer prints `TypeError: bundle.getNoteState is not a function`"
|
||||
|
||||
The bundle field has been part of the contract since slopsmith#254. If `bundle.getNoteState` is missing, the host slopsmith you're running against predates that issue — you're on an old build. Check the version with `cat VERSION` in the slopsmith repo; anything ≥ 0.2 should have it. The field is set to a stable reference, never undefined.
|
||||
|
||||
### "I want to draw something on hit that isn't a gem (a particle burst, a sound, anything)"
|
||||
|
||||
Two options:
|
||||
|
||||
1. **Subscribe to events from inside your renderer.** `window.slopsmith.on('note:hit', judgment => { … })`. The judgment object has the chord/note info, timing/pitch error, etc. Fires once per detection, not per frame. Good for one-shots (sounds, screen shakes, achievement triggers).
|
||||
2. **Watch for state transitions in your draw loop.** Cache the last-seen state per note key in your renderer; when this frame's state changes from `null` to truthy, you know an event just landed. Good for cumulative effects (particle bursts that need to know their start time).
|
||||
|
||||
The provider doesn't itself give you an event stream — it gives you a sample-per-frame view of current state. Pick the right tool for your effect.
|
||||
|
||||
## Where to look next
|
||||
|
||||
- [`realtime-scoring-pipeline.md`](realtime-scoring-pipeline.md) — full data flow, top to bottom.
|
||||
- [`note-state-provider.md`](note-state-provider.md) — the producer contract (write your own scorer).
|
||||
- [`../CLAUDE.md`](../CLAUDE.md) — the canonical `setRenderer` / overlay contract.
|
||||
- [`../plugins/highway_3d/screen.js`](../plugins/highway_3d/screen.js) — case-study renderer.
|
||||
- [`note-detect-tuning.md`](note-detect-tuning.md) — what to do when detection itself is flaky (your viz works fine but the underlying hits / misses look wrong).
|
||||
@@ -6,6 +6,11 @@ The whole renderer is **one file** — `screen.js`, wrapped in an IIFE, register
|
||||
|
||||
**Styling (slopsmith `styles` capability).** This plugin owns its Tailwind CSS: it ships `assets/plugin.css` and declares `"styles": "assets/plugin.css"` in `plugin.json`, so core's prebuilt `static/tailwind.min.css` no longer scans it (it's excluded from core's content globs). The frontend injects `assets/plugin.css` as a `<link>` when the renderer activates. This is the one maintainer-time build step: after you add/change a Tailwind class in `screen.js` or `settings.html`, run `bash build-tailwind.sh` (pinned `tailwindcss@3.4.19`, `corePlugins.preflight=false` — utilities only) and **bump the `version` in `plugin.json`** so the injected `<link>`'s `?v=` cache-buster fetches the fresh file. The generated `assets/plugin.css` is committed; end users never build. See [docs/plugin-styles.md](../../docs/plugin-styles.md).
|
||||
|
||||
**For the contracts this plugin implements** (setRenderer, the note-state provider, the real-time scoring pipeline), see the repo-root docs:
|
||||
- [`../../docs/realtime-scoring-pipeline.md`](../../docs/realtime-scoring-pipeline.md) — end-to-end audio → lit gem trace. This plugin is one of the renderers in the final step.
|
||||
- [`../../docs/note-state-provider.md`](../../docs/note-state-provider.md) — the `bundle.getNoteState(note, chartTime)` contract this plugin consumes for hit/active/miss feedback.
|
||||
- [`../../docs/visualization-feedback-guide.md`](../../docs/visualization-feedback-guide.md) — practical "how do I add hit feedback to a viz" walkthrough, using this plugin as a case study.
|
||||
|
||||
> **Navigation note:** This guide references functions by name and uses the existing banner comments (`/* ── Scene initialisation ─ */`, etc.) as section anchors. Line numbers are deliberately avoided so this stays correct as the file evolves. Use `Grep` for the function name or banner text to jump to a section.
|
||||
|
||||
## File structure at a glance
|
||||
|
||||
Reference in New Issue
Block a user