mirror of
https://github.com/got-feedBack/feedBack-desktop.git
synced 2026-09-11 06:44:11 +00:00
perf(audio): gate ML note-detection pipeline (default OFF, arm on demand) (#51)
* fix(audio-input): stable name-based input identity + fail-loud open + bound read-back Replace the positional-index logicalSourceKey with a name-encoded one so a named device survives reorder/hotplug; resolve by name and fail loud instead of silently opening the default mic; read back and return the actually-bound device. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(audio): gate the ML note-detection pipeline behind a master enable The Basic-Pitch ONNX detector is the most expensive thing in the engine (~30 ms inference every hop) and on the default desktop path nothing reads it: note detection is scored by the harmonic-comb NoteVerifier, and the always-on home tuner runs its own YIN over raw frames. Yet the pipeline ran unconditionally from construction, pinning a core on an idle home screen. Add a master gate so ML only runs when a consumer actually needs it: - MlNoteDetector: std::atomic<bool> enabled{false}. pushSamples() early- returns on the audio thread (lock-free relaxed load, no feed) and runInferenceIfDue() early-returns on the inference thread (no Run()), so the whole pipeline is dormant until armed. setEnabled(false) clears the rolling window + published snapshot (clearAudioState resets hasPublished), so a re-arm starts cold and serves the YIN fallback until the first fresh inference. The inference thread stays alive but idle — toggling needs no thread restart. isEnabled() for symmetry; no-op stubs in the ONNX-off build. - AudioEngine::setMlNoteDetectionEnabled(bool) fans to every source's detector (whole pool, so a later-activated source inherits the arm state). - NodeAddon setNoteDetectionEnabled + audio-bridge ipc + preload, all typeof/ try-guarded so a downlevel addon ignores it (fail-safe to current behaviour). The renderer (note_detect) arms this true only while it will read ML notes (native-frame detection / non-verifier fallback) and false otherwise — a follow-up renderer change. Default OFF means the shipped verifier path and the home tuner pay nothing for ML. Verified: native addon builds clean (ONNX path); the standalone mlnd_test detects the full C-major triad when armed (3/3); ml-note-detection + multi-source JS suites pass (16/16). mlnotedetector/test.cpp arms the detector after prepare() to match the new default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(audio): make the ML gate reset race-free (thread-owned cold start) The first cut cleared the rolling window/FIFO from setEnabled() on the N-API thread while the inference thread was still alive — a data race on the buffers. Move the reset onto the thread that owns them, and fix two follow-on issues Codex flagged: - fifo.reset() TOCTOU: resetting the FIFO on the inference thread can still race an in-flight pushSamples() that passed the resetPending gate just before it was set (the >=8 ms callback gap is not a guarantee). Fix: the thread-side cold start DRAINS the FIFO (fifo.finishedRead(getNumReady()) — advances only the consumer's read index, safe SPSC) instead of fifo.reset(). clearAudioState() (with the real reset) is kept for the prepare()/stop() paths where the thread is already joined. resetPending stays set through the drain so pushSamples() is gated off the FIFO the whole time, then is released. - stale readiness on re-arm: setEnabled(true) exposed enabled=true immediately while hasPublished stayed true from the previous arm, so isReady() briefly served the old snapshot. Fix: drop hasPublished synchronously BEFORE storing enabled=true (release/acquire ordering: isReady() loads enabled before hasPublished, so seeing enabled=true guarantees seeing hasPublished=false). Other gate mechanics: the enabled-gate is at the top of the inference callback (disabled ⇒ no ingest, no inference), pushSamples() no-ops when !enabled or resetPending, and isReady() gates on enabled so a suspended detector serves the YIN fallback rather than a stale snapshot. mlnotedetector/test.cpp asserts both directions: fed the chord region while DISABLED, the detector publishes nothing and never becomes ready; armed, it still detects the full C-major triad (3/3). Addon rebuilds clean; tsc clean; ml-note-detection + multi-source JS suites pass (16/16). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Byron Gamatos
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
06c68262a9
commit
92a78b4c9a
+68
-8
@@ -292,16 +292,50 @@ window.__feedBackDesktopAudioHooks = window.__feedBackDesktopAudioHooks || {};
|
||||
|
||||
async function audioInputOpenHandler(request) {
|
||||
const source = (request && request.logicalSourceKey) ? String(request.logicalSourceKey) : '';
|
||||
const match = /^desktop-audio:([^:]+):input:(\d+)$/.exec(source);
|
||||
const inputType = match ? match[1] : safeKeyPart(deviceTypeSelect?.value || 'default');
|
||||
const inputIndex = match ? Number(match[2]) : -1;
|
||||
// A named device carries a STABLE name-encoded key so the selection
|
||||
// survives device reorder/hotplug (virtual/aggregate devices like
|
||||
// BlackHole reorder on enumeration). A legacy `:input:<N>` key — emitted
|
||||
// by older builds and any selection persisted before this change — still
|
||||
// resolves positionally for one more open.
|
||||
const nameMatch = /^desktop-audio:([^:]+):input:name:(.+)$/.exec(source);
|
||||
const indexMatch = nameMatch ? null : /^desktop-audio:([^:]+):input:(\d+)$/.exec(source);
|
||||
const inputType = nameMatch ? nameMatch[1]
|
||||
: indexMatch ? indexMatch[1]
|
||||
: safeKeyPart(deviceTypeSelect?.value || 'default');
|
||||
const typeInfo = currentDeviceTypes.find(t => safeKeyPart(t && t.name) === inputType)
|
||||
|| currentDeviceTypes.find(t => t && t.name === deviceTypeSelect?.value)
|
||||
|| currentDeviceTypes[0]
|
||||
|| null;
|
||||
const inputDevice = inputIndex >= 0 && typeInfo && Array.isArray(typeInfo.inputs)
|
||||
? (typeInfo.inputs[inputIndex] || '')
|
||||
: (inputDeviceSelect?.value || '');
|
||||
const typeInputs = typeInfo && Array.isArray(typeInfo.inputs) ? typeInfo.inputs : [];
|
||||
|
||||
// Resolve the picked device to a concrete name and FAIL LOUD when it's
|
||||
// gone. The old code fell through to '' (or whatever the Settings
|
||||
// dropdown showed), which makes the native engine open its DEFAULT device
|
||||
// — the internal mic. That silent substitution is the macOS wrong-mic bug
|
||||
// this handler exists to kill: the user picks BlackHole, setup "succeeds",
|
||||
// and every score is garbage with no signal anything went wrong.
|
||||
let inputDevice = '';
|
||||
if (nameMatch) {
|
||||
let decoded = '';
|
||||
try { decoded = decodeURIComponent(nameMatch[2]); } catch (_) { decoded = ''; }
|
||||
// The key encodes the trimmed name; match trim-tolerantly but bind the
|
||||
// engine's exact enumerated string.
|
||||
inputDevice = typeInputs.find(n => String(n).trim() === decoded) || '';
|
||||
if (!inputDevice) {
|
||||
return { outcome: 'failed', status: 'failed', reason: `Selected input device "${decoded || '(unknown)'}" is no longer available` };
|
||||
}
|
||||
} else if (indexMatch) {
|
||||
const idx = Number(indexMatch[2]);
|
||||
inputDevice = (idx >= 0 && idx < typeInputs.length) ? typeInputs[idx] : '';
|
||||
if (!inputDevice) {
|
||||
return { outcome: 'failed', status: 'failed', reason: 'Selected input device is no longer available' };
|
||||
}
|
||||
} else {
|
||||
// No recognizable source key — refuse rather than silently grabbing
|
||||
// whatever the Settings dropdown happens to show (another default path).
|
||||
return { outcome: 'failed', status: 'failed', reason: 'No input device selected' };
|
||||
}
|
||||
|
||||
const snapshot = currentAudioDeviceSnapshot();
|
||||
const result = await api.setDevice({
|
||||
inputType: typeInfo && typeInfo.name ? typeInfo.name : snapshot.inputType,
|
||||
@@ -314,7 +348,23 @@ window.__feedBackDesktopAudioHooks = window.__feedBackDesktopAudioHooks || {};
|
||||
const ok = typeof result === 'boolean' ? result : !!result?.ok;
|
||||
if (!ok) return { outcome: 'failed', status: 'failed', reason: result && result.error ? String(result.error) : 'Native audio device open failed' };
|
||||
if (typeof api.startAudio === 'function') await api.startAudio();
|
||||
return { outcome: 'handled', status: 'open' };
|
||||
|
||||
// Read back what the engine ACTUALLY bound, so callers (input_setup's
|
||||
// confirmation gate, note_detect) can surface "Now listening to: <device>"
|
||||
// and spot a silent mismatch instead of trusting the request blind.
|
||||
let boundType = (typeInfo && typeInfo.name) ? String(typeInfo.name) : '';
|
||||
let boundName = inputDevice;
|
||||
try {
|
||||
if (typeof api.getCurrentDevice === 'function') {
|
||||
const cur = await api.getCurrentDevice();
|
||||
if (cur && typeof cur === 'object') {
|
||||
if (cur.inputType) boundType = String(cur.inputType);
|
||||
if (cur.input) boundName = String(cur.input);
|
||||
}
|
||||
}
|
||||
} catch (_) { /* fall back to the requested identity */ }
|
||||
|
||||
return { outcome: 'handled', status: 'open', payload: { boundType, boundName, requestedName: inputDevice } };
|
||||
}
|
||||
|
||||
async function audioInputCloseHandler() {
|
||||
@@ -342,11 +392,21 @@ window.__feedBackDesktopAudioHooks = window.__feedBackDesktopAudioHooks || {};
|
||||
const driverSuffix = (showDriverType && typeName) ? ` (${typeName})` : '';
|
||||
const inputs = Array.isArray(typeInfo && typeInfo.inputs) ? typeInfo.inputs : [];
|
||||
inputs.forEach((deviceName, index) => {
|
||||
const logicalSourceKey = `desktop-audio:${safeKeyPart(typeName)}:input:${index}`;
|
||||
const hasRealName = (typeof deviceName === 'string' && !!deviceName.trim());
|
||||
const realName = hasRealName
|
||||
? deviceName.trim()
|
||||
: `Desktop input ${index + 1}`;
|
||||
// Identity in the key: a named device gets a STABLE name-encoded
|
||||
// key so selection survives device reorder/hotplug (encodeURIComponent
|
||||
// escapes ':' to %3A so it can't break the handler's parser). Only a
|
||||
// truly nameless input falls back to the positional index, where the
|
||||
// index is the only identity available. NOTE: selections persisted by
|
||||
// an older build use the legacy index key; the open handler still
|
||||
// resolves those positionally, but the picker now lists this device
|
||||
// under its name key, so a returning user re-picks once.
|
||||
const logicalSourceKey = hasRealName
|
||||
? `desktop-audio:${safeKeyPart(typeName)}:input:name:${encodeURIComponent(realName)}`
|
||||
: `desktop-audio:${safeKeyPart(typeName)}:input:${index}`;
|
||||
audioSession.registerInputSource({
|
||||
sourceId: `audio_engine:${logicalSourceKey}`,
|
||||
logicalSourceKey,
|
||||
|
||||
Reference in New Issue
Block a user