mirror of
https://github.com/got-feedBack/feedBack-desktop.git
synced 2026-08-16 05:37:40 +00:00
perf(audio): gate ML note-detection pipeline (default OFF, arm on demand) (#51)
* fix(audio-input): stable name-based input identity + fail-loud open + bound read-back Replace the positional-index logicalSourceKey with a name-encoded one so a named device survives reorder/hotplug; resolve by name and fail loud instead of silently opening the default mic; read back and return the actually-bound device. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(audio): gate the ML note-detection pipeline behind a master enable The Basic-Pitch ONNX detector is the most expensive thing in the engine (~30 ms inference every hop) and on the default desktop path nothing reads it: note detection is scored by the harmonic-comb NoteVerifier, and the always-on home tuner runs its own YIN over raw frames. Yet the pipeline ran unconditionally from construction, pinning a core on an idle home screen. Add a master gate so ML only runs when a consumer actually needs it: - MlNoteDetector: std::atomic<bool> enabled{false}. pushSamples() early- returns on the audio thread (lock-free relaxed load, no feed) and runInferenceIfDue() early-returns on the inference thread (no Run()), so the whole pipeline is dormant until armed. setEnabled(false) clears the rolling window + published snapshot (clearAudioState resets hasPublished), so a re-arm starts cold and serves the YIN fallback until the first fresh inference. The inference thread stays alive but idle — toggling needs no thread restart. isEnabled() for symmetry; no-op stubs in the ONNX-off build. - AudioEngine::setMlNoteDetectionEnabled(bool) fans to every source's detector (whole pool, so a later-activated source inherits the arm state). - NodeAddon setNoteDetectionEnabled + audio-bridge ipc + preload, all typeof/ try-guarded so a downlevel addon ignores it (fail-safe to current behaviour). The renderer (note_detect) arms this true only while it will read ML notes (native-frame detection / non-verifier fallback) and false otherwise — a follow-up renderer change. Default OFF means the shipped verifier path and the home tuner pay nothing for ML. Verified: native addon builds clean (ONNX path); the standalone mlnd_test detects the full C-major triad when armed (3/3); ml-note-detection + multi-source JS suites pass (16/16). mlnotedetector/test.cpp arms the detector after prepare() to match the new default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(audio): make the ML gate reset race-free (thread-owned cold start) The first cut cleared the rolling window/FIFO from setEnabled() on the N-API thread while the inference thread was still alive — a data race on the buffers. Move the reset onto the thread that owns them, and fix two follow-on issues Codex flagged: - fifo.reset() TOCTOU: resetting the FIFO on the inference thread can still race an in-flight pushSamples() that passed the resetPending gate just before it was set (the >=8 ms callback gap is not a guarantee). Fix: the thread-side cold start DRAINS the FIFO (fifo.finishedRead(getNumReady()) — advances only the consumer's read index, safe SPSC) instead of fifo.reset(). clearAudioState() (with the real reset) is kept for the prepare()/stop() paths where the thread is already joined. resetPending stays set through the drain so pushSamples() is gated off the FIFO the whole time, then is released. - stale readiness on re-arm: setEnabled(true) exposed enabled=true immediately while hasPublished stayed true from the previous arm, so isReady() briefly served the old snapshot. Fix: drop hasPublished synchronously BEFORE storing enabled=true (release/acquire ordering: isReady() loads enabled before hasPublished, so seeing enabled=true guarantees seeing hasPublished=false). Other gate mechanics: the enabled-gate is at the top of the inference callback (disabled ⇒ no ingest, no inference), pushSamples() no-ops when !enabled or resetPending, and isReady() gates on enabled so a suspended detector serves the YIN fallback rather than a stale snapshot. mlnotedetector/test.cpp asserts both directions: fed the chord region while DISABLED, the detector publishes nothing and never becomes ready; armed, it still detects the full C-major triad (3/3). Addon rebuilds clean; tsc clean; ml-note-detection + multi-source JS suites pass (16/16). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
06c68262a9
commit
92a78b4c9a
@@ -1577,6 +1577,16 @@ SourceChain* AudioEngine::getSource(int id)
|
|||||||
return (id == 0 || src.isActive()) ? &src : nullptr;
|
return (id == 0 || src.isActive()) ? &src : nullptr;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void AudioEngine::setMlNoteDetectionEnabled(bool e)
|
||||||
|
{
|
||||||
|
// Fan to every source in the fixed pool (not just active ones) so a source
|
||||||
|
// activated later inherits the current arm state instead of silently
|
||||||
|
// staying dormant. Each MlNoteDetector::setEnabled is a cheap atomic + a
|
||||||
|
// cold-state clear on a real transition.
|
||||||
|
for (int i = 0; i < kMaxSources; ++i)
|
||||||
|
sources[(size_t) i]->getMlNoteDetector().setEnabled(e);
|
||||||
|
}
|
||||||
|
|
||||||
std::vector<AudioEngine::SourceInfo> AudioEngine::listSources() const
|
std::vector<AudioEngine::SourceInfo> AudioEngine::listSources() const
|
||||||
{
|
{
|
||||||
std::vector<SourceInfo> out;
|
std::vector<SourceInfo> out;
|
||||||
|
|||||||
@@ -30,6 +30,13 @@ public:
|
|||||||
PitchDetector& getPitchDetector() { return source0().getPitchDetector(); }
|
PitchDetector& getPitchDetector() { return source0().getPitchDetector(); }
|
||||||
MlNoteDetector& getMlNoteDetector() { return source0().getMlNoteDetector(); }
|
MlNoteDetector& getMlNoteDetector() { return source0().getMlNoteDetector(); }
|
||||||
|
|
||||||
|
// Arm/suspend the ML note-detection pipeline across every source's detector.
|
||||||
|
// Defaults off; the renderer (note_detect) calls this true only while a
|
||||||
|
// consumer actually reads ML notes (native-frame detection / non-verifier
|
||||||
|
// fallback) and false otherwise, so the default harmonic-comb verifier path
|
||||||
|
// — and the always-on home tuner — never pay for ONNX inference. Main thread.
|
||||||
|
void setMlNoteDetectionEnabled(bool e);
|
||||||
|
|
||||||
// Load the Basic Pitch ONNX model for the polyphonic ML detector. When a
|
// Load the Basic Pitch ONNX model for the polyphonic ML detector. When a
|
||||||
// model is loaded, getActiveDetection() / scoreChord() route through it;
|
// model is loaded, getActiveDetection() / scoreChord() route through it;
|
||||||
// otherwise they fall back to the YIN PitchDetector / ChordScorer.
|
// otherwise they fall back to the YIN PitchDetector / ChordScorer.
|
||||||
|
|||||||
@@ -141,6 +141,25 @@ struct MlNoteDetector::Impl
|
|||||||
// than an ML scorer that would return all-misses until the window fills.
|
// than an ML scorer that would return all-misses until the window fills.
|
||||||
std::atomic<bool> hasPublished{ false };
|
std::atomic<bool> hasPublished{ false };
|
||||||
|
|
||||||
|
// Master gate. The ML pipeline is the single most expensive thing in the
|
||||||
|
// engine (~30 ms ONNX inference every hop), and on the default desktop path
|
||||||
|
// note detection is scored by the harmonic-comb NoteVerifier — nothing reads
|
||||||
|
// the ML detector. So it defaults OFF and the renderer arms it (via
|
||||||
|
// setNoteDetectionEnabled) only when a consumer actually needs it
|
||||||
|
// (native-frame detection / non-verifier fallback). When false the audio
|
||||||
|
// thread stops feeding pushSamples and the inference thread runs no Run(),
|
||||||
|
// so a silent home tuner — or an in-song verifier session — costs nothing.
|
||||||
|
std::atomic<bool> enabled{ false };
|
||||||
|
|
||||||
|
// Requested by setEnabled(true): the INFERENCE THREAD clears the rolling
|
||||||
|
// window/FIFO/snapshot on its next wake before the first inference, giving a
|
||||||
|
// cold start on (re)arm instead of scoring against audio left over from a
|
||||||
|
// previous arm. The reset runs on the thread (never the N-API thread) so it
|
||||||
|
// can't race ingest(); pushSamples() no-ops while it's pending so the audio
|
||||||
|
// thread isn't writing the FIFO mid-reset. This preserves the invariant that
|
||||||
|
// clearAudioState() only runs when nothing else is touching the buffers.
|
||||||
|
std::atomic<bool> resetPending{ false };
|
||||||
|
|
||||||
// Start / stop the background inference thread. prepare() and stop() use
|
// Start / stop the background inference thread. prepare() and stop() use
|
||||||
// these so the thread is never alive while clearAudioState() mutates the
|
// these so the thread is never alive while clearAudioState() mutates the
|
||||||
// FIFO / inQueue / circular buffer — otherwise a device stop→start cycle
|
// FIFO / inQueue / circular buffer — otherwise a device stop→start cycle
|
||||||
@@ -149,6 +168,19 @@ struct MlNoteDetector::Impl
|
|||||||
{
|
{
|
||||||
thread = std::make_unique<MlInferenceThread>([this]()
|
thread = std::make_unique<MlInferenceThread>([this]()
|
||||||
{
|
{
|
||||||
|
// Cold-start reset requested by setEnabled(true) — runs here, on the
|
||||||
|
// thread that owns the buffers. Keep resetPending set THROUGH the
|
||||||
|
// reset (clear it only after) so pushSamples() stays gated off the
|
||||||
|
// FIFO for the whole drain, then release it.
|
||||||
|
if (resetPending.load(std::memory_order_acquire))
|
||||||
|
{
|
||||||
|
threadColdStart();
|
||||||
|
resetPending.store(false, std::memory_order_release);
|
||||||
|
}
|
||||||
|
// Master gate: when no consumer wants ML, run nothing — no ingest,
|
||||||
|
// no inference — so the thread sits idle instead of burning CPU.
|
||||||
|
if (! enabled.load(std::memory_order_acquire))
|
||||||
|
return;
|
||||||
ingest();
|
ingest();
|
||||||
runInferenceIfDue();
|
runInferenceIfDue();
|
||||||
});
|
});
|
||||||
@@ -169,7 +201,11 @@ struct MlNoteDetector::Impl
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
void clearAudioState()
|
// Reset everything EXCEPT the FIFO. These fields are touched only by the
|
||||||
|
// inference thread (resampler/inQueue/circ/window counters) or under
|
||||||
|
// snapshotLock (the published snapshot), so this is safe to call either with
|
||||||
|
// the thread stopped (prepare/stop) or on the thread itself (cold re-arm).
|
||||||
|
void clearBuffersExceptFifo()
|
||||||
{
|
{
|
||||||
resampler.reset();
|
resampler.reset();
|
||||||
inQueue.clear();
|
inQueue.clear();
|
||||||
@@ -177,7 +213,6 @@ struct MlNoteDetector::Impl
|
|||||||
circWrite = 0;
|
circWrite = 0;
|
||||||
totalResampled = 0;
|
totalResampled = 0;
|
||||||
sinceInference = 0;
|
sinceInference = 0;
|
||||||
fifo.reset();
|
|
||||||
onsetTimeMs.fill(0.0);
|
onsetTimeMs.fill(0.0);
|
||||||
onsetSeq.fill(0);
|
onsetSeq.fill(0);
|
||||||
onsetConf.fill(0.0f);
|
onsetConf.fill(0.0f);
|
||||||
@@ -197,6 +232,27 @@ struct MlNoteDetector::Impl
|
|||||||
hasPublished.store(false, std::memory_order_relaxed);
|
hasPublished.store(false, std::memory_order_relaxed);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Full reset including fifo.reset(). fifo.reset() mutates BOTH FIFO indices
|
||||||
|
// and is NOT safe against a concurrent producer, so this must only be called
|
||||||
|
// with the audio callback quiescent: prepare()/stop() call it after
|
||||||
|
// stopThread() joins the inference thread, and the device is not streaming.
|
||||||
|
void clearAudioState()
|
||||||
|
{
|
||||||
|
fifo.reset();
|
||||||
|
clearBuffersExceptFifo();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cold re-arm on the inference thread. Cannot fifo.reset() here — the audio
|
||||||
|
// thread may still be producing — so DRAIN the FIFO instead (advancing only
|
||||||
|
// the read index, which is the consumer's own; safe SPSC against a
|
||||||
|
// concurrent pushSamples). pushSamples() also gates on resetPending while
|
||||||
|
// this runs, so in practice the producer is quiescent anyway.
|
||||||
|
void threadColdStart()
|
||||||
|
{
|
||||||
|
fifo.finishedRead(fifo.getNumReady());
|
||||||
|
clearBuffersExceptFifo();
|
||||||
|
}
|
||||||
|
|
||||||
// Drain the FIFO, resample to 22050 Hz, append to the rolling window.
|
// Drain the FIFO, resample to 22050 Hz, append to the rolling window.
|
||||||
void ingest()
|
void ingest()
|
||||||
{
|
{
|
||||||
@@ -371,10 +427,13 @@ bool MlNoteDetector::isAvailable() const
|
|||||||
|
|
||||||
bool MlNoteDetector::isReady() const
|
bool MlNoteDetector::isReady() const
|
||||||
{
|
{
|
||||||
// Available AND has published at least one inference snapshot — the
|
// Available AND armed AND has published at least one inference snapshot.
|
||||||
// engine gates ML routing on this so the cold-start window after an
|
// The engine gates ML routing on this, so: the cold-start window after an
|
||||||
// audio start/restart uses the YIN/ChordScorer fallback.
|
// audio start/restart (or a re-arm) uses the YIN/ChordScorer fallback, and
|
||||||
|
// a suspended detector (enabled=false) routes to YIN rather than serving the
|
||||||
|
// stale snapshot left from its last arm.
|
||||||
return isAvailable()
|
return isAvailable()
|
||||||
|
&& impl->enabled.load(std::memory_order_acquire)
|
||||||
&& impl->hasPublished.load(std::memory_order_relaxed);
|
&& impl->hasPublished.load(std::memory_order_relaxed);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -452,9 +511,41 @@ void MlNoteDetector::stop()
|
|||||||
impl->clearAudioState();
|
impl->clearAudioState();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void MlNoteDetector::setEnabled(bool e)
|
||||||
|
{
|
||||||
|
if (impl->enabled.load(std::memory_order_relaxed) == e) return; // dedup; no spurious re-arm
|
||||||
|
if (e)
|
||||||
|
{
|
||||||
|
// Arming. Drop readiness SYNCHRONOUSLY, BEFORE exposing enabled=true, so
|
||||||
|
// no reader (isReady() → DetectNotes/getActiveDetection) can observe the
|
||||||
|
// previous arm's snapshot during the cold-start window. isReady() loads
|
||||||
|
// enabled (acquire) before hasPublished, so seeing enabled=true here
|
||||||
|
// guarantees seeing hasPublished=false. Also request the thread-side
|
||||||
|
// buffer cold start (drains the FIFO + clears the window on the thread).
|
||||||
|
impl->hasPublished.store(false, std::memory_order_release);
|
||||||
|
impl->resetPending.store(true, std::memory_order_release);
|
||||||
|
}
|
||||||
|
// Publish the new state last (release). Disarming needs no buffer work: the
|
||||||
|
// thread sees !enabled on its next wake and idles, and isReady() gates on
|
||||||
|
// enabled so a suspended detector serves the YIN fallback, not a stale snapshot.
|
||||||
|
impl->enabled.store(e, std::memory_order_release);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool MlNoteDetector::isEnabled() const
|
||||||
|
{
|
||||||
|
return impl->enabled.load(std::memory_order_relaxed);
|
||||||
|
}
|
||||||
|
|
||||||
void MlNoteDetector::pushSamples(const float* data, int numSamples)
|
void MlNoteDetector::pushSamples(const float* data, int numSamples)
|
||||||
{
|
{
|
||||||
if (numSamples <= 0) return;
|
if (numSamples <= 0) return;
|
||||||
|
// Master gate (audio thread, relaxed atomic loads — no lock, no allocation):
|
||||||
|
// don't feed the FIFO when no consumer wants ML, nor while a thread-side
|
||||||
|
// cold-start reset is pending (the reset clears the FIFO — keep the audio
|
||||||
|
// thread off it until that's done). The inference thread also early-returns,
|
||||||
|
// so the whole pipeline is dormant until armed.
|
||||||
|
if (! impl->enabled.load(std::memory_order_relaxed)
|
||||||
|
|| impl->resetPending.load(std::memory_order_relaxed)) return;
|
||||||
// Lock-free write; if the FIFO is full (inference stalled) the oldest
|
// Lock-free write; if the FIFO is full (inference stalled) the oldest
|
||||||
// unread samples are simply not overwritten — we drop the newest instead,
|
// unread samples are simply not overwritten — we drop the newest instead,
|
||||||
// which never blocks or allocates on the audio thread.
|
// which never blocks or allocates on the audio thread.
|
||||||
@@ -563,6 +654,8 @@ bool MlNoteDetector::isReady() const { return false; }
|
|||||||
bool MlNoteDetector::loadModel(const juce::File&) { return false; }
|
bool MlNoteDetector::loadModel(const juce::File&) { return false; }
|
||||||
void MlNoteDetector::prepare(double, int) {}
|
void MlNoteDetector::prepare(double, int) {}
|
||||||
void MlNoteDetector::stop() {}
|
void MlNoteDetector::stop() {}
|
||||||
|
void MlNoteDetector::setEnabled(bool) {}
|
||||||
|
bool MlNoteDetector::isEnabled() const { return false; }
|
||||||
void MlNoteDetector::pushSamples(const float*, int) {}
|
void MlNoteDetector::pushSamples(const float*, int) {}
|
||||||
std::vector<MlNoteDetector::ActiveNote> MlNoteDetector::getActiveNotes() const { return {}; }
|
std::vector<MlNoteDetector::ActiveNote> MlNoteDetector::getActiveNotes() const { return {}; }
|
||||||
MlNoteDetector::ActiveNote MlNoteDetector::getDominantNote() const { return {}; }
|
MlNoteDetector::ActiveNote MlNoteDetector::getDominantNote() const { return {}; }
|
||||||
|
|||||||
@@ -47,6 +47,18 @@ public:
|
|||||||
void prepare(double sampleRate, int blockSize);
|
void prepare(double sampleRate, int blockSize);
|
||||||
void stop();
|
void stop();
|
||||||
|
|
||||||
|
// Master gate. Defaults OFF: the ML pipeline only runs inference (and only
|
||||||
|
// accepts pushSamples) while enabled. The renderer arms it via the
|
||||||
|
// setNoteDetectionEnabled bridge when a consumer actually needs ML notes
|
||||||
|
// (native-frame detection / non-verifier fallback); the default desktop path
|
||||||
|
// scores with the harmonic-comb NoteVerifier and leaves this off, so a home
|
||||||
|
// tuner — or an in-song verifier session — pays nothing for ML. Disabling
|
||||||
|
// clears the rolling window + published snapshot (cold re-arm); the inference
|
||||||
|
// thread stays alive but idle, so toggling needs no thread restart. Safe to
|
||||||
|
// call from the N-API/main thread; the audio thread reads the flag lock-free.
|
||||||
|
void setEnabled(bool e);
|
||||||
|
bool isEnabled() const;
|
||||||
|
|
||||||
// Audio thread — lock-free, no allocation.
|
// Audio thread — lock-free, no allocation.
|
||||||
void pushSamples(const float* data, int numSamples);
|
void pushSamples(const float* data, int numSamples);
|
||||||
|
|
||||||
|
|||||||
@@ -616,6 +616,20 @@ static Napi::Value SetMonitorMute(const Napi::CallbackInfo& info)
|
|||||||
return info.Env().Undefined();
|
return info.Env().Undefined();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// setNoteDetectionEnabled(bool) -> undefined. Arms/suspends the polyphonic ML
|
||||||
|
// note-detection pipeline across all sources. The renderer (note_detect) calls
|
||||||
|
// this true only while a consumer actually reads ML notes (native-frame
|
||||||
|
// detection / non-verifier fallback) and false otherwise — the default
|
||||||
|
// harmonic-comb verifier path and the always-on home tuner leave ML suspended,
|
||||||
|
// so the engine runs no ONNX inference when nothing needs it.
|
||||||
|
static Napi::Value SetNoteDetectionEnabled(const Napi::CallbackInfo& info)
|
||||||
|
{
|
||||||
|
auto liveEngine = snapshotEngine();
|
||||||
|
if (liveEngine && info.Length() > 0)
|
||||||
|
liveEngine->setMlNoteDetectionEnabled(info[0].As<Napi::Boolean>().Value());
|
||||||
|
return info.Env().Undefined();
|
||||||
|
}
|
||||||
|
|
||||||
static Napi::Value SetMonitorMuteSuppressed(const Napi::CallbackInfo& info)
|
static Napi::Value SetMonitorMuteSuppressed(const Napi::CallbackInfo& info)
|
||||||
{
|
{
|
||||||
// IsBoolean()-guarded so a mismatched renderer build / manual caller
|
// IsBoolean()-guarded so a mismatched renderer build / manual caller
|
||||||
@@ -3259,6 +3273,7 @@ static Napi::Object InitModule(Napi::Env env, Napi::Object exports)
|
|||||||
exports.Set("getSampleRate", Napi::Function::New(env, GetSampleRate));
|
exports.Set("getSampleRate", Napi::Function::New(env, GetSampleRate));
|
||||||
exports.Set("loadNoteModel", Napi::Function::New(env, LoadNoteModel));
|
exports.Set("loadNoteModel", Napi::Function::New(env, LoadNoteModel));
|
||||||
exports.Set("isMlNoteDetection", Napi::Function::New(env, IsMlNoteDetection));
|
exports.Set("isMlNoteDetection", Napi::Function::New(env, IsMlNoteDetection));
|
||||||
|
exports.Set("setNoteDetectionEnabled", Napi::Function::New(env, SetNoteDetectionEnabled));
|
||||||
exports.Set("detectNotes", Napi::Function::New(env, DetectNotes));
|
exports.Set("detectNotes", Napi::Function::New(env, DetectNotes));
|
||||||
|
|
||||||
// VST scanning
|
// VST scanning
|
||||||
|
|||||||
@@ -655,6 +655,21 @@ export function initAudioBridge(): void {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Arm/suspend the ML note-detection pipeline. note_detect calls this true
|
||||||
|
// only while a consumer actually reads ML notes (native-frame detection /
|
||||||
|
// non-verifier fallback) and false otherwise, so the default harmonic-comb
|
||||||
|
// verifier path and the always-on home tuner cost no ONNX inference.
|
||||||
|
// typeof-guarded so a downlevel addon (no gate) simply ignores it — ML then
|
||||||
|
// runs as before, i.e. fail-safe to current behaviour.
|
||||||
|
ipcMain.handle('audio:setNoteDetectionEnabled', (_event, enabled: boolean) => {
|
||||||
|
if (!audio || typeof audio.setNoteDetectionEnabled !== 'function') return;
|
||||||
|
try {
|
||||||
|
audio.setNoteDetectionEnabled(Boolean(enabled));
|
||||||
|
} catch (e) {
|
||||||
|
console.warn(`[audio] setNoteDetectionEnabled failed: ${e instanceof Error ? e.message : String(e)}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
// ── Chord Scoring (polyphonic) ─────────────────────────────────────────
|
// ── Chord Scoring (polyphonic) ─────────────────────────────────────────
|
||||||
// The notedetect plugin's chord-scoring branch hands us a chord
|
// The notedetect plugin's chord-scoring branch hands us a chord
|
||||||
// context — notes, arrangement, tuning offsets, thresholds — and
|
// context — notes, arrangement, tuning offsets, thresholds — and
|
||||||
|
|||||||
@@ -280,6 +280,13 @@ const feedBackDesktopApi = {
|
|||||||
// fallback. Resolves false on a downlevel addon.
|
// fallback. Resolves false on a downlevel addon.
|
||||||
isMlNoteDetection: (): Promise<boolean> => ipcRenderer.invoke('audio:isMlNoteDetection'),
|
isMlNoteDetection: (): Promise<boolean> => ipcRenderer.invoke('audio:isMlNoteDetection'),
|
||||||
|
|
||||||
|
// Arm/suspend the polyphonic ML note-detection pipeline. note_detect
|
||||||
|
// calls this true only when it will actually read ML notes; the default
|
||||||
|
// harmonic-comb verifier path leaves it false so a home tuner / verifier
|
||||||
|
// session runs no ONNX inference. No-op on a downlevel addon.
|
||||||
|
setNoteDetectionEnabled: (enabled: boolean): Promise<void> =>
|
||||||
|
ipcRenderer.invoke('audio:setNoteDetectionEnabled', enabled),
|
||||||
|
|
||||||
// Current engine sample rate — needed by notedetect's chord
|
// Current engine sample rate — needed by notedetect's chord
|
||||||
// scorer to map FFT bins to Hz on the bridge path (no
|
// scorer to map FFT bins to Hz on the bridge path (no
|
||||||
// AudioContext to read it from). Queried once at startAudio.
|
// AudioContext to read it from). Queried once at startAudio.
|
||||||
|
|||||||
+68
-8
@@ -292,16 +292,50 @@ window.__feedBackDesktopAudioHooks = window.__feedBackDesktopAudioHooks || {};
|
|||||||
|
|
||||||
async function audioInputOpenHandler(request) {
|
async function audioInputOpenHandler(request) {
|
||||||
const source = (request && request.logicalSourceKey) ? String(request.logicalSourceKey) : '';
|
const source = (request && request.logicalSourceKey) ? String(request.logicalSourceKey) : '';
|
||||||
const match = /^desktop-audio:([^:]+):input:(\d+)$/.exec(source);
|
// A named device carries a STABLE name-encoded key so the selection
|
||||||
const inputType = match ? match[1] : safeKeyPart(deviceTypeSelect?.value || 'default');
|
// survives device reorder/hotplug (virtual/aggregate devices like
|
||||||
const inputIndex = match ? Number(match[2]) : -1;
|
// BlackHole reorder on enumeration). A legacy `:input:<N>` key — emitted
|
||||||
|
// by older builds and any selection persisted before this change — still
|
||||||
|
// resolves positionally for one more open.
|
||||||
|
const nameMatch = /^desktop-audio:([^:]+):input:name:(.+)$/.exec(source);
|
||||||
|
const indexMatch = nameMatch ? null : /^desktop-audio:([^:]+):input:(\d+)$/.exec(source);
|
||||||
|
const inputType = nameMatch ? nameMatch[1]
|
||||||
|
: indexMatch ? indexMatch[1]
|
||||||
|
: safeKeyPart(deviceTypeSelect?.value || 'default');
|
||||||
const typeInfo = currentDeviceTypes.find(t => safeKeyPart(t && t.name) === inputType)
|
const typeInfo = currentDeviceTypes.find(t => safeKeyPart(t && t.name) === inputType)
|
||||||
|| currentDeviceTypes.find(t => t && t.name === deviceTypeSelect?.value)
|
|| currentDeviceTypes.find(t => t && t.name === deviceTypeSelect?.value)
|
||||||
|| currentDeviceTypes[0]
|
|| currentDeviceTypes[0]
|
||||||
|| null;
|
|| null;
|
||||||
const inputDevice = inputIndex >= 0 && typeInfo && Array.isArray(typeInfo.inputs)
|
const typeInputs = typeInfo && Array.isArray(typeInfo.inputs) ? typeInfo.inputs : [];
|
||||||
? (typeInfo.inputs[inputIndex] || '')
|
|
||||||
: (inputDeviceSelect?.value || '');
|
// Resolve the picked device to a concrete name and FAIL LOUD when it's
|
||||||
|
// gone. The old code fell through to '' (or whatever the Settings
|
||||||
|
// dropdown showed), which makes the native engine open its DEFAULT device
|
||||||
|
// — the internal mic. That silent substitution is the macOS wrong-mic bug
|
||||||
|
// this handler exists to kill: the user picks BlackHole, setup "succeeds",
|
||||||
|
// and every score is garbage with no signal anything went wrong.
|
||||||
|
let inputDevice = '';
|
||||||
|
if (nameMatch) {
|
||||||
|
let decoded = '';
|
||||||
|
try { decoded = decodeURIComponent(nameMatch[2]); } catch (_) { decoded = ''; }
|
||||||
|
// The key encodes the trimmed name; match trim-tolerantly but bind the
|
||||||
|
// engine's exact enumerated string.
|
||||||
|
inputDevice = typeInputs.find(n => String(n).trim() === decoded) || '';
|
||||||
|
if (!inputDevice) {
|
||||||
|
return { outcome: 'failed', status: 'failed', reason: `Selected input device "${decoded || '(unknown)'}" is no longer available` };
|
||||||
|
}
|
||||||
|
} else if (indexMatch) {
|
||||||
|
const idx = Number(indexMatch[2]);
|
||||||
|
inputDevice = (idx >= 0 && idx < typeInputs.length) ? typeInputs[idx] : '';
|
||||||
|
if (!inputDevice) {
|
||||||
|
return { outcome: 'failed', status: 'failed', reason: 'Selected input device is no longer available' };
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// No recognizable source key — refuse rather than silently grabbing
|
||||||
|
// whatever the Settings dropdown happens to show (another default path).
|
||||||
|
return { outcome: 'failed', status: 'failed', reason: 'No input device selected' };
|
||||||
|
}
|
||||||
|
|
||||||
const snapshot = currentAudioDeviceSnapshot();
|
const snapshot = currentAudioDeviceSnapshot();
|
||||||
const result = await api.setDevice({
|
const result = await api.setDevice({
|
||||||
inputType: typeInfo && typeInfo.name ? typeInfo.name : snapshot.inputType,
|
inputType: typeInfo && typeInfo.name ? typeInfo.name : snapshot.inputType,
|
||||||
@@ -314,7 +348,23 @@ window.__feedBackDesktopAudioHooks = window.__feedBackDesktopAudioHooks || {};
|
|||||||
const ok = typeof result === 'boolean' ? result : !!result?.ok;
|
const ok = typeof result === 'boolean' ? result : !!result?.ok;
|
||||||
if (!ok) return { outcome: 'failed', status: 'failed', reason: result && result.error ? String(result.error) : 'Native audio device open failed' };
|
if (!ok) return { outcome: 'failed', status: 'failed', reason: result && result.error ? String(result.error) : 'Native audio device open failed' };
|
||||||
if (typeof api.startAudio === 'function') await api.startAudio();
|
if (typeof api.startAudio === 'function') await api.startAudio();
|
||||||
return { outcome: 'handled', status: 'open' };
|
|
||||||
|
// Read back what the engine ACTUALLY bound, so callers (input_setup's
|
||||||
|
// confirmation gate, note_detect) can surface "Now listening to: <device>"
|
||||||
|
// and spot a silent mismatch instead of trusting the request blind.
|
||||||
|
let boundType = (typeInfo && typeInfo.name) ? String(typeInfo.name) : '';
|
||||||
|
let boundName = inputDevice;
|
||||||
|
try {
|
||||||
|
if (typeof api.getCurrentDevice === 'function') {
|
||||||
|
const cur = await api.getCurrentDevice();
|
||||||
|
if (cur && typeof cur === 'object') {
|
||||||
|
if (cur.inputType) boundType = String(cur.inputType);
|
||||||
|
if (cur.input) boundName = String(cur.input);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (_) { /* fall back to the requested identity */ }
|
||||||
|
|
||||||
|
return { outcome: 'handled', status: 'open', payload: { boundType, boundName, requestedName: inputDevice } };
|
||||||
}
|
}
|
||||||
|
|
||||||
async function audioInputCloseHandler() {
|
async function audioInputCloseHandler() {
|
||||||
@@ -342,11 +392,21 @@ window.__feedBackDesktopAudioHooks = window.__feedBackDesktopAudioHooks || {};
|
|||||||
const driverSuffix = (showDriverType && typeName) ? ` (${typeName})` : '';
|
const driverSuffix = (showDriverType && typeName) ? ` (${typeName})` : '';
|
||||||
const inputs = Array.isArray(typeInfo && typeInfo.inputs) ? typeInfo.inputs : [];
|
const inputs = Array.isArray(typeInfo && typeInfo.inputs) ? typeInfo.inputs : [];
|
||||||
inputs.forEach((deviceName, index) => {
|
inputs.forEach((deviceName, index) => {
|
||||||
const logicalSourceKey = `desktop-audio:${safeKeyPart(typeName)}:input:${index}`;
|
|
||||||
const hasRealName = (typeof deviceName === 'string' && !!deviceName.trim());
|
const hasRealName = (typeof deviceName === 'string' && !!deviceName.trim());
|
||||||
const realName = hasRealName
|
const realName = hasRealName
|
||||||
? deviceName.trim()
|
? deviceName.trim()
|
||||||
: `Desktop input ${index + 1}`;
|
: `Desktop input ${index + 1}`;
|
||||||
|
// Identity in the key: a named device gets a STABLE name-encoded
|
||||||
|
// key so selection survives device reorder/hotplug (encodeURIComponent
|
||||||
|
// escapes ':' to %3A so it can't break the handler's parser). Only a
|
||||||
|
// truly nameless input falls back to the positional index, where the
|
||||||
|
// index is the only identity available. NOTE: selections persisted by
|
||||||
|
// an older build use the legacy index key; the open handler still
|
||||||
|
// resolves those positionally, but the picker now lists this device
|
||||||
|
// under its name key, so a returning user re-picks once.
|
||||||
|
const logicalSourceKey = hasRealName
|
||||||
|
? `desktop-audio:${safeKeyPart(typeName)}:input:name:${encodeURIComponent(realName)}`
|
||||||
|
: `desktop-audio:${safeKeyPart(typeName)}:input:${index}`;
|
||||||
audioSession.registerInputSource({
|
audioSession.registerInputSource({
|
||||||
sourceId: `audio_engine:${logicalSourceKey}`,
|
sourceId: `audio_engine:${logicalSourceKey}`,
|
||||||
logicalSourceKey,
|
logicalSourceKey,
|
||||||
|
|||||||
@@ -93,6 +93,33 @@ int main(int argc, char** argv)
|
|||||||
|
|
||||||
det.prepare((double) sampleRate, 256);
|
det.prepare((double) sampleRate, 256);
|
||||||
|
|
||||||
|
// Gate check: the pipeline defaults OFF. Feed the chord region (~1 s of
|
||||||
|
// audio that detects cleanly once armed) while still DISABLED and confirm
|
||||||
|
// the detector publishes nothing and never becomes ready — pushSamples must
|
||||||
|
// no-op and the inference thread must run no Run().
|
||||||
|
{
|
||||||
|
const int gateBlock = 256;
|
||||||
|
const size_t gateStart = (size_t) (3.6 * sampleRate);
|
||||||
|
const size_t gateEnd = std::min(wav.size(), (size_t) (4.8 * sampleRate));
|
||||||
|
for (size_t i = gateStart; i < gateEnd; i += gateBlock)
|
||||||
|
{
|
||||||
|
const int n = (int) std::min<size_t>(gateBlock, gateEnd - i);
|
||||||
|
det.pushSamples(wav.data() + i, n);
|
||||||
|
std::this_thread::sleep_for(std::chrono::microseconds(
|
||||||
|
(long long) (1e6 * n / sampleRate)));
|
||||||
|
if (det.isReady() || ! det.getActiveNotes().empty())
|
||||||
|
{
|
||||||
|
std::cerr << "FAIL: detector active while disabled (gate leak)\n";
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
std::cout << "gate OK: no detection while disabled\n";
|
||||||
|
}
|
||||||
|
|
||||||
|
// Arm it so pushSamples feeds and the inference thread runs. setEnabled(true)
|
||||||
|
// requests a thread-side cold-start reset before the first inference.
|
||||||
|
det.setEnabled(true);
|
||||||
|
|
||||||
// Feed the WAV in 256-sample blocks at ~real time so the background
|
// Feed the WAV in 256-sample blocks at ~real time so the background
|
||||||
// inference thread drains the FIFO instead of overflowing it. The
|
// inference thread drains the FIFO instead of overflowing it. The
|
||||||
// detector reports "what is sounding now", so we poll the active set
|
// detector reports "what is sounding now", so we poll the active set
|
||||||
|
|||||||
Reference in New Issue
Block a user