got-feedBack/feedBack-desktop
1
0
Fork 0
mirror of https://github.com/got-feedBack/feedBack-desktop.git synced 2026-08-14 12:47:10 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Clean release snapshot

Browse Source
This commit is contained in:
Byron Gamatos
2026-06-16 18:48:12 +02:00
commit bd603184d5
291 changed files with 47318 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/spike/CMakeLists.txt
+28
View File
@@ -0,0 +1,28 @@
# Phase 0 de-risk spike — standalone, NOT part of the addon build.
#
# Build:
# cmake -B build -DONNXRUNTIME_ROOT=/path/to/onnxruntime-linux-x64-1.20.1
# cmake --build build
#
# ONNXRUNTIME_ROOT must point at an extracted prebuilt ONNX Runtime release
# (the dir containing include/ and lib/).
cmake_minimum_required(VERSION 3.22)
project(bp_spike CXX)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
if(NOT DEFINED ONNXRUNTIME_ROOT)
message(FATAL_ERROR "Set -DONNXRUNTIME_ROOT=/path/to/onnxruntime-<os>-<arch>-<ver>")
endif()
add_executable(spike main.cpp)
target_include_directories(spike PRIVATE "${ONNXRUNTIME_ROOT}/include")
target_link_directories(spike PRIVATE "${ONNXRUNTIME_ROOT}/lib")
target_link_libraries(spike PRIVATE onnxruntime)
# Load libonnxruntime.so from next to the binary or from ONNXRUNTIME_ROOT/lib.
set_target_properties(spike PROPERTIES
BUILD_RPATH "${ONNXRUNTIME_ROOT}/lib"
INSTALL_RPATH "$ORIGIN")
tests/spike/README.md
+63
View File
@@ -0,0 +1,63 @@
# Phase 0 de-risk spike — polyphonic ML note detection
**Throwaway.** Not part of the addon build. Proves Spotify Basic Pitch runs
under ONNX Runtime's C++ API before the real engine integration begins.
## Provenance
- **Model:** `nmp.onnx` from the `basic-pitch` PyPI package, v0.4.0
(`basic_pitch/saved_models/icassp_2022/nmp.onnx`, 230 KB).
Spotify Basic Pitch, **Apache-2.0**. The package ships a clean ONNX export —
no `tf2onnx` conversion needed.
- **ONNX Runtime:** v1.20.1, official prebuilt CPU release.
- Linux x64: `onnxruntime-linux-x64-1.20.1.tgz`
SHA-256 `67db4dc1561f1e3fd42e619575c82c601ef89849afc7ea85a003abbac1a1a105`
- URL pattern: `https://github.com/microsoft/onnxruntime/releases/download/v1.20.1/onnxruntime-<os-arch>-1.20.1.<ext>`
## Build & run
```sh
cmake -B build -DONNXRUNTIME_ROOT=/path/to/onnxruntime-linux-x64-1.20.1
cmake --build build
./build/spike /path/to/nmp.onnx test_guitar.wav
```
`test_guitar.wav` is a 48 kHz synthetic Karplus-Strong guitar clip: single
notes A2 / D3 / G3 at t≈0.3/1.3/2.3 s, then a C-major triad (C3+E3+G3) at
t≈3.3 s.
## Model I/O contract (verified)
- **Input** `serving_default_input_2:0` — `[batch, 43844, 1]` float32.
43844 = 22050·2 − 256, a ~2 s mono window at **22050 Hz**.
- **Outputs** (3 posteriorgrams, ~86 frames/s, 172 frames/window):
- `StatefulPartitionedCall:1` — **note/frame** `[batch, 172, 88]`
- `StatefulPartitionedCall:2` — **onset** `[batch, 172, 88]`
- `StatefulPartitionedCall:0` — **contour** `[batch, 172, 264]` (unused)
- 88 pitches = MIDI 21..108 (pitch index `p` → MIDI `21 + p`).
## Post-processing (minimal slice ported to C++)
A note onset = a rising edge of the onset posteriorgram past 0.5, gated by the
frame posteriorgram past 0.3:
`onset[f,p] ≥ 0.5 && onset[f-1,p] < 0.5 && note[f,p] ≥ 0.3`.
This is all the live hit/miss path needs — no full offline note-event
reconstruction.
## Findings
- **Accuracy:** all 4 events detected at the correct MIDI and time; the C-major
triad resolved polyphonically (C3+E3+G3). Zero false positives in the C++ run.
- **Latency:** 33 ms/window inference, single-threaded
(`IntraOpNumThreads=1`), ONNX Runtime 1.20.1, CPU EP. With a 64 ms hop,
end-to-end detection latency (hop + inference + model onset lag) lands
≈100–150 ms — the ≤150 ms target is reachable; a 128 ms hop trades latency
(~180–200 ms) for lower CPU.
- **Window boundaries:** non-overlapping ~2 s windows can re-onset a sustained
note at a window edge. The production `MlNoteDetector` avoids this with a
rolling 22050 Hz buffer, reading only the freshest frames each hop.
- **Resampling:** the spike uses Catmull-Rom cubic interpolation for
48000→22050; the production detector will use `juce::LagrangeInterpolator`.
**Conclusion: de-risked. Model + ONNX Runtime C++ work; output is
interpretable. Proceed to Phase 1.**
tests/spike/main.cpp
+252
View File
@@ -0,0 +1,252 @@
// Phase 0 de-risk spike — TST-style polyphonic ML note detection.
//
// THROWAWAY. Not wired into the addon build. Proves that Spotify Basic Pitch
// (nmp.onnx) loads and runs under ONNX Runtime's C++ API, that the I/O contract
// matches expectations, and that a minimal onset/frame post-processing yields
// interpretable MIDI notes. See README.md for provenance.
//
// Build: see CMakeLists.txt in this directory.
// Run: ./spike <model.onnx> <audio.wav>
#include <onnxruntime_cxx_api.h>
#include <algorithm>
#include <chrono>
#include <cmath>
#include <cstdint>
#include <cstdio>
#include <cstring>
#include <fstream>
#include <iostream>
#include <string>
#include <vector>
// --- Basic Pitch constants (basic_pitch/constants.py) ---------------------
static constexpr int kModelSampleRate = 22050;
static constexpr int kFftHop = 256;
static constexpr int kAudioNSamples = 22050 * 2 - 256; // 43844, ~2 s window
static constexpr int kFramesPerSecond = 22050 / 256; // 86
static constexpr int kNumPitches = 88; // MIDI 21..108
static constexpr int kLowestMidi = 21; // A0, base freq 27.5 Hz
static constexpr float kOnsetThreshold = 0.5f;
static constexpr float kFrameThreshold = 0.3f;
// nmp.onnx tensor names (verified via the Python ONNX Runtime in Phase 0).
static const char* kInputName = "serving_default_input_2:0";
static const char* kNoteOutput = "StatefulPartitionedCall:1"; // frame/note posteriorgram
static const char* kOnsetOutput= "StatefulPartitionedCall:2"; // onset posteriorgram
// --------------------------------------------------------------------------
// Minimal WAV reader: 16-bit PCM or 32-bit float, any channel count -> mono.
// --------------------------------------------------------------------------
struct Wav { std::vector<float> samples; int sampleRate = 0; };
static uint32_t rdU32(const uint8_t* p) { return p[0] | (p[1]<<8) | (p[2]<<16) | (uint32_t(p[3])<<24); }
static uint16_t rdU16(const uint8_t* p) { return uint16_t(p[0] | (p[1]<<8)); }
static bool readWav(const std::string& path, Wav& out)
{
std::ifstream f(path, std::ios::binary);
if (!f) { std::cerr << "cannot open " << path << "\n"; return false; }
std::vector<uint8_t> buf((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
if (buf.size() < 44 || std::memcmp(buf.data(), "RIFF", 4) || std::memcmp(buf.data()+8, "WAVE", 4))
{ std::cerr << "not a RIFF/WAVE file\n"; return false; }
uint16_t fmt = 0, channels = 0, bits = 0;
uint32_t rate = 0;
const uint8_t* data = nullptr;
uint32_t dataLen = 0;
size_t pos = 12;
while (pos + 8 <= buf.size())
{
const char* id = reinterpret_cast<const char*>(buf.data() + pos);
uint32_t sz = rdU32(buf.data() + pos + 4);
const uint8_t* body = buf.data() + pos + 8;
// Guard the fmt-body reads (up to body+14, i.e. 16 bytes) against a
// truncated file: a declared sz >= 16 doesn't mean 16 bytes are
// actually present.
if (!std::memcmp(id, "fmt ", 4) && sz >= 16 && pos + 8 + 16 <= buf.size())
{
fmt = rdU16(body); channels = rdU16(body+2); rate = rdU32(body+4); bits = rdU16(body+14);
}
else if (!std::memcmp(id, "data", 4))
{
data = body;
dataLen = std::min<uint32_t>(sz, uint32_t(buf.size() - (pos + 8)));
}
pos += 8 + sz + (sz & 1); // chunks are word-aligned
}
// Guard the header fields before the rate/size math below: a malformed
// header (rate 0, bits 0) would otherwise divide by zero.
if (!data || channels == 0 || rate == 0 || bits < 8)
{ std::cerr << "invalid or missing fmt/data chunk\n"; return false; }
out.sampleRate = int(rate);
const int bytesPerSample = bits / 8;
const size_t frames = dataLen / (bytesPerSample * channels);
out.samples.resize(frames);
for (size_t i = 0; i < frames; ++i)
{
double acc = 0.0;
for (int c = 0; c < channels; ++c)
{
const uint8_t* s = data + (i * channels + c) * bytesPerSample;
if (fmt == 3 && bits == 32) { float v; std::memcpy(&v, s, 4); acc += v; }
else if (fmt == 1 && bits == 16) { acc += int16_t(rdU16(s)) / 32768.0; }
else if (fmt == 1 && bits == 32) { acc += int32_t(rdU32(s)) / 2147483648.0; }
else { std::cerr << "unsupported fmt=" << fmt << " bits=" << bits << "\n"; return false; }
}
out.samples[i] = float(acc / channels);
}
return true;
}
// --------------------------------------------------------------------------
// Resample to 22050 Hz. Catmull-Rom cubic interpolation — adequate for the
// spike; the real MlNoteDetector will use juce::LagrangeInterpolator.
// --------------------------------------------------------------------------
static std::vector<float> resampleTo22050(const std::vector<float>& in, int srcRate)
{
if (srcRate == kModelSampleRate) return in;
const double ratio = double(srcRate) / kModelSampleRate;
const size_t outLen = size_t(in.size() / ratio);
std::vector<float> out(outLen);
auto at = [&](long i) -> float {
if (i < 0) i = 0;
if (i >= long(in.size())) i = long(in.size()) - 1;
return in[size_t(i)];
};
for (size_t n = 0; n < outLen; ++n)
{
const double srcPos = n * ratio;
const long i = long(srcPos);
const float t = float(srcPos - i);
const float p0 = at(i-1), p1 = at(i), p2 = at(i+1), p3 = at(i+2);
out[n] = p1 + 0.5f * t * ((p2 - p0) + t * ((2*p0 - 5*p1 + 4*p2 - p3)
+ t * (3*(p1 - p2) + p3 - p0)));
}
return out;
}
static const char* noteName(int midi)
{
static const char* n[12] = {"C","C#","D","D#","E","F","F#","G","G#","A","A#","B"};
static char buf[8];
std::snprintf(buf, sizeof(buf), "%s%d", n[midi % 12], midi / 12 - 1);
return buf;
}
int main(int argc, char** argv)
{
if (argc < 3) { std::cerr << "usage: spike <model.onnx> <audio.wav>\n"; return 2; }
const std::string modelPath = argv[1];
const std::string wavPath = argv[2];
Wav wav;
if (!readWav(wavPath, wav)) return 1;
std::cout << "WAV: " << wav.samples.size() << " samples @ " << wav.sampleRate << " Hz\n";
const std::vector<float> mono = resampleTo22050(wav.samples, wav.sampleRate);
std::cout << "resampled: " << mono.size() << " samples @ " << kModelSampleRate << " Hz\n";
// --- ONNX Runtime session ---
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "bp-spike");
Ort::SessionOptions opts;
opts.SetIntraOpNumThreads(1);
opts.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);
Ort::Session session(env, modelPath.c_str(), opts);
{
Ort::AllocatorWithDefaultOptions alloc;
std::cout << "model inputs:\n";
for (size_t i = 0; i < session.GetInputCount(); ++i)
{
auto name = session.GetInputNameAllocated(i, alloc);
auto shp = session.GetInputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape();
std::cout << " " << name.get() << " [";
for (auto d : shp) std::cout << d << " ";
std::cout << "]\n";
}
std::cout << "model outputs:\n";
for (size_t i = 0; i < session.GetOutputCount(); ++i)
{
auto name = session.GetOutputNameAllocated(i, alloc);
auto shp = session.GetOutputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape();
std::cout << " " << name.get() << " [";
for (auto d : shp) std::cout << d << " ";
std::cout << "]\n";
}
}
auto memInfo = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
const char* inNames[] = { kInputName };
const char* outNames[] = { kNoteOutput, kOnsetOutput };
// Process non-overlapping ~2 s windows. (The production detector uses a
// rolling buffer reading only fresh frames; non-overlapping is fine here
// and is the source of the boundary re-onsets noted in README.md.)
struct Hit { double timeSec; int midi; float conf; };
std::vector<Hit> hits;
int windows = 0;
double totalInferMs = 0.0;
for (size_t base = 0; base < mono.size(); base += kAudioNSamples)
{
std::vector<float> window(kAudioNSamples, 0.0f);
const size_t n = std::min<size_t>(kAudioNSamples, mono.size() - base);
std::memcpy(window.data(), mono.data() + base, n * sizeof(float));
const int64_t inShape[3] = { 1, kAudioNSamples, 1 };
Ort::Value inTensor = Ort::Value::CreateTensor<float>(
memInfo, window.data(), window.size(), inShape, 3);
const auto t0 = std::chrono::steady_clock::now();
auto out = session.Run(Ort::RunOptions{nullptr}, inNames, &inTensor, 1, outNames, 2);
const auto t1 = std::chrono::steady_clock::now();
totalInferMs += std::chrono::duration<double, std::milli>(t1 - t0).count();
const float* note = out[0].GetTensorData<float>();
const float* onset = out[1].GetTensorData<float>();
// Validate the output shapes before indexing: both heads must be
// rank-3 [1, frames, kNumPitches] and agree on the frame count, or
// the flat indexing below would read out of bounds.
const auto noteShape = out[0].GetTensorTypeAndShapeInfo().GetShape();
const auto onsetShape = out[1].GetTensorTypeAndShapeInfo().GetShape();
if (noteShape.size() != 3 || onsetShape.size() != 3
|| noteShape[2] != kNumPitches || onsetShape[2] != kNumPitches
|| noteShape[1] != onsetShape[1])
{
std::cerr << "FAIL: unexpected model output shape\n";
return 1;
}
const int frames = int(noteShape[1]); // 172
const double windowStartSec = double(base) / kModelSampleRate;
for (int p = 0; p < kNumPitches; ++p)
for (int f = 1; f < frames; ++f)
{
const float on = onset[f * kNumPitches + p];
const float onPrev = onset[(f-1) * kNumPitches + p];
const float fr = note[f * kNumPitches + p];
if (on >= kOnsetThreshold && onPrev < kOnsetThreshold && fr >= kFrameThreshold)
hits.push_back({ windowStartSec + double(f) / kFramesPerSecond,
kLowestMidi + p, on });
}
++windows;
}
std::sort(hits.begin(), hits.end(),
[](const Hit& a, const Hit& b){ return a.timeSec < b.timeSec; });
std::cout << "\n" << windows << " windows, total inference "
<< totalInferMs << " ms, " << (totalInferMs / std::max(1, windows))
<< " ms/window\n";
std::cout << "\nDETECTED onsets (time_s, midi, note, onset_conf):\n";
for (const auto& h : hits)
std::cout << " t=" << h.timeSec << " midi=" << h.midi
<< " " << noteName(h.midi) << " conf=" << h.conf << "\n";
std::cout << "\nspike OK\n";
return 0;
}
tests/spike/test_guitar.wav
BIN
View File
Binary file not shown.