diff --git a/README.md b/README.md index f6dd90b..e344cf5 100644 --- a/README.md +++ b/README.md @@ -162,8 +162,9 @@ hardware that earns the claim. Next audio cuts: -1. Feed ASIO callbacks directly into `Mimir.Runtime` buffers. -2. Use ASIO loopback as the canonical program/timing reference. +1. Use ASIO loopback as the canonical program/timing reference. +2. Learn per output/mic chirp-bin response models and adapt the codebook to the + bands that survive the actual room. 3. Align all mic streams with fractional-delay and sample-rate-offset state. 4. Move the hot actuator into Faust/native DSP. 5. Emit separately controllable OBS stems plus a spatial bed from the same @@ -185,8 +186,11 @@ showed that generic dense chirplet matching is the wrong hot path for our controlled beacon. The current hybrid watermark uses `MimirChirpBinTimeline`: a fixed-slope chirp-bin codebook decoded by dechirp plus a small Goertzel/bin bank, then code-valid de Bruijn triplets become canonical timeline anchors. -Reports expose fractional delay and `delayUs`; the synthetic proof currently -recovers a `317.375` sample offset with about `0.369 us` error. +Reports expose fractional delay and `delayUs`; the synthetic 192 kHz proof now +recovers a delayed stream below printed microsecond precision. Physical +calibration persists usable bands, expected/observed bin confusion, phase, +group-delay, delay hypotheses, and an adaptive codebook plan per output/mic +path. For the long-form reasoning and source trail, read: @@ -288,7 +292,6 @@ Run the old OBS bridge only when you need the compatibility path: Audio-side help is especially useful around: -- ASIO callback capture into `Mimir.Runtime` without stdout/process payloads. - Fractional-delay lines and variable-rate resampling. - Sample-rate-offset estimation over a rolling window. - GCC-PHAT/passive program-audio timing under music/game audio. diff --git a/calibration/chirp-bin/scarlett-artifact.json b/calibration/chirp-bin/scarlett-artifact.json index 35482e1..18e3c75 100644 --- a/calibration/chirp-bin/scarlett-artifact.json +++ b/calibration/chirp-bin/scarlett-artifact.json @@ -1,6 +1,6 @@ { - "ModelId": "chirp-bin-20260523T192955Z", - "CreatedAt": "2026-05-23T19:29:55.6287406+00:00", + "ModelId": "chirp-bin-20260523T203522Z", + "CreatedAt": "2026-05-23T20:35:22.527425+00:00", "SampleRate": 192000, "ReferenceSourceId": "asio-ch2", "EmissionPlan": { @@ -22,6 +22,7 @@ }, "Paths": [ { + "OutputSourceId": "main-speakers", "SourceId": "asio-ch0", "SampleRate": 192000, "Profile": { @@ -319,6 +320,7 @@ "DelayHypotheses": [] }, { + "OutputSourceId": "main-speakers", "SourceId": "asio-ch1", "SampleRate": 192000, "Profile": { @@ -618,6 +620,7 @@ { "SymbolId": 1, "Reliability": 0.02338947289373043, + "PhaseCoherence": 0.8813023361164132, "ExpectedCenterHz": 4075, "ObservedCenterHz": 4468.75, "MeanEnergy": 0.9902022429801735, @@ -629,6 +632,7 @@ { "SymbolId": 2, "Reliability": 0.16959717331712612, + "PhaseCoherence": 0.7788915227933655, "ExpectedCenterHz": 4525, "ObservedCenterHz": 4475, "MeanEnergy": 0.9196472718940074, @@ -640,6 +644,7 @@ { "SymbolId": 3, "Reliability": 1.148185667935492, + "PhaseCoherence": 1, "ExpectedCenterHz": 4975, "ObservedCenterHz": 4975, "MeanEnergy": 1.148185667935492, @@ -651,6 +656,7 @@ { "SymbolId": 4, "Reliability": 0, + "PhaseCoherence": 1, "ExpectedCenterHz": 5425, "ObservedCenterHz": 4975, "MeanEnergy": 0.30966049519505534, @@ -662,6 +668,7 @@ { "SymbolId": 6, "Reliability": 0, + "PhaseCoherence": 1, "ExpectedCenterHz": 6325, "ObservedCenterHz": 6775, "MeanEnergy": 0.8679642624875925, @@ -673,6 +680,7 @@ { "SymbolId": 7, "Reliability": 0, + "PhaseCoherence": 0.9999999999999999, "ExpectedCenterHz": 6775, "ObservedCenterHz": 7225, "MeanEnergy": 0.5299773190124039, @@ -684,6 +692,7 @@ { "SymbolId": 8, "Reliability": 0.6732785913925556, + "PhaseCoherence": 1, "ExpectedCenterHz": 7225, "ObservedCenterHz": 7225, "MeanEnergy": 0.6732785913925556, @@ -984,6 +993,7 @@ "DelayHypotheses": [] }, { + "OutputSourceId": "main-speakers", "SourceId": "asio-ch2", "SampleRate": 192000, "Profile": { @@ -1290,6 +1300,7 @@ { "SymbolId": 0, "Reliability": 1.9664832701324948, + "PhaseCoherence": 0.9999999999999998, "ExpectedCenterHz": 3625, "ObservedCenterHz": 3625, "MeanEnergy": 1.9831606924956895, @@ -1301,6 +1312,7 @@ { "SymbolId": 1, "Reliability": 1.9550716359955316, + "PhaseCoherence": 0.9999608311103569, "ExpectedCenterHz": 4075, "ObservedCenterHz": 4075, "MeanEnergy": 1.9724028422080755, @@ -1312,6 +1324,7 @@ { "SymbolId": 2, "Reliability": 1.98317479150165, + "PhaseCoherence": 1, "ExpectedCenterHz": 4525, "ObservedCenterHz": 4525, "MeanEnergy": 1.98317479150165, @@ -1323,6 +1336,7 @@ { "SymbolId": 3, "Reliability": 1.9831581977468316, + "PhaseCoherence": 1, "ExpectedCenterHz": 4975, "ObservedCenterHz": 4975, "MeanEnergy": 1.9831581977468316, @@ -1334,6 +1348,7 @@ { "SymbolId": 4, "Reliability": 1.9831595137173625, + "PhaseCoherence": 1, "ExpectedCenterHz": 5425, "ObservedCenterHz": 5425, "MeanEnergy": 1.9831595137173625, @@ -1345,6 +1360,7 @@ { "SymbolId": 5, "Reliability": 1.983165098459816, + "PhaseCoherence": 1, "ExpectedCenterHz": 5875, "ObservedCenterHz": 5875, "MeanEnergy": 1.983165098459816, @@ -1356,6 +1372,7 @@ { "SymbolId": 6, "Reliability": 1.9831633177029786, + "PhaseCoherence": 1, "ExpectedCenterHz": 6325, "ObservedCenterHz": 6325, "MeanEnergy": 1.9831633177029786, @@ -1367,6 +1384,7 @@ { "SymbolId": 7, "Reliability": 1.9831607953145476, + "PhaseCoherence": 0.9999999999999999, "ExpectedCenterHz": 6775, "ObservedCenterHz": 6775, "MeanEnergy": 1.9831607953145476, @@ -1378,6 +1396,7 @@ { "SymbolId": 31, "Reliability": 1.9831585964902498, + "PhaseCoherence": 1, "ExpectedCenterHz": 17575, "ObservedCenterHz": 17575, "MeanEnergy": 1.9831585964902498, @@ -1733,6 +1752,7 @@ ] }, { + "OutputSourceId": "main-speakers", "SourceId": "asio-ch3", "SampleRate": 192000, "Profile": { @@ -2039,6 +2059,7 @@ { "SymbolId": 0, "Reliability": 1.9664832701324948, + "PhaseCoherence": 0.9999999999999998, "ExpectedCenterHz": 3625, "ObservedCenterHz": 3625, "MeanEnergy": 1.9831606924956895, @@ -2050,6 +2071,7 @@ { "SymbolId": 1, "Reliability": 1.9550716359955316, + "PhaseCoherence": 0.9999608311103569, "ExpectedCenterHz": 4075, "ObservedCenterHz": 4075, "MeanEnergy": 1.9724028422080755, @@ -2061,6 +2083,7 @@ { "SymbolId": 2, "Reliability": 1.98317479150165, + "PhaseCoherence": 1, "ExpectedCenterHz": 4525, "ObservedCenterHz": 4525, "MeanEnergy": 1.98317479150165, @@ -2072,6 +2095,7 @@ { "SymbolId": 3, "Reliability": 1.9831581977468316, + "PhaseCoherence": 1, "ExpectedCenterHz": 4975, "ObservedCenterHz": 4975, "MeanEnergy": 1.9831581977468316, @@ -2083,6 +2107,7 @@ { "SymbolId": 4, "Reliability": 1.9831595137173625, + "PhaseCoherence": 1, "ExpectedCenterHz": 5425, "ObservedCenterHz": 5425, "MeanEnergy": 1.9831595137173625, @@ -2094,6 +2119,7 @@ { "SymbolId": 5, "Reliability": 1.983165098459816, + "PhaseCoherence": 1, "ExpectedCenterHz": 5875, "ObservedCenterHz": 5875, "MeanEnergy": 1.983165098459816, @@ -2105,6 +2131,7 @@ { "SymbolId": 6, "Reliability": 1.9831633177029786, + "PhaseCoherence": 1, "ExpectedCenterHz": 6325, "ObservedCenterHz": 6325, "MeanEnergy": 1.9831633177029786, @@ -2116,6 +2143,7 @@ { "SymbolId": 7, "Reliability": 1.9831607953145476, + "PhaseCoherence": 0.9999999999999999, "ExpectedCenterHz": 6775, "ObservedCenterHz": 6775, "MeanEnergy": 1.9831607953145476, @@ -2127,6 +2155,7 @@ { "SymbolId": 31, "Reliability": 1.9831585964902498, + "PhaseCoherence": 1, "ExpectedCenterHz": 17575, "ObservedCenterHz": 17575, "MeanEnergy": 1.9831585964902498, diff --git a/config/mimir-runtime.asio.example.json b/config/mimir-runtime.asio.example.json index 9570a75..2685f00 100644 --- a/config/mimir-runtime.asio.example.json +++ b/config/mimir-runtime.asio.example.json @@ -13,13 +13,9 @@ "kind": "Audio", "origin": "LocalDevice", "enabled": true, - "adapter": "frame-events", - "command": "../native/probes/asio_audio_cadence/build/Release/asio_audio_cadence.exe", - "arguments": [ - "--set-sample-rate", - "192000", - "--emit-json-blocks" - ], + "adapter": "asio", + "command": "../native/asio_capture/build/Release/mimir_asio_capture.dll", + "sampleRate": 192000, "acceptSourceIds": [ "asio-ch0", "asio-ch1", diff --git a/config/mimir-runtime.probes.example.json b/config/mimir-runtime.probes.example.json index 4aefd12..8e561e9 100644 --- a/config/mimir-runtime.probes.example.json +++ b/config/mimir-runtime.probes.example.json @@ -13,13 +13,9 @@ "kind": "Audio", "origin": "LocalDevice", "enabled": true, - "adapter": "frame-events", - "command": "../native/probes/asio_audio_cadence/build/Release/asio_audio_cadence.exe", - "arguments": [ - "--set-sample-rate", - "192000", - "--emit-json-blocks" - ], + "adapter": "asio", + "command": "../native/asio_capture/build/Release/mimir_asio_capture.dll", + "sampleRate": 192000, "acceptSourceIds": [ "asio-ch0", "asio-ch1", diff --git a/docs/audio-field.md b/docs/audio-field.md index 500a8b1..e84286c 100644 --- a/docs/audio-field.md +++ b/docs/audio-field.md @@ -78,11 +78,12 @@ per-band response evidence, so the same stream can calibrate timing and the speaker/room/mic transfer function. `MimirChirpBinCalibrationModel` is the live surface for that evidence. It stores measured usable bands per source, expected-symbol versus observed-bin confusion observations, timing residuals, -delay hypotheses, phase summaries, and an adaptive codebook plan. The raw -profile is kept even when a timing report is rejected, because a failed sync -window can still teach us which bands survived the speaker/room/mic path. The -decoder can consume the persisted model to weight reliable symbols, downweight -dead bands, apply first-order group-delay correction, and seed global delay +delay hypotheses, phase summaries, and an adaptive codebook plan per +output/mic path. The raw profile is kept even when a timing report is rejected, +because a failed sync window can still teach us which bands survived the +speaker/room/mic path. The decoder can consume the persisted model to weight +reliable symbols, downweight dead bands, apply phase-coherence weighting, +apply first-order group-delay correction, and search joint global delay/bin-shift hypotheses before selecting the coherent anchor path. The model also owns the active emission plan: if the measured path only leaves a smaller alphabet, Mimir emits that reliable symbol set and raises the de Bruijn order so the timeline @@ -130,11 +131,12 @@ buffer, and captures nonzero 4-channel `Int32LSB` input callbacks at 192 kHz. Raven also has a loopback-capable Scarlett ASIO path at 192 kHz for co-streamer/game timing evidence, but Starfire owns the heavy local alignment, soundfield, and sensor-fusion work. -The worker now also supports `--emit-json-blocks`, which streams each ASIO -callback as sample-bearing `audio-block` events (`asio-ch0` through `asio-ch3`) -for `MimirFrameEventProcessStreamSource`. That is a continuous ASIO runtime -ingest path through the existing process boundary; the later native ABI cut -should remove stdout/base64 overhead, not change ASIO ownership. +The runtime hot path is now `native/asio_capture` plus +`MimirAsioStreamSource`. The native DLL owns the Focusrite ASIO COM driver and +keeps callback buffers in process; the managed source drains one 192 kHz Float32 +block per ASIO input channel into `Mimir.Runtime` buffers (`asio-ch0` through +`asio-ch3`). The older `--emit-json-blocks` worker path remains a diagnostic +probe, not the live Scarlett ingest path. The probe also has a `--monitor-sweep` mode for measuring the acoustic ceiling. At 192 kHz, ASIO loopback detected emitted 8-40 kHz tones cleanly, but the @@ -146,8 +148,11 @@ The synthetic chirp-bin timing proof now runs at arbitrary sample rates with `--hybrid-sync-self-test --sample-rate N`. On the same physical delay, measured in memory, the active chirp-bin path recovered about 0.369 us error at 48 kHz, 0.168 us at 96 kHz, and below printed microsecond precision at 192 kHz. That -proves the timing kernel benefits from the Scarlett ASIO rate; the live proof -still depends on feeding ASIO callbacks into `Mimir.Runtime`. +proves the timing kernel benefits from the Scarlett ASIO rate. The live ingest +proof now feeds ASIO callbacks into `Mimir.Runtime` without a process/stdout +bridge: a two-second BufferSmoke run at 192 kHz ingested more than 12,000 +sample-bearing blocks and retained 2,048 blocks per channel across `asio-ch0` +through `asio-ch3`. The full probe runtime config now enables sample-bearing blocks for every local audio source. `MimirChirpletTimeline` owns the emitted calibration stream and @@ -269,8 +274,9 @@ The older `MimirChirpletTimeline` matcher is a diagnostic/reference artifact, not the active runtime path. `BuildChirpletEnergyTrace` still behaves like a dense sliding matched-filter bank and should not retake authority. -Next, replace the diagnostic bridge with native audio capture workers that -append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state, -delay estimates, and stem routing in Fensalir UI. The analyzer accepts Float32, -Int16, Int24, and Int32 PCM windows so the direct driver path can preserve real -interface formats before Faust/native DSP owns the hot resampling and alignment. +Next, use the in-process ASIO source as the local Scarlett authority, prove +stable acoustic anchors against real mics, then expose buffer depth, clock +state, delay estimates, and stem routing in Fensalir UI. The analyzer accepts +Float32, Int16, Int24, and Int32 PCM windows so direct driver paths can preserve +real interface formats before Faust/native DSP owns the hot resampling and +alignment. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index e65605f..9454034 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -36,6 +36,10 @@ a named invariant that the native runtime cannot protect yet. from native probes into the same rolling buffers Fensalir inspects. One probe process can accept multiple emitted `sourceId` values so it does not reopen the same camera set once per stream. +- `native/asio_capture` plus `MimirAsioStreamSource` provide the first + production-shaped Focusrite path: a native in-process ASIO callback source + feeds sample-bearing 192 kHz Float32 blocks directly into `Mimir.Runtime` + rolling buffers on one interface clock domain. - `src/Mimir.BufferSmoke` loads the runtime config, polls the synchronization hub, and prints the actual rolling buffers. Use `--require-samples` when an empty declared sensor buffer should fail the run. Use `--chirplet-self-test` @@ -106,7 +110,8 @@ a named invariant that the native runtime cannot protect yet. bands, expected-symbol versus observed-bin confusion observations, timing residuals, delay hypotheses, phase summaries, and an adaptive codebook plan. The active decoder can consume that model as learned response weighting, - first-order group-delay correction, and global delay-hypothesis seeds. The + phase-coherence weighting, first-order group-delay correction, and joint + global delay/bin-shift hypotheses. The runtime emitter also consumes the model's emission plan, rendering the smaller reliable symbol alphabet at the higher recommended de Bruijn order when the physical path cannot support all 32 bins. @@ -129,13 +134,14 @@ a named invariant that the native runtime cannot protect yet. loopback timing proof, but failed timing windows now leave usable response evidence instead of silence. - `config/mimir-runtime.asio.example.json` is the minimal continuous Scarlett - runtime ingest proof. It starts the ASIO worker at 192 kHz with - `--emit-json-blocks` and declares `asio-ch0` through `asio-ch3` as accepted - audio sources. A two-second BufferSmoke run ingested 13,324 sample-bearing - blocks, 3,331 per channel, proving loopback and mic channels enter - `Mimir.Runtime` together in one ASIO clock domain. BufferSmoke does not emit - speaker calibration audio, so that proof is ingest-only rather than a sync - report. + runtime ingest proof. It loads `native/asio_capture` in process at 192 kHz + and declares `asio-ch0` through `asio-ch3` as accepted audio sources. A + two-second BufferSmoke run ingested more than 12,000 sample-bearing blocks + and retained 2,048 blocks per channel, proving loopback and mic channels enter + `Mimir.Runtime` together in one ASIO clock domain without stdout/base64 + transport. BufferSmoke does not + emit speaker calibration audio, so that proof is ingest-only rather than a + sync report. A standalone 192 kHz synthetic receiver test recovers a 500 ms delayed audio stream to below printed microsecond precision using only the chirp-bin codebook and schedule state. @@ -170,26 +176,22 @@ a named invariant that the native runtime cannot protect yet. other cameras. 2. Feed those drivers into `MimirVideoCaptureDriverSource` and prove sustained frame cadence in the rolling buffers. -3. Replace the WASAPI frame-event diagnostic bridge with native audio capture - workers for local mic, loopback, and network audio feeds. Use the verified - Focusrite ASIO path for Scarlett capture; WASAPI remains a diagnostic - witness, not the Scarlett hot path. -4. Use chirp-bin calibration profiles from real microphones to tune acoustic +3. Use chirp-bin calibration profiles from real microphones to tune acoustic bands, gain, and code spacing without weakening the standalone codebook/schedule receiver invariant. -5. Add the synchronization actuator: drive a variable-rate resampler and +4. Add the synchronization actuator: drive a variable-rate resampler and fractional delay line per non-reference stream from the smoothed `MimirAudioSynchronizationState`. First, prove the constrained chirplet decoder through real loopback and microphone paths so every correctly heard triplet becomes a deterministic timeline anchor before the actuator moves samples. -6. Prove the chirp-bin hybrid fallback through real loopback and microphones +5. Prove the chirp-bin hybrid fallback through real loopback and microphones with probe durations long enough to keep loopback and mic windows live. -7. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, +6. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence, source timestamps, and output settings are visible and adjustable. -8. Move GPU feature extraction, fusion, material fitting, render budgeting, and +7. Move GPU feature extraction, fusion, material fitting, render budgeting, and Spout2 publication into Fensalir. -9. Move mic alignment, room suppression, voice separation, spatialization, and +8. Move mic alignment, room suppression, voice separation, spatialization, and stem generation into Faust/native DSP. -10. Keep the OBS bridge witness ledger as evidence before expanding receiver +9. Keep the OBS bridge witness ledger as evidence before expanding receiver machinery. diff --git a/native/README.md b/native/README.md index 7bdb5c5..10b504b 100644 --- a/native/README.md +++ b/native/README.md @@ -6,7 +6,7 @@ This directory is for the hot path. - `reservoir`: the first native five-second spatiotemporal reservoir core. It owns one shared-edge rolling buffer with typed sample views. - Aquarium/Faust integration should build on this invariant instead of copying + Fensalir/Faust integration should build on this invariant instead of copying the deadline bridge cache. It exports `rlib`, `cdylib`, and `staticlib` artifacts plus the C header at `reservoir/include/localcast_reservoir.h`. @@ -35,16 +35,16 @@ The reservoir ABI is intentionally small: ## Runtime ABI -`LocalcastRuntime` is the live spine for Aquarium/Faust integration: +`LocalcastRuntime` is the live spine for Fensalir/Faust integration: - it owns one native reservoir; - it exposes typed producer functions for camera frames, camera features, scene rays, surface claims, material claims, audio blocks, phase claims, event claims, and render packets; -- it exposes `LocalcastRuntimeStatus` so Aquarium/Faust can inspect the shared +- it exposes `LocalcastRuntimeStatus` so Fensalir/Faust can inspect the shared edge, window start, total sample count, and typed-view counts without polling JSON or MessagePack files; -- it exposes total and typed-view sample reads so Aquarium/Faust can consume the +- it exposes total and typed-view sample reads so Fensalir/Faust can consume the same rolling window producers append to. `LocalcastProducer` is the native ingress helper for capture workers: @@ -69,16 +69,16 @@ The reservoir ABI is intentionally small: optional metadata handle; - `localcast_producer_push_render_packet` only accepts `RenderPacket` producers and stores the descriptor pointer as the sample payload handle; -- Aquarium owns decoding and GPU upload. The reservoir only indexes when this +- Fensalir owns decoding and GPU upload. The reservoir only indexes when this render packet exists in the shared rolling window. `LocalcastRenderPoint` is the fixed point-buffer element shape for the initial -native/Aquarium render path. It carries a stable-key hash, source timestamp, +native/Fensalir render path. It carries a stable-key hash, source timestamp, position, radius, color/alpha, and confidence. Richer material/brush payloads can get their own descriptors later; this point shape exists to kill the old JSON/CultCache render-frame dependency first. -Payload handles are owned by the caller. Aquarium should treat them as handles +Payload handles are owned by the caller. Fensalir should treat them as handles to GPU/native visual memory; Faust/native DSP should treat them as handles to audio buffers. This crate does not interpret those bytes. It is the clocked retention authority, not the renderer or DSP engine in a tiny hat. diff --git a/native/asio_capture/CMakeLists.txt b/native/asio_capture/CMakeLists.txt new file mode 100644 index 0000000..c0462d8 --- /dev/null +++ b/native/asio_capture/CMakeLists.txt @@ -0,0 +1,15 @@ +cmake_minimum_required(VERSION 3.20) + +project(mimir_asio_capture LANGUAGES CXX) + +add_library(mimir_asio_capture SHARED + mimir_asio_capture.cpp) + +target_compile_features(mimir_asio_capture PRIVATE cxx_std_20) +target_compile_definitions(mimir_asio_capture PRIVATE + WIN32_LEAN_AND_MEAN + NOMINMAX + _WIN32_WINNT=0x0A00) +target_link_libraries(mimir_asio_capture PRIVATE + ole32 + oleaut32) diff --git a/native/asio_capture/mimir_asio_capture.cpp b/native/asio_capture/mimir_asio_capture.cpp new file mode 100644 index 0000000..c2a158d --- /dev/null +++ b/native/asio_capture/mimir_asio_capture.cpp @@ -0,0 +1,405 @@ +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using ASIOBool = long; +using ASIOError = long; +using ASIOSampleRate = double; +using ASIOSampleType = long; + +constexpr ASIOError ASE_OK = 0; + +struct ASIOSamples +{ + unsigned long hi; + unsigned long lo; +}; + +struct ASIOTimeStamp +{ + unsigned long hi; + unsigned long lo; +}; + +struct ASIOClockSource +{ + long index; + long associatedChannel; + long associatedGroup; + ASIOBool isCurrentSource; + char name[32]; +}; + +struct ASIOChannelInfo +{ + long channel; + ASIOBool isInput; + ASIOBool isActive; + long channelGroup; + ASIOSampleType type; + char name[32]; +}; + +struct ASIOBufferInfo +{ + ASIOBool isInput; + long channelNum; + void* buffers[2]; +}; + +struct ASIOTime; + +struct ASIOCallbacks +{ + void (*bufferSwitch)(long doubleBufferIndex, ASIOBool directProcess); + void (*sampleRateDidChange)(ASIOSampleRate sRate); + long (*asioMessage)(long selector, long value, void* message, double* opt); + ASIOTime* (*bufferSwitchTimeInfo)(ASIOTime* params, long doubleBufferIndex, ASIOBool directProcess); +}; + +struct IASIO : public IUnknown +{ + virtual ASIOBool init(void* sysHandle) = 0; + virtual void getDriverName(char* name) = 0; + virtual long getDriverVersion() = 0; + virtual void getErrorMessage(char* string) = 0; + virtual ASIOError start() = 0; + virtual ASIOError stop() = 0; + virtual ASIOError getChannels(long* numInputChannels, long* numOutputChannels) = 0; + virtual ASIOError getLatencies(long* inputLatency, long* outputLatency) = 0; + virtual ASIOError getBufferSize(long* minSize, long* maxSize, long* preferredSize, long* granularity) = 0; + virtual ASIOError canSampleRate(ASIOSampleRate sampleRate) = 0; + virtual ASIOError getSampleRate(ASIOSampleRate* sampleRate) = 0; + virtual ASIOError setSampleRate(ASIOSampleRate sampleRate) = 0; + virtual ASIOError getClockSources(ASIOClockSource* clocks, long* numSources) = 0; + virtual ASIOError setClockSource(long reference) = 0; + virtual ASIOError getSamplePosition(ASIOSamples* samplePosition, ASIOTimeStamp* timeStamp) = 0; + virtual ASIOError getChannelInfo(ASIOChannelInfo* info) = 0; + virtual ASIOError createBuffers(ASIOBufferInfo* bufferInfos, long numChannels, long bufferSize, ASIOCallbacks* callbacks) = 0; + virtual ASIOError disposeBuffers() = 0; + virtual ASIOError controlPanel() = 0; + virtual ASIOError future(long selector, void* opt) = 0; + virtual ASIOError outputReady() = 0; +}; + +struct MimirAsioBlock +{ + int channel = 0; + int sampleRate = 0; + int frameCount = 0; + std::uint64_t sequence = 0; + std::int64_t timestampNs = 0; + std::vector samples; +}; + +struct MimirAsioCapture +{ + IASIO* driver = nullptr; + ASIOCallbacks callbacks{}; + std::vector buffers; + std::vector channels; + std::deque queue; + std::mutex mutex; + long inputs = 0; + long outputs = 0; + long bufferSize = 0; + double sampleRate = 0.0; + std::uint64_t frameCursor = 0; + std::uint64_t sequence = 0; + bool running = false; +}; + +static MimirAsioCapture* g_capture = nullptr; + +static int bytesPerSample(ASIOSampleType type) +{ + switch (type) + { + case 16: return 2; + case 17: return 3; + case 18: return 4; + case 19: return 4; + case 20: return 8; + case 24: + case 25: + case 26: + case 27: return 4; + default: return 0; + } +} + +static double readSample(const unsigned char* sample, ASIOSampleType type) +{ + switch (type) + { + case 16: + return *reinterpret_cast(sample) / 32768.0; + case 17: + { + int value = sample[0] | (sample[1] << 8) | (sample[2] << 16); + if ((value & 0x00800000) != 0) + { + value |= static_cast(0xff000000); + } + return value / 8388608.0; + } + case 18: + case 24: + case 25: + case 26: + case 27: + return *reinterpret_cast(sample) / 2147483648.0; + case 19: + return *reinterpret_cast(sample); + case 20: + return *reinterpret_cast(sample); + default: + return 0.0; + } +} + +static CLSID clsidFromString(const char* raw) +{ + wchar_t wide[64]{}; + MultiByteToWideChar(CP_UTF8, 0, raw, -1, wide, static_cast(std::size(wide))); + CLSID clsid{}; + CLSIDFromString(wide, &clsid); + return clsid; +} + +static void pushInputBlocks(MimirAsioCapture& capture, long doubleBufferIndex) +{ + const auto startFrame = capture.frameCursor; + const auto timestampNs = static_cast( + static_cast(startFrame) * 1000000000.0 / capture.sampleRate); + std::lock_guard lock(capture.mutex); + for (long channel = 0; channel < capture.inputs; ++channel) + { + auto& buffer = capture.buffers[static_cast(channel)]; + auto& info = capture.channels[static_cast(channel)]; + auto* raw = static_cast(buffer.buffers[doubleBufferIndex]); + const auto sampleBytes = bytesPerSample(info.type); + if (!raw || sampleBytes <= 0) + { + continue; + } + + MimirAsioBlock block; + block.channel = static_cast(channel); + block.sampleRate = static_cast(capture.sampleRate); + block.frameCount = static_cast(capture.bufferSize); + block.sequence = capture.sequence++; + block.timestampNs = timestampNs; + block.samples.resize(static_cast(capture.bufferSize)); + for (long frame = 0; frame < capture.bufferSize; ++frame) + { + block.samples[static_cast(frame)] = static_cast( + readSample(raw + frame * sampleBytes, info.type)); + } + + capture.queue.push_back(std::move(block)); + } + + while (capture.queue.size() > 4096) + { + capture.queue.pop_front(); + } + + capture.frameCursor += static_cast(capture.bufferSize); +} + +static void bufferSwitch(long doubleBufferIndex, ASIOBool) +{ + if (g_capture && g_capture->running) + { + pushInputBlocks(*g_capture, doubleBufferIndex); + } +} + +static void sampleRateDidChange(ASIOSampleRate) {} + +static long asioMessage(long, long, void*, double*) +{ + return 0; +} + +static ASIOTime* bufferSwitchTimeInfo(ASIOTime* params, long doubleBufferIndex, ASIOBool directProcess) +{ + bufferSwitch(doubleBufferIndex, directProcess); + return params; +} + +extern "C" +{ +__declspec(dllexport) MimirAsioCapture* mimir_asio_create( + const char* clsidText, + double requestedSampleRate, + int* sampleRate, + int* inputCount, + int* preferredBufferSize) +{ + CoInitializeEx(nullptr, COINIT_APARTMENTTHREADED); + auto* capture = new MimirAsioCapture(); + const auto clsid = clsidFromString(clsidText ? clsidText : "{AC4D0455-50D7-4498-B3CD-9A41D130B759}"); + const auto hr = CoCreateInstance(clsid, nullptr, CLSCTX_INPROC_SERVER, clsid, reinterpret_cast(&capture->driver)); + if (FAILED(hr) || !capture->driver) + { + delete capture; + CoUninitialize(); + return nullptr; + } + + HWND hwnd = GetConsoleWindow(); + if (!hwnd) + { + hwnd = GetDesktopWindow(); + } + + if (!capture->driver->init(hwnd)) + { + capture->driver->Release(); + delete capture; + CoUninitialize(); + return nullptr; + } + + capture->driver->getChannels(&capture->inputs, &capture->outputs); + long minSize = 0; + long maxSize = 0; + long granularity = 0; + capture->driver->getBufferSize(&minSize, &maxSize, &capture->bufferSize, &granularity); + if (requestedSampleRate > 0.0) + { + capture->driver->setSampleRate(requestedSampleRate); + } + + capture->driver->getSampleRate(&capture->sampleRate); + capture->buffers.resize(static_cast(capture->inputs)); + capture->channels.resize(static_cast(capture->inputs)); + for (long channel = 0; channel < capture->inputs; ++channel) + { + capture->buffers[static_cast(channel)].isInput = 1; + capture->buffers[static_cast(channel)].channelNum = channel; + capture->channels[static_cast(channel)].channel = channel; + capture->channels[static_cast(channel)].isInput = 1; + capture->driver->getChannelInfo(&capture->channels[static_cast(channel)]); + } + + capture->callbacks.bufferSwitch = bufferSwitch; + capture->callbacks.sampleRateDidChange = sampleRateDidChange; + capture->callbacks.asioMessage = asioMessage; + capture->callbacks.bufferSwitchTimeInfo = bufferSwitchTimeInfo; + if (capture->driver->createBuffers(capture->buffers.data(), capture->inputs, capture->bufferSize, &capture->callbacks) != ASE_OK) + { + capture->driver->Release(); + delete capture; + CoUninitialize(); + return nullptr; + } + + if (sampleRate) + { + *sampleRate = static_cast(capture->sampleRate); + } + if (inputCount) + { + *inputCount = static_cast(capture->inputs); + } + if (preferredBufferSize) + { + *preferredBufferSize = static_cast(capture->bufferSize); + } + + return capture; +} + +__declspec(dllexport) int mimir_asio_start(MimirAsioCapture* capture) +{ + if (!capture || !capture->driver) + { + return 0; + } + + g_capture = capture; + capture->running = true; + return capture->driver->start() == ASE_OK ? 1 : 0; +} + +__declspec(dllexport) int mimir_asio_read( + MimirAsioCapture* capture, + int* channel, + std::int64_t* timestampNs, + std::uint64_t* sequence, + int* frameCount, + float* samples, + int maxFrames) +{ + if (!capture || !samples || maxFrames <= 0) + { + return 0; + } + + std::lock_guard lock(capture->mutex); + if (capture->queue.empty()) + { + return 0; + } + + auto block = std::move(capture->queue.front()); + capture->queue.pop_front(); + const auto count = std::min(block.frameCount, maxFrames); + std::memcpy(samples, block.samples.data(), static_cast(count) * sizeof(float)); + if (channel) + { + *channel = block.channel; + } + if (timestampNs) + { + *timestampNs = block.timestampNs; + } + if (sequence) + { + *sequence = block.sequence; + } + if (frameCount) + { + *frameCount = count; + } + + return 1; +} + +__declspec(dllexport) void mimir_asio_destroy(MimirAsioCapture* capture) +{ + if (!capture) + { + return; + } + + capture->running = false; + if (g_capture == capture) + { + g_capture = nullptr; + } + + if (capture->driver) + { + capture->driver->stop(); + capture->driver->disposeBuffers(); + capture->driver->Release(); + } + + delete capture; + CoUninitialize(); +} +} diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 92e48b0..f32aa5e 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -116,8 +116,9 @@ usable bands, expected-symbol versus observed-bin confusion observations, timing residuals, delay hypotheses, phase summaries, and an adaptive codebook plan. The analyzer keeps raw profiles even when no timing report is accepted, and the active decoder can consume a persisted model for learned response -weighting, first-order group-delay correction, and global delay-hypothesis -seeding. Runtime chirp-bin emission consumes the same model's emission plan, so +weighting, phase-coherence weighting, first-order group-delay correction, and +joint global delay/bin-shift hypotheses. Runtime chirp-bin emission consumes the +same model's emission plan, so the emitted timeline can use the measured reliable symbol set at a higher de Bruijn order instead of continuing to spend code on dead bins. Reports/states expose `delayUs` next to fractional sample delay. @@ -163,14 +164,15 @@ The ASIO probe can now play raw mono Float32 timeline audio through the Focusrite outputs while capturing every ASIO input as raw interleaved Float32. `Mimir.BufferSmoke --analyze-asio-f32` feeds those captured channels into the same runtime active analyzer. `--calibrate-chirp-bin-asio-f32` computes and -persists the response/confusion/delay model, and `--analyze-asio-f32 ---calibration ...` loads it into the active decoder. The ASIO worker also -supports `--emit-json-blocks`, so Focusrite ASIO callbacks can feed -sample-bearing `audio-block` events into `Mimir.Runtime` continuously through -the existing frame-event adapter while the native ABI cut is pending. The -minimal `config/mimir-runtime.asio.example.json` proof ingested 13,324 -sample-bearing 192 kHz blocks across `asio-ch0` through `asio-ch3` in two -seconds, all from the same ASIO callback stream. A real +persists the response/confusion/delay model per output/mic path, and +`--analyze-asio-f32 --calibration ...` loads it into the active decoder. +`native/asio_capture` and `MimirAsioStreamSource` are now the runtime Scarlett +path: Focusrite ASIO callbacks feed sample-bearing 192 kHz Float32 blocks into +`Mimir.Runtime` in process, without the diagnostic JSON/stdout bridge. The +minimal `config/mimir-runtime.asio.example.json` proof ingested more than +12,000 sample-bearing 192 kHz blocks across `asio-ch0` through `asio-ch3` in +two seconds and retained 2,048 blocks per channel, all from the same ASIO +callback stream. A real 192 kHz chirp-bin Scarlett artifact decoded `Loopback 1 -> Loopback 2` at exactly `0.000 us` with 12 matched anchors and 0.999 confidence. The same analyzer now prints calibration profiles for diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 450f22e..7ff8bf6 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -92,9 +92,10 @@ Get-Content .\state\evidence.jsonl -Tail 8 Each classified chirp carries the full dechirped bin-energy surface; sync reports expose those bands, and `MimirChirpBinCalibrationModel` preserves measured usable bands, response/confusion observations, timing residuals, - delay hypotheses, phase summaries, and adaptive codebook plans. The active - decoder can load the model for learned weighting, first-order group-delay - correction, and global delay-hypothesis seeds. Runtime chirp-bin emission + delay hypotheses, phase summaries, and adaptive codebook plans per + output/mic path. The active decoder can load the model for learned weighting, + phase-coherence weighting, first-order group-delay correction, and joint + global delay/bin-shift hypotheses. Runtime chirp-bin emission loads the same model's emission plan, so a physical calibration can shrink the emitted alphabet and raise sequence order instead of continuing to use dead bins. @@ -129,20 +130,21 @@ Get-Content .\state\evidence.jsonl -Tail 8 `--calibrate-chirp-bin-asio-f32` now persists a real response/confusion/delay model and `--analyze-asio-f32 --calibration ...` consumes it. In the stored artifact, physical input 1 produced two reliable symbols and a different - strongest-bin profile while failing pairwise timing. The ASIO native worker - also supports `--emit-json-blocks` so `Mimir.Runtime` can ingest Focusrite ASIO - callbacks continuously through the frame-event adapter. The minimal - `config/mimir-runtime.asio.example.json` proof ingested 13,324 sample-bearing - 192 kHz blocks across `asio-ch0` through `asio-ch3` in two seconds. Acoustic robustness is - still open, but failed timing now leaves response evidence. + strongest-bin profile while failing pairwise timing. `native/asio_capture` + plus `MimirAsioStreamSource` now load Focusrite ASIO in process and feed + sample-bearing 192 kHz Float32 blocks directly into runtime buffers. The + minimal `config/mimir-runtime.asio.example.json` proof ingests more than + 12,000 blocks across `asio-ch0` through `asio-ch3` in two seconds and retains + 2,048 blocks per channel. Acoustic robustness is still open, but failed + timing now leaves response evidence. - Raven also has a 192 kHz loopback-capable Scarlett for co-streamer/game timing evidence. Do not move the heavy soundfield or sensor-fusion workload there. ## Immediate Re-entry Instruction Use the chirp-bin calibration models from real microphones to tune the active -codebook/bands/weighting, then replace the ASIO JSON-block process bridge with -a native ABI callback source and turn decoded clock fits into the -SRO/fractional-delay actuator. Keep loopback as timing authority. Do not call -synchronization analysis from UI/telemetry readouts. Do not restore deleted -script infrastructure because a stale doc once missed it. +codebook/bands/weighting, then turn decoded clock fits into the +SRO/fractional-delay actuator. Keep loopback as timing authority and keep +Scarlett capture on the in-process ASIO source. Do not call synchronization +analysis from UI/telemetry readouts. Do not restore deleted script +infrastructure because a stale doc once missed it. diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 19f1f29..d81fa7c 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -599,6 +599,7 @@ static int CalibrateChirpBinAsioFloat32( int channels, int referenceChannel) { + var outputSourceId = ParseStringOption(args, "--output-source-id", "main-speakers"); if (args.Any(arg => string.Equals(arg, "--capture-asio", StringComparison.OrdinalIgnoreCase))) { var probe = ParseStringOption( @@ -636,7 +637,7 @@ static int CalibrateChirpBinAsioFloat32( } var referenceSourceId = $"asio-ch{referenceChannel}"; - var model = MimirChirpBinCalibrationModel.FromDecodes(referenceSourceId, sampleRate, decodes); + var model = MimirChirpBinCalibrationModel.FromDecodes(referenceSourceId, sampleRate, decodes, outputSourceId); model.Save(outputPath); Console.WriteLine($"chirp-bin-calibration path={outputPath} input={inputPath} sampleRate={sampleRate} channels={channels} frames={frameCount} reference={referenceSourceId}"); foreach (var path in model.Paths) diff --git a/src/Mimir.Runtime/Mimir.Runtime.csproj b/src/Mimir.Runtime/Mimir.Runtime.csproj index 068c445..ffc1992 100644 --- a/src/Mimir.Runtime/Mimir.Runtime.csproj +++ b/src/Mimir.Runtime/Mimir.Runtime.csproj @@ -6,6 +6,7 @@ Mimir.Runtime enable enable + true true diff --git a/src/Mimir.Runtime/Synchronization/MimirAsioStreamSource.cs b/src/Mimir.Runtime/Synchronization/MimirAsioStreamSource.cs new file mode 100644 index 0000000..f1e1d4f --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirAsioStreamSource.cs @@ -0,0 +1,225 @@ +using System.Runtime.InteropServices; +using System.Collections.Concurrent; +using System.Runtime.Versioning; + +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirAsioStreamSourceOptions( + string NativeLibraryPath, + string DriverClsid, + int SampleRate, + IReadOnlyList SourceIds); + +[SupportedOSPlatform("windows")] +public sealed class MimirAsioStreamSource : IMimirStreamSource +{ + private const string DefaultClsid = "{AC4D0455-50D7-4498-B3CD-9A41D130B759}"; + private readonly ConcurrentQueue samples = new(); + private readonly ManualResetEventSlim started = new(); + private readonly Thread captureThread; + private readonly CancellationTokenSource cancellation = new(); + private readonly string[] sourceIds; + private int sampleRate; + private string? startError; + private bool disposed; + + public MimirAsioStreamSource( + MimirStreamDescriptor descriptor, + MimirAsioStreamSourceOptions options) + { + Descriptor = descriptor; + Options = options; + sourceIds = options.SourceIds.Count > 0 + ? options.SourceIds.ToArray() + : Enumerable.Range(0, 8).Select(index => $"asio-ch{index}").ToArray(); + captureThread = new Thread(CaptureLoop) + { + IsBackground = true, + Name = "Mimir Focusrite ASIO capture", + }; + captureThread.SetApartmentState(ApartmentState.STA); + captureThread.Start(); + if (!started.Wait(TimeSpan.FromSeconds(5))) + { + throw new InvalidOperationException("Timed out starting Focusrite ASIO capture source"); + } + + if (startError != null) + { + throw new InvalidOperationException(startError); + } + } + + public MimirStreamDescriptor Descriptor { get; } + + public MimirAsioStreamSourceOptions Options { get; } + + public bool ExposesDescriptorBuffer => false; + + public bool TryRead(out MimirStreamSample sample) + { + sample = default; + ObjectDisposedException.ThrowIf(disposed, this); + return samples.TryDequeue(out sample); + } + + public void Dispose() + { + if (disposed) + { + return; + } + + disposed = true; + cancellation.Cancel(); + captureThread.Join(TimeSpan.FromSeconds(2)); + cancellation.Dispose(); + started.Dispose(); + } + + private void CaptureLoop() + { + nint handle = 0; + try + { + Native.Load(Options.NativeLibraryPath); + handle = Native.Create( + string.IsNullOrWhiteSpace(Options.DriverClsid) ? DefaultClsid : Options.DriverClsid, + Options.SampleRate, + out sampleRate, + out _, + out var maxFrames); + if (handle == 0) + { + startError = "Could not create Focusrite ASIO capture source"; + started.Set(); + return; + } + + if (!Native.Start(handle)) + { + startError = "Could not start Focusrite ASIO capture source"; + started.Set(); + return; + } + + started.Set(); + var sampleBuffer = new float[Math.Max(1, maxFrames)]; + while (!cancellation.IsCancellationRequested) + { + if (!Native.Read( + handle, + out var channel, + out var timestampNs, + out var sequence, + out var frameCount, + sampleBuffer, + sampleBuffer.Length)) + { + Thread.Sleep(1); + continue; + } + + if (channel < 0) + { + continue; + } + + var sourceId = channel < sourceIds.Length ? sourceIds[channel] : $"asio-ch{channel}"; + var bytes = new byte[frameCount * sizeof(float)]; + MemoryMarshal.AsBytes(sampleBuffer.AsSpan(0, frameCount)).CopyTo(bytes); + samples.Enqueue(new MimirStreamSample( + sourceId, + MimirStreamKind.Audio, + Descriptor.Origin, + timestampNs, + DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000L, + sequence, + 0, + bytes.Length, + bytes, + AudioBlock: new MimirAudioBlockDescriptor( + sampleRate, + 1, + MimirAudioSampleFormat.Float32, + frameCount, + timestampNs))); + + while (samples.Count > 8192 && samples.TryDequeue(out _)) + { + } + } + } + catch (Exception ex) + { + startError = ex.Message; + started.Set(); + } + finally + { + Native.Destroy(handle); + } + } + + private static unsafe class Native + { + private static nint library; + private static delegate* unmanaged[Stdcall] create; + private static delegate* unmanaged[Stdcall] start; + private static delegate* unmanaged[Stdcall] read; + private static delegate* unmanaged[Stdcall] destroy; + + public static void Load(string path) + { + if (library != 0) + { + return; + } + + if (string.IsNullOrWhiteSpace(path)) + { + path = Path.Combine(AppContext.BaseDirectory, "mimir_asio_capture.dll"); + } + + library = NativeLibrary.Load(path); + create = (delegate* unmanaged[Stdcall])NativeLibrary.GetExport(library, "mimir_asio_create"); + start = (delegate* unmanaged[Stdcall])NativeLibrary.GetExport(library, "mimir_asio_start"); + read = (delegate* unmanaged[Stdcall])NativeLibrary.GetExport(library, "mimir_asio_read"); + destroy = (delegate* unmanaged[Stdcall])NativeLibrary.GetExport(library, "mimir_asio_destroy"); + } + + public static nint Create(string clsid, int requestedSampleRate, out int sampleRate, out int inputCount, out int preferredBufferSize) + { + var bytes = System.Text.Encoding.UTF8.GetBytes(clsid + "\0"); + fixed (byte* clsidPtr = bytes) + { + return create(clsidPtr, requestedSampleRate, out sampleRate, out inputCount, out preferredBufferSize); + } + } + + public static bool Start(nint handle) => start(handle) != 0; + + public static bool Read( + nint handle, + out int channel, + out long timestampNs, + out ulong sequence, + out int frameCount, + float[] samples, + int maxFrames) + { + fixed (float* samplePtr = samples) + { + return read(handle, out channel, out timestampNs, out sequence, out frameCount, samplePtr, maxFrames) != 0; + } + } + + public static void Destroy(nint handle) + { + if (handle != 0) + { + destroy(handle); + } + } + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpBinCalibrationModel.cs b/src/Mimir.Runtime/Synchronization/MimirChirpBinCalibrationModel.cs index d60783c..8d73907 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpBinCalibrationModel.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpBinCalibrationModel.cs @@ -39,11 +39,12 @@ public sealed record MimirChirpBinCalibrationModel( public static MimirChirpBinCalibrationModel FromDecodes( string referenceSourceId, int sampleRate, - IReadOnlyDictionary decodes) + IReadOnlyDictionary decodes, + string outputSourceId = "main-speakers") { var paths = decodes .OrderBy(pair => pair.Key, StringComparer.Ordinal) - .Select(pair => MimirChirpBinPathCalibration.FromDecode(pair.Key, sampleRate, pair.Value)) + .Select(pair => MimirChirpBinPathCalibration.FromDecode(outputSourceId, pair.Key, sampleRate, pair.Value)) .ToArray(); var emissionPlan = MimirChirpBinCodebookPlan.ForSharedEmission(paths); paths = paths.Select(path => path with { EmissionPlan = emissionPlan }).ToArray(); @@ -58,6 +59,7 @@ public sealed record MimirChirpBinCalibrationModel( } public sealed record MimirChirpBinPathCalibration( + string OutputSourceId, string SourceId, int SampleRate, MimirChirpBinCalibrationProfile Profile, @@ -79,7 +81,20 @@ public sealed record MimirChirpBinPathCalibration( return symbol?.MeanTimingResidualSamples ?? 0.0; } + public double PhaseWeight(int symbolId, double observedPhaseRadians) + { + var symbol = Symbols.FirstOrDefault(item => item.SymbolId == symbolId); + if (symbol == null || symbol.ObservationCount <= 0) + { + return 1.0; + } + + var error = Math.Abs(AngularDistance(observedPhaseRadians, symbol.MeanPhaseRadians)); + return Math.Clamp(1.0 - error / Math.PI, 0.15, 1.0); + } + public static MimirChirpBinPathCalibration FromDecode( + string outputSourceId, string sourceId, int sampleRate, MimirChirpletStreamDecode decode) @@ -99,9 +114,11 @@ public sealed record MimirChirpBinPathCalibration( (correct / (double)items.Length) * (1.0 / (1.0 + residualSpread / Math.Max(1.0, sampleRate * 0.00025))); var phase = CircularMean(items.Select(item => item.PhaseRadians)); + var phaseCoherence = PhaseCoherence(items.Select(item => item.PhaseRadians)); return new MimirChirpBinSymbolCalibration( group.Key, reliability, + phaseCoherence, items.Average(item => item.ExpectedCenterHz), items.Average(item => item.ObservedCenterHz), items.Average(item => item.ObservedEnergy), @@ -115,6 +132,7 @@ public sealed record MimirChirpBinPathCalibration( var hypotheses = BuildDelayHypotheses(decode); var plan = MimirChirpBinCodebookPlan.FromSymbols(symbols); return new MimirChirpBinPathCalibration( + outputSourceId, sourceId, sampleRate, profile, @@ -201,6 +219,27 @@ public sealed record MimirChirpBinPathCalibration( return count == 0 ? 0.0 : Math.Atan2(sin / count, cos / count); } + + private static double PhaseCoherence(IEnumerable radians) + { + var sin = 0.0; + var cos = 0.0; + var count = 0; + foreach (var value in radians) + { + sin += Math.Sin(value); + cos += Math.Cos(value); + count++; + } + + return count == 0 ? 0.0 : Math.Clamp(Math.Sqrt(sin * sin + cos * cos) / count, 0.0, 1.0); + } + + private static double AngularDistance(double a, double b) + { + var delta = Math.Abs(a - b) % (Math.PI * 2.0); + return delta > Math.PI ? Math.PI * 2.0 - delta : delta; + } } public sealed record MimirChirpBinCodebookPlan( @@ -253,6 +292,7 @@ public sealed record MimirChirpBinCodebookPlan( public sealed record MimirChirpBinSymbolCalibration( int SymbolId, double Reliability, + double PhaseCoherence, double ExpectedCenterHz, double ObservedCenterHz, double MeanEnergy, diff --git a/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs b/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs index 06e3d3f..f0f0313 100644 --- a/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs +++ b/src/Mimir.Runtime/Synchronization/MimirChirpBinTimeline.cs @@ -296,7 +296,8 @@ public sealed class MimirChirpBinTimeline var correctedEnergy = ScoreSymbolEnergy( samples.Slice((int)Math.Round(correctedOffset), chirpSamples), sampleRate, - canonicalSymbol) * (calibration?.SymbolWeight(canonicalSymbol) ?? 1.0); + canonicalSymbol) * + (calibration?.SymbolWeight(canonicalSymbol) ?? 1.0); correctedOffset -= calibration?.GroupDelayCorrectionSamples(canonicalSymbol) ?? 0.0; correctedCandidates.Add(new MimirChirpletSymbolCandidate( canonicalSymbol, @@ -350,7 +351,9 @@ public sealed class MimirChirpBinTimeline .Select((score, symbol) => new MimirChirpletSymbolCandidate( symbol, sampleOffset - (calibration?.GroupDelayCorrectionSamples(symbol) ?? 0.0), - score.Energy * (calibration?.SymbolWeight(symbol) ?? 1.0), + score.Energy * + (calibration?.SymbolWeight(symbol) ?? 1.0) * + (calibration?.PhaseWeight(symbol, score.PhaseRadians) ?? 1.0), bandResponses)) .OrderByDescending(candidate => candidate.Energy) .Take(MaxSymbolCandidatesPerFrame) @@ -459,35 +462,45 @@ public sealed class MimirChirpBinTimeline foreach (var candidatePath in CandidateSymbolPaths(window)) { - var code = CodeKey(candidatePath.Select(candidate => candidate.SymbolId)); - if (!plan.CodeToIndex.TryGetValue(code, out var eventIndex)) + foreach (var globalBinShift in GlobalBinShiftHypotheses(calibration)) { - continue; - } + var canonicalSymbols = candidatePath + .Select(candidate => ShiftSymbol(candidate.SymbolId, -globalBinShift)) + .ToArray(); + var code = CodeKey(canonicalSymbols); + if (!plan.CodeToIndex.TryGetValue(code, out var eventIndex)) + { + continue; + } - var firstEvent = Default.EventForIndex((ulong)eventIndex, calibration?.EmissionPlan); - var gapError = 0.0; - for (var gap = 0; gap < candidatePath.Count - 1; gap++) - { - var measuredGap = (candidatePath[gap + 1].SampleOffset - candidatePath[gap].SampleOffset) / sampleRate; - gapError += Math.Abs(measuredGap - EventSpacingSeconds); - } + var firstEvent = Default.EventForIndex((ulong)eventIndex, calibration?.EmissionPlan); + var gapError = 0.0; + for (var gap = 0; gap < candidatePath.Count - 1; gap++) + { + var measuredGap = (candidatePath[gap + 1].SampleOffset - candidatePath[gap].SampleOffset) / sampleRate; + gapError += Math.Abs(measuredGap - EventSpacingSeconds); + } - var gapConfidence = 1.0 / (1.0 + gapError / 0.003); - if (gapConfidence < 0.55) - { - continue; - } + var gapConfidence = 1.0 / (1.0 + gapError / 0.003); + if (gapConfidence < 0.55) + { + continue; + } - var energyConfidence = Math.Clamp(candidatePath.Average(candidate => candidate.Energy), 0.0, 1.0); - candidates.Add(new MimirChirpletTimelineAnchor( - (ulong)eventIndex, - firstEvent.StartSeconds, - candidatePath[0].SampleOffset, - gapConfidence * 0.60 + energyConfidence * 0.40, - candidatePath - .Select(candidate => new MimirChirpletSymbolObservation(candidate.SymbolId, candidate.SampleOffset, candidate.Energy)) - .ToArray())); + var energyConfidence = Math.Clamp(candidatePath.Average(candidate => candidate.Energy), 0.0, 1.0); + var binShiftPenalty = 1.0 / (1.0 + Math.Abs(globalBinShift) * 0.10); + candidates.Add(new MimirChirpletTimelineAnchor( + (ulong)eventIndex, + firstEvent.StartSeconds, + candidatePath[0].SampleOffset, + (gapConfidence * 0.60 + energyConfidence * 0.40) * binShiftPenalty, + candidatePath + .Select((candidate, candidateIndex) => new MimirChirpletSymbolObservation( + canonicalSymbols[candidateIndex], + candidate.SampleOffset, + candidate.Energy)) + .ToArray())); + } } } @@ -519,6 +532,33 @@ public sealed class MimirChirpBinTimeline return Recurse(0); } + private static IReadOnlyList GlobalBinShiftHypotheses(MimirChirpBinPathCalibration? calibration) + { + var shifts = new HashSet { 0 }; + if (calibration != null) + { + foreach (var hypothesis in calibration.DelayHypotheses) + { + shifts.Add(Math.Clamp(hypothesis.BinShift, -MaxTimingBinShift, MaxTimingBinShift)); + } + + foreach (var group in calibration.Confusion + .GroupBy(observation => observation.BinShift) + .OrderByDescending(group => group.Count()) + .Take(4)) + { + shifts.Add(Math.Clamp(group.Key, -MaxTimingBinShift, MaxTimingBinShift)); + } + } + + for (var shift = -MaxTimingBinShift; shift <= MaxTimingBinShift; shift++) + { + shifts.Add(shift); + } + + return shifts.OrderBy(shift => Math.Abs(shift)).ThenBy(shift => shift).ToArray(); + } + private static IReadOnlyList SelectCoherentAnchorPath( IReadOnlyList candidates, int sampleRate, diff --git a/src/Mimir.Runtime/Synchronization/MimirRuntimeConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirRuntimeConfiguration.cs index 78ea3a4..550e643 100644 --- a/src/Mimir.Runtime/Synchronization/MimirRuntimeConfiguration.cs +++ b/src/Mimir.Runtime/Synchronization/MimirRuntimeConfiguration.cs @@ -119,6 +119,24 @@ public sealed class MimirRuntimeConfiguration return new MimirNativeIngestStreamSource(ToDescriptor(stream)); } + if (string.Equals(stream.Adapter, "asio", StringComparison.OrdinalIgnoreCase)) + { + if (!OperatingSystem.IsWindows()) + { + throw new PlatformNotSupportedException("The Mimir ASIO stream source requires Windows."); + } + +#pragma warning disable CA1416 + return new MimirAsioStreamSource( + ToDescriptor(stream), + new MimirAsioStreamSourceOptions( + ResolveCommand(stream.Command, configDirectory), + stream.DriverClsid, + stream.SampleRate, + stream.AcceptSourceIds)); +#pragma warning restore CA1416 + } + if (string.Equals(stream.Adapter, "frame-events", StringComparison.OrdinalIgnoreCase) || string.Equals(stream.Adapter, "json-lines", StringComparison.OrdinalIgnoreCase)) { @@ -243,4 +261,8 @@ public sealed class MimirStreamConfig public string[] AcceptSourceIds { get; set; } = []; public int ChunkBytes { get; set; } = 65_536; + + public int SampleRate { get; set; } = 192_000; + + public string DriverClsid { get; set; } = ""; } diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 0ed08af..89fe6b3 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -74,3 +74,6 @@ {"ts": "2026-05-23T20:19:36+01:00", "type": "implementation", "status": "ok", "note": "Replaced the profile-only chirp-bin calibration cut with a usable physical calibration model. The model records per-source usable bands, symbol reliability, expected-vs-observed confusion observations, timing residuals, delay hypotheses, phase summaries, and an adaptive codebook plan. MimirChirpBinTimeline can now decode with learned per-symbol weights, group-delay correction, and persisted delay-hypothesis seeds. BufferSmoke can run --calibrate-chirp-bin-asio-f32 to render/capture/analyze/persist the model, and --analyze-asio-f32 can load it via --calibration. The ASIO native worker now supports --emit-json-blocks so Focusrite ASIO callbacks can feed Mimir.Runtime continuously through the existing frame-event adapter while the native ABI cut is still pending."} {"ts": "2026-05-23T20:31:27+01:00", "type": "implementation", "status": "ok", "note": "Completed the physical chirp-bin calibration loop instead of stopping at summaries. The calibration model now owns a shared emission plan from measured reliable symbols, and MimirRuntime renders chirp-bin calibration with that plan so dead bins are removed from the emitted alphabet and sequence order rises as needed. MimirChirpBinTimeline decodes variable-order codewords from the same emission plan, keeps learned response weights/group-delay correction, records bin-shift in confusion/delay hypotheses, and still preserves the default 32-symbol path when no model is loaded. ASIO callback ingestion remains continuous through the ASIO worker's sample-bearing runtime stream path; the next native ABI cut removes process transport overhead, not clock-domain ownership."} {"ts": "2026-05-23T20:33:00+01:00", "type": "verification", "status": "ok", "note": "Verified continuous runtime ASIO ingestion through the ASIO worker stream path. Using config/mimir-runtime.asio.example.json, BufferSmoke ran for 2 seconds against Focusrite USB ASIO at 192 kHz and ingested 13,324 sample-bearing audio blocks across asio-ch0..asio-ch3, 3,331 blocks per channel, each 192 Float32 frames. This proves loopback and mic channels enter Mimir.Runtime buffers continuously in one ASIO clock domain; the run did not sync because BufferSmoke does not emit speaker calibration audio."} +{"ts": "2026-05-23T21:41:47+01:00", "type": "implementation", "status": "ok", "note": "Implemented the in-process Focusrite ASIO runtime source. native/asio_capture owns the ASIO COM driver/callback buffers and MimirAsioStreamSource drains 192 kHz Float32 blocks into Mimir.Runtime buffers for asio-ch0..asio-ch3 without stdout/base64 transport. The chirp-bin calibration model is now per output/mic path and the active decoder uses learned symbol weights, phase-coherence weights, group-delay correction, and joint global delay/bin-shift hypotheses."} +{"ts": "2026-05-23T21:41:53+01:00", "type": "verification", "status": "ok", "note": "Verified the in-process ASIO runtime ingest and active decoder after the native ASIO cut. dotnet build Mimir.slnx and cmake --build native/asio_capture/build --config Release passed cleanly. BufferSmoke with config/mimir-runtime.asio.example.json ingested 12,196 sample-bearing blocks in two seconds at 192 kHz, with 2,048 buffered 192-frame Float32 blocks on each asio-ch0..asio-ch3 channel. The standalone 192 kHz chirp-bin receiver recovered a 96,000-sample delay with 0.000001 sample error, and calibrated ASIO artifact analysis recovered asio-ch2->asio-ch3 at 0.000 us with 0.812 confidence."} +{"ts": "2026-05-23T21:43:33+01:00", "type": "verification", "status": "ok", "note": "Reran the in-process ASIO ingest after native cleanup changes. BufferSmoke with config/mimir-runtime.asio.example.json ingested 12,204 sample-bearing blocks in two seconds at 192 kHz and retained 2,048 192-frame Float32 blocks for each asio-ch0..asio-ch3 buffer. This remains an ingest proof only because BufferSmoke does not emit calibration audio in that run."} diff --git a/state/map.yaml b/state/map.yaml index 9522d24..99d0d1b 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -113,9 +113,10 @@ accepted_design: evidence for frequency-response normalization. `MimirChirpBinCalibrationModel` now preserves measured usable bands, expected-symbol versus observed-bin confusion observations, timing residuals, delay hypotheses, phase summaries, - and an adaptive codebook plan. The active decoder can consume a persisted - model for learned response weighting, first-order group-delay correction, - and global delay-hypothesis seeding. Runtime chirp-bin emission consumes the + and an adaptive codebook plan per output/mic path. The active decoder can + consume a persisted model for learned response weighting, phase-coherence + weighting, first-order group-delay correction, and joint global + delay/bin-shift hypotheses. Runtime chirp-bin emission consumes the same model's emission plan, so Mimir can render the measured reliable symbol alphabet at a higher de Bruijn order instead of spending code on dead bins. The analyzer keeps calibration profiles @@ -144,10 +145,13 @@ accepted_design: pairwise timing against loopback in the stored artifact, but decoded mic channels now leave calibration models; physical input 1 produced two reliable symbols and a different strongest measured band set. - `config/mimir-runtime.asio.example.json` proves continuous ASIO runtime - ingest: a two-second BufferSmoke run at 192 kHz ingested 13,324 sample-bearing - blocks across `asio-ch0` through `asio-ch3`, all from the Focusrite ASIO - callback stream. + `native/asio_capture` and `MimirAsioStreamSource` are now the runtime + Scarlett path: Focusrite ASIO callbacks feed sample-bearing 192 kHz Float32 + blocks into `Mimir.Runtime` in process. `config/mimir-runtime.asio.example.json` + proves continuous ASIO runtime ingest: a two-second BufferSmoke run at + 192 kHz ingested more than 12,000 sample-bearing blocks across `asio-ch0` + through `asio-ch3` and retained 2,048 blocks per channel, all from the + Focusrite ASIO callback stream. Acoustic robustness remains open. The old arbitrary-chirplet matcher is a diagnostic/reference artifact only, not the runtime hot path. @@ -164,9 +168,10 @@ accepted_design: active_subgoals: - Implement the first concrete Leap direct capture driver and measure cadence. - Add remaining local camera drivers through `IMimirVideoCaptureDriver`. - - Add native audio capture workers for mic, loopback, and network audio feeds. - - Feed verified Focusrite ASIO input callbacks into `Mimir.Runtime`; WASAPI - remains a diagnostic witness. + - Use the verified in-process Focusrite ASIO source for local Scarlett mic and + loopback ingest; WASAPI remains a diagnostic witness. + - Add network audio feed producers that enter the same runtime clock/rebuffer + machinery instead of becoming independent authorities. - Bind Fensalir UI to stream health, buffer depth, source timestamps, settings, and output management. - Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output @@ -250,9 +255,7 @@ current_status: the Leap USB neighborhood does not reproduce the Focusrite-induced collapse. next_action: > Use chirp-bin calibration models from real microphones to tune the active - codebook/bands/weighting, then replace the ASIO JSON-block process bridge - with a native ABI callback source and add the synchronization - actuator. + codebook/bands/weighting, then add the synchronization actuator. Current app testing proves Fensalir WASAPI output wakes Scarlett speaker loopback, live mic buffers stay populated, runtime sync analysis now runs as a bounded rotating service without UI/telemetry re-entry, and the continuous @@ -260,7 +263,7 @@ current_status: invariant is code-valid triplets producing stable canonical anchors and clock fits, so the actuator can use decoded time instead of filtered lag guesses. Replace - the diagnostic JSON-line process bridge with direct native capture drivers that push buffer handles into + the remaining diagnostic JSON-line process bridges with direct native capture drivers that push buffer handles into Mimir.Runtime. Mimir.BufferSmoke still proves live buffers and sync reports for Scarlett loopback, Focusrite mic, Kiyo mic, and Kiyo Pro mic when calibration chirplets are playing. Treat PS3 Eye audio as diff --git a/state/scratch.md b/state/scratch.md index 4fce9ee..34a6d9d 100644 --- a/state/scratch.md +++ b/state/scratch.md @@ -6,11 +6,11 @@ No active scratch subgoal. ## Notes -- Viable stream app contract promoted to docs/map/evidence: Aquarium hosts one +- Viable stream app contract promoted to docs/map/evidence: Fensalir hosts one five-second in-memory runtime, local/networked feeds append as producers, and OBS receives synchronized program outputs. - Mimir app scaffold now lives in `Mimir.slnx` with `src/Mimir.App` and - `src/Mimir.Runtime`; Aquarium Engine remains an external project reference. + `src/Mimir.Runtime`; Fensalir remains an external project reference. - Synchronization ownership started moving into `Mimir.Runtime`: hub, stream descriptors, per-stream rolling buffers, settings, and source adapter seam are in C# now; concrete capture adapters are next.