Deepen Perfect Machine study index

This commit is contained in:
Meta Vens 2026-05-24 01:56:15 +01:00
commit 8b151294e7
39 changed files with 5005 additions and 4 deletions

View file

@ -15,8 +15,14 @@ Read this with:
the long-form design argument.
- `research/perfect-machine-study-2026-05-23/index.md` for the study directory
and sample-code index.
- `research/perfect-machine-study-2026-05-23/reading-guide.md` for task-oriented
navigation through the study.
- `research/perfect-machine-study-2026-05-23/optimization-ledger.md` for hot
loop candidates.
- `research/perfect-machine-study-2026-05-23/fensalir-integration-map.md` for
the Mimir/Fensalir ownership cut.
- `research/perfect-machine-study-2026-05-23/calibration-session-spec.md` for
the planned physical-path calibration command.
- `research/perfect-machine-study-2026-05-23/samples/README.md` for sample
implementation sketches.

View file

@ -0,0 +1,172 @@
# Acoustic Field Models
## Purpose
Mimir's audio goal is not just "delay-align microphones." Delay alignment is
the entry fee. The larger target is a live volumetric audio field: where sound
is, how the room moves it, which mic/speaker paths are trustworthy, and how to
emit OBS-facing stems or a spatial bed that are better than raw source feeds.
## The Three Layers
### 1. Clock And Path Layer
Questions:
- Where is this sample on the canonical timeline?
- How much sample-rate offset is present?
- What delay, phase, and group-delay shape does this path impose?
Tools:
- active chirp-bin/codebook anchors;
- passive GCC-PHAT on program audio;
- loopback as emitter-side timing authority;
- Farrow/ASRC actuator.
Output:
- delay and SRO state;
- per-band magnitude and phase response;
- confidence and residuals.
This layer must work before any field reconstruction deserves trust.
### 2. Source/Geometry Layer
Questions:
- Which source positions explain the synchronized microphones?
- Which reflections are direct path versus room response?
- What evidence should become a stable field claim?
Tools:
- TDOA;
- SRP-PHAT;
- beamforming;
- known mic geometry;
- visual constraints from cameras/Leap when available.
Output:
- source position candidates;
- source confidence;
- reflection/room residuals;
- constraints for Fensalir's temporal evidence reservoir.
### 3. Field Reconstruction Layer
Questions:
- What pressure/energy field exists in the room?
- Can the field be represented as sources, ambisonics, equivalent sources, or a
hybrid?
- What should OBS get: isolated stems, spatial bed, diagnostics, or all three?
Tools:
- ambisonics/spherical harmonics for compact field representation where geometry
fits;
- equivalent source method / near-field acoustic holography when treating sound
as sparse physical sources;
- compressed sensing or Bayesian sparse recovery when source count is low and
mic count is limited;
- learned priors only after deterministic models fail cleanly.
Output:
- aligned stems;
- sound-source tracks;
- spatial bed/ambisonic representation;
- visualizable acoustic field constraints.
## Why Six Mics Are Not An Eigenmike
A commercial spherical array might have dozens of capsules in known geometry.
Mimir has an irregular consumer/pro-audio constellation: Scarlett mic, camera
mics, possibly phone/Raven mics, and loopback. That pushes the machine away from
pure textbook ambisonics and toward an evidence model:
- synchronize first;
- calibrate each path;
- infer source constraints;
- reconstruct only what the geometry supports.
If the geometry cannot support a high-order field, the honest output is lower
order or sparse-source estimates with uncertainty. Pretending otherwise would
be the audio equivalent of drawing a ruler on fog and calling it engineering.
## Candidate Models
### Ambisonic Fit
Use when:
- mic geometry can support a stable low-order spherical harmonic fit;
- the goal is a spatial bed, not precise source localization;
- the scene is diffuse or ambient.
Risk:
- irregular sparse mics produce unstable coefficients;
- near-field sources and room boundaries violate simple assumptions.
### Beamforming / SRP-PHAT
Use when:
- seeking source direction/position;
- synchronized mic geometry is known;
- direct-path energy exists.
Risk:
- reflections can produce plausible false peaks;
- small baseline limits low-frequency localization.
### Equivalent Source / Sparse Acoustic Holography
Use when:
- sources can be represented sparsely;
- a grid or source dictionary is acceptable;
- response calibration is available.
Risk:
- inverse problems are ill-conditioned;
- dense grids are expensive;
- sparse recovery can hallucinate when the measurement model is wrong.
### Hybrid Evidence Field
Use when:
- Mimir has mixed-quality sensors;
- visual evidence can constrain likely source regions;
- acoustic output should remain honest about uncertainty.
Shape:
- clock/path layer feeds calibrated observations;
- localization layer creates candidates;
- Fensalir temporal reservoir stabilizes candidates across time;
- DSP/output layer uses stable tracks as weights, not commandments.
This is the likely Mimir path.
## Practical First Milestone
Build one static acoustic source proof:
1. Record loopback and all live mics while emitting calibrated active witness.
2. Recover canonical anchors on loopback and at least one physical mic.
3. Estimate per-mic delay/SRO/response.
4. Use known speaker/mic geometry to form a TDOA/SRP-PHAT position hypothesis.
5. Lower that as an `AquariumAcousticConstraint`.
6. Display/log the constraint without letting it alter program audio yet.
The proof is deliberately modest. It forces the field machine to earn geometry
from timing instead of leaping straight into theatrical volumetric fog.

View file

@ -6,6 +6,9 @@
dedicated pass mapping, researching, distilling, ruminating, and writing
sample options.
- Active pass start recorded locally: `2026-05-23T23:41:19+01:00`.
- Corrected no-shortcut pass start recorded locally after the final instruction:
`2026-05-23T23:54:42.9041588+01:00`. Do not treat this directory as complete
until at least `2026-05-24T01:54:42.9041588+01:00`.
## Prime Cut
@ -210,4 +213,3 @@ Reason:
that is how bandwidth dies in a formal suit.
- Treating 192 kHz as magic: it improves timing granularity and ultrasonic
options only where transducers and path response actually preserve signal.

View file

@ -0,0 +1,274 @@
# Benchmark Plan
## Purpose
This is the measurement spine for future implementation passes. It exists to
keep the machine honest: no hot-path change earns production status because it
sounds clever. It earns it by moving measured latency, jitter, throughput,
allocation, cache pressure, confidence, or reconstruction quality in the right
direction.
## Global Rules
- Measure wall-clock throughput and canonical timing error separately.
- Keep synthetic, artifact, and live-device runs distinct.
- Always record sample rate, buffer size, device clock, channel mapping, and
mode (`passive`, `chirp-only`, `hybrid`).
- Allocation-free claims require runtime allocation counters, not code vibes.
- A passing self-test is a unit proof, not a physical proof.
- A physical proof without artifact retention is only a campfire story.
## Audio Synchronization Benchmarks
### Synthetic Chirp-Bin Decoder
Goal: prove canonical-time recovery without acoustic path damage.
Inputs:
- generated chirp-bin timeline;
- known integer delay;
- known fractional delay;
- optional sample-rate offset;
- optional colored noise and band dropouts.
Metrics:
- decoded anchors per second;
- false anchors per second;
- delay error in microseconds;
- fractional sample residual;
- SRO estimate error in ppm;
- allocations per second;
- CPU cycles or elapsed time per decoded second.
Current command family:
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --standalone-chirp-bin-self-test --sample-rate 192000 --delay-samples 96000
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 192000
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --hybrid-sync-self-test --sample-rate 192000
```
Missing benchmark:
- a loop over SNR, reliable-bin count, fractional delay, and SRO;
- output JSON/CSV artifact;
- CI-friendly threshold for the synthetic path.
### Scarlett Loopback Artifact
Goal: prove the analyzer survives the real ASIO clock and converter path when
the signal remains electrical/loopback-clean.
Inputs:
- rendered mono float chirp-bin witness;
- ASIO output through Scarlett;
- ASIO capture of loopback channels;
- optional persisted calibration profile.
Metrics:
- loopback channel-to-channel delay;
- confidence;
- anchor count;
- response matrix diagonal strength;
- per-band phase/group-delay residual;
- callback block jitter;
- dropped/late ASIO blocks.
Current command family:
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --render-chirp-bin-f32 --sample-rate 192000 --seconds 6 --output .\artifacts\chirp-bin.f32
.\native\probes\asio_audio_cadence\build\Release\asio_audio_cadence.exe --play-f32-mono .\artifacts\chirp-bin.f32 --record-f32-interleaved .\artifacts\scarlett.f32 --sample-rate 192000 --seconds 6
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --analyze-asio-f32 .\artifacts\scarlett.f32 --sample-rate 192000
```
Missing benchmark:
- stable artifact directory naming;
- automatic device/channel manifest capture;
- repeated run variance.
### Meatspace Mic Path
Goal: prove acoustic propagation, monitor response, mic response, room
reflections, and gain staging still permit deterministic code-valid anchors.
Inputs:
- Scarlett output through monitors;
- Scarlett mic and camera mics;
- calibration profile generated from the same room/output/mic path;
- optional passive music bed.
Metrics:
- usable bands per mic;
- confusion matrix entropy;
- strongest alias bins;
- group-delay slope by bin;
- decoded anchors per second;
- rejected anchors and why;
- global delay/bin-shift hypothesis likelihood;
- time-to-lock and lock-hold duration;
- SRO drift estimate stability.
Acceptance:
- a calibrated physical mic should produce stable canonical anchors within the
five-second rolling buffer;
- delay should be reported in microseconds with residuals low enough for a
fractional-delay actuator to matter;
- dead bins must be excluded or downweighted by the emitted codebook, not merely
filtered after failure.
## Audio Field Benchmarks
### Passive Program-Audio Alignment
Goal: use loopback music as continuous timing evidence when active chirps are
unnecessary.
Metrics:
- PHAT peak sharpness;
- delay confidence over time;
- failure rate on low-frequency-only material;
- behavior during silence;
- agreement with active chirp anchors.
Cut line:
- passive alignment is advisory unless it agrees with active/codebook anchors or
repeated state earns confidence.
### Acoustic Source Localization
Goal: turn synchronized mics into spatial estimates rather than just aligned
waveforms.
Inputs:
- known source positions if available;
- synthetic room model for baseline;
- live mic geometry.
Metrics:
- TDOA residual;
- SRP-PHAT peak width;
- source position error;
- update rate;
- ambiguity count.
First proof:
- one static speaker position recovered from loopback/chirp evidence and mic
geometry.
## Video Capture Benchmarks
### Native Camera Cadence
Goal: prove each camera's raw driver path maintains sustained cadence without
framework buffering lies.
Metrics:
- delivered fps after warm-up;
- frame interval p50/p95/p99;
- missing sequence count;
- device timestamp monotonicity;
- CPU time in callback;
- transfer size and USB topology.
Known baselines:
- Leap full-height stereo IR: roughly 110 fps when the USB neighborhood is not
polluted by the Focusrite.
- PS3 Eyes: roughly 187 fps at 320x240 and roughly 58-60 fps at 640x480.
- Kiyo Pro: currently about 25 fps despite 60 fps advertisement and HighSpeed
negotiation.
- Kiyo basic: about 30 fps in current usable modes.
### Multi-Camera Contention
Goal: discover whether frame loss is USB topology, driver scheduling, CPU copy,
or runtime ingest pressure.
Metrics:
- per-device fps with all sources open;
- callback jitter correlation across devices;
- host controller/root hub mapping;
- frame-size bandwidth estimate versus observed cadence;
- runtime buffer enqueue cost.
First implementation proof:
- all local cameras feed `Mimir.Runtime` descriptors in one app run with no
JSON process bridge.
## Fensalir Lowering Benchmarks
Goal: prove Mimir can create Fensalir contract frames every render tick without
turning the app into a garbage factory.
Metrics:
- lowering time per frame;
- managed allocations per frame;
- number of sensor descriptors lowered;
- D3D12 upload/import wait time once connected;
- stale observation count.
Acceptance:
- lowering is pure mapping over current buffer snapshots;
- analysis/capture does not run from the render tick;
- UI reads cached state only.
## Gaussian Splat / Sensor Fusion Benchmarks
Goal: decide which visual fusion representation deserves production attention.
Metrics:
- observations accepted per second;
- splat claims updated per second;
- reprojection error;
- track lifetime;
- sort/tile/render cost;
- memory footprint per claim;
- drift under camera motion.
Candidate approaches:
- sparse feature claims lowered into Fensalir temporal evidence reservoir;
- live Gaussian claims updated from synchronized camera features;
- hybrid point/splat field where Mimir supplies timing and Fensalir owns
rendering/reuse.
## Reporting Format
Every serious benchmark should write one manifest next to artifacts:
```json
{
"machine": "Starfire",
"timestampUtc": "2026-05-24T00:00:00Z",
"mode": "chirp-only",
"sampleRate": 192000,
"bufferFrames": 192,
"sources": [],
"artifacts": [],
"metrics": {},
"notes": []
}
```
The manifest is not the hot path. It is the lab notebook.

View file

@ -0,0 +1,179 @@
# Calibration Session Spec
## Purpose
The calibration command should be the machine that turns a real room into a
path model. It should not be a pile of one-off artifact commands that only the
current operator remembers how to arrange.
## Objective
Emit a known active witness, capture loopback and every microphone in the same
clock domain where possible, compute per-output/mic response/confusion/delay
models, persist them, and immediately prove that the live decoder improves when
those models are loaded.
## Inputs
- output path id;
- ASIO driver id / source manifest;
- sample rate;
- duration;
- emission mode (`chirp-only`, `hybrid-watermark`, `silent-passive-baseline`);
- candidate symbol plan;
- mic geometry file if available;
- artifact directory.
## Outputs
- raw capture artifact;
- rendered witness artifact;
- manifest with device/channel/sample-rate/buffer-size/gain notes;
- `MimirChirpBinCalibrationModel`;
- per-path usable-band table;
- confusion matrix;
- timing residual table;
- group-delay estimate;
- recommended codebook plan;
- before/after decode report.
## Command Shape
Future production command:
```powershell
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- `
--calibration-session `
--config .\config\mimir-runtime.asio.json `
--output-path scarlett-main-out `
--seconds 30 `
--sample-rate 192000 `
--artifact-dir .\calibration\runs\2026-05-24-starfire
```
This can start as `BufferSmoke`, but the final owner should be a runtime command
or app action that uses the same ASIO/native path as production.
## Session Phases
### 1. Preflight
- enumerate ASIO inputs/outputs;
- verify requested sample rate and buffer size;
- verify loopback channels are present;
- verify input channels are non-silent when expected;
- record channel names and source ids;
- load existing calibration for comparison.
Fail fast if loopback is absent. Without loopback, the session can still record
diagnostics, but it cannot become canonical calibration.
### 2. Passive Baseline
Capture room/program audio without active witness for a short window.
Purpose:
- noise floor;
- current music/program spectrum;
- passive correlation strength;
- mic gain sanity.
### 3. Active Emission
Render a deterministic chirp-bin/watermark witness using the candidate symbol
plan. Capture loopback and all inputs.
Important:
- event ids and schedule seed go into the manifest;
- emitted codebook plan goes into the artifact;
- output gain should be recorded manually or through driver state if possible.
### 4. Decode And Score
For each input:
- decode loopback anchors;
- decode mic anchors;
- compute matched anchor delays;
- compute expected-symbol versus observed-bin matrix;
- compute timing residuals;
- estimate magnitude response;
- estimate phase/group delay where phase evidence is stable;
- generate delay/bin-shift hypotheses.
### 5. Plan Adaptation
Across all selected mics:
- identify reliable bins;
- identify dead/aliased bins;
- choose reduced alphabet;
- choose higher de Bruijn order if necessary;
- estimate expected time-to-unique-lock;
- write recommended emission plan.
### 6. Replay Proof
Reload the persisted model and decode the same artifact:
- unweighted decode report;
- weighted/adapted decode report;
- anchor gain/loss;
- confidence gain/loss;
- delay residual gain/loss.
If the calibrated model does not improve the capture, the command should say so
plainly. A calibration file that cannot improve the decoder is just decorative
paperwork with a badge.
## Persisted Model Shape
Minimum:
```json
{
"schema": "gamecult.mimir.chirp_bin_calibration.v1",
"createdUtc": "2026-05-24T00:00:00Z",
"sampleRate": 192000,
"outputPathId": "scarlett-main-out",
"emissionPlan": {},
"paths": [
{
"sourceId": "asio-ch0",
"usableBands": [],
"symbols": [],
"delayHypotheses": [],
"phaseModel": {},
"groupDelayModel": {},
"confidence": 0.0
}
]
}
```
## Acceptance Criteria
- loopback decodes confidently;
- at least one physical mic produces code-valid anchors;
- calibration identifies reliable and unreliable bands;
- adapted emission plan changes the symbol set when the original bands are bad;
- loaded calibration affects likelihood before codebook solving;
- replay proof is stored with the model.
## Failure Records
A failed calibration is still useful if it records:
- no loopback;
- silent mic;
- clipping;
- insufficient high-frequency response;
- strong alias/confusion;
- group delay too unstable;
- active witness too quiet/loud;
- room reflection dominated direct path.
Do not delete failed runs unless they are accidental trash. They are evidence.

View file

@ -0,0 +1,169 @@
# Chirplet Transform And Controlled Chirp Codebook Deep Dive
## Core Distinction
There are two different machines that share similar math:
1. A general chirplet transform asks: "what chirp-like atoms exist in this
signal?"
2. Mimir's active receiver asks: "which known emitted atom arrived when, through
which distorted path?"
The second machine is much cheaper because the codebook, slopes, timing plan,
and symbol grammar are known.
## Uses For Mimir
### Timing Synchronization
Each decoded code-valid chirp tuple can become a canonical timeline anchor.
With enough anchors, fit:
- offset;
- sample-rate offset;
- confidence;
- jitter/residual.
### Frequency Response Normalization
Each classified chirp yields observed energy across bins. Aggregated over time:
- reliable bands;
- dead bands;
- alias/confusion bands;
- path coloration.
### Phase / Group Delay
If phase is coherent enough, dechirped complex bin responses can estimate:
- per-bin phase;
- phase slope across frequency;
- group delay correction;
- reflection/path stability.
### Localization
Known emitted time plus observed receiver time gives a distance constraint when
the source and receiver clocks are modeled. Multiple receivers turn this into a
position estimate. Multiple emitters/receivers turn it into room calibration.
### Watermarking
Hybrid mode can embed low-gain coded chirps under program audio. The watermark
should be:
- sparse enough not to annoy;
- deterministic enough to identify time;
- adaptive enough to avoid dead acoustic bands;
- continuous enough to track drift.
## Decoder Pipeline
```mermaid
flowchart TD
PCM["PCM stream"] --> Energy["streaming energy/onset proposal"]
Energy --> Cand["candidate event windows"]
Cand --> Dechirp["dechirp with known slope"]
Dechirp --> Bins["bin / Goertzel / FFT score"]
Bins --> Likelihood["calibration-weighted likelihood"]
Likelihood --> Grammar["codebook grammar / de Bruijn tuples"]
Grammar --> Clock["clock fit"]
Clock --> Model["response + delay model"]
```
## What The Current Code Has
- Fixed-slope chirp-bin renderer.
- De Bruijn codebook plan.
- Energy/onset proposals.
- Dechirp/bin scoring.
- Ambiguity candidates.
- Global bin-shift hypotheses.
- Clock fit and source-offset refinement.
- Calibration model with response/confusion/delay/phase/adaptive codebook.
## What Is Missing
- Streaming candidate state.
- Calibration-weighted likelihood before path selection as the default physical
path, not just optional model support.
- Better group-delay model.
- Reduced reliable-bin physical proof.
- Native/SIMD/GPU scoring path after correctness is proven.
## Algorithm Options
### Option A: Goertzel Bank
Best when:
- bin count is small;
- sample window is short;
- CPU SIMD is available;
- frequencies are fixed.
Pros:
- No FFT plan overhead.
- Easy to weight per bin.
- Simple to port to AVX2.
Cons:
- Repeats work per bin.
- Less attractive if bin count/window count grows.
### Option B: Dechirp + FFT
Best when:
- many bins;
- many candidates;
- batching is possible.
Pros:
- Canonical CSS receiver shape.
- Good fit for GPU/native FFT batches.
Cons:
- FFT overhead and memory motion can dominate small cases.
### Option C: Chirp Z Transform
Best when:
- only a narrow frequency band is needed at high resolution;
- arbitrary frequency grid matters.
Pros:
- Efficient zoomed spectral analysis.
Cons:
- More complex than current fixed bins.
- Might be premature unless physical data shows bin boundaries are the problem.
### Option D: Matched Chirplet Atoms
Best when:
- emitted atoms are varied and not reducible to one common slope.
Pros:
- General.
Cons:
- Too expensive and too unconstrained for the current controlled beacon.
## Implementation Rule
Build the controlled CSS-like receiver first. Pull in full chirplet machinery
only when the codebook stops being controlled or when a research proof shows a
specific invariant the controlled receiver cannot protect.

View file

@ -0,0 +1,174 @@
# Current Hotspot Audit
## What This Is
This is a static audit of the current repo for likely allocation, scan, and
driver-pressure hotspots. It is not a profiler output. Treat it as a list of
places to benchmark once the next implementation cut starts.
## High-Risk Runtime Hotspots
### `MimirAudioSynchronizationAnalyzer.Analyze`
Why it is hot:
- Called repeatedly by `MimirSynchronizationHub`.
- Extracts windows from rolling buffers.
- Converts PCM formats.
- Resamples candidate windows.
- Invokes passive or active decode.
Risk shape:
- `ExtractMonoWindow` builds a `List<float>` and returns `ToArray`.
- Candidate windows can be re-extracted on every analysis step.
- Resampling allocates a new output array.
- Reports/traces allocate lists and records.
Coherent next cut:
- Add an analyzer scratch/context object owning reusable buffers.
- Add streaming active decoder state so repeated windows are not rescanned.
- Cache reference extracted window/spectrum per analysis edge.
### `MimirChirpBinTimeline.DecodeStreamWindow`
Why it is hot:
- Current active path for chirp-only/hybrid sync.
- Runs event proposal, classification, anchor decode, clock fit.
Risk shape:
- `BuildWindowEnergyTrace` allocates prefix/output arrays.
- `DetectFrames` sorts proposals by energy.
- `ScoreBins` scores every symbol for each candidate.
- `DecodeAnchors` creates candidate windows and path arrays.
- LINQ/`ToArray` appears in several inner-ish paths.
Coherent next cut:
- Split into streaming proposal state and stateless symbol scorer.
- Replace LINQ in candidate/code paths with bounded arrays/spans once behavior
is frozen.
- Move calibration likelihood into the scoring stage before candidate pruning.
### `MimirPassiveAudioSynchronizationEstimator.Estimate`
Why it is hot:
- Passive/hybrid mode can compare multiple sources.
- FFT arrays allocate per estimate.
Risk shape:
- `Complex[]` reference/candidate arrays per call.
- Hann/preemphasis recomputed per call.
- Reference FFT repeated for each candidate.
Coherent next cut:
- Scratch object with reusable complex buffers.
- Precomputed window.
- Reference spectrum cache per edge.
- Limit lag search by known physical/network horizon.
### `MimirRuntime.QueueCalibrationTimeline`
Why it is hot-ish:
- Renders/encodes active witness audio inside runtime update.
Risk shape:
- Allocates PCM bytes and base64 every batch.
- Renders sine chirps with scalar `Math.Sin`.
Coherent next cut:
- For production, move witness generation to a reusable native/DSP oscillator or
pre-rendered segment ring. Keep runtime policy, move audio hot work.
## Native Hotspots
### `native/asio_capture/pushInputBlocks`
Why it is hot:
- Runs from ASIO callback.
- Copies every input channel into a queued `std::vector<float>`.
Risk shape:
- Allocates/copies per channel block.
- Uses mutex-protected deque.
Current acceptability:
- Fine for proof and likely fine for four Scarlett channels at 192 kHz in short
term.
Coherent next cut:
- Preallocated block pool or SPSC ring if profiling shows callback pressure.
### `native/probes/ks_camera_cadence/measureCandidate`
Why it is hot:
- Direct KS async read queue for camera cadence.
Risk shape:
- Diagnostic stdout and optional JSON events.
- Probe code owns too much CLI behavior for production.
Coherent next cut:
- Extract the async read queue shape into a direct driver worker, minus stdout
and measurement ceremony.
### `native/reservoir/RollingReservoir`
Why it matters:
- Intended lower shared-edge retention authority.
Risk shape:
- Current Rust `VecDeque` and typed-view iteration are conceptually clean but
not necessarily the final cache-optimal layout.
Coherent next cut:
- Keep invariant. Replace storage only when Fensalir/Faust integration proves
the need.
## Allocation Smells Found By Static Scan
Likely production-relevant:
- `MimirChirpBinTimeline`: repeated `List<>`, `ToArray`, sorting, grouping in
decode path.
- `MimirAudioSynchronizationAnalyzer`: window extraction and resampling arrays.
- `MimirPassiveAudioSynchronizationEstimator`: per-call FFT arrays.
- `MimirAsioStreamSource`: per-block byte arrays after native read.
- `MimirRollingStreamBuffer.Snapshot`: full array snapshot for readers.
Mostly diagnostic:
- `Mimir.BufferSmoke` allocations.
- Native probe stdout/JSON allocations.
- Bridge scripts.
## Rule For Next Optimization
Do not optimize the whole repo. Pick one measured hot path:
1. physical chirp-bin decode;
2. passive FFT reuse;
3. ASIO queue pressure;
4. camera driver handoff.
Then make one ownership-improving cut and benchmark before widening.

View file

@ -0,0 +1,214 @@
# Data Dictionary
## Purpose
This dictionary names the data objects that keep recurring across code, docs,
samples, and planned commands.
## Stream Descriptor
Identity for one source stream.
Fields:
- source id;
- kind (`audio`, `video`, etc.);
- origin (`local`, `network`, `process`, `native`);
- human label;
- optional device/config metadata.
Invariant:
- one descriptor maps to one rolling buffer.
## Stream Sample
One runtime sample envelope.
Fields:
- descriptor/source id;
- timestamp;
- sequence;
- audio block descriptor or video frame descriptor;
- optional payload bytes/handle.
Invariant:
- samples are indexed by runtime buffers; they are not long-term history.
## Audio Block Descriptor
Metadata for a block of audio.
Fields:
- sample rate;
- channel count;
- sample format;
- frame count;
- timestamp;
- optional native handle.
Invariant:
- production payload should move toward native handles, not managed byte arrays.
## Video Frame Descriptor
Metadata for one camera frame.
Fields:
- width;
- height;
- pixel format;
- stride;
- device timestamp;
- byte length;
- optional native/GPU handle.
Invariant:
- frame descriptor must preserve enough information for Fensalir to import or
upload without guessing.
## Chirp Symbol
One emitted active-witness event.
Fields:
- symbol id;
- frequency bin;
- chirp slope/window/duration;
- expected event time;
- optional rhythmic/gap evidence.
Invariant:
- symbol shape is controlled by the emitter and known by the receiver.
## Chirp Anchor
A decoded observation that maps a local sample offset to a canonical event.
Fields:
- event index;
- symbol id;
- local sample offset;
- canonical time;
- confidence;
- residual/error;
- bin shift hypothesis.
Invariant:
- anchors are the bridge between local sample time and canonical time.
## Clock Fit
Affine map between local samples and canonical time.
Fields:
- source offset samples;
- effective sample rate;
- confidence;
- anchor count;
- mean absolute error.
Invariant:
- clock fit is evidence state, not a sample-moving actuator.
## Calibration Path
Per-output/mic model.
Fields:
- output path id;
- source/mic id;
- sample rate;
- usable bands;
- symbol calibrations;
- confusion observations;
- delay hypotheses;
- phase/group-delay summaries;
- codebook plan.
Invariant:
- calibration alters likelihood and emission planning before decode failure
happens.
## Sync Report
Analyzer output for one source relative to reference.
Fields:
- source id;
- delay samples;
- delay milliseconds/microseconds;
- fractional delay;
- confidence;
- band response;
- decode trace.
Invariant:
- report describes measurement; DSP decides how to move audio.
## Sync State
Smoothed runtime belief.
Fields:
- latest delay;
- smoothed delay;
- confidence;
- delay slope / SRO ppm;
- per-band evidence.
Invariant:
- UI reads this; analyzers write it; render tick does not recompute it.
## Acoustic Constraint
Fensalir-facing spatial/timing evidence.
Fields:
- constraint id;
- kind;
- position/velocity/radius;
- confidence;
- timestamp;
- timing uncertainty.
Invariant:
- raw PCM stays out of Fensalir contract frames.
## Visual Observation
Fensalir-facing camera evidence.
Fields:
- source id;
- native/shared texture handle or payload handle;
- pixel format;
- dimensions;
- calibration id;
- canonical timestamp.
Invariant:
- Fensalir owns GPU lifetime after import/lowering.

View file

@ -0,0 +1,196 @@
# Decoder Architecture Options
## Purpose
The active decoder must eventually be small enough to run on a phone, robust
enough for meatspace microphones, and cheap enough not to eat a CPU core per
stream. This file separates the receiver shapes so the next cut can be made on
invariants instead of frustration.
## Current Runtime Shape
`MimirChirpBinTimeline.DecodeStreamWindow` currently does the whole job for a
bounded PCM window:
1. build an energy trace;
2. find candidate event windows;
3. classify each window with chirp-bin scoring;
4. preserve ambiguity candidates;
5. build code-valid anchors from the de Bruijn schedule;
6. search delay/bin-shift hypotheses;
7. fit an affine source clock;
8. emit calibration response/confusion evidence.
This is coherent as a proof machine. It is not yet the final streaming receiver.
## Option A: Streaming Scalar Receiver
### Shape
- Maintain a rolling PCM ring.
- Maintain an energy/proposal ring keyed by absolute sample index.
- When a proposal crosses threshold, score only the local candidate window.
- Keep the top K symbol hypotheses per proposal.
- Run a small codebook trellis over recent candidates.
- Update clock fit incrementally.
### Why It Fits
The emitter controls the symbol vocabulary and schedule. The receiver does not
need to search an unconstrained chirplet dictionary. It needs to notice likely
events and map them through a codebook.
### Risks
- Scalar scoring may still be too expensive at 192 kHz across many mics.
- Proposal thresholds can hide weak physical events if calibration is not used
before rejection.
- Trellis state must preserve ambiguity, not collapse to one wrong symbol.
### Promote When
It matches the current window decoder on synthetic and artifact captures while
reducing repeated work by at least an order of magnitude.
## Option B: SIMD Dechirp + Goertzel Receiver
### Shape
- Precompute symbol reference chirps or dechirp oscillators.
- Process candidate windows in AVX2/FMA blocks.
- Accumulate real/imag Goertzel-style bin responses.
- Produce dense scores for the codebook trellis.
### Why It Fits
The hot math is mostly multiply-add and reduction over short windows. That is
exactly where SIMD is boring and useful. Boring and useful beats clever and
fragile here.
### Risks
- Small windows and few bins may be memory-bound or dispatch-bound rather than
arithmetic-bound.
- Managed/native transition cost can erase wins if called per candidate instead
of in batches.
- Alignment and denormal behavior need explicit treatment.
### Promote When
Profiling shows scalar C# scoring dominates, and a batched native scorer is
faster after interop overhead is included.
## Option C: Dechirp + FFT/CZT Receiver
### Shape
- Dechirp candidate windows against known slope.
- Use FFT or chirp-Z transform to estimate symbol bin/frequency offset.
- Feed bin likelihoods into codebook path selection.
### Why It Fits
CSS/LoRa receivers validate this family. It also handles frequency offset and
bin interpolation naturally when the symbol set is dense.
### Risks
- FFT setup and memory traffic can dominate for 32 bins and small windows.
- A full FFT is wasteful if only a small fixed bin bank is needed.
- CZT is attractive for narrowband zooming, but more complex than necessary
until measured bin interpolation is the problem.
### Promote When
The codebook expands, frequency offset becomes first-order, or batched GPU/native
FFT plans beat SIMD Goertzel on actual candidate batches.
## Option D: Full Chirplet Transform / Fast Chirplet Transform
### Shape
- Search a chirplet parameter space over time, center frequency, duration,
chirp rate, and scale.
- Use fast chirplet transform variants or maximum chirplet decomposition to
locate arbitrary chirp-like atoms.
### Why It Fits
It is the canonical general tool when the emitted atoms are unknown or when we
are analyzing natural signals.
### Why It Does Not Fit The Hot Path Yet
Mimir controls the emitter. Searching a large affine dictionary to rediscover a
known alphabet is buying a forklift to move a teacup. Good forklift, wrong
room.
### Promote When
Only if the controlled-codebook receiver cannot survive physical paths even
after response weighting, group-delay correction, and codebook adaptation.
## Option E: Learned / Adaptive Receiver
### Shape
- Train or fit a compact classifier on calibration captures.
- Use learned symbol likelihood surfaces and delay priors.
- Keep codebook trellis deterministic after likelihood generation.
### Why It Fits
Physical paths produce systematic distortion: dead bands, aliases, reflections,
group delay, and mic-specific response. A learned likelihood surface may capture
what hand weighting misses.
### Risks
- A learned classifier can become an uninspectable compensator.
- Training data may overfit one room/gain setting.
- It must not replace the deterministic schedule map.
### Promote When
Calibration-weighted deterministic decoding fails in repeatable, explainable
ways that a compact trained likelihood model can resolve on held-out captures.
## Recommended Cut
Build Option A first, but design the scoring call as a replaceable batched
kernel so Option B or C can occupy it without changing ownership:
```text
PCM ring -> proposal ring -> batched score kernel -> ambiguity trellis -> clock fit
```
Only the score kernel should vary. The codebook, schedule, calibration model,
and clock fit remain the receiver's authority.
## Phone Receiver Shape
A phone should need:
- source codebook id;
- current schedule seed/epoch;
- sample rate estimate;
- rolling PCM buffer;
- compact path calibration if known.
It should not need:
- Starfire round trips for each event;
- Raven transport timing;
- Fensalir GPU state;
- OBS state.
The phone receiver emits:
- decoded anchor observations;
- fitted local clock to canonical time;
- confidence/residuals;
- response/confusion profile.
That profile can come home later. The phone does not need to be trusted as a
clock authority; it is a self-locating sensor.

View file

@ -0,0 +1,150 @@
# Distributed Receiver Spec
## Purpose
Mimir must work when Raven, a phone, or another networked sensor hears the same
active witness late, compressed, resampled, and out of Starfire's immediate
clock domain. The active codebook is meant to make that possible.
## Objective
A remote receiver that knows the source codebook and schedule should be able to
place local samples on the canonical timeline without round-tripping every
sample through Starfire. Network timestamps are metadata. Decoded anchors are
the timing truth.
## Receiver Inputs
- codebook id;
- schedule seed/epoch;
- nominal sample rate;
- local audio stream;
- optional path calibration;
- optional expected frequency band limits;
- optional network time estimate for coarse acquisition only.
## Receiver State
Minimum:
- rolling PCM ring;
- proposal ring;
- candidate symbol ring;
- codebook trellis state;
- local clock fit to canonical time;
- response/confusion accumulator;
- health counters.
It does not need:
- Fensalir render state;
- OBS state;
- Starfire ASIO state;
- per-sample network acknowledgements.
## Receiver Output
```json
{
"sourceId": "phone-left",
"codebookId": "main-room-2026-05-24",
"canonicalClockFit": {
"localSampleAtEpoch": 12345678.5,
"effectiveSampleRate": 47999.92,
"confidence": 0.91,
"meanAbsErrorSamples": 0.37
},
"anchors": [],
"responseProfile": {},
"health": {}
}
```
The output is compact enough for LAN transport. Raw audio can still be streamed
when needed, but the timing machine should not depend on raw audio arriving
with low latency.
## Canonical Mapping
The receiver fits:
```text
local_sample = source_offset + canonical_seconds * effective_sample_rate
```
The inverse places a local sample on the source timeline:
```text
canonical_seconds = (local_sample - source_offset) / effective_sample_rate
```
Any code-valid anchor improves the fit. A triplet or higher-order tuple
identifies which canonical event was heard. Repeated anchors estimate drift.
## Acquisition Strategy
1. Use broad energy/onset proposals to avoid scanning every symbol everywhere.
2. Score candidates through the calibrated symbol likelihood.
3. Preserve top-K ambiguities.
4. Search code-valid tuples against the known schedule.
5. Fit local clock.
6. Narrow future search around predicted event times.
That last step is the efficiency win. Once locked, the receiver is not hunting
the whole universe; it is checking whether expected events arrived where the
current clock predicts.
## Network Behavior
Remote nodes should send:
- periodic clock-fit summaries;
- decoded anchor batches;
- response/confusion updates;
- buffer health;
- optional compressed/raw audio windows for debugging or program use.
Starfire should not assume:
- network latency is stable;
- transport order is canonical order;
- remote capture sample rate is exact;
- packet timestamps are better than decoded anchors.
## Raven Shape
Raven is special because it is a co-stream/game machine with its own Scarlett.
It can:
- hear/emit the same active witness;
- decode local clock against Starfire's schedule;
- stream compact anchor/response state to Starfire;
- optionally stream audio/video payloads into Starfire's rolling buffers with
canonical-time metadata attached.
Raven should not:
- do heavy soundfield reconstruction;
- become the global clock authority unless explicitly selected;
- depend on OBS or game timing as source truth.
## Phone Shape
A phone is the harsh receiver target:
- unknown mic response;
- OS audio resampling;
- possible AGC/noise suppression;
- weaker CPU budget;
- awkward latency.
If the phone can decode enough anchors to estimate canonical time, the codebook
is doing its job. If it cannot, the failure is likely response/codebook design,
not network transport.
## Security / Identity Note
Codebook possession should be enough to decode time, not enough to control
Mimir. Remote timing reports need source identity and trust policy before they
affect program output.

View file

@ -0,0 +1,236 @@
# Failure Mode Ledger
## Purpose
This ledger names failure modes as data-flow problems. The fix should generally
move authority or measurement to the right place, not add another after-the-fact
filter.
## Active Witness Failures
### Dead Frequency Band
Symptom:
- emitted symbol is consistently absent or weak on one mic.
Coherent fix:
- calibration marks band unreliable;
- codebook emission avoids it for that path or shared room plan;
- decoder weights it down before trellis selection.
Bad fix:
- classify as normal, then reject "outliers" later.
### Symbol Alias
Symptom:
- expected symbol often scores as a neighboring bin.
Coherent fix:
- confusion matrix changes likelihood surface;
- global bin-shift hypotheses are searched as part of the window;
- codebook spacing/order adapts.
Bad fix:
- hard-code a special-case symbol remap without path identity.
### Reflection-Dominated Timing
Symptom:
- anchors decode, but delay residuals jump between plausible paths.
Coherent fix:
- preserve multiple delay hypotheses;
- combine with mic/speaker geometry and repeated state;
- report uncertainty to Fensalir/DSP.
Bad fix:
- smooth the jump until it looks calm.
### Gain / Clipping
Symptom:
- loopback or mic confidence collapses despite audible witness.
Coherent fix:
- calibration preflight checks clipping/noise floor;
- artifacts record gain state;
- witness amplitude adapts within safe bounds.
Bad fix:
- lower thresholds globally.
## Passive Sync Failures
### Silence
Symptom:
- passive estimator emits weak or meaningless delay.
Coherent fix:
- confidence falls;
- hybrid emits active watermark/chirp witness.
Bad fix:
- hold stale passive delay as if it were current truth.
### Narrowband Music
Symptom:
- PHAT correlation peak is broad or ambiguous.
Coherent fix:
- passive confidence remains advisory;
- active witness or longer evidence window takes authority.
Bad fix:
- pretend a low-frequency beat gives microsecond timing.
## Audio Actuator Failures
### Delay Corrected, Drift Ignored
Symptom:
- streams align briefly and then walk apart.
Coherent fix:
- separate delay and SRO loops;
- drive ASRC/fractional delay continuously.
Bad fix:
- periodically snap delay back with audible discontinuity.
### Actuator Chases Source Motion
Symptom:
- moving speaker/mic looks like sample-rate drift.
Coherent fix:
- slow SRO loop uses long-term clock evidence;
- fast TDOA/source motion remains separate.
Bad fix:
- one smoothing constant for everything.
## Camera Failures
### Advertised FPS Lies
Symptom:
- device reports 60/120 fps but delivers less.
Coherent fix:
- measure actual cadence after warm-up;
- record topology/control settings;
- treat measured cadence as truth.
Bad fix:
- trust media type advertisements.
### Shared USB Collapse
Symptom:
- one device's cadence drops when another device appears.
Coherent fix:
- topology/cadence benchmark;
- distribute devices across controllers/hosts;
- record known bad neighborhoods.
Bad fix:
- add buffering and call latency acceptable.
### JSON Bridge Becomes Production
Symptom:
- diagnostic frame events are relied on for live six-camera ingest.
Coherent fix:
- promote probe logic into native worker ABI;
- payload handles enter runtime/reservoir.
Bad fix:
- optimize JSON parsing.
## Fusion Failures
### CPU Pixel Furnace
Symptom:
- managed code demosaics/converts/processes every camera frame.
Coherent fix:
- push feature extraction/upload into Fensalir GPU path;
- keep CPU metadata small.
Bad fix:
- add worker threads until the thermal budget files a complaint.
### Parallel Stable Track Owners
Symptom:
- Mimir and Fensalir both maintain stable visual/acoustic track identity.
Coherent fix:
- Mimir owns observations;
- Fensalir temporal evidence reservoir owns tracks after lowering.
Bad fix:
- add id synchronization glue.
## Network Failures
### Transport Timestamp Worship
Symptom:
- Raven/phone timing is derived from packet arrival time.
Coherent fix:
- remote receiver decodes codebook anchors locally;
- network time is coarse metadata only.
Bad fix:
- add latency compensation constants per sender.

View file

@ -0,0 +1,180 @@
# Fensalir Integration Map
## Purpose
Mimir should not grow a second renderer, second temporal evidence reservoir, or
second UI engine. Fensalir already owns the window, D3D12 device, render graph,
GPU sensor fusion packets, temporal Gaussian field, acoustic constraint packet,
debug chrome, and future Spout/program output. Mimir owns the physical truth:
device capture, codebook schedules, calibration state, rolling runtime buffers,
network feed ingress, and the decision about which evidence enters the field.
The integration cut is therefore simple:
- Mimir produces timestamped observations and calibration constraints.
- Fensalir lowers those observations into D3D12 resources and temporal fields.
- Mimir reads health/status and manages policy.
- OBS receives final program surfaces/stems, not raw driver noise.
## Current Fensalir Surfaces
This pass inspected `E:\Projects\Fensalir` because the engine moved out of this
repo and the old app-specific mental model is now stale.
Relevant contract surfaces:
- `AquariumSceneState.GpuSensorFrame`
- `AquariumSceneState.AcousticFieldFrame`
- `AquariumSceneState.CalibrationEventFrame`
- `AquariumSceneState.GpuFusionField`
- `AquariumSceneState.TemporalGaussianField`
- `AquariumRenderPlan`
- `AquariumShaderManifest`
Relevant engine concepts:
- D3D12 renderer owns swapchain, command lists, descriptor heaps, upload rings,
shader compilation, and render graph execution.
- `D3D12ExternalSensorTexture` is the natural landing point for imported camera
frames once the native capture path can provide a shareable GPU handle.
- `D3D12GpuSensorFusion.hlsl` and `D3D12TemporalGaussian.hlsl` already express
the shape of a GPU-resident fusion pass.
- `TemporalSpatialEvidenceReservoir` owns stable evidence tracks. Mimir should
not invent a parallel stable-key cache for visual/acoustic claims.
- `D3D12AcousticConstraintPacket` is the visible contract slot for decoded
audio geometry constraints.
- `D3D12Renderer` already converts `AquariumAcousticConstraint` and clap
calibration events into acoustic constraint packets, so Mimir should lower
decoded audio evidence into those contracts rather than creating a side
debug-only renderer path.
- Fensalir tests already assert that `AquariumSceneState` can carry a
`GpuSensorFrame` with shared texture metadata and an `AcousticFieldFrame`
with timing confidence, uncertainty, accumulation window, presentation delay,
and spatial constraints.
## Target Pipeline
```mermaid
flowchart TD
A["native camera callbacks"] --> B["Mimir.Runtime rolling buffer"]
C["ASIO callbacks"] --> B
D["network feed packets"] --> B
B --> E["Mimir calibration and synchronization kernel"]
E --> F["AquariumGpuSensorFrame"]
E --> G["AquariumAcousticFieldFrame"]
E --> H["AquariumCalibrationEventFrame"]
F --> I["Fensalir D3D12 fusion"]
G --> I
H --> I
I --> J["Temporal Gaussian / evidence field"]
J --> K["Program video / diagnostics"]
```
## Ownership Invariants
- Fensalir owns GPU resource lifetime. Mimir may describe external memory, but
it must not build a private D3D12 lifetime system.
- Fensalir owns temporal evidence reuse once observations are lowered into the
shared reservoir. Mimir owns observation truth before lowering.
- Mimir owns calibration identity: device ids, mic/speaker path ids, codebook
generation, response/confusion matrices, delay/group-delay models, and
reliable-bin plans.
- Mimir owns the rolling buffer retention boundary. Fensalir can snapshot or
consume the live window, but it must not silently become the history owner.
- `AquariumAudioDocument.EnqueuePcm16Base64` is not a production Mimir audio
path. It is suitable for UI/demo audio, not 192 kHz ASIO timing evidence.
## Concrete Lowering Contracts
### Camera Frames
Mimir source:
- source id;
- canonical timestamp estimate;
- device timestamp;
- frame sequence number;
- dimensions and pixel format;
- native CPU pointer or GPU shared handle;
- calibration id and lens model;
- confidence/status bits.
Fensalir target:
- `AquariumGpuSensorFrame`
- `AquariumGpuSensorCamera`
- `AquariumExternalGpuTexture`
The first production proof can use CPU upload because it is easier to inspect.
The final six-camera path wants GPU import or a pinned/native staging ring.
### Acoustic Constraints
Mimir source:
- decoded chirp anchors;
- passive program-audio anchors;
- loopback canonical clock fit;
- mic delay/SRO estimates;
- per-band magnitude/phase response;
- TDOA/source localization candidates;
- confidence and residual statistics.
Fensalir target:
- `AquariumAcousticFieldFrame`
- `AquariumAcousticConstraint`
The acoustic packet should not carry raw PCM. Raw PCM stays in the Mimir/native
audio reservoir; Fensalir receives constraints and optional debug surfaces.
The field frame can also carry timing oracle values. Mimir should derive these
from the loopback/reference clock fit:
- timing oracle timestamp;
- timing confidence;
- timing uncertainty in microseconds;
- accumulation window;
- presentation delay.
That gives Fensalir a scalar timing truth for visualization while keeping the
per-source audio correction state in Mimir/Faust.
### Calibration Events
Mimir source:
- emitted chirp/watermark event id;
- clap/impulse event id;
- detected mic/camera event timestamps;
- spatial hypothesis if visual evidence exists.
Fensalir target:
- `AquariumCalibrationEventFrame`
- `AquariumClapCalibrationEvent`
The important cut: calibration events are high-confidence anchors, not a
separate timeline authority.
## Avoided Machines
- A Mimir-side render graph. Fensalir already owns it.
- A Mimir-side stable Gaussian track database. Fensalir's temporal evidence
reservoir is the correct owner once observations become tracks.
- A second app-specific debug panel inside Mimir. Mimir should provide state;
Fensalir should render the debugging UI.
- Base64 or JSON transport in hot local audio/video paths. Those remain
diagnostics and artifact tools.
## Immediate Implementation Cut
1. Add a small Mimir-to-Fensalir bridge class that builds an
`AquariumGpuSensorFrame` from current `MimirStreamBufferSet` video
descriptors.
2. Add an acoustic lowering class that turns current sync reports and
calibration profiles into `AquariumAcousticFieldFrame`.
3. Keep both as pure, allocation-measured mapping code. They should not read
devices, run analyzers, or own timers.
4. Benchmark one second of repeated lowering against a populated synthetic
buffer before connecting it to the app render tick.

View file

@ -0,0 +1,162 @@
# Glossary
This glossary is deliberately blunt. It is here so the next implementation pass
does not burn half an hour rediscovering which word owns which invariant.
## Active Witness
A signal Mimir deliberately emits so receivers can recover canonical time and
path response. In this repo the active witness is the chirp-bin/chirplet
timeline. It is not only a sync beep; it is also a calibration probe.
## ASIO
Low-latency pro-audio driver API. In Mimir, ASIO is the production path for
Scarlett loopback and mic capture because it can put output loopback and inputs
in the same driver clock domain at 192 kHz.
## Canonical Timeline
The emitter-owned time axis. A receiver that knows the codebook and schedule
should infer where its captured samples sit on this timeline.
## Chirp Spread Spectrum
A communications family that encodes symbols in chirps. LoRa/CSS demodulators
matter to Mimir because they show a mature version of dechirp, FFT/bin scoring,
timing/frequency-offset handling, and low-power receiver design.
## Chirplet
A localized chirp atom: time-limited, frequency-localized, and slope-bearing.
The generic chirplet transform searches a broad chirplet parameter space. Mimir
uses a constrained chirplet receiver because the emitter controls the codebook.
## Codebook
The set of allowed emitted symbols and their shapes. In Mimir this includes
frequency bin, glide direction/slope, duration, and rhythmic spacing. It should
adapt to the measured physical path by avoiding dead or ambiguous bins.
## Confusion Matrix
For each expected symbol, the observed energy distribution across possible
symbols/bins. This is how Mimir learns that a mic hears symbol 12 as symbol 10,
or does not hear a band at all.
## De Bruijn Sequence
A sequence where every length-N symbol tuple appears once in a cycle. Mimir uses
the idea so a small number of consecutive chirp events can identify a unique
timeline position.
## Dechirp
Mix a received chirp against the conjugate/reference chirp so its energy
collapses toward a tone/bin. This turns chirp classification into a cheap
frequency-bin measurement.
## Farrow Filter
A fractional-delay filter structure where coefficients are polynomial functions
of the desired fractional sample delay. Useful for continuous delay actuation
without rebuilding FIR kernels at audio rate.
## Fensalir
The engine repo. It owns windowing, rendering, D3D12, render graph execution,
GPU fusion packets, temporal Gaussian/evidence fields, debug UI, and eventual
program surface publication.
## Frequency Response
How a speaker/room/mic path changes signal magnitude by frequency. Mimir should
learn it from the active witness and use it to normalize both calibration and
program audio.
## GCC-PHAT
Generalized cross-correlation with phase transform weighting. A common robust
delay estimator. It is useful for passive program-audio alignment and TDOA, but
it is not a substitute for controlled codebook anchors.
## Gaussian Splat
A 3D/4D ellipsoidal radiance primitive. For Mimir, splats are a plausible
runtime field representation for fused camera evidence, but Fensalir should own
the GPU field and renderer.
## Group Delay
Frequency-dependent delay through a path. If high bands arrive shifted relative
to low bands, symbol timing and response normalization both suffer. The active
witness should estimate and correct this.
## Hybrid Mode
Use passive program-audio timing by default, emit active witness only when
confidence falls. In Mimir this should eventually be a shaped watermark, not a
burst of rude test tones.
## Loopback
Audio captured from the output path. Scarlett loopback is the current timing
authority because it observes what Mimir attempted to emit before acoustic room
damage.
## Near-Field Acoustic Holography
A family of techniques for reconstructing sound fields from microphone array
measurements. It matters because Mimir is not only aligning mics; it is trying
to infer a room-scale sound field from sparse sensors.
## Passive Mode
No active emitted witness. Mimir estimates delay from program audio such as
music/game sound. Useful when the content is rich; fragile during silence or
band-limited material.
## PLL
Phase-locked loop. In Mimir, a control-rate PLL can turn repeated delay
observations into delay and sample-rate-offset corrections for the audio
actuator.
## Reservoir
A bounded evidence store. The Mimir rolling buffer owns recent raw stream
history; Fensalir's temporal evidence reservoir owns stable resolved visual and
acoustic field claims after lowering.
## Sample-Rate Offset
Clock drift between devices, usually described in ppm. Delay alone aligns one
moment; SRO correction keeps streams aligned over time.
## SPSC Ring
Single-producer/single-consumer ring buffer. The natural shape for hot native
capture callbacks handing blocks to one runtime consumer without locks.
## SRP-PHAT
Steered response power with PHAT weighting. It searches source positions by
testing which spatial hypothesis best explains inter-mic phase/delay evidence.
## TDOA
Time difference of arrival. With known microphone positions, TDOA constraints
help localize sound sources.
## Temporal Gaussian Field
Fensalir's GPU-side field representation for temporally coherent splat-like
evidence. Mimir should feed it observations and constraints, not own its
internal rendering state.
## Watermark
An active witness shaped to be unobtrusive under program audio. The best Mimir
watermark still carries deterministic canonical-time identity and response
calibration energy.

View file

@ -0,0 +1,159 @@
# Hot Path Pseudocode
## Purpose
This note writes the intended hot paths as code-shaped prose. It is not
production code. It is a guard against implementation drift.
## Streaming Active Decoder
```text
onAudioBlock(source, absoluteStart, samples):
pcmRing.append(absoluteStart, samples)
for each analysisHop completed by this block:
energy = updateProposalEnergy(pcmRing, analysisHop)
if energy is plausible:
proposalRing.push(analysisHop, energy)
candidates = proposalRing.readySince(lastScoredSample)
if candidates is empty:
return currentClockFit
scoreRows = scoreKernel.scoreBatch(pcmRing, candidates, codebook, calibration)
for row in scoreRows:
ambiguityRing.push(row)
anchors = trellis.solveRecent(ambiguityRing, schedule, calibration)
clockFit.update(anchors)
responseModel.observe(scoreRows, anchors)
return clockFit
```
Invariant:
- proposal, scoring, schedule solving, and clock fit are separate phases;
- calibration changes likelihood before trellis solving;
- old PCM expires with the rolling window.
## Calibration-Weighted Symbol Score
```text
score(symbol, observedBinEnergy, phase, timing):
magnitude = normalize(observedBinEnergy)
responseWeight = path.usableBandWeight(symbol)
confusion = path.confusionProbability(expected=symbol, observed=bin)
phaseWeight = path.phaseCoherence(symbol, phase)
groupDelay = path.groupDelayCorrection(symbol)
correctedTime = timing - groupDelay
return likelihood(magnitude, responseWeight, confusion, phaseWeight, correctedTime)
```
Invariant:
- dead bands lose authority before code selection;
- group delay changes event timing before clock fit;
- phase is evidence when stable, not mandatory when unstable.
## Pairwise Delay From Anchors
```text
estimateDelay(referenceAnchors, sourceAnchors):
matches = joinByEventIndex(referenceAnchors, sourceAnchors)
if enough(matches):
return weightedMean(source.sampleOffset - reference.sampleOffset)
if bothClockFitsConfident:
return sourceFit.sampleFor(t) - referenceFit.sampleFor(t)
return noReport
```
Invariant:
- matched event anchors beat independent fits;
- independent fits must stay inside live horizon;
- no report is better than absurd confidence.
## DSP Actuator Control
```text
onSyncState(source, delaySamples, sroPpm, confidence):
if confidence < threshold:
holdOrFadeCorrection()
return
pll.update(delaySamples, sroPpm)
actuator.setTargetDelay(pll.delay)
actuator.setResampleRatio(1.0 - pll.sroPpm * 1e-6)
```
Invariant:
- control loop runs at control rate;
- sample movement runs at audio rate;
- drift and delay are not the same state.
## Native ASIO Ring
```text
onBufferSwitch(index, samplePosition):
block = ring.tryReserve()
if block is unavailable:
droppedBlocks++
return
block.sourceChannel = channel
block.startFrame = samplePosition
block.sampleRate = currentRate
convertOrCopyDeviceBuffer(block.payload)
ring.publish(block)
```
Invariant:
- callback never allocates;
- queue is bounded;
- dropped blocks are counted, not hidden.
## Camera Frame Ring
```text
onReadComplete(slot):
frame = ring.tryReserve()
if frame unavailable:
droppedFrames++
resubmit(slot)
return
frame.source = deviceId
frame.sequence = sequence++
frame.deviceTimestamp = slot.timestamp
frame.payloadHandle = slot.payload
frame.format = negotiatedFormat
ring.publish(frame)
resubmit(nextSlot)
```
Invariant:
- read queue stays full;
- frame metadata is typed;
- payload ownership is explicit.
## Fensalir Lowering
```text
buildSceneState(runtimeSnapshot):
gpuSensors = lowerVideoDescriptors(runtimeSnapshot.videoWindow)
acoustic = lowerSyncAndCalibration(runtimeSnapshot.audioState)
calibrationEvents = lowerHighConfidenceEvents(runtimeSnapshot.events)
return AquariumSceneState(gpuSensors, acoustic, calibrationEvents)
```
Invariant:
- lowering reads cached/snapshotted state;
- lowering does not poll devices;
- lowering does not run analyzers;
- Fensalir owns what happens after contract construction.

View file

@ -0,0 +1,173 @@
# Implementation Roadmap From Study
## Goal
Turn the study into implementation passes that improve coherence instead of
adding another ring of machinery around the current code.
## Phase 1: Make Active Decode Streaming
Objective:
- Stop rescanning full active windows every analysis tick.
Current mechanism:
- Analyzer extracts mono windows and calls `MimirChirpBinTimeline.DecodeStreamWindow`.
- Decode rebuilds proposals/frames/anchors from the provided span.
Intended mechanism:
- A per-source active decoder owns a rolling PCM window, energy proposal ring,
candidate-frame cache, anchor trellis tail, and clock fit state.
- Analyzer asks the decoder for the current snapshot.
Invariants:
- Same accepted reports as current analyzer on synthetic and loopback captures.
- No UI/telemetry re-entry.
- Calibration weighting remains optional until physical proof is run.
Cut line:
- If streaming state becomes harder to explain than the current decode, cut it
back to proposal/frame caching only.
Proof:
- Synthetic chirp-only/hybrid tests still recover known delay.
- Stored loopback ASIO artifact still decodes 0 us.
- Allocation count per analysis tick drops.
## Phase 2: Physical Calibration-Weighted Decode
Objective:
- Make meatspace chirp-bin decode use what calibration already learned.
Current mechanism:
- Calibration model can be loaded.
- Symbol weights, phase weights, group-delay correction, and bin-shift
hypotheses exist but need physical proof.
Intended mechanism:
- Symbol likelihood is shaped by reliability/confusion/phase before candidate
pruning.
- Global delay/bin-shift hypotheses score the whole observation window.
- Reduced reliable-bin codebook can be emitted and decoded.
Invariants:
- The codebook remains bijective over the intended horizon.
- Dead bands lose authority.
- Failed timing still emits calibration evidence.
Proof:
- Compare unweighted versus weighted decode on stored hot-mic/physical captures.
- Emit reliable-bin plan and measure anchor rate through the real monitor/mic
path.
## Phase 3: Actuator Proof
Objective:
- Move from "we know the delay" to "samples are aligned."
Current mechanism:
- `MimirAudioSynchronizationState` estimates smoothed delay and SRO.
Intended mechanism:
- Native/Faust DSP consumes delay/SRO state.
- Fractional delay line corrects sub-sample offset.
- ASRC/PLL corrects drift.
Invariants:
- Runtime estimates; DSP moves samples.
- Actuator state is inspectable but not owned by UI.
- Output remains separately stemmable for OBS.
Proof:
- Offline artifact: apply actuator to delayed candidate and reduce residual.
- Live loopback/mic: show residual delay converges and remains bounded.
## Phase 4: Direct Camera Driver Runtime Path
Objective:
- Replace diagnostic frame-event probe transport for local cameras.
Current mechanism:
- KS/PS3 probes can emit JSON frame events.
- Runtime has `IMimirVideoCaptureDriver` and source adapter seam.
Intended mechanism:
- One native/direct driver worker per camera family.
- Capture worker preserves timestamp and payload handle.
- Runtime indexes sample envelope.
- Fensalir consumes handle/window for GPU feature extraction.
Invariants:
- No local camera pixels through stdout/base64.
- Device/arrival timestamp survives the boundary.
- Capture worker owns device pressure, runtime owns identity and buffer.
Proof:
- Leap or PS3 Eye frames enter runtime buffers with sustained measured cadence.
- Fensalir UI can inspect buffer health without touching pixel hot path.
## Phase 5: Native Reservoir/Fensalir/Faust Integration
Objective:
- Make the native reservoir the shared lower memory/timing authority.
Current mechanism:
- C# buffers own runtime proof state.
- Rust reservoir exists with C ABI and tests.
Intended mechanism:
- Capture workers push handles into native reservoir.
- Runtime and Fensalir read typed views.
- Faust reads audio block handles and alignment state.
Invariants:
- One edge owns retention.
- Typed views do not own separate windows.
- Payload ownership stays with producer/consumer domain.
Proof:
- Runtime status matches C# buffer counts for a mirrored proof.
- Fensalir renders a simple current-window visual packet from reservoir claims.
## Benchmark Harnesses Needed
1. `chirp-bin-decode-bench`: synthetic and artifact decode timing/allocation.
2. `passive-sync-bench`: FFT allocation/reuse comparison.
3. `asio-callback-pressure-bench`: native queue/block pool comparison.
4. `camera-driver-buffer-bench`: direct driver payload handoff cadence.
5. `actuator-residual-bench`: before/after residual delay and drift.
## Decision Gates
- SIMD scorer only after decode timing is the measured bottleneck.
- GPU scorer only after candidate/bin batches exceed CPU SIMD advantage.
- Neural/acoustic-field models only after classical timing/calibration/source
localization is coherent.
- Dynamic 4DGS only after direct synchronized visual evidence exists in
Fensalir.

View file

@ -13,6 +13,32 @@ needs the reason a cut exists.
| `architecture-rumination.md` | System ownership, current machine critique, coherent next cuts. |
| `optimization-ledger.md` | Hot loop risks, low-level implementation options, micro-optimization candidates. |
| `references.md` | Research links for chirplets, CSS/LoRa, GCC-PHAT, D3D12/FFT/GPU compute. |
| `current-hotspot-audit.md` | Static audit of likely runtime allocation/scan/driver-pressure hotspots. |
| `implementation-roadmap.md` | Phase plan, invariants, proofs, and benchmark harnesses. |
| `questions-and-hypotheses.md` | Open questions and falsifiable hypotheses by domain. |
| `reading-guide.md` | Task-oriented route through the study docs and samples. |
| `data-dictionary.md` | Data object definitions for streams, samples, chirp anchors, clock fits, calibration paths, reports, and Fensalir constraints. |
| `volumetric-audio-field.md` | Sound-field reconstruction decomposition and first practical audio target. |
| `acoustic-field-models.md` | Field reconstruction models and the first honest acoustic source proof. |
| `chirplet-transform-deep-dive.md` | Controlled chirp receiver versus generic chirplet transform. |
| `decoder-architecture-options.md` | Active receiver implementation options and promotion criteria. |
| `distributed-receiver-spec.md` | Raven/phone receiver shape for self-locating remote sensors using codebook and schedule state. |
| `visual-fusion-4dgs-study.md` | Sensor fusion and realtime Gaussian splatting architecture study. |
| `open-implementation-catalog.md` | Implementation families to study before inventing local machinery. |
| `prior-research-synthesis.md` | Distillation of the older chirplet, adaptive sync, ambisonic, splatting, calibration, and visual archives. |
| `research-claims-digest.md` | Claim-level digest of external literature and how it steers Mimir. |
| `fensalir-integration-map.md` | Contract-level cut between Mimir runtime truth and Fensalir GPU/render ownership. |
| `native-boundary-map.md` | Native/managed/Fensalir ownership boundaries for ASIO, cameras, reservoir, and GPU payloads. |
| `option-matrix.md` | Implementation alternatives, failure modes, and deciding proofs for the major subsystems. |
| `benchmark-plan.md` | Measurement plan for audio sync, acoustic field, camera capture, Fensalir lowering, and splat fusion. |
| `calibration-session-spec.md` | Command/session shape for learning and validating real output/mic path models. |
| `low-level-implementation-notes.md` | Concrete memory layout, SIMD/GPU, ASIO, camera, and instrumentation notes. |
| `microsecond-sync-math.md` | Timing math for sample period, bandwidth, SNR, subsample delay, and acoustic distance. |
| `state-machine-invariants.md` | Ownership limits for runtime, hub, buffers, decoders, calibration, DSP, capture, Fensalir, network, and OBS. |
| `failure-mode-ledger.md` | Failure modes and coherent fixes across active witness, passive sync, actuator, camera, fusion, and network paths. |
| `hot-path-pseudocode.md` | Code-shaped intended hot paths for streaming decode, calibration weighting, actuator, native rings, camera rings, and Fensalir lowering. |
| `glossary.md` | Short definitions for the synchronization, acoustic, visual, and reservoir terms used across the study. |
| `sample-code-index.json` | Machine-readable index for sample sketches. |
| `samples/README.md` | Index of sample implementation sketches. |
## Sample Code
@ -21,9 +47,17 @@ needs the reason a cut exists.
| --- | --- | --- |
| `samples/StreamingChirpBinDecoderSketch.cs` | `MimirChirpBinStreamingDecoder` or equivalent runtime class | What state removes full-window rescans? |
| `samples/Avx2DechirpGoertzelSketch.cpp` | Native CPU scorer for chirp-bin candidates | What does SIMD dechirp/bin scoring look like? |
| `samples/BatchedChirpScoreAbiSketch.h` | Native scorer ABI | What batch boundary avoids per-candidate managed/native overhead? |
| `samples/ChirpBinScore.compute.hlsl` | Fensalir/D3D12 compute scorer | How would batched candidate/bin scoring map to GPU? |
| `samples/FarrowFractionalDelaySketch.cpp` | Faust/native DSP actuator | What is the smallest fractional delay proof? |
| `samples/SpscAudioBlockRingSketch.cpp` | Native capture worker handoff | What shape should low-jitter audio block transfer take? |
| `samples/CalibrationWeightedChirpLikelihoodSketch.cs` | Physical-path active decoder | Where does calibration enter symbol likelihood? |
| `samples/SroPllAsrcControllerSketch.cpp` | DSP actuator control | How does delay/SRO state become a resampler command? |
| `samples/GaussianSplatClaimUpdate.compute.hlsl` | Fensalir visual fusion | What is the smallest live splat-claim compute shape? |
| `samples/AquariumGpuSensorFrameBridgeSketch.cs` | Mimir-to-Fensalir visual lowering | How should runtime video observations become Fensalir sensor frames? |
| `samples/AcousticConstraintLoweringSketch.cs` | Mimir-to-Fensalir acoustic lowering | How should decoded anchors/geometry become acoustic constraints? |
| `samples/TemporalEvidenceCandidateSketch.cs` | Shared temporal evidence candidates | How can Mimir lower candidates without owning Fensalir tracks? |
| `samples/SrpPhatGridSketch.cpp` | Acoustic source localization | What is the smallest TDOA/SRP-PHAT-style grid scoring proof? |
## Best Next Implementation Sequence
@ -55,6 +89,8 @@ needs the reason a cut exists.
- group delay
- LoRa CSS
- native reservoir
- Fensalir
- Gaussian splatting
- passive sync
- Raven
- response/confusion matrix
@ -64,4 +100,3 @@ needs the reason a cut exists.
- streaming decoder
- volumetric audio
- volumetric visual fusion

View file

@ -0,0 +1,199 @@
# Low-Level Implementation Notes
## Purpose
This file collects the implementation details that are easy to lose when the
conversation stays at the architecture level. None of these notes are promises;
they are candidates to benchmark.
## Audio Memory Layout
### Preferred Analysis Layout
For decoder kernels:
```text
channel0 samples: [s0 s1 s2 ...]
channel1 samples: [s0 s1 s2 ...]
...
```
Separate channel spans make per-channel scoring straightforward and avoid
striding through interleaved audio during analysis. Interleaving is useful for
device I/O and some DSP kernels; it is not automatically the right analysis
shape.
### Preferred Scoring Layout
```text
candidateStartSamples[]
candidateEnergy[]
symbolReliability[]
binFrequencies[]
binSin[]
binCos[]
```
Dense arrays let SIMD/native/GPU kernels process candidates without object
chasing. The codebook trellis can stay managed/CPU because it is small and
branchy.
## SIMD Chirp-Bin Scoring
Likely CPU kernel shape:
1. load candidate window samples in 8-float AVX2 chunks;
2. multiply by dechirp oscillator/window;
3. accumulate bin real/imag sums;
4. compute magnitude or normalized energy;
5. write top K or full dense bin vector.
Important:
- keep oscillator tables aligned;
- keep candidate windows contiguous;
- batch multiple candidates per interop call;
- avoid denormals;
- normalize outside the deepest loop where possible.
## FFT / Goertzel Decision
Use Goertzel/fixed-bin DFT when:
- symbol count is small;
- bins are known;
- candidate count is modest;
- native dispatch overhead matters.
Use FFT/CZT when:
- many bins are needed;
- frequency offset interpolation dominates;
- candidate batches are large enough;
- GPU/native FFT plans can be reused.
Do not choose FFT because it sounds more canonical. Canonical is not a profiler.
## GPU Chirp Scoring
The GPU scoring pass only makes sense if data is already resident or batches are
large enough to pay dispatch overhead.
Good GPU candidates:
- many mic streams;
- many remote receiver windows;
- wide candidate sets during acquisition;
- offline/artifact calibration sweeps.
Bad GPU candidates:
- one or two channels;
- tiny candidate counts;
- paths that require CPU readback before codebook solving every event.
## ASIO Capture Hot Path
Current observed risks:
- per-block `std::vector<float>` resize;
- `std::deque` queue;
- native-to-managed polling copies into `float[]`;
- managed byte array allocation for sample payload;
- `ConcurrentQueue.Count` pressure.
Preferred cut:
- native SPSC ring;
- fixed block pool;
- monotonic frame counters;
- block descriptors with handle/index;
- C# metadata only until DSP needs samples.
## Camera Capture Hot Path
Current probe strengths:
- KS async queueing is already close to the driver;
- PS3 Eye raw path proves high fps;
- JSON frame events make cadence visible.
Current probe risks:
- JSON must not become production payload;
- CPU frame buffers are diagnostic until Fensalir import/upload path exists;
- one-off command-line knobs must become configuration/stateful driver policy.
Preferred cut:
- one native capture worker per camera family;
- fixed frame ring;
- descriptor ABI;
- optional shared GPU texture handle;
- Fensalir owns texture import and GPU staging.
## Fensalir GPU Fusion
Fensalir contracts already distinguish:
- `GpuSensorFrame` for calibrated camera inputs and external textures;
- `AcousticFieldFrame` for timing/room constraints;
- `CalibrationEventFrame` for high-confidence event anchors;
- `GpuFusionField` for point buffers/seeds;
- `TemporalGaussianField` for stable gaussian observations.
Mimir should produce these rows. It should not pack shader structs directly.
## Gaussian Splat Rendering Levers
From modern 3DGS implementations:
- tile size matters;
- sorting strategy matters;
- cull before sort;
- packed buffers matter;
- anti-aliasing and opacity/radius thresholds matter;
- view-consistency fixes cost real time;
- depth/feature render modes are useful for diagnostics.
Mimir-specific cut:
- first update live claims;
- then render simple claims;
- only later optimize dynamic radiance fields.
## Benchmark Counters To Add
Decoder:
- samples consumed;
- proposals created;
- candidates scored;
- anchors accepted/rejected;
- elapsed scoring time;
- allocations per analysis tick.
ASIO:
- callback count;
- block count;
- queue/ring high-water mark;
- dropped blocks;
- conversion time if measurable.
Camera:
- frame interval p50/p95/p99;
- async queue depth;
- dropped/late frames;
- payload copy time;
- upload/import time.
Fensalir:
- contract lowering time;
- upload time;
- GPU pass timings;
- temporal field count;
- acoustic constraint count.

View file

@ -0,0 +1,102 @@
# Microsecond Sync Math
## Purpose
The project target is microsecond-class synchronization. This note keeps the
math visible so we do not confuse sample-rate settings, transducer bandwidth,
and actual timing certainty.
## Sample Periods
| Sample Rate | Sample Period |
| --- | --- |
| 48 kHz | 20.833 us |
| 96 kHz | 10.417 us |
| 192 kHz | 5.208 us |
Subsample estimation is mandatory for microsecond delay estimates at every
practical audio sample rate. 192 kHz helps by reducing the interpolation gap and
allowing higher witness frequencies, but it does not magically provide
one-microsecond truth.
## What Enables Subsample Timing
- known emitted waveform;
- high SNR;
- broad enough bandwidth;
- stable phase response;
- clean enough direct path;
- coherent clock model over multiple anchors;
- interpolation or model fitting around correlation/likelihood peaks.
The active witness helps because its shape is known. Passive music can help when
it has enough broadband content, but it is not guaranteed.
## Cramer-Rao Intuition
Delay estimation improves with:
- signal bandwidth;
- SNR;
- observation duration / repeated independent evidence.
It degrades with:
- narrowband signals;
- reverberation;
- unknown filtering;
- low SNR;
- nonlinear processing;
- clock drift;
- false symbol aliases.
This is why the chirp signal has a dual role: it spreads energy across time and
frequency for timing, while also measuring which parts of the path survived.
## 192 kHz Is Useful When
- the interface can capture output and mics in one stable clock domain;
- the monitors/mics preserve enough high-frequency energy;
- phase/group delay can be calibrated;
- the actuator can use fractional delay/SRO information;
- the extra CPU/memory load stays within budget.
## 192 kHz Is Not Useful When
- the acoustic path low-passes the witness below the useful band;
- the mic/driver applies hidden processing;
- the decoder only reports integer samples;
- the system still lacks a resampling/fractional-delay actuator;
- the bottleneck is symbol confusion rather than sample quantization.
## Timing Evidence Hierarchy
1. Electrical loopback: best for emitted timeline and interface clock.
2. Physical mic active witness: best for acoustic path timing/response.
3. Passive program audio: useful continuous drift evidence when content allows.
4. Network timestamps: metadata, not truth.
## Delay To Distance
At roughly 343 m/s:
- 1 us is about 0.343 mm of acoustic path length;
- 10 us is about 3.43 mm;
- 100 us is about 34.3 mm.
That scale is useful and humbling. A room reflection or mic position error can
swamp the number we are trying to measure. The machine must keep uncertainty
visible.
## Practical Acceptance
For the next physical proof, "microsecond sync" should mean:
- loopback pair recovers below printed microsecond precision;
- physical mic reports fractional-sample delay with stable residuals;
- repeated anchors show bounded SRO drift;
- delay uncertainty is reported, not hidden;
- actuator demonstrably reduces residual misalignment.
Without the actuator, the system is measuring microseconds, not delivering them.

View file

@ -0,0 +1,127 @@
# Native Boundary Map
## Purpose
Mimir's managed runtime should own identity, policy, and orchestration. It
should not sit inside the six-camera or 192 kHz audio hot loops doing avoidable
copies, JSON parsing, or queue churn. This file marks the boundaries that should
be native, managed, or Fensalir-owned.
## Current Native Pieces
### ASIO Capture
`native/asio_capture/mimir_asio_capture.cpp` currently:
- creates the Focusrite ASIO driver;
- configures input buffers;
- receives `bufferSwitch` / `bufferSwitchTimeInfo` callbacks;
- converts device sample formats to mono float blocks;
- queues `MimirAsioBlock` objects;
- exposes a polling ABI to C#.
This proves in-process ASIO ingest. It is not the final zero-copy hot path:
`std::deque`, `std::vector<float>`, per-block resize, and managed byte-array
copies are all visible future costs.
### Camera Probes
The KS probe currently uses low-level Kernel Streaming / AVStream surfaces:
- property negotiation;
- async `NtDeviceIoControlFile` reads;
- queued read slots;
- frame JSON emission for diagnostics.
The PS3 Eye probe uses the raw driver/library path and emits frame metadata. It
proves throughput and topology, not final app ingestion.
### Reservoir
`native/reservoir` is the lower-level memory invariant: one rolling edge, typed
views, no per-kind retention. This should become the shared payload/index owner
when C# descriptors become too expensive for full payloads.
## Boundary Decisions
| Concern | Owner | Reason |
| --- | --- | --- |
| ASIO COM driver calls | native | COM/callback lifetime and sample-format conversion do not belong in C#. |
| ASIO block timing/counters | native first, Mimir indexed | The callback sees the driver clock; Mimir maps it to canonical time. |
| Audio payload memory | native/Faust | Program DSP should not copy through managed byte arrays. |
| Audio sync reports | Mimir.Runtime | Reports are policy/belief, not raw DSP. |
| Camera device control | native | Driver calls, async reads, libusb, and vendor knobs are hardware work. |
| Camera payload memory | native/Fensalir | Pixels should land in native or GPU memory. |
| GPU texture lifetime | Fensalir | The engine owns D3D12 resources and synchronization. |
| Calibration JSON/artifacts | Mimir | Persistence and reproducibility are repo/runtime concerns. |
## Desired ASIO Evolution
```mermaid
flowchart TD
A["ASIO callback"] --> B["native SPSC block ring"]
B --> C["native reservoir audio view"]
C --> D["Mimir.Runtime metadata/index"]
C --> E["Faust/native DSP"]
D --> F["sync analyzer state"]
F --> E
```
Immediate cut:
- replace `std::deque` with a bounded SPSC ring;
- preallocate per-channel block buffers;
- copy/convert once from ASIO format into the block ring;
- expose block handles or spans instead of repacking to managed arrays.
Final cut:
- keep raw/current device format where possible;
- run hot alignment and filtering in native/Faust;
- hand Mimir descriptors and diagnostics rather than payload copies.
## Desired Camera Evolution
```mermaid
flowchart TD
A["driver read completion"] --> B["native frame ring"]
B --> C["timestamp/sequence descriptor"]
B --> D["CPU pinned or GPU shared payload"]
C --> E["Mimir.Runtime rolling buffer"]
D --> F["Fensalir texture import/upload"]
```
Immediate cut:
- one native worker per camera family;
- frame descriptor ABI with source id, sequence, timestamp, dimensions, format,
stride, byte length, and payload handle;
- no JSON in the production app path.
Final cut:
- D3D12-compatible shared resources where the driver path can support them;
- CPU staging only where direct GPU import is not practical;
- feature extraction as early GPU compute, not managed pixel processing.
## External Guidance Anchors
- Microsoft Kernel Streaming docs confirm KS/AVStream is the low-level Windows
stream surface beneath higher media frameworks.
- `IDXGIResource1::CreateSharedHandle` and D3D11/12 shared NT handles are the
relevant family for resource sharing once capture can land in shareable
textures.
- ASIO `bufferSwitchTimeInfo` / sample position is the right place to preserve
driver-clock timing, not a managed polling timestamp after the fact.
- Focusrite channel-order docs matter because loopback availability and channel
layout can change with sample rate and model.
## Watch List
- Per-block `new float[]` / `byte[]` in managed ingest.
- `ConcurrentQueue.Count` in hot loops.
- `std::deque` under callback pressure.
- One process per camera.
- Any "temporary" JSON bridge that survives once payloads matter.
- GPU resource ownership creeping into Mimir instead of Fensalir.

View file

@ -0,0 +1,99 @@
# Open Implementation Catalog
This catalog lists implementation families worth stealing shape from. Do not
vendor code casually. The point is to understand mature cuts.
## Chirp / CSS / LoRa
What to look for:
- dechirp implementation;
- symbol timing synchronization;
- FFT/bin selection;
- CFO/bin-shift correction;
- preamble/sync word handling;
- soft decision metrics.
Candidate references:
- OpenLoRa demodulator work.
- SDR LoRa implementations.
- GNU Radio LoRa/CSS blocks.
- Audio "data over sound" projects for acoustic channel constraints.
Mimir translation:
- Preamble becomes deterministic timeline/codebook schedule.
- CFO becomes acoustic bin shift / sample-rate mismatch.
- Channel response becomes output/mic path calibration.
## Audio Delay / Beamforming
What to look for:
- GCC-PHAT implementations;
- SRP-PHAT acceleration;
- microphone array calibration;
- fractional delay filters;
- asynchronous sample-rate conversion.
Candidate references:
- Pyroomacoustics for algorithm shape.
- WebRTC audio processing for pragmatic realtime constraints.
- JACK/PipeWire/ASIO style callback boundaries.
- Faust examples for DSP graph ownership.
Mimir translation:
- Keep algorithm shape, not necessarily library dependency.
- Faust/native DSP is the actuator home.
## Gaussian Splatting
What to look for:
- 3D Gaussian Splatting CUDA rasterizer.
- Dynamic/4D Gaussian splatting repos.
- Differentiable Gaussian rasterization data layouts.
- Real-time viewer/update pipelines.
Mimir translation:
- Fensalir owns render/update.
- Native reservoir owns time-indexed claims.
- Direct camera drivers feed evidence; do not train/reconstruct from disk
batches in the live path.
## Native Capture
What to look for:
- ASIO callback isolation.
- Kernel Streaming async read queues.
- libusb/WinUSB transfer queues.
- D3D12 shared texture/resource handoff.
- lock-free SPSC rings.
Mimir translation:
- One capture worker owns one device.
- Payload handles cross subsystem boundaries.
- Runtime indexes metadata and timing.
## SIMD/GPU Compute
What to look for:
- AVX2/FMA complex dot products.
- batched FFT plan reuse.
- HLSL wave reductions.
- CUDA/cuFFT callback fusion.
- cache-aware ring buffers.
Mimir translation:
- SIMD first for fixed small bin banks.
- GPU when candidate/bin/channel batches are large enough to amortize dispatch
and memory movement.

View file

@ -0,0 +1,72 @@
# Implementation Option Matrix
## Purpose
This matrix keeps future cuts from turning into flavor arguments. Each row names
the invariant a subsystem must protect, the conservative implementation, the
aggressive implementation, and the proof that should decide between them.
## Audio Sync Receiver
| Option | Protects | Upside | Failure Mode | Proof |
| --- | --- | --- | --- | --- |
| Current window decoder | Correctness oracle | Easy to inspect; already wired | Repeated full-window work | Keep as reference until streaming matches |
| Streaming scalar decoder | One-pass receiver state | Small, portable, phone-shaped | Weak proposals can miss path-damaged chirps | Synthetic/artifact parity with fewer scans |
| AVX2 scorer | CPU throughput | Fast fixed-bin scoring | Interop and memory traffic erase wins | Batched benchmark beats scalar end-to-end |
| GPU scorer | Large batch throughput | Many channels/candidates at once | Dispatch/readback overhead | Only promote when batches are large and GPU stays resident |
| Full chirplet transform | General signal analysis | Handles unknown chirps | Too broad for controlled codebook | Only if controlled receiver fails after calibration |
## Active Witness Shape
| Option | Protects | Upside | Failure Mode | Proof |
| --- | --- | --- | --- | --- |
| 32 fixed bins | Short triplets | Dense code space | Dead/aliased bands in room | Works only where response is good |
| Adaptive reliable bins + higher order | Physical robustness | Avoids dead bands | Longer lock window | Meatspace anchors increase after calibration |
| Watermarked hybrid | Human tolerance | Less annoying | Masked too well to decode | Passive+active confidence improves without audible abuse |
| Ultrasonic-only | Low distraction | Clean separation from music | Monitors/mics may not pass band | Measured SNR and response beyond hearing |
## Audio Actuator
| Option | Protects | Upside | Failure Mode | Proof |
| --- | --- | --- | --- | --- |
| Integer delay | Coarse alignment | Simple | Not microsecond/subsample | Useful only as baseline |
| Farrow fractional delay | Subsample delay | Cheap, time-varying | Magnitude/phase ripple | Delay residual falls on artifact and live captures |
| Variable ASRC | Long-term SRO | Holds sync over time | More complex buffering | Drift stays bounded over 20+ minutes |
| Frequency-domain phase correction | STFT coherence | Good for analysis frames | Latency/window artifacts | Program material remains clean |
## Audio Field Reconstruction
| Option | Protects | Upside | Failure Mode | Proof |
| --- | --- | --- | --- | --- |
| Aligned stems | OBS usefulness | Immediate production value | Not volumetric | Stems stay synchronized and mixable |
| Source-based spatial bus | Coherent performance mix | Honest with sparse mics | Depends on source metadata | Stable source positions/pans |
| Ambisonic fit | Compact spatial field | Standard ecosystem | Sparse irregular mics unstable | Calibration proves usable order/band |
| Sparse equivalent sources | Physical field estimate | Handles arbitrary geometry | Ill-conditioned inverse problem | Known source reconstructed with residuals |
| Hybrid evidence field | General Mimir shape | Combines audio/visual constraints | Complexity creep | Fensalir reservoir stabilizes claims |
## Camera Ingest
| Option | Protects | Upside | Failure Mode | Proof |
| --- | --- | --- | --- | --- |
| JSON probes | Research observability | Fast to inspect | Not a hot path | Keep diagnostic only |
| Managed wrapper over native reads | App integration | Easier first driver | Copies/GC pressure | One camera sustained cadence |
| Native SPSC frame rings | Throughput | Predictable hot loop | ABI/lifetime complexity | All local cameras sustained |
| GPU shared textures | Copy avoidance | Best final path | Driver/API friction | Fensalir imports and renders frames |
## Visual Fusion
| Option | Protects | Upside | Failure Mode | Proof |
| --- | --- | --- | --- | --- |
| Feature tracks | Sparse evidence | Debuggable | Not visually rich | Tracks align across cameras |
| Point/splat claims | Live volumetric surface | GPU-friendly | Needs calibration | Stable live render from observations |
| Offline 3DGS-style optimization | Visual quality | Mature papers/code | Wrong latency/ownership | Use only for reference or static assets |
| Dynamic/4D splats | Live field | Target shape | Research complexity | Start after direct observations work |
## Fensalir Boundary
| Option | Protects | Upside | Failure Mode | Proof |
| --- | --- | --- | --- | --- |
| Mimir builds render state | Quick local control | Tempting | Duplicates engine | Reject |
| Mimir lowers contract frames | Ownership clarity | Clean boundary | Needs stable contract mapping | Allocation-free lowering benchmark |
| Fensalir owns temporal evidence | Single field authority | Reuses engine work | Requires good candidates | Acoustic/visual claims visible in engine |

View file

@ -0,0 +1,142 @@
# Prior Research Synthesis
## Purpose
Mimir already carries useful research archives. This note distills the pieces
that still steer the live machine so the next pass does not reread every mirror
before touching code.
## Chirplet Sync Decoder Archive
Location: `research/chirplet-sync-decoder/`
Durable lesson:
- Generic chirplet matching is valid signal analysis, but the controlled Mimir
beacon should be receiver-cheap by construction.
- Dechirp plus FFT/Goertzel is the important communications trick.
- Symbol identity should be built from common chirp duration/slope/window plus
decodable frequency bin; rhythm and gaps are additional evidence.
Impact on current work:
- `MimirChirpBinTimeline` is directionally right.
- The next work is streaming state and calibrated likelihood, not resurrecting
dense matched-filter banks.
## Live Adaptive Sync Archive
Location: `research/live-adaptive-sync/`
Durable lesson:
- Initial delay and sample-rate offset are separate loops.
- SRO is inevitable across independent clocks and must be estimated/compensated.
- Resampling/fractional delay is the actuator when hardware clocks cannot be
controlled.
- GCC-PHAT/TDOA is evidence for relative arrival and localization, not a
canonical timeline by itself.
Impact on current work:
- `MimirAudioSynchronizationState` needs to drive a real actuator.
- Delay/SRO estimates should be control inputs to Faust/native DSP.
- Passive mode remains valuable, but it does not replace active codebook time.
## Ambisonic / AquaSynth Archive
Location: `research/ambisonic-aquasynth/`
Durable lesson:
- Arbitrary microphones are not automatically an ambisonic mic.
- Ambisonic output needs declared order, channel ordering, normalization, sample
rate, and target.
- Source-based spatialization is the honest first path; arbitrary-array field
reconstruction requires calibration and regularization.
Impact on current work:
- Mimir should not promise high-order field truth until mic geometry and path
calibration prove it.
- Faust `hoa.lib`, IEM, SPARTA, and SAF are verification/reference tools.
- The initial output can be aligned stems plus a lower-order spatial bed.
## GPU Fusion / Splatting Archive
Location: `research/gpu-fusion-splatting/`
Durable lesson:
- Gaussian splatting is useful for packed GPU representations, tile/bin/sort
renderers, anisotropic radiance primitives, and dynamic field ideas.
- Mimir's live problem is not offline static-scene 3DGS training.
- CPU readback of dense visual state is poison for the hot path.
Impact on current work:
- Fensalir owns temporal Gaussian/evidence fields.
- Mimir should feed synchronized observations and constraints.
- The first visual fusion proof should be a small live observation-to-splat
update, not full 4D training.
## Playback / Feedback Calibration Archives
Locations:
- `research/playback-calibration/`
- `research/feedback-calibration/`
Durable lesson:
- Room/speaker/mic paths can be measured as impulse or transfer functions.
- Online acoustic system identification and adaptive echo cancellation literature
is relevant because Mimir is emitting and listening in the same room.
- Magnitude-only response is insufficient; phase/group delay are part of the
timing machine.
Impact on current work:
- The chirp-bin calibration model should keep becoming a path transfer model.
- Active witness design should support both timeline decoding and transfer
function estimation.
- Hybrid watermarking should learn from echo-cancellation and system-identification
practice rather than just hiding chirps under music.
## Visual Spatial Map Archive
Location: `research/visual-spatial-map/`
Durable lesson:
- Direct device paths and camera-specific quirks matter. Framework convenience
layers hide timing, buffering, exposure, and bandwidth behavior.
- Leap, Kiyos, and PS3 Eyes have different roles: timing/IR, RGB ground truth,
and high-rate tracking.
- Multi-camera calibration is a geometry problem before it is a rendering
problem.
Impact on current work:
- `IMimirVideoCaptureDriver` is the correct production seam.
- Native probes should be promoted into workers only after they preserve
timestamps, payload handles, and cadence.
- Fensalir should consume calibrated camera observations, not raw app-specific
UI experiments.
## Repository-Level Conclusion
The archives agree on one architecture:
```text
calibrated sensors -> rolling evidence window -> canonical time -> native/GPU/DSP lowering -> program outputs
```
They do not support:
- framework-mediated capture in the six-camera hot loop;
- generic chirplet transforms as the default controlled-beacon receiver;
- arbitrary ambisonic claims from unsynchronized heterogeneous mics;
- CPU-owned dense visual fusion;
- OBS as synchronization authority.

View file

@ -0,0 +1,99 @@
# Questions And Hypotheses
## Audio
### Q: Is the current physical failure mostly SNR, frequency response, or timing ambiguity?
Hypothesis:
- It is primarily response/confusion plus weak direct-path SNR, not a fundamental
codebook failure.
Test:
- Compare calibration-weighted likelihood on stored hot mic captures.
### Q: Is 192 kHz worth keeping for active sync?
Hypothesis:
- Yes for fractional timing, loopback, and high-frequency option space; no as a
magic acoustic fix when speakers/mics do not preserve ultrasonic energy.
Test:
- Run same reliable-bin codebook at 48/96/192 kHz through physical path and
compare anchor residuals, not just sample granularity.
### Q: Should chirp-only and hybrid share one decoder?
Hypothesis:
- Yes. Mode changes emission policy and confidence gating, not decode
semantics.
### Q: Can any three chirps always locate timeline position?
Hypothesis:
- Only if they are correctly classified inside the active codebook and the
receiver keeps enough timing/frequency spacing evidence. In meatspace,
"any three heard chirps" means "any three reliable decoded symbols after path
calibration," not raw peaks.
## Visual
### Q: Should Leap or PS3 Eye be first direct runtime driver?
Hypothesis:
- Leap is semantically highest value as timing/depth/near-field witness; PS3 Eye
may be lower-risk for proving direct high-rate runtime plumbing.
Test:
- Build the smaller driver first if it proves the seam faster, but do not lose
the Leap timing-camera goal.
### Q: Is realtime Gaussian splatting the first visual fusion target?
Hypothesis:
- No. The first target is GPU feature/claim publication from synchronized
frames. Splatting is the render/update representation after that.
## Distributed
### Q: Should Raven send raw audio/video or decoded observations?
Hypothesis:
- Both may be needed, but timing authority should come from locally decoded
codebook/schedule observations where possible. Raw streams are evidence; local
decoded anchors are stronger over variable network latency.
### Q: Can a phone know canonical time from codebook alone?
Hypothesis:
- Yes if it has schedule epoch, symbol plan, sample-rate estimate, and enough
reliable decoded anchors. It should report confidence and path model, not just
"time."
## Implementation
### Q: Is the native reservoir ready to replace C# buffers?
Hypothesis:
- Conceptually yes, operationally not yet. First mirror C# buffer proof through
reservoir status/views before making it the only path.
### Q: Where does micro-optimization start?
Hypothesis:
- Not with HLSL. First benchmark current active decode; if scalar bin scoring
dominates, AVX2/FMA scorer is likely the best first optimization. GPU wins
only when batches are large.

View file

@ -0,0 +1,132 @@
# Reading Guide
## Purpose
This directory is now large enough to need its own map. Use this guide when
entering a future implementation pass cold.
## If You Are Building The Active Decoder
Read in order:
1. `chirplet-transform-deep-dive.md`
2. `decoder-architecture-options.md`
3. `microsecond-sync-math.md`
4. `current-hotspot-audit.md`
5. `optimization-ledger.md`
6. `low-level-implementation-notes.md`
7. `benchmark-plan.md`
6. sample sketches:
- `samples/StreamingChirpBinDecoderSketch.cs`
- `samples/BatchedChirpScoreAbiSketch.h`
- `samples/CalibrationWeightedChirpLikelihoodSketch.cs`
- `samples/Avx2DechirpGoertzelSketch.cpp`
- `samples/ChirpBinScore.compute.hlsl`
Decision:
- implement streaming state first;
- preserve the existing window decoder as oracle;
- promote native/GPU scoring only after benchmarks.
## If You Are Building Meatspace Calibration
Read in order:
1. `volumetric-audio-field.md`
2. `acoustic-field-models.md`
3. `calibration-session-spec.md`
4. `prior-research-synthesis.md`
5. `benchmark-plan.md`
6. `references.md`
Decision:
- learn usable bands per path;
- shape symbol likelihood before codebook solving;
- correct magnitude and phase/group delay;
- adapt the emitted alphabet rather than filtering failure afterward.
## If You Are Building The Audio Actuator
Read in order:
1. `prior-research-synthesis.md`
2. `optimization-ledger.md`
3. `benchmark-plan.md`
4. sample sketches:
- `samples/FarrowFractionalDelaySketch.cpp`
- `samples/SroPllAsrcControllerSketch.cpp`
Decision:
- delay and SRO are separate loops;
- the runtime estimates state;
- Faust/native DSP moves samples.
## If You Are Building Camera Ingest
Read in order:
1. `native-boundary-map.md`
2. `visual-fusion-4dgs-study.md`
3. `low-level-implementation-notes.md`
4. `current-hotspot-audit.md`
5. `benchmark-plan.md`
6. `docs/native-capture-cadence.md`
7. `docs/native-rebuild-plan.md`
Decision:
- JSON probes are diagnostics;
- production capture is native workers plus payload handles;
- Fensalir owns GPU import and resource lifetime.
## If You Are Building Fensalir Integration
Read in order:
1. `fensalir-integration-map.md`
2. `native-boundary-map.md`
3. `visual-fusion-4dgs-study.md`
4. sample sketches:
- `samples/AquariumGpuSensorFrameBridgeSketch.cs`
- `samples/AcousticConstraintLoweringSketch.cs`
- `samples/TemporalEvidenceCandidateSketch.cs`
Decision:
- Mimir lowers observations/constraints;
- Fensalir owns temporal evidence and render state;
- the render tick must not run capture or analysis.
## If You Are Looking For The Next Cut
Read in order:
1. `implementation-roadmap.md`
2. `option-matrix.md`
3. `research-claims-digest.md`
4. `questions-and-hypotheses.md`
5. `distributed-receiver-spec.md`
6. `current-hotspot-audit.md`
Recommended next code cut:
- streaming active decoder state with calibration-weighted likelihood;
- benchmark it against the current window decoder;
- then wire the actuator.
## If You Are Updating Public Docs
Read:
1. `docs/perfect-machine-domain-index.md`
2. `docs/code-algorithm-map.md`
3. `notes/current-system-map.md`
4. this directory's `index.md`
Rule:
- public docs describe the live machine and current constraints;
- research docs may hold scars, rejected paths, and implementation options.

View file

@ -9,6 +9,10 @@
https://www.sciencedirect.com/science/article/pii/S0888327015003994
- High-resolution chirplet transform parameter analysis:
https://arxiv.org/abs/2108.00572
- Fast Chirplet Transform with FPGA-based implementation:
https://www.ece.iit.edu/~eoruklu/IIT/Publications_files/04639630.pdf
- Speed-optimized fast chirplet implementation:
https://ecasp.ece.iit.edu/publications/2012-present/2023-13.pdf
## Chirp Spread Spectrum / LoRa Receiver Shape
@ -42,6 +46,10 @@
- HLSL Shader Model 6 wave intrinsics:
https://learn.microsoft.com/windows/win32/direct3dhlsl/hlsl-shader-model-6-0-features-for-direct3d-12
- Microsoft Kernel Streaming / AVStream overview:
https://learn.microsoft.com/en-gb/windows-hardware/drivers/stream/kernel-streaming
- DXGI shared resource handle creation:
https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_2/nf-dxgi1_2-idxgiresource1-createsharedhandle
- DirectX Shader Compiler wave intrinsics notes:
https://github.com/microsoft/directxshadercompiler/wiki/wave-intrinsics
- cuFFT callback documentation:
@ -55,6 +63,82 @@
- Chirp Z-transform theory:
https://pyffs.readthedocs.io/en/stable/theory/CZT.html
## Volumetric Audio / Room Acoustics / Actuation
- Pyroomacoustics beamforming module:
https://pyroomacoustics.readthedocs.io/en/pypi-release/pyroomacoustics.beamforming.html
- Pyroomacoustics repository and room/array algorithm examples:
https://github.com/LCAV/pyroomacoustics
- Pyroomacoustics paper:
https://arxiv.org/abs/1710.04196
- ManyEars sound source localization, tracking, and separation:
https://github.com/introlab/manyears
- Spatial Audio Framework, open C/C++ spatial-audio toolkit:
https://github.com/leomccormack/Spatial_Audio_Framework
- Sound field reconstruction in reverberant environments with rigid spherical
microphone arrays:
https://journals.sagepub.com/doi/10.1177/14613484261447569
- Sparse Reconstruction of Sound Field Using Bayesian Compressive Sensing and
Equivalent Source Method:
https://pmc.ncbi.nlm.nih.gov/articles/PMC10301025/
- Near-field acoustic holography with compressive sensing:
https://docs.lib.purdue.edu/herrick/208/
- Ambisonic encoding from equatorial microphone arrays:
https://arxiv.org/abs/2211.00584
- Spheroidal ambisonics:
https://arxiv.org/abs/2103.05719
- Real-time spherical array renderer:
https://research.chalmers.se/publication/509494
- Farrow fractional delay overview:
https://engee.com/helpcenter/stable/en/interactive-scripts/dsp/fractional_delay.html
- AMD Vitis fractional-delay Farrow filter note:
https://docs.amd.com/r/2024.1-English/Vitis-Tutorials-AI-Engine-Development/Fractional-Delay-Farrow-Filter
- Faust delay/fractional delay reference:
https://ringbuffer.org/faust/faust_basics/faust-delay/
- MATLAB Farrow fractional delay example:
https://www.mathworks.com/help/dsp/ug/fractional-delay-filters-using-farrow-structures.html
- Joint SFO estimation and compensation based on Farrow structure:
https://arxiv.org/abs/2503.07577
- Joint SFO estimation and compensation algorithms based on Farrow structure:
https://arxiv.org/abs/2603.00627
## Realtime Gaussian Splatting / Dynamic Visual Fields
- Official 3D Gaussian Splatting implementation:
https://github.com/graphdeco-inria/gaussian-splatting
- 3D Gaussian Splatting paper:
https://arxiv.org/abs/2308.04079
- 4D Gaussian Splatting implementation:
https://github.com/fudan-zvg/4d-gaussian-splatting
- 4D Gaussian Splatting paper:
https://openaccess.thecvf.com/content/CVPR2024/papers/Wu_4D_Gaussian_Splatting_for_Real-Time_Dynamic_Scene_Rendering_CVPR_2024_paper.pdf
- Spacetime Gaussian Feature Splatting implementation:
https://github.com/oppo-us-research/SpacetimeGaussians
- Spacetime Gaussian Feature Splatting paper:
https://arxiv.org/abs/2312.16812
- PlayCanvas engine, an open WebGL/WebGPU engine with Gaussian splatting support:
https://github.com/playcanvas/engine
- PlayCanvas generic Gaussian splat processing docs:
https://developer.playcanvas.com/user-manual/gaussian-splatting/building/unified-rendering/splat-processing/
- NVIDIA vk_gaussian_splatting technical blog:
https://developer.nvidia.com/blog/real-time-gpu-accelerated-gaussian-splatting-with-nvidia-designworks-sample-vk_gaussian_splatting/
- gsplat documentation:
https://docs.gsplat.studio/main/
- gsplat rasterization API:
https://docs.gsplat.studio/main/apis/rasterization.html
- gsplat paper:
https://arxiv.org/abs/2409.06765
- StopThePop sorted Gaussian splatting implementation:
https://github.com/r4dl/StopThePop
- FlashGS paper:
https://arxiv.org/abs/2408.07967
- Gaussian-LIC, Gaussian splatting plus LiDAR-inertial-camera SLAM:
https://arxiv.org/abs/2404.06926
- RMGS-SLAM, real-time multi-sensor Gaussian splatting SLAM:
https://arxiv.org/abs/2604.12942
- Multi-Calib, online multi-sensor calibration direction:
https://pmc.ncbi.nlm.nih.gov/articles/PMC12693731/
## How These References Steer Mimir
- Dechirp-plus-bin scoring is the right hot active receiver shape for controlled
@ -67,4 +151,6 @@
a small fixed bin bank, SIMD Goertzel/dechirp may beat FFT overhead.
- If a transform path moves to GPU, fuse dechirp/window/preprocess with transform
load/store where possible to avoid memory bandwidth waste.
- Visual fusion should borrow Gaussian splatting data/layout ideas, but the live
Mimir problem is online synchronized evidence update, not offline novel-view
training from a static capture folder.

View file

@ -0,0 +1,168 @@
# Research Claims Digest
## Purpose
This is the claim-level digest from the research pass. It records what the
external literature and open implementations change about Mimir's design.
## Chirplets / CSS / Receiver Design
Claim:
- A generic chirplet transform is the broad signal-analysis tool.
- A controlled chirp communication receiver should usually exploit waveform
design: dechirp, bin score, then decode.
Mimir consequence:
- The hot receiver should remain constrained by the known codebook and schedule.
- Full chirplet transforms are reference tools, not first-line runtime code.
- Symbol design should serve demodulation.
References:
- Steve Mann chirplet transform.
- Fast chirplet transform / FPGA implementation work.
- LoRa/CSS demodulator literature and OpenLoRa.
## Passive Delay / Distributed Microphones
Claim:
- Delay and sampling-rate offset are different phenomena.
- Distributed microphone arrays require explicit SRO estimation/compensation.
- GCC-PHAT is useful for TDOA/delay evidence, not enough to own canonical time.
Mimir consequence:
- Passive mode is a confidence-bearing witness.
- Active codebook anchors remain the canonical-time path.
- The actuator must include both fractional delay and SRO control.
References:
- Wang/Doclo SRO estimation.
- Schmalenstroeer/Haeb-Umbach clock skew work.
- Didier distributed adaptive node-specific signal estimation.
- GCC-PHAT refinements.
## Room / Sound Field Reconstruction
Claim:
- Sound-field reconstruction from sparse arrays is an inverse problem with
geometry, bandwidth, regularization, and noise constraints.
- Ambisonics works cleanly when the microphone array and calibration support the
basis.
- Equivalent-source / sparse reconstruction methods are useful when the source
model is sparse and measurement model is known.
Mimir consequence:
- Six heterogeneous mics are not automatically a high-order field mic.
- The first honest target is synchronized stems plus source/constraint estimates.
- Field reconstruction should expose uncertainty and avoid pretending unsupported
spatial detail exists.
References:
- Pyroomacoustics.
- Spatial Audio Framework.
- ManyEars.
- near-field acoustic holography / equivalent source / compressive sensing
papers.
- ambisonics/arbitrary-array encoding notes.
## Fractional Delay / SRO Actuation
Claim:
- Farrow structures are standard for continuously variable fractional delay.
- Variable-rate resampling or equivalent phase correction is required for drift.
Mimir consequence:
- Microsecond reporting is incomplete until an actuator moves samples.
- A Farrow proof is the right first native/Faust actuator slice.
- Higher-quality polyphase/ASRC work comes after the control loop is proven.
References:
- AMD Vitis Farrow filter docs.
- MATLAB Farrow fractional delay notes.
- joint SFO/Farrow papers.
## Gaussian Splatting / Visual Fusion
Claim:
- Practical 3DGS renderers use packed GPU buffers, projection, tile binning,
sorting/order strategies, and optimized rasterization kernels.
- Dynamic/4D Gaussian splatting extends splats over time, but production code is
still research-shaped.
- Online SLAM/sensor fusion variants are more relevant to Mimir than offline
static scene training alone.
Mimir consequence:
- Fensalir should own the GPU field.
- Mimir should feed synchronized observations and constraints.
- The first proof is live observation-to-claim-to-render, not full 4DGS training.
References:
- GraphDeco 3DGS.
- 4D Gaussian Splatting.
- Spacetime Gaussians.
- gsplat.
- NVIDIA `vk_gaussian_splatting`.
- StopThePop.
- FlashGS.
- Gaussian-SLAM / multi-sensor calibration papers.
## Native Capture / GPU Resource Boundaries
Claim:
- Windows KS/AVStream is the low-level video streaming surface under higher media
frameworks.
- D3D shared handles are the relevant resource-sharing family once camera
payloads can become GPU resources.
- ASIO callbacks and sample positions are the correct place to preserve driver
clock timing.
Mimir consequence:
- Media frameworks may remain diagnostics/fallbacks, but direct workers own
production capture.
- Fensalir owns D3D12 resource lifetime and imports.
- Mimir should not route six cameras through stdout/JSON or managed pixel
processing.
References:
- Microsoft Kernel Streaming docs.
- Microsoft DXGI shared handle docs.
- ASIO specification / callback notes.
- Focusrite channel/loopback documentation.
## Strongest Design Conclusion
The research converges on one architecture:
```text
controlled witness + calibrated sensors + rolling buffers
-> deterministic timeline/response models
-> native DSP and GPU field lowering
-> OBS-facing program output
```
The critical missing production step is not another filter. It is connecting
measurement to actuation and field lowering:
- streaming decoder state;
- calibration-weighted likelihood;
- fractional delay/SRO actuator;
- native camera payload handles;
- Fensalir contract lowering.

View file

@ -0,0 +1,97 @@
{
"schema": "gamecult.mimir.sample_code_index.v1",
"created": "2026-05-23",
"samples": [
{
"path": "research/perfect-machine-study-2026-05-23/samples/StreamingChirpBinDecoderSketch.cs",
"domain": "audio-sync",
"purpose": "Incremental chirp-bin decoder state model",
"productionTarget": "Mimir.Runtime streaming active decoder",
"promoteWhen": "Matches current analyzer outputs while avoiding full-window rescans"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/Avx2DechirpGoertzelSketch.cpp",
"domain": "audio-sync",
"purpose": "AVX2/FMA dechirp plus bin scoring sketch",
"productionTarget": "native chirp-bin scorer",
"promoteWhen": "Profiling shows C# scalar bin scoring dominates"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/BatchedChirpScoreAbiSketch.h",
"domain": "audio-sync",
"purpose": "Native batched scoring ABI boundary for chirp candidate/bin work",
"productionTarget": "Mimir native scorer interop layer",
"promoteWhen": "Scalar scoring dominates and batching proves interop cost is amortized"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/ChirpBinScore.compute.hlsl",
"domain": "audio-sync-gpu",
"purpose": "D3D12 compute candidate/bin scoring sketch",
"productionTarget": "Fensalir compute scoring pass",
"promoteWhen": "Candidate/bin/channel batches are large enough to beat CPU SIMD"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/FarrowFractionalDelaySketch.cpp",
"domain": "audio-dsp",
"purpose": "Cubic Farrow fractional delay actuator sketch",
"productionTarget": "Faust/native DSP alignment actuator",
"promoteWhen": "Runtime state can drive a test resampler/delay line with measured improvement"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/SpscAudioBlockRingSketch.cpp",
"domain": "native-capture",
"purpose": "Low-jitter SPSC audio block handoff sketch",
"productionTarget": "native ASIO/camera capture handoff",
"promoteWhen": "Managed queues or payload copies show measurable capture pressure"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/CalibrationWeightedChirpLikelihoodSketch.cs",
"domain": "audio-sync",
"purpose": "Calibration-weighted symbol likelihood before codebook solving",
"productionTarget": "MimirChirpBinTimeline scoring path",
"promoteWhen": "Physical mic captures show unweighted symbol likelihood is the failure point"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/SroPllAsrcControllerSketch.cpp",
"domain": "audio-dsp",
"purpose": "Control loop turning delay observations into ASRC/fractional delay commands",
"productionTarget": "Faust/native DSP actuator control",
"promoteWhen": "Runtime reports are stable enough to move samples instead of only logging state"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/GaussianSplatClaimUpdate.compute.hlsl",
"domain": "visual-fusion",
"purpose": "D3D12 compute sketch for live splat claim updates",
"productionTarget": "Fensalir visual fusion pass",
"promoteWhen": "Direct camera drivers feed synchronized feature observations"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/AquariumGpuSensorFrameBridgeSketch.cs",
"domain": "fensalir-integration",
"purpose": "Lower runtime video observations into Fensalir GPU sensor frame contracts",
"productionTarget": "Mimir app-to-Fensalir bridge",
"promoteWhen": "Runtime buffers carry pixel/native handles and lowering benchmarks allocation-free"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/AcousticConstraintLoweringSketch.cs",
"domain": "fensalir-integration",
"purpose": "Lower decoded audio anchors and mic geometry into Fensalir acoustic constraints",
"productionTarget": "Mimir acoustic field bridge",
"promoteWhen": "Decoder reports stable anchors and Fensalir consumes constraints in diagnostics"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/TemporalEvidenceCandidateSketch.cs",
"domain": "sensor-fusion",
"purpose": "Convert Mimir observations into reservoir candidates without owning stable tracks",
"productionTarget": "Fensalir temporal evidence lowering path",
"promoteWhen": "Mimir emits synchronized visual/acoustic candidates and Fensalir reservoir accepts them"
},
{
"path": "research/perfect-machine-study-2026-05-23/samples/SrpPhatGridSketch.cpp",
"domain": "volumetric-audio",
"purpose": "Minimal grid scoring from pairwise mic delays to source-position hypotheses",
"productionTarget": "native acoustic source localization proof",
"promoteWhen": "At least two physical mics produce calibrated stable delay anchors"
}
]
}

View file

@ -0,0 +1,104 @@
// Sketch only. Purpose: keep acoustic raw buffers in Mimir/native DSP while
// sending Fensalir compact constraints for spatial fusion and diagnostics.
using System;
using System.Collections.Generic;
namespace Mimir.PerfectMachineStudy;
public readonly record struct MimirDecodedAnchorSketch(
string SourceId,
int EventIndex,
double SourceSampleTimeSeconds,
double CanonicalTimeSeconds,
double DelaySeconds,
double FrequencyHz,
float Confidence,
float PhaseRadians,
float GroupDelaySeconds);
public readonly record struct MimirMicGeometrySketch(
string SourceId,
float X,
float Y,
float Z);
public sealed class AcousticConstraintLoweringSketch
{
private readonly List<AquariumAcousticConstraintSketch> _constraints = new(128);
public AquariumAcousticFieldFrameSketch Lower(
ReadOnlySpan<MimirDecodedAnchorSketch> anchors,
ReadOnlySpan<MimirMicGeometrySketch> geometry,
double canonicalNowSeconds)
{
_constraints.Clear();
for (var i = 0; i < anchors.Length; i++)
{
ref readonly var anchor = ref anchors[i];
if (anchor.Confidence < 0.4f)
{
continue;
}
var mic = FindGeometry(geometry, anchor.SourceId);
if (mic.SourceId is null)
{
continue;
}
_constraints.Add(new AquariumAcousticConstraintSketch(
anchor.SourceId,
anchor.EventIndex,
canonicalNowSeconds,
anchor.CanonicalTimeSeconds,
anchor.DelaySeconds,
anchor.FrequencyHz,
mic.X,
mic.Y,
mic.Z,
anchor.Confidence,
anchor.PhaseRadians,
anchor.GroupDelaySeconds));
}
// Production version should avoid this copy and hand Fensalir a stable
// span/packet buffer whose lifetime is owned by the frame boundary.
return new AquariumAcousticFieldFrameSketch(canonicalNowSeconds, _constraints.ToArray());
}
private static MimirMicGeometrySketch FindGeometry(
ReadOnlySpan<MimirMicGeometrySketch> geometry,
string sourceId)
{
for (var i = 0; i < geometry.Length; i++)
{
if (StringComparer.Ordinal.Equals(geometry[i].SourceId, sourceId))
{
return geometry[i];
}
}
return default;
}
}
public readonly record struct AquariumAcousticFieldFrameSketch(
double CanonicalTimeSeconds,
AquariumAcousticConstraintSketch[] Constraints);
public readonly record struct AquariumAcousticConstraintSketch(
string SourceId,
int EventIndex,
double FrameCanonicalTimeSeconds,
double EventCanonicalTimeSeconds,
double DelaySeconds,
double FrequencyHz,
float MicX,
float MicY,
float MicZ,
float Confidence,
float PhaseRadians,
float GroupDelaySeconds);

View file

@ -0,0 +1,78 @@
// Sketch only. This is not wired to production contracts yet.
// Purpose: show the desired shape of the Mimir -> Fensalir camera lowering cut.
using System;
using System.Collections.Generic;
namespace Mimir.PerfectMachineStudy;
public readonly record struct MimirVideoObservation(
string SourceId,
long Sequence,
double CanonicalTimeSeconds,
double DeviceTimeSeconds,
int Width,
int Height,
string PixelFormat,
IntPtr NativeHandle,
int NativeHandleKind,
string CalibrationId,
float Confidence);
public sealed class AquariumGpuSensorFrameBridgeSketch
{
private readonly List<AquariumGpuSensorCameraSketch> _cameras = new(8);
public AquariumGpuSensorFrameSketch Lower(
ReadOnlySpan<MimirVideoObservation> observations,
double frameCanonicalTimeSeconds)
{
_cameras.Clear();
for (var i = 0; i < observations.Length; i++)
{
ref readonly var source = ref observations[i];
if (source.NativeHandle == IntPtr.Zero || source.Confidence <= 0.0f)
{
continue;
}
_cameras.Add(new AquariumGpuSensorCameraSketch(
source.SourceId,
source.Sequence,
source.CanonicalTimeSeconds,
source.DeviceTimeSeconds,
source.Width,
source.Height,
source.PixelFormat,
source.NativeHandle,
source.NativeHandleKind,
source.CalibrationId,
source.Confidence));
}
// Production version should fill a reusable contract-owned array or pooled
// struct buffer instead of allocating here.
return new AquariumGpuSensorFrameSketch(
frameCanonicalTimeSeconds,
_cameras.ToArray());
}
}
public readonly record struct AquariumGpuSensorFrameSketch(
double CanonicalTimeSeconds,
AquariumGpuSensorCameraSketch[] Cameras);
public readonly record struct AquariumGpuSensorCameraSketch(
string SourceId,
long Sequence,
double CanonicalTimeSeconds,
double DeviceTimeSeconds,
int Width,
int Height,
string PixelFormat,
IntPtr NativeHandle,
int NativeHandleKind,
string CalibrationId,
float Confidence);

View file

@ -0,0 +1,53 @@
// Sketch only. Purpose: native ABI shape for batched chirp-bin scoring.
// The point is to batch enough work that managed/native transition cost does
// not dominate the actual multiply-adds.
#pragma once
#include <cstdint>
extern "C"
{
struct MimirChirpScoreCandidate
{
std::int64_t absoluteSampleOffset;
std::int32_t sampleOffsetInBuffer;
std::int32_t sampleCount;
float proposalEnergy;
};
struct MimirChirpScoreKernel
{
const float* window;
const float* dechirpReal;
const float* dechirpImag;
std::int32_t sampleCount;
};
struct MimirChirpScoreRequest
{
const float* samples;
std::int32_t sampleCount;
const MimirChirpScoreCandidate* candidates;
std::int32_t candidateCount;
const MimirChirpScoreKernel* kernels;
std::int32_t kernelCount;
std::int32_t topK;
};
struct MimirChirpScoreResult
{
std::int64_t absoluteSampleOffset;
std::int32_t kernelIndex;
float energy;
float phaseRadians;
};
// Returns the number of result rows written. Production ABI would also
// return an error/status code and expose required result capacity.
std::int32_t mimir_score_chirp_candidates(
const MimirChirpScoreRequest* request,
MimirChirpScoreResult* results,
std::int32_t resultCapacity);
}

View file

@ -0,0 +1,105 @@
// Sketch only. This shows where meatspace calibration should enter the active
// decoder: before code-valid path selection, as likelihood shaping.
namespace Mimir.Research.Samples;
public sealed class CalibrationWeightedChirpLikelihoodSketch
{
private readonly PathCalibration calibration;
public CalibrationWeightedChirpLikelihoodSketch(PathCalibration calibration)
{
this.calibration = calibration;
}
public SymbolLikelihood[] ScoreFrame(FrameBinEnergy frame)
{
var result = new SymbolLikelihood[calibration.Symbols.Length];
for (var symbolId = 0; symbolId < calibration.Symbols.Length; symbolId++)
{
var symbol = calibration.Symbols[symbolId];
var observed = frame.EnergyByBin[symbolId];
var reliability = symbol.Reliability;
var confusionPenalty = ConfusionPenalty(symbolId, frame);
var phaseWeight = PhaseWeight(symbol, frame.PhaseByBin[symbolId]);
var delayCorrection = symbol.GroupDelaySamples;
var logLikelihood =
Math.Log(Math.Max(observed, 1.0e-12)) * reliability +
Math.Log(Math.Max(phaseWeight, 1.0e-6)) -
confusionPenalty;
result[symbolId] = new SymbolLikelihood(
symbolId,
logLikelihood,
delayCorrection,
reliability,
observed);
}
return result
.OrderByDescending(score => score.LogLikelihood)
.Take(calibration.MaxCandidatesPerFrame)
.ToArray();
}
private double ConfusionPenalty(int expectedSymbol, FrameBinEnergy frame)
{
var penalty = 0.0;
foreach (var alias in calibration.Confusions.Where(c => c.ExpectedSymbol == expectedSymbol))
{
var aliasEnergy = frame.EnergyByBin[alias.ObservedSymbol];
penalty += alias.Probability * Math.Log1p(aliasEnergy);
}
return penalty;
}
private static double PhaseWeight(SymbolCalibration symbol, double observedPhase)
{
if (symbol.PhaseCoherence <= 0.0)
{
return 1.0;
}
var distance = AngularDistance(observedPhase, symbol.MeanPhaseRadians);
return 1.0 - symbol.PhaseCoherence * Math.Min(1.0, distance / Math.PI);
}
private static double AngularDistance(double a, double b)
{
var d = Math.Abs(a - b) % (Math.PI * 2.0);
return d > Math.PI ? Math.PI * 2.0 - d : d;
}
}
public sealed record FrameBinEnergy(
double[] EnergyByBin,
double[] PhaseByBin,
double InitialOffsetSamples);
public sealed record PathCalibration(
SymbolCalibration[] Symbols,
Confusion[] Confusions,
int MaxCandidatesPerFrame);
public sealed record SymbolCalibration(
int SymbolId,
double Reliability,
double MeanEnergy,
double MeanPhaseRadians,
double PhaseCoherence,
double GroupDelaySamples);
public sealed record Confusion(
int ExpectedSymbol,
int ObservedSymbol,
double Probability);
public sealed record SymbolLikelihood(
int SymbolId,
double LogLikelihood,
double GroupDelayCorrectionSamples,
double Reliability,
double ObservedEnergy);

View file

@ -0,0 +1,68 @@
// Sketch only. Fensalir-side compute shape for updating simple live splat
// claims from synchronized feature observations.
struct FeatureObservation
{
uint StableKeyLow;
uint StableKeyHigh;
float3 PositionMeters;
float3 Normal;
float4 Color;
float Confidence;
uint SourceIndex;
uint TimestampLo;
uint TimestampHi;
};
struct SplatClaim
{
uint StableKeyLow;
uint StableKeyHigh;
float3 PositionMeters;
float RadiusMeters;
float4 Color;
float Confidence;
uint TimeMinLo;
uint TimeMinHi;
uint TimeMaxLo;
uint TimeMaxHi;
};
cbuffer SplatUpdateConstants : register(b0)
{
uint ObservationCount;
float Decay;
float Blend;
float DefaultRadius;
};
StructuredBuffer<FeatureObservation> Observations : register(t0);
RWStructuredBuffer<SplatClaim> Claims : register(u0);
[numthreads(128, 1, 1)]
void main(uint3 id : SV_DispatchThreadID)
{
uint index = id.x;
if (index >= ObservationCount)
{
return;
}
FeatureObservation obs = Observations[index];
// Sketch simplification: production needs a hash table or sorted stable-key
// compaction pass. This writes one observation to one claim slot.
SplatClaim claim;
claim.StableKeyLow = obs.StableKeyLow;
claim.StableKeyHigh = obs.StableKeyHigh;
claim.PositionMeters = obs.PositionMeters;
claim.RadiusMeters = DefaultRadius;
claim.Color = obs.Color;
claim.Confidence = saturate(obs.Confidence);
claim.TimeMinLo = obs.TimestampLo;
claim.TimeMinHi = obs.TimestampHi;
claim.TimeMaxLo = obs.TimestampLo;
claim.TimeMaxHi = obs.TimestampHi;
Claims[index] = claim;
}

View file

@ -9,14 +9,30 @@ machinery into the app.
- `StreamingChirpBinDecoderSketch.cs`: C# state model for an incremental
chirp-bin decoder that avoids full-window rescans.
- `Avx2DechirpGoertzelSketch.cpp`: AVX2/FMA-shaped CPU scoring kernel sketch.
- `BatchedChirpScoreAbiSketch.h`: native scoring ABI shape for batching
candidate/bin work across the managed/native boundary.
- `ChirpBinScore.compute.hlsl`: D3D12 compute shader shape for batched
candidate/bin scoring.
- `FarrowFractionalDelaySketch.cpp`: native fractional delay actuator sketch.
- `SpscAudioBlockRingSketch.cpp`: native single-producer/single-consumer audio
block ring sketch for capture worker handoff.
- `CalibrationWeightedChirpLikelihoodSketch.cs`: shows how response/confusion/
phase/group-delay calibration should shape symbol likelihood before codebook
path selection.
- `SroPllAsrcControllerSketch.cpp`: control-rate delay/SRO loop for driving an
ASRC or fractional-delay actuator.
- `GaussianSplatClaimUpdate.compute.hlsl`: minimal Fensalir compute shape for
turning synchronized visual observations into live splat claims.
- `AquariumGpuSensorFrameBridgeSketch.cs`: contract sketch for lowering runtime
video observations into Fensalir GPU sensor frames.
- `AcousticConstraintLoweringSketch.cs`: contract sketch for lowering decoded
chirp/passive anchors into Fensalir acoustic constraints.
- `TemporalEvidenceCandidateSketch.cs`: candidate lowering sketch that keeps
stable track ownership in Fensalir.
- `SrpPhatGridSketch.cpp`: minimal TDOA/SRP-PHAT-style grid scoring proof for a
synchronized mic constellation.
## Rule
Promote one of these only after a benchmark or integration test proves it owns a
real invariant. Sample code is a microscope, not a new wall in the house.

View file

@ -0,0 +1,60 @@
// Sketch only. Control-rate PLL for driving an ASRC/fractional delay actuator.
#include <algorithm>
#include <cmath>
struct SyncObservation
{
double delaySamples;
double confidence;
double dtSeconds;
};
struct ActuatorCommand
{
double targetDelaySamples;
double resampleRatio;
double confidence;
};
class SroPllAsrcController
{
public:
ActuatorCommand update(const SyncObservation& obs)
{
if (obs.confidence < 0.05 || obs.dtSeconds <= 0.0)
{
return command(obs.confidence);
}
double error = obs.delaySamples - delaySamples_;
double alpha = clamp(obs.confidence * 0.08, 0.005, 0.05);
double beta = clamp(obs.confidence * 0.01, 0.0005, 0.006);
delaySamples_ += alpha * error;
sroPpm_ += beta * error / obs.dtSeconds;
sroPpm_ = clamp(sroPpm_, -300.0, 300.0);
confidence_ += (obs.confidence - confidence_) * 0.05;
return command(confidence_);
}
private:
double delaySamples_ = 0.0;
double sroPpm_ = 0.0;
double confidence_ = 0.0;
ActuatorCommand command(double confidence) const
{
// Positive SRO means candidate appears to drift later; playback/capture
// correction convention must be fixed against the DSP graph before use.
double ratio = 1.0 - sroPpm_ * 1.0e-6;
return { delaySamples_, ratio, confidence };
}
static double clamp(double value, double lo, double hi)
{
return std::max(lo, std::min(hi, value));
}
};

View file

@ -0,0 +1,95 @@
// Sketch only. Purpose: first acoustic source-position proof after mic timing is
// calibrated. This is intentionally small and direct, not a full acoustic camera.
#include <cmath>
#include <cstdint>
#include <span>
struct MicPoseSketch
{
float x;
float y;
float z;
};
struct CandidatePointSketch
{
float x;
float y;
float z;
};
struct PairDelaySketch
{
int micA;
int micB;
double delaySeconds;
float confidence;
};
struct SrpResultSketch
{
int bestIndex;
float score;
};
static inline double distanceMeters(
const MicPoseSketch& mic,
const CandidatePointSketch& point)
{
const double dx = static_cast<double>(point.x) - mic.x;
const double dy = static_cast<double>(point.y) - mic.y;
const double dz = static_cast<double>(point.z) - mic.z;
return std::sqrt(dx * dx + dy * dy + dz * dz);
}
SrpResultSketch scoreGrid(
std::span<const MicPoseSketch> mics,
std::span<const CandidatePointSketch> grid,
std::span<const PairDelaySketch> observedPairs,
double speedOfSoundMetersPerSecond)
{
auto best = SrpResultSketch{-1, -1.0f};
for (int pointIndex = 0; pointIndex < static_cast<int>(grid.size()); pointIndex++)
{
const auto& point = grid[static_cast<size_t>(pointIndex)];
double score = 0.0;
double weight = 0.0;
for (const auto& pair : observedPairs)
{
if (pair.micA < 0 || pair.micB < 0 ||
pair.micA >= static_cast<int>(mics.size()) ||
pair.micB >= static_cast<int>(mics.size()) ||
pair.confidence <= 0.0f)
{
continue;
}
const double da = distanceMeters(mics[static_cast<size_t>(pair.micA)], point);
const double db = distanceMeters(mics[static_cast<size_t>(pair.micB)], point);
const double predicted = (db - da) / speedOfSoundMetersPerSecond;
const double errorUs = std::abs(predicted - pair.delaySeconds) * 1'000'000.0;
// Smooth robust score: 1 at zero error, half around 100 us.
const double local = 1.0 / (1.0 + errorUs / 100.0);
score += local * pair.confidence;
weight += pair.confidence;
}
if (weight <= 0.0)
{
continue;
}
const float normalized = static_cast<float>(score / weight);
if (normalized > best.score)
{
best = SrpResultSketch{pointIndex, normalized};
}
}
return best;
}

View file

@ -0,0 +1,80 @@
// Sketch only. Purpose: show how Mimir evidence can be lowered as candidates
// without becoming a second stable-track database beside Fensalir.
using System;
namespace Mimir.PerfectMachineStudy;
public enum MimirEvidenceKindSketch
{
Unknown = 0,
VisualFeature = 1,
ChirpAnchor = 2,
PassiveAudioAnchor = 3,
AcousticSource = 4
}
public readonly record struct MimirEvidenceCandidateSketch(
MimirEvidenceKindSketch Kind,
ulong StableSourceKey,
double CanonicalTimeSeconds,
float X,
float Y,
float Z,
float Radius,
float Confidence,
float Residual,
uint PayloadOffset,
uint PayloadLength);
public readonly record struct FensalirReservoirCandidateSketch(
ulong StableSourceKey,
double CanonicalTimeSeconds,
float X,
float Y,
float Z,
float Radius,
float Weight,
uint Domain,
uint PayloadOffset,
uint PayloadLength);
public static class TemporalEvidenceCandidateLoweringSketch
{
public static int Lower(
ReadOnlySpan<MimirEvidenceCandidateSketch> input,
Span<FensalirReservoirCandidateSketch> output)
{
var count = 0;
for (var i = 0; i < input.Length && count < output.Length; i++)
{
ref readonly var candidate = ref input[i];
if (candidate.Confidence <= 0.0f || candidate.Radius <= 0.0f)
{
continue;
}
var residualPenalty = 1.0f / (1.0f + MathF.Max(0.0f, candidate.Residual));
var weight = candidate.Confidence * residualPenalty;
if (weight <= 0.0f)
{
continue;
}
output[count++] = new FensalirReservoirCandidateSketch(
candidate.StableSourceKey,
candidate.CanonicalTimeSeconds,
candidate.X,
candidate.Y,
candidate.Z,
candidate.Radius,
weight,
(uint)candidate.Kind,
candidate.PayloadOffset,
candidate.PayloadLength);
}
return count;
}
}

View file

@ -0,0 +1,178 @@
# State Machine Invariants
## Purpose
The Perfect Machine is mostly a state discipline problem. This note lists the
state each subsystem may own and the state it must not smuggle in through a
friendly-looking helper.
## Runtime State
`MimirRuntime` may own:
- runtime lifecycle;
- loaded configuration;
- witness emission policy;
- Fensalir scene state assembly;
- UI/debug readout surfaces.
It must not own:
- raw driver callback loops;
- audio DSP sample movement;
- dense visual tracks;
- OBS composition.
## Hub State
`MimirSynchronizationHub` may own:
- registered sources;
- rolling buffer set;
- polling/update cadence;
- latest reports;
- smoothed synchronization states.
It must not own:
- analyzer internals;
- render tick work;
- hidden per-source DSP buffers;
- long-lived history outside retention window.
## Rolling Buffer State
Each rolling buffer owns:
- one stream descriptor;
- bounded sample window;
- current edge;
- snapshot semantics.
It must not own:
- a second clock model;
- per-kind retention policy;
- decoded field tracks;
- calibration truth.
## Active Decoder State
The streaming decoder should own:
- PCM window/ring;
- event proposal ring;
- candidate ambiguity ring;
- codebook/schedule view;
- local clock fit;
- response/confusion accumulation.
It must not own:
- UI state;
- emitter policy;
- DSP actuator state;
- global app timing authority.
## Calibration State
Calibration owns:
- output/mic path identity;
- usable bands;
- confusion matrix;
- delay/bin-shift hypotheses;
- magnitude response;
- phase/group delay;
- recommended emission plan.
It must not own:
- raw capture buffers forever;
- UI-specific names as primary identity;
- unverified learned patches that cannot be replayed.
## DSP Actuator State
The actuator should own:
- fractional delay line state;
- resampler phase;
- SRO control state;
- per-source gain/phase correction;
- audio-rate buffers.
It must not own:
- source discovery;
- calibration file persistence;
- UI controls except exposed parameters;
- timeline decoding.
## Native Capture State
Native capture workers own:
- device handles;
- callback/read state;
- block/frame rings;
- device timestamps/counters;
- low-level format conversion where unavoidable.
They must not own:
- canonical timeline decisions;
- OBS routing;
- Fensalir render state;
- long-term calibration policy.
## Fensalir State
Fensalir owns:
- window/input/render loop;
- D3D12 resource lifetime;
- external texture import;
- GPU fusion buffers;
- temporal evidence reservoir after lowering;
- program render surfaces.
It must not own:
- source capture configuration;
- audio clock authority;
- Mimir calibration truth;
- app-specific device policy.
## Network Receiver State
Remote nodes own:
- local PCM ring;
- local decode state;
- local clock fit;
- local response/confusion observations;
- transport health.
They must not own:
- Starfire canonical clock authority;
- program output mixing;
- global calibration policy;
- trust decisions.
## OBS State
OBS owns:
- final scene composition;
- stream/record controls;
- operator mix choices.
It must not own:
- stream synchronization;
- canonical timing;
- raw calibration truth;
- volumetric field reconstruction.

View file

@ -0,0 +1,169 @@
# Visual Fusion And Realtime Gaussian Splatting Study
## Thesis
Mimir's visual side should not become a CPU image processing application. The
camera rig exists to produce time-indexed visual evidence that Fensalir can fuse
on GPU into a live volumetric program surface.
The direct-driver work matters because sensor fusion is brutally sensitive to:
- timestamp error;
- rolling/shutter latency;
- frame drops;
- calibration drift;
- duplicated CPU copies;
- inconsistent exposure/format state.
## Current Sensor Roles
- Leap stereo IR: near-field timing/depth/tracking candidate.
- PS3 Eyes: high-rate feature/tracking witnesses.
- Kiyo: stable RGB context.
- Kiyo Pro: intended RGB ground truth, currently high-speed/cadence limited.
- Raven-side devices: remote evidence producers, not local compute authority.
## Evidence Layers
### Raw Frame
Minimum:
- source id;
- device/arrival timestamp;
- dimensions/format/stride;
- native handle or byte span.
### Feature Frame
Possible GPU outputs:
- corners/keypoints;
- descriptors;
- optical flow;
- silhouettes/masks;
- depth/disparity hints;
- confidence per feature.
### Scene Claims
Rolling reservoir views:
- scene rays;
- surface claims;
- material claims;
- render packets.
### Program Surface
Fensalir output:
- Gaussian splat/point cloud render;
- avatar/AR overlay;
- Spout2 program video.
## Realtime Gaussian Splatting Relevance
Gaussian splatting is attractive because it represents scenes as many local
anisotropic primitives that can be rasterized quickly. For Mimir, the hard part
is not rendering a static splat set. The hard part is maintaining a dynamic
time-indexed splat field from live unsynchronized cameras.
Mimir needs:
- synchronized observations;
- camera calibration;
- dynamic object handling;
- confidence and decay;
- GPU-resident update path;
- output that can be inspected/debugged live.
The low-level implementation references all point at the same renderer spine:
project Gaussians, bin/intersect them into tiles, sort or partially order for
view-consistent blending, then rasterize tiles with tightly packed GPU buffers.
`gsplat`, NVIDIA's Vulkan sample, StopThePop, and FlashGS are useful because
they expose performance levers Mimir/Fensalir can reuse without pretending the
live rig is an offline training pipeline.
Useful borrowed ideas:
- packed structure-of-arrays splat buffers;
- tile-size as a first-order performance parameter;
- radius/opacity culling before sort/raster;
- global versus local/tile order tradeoffs;
- asynchronous CPU/GPU sorting only if it does not add frame latency;
- feature/depth render modes as diagnostics, not only RGB output.
## 4D / Dynamic Splatting Shape
```mermaid
flowchart TD
Frames["time-aligned camera frames"] --> Features["GPU features / masks / flow"]
Features --> Tracks["temporal tracks"]
Tracks --> Claims["surface/material claims"]
Claims --> Splats["dynamic Gaussian / point primitives"]
Audio["audio source estimates"] --> Claims
Splats --> Render["Fensalir program render"]
```
## Data Model Sketch
```cpp
struct MimirVisualObservation
{
uint64_t sourceHash;
uint64_t timestampNs;
uint32_t width;
uint32_t height;
uint32_t format;
uint64_t nativeHandle;
};
struct MimirSplatClaim
{
uint64_t stableKeyHash;
float position[3];
float covariance[6];
float color[4];
float confidence;
uint64_t sourceTimeMinNs;
uint64_t sourceTimeMaxNs;
};
```
## Immediate Architecture Cut
Do not start with "implement 4DGS." Start with the ownership:
1. direct camera driver pushes frame handles;
2. runtime/reservoir indexes them;
3. Fensalir reads current synchronized window;
4. Fensalir produces simple point/render packet claims;
5. only then expand to dynamic splat update.
## Micro-Optimization Risks
- CPU demosaic/format conversion before GPU upload.
- Multiple copies of compressed camera frames.
- Re-decoding frame data per feature stage.
- Per-frame heap allocations in capture workers.
- Blocking on GPU readback for UI/debug.
- Sorting every splat globally every frame before the field is even stable.
- Treating 4DGS optimization loops as live capture prerequisites.
## Research Threads To Continue
- 3D Gaussian Splatting reference CUDA rasterizer.
- `gsplat` tile/raster APIs and CUDA memory layout.
- NVIDIA `vk_gaussian_splatting` mesh-shader versus vertex-shader paths.
- StopThePop / sorted Gaussian splatting for view-consistency tradeoffs.
- FlashGS-style kernel-level optimization and redundant-sort reduction.
- Dynamic/4D Gaussian splatting update strategies.
- PlayCanvas/WebGPU splat processing as a practical open implementation of
GPU-side splat mutation and browser-scale rendering constraints.
- Gaussian-SLAM and multi-sensor calibration papers as shape references for
online pose/calibration drift, even though Mimir's rig is room-bound rather
than vehicle-mounted.
- Online multi-camera calibration and temporal alignment.
- GPU optical flow/features for live camera arrays.
- Sensor-fusion priors from audio localization.

View file

@ -0,0 +1,191 @@
# Volumetric Audio Field Study
## Thesis
Mimir's audio target is not "sync the mics." That is only the entry fee. The
target is a continuously updated room-scale sound field: where sources are,
which surfaces colored them, how each microphone hears them, and which
OBS-facing stems can be emitted without flattening useful spatial evidence.
The current ASIO/chirp-bin work is the correct primitive because a microphone
array without precise timing is just a pile of unrelated waveforms.
## Problem Decomposition
### 1. Clock And Delay
Every input stream needs:
- delay relative to loopback/canonical program time;
- fractional delay below one sample;
- sampling-rate offset over time;
- confidence and failure state.
This is the current `MimirAudioSynchronizationState` surface.
### 2. Transfer Function
Every output/mic path needs:
- magnitude response by frequency;
- phase/group-delay response;
- confusion/alias behavior for coded active signals;
- reflection-prone bands and time regions.
This is the current `MimirChirpBinCalibrationModel` surface, but it should grow
from "decoder helper" into "room path model."
### 3. Source Localization
Once streams are aligned, source localization can use:
- TDOA between microphones;
- direct-path chirp time-of-arrival;
- beamforming/steered response;
- visual priors from camera/pose/splat state;
- semantic priors from dialogue/game sources.
For Mimir, the first robust source localizer should probably be hybrid:
1. use active chirp emitters for calibration and known anchors;
2. use TDOA/GCC-PHAT/SRP-PHAT for transient/passive events;
3. use visual tracking as a prior for voices/hands/objects.
### 4. Field Reconstruction
Field reconstruction can be represented at several resolutions:
- low-order ambisonic bed for OBS-friendly spatial ambience;
- per-speaker/mic transfer function models;
- localized source objects with direction, distance, and confidence;
- room response map for reflections/suppression;
- neural or Gaussian acoustic field later, if justified.
The first practical target is not a full pressure-field simulator. It is:
- aligned dialogue anchors;
- camera mics as spatial witnesses;
- local loopback as program reference;
- co-streamer/Raven timing evidence;
- separable stems plus spatial bed.
## Methods Worth Keeping Separate
### GCC-PHAT / SRP-PHAT
Use:
- passive TDOA;
- transient localization;
- drift hints when program audio is correlated.
Do not use:
- canonical timeline identity.
Reason:
- It estimates relative delay from content, but content is not a unique clock
unless it is actively coded.
### Beamforming
Use:
- voice/source enhancement once mic positions and delays are known.
Risks:
- Bad geometry or uncorrected phase response creates false confidence.
- Camera mics are low quality but spatially useful; Focusrites are high quality
but fewer.
### Ambisonics
Use:
- OBS-facing spatial bed.
- Compact representation after alignment.
Risks:
- Ambisonics is an output/control representation, not the raw truth. Do not
throw away per-mic evidence too early.
### Room Impulse Response Estimation
Use:
- speaker/mic path model;
- frequency response normalization;
- reflection windows;
- phase/group-delay calibration.
Best current route:
- Use chirp-bin codebook for continuous/lightweight operation.
- Use fuller sweeps or exponential sine sweeps for deliberate calibration
sessions.
## Architecture Proposal
```mermaid
flowchart TD
ASIO["ASIO inputs + loopback"] --> Align["delay/SRO estimator"]
Chirps["chirp-bin active witness"] --> Align
Music["program audio"] --> Passive["passive GCC-PHAT"]
Passive --> Align
Align --> Model["path response model"]
Align --> Actuator["fractional delay + ASRC"]
Model --> Actuator
Actuator --> Field["sound field estimator"]
Visual["Fensalir visual priors"] --> Field
Field --> Stems["voice/ambient/transient stems"]
Field --> Bed["spatial bed"]
Stems --> OBS["OBS"]
Bed --> OBS
```
## Data Model Sketch
```csharp
public sealed record MimirAudioPathModel(
string OutputSourceId,
string ReceiverSourceId,
int SampleRate,
IReadOnlyList<FrequencyBandResponse> Magnitude,
IReadOnlyList<GroupDelayPoint> GroupDelay,
IReadOnlyList<ReflectionWindow> Reflections,
IReadOnlyList<SymbolConfusion> ActiveConfusion,
double DirectPathDelaySamples,
double Confidence);
public sealed record MimirSoundSourceEstimate(
string SourceClass,
Vector3 PositionMeters,
Vector3 VelocityMetersPerSecond,
double StartTimeNs,
double Confidence,
IReadOnlyList<string> SupportingReceivers);
```
## Hot-Path Warning
Do not implement volumetric audio as repeated whole-window cross-correlations
between every pair of streams. Pairwise all-to-all is fine for diagnosis and
small proofs; production needs:
- reference-first timing;
- cached spectra;
- bounded candidate pairs;
- geometry-informed search;
- streaming state.
## Research Threads To Continue
- SRP-PHAT acceleration on GPU for small mic arrays.
- Low-order ambisonic encoding from irregular microphone arrays.
- Joint audio-visual source localization with tracked speaker/head priors.
- Online room impulse response tracking under moving sources.
- Neural acoustic fields only after classical calibration stops scaling.