Deepen Perfect Machine study index
This commit is contained in:
parent
72d691b798
commit
8b151294e7
39 changed files with 5005 additions and 4 deletions
|
|
@ -15,8 +15,14 @@ Read this with:
|
|||
the long-form design argument.
|
||||
- `research/perfect-machine-study-2026-05-23/index.md` for the study directory
|
||||
and sample-code index.
|
||||
- `research/perfect-machine-study-2026-05-23/reading-guide.md` for task-oriented
|
||||
navigation through the study.
|
||||
- `research/perfect-machine-study-2026-05-23/optimization-ledger.md` for hot
|
||||
loop candidates.
|
||||
- `research/perfect-machine-study-2026-05-23/fensalir-integration-map.md` for
|
||||
the Mimir/Fensalir ownership cut.
|
||||
- `research/perfect-machine-study-2026-05-23/calibration-session-spec.md` for
|
||||
the planned physical-path calibration command.
|
||||
- `research/perfect-machine-study-2026-05-23/samples/README.md` for sample
|
||||
implementation sketches.
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,172 @@
|
|||
# Acoustic Field Models
|
||||
|
||||
## Purpose
|
||||
|
||||
Mimir's audio goal is not just "delay-align microphones." Delay alignment is
|
||||
the entry fee. The larger target is a live volumetric audio field: where sound
|
||||
is, how the room moves it, which mic/speaker paths are trustworthy, and how to
|
||||
emit OBS-facing stems or a spatial bed that are better than raw source feeds.
|
||||
|
||||
## The Three Layers
|
||||
|
||||
### 1. Clock And Path Layer
|
||||
|
||||
Questions:
|
||||
|
||||
- Where is this sample on the canonical timeline?
|
||||
- How much sample-rate offset is present?
|
||||
- What delay, phase, and group-delay shape does this path impose?
|
||||
|
||||
Tools:
|
||||
|
||||
- active chirp-bin/codebook anchors;
|
||||
- passive GCC-PHAT on program audio;
|
||||
- loopback as emitter-side timing authority;
|
||||
- Farrow/ASRC actuator.
|
||||
|
||||
Output:
|
||||
|
||||
- delay and SRO state;
|
||||
- per-band magnitude and phase response;
|
||||
- confidence and residuals.
|
||||
|
||||
This layer must work before any field reconstruction deserves trust.
|
||||
|
||||
### 2. Source/Geometry Layer
|
||||
|
||||
Questions:
|
||||
|
||||
- Which source positions explain the synchronized microphones?
|
||||
- Which reflections are direct path versus room response?
|
||||
- What evidence should become a stable field claim?
|
||||
|
||||
Tools:
|
||||
|
||||
- TDOA;
|
||||
- SRP-PHAT;
|
||||
- beamforming;
|
||||
- known mic geometry;
|
||||
- visual constraints from cameras/Leap when available.
|
||||
|
||||
Output:
|
||||
|
||||
- source position candidates;
|
||||
- source confidence;
|
||||
- reflection/room residuals;
|
||||
- constraints for Fensalir's temporal evidence reservoir.
|
||||
|
||||
### 3. Field Reconstruction Layer
|
||||
|
||||
Questions:
|
||||
|
||||
- What pressure/energy field exists in the room?
|
||||
- Can the field be represented as sources, ambisonics, equivalent sources, or a
|
||||
hybrid?
|
||||
- What should OBS get: isolated stems, spatial bed, diagnostics, or all three?
|
||||
|
||||
Tools:
|
||||
|
||||
- ambisonics/spherical harmonics for compact field representation where geometry
|
||||
fits;
|
||||
- equivalent source method / near-field acoustic holography when treating sound
|
||||
as sparse physical sources;
|
||||
- compressed sensing or Bayesian sparse recovery when source count is low and
|
||||
mic count is limited;
|
||||
- learned priors only after deterministic models fail cleanly.
|
||||
|
||||
Output:
|
||||
|
||||
- aligned stems;
|
||||
- sound-source tracks;
|
||||
- spatial bed/ambisonic representation;
|
||||
- visualizable acoustic field constraints.
|
||||
|
||||
## Why Six Mics Are Not An Eigenmike
|
||||
|
||||
A commercial spherical array might have dozens of capsules in known geometry.
|
||||
Mimir has an irregular consumer/pro-audio constellation: Scarlett mic, camera
|
||||
mics, possibly phone/Raven mics, and loopback. That pushes the machine away from
|
||||
pure textbook ambisonics and toward an evidence model:
|
||||
|
||||
- synchronize first;
|
||||
- calibrate each path;
|
||||
- infer source constraints;
|
||||
- reconstruct only what the geometry supports.
|
||||
|
||||
If the geometry cannot support a high-order field, the honest output is lower
|
||||
order or sparse-source estimates with uncertainty. Pretending otherwise would
|
||||
be the audio equivalent of drawing a ruler on fog and calling it engineering.
|
||||
|
||||
## Candidate Models
|
||||
|
||||
### Ambisonic Fit
|
||||
|
||||
Use when:
|
||||
|
||||
- mic geometry can support a stable low-order spherical harmonic fit;
|
||||
- the goal is a spatial bed, not precise source localization;
|
||||
- the scene is diffuse or ambient.
|
||||
|
||||
Risk:
|
||||
|
||||
- irregular sparse mics produce unstable coefficients;
|
||||
- near-field sources and room boundaries violate simple assumptions.
|
||||
|
||||
### Beamforming / SRP-PHAT
|
||||
|
||||
Use when:
|
||||
|
||||
- seeking source direction/position;
|
||||
- synchronized mic geometry is known;
|
||||
- direct-path energy exists.
|
||||
|
||||
Risk:
|
||||
|
||||
- reflections can produce plausible false peaks;
|
||||
- small baseline limits low-frequency localization.
|
||||
|
||||
### Equivalent Source / Sparse Acoustic Holography
|
||||
|
||||
Use when:
|
||||
|
||||
- sources can be represented sparsely;
|
||||
- a grid or source dictionary is acceptable;
|
||||
- response calibration is available.
|
||||
|
||||
Risk:
|
||||
|
||||
- inverse problems are ill-conditioned;
|
||||
- dense grids are expensive;
|
||||
- sparse recovery can hallucinate when the measurement model is wrong.
|
||||
|
||||
### Hybrid Evidence Field
|
||||
|
||||
Use when:
|
||||
|
||||
- Mimir has mixed-quality sensors;
|
||||
- visual evidence can constrain likely source regions;
|
||||
- acoustic output should remain honest about uncertainty.
|
||||
|
||||
Shape:
|
||||
|
||||
- clock/path layer feeds calibrated observations;
|
||||
- localization layer creates candidates;
|
||||
- Fensalir temporal reservoir stabilizes candidates across time;
|
||||
- DSP/output layer uses stable tracks as weights, not commandments.
|
||||
|
||||
This is the likely Mimir path.
|
||||
|
||||
## Practical First Milestone
|
||||
|
||||
Build one static acoustic source proof:
|
||||
|
||||
1. Record loopback and all live mics while emitting calibrated active witness.
|
||||
2. Recover canonical anchors on loopback and at least one physical mic.
|
||||
3. Estimate per-mic delay/SRO/response.
|
||||
4. Use known speaker/mic geometry to form a TDOA/SRP-PHAT position hypothesis.
|
||||
5. Lower that as an `AquariumAcousticConstraint`.
|
||||
6. Display/log the constraint without letting it alter program audio yet.
|
||||
|
||||
The proof is deliberately modest. It forces the field machine to earn geometry
|
||||
from timing instead of leaping straight into theatrical volumetric fog.
|
||||
|
||||
|
|
@ -6,6 +6,9 @@
|
|||
dedicated pass mapping, researching, distilling, ruminating, and writing
|
||||
sample options.
|
||||
- Active pass start recorded locally: `2026-05-23T23:41:19+01:00`.
|
||||
- Corrected no-shortcut pass start recorded locally after the final instruction:
|
||||
`2026-05-23T23:54:42.9041588+01:00`. Do not treat this directory as complete
|
||||
until at least `2026-05-24T01:54:42.9041588+01:00`.
|
||||
|
||||
## Prime Cut
|
||||
|
||||
|
|
@ -210,4 +213,3 @@ Reason:
|
|||
that is how bandwidth dies in a formal suit.
|
||||
- Treating 192 kHz as magic: it improves timing granularity and ultrasonic
|
||||
options only where transducers and path response actually preserve signal.
|
||||
|
||||
|
|
|
|||
274
research/perfect-machine-study-2026-05-23/benchmark-plan.md
Normal file
274
research/perfect-machine-study-2026-05-23/benchmark-plan.md
Normal file
|
|
@ -0,0 +1,274 @@
|
|||
# Benchmark Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
This is the measurement spine for future implementation passes. It exists to
|
||||
keep the machine honest: no hot-path change earns production status because it
|
||||
sounds clever. It earns it by moving measured latency, jitter, throughput,
|
||||
allocation, cache pressure, confidence, or reconstruction quality in the right
|
||||
direction.
|
||||
|
||||
## Global Rules
|
||||
|
||||
- Measure wall-clock throughput and canonical timing error separately.
|
||||
- Keep synthetic, artifact, and live-device runs distinct.
|
||||
- Always record sample rate, buffer size, device clock, channel mapping, and
|
||||
mode (`passive`, `chirp-only`, `hybrid`).
|
||||
- Allocation-free claims require runtime allocation counters, not code vibes.
|
||||
- A passing self-test is a unit proof, not a physical proof.
|
||||
- A physical proof without artifact retention is only a campfire story.
|
||||
|
||||
## Audio Synchronization Benchmarks
|
||||
|
||||
### Synthetic Chirp-Bin Decoder
|
||||
|
||||
Goal: prove canonical-time recovery without acoustic path damage.
|
||||
|
||||
Inputs:
|
||||
|
||||
- generated chirp-bin timeline;
|
||||
- known integer delay;
|
||||
- known fractional delay;
|
||||
- optional sample-rate offset;
|
||||
- optional colored noise and band dropouts.
|
||||
|
||||
Metrics:
|
||||
|
||||
- decoded anchors per second;
|
||||
- false anchors per second;
|
||||
- delay error in microseconds;
|
||||
- fractional sample residual;
|
||||
- SRO estimate error in ppm;
|
||||
- allocations per second;
|
||||
- CPU cycles or elapsed time per decoded second.
|
||||
|
||||
Current command family:
|
||||
|
||||
```powershell
|
||||
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --standalone-chirp-bin-self-test --sample-rate 192000 --delay-samples 96000
|
||||
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 192000
|
||||
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --hybrid-sync-self-test --sample-rate 192000
|
||||
```
|
||||
|
||||
Missing benchmark:
|
||||
|
||||
- a loop over SNR, reliable-bin count, fractional delay, and SRO;
|
||||
- output JSON/CSV artifact;
|
||||
- CI-friendly threshold for the synthetic path.
|
||||
|
||||
### Scarlett Loopback Artifact
|
||||
|
||||
Goal: prove the analyzer survives the real ASIO clock and converter path when
|
||||
the signal remains electrical/loopback-clean.
|
||||
|
||||
Inputs:
|
||||
|
||||
- rendered mono float chirp-bin witness;
|
||||
- ASIO output through Scarlett;
|
||||
- ASIO capture of loopback channels;
|
||||
- optional persisted calibration profile.
|
||||
|
||||
Metrics:
|
||||
|
||||
- loopback channel-to-channel delay;
|
||||
- confidence;
|
||||
- anchor count;
|
||||
- response matrix diagonal strength;
|
||||
- per-band phase/group-delay residual;
|
||||
- callback block jitter;
|
||||
- dropped/late ASIO blocks.
|
||||
|
||||
Current command family:
|
||||
|
||||
```powershell
|
||||
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --render-chirp-bin-f32 --sample-rate 192000 --seconds 6 --output .\artifacts\chirp-bin.f32
|
||||
.\native\probes\asio_audio_cadence\build\Release\asio_audio_cadence.exe --play-f32-mono .\artifacts\chirp-bin.f32 --record-f32-interleaved .\artifacts\scarlett.f32 --sample-rate 192000 --seconds 6
|
||||
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --analyze-asio-f32 .\artifacts\scarlett.f32 --sample-rate 192000
|
||||
```
|
||||
|
||||
Missing benchmark:
|
||||
|
||||
- stable artifact directory naming;
|
||||
- automatic device/channel manifest capture;
|
||||
- repeated run variance.
|
||||
|
||||
### Meatspace Mic Path
|
||||
|
||||
Goal: prove acoustic propagation, monitor response, mic response, room
|
||||
reflections, and gain staging still permit deterministic code-valid anchors.
|
||||
|
||||
Inputs:
|
||||
|
||||
- Scarlett output through monitors;
|
||||
- Scarlett mic and camera mics;
|
||||
- calibration profile generated from the same room/output/mic path;
|
||||
- optional passive music bed.
|
||||
|
||||
Metrics:
|
||||
|
||||
- usable bands per mic;
|
||||
- confusion matrix entropy;
|
||||
- strongest alias bins;
|
||||
- group-delay slope by bin;
|
||||
- decoded anchors per second;
|
||||
- rejected anchors and why;
|
||||
- global delay/bin-shift hypothesis likelihood;
|
||||
- time-to-lock and lock-hold duration;
|
||||
- SRO drift estimate stability.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- a calibrated physical mic should produce stable canonical anchors within the
|
||||
five-second rolling buffer;
|
||||
- delay should be reported in microseconds with residuals low enough for a
|
||||
fractional-delay actuator to matter;
|
||||
- dead bins must be excluded or downweighted by the emitted codebook, not merely
|
||||
filtered after failure.
|
||||
|
||||
## Audio Field Benchmarks
|
||||
|
||||
### Passive Program-Audio Alignment
|
||||
|
||||
Goal: use loopback music as continuous timing evidence when active chirps are
|
||||
unnecessary.
|
||||
|
||||
Metrics:
|
||||
|
||||
- PHAT peak sharpness;
|
||||
- delay confidence over time;
|
||||
- failure rate on low-frequency-only material;
|
||||
- behavior during silence;
|
||||
- agreement with active chirp anchors.
|
||||
|
||||
Cut line:
|
||||
|
||||
- passive alignment is advisory unless it agrees with active/codebook anchors or
|
||||
repeated state earns confidence.
|
||||
|
||||
### Acoustic Source Localization
|
||||
|
||||
Goal: turn synchronized mics into spatial estimates rather than just aligned
|
||||
waveforms.
|
||||
|
||||
Inputs:
|
||||
|
||||
- known source positions if available;
|
||||
- synthetic room model for baseline;
|
||||
- live mic geometry.
|
||||
|
||||
Metrics:
|
||||
|
||||
- TDOA residual;
|
||||
- SRP-PHAT peak width;
|
||||
- source position error;
|
||||
- update rate;
|
||||
- ambiguity count.
|
||||
|
||||
First proof:
|
||||
|
||||
- one static speaker position recovered from loopback/chirp evidence and mic
|
||||
geometry.
|
||||
|
||||
## Video Capture Benchmarks
|
||||
|
||||
### Native Camera Cadence
|
||||
|
||||
Goal: prove each camera's raw driver path maintains sustained cadence without
|
||||
framework buffering lies.
|
||||
|
||||
Metrics:
|
||||
|
||||
- delivered fps after warm-up;
|
||||
- frame interval p50/p95/p99;
|
||||
- missing sequence count;
|
||||
- device timestamp monotonicity;
|
||||
- CPU time in callback;
|
||||
- transfer size and USB topology.
|
||||
|
||||
Known baselines:
|
||||
|
||||
- Leap full-height stereo IR: roughly 110 fps when the USB neighborhood is not
|
||||
polluted by the Focusrite.
|
||||
- PS3 Eyes: roughly 187 fps at 320x240 and roughly 58-60 fps at 640x480.
|
||||
- Kiyo Pro: currently about 25 fps despite 60 fps advertisement and HighSpeed
|
||||
negotiation.
|
||||
- Kiyo basic: about 30 fps in current usable modes.
|
||||
|
||||
### Multi-Camera Contention
|
||||
|
||||
Goal: discover whether frame loss is USB topology, driver scheduling, CPU copy,
|
||||
or runtime ingest pressure.
|
||||
|
||||
Metrics:
|
||||
|
||||
- per-device fps with all sources open;
|
||||
- callback jitter correlation across devices;
|
||||
- host controller/root hub mapping;
|
||||
- frame-size bandwidth estimate versus observed cadence;
|
||||
- runtime buffer enqueue cost.
|
||||
|
||||
First implementation proof:
|
||||
|
||||
- all local cameras feed `Mimir.Runtime` descriptors in one app run with no
|
||||
JSON process bridge.
|
||||
|
||||
## Fensalir Lowering Benchmarks
|
||||
|
||||
Goal: prove Mimir can create Fensalir contract frames every render tick without
|
||||
turning the app into a garbage factory.
|
||||
|
||||
Metrics:
|
||||
|
||||
- lowering time per frame;
|
||||
- managed allocations per frame;
|
||||
- number of sensor descriptors lowered;
|
||||
- D3D12 upload/import wait time once connected;
|
||||
- stale observation count.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- lowering is pure mapping over current buffer snapshots;
|
||||
- analysis/capture does not run from the render tick;
|
||||
- UI reads cached state only.
|
||||
|
||||
## Gaussian Splat / Sensor Fusion Benchmarks
|
||||
|
||||
Goal: decide which visual fusion representation deserves production attention.
|
||||
|
||||
Metrics:
|
||||
|
||||
- observations accepted per second;
|
||||
- splat claims updated per second;
|
||||
- reprojection error;
|
||||
- track lifetime;
|
||||
- sort/tile/render cost;
|
||||
- memory footprint per claim;
|
||||
- drift under camera motion.
|
||||
|
||||
Candidate approaches:
|
||||
|
||||
- sparse feature claims lowered into Fensalir temporal evidence reservoir;
|
||||
- live Gaussian claims updated from synchronized camera features;
|
||||
- hybrid point/splat field where Mimir supplies timing and Fensalir owns
|
||||
rendering/reuse.
|
||||
|
||||
## Reporting Format
|
||||
|
||||
Every serious benchmark should write one manifest next to artifacts:
|
||||
|
||||
```json
|
||||
{
|
||||
"machine": "Starfire",
|
||||
"timestampUtc": "2026-05-24T00:00:00Z",
|
||||
"mode": "chirp-only",
|
||||
"sampleRate": 192000,
|
||||
"bufferFrames": 192,
|
||||
"sources": [],
|
||||
"artifacts": [],
|
||||
"metrics": {},
|
||||
"notes": []
|
||||
}
|
||||
```
|
||||
|
||||
The manifest is not the hot path. It is the lab notebook.
|
||||
|
||||
|
|
@ -0,0 +1,179 @@
|
|||
# Calibration Session Spec
|
||||
|
||||
## Purpose
|
||||
|
||||
The calibration command should be the machine that turns a real room into a
|
||||
path model. It should not be a pile of one-off artifact commands that only the
|
||||
current operator remembers how to arrange.
|
||||
|
||||
## Objective
|
||||
|
||||
Emit a known active witness, capture loopback and every microphone in the same
|
||||
clock domain where possible, compute per-output/mic response/confusion/delay
|
||||
models, persist them, and immediately prove that the live decoder improves when
|
||||
those models are loaded.
|
||||
|
||||
## Inputs
|
||||
|
||||
- output path id;
|
||||
- ASIO driver id / source manifest;
|
||||
- sample rate;
|
||||
- duration;
|
||||
- emission mode (`chirp-only`, `hybrid-watermark`, `silent-passive-baseline`);
|
||||
- candidate symbol plan;
|
||||
- mic geometry file if available;
|
||||
- artifact directory.
|
||||
|
||||
## Outputs
|
||||
|
||||
- raw capture artifact;
|
||||
- rendered witness artifact;
|
||||
- manifest with device/channel/sample-rate/buffer-size/gain notes;
|
||||
- `MimirChirpBinCalibrationModel`;
|
||||
- per-path usable-band table;
|
||||
- confusion matrix;
|
||||
- timing residual table;
|
||||
- group-delay estimate;
|
||||
- recommended codebook plan;
|
||||
- before/after decode report.
|
||||
|
||||
## Command Shape
|
||||
|
||||
Future production command:
|
||||
|
||||
```powershell
|
||||
dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- `
|
||||
--calibration-session `
|
||||
--config .\config\mimir-runtime.asio.json `
|
||||
--output-path scarlett-main-out `
|
||||
--seconds 30 `
|
||||
--sample-rate 192000 `
|
||||
--artifact-dir .\calibration\runs\2026-05-24-starfire
|
||||
```
|
||||
|
||||
This can start as `BufferSmoke`, but the final owner should be a runtime command
|
||||
or app action that uses the same ASIO/native path as production.
|
||||
|
||||
## Session Phases
|
||||
|
||||
### 1. Preflight
|
||||
|
||||
- enumerate ASIO inputs/outputs;
|
||||
- verify requested sample rate and buffer size;
|
||||
- verify loopback channels are present;
|
||||
- verify input channels are non-silent when expected;
|
||||
- record channel names and source ids;
|
||||
- load existing calibration for comparison.
|
||||
|
||||
Fail fast if loopback is absent. Without loopback, the session can still record
|
||||
diagnostics, but it cannot become canonical calibration.
|
||||
|
||||
### 2. Passive Baseline
|
||||
|
||||
Capture room/program audio without active witness for a short window.
|
||||
|
||||
Purpose:
|
||||
|
||||
- noise floor;
|
||||
- current music/program spectrum;
|
||||
- passive correlation strength;
|
||||
- mic gain sanity.
|
||||
|
||||
### 3. Active Emission
|
||||
|
||||
Render a deterministic chirp-bin/watermark witness using the candidate symbol
|
||||
plan. Capture loopback and all inputs.
|
||||
|
||||
Important:
|
||||
|
||||
- event ids and schedule seed go into the manifest;
|
||||
- emitted codebook plan goes into the artifact;
|
||||
- output gain should be recorded manually or through driver state if possible.
|
||||
|
||||
### 4. Decode And Score
|
||||
|
||||
For each input:
|
||||
|
||||
- decode loopback anchors;
|
||||
- decode mic anchors;
|
||||
- compute matched anchor delays;
|
||||
- compute expected-symbol versus observed-bin matrix;
|
||||
- compute timing residuals;
|
||||
- estimate magnitude response;
|
||||
- estimate phase/group delay where phase evidence is stable;
|
||||
- generate delay/bin-shift hypotheses.
|
||||
|
||||
### 5. Plan Adaptation
|
||||
|
||||
Across all selected mics:
|
||||
|
||||
- identify reliable bins;
|
||||
- identify dead/aliased bins;
|
||||
- choose reduced alphabet;
|
||||
- choose higher de Bruijn order if necessary;
|
||||
- estimate expected time-to-unique-lock;
|
||||
- write recommended emission plan.
|
||||
|
||||
### 6. Replay Proof
|
||||
|
||||
Reload the persisted model and decode the same artifact:
|
||||
|
||||
- unweighted decode report;
|
||||
- weighted/adapted decode report;
|
||||
- anchor gain/loss;
|
||||
- confidence gain/loss;
|
||||
- delay residual gain/loss.
|
||||
|
||||
If the calibrated model does not improve the capture, the command should say so
|
||||
plainly. A calibration file that cannot improve the decoder is just decorative
|
||||
paperwork with a badge.
|
||||
|
||||
## Persisted Model Shape
|
||||
|
||||
Minimum:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": "gamecult.mimir.chirp_bin_calibration.v1",
|
||||
"createdUtc": "2026-05-24T00:00:00Z",
|
||||
"sampleRate": 192000,
|
||||
"outputPathId": "scarlett-main-out",
|
||||
"emissionPlan": {},
|
||||
"paths": [
|
||||
{
|
||||
"sourceId": "asio-ch0",
|
||||
"usableBands": [],
|
||||
"symbols": [],
|
||||
"delayHypotheses": [],
|
||||
"phaseModel": {},
|
||||
"groupDelayModel": {},
|
||||
"confidence": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- loopback decodes confidently;
|
||||
- at least one physical mic produces code-valid anchors;
|
||||
- calibration identifies reliable and unreliable bands;
|
||||
- adapted emission plan changes the symbol set when the original bands are bad;
|
||||
- loaded calibration affects likelihood before codebook solving;
|
||||
- replay proof is stored with the model.
|
||||
|
||||
## Failure Records
|
||||
|
||||
A failed calibration is still useful if it records:
|
||||
|
||||
- no loopback;
|
||||
- silent mic;
|
||||
- clipping;
|
||||
- insufficient high-frequency response;
|
||||
- strong alias/confusion;
|
||||
- group delay too unstable;
|
||||
- active witness too quiet/loud;
|
||||
- room reflection dominated direct path.
|
||||
|
||||
Do not delete failed runs unless they are accidental trash. They are evidence.
|
||||
|
||||
|
|
@ -0,0 +1,169 @@
|
|||
# Chirplet Transform And Controlled Chirp Codebook Deep Dive
|
||||
|
||||
## Core Distinction
|
||||
|
||||
There are two different machines that share similar math:
|
||||
|
||||
1. A general chirplet transform asks: "what chirp-like atoms exist in this
|
||||
signal?"
|
||||
2. Mimir's active receiver asks: "which known emitted atom arrived when, through
|
||||
which distorted path?"
|
||||
|
||||
The second machine is much cheaper because the codebook, slopes, timing plan,
|
||||
and symbol grammar are known.
|
||||
|
||||
## Uses For Mimir
|
||||
|
||||
### Timing Synchronization
|
||||
|
||||
Each decoded code-valid chirp tuple can become a canonical timeline anchor.
|
||||
With enough anchors, fit:
|
||||
|
||||
- offset;
|
||||
- sample-rate offset;
|
||||
- confidence;
|
||||
- jitter/residual.
|
||||
|
||||
### Frequency Response Normalization
|
||||
|
||||
Each classified chirp yields observed energy across bins. Aggregated over time:
|
||||
|
||||
- reliable bands;
|
||||
- dead bands;
|
||||
- alias/confusion bands;
|
||||
- path coloration.
|
||||
|
||||
### Phase / Group Delay
|
||||
|
||||
If phase is coherent enough, dechirped complex bin responses can estimate:
|
||||
|
||||
- per-bin phase;
|
||||
- phase slope across frequency;
|
||||
- group delay correction;
|
||||
- reflection/path stability.
|
||||
|
||||
### Localization
|
||||
|
||||
Known emitted time plus observed receiver time gives a distance constraint when
|
||||
the source and receiver clocks are modeled. Multiple receivers turn this into a
|
||||
position estimate. Multiple emitters/receivers turn it into room calibration.
|
||||
|
||||
### Watermarking
|
||||
|
||||
Hybrid mode can embed low-gain coded chirps under program audio. The watermark
|
||||
should be:
|
||||
|
||||
- sparse enough not to annoy;
|
||||
- deterministic enough to identify time;
|
||||
- adaptive enough to avoid dead acoustic bands;
|
||||
- continuous enough to track drift.
|
||||
|
||||
## Decoder Pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
PCM["PCM stream"] --> Energy["streaming energy/onset proposal"]
|
||||
Energy --> Cand["candidate event windows"]
|
||||
Cand --> Dechirp["dechirp with known slope"]
|
||||
Dechirp --> Bins["bin / Goertzel / FFT score"]
|
||||
Bins --> Likelihood["calibration-weighted likelihood"]
|
||||
Likelihood --> Grammar["codebook grammar / de Bruijn tuples"]
|
||||
Grammar --> Clock["clock fit"]
|
||||
Clock --> Model["response + delay model"]
|
||||
```
|
||||
|
||||
## What The Current Code Has
|
||||
|
||||
- Fixed-slope chirp-bin renderer.
|
||||
- De Bruijn codebook plan.
|
||||
- Energy/onset proposals.
|
||||
- Dechirp/bin scoring.
|
||||
- Ambiguity candidates.
|
||||
- Global bin-shift hypotheses.
|
||||
- Clock fit and source-offset refinement.
|
||||
- Calibration model with response/confusion/delay/phase/adaptive codebook.
|
||||
|
||||
## What Is Missing
|
||||
|
||||
- Streaming candidate state.
|
||||
- Calibration-weighted likelihood before path selection as the default physical
|
||||
path, not just optional model support.
|
||||
- Better group-delay model.
|
||||
- Reduced reliable-bin physical proof.
|
||||
- Native/SIMD/GPU scoring path after correctness is proven.
|
||||
|
||||
## Algorithm Options
|
||||
|
||||
### Option A: Goertzel Bank
|
||||
|
||||
Best when:
|
||||
|
||||
- bin count is small;
|
||||
- sample window is short;
|
||||
- CPU SIMD is available;
|
||||
- frequencies are fixed.
|
||||
|
||||
Pros:
|
||||
|
||||
- No FFT plan overhead.
|
||||
- Easy to weight per bin.
|
||||
- Simple to port to AVX2.
|
||||
|
||||
Cons:
|
||||
|
||||
- Repeats work per bin.
|
||||
- Less attractive if bin count/window count grows.
|
||||
|
||||
### Option B: Dechirp + FFT
|
||||
|
||||
Best when:
|
||||
|
||||
- many bins;
|
||||
- many candidates;
|
||||
- batching is possible.
|
||||
|
||||
Pros:
|
||||
|
||||
- Canonical CSS receiver shape.
|
||||
- Good fit for GPU/native FFT batches.
|
||||
|
||||
Cons:
|
||||
|
||||
- FFT overhead and memory motion can dominate small cases.
|
||||
|
||||
### Option C: Chirp Z Transform
|
||||
|
||||
Best when:
|
||||
|
||||
- only a narrow frequency band is needed at high resolution;
|
||||
- arbitrary frequency grid matters.
|
||||
|
||||
Pros:
|
||||
|
||||
- Efficient zoomed spectral analysis.
|
||||
|
||||
Cons:
|
||||
|
||||
- More complex than current fixed bins.
|
||||
- Might be premature unless physical data shows bin boundaries are the problem.
|
||||
|
||||
### Option D: Matched Chirplet Atoms
|
||||
|
||||
Best when:
|
||||
|
||||
- emitted atoms are varied and not reducible to one common slope.
|
||||
|
||||
Pros:
|
||||
|
||||
- General.
|
||||
|
||||
Cons:
|
||||
|
||||
- Too expensive and too unconstrained for the current controlled beacon.
|
||||
|
||||
## Implementation Rule
|
||||
|
||||
Build the controlled CSS-like receiver first. Pull in full chirplet machinery
|
||||
only when the codebook stops being controlled or when a research proof shows a
|
||||
specific invariant the controlled receiver cannot protect.
|
||||
|
||||
|
|
@ -0,0 +1,174 @@
|
|||
# Current Hotspot Audit
|
||||
|
||||
## What This Is
|
||||
|
||||
This is a static audit of the current repo for likely allocation, scan, and
|
||||
driver-pressure hotspots. It is not a profiler output. Treat it as a list of
|
||||
places to benchmark once the next implementation cut starts.
|
||||
|
||||
## High-Risk Runtime Hotspots
|
||||
|
||||
### `MimirAudioSynchronizationAnalyzer.Analyze`
|
||||
|
||||
Why it is hot:
|
||||
|
||||
- Called repeatedly by `MimirSynchronizationHub`.
|
||||
- Extracts windows from rolling buffers.
|
||||
- Converts PCM formats.
|
||||
- Resamples candidate windows.
|
||||
- Invokes passive or active decode.
|
||||
|
||||
Risk shape:
|
||||
|
||||
- `ExtractMonoWindow` builds a `List<float>` and returns `ToArray`.
|
||||
- Candidate windows can be re-extracted on every analysis step.
|
||||
- Resampling allocates a new output array.
|
||||
- Reports/traces allocate lists and records.
|
||||
|
||||
Coherent next cut:
|
||||
|
||||
- Add an analyzer scratch/context object owning reusable buffers.
|
||||
- Add streaming active decoder state so repeated windows are not rescanned.
|
||||
- Cache reference extracted window/spectrum per analysis edge.
|
||||
|
||||
### `MimirChirpBinTimeline.DecodeStreamWindow`
|
||||
|
||||
Why it is hot:
|
||||
|
||||
- Current active path for chirp-only/hybrid sync.
|
||||
- Runs event proposal, classification, anchor decode, clock fit.
|
||||
|
||||
Risk shape:
|
||||
|
||||
- `BuildWindowEnergyTrace` allocates prefix/output arrays.
|
||||
- `DetectFrames` sorts proposals by energy.
|
||||
- `ScoreBins` scores every symbol for each candidate.
|
||||
- `DecodeAnchors` creates candidate windows and path arrays.
|
||||
- LINQ/`ToArray` appears in several inner-ish paths.
|
||||
|
||||
Coherent next cut:
|
||||
|
||||
- Split into streaming proposal state and stateless symbol scorer.
|
||||
- Replace LINQ in candidate/code paths with bounded arrays/spans once behavior
|
||||
is frozen.
|
||||
- Move calibration likelihood into the scoring stage before candidate pruning.
|
||||
|
||||
### `MimirPassiveAudioSynchronizationEstimator.Estimate`
|
||||
|
||||
Why it is hot:
|
||||
|
||||
- Passive/hybrid mode can compare multiple sources.
|
||||
- FFT arrays allocate per estimate.
|
||||
|
||||
Risk shape:
|
||||
|
||||
- `Complex[]` reference/candidate arrays per call.
|
||||
- Hann/preemphasis recomputed per call.
|
||||
- Reference FFT repeated for each candidate.
|
||||
|
||||
Coherent next cut:
|
||||
|
||||
- Scratch object with reusable complex buffers.
|
||||
- Precomputed window.
|
||||
- Reference spectrum cache per edge.
|
||||
- Limit lag search by known physical/network horizon.
|
||||
|
||||
### `MimirRuntime.QueueCalibrationTimeline`
|
||||
|
||||
Why it is hot-ish:
|
||||
|
||||
- Renders/encodes active witness audio inside runtime update.
|
||||
|
||||
Risk shape:
|
||||
|
||||
- Allocates PCM bytes and base64 every batch.
|
||||
- Renders sine chirps with scalar `Math.Sin`.
|
||||
|
||||
Coherent next cut:
|
||||
|
||||
- For production, move witness generation to a reusable native/DSP oscillator or
|
||||
pre-rendered segment ring. Keep runtime policy, move audio hot work.
|
||||
|
||||
## Native Hotspots
|
||||
|
||||
### `native/asio_capture/pushInputBlocks`
|
||||
|
||||
Why it is hot:
|
||||
|
||||
- Runs from ASIO callback.
|
||||
- Copies every input channel into a queued `std::vector<float>`.
|
||||
|
||||
Risk shape:
|
||||
|
||||
- Allocates/copies per channel block.
|
||||
- Uses mutex-protected deque.
|
||||
|
||||
Current acceptability:
|
||||
|
||||
- Fine for proof and likely fine for four Scarlett channels at 192 kHz in short
|
||||
term.
|
||||
|
||||
Coherent next cut:
|
||||
|
||||
- Preallocated block pool or SPSC ring if profiling shows callback pressure.
|
||||
|
||||
### `native/probes/ks_camera_cadence/measureCandidate`
|
||||
|
||||
Why it is hot:
|
||||
|
||||
- Direct KS async read queue for camera cadence.
|
||||
|
||||
Risk shape:
|
||||
|
||||
- Diagnostic stdout and optional JSON events.
|
||||
- Probe code owns too much CLI behavior for production.
|
||||
|
||||
Coherent next cut:
|
||||
|
||||
- Extract the async read queue shape into a direct driver worker, minus stdout
|
||||
and measurement ceremony.
|
||||
|
||||
### `native/reservoir/RollingReservoir`
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Intended lower shared-edge retention authority.
|
||||
|
||||
Risk shape:
|
||||
|
||||
- Current Rust `VecDeque` and typed-view iteration are conceptually clean but
|
||||
not necessarily the final cache-optimal layout.
|
||||
|
||||
Coherent next cut:
|
||||
|
||||
- Keep invariant. Replace storage only when Fensalir/Faust integration proves
|
||||
the need.
|
||||
|
||||
## Allocation Smells Found By Static Scan
|
||||
|
||||
Likely production-relevant:
|
||||
|
||||
- `MimirChirpBinTimeline`: repeated `List<>`, `ToArray`, sorting, grouping in
|
||||
decode path.
|
||||
- `MimirAudioSynchronizationAnalyzer`: window extraction and resampling arrays.
|
||||
- `MimirPassiveAudioSynchronizationEstimator`: per-call FFT arrays.
|
||||
- `MimirAsioStreamSource`: per-block byte arrays after native read.
|
||||
- `MimirRollingStreamBuffer.Snapshot`: full array snapshot for readers.
|
||||
|
||||
Mostly diagnostic:
|
||||
|
||||
- `Mimir.BufferSmoke` allocations.
|
||||
- Native probe stdout/JSON allocations.
|
||||
- Bridge scripts.
|
||||
|
||||
## Rule For Next Optimization
|
||||
|
||||
Do not optimize the whole repo. Pick one measured hot path:
|
||||
|
||||
1. physical chirp-bin decode;
|
||||
2. passive FFT reuse;
|
||||
3. ASIO queue pressure;
|
||||
4. camera driver handoff.
|
||||
|
||||
Then make one ownership-improving cut and benchmark before widening.
|
||||
|
||||
214
research/perfect-machine-study-2026-05-23/data-dictionary.md
Normal file
214
research/perfect-machine-study-2026-05-23/data-dictionary.md
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
# Data Dictionary
|
||||
|
||||
## Purpose
|
||||
|
||||
This dictionary names the data objects that keep recurring across code, docs,
|
||||
samples, and planned commands.
|
||||
|
||||
## Stream Descriptor
|
||||
|
||||
Identity for one source stream.
|
||||
|
||||
Fields:
|
||||
|
||||
- source id;
|
||||
- kind (`audio`, `video`, etc.);
|
||||
- origin (`local`, `network`, `process`, `native`);
|
||||
- human label;
|
||||
- optional device/config metadata.
|
||||
|
||||
Invariant:
|
||||
|
||||
- one descriptor maps to one rolling buffer.
|
||||
|
||||
## Stream Sample
|
||||
|
||||
One runtime sample envelope.
|
||||
|
||||
Fields:
|
||||
|
||||
- descriptor/source id;
|
||||
- timestamp;
|
||||
- sequence;
|
||||
- audio block descriptor or video frame descriptor;
|
||||
- optional payload bytes/handle.
|
||||
|
||||
Invariant:
|
||||
|
||||
- samples are indexed by runtime buffers; they are not long-term history.
|
||||
|
||||
## Audio Block Descriptor
|
||||
|
||||
Metadata for a block of audio.
|
||||
|
||||
Fields:
|
||||
|
||||
- sample rate;
|
||||
- channel count;
|
||||
- sample format;
|
||||
- frame count;
|
||||
- timestamp;
|
||||
- optional native handle.
|
||||
|
||||
Invariant:
|
||||
|
||||
- production payload should move toward native handles, not managed byte arrays.
|
||||
|
||||
## Video Frame Descriptor
|
||||
|
||||
Metadata for one camera frame.
|
||||
|
||||
Fields:
|
||||
|
||||
- width;
|
||||
- height;
|
||||
- pixel format;
|
||||
- stride;
|
||||
- device timestamp;
|
||||
- byte length;
|
||||
- optional native/GPU handle.
|
||||
|
||||
Invariant:
|
||||
|
||||
- frame descriptor must preserve enough information for Fensalir to import or
|
||||
upload without guessing.
|
||||
|
||||
## Chirp Symbol
|
||||
|
||||
One emitted active-witness event.
|
||||
|
||||
Fields:
|
||||
|
||||
- symbol id;
|
||||
- frequency bin;
|
||||
- chirp slope/window/duration;
|
||||
- expected event time;
|
||||
- optional rhythmic/gap evidence.
|
||||
|
||||
Invariant:
|
||||
|
||||
- symbol shape is controlled by the emitter and known by the receiver.
|
||||
|
||||
## Chirp Anchor
|
||||
|
||||
A decoded observation that maps a local sample offset to a canonical event.
|
||||
|
||||
Fields:
|
||||
|
||||
- event index;
|
||||
- symbol id;
|
||||
- local sample offset;
|
||||
- canonical time;
|
||||
- confidence;
|
||||
- residual/error;
|
||||
- bin shift hypothesis.
|
||||
|
||||
Invariant:
|
||||
|
||||
- anchors are the bridge between local sample time and canonical time.
|
||||
|
||||
## Clock Fit
|
||||
|
||||
Affine map between local samples and canonical time.
|
||||
|
||||
Fields:
|
||||
|
||||
- source offset samples;
|
||||
- effective sample rate;
|
||||
- confidence;
|
||||
- anchor count;
|
||||
- mean absolute error.
|
||||
|
||||
Invariant:
|
||||
|
||||
- clock fit is evidence state, not a sample-moving actuator.
|
||||
|
||||
## Calibration Path
|
||||
|
||||
Per-output/mic model.
|
||||
|
||||
Fields:
|
||||
|
||||
- output path id;
|
||||
- source/mic id;
|
||||
- sample rate;
|
||||
- usable bands;
|
||||
- symbol calibrations;
|
||||
- confusion observations;
|
||||
- delay hypotheses;
|
||||
- phase/group-delay summaries;
|
||||
- codebook plan.
|
||||
|
||||
Invariant:
|
||||
|
||||
- calibration alters likelihood and emission planning before decode failure
|
||||
happens.
|
||||
|
||||
## Sync Report
|
||||
|
||||
Analyzer output for one source relative to reference.
|
||||
|
||||
Fields:
|
||||
|
||||
- source id;
|
||||
- delay samples;
|
||||
- delay milliseconds/microseconds;
|
||||
- fractional delay;
|
||||
- confidence;
|
||||
- band response;
|
||||
- decode trace.
|
||||
|
||||
Invariant:
|
||||
|
||||
- report describes measurement; DSP decides how to move audio.
|
||||
|
||||
## Sync State
|
||||
|
||||
Smoothed runtime belief.
|
||||
|
||||
Fields:
|
||||
|
||||
- latest delay;
|
||||
- smoothed delay;
|
||||
- confidence;
|
||||
- delay slope / SRO ppm;
|
||||
- per-band evidence.
|
||||
|
||||
Invariant:
|
||||
|
||||
- UI reads this; analyzers write it; render tick does not recompute it.
|
||||
|
||||
## Acoustic Constraint
|
||||
|
||||
Fensalir-facing spatial/timing evidence.
|
||||
|
||||
Fields:
|
||||
|
||||
- constraint id;
|
||||
- kind;
|
||||
- position/velocity/radius;
|
||||
- confidence;
|
||||
- timestamp;
|
||||
- timing uncertainty.
|
||||
|
||||
Invariant:
|
||||
|
||||
- raw PCM stays out of Fensalir contract frames.
|
||||
|
||||
## Visual Observation
|
||||
|
||||
Fensalir-facing camera evidence.
|
||||
|
||||
Fields:
|
||||
|
||||
- source id;
|
||||
- native/shared texture handle or payload handle;
|
||||
- pixel format;
|
||||
- dimensions;
|
||||
- calibration id;
|
||||
- canonical timestamp.
|
||||
|
||||
Invariant:
|
||||
|
||||
- Fensalir owns GPU lifetime after import/lowering.
|
||||
|
||||
|
|
@ -0,0 +1,196 @@
|
|||
# Decoder Architecture Options
|
||||
|
||||
## Purpose
|
||||
|
||||
The active decoder must eventually be small enough to run on a phone, robust
|
||||
enough for meatspace microphones, and cheap enough not to eat a CPU core per
|
||||
stream. This file separates the receiver shapes so the next cut can be made on
|
||||
invariants instead of frustration.
|
||||
|
||||
## Current Runtime Shape
|
||||
|
||||
`MimirChirpBinTimeline.DecodeStreamWindow` currently does the whole job for a
|
||||
bounded PCM window:
|
||||
|
||||
1. build an energy trace;
|
||||
2. find candidate event windows;
|
||||
3. classify each window with chirp-bin scoring;
|
||||
4. preserve ambiguity candidates;
|
||||
5. build code-valid anchors from the de Bruijn schedule;
|
||||
6. search delay/bin-shift hypotheses;
|
||||
7. fit an affine source clock;
|
||||
8. emit calibration response/confusion evidence.
|
||||
|
||||
This is coherent as a proof machine. It is not yet the final streaming receiver.
|
||||
|
||||
## Option A: Streaming Scalar Receiver
|
||||
|
||||
### Shape
|
||||
|
||||
- Maintain a rolling PCM ring.
|
||||
- Maintain an energy/proposal ring keyed by absolute sample index.
|
||||
- When a proposal crosses threshold, score only the local candidate window.
|
||||
- Keep the top K symbol hypotheses per proposal.
|
||||
- Run a small codebook trellis over recent candidates.
|
||||
- Update clock fit incrementally.
|
||||
|
||||
### Why It Fits
|
||||
|
||||
The emitter controls the symbol vocabulary and schedule. The receiver does not
|
||||
need to search an unconstrained chirplet dictionary. It needs to notice likely
|
||||
events and map them through a codebook.
|
||||
|
||||
### Risks
|
||||
|
||||
- Scalar scoring may still be too expensive at 192 kHz across many mics.
|
||||
- Proposal thresholds can hide weak physical events if calibration is not used
|
||||
before rejection.
|
||||
- Trellis state must preserve ambiguity, not collapse to one wrong symbol.
|
||||
|
||||
### Promote When
|
||||
|
||||
It matches the current window decoder on synthetic and artifact captures while
|
||||
reducing repeated work by at least an order of magnitude.
|
||||
|
||||
## Option B: SIMD Dechirp + Goertzel Receiver
|
||||
|
||||
### Shape
|
||||
|
||||
- Precompute symbol reference chirps or dechirp oscillators.
|
||||
- Process candidate windows in AVX2/FMA blocks.
|
||||
- Accumulate real/imag Goertzel-style bin responses.
|
||||
- Produce dense scores for the codebook trellis.
|
||||
|
||||
### Why It Fits
|
||||
|
||||
The hot math is mostly multiply-add and reduction over short windows. That is
|
||||
exactly where SIMD is boring and useful. Boring and useful beats clever and
|
||||
fragile here.
|
||||
|
||||
### Risks
|
||||
|
||||
- Small windows and few bins may be memory-bound or dispatch-bound rather than
|
||||
arithmetic-bound.
|
||||
- Managed/native transition cost can erase wins if called per candidate instead
|
||||
of in batches.
|
||||
- Alignment and denormal behavior need explicit treatment.
|
||||
|
||||
### Promote When
|
||||
|
||||
Profiling shows scalar C# scoring dominates, and a batched native scorer is
|
||||
faster after interop overhead is included.
|
||||
|
||||
## Option C: Dechirp + FFT/CZT Receiver
|
||||
|
||||
### Shape
|
||||
|
||||
- Dechirp candidate windows against known slope.
|
||||
- Use FFT or chirp-Z transform to estimate symbol bin/frequency offset.
|
||||
- Feed bin likelihoods into codebook path selection.
|
||||
|
||||
### Why It Fits
|
||||
|
||||
CSS/LoRa receivers validate this family. It also handles frequency offset and
|
||||
bin interpolation naturally when the symbol set is dense.
|
||||
|
||||
### Risks
|
||||
|
||||
- FFT setup and memory traffic can dominate for 32 bins and small windows.
|
||||
- A full FFT is wasteful if only a small fixed bin bank is needed.
|
||||
- CZT is attractive for narrowband zooming, but more complex than necessary
|
||||
until measured bin interpolation is the problem.
|
||||
|
||||
### Promote When
|
||||
|
||||
The codebook expands, frequency offset becomes first-order, or batched GPU/native
|
||||
FFT plans beat SIMD Goertzel on actual candidate batches.
|
||||
|
||||
## Option D: Full Chirplet Transform / Fast Chirplet Transform
|
||||
|
||||
### Shape
|
||||
|
||||
- Search a chirplet parameter space over time, center frequency, duration,
|
||||
chirp rate, and scale.
|
||||
- Use fast chirplet transform variants or maximum chirplet decomposition to
|
||||
locate arbitrary chirp-like atoms.
|
||||
|
||||
### Why It Fits
|
||||
|
||||
It is the canonical general tool when the emitted atoms are unknown or when we
|
||||
are analyzing natural signals.
|
||||
|
||||
### Why It Does Not Fit The Hot Path Yet
|
||||
|
||||
Mimir controls the emitter. Searching a large affine dictionary to rediscover a
|
||||
known alphabet is buying a forklift to move a teacup. Good forklift, wrong
|
||||
room.
|
||||
|
||||
### Promote When
|
||||
|
||||
Only if the controlled-codebook receiver cannot survive physical paths even
|
||||
after response weighting, group-delay correction, and codebook adaptation.
|
||||
|
||||
## Option E: Learned / Adaptive Receiver
|
||||
|
||||
### Shape
|
||||
|
||||
- Train or fit a compact classifier on calibration captures.
|
||||
- Use learned symbol likelihood surfaces and delay priors.
|
||||
- Keep codebook trellis deterministic after likelihood generation.
|
||||
|
||||
### Why It Fits
|
||||
|
||||
Physical paths produce systematic distortion: dead bands, aliases, reflections,
|
||||
group delay, and mic-specific response. A learned likelihood surface may capture
|
||||
what hand weighting misses.
|
||||
|
||||
### Risks
|
||||
|
||||
- A learned classifier can become an uninspectable compensator.
|
||||
- Training data may overfit one room/gain setting.
|
||||
- It must not replace the deterministic schedule map.
|
||||
|
||||
### Promote When
|
||||
|
||||
Calibration-weighted deterministic decoding fails in repeatable, explainable
|
||||
ways that a compact trained likelihood model can resolve on held-out captures.
|
||||
|
||||
## Recommended Cut
|
||||
|
||||
Build Option A first, but design the scoring call as a replaceable batched
|
||||
kernel so Option B or C can occupy it without changing ownership:
|
||||
|
||||
```text
|
||||
PCM ring -> proposal ring -> batched score kernel -> ambiguity trellis -> clock fit
|
||||
```
|
||||
|
||||
Only the score kernel should vary. The codebook, schedule, calibration model,
|
||||
and clock fit remain the receiver's authority.
|
||||
|
||||
## Phone Receiver Shape
|
||||
|
||||
A phone should need:
|
||||
|
||||
- source codebook id;
|
||||
- current schedule seed/epoch;
|
||||
- sample rate estimate;
|
||||
- rolling PCM buffer;
|
||||
- compact path calibration if known.
|
||||
|
||||
It should not need:
|
||||
|
||||
- Starfire round trips for each event;
|
||||
- Raven transport timing;
|
||||
- Fensalir GPU state;
|
||||
- OBS state.
|
||||
|
||||
The phone receiver emits:
|
||||
|
||||
- decoded anchor observations;
|
||||
- fitted local clock to canonical time;
|
||||
- confidence/residuals;
|
||||
- response/confusion profile.
|
||||
|
||||
That profile can come home later. The phone does not need to be trusted as a
|
||||
clock authority; it is a self-locating sensor.
|
||||
|
||||
|
|
@ -0,0 +1,150 @@
|
|||
# Distributed Receiver Spec
|
||||
|
||||
## Purpose
|
||||
|
||||
Mimir must work when Raven, a phone, or another networked sensor hears the same
|
||||
active witness late, compressed, resampled, and out of Starfire's immediate
|
||||
clock domain. The active codebook is meant to make that possible.
|
||||
|
||||
## Objective
|
||||
|
||||
A remote receiver that knows the source codebook and schedule should be able to
|
||||
place local samples on the canonical timeline without round-tripping every
|
||||
sample through Starfire. Network timestamps are metadata. Decoded anchors are
|
||||
the timing truth.
|
||||
|
||||
## Receiver Inputs
|
||||
|
||||
- codebook id;
|
||||
- schedule seed/epoch;
|
||||
- nominal sample rate;
|
||||
- local audio stream;
|
||||
- optional path calibration;
|
||||
- optional expected frequency band limits;
|
||||
- optional network time estimate for coarse acquisition only.
|
||||
|
||||
## Receiver State
|
||||
|
||||
Minimum:
|
||||
|
||||
- rolling PCM ring;
|
||||
- proposal ring;
|
||||
- candidate symbol ring;
|
||||
- codebook trellis state;
|
||||
- local clock fit to canonical time;
|
||||
- response/confusion accumulator;
|
||||
- health counters.
|
||||
|
||||
It does not need:
|
||||
|
||||
- Fensalir render state;
|
||||
- OBS state;
|
||||
- Starfire ASIO state;
|
||||
- per-sample network acknowledgements.
|
||||
|
||||
## Receiver Output
|
||||
|
||||
```json
|
||||
{
|
||||
"sourceId": "phone-left",
|
||||
"codebookId": "main-room-2026-05-24",
|
||||
"canonicalClockFit": {
|
||||
"localSampleAtEpoch": 12345678.5,
|
||||
"effectiveSampleRate": 47999.92,
|
||||
"confidence": 0.91,
|
||||
"meanAbsErrorSamples": 0.37
|
||||
},
|
||||
"anchors": [],
|
||||
"responseProfile": {},
|
||||
"health": {}
|
||||
}
|
||||
```
|
||||
|
||||
The output is compact enough for LAN transport. Raw audio can still be streamed
|
||||
when needed, but the timing machine should not depend on raw audio arriving
|
||||
with low latency.
|
||||
|
||||
## Canonical Mapping
|
||||
|
||||
The receiver fits:
|
||||
|
||||
```text
|
||||
local_sample = source_offset + canonical_seconds * effective_sample_rate
|
||||
```
|
||||
|
||||
The inverse places a local sample on the source timeline:
|
||||
|
||||
```text
|
||||
canonical_seconds = (local_sample - source_offset) / effective_sample_rate
|
||||
```
|
||||
|
||||
Any code-valid anchor improves the fit. A triplet or higher-order tuple
|
||||
identifies which canonical event was heard. Repeated anchors estimate drift.
|
||||
|
||||
## Acquisition Strategy
|
||||
|
||||
1. Use broad energy/onset proposals to avoid scanning every symbol everywhere.
|
||||
2. Score candidates through the calibrated symbol likelihood.
|
||||
3. Preserve top-K ambiguities.
|
||||
4. Search code-valid tuples against the known schedule.
|
||||
5. Fit local clock.
|
||||
6. Narrow future search around predicted event times.
|
||||
|
||||
That last step is the efficiency win. Once locked, the receiver is not hunting
|
||||
the whole universe; it is checking whether expected events arrived where the
|
||||
current clock predicts.
|
||||
|
||||
## Network Behavior
|
||||
|
||||
Remote nodes should send:
|
||||
|
||||
- periodic clock-fit summaries;
|
||||
- decoded anchor batches;
|
||||
- response/confusion updates;
|
||||
- buffer health;
|
||||
- optional compressed/raw audio windows for debugging or program use.
|
||||
|
||||
Starfire should not assume:
|
||||
|
||||
- network latency is stable;
|
||||
- transport order is canonical order;
|
||||
- remote capture sample rate is exact;
|
||||
- packet timestamps are better than decoded anchors.
|
||||
|
||||
## Raven Shape
|
||||
|
||||
Raven is special because it is a co-stream/game machine with its own Scarlett.
|
||||
It can:
|
||||
|
||||
- hear/emit the same active witness;
|
||||
- decode local clock against Starfire's schedule;
|
||||
- stream compact anchor/response state to Starfire;
|
||||
- optionally stream audio/video payloads into Starfire's rolling buffers with
|
||||
canonical-time metadata attached.
|
||||
|
||||
Raven should not:
|
||||
|
||||
- do heavy soundfield reconstruction;
|
||||
- become the global clock authority unless explicitly selected;
|
||||
- depend on OBS or game timing as source truth.
|
||||
|
||||
## Phone Shape
|
||||
|
||||
A phone is the harsh receiver target:
|
||||
|
||||
- unknown mic response;
|
||||
- OS audio resampling;
|
||||
- possible AGC/noise suppression;
|
||||
- weaker CPU budget;
|
||||
- awkward latency.
|
||||
|
||||
If the phone can decode enough anchors to estimate canonical time, the codebook
|
||||
is doing its job. If it cannot, the failure is likely response/codebook design,
|
||||
not network transport.
|
||||
|
||||
## Security / Identity Note
|
||||
|
||||
Codebook possession should be enough to decode time, not enough to control
|
||||
Mimir. Remote timing reports need source identity and trust policy before they
|
||||
affect program output.
|
||||
|
||||
236
research/perfect-machine-study-2026-05-23/failure-mode-ledger.md
Normal file
236
research/perfect-machine-study-2026-05-23/failure-mode-ledger.md
Normal file
|
|
@ -0,0 +1,236 @@
|
|||
# Failure Mode Ledger
|
||||
|
||||
## Purpose
|
||||
|
||||
This ledger names failure modes as data-flow problems. The fix should generally
|
||||
move authority or measurement to the right place, not add another after-the-fact
|
||||
filter.
|
||||
|
||||
## Active Witness Failures
|
||||
|
||||
### Dead Frequency Band
|
||||
|
||||
Symptom:
|
||||
|
||||
- emitted symbol is consistently absent or weak on one mic.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- calibration marks band unreliable;
|
||||
- codebook emission avoids it for that path or shared room plan;
|
||||
- decoder weights it down before trellis selection.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- classify as normal, then reject "outliers" later.
|
||||
|
||||
### Symbol Alias
|
||||
|
||||
Symptom:
|
||||
|
||||
- expected symbol often scores as a neighboring bin.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- confusion matrix changes likelihood surface;
|
||||
- global bin-shift hypotheses are searched as part of the window;
|
||||
- codebook spacing/order adapts.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- hard-code a special-case symbol remap without path identity.
|
||||
|
||||
### Reflection-Dominated Timing
|
||||
|
||||
Symptom:
|
||||
|
||||
- anchors decode, but delay residuals jump between plausible paths.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- preserve multiple delay hypotheses;
|
||||
- combine with mic/speaker geometry and repeated state;
|
||||
- report uncertainty to Fensalir/DSP.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- smooth the jump until it looks calm.
|
||||
|
||||
### Gain / Clipping
|
||||
|
||||
Symptom:
|
||||
|
||||
- loopback or mic confidence collapses despite audible witness.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- calibration preflight checks clipping/noise floor;
|
||||
- artifacts record gain state;
|
||||
- witness amplitude adapts within safe bounds.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- lower thresholds globally.
|
||||
|
||||
## Passive Sync Failures
|
||||
|
||||
### Silence
|
||||
|
||||
Symptom:
|
||||
|
||||
- passive estimator emits weak or meaningless delay.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- confidence falls;
|
||||
- hybrid emits active watermark/chirp witness.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- hold stale passive delay as if it were current truth.
|
||||
|
||||
### Narrowband Music
|
||||
|
||||
Symptom:
|
||||
|
||||
- PHAT correlation peak is broad or ambiguous.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- passive confidence remains advisory;
|
||||
- active witness or longer evidence window takes authority.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- pretend a low-frequency beat gives microsecond timing.
|
||||
|
||||
## Audio Actuator Failures
|
||||
|
||||
### Delay Corrected, Drift Ignored
|
||||
|
||||
Symptom:
|
||||
|
||||
- streams align briefly and then walk apart.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- separate delay and SRO loops;
|
||||
- drive ASRC/fractional delay continuously.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- periodically snap delay back with audible discontinuity.
|
||||
|
||||
### Actuator Chases Source Motion
|
||||
|
||||
Symptom:
|
||||
|
||||
- moving speaker/mic looks like sample-rate drift.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- slow SRO loop uses long-term clock evidence;
|
||||
- fast TDOA/source motion remains separate.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- one smoothing constant for everything.
|
||||
|
||||
## Camera Failures
|
||||
|
||||
### Advertised FPS Lies
|
||||
|
||||
Symptom:
|
||||
|
||||
- device reports 60/120 fps but delivers less.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- measure actual cadence after warm-up;
|
||||
- record topology/control settings;
|
||||
- treat measured cadence as truth.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- trust media type advertisements.
|
||||
|
||||
### Shared USB Collapse
|
||||
|
||||
Symptom:
|
||||
|
||||
- one device's cadence drops when another device appears.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- topology/cadence benchmark;
|
||||
- distribute devices across controllers/hosts;
|
||||
- record known bad neighborhoods.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- add buffering and call latency acceptable.
|
||||
|
||||
### JSON Bridge Becomes Production
|
||||
|
||||
Symptom:
|
||||
|
||||
- diagnostic frame events are relied on for live six-camera ingest.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- promote probe logic into native worker ABI;
|
||||
- payload handles enter runtime/reservoir.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- optimize JSON parsing.
|
||||
|
||||
## Fusion Failures
|
||||
|
||||
### CPU Pixel Furnace
|
||||
|
||||
Symptom:
|
||||
|
||||
- managed code demosaics/converts/processes every camera frame.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- push feature extraction/upload into Fensalir GPU path;
|
||||
- keep CPU metadata small.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- add worker threads until the thermal budget files a complaint.
|
||||
|
||||
### Parallel Stable Track Owners
|
||||
|
||||
Symptom:
|
||||
|
||||
- Mimir and Fensalir both maintain stable visual/acoustic track identity.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- Mimir owns observations;
|
||||
- Fensalir temporal evidence reservoir owns tracks after lowering.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- add id synchronization glue.
|
||||
|
||||
## Network Failures
|
||||
|
||||
### Transport Timestamp Worship
|
||||
|
||||
Symptom:
|
||||
|
||||
- Raven/phone timing is derived from packet arrival time.
|
||||
|
||||
Coherent fix:
|
||||
|
||||
- remote receiver decodes codebook anchors locally;
|
||||
- network time is coarse metadata only.
|
||||
|
||||
Bad fix:
|
||||
|
||||
- add latency compensation constants per sender.
|
||||
|
||||
|
|
@ -0,0 +1,180 @@
|
|||
# Fensalir Integration Map
|
||||
|
||||
## Purpose
|
||||
|
||||
Mimir should not grow a second renderer, second temporal evidence reservoir, or
|
||||
second UI engine. Fensalir already owns the window, D3D12 device, render graph,
|
||||
GPU sensor fusion packets, temporal Gaussian field, acoustic constraint packet,
|
||||
debug chrome, and future Spout/program output. Mimir owns the physical truth:
|
||||
device capture, codebook schedules, calibration state, rolling runtime buffers,
|
||||
network feed ingress, and the decision about which evidence enters the field.
|
||||
|
||||
The integration cut is therefore simple:
|
||||
|
||||
- Mimir produces timestamped observations and calibration constraints.
|
||||
- Fensalir lowers those observations into D3D12 resources and temporal fields.
|
||||
- Mimir reads health/status and manages policy.
|
||||
- OBS receives final program surfaces/stems, not raw driver noise.
|
||||
|
||||
## Current Fensalir Surfaces
|
||||
|
||||
This pass inspected `E:\Projects\Fensalir` because the engine moved out of this
|
||||
repo and the old app-specific mental model is now stale.
|
||||
|
||||
Relevant contract surfaces:
|
||||
|
||||
- `AquariumSceneState.GpuSensorFrame`
|
||||
- `AquariumSceneState.AcousticFieldFrame`
|
||||
- `AquariumSceneState.CalibrationEventFrame`
|
||||
- `AquariumSceneState.GpuFusionField`
|
||||
- `AquariumSceneState.TemporalGaussianField`
|
||||
- `AquariumRenderPlan`
|
||||
- `AquariumShaderManifest`
|
||||
|
||||
Relevant engine concepts:
|
||||
|
||||
- D3D12 renderer owns swapchain, command lists, descriptor heaps, upload rings,
|
||||
shader compilation, and render graph execution.
|
||||
- `D3D12ExternalSensorTexture` is the natural landing point for imported camera
|
||||
frames once the native capture path can provide a shareable GPU handle.
|
||||
- `D3D12GpuSensorFusion.hlsl` and `D3D12TemporalGaussian.hlsl` already express
|
||||
the shape of a GPU-resident fusion pass.
|
||||
- `TemporalSpatialEvidenceReservoir` owns stable evidence tracks. Mimir should
|
||||
not invent a parallel stable-key cache for visual/acoustic claims.
|
||||
- `D3D12AcousticConstraintPacket` is the visible contract slot for decoded
|
||||
audio geometry constraints.
|
||||
- `D3D12Renderer` already converts `AquariumAcousticConstraint` and clap
|
||||
calibration events into acoustic constraint packets, so Mimir should lower
|
||||
decoded audio evidence into those contracts rather than creating a side
|
||||
debug-only renderer path.
|
||||
- Fensalir tests already assert that `AquariumSceneState` can carry a
|
||||
`GpuSensorFrame` with shared texture metadata and an `AcousticFieldFrame`
|
||||
with timing confidence, uncertainty, accumulation window, presentation delay,
|
||||
and spatial constraints.
|
||||
|
||||
## Target Pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["native camera callbacks"] --> B["Mimir.Runtime rolling buffer"]
|
||||
C["ASIO callbacks"] --> B
|
||||
D["network feed packets"] --> B
|
||||
B --> E["Mimir calibration and synchronization kernel"]
|
||||
E --> F["AquariumGpuSensorFrame"]
|
||||
E --> G["AquariumAcousticFieldFrame"]
|
||||
E --> H["AquariumCalibrationEventFrame"]
|
||||
F --> I["Fensalir D3D12 fusion"]
|
||||
G --> I
|
||||
H --> I
|
||||
I --> J["Temporal Gaussian / evidence field"]
|
||||
J --> K["Program video / diagnostics"]
|
||||
```
|
||||
|
||||
## Ownership Invariants
|
||||
|
||||
- Fensalir owns GPU resource lifetime. Mimir may describe external memory, but
|
||||
it must not build a private D3D12 lifetime system.
|
||||
- Fensalir owns temporal evidence reuse once observations are lowered into the
|
||||
shared reservoir. Mimir owns observation truth before lowering.
|
||||
- Mimir owns calibration identity: device ids, mic/speaker path ids, codebook
|
||||
generation, response/confusion matrices, delay/group-delay models, and
|
||||
reliable-bin plans.
|
||||
- Mimir owns the rolling buffer retention boundary. Fensalir can snapshot or
|
||||
consume the live window, but it must not silently become the history owner.
|
||||
- `AquariumAudioDocument.EnqueuePcm16Base64` is not a production Mimir audio
|
||||
path. It is suitable for UI/demo audio, not 192 kHz ASIO timing evidence.
|
||||
|
||||
## Concrete Lowering Contracts
|
||||
|
||||
### Camera Frames
|
||||
|
||||
Mimir source:
|
||||
|
||||
- source id;
|
||||
- canonical timestamp estimate;
|
||||
- device timestamp;
|
||||
- frame sequence number;
|
||||
- dimensions and pixel format;
|
||||
- native CPU pointer or GPU shared handle;
|
||||
- calibration id and lens model;
|
||||
- confidence/status bits.
|
||||
|
||||
Fensalir target:
|
||||
|
||||
- `AquariumGpuSensorFrame`
|
||||
- `AquariumGpuSensorCamera`
|
||||
- `AquariumExternalGpuTexture`
|
||||
|
||||
The first production proof can use CPU upload because it is easier to inspect.
|
||||
The final six-camera path wants GPU import or a pinned/native staging ring.
|
||||
|
||||
### Acoustic Constraints
|
||||
|
||||
Mimir source:
|
||||
|
||||
- decoded chirp anchors;
|
||||
- passive program-audio anchors;
|
||||
- loopback canonical clock fit;
|
||||
- mic delay/SRO estimates;
|
||||
- per-band magnitude/phase response;
|
||||
- TDOA/source localization candidates;
|
||||
- confidence and residual statistics.
|
||||
|
||||
Fensalir target:
|
||||
|
||||
- `AquariumAcousticFieldFrame`
|
||||
- `AquariumAcousticConstraint`
|
||||
|
||||
The acoustic packet should not carry raw PCM. Raw PCM stays in the Mimir/native
|
||||
audio reservoir; Fensalir receives constraints and optional debug surfaces.
|
||||
|
||||
The field frame can also carry timing oracle values. Mimir should derive these
|
||||
from the loopback/reference clock fit:
|
||||
|
||||
- timing oracle timestamp;
|
||||
- timing confidence;
|
||||
- timing uncertainty in microseconds;
|
||||
- accumulation window;
|
||||
- presentation delay.
|
||||
|
||||
That gives Fensalir a scalar timing truth for visualization while keeping the
|
||||
per-source audio correction state in Mimir/Faust.
|
||||
|
||||
### Calibration Events
|
||||
|
||||
Mimir source:
|
||||
|
||||
- emitted chirp/watermark event id;
|
||||
- clap/impulse event id;
|
||||
- detected mic/camera event timestamps;
|
||||
- spatial hypothesis if visual evidence exists.
|
||||
|
||||
Fensalir target:
|
||||
|
||||
- `AquariumCalibrationEventFrame`
|
||||
- `AquariumClapCalibrationEvent`
|
||||
|
||||
The important cut: calibration events are high-confidence anchors, not a
|
||||
separate timeline authority.
|
||||
|
||||
## Avoided Machines
|
||||
|
||||
- A Mimir-side render graph. Fensalir already owns it.
|
||||
- A Mimir-side stable Gaussian track database. Fensalir's temporal evidence
|
||||
reservoir is the correct owner once observations become tracks.
|
||||
- A second app-specific debug panel inside Mimir. Mimir should provide state;
|
||||
Fensalir should render the debugging UI.
|
||||
- Base64 or JSON transport in hot local audio/video paths. Those remain
|
||||
diagnostics and artifact tools.
|
||||
|
||||
## Immediate Implementation Cut
|
||||
|
||||
1. Add a small Mimir-to-Fensalir bridge class that builds an
|
||||
`AquariumGpuSensorFrame` from current `MimirStreamBufferSet` video
|
||||
descriptors.
|
||||
2. Add an acoustic lowering class that turns current sync reports and
|
||||
calibration profiles into `AquariumAcousticFieldFrame`.
|
||||
3. Keep both as pure, allocation-measured mapping code. They should not read
|
||||
devices, run analyzers, or own timers.
|
||||
4. Benchmark one second of repeated lowering against a populated synthetic
|
||||
buffer before connecting it to the app render tick.
|
||||
162
research/perfect-machine-study-2026-05-23/glossary.md
Normal file
162
research/perfect-machine-study-2026-05-23/glossary.md
Normal file
|
|
@ -0,0 +1,162 @@
|
|||
# Glossary
|
||||
|
||||
This glossary is deliberately blunt. It is here so the next implementation pass
|
||||
does not burn half an hour rediscovering which word owns which invariant.
|
||||
|
||||
## Active Witness
|
||||
|
||||
A signal Mimir deliberately emits so receivers can recover canonical time and
|
||||
path response. In this repo the active witness is the chirp-bin/chirplet
|
||||
timeline. It is not only a sync beep; it is also a calibration probe.
|
||||
|
||||
## ASIO
|
||||
|
||||
Low-latency pro-audio driver API. In Mimir, ASIO is the production path for
|
||||
Scarlett loopback and mic capture because it can put output loopback and inputs
|
||||
in the same driver clock domain at 192 kHz.
|
||||
|
||||
## Canonical Timeline
|
||||
|
||||
The emitter-owned time axis. A receiver that knows the codebook and schedule
|
||||
should infer where its captured samples sit on this timeline.
|
||||
|
||||
## Chirp Spread Spectrum
|
||||
|
||||
A communications family that encodes symbols in chirps. LoRa/CSS demodulators
|
||||
matter to Mimir because they show a mature version of dechirp, FFT/bin scoring,
|
||||
timing/frequency-offset handling, and low-power receiver design.
|
||||
|
||||
## Chirplet
|
||||
|
||||
A localized chirp atom: time-limited, frequency-localized, and slope-bearing.
|
||||
The generic chirplet transform searches a broad chirplet parameter space. Mimir
|
||||
uses a constrained chirplet receiver because the emitter controls the codebook.
|
||||
|
||||
## Codebook
|
||||
|
||||
The set of allowed emitted symbols and their shapes. In Mimir this includes
|
||||
frequency bin, glide direction/slope, duration, and rhythmic spacing. It should
|
||||
adapt to the measured physical path by avoiding dead or ambiguous bins.
|
||||
|
||||
## Confusion Matrix
|
||||
|
||||
For each expected symbol, the observed energy distribution across possible
|
||||
symbols/bins. This is how Mimir learns that a mic hears symbol 12 as symbol 10,
|
||||
or does not hear a band at all.
|
||||
|
||||
## De Bruijn Sequence
|
||||
|
||||
A sequence where every length-N symbol tuple appears once in a cycle. Mimir uses
|
||||
the idea so a small number of consecutive chirp events can identify a unique
|
||||
timeline position.
|
||||
|
||||
## Dechirp
|
||||
|
||||
Mix a received chirp against the conjugate/reference chirp so its energy
|
||||
collapses toward a tone/bin. This turns chirp classification into a cheap
|
||||
frequency-bin measurement.
|
||||
|
||||
## Farrow Filter
|
||||
|
||||
A fractional-delay filter structure where coefficients are polynomial functions
|
||||
of the desired fractional sample delay. Useful for continuous delay actuation
|
||||
without rebuilding FIR kernels at audio rate.
|
||||
|
||||
## Fensalir
|
||||
|
||||
The engine repo. It owns windowing, rendering, D3D12, render graph execution,
|
||||
GPU fusion packets, temporal Gaussian/evidence fields, debug UI, and eventual
|
||||
program surface publication.
|
||||
|
||||
## Frequency Response
|
||||
|
||||
How a speaker/room/mic path changes signal magnitude by frequency. Mimir should
|
||||
learn it from the active witness and use it to normalize both calibration and
|
||||
program audio.
|
||||
|
||||
## GCC-PHAT
|
||||
|
||||
Generalized cross-correlation with phase transform weighting. A common robust
|
||||
delay estimator. It is useful for passive program-audio alignment and TDOA, but
|
||||
it is not a substitute for controlled codebook anchors.
|
||||
|
||||
## Gaussian Splat
|
||||
|
||||
A 3D/4D ellipsoidal radiance primitive. For Mimir, splats are a plausible
|
||||
runtime field representation for fused camera evidence, but Fensalir should own
|
||||
the GPU field and renderer.
|
||||
|
||||
## Group Delay
|
||||
|
||||
Frequency-dependent delay through a path. If high bands arrive shifted relative
|
||||
to low bands, symbol timing and response normalization both suffer. The active
|
||||
witness should estimate and correct this.
|
||||
|
||||
## Hybrid Mode
|
||||
|
||||
Use passive program-audio timing by default, emit active witness only when
|
||||
confidence falls. In Mimir this should eventually be a shaped watermark, not a
|
||||
burst of rude test tones.
|
||||
|
||||
## Loopback
|
||||
|
||||
Audio captured from the output path. Scarlett loopback is the current timing
|
||||
authority because it observes what Mimir attempted to emit before acoustic room
|
||||
damage.
|
||||
|
||||
## Near-Field Acoustic Holography
|
||||
|
||||
A family of techniques for reconstructing sound fields from microphone array
|
||||
measurements. It matters because Mimir is not only aligning mics; it is trying
|
||||
to infer a room-scale sound field from sparse sensors.
|
||||
|
||||
## Passive Mode
|
||||
|
||||
No active emitted witness. Mimir estimates delay from program audio such as
|
||||
music/game sound. Useful when the content is rich; fragile during silence or
|
||||
band-limited material.
|
||||
|
||||
## PLL
|
||||
|
||||
Phase-locked loop. In Mimir, a control-rate PLL can turn repeated delay
|
||||
observations into delay and sample-rate-offset corrections for the audio
|
||||
actuator.
|
||||
|
||||
## Reservoir
|
||||
|
||||
A bounded evidence store. The Mimir rolling buffer owns recent raw stream
|
||||
history; Fensalir's temporal evidence reservoir owns stable resolved visual and
|
||||
acoustic field claims after lowering.
|
||||
|
||||
## Sample-Rate Offset
|
||||
|
||||
Clock drift between devices, usually described in ppm. Delay alone aligns one
|
||||
moment; SRO correction keeps streams aligned over time.
|
||||
|
||||
## SPSC Ring
|
||||
|
||||
Single-producer/single-consumer ring buffer. The natural shape for hot native
|
||||
capture callbacks handing blocks to one runtime consumer without locks.
|
||||
|
||||
## SRP-PHAT
|
||||
|
||||
Steered response power with PHAT weighting. It searches source positions by
|
||||
testing which spatial hypothesis best explains inter-mic phase/delay evidence.
|
||||
|
||||
## TDOA
|
||||
|
||||
Time difference of arrival. With known microphone positions, TDOA constraints
|
||||
help localize sound sources.
|
||||
|
||||
## Temporal Gaussian Field
|
||||
|
||||
Fensalir's GPU-side field representation for temporally coherent splat-like
|
||||
evidence. Mimir should feed it observations and constraints, not own its
|
||||
internal rendering state.
|
||||
|
||||
## Watermark
|
||||
|
||||
An active witness shaped to be unobtrusive under program audio. The best Mimir
|
||||
watermark still carries deterministic canonical-time identity and response
|
||||
calibration energy.
|
||||
|
||||
159
research/perfect-machine-study-2026-05-23/hot-path-pseudocode.md
Normal file
159
research/perfect-machine-study-2026-05-23/hot-path-pseudocode.md
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
# Hot Path Pseudocode
|
||||
|
||||
## Purpose
|
||||
|
||||
This note writes the intended hot paths as code-shaped prose. It is not
|
||||
production code. It is a guard against implementation drift.
|
||||
|
||||
## Streaming Active Decoder
|
||||
|
||||
```text
|
||||
onAudioBlock(source, absoluteStart, samples):
|
||||
pcmRing.append(absoluteStart, samples)
|
||||
for each analysisHop completed by this block:
|
||||
energy = updateProposalEnergy(pcmRing, analysisHop)
|
||||
if energy is plausible:
|
||||
proposalRing.push(analysisHop, energy)
|
||||
|
||||
candidates = proposalRing.readySince(lastScoredSample)
|
||||
if candidates is empty:
|
||||
return currentClockFit
|
||||
|
||||
scoreRows = scoreKernel.scoreBatch(pcmRing, candidates, codebook, calibration)
|
||||
for row in scoreRows:
|
||||
ambiguityRing.push(row)
|
||||
|
||||
anchors = trellis.solveRecent(ambiguityRing, schedule, calibration)
|
||||
clockFit.update(anchors)
|
||||
responseModel.observe(scoreRows, anchors)
|
||||
return clockFit
|
||||
```
|
||||
|
||||
Invariant:
|
||||
|
||||
- proposal, scoring, schedule solving, and clock fit are separate phases;
|
||||
- calibration changes likelihood before trellis solving;
|
||||
- old PCM expires with the rolling window.
|
||||
|
||||
## Calibration-Weighted Symbol Score
|
||||
|
||||
```text
|
||||
score(symbol, observedBinEnergy, phase, timing):
|
||||
magnitude = normalize(observedBinEnergy)
|
||||
responseWeight = path.usableBandWeight(symbol)
|
||||
confusion = path.confusionProbability(expected=symbol, observed=bin)
|
||||
phaseWeight = path.phaseCoherence(symbol, phase)
|
||||
groupDelay = path.groupDelayCorrection(symbol)
|
||||
correctedTime = timing - groupDelay
|
||||
return likelihood(magnitude, responseWeight, confusion, phaseWeight, correctedTime)
|
||||
```
|
||||
|
||||
Invariant:
|
||||
|
||||
- dead bands lose authority before code selection;
|
||||
- group delay changes event timing before clock fit;
|
||||
- phase is evidence when stable, not mandatory when unstable.
|
||||
|
||||
## Pairwise Delay From Anchors
|
||||
|
||||
```text
|
||||
estimateDelay(referenceAnchors, sourceAnchors):
|
||||
matches = joinByEventIndex(referenceAnchors, sourceAnchors)
|
||||
if enough(matches):
|
||||
return weightedMean(source.sampleOffset - reference.sampleOffset)
|
||||
|
||||
if bothClockFitsConfident:
|
||||
return sourceFit.sampleFor(t) - referenceFit.sampleFor(t)
|
||||
|
||||
return noReport
|
||||
```
|
||||
|
||||
Invariant:
|
||||
|
||||
- matched event anchors beat independent fits;
|
||||
- independent fits must stay inside live horizon;
|
||||
- no report is better than absurd confidence.
|
||||
|
||||
## DSP Actuator Control
|
||||
|
||||
```text
|
||||
onSyncState(source, delaySamples, sroPpm, confidence):
|
||||
if confidence < threshold:
|
||||
holdOrFadeCorrection()
|
||||
return
|
||||
|
||||
pll.update(delaySamples, sroPpm)
|
||||
actuator.setTargetDelay(pll.delay)
|
||||
actuator.setResampleRatio(1.0 - pll.sroPpm * 1e-6)
|
||||
```
|
||||
|
||||
Invariant:
|
||||
|
||||
- control loop runs at control rate;
|
||||
- sample movement runs at audio rate;
|
||||
- drift and delay are not the same state.
|
||||
|
||||
## Native ASIO Ring
|
||||
|
||||
```text
|
||||
onBufferSwitch(index, samplePosition):
|
||||
block = ring.tryReserve()
|
||||
if block is unavailable:
|
||||
droppedBlocks++
|
||||
return
|
||||
|
||||
block.sourceChannel = channel
|
||||
block.startFrame = samplePosition
|
||||
block.sampleRate = currentRate
|
||||
convertOrCopyDeviceBuffer(block.payload)
|
||||
ring.publish(block)
|
||||
```
|
||||
|
||||
Invariant:
|
||||
|
||||
- callback never allocates;
|
||||
- queue is bounded;
|
||||
- dropped blocks are counted, not hidden.
|
||||
|
||||
## Camera Frame Ring
|
||||
|
||||
```text
|
||||
onReadComplete(slot):
|
||||
frame = ring.tryReserve()
|
||||
if frame unavailable:
|
||||
droppedFrames++
|
||||
resubmit(slot)
|
||||
return
|
||||
|
||||
frame.source = deviceId
|
||||
frame.sequence = sequence++
|
||||
frame.deviceTimestamp = slot.timestamp
|
||||
frame.payloadHandle = slot.payload
|
||||
frame.format = negotiatedFormat
|
||||
ring.publish(frame)
|
||||
resubmit(nextSlot)
|
||||
```
|
||||
|
||||
Invariant:
|
||||
|
||||
- read queue stays full;
|
||||
- frame metadata is typed;
|
||||
- payload ownership is explicit.
|
||||
|
||||
## Fensalir Lowering
|
||||
|
||||
```text
|
||||
buildSceneState(runtimeSnapshot):
|
||||
gpuSensors = lowerVideoDescriptors(runtimeSnapshot.videoWindow)
|
||||
acoustic = lowerSyncAndCalibration(runtimeSnapshot.audioState)
|
||||
calibrationEvents = lowerHighConfidenceEvents(runtimeSnapshot.events)
|
||||
return AquariumSceneState(gpuSensors, acoustic, calibrationEvents)
|
||||
```
|
||||
|
||||
Invariant:
|
||||
|
||||
- lowering reads cached/snapshotted state;
|
||||
- lowering does not poll devices;
|
||||
- lowering does not run analyzers;
|
||||
- Fensalir owns what happens after contract construction.
|
||||
|
||||
|
|
@ -0,0 +1,173 @@
|
|||
# Implementation Roadmap From Study
|
||||
|
||||
## Goal
|
||||
|
||||
Turn the study into implementation passes that improve coherence instead of
|
||||
adding another ring of machinery around the current code.
|
||||
|
||||
## Phase 1: Make Active Decode Streaming
|
||||
|
||||
Objective:
|
||||
|
||||
- Stop rescanning full active windows every analysis tick.
|
||||
|
||||
Current mechanism:
|
||||
|
||||
- Analyzer extracts mono windows and calls `MimirChirpBinTimeline.DecodeStreamWindow`.
|
||||
- Decode rebuilds proposals/frames/anchors from the provided span.
|
||||
|
||||
Intended mechanism:
|
||||
|
||||
- A per-source active decoder owns a rolling PCM window, energy proposal ring,
|
||||
candidate-frame cache, anchor trellis tail, and clock fit state.
|
||||
- Analyzer asks the decoder for the current snapshot.
|
||||
|
||||
Invariants:
|
||||
|
||||
- Same accepted reports as current analyzer on synthetic and loopback captures.
|
||||
- No UI/telemetry re-entry.
|
||||
- Calibration weighting remains optional until physical proof is run.
|
||||
|
||||
Cut line:
|
||||
|
||||
- If streaming state becomes harder to explain than the current decode, cut it
|
||||
back to proposal/frame caching only.
|
||||
|
||||
Proof:
|
||||
|
||||
- Synthetic chirp-only/hybrid tests still recover known delay.
|
||||
- Stored loopback ASIO artifact still decodes 0 us.
|
||||
- Allocation count per analysis tick drops.
|
||||
|
||||
## Phase 2: Physical Calibration-Weighted Decode
|
||||
|
||||
Objective:
|
||||
|
||||
- Make meatspace chirp-bin decode use what calibration already learned.
|
||||
|
||||
Current mechanism:
|
||||
|
||||
- Calibration model can be loaded.
|
||||
- Symbol weights, phase weights, group-delay correction, and bin-shift
|
||||
hypotheses exist but need physical proof.
|
||||
|
||||
Intended mechanism:
|
||||
|
||||
- Symbol likelihood is shaped by reliability/confusion/phase before candidate
|
||||
pruning.
|
||||
- Global delay/bin-shift hypotheses score the whole observation window.
|
||||
- Reduced reliable-bin codebook can be emitted and decoded.
|
||||
|
||||
Invariants:
|
||||
|
||||
- The codebook remains bijective over the intended horizon.
|
||||
- Dead bands lose authority.
|
||||
- Failed timing still emits calibration evidence.
|
||||
|
||||
Proof:
|
||||
|
||||
- Compare unweighted versus weighted decode on stored hot-mic/physical captures.
|
||||
- Emit reliable-bin plan and measure anchor rate through the real monitor/mic
|
||||
path.
|
||||
|
||||
## Phase 3: Actuator Proof
|
||||
|
||||
Objective:
|
||||
|
||||
- Move from "we know the delay" to "samples are aligned."
|
||||
|
||||
Current mechanism:
|
||||
|
||||
- `MimirAudioSynchronizationState` estimates smoothed delay and SRO.
|
||||
|
||||
Intended mechanism:
|
||||
|
||||
- Native/Faust DSP consumes delay/SRO state.
|
||||
- Fractional delay line corrects sub-sample offset.
|
||||
- ASRC/PLL corrects drift.
|
||||
|
||||
Invariants:
|
||||
|
||||
- Runtime estimates; DSP moves samples.
|
||||
- Actuator state is inspectable but not owned by UI.
|
||||
- Output remains separately stemmable for OBS.
|
||||
|
||||
Proof:
|
||||
|
||||
- Offline artifact: apply actuator to delayed candidate and reduce residual.
|
||||
- Live loopback/mic: show residual delay converges and remains bounded.
|
||||
|
||||
## Phase 4: Direct Camera Driver Runtime Path
|
||||
|
||||
Objective:
|
||||
|
||||
- Replace diagnostic frame-event probe transport for local cameras.
|
||||
|
||||
Current mechanism:
|
||||
|
||||
- KS/PS3 probes can emit JSON frame events.
|
||||
- Runtime has `IMimirVideoCaptureDriver` and source adapter seam.
|
||||
|
||||
Intended mechanism:
|
||||
|
||||
- One native/direct driver worker per camera family.
|
||||
- Capture worker preserves timestamp and payload handle.
|
||||
- Runtime indexes sample envelope.
|
||||
- Fensalir consumes handle/window for GPU feature extraction.
|
||||
|
||||
Invariants:
|
||||
|
||||
- No local camera pixels through stdout/base64.
|
||||
- Device/arrival timestamp survives the boundary.
|
||||
- Capture worker owns device pressure, runtime owns identity and buffer.
|
||||
|
||||
Proof:
|
||||
|
||||
- Leap or PS3 Eye frames enter runtime buffers with sustained measured cadence.
|
||||
- Fensalir UI can inspect buffer health without touching pixel hot path.
|
||||
|
||||
## Phase 5: Native Reservoir/Fensalir/Faust Integration
|
||||
|
||||
Objective:
|
||||
|
||||
- Make the native reservoir the shared lower memory/timing authority.
|
||||
|
||||
Current mechanism:
|
||||
|
||||
- C# buffers own runtime proof state.
|
||||
- Rust reservoir exists with C ABI and tests.
|
||||
|
||||
Intended mechanism:
|
||||
|
||||
- Capture workers push handles into native reservoir.
|
||||
- Runtime and Fensalir read typed views.
|
||||
- Faust reads audio block handles and alignment state.
|
||||
|
||||
Invariants:
|
||||
|
||||
- One edge owns retention.
|
||||
- Typed views do not own separate windows.
|
||||
- Payload ownership stays with producer/consumer domain.
|
||||
|
||||
Proof:
|
||||
|
||||
- Runtime status matches C# buffer counts for a mirrored proof.
|
||||
- Fensalir renders a simple current-window visual packet from reservoir claims.
|
||||
|
||||
## Benchmark Harnesses Needed
|
||||
|
||||
1. `chirp-bin-decode-bench`: synthetic and artifact decode timing/allocation.
|
||||
2. `passive-sync-bench`: FFT allocation/reuse comparison.
|
||||
3. `asio-callback-pressure-bench`: native queue/block pool comparison.
|
||||
4. `camera-driver-buffer-bench`: direct driver payload handoff cadence.
|
||||
5. `actuator-residual-bench`: before/after residual delay and drift.
|
||||
|
||||
## Decision Gates
|
||||
|
||||
- SIMD scorer only after decode timing is the measured bottleneck.
|
||||
- GPU scorer only after candidate/bin batches exceed CPU SIMD advantage.
|
||||
- Neural/acoustic-field models only after classical timing/calibration/source
|
||||
localization is coherent.
|
||||
- Dynamic 4DGS only after direct synchronized visual evidence exists in
|
||||
Fensalir.
|
||||
|
||||
|
|
@ -13,6 +13,32 @@ needs the reason a cut exists.
|
|||
| `architecture-rumination.md` | System ownership, current machine critique, coherent next cuts. |
|
||||
| `optimization-ledger.md` | Hot loop risks, low-level implementation options, micro-optimization candidates. |
|
||||
| `references.md` | Research links for chirplets, CSS/LoRa, GCC-PHAT, D3D12/FFT/GPU compute. |
|
||||
| `current-hotspot-audit.md` | Static audit of likely runtime allocation/scan/driver-pressure hotspots. |
|
||||
| `implementation-roadmap.md` | Phase plan, invariants, proofs, and benchmark harnesses. |
|
||||
| `questions-and-hypotheses.md` | Open questions and falsifiable hypotheses by domain. |
|
||||
| `reading-guide.md` | Task-oriented route through the study docs and samples. |
|
||||
| `data-dictionary.md` | Data object definitions for streams, samples, chirp anchors, clock fits, calibration paths, reports, and Fensalir constraints. |
|
||||
| `volumetric-audio-field.md` | Sound-field reconstruction decomposition and first practical audio target. |
|
||||
| `acoustic-field-models.md` | Field reconstruction models and the first honest acoustic source proof. |
|
||||
| `chirplet-transform-deep-dive.md` | Controlled chirp receiver versus generic chirplet transform. |
|
||||
| `decoder-architecture-options.md` | Active receiver implementation options and promotion criteria. |
|
||||
| `distributed-receiver-spec.md` | Raven/phone receiver shape for self-locating remote sensors using codebook and schedule state. |
|
||||
| `visual-fusion-4dgs-study.md` | Sensor fusion and realtime Gaussian splatting architecture study. |
|
||||
| `open-implementation-catalog.md` | Implementation families to study before inventing local machinery. |
|
||||
| `prior-research-synthesis.md` | Distillation of the older chirplet, adaptive sync, ambisonic, splatting, calibration, and visual archives. |
|
||||
| `research-claims-digest.md` | Claim-level digest of external literature and how it steers Mimir. |
|
||||
| `fensalir-integration-map.md` | Contract-level cut between Mimir runtime truth and Fensalir GPU/render ownership. |
|
||||
| `native-boundary-map.md` | Native/managed/Fensalir ownership boundaries for ASIO, cameras, reservoir, and GPU payloads. |
|
||||
| `option-matrix.md` | Implementation alternatives, failure modes, and deciding proofs for the major subsystems. |
|
||||
| `benchmark-plan.md` | Measurement plan for audio sync, acoustic field, camera capture, Fensalir lowering, and splat fusion. |
|
||||
| `calibration-session-spec.md` | Command/session shape for learning and validating real output/mic path models. |
|
||||
| `low-level-implementation-notes.md` | Concrete memory layout, SIMD/GPU, ASIO, camera, and instrumentation notes. |
|
||||
| `microsecond-sync-math.md` | Timing math for sample period, bandwidth, SNR, subsample delay, and acoustic distance. |
|
||||
| `state-machine-invariants.md` | Ownership limits for runtime, hub, buffers, decoders, calibration, DSP, capture, Fensalir, network, and OBS. |
|
||||
| `failure-mode-ledger.md` | Failure modes and coherent fixes across active witness, passive sync, actuator, camera, fusion, and network paths. |
|
||||
| `hot-path-pseudocode.md` | Code-shaped intended hot paths for streaming decode, calibration weighting, actuator, native rings, camera rings, and Fensalir lowering. |
|
||||
| `glossary.md` | Short definitions for the synchronization, acoustic, visual, and reservoir terms used across the study. |
|
||||
| `sample-code-index.json` | Machine-readable index for sample sketches. |
|
||||
| `samples/README.md` | Index of sample implementation sketches. |
|
||||
|
||||
## Sample Code
|
||||
|
|
@ -21,9 +47,17 @@ needs the reason a cut exists.
|
|||
| --- | --- | --- |
|
||||
| `samples/StreamingChirpBinDecoderSketch.cs` | `MimirChirpBinStreamingDecoder` or equivalent runtime class | What state removes full-window rescans? |
|
||||
| `samples/Avx2DechirpGoertzelSketch.cpp` | Native CPU scorer for chirp-bin candidates | What does SIMD dechirp/bin scoring look like? |
|
||||
| `samples/BatchedChirpScoreAbiSketch.h` | Native scorer ABI | What batch boundary avoids per-candidate managed/native overhead? |
|
||||
| `samples/ChirpBinScore.compute.hlsl` | Fensalir/D3D12 compute scorer | How would batched candidate/bin scoring map to GPU? |
|
||||
| `samples/FarrowFractionalDelaySketch.cpp` | Faust/native DSP actuator | What is the smallest fractional delay proof? |
|
||||
| `samples/SpscAudioBlockRingSketch.cpp` | Native capture worker handoff | What shape should low-jitter audio block transfer take? |
|
||||
| `samples/CalibrationWeightedChirpLikelihoodSketch.cs` | Physical-path active decoder | Where does calibration enter symbol likelihood? |
|
||||
| `samples/SroPllAsrcControllerSketch.cpp` | DSP actuator control | How does delay/SRO state become a resampler command? |
|
||||
| `samples/GaussianSplatClaimUpdate.compute.hlsl` | Fensalir visual fusion | What is the smallest live splat-claim compute shape? |
|
||||
| `samples/AquariumGpuSensorFrameBridgeSketch.cs` | Mimir-to-Fensalir visual lowering | How should runtime video observations become Fensalir sensor frames? |
|
||||
| `samples/AcousticConstraintLoweringSketch.cs` | Mimir-to-Fensalir acoustic lowering | How should decoded anchors/geometry become acoustic constraints? |
|
||||
| `samples/TemporalEvidenceCandidateSketch.cs` | Shared temporal evidence candidates | How can Mimir lower candidates without owning Fensalir tracks? |
|
||||
| `samples/SrpPhatGridSketch.cpp` | Acoustic source localization | What is the smallest TDOA/SRP-PHAT-style grid scoring proof? |
|
||||
|
||||
## Best Next Implementation Sequence
|
||||
|
||||
|
|
@ -55,6 +89,8 @@ needs the reason a cut exists.
|
|||
- group delay
|
||||
- LoRa CSS
|
||||
- native reservoir
|
||||
- Fensalir
|
||||
- Gaussian splatting
|
||||
- passive sync
|
||||
- Raven
|
||||
- response/confusion matrix
|
||||
|
|
@ -64,4 +100,3 @@ needs the reason a cut exists.
|
|||
- streaming decoder
|
||||
- volumetric audio
|
||||
- volumetric visual fusion
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,199 @@
|
|||
# Low-Level Implementation Notes
|
||||
|
||||
## Purpose
|
||||
|
||||
This file collects the implementation details that are easy to lose when the
|
||||
conversation stays at the architecture level. None of these notes are promises;
|
||||
they are candidates to benchmark.
|
||||
|
||||
## Audio Memory Layout
|
||||
|
||||
### Preferred Analysis Layout
|
||||
|
||||
For decoder kernels:
|
||||
|
||||
```text
|
||||
channel0 samples: [s0 s1 s2 ...]
|
||||
channel1 samples: [s0 s1 s2 ...]
|
||||
...
|
||||
```
|
||||
|
||||
Separate channel spans make per-channel scoring straightforward and avoid
|
||||
striding through interleaved audio during analysis. Interleaving is useful for
|
||||
device I/O and some DSP kernels; it is not automatically the right analysis
|
||||
shape.
|
||||
|
||||
### Preferred Scoring Layout
|
||||
|
||||
```text
|
||||
candidateStartSamples[]
|
||||
candidateEnergy[]
|
||||
symbolReliability[]
|
||||
binFrequencies[]
|
||||
binSin[]
|
||||
binCos[]
|
||||
```
|
||||
|
||||
Dense arrays let SIMD/native/GPU kernels process candidates without object
|
||||
chasing. The codebook trellis can stay managed/CPU because it is small and
|
||||
branchy.
|
||||
|
||||
## SIMD Chirp-Bin Scoring
|
||||
|
||||
Likely CPU kernel shape:
|
||||
|
||||
1. load candidate window samples in 8-float AVX2 chunks;
|
||||
2. multiply by dechirp oscillator/window;
|
||||
3. accumulate bin real/imag sums;
|
||||
4. compute magnitude or normalized energy;
|
||||
5. write top K or full dense bin vector.
|
||||
|
||||
Important:
|
||||
|
||||
- keep oscillator tables aligned;
|
||||
- keep candidate windows contiguous;
|
||||
- batch multiple candidates per interop call;
|
||||
- avoid denormals;
|
||||
- normalize outside the deepest loop where possible.
|
||||
|
||||
## FFT / Goertzel Decision
|
||||
|
||||
Use Goertzel/fixed-bin DFT when:
|
||||
|
||||
- symbol count is small;
|
||||
- bins are known;
|
||||
- candidate count is modest;
|
||||
- native dispatch overhead matters.
|
||||
|
||||
Use FFT/CZT when:
|
||||
|
||||
- many bins are needed;
|
||||
- frequency offset interpolation dominates;
|
||||
- candidate batches are large enough;
|
||||
- GPU/native FFT plans can be reused.
|
||||
|
||||
Do not choose FFT because it sounds more canonical. Canonical is not a profiler.
|
||||
|
||||
## GPU Chirp Scoring
|
||||
|
||||
The GPU scoring pass only makes sense if data is already resident or batches are
|
||||
large enough to pay dispatch overhead.
|
||||
|
||||
Good GPU candidates:
|
||||
|
||||
- many mic streams;
|
||||
- many remote receiver windows;
|
||||
- wide candidate sets during acquisition;
|
||||
- offline/artifact calibration sweeps.
|
||||
|
||||
Bad GPU candidates:
|
||||
|
||||
- one or two channels;
|
||||
- tiny candidate counts;
|
||||
- paths that require CPU readback before codebook solving every event.
|
||||
|
||||
## ASIO Capture Hot Path
|
||||
|
||||
Current observed risks:
|
||||
|
||||
- per-block `std::vector<float>` resize;
|
||||
- `std::deque` queue;
|
||||
- native-to-managed polling copies into `float[]`;
|
||||
- managed byte array allocation for sample payload;
|
||||
- `ConcurrentQueue.Count` pressure.
|
||||
|
||||
Preferred cut:
|
||||
|
||||
- native SPSC ring;
|
||||
- fixed block pool;
|
||||
- monotonic frame counters;
|
||||
- block descriptors with handle/index;
|
||||
- C# metadata only until DSP needs samples.
|
||||
|
||||
## Camera Capture Hot Path
|
||||
|
||||
Current probe strengths:
|
||||
|
||||
- KS async queueing is already close to the driver;
|
||||
- PS3 Eye raw path proves high fps;
|
||||
- JSON frame events make cadence visible.
|
||||
|
||||
Current probe risks:
|
||||
|
||||
- JSON must not become production payload;
|
||||
- CPU frame buffers are diagnostic until Fensalir import/upload path exists;
|
||||
- one-off command-line knobs must become configuration/stateful driver policy.
|
||||
|
||||
Preferred cut:
|
||||
|
||||
- one native capture worker per camera family;
|
||||
- fixed frame ring;
|
||||
- descriptor ABI;
|
||||
- optional shared GPU texture handle;
|
||||
- Fensalir owns texture import and GPU staging.
|
||||
|
||||
## Fensalir GPU Fusion
|
||||
|
||||
Fensalir contracts already distinguish:
|
||||
|
||||
- `GpuSensorFrame` for calibrated camera inputs and external textures;
|
||||
- `AcousticFieldFrame` for timing/room constraints;
|
||||
- `CalibrationEventFrame` for high-confidence event anchors;
|
||||
- `GpuFusionField` for point buffers/seeds;
|
||||
- `TemporalGaussianField` for stable gaussian observations.
|
||||
|
||||
Mimir should produce these rows. It should not pack shader structs directly.
|
||||
|
||||
## Gaussian Splat Rendering Levers
|
||||
|
||||
From modern 3DGS implementations:
|
||||
|
||||
- tile size matters;
|
||||
- sorting strategy matters;
|
||||
- cull before sort;
|
||||
- packed buffers matter;
|
||||
- anti-aliasing and opacity/radius thresholds matter;
|
||||
- view-consistency fixes cost real time;
|
||||
- depth/feature render modes are useful for diagnostics.
|
||||
|
||||
Mimir-specific cut:
|
||||
|
||||
- first update live claims;
|
||||
- then render simple claims;
|
||||
- only later optimize dynamic radiance fields.
|
||||
|
||||
## Benchmark Counters To Add
|
||||
|
||||
Decoder:
|
||||
|
||||
- samples consumed;
|
||||
- proposals created;
|
||||
- candidates scored;
|
||||
- anchors accepted/rejected;
|
||||
- elapsed scoring time;
|
||||
- allocations per analysis tick.
|
||||
|
||||
ASIO:
|
||||
|
||||
- callback count;
|
||||
- block count;
|
||||
- queue/ring high-water mark;
|
||||
- dropped blocks;
|
||||
- conversion time if measurable.
|
||||
|
||||
Camera:
|
||||
|
||||
- frame interval p50/p95/p99;
|
||||
- async queue depth;
|
||||
- dropped/late frames;
|
||||
- payload copy time;
|
||||
- upload/import time.
|
||||
|
||||
Fensalir:
|
||||
|
||||
- contract lowering time;
|
||||
- upload time;
|
||||
- GPU pass timings;
|
||||
- temporal field count;
|
||||
- acoustic constraint count.
|
||||
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
# Microsecond Sync Math
|
||||
|
||||
## Purpose
|
||||
|
||||
The project target is microsecond-class synchronization. This note keeps the
|
||||
math visible so we do not confuse sample-rate settings, transducer bandwidth,
|
||||
and actual timing certainty.
|
||||
|
||||
## Sample Periods
|
||||
|
||||
| Sample Rate | Sample Period |
|
||||
| --- | --- |
|
||||
| 48 kHz | 20.833 us |
|
||||
| 96 kHz | 10.417 us |
|
||||
| 192 kHz | 5.208 us |
|
||||
|
||||
Subsample estimation is mandatory for microsecond delay estimates at every
|
||||
practical audio sample rate. 192 kHz helps by reducing the interpolation gap and
|
||||
allowing higher witness frequencies, but it does not magically provide
|
||||
one-microsecond truth.
|
||||
|
||||
## What Enables Subsample Timing
|
||||
|
||||
- known emitted waveform;
|
||||
- high SNR;
|
||||
- broad enough bandwidth;
|
||||
- stable phase response;
|
||||
- clean enough direct path;
|
||||
- coherent clock model over multiple anchors;
|
||||
- interpolation or model fitting around correlation/likelihood peaks.
|
||||
|
||||
The active witness helps because its shape is known. Passive music can help when
|
||||
it has enough broadband content, but it is not guaranteed.
|
||||
|
||||
## Cramer-Rao Intuition
|
||||
|
||||
Delay estimation improves with:
|
||||
|
||||
- signal bandwidth;
|
||||
- SNR;
|
||||
- observation duration / repeated independent evidence.
|
||||
|
||||
It degrades with:
|
||||
|
||||
- narrowband signals;
|
||||
- reverberation;
|
||||
- unknown filtering;
|
||||
- low SNR;
|
||||
- nonlinear processing;
|
||||
- clock drift;
|
||||
- false symbol aliases.
|
||||
|
||||
This is why the chirp signal has a dual role: it spreads energy across time and
|
||||
frequency for timing, while also measuring which parts of the path survived.
|
||||
|
||||
## 192 kHz Is Useful When
|
||||
|
||||
- the interface can capture output and mics in one stable clock domain;
|
||||
- the monitors/mics preserve enough high-frequency energy;
|
||||
- phase/group delay can be calibrated;
|
||||
- the actuator can use fractional delay/SRO information;
|
||||
- the extra CPU/memory load stays within budget.
|
||||
|
||||
## 192 kHz Is Not Useful When
|
||||
|
||||
- the acoustic path low-passes the witness below the useful band;
|
||||
- the mic/driver applies hidden processing;
|
||||
- the decoder only reports integer samples;
|
||||
- the system still lacks a resampling/fractional-delay actuator;
|
||||
- the bottleneck is symbol confusion rather than sample quantization.
|
||||
|
||||
## Timing Evidence Hierarchy
|
||||
|
||||
1. Electrical loopback: best for emitted timeline and interface clock.
|
||||
2. Physical mic active witness: best for acoustic path timing/response.
|
||||
3. Passive program audio: useful continuous drift evidence when content allows.
|
||||
4. Network timestamps: metadata, not truth.
|
||||
|
||||
## Delay To Distance
|
||||
|
||||
At roughly 343 m/s:
|
||||
|
||||
- 1 us is about 0.343 mm of acoustic path length;
|
||||
- 10 us is about 3.43 mm;
|
||||
- 100 us is about 34.3 mm.
|
||||
|
||||
That scale is useful and humbling. A room reflection or mic position error can
|
||||
swamp the number we are trying to measure. The machine must keep uncertainty
|
||||
visible.
|
||||
|
||||
## Practical Acceptance
|
||||
|
||||
For the next physical proof, "microsecond sync" should mean:
|
||||
|
||||
- loopback pair recovers below printed microsecond precision;
|
||||
- physical mic reports fractional-sample delay with stable residuals;
|
||||
- repeated anchors show bounded SRO drift;
|
||||
- delay uncertainty is reported, not hidden;
|
||||
- actuator demonstrably reduces residual misalignment.
|
||||
|
||||
Without the actuator, the system is measuring microseconds, not delivering them.
|
||||
|
||||
127
research/perfect-machine-study-2026-05-23/native-boundary-map.md
Normal file
127
research/perfect-machine-study-2026-05-23/native-boundary-map.md
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
# Native Boundary Map
|
||||
|
||||
## Purpose
|
||||
|
||||
Mimir's managed runtime should own identity, policy, and orchestration. It
|
||||
should not sit inside the six-camera or 192 kHz audio hot loops doing avoidable
|
||||
copies, JSON parsing, or queue churn. This file marks the boundaries that should
|
||||
be native, managed, or Fensalir-owned.
|
||||
|
||||
## Current Native Pieces
|
||||
|
||||
### ASIO Capture
|
||||
|
||||
`native/asio_capture/mimir_asio_capture.cpp` currently:
|
||||
|
||||
- creates the Focusrite ASIO driver;
|
||||
- configures input buffers;
|
||||
- receives `bufferSwitch` / `bufferSwitchTimeInfo` callbacks;
|
||||
- converts device sample formats to mono float blocks;
|
||||
- queues `MimirAsioBlock` objects;
|
||||
- exposes a polling ABI to C#.
|
||||
|
||||
This proves in-process ASIO ingest. It is not the final zero-copy hot path:
|
||||
`std::deque`, `std::vector<float>`, per-block resize, and managed byte-array
|
||||
copies are all visible future costs.
|
||||
|
||||
### Camera Probes
|
||||
|
||||
The KS probe currently uses low-level Kernel Streaming / AVStream surfaces:
|
||||
|
||||
- property negotiation;
|
||||
- async `NtDeviceIoControlFile` reads;
|
||||
- queued read slots;
|
||||
- frame JSON emission for diagnostics.
|
||||
|
||||
The PS3 Eye probe uses the raw driver/library path and emits frame metadata. It
|
||||
proves throughput and topology, not final app ingestion.
|
||||
|
||||
### Reservoir
|
||||
|
||||
`native/reservoir` is the lower-level memory invariant: one rolling edge, typed
|
||||
views, no per-kind retention. This should become the shared payload/index owner
|
||||
when C# descriptors become too expensive for full payloads.
|
||||
|
||||
## Boundary Decisions
|
||||
|
||||
| Concern | Owner | Reason |
|
||||
| --- | --- | --- |
|
||||
| ASIO COM driver calls | native | COM/callback lifetime and sample-format conversion do not belong in C#. |
|
||||
| ASIO block timing/counters | native first, Mimir indexed | The callback sees the driver clock; Mimir maps it to canonical time. |
|
||||
| Audio payload memory | native/Faust | Program DSP should not copy through managed byte arrays. |
|
||||
| Audio sync reports | Mimir.Runtime | Reports are policy/belief, not raw DSP. |
|
||||
| Camera device control | native | Driver calls, async reads, libusb, and vendor knobs are hardware work. |
|
||||
| Camera payload memory | native/Fensalir | Pixels should land in native or GPU memory. |
|
||||
| GPU texture lifetime | Fensalir | The engine owns D3D12 resources and synchronization. |
|
||||
| Calibration JSON/artifacts | Mimir | Persistence and reproducibility are repo/runtime concerns. |
|
||||
|
||||
## Desired ASIO Evolution
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["ASIO callback"] --> B["native SPSC block ring"]
|
||||
B --> C["native reservoir audio view"]
|
||||
C --> D["Mimir.Runtime metadata/index"]
|
||||
C --> E["Faust/native DSP"]
|
||||
D --> F["sync analyzer state"]
|
||||
F --> E
|
||||
```
|
||||
|
||||
Immediate cut:
|
||||
|
||||
- replace `std::deque` with a bounded SPSC ring;
|
||||
- preallocate per-channel block buffers;
|
||||
- copy/convert once from ASIO format into the block ring;
|
||||
- expose block handles or spans instead of repacking to managed arrays.
|
||||
|
||||
Final cut:
|
||||
|
||||
- keep raw/current device format where possible;
|
||||
- run hot alignment and filtering in native/Faust;
|
||||
- hand Mimir descriptors and diagnostics rather than payload copies.
|
||||
|
||||
## Desired Camera Evolution
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["driver read completion"] --> B["native frame ring"]
|
||||
B --> C["timestamp/sequence descriptor"]
|
||||
B --> D["CPU pinned or GPU shared payload"]
|
||||
C --> E["Mimir.Runtime rolling buffer"]
|
||||
D --> F["Fensalir texture import/upload"]
|
||||
```
|
||||
|
||||
Immediate cut:
|
||||
|
||||
- one native worker per camera family;
|
||||
- frame descriptor ABI with source id, sequence, timestamp, dimensions, format,
|
||||
stride, byte length, and payload handle;
|
||||
- no JSON in the production app path.
|
||||
|
||||
Final cut:
|
||||
|
||||
- D3D12-compatible shared resources where the driver path can support them;
|
||||
- CPU staging only where direct GPU import is not practical;
|
||||
- feature extraction as early GPU compute, not managed pixel processing.
|
||||
|
||||
## External Guidance Anchors
|
||||
|
||||
- Microsoft Kernel Streaming docs confirm KS/AVStream is the low-level Windows
|
||||
stream surface beneath higher media frameworks.
|
||||
- `IDXGIResource1::CreateSharedHandle` and D3D11/12 shared NT handles are the
|
||||
relevant family for resource sharing once capture can land in shareable
|
||||
textures.
|
||||
- ASIO `bufferSwitchTimeInfo` / sample position is the right place to preserve
|
||||
driver-clock timing, not a managed polling timestamp after the fact.
|
||||
- Focusrite channel-order docs matter because loopback availability and channel
|
||||
layout can change with sample rate and model.
|
||||
|
||||
## Watch List
|
||||
|
||||
- Per-block `new float[]` / `byte[]` in managed ingest.
|
||||
- `ConcurrentQueue.Count` in hot loops.
|
||||
- `std::deque` under callback pressure.
|
||||
- One process per camera.
|
||||
- Any "temporary" JSON bridge that survives once payloads matter.
|
||||
- GPU resource ownership creeping into Mimir instead of Fensalir.
|
||||
|
||||
|
|
@ -0,0 +1,99 @@
|
|||
# Open Implementation Catalog
|
||||
|
||||
This catalog lists implementation families worth stealing shape from. Do not
|
||||
vendor code casually. The point is to understand mature cuts.
|
||||
|
||||
## Chirp / CSS / LoRa
|
||||
|
||||
What to look for:
|
||||
|
||||
- dechirp implementation;
|
||||
- symbol timing synchronization;
|
||||
- FFT/bin selection;
|
||||
- CFO/bin-shift correction;
|
||||
- preamble/sync word handling;
|
||||
- soft decision metrics.
|
||||
|
||||
Candidate references:
|
||||
|
||||
- OpenLoRa demodulator work.
|
||||
- SDR LoRa implementations.
|
||||
- GNU Radio LoRa/CSS blocks.
|
||||
- Audio "data over sound" projects for acoustic channel constraints.
|
||||
|
||||
Mimir translation:
|
||||
|
||||
- Preamble becomes deterministic timeline/codebook schedule.
|
||||
- CFO becomes acoustic bin shift / sample-rate mismatch.
|
||||
- Channel response becomes output/mic path calibration.
|
||||
|
||||
## Audio Delay / Beamforming
|
||||
|
||||
What to look for:
|
||||
|
||||
- GCC-PHAT implementations;
|
||||
- SRP-PHAT acceleration;
|
||||
- microphone array calibration;
|
||||
- fractional delay filters;
|
||||
- asynchronous sample-rate conversion.
|
||||
|
||||
Candidate references:
|
||||
|
||||
- Pyroomacoustics for algorithm shape.
|
||||
- WebRTC audio processing for pragmatic realtime constraints.
|
||||
- JACK/PipeWire/ASIO style callback boundaries.
|
||||
- Faust examples for DSP graph ownership.
|
||||
|
||||
Mimir translation:
|
||||
|
||||
- Keep algorithm shape, not necessarily library dependency.
|
||||
- Faust/native DSP is the actuator home.
|
||||
|
||||
## Gaussian Splatting
|
||||
|
||||
What to look for:
|
||||
|
||||
- 3D Gaussian Splatting CUDA rasterizer.
|
||||
- Dynamic/4D Gaussian splatting repos.
|
||||
- Differentiable Gaussian rasterization data layouts.
|
||||
- Real-time viewer/update pipelines.
|
||||
|
||||
Mimir translation:
|
||||
|
||||
- Fensalir owns render/update.
|
||||
- Native reservoir owns time-indexed claims.
|
||||
- Direct camera drivers feed evidence; do not train/reconstruct from disk
|
||||
batches in the live path.
|
||||
|
||||
## Native Capture
|
||||
|
||||
What to look for:
|
||||
|
||||
- ASIO callback isolation.
|
||||
- Kernel Streaming async read queues.
|
||||
- libusb/WinUSB transfer queues.
|
||||
- D3D12 shared texture/resource handoff.
|
||||
- lock-free SPSC rings.
|
||||
|
||||
Mimir translation:
|
||||
|
||||
- One capture worker owns one device.
|
||||
- Payload handles cross subsystem boundaries.
|
||||
- Runtime indexes metadata and timing.
|
||||
|
||||
## SIMD/GPU Compute
|
||||
|
||||
What to look for:
|
||||
|
||||
- AVX2/FMA complex dot products.
|
||||
- batched FFT plan reuse.
|
||||
- HLSL wave reductions.
|
||||
- CUDA/cuFFT callback fusion.
|
||||
- cache-aware ring buffers.
|
||||
|
||||
Mimir translation:
|
||||
|
||||
- SIMD first for fixed small bin banks.
|
||||
- GPU when candidate/bin/channel batches are large enough to amortize dispatch
|
||||
and memory movement.
|
||||
|
||||
72
research/perfect-machine-study-2026-05-23/option-matrix.md
Normal file
72
research/perfect-machine-study-2026-05-23/option-matrix.md
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
# Implementation Option Matrix
|
||||
|
||||
## Purpose
|
||||
|
||||
This matrix keeps future cuts from turning into flavor arguments. Each row names
|
||||
the invariant a subsystem must protect, the conservative implementation, the
|
||||
aggressive implementation, and the proof that should decide between them.
|
||||
|
||||
## Audio Sync Receiver
|
||||
|
||||
| Option | Protects | Upside | Failure Mode | Proof |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Current window decoder | Correctness oracle | Easy to inspect; already wired | Repeated full-window work | Keep as reference until streaming matches |
|
||||
| Streaming scalar decoder | One-pass receiver state | Small, portable, phone-shaped | Weak proposals can miss path-damaged chirps | Synthetic/artifact parity with fewer scans |
|
||||
| AVX2 scorer | CPU throughput | Fast fixed-bin scoring | Interop and memory traffic erase wins | Batched benchmark beats scalar end-to-end |
|
||||
| GPU scorer | Large batch throughput | Many channels/candidates at once | Dispatch/readback overhead | Only promote when batches are large and GPU stays resident |
|
||||
| Full chirplet transform | General signal analysis | Handles unknown chirps | Too broad for controlled codebook | Only if controlled receiver fails after calibration |
|
||||
|
||||
## Active Witness Shape
|
||||
|
||||
| Option | Protects | Upside | Failure Mode | Proof |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 32 fixed bins | Short triplets | Dense code space | Dead/aliased bands in room | Works only where response is good |
|
||||
| Adaptive reliable bins + higher order | Physical robustness | Avoids dead bands | Longer lock window | Meatspace anchors increase after calibration |
|
||||
| Watermarked hybrid | Human tolerance | Less annoying | Masked too well to decode | Passive+active confidence improves without audible abuse |
|
||||
| Ultrasonic-only | Low distraction | Clean separation from music | Monitors/mics may not pass band | Measured SNR and response beyond hearing |
|
||||
|
||||
## Audio Actuator
|
||||
|
||||
| Option | Protects | Upside | Failure Mode | Proof |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Integer delay | Coarse alignment | Simple | Not microsecond/subsample | Useful only as baseline |
|
||||
| Farrow fractional delay | Subsample delay | Cheap, time-varying | Magnitude/phase ripple | Delay residual falls on artifact and live captures |
|
||||
| Variable ASRC | Long-term SRO | Holds sync over time | More complex buffering | Drift stays bounded over 20+ minutes |
|
||||
| Frequency-domain phase correction | STFT coherence | Good for analysis frames | Latency/window artifacts | Program material remains clean |
|
||||
|
||||
## Audio Field Reconstruction
|
||||
|
||||
| Option | Protects | Upside | Failure Mode | Proof |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Aligned stems | OBS usefulness | Immediate production value | Not volumetric | Stems stay synchronized and mixable |
|
||||
| Source-based spatial bus | Coherent performance mix | Honest with sparse mics | Depends on source metadata | Stable source positions/pans |
|
||||
| Ambisonic fit | Compact spatial field | Standard ecosystem | Sparse irregular mics unstable | Calibration proves usable order/band |
|
||||
| Sparse equivalent sources | Physical field estimate | Handles arbitrary geometry | Ill-conditioned inverse problem | Known source reconstructed with residuals |
|
||||
| Hybrid evidence field | General Mimir shape | Combines audio/visual constraints | Complexity creep | Fensalir reservoir stabilizes claims |
|
||||
|
||||
## Camera Ingest
|
||||
|
||||
| Option | Protects | Upside | Failure Mode | Proof |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| JSON probes | Research observability | Fast to inspect | Not a hot path | Keep diagnostic only |
|
||||
| Managed wrapper over native reads | App integration | Easier first driver | Copies/GC pressure | One camera sustained cadence |
|
||||
| Native SPSC frame rings | Throughput | Predictable hot loop | ABI/lifetime complexity | All local cameras sustained |
|
||||
| GPU shared textures | Copy avoidance | Best final path | Driver/API friction | Fensalir imports and renders frames |
|
||||
|
||||
## Visual Fusion
|
||||
|
||||
| Option | Protects | Upside | Failure Mode | Proof |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Feature tracks | Sparse evidence | Debuggable | Not visually rich | Tracks align across cameras |
|
||||
| Point/splat claims | Live volumetric surface | GPU-friendly | Needs calibration | Stable live render from observations |
|
||||
| Offline 3DGS-style optimization | Visual quality | Mature papers/code | Wrong latency/ownership | Use only for reference or static assets |
|
||||
| Dynamic/4D splats | Live field | Target shape | Research complexity | Start after direct observations work |
|
||||
|
||||
## Fensalir Boundary
|
||||
|
||||
| Option | Protects | Upside | Failure Mode | Proof |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Mimir builds render state | Quick local control | Tempting | Duplicates engine | Reject |
|
||||
| Mimir lowers contract frames | Ownership clarity | Clean boundary | Needs stable contract mapping | Allocation-free lowering benchmark |
|
||||
| Fensalir owns temporal evidence | Single field authority | Reuses engine work | Requires good candidates | Acoustic/visual claims visible in engine |
|
||||
|
||||
|
|
@ -0,0 +1,142 @@
|
|||
# Prior Research Synthesis
|
||||
|
||||
## Purpose
|
||||
|
||||
Mimir already carries useful research archives. This note distills the pieces
|
||||
that still steer the live machine so the next pass does not reread every mirror
|
||||
before touching code.
|
||||
|
||||
## Chirplet Sync Decoder Archive
|
||||
|
||||
Location: `research/chirplet-sync-decoder/`
|
||||
|
||||
Durable lesson:
|
||||
|
||||
- Generic chirplet matching is valid signal analysis, but the controlled Mimir
|
||||
beacon should be receiver-cheap by construction.
|
||||
- Dechirp plus FFT/Goertzel is the important communications trick.
|
||||
- Symbol identity should be built from common chirp duration/slope/window plus
|
||||
decodable frequency bin; rhythm and gaps are additional evidence.
|
||||
|
||||
Impact on current work:
|
||||
|
||||
- `MimirChirpBinTimeline` is directionally right.
|
||||
- The next work is streaming state and calibrated likelihood, not resurrecting
|
||||
dense matched-filter banks.
|
||||
|
||||
## Live Adaptive Sync Archive
|
||||
|
||||
Location: `research/live-adaptive-sync/`
|
||||
|
||||
Durable lesson:
|
||||
|
||||
- Initial delay and sample-rate offset are separate loops.
|
||||
- SRO is inevitable across independent clocks and must be estimated/compensated.
|
||||
- Resampling/fractional delay is the actuator when hardware clocks cannot be
|
||||
controlled.
|
||||
- GCC-PHAT/TDOA is evidence for relative arrival and localization, not a
|
||||
canonical timeline by itself.
|
||||
|
||||
Impact on current work:
|
||||
|
||||
- `MimirAudioSynchronizationState` needs to drive a real actuator.
|
||||
- Delay/SRO estimates should be control inputs to Faust/native DSP.
|
||||
- Passive mode remains valuable, but it does not replace active codebook time.
|
||||
|
||||
## Ambisonic / AquaSynth Archive
|
||||
|
||||
Location: `research/ambisonic-aquasynth/`
|
||||
|
||||
Durable lesson:
|
||||
|
||||
- Arbitrary microphones are not automatically an ambisonic mic.
|
||||
- Ambisonic output needs declared order, channel ordering, normalization, sample
|
||||
rate, and target.
|
||||
- Source-based spatialization is the honest first path; arbitrary-array field
|
||||
reconstruction requires calibration and regularization.
|
||||
|
||||
Impact on current work:
|
||||
|
||||
- Mimir should not promise high-order field truth until mic geometry and path
|
||||
calibration prove it.
|
||||
- Faust `hoa.lib`, IEM, SPARTA, and SAF are verification/reference tools.
|
||||
- The initial output can be aligned stems plus a lower-order spatial bed.
|
||||
|
||||
## GPU Fusion / Splatting Archive
|
||||
|
||||
Location: `research/gpu-fusion-splatting/`
|
||||
|
||||
Durable lesson:
|
||||
|
||||
- Gaussian splatting is useful for packed GPU representations, tile/bin/sort
|
||||
renderers, anisotropic radiance primitives, and dynamic field ideas.
|
||||
- Mimir's live problem is not offline static-scene 3DGS training.
|
||||
- CPU readback of dense visual state is poison for the hot path.
|
||||
|
||||
Impact on current work:
|
||||
|
||||
- Fensalir owns temporal Gaussian/evidence fields.
|
||||
- Mimir should feed synchronized observations and constraints.
|
||||
- The first visual fusion proof should be a small live observation-to-splat
|
||||
update, not full 4D training.
|
||||
|
||||
## Playback / Feedback Calibration Archives
|
||||
|
||||
Locations:
|
||||
|
||||
- `research/playback-calibration/`
|
||||
- `research/feedback-calibration/`
|
||||
|
||||
Durable lesson:
|
||||
|
||||
- Room/speaker/mic paths can be measured as impulse or transfer functions.
|
||||
- Online acoustic system identification and adaptive echo cancellation literature
|
||||
is relevant because Mimir is emitting and listening in the same room.
|
||||
- Magnitude-only response is insufficient; phase/group delay are part of the
|
||||
timing machine.
|
||||
|
||||
Impact on current work:
|
||||
|
||||
- The chirp-bin calibration model should keep becoming a path transfer model.
|
||||
- Active witness design should support both timeline decoding and transfer
|
||||
function estimation.
|
||||
- Hybrid watermarking should learn from echo-cancellation and system-identification
|
||||
practice rather than just hiding chirps under music.
|
||||
|
||||
## Visual Spatial Map Archive
|
||||
|
||||
Location: `research/visual-spatial-map/`
|
||||
|
||||
Durable lesson:
|
||||
|
||||
- Direct device paths and camera-specific quirks matter. Framework convenience
|
||||
layers hide timing, buffering, exposure, and bandwidth behavior.
|
||||
- Leap, Kiyos, and PS3 Eyes have different roles: timing/IR, RGB ground truth,
|
||||
and high-rate tracking.
|
||||
- Multi-camera calibration is a geometry problem before it is a rendering
|
||||
problem.
|
||||
|
||||
Impact on current work:
|
||||
|
||||
- `IMimirVideoCaptureDriver` is the correct production seam.
|
||||
- Native probes should be promoted into workers only after they preserve
|
||||
timestamps, payload handles, and cadence.
|
||||
- Fensalir should consume calibrated camera observations, not raw app-specific
|
||||
UI experiments.
|
||||
|
||||
## Repository-Level Conclusion
|
||||
|
||||
The archives agree on one architecture:
|
||||
|
||||
```text
|
||||
calibrated sensors -> rolling evidence window -> canonical time -> native/GPU/DSP lowering -> program outputs
|
||||
```
|
||||
|
||||
They do not support:
|
||||
|
||||
- framework-mediated capture in the six-camera hot loop;
|
||||
- generic chirplet transforms as the default controlled-beacon receiver;
|
||||
- arbitrary ambisonic claims from unsynchronized heterogeneous mics;
|
||||
- CPU-owned dense visual fusion;
|
||||
- OBS as synchronization authority.
|
||||
|
||||
|
|
@ -0,0 +1,99 @@
|
|||
# Questions And Hypotheses
|
||||
|
||||
## Audio
|
||||
|
||||
### Q: Is the current physical failure mostly SNR, frequency response, or timing ambiguity?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- It is primarily response/confusion plus weak direct-path SNR, not a fundamental
|
||||
codebook failure.
|
||||
|
||||
Test:
|
||||
|
||||
- Compare calibration-weighted likelihood on stored hot mic captures.
|
||||
|
||||
### Q: Is 192 kHz worth keeping for active sync?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- Yes for fractional timing, loopback, and high-frequency option space; no as a
|
||||
magic acoustic fix when speakers/mics do not preserve ultrasonic energy.
|
||||
|
||||
Test:
|
||||
|
||||
- Run same reliable-bin codebook at 48/96/192 kHz through physical path and
|
||||
compare anchor residuals, not just sample granularity.
|
||||
|
||||
### Q: Should chirp-only and hybrid share one decoder?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- Yes. Mode changes emission policy and confidence gating, not decode
|
||||
semantics.
|
||||
|
||||
### Q: Can any three chirps always locate timeline position?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- Only if they are correctly classified inside the active codebook and the
|
||||
receiver keeps enough timing/frequency spacing evidence. In meatspace,
|
||||
"any three heard chirps" means "any three reliable decoded symbols after path
|
||||
calibration," not raw peaks.
|
||||
|
||||
## Visual
|
||||
|
||||
### Q: Should Leap or PS3 Eye be first direct runtime driver?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- Leap is semantically highest value as timing/depth/near-field witness; PS3 Eye
|
||||
may be lower-risk for proving direct high-rate runtime plumbing.
|
||||
|
||||
Test:
|
||||
|
||||
- Build the smaller driver first if it proves the seam faster, but do not lose
|
||||
the Leap timing-camera goal.
|
||||
|
||||
### Q: Is realtime Gaussian splatting the first visual fusion target?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- No. The first target is GPU feature/claim publication from synchronized
|
||||
frames. Splatting is the render/update representation after that.
|
||||
|
||||
## Distributed
|
||||
|
||||
### Q: Should Raven send raw audio/video or decoded observations?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- Both may be needed, but timing authority should come from locally decoded
|
||||
codebook/schedule observations where possible. Raw streams are evidence; local
|
||||
decoded anchors are stronger over variable network latency.
|
||||
|
||||
### Q: Can a phone know canonical time from codebook alone?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- Yes if it has schedule epoch, symbol plan, sample-rate estimate, and enough
|
||||
reliable decoded anchors. It should report confidence and path model, not just
|
||||
"time."
|
||||
|
||||
## Implementation
|
||||
|
||||
### Q: Is the native reservoir ready to replace C# buffers?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- Conceptually yes, operationally not yet. First mirror C# buffer proof through
|
||||
reservoir status/views before making it the only path.
|
||||
|
||||
### Q: Where does micro-optimization start?
|
||||
|
||||
Hypothesis:
|
||||
|
||||
- Not with HLSL. First benchmark current active decode; if scalar bin scoring
|
||||
dominates, AVX2/FMA scorer is likely the best first optimization. GPU wins
|
||||
only when batches are large.
|
||||
|
||||
132
research/perfect-machine-study-2026-05-23/reading-guide.md
Normal file
132
research/perfect-machine-study-2026-05-23/reading-guide.md
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
# Reading Guide
|
||||
|
||||
## Purpose
|
||||
|
||||
This directory is now large enough to need its own map. Use this guide when
|
||||
entering a future implementation pass cold.
|
||||
|
||||
## If You Are Building The Active Decoder
|
||||
|
||||
Read in order:
|
||||
|
||||
1. `chirplet-transform-deep-dive.md`
|
||||
2. `decoder-architecture-options.md`
|
||||
3. `microsecond-sync-math.md`
|
||||
4. `current-hotspot-audit.md`
|
||||
5. `optimization-ledger.md`
|
||||
6. `low-level-implementation-notes.md`
|
||||
7. `benchmark-plan.md`
|
||||
6. sample sketches:
|
||||
- `samples/StreamingChirpBinDecoderSketch.cs`
|
||||
- `samples/BatchedChirpScoreAbiSketch.h`
|
||||
- `samples/CalibrationWeightedChirpLikelihoodSketch.cs`
|
||||
- `samples/Avx2DechirpGoertzelSketch.cpp`
|
||||
- `samples/ChirpBinScore.compute.hlsl`
|
||||
|
||||
Decision:
|
||||
|
||||
- implement streaming state first;
|
||||
- preserve the existing window decoder as oracle;
|
||||
- promote native/GPU scoring only after benchmarks.
|
||||
|
||||
## If You Are Building Meatspace Calibration
|
||||
|
||||
Read in order:
|
||||
|
||||
1. `volumetric-audio-field.md`
|
||||
2. `acoustic-field-models.md`
|
||||
3. `calibration-session-spec.md`
|
||||
4. `prior-research-synthesis.md`
|
||||
5. `benchmark-plan.md`
|
||||
6. `references.md`
|
||||
|
||||
Decision:
|
||||
|
||||
- learn usable bands per path;
|
||||
- shape symbol likelihood before codebook solving;
|
||||
- correct magnitude and phase/group delay;
|
||||
- adapt the emitted alphabet rather than filtering failure afterward.
|
||||
|
||||
## If You Are Building The Audio Actuator
|
||||
|
||||
Read in order:
|
||||
|
||||
1. `prior-research-synthesis.md`
|
||||
2. `optimization-ledger.md`
|
||||
3. `benchmark-plan.md`
|
||||
4. sample sketches:
|
||||
- `samples/FarrowFractionalDelaySketch.cpp`
|
||||
- `samples/SroPllAsrcControllerSketch.cpp`
|
||||
|
||||
Decision:
|
||||
|
||||
- delay and SRO are separate loops;
|
||||
- the runtime estimates state;
|
||||
- Faust/native DSP moves samples.
|
||||
|
||||
## If You Are Building Camera Ingest
|
||||
|
||||
Read in order:
|
||||
|
||||
1. `native-boundary-map.md`
|
||||
2. `visual-fusion-4dgs-study.md`
|
||||
3. `low-level-implementation-notes.md`
|
||||
4. `current-hotspot-audit.md`
|
||||
5. `benchmark-plan.md`
|
||||
6. `docs/native-capture-cadence.md`
|
||||
7. `docs/native-rebuild-plan.md`
|
||||
|
||||
Decision:
|
||||
|
||||
- JSON probes are diagnostics;
|
||||
- production capture is native workers plus payload handles;
|
||||
- Fensalir owns GPU import and resource lifetime.
|
||||
|
||||
## If You Are Building Fensalir Integration
|
||||
|
||||
Read in order:
|
||||
|
||||
1. `fensalir-integration-map.md`
|
||||
2. `native-boundary-map.md`
|
||||
3. `visual-fusion-4dgs-study.md`
|
||||
4. sample sketches:
|
||||
- `samples/AquariumGpuSensorFrameBridgeSketch.cs`
|
||||
- `samples/AcousticConstraintLoweringSketch.cs`
|
||||
- `samples/TemporalEvidenceCandidateSketch.cs`
|
||||
|
||||
Decision:
|
||||
|
||||
- Mimir lowers observations/constraints;
|
||||
- Fensalir owns temporal evidence and render state;
|
||||
- the render tick must not run capture or analysis.
|
||||
|
||||
## If You Are Looking For The Next Cut
|
||||
|
||||
Read in order:
|
||||
|
||||
1. `implementation-roadmap.md`
|
||||
2. `option-matrix.md`
|
||||
3. `research-claims-digest.md`
|
||||
4. `questions-and-hypotheses.md`
|
||||
5. `distributed-receiver-spec.md`
|
||||
6. `current-hotspot-audit.md`
|
||||
|
||||
Recommended next code cut:
|
||||
|
||||
- streaming active decoder state with calibration-weighted likelihood;
|
||||
- benchmark it against the current window decoder;
|
||||
- then wire the actuator.
|
||||
|
||||
## If You Are Updating Public Docs
|
||||
|
||||
Read:
|
||||
|
||||
1. `docs/perfect-machine-domain-index.md`
|
||||
2. `docs/code-algorithm-map.md`
|
||||
3. `notes/current-system-map.md`
|
||||
4. this directory's `index.md`
|
||||
|
||||
Rule:
|
||||
|
||||
- public docs describe the live machine and current constraints;
|
||||
- research docs may hold scars, rejected paths, and implementation options.
|
||||
|
|
@ -9,6 +9,10 @@
|
|||
https://www.sciencedirect.com/science/article/pii/S0888327015003994
|
||||
- High-resolution chirplet transform parameter analysis:
|
||||
https://arxiv.org/abs/2108.00572
|
||||
- Fast Chirplet Transform with FPGA-based implementation:
|
||||
https://www.ece.iit.edu/~eoruklu/IIT/Publications_files/04639630.pdf
|
||||
- Speed-optimized fast chirplet implementation:
|
||||
https://ecasp.ece.iit.edu/publications/2012-present/2023-13.pdf
|
||||
|
||||
## Chirp Spread Spectrum / LoRa Receiver Shape
|
||||
|
||||
|
|
@ -42,6 +46,10 @@
|
|||
|
||||
- HLSL Shader Model 6 wave intrinsics:
|
||||
https://learn.microsoft.com/windows/win32/direct3dhlsl/hlsl-shader-model-6-0-features-for-direct3d-12
|
||||
- Microsoft Kernel Streaming / AVStream overview:
|
||||
https://learn.microsoft.com/en-gb/windows-hardware/drivers/stream/kernel-streaming
|
||||
- DXGI shared resource handle creation:
|
||||
https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_2/nf-dxgi1_2-idxgiresource1-createsharedhandle
|
||||
- DirectX Shader Compiler wave intrinsics notes:
|
||||
https://github.com/microsoft/directxshadercompiler/wiki/wave-intrinsics
|
||||
- cuFFT callback documentation:
|
||||
|
|
@ -55,6 +63,82 @@
|
|||
- Chirp Z-transform theory:
|
||||
https://pyffs.readthedocs.io/en/stable/theory/CZT.html
|
||||
|
||||
## Volumetric Audio / Room Acoustics / Actuation
|
||||
|
||||
- Pyroomacoustics beamforming module:
|
||||
https://pyroomacoustics.readthedocs.io/en/pypi-release/pyroomacoustics.beamforming.html
|
||||
- Pyroomacoustics repository and room/array algorithm examples:
|
||||
https://github.com/LCAV/pyroomacoustics
|
||||
- Pyroomacoustics paper:
|
||||
https://arxiv.org/abs/1710.04196
|
||||
- ManyEars sound source localization, tracking, and separation:
|
||||
https://github.com/introlab/manyears
|
||||
- Spatial Audio Framework, open C/C++ spatial-audio toolkit:
|
||||
https://github.com/leomccormack/Spatial_Audio_Framework
|
||||
- Sound field reconstruction in reverberant environments with rigid spherical
|
||||
microphone arrays:
|
||||
https://journals.sagepub.com/doi/10.1177/14613484261447569
|
||||
- Sparse Reconstruction of Sound Field Using Bayesian Compressive Sensing and
|
||||
Equivalent Source Method:
|
||||
https://pmc.ncbi.nlm.nih.gov/articles/PMC10301025/
|
||||
- Near-field acoustic holography with compressive sensing:
|
||||
https://docs.lib.purdue.edu/herrick/208/
|
||||
- Ambisonic encoding from equatorial microphone arrays:
|
||||
https://arxiv.org/abs/2211.00584
|
||||
- Spheroidal ambisonics:
|
||||
https://arxiv.org/abs/2103.05719
|
||||
- Real-time spherical array renderer:
|
||||
https://research.chalmers.se/publication/509494
|
||||
- Farrow fractional delay overview:
|
||||
https://engee.com/helpcenter/stable/en/interactive-scripts/dsp/fractional_delay.html
|
||||
- AMD Vitis fractional-delay Farrow filter note:
|
||||
https://docs.amd.com/r/2024.1-English/Vitis-Tutorials-AI-Engine-Development/Fractional-Delay-Farrow-Filter
|
||||
- Faust delay/fractional delay reference:
|
||||
https://ringbuffer.org/faust/faust_basics/faust-delay/
|
||||
- MATLAB Farrow fractional delay example:
|
||||
https://www.mathworks.com/help/dsp/ug/fractional-delay-filters-using-farrow-structures.html
|
||||
- Joint SFO estimation and compensation based on Farrow structure:
|
||||
https://arxiv.org/abs/2503.07577
|
||||
- Joint SFO estimation and compensation algorithms based on Farrow structure:
|
||||
https://arxiv.org/abs/2603.00627
|
||||
|
||||
## Realtime Gaussian Splatting / Dynamic Visual Fields
|
||||
|
||||
- Official 3D Gaussian Splatting implementation:
|
||||
https://github.com/graphdeco-inria/gaussian-splatting
|
||||
- 3D Gaussian Splatting paper:
|
||||
https://arxiv.org/abs/2308.04079
|
||||
- 4D Gaussian Splatting implementation:
|
||||
https://github.com/fudan-zvg/4d-gaussian-splatting
|
||||
- 4D Gaussian Splatting paper:
|
||||
https://openaccess.thecvf.com/content/CVPR2024/papers/Wu_4D_Gaussian_Splatting_for_Real-Time_Dynamic_Scene_Rendering_CVPR_2024_paper.pdf
|
||||
- Spacetime Gaussian Feature Splatting implementation:
|
||||
https://github.com/oppo-us-research/SpacetimeGaussians
|
||||
- Spacetime Gaussian Feature Splatting paper:
|
||||
https://arxiv.org/abs/2312.16812
|
||||
- PlayCanvas engine, an open WebGL/WebGPU engine with Gaussian splatting support:
|
||||
https://github.com/playcanvas/engine
|
||||
- PlayCanvas generic Gaussian splat processing docs:
|
||||
https://developer.playcanvas.com/user-manual/gaussian-splatting/building/unified-rendering/splat-processing/
|
||||
- NVIDIA vk_gaussian_splatting technical blog:
|
||||
https://developer.nvidia.com/blog/real-time-gpu-accelerated-gaussian-splatting-with-nvidia-designworks-sample-vk_gaussian_splatting/
|
||||
- gsplat documentation:
|
||||
https://docs.gsplat.studio/main/
|
||||
- gsplat rasterization API:
|
||||
https://docs.gsplat.studio/main/apis/rasterization.html
|
||||
- gsplat paper:
|
||||
https://arxiv.org/abs/2409.06765
|
||||
- StopThePop sorted Gaussian splatting implementation:
|
||||
https://github.com/r4dl/StopThePop
|
||||
- FlashGS paper:
|
||||
https://arxiv.org/abs/2408.07967
|
||||
- Gaussian-LIC, Gaussian splatting plus LiDAR-inertial-camera SLAM:
|
||||
https://arxiv.org/abs/2404.06926
|
||||
- RMGS-SLAM, real-time multi-sensor Gaussian splatting SLAM:
|
||||
https://arxiv.org/abs/2604.12942
|
||||
- Multi-Calib, online multi-sensor calibration direction:
|
||||
https://pmc.ncbi.nlm.nih.gov/articles/PMC12693731/
|
||||
|
||||
## How These References Steer Mimir
|
||||
|
||||
- Dechirp-plus-bin scoring is the right hot active receiver shape for controlled
|
||||
|
|
@ -67,4 +151,6 @@
|
|||
a small fixed bin bank, SIMD Goertzel/dechirp may beat FFT overhead.
|
||||
- If a transform path moves to GPU, fuse dechirp/window/preprocess with transform
|
||||
load/store where possible to avoid memory bandwidth waste.
|
||||
|
||||
- Visual fusion should borrow Gaussian splatting data/layout ideas, but the live
|
||||
Mimir problem is online synchronized evidence update, not offline novel-view
|
||||
training from a static capture folder.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,168 @@
|
|||
# Research Claims Digest
|
||||
|
||||
## Purpose
|
||||
|
||||
This is the claim-level digest from the research pass. It records what the
|
||||
external literature and open implementations change about Mimir's design.
|
||||
|
||||
## Chirplets / CSS / Receiver Design
|
||||
|
||||
Claim:
|
||||
|
||||
- A generic chirplet transform is the broad signal-analysis tool.
|
||||
- A controlled chirp communication receiver should usually exploit waveform
|
||||
design: dechirp, bin score, then decode.
|
||||
|
||||
Mimir consequence:
|
||||
|
||||
- The hot receiver should remain constrained by the known codebook and schedule.
|
||||
- Full chirplet transforms are reference tools, not first-line runtime code.
|
||||
- Symbol design should serve demodulation.
|
||||
|
||||
References:
|
||||
|
||||
- Steve Mann chirplet transform.
|
||||
- Fast chirplet transform / FPGA implementation work.
|
||||
- LoRa/CSS demodulator literature and OpenLoRa.
|
||||
|
||||
## Passive Delay / Distributed Microphones
|
||||
|
||||
Claim:
|
||||
|
||||
- Delay and sampling-rate offset are different phenomena.
|
||||
- Distributed microphone arrays require explicit SRO estimation/compensation.
|
||||
- GCC-PHAT is useful for TDOA/delay evidence, not enough to own canonical time.
|
||||
|
||||
Mimir consequence:
|
||||
|
||||
- Passive mode is a confidence-bearing witness.
|
||||
- Active codebook anchors remain the canonical-time path.
|
||||
- The actuator must include both fractional delay and SRO control.
|
||||
|
||||
References:
|
||||
|
||||
- Wang/Doclo SRO estimation.
|
||||
- Schmalenstroeer/Haeb-Umbach clock skew work.
|
||||
- Didier distributed adaptive node-specific signal estimation.
|
||||
- GCC-PHAT refinements.
|
||||
|
||||
## Room / Sound Field Reconstruction
|
||||
|
||||
Claim:
|
||||
|
||||
- Sound-field reconstruction from sparse arrays is an inverse problem with
|
||||
geometry, bandwidth, regularization, and noise constraints.
|
||||
- Ambisonics works cleanly when the microphone array and calibration support the
|
||||
basis.
|
||||
- Equivalent-source / sparse reconstruction methods are useful when the source
|
||||
model is sparse and measurement model is known.
|
||||
|
||||
Mimir consequence:
|
||||
|
||||
- Six heterogeneous mics are not automatically a high-order field mic.
|
||||
- The first honest target is synchronized stems plus source/constraint estimates.
|
||||
- Field reconstruction should expose uncertainty and avoid pretending unsupported
|
||||
spatial detail exists.
|
||||
|
||||
References:
|
||||
|
||||
- Pyroomacoustics.
|
||||
- Spatial Audio Framework.
|
||||
- ManyEars.
|
||||
- near-field acoustic holography / equivalent source / compressive sensing
|
||||
papers.
|
||||
- ambisonics/arbitrary-array encoding notes.
|
||||
|
||||
## Fractional Delay / SRO Actuation
|
||||
|
||||
Claim:
|
||||
|
||||
- Farrow structures are standard for continuously variable fractional delay.
|
||||
- Variable-rate resampling or equivalent phase correction is required for drift.
|
||||
|
||||
Mimir consequence:
|
||||
|
||||
- Microsecond reporting is incomplete until an actuator moves samples.
|
||||
- A Farrow proof is the right first native/Faust actuator slice.
|
||||
- Higher-quality polyphase/ASRC work comes after the control loop is proven.
|
||||
|
||||
References:
|
||||
|
||||
- AMD Vitis Farrow filter docs.
|
||||
- MATLAB Farrow fractional delay notes.
|
||||
- joint SFO/Farrow papers.
|
||||
|
||||
## Gaussian Splatting / Visual Fusion
|
||||
|
||||
Claim:
|
||||
|
||||
- Practical 3DGS renderers use packed GPU buffers, projection, tile binning,
|
||||
sorting/order strategies, and optimized rasterization kernels.
|
||||
- Dynamic/4D Gaussian splatting extends splats over time, but production code is
|
||||
still research-shaped.
|
||||
- Online SLAM/sensor fusion variants are more relevant to Mimir than offline
|
||||
static scene training alone.
|
||||
|
||||
Mimir consequence:
|
||||
|
||||
- Fensalir should own the GPU field.
|
||||
- Mimir should feed synchronized observations and constraints.
|
||||
- The first proof is live observation-to-claim-to-render, not full 4DGS training.
|
||||
|
||||
References:
|
||||
|
||||
- GraphDeco 3DGS.
|
||||
- 4D Gaussian Splatting.
|
||||
- Spacetime Gaussians.
|
||||
- gsplat.
|
||||
- NVIDIA `vk_gaussian_splatting`.
|
||||
- StopThePop.
|
||||
- FlashGS.
|
||||
- Gaussian-SLAM / multi-sensor calibration papers.
|
||||
|
||||
## Native Capture / GPU Resource Boundaries
|
||||
|
||||
Claim:
|
||||
|
||||
- Windows KS/AVStream is the low-level video streaming surface under higher media
|
||||
frameworks.
|
||||
- D3D shared handles are the relevant resource-sharing family once camera
|
||||
payloads can become GPU resources.
|
||||
- ASIO callbacks and sample positions are the correct place to preserve driver
|
||||
clock timing.
|
||||
|
||||
Mimir consequence:
|
||||
|
||||
- Media frameworks may remain diagnostics/fallbacks, but direct workers own
|
||||
production capture.
|
||||
- Fensalir owns D3D12 resource lifetime and imports.
|
||||
- Mimir should not route six cameras through stdout/JSON or managed pixel
|
||||
processing.
|
||||
|
||||
References:
|
||||
|
||||
- Microsoft Kernel Streaming docs.
|
||||
- Microsoft DXGI shared handle docs.
|
||||
- ASIO specification / callback notes.
|
||||
- Focusrite channel/loopback documentation.
|
||||
|
||||
## Strongest Design Conclusion
|
||||
|
||||
The research converges on one architecture:
|
||||
|
||||
```text
|
||||
controlled witness + calibrated sensors + rolling buffers
|
||||
-> deterministic timeline/response models
|
||||
-> native DSP and GPU field lowering
|
||||
-> OBS-facing program output
|
||||
```
|
||||
|
||||
The critical missing production step is not another filter. It is connecting
|
||||
measurement to actuation and field lowering:
|
||||
|
||||
- streaming decoder state;
|
||||
- calibration-weighted likelihood;
|
||||
- fractional delay/SRO actuator;
|
||||
- native camera payload handles;
|
||||
- Fensalir contract lowering.
|
||||
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
{
|
||||
"schema": "gamecult.mimir.sample_code_index.v1",
|
||||
"created": "2026-05-23",
|
||||
"samples": [
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/StreamingChirpBinDecoderSketch.cs",
|
||||
"domain": "audio-sync",
|
||||
"purpose": "Incremental chirp-bin decoder state model",
|
||||
"productionTarget": "Mimir.Runtime streaming active decoder",
|
||||
"promoteWhen": "Matches current analyzer outputs while avoiding full-window rescans"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/Avx2DechirpGoertzelSketch.cpp",
|
||||
"domain": "audio-sync",
|
||||
"purpose": "AVX2/FMA dechirp plus bin scoring sketch",
|
||||
"productionTarget": "native chirp-bin scorer",
|
||||
"promoteWhen": "Profiling shows C# scalar bin scoring dominates"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/BatchedChirpScoreAbiSketch.h",
|
||||
"domain": "audio-sync",
|
||||
"purpose": "Native batched scoring ABI boundary for chirp candidate/bin work",
|
||||
"productionTarget": "Mimir native scorer interop layer",
|
||||
"promoteWhen": "Scalar scoring dominates and batching proves interop cost is amortized"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/ChirpBinScore.compute.hlsl",
|
||||
"domain": "audio-sync-gpu",
|
||||
"purpose": "D3D12 compute candidate/bin scoring sketch",
|
||||
"productionTarget": "Fensalir compute scoring pass",
|
||||
"promoteWhen": "Candidate/bin/channel batches are large enough to beat CPU SIMD"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/FarrowFractionalDelaySketch.cpp",
|
||||
"domain": "audio-dsp",
|
||||
"purpose": "Cubic Farrow fractional delay actuator sketch",
|
||||
"productionTarget": "Faust/native DSP alignment actuator",
|
||||
"promoteWhen": "Runtime state can drive a test resampler/delay line with measured improvement"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/SpscAudioBlockRingSketch.cpp",
|
||||
"domain": "native-capture",
|
||||
"purpose": "Low-jitter SPSC audio block handoff sketch",
|
||||
"productionTarget": "native ASIO/camera capture handoff",
|
||||
"promoteWhen": "Managed queues or payload copies show measurable capture pressure"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/CalibrationWeightedChirpLikelihoodSketch.cs",
|
||||
"domain": "audio-sync",
|
||||
"purpose": "Calibration-weighted symbol likelihood before codebook solving",
|
||||
"productionTarget": "MimirChirpBinTimeline scoring path",
|
||||
"promoteWhen": "Physical mic captures show unweighted symbol likelihood is the failure point"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/SroPllAsrcControllerSketch.cpp",
|
||||
"domain": "audio-dsp",
|
||||
"purpose": "Control loop turning delay observations into ASRC/fractional delay commands",
|
||||
"productionTarget": "Faust/native DSP actuator control",
|
||||
"promoteWhen": "Runtime reports are stable enough to move samples instead of only logging state"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/GaussianSplatClaimUpdate.compute.hlsl",
|
||||
"domain": "visual-fusion",
|
||||
"purpose": "D3D12 compute sketch for live splat claim updates",
|
||||
"productionTarget": "Fensalir visual fusion pass",
|
||||
"promoteWhen": "Direct camera drivers feed synchronized feature observations"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/AquariumGpuSensorFrameBridgeSketch.cs",
|
||||
"domain": "fensalir-integration",
|
||||
"purpose": "Lower runtime video observations into Fensalir GPU sensor frame contracts",
|
||||
"productionTarget": "Mimir app-to-Fensalir bridge",
|
||||
"promoteWhen": "Runtime buffers carry pixel/native handles and lowering benchmarks allocation-free"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/AcousticConstraintLoweringSketch.cs",
|
||||
"domain": "fensalir-integration",
|
||||
"purpose": "Lower decoded audio anchors and mic geometry into Fensalir acoustic constraints",
|
||||
"productionTarget": "Mimir acoustic field bridge",
|
||||
"promoteWhen": "Decoder reports stable anchors and Fensalir consumes constraints in diagnostics"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/TemporalEvidenceCandidateSketch.cs",
|
||||
"domain": "sensor-fusion",
|
||||
"purpose": "Convert Mimir observations into reservoir candidates without owning stable tracks",
|
||||
"productionTarget": "Fensalir temporal evidence lowering path",
|
||||
"promoteWhen": "Mimir emits synchronized visual/acoustic candidates and Fensalir reservoir accepts them"
|
||||
},
|
||||
{
|
||||
"path": "research/perfect-machine-study-2026-05-23/samples/SrpPhatGridSketch.cpp",
|
||||
"domain": "volumetric-audio",
|
||||
"purpose": "Minimal grid scoring from pairwise mic delays to source-position hypotheses",
|
||||
"productionTarget": "native acoustic source localization proof",
|
||||
"promoteWhen": "At least two physical mics produce calibrated stable delay anchors"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
// Sketch only. Purpose: keep acoustic raw buffers in Mimir/native DSP while
|
||||
// sending Fensalir compact constraints for spatial fusion and diagnostics.
|
||||
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace Mimir.PerfectMachineStudy;
|
||||
|
||||
public readonly record struct MimirDecodedAnchorSketch(
|
||||
string SourceId,
|
||||
int EventIndex,
|
||||
double SourceSampleTimeSeconds,
|
||||
double CanonicalTimeSeconds,
|
||||
double DelaySeconds,
|
||||
double FrequencyHz,
|
||||
float Confidence,
|
||||
float PhaseRadians,
|
||||
float GroupDelaySeconds);
|
||||
|
||||
public readonly record struct MimirMicGeometrySketch(
|
||||
string SourceId,
|
||||
float X,
|
||||
float Y,
|
||||
float Z);
|
||||
|
||||
public sealed class AcousticConstraintLoweringSketch
|
||||
{
|
||||
private readonly List<AquariumAcousticConstraintSketch> _constraints = new(128);
|
||||
|
||||
public AquariumAcousticFieldFrameSketch Lower(
|
||||
ReadOnlySpan<MimirDecodedAnchorSketch> anchors,
|
||||
ReadOnlySpan<MimirMicGeometrySketch> geometry,
|
||||
double canonicalNowSeconds)
|
||||
{
|
||||
_constraints.Clear();
|
||||
|
||||
for (var i = 0; i < anchors.Length; i++)
|
||||
{
|
||||
ref readonly var anchor = ref anchors[i];
|
||||
if (anchor.Confidence < 0.4f)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var mic = FindGeometry(geometry, anchor.SourceId);
|
||||
if (mic.SourceId is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
_constraints.Add(new AquariumAcousticConstraintSketch(
|
||||
anchor.SourceId,
|
||||
anchor.EventIndex,
|
||||
canonicalNowSeconds,
|
||||
anchor.CanonicalTimeSeconds,
|
||||
anchor.DelaySeconds,
|
||||
anchor.FrequencyHz,
|
||||
mic.X,
|
||||
mic.Y,
|
||||
mic.Z,
|
||||
anchor.Confidence,
|
||||
anchor.PhaseRadians,
|
||||
anchor.GroupDelaySeconds));
|
||||
}
|
||||
|
||||
// Production version should avoid this copy and hand Fensalir a stable
|
||||
// span/packet buffer whose lifetime is owned by the frame boundary.
|
||||
return new AquariumAcousticFieldFrameSketch(canonicalNowSeconds, _constraints.ToArray());
|
||||
}
|
||||
|
||||
private static MimirMicGeometrySketch FindGeometry(
|
||||
ReadOnlySpan<MimirMicGeometrySketch> geometry,
|
||||
string sourceId)
|
||||
{
|
||||
for (var i = 0; i < geometry.Length; i++)
|
||||
{
|
||||
if (StringComparer.Ordinal.Equals(geometry[i].SourceId, sourceId))
|
||||
{
|
||||
return geometry[i];
|
||||
}
|
||||
}
|
||||
|
||||
return default;
|
||||
}
|
||||
}
|
||||
|
||||
public readonly record struct AquariumAcousticFieldFrameSketch(
|
||||
double CanonicalTimeSeconds,
|
||||
AquariumAcousticConstraintSketch[] Constraints);
|
||||
|
||||
public readonly record struct AquariumAcousticConstraintSketch(
|
||||
string SourceId,
|
||||
int EventIndex,
|
||||
double FrameCanonicalTimeSeconds,
|
||||
double EventCanonicalTimeSeconds,
|
||||
double DelaySeconds,
|
||||
double FrequencyHz,
|
||||
float MicX,
|
||||
float MicY,
|
||||
float MicZ,
|
||||
float Confidence,
|
||||
float PhaseRadians,
|
||||
float GroupDelaySeconds);
|
||||
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
// Sketch only. This is not wired to production contracts yet.
|
||||
// Purpose: show the desired shape of the Mimir -> Fensalir camera lowering cut.
|
||||
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace Mimir.PerfectMachineStudy;
|
||||
|
||||
public readonly record struct MimirVideoObservation(
|
||||
string SourceId,
|
||||
long Sequence,
|
||||
double CanonicalTimeSeconds,
|
||||
double DeviceTimeSeconds,
|
||||
int Width,
|
||||
int Height,
|
||||
string PixelFormat,
|
||||
IntPtr NativeHandle,
|
||||
int NativeHandleKind,
|
||||
string CalibrationId,
|
||||
float Confidence);
|
||||
|
||||
public sealed class AquariumGpuSensorFrameBridgeSketch
|
||||
{
|
||||
private readonly List<AquariumGpuSensorCameraSketch> _cameras = new(8);
|
||||
|
||||
public AquariumGpuSensorFrameSketch Lower(
|
||||
ReadOnlySpan<MimirVideoObservation> observations,
|
||||
double frameCanonicalTimeSeconds)
|
||||
{
|
||||
_cameras.Clear();
|
||||
|
||||
for (var i = 0; i < observations.Length; i++)
|
||||
{
|
||||
ref readonly var source = ref observations[i];
|
||||
if (source.NativeHandle == IntPtr.Zero || source.Confidence <= 0.0f)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
_cameras.Add(new AquariumGpuSensorCameraSketch(
|
||||
source.SourceId,
|
||||
source.Sequence,
|
||||
source.CanonicalTimeSeconds,
|
||||
source.DeviceTimeSeconds,
|
||||
source.Width,
|
||||
source.Height,
|
||||
source.PixelFormat,
|
||||
source.NativeHandle,
|
||||
source.NativeHandleKind,
|
||||
source.CalibrationId,
|
||||
source.Confidence));
|
||||
}
|
||||
|
||||
// Production version should fill a reusable contract-owned array or pooled
|
||||
// struct buffer instead of allocating here.
|
||||
return new AquariumGpuSensorFrameSketch(
|
||||
frameCanonicalTimeSeconds,
|
||||
_cameras.ToArray());
|
||||
}
|
||||
}
|
||||
|
||||
public readonly record struct AquariumGpuSensorFrameSketch(
|
||||
double CanonicalTimeSeconds,
|
||||
AquariumGpuSensorCameraSketch[] Cameras);
|
||||
|
||||
public readonly record struct AquariumGpuSensorCameraSketch(
|
||||
string SourceId,
|
||||
long Sequence,
|
||||
double CanonicalTimeSeconds,
|
||||
double DeviceTimeSeconds,
|
||||
int Width,
|
||||
int Height,
|
||||
string PixelFormat,
|
||||
IntPtr NativeHandle,
|
||||
int NativeHandleKind,
|
||||
string CalibrationId,
|
||||
float Confidence);
|
||||
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
// Sketch only. Purpose: native ABI shape for batched chirp-bin scoring.
|
||||
// The point is to batch enough work that managed/native transition cost does
|
||||
// not dominate the actual multiply-adds.
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
|
||||
extern "C"
|
||||
{
|
||||
struct MimirChirpScoreCandidate
|
||||
{
|
||||
std::int64_t absoluteSampleOffset;
|
||||
std::int32_t sampleOffsetInBuffer;
|
||||
std::int32_t sampleCount;
|
||||
float proposalEnergy;
|
||||
};
|
||||
|
||||
struct MimirChirpScoreKernel
|
||||
{
|
||||
const float* window;
|
||||
const float* dechirpReal;
|
||||
const float* dechirpImag;
|
||||
std::int32_t sampleCount;
|
||||
};
|
||||
|
||||
struct MimirChirpScoreRequest
|
||||
{
|
||||
const float* samples;
|
||||
std::int32_t sampleCount;
|
||||
const MimirChirpScoreCandidate* candidates;
|
||||
std::int32_t candidateCount;
|
||||
const MimirChirpScoreKernel* kernels;
|
||||
std::int32_t kernelCount;
|
||||
std::int32_t topK;
|
||||
};
|
||||
|
||||
struct MimirChirpScoreResult
|
||||
{
|
||||
std::int64_t absoluteSampleOffset;
|
||||
std::int32_t kernelIndex;
|
||||
float energy;
|
||||
float phaseRadians;
|
||||
};
|
||||
|
||||
// Returns the number of result rows written. Production ABI would also
|
||||
// return an error/status code and expose required result capacity.
|
||||
std::int32_t mimir_score_chirp_candidates(
|
||||
const MimirChirpScoreRequest* request,
|
||||
MimirChirpScoreResult* results,
|
||||
std::int32_t resultCapacity);
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,105 @@
|
|||
// Sketch only. This shows where meatspace calibration should enter the active
|
||||
// decoder: before code-valid path selection, as likelihood shaping.
|
||||
|
||||
namespace Mimir.Research.Samples;
|
||||
|
||||
public sealed class CalibrationWeightedChirpLikelihoodSketch
|
||||
{
|
||||
private readonly PathCalibration calibration;
|
||||
|
||||
public CalibrationWeightedChirpLikelihoodSketch(PathCalibration calibration)
|
||||
{
|
||||
this.calibration = calibration;
|
||||
}
|
||||
|
||||
public SymbolLikelihood[] ScoreFrame(FrameBinEnergy frame)
|
||||
{
|
||||
var result = new SymbolLikelihood[calibration.Symbols.Length];
|
||||
for (var symbolId = 0; symbolId < calibration.Symbols.Length; symbolId++)
|
||||
{
|
||||
var symbol = calibration.Symbols[symbolId];
|
||||
var observed = frame.EnergyByBin[symbolId];
|
||||
var reliability = symbol.Reliability;
|
||||
var confusionPenalty = ConfusionPenalty(symbolId, frame);
|
||||
var phaseWeight = PhaseWeight(symbol, frame.PhaseByBin[symbolId]);
|
||||
var delayCorrection = symbol.GroupDelaySamples;
|
||||
|
||||
var logLikelihood =
|
||||
Math.Log(Math.Max(observed, 1.0e-12)) * reliability +
|
||||
Math.Log(Math.Max(phaseWeight, 1.0e-6)) -
|
||||
confusionPenalty;
|
||||
|
||||
result[symbolId] = new SymbolLikelihood(
|
||||
symbolId,
|
||||
logLikelihood,
|
||||
delayCorrection,
|
||||
reliability,
|
||||
observed);
|
||||
}
|
||||
|
||||
return result
|
||||
.OrderByDescending(score => score.LogLikelihood)
|
||||
.Take(calibration.MaxCandidatesPerFrame)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private double ConfusionPenalty(int expectedSymbol, FrameBinEnergy frame)
|
||||
{
|
||||
var penalty = 0.0;
|
||||
foreach (var alias in calibration.Confusions.Where(c => c.ExpectedSymbol == expectedSymbol))
|
||||
{
|
||||
var aliasEnergy = frame.EnergyByBin[alias.ObservedSymbol];
|
||||
penalty += alias.Probability * Math.Log1p(aliasEnergy);
|
||||
}
|
||||
|
||||
return penalty;
|
||||
}
|
||||
|
||||
private static double PhaseWeight(SymbolCalibration symbol, double observedPhase)
|
||||
{
|
||||
if (symbol.PhaseCoherence <= 0.0)
|
||||
{
|
||||
return 1.0;
|
||||
}
|
||||
|
||||
var distance = AngularDistance(observedPhase, symbol.MeanPhaseRadians);
|
||||
return 1.0 - symbol.PhaseCoherence * Math.Min(1.0, distance / Math.PI);
|
||||
}
|
||||
|
||||
private static double AngularDistance(double a, double b)
|
||||
{
|
||||
var d = Math.Abs(a - b) % (Math.PI * 2.0);
|
||||
return d > Math.PI ? Math.PI * 2.0 - d : d;
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record FrameBinEnergy(
|
||||
double[] EnergyByBin,
|
||||
double[] PhaseByBin,
|
||||
double InitialOffsetSamples);
|
||||
|
||||
public sealed record PathCalibration(
|
||||
SymbolCalibration[] Symbols,
|
||||
Confusion[] Confusions,
|
||||
int MaxCandidatesPerFrame);
|
||||
|
||||
public sealed record SymbolCalibration(
|
||||
int SymbolId,
|
||||
double Reliability,
|
||||
double MeanEnergy,
|
||||
double MeanPhaseRadians,
|
||||
double PhaseCoherence,
|
||||
double GroupDelaySamples);
|
||||
|
||||
public sealed record Confusion(
|
||||
int ExpectedSymbol,
|
||||
int ObservedSymbol,
|
||||
double Probability);
|
||||
|
||||
public sealed record SymbolLikelihood(
|
||||
int SymbolId,
|
||||
double LogLikelihood,
|
||||
double GroupDelayCorrectionSamples,
|
||||
double Reliability,
|
||||
double ObservedEnergy);
|
||||
|
||||
|
|
@ -0,0 +1,68 @@
|
|||
// Sketch only. Fensalir-side compute shape for updating simple live splat
|
||||
// claims from synchronized feature observations.
|
||||
|
||||
struct FeatureObservation
|
||||
{
|
||||
uint StableKeyLow;
|
||||
uint StableKeyHigh;
|
||||
float3 PositionMeters;
|
||||
float3 Normal;
|
||||
float4 Color;
|
||||
float Confidence;
|
||||
uint SourceIndex;
|
||||
uint TimestampLo;
|
||||
uint TimestampHi;
|
||||
};
|
||||
|
||||
struct SplatClaim
|
||||
{
|
||||
uint StableKeyLow;
|
||||
uint StableKeyHigh;
|
||||
float3 PositionMeters;
|
||||
float RadiusMeters;
|
||||
float4 Color;
|
||||
float Confidence;
|
||||
uint TimeMinLo;
|
||||
uint TimeMinHi;
|
||||
uint TimeMaxLo;
|
||||
uint TimeMaxHi;
|
||||
};
|
||||
|
||||
cbuffer SplatUpdateConstants : register(b0)
|
||||
{
|
||||
uint ObservationCount;
|
||||
float Decay;
|
||||
float Blend;
|
||||
float DefaultRadius;
|
||||
};
|
||||
|
||||
StructuredBuffer<FeatureObservation> Observations : register(t0);
|
||||
RWStructuredBuffer<SplatClaim> Claims : register(u0);
|
||||
|
||||
[numthreads(128, 1, 1)]
|
||||
void main(uint3 id : SV_DispatchThreadID)
|
||||
{
|
||||
uint index = id.x;
|
||||
if (index >= ObservationCount)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
FeatureObservation obs = Observations[index];
|
||||
|
||||
// Sketch simplification: production needs a hash table or sorted stable-key
|
||||
// compaction pass. This writes one observation to one claim slot.
|
||||
SplatClaim claim;
|
||||
claim.StableKeyLow = obs.StableKeyLow;
|
||||
claim.StableKeyHigh = obs.StableKeyHigh;
|
||||
claim.PositionMeters = obs.PositionMeters;
|
||||
claim.RadiusMeters = DefaultRadius;
|
||||
claim.Color = obs.Color;
|
||||
claim.Confidence = saturate(obs.Confidence);
|
||||
claim.TimeMinLo = obs.TimestampLo;
|
||||
claim.TimeMinHi = obs.TimestampHi;
|
||||
claim.TimeMaxLo = obs.TimestampLo;
|
||||
claim.TimeMaxHi = obs.TimestampHi;
|
||||
Claims[index] = claim;
|
||||
}
|
||||
|
||||
|
|
@ -9,14 +9,30 @@ machinery into the app.
|
|||
- `StreamingChirpBinDecoderSketch.cs`: C# state model for an incremental
|
||||
chirp-bin decoder that avoids full-window rescans.
|
||||
- `Avx2DechirpGoertzelSketch.cpp`: AVX2/FMA-shaped CPU scoring kernel sketch.
|
||||
- `BatchedChirpScoreAbiSketch.h`: native scoring ABI shape for batching
|
||||
candidate/bin work across the managed/native boundary.
|
||||
- `ChirpBinScore.compute.hlsl`: D3D12 compute shader shape for batched
|
||||
candidate/bin scoring.
|
||||
- `FarrowFractionalDelaySketch.cpp`: native fractional delay actuator sketch.
|
||||
- `SpscAudioBlockRingSketch.cpp`: native single-producer/single-consumer audio
|
||||
block ring sketch for capture worker handoff.
|
||||
- `CalibrationWeightedChirpLikelihoodSketch.cs`: shows how response/confusion/
|
||||
phase/group-delay calibration should shape symbol likelihood before codebook
|
||||
path selection.
|
||||
- `SroPllAsrcControllerSketch.cpp`: control-rate delay/SRO loop for driving an
|
||||
ASRC or fractional-delay actuator.
|
||||
- `GaussianSplatClaimUpdate.compute.hlsl`: minimal Fensalir compute shape for
|
||||
turning synchronized visual observations into live splat claims.
|
||||
- `AquariumGpuSensorFrameBridgeSketch.cs`: contract sketch for lowering runtime
|
||||
video observations into Fensalir GPU sensor frames.
|
||||
- `AcousticConstraintLoweringSketch.cs`: contract sketch for lowering decoded
|
||||
chirp/passive anchors into Fensalir acoustic constraints.
|
||||
- `TemporalEvidenceCandidateSketch.cs`: candidate lowering sketch that keeps
|
||||
stable track ownership in Fensalir.
|
||||
- `SrpPhatGridSketch.cpp`: minimal TDOA/SRP-PHAT-style grid scoring proof for a
|
||||
synchronized mic constellation.
|
||||
|
||||
## Rule
|
||||
|
||||
Promote one of these only after a benchmark or integration test proves it owns a
|
||||
real invariant. Sample code is a microscope, not a new wall in the house.
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,60 @@
|
|||
// Sketch only. Control-rate PLL for driving an ASRC/fractional delay actuator.
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
|
||||
struct SyncObservation
|
||||
{
|
||||
double delaySamples;
|
||||
double confidence;
|
||||
double dtSeconds;
|
||||
};
|
||||
|
||||
struct ActuatorCommand
|
||||
{
|
||||
double targetDelaySamples;
|
||||
double resampleRatio;
|
||||
double confidence;
|
||||
};
|
||||
|
||||
class SroPllAsrcController
|
||||
{
|
||||
public:
|
||||
ActuatorCommand update(const SyncObservation& obs)
|
||||
{
|
||||
if (obs.confidence < 0.05 || obs.dtSeconds <= 0.0)
|
||||
{
|
||||
return command(obs.confidence);
|
||||
}
|
||||
|
||||
double error = obs.delaySamples - delaySamples_;
|
||||
double alpha = clamp(obs.confidence * 0.08, 0.005, 0.05);
|
||||
double beta = clamp(obs.confidence * 0.01, 0.0005, 0.006);
|
||||
|
||||
delaySamples_ += alpha * error;
|
||||
sroPpm_ += beta * error / obs.dtSeconds;
|
||||
sroPpm_ = clamp(sroPpm_, -300.0, 300.0);
|
||||
confidence_ += (obs.confidence - confidence_) * 0.05;
|
||||
|
||||
return command(confidence_);
|
||||
}
|
||||
|
||||
private:
|
||||
double delaySamples_ = 0.0;
|
||||
double sroPpm_ = 0.0;
|
||||
double confidence_ = 0.0;
|
||||
|
||||
ActuatorCommand command(double confidence) const
|
||||
{
|
||||
// Positive SRO means candidate appears to drift later; playback/capture
|
||||
// correction convention must be fixed against the DSP graph before use.
|
||||
double ratio = 1.0 - sroPpm_ * 1.0e-6;
|
||||
return { delaySamples_, ratio, confidence };
|
||||
}
|
||||
|
||||
static double clamp(double value, double lo, double hi)
|
||||
{
|
||||
return std::max(lo, std::min(hi, value));
|
||||
}
|
||||
};
|
||||
|
||||
|
|
@ -0,0 +1,95 @@
|
|||
// Sketch only. Purpose: first acoustic source-position proof after mic timing is
|
||||
// calibrated. This is intentionally small and direct, not a full acoustic camera.
|
||||
|
||||
#include <cmath>
|
||||
#include <cstdint>
|
||||
#include <span>
|
||||
|
||||
struct MicPoseSketch
|
||||
{
|
||||
float x;
|
||||
float y;
|
||||
float z;
|
||||
};
|
||||
|
||||
struct CandidatePointSketch
|
||||
{
|
||||
float x;
|
||||
float y;
|
||||
float z;
|
||||
};
|
||||
|
||||
struct PairDelaySketch
|
||||
{
|
||||
int micA;
|
||||
int micB;
|
||||
double delaySeconds;
|
||||
float confidence;
|
||||
};
|
||||
|
||||
struct SrpResultSketch
|
||||
{
|
||||
int bestIndex;
|
||||
float score;
|
||||
};
|
||||
|
||||
static inline double distanceMeters(
|
||||
const MicPoseSketch& mic,
|
||||
const CandidatePointSketch& point)
|
||||
{
|
||||
const double dx = static_cast<double>(point.x) - mic.x;
|
||||
const double dy = static_cast<double>(point.y) - mic.y;
|
||||
const double dz = static_cast<double>(point.z) - mic.z;
|
||||
return std::sqrt(dx * dx + dy * dy + dz * dz);
|
||||
}
|
||||
|
||||
SrpResultSketch scoreGrid(
|
||||
std::span<const MicPoseSketch> mics,
|
||||
std::span<const CandidatePointSketch> grid,
|
||||
std::span<const PairDelaySketch> observedPairs,
|
||||
double speedOfSoundMetersPerSecond)
|
||||
{
|
||||
auto best = SrpResultSketch{-1, -1.0f};
|
||||
|
||||
for (int pointIndex = 0; pointIndex < static_cast<int>(grid.size()); pointIndex++)
|
||||
{
|
||||
const auto& point = grid[static_cast<size_t>(pointIndex)];
|
||||
double score = 0.0;
|
||||
double weight = 0.0;
|
||||
|
||||
for (const auto& pair : observedPairs)
|
||||
{
|
||||
if (pair.micA < 0 || pair.micB < 0 ||
|
||||
pair.micA >= static_cast<int>(mics.size()) ||
|
||||
pair.micB >= static_cast<int>(mics.size()) ||
|
||||
pair.confidence <= 0.0f)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
const double da = distanceMeters(mics[static_cast<size_t>(pair.micA)], point);
|
||||
const double db = distanceMeters(mics[static_cast<size_t>(pair.micB)], point);
|
||||
const double predicted = (db - da) / speedOfSoundMetersPerSecond;
|
||||
const double errorUs = std::abs(predicted - pair.delaySeconds) * 1'000'000.0;
|
||||
|
||||
// Smooth robust score: 1 at zero error, half around 100 us.
|
||||
const double local = 1.0 / (1.0 + errorUs / 100.0);
|
||||
score += local * pair.confidence;
|
||||
weight += pair.confidence;
|
||||
}
|
||||
|
||||
if (weight <= 0.0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
const float normalized = static_cast<float>(score / weight);
|
||||
if (normalized > best.score)
|
||||
{
|
||||
best = SrpResultSketch{pointIndex, normalized};
|
||||
}
|
||||
}
|
||||
|
||||
return best;
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
// Sketch only. Purpose: show how Mimir evidence can be lowered as candidates
|
||||
// without becoming a second stable-track database beside Fensalir.
|
||||
|
||||
using System;
|
||||
|
||||
namespace Mimir.PerfectMachineStudy;
|
||||
|
||||
public enum MimirEvidenceKindSketch
|
||||
{
|
||||
Unknown = 0,
|
||||
VisualFeature = 1,
|
||||
ChirpAnchor = 2,
|
||||
PassiveAudioAnchor = 3,
|
||||
AcousticSource = 4
|
||||
}
|
||||
|
||||
public readonly record struct MimirEvidenceCandidateSketch(
|
||||
MimirEvidenceKindSketch Kind,
|
||||
ulong StableSourceKey,
|
||||
double CanonicalTimeSeconds,
|
||||
float X,
|
||||
float Y,
|
||||
float Z,
|
||||
float Radius,
|
||||
float Confidence,
|
||||
float Residual,
|
||||
uint PayloadOffset,
|
||||
uint PayloadLength);
|
||||
|
||||
public readonly record struct FensalirReservoirCandidateSketch(
|
||||
ulong StableSourceKey,
|
||||
double CanonicalTimeSeconds,
|
||||
float X,
|
||||
float Y,
|
||||
float Z,
|
||||
float Radius,
|
||||
float Weight,
|
||||
uint Domain,
|
||||
uint PayloadOffset,
|
||||
uint PayloadLength);
|
||||
|
||||
public static class TemporalEvidenceCandidateLoweringSketch
|
||||
{
|
||||
public static int Lower(
|
||||
ReadOnlySpan<MimirEvidenceCandidateSketch> input,
|
||||
Span<FensalirReservoirCandidateSketch> output)
|
||||
{
|
||||
var count = 0;
|
||||
for (var i = 0; i < input.Length && count < output.Length; i++)
|
||||
{
|
||||
ref readonly var candidate = ref input[i];
|
||||
if (candidate.Confidence <= 0.0f || candidate.Radius <= 0.0f)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var residualPenalty = 1.0f / (1.0f + MathF.Max(0.0f, candidate.Residual));
|
||||
var weight = candidate.Confidence * residualPenalty;
|
||||
if (weight <= 0.0f)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
output[count++] = new FensalirReservoirCandidateSketch(
|
||||
candidate.StableSourceKey,
|
||||
candidate.CanonicalTimeSeconds,
|
||||
candidate.X,
|
||||
candidate.Y,
|
||||
candidate.Z,
|
||||
candidate.Radius,
|
||||
weight,
|
||||
(uint)candidate.Kind,
|
||||
candidate.PayloadOffset,
|
||||
candidate.PayloadLength);
|
||||
}
|
||||
|
||||
return count;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,178 @@
|
|||
# State Machine Invariants
|
||||
|
||||
## Purpose
|
||||
|
||||
The Perfect Machine is mostly a state discipline problem. This note lists the
|
||||
state each subsystem may own and the state it must not smuggle in through a
|
||||
friendly-looking helper.
|
||||
|
||||
## Runtime State
|
||||
|
||||
`MimirRuntime` may own:
|
||||
|
||||
- runtime lifecycle;
|
||||
- loaded configuration;
|
||||
- witness emission policy;
|
||||
- Fensalir scene state assembly;
|
||||
- UI/debug readout surfaces.
|
||||
|
||||
It must not own:
|
||||
|
||||
- raw driver callback loops;
|
||||
- audio DSP sample movement;
|
||||
- dense visual tracks;
|
||||
- OBS composition.
|
||||
|
||||
## Hub State
|
||||
|
||||
`MimirSynchronizationHub` may own:
|
||||
|
||||
- registered sources;
|
||||
- rolling buffer set;
|
||||
- polling/update cadence;
|
||||
- latest reports;
|
||||
- smoothed synchronization states.
|
||||
|
||||
It must not own:
|
||||
|
||||
- analyzer internals;
|
||||
- render tick work;
|
||||
- hidden per-source DSP buffers;
|
||||
- long-lived history outside retention window.
|
||||
|
||||
## Rolling Buffer State
|
||||
|
||||
Each rolling buffer owns:
|
||||
|
||||
- one stream descriptor;
|
||||
- bounded sample window;
|
||||
- current edge;
|
||||
- snapshot semantics.
|
||||
|
||||
It must not own:
|
||||
|
||||
- a second clock model;
|
||||
- per-kind retention policy;
|
||||
- decoded field tracks;
|
||||
- calibration truth.
|
||||
|
||||
## Active Decoder State
|
||||
|
||||
The streaming decoder should own:
|
||||
|
||||
- PCM window/ring;
|
||||
- event proposal ring;
|
||||
- candidate ambiguity ring;
|
||||
- codebook/schedule view;
|
||||
- local clock fit;
|
||||
- response/confusion accumulation.
|
||||
|
||||
It must not own:
|
||||
|
||||
- UI state;
|
||||
- emitter policy;
|
||||
- DSP actuator state;
|
||||
- global app timing authority.
|
||||
|
||||
## Calibration State
|
||||
|
||||
Calibration owns:
|
||||
|
||||
- output/mic path identity;
|
||||
- usable bands;
|
||||
- confusion matrix;
|
||||
- delay/bin-shift hypotheses;
|
||||
- magnitude response;
|
||||
- phase/group delay;
|
||||
- recommended emission plan.
|
||||
|
||||
It must not own:
|
||||
|
||||
- raw capture buffers forever;
|
||||
- UI-specific names as primary identity;
|
||||
- unverified learned patches that cannot be replayed.
|
||||
|
||||
## DSP Actuator State
|
||||
|
||||
The actuator should own:
|
||||
|
||||
- fractional delay line state;
|
||||
- resampler phase;
|
||||
- SRO control state;
|
||||
- per-source gain/phase correction;
|
||||
- audio-rate buffers.
|
||||
|
||||
It must not own:
|
||||
|
||||
- source discovery;
|
||||
- calibration file persistence;
|
||||
- UI controls except exposed parameters;
|
||||
- timeline decoding.
|
||||
|
||||
## Native Capture State
|
||||
|
||||
Native capture workers own:
|
||||
|
||||
- device handles;
|
||||
- callback/read state;
|
||||
- block/frame rings;
|
||||
- device timestamps/counters;
|
||||
- low-level format conversion where unavoidable.
|
||||
|
||||
They must not own:
|
||||
|
||||
- canonical timeline decisions;
|
||||
- OBS routing;
|
||||
- Fensalir render state;
|
||||
- long-term calibration policy.
|
||||
|
||||
## Fensalir State
|
||||
|
||||
Fensalir owns:
|
||||
|
||||
- window/input/render loop;
|
||||
- D3D12 resource lifetime;
|
||||
- external texture import;
|
||||
- GPU fusion buffers;
|
||||
- temporal evidence reservoir after lowering;
|
||||
- program render surfaces.
|
||||
|
||||
It must not own:
|
||||
|
||||
- source capture configuration;
|
||||
- audio clock authority;
|
||||
- Mimir calibration truth;
|
||||
- app-specific device policy.
|
||||
|
||||
## Network Receiver State
|
||||
|
||||
Remote nodes own:
|
||||
|
||||
- local PCM ring;
|
||||
- local decode state;
|
||||
- local clock fit;
|
||||
- local response/confusion observations;
|
||||
- transport health.
|
||||
|
||||
They must not own:
|
||||
|
||||
- Starfire canonical clock authority;
|
||||
- program output mixing;
|
||||
- global calibration policy;
|
||||
- trust decisions.
|
||||
|
||||
## OBS State
|
||||
|
||||
OBS owns:
|
||||
|
||||
- final scene composition;
|
||||
- stream/record controls;
|
||||
- operator mix choices.
|
||||
|
||||
It must not own:
|
||||
|
||||
- stream synchronization;
|
||||
- canonical timing;
|
||||
- raw calibration truth;
|
||||
- volumetric field reconstruction.
|
||||
|
||||
|
|
@ -0,0 +1,169 @@
|
|||
# Visual Fusion And Realtime Gaussian Splatting Study
|
||||
|
||||
## Thesis
|
||||
|
||||
Mimir's visual side should not become a CPU image processing application. The
|
||||
camera rig exists to produce time-indexed visual evidence that Fensalir can fuse
|
||||
on GPU into a live volumetric program surface.
|
||||
|
||||
The direct-driver work matters because sensor fusion is brutally sensitive to:
|
||||
|
||||
- timestamp error;
|
||||
- rolling/shutter latency;
|
||||
- frame drops;
|
||||
- calibration drift;
|
||||
- duplicated CPU copies;
|
||||
- inconsistent exposure/format state.
|
||||
|
||||
## Current Sensor Roles
|
||||
|
||||
- Leap stereo IR: near-field timing/depth/tracking candidate.
|
||||
- PS3 Eyes: high-rate feature/tracking witnesses.
|
||||
- Kiyo: stable RGB context.
|
||||
- Kiyo Pro: intended RGB ground truth, currently high-speed/cadence limited.
|
||||
- Raven-side devices: remote evidence producers, not local compute authority.
|
||||
|
||||
## Evidence Layers
|
||||
|
||||
### Raw Frame
|
||||
|
||||
Minimum:
|
||||
|
||||
- source id;
|
||||
- device/arrival timestamp;
|
||||
- dimensions/format/stride;
|
||||
- native handle or byte span.
|
||||
|
||||
### Feature Frame
|
||||
|
||||
Possible GPU outputs:
|
||||
|
||||
- corners/keypoints;
|
||||
- descriptors;
|
||||
- optical flow;
|
||||
- silhouettes/masks;
|
||||
- depth/disparity hints;
|
||||
- confidence per feature.
|
||||
|
||||
### Scene Claims
|
||||
|
||||
Rolling reservoir views:
|
||||
|
||||
- scene rays;
|
||||
- surface claims;
|
||||
- material claims;
|
||||
- render packets.
|
||||
|
||||
### Program Surface
|
||||
|
||||
Fensalir output:
|
||||
|
||||
- Gaussian splat/point cloud render;
|
||||
- avatar/AR overlay;
|
||||
- Spout2 program video.
|
||||
|
||||
## Realtime Gaussian Splatting Relevance
|
||||
|
||||
Gaussian splatting is attractive because it represents scenes as many local
|
||||
anisotropic primitives that can be rasterized quickly. For Mimir, the hard part
|
||||
is not rendering a static splat set. The hard part is maintaining a dynamic
|
||||
time-indexed splat field from live unsynchronized cameras.
|
||||
|
||||
Mimir needs:
|
||||
|
||||
- synchronized observations;
|
||||
- camera calibration;
|
||||
- dynamic object handling;
|
||||
- confidence and decay;
|
||||
- GPU-resident update path;
|
||||
- output that can be inspected/debugged live.
|
||||
|
||||
The low-level implementation references all point at the same renderer spine:
|
||||
project Gaussians, bin/intersect them into tiles, sort or partially order for
|
||||
view-consistent blending, then rasterize tiles with tightly packed GPU buffers.
|
||||
`gsplat`, NVIDIA's Vulkan sample, StopThePop, and FlashGS are useful because
|
||||
they expose performance levers Mimir/Fensalir can reuse without pretending the
|
||||
live rig is an offline training pipeline.
|
||||
|
||||
Useful borrowed ideas:
|
||||
|
||||
- packed structure-of-arrays splat buffers;
|
||||
- tile-size as a first-order performance parameter;
|
||||
- radius/opacity culling before sort/raster;
|
||||
- global versus local/tile order tradeoffs;
|
||||
- asynchronous CPU/GPU sorting only if it does not add frame latency;
|
||||
- feature/depth render modes as diagnostics, not only RGB output.
|
||||
|
||||
## 4D / Dynamic Splatting Shape
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Frames["time-aligned camera frames"] --> Features["GPU features / masks / flow"]
|
||||
Features --> Tracks["temporal tracks"]
|
||||
Tracks --> Claims["surface/material claims"]
|
||||
Claims --> Splats["dynamic Gaussian / point primitives"]
|
||||
Audio["audio source estimates"] --> Claims
|
||||
Splats --> Render["Fensalir program render"]
|
||||
```
|
||||
|
||||
## Data Model Sketch
|
||||
|
||||
```cpp
|
||||
struct MimirVisualObservation
|
||||
{
|
||||
uint64_t sourceHash;
|
||||
uint64_t timestampNs;
|
||||
uint32_t width;
|
||||
uint32_t height;
|
||||
uint32_t format;
|
||||
uint64_t nativeHandle;
|
||||
};
|
||||
|
||||
struct MimirSplatClaim
|
||||
{
|
||||
uint64_t stableKeyHash;
|
||||
float position[3];
|
||||
float covariance[6];
|
||||
float color[4];
|
||||
float confidence;
|
||||
uint64_t sourceTimeMinNs;
|
||||
uint64_t sourceTimeMaxNs;
|
||||
};
|
||||
```
|
||||
|
||||
## Immediate Architecture Cut
|
||||
|
||||
Do not start with "implement 4DGS." Start with the ownership:
|
||||
|
||||
1. direct camera driver pushes frame handles;
|
||||
2. runtime/reservoir indexes them;
|
||||
3. Fensalir reads current synchronized window;
|
||||
4. Fensalir produces simple point/render packet claims;
|
||||
5. only then expand to dynamic splat update.
|
||||
|
||||
## Micro-Optimization Risks
|
||||
|
||||
- CPU demosaic/format conversion before GPU upload.
|
||||
- Multiple copies of compressed camera frames.
|
||||
- Re-decoding frame data per feature stage.
|
||||
- Per-frame heap allocations in capture workers.
|
||||
- Blocking on GPU readback for UI/debug.
|
||||
- Sorting every splat globally every frame before the field is even stable.
|
||||
- Treating 4DGS optimization loops as live capture prerequisites.
|
||||
|
||||
## Research Threads To Continue
|
||||
|
||||
- 3D Gaussian Splatting reference CUDA rasterizer.
|
||||
- `gsplat` tile/raster APIs and CUDA memory layout.
|
||||
- NVIDIA `vk_gaussian_splatting` mesh-shader versus vertex-shader paths.
|
||||
- StopThePop / sorted Gaussian splatting for view-consistency tradeoffs.
|
||||
- FlashGS-style kernel-level optimization and redundant-sort reduction.
|
||||
- Dynamic/4D Gaussian splatting update strategies.
|
||||
- PlayCanvas/WebGPU splat processing as a practical open implementation of
|
||||
GPU-side splat mutation and browser-scale rendering constraints.
|
||||
- Gaussian-SLAM and multi-sensor calibration papers as shape references for
|
||||
online pose/calibration drift, even though Mimir's rig is room-bound rather
|
||||
than vehicle-mounted.
|
||||
- Online multi-camera calibration and temporal alignment.
|
||||
- GPU optical flow/features for live camera arrays.
|
||||
- Sensor-fusion priors from audio localization.
|
||||
|
|
@ -0,0 +1,191 @@
|
|||
# Volumetric Audio Field Study
|
||||
|
||||
## Thesis
|
||||
|
||||
Mimir's audio target is not "sync the mics." That is only the entry fee. The
|
||||
target is a continuously updated room-scale sound field: where sources are,
|
||||
which surfaces colored them, how each microphone hears them, and which
|
||||
OBS-facing stems can be emitted without flattening useful spatial evidence.
|
||||
|
||||
The current ASIO/chirp-bin work is the correct primitive because a microphone
|
||||
array without precise timing is just a pile of unrelated waveforms.
|
||||
|
||||
## Problem Decomposition
|
||||
|
||||
### 1. Clock And Delay
|
||||
|
||||
Every input stream needs:
|
||||
|
||||
- delay relative to loopback/canonical program time;
|
||||
- fractional delay below one sample;
|
||||
- sampling-rate offset over time;
|
||||
- confidence and failure state.
|
||||
|
||||
This is the current `MimirAudioSynchronizationState` surface.
|
||||
|
||||
### 2. Transfer Function
|
||||
|
||||
Every output/mic path needs:
|
||||
|
||||
- magnitude response by frequency;
|
||||
- phase/group-delay response;
|
||||
- confusion/alias behavior for coded active signals;
|
||||
- reflection-prone bands and time regions.
|
||||
|
||||
This is the current `MimirChirpBinCalibrationModel` surface, but it should grow
|
||||
from "decoder helper" into "room path model."
|
||||
|
||||
### 3. Source Localization
|
||||
|
||||
Once streams are aligned, source localization can use:
|
||||
|
||||
- TDOA between microphones;
|
||||
- direct-path chirp time-of-arrival;
|
||||
- beamforming/steered response;
|
||||
- visual priors from camera/pose/splat state;
|
||||
- semantic priors from dialogue/game sources.
|
||||
|
||||
For Mimir, the first robust source localizer should probably be hybrid:
|
||||
|
||||
1. use active chirp emitters for calibration and known anchors;
|
||||
2. use TDOA/GCC-PHAT/SRP-PHAT for transient/passive events;
|
||||
3. use visual tracking as a prior for voices/hands/objects.
|
||||
|
||||
### 4. Field Reconstruction
|
||||
|
||||
Field reconstruction can be represented at several resolutions:
|
||||
|
||||
- low-order ambisonic bed for OBS-friendly spatial ambience;
|
||||
- per-speaker/mic transfer function models;
|
||||
- localized source objects with direction, distance, and confidence;
|
||||
- room response map for reflections/suppression;
|
||||
- neural or Gaussian acoustic field later, if justified.
|
||||
|
||||
The first practical target is not a full pressure-field simulator. It is:
|
||||
|
||||
- aligned dialogue anchors;
|
||||
- camera mics as spatial witnesses;
|
||||
- local loopback as program reference;
|
||||
- co-streamer/Raven timing evidence;
|
||||
- separable stems plus spatial bed.
|
||||
|
||||
## Methods Worth Keeping Separate
|
||||
|
||||
### GCC-PHAT / SRP-PHAT
|
||||
|
||||
Use:
|
||||
|
||||
- passive TDOA;
|
||||
- transient localization;
|
||||
- drift hints when program audio is correlated.
|
||||
|
||||
Do not use:
|
||||
|
||||
- canonical timeline identity.
|
||||
|
||||
Reason:
|
||||
|
||||
- It estimates relative delay from content, but content is not a unique clock
|
||||
unless it is actively coded.
|
||||
|
||||
### Beamforming
|
||||
|
||||
Use:
|
||||
|
||||
- voice/source enhancement once mic positions and delays are known.
|
||||
|
||||
Risks:
|
||||
|
||||
- Bad geometry or uncorrected phase response creates false confidence.
|
||||
- Camera mics are low quality but spatially useful; Focusrites are high quality
|
||||
but fewer.
|
||||
|
||||
### Ambisonics
|
||||
|
||||
Use:
|
||||
|
||||
- OBS-facing spatial bed.
|
||||
- Compact representation after alignment.
|
||||
|
||||
Risks:
|
||||
|
||||
- Ambisonics is an output/control representation, not the raw truth. Do not
|
||||
throw away per-mic evidence too early.
|
||||
|
||||
### Room Impulse Response Estimation
|
||||
|
||||
Use:
|
||||
|
||||
- speaker/mic path model;
|
||||
- frequency response normalization;
|
||||
- reflection windows;
|
||||
- phase/group-delay calibration.
|
||||
|
||||
Best current route:
|
||||
|
||||
- Use chirp-bin codebook for continuous/lightweight operation.
|
||||
- Use fuller sweeps or exponential sine sweeps for deliberate calibration
|
||||
sessions.
|
||||
|
||||
## Architecture Proposal
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
ASIO["ASIO inputs + loopback"] --> Align["delay/SRO estimator"]
|
||||
Chirps["chirp-bin active witness"] --> Align
|
||||
Music["program audio"] --> Passive["passive GCC-PHAT"]
|
||||
Passive --> Align
|
||||
Align --> Model["path response model"]
|
||||
Align --> Actuator["fractional delay + ASRC"]
|
||||
Model --> Actuator
|
||||
Actuator --> Field["sound field estimator"]
|
||||
Visual["Fensalir visual priors"] --> Field
|
||||
Field --> Stems["voice/ambient/transient stems"]
|
||||
Field --> Bed["spatial bed"]
|
||||
Stems --> OBS["OBS"]
|
||||
Bed --> OBS
|
||||
```
|
||||
|
||||
## Data Model Sketch
|
||||
|
||||
```csharp
|
||||
public sealed record MimirAudioPathModel(
|
||||
string OutputSourceId,
|
||||
string ReceiverSourceId,
|
||||
int SampleRate,
|
||||
IReadOnlyList<FrequencyBandResponse> Magnitude,
|
||||
IReadOnlyList<GroupDelayPoint> GroupDelay,
|
||||
IReadOnlyList<ReflectionWindow> Reflections,
|
||||
IReadOnlyList<SymbolConfusion> ActiveConfusion,
|
||||
double DirectPathDelaySamples,
|
||||
double Confidence);
|
||||
|
||||
public sealed record MimirSoundSourceEstimate(
|
||||
string SourceClass,
|
||||
Vector3 PositionMeters,
|
||||
Vector3 VelocityMetersPerSecond,
|
||||
double StartTimeNs,
|
||||
double Confidence,
|
||||
IReadOnlyList<string> SupportingReceivers);
|
||||
```
|
||||
|
||||
## Hot-Path Warning
|
||||
|
||||
Do not implement volumetric audio as repeated whole-window cross-correlations
|
||||
between every pair of streams. Pairwise all-to-all is fine for diagnosis and
|
||||
small proofs; production needs:
|
||||
|
||||
- reference-first timing;
|
||||
- cached spectra;
|
||||
- bounded candidate pairs;
|
||||
- geometry-informed search;
|
||||
- streaming state.
|
||||
|
||||
## Research Threads To Continue
|
||||
|
||||
- SRP-PHAT acceleration on GPU for small mic arrays.
|
||||
- Low-order ambisonic encoding from irregular microphone arrays.
|
||||
- Joint audio-visual source localization with tracked speaker/head priors.
|
||||
- Online room impulse response tracking under moving sources.
|
||||
- Neural acoustic fields only after classical calibration stops scaling.
|
||||
|
||||
Loading…
Reference in a new issue