diff --git a/docs/perfect-machine-domain-index.md b/docs/perfect-machine-domain-index.md index 15e5133..74a4920 100644 --- a/docs/perfect-machine-domain-index.md +++ b/docs/perfect-machine-domain-index.md @@ -15,8 +15,14 @@ Read this with: the long-form design argument. - `research/perfect-machine-study-2026-05-23/index.md` for the study directory and sample-code index. +- `research/perfect-machine-study-2026-05-23/reading-guide.md` for task-oriented + navigation through the study. - `research/perfect-machine-study-2026-05-23/optimization-ledger.md` for hot loop candidates. +- `research/perfect-machine-study-2026-05-23/fensalir-integration-map.md` for + the Mimir/Fensalir ownership cut. +- `research/perfect-machine-study-2026-05-23/calibration-session-spec.md` for + the planned physical-path calibration command. - `research/perfect-machine-study-2026-05-23/samples/README.md` for sample implementation sketches. diff --git a/research/perfect-machine-study-2026-05-23/acoustic-field-models.md b/research/perfect-machine-study-2026-05-23/acoustic-field-models.md new file mode 100644 index 0000000..95c71af --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/acoustic-field-models.md @@ -0,0 +1,172 @@ +# Acoustic Field Models + +## Purpose + +Mimir's audio goal is not just "delay-align microphones." Delay alignment is +the entry fee. The larger target is a live volumetric audio field: where sound +is, how the room moves it, which mic/speaker paths are trustworthy, and how to +emit OBS-facing stems or a spatial bed that are better than raw source feeds. + +## The Three Layers + +### 1. Clock And Path Layer + +Questions: + +- Where is this sample on the canonical timeline? +- How much sample-rate offset is present? +- What delay, phase, and group-delay shape does this path impose? + +Tools: + +- active chirp-bin/codebook anchors; +- passive GCC-PHAT on program audio; +- loopback as emitter-side timing authority; +- Farrow/ASRC actuator. + +Output: + +- delay and SRO state; +- per-band magnitude and phase response; +- confidence and residuals. + +This layer must work before any field reconstruction deserves trust. + +### 2. Source/Geometry Layer + +Questions: + +- Which source positions explain the synchronized microphones? +- Which reflections are direct path versus room response? +- What evidence should become a stable field claim? + +Tools: + +- TDOA; +- SRP-PHAT; +- beamforming; +- known mic geometry; +- visual constraints from cameras/Leap when available. + +Output: + +- source position candidates; +- source confidence; +- reflection/room residuals; +- constraints for Fensalir's temporal evidence reservoir. + +### 3. Field Reconstruction Layer + +Questions: + +- What pressure/energy field exists in the room? +- Can the field be represented as sources, ambisonics, equivalent sources, or a + hybrid? +- What should OBS get: isolated stems, spatial bed, diagnostics, or all three? + +Tools: + +- ambisonics/spherical harmonics for compact field representation where geometry + fits; +- equivalent source method / near-field acoustic holography when treating sound + as sparse physical sources; +- compressed sensing or Bayesian sparse recovery when source count is low and + mic count is limited; +- learned priors only after deterministic models fail cleanly. + +Output: + +- aligned stems; +- sound-source tracks; +- spatial bed/ambisonic representation; +- visualizable acoustic field constraints. + +## Why Six Mics Are Not An Eigenmike + +A commercial spherical array might have dozens of capsules in known geometry. +Mimir has an irregular consumer/pro-audio constellation: Scarlett mic, camera +mics, possibly phone/Raven mics, and loopback. That pushes the machine away from +pure textbook ambisonics and toward an evidence model: + +- synchronize first; +- calibrate each path; +- infer source constraints; +- reconstruct only what the geometry supports. + +If the geometry cannot support a high-order field, the honest output is lower +order or sparse-source estimates with uncertainty. Pretending otherwise would +be the audio equivalent of drawing a ruler on fog and calling it engineering. + +## Candidate Models + +### Ambisonic Fit + +Use when: + +- mic geometry can support a stable low-order spherical harmonic fit; +- the goal is a spatial bed, not precise source localization; +- the scene is diffuse or ambient. + +Risk: + +- irregular sparse mics produce unstable coefficients; +- near-field sources and room boundaries violate simple assumptions. + +### Beamforming / SRP-PHAT + +Use when: + +- seeking source direction/position; +- synchronized mic geometry is known; +- direct-path energy exists. + +Risk: + +- reflections can produce plausible false peaks; +- small baseline limits low-frequency localization. + +### Equivalent Source / Sparse Acoustic Holography + +Use when: + +- sources can be represented sparsely; +- a grid or source dictionary is acceptable; +- response calibration is available. + +Risk: + +- inverse problems are ill-conditioned; +- dense grids are expensive; +- sparse recovery can hallucinate when the measurement model is wrong. + +### Hybrid Evidence Field + +Use when: + +- Mimir has mixed-quality sensors; +- visual evidence can constrain likely source regions; +- acoustic output should remain honest about uncertainty. + +Shape: + +- clock/path layer feeds calibrated observations; +- localization layer creates candidates; +- Fensalir temporal reservoir stabilizes candidates across time; +- DSP/output layer uses stable tracks as weights, not commandments. + +This is the likely Mimir path. + +## Practical First Milestone + +Build one static acoustic source proof: + +1. Record loopback and all live mics while emitting calibrated active witness. +2. Recover canonical anchors on loopback and at least one physical mic. +3. Estimate per-mic delay/SRO/response. +4. Use known speaker/mic geometry to form a TDOA/SRP-PHAT position hypothesis. +5. Lower that as an `AquariumAcousticConstraint`. +6. Display/log the constraint without letting it alter program audio yet. + +The proof is deliberately modest. It forces the field machine to earn geometry +from timing instead of leaping straight into theatrical volumetric fog. + diff --git a/research/perfect-machine-study-2026-05-23/architecture-rumination.md b/research/perfect-machine-study-2026-05-23/architecture-rumination.md index 2a6f736..3d98312 100644 --- a/research/perfect-machine-study-2026-05-23/architecture-rumination.md +++ b/research/perfect-machine-study-2026-05-23/architecture-rumination.md @@ -6,6 +6,9 @@ dedicated pass mapping, researching, distilling, ruminating, and writing sample options. - Active pass start recorded locally: `2026-05-23T23:41:19+01:00`. +- Corrected no-shortcut pass start recorded locally after the final instruction: + `2026-05-23T23:54:42.9041588+01:00`. Do not treat this directory as complete + until at least `2026-05-24T01:54:42.9041588+01:00`. ## Prime Cut @@ -210,4 +213,3 @@ Reason: that is how bandwidth dies in a formal suit. - Treating 192 kHz as magic: it improves timing granularity and ultrasonic options only where transducers and path response actually preserve signal. - diff --git a/research/perfect-machine-study-2026-05-23/benchmark-plan.md b/research/perfect-machine-study-2026-05-23/benchmark-plan.md new file mode 100644 index 0000000..05f8cc1 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/benchmark-plan.md @@ -0,0 +1,274 @@ +# Benchmark Plan + +## Purpose + +This is the measurement spine for future implementation passes. It exists to +keep the machine honest: no hot-path change earns production status because it +sounds clever. It earns it by moving measured latency, jitter, throughput, +allocation, cache pressure, confidence, or reconstruction quality in the right +direction. + +## Global Rules + +- Measure wall-clock throughput and canonical timing error separately. +- Keep synthetic, artifact, and live-device runs distinct. +- Always record sample rate, buffer size, device clock, channel mapping, and + mode (`passive`, `chirp-only`, `hybrid`). +- Allocation-free claims require runtime allocation counters, not code vibes. +- A passing self-test is a unit proof, not a physical proof. +- A physical proof without artifact retention is only a campfire story. + +## Audio Synchronization Benchmarks + +### Synthetic Chirp-Bin Decoder + +Goal: prove canonical-time recovery without acoustic path damage. + +Inputs: + +- generated chirp-bin timeline; +- known integer delay; +- known fractional delay; +- optional sample-rate offset; +- optional colored noise and band dropouts. + +Metrics: + +- decoded anchors per second; +- false anchors per second; +- delay error in microseconds; +- fractional sample residual; +- SRO estimate error in ppm; +- allocations per second; +- CPU cycles or elapsed time per decoded second. + +Current command family: + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --standalone-chirp-bin-self-test --sample-rate 192000 --delay-samples 96000 +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 192000 +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --hybrid-sync-self-test --sample-rate 192000 +``` + +Missing benchmark: + +- a loop over SNR, reliable-bin count, fractional delay, and SRO; +- output JSON/CSV artifact; +- CI-friendly threshold for the synthetic path. + +### Scarlett Loopback Artifact + +Goal: prove the analyzer survives the real ASIO clock and converter path when +the signal remains electrical/loopback-clean. + +Inputs: + +- rendered mono float chirp-bin witness; +- ASIO output through Scarlett; +- ASIO capture of loopback channels; +- optional persisted calibration profile. + +Metrics: + +- loopback channel-to-channel delay; +- confidence; +- anchor count; +- response matrix diagonal strength; +- per-band phase/group-delay residual; +- callback block jitter; +- dropped/late ASIO blocks. + +Current command family: + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --render-chirp-bin-f32 --sample-rate 192000 --seconds 6 --output .\artifacts\chirp-bin.f32 +.\native\probes\asio_audio_cadence\build\Release\asio_audio_cadence.exe --play-f32-mono .\artifacts\chirp-bin.f32 --record-f32-interleaved .\artifacts\scarlett.f32 --sample-rate 192000 --seconds 6 +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --analyze-asio-f32 .\artifacts\scarlett.f32 --sample-rate 192000 +``` + +Missing benchmark: + +- stable artifact directory naming; +- automatic device/channel manifest capture; +- repeated run variance. + +### Meatspace Mic Path + +Goal: prove acoustic propagation, monitor response, mic response, room +reflections, and gain staging still permit deterministic code-valid anchors. + +Inputs: + +- Scarlett output through monitors; +- Scarlett mic and camera mics; +- calibration profile generated from the same room/output/mic path; +- optional passive music bed. + +Metrics: + +- usable bands per mic; +- confusion matrix entropy; +- strongest alias bins; +- group-delay slope by bin; +- decoded anchors per second; +- rejected anchors and why; +- global delay/bin-shift hypothesis likelihood; +- time-to-lock and lock-hold duration; +- SRO drift estimate stability. + +Acceptance: + +- a calibrated physical mic should produce stable canonical anchors within the + five-second rolling buffer; +- delay should be reported in microseconds with residuals low enough for a + fractional-delay actuator to matter; +- dead bins must be excluded or downweighted by the emitted codebook, not merely + filtered after failure. + +## Audio Field Benchmarks + +### Passive Program-Audio Alignment + +Goal: use loopback music as continuous timing evidence when active chirps are +unnecessary. + +Metrics: + +- PHAT peak sharpness; +- delay confidence over time; +- failure rate on low-frequency-only material; +- behavior during silence; +- agreement with active chirp anchors. + +Cut line: + +- passive alignment is advisory unless it agrees with active/codebook anchors or + repeated state earns confidence. + +### Acoustic Source Localization + +Goal: turn synchronized mics into spatial estimates rather than just aligned +waveforms. + +Inputs: + +- known source positions if available; +- synthetic room model for baseline; +- live mic geometry. + +Metrics: + +- TDOA residual; +- SRP-PHAT peak width; +- source position error; +- update rate; +- ambiguity count. + +First proof: + +- one static speaker position recovered from loopback/chirp evidence and mic + geometry. + +## Video Capture Benchmarks + +### Native Camera Cadence + +Goal: prove each camera's raw driver path maintains sustained cadence without +framework buffering lies. + +Metrics: + +- delivered fps after warm-up; +- frame interval p50/p95/p99; +- missing sequence count; +- device timestamp monotonicity; +- CPU time in callback; +- transfer size and USB topology. + +Known baselines: + +- Leap full-height stereo IR: roughly 110 fps when the USB neighborhood is not + polluted by the Focusrite. +- PS3 Eyes: roughly 187 fps at 320x240 and roughly 58-60 fps at 640x480. +- Kiyo Pro: currently about 25 fps despite 60 fps advertisement and HighSpeed + negotiation. +- Kiyo basic: about 30 fps in current usable modes. + +### Multi-Camera Contention + +Goal: discover whether frame loss is USB topology, driver scheduling, CPU copy, +or runtime ingest pressure. + +Metrics: + +- per-device fps with all sources open; +- callback jitter correlation across devices; +- host controller/root hub mapping; +- frame-size bandwidth estimate versus observed cadence; +- runtime buffer enqueue cost. + +First implementation proof: + +- all local cameras feed `Mimir.Runtime` descriptors in one app run with no + JSON process bridge. + +## Fensalir Lowering Benchmarks + +Goal: prove Mimir can create Fensalir contract frames every render tick without +turning the app into a garbage factory. + +Metrics: + +- lowering time per frame; +- managed allocations per frame; +- number of sensor descriptors lowered; +- D3D12 upload/import wait time once connected; +- stale observation count. + +Acceptance: + +- lowering is pure mapping over current buffer snapshots; +- analysis/capture does not run from the render tick; +- UI reads cached state only. + +## Gaussian Splat / Sensor Fusion Benchmarks + +Goal: decide which visual fusion representation deserves production attention. + +Metrics: + +- observations accepted per second; +- splat claims updated per second; +- reprojection error; +- track lifetime; +- sort/tile/render cost; +- memory footprint per claim; +- drift under camera motion. + +Candidate approaches: + +- sparse feature claims lowered into Fensalir temporal evidence reservoir; +- live Gaussian claims updated from synchronized camera features; +- hybrid point/splat field where Mimir supplies timing and Fensalir owns + rendering/reuse. + +## Reporting Format + +Every serious benchmark should write one manifest next to artifacts: + +```json +{ + "machine": "Starfire", + "timestampUtc": "2026-05-24T00:00:00Z", + "mode": "chirp-only", + "sampleRate": 192000, + "bufferFrames": 192, + "sources": [], + "artifacts": [], + "metrics": {}, + "notes": [] +} +``` + +The manifest is not the hot path. It is the lab notebook. + diff --git a/research/perfect-machine-study-2026-05-23/calibration-session-spec.md b/research/perfect-machine-study-2026-05-23/calibration-session-spec.md new file mode 100644 index 0000000..d0112d0 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/calibration-session-spec.md @@ -0,0 +1,179 @@ +# Calibration Session Spec + +## Purpose + +The calibration command should be the machine that turns a real room into a +path model. It should not be a pile of one-off artifact commands that only the +current operator remembers how to arrange. + +## Objective + +Emit a known active witness, capture loopback and every microphone in the same +clock domain where possible, compute per-output/mic response/confusion/delay +models, persist them, and immediately prove that the live decoder improves when +those models are loaded. + +## Inputs + +- output path id; +- ASIO driver id / source manifest; +- sample rate; +- duration; +- emission mode (`chirp-only`, `hybrid-watermark`, `silent-passive-baseline`); +- candidate symbol plan; +- mic geometry file if available; +- artifact directory. + +## Outputs + +- raw capture artifact; +- rendered witness artifact; +- manifest with device/channel/sample-rate/buffer-size/gain notes; +- `MimirChirpBinCalibrationModel`; +- per-path usable-band table; +- confusion matrix; +- timing residual table; +- group-delay estimate; +- recommended codebook plan; +- before/after decode report. + +## Command Shape + +Future production command: + +```powershell +dotnet run --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- ` + --calibration-session ` + --config .\config\mimir-runtime.asio.json ` + --output-path scarlett-main-out ` + --seconds 30 ` + --sample-rate 192000 ` + --artifact-dir .\calibration\runs\2026-05-24-starfire +``` + +This can start as `BufferSmoke`, but the final owner should be a runtime command +or app action that uses the same ASIO/native path as production. + +## Session Phases + +### 1. Preflight + +- enumerate ASIO inputs/outputs; +- verify requested sample rate and buffer size; +- verify loopback channels are present; +- verify input channels are non-silent when expected; +- record channel names and source ids; +- load existing calibration for comparison. + +Fail fast if loopback is absent. Without loopback, the session can still record +diagnostics, but it cannot become canonical calibration. + +### 2. Passive Baseline + +Capture room/program audio without active witness for a short window. + +Purpose: + +- noise floor; +- current music/program spectrum; +- passive correlation strength; +- mic gain sanity. + +### 3. Active Emission + +Render a deterministic chirp-bin/watermark witness using the candidate symbol +plan. Capture loopback and all inputs. + +Important: + +- event ids and schedule seed go into the manifest; +- emitted codebook plan goes into the artifact; +- output gain should be recorded manually or through driver state if possible. + +### 4. Decode And Score + +For each input: + +- decode loopback anchors; +- decode mic anchors; +- compute matched anchor delays; +- compute expected-symbol versus observed-bin matrix; +- compute timing residuals; +- estimate magnitude response; +- estimate phase/group delay where phase evidence is stable; +- generate delay/bin-shift hypotheses. + +### 5. Plan Adaptation + +Across all selected mics: + +- identify reliable bins; +- identify dead/aliased bins; +- choose reduced alphabet; +- choose higher de Bruijn order if necessary; +- estimate expected time-to-unique-lock; +- write recommended emission plan. + +### 6. Replay Proof + +Reload the persisted model and decode the same artifact: + +- unweighted decode report; +- weighted/adapted decode report; +- anchor gain/loss; +- confidence gain/loss; +- delay residual gain/loss. + +If the calibrated model does not improve the capture, the command should say so +plainly. A calibration file that cannot improve the decoder is just decorative +paperwork with a badge. + +## Persisted Model Shape + +Minimum: + +```json +{ + "schema": "gamecult.mimir.chirp_bin_calibration.v1", + "createdUtc": "2026-05-24T00:00:00Z", + "sampleRate": 192000, + "outputPathId": "scarlett-main-out", + "emissionPlan": {}, + "paths": [ + { + "sourceId": "asio-ch0", + "usableBands": [], + "symbols": [], + "delayHypotheses": [], + "phaseModel": {}, + "groupDelayModel": {}, + "confidence": 0.0 + } + ] +} +``` + +## Acceptance Criteria + +- loopback decodes confidently; +- at least one physical mic produces code-valid anchors; +- calibration identifies reliable and unreliable bands; +- adapted emission plan changes the symbol set when the original bands are bad; +- loaded calibration affects likelihood before codebook solving; +- replay proof is stored with the model. + +## Failure Records + +A failed calibration is still useful if it records: + +- no loopback; +- silent mic; +- clipping; +- insufficient high-frequency response; +- strong alias/confusion; +- group delay too unstable; +- active witness too quiet/loud; +- room reflection dominated direct path. + +Do not delete failed runs unless they are accidental trash. They are evidence. + diff --git a/research/perfect-machine-study-2026-05-23/chirplet-transform-deep-dive.md b/research/perfect-machine-study-2026-05-23/chirplet-transform-deep-dive.md new file mode 100644 index 0000000..8a2171e --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/chirplet-transform-deep-dive.md @@ -0,0 +1,169 @@ +# Chirplet Transform And Controlled Chirp Codebook Deep Dive + +## Core Distinction + +There are two different machines that share similar math: + +1. A general chirplet transform asks: "what chirp-like atoms exist in this + signal?" +2. Mimir's active receiver asks: "which known emitted atom arrived when, through + which distorted path?" + +The second machine is much cheaper because the codebook, slopes, timing plan, +and symbol grammar are known. + +## Uses For Mimir + +### Timing Synchronization + +Each decoded code-valid chirp tuple can become a canonical timeline anchor. +With enough anchors, fit: + +- offset; +- sample-rate offset; +- confidence; +- jitter/residual. + +### Frequency Response Normalization + +Each classified chirp yields observed energy across bins. Aggregated over time: + +- reliable bands; +- dead bands; +- alias/confusion bands; +- path coloration. + +### Phase / Group Delay + +If phase is coherent enough, dechirped complex bin responses can estimate: + +- per-bin phase; +- phase slope across frequency; +- group delay correction; +- reflection/path stability. + +### Localization + +Known emitted time plus observed receiver time gives a distance constraint when +the source and receiver clocks are modeled. Multiple receivers turn this into a +position estimate. Multiple emitters/receivers turn it into room calibration. + +### Watermarking + +Hybrid mode can embed low-gain coded chirps under program audio. The watermark +should be: + +- sparse enough not to annoy; +- deterministic enough to identify time; +- adaptive enough to avoid dead acoustic bands; +- continuous enough to track drift. + +## Decoder Pipeline + +```mermaid +flowchart TD + PCM["PCM stream"] --> Energy["streaming energy/onset proposal"] + Energy --> Cand["candidate event windows"] + Cand --> Dechirp["dechirp with known slope"] + Dechirp --> Bins["bin / Goertzel / FFT score"] + Bins --> Likelihood["calibration-weighted likelihood"] + Likelihood --> Grammar["codebook grammar / de Bruijn tuples"] + Grammar --> Clock["clock fit"] + Clock --> Model["response + delay model"] +``` + +## What The Current Code Has + +- Fixed-slope chirp-bin renderer. +- De Bruijn codebook plan. +- Energy/onset proposals. +- Dechirp/bin scoring. +- Ambiguity candidates. +- Global bin-shift hypotheses. +- Clock fit and source-offset refinement. +- Calibration model with response/confusion/delay/phase/adaptive codebook. + +## What Is Missing + +- Streaming candidate state. +- Calibration-weighted likelihood before path selection as the default physical + path, not just optional model support. +- Better group-delay model. +- Reduced reliable-bin physical proof. +- Native/SIMD/GPU scoring path after correctness is proven. + +## Algorithm Options + +### Option A: Goertzel Bank + +Best when: + +- bin count is small; +- sample window is short; +- CPU SIMD is available; +- frequencies are fixed. + +Pros: + +- No FFT plan overhead. +- Easy to weight per bin. +- Simple to port to AVX2. + +Cons: + +- Repeats work per bin. +- Less attractive if bin count/window count grows. + +### Option B: Dechirp + FFT + +Best when: + +- many bins; +- many candidates; +- batching is possible. + +Pros: + +- Canonical CSS receiver shape. +- Good fit for GPU/native FFT batches. + +Cons: + +- FFT overhead and memory motion can dominate small cases. + +### Option C: Chirp Z Transform + +Best when: + +- only a narrow frequency band is needed at high resolution; +- arbitrary frequency grid matters. + +Pros: + +- Efficient zoomed spectral analysis. + +Cons: + +- More complex than current fixed bins. +- Might be premature unless physical data shows bin boundaries are the problem. + +### Option D: Matched Chirplet Atoms + +Best when: + +- emitted atoms are varied and not reducible to one common slope. + +Pros: + +- General. + +Cons: + +- Too expensive and too unconstrained for the current controlled beacon. + +## Implementation Rule + +Build the controlled CSS-like receiver first. Pull in full chirplet machinery +only when the codebook stops being controlled or when a research proof shows a +specific invariant the controlled receiver cannot protect. + diff --git a/research/perfect-machine-study-2026-05-23/current-hotspot-audit.md b/research/perfect-machine-study-2026-05-23/current-hotspot-audit.md new file mode 100644 index 0000000..e6866c4 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/current-hotspot-audit.md @@ -0,0 +1,174 @@ +# Current Hotspot Audit + +## What This Is + +This is a static audit of the current repo for likely allocation, scan, and +driver-pressure hotspots. It is not a profiler output. Treat it as a list of +places to benchmark once the next implementation cut starts. + +## High-Risk Runtime Hotspots + +### `MimirAudioSynchronizationAnalyzer.Analyze` + +Why it is hot: + +- Called repeatedly by `MimirSynchronizationHub`. +- Extracts windows from rolling buffers. +- Converts PCM formats. +- Resamples candidate windows. +- Invokes passive or active decode. + +Risk shape: + +- `ExtractMonoWindow` builds a `List` and returns `ToArray`. +- Candidate windows can be re-extracted on every analysis step. +- Resampling allocates a new output array. +- Reports/traces allocate lists and records. + +Coherent next cut: + +- Add an analyzer scratch/context object owning reusable buffers. +- Add streaming active decoder state so repeated windows are not rescanned. +- Cache reference extracted window/spectrum per analysis edge. + +### `MimirChirpBinTimeline.DecodeStreamWindow` + +Why it is hot: + +- Current active path for chirp-only/hybrid sync. +- Runs event proposal, classification, anchor decode, clock fit. + +Risk shape: + +- `BuildWindowEnergyTrace` allocates prefix/output arrays. +- `DetectFrames` sorts proposals by energy. +- `ScoreBins` scores every symbol for each candidate. +- `DecodeAnchors` creates candidate windows and path arrays. +- LINQ/`ToArray` appears in several inner-ish paths. + +Coherent next cut: + +- Split into streaming proposal state and stateless symbol scorer. +- Replace LINQ in candidate/code paths with bounded arrays/spans once behavior + is frozen. +- Move calibration likelihood into the scoring stage before candidate pruning. + +### `MimirPassiveAudioSynchronizationEstimator.Estimate` + +Why it is hot: + +- Passive/hybrid mode can compare multiple sources. +- FFT arrays allocate per estimate. + +Risk shape: + +- `Complex[]` reference/candidate arrays per call. +- Hann/preemphasis recomputed per call. +- Reference FFT repeated for each candidate. + +Coherent next cut: + +- Scratch object with reusable complex buffers. +- Precomputed window. +- Reference spectrum cache per edge. +- Limit lag search by known physical/network horizon. + +### `MimirRuntime.QueueCalibrationTimeline` + +Why it is hot-ish: + +- Renders/encodes active witness audio inside runtime update. + +Risk shape: + +- Allocates PCM bytes and base64 every batch. +- Renders sine chirps with scalar `Math.Sin`. + +Coherent next cut: + +- For production, move witness generation to a reusable native/DSP oscillator or + pre-rendered segment ring. Keep runtime policy, move audio hot work. + +## Native Hotspots + +### `native/asio_capture/pushInputBlocks` + +Why it is hot: + +- Runs from ASIO callback. +- Copies every input channel into a queued `std::vector`. + +Risk shape: + +- Allocates/copies per channel block. +- Uses mutex-protected deque. + +Current acceptability: + +- Fine for proof and likely fine for four Scarlett channels at 192 kHz in short + term. + +Coherent next cut: + +- Preallocated block pool or SPSC ring if profiling shows callback pressure. + +### `native/probes/ks_camera_cadence/measureCandidate` + +Why it is hot: + +- Direct KS async read queue for camera cadence. + +Risk shape: + +- Diagnostic stdout and optional JSON events. +- Probe code owns too much CLI behavior for production. + +Coherent next cut: + +- Extract the async read queue shape into a direct driver worker, minus stdout + and measurement ceremony. + +### `native/reservoir/RollingReservoir` + +Why it matters: + +- Intended lower shared-edge retention authority. + +Risk shape: + +- Current Rust `VecDeque` and typed-view iteration are conceptually clean but + not necessarily the final cache-optimal layout. + +Coherent next cut: + +- Keep invariant. Replace storage only when Fensalir/Faust integration proves + the need. + +## Allocation Smells Found By Static Scan + +Likely production-relevant: + +- `MimirChirpBinTimeline`: repeated `List<>`, `ToArray`, sorting, grouping in + decode path. +- `MimirAudioSynchronizationAnalyzer`: window extraction and resampling arrays. +- `MimirPassiveAudioSynchronizationEstimator`: per-call FFT arrays. +- `MimirAsioStreamSource`: per-block byte arrays after native read. +- `MimirRollingStreamBuffer.Snapshot`: full array snapshot for readers. + +Mostly diagnostic: + +- `Mimir.BufferSmoke` allocations. +- Native probe stdout/JSON allocations. +- Bridge scripts. + +## Rule For Next Optimization + +Do not optimize the whole repo. Pick one measured hot path: + +1. physical chirp-bin decode; +2. passive FFT reuse; +3. ASIO queue pressure; +4. camera driver handoff. + +Then make one ownership-improving cut and benchmark before widening. + diff --git a/research/perfect-machine-study-2026-05-23/data-dictionary.md b/research/perfect-machine-study-2026-05-23/data-dictionary.md new file mode 100644 index 0000000..5b7cfe5 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/data-dictionary.md @@ -0,0 +1,214 @@ +# Data Dictionary + +## Purpose + +This dictionary names the data objects that keep recurring across code, docs, +samples, and planned commands. + +## Stream Descriptor + +Identity for one source stream. + +Fields: + +- source id; +- kind (`audio`, `video`, etc.); +- origin (`local`, `network`, `process`, `native`); +- human label; +- optional device/config metadata. + +Invariant: + +- one descriptor maps to one rolling buffer. + +## Stream Sample + +One runtime sample envelope. + +Fields: + +- descriptor/source id; +- timestamp; +- sequence; +- audio block descriptor or video frame descriptor; +- optional payload bytes/handle. + +Invariant: + +- samples are indexed by runtime buffers; they are not long-term history. + +## Audio Block Descriptor + +Metadata for a block of audio. + +Fields: + +- sample rate; +- channel count; +- sample format; +- frame count; +- timestamp; +- optional native handle. + +Invariant: + +- production payload should move toward native handles, not managed byte arrays. + +## Video Frame Descriptor + +Metadata for one camera frame. + +Fields: + +- width; +- height; +- pixel format; +- stride; +- device timestamp; +- byte length; +- optional native/GPU handle. + +Invariant: + +- frame descriptor must preserve enough information for Fensalir to import or + upload without guessing. + +## Chirp Symbol + +One emitted active-witness event. + +Fields: + +- symbol id; +- frequency bin; +- chirp slope/window/duration; +- expected event time; +- optional rhythmic/gap evidence. + +Invariant: + +- symbol shape is controlled by the emitter and known by the receiver. + +## Chirp Anchor + +A decoded observation that maps a local sample offset to a canonical event. + +Fields: + +- event index; +- symbol id; +- local sample offset; +- canonical time; +- confidence; +- residual/error; +- bin shift hypothesis. + +Invariant: + +- anchors are the bridge between local sample time and canonical time. + +## Clock Fit + +Affine map between local samples and canonical time. + +Fields: + +- source offset samples; +- effective sample rate; +- confidence; +- anchor count; +- mean absolute error. + +Invariant: + +- clock fit is evidence state, not a sample-moving actuator. + +## Calibration Path + +Per-output/mic model. + +Fields: + +- output path id; +- source/mic id; +- sample rate; +- usable bands; +- symbol calibrations; +- confusion observations; +- delay hypotheses; +- phase/group-delay summaries; +- codebook plan. + +Invariant: + +- calibration alters likelihood and emission planning before decode failure + happens. + +## Sync Report + +Analyzer output for one source relative to reference. + +Fields: + +- source id; +- delay samples; +- delay milliseconds/microseconds; +- fractional delay; +- confidence; +- band response; +- decode trace. + +Invariant: + +- report describes measurement; DSP decides how to move audio. + +## Sync State + +Smoothed runtime belief. + +Fields: + +- latest delay; +- smoothed delay; +- confidence; +- delay slope / SRO ppm; +- per-band evidence. + +Invariant: + +- UI reads this; analyzers write it; render tick does not recompute it. + +## Acoustic Constraint + +Fensalir-facing spatial/timing evidence. + +Fields: + +- constraint id; +- kind; +- position/velocity/radius; +- confidence; +- timestamp; +- timing uncertainty. + +Invariant: + +- raw PCM stays out of Fensalir contract frames. + +## Visual Observation + +Fensalir-facing camera evidence. + +Fields: + +- source id; +- native/shared texture handle or payload handle; +- pixel format; +- dimensions; +- calibration id; +- canonical timestamp. + +Invariant: + +- Fensalir owns GPU lifetime after import/lowering. + diff --git a/research/perfect-machine-study-2026-05-23/decoder-architecture-options.md b/research/perfect-machine-study-2026-05-23/decoder-architecture-options.md new file mode 100644 index 0000000..b144fee --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/decoder-architecture-options.md @@ -0,0 +1,196 @@ +# Decoder Architecture Options + +## Purpose + +The active decoder must eventually be small enough to run on a phone, robust +enough for meatspace microphones, and cheap enough not to eat a CPU core per +stream. This file separates the receiver shapes so the next cut can be made on +invariants instead of frustration. + +## Current Runtime Shape + +`MimirChirpBinTimeline.DecodeStreamWindow` currently does the whole job for a +bounded PCM window: + +1. build an energy trace; +2. find candidate event windows; +3. classify each window with chirp-bin scoring; +4. preserve ambiguity candidates; +5. build code-valid anchors from the de Bruijn schedule; +6. search delay/bin-shift hypotheses; +7. fit an affine source clock; +8. emit calibration response/confusion evidence. + +This is coherent as a proof machine. It is not yet the final streaming receiver. + +## Option A: Streaming Scalar Receiver + +### Shape + +- Maintain a rolling PCM ring. +- Maintain an energy/proposal ring keyed by absolute sample index. +- When a proposal crosses threshold, score only the local candidate window. +- Keep the top K symbol hypotheses per proposal. +- Run a small codebook trellis over recent candidates. +- Update clock fit incrementally. + +### Why It Fits + +The emitter controls the symbol vocabulary and schedule. The receiver does not +need to search an unconstrained chirplet dictionary. It needs to notice likely +events and map them through a codebook. + +### Risks + +- Scalar scoring may still be too expensive at 192 kHz across many mics. +- Proposal thresholds can hide weak physical events if calibration is not used + before rejection. +- Trellis state must preserve ambiguity, not collapse to one wrong symbol. + +### Promote When + +It matches the current window decoder on synthetic and artifact captures while +reducing repeated work by at least an order of magnitude. + +## Option B: SIMD Dechirp + Goertzel Receiver + +### Shape + +- Precompute symbol reference chirps or dechirp oscillators. +- Process candidate windows in AVX2/FMA blocks. +- Accumulate real/imag Goertzel-style bin responses. +- Produce dense scores for the codebook trellis. + +### Why It Fits + +The hot math is mostly multiply-add and reduction over short windows. That is +exactly where SIMD is boring and useful. Boring and useful beats clever and +fragile here. + +### Risks + +- Small windows and few bins may be memory-bound or dispatch-bound rather than + arithmetic-bound. +- Managed/native transition cost can erase wins if called per candidate instead + of in batches. +- Alignment and denormal behavior need explicit treatment. + +### Promote When + +Profiling shows scalar C# scoring dominates, and a batched native scorer is +faster after interop overhead is included. + +## Option C: Dechirp + FFT/CZT Receiver + +### Shape + +- Dechirp candidate windows against known slope. +- Use FFT or chirp-Z transform to estimate symbol bin/frequency offset. +- Feed bin likelihoods into codebook path selection. + +### Why It Fits + +CSS/LoRa receivers validate this family. It also handles frequency offset and +bin interpolation naturally when the symbol set is dense. + +### Risks + +- FFT setup and memory traffic can dominate for 32 bins and small windows. +- A full FFT is wasteful if only a small fixed bin bank is needed. +- CZT is attractive for narrowband zooming, but more complex than necessary + until measured bin interpolation is the problem. + +### Promote When + +The codebook expands, frequency offset becomes first-order, or batched GPU/native +FFT plans beat SIMD Goertzel on actual candidate batches. + +## Option D: Full Chirplet Transform / Fast Chirplet Transform + +### Shape + +- Search a chirplet parameter space over time, center frequency, duration, + chirp rate, and scale. +- Use fast chirplet transform variants or maximum chirplet decomposition to + locate arbitrary chirp-like atoms. + +### Why It Fits + +It is the canonical general tool when the emitted atoms are unknown or when we +are analyzing natural signals. + +### Why It Does Not Fit The Hot Path Yet + +Mimir controls the emitter. Searching a large affine dictionary to rediscover a +known alphabet is buying a forklift to move a teacup. Good forklift, wrong +room. + +### Promote When + +Only if the controlled-codebook receiver cannot survive physical paths even +after response weighting, group-delay correction, and codebook adaptation. + +## Option E: Learned / Adaptive Receiver + +### Shape + +- Train or fit a compact classifier on calibration captures. +- Use learned symbol likelihood surfaces and delay priors. +- Keep codebook trellis deterministic after likelihood generation. + +### Why It Fits + +Physical paths produce systematic distortion: dead bands, aliases, reflections, +group delay, and mic-specific response. A learned likelihood surface may capture +what hand weighting misses. + +### Risks + +- A learned classifier can become an uninspectable compensator. +- Training data may overfit one room/gain setting. +- It must not replace the deterministic schedule map. + +### Promote When + +Calibration-weighted deterministic decoding fails in repeatable, explainable +ways that a compact trained likelihood model can resolve on held-out captures. + +## Recommended Cut + +Build Option A first, but design the scoring call as a replaceable batched +kernel so Option B or C can occupy it without changing ownership: + +```text +PCM ring -> proposal ring -> batched score kernel -> ambiguity trellis -> clock fit +``` + +Only the score kernel should vary. The codebook, schedule, calibration model, +and clock fit remain the receiver's authority. + +## Phone Receiver Shape + +A phone should need: + +- source codebook id; +- current schedule seed/epoch; +- sample rate estimate; +- rolling PCM buffer; +- compact path calibration if known. + +It should not need: + +- Starfire round trips for each event; +- Raven transport timing; +- Fensalir GPU state; +- OBS state. + +The phone receiver emits: + +- decoded anchor observations; +- fitted local clock to canonical time; +- confidence/residuals; +- response/confusion profile. + +That profile can come home later. The phone does not need to be trusted as a +clock authority; it is a self-locating sensor. + diff --git a/research/perfect-machine-study-2026-05-23/distributed-receiver-spec.md b/research/perfect-machine-study-2026-05-23/distributed-receiver-spec.md new file mode 100644 index 0000000..29c72f8 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/distributed-receiver-spec.md @@ -0,0 +1,150 @@ +# Distributed Receiver Spec + +## Purpose + +Mimir must work when Raven, a phone, or another networked sensor hears the same +active witness late, compressed, resampled, and out of Starfire's immediate +clock domain. The active codebook is meant to make that possible. + +## Objective + +A remote receiver that knows the source codebook and schedule should be able to +place local samples on the canonical timeline without round-tripping every +sample through Starfire. Network timestamps are metadata. Decoded anchors are +the timing truth. + +## Receiver Inputs + +- codebook id; +- schedule seed/epoch; +- nominal sample rate; +- local audio stream; +- optional path calibration; +- optional expected frequency band limits; +- optional network time estimate for coarse acquisition only. + +## Receiver State + +Minimum: + +- rolling PCM ring; +- proposal ring; +- candidate symbol ring; +- codebook trellis state; +- local clock fit to canonical time; +- response/confusion accumulator; +- health counters. + +It does not need: + +- Fensalir render state; +- OBS state; +- Starfire ASIO state; +- per-sample network acknowledgements. + +## Receiver Output + +```json +{ + "sourceId": "phone-left", + "codebookId": "main-room-2026-05-24", + "canonicalClockFit": { + "localSampleAtEpoch": 12345678.5, + "effectiveSampleRate": 47999.92, + "confidence": 0.91, + "meanAbsErrorSamples": 0.37 + }, + "anchors": [], + "responseProfile": {}, + "health": {} +} +``` + +The output is compact enough for LAN transport. Raw audio can still be streamed +when needed, but the timing machine should not depend on raw audio arriving +with low latency. + +## Canonical Mapping + +The receiver fits: + +```text +local_sample = source_offset + canonical_seconds * effective_sample_rate +``` + +The inverse places a local sample on the source timeline: + +```text +canonical_seconds = (local_sample - source_offset) / effective_sample_rate +``` + +Any code-valid anchor improves the fit. A triplet or higher-order tuple +identifies which canonical event was heard. Repeated anchors estimate drift. + +## Acquisition Strategy + +1. Use broad energy/onset proposals to avoid scanning every symbol everywhere. +2. Score candidates through the calibrated symbol likelihood. +3. Preserve top-K ambiguities. +4. Search code-valid tuples against the known schedule. +5. Fit local clock. +6. Narrow future search around predicted event times. + +That last step is the efficiency win. Once locked, the receiver is not hunting +the whole universe; it is checking whether expected events arrived where the +current clock predicts. + +## Network Behavior + +Remote nodes should send: + +- periodic clock-fit summaries; +- decoded anchor batches; +- response/confusion updates; +- buffer health; +- optional compressed/raw audio windows for debugging or program use. + +Starfire should not assume: + +- network latency is stable; +- transport order is canonical order; +- remote capture sample rate is exact; +- packet timestamps are better than decoded anchors. + +## Raven Shape + +Raven is special because it is a co-stream/game machine with its own Scarlett. +It can: + +- hear/emit the same active witness; +- decode local clock against Starfire's schedule; +- stream compact anchor/response state to Starfire; +- optionally stream audio/video payloads into Starfire's rolling buffers with + canonical-time metadata attached. + +Raven should not: + +- do heavy soundfield reconstruction; +- become the global clock authority unless explicitly selected; +- depend on OBS or game timing as source truth. + +## Phone Shape + +A phone is the harsh receiver target: + +- unknown mic response; +- OS audio resampling; +- possible AGC/noise suppression; +- weaker CPU budget; +- awkward latency. + +If the phone can decode enough anchors to estimate canonical time, the codebook +is doing its job. If it cannot, the failure is likely response/codebook design, +not network transport. + +## Security / Identity Note + +Codebook possession should be enough to decode time, not enough to control +Mimir. Remote timing reports need source identity and trust policy before they +affect program output. + diff --git a/research/perfect-machine-study-2026-05-23/failure-mode-ledger.md b/research/perfect-machine-study-2026-05-23/failure-mode-ledger.md new file mode 100644 index 0000000..9734963 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/failure-mode-ledger.md @@ -0,0 +1,236 @@ +# Failure Mode Ledger + +## Purpose + +This ledger names failure modes as data-flow problems. The fix should generally +move authority or measurement to the right place, not add another after-the-fact +filter. + +## Active Witness Failures + +### Dead Frequency Band + +Symptom: + +- emitted symbol is consistently absent or weak on one mic. + +Coherent fix: + +- calibration marks band unreliable; +- codebook emission avoids it for that path or shared room plan; +- decoder weights it down before trellis selection. + +Bad fix: + +- classify as normal, then reject "outliers" later. + +### Symbol Alias + +Symptom: + +- expected symbol often scores as a neighboring bin. + +Coherent fix: + +- confusion matrix changes likelihood surface; +- global bin-shift hypotheses are searched as part of the window; +- codebook spacing/order adapts. + +Bad fix: + +- hard-code a special-case symbol remap without path identity. + +### Reflection-Dominated Timing + +Symptom: + +- anchors decode, but delay residuals jump between plausible paths. + +Coherent fix: + +- preserve multiple delay hypotheses; +- combine with mic/speaker geometry and repeated state; +- report uncertainty to Fensalir/DSP. + +Bad fix: + +- smooth the jump until it looks calm. + +### Gain / Clipping + +Symptom: + +- loopback or mic confidence collapses despite audible witness. + +Coherent fix: + +- calibration preflight checks clipping/noise floor; +- artifacts record gain state; +- witness amplitude adapts within safe bounds. + +Bad fix: + +- lower thresholds globally. + +## Passive Sync Failures + +### Silence + +Symptom: + +- passive estimator emits weak or meaningless delay. + +Coherent fix: + +- confidence falls; +- hybrid emits active watermark/chirp witness. + +Bad fix: + +- hold stale passive delay as if it were current truth. + +### Narrowband Music + +Symptom: + +- PHAT correlation peak is broad or ambiguous. + +Coherent fix: + +- passive confidence remains advisory; +- active witness or longer evidence window takes authority. + +Bad fix: + +- pretend a low-frequency beat gives microsecond timing. + +## Audio Actuator Failures + +### Delay Corrected, Drift Ignored + +Symptom: + +- streams align briefly and then walk apart. + +Coherent fix: + +- separate delay and SRO loops; +- drive ASRC/fractional delay continuously. + +Bad fix: + +- periodically snap delay back with audible discontinuity. + +### Actuator Chases Source Motion + +Symptom: + +- moving speaker/mic looks like sample-rate drift. + +Coherent fix: + +- slow SRO loop uses long-term clock evidence; +- fast TDOA/source motion remains separate. + +Bad fix: + +- one smoothing constant for everything. + +## Camera Failures + +### Advertised FPS Lies + +Symptom: + +- device reports 60/120 fps but delivers less. + +Coherent fix: + +- measure actual cadence after warm-up; +- record topology/control settings; +- treat measured cadence as truth. + +Bad fix: + +- trust media type advertisements. + +### Shared USB Collapse + +Symptom: + +- one device's cadence drops when another device appears. + +Coherent fix: + +- topology/cadence benchmark; +- distribute devices across controllers/hosts; +- record known bad neighborhoods. + +Bad fix: + +- add buffering and call latency acceptable. + +### JSON Bridge Becomes Production + +Symptom: + +- diagnostic frame events are relied on for live six-camera ingest. + +Coherent fix: + +- promote probe logic into native worker ABI; +- payload handles enter runtime/reservoir. + +Bad fix: + +- optimize JSON parsing. + +## Fusion Failures + +### CPU Pixel Furnace + +Symptom: + +- managed code demosaics/converts/processes every camera frame. + +Coherent fix: + +- push feature extraction/upload into Fensalir GPU path; +- keep CPU metadata small. + +Bad fix: + +- add worker threads until the thermal budget files a complaint. + +### Parallel Stable Track Owners + +Symptom: + +- Mimir and Fensalir both maintain stable visual/acoustic track identity. + +Coherent fix: + +- Mimir owns observations; +- Fensalir temporal evidence reservoir owns tracks after lowering. + +Bad fix: + +- add id synchronization glue. + +## Network Failures + +### Transport Timestamp Worship + +Symptom: + +- Raven/phone timing is derived from packet arrival time. + +Coherent fix: + +- remote receiver decodes codebook anchors locally; +- network time is coarse metadata only. + +Bad fix: + +- add latency compensation constants per sender. + diff --git a/research/perfect-machine-study-2026-05-23/fensalir-integration-map.md b/research/perfect-machine-study-2026-05-23/fensalir-integration-map.md new file mode 100644 index 0000000..461dcd6 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/fensalir-integration-map.md @@ -0,0 +1,180 @@ +# Fensalir Integration Map + +## Purpose + +Mimir should not grow a second renderer, second temporal evidence reservoir, or +second UI engine. Fensalir already owns the window, D3D12 device, render graph, +GPU sensor fusion packets, temporal Gaussian field, acoustic constraint packet, +debug chrome, and future Spout/program output. Mimir owns the physical truth: +device capture, codebook schedules, calibration state, rolling runtime buffers, +network feed ingress, and the decision about which evidence enters the field. + +The integration cut is therefore simple: + +- Mimir produces timestamped observations and calibration constraints. +- Fensalir lowers those observations into D3D12 resources and temporal fields. +- Mimir reads health/status and manages policy. +- OBS receives final program surfaces/stems, not raw driver noise. + +## Current Fensalir Surfaces + +This pass inspected `E:\Projects\Fensalir` because the engine moved out of this +repo and the old app-specific mental model is now stale. + +Relevant contract surfaces: + +- `AquariumSceneState.GpuSensorFrame` +- `AquariumSceneState.AcousticFieldFrame` +- `AquariumSceneState.CalibrationEventFrame` +- `AquariumSceneState.GpuFusionField` +- `AquariumSceneState.TemporalGaussianField` +- `AquariumRenderPlan` +- `AquariumShaderManifest` + +Relevant engine concepts: + +- D3D12 renderer owns swapchain, command lists, descriptor heaps, upload rings, + shader compilation, and render graph execution. +- `D3D12ExternalSensorTexture` is the natural landing point for imported camera + frames once the native capture path can provide a shareable GPU handle. +- `D3D12GpuSensorFusion.hlsl` and `D3D12TemporalGaussian.hlsl` already express + the shape of a GPU-resident fusion pass. +- `TemporalSpatialEvidenceReservoir` owns stable evidence tracks. Mimir should + not invent a parallel stable-key cache for visual/acoustic claims. +- `D3D12AcousticConstraintPacket` is the visible contract slot for decoded + audio geometry constraints. +- `D3D12Renderer` already converts `AquariumAcousticConstraint` and clap + calibration events into acoustic constraint packets, so Mimir should lower + decoded audio evidence into those contracts rather than creating a side + debug-only renderer path. +- Fensalir tests already assert that `AquariumSceneState` can carry a + `GpuSensorFrame` with shared texture metadata and an `AcousticFieldFrame` + with timing confidence, uncertainty, accumulation window, presentation delay, + and spatial constraints. + +## Target Pipeline + +```mermaid +flowchart TD + A["native camera callbacks"] --> B["Mimir.Runtime rolling buffer"] + C["ASIO callbacks"] --> B + D["network feed packets"] --> B + B --> E["Mimir calibration and synchronization kernel"] + E --> F["AquariumGpuSensorFrame"] + E --> G["AquariumAcousticFieldFrame"] + E --> H["AquariumCalibrationEventFrame"] + F --> I["Fensalir D3D12 fusion"] + G --> I + H --> I + I --> J["Temporal Gaussian / evidence field"] + J --> K["Program video / diagnostics"] +``` + +## Ownership Invariants + +- Fensalir owns GPU resource lifetime. Mimir may describe external memory, but + it must not build a private D3D12 lifetime system. +- Fensalir owns temporal evidence reuse once observations are lowered into the + shared reservoir. Mimir owns observation truth before lowering. +- Mimir owns calibration identity: device ids, mic/speaker path ids, codebook + generation, response/confusion matrices, delay/group-delay models, and + reliable-bin plans. +- Mimir owns the rolling buffer retention boundary. Fensalir can snapshot or + consume the live window, but it must not silently become the history owner. +- `AquariumAudioDocument.EnqueuePcm16Base64` is not a production Mimir audio + path. It is suitable for UI/demo audio, not 192 kHz ASIO timing evidence. + +## Concrete Lowering Contracts + +### Camera Frames + +Mimir source: + +- source id; +- canonical timestamp estimate; +- device timestamp; +- frame sequence number; +- dimensions and pixel format; +- native CPU pointer or GPU shared handle; +- calibration id and lens model; +- confidence/status bits. + +Fensalir target: + +- `AquariumGpuSensorFrame` +- `AquariumGpuSensorCamera` +- `AquariumExternalGpuTexture` + +The first production proof can use CPU upload because it is easier to inspect. +The final six-camera path wants GPU import or a pinned/native staging ring. + +### Acoustic Constraints + +Mimir source: + +- decoded chirp anchors; +- passive program-audio anchors; +- loopback canonical clock fit; +- mic delay/SRO estimates; +- per-band magnitude/phase response; +- TDOA/source localization candidates; +- confidence and residual statistics. + +Fensalir target: + +- `AquariumAcousticFieldFrame` +- `AquariumAcousticConstraint` + +The acoustic packet should not carry raw PCM. Raw PCM stays in the Mimir/native +audio reservoir; Fensalir receives constraints and optional debug surfaces. + +The field frame can also carry timing oracle values. Mimir should derive these +from the loopback/reference clock fit: + +- timing oracle timestamp; +- timing confidence; +- timing uncertainty in microseconds; +- accumulation window; +- presentation delay. + +That gives Fensalir a scalar timing truth for visualization while keeping the +per-source audio correction state in Mimir/Faust. + +### Calibration Events + +Mimir source: + +- emitted chirp/watermark event id; +- clap/impulse event id; +- detected mic/camera event timestamps; +- spatial hypothesis if visual evidence exists. + +Fensalir target: + +- `AquariumCalibrationEventFrame` +- `AquariumClapCalibrationEvent` + +The important cut: calibration events are high-confidence anchors, not a +separate timeline authority. + +## Avoided Machines + +- A Mimir-side render graph. Fensalir already owns it. +- A Mimir-side stable Gaussian track database. Fensalir's temporal evidence + reservoir is the correct owner once observations become tracks. +- A second app-specific debug panel inside Mimir. Mimir should provide state; + Fensalir should render the debugging UI. +- Base64 or JSON transport in hot local audio/video paths. Those remain + diagnostics and artifact tools. + +## Immediate Implementation Cut + +1. Add a small Mimir-to-Fensalir bridge class that builds an + `AquariumGpuSensorFrame` from current `MimirStreamBufferSet` video + descriptors. +2. Add an acoustic lowering class that turns current sync reports and + calibration profiles into `AquariumAcousticFieldFrame`. +3. Keep both as pure, allocation-measured mapping code. They should not read + devices, run analyzers, or own timers. +4. Benchmark one second of repeated lowering against a populated synthetic + buffer before connecting it to the app render tick. diff --git a/research/perfect-machine-study-2026-05-23/glossary.md b/research/perfect-machine-study-2026-05-23/glossary.md new file mode 100644 index 0000000..098b5b9 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/glossary.md @@ -0,0 +1,162 @@ +# Glossary + +This glossary is deliberately blunt. It is here so the next implementation pass +does not burn half an hour rediscovering which word owns which invariant. + +## Active Witness + +A signal Mimir deliberately emits so receivers can recover canonical time and +path response. In this repo the active witness is the chirp-bin/chirplet +timeline. It is not only a sync beep; it is also a calibration probe. + +## ASIO + +Low-latency pro-audio driver API. In Mimir, ASIO is the production path for +Scarlett loopback and mic capture because it can put output loopback and inputs +in the same driver clock domain at 192 kHz. + +## Canonical Timeline + +The emitter-owned time axis. A receiver that knows the codebook and schedule +should infer where its captured samples sit on this timeline. + +## Chirp Spread Spectrum + +A communications family that encodes symbols in chirps. LoRa/CSS demodulators +matter to Mimir because they show a mature version of dechirp, FFT/bin scoring, +timing/frequency-offset handling, and low-power receiver design. + +## Chirplet + +A localized chirp atom: time-limited, frequency-localized, and slope-bearing. +The generic chirplet transform searches a broad chirplet parameter space. Mimir +uses a constrained chirplet receiver because the emitter controls the codebook. + +## Codebook + +The set of allowed emitted symbols and their shapes. In Mimir this includes +frequency bin, glide direction/slope, duration, and rhythmic spacing. It should +adapt to the measured physical path by avoiding dead or ambiguous bins. + +## Confusion Matrix + +For each expected symbol, the observed energy distribution across possible +symbols/bins. This is how Mimir learns that a mic hears symbol 12 as symbol 10, +or does not hear a band at all. + +## De Bruijn Sequence + +A sequence where every length-N symbol tuple appears once in a cycle. Mimir uses +the idea so a small number of consecutive chirp events can identify a unique +timeline position. + +## Dechirp + +Mix a received chirp against the conjugate/reference chirp so its energy +collapses toward a tone/bin. This turns chirp classification into a cheap +frequency-bin measurement. + +## Farrow Filter + +A fractional-delay filter structure where coefficients are polynomial functions +of the desired fractional sample delay. Useful for continuous delay actuation +without rebuilding FIR kernels at audio rate. + +## Fensalir + +The engine repo. It owns windowing, rendering, D3D12, render graph execution, +GPU fusion packets, temporal Gaussian/evidence fields, debug UI, and eventual +program surface publication. + +## Frequency Response + +How a speaker/room/mic path changes signal magnitude by frequency. Mimir should +learn it from the active witness and use it to normalize both calibration and +program audio. + +## GCC-PHAT + +Generalized cross-correlation with phase transform weighting. A common robust +delay estimator. It is useful for passive program-audio alignment and TDOA, but +it is not a substitute for controlled codebook anchors. + +## Gaussian Splat + +A 3D/4D ellipsoidal radiance primitive. For Mimir, splats are a plausible +runtime field representation for fused camera evidence, but Fensalir should own +the GPU field and renderer. + +## Group Delay + +Frequency-dependent delay through a path. If high bands arrive shifted relative +to low bands, symbol timing and response normalization both suffer. The active +witness should estimate and correct this. + +## Hybrid Mode + +Use passive program-audio timing by default, emit active witness only when +confidence falls. In Mimir this should eventually be a shaped watermark, not a +burst of rude test tones. + +## Loopback + +Audio captured from the output path. Scarlett loopback is the current timing +authority because it observes what Mimir attempted to emit before acoustic room +damage. + +## Near-Field Acoustic Holography + +A family of techniques for reconstructing sound fields from microphone array +measurements. It matters because Mimir is not only aligning mics; it is trying +to infer a room-scale sound field from sparse sensors. + +## Passive Mode + +No active emitted witness. Mimir estimates delay from program audio such as +music/game sound. Useful when the content is rich; fragile during silence or +band-limited material. + +## PLL + +Phase-locked loop. In Mimir, a control-rate PLL can turn repeated delay +observations into delay and sample-rate-offset corrections for the audio +actuator. + +## Reservoir + +A bounded evidence store. The Mimir rolling buffer owns recent raw stream +history; Fensalir's temporal evidence reservoir owns stable resolved visual and +acoustic field claims after lowering. + +## Sample-Rate Offset + +Clock drift between devices, usually described in ppm. Delay alone aligns one +moment; SRO correction keeps streams aligned over time. + +## SPSC Ring + +Single-producer/single-consumer ring buffer. The natural shape for hot native +capture callbacks handing blocks to one runtime consumer without locks. + +## SRP-PHAT + +Steered response power with PHAT weighting. It searches source positions by +testing which spatial hypothesis best explains inter-mic phase/delay evidence. + +## TDOA + +Time difference of arrival. With known microphone positions, TDOA constraints +help localize sound sources. + +## Temporal Gaussian Field + +Fensalir's GPU-side field representation for temporally coherent splat-like +evidence. Mimir should feed it observations and constraints, not own its +internal rendering state. + +## Watermark + +An active witness shaped to be unobtrusive under program audio. The best Mimir +watermark still carries deterministic canonical-time identity and response +calibration energy. + diff --git a/research/perfect-machine-study-2026-05-23/hot-path-pseudocode.md b/research/perfect-machine-study-2026-05-23/hot-path-pseudocode.md new file mode 100644 index 0000000..f81e5ec --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/hot-path-pseudocode.md @@ -0,0 +1,159 @@ +# Hot Path Pseudocode + +## Purpose + +This note writes the intended hot paths as code-shaped prose. It is not +production code. It is a guard against implementation drift. + +## Streaming Active Decoder + +```text +onAudioBlock(source, absoluteStart, samples): + pcmRing.append(absoluteStart, samples) + for each analysisHop completed by this block: + energy = updateProposalEnergy(pcmRing, analysisHop) + if energy is plausible: + proposalRing.push(analysisHop, energy) + + candidates = proposalRing.readySince(lastScoredSample) + if candidates is empty: + return currentClockFit + + scoreRows = scoreKernel.scoreBatch(pcmRing, candidates, codebook, calibration) + for row in scoreRows: + ambiguityRing.push(row) + + anchors = trellis.solveRecent(ambiguityRing, schedule, calibration) + clockFit.update(anchors) + responseModel.observe(scoreRows, anchors) + return clockFit +``` + +Invariant: + +- proposal, scoring, schedule solving, and clock fit are separate phases; +- calibration changes likelihood before trellis solving; +- old PCM expires with the rolling window. + +## Calibration-Weighted Symbol Score + +```text +score(symbol, observedBinEnergy, phase, timing): + magnitude = normalize(observedBinEnergy) + responseWeight = path.usableBandWeight(symbol) + confusion = path.confusionProbability(expected=symbol, observed=bin) + phaseWeight = path.phaseCoherence(symbol, phase) + groupDelay = path.groupDelayCorrection(symbol) + correctedTime = timing - groupDelay + return likelihood(magnitude, responseWeight, confusion, phaseWeight, correctedTime) +``` + +Invariant: + +- dead bands lose authority before code selection; +- group delay changes event timing before clock fit; +- phase is evidence when stable, not mandatory when unstable. + +## Pairwise Delay From Anchors + +```text +estimateDelay(referenceAnchors, sourceAnchors): + matches = joinByEventIndex(referenceAnchors, sourceAnchors) + if enough(matches): + return weightedMean(source.sampleOffset - reference.sampleOffset) + + if bothClockFitsConfident: + return sourceFit.sampleFor(t) - referenceFit.sampleFor(t) + + return noReport +``` + +Invariant: + +- matched event anchors beat independent fits; +- independent fits must stay inside live horizon; +- no report is better than absurd confidence. + +## DSP Actuator Control + +```text +onSyncState(source, delaySamples, sroPpm, confidence): + if confidence < threshold: + holdOrFadeCorrection() + return + + pll.update(delaySamples, sroPpm) + actuator.setTargetDelay(pll.delay) + actuator.setResampleRatio(1.0 - pll.sroPpm * 1e-6) +``` + +Invariant: + +- control loop runs at control rate; +- sample movement runs at audio rate; +- drift and delay are not the same state. + +## Native ASIO Ring + +```text +onBufferSwitch(index, samplePosition): + block = ring.tryReserve() + if block is unavailable: + droppedBlocks++ + return + + block.sourceChannel = channel + block.startFrame = samplePosition + block.sampleRate = currentRate + convertOrCopyDeviceBuffer(block.payload) + ring.publish(block) +``` + +Invariant: + +- callback never allocates; +- queue is bounded; +- dropped blocks are counted, not hidden. + +## Camera Frame Ring + +```text +onReadComplete(slot): + frame = ring.tryReserve() + if frame unavailable: + droppedFrames++ + resubmit(slot) + return + + frame.source = deviceId + frame.sequence = sequence++ + frame.deviceTimestamp = slot.timestamp + frame.payloadHandle = slot.payload + frame.format = negotiatedFormat + ring.publish(frame) + resubmit(nextSlot) +``` + +Invariant: + +- read queue stays full; +- frame metadata is typed; +- payload ownership is explicit. + +## Fensalir Lowering + +```text +buildSceneState(runtimeSnapshot): + gpuSensors = lowerVideoDescriptors(runtimeSnapshot.videoWindow) + acoustic = lowerSyncAndCalibration(runtimeSnapshot.audioState) + calibrationEvents = lowerHighConfidenceEvents(runtimeSnapshot.events) + return AquariumSceneState(gpuSensors, acoustic, calibrationEvents) +``` + +Invariant: + +- lowering reads cached/snapshotted state; +- lowering does not poll devices; +- lowering does not run analyzers; +- Fensalir owns what happens after contract construction. + diff --git a/research/perfect-machine-study-2026-05-23/implementation-roadmap.md b/research/perfect-machine-study-2026-05-23/implementation-roadmap.md new file mode 100644 index 0000000..1d18b0a --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/implementation-roadmap.md @@ -0,0 +1,173 @@ +# Implementation Roadmap From Study + +## Goal + +Turn the study into implementation passes that improve coherence instead of +adding another ring of machinery around the current code. + +## Phase 1: Make Active Decode Streaming + +Objective: + +- Stop rescanning full active windows every analysis tick. + +Current mechanism: + +- Analyzer extracts mono windows and calls `MimirChirpBinTimeline.DecodeStreamWindow`. +- Decode rebuilds proposals/frames/anchors from the provided span. + +Intended mechanism: + +- A per-source active decoder owns a rolling PCM window, energy proposal ring, + candidate-frame cache, anchor trellis tail, and clock fit state. +- Analyzer asks the decoder for the current snapshot. + +Invariants: + +- Same accepted reports as current analyzer on synthetic and loopback captures. +- No UI/telemetry re-entry. +- Calibration weighting remains optional until physical proof is run. + +Cut line: + +- If streaming state becomes harder to explain than the current decode, cut it + back to proposal/frame caching only. + +Proof: + +- Synthetic chirp-only/hybrid tests still recover known delay. +- Stored loopback ASIO artifact still decodes 0 us. +- Allocation count per analysis tick drops. + +## Phase 2: Physical Calibration-Weighted Decode + +Objective: + +- Make meatspace chirp-bin decode use what calibration already learned. + +Current mechanism: + +- Calibration model can be loaded. +- Symbol weights, phase weights, group-delay correction, and bin-shift + hypotheses exist but need physical proof. + +Intended mechanism: + +- Symbol likelihood is shaped by reliability/confusion/phase before candidate + pruning. +- Global delay/bin-shift hypotheses score the whole observation window. +- Reduced reliable-bin codebook can be emitted and decoded. + +Invariants: + +- The codebook remains bijective over the intended horizon. +- Dead bands lose authority. +- Failed timing still emits calibration evidence. + +Proof: + +- Compare unweighted versus weighted decode on stored hot-mic/physical captures. +- Emit reliable-bin plan and measure anchor rate through the real monitor/mic + path. + +## Phase 3: Actuator Proof + +Objective: + +- Move from "we know the delay" to "samples are aligned." + +Current mechanism: + +- `MimirAudioSynchronizationState` estimates smoothed delay and SRO. + +Intended mechanism: + +- Native/Faust DSP consumes delay/SRO state. +- Fractional delay line corrects sub-sample offset. +- ASRC/PLL corrects drift. + +Invariants: + +- Runtime estimates; DSP moves samples. +- Actuator state is inspectable but not owned by UI. +- Output remains separately stemmable for OBS. + +Proof: + +- Offline artifact: apply actuator to delayed candidate and reduce residual. +- Live loopback/mic: show residual delay converges and remains bounded. + +## Phase 4: Direct Camera Driver Runtime Path + +Objective: + +- Replace diagnostic frame-event probe transport for local cameras. + +Current mechanism: + +- KS/PS3 probes can emit JSON frame events. +- Runtime has `IMimirVideoCaptureDriver` and source adapter seam. + +Intended mechanism: + +- One native/direct driver worker per camera family. +- Capture worker preserves timestamp and payload handle. +- Runtime indexes sample envelope. +- Fensalir consumes handle/window for GPU feature extraction. + +Invariants: + +- No local camera pixels through stdout/base64. +- Device/arrival timestamp survives the boundary. +- Capture worker owns device pressure, runtime owns identity and buffer. + +Proof: + +- Leap or PS3 Eye frames enter runtime buffers with sustained measured cadence. +- Fensalir UI can inspect buffer health without touching pixel hot path. + +## Phase 5: Native Reservoir/Fensalir/Faust Integration + +Objective: + +- Make the native reservoir the shared lower memory/timing authority. + +Current mechanism: + +- C# buffers own runtime proof state. +- Rust reservoir exists with C ABI and tests. + +Intended mechanism: + +- Capture workers push handles into native reservoir. +- Runtime and Fensalir read typed views. +- Faust reads audio block handles and alignment state. + +Invariants: + +- One edge owns retention. +- Typed views do not own separate windows. +- Payload ownership stays with producer/consumer domain. + +Proof: + +- Runtime status matches C# buffer counts for a mirrored proof. +- Fensalir renders a simple current-window visual packet from reservoir claims. + +## Benchmark Harnesses Needed + +1. `chirp-bin-decode-bench`: synthetic and artifact decode timing/allocation. +2. `passive-sync-bench`: FFT allocation/reuse comparison. +3. `asio-callback-pressure-bench`: native queue/block pool comparison. +4. `camera-driver-buffer-bench`: direct driver payload handoff cadence. +5. `actuator-residual-bench`: before/after residual delay and drift. + +## Decision Gates + +- SIMD scorer only after decode timing is the measured bottleneck. +- GPU scorer only after candidate/bin batches exceed CPU SIMD advantage. +- Neural/acoustic-field models only after classical timing/calibration/source + localization is coherent. +- Dynamic 4DGS only after direct synchronized visual evidence exists in + Fensalir. + diff --git a/research/perfect-machine-study-2026-05-23/index.md b/research/perfect-machine-study-2026-05-23/index.md index f19e020..d5d1a5b 100644 --- a/research/perfect-machine-study-2026-05-23/index.md +++ b/research/perfect-machine-study-2026-05-23/index.md @@ -13,6 +13,32 @@ needs the reason a cut exists. | `architecture-rumination.md` | System ownership, current machine critique, coherent next cuts. | | `optimization-ledger.md` | Hot loop risks, low-level implementation options, micro-optimization candidates. | | `references.md` | Research links for chirplets, CSS/LoRa, GCC-PHAT, D3D12/FFT/GPU compute. | +| `current-hotspot-audit.md` | Static audit of likely runtime allocation/scan/driver-pressure hotspots. | +| `implementation-roadmap.md` | Phase plan, invariants, proofs, and benchmark harnesses. | +| `questions-and-hypotheses.md` | Open questions and falsifiable hypotheses by domain. | +| `reading-guide.md` | Task-oriented route through the study docs and samples. | +| `data-dictionary.md` | Data object definitions for streams, samples, chirp anchors, clock fits, calibration paths, reports, and Fensalir constraints. | +| `volumetric-audio-field.md` | Sound-field reconstruction decomposition and first practical audio target. | +| `acoustic-field-models.md` | Field reconstruction models and the first honest acoustic source proof. | +| `chirplet-transform-deep-dive.md` | Controlled chirp receiver versus generic chirplet transform. | +| `decoder-architecture-options.md` | Active receiver implementation options and promotion criteria. | +| `distributed-receiver-spec.md` | Raven/phone receiver shape for self-locating remote sensors using codebook and schedule state. | +| `visual-fusion-4dgs-study.md` | Sensor fusion and realtime Gaussian splatting architecture study. | +| `open-implementation-catalog.md` | Implementation families to study before inventing local machinery. | +| `prior-research-synthesis.md` | Distillation of the older chirplet, adaptive sync, ambisonic, splatting, calibration, and visual archives. | +| `research-claims-digest.md` | Claim-level digest of external literature and how it steers Mimir. | +| `fensalir-integration-map.md` | Contract-level cut between Mimir runtime truth and Fensalir GPU/render ownership. | +| `native-boundary-map.md` | Native/managed/Fensalir ownership boundaries for ASIO, cameras, reservoir, and GPU payloads. | +| `option-matrix.md` | Implementation alternatives, failure modes, and deciding proofs for the major subsystems. | +| `benchmark-plan.md` | Measurement plan for audio sync, acoustic field, camera capture, Fensalir lowering, and splat fusion. | +| `calibration-session-spec.md` | Command/session shape for learning and validating real output/mic path models. | +| `low-level-implementation-notes.md` | Concrete memory layout, SIMD/GPU, ASIO, camera, and instrumentation notes. | +| `microsecond-sync-math.md` | Timing math for sample period, bandwidth, SNR, subsample delay, and acoustic distance. | +| `state-machine-invariants.md` | Ownership limits for runtime, hub, buffers, decoders, calibration, DSP, capture, Fensalir, network, and OBS. | +| `failure-mode-ledger.md` | Failure modes and coherent fixes across active witness, passive sync, actuator, camera, fusion, and network paths. | +| `hot-path-pseudocode.md` | Code-shaped intended hot paths for streaming decode, calibration weighting, actuator, native rings, camera rings, and Fensalir lowering. | +| `glossary.md` | Short definitions for the synchronization, acoustic, visual, and reservoir terms used across the study. | +| `sample-code-index.json` | Machine-readable index for sample sketches. | | `samples/README.md` | Index of sample implementation sketches. | ## Sample Code @@ -21,9 +47,17 @@ needs the reason a cut exists. | --- | --- | --- | | `samples/StreamingChirpBinDecoderSketch.cs` | `MimirChirpBinStreamingDecoder` or equivalent runtime class | What state removes full-window rescans? | | `samples/Avx2DechirpGoertzelSketch.cpp` | Native CPU scorer for chirp-bin candidates | What does SIMD dechirp/bin scoring look like? | +| `samples/BatchedChirpScoreAbiSketch.h` | Native scorer ABI | What batch boundary avoids per-candidate managed/native overhead? | | `samples/ChirpBinScore.compute.hlsl` | Fensalir/D3D12 compute scorer | How would batched candidate/bin scoring map to GPU? | | `samples/FarrowFractionalDelaySketch.cpp` | Faust/native DSP actuator | What is the smallest fractional delay proof? | | `samples/SpscAudioBlockRingSketch.cpp` | Native capture worker handoff | What shape should low-jitter audio block transfer take? | +| `samples/CalibrationWeightedChirpLikelihoodSketch.cs` | Physical-path active decoder | Where does calibration enter symbol likelihood? | +| `samples/SroPllAsrcControllerSketch.cpp` | DSP actuator control | How does delay/SRO state become a resampler command? | +| `samples/GaussianSplatClaimUpdate.compute.hlsl` | Fensalir visual fusion | What is the smallest live splat-claim compute shape? | +| `samples/AquariumGpuSensorFrameBridgeSketch.cs` | Mimir-to-Fensalir visual lowering | How should runtime video observations become Fensalir sensor frames? | +| `samples/AcousticConstraintLoweringSketch.cs` | Mimir-to-Fensalir acoustic lowering | How should decoded anchors/geometry become acoustic constraints? | +| `samples/TemporalEvidenceCandidateSketch.cs` | Shared temporal evidence candidates | How can Mimir lower candidates without owning Fensalir tracks? | +| `samples/SrpPhatGridSketch.cpp` | Acoustic source localization | What is the smallest TDOA/SRP-PHAT-style grid scoring proof? | ## Best Next Implementation Sequence @@ -55,6 +89,8 @@ needs the reason a cut exists. - group delay - LoRa CSS - native reservoir +- Fensalir +- Gaussian splatting - passive sync - Raven - response/confusion matrix @@ -64,4 +100,3 @@ needs the reason a cut exists. - streaming decoder - volumetric audio - volumetric visual fusion - diff --git a/research/perfect-machine-study-2026-05-23/low-level-implementation-notes.md b/research/perfect-machine-study-2026-05-23/low-level-implementation-notes.md new file mode 100644 index 0000000..f418392 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/low-level-implementation-notes.md @@ -0,0 +1,199 @@ +# Low-Level Implementation Notes + +## Purpose + +This file collects the implementation details that are easy to lose when the +conversation stays at the architecture level. None of these notes are promises; +they are candidates to benchmark. + +## Audio Memory Layout + +### Preferred Analysis Layout + +For decoder kernels: + +```text +channel0 samples: [s0 s1 s2 ...] +channel1 samples: [s0 s1 s2 ...] +... +``` + +Separate channel spans make per-channel scoring straightforward and avoid +striding through interleaved audio during analysis. Interleaving is useful for +device I/O and some DSP kernels; it is not automatically the right analysis +shape. + +### Preferred Scoring Layout + +```text +candidateStartSamples[] +candidateEnergy[] +symbolReliability[] +binFrequencies[] +binSin[] +binCos[] +``` + +Dense arrays let SIMD/native/GPU kernels process candidates without object +chasing. The codebook trellis can stay managed/CPU because it is small and +branchy. + +## SIMD Chirp-Bin Scoring + +Likely CPU kernel shape: + +1. load candidate window samples in 8-float AVX2 chunks; +2. multiply by dechirp oscillator/window; +3. accumulate bin real/imag sums; +4. compute magnitude or normalized energy; +5. write top K or full dense bin vector. + +Important: + +- keep oscillator tables aligned; +- keep candidate windows contiguous; +- batch multiple candidates per interop call; +- avoid denormals; +- normalize outside the deepest loop where possible. + +## FFT / Goertzel Decision + +Use Goertzel/fixed-bin DFT when: + +- symbol count is small; +- bins are known; +- candidate count is modest; +- native dispatch overhead matters. + +Use FFT/CZT when: + +- many bins are needed; +- frequency offset interpolation dominates; +- candidate batches are large enough; +- GPU/native FFT plans can be reused. + +Do not choose FFT because it sounds more canonical. Canonical is not a profiler. + +## GPU Chirp Scoring + +The GPU scoring pass only makes sense if data is already resident or batches are +large enough to pay dispatch overhead. + +Good GPU candidates: + +- many mic streams; +- many remote receiver windows; +- wide candidate sets during acquisition; +- offline/artifact calibration sweeps. + +Bad GPU candidates: + +- one or two channels; +- tiny candidate counts; +- paths that require CPU readback before codebook solving every event. + +## ASIO Capture Hot Path + +Current observed risks: + +- per-block `std::vector` resize; +- `std::deque` queue; +- native-to-managed polling copies into `float[]`; +- managed byte array allocation for sample payload; +- `ConcurrentQueue.Count` pressure. + +Preferred cut: + +- native SPSC ring; +- fixed block pool; +- monotonic frame counters; +- block descriptors with handle/index; +- C# metadata only until DSP needs samples. + +## Camera Capture Hot Path + +Current probe strengths: + +- KS async queueing is already close to the driver; +- PS3 Eye raw path proves high fps; +- JSON frame events make cadence visible. + +Current probe risks: + +- JSON must not become production payload; +- CPU frame buffers are diagnostic until Fensalir import/upload path exists; +- one-off command-line knobs must become configuration/stateful driver policy. + +Preferred cut: + +- one native capture worker per camera family; +- fixed frame ring; +- descriptor ABI; +- optional shared GPU texture handle; +- Fensalir owns texture import and GPU staging. + +## Fensalir GPU Fusion + +Fensalir contracts already distinguish: + +- `GpuSensorFrame` for calibrated camera inputs and external textures; +- `AcousticFieldFrame` for timing/room constraints; +- `CalibrationEventFrame` for high-confidence event anchors; +- `GpuFusionField` for point buffers/seeds; +- `TemporalGaussianField` for stable gaussian observations. + +Mimir should produce these rows. It should not pack shader structs directly. + +## Gaussian Splat Rendering Levers + +From modern 3DGS implementations: + +- tile size matters; +- sorting strategy matters; +- cull before sort; +- packed buffers matter; +- anti-aliasing and opacity/radius thresholds matter; +- view-consistency fixes cost real time; +- depth/feature render modes are useful for diagnostics. + +Mimir-specific cut: + +- first update live claims; +- then render simple claims; +- only later optimize dynamic radiance fields. + +## Benchmark Counters To Add + +Decoder: + +- samples consumed; +- proposals created; +- candidates scored; +- anchors accepted/rejected; +- elapsed scoring time; +- allocations per analysis tick. + +ASIO: + +- callback count; +- block count; +- queue/ring high-water mark; +- dropped blocks; +- conversion time if measurable. + +Camera: + +- frame interval p50/p95/p99; +- async queue depth; +- dropped/late frames; +- payload copy time; +- upload/import time. + +Fensalir: + +- contract lowering time; +- upload time; +- GPU pass timings; +- temporal field count; +- acoustic constraint count. + diff --git a/research/perfect-machine-study-2026-05-23/microsecond-sync-math.md b/research/perfect-machine-study-2026-05-23/microsecond-sync-math.md new file mode 100644 index 0000000..f9ee635 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/microsecond-sync-math.md @@ -0,0 +1,102 @@ +# Microsecond Sync Math + +## Purpose + +The project target is microsecond-class synchronization. This note keeps the +math visible so we do not confuse sample-rate settings, transducer bandwidth, +and actual timing certainty. + +## Sample Periods + +| Sample Rate | Sample Period | +| --- | --- | +| 48 kHz | 20.833 us | +| 96 kHz | 10.417 us | +| 192 kHz | 5.208 us | + +Subsample estimation is mandatory for microsecond delay estimates at every +practical audio sample rate. 192 kHz helps by reducing the interpolation gap and +allowing higher witness frequencies, but it does not magically provide +one-microsecond truth. + +## What Enables Subsample Timing + +- known emitted waveform; +- high SNR; +- broad enough bandwidth; +- stable phase response; +- clean enough direct path; +- coherent clock model over multiple anchors; +- interpolation or model fitting around correlation/likelihood peaks. + +The active witness helps because its shape is known. Passive music can help when +it has enough broadband content, but it is not guaranteed. + +## Cramer-Rao Intuition + +Delay estimation improves with: + +- signal bandwidth; +- SNR; +- observation duration / repeated independent evidence. + +It degrades with: + +- narrowband signals; +- reverberation; +- unknown filtering; +- low SNR; +- nonlinear processing; +- clock drift; +- false symbol aliases. + +This is why the chirp signal has a dual role: it spreads energy across time and +frequency for timing, while also measuring which parts of the path survived. + +## 192 kHz Is Useful When + +- the interface can capture output and mics in one stable clock domain; +- the monitors/mics preserve enough high-frequency energy; +- phase/group delay can be calibrated; +- the actuator can use fractional delay/SRO information; +- the extra CPU/memory load stays within budget. + +## 192 kHz Is Not Useful When + +- the acoustic path low-passes the witness below the useful band; +- the mic/driver applies hidden processing; +- the decoder only reports integer samples; +- the system still lacks a resampling/fractional-delay actuator; +- the bottleneck is symbol confusion rather than sample quantization. + +## Timing Evidence Hierarchy + +1. Electrical loopback: best for emitted timeline and interface clock. +2. Physical mic active witness: best for acoustic path timing/response. +3. Passive program audio: useful continuous drift evidence when content allows. +4. Network timestamps: metadata, not truth. + +## Delay To Distance + +At roughly 343 m/s: + +- 1 us is about 0.343 mm of acoustic path length; +- 10 us is about 3.43 mm; +- 100 us is about 34.3 mm. + +That scale is useful and humbling. A room reflection or mic position error can +swamp the number we are trying to measure. The machine must keep uncertainty +visible. + +## Practical Acceptance + +For the next physical proof, "microsecond sync" should mean: + +- loopback pair recovers below printed microsecond precision; +- physical mic reports fractional-sample delay with stable residuals; +- repeated anchors show bounded SRO drift; +- delay uncertainty is reported, not hidden; +- actuator demonstrably reduces residual misalignment. + +Without the actuator, the system is measuring microseconds, not delivering them. + diff --git a/research/perfect-machine-study-2026-05-23/native-boundary-map.md b/research/perfect-machine-study-2026-05-23/native-boundary-map.md new file mode 100644 index 0000000..90c6531 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/native-boundary-map.md @@ -0,0 +1,127 @@ +# Native Boundary Map + +## Purpose + +Mimir's managed runtime should own identity, policy, and orchestration. It +should not sit inside the six-camera or 192 kHz audio hot loops doing avoidable +copies, JSON parsing, or queue churn. This file marks the boundaries that should +be native, managed, or Fensalir-owned. + +## Current Native Pieces + +### ASIO Capture + +`native/asio_capture/mimir_asio_capture.cpp` currently: + +- creates the Focusrite ASIO driver; +- configures input buffers; +- receives `bufferSwitch` / `bufferSwitchTimeInfo` callbacks; +- converts device sample formats to mono float blocks; +- queues `MimirAsioBlock` objects; +- exposes a polling ABI to C#. + +This proves in-process ASIO ingest. It is not the final zero-copy hot path: +`std::deque`, `std::vector`, per-block resize, and managed byte-array +copies are all visible future costs. + +### Camera Probes + +The KS probe currently uses low-level Kernel Streaming / AVStream surfaces: + +- property negotiation; +- async `NtDeviceIoControlFile` reads; +- queued read slots; +- frame JSON emission for diagnostics. + +The PS3 Eye probe uses the raw driver/library path and emits frame metadata. It +proves throughput and topology, not final app ingestion. + +### Reservoir + +`native/reservoir` is the lower-level memory invariant: one rolling edge, typed +views, no per-kind retention. This should become the shared payload/index owner +when C# descriptors become too expensive for full payloads. + +## Boundary Decisions + +| Concern | Owner | Reason | +| --- | --- | --- | +| ASIO COM driver calls | native | COM/callback lifetime and sample-format conversion do not belong in C#. | +| ASIO block timing/counters | native first, Mimir indexed | The callback sees the driver clock; Mimir maps it to canonical time. | +| Audio payload memory | native/Faust | Program DSP should not copy through managed byte arrays. | +| Audio sync reports | Mimir.Runtime | Reports are policy/belief, not raw DSP. | +| Camera device control | native | Driver calls, async reads, libusb, and vendor knobs are hardware work. | +| Camera payload memory | native/Fensalir | Pixels should land in native or GPU memory. | +| GPU texture lifetime | Fensalir | The engine owns D3D12 resources and synchronization. | +| Calibration JSON/artifacts | Mimir | Persistence and reproducibility are repo/runtime concerns. | + +## Desired ASIO Evolution + +```mermaid +flowchart TD + A["ASIO callback"] --> B["native SPSC block ring"] + B --> C["native reservoir audio view"] + C --> D["Mimir.Runtime metadata/index"] + C --> E["Faust/native DSP"] + D --> F["sync analyzer state"] + F --> E +``` + +Immediate cut: + +- replace `std::deque` with a bounded SPSC ring; +- preallocate per-channel block buffers; +- copy/convert once from ASIO format into the block ring; +- expose block handles or spans instead of repacking to managed arrays. + +Final cut: + +- keep raw/current device format where possible; +- run hot alignment and filtering in native/Faust; +- hand Mimir descriptors and diagnostics rather than payload copies. + +## Desired Camera Evolution + +```mermaid +flowchart TD + A["driver read completion"] --> B["native frame ring"] + B --> C["timestamp/sequence descriptor"] + B --> D["CPU pinned or GPU shared payload"] + C --> E["Mimir.Runtime rolling buffer"] + D --> F["Fensalir texture import/upload"] +``` + +Immediate cut: + +- one native worker per camera family; +- frame descriptor ABI with source id, sequence, timestamp, dimensions, format, + stride, byte length, and payload handle; +- no JSON in the production app path. + +Final cut: + +- D3D12-compatible shared resources where the driver path can support them; +- CPU staging only where direct GPU import is not practical; +- feature extraction as early GPU compute, not managed pixel processing. + +## External Guidance Anchors + +- Microsoft Kernel Streaming docs confirm KS/AVStream is the low-level Windows + stream surface beneath higher media frameworks. +- `IDXGIResource1::CreateSharedHandle` and D3D11/12 shared NT handles are the + relevant family for resource sharing once capture can land in shareable + textures. +- ASIO `bufferSwitchTimeInfo` / sample position is the right place to preserve + driver-clock timing, not a managed polling timestamp after the fact. +- Focusrite channel-order docs matter because loopback availability and channel + layout can change with sample rate and model. + +## Watch List + +- Per-block `new float[]` / `byte[]` in managed ingest. +- `ConcurrentQueue.Count` in hot loops. +- `std::deque` under callback pressure. +- One process per camera. +- Any "temporary" JSON bridge that survives once payloads matter. +- GPU resource ownership creeping into Mimir instead of Fensalir. + diff --git a/research/perfect-machine-study-2026-05-23/open-implementation-catalog.md b/research/perfect-machine-study-2026-05-23/open-implementation-catalog.md new file mode 100644 index 0000000..6c8f234 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/open-implementation-catalog.md @@ -0,0 +1,99 @@ +# Open Implementation Catalog + +This catalog lists implementation families worth stealing shape from. Do not +vendor code casually. The point is to understand mature cuts. + +## Chirp / CSS / LoRa + +What to look for: + +- dechirp implementation; +- symbol timing synchronization; +- FFT/bin selection; +- CFO/bin-shift correction; +- preamble/sync word handling; +- soft decision metrics. + +Candidate references: + +- OpenLoRa demodulator work. +- SDR LoRa implementations. +- GNU Radio LoRa/CSS blocks. +- Audio "data over sound" projects for acoustic channel constraints. + +Mimir translation: + +- Preamble becomes deterministic timeline/codebook schedule. +- CFO becomes acoustic bin shift / sample-rate mismatch. +- Channel response becomes output/mic path calibration. + +## Audio Delay / Beamforming + +What to look for: + +- GCC-PHAT implementations; +- SRP-PHAT acceleration; +- microphone array calibration; +- fractional delay filters; +- asynchronous sample-rate conversion. + +Candidate references: + +- Pyroomacoustics for algorithm shape. +- WebRTC audio processing for pragmatic realtime constraints. +- JACK/PipeWire/ASIO style callback boundaries. +- Faust examples for DSP graph ownership. + +Mimir translation: + +- Keep algorithm shape, not necessarily library dependency. +- Faust/native DSP is the actuator home. + +## Gaussian Splatting + +What to look for: + +- 3D Gaussian Splatting CUDA rasterizer. +- Dynamic/4D Gaussian splatting repos. +- Differentiable Gaussian rasterization data layouts. +- Real-time viewer/update pipelines. + +Mimir translation: + +- Fensalir owns render/update. +- Native reservoir owns time-indexed claims. +- Direct camera drivers feed evidence; do not train/reconstruct from disk + batches in the live path. + +## Native Capture + +What to look for: + +- ASIO callback isolation. +- Kernel Streaming async read queues. +- libusb/WinUSB transfer queues. +- D3D12 shared texture/resource handoff. +- lock-free SPSC rings. + +Mimir translation: + +- One capture worker owns one device. +- Payload handles cross subsystem boundaries. +- Runtime indexes metadata and timing. + +## SIMD/GPU Compute + +What to look for: + +- AVX2/FMA complex dot products. +- batched FFT plan reuse. +- HLSL wave reductions. +- CUDA/cuFFT callback fusion. +- cache-aware ring buffers. + +Mimir translation: + +- SIMD first for fixed small bin banks. +- GPU when candidate/bin/channel batches are large enough to amortize dispatch + and memory movement. + diff --git a/research/perfect-machine-study-2026-05-23/option-matrix.md b/research/perfect-machine-study-2026-05-23/option-matrix.md new file mode 100644 index 0000000..9df556a --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/option-matrix.md @@ -0,0 +1,72 @@ +# Implementation Option Matrix + +## Purpose + +This matrix keeps future cuts from turning into flavor arguments. Each row names +the invariant a subsystem must protect, the conservative implementation, the +aggressive implementation, and the proof that should decide between them. + +## Audio Sync Receiver + +| Option | Protects | Upside | Failure Mode | Proof | +| --- | --- | --- | --- | --- | +| Current window decoder | Correctness oracle | Easy to inspect; already wired | Repeated full-window work | Keep as reference until streaming matches | +| Streaming scalar decoder | One-pass receiver state | Small, portable, phone-shaped | Weak proposals can miss path-damaged chirps | Synthetic/artifact parity with fewer scans | +| AVX2 scorer | CPU throughput | Fast fixed-bin scoring | Interop and memory traffic erase wins | Batched benchmark beats scalar end-to-end | +| GPU scorer | Large batch throughput | Many channels/candidates at once | Dispatch/readback overhead | Only promote when batches are large and GPU stays resident | +| Full chirplet transform | General signal analysis | Handles unknown chirps | Too broad for controlled codebook | Only if controlled receiver fails after calibration | + +## Active Witness Shape + +| Option | Protects | Upside | Failure Mode | Proof | +| --- | --- | --- | --- | --- | +| 32 fixed bins | Short triplets | Dense code space | Dead/aliased bands in room | Works only where response is good | +| Adaptive reliable bins + higher order | Physical robustness | Avoids dead bands | Longer lock window | Meatspace anchors increase after calibration | +| Watermarked hybrid | Human tolerance | Less annoying | Masked too well to decode | Passive+active confidence improves without audible abuse | +| Ultrasonic-only | Low distraction | Clean separation from music | Monitors/mics may not pass band | Measured SNR and response beyond hearing | + +## Audio Actuator + +| Option | Protects | Upside | Failure Mode | Proof | +| --- | --- | --- | --- | --- | +| Integer delay | Coarse alignment | Simple | Not microsecond/subsample | Useful only as baseline | +| Farrow fractional delay | Subsample delay | Cheap, time-varying | Magnitude/phase ripple | Delay residual falls on artifact and live captures | +| Variable ASRC | Long-term SRO | Holds sync over time | More complex buffering | Drift stays bounded over 20+ minutes | +| Frequency-domain phase correction | STFT coherence | Good for analysis frames | Latency/window artifacts | Program material remains clean | + +## Audio Field Reconstruction + +| Option | Protects | Upside | Failure Mode | Proof | +| --- | --- | --- | --- | --- | +| Aligned stems | OBS usefulness | Immediate production value | Not volumetric | Stems stay synchronized and mixable | +| Source-based spatial bus | Coherent performance mix | Honest with sparse mics | Depends on source metadata | Stable source positions/pans | +| Ambisonic fit | Compact spatial field | Standard ecosystem | Sparse irregular mics unstable | Calibration proves usable order/band | +| Sparse equivalent sources | Physical field estimate | Handles arbitrary geometry | Ill-conditioned inverse problem | Known source reconstructed with residuals | +| Hybrid evidence field | General Mimir shape | Combines audio/visual constraints | Complexity creep | Fensalir reservoir stabilizes claims | + +## Camera Ingest + +| Option | Protects | Upside | Failure Mode | Proof | +| --- | --- | --- | --- | --- | +| JSON probes | Research observability | Fast to inspect | Not a hot path | Keep diagnostic only | +| Managed wrapper over native reads | App integration | Easier first driver | Copies/GC pressure | One camera sustained cadence | +| Native SPSC frame rings | Throughput | Predictable hot loop | ABI/lifetime complexity | All local cameras sustained | +| GPU shared textures | Copy avoidance | Best final path | Driver/API friction | Fensalir imports and renders frames | + +## Visual Fusion + +| Option | Protects | Upside | Failure Mode | Proof | +| --- | --- | --- | --- | --- | +| Feature tracks | Sparse evidence | Debuggable | Not visually rich | Tracks align across cameras | +| Point/splat claims | Live volumetric surface | GPU-friendly | Needs calibration | Stable live render from observations | +| Offline 3DGS-style optimization | Visual quality | Mature papers/code | Wrong latency/ownership | Use only for reference or static assets | +| Dynamic/4D splats | Live field | Target shape | Research complexity | Start after direct observations work | + +## Fensalir Boundary + +| Option | Protects | Upside | Failure Mode | Proof | +| --- | --- | --- | --- | --- | +| Mimir builds render state | Quick local control | Tempting | Duplicates engine | Reject | +| Mimir lowers contract frames | Ownership clarity | Clean boundary | Needs stable contract mapping | Allocation-free lowering benchmark | +| Fensalir owns temporal evidence | Single field authority | Reuses engine work | Requires good candidates | Acoustic/visual claims visible in engine | + diff --git a/research/perfect-machine-study-2026-05-23/prior-research-synthesis.md b/research/perfect-machine-study-2026-05-23/prior-research-synthesis.md new file mode 100644 index 0000000..884cc4b --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/prior-research-synthesis.md @@ -0,0 +1,142 @@ +# Prior Research Synthesis + +## Purpose + +Mimir already carries useful research archives. This note distills the pieces +that still steer the live machine so the next pass does not reread every mirror +before touching code. + +## Chirplet Sync Decoder Archive + +Location: `research/chirplet-sync-decoder/` + +Durable lesson: + +- Generic chirplet matching is valid signal analysis, but the controlled Mimir + beacon should be receiver-cheap by construction. +- Dechirp plus FFT/Goertzel is the important communications trick. +- Symbol identity should be built from common chirp duration/slope/window plus + decodable frequency bin; rhythm and gaps are additional evidence. + +Impact on current work: + +- `MimirChirpBinTimeline` is directionally right. +- The next work is streaming state and calibrated likelihood, not resurrecting + dense matched-filter banks. + +## Live Adaptive Sync Archive + +Location: `research/live-adaptive-sync/` + +Durable lesson: + +- Initial delay and sample-rate offset are separate loops. +- SRO is inevitable across independent clocks and must be estimated/compensated. +- Resampling/fractional delay is the actuator when hardware clocks cannot be + controlled. +- GCC-PHAT/TDOA is evidence for relative arrival and localization, not a + canonical timeline by itself. + +Impact on current work: + +- `MimirAudioSynchronizationState` needs to drive a real actuator. +- Delay/SRO estimates should be control inputs to Faust/native DSP. +- Passive mode remains valuable, but it does not replace active codebook time. + +## Ambisonic / AquaSynth Archive + +Location: `research/ambisonic-aquasynth/` + +Durable lesson: + +- Arbitrary microphones are not automatically an ambisonic mic. +- Ambisonic output needs declared order, channel ordering, normalization, sample + rate, and target. +- Source-based spatialization is the honest first path; arbitrary-array field + reconstruction requires calibration and regularization. + +Impact on current work: + +- Mimir should not promise high-order field truth until mic geometry and path + calibration prove it. +- Faust `hoa.lib`, IEM, SPARTA, and SAF are verification/reference tools. +- The initial output can be aligned stems plus a lower-order spatial bed. + +## GPU Fusion / Splatting Archive + +Location: `research/gpu-fusion-splatting/` + +Durable lesson: + +- Gaussian splatting is useful for packed GPU representations, tile/bin/sort + renderers, anisotropic radiance primitives, and dynamic field ideas. +- Mimir's live problem is not offline static-scene 3DGS training. +- CPU readback of dense visual state is poison for the hot path. + +Impact on current work: + +- Fensalir owns temporal Gaussian/evidence fields. +- Mimir should feed synchronized observations and constraints. +- The first visual fusion proof should be a small live observation-to-splat + update, not full 4D training. + +## Playback / Feedback Calibration Archives + +Locations: + +- `research/playback-calibration/` +- `research/feedback-calibration/` + +Durable lesson: + +- Room/speaker/mic paths can be measured as impulse or transfer functions. +- Online acoustic system identification and adaptive echo cancellation literature + is relevant because Mimir is emitting and listening in the same room. +- Magnitude-only response is insufficient; phase/group delay are part of the + timing machine. + +Impact on current work: + +- The chirp-bin calibration model should keep becoming a path transfer model. +- Active witness design should support both timeline decoding and transfer + function estimation. +- Hybrid watermarking should learn from echo-cancellation and system-identification + practice rather than just hiding chirps under music. + +## Visual Spatial Map Archive + +Location: `research/visual-spatial-map/` + +Durable lesson: + +- Direct device paths and camera-specific quirks matter. Framework convenience + layers hide timing, buffering, exposure, and bandwidth behavior. +- Leap, Kiyos, and PS3 Eyes have different roles: timing/IR, RGB ground truth, + and high-rate tracking. +- Multi-camera calibration is a geometry problem before it is a rendering + problem. + +Impact on current work: + +- `IMimirVideoCaptureDriver` is the correct production seam. +- Native probes should be promoted into workers only after they preserve + timestamps, payload handles, and cadence. +- Fensalir should consume calibrated camera observations, not raw app-specific + UI experiments. + +## Repository-Level Conclusion + +The archives agree on one architecture: + +```text +calibrated sensors -> rolling evidence window -> canonical time -> native/GPU/DSP lowering -> program outputs +``` + +They do not support: + +- framework-mediated capture in the six-camera hot loop; +- generic chirplet transforms as the default controlled-beacon receiver; +- arbitrary ambisonic claims from unsynchronized heterogeneous mics; +- CPU-owned dense visual fusion; +- OBS as synchronization authority. + diff --git a/research/perfect-machine-study-2026-05-23/questions-and-hypotheses.md b/research/perfect-machine-study-2026-05-23/questions-and-hypotheses.md new file mode 100644 index 0000000..77bbfbf --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/questions-and-hypotheses.md @@ -0,0 +1,99 @@ +# Questions And Hypotheses + +## Audio + +### Q: Is the current physical failure mostly SNR, frequency response, or timing ambiguity? + +Hypothesis: + +- It is primarily response/confusion plus weak direct-path SNR, not a fundamental + codebook failure. + +Test: + +- Compare calibration-weighted likelihood on stored hot mic captures. + +### Q: Is 192 kHz worth keeping for active sync? + +Hypothesis: + +- Yes for fractional timing, loopback, and high-frequency option space; no as a + magic acoustic fix when speakers/mics do not preserve ultrasonic energy. + +Test: + +- Run same reliable-bin codebook at 48/96/192 kHz through physical path and + compare anchor residuals, not just sample granularity. + +### Q: Should chirp-only and hybrid share one decoder? + +Hypothesis: + +- Yes. Mode changes emission policy and confidence gating, not decode + semantics. + +### Q: Can any three chirps always locate timeline position? + +Hypothesis: + +- Only if they are correctly classified inside the active codebook and the + receiver keeps enough timing/frequency spacing evidence. In meatspace, + "any three heard chirps" means "any three reliable decoded symbols after path + calibration," not raw peaks. + +## Visual + +### Q: Should Leap or PS3 Eye be first direct runtime driver? + +Hypothesis: + +- Leap is semantically highest value as timing/depth/near-field witness; PS3 Eye + may be lower-risk for proving direct high-rate runtime plumbing. + +Test: + +- Build the smaller driver first if it proves the seam faster, but do not lose + the Leap timing-camera goal. + +### Q: Is realtime Gaussian splatting the first visual fusion target? + +Hypothesis: + +- No. The first target is GPU feature/claim publication from synchronized + frames. Splatting is the render/update representation after that. + +## Distributed + +### Q: Should Raven send raw audio/video or decoded observations? + +Hypothesis: + +- Both may be needed, but timing authority should come from locally decoded + codebook/schedule observations where possible. Raw streams are evidence; local + decoded anchors are stronger over variable network latency. + +### Q: Can a phone know canonical time from codebook alone? + +Hypothesis: + +- Yes if it has schedule epoch, symbol plan, sample-rate estimate, and enough + reliable decoded anchors. It should report confidence and path model, not just + "time." + +## Implementation + +### Q: Is the native reservoir ready to replace C# buffers? + +Hypothesis: + +- Conceptually yes, operationally not yet. First mirror C# buffer proof through + reservoir status/views before making it the only path. + +### Q: Where does micro-optimization start? + +Hypothesis: + +- Not with HLSL. First benchmark current active decode; if scalar bin scoring + dominates, AVX2/FMA scorer is likely the best first optimization. GPU wins + only when batches are large. + diff --git a/research/perfect-machine-study-2026-05-23/reading-guide.md b/research/perfect-machine-study-2026-05-23/reading-guide.md new file mode 100644 index 0000000..724bc6f --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/reading-guide.md @@ -0,0 +1,132 @@ +# Reading Guide + +## Purpose + +This directory is now large enough to need its own map. Use this guide when +entering a future implementation pass cold. + +## If You Are Building The Active Decoder + +Read in order: + +1. `chirplet-transform-deep-dive.md` +2. `decoder-architecture-options.md` +3. `microsecond-sync-math.md` +4. `current-hotspot-audit.md` +5. `optimization-ledger.md` +6. `low-level-implementation-notes.md` +7. `benchmark-plan.md` +6. sample sketches: + - `samples/StreamingChirpBinDecoderSketch.cs` + - `samples/BatchedChirpScoreAbiSketch.h` + - `samples/CalibrationWeightedChirpLikelihoodSketch.cs` + - `samples/Avx2DechirpGoertzelSketch.cpp` + - `samples/ChirpBinScore.compute.hlsl` + +Decision: + +- implement streaming state first; +- preserve the existing window decoder as oracle; +- promote native/GPU scoring only after benchmarks. + +## If You Are Building Meatspace Calibration + +Read in order: + +1. `volumetric-audio-field.md` +2. `acoustic-field-models.md` +3. `calibration-session-spec.md` +4. `prior-research-synthesis.md` +5. `benchmark-plan.md` +6. `references.md` + +Decision: + +- learn usable bands per path; +- shape symbol likelihood before codebook solving; +- correct magnitude and phase/group delay; +- adapt the emitted alphabet rather than filtering failure afterward. + +## If You Are Building The Audio Actuator + +Read in order: + +1. `prior-research-synthesis.md` +2. `optimization-ledger.md` +3. `benchmark-plan.md` +4. sample sketches: + - `samples/FarrowFractionalDelaySketch.cpp` + - `samples/SroPllAsrcControllerSketch.cpp` + +Decision: + +- delay and SRO are separate loops; +- the runtime estimates state; +- Faust/native DSP moves samples. + +## If You Are Building Camera Ingest + +Read in order: + +1. `native-boundary-map.md` +2. `visual-fusion-4dgs-study.md` +3. `low-level-implementation-notes.md` +4. `current-hotspot-audit.md` +5. `benchmark-plan.md` +6. `docs/native-capture-cadence.md` +7. `docs/native-rebuild-plan.md` + +Decision: + +- JSON probes are diagnostics; +- production capture is native workers plus payload handles; +- Fensalir owns GPU import and resource lifetime. + +## If You Are Building Fensalir Integration + +Read in order: + +1. `fensalir-integration-map.md` +2. `native-boundary-map.md` +3. `visual-fusion-4dgs-study.md` +4. sample sketches: + - `samples/AquariumGpuSensorFrameBridgeSketch.cs` + - `samples/AcousticConstraintLoweringSketch.cs` + - `samples/TemporalEvidenceCandidateSketch.cs` + +Decision: + +- Mimir lowers observations/constraints; +- Fensalir owns temporal evidence and render state; +- the render tick must not run capture or analysis. + +## If You Are Looking For The Next Cut + +Read in order: + +1. `implementation-roadmap.md` +2. `option-matrix.md` +3. `research-claims-digest.md` +4. `questions-and-hypotheses.md` +5. `distributed-receiver-spec.md` +6. `current-hotspot-audit.md` + +Recommended next code cut: + +- streaming active decoder state with calibration-weighted likelihood; +- benchmark it against the current window decoder; +- then wire the actuator. + +## If You Are Updating Public Docs + +Read: + +1. `docs/perfect-machine-domain-index.md` +2. `docs/code-algorithm-map.md` +3. `notes/current-system-map.md` +4. this directory's `index.md` + +Rule: + +- public docs describe the live machine and current constraints; +- research docs may hold scars, rejected paths, and implementation options. diff --git a/research/perfect-machine-study-2026-05-23/references.md b/research/perfect-machine-study-2026-05-23/references.md index 47b0faf..c16335b 100644 --- a/research/perfect-machine-study-2026-05-23/references.md +++ b/research/perfect-machine-study-2026-05-23/references.md @@ -9,6 +9,10 @@ https://www.sciencedirect.com/science/article/pii/S0888327015003994 - High-resolution chirplet transform parameter analysis: https://arxiv.org/abs/2108.00572 +- Fast Chirplet Transform with FPGA-based implementation: + https://www.ece.iit.edu/~eoruklu/IIT/Publications_files/04639630.pdf +- Speed-optimized fast chirplet implementation: + https://ecasp.ece.iit.edu/publications/2012-present/2023-13.pdf ## Chirp Spread Spectrum / LoRa Receiver Shape @@ -42,6 +46,10 @@ - HLSL Shader Model 6 wave intrinsics: https://learn.microsoft.com/windows/win32/direct3dhlsl/hlsl-shader-model-6-0-features-for-direct3d-12 +- Microsoft Kernel Streaming / AVStream overview: + https://learn.microsoft.com/en-gb/windows-hardware/drivers/stream/kernel-streaming +- DXGI shared resource handle creation: + https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_2/nf-dxgi1_2-idxgiresource1-createsharedhandle - DirectX Shader Compiler wave intrinsics notes: https://github.com/microsoft/directxshadercompiler/wiki/wave-intrinsics - cuFFT callback documentation: @@ -55,6 +63,82 @@ - Chirp Z-transform theory: https://pyffs.readthedocs.io/en/stable/theory/CZT.html +## Volumetric Audio / Room Acoustics / Actuation + +- Pyroomacoustics beamforming module: + https://pyroomacoustics.readthedocs.io/en/pypi-release/pyroomacoustics.beamforming.html +- Pyroomacoustics repository and room/array algorithm examples: + https://github.com/LCAV/pyroomacoustics +- Pyroomacoustics paper: + https://arxiv.org/abs/1710.04196 +- ManyEars sound source localization, tracking, and separation: + https://github.com/introlab/manyears +- Spatial Audio Framework, open C/C++ spatial-audio toolkit: + https://github.com/leomccormack/Spatial_Audio_Framework +- Sound field reconstruction in reverberant environments with rigid spherical + microphone arrays: + https://journals.sagepub.com/doi/10.1177/14613484261447569 +- Sparse Reconstruction of Sound Field Using Bayesian Compressive Sensing and + Equivalent Source Method: + https://pmc.ncbi.nlm.nih.gov/articles/PMC10301025/ +- Near-field acoustic holography with compressive sensing: + https://docs.lib.purdue.edu/herrick/208/ +- Ambisonic encoding from equatorial microphone arrays: + https://arxiv.org/abs/2211.00584 +- Spheroidal ambisonics: + https://arxiv.org/abs/2103.05719 +- Real-time spherical array renderer: + https://research.chalmers.se/publication/509494 +- Farrow fractional delay overview: + https://engee.com/helpcenter/stable/en/interactive-scripts/dsp/fractional_delay.html +- AMD Vitis fractional-delay Farrow filter note: + https://docs.amd.com/r/2024.1-English/Vitis-Tutorials-AI-Engine-Development/Fractional-Delay-Farrow-Filter +- Faust delay/fractional delay reference: + https://ringbuffer.org/faust/faust_basics/faust-delay/ +- MATLAB Farrow fractional delay example: + https://www.mathworks.com/help/dsp/ug/fractional-delay-filters-using-farrow-structures.html +- Joint SFO estimation and compensation based on Farrow structure: + https://arxiv.org/abs/2503.07577 +- Joint SFO estimation and compensation algorithms based on Farrow structure: + https://arxiv.org/abs/2603.00627 + +## Realtime Gaussian Splatting / Dynamic Visual Fields + +- Official 3D Gaussian Splatting implementation: + https://github.com/graphdeco-inria/gaussian-splatting +- 3D Gaussian Splatting paper: + https://arxiv.org/abs/2308.04079 +- 4D Gaussian Splatting implementation: + https://github.com/fudan-zvg/4d-gaussian-splatting +- 4D Gaussian Splatting paper: + https://openaccess.thecvf.com/content/CVPR2024/papers/Wu_4D_Gaussian_Splatting_for_Real-Time_Dynamic_Scene_Rendering_CVPR_2024_paper.pdf +- Spacetime Gaussian Feature Splatting implementation: + https://github.com/oppo-us-research/SpacetimeGaussians +- Spacetime Gaussian Feature Splatting paper: + https://arxiv.org/abs/2312.16812 +- PlayCanvas engine, an open WebGL/WebGPU engine with Gaussian splatting support: + https://github.com/playcanvas/engine +- PlayCanvas generic Gaussian splat processing docs: + https://developer.playcanvas.com/user-manual/gaussian-splatting/building/unified-rendering/splat-processing/ +- NVIDIA vk_gaussian_splatting technical blog: + https://developer.nvidia.com/blog/real-time-gpu-accelerated-gaussian-splatting-with-nvidia-designworks-sample-vk_gaussian_splatting/ +- gsplat documentation: + https://docs.gsplat.studio/main/ +- gsplat rasterization API: + https://docs.gsplat.studio/main/apis/rasterization.html +- gsplat paper: + https://arxiv.org/abs/2409.06765 +- StopThePop sorted Gaussian splatting implementation: + https://github.com/r4dl/StopThePop +- FlashGS paper: + https://arxiv.org/abs/2408.07967 +- Gaussian-LIC, Gaussian splatting plus LiDAR-inertial-camera SLAM: + https://arxiv.org/abs/2404.06926 +- RMGS-SLAM, real-time multi-sensor Gaussian splatting SLAM: + https://arxiv.org/abs/2604.12942 +- Multi-Calib, online multi-sensor calibration direction: + https://pmc.ncbi.nlm.nih.gov/articles/PMC12693731/ + ## How These References Steer Mimir - Dechirp-plus-bin scoring is the right hot active receiver shape for controlled @@ -67,4 +151,6 @@ a small fixed bin bank, SIMD Goertzel/dechirp may beat FFT overhead. - If a transform path moves to GPU, fuse dechirp/window/preprocess with transform load/store where possible to avoid memory bandwidth waste. - +- Visual fusion should borrow Gaussian splatting data/layout ideas, but the live + Mimir problem is online synchronized evidence update, not offline novel-view + training from a static capture folder. diff --git a/research/perfect-machine-study-2026-05-23/research-claims-digest.md b/research/perfect-machine-study-2026-05-23/research-claims-digest.md new file mode 100644 index 0000000..8d66481 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/research-claims-digest.md @@ -0,0 +1,168 @@ +# Research Claims Digest + +## Purpose + +This is the claim-level digest from the research pass. It records what the +external literature and open implementations change about Mimir's design. + +## Chirplets / CSS / Receiver Design + +Claim: + +- A generic chirplet transform is the broad signal-analysis tool. +- A controlled chirp communication receiver should usually exploit waveform + design: dechirp, bin score, then decode. + +Mimir consequence: + +- The hot receiver should remain constrained by the known codebook and schedule. +- Full chirplet transforms are reference tools, not first-line runtime code. +- Symbol design should serve demodulation. + +References: + +- Steve Mann chirplet transform. +- Fast chirplet transform / FPGA implementation work. +- LoRa/CSS demodulator literature and OpenLoRa. + +## Passive Delay / Distributed Microphones + +Claim: + +- Delay and sampling-rate offset are different phenomena. +- Distributed microphone arrays require explicit SRO estimation/compensation. +- GCC-PHAT is useful for TDOA/delay evidence, not enough to own canonical time. + +Mimir consequence: + +- Passive mode is a confidence-bearing witness. +- Active codebook anchors remain the canonical-time path. +- The actuator must include both fractional delay and SRO control. + +References: + +- Wang/Doclo SRO estimation. +- Schmalenstroeer/Haeb-Umbach clock skew work. +- Didier distributed adaptive node-specific signal estimation. +- GCC-PHAT refinements. + +## Room / Sound Field Reconstruction + +Claim: + +- Sound-field reconstruction from sparse arrays is an inverse problem with + geometry, bandwidth, regularization, and noise constraints. +- Ambisonics works cleanly when the microphone array and calibration support the + basis. +- Equivalent-source / sparse reconstruction methods are useful when the source + model is sparse and measurement model is known. + +Mimir consequence: + +- Six heterogeneous mics are not automatically a high-order field mic. +- The first honest target is synchronized stems plus source/constraint estimates. +- Field reconstruction should expose uncertainty and avoid pretending unsupported + spatial detail exists. + +References: + +- Pyroomacoustics. +- Spatial Audio Framework. +- ManyEars. +- near-field acoustic holography / equivalent source / compressive sensing + papers. +- ambisonics/arbitrary-array encoding notes. + +## Fractional Delay / SRO Actuation + +Claim: + +- Farrow structures are standard for continuously variable fractional delay. +- Variable-rate resampling or equivalent phase correction is required for drift. + +Mimir consequence: + +- Microsecond reporting is incomplete until an actuator moves samples. +- A Farrow proof is the right first native/Faust actuator slice. +- Higher-quality polyphase/ASRC work comes after the control loop is proven. + +References: + +- AMD Vitis Farrow filter docs. +- MATLAB Farrow fractional delay notes. +- joint SFO/Farrow papers. + +## Gaussian Splatting / Visual Fusion + +Claim: + +- Practical 3DGS renderers use packed GPU buffers, projection, tile binning, + sorting/order strategies, and optimized rasterization kernels. +- Dynamic/4D Gaussian splatting extends splats over time, but production code is + still research-shaped. +- Online SLAM/sensor fusion variants are more relevant to Mimir than offline + static scene training alone. + +Mimir consequence: + +- Fensalir should own the GPU field. +- Mimir should feed synchronized observations and constraints. +- The first proof is live observation-to-claim-to-render, not full 4DGS training. + +References: + +- GraphDeco 3DGS. +- 4D Gaussian Splatting. +- Spacetime Gaussians. +- gsplat. +- NVIDIA `vk_gaussian_splatting`. +- StopThePop. +- FlashGS. +- Gaussian-SLAM / multi-sensor calibration papers. + +## Native Capture / GPU Resource Boundaries + +Claim: + +- Windows KS/AVStream is the low-level video streaming surface under higher media + frameworks. +- D3D shared handles are the relevant resource-sharing family once camera + payloads can become GPU resources. +- ASIO callbacks and sample positions are the correct place to preserve driver + clock timing. + +Mimir consequence: + +- Media frameworks may remain diagnostics/fallbacks, but direct workers own + production capture. +- Fensalir owns D3D12 resource lifetime and imports. +- Mimir should not route six cameras through stdout/JSON or managed pixel + processing. + +References: + +- Microsoft Kernel Streaming docs. +- Microsoft DXGI shared handle docs. +- ASIO specification / callback notes. +- Focusrite channel/loopback documentation. + +## Strongest Design Conclusion + +The research converges on one architecture: + +```text +controlled witness + calibrated sensors + rolling buffers +-> deterministic timeline/response models +-> native DSP and GPU field lowering +-> OBS-facing program output +``` + +The critical missing production step is not another filter. It is connecting +measurement to actuation and field lowering: + +- streaming decoder state; +- calibration-weighted likelihood; +- fractional delay/SRO actuator; +- native camera payload handles; +- Fensalir contract lowering. + diff --git a/research/perfect-machine-study-2026-05-23/sample-code-index.json b/research/perfect-machine-study-2026-05-23/sample-code-index.json new file mode 100644 index 0000000..b420c75 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/sample-code-index.json @@ -0,0 +1,97 @@ +{ + "schema": "gamecult.mimir.sample_code_index.v1", + "created": "2026-05-23", + "samples": [ + { + "path": "research/perfect-machine-study-2026-05-23/samples/StreamingChirpBinDecoderSketch.cs", + "domain": "audio-sync", + "purpose": "Incremental chirp-bin decoder state model", + "productionTarget": "Mimir.Runtime streaming active decoder", + "promoteWhen": "Matches current analyzer outputs while avoiding full-window rescans" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/Avx2DechirpGoertzelSketch.cpp", + "domain": "audio-sync", + "purpose": "AVX2/FMA dechirp plus bin scoring sketch", + "productionTarget": "native chirp-bin scorer", + "promoteWhen": "Profiling shows C# scalar bin scoring dominates" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/BatchedChirpScoreAbiSketch.h", + "domain": "audio-sync", + "purpose": "Native batched scoring ABI boundary for chirp candidate/bin work", + "productionTarget": "Mimir native scorer interop layer", + "promoteWhen": "Scalar scoring dominates and batching proves interop cost is amortized" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/ChirpBinScore.compute.hlsl", + "domain": "audio-sync-gpu", + "purpose": "D3D12 compute candidate/bin scoring sketch", + "productionTarget": "Fensalir compute scoring pass", + "promoteWhen": "Candidate/bin/channel batches are large enough to beat CPU SIMD" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/FarrowFractionalDelaySketch.cpp", + "domain": "audio-dsp", + "purpose": "Cubic Farrow fractional delay actuator sketch", + "productionTarget": "Faust/native DSP alignment actuator", + "promoteWhen": "Runtime state can drive a test resampler/delay line with measured improvement" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/SpscAudioBlockRingSketch.cpp", + "domain": "native-capture", + "purpose": "Low-jitter SPSC audio block handoff sketch", + "productionTarget": "native ASIO/camera capture handoff", + "promoteWhen": "Managed queues or payload copies show measurable capture pressure" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/CalibrationWeightedChirpLikelihoodSketch.cs", + "domain": "audio-sync", + "purpose": "Calibration-weighted symbol likelihood before codebook solving", + "productionTarget": "MimirChirpBinTimeline scoring path", + "promoteWhen": "Physical mic captures show unweighted symbol likelihood is the failure point" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/SroPllAsrcControllerSketch.cpp", + "domain": "audio-dsp", + "purpose": "Control loop turning delay observations into ASRC/fractional delay commands", + "productionTarget": "Faust/native DSP actuator control", + "promoteWhen": "Runtime reports are stable enough to move samples instead of only logging state" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/GaussianSplatClaimUpdate.compute.hlsl", + "domain": "visual-fusion", + "purpose": "D3D12 compute sketch for live splat claim updates", + "productionTarget": "Fensalir visual fusion pass", + "promoteWhen": "Direct camera drivers feed synchronized feature observations" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/AquariumGpuSensorFrameBridgeSketch.cs", + "domain": "fensalir-integration", + "purpose": "Lower runtime video observations into Fensalir GPU sensor frame contracts", + "productionTarget": "Mimir app-to-Fensalir bridge", + "promoteWhen": "Runtime buffers carry pixel/native handles and lowering benchmarks allocation-free" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/AcousticConstraintLoweringSketch.cs", + "domain": "fensalir-integration", + "purpose": "Lower decoded audio anchors and mic geometry into Fensalir acoustic constraints", + "productionTarget": "Mimir acoustic field bridge", + "promoteWhen": "Decoder reports stable anchors and Fensalir consumes constraints in diagnostics" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/TemporalEvidenceCandidateSketch.cs", + "domain": "sensor-fusion", + "purpose": "Convert Mimir observations into reservoir candidates without owning stable tracks", + "productionTarget": "Fensalir temporal evidence lowering path", + "promoteWhen": "Mimir emits synchronized visual/acoustic candidates and Fensalir reservoir accepts them" + }, + { + "path": "research/perfect-machine-study-2026-05-23/samples/SrpPhatGridSketch.cpp", + "domain": "volumetric-audio", + "purpose": "Minimal grid scoring from pairwise mic delays to source-position hypotheses", + "productionTarget": "native acoustic source localization proof", + "promoteWhen": "At least two physical mics produce calibrated stable delay anchors" + } + ] +} diff --git a/research/perfect-machine-study-2026-05-23/samples/AcousticConstraintLoweringSketch.cs b/research/perfect-machine-study-2026-05-23/samples/AcousticConstraintLoweringSketch.cs new file mode 100644 index 0000000..e1e13a9 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/samples/AcousticConstraintLoweringSketch.cs @@ -0,0 +1,104 @@ +// Sketch only. Purpose: keep acoustic raw buffers in Mimir/native DSP while +// sending Fensalir compact constraints for spatial fusion and diagnostics. + +using System; +using System.Collections.Generic; + +namespace Mimir.PerfectMachineStudy; + +public readonly record struct MimirDecodedAnchorSketch( + string SourceId, + int EventIndex, + double SourceSampleTimeSeconds, + double CanonicalTimeSeconds, + double DelaySeconds, + double FrequencyHz, + float Confidence, + float PhaseRadians, + float GroupDelaySeconds); + +public readonly record struct MimirMicGeometrySketch( + string SourceId, + float X, + float Y, + float Z); + +public sealed class AcousticConstraintLoweringSketch +{ + private readonly List _constraints = new(128); + + public AquariumAcousticFieldFrameSketch Lower( + ReadOnlySpan anchors, + ReadOnlySpan geometry, + double canonicalNowSeconds) + { + _constraints.Clear(); + + for (var i = 0; i < anchors.Length; i++) + { + ref readonly var anchor = ref anchors[i]; + if (anchor.Confidence < 0.4f) + { + continue; + } + + var mic = FindGeometry(geometry, anchor.SourceId); + if (mic.SourceId is null) + { + continue; + } + + _constraints.Add(new AquariumAcousticConstraintSketch( + anchor.SourceId, + anchor.EventIndex, + canonicalNowSeconds, + anchor.CanonicalTimeSeconds, + anchor.DelaySeconds, + anchor.FrequencyHz, + mic.X, + mic.Y, + mic.Z, + anchor.Confidence, + anchor.PhaseRadians, + anchor.GroupDelaySeconds)); + } + + // Production version should avoid this copy and hand Fensalir a stable + // span/packet buffer whose lifetime is owned by the frame boundary. + return new AquariumAcousticFieldFrameSketch(canonicalNowSeconds, _constraints.ToArray()); + } + + private static MimirMicGeometrySketch FindGeometry( + ReadOnlySpan geometry, + string sourceId) + { + for (var i = 0; i < geometry.Length; i++) + { + if (StringComparer.Ordinal.Equals(geometry[i].SourceId, sourceId)) + { + return geometry[i]; + } + } + + return default; + } +} + +public readonly record struct AquariumAcousticFieldFrameSketch( + double CanonicalTimeSeconds, + AquariumAcousticConstraintSketch[] Constraints); + +public readonly record struct AquariumAcousticConstraintSketch( + string SourceId, + int EventIndex, + double FrameCanonicalTimeSeconds, + double EventCanonicalTimeSeconds, + double DelaySeconds, + double FrequencyHz, + float MicX, + float MicY, + float MicZ, + float Confidence, + float PhaseRadians, + float GroupDelaySeconds); + diff --git a/research/perfect-machine-study-2026-05-23/samples/AquariumGpuSensorFrameBridgeSketch.cs b/research/perfect-machine-study-2026-05-23/samples/AquariumGpuSensorFrameBridgeSketch.cs new file mode 100644 index 0000000..02b956a --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/samples/AquariumGpuSensorFrameBridgeSketch.cs @@ -0,0 +1,78 @@ +// Sketch only. This is not wired to production contracts yet. +// Purpose: show the desired shape of the Mimir -> Fensalir camera lowering cut. + +using System; +using System.Collections.Generic; + +namespace Mimir.PerfectMachineStudy; + +public readonly record struct MimirVideoObservation( + string SourceId, + long Sequence, + double CanonicalTimeSeconds, + double DeviceTimeSeconds, + int Width, + int Height, + string PixelFormat, + IntPtr NativeHandle, + int NativeHandleKind, + string CalibrationId, + float Confidence); + +public sealed class AquariumGpuSensorFrameBridgeSketch +{ + private readonly List _cameras = new(8); + + public AquariumGpuSensorFrameSketch Lower( + ReadOnlySpan observations, + double frameCanonicalTimeSeconds) + { + _cameras.Clear(); + + for (var i = 0; i < observations.Length; i++) + { + ref readonly var source = ref observations[i]; + if (source.NativeHandle == IntPtr.Zero || source.Confidence <= 0.0f) + { + continue; + } + + _cameras.Add(new AquariumGpuSensorCameraSketch( + source.SourceId, + source.Sequence, + source.CanonicalTimeSeconds, + source.DeviceTimeSeconds, + source.Width, + source.Height, + source.PixelFormat, + source.NativeHandle, + source.NativeHandleKind, + source.CalibrationId, + source.Confidence)); + } + + // Production version should fill a reusable contract-owned array or pooled + // struct buffer instead of allocating here. + return new AquariumGpuSensorFrameSketch( + frameCanonicalTimeSeconds, + _cameras.ToArray()); + } +} + +public readonly record struct AquariumGpuSensorFrameSketch( + double CanonicalTimeSeconds, + AquariumGpuSensorCameraSketch[] Cameras); + +public readonly record struct AquariumGpuSensorCameraSketch( + string SourceId, + long Sequence, + double CanonicalTimeSeconds, + double DeviceTimeSeconds, + int Width, + int Height, + string PixelFormat, + IntPtr NativeHandle, + int NativeHandleKind, + string CalibrationId, + float Confidence); + diff --git a/research/perfect-machine-study-2026-05-23/samples/BatchedChirpScoreAbiSketch.h b/research/perfect-machine-study-2026-05-23/samples/BatchedChirpScoreAbiSketch.h new file mode 100644 index 0000000..83d7dac --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/samples/BatchedChirpScoreAbiSketch.h @@ -0,0 +1,53 @@ +// Sketch only. Purpose: native ABI shape for batched chirp-bin scoring. +// The point is to batch enough work that managed/native transition cost does +// not dominate the actual multiply-adds. + +#pragma once + +#include + +extern "C" +{ + struct MimirChirpScoreCandidate + { + std::int64_t absoluteSampleOffset; + std::int32_t sampleOffsetInBuffer; + std::int32_t sampleCount; + float proposalEnergy; + }; + + struct MimirChirpScoreKernel + { + const float* window; + const float* dechirpReal; + const float* dechirpImag; + std::int32_t sampleCount; + }; + + struct MimirChirpScoreRequest + { + const float* samples; + std::int32_t sampleCount; + const MimirChirpScoreCandidate* candidates; + std::int32_t candidateCount; + const MimirChirpScoreKernel* kernels; + std::int32_t kernelCount; + std::int32_t topK; + }; + + struct MimirChirpScoreResult + { + std::int64_t absoluteSampleOffset; + std::int32_t kernelIndex; + float energy; + float phaseRadians; + }; + + // Returns the number of result rows written. Production ABI would also + // return an error/status code and expose required result capacity. + std::int32_t mimir_score_chirp_candidates( + const MimirChirpScoreRequest* request, + MimirChirpScoreResult* results, + std::int32_t resultCapacity); +} + diff --git a/research/perfect-machine-study-2026-05-23/samples/CalibrationWeightedChirpLikelihoodSketch.cs b/research/perfect-machine-study-2026-05-23/samples/CalibrationWeightedChirpLikelihoodSketch.cs new file mode 100644 index 0000000..3952dfa --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/samples/CalibrationWeightedChirpLikelihoodSketch.cs @@ -0,0 +1,105 @@ +// Sketch only. This shows where meatspace calibration should enter the active +// decoder: before code-valid path selection, as likelihood shaping. + +namespace Mimir.Research.Samples; + +public sealed class CalibrationWeightedChirpLikelihoodSketch +{ + private readonly PathCalibration calibration; + + public CalibrationWeightedChirpLikelihoodSketch(PathCalibration calibration) + { + this.calibration = calibration; + } + + public SymbolLikelihood[] ScoreFrame(FrameBinEnergy frame) + { + var result = new SymbolLikelihood[calibration.Symbols.Length]; + for (var symbolId = 0; symbolId < calibration.Symbols.Length; symbolId++) + { + var symbol = calibration.Symbols[symbolId]; + var observed = frame.EnergyByBin[symbolId]; + var reliability = symbol.Reliability; + var confusionPenalty = ConfusionPenalty(symbolId, frame); + var phaseWeight = PhaseWeight(symbol, frame.PhaseByBin[symbolId]); + var delayCorrection = symbol.GroupDelaySamples; + + var logLikelihood = + Math.Log(Math.Max(observed, 1.0e-12)) * reliability + + Math.Log(Math.Max(phaseWeight, 1.0e-6)) - + confusionPenalty; + + result[symbolId] = new SymbolLikelihood( + symbolId, + logLikelihood, + delayCorrection, + reliability, + observed); + } + + return result + .OrderByDescending(score => score.LogLikelihood) + .Take(calibration.MaxCandidatesPerFrame) + .ToArray(); + } + + private double ConfusionPenalty(int expectedSymbol, FrameBinEnergy frame) + { + var penalty = 0.0; + foreach (var alias in calibration.Confusions.Where(c => c.ExpectedSymbol == expectedSymbol)) + { + var aliasEnergy = frame.EnergyByBin[alias.ObservedSymbol]; + penalty += alias.Probability * Math.Log1p(aliasEnergy); + } + + return penalty; + } + + private static double PhaseWeight(SymbolCalibration symbol, double observedPhase) + { + if (symbol.PhaseCoherence <= 0.0) + { + return 1.0; + } + + var distance = AngularDistance(observedPhase, symbol.MeanPhaseRadians); + return 1.0 - symbol.PhaseCoherence * Math.Min(1.0, distance / Math.PI); + } + + private static double AngularDistance(double a, double b) + { + var d = Math.Abs(a - b) % (Math.PI * 2.0); + return d > Math.PI ? Math.PI * 2.0 - d : d; + } +} + +public sealed record FrameBinEnergy( + double[] EnergyByBin, + double[] PhaseByBin, + double InitialOffsetSamples); + +public sealed record PathCalibration( + SymbolCalibration[] Symbols, + Confusion[] Confusions, + int MaxCandidatesPerFrame); + +public sealed record SymbolCalibration( + int SymbolId, + double Reliability, + double MeanEnergy, + double MeanPhaseRadians, + double PhaseCoherence, + double GroupDelaySamples); + +public sealed record Confusion( + int ExpectedSymbol, + int ObservedSymbol, + double Probability); + +public sealed record SymbolLikelihood( + int SymbolId, + double LogLikelihood, + double GroupDelayCorrectionSamples, + double Reliability, + double ObservedEnergy); + diff --git a/research/perfect-machine-study-2026-05-23/samples/GaussianSplatClaimUpdate.compute.hlsl b/research/perfect-machine-study-2026-05-23/samples/GaussianSplatClaimUpdate.compute.hlsl new file mode 100644 index 0000000..0a62c06 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/samples/GaussianSplatClaimUpdate.compute.hlsl @@ -0,0 +1,68 @@ +// Sketch only. Fensalir-side compute shape for updating simple live splat +// claims from synchronized feature observations. + +struct FeatureObservation +{ + uint StableKeyLow; + uint StableKeyHigh; + float3 PositionMeters; + float3 Normal; + float4 Color; + float Confidence; + uint SourceIndex; + uint TimestampLo; + uint TimestampHi; +}; + +struct SplatClaim +{ + uint StableKeyLow; + uint StableKeyHigh; + float3 PositionMeters; + float RadiusMeters; + float4 Color; + float Confidence; + uint TimeMinLo; + uint TimeMinHi; + uint TimeMaxLo; + uint TimeMaxHi; +}; + +cbuffer SplatUpdateConstants : register(b0) +{ + uint ObservationCount; + float Decay; + float Blend; + float DefaultRadius; +}; + +StructuredBuffer Observations : register(t0); +RWStructuredBuffer Claims : register(u0); + +[numthreads(128, 1, 1)] +void main(uint3 id : SV_DispatchThreadID) +{ + uint index = id.x; + if (index >= ObservationCount) + { + return; + } + + FeatureObservation obs = Observations[index]; + + // Sketch simplification: production needs a hash table or sorted stable-key + // compaction pass. This writes one observation to one claim slot. + SplatClaim claim; + claim.StableKeyLow = obs.StableKeyLow; + claim.StableKeyHigh = obs.StableKeyHigh; + claim.PositionMeters = obs.PositionMeters; + claim.RadiusMeters = DefaultRadius; + claim.Color = obs.Color; + claim.Confidence = saturate(obs.Confidence); + claim.TimeMinLo = obs.TimestampLo; + claim.TimeMinHi = obs.TimestampHi; + claim.TimeMaxLo = obs.TimestampLo; + claim.TimeMaxHi = obs.TimestampHi; + Claims[index] = claim; +} + diff --git a/research/perfect-machine-study-2026-05-23/samples/README.md b/research/perfect-machine-study-2026-05-23/samples/README.md index b1f916b..20af71c 100644 --- a/research/perfect-machine-study-2026-05-23/samples/README.md +++ b/research/perfect-machine-study-2026-05-23/samples/README.md @@ -9,14 +9,30 @@ machinery into the app. - `StreamingChirpBinDecoderSketch.cs`: C# state model for an incremental chirp-bin decoder that avoids full-window rescans. - `Avx2DechirpGoertzelSketch.cpp`: AVX2/FMA-shaped CPU scoring kernel sketch. +- `BatchedChirpScoreAbiSketch.h`: native scoring ABI shape for batching + candidate/bin work across the managed/native boundary. - `ChirpBinScore.compute.hlsl`: D3D12 compute shader shape for batched candidate/bin scoring. - `FarrowFractionalDelaySketch.cpp`: native fractional delay actuator sketch. - `SpscAudioBlockRingSketch.cpp`: native single-producer/single-consumer audio block ring sketch for capture worker handoff. +- `CalibrationWeightedChirpLikelihoodSketch.cs`: shows how response/confusion/ + phase/group-delay calibration should shape symbol likelihood before codebook + path selection. +- `SroPllAsrcControllerSketch.cpp`: control-rate delay/SRO loop for driving an + ASRC or fractional-delay actuator. +- `GaussianSplatClaimUpdate.compute.hlsl`: minimal Fensalir compute shape for + turning synchronized visual observations into live splat claims. +- `AquariumGpuSensorFrameBridgeSketch.cs`: contract sketch for lowering runtime + video observations into Fensalir GPU sensor frames. +- `AcousticConstraintLoweringSketch.cs`: contract sketch for lowering decoded + chirp/passive anchors into Fensalir acoustic constraints. +- `TemporalEvidenceCandidateSketch.cs`: candidate lowering sketch that keeps + stable track ownership in Fensalir. +- `SrpPhatGridSketch.cpp`: minimal TDOA/SRP-PHAT-style grid scoring proof for a + synchronized mic constellation. ## Rule Promote one of these only after a benchmark or integration test proves it owns a real invariant. Sample code is a microscope, not a new wall in the house. - diff --git a/research/perfect-machine-study-2026-05-23/samples/SroPllAsrcControllerSketch.cpp b/research/perfect-machine-study-2026-05-23/samples/SroPllAsrcControllerSketch.cpp new file mode 100644 index 0000000..0ae3afb --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/samples/SroPllAsrcControllerSketch.cpp @@ -0,0 +1,60 @@ +// Sketch only. Control-rate PLL for driving an ASRC/fractional delay actuator. + +#include +#include + +struct SyncObservation +{ + double delaySamples; + double confidence; + double dtSeconds; +}; + +struct ActuatorCommand +{ + double targetDelaySamples; + double resampleRatio; + double confidence; +}; + +class SroPllAsrcController +{ +public: + ActuatorCommand update(const SyncObservation& obs) + { + if (obs.confidence < 0.05 || obs.dtSeconds <= 0.0) + { + return command(obs.confidence); + } + + double error = obs.delaySamples - delaySamples_; + double alpha = clamp(obs.confidence * 0.08, 0.005, 0.05); + double beta = clamp(obs.confidence * 0.01, 0.0005, 0.006); + + delaySamples_ += alpha * error; + sroPpm_ += beta * error / obs.dtSeconds; + sroPpm_ = clamp(sroPpm_, -300.0, 300.0); + confidence_ += (obs.confidence - confidence_) * 0.05; + + return command(confidence_); + } + +private: + double delaySamples_ = 0.0; + double sroPpm_ = 0.0; + double confidence_ = 0.0; + + ActuatorCommand command(double confidence) const + { + // Positive SRO means candidate appears to drift later; playback/capture + // correction convention must be fixed against the DSP graph before use. + double ratio = 1.0 - sroPpm_ * 1.0e-6; + return { delaySamples_, ratio, confidence }; + } + + static double clamp(double value, double lo, double hi) + { + return std::max(lo, std::min(hi, value)); + } +}; + diff --git a/research/perfect-machine-study-2026-05-23/samples/SrpPhatGridSketch.cpp b/research/perfect-machine-study-2026-05-23/samples/SrpPhatGridSketch.cpp new file mode 100644 index 0000000..51b3a05 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/samples/SrpPhatGridSketch.cpp @@ -0,0 +1,95 @@ +// Sketch only. Purpose: first acoustic source-position proof after mic timing is +// calibrated. This is intentionally small and direct, not a full acoustic camera. + +#include +#include +#include + +struct MicPoseSketch +{ + float x; + float y; + float z; +}; + +struct CandidatePointSketch +{ + float x; + float y; + float z; +}; + +struct PairDelaySketch +{ + int micA; + int micB; + double delaySeconds; + float confidence; +}; + +struct SrpResultSketch +{ + int bestIndex; + float score; +}; + +static inline double distanceMeters( + const MicPoseSketch& mic, + const CandidatePointSketch& point) +{ + const double dx = static_cast(point.x) - mic.x; + const double dy = static_cast(point.y) - mic.y; + const double dz = static_cast(point.z) - mic.z; + return std::sqrt(dx * dx + dy * dy + dz * dz); +} + +SrpResultSketch scoreGrid( + std::span mics, + std::span grid, + std::span observedPairs, + double speedOfSoundMetersPerSecond) +{ + auto best = SrpResultSketch{-1, -1.0f}; + + for (int pointIndex = 0; pointIndex < static_cast(grid.size()); pointIndex++) + { + const auto& point = grid[static_cast(pointIndex)]; + double score = 0.0; + double weight = 0.0; + + for (const auto& pair : observedPairs) + { + if (pair.micA < 0 || pair.micB < 0 || + pair.micA >= static_cast(mics.size()) || + pair.micB >= static_cast(mics.size()) || + pair.confidence <= 0.0f) + { + continue; + } + + const double da = distanceMeters(mics[static_cast(pair.micA)], point); + const double db = distanceMeters(mics[static_cast(pair.micB)], point); + const double predicted = (db - da) / speedOfSoundMetersPerSecond; + const double errorUs = std::abs(predicted - pair.delaySeconds) * 1'000'000.0; + + // Smooth robust score: 1 at zero error, half around 100 us. + const double local = 1.0 / (1.0 + errorUs / 100.0); + score += local * pair.confidence; + weight += pair.confidence; + } + + if (weight <= 0.0) + { + continue; + } + + const float normalized = static_cast(score / weight); + if (normalized > best.score) + { + best = SrpResultSketch{pointIndex, normalized}; + } + } + + return best; +} + diff --git a/research/perfect-machine-study-2026-05-23/samples/TemporalEvidenceCandidateSketch.cs b/research/perfect-machine-study-2026-05-23/samples/TemporalEvidenceCandidateSketch.cs new file mode 100644 index 0000000..d9ae929 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/samples/TemporalEvidenceCandidateSketch.cs @@ -0,0 +1,80 @@ +// Sketch only. Purpose: show how Mimir evidence can be lowered as candidates +// without becoming a second stable-track database beside Fensalir. + +using System; + +namespace Mimir.PerfectMachineStudy; + +public enum MimirEvidenceKindSketch +{ + Unknown = 0, + VisualFeature = 1, + ChirpAnchor = 2, + PassiveAudioAnchor = 3, + AcousticSource = 4 +} + +public readonly record struct MimirEvidenceCandidateSketch( + MimirEvidenceKindSketch Kind, + ulong StableSourceKey, + double CanonicalTimeSeconds, + float X, + float Y, + float Z, + float Radius, + float Confidence, + float Residual, + uint PayloadOffset, + uint PayloadLength); + +public readonly record struct FensalirReservoirCandidateSketch( + ulong StableSourceKey, + double CanonicalTimeSeconds, + float X, + float Y, + float Z, + float Radius, + float Weight, + uint Domain, + uint PayloadOffset, + uint PayloadLength); + +public static class TemporalEvidenceCandidateLoweringSketch +{ + public static int Lower( + ReadOnlySpan input, + Span output) + { + var count = 0; + for (var i = 0; i < input.Length && count < output.Length; i++) + { + ref readonly var candidate = ref input[i]; + if (candidate.Confidence <= 0.0f || candidate.Radius <= 0.0f) + { + continue; + } + + var residualPenalty = 1.0f / (1.0f + MathF.Max(0.0f, candidate.Residual)); + var weight = candidate.Confidence * residualPenalty; + if (weight <= 0.0f) + { + continue; + } + + output[count++] = new FensalirReservoirCandidateSketch( + candidate.StableSourceKey, + candidate.CanonicalTimeSeconds, + candidate.X, + candidate.Y, + candidate.Z, + candidate.Radius, + weight, + (uint)candidate.Kind, + candidate.PayloadOffset, + candidate.PayloadLength); + } + + return count; + } +} + diff --git a/research/perfect-machine-study-2026-05-23/state-machine-invariants.md b/research/perfect-machine-study-2026-05-23/state-machine-invariants.md new file mode 100644 index 0000000..385a19f --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/state-machine-invariants.md @@ -0,0 +1,178 @@ +# State Machine Invariants + +## Purpose + +The Perfect Machine is mostly a state discipline problem. This note lists the +state each subsystem may own and the state it must not smuggle in through a +friendly-looking helper. + +## Runtime State + +`MimirRuntime` may own: + +- runtime lifecycle; +- loaded configuration; +- witness emission policy; +- Fensalir scene state assembly; +- UI/debug readout surfaces. + +It must not own: + +- raw driver callback loops; +- audio DSP sample movement; +- dense visual tracks; +- OBS composition. + +## Hub State + +`MimirSynchronizationHub` may own: + +- registered sources; +- rolling buffer set; +- polling/update cadence; +- latest reports; +- smoothed synchronization states. + +It must not own: + +- analyzer internals; +- render tick work; +- hidden per-source DSP buffers; +- long-lived history outside retention window. + +## Rolling Buffer State + +Each rolling buffer owns: + +- one stream descriptor; +- bounded sample window; +- current edge; +- snapshot semantics. + +It must not own: + +- a second clock model; +- per-kind retention policy; +- decoded field tracks; +- calibration truth. + +## Active Decoder State + +The streaming decoder should own: + +- PCM window/ring; +- event proposal ring; +- candidate ambiguity ring; +- codebook/schedule view; +- local clock fit; +- response/confusion accumulation. + +It must not own: + +- UI state; +- emitter policy; +- DSP actuator state; +- global app timing authority. + +## Calibration State + +Calibration owns: + +- output/mic path identity; +- usable bands; +- confusion matrix; +- delay/bin-shift hypotheses; +- magnitude response; +- phase/group delay; +- recommended emission plan. + +It must not own: + +- raw capture buffers forever; +- UI-specific names as primary identity; +- unverified learned patches that cannot be replayed. + +## DSP Actuator State + +The actuator should own: + +- fractional delay line state; +- resampler phase; +- SRO control state; +- per-source gain/phase correction; +- audio-rate buffers. + +It must not own: + +- source discovery; +- calibration file persistence; +- UI controls except exposed parameters; +- timeline decoding. + +## Native Capture State + +Native capture workers own: + +- device handles; +- callback/read state; +- block/frame rings; +- device timestamps/counters; +- low-level format conversion where unavoidable. + +They must not own: + +- canonical timeline decisions; +- OBS routing; +- Fensalir render state; +- long-term calibration policy. + +## Fensalir State + +Fensalir owns: + +- window/input/render loop; +- D3D12 resource lifetime; +- external texture import; +- GPU fusion buffers; +- temporal evidence reservoir after lowering; +- program render surfaces. + +It must not own: + +- source capture configuration; +- audio clock authority; +- Mimir calibration truth; +- app-specific device policy. + +## Network Receiver State + +Remote nodes own: + +- local PCM ring; +- local decode state; +- local clock fit; +- local response/confusion observations; +- transport health. + +They must not own: + +- Starfire canonical clock authority; +- program output mixing; +- global calibration policy; +- trust decisions. + +## OBS State + +OBS owns: + +- final scene composition; +- stream/record controls; +- operator mix choices. + +It must not own: + +- stream synchronization; +- canonical timing; +- raw calibration truth; +- volumetric field reconstruction. + diff --git a/research/perfect-machine-study-2026-05-23/visual-fusion-4dgs-study.md b/research/perfect-machine-study-2026-05-23/visual-fusion-4dgs-study.md new file mode 100644 index 0000000..5305c38 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/visual-fusion-4dgs-study.md @@ -0,0 +1,169 @@ +# Visual Fusion And Realtime Gaussian Splatting Study + +## Thesis + +Mimir's visual side should not become a CPU image processing application. The +camera rig exists to produce time-indexed visual evidence that Fensalir can fuse +on GPU into a live volumetric program surface. + +The direct-driver work matters because sensor fusion is brutally sensitive to: + +- timestamp error; +- rolling/shutter latency; +- frame drops; +- calibration drift; +- duplicated CPU copies; +- inconsistent exposure/format state. + +## Current Sensor Roles + +- Leap stereo IR: near-field timing/depth/tracking candidate. +- PS3 Eyes: high-rate feature/tracking witnesses. +- Kiyo: stable RGB context. +- Kiyo Pro: intended RGB ground truth, currently high-speed/cadence limited. +- Raven-side devices: remote evidence producers, not local compute authority. + +## Evidence Layers + +### Raw Frame + +Minimum: + +- source id; +- device/arrival timestamp; +- dimensions/format/stride; +- native handle or byte span. + +### Feature Frame + +Possible GPU outputs: + +- corners/keypoints; +- descriptors; +- optical flow; +- silhouettes/masks; +- depth/disparity hints; +- confidence per feature. + +### Scene Claims + +Rolling reservoir views: + +- scene rays; +- surface claims; +- material claims; +- render packets. + +### Program Surface + +Fensalir output: + +- Gaussian splat/point cloud render; +- avatar/AR overlay; +- Spout2 program video. + +## Realtime Gaussian Splatting Relevance + +Gaussian splatting is attractive because it represents scenes as many local +anisotropic primitives that can be rasterized quickly. For Mimir, the hard part +is not rendering a static splat set. The hard part is maintaining a dynamic +time-indexed splat field from live unsynchronized cameras. + +Mimir needs: + +- synchronized observations; +- camera calibration; +- dynamic object handling; +- confidence and decay; +- GPU-resident update path; +- output that can be inspected/debugged live. + +The low-level implementation references all point at the same renderer spine: +project Gaussians, bin/intersect them into tiles, sort or partially order for +view-consistent blending, then rasterize tiles with tightly packed GPU buffers. +`gsplat`, NVIDIA's Vulkan sample, StopThePop, and FlashGS are useful because +they expose performance levers Mimir/Fensalir can reuse without pretending the +live rig is an offline training pipeline. + +Useful borrowed ideas: + +- packed structure-of-arrays splat buffers; +- tile-size as a first-order performance parameter; +- radius/opacity culling before sort/raster; +- global versus local/tile order tradeoffs; +- asynchronous CPU/GPU sorting only if it does not add frame latency; +- feature/depth render modes as diagnostics, not only RGB output. + +## 4D / Dynamic Splatting Shape + +```mermaid +flowchart TD + Frames["time-aligned camera frames"] --> Features["GPU features / masks / flow"] + Features --> Tracks["temporal tracks"] + Tracks --> Claims["surface/material claims"] + Claims --> Splats["dynamic Gaussian / point primitives"] + Audio["audio source estimates"] --> Claims + Splats --> Render["Fensalir program render"] +``` + +## Data Model Sketch + +```cpp +struct MimirVisualObservation +{ + uint64_t sourceHash; + uint64_t timestampNs; + uint32_t width; + uint32_t height; + uint32_t format; + uint64_t nativeHandle; +}; + +struct MimirSplatClaim +{ + uint64_t stableKeyHash; + float position[3]; + float covariance[6]; + float color[4]; + float confidence; + uint64_t sourceTimeMinNs; + uint64_t sourceTimeMaxNs; +}; +``` + +## Immediate Architecture Cut + +Do not start with "implement 4DGS." Start with the ownership: + +1. direct camera driver pushes frame handles; +2. runtime/reservoir indexes them; +3. Fensalir reads current synchronized window; +4. Fensalir produces simple point/render packet claims; +5. only then expand to dynamic splat update. + +## Micro-Optimization Risks + +- CPU demosaic/format conversion before GPU upload. +- Multiple copies of compressed camera frames. +- Re-decoding frame data per feature stage. +- Per-frame heap allocations in capture workers. +- Blocking on GPU readback for UI/debug. +- Sorting every splat globally every frame before the field is even stable. +- Treating 4DGS optimization loops as live capture prerequisites. + +## Research Threads To Continue + +- 3D Gaussian Splatting reference CUDA rasterizer. +- `gsplat` tile/raster APIs and CUDA memory layout. +- NVIDIA `vk_gaussian_splatting` mesh-shader versus vertex-shader paths. +- StopThePop / sorted Gaussian splatting for view-consistency tradeoffs. +- FlashGS-style kernel-level optimization and redundant-sort reduction. +- Dynamic/4D Gaussian splatting update strategies. +- PlayCanvas/WebGPU splat processing as a practical open implementation of + GPU-side splat mutation and browser-scale rendering constraints. +- Gaussian-SLAM and multi-sensor calibration papers as shape references for + online pose/calibration drift, even though Mimir's rig is room-bound rather + than vehicle-mounted. +- Online multi-camera calibration and temporal alignment. +- GPU optical flow/features for live camera arrays. +- Sensor-fusion priors from audio localization. diff --git a/research/perfect-machine-study-2026-05-23/volumetric-audio-field.md b/research/perfect-machine-study-2026-05-23/volumetric-audio-field.md new file mode 100644 index 0000000..2d30c50 --- /dev/null +++ b/research/perfect-machine-study-2026-05-23/volumetric-audio-field.md @@ -0,0 +1,191 @@ +# Volumetric Audio Field Study + +## Thesis + +Mimir's audio target is not "sync the mics." That is only the entry fee. The +target is a continuously updated room-scale sound field: where sources are, +which surfaces colored them, how each microphone hears them, and which +OBS-facing stems can be emitted without flattening useful spatial evidence. + +The current ASIO/chirp-bin work is the correct primitive because a microphone +array without precise timing is just a pile of unrelated waveforms. + +## Problem Decomposition + +### 1. Clock And Delay + +Every input stream needs: + +- delay relative to loopback/canonical program time; +- fractional delay below one sample; +- sampling-rate offset over time; +- confidence and failure state. + +This is the current `MimirAudioSynchronizationState` surface. + +### 2. Transfer Function + +Every output/mic path needs: + +- magnitude response by frequency; +- phase/group-delay response; +- confusion/alias behavior for coded active signals; +- reflection-prone bands and time regions. + +This is the current `MimirChirpBinCalibrationModel` surface, but it should grow +from "decoder helper" into "room path model." + +### 3. Source Localization + +Once streams are aligned, source localization can use: + +- TDOA between microphones; +- direct-path chirp time-of-arrival; +- beamforming/steered response; +- visual priors from camera/pose/splat state; +- semantic priors from dialogue/game sources. + +For Mimir, the first robust source localizer should probably be hybrid: + +1. use active chirp emitters for calibration and known anchors; +2. use TDOA/GCC-PHAT/SRP-PHAT for transient/passive events; +3. use visual tracking as a prior for voices/hands/objects. + +### 4. Field Reconstruction + +Field reconstruction can be represented at several resolutions: + +- low-order ambisonic bed for OBS-friendly spatial ambience; +- per-speaker/mic transfer function models; +- localized source objects with direction, distance, and confidence; +- room response map for reflections/suppression; +- neural or Gaussian acoustic field later, if justified. + +The first practical target is not a full pressure-field simulator. It is: + +- aligned dialogue anchors; +- camera mics as spatial witnesses; +- local loopback as program reference; +- co-streamer/Raven timing evidence; +- separable stems plus spatial bed. + +## Methods Worth Keeping Separate + +### GCC-PHAT / SRP-PHAT + +Use: + +- passive TDOA; +- transient localization; +- drift hints when program audio is correlated. + +Do not use: + +- canonical timeline identity. + +Reason: + +- It estimates relative delay from content, but content is not a unique clock + unless it is actively coded. + +### Beamforming + +Use: + +- voice/source enhancement once mic positions and delays are known. + +Risks: + +- Bad geometry or uncorrected phase response creates false confidence. +- Camera mics are low quality but spatially useful; Focusrites are high quality + but fewer. + +### Ambisonics + +Use: + +- OBS-facing spatial bed. +- Compact representation after alignment. + +Risks: + +- Ambisonics is an output/control representation, not the raw truth. Do not + throw away per-mic evidence too early. + +### Room Impulse Response Estimation + +Use: + +- speaker/mic path model; +- frequency response normalization; +- reflection windows; +- phase/group-delay calibration. + +Best current route: + +- Use chirp-bin codebook for continuous/lightweight operation. +- Use fuller sweeps or exponential sine sweeps for deliberate calibration + sessions. + +## Architecture Proposal + +```mermaid +flowchart TD + ASIO["ASIO inputs + loopback"] --> Align["delay/SRO estimator"] + Chirps["chirp-bin active witness"] --> Align + Music["program audio"] --> Passive["passive GCC-PHAT"] + Passive --> Align + Align --> Model["path response model"] + Align --> Actuator["fractional delay + ASRC"] + Model --> Actuator + Actuator --> Field["sound field estimator"] + Visual["Fensalir visual priors"] --> Field + Field --> Stems["voice/ambient/transient stems"] + Field --> Bed["spatial bed"] + Stems --> OBS["OBS"] + Bed --> OBS +``` + +## Data Model Sketch + +```csharp +public sealed record MimirAudioPathModel( + string OutputSourceId, + string ReceiverSourceId, + int SampleRate, + IReadOnlyList Magnitude, + IReadOnlyList GroupDelay, + IReadOnlyList Reflections, + IReadOnlyList ActiveConfusion, + double DirectPathDelaySamples, + double Confidence); + +public sealed record MimirSoundSourceEstimate( + string SourceClass, + Vector3 PositionMeters, + Vector3 VelocityMetersPerSecond, + double StartTimeNs, + double Confidence, + IReadOnlyList SupportingReceivers); +``` + +## Hot-Path Warning + +Do not implement volumetric audio as repeated whole-window cross-correlations +between every pair of streams. Pairwise all-to-all is fine for diagnosis and +small proofs; production needs: + +- reference-first timing; +- cached spectra; +- bounded candidate pairs; +- geometry-informed search; +- streaming state. + +## Research Threads To Continue + +- SRP-PHAT acceleration on GPU for small mic arrays. +- Low-order ambisonic encoding from irregular microphone arrays. +- Joint audio-visual source localization with tracked speaker/head priors. +- Online room impulse response tracking under moving sources. +- Neural acoustic fields only after classical calibration stops scaling. +