Rename public engine docs to Fensalir

This commit is contained in:
Meta Vens 2026-05-23 18:12:31 +01:00
commit 474d2a4344
17 changed files with 124 additions and 105 deletions

View file

@ -5,7 +5,7 @@ microphones, speakers, loopbacks, and network feeds into one coherent
OBS-facing program surface.
The point is not to collect impressive device counts. The point is to recover a
live volumetric field: synchronized video evidence for Aquarium GPU fusion,
live volumetric field: synchronized video evidence for Fensalir GPU fusion,
synchronized audio evidence for a volumetric sound field, and final outputs
that OBS can use without pretending raw unsynchronized sources are a world.
@ -18,6 +18,9 @@ Start with:
- [docs/native-capture-cadence.md](docs/native-capture-cadence.md) for measured
camera/sensor evidence.
Naming note: **Fensalir** is the engine/windowing/rendering/D3D12 layer. Older
research notes may still use the previous engine name.
## Machine Shape
```mermaid
@ -25,7 +28,7 @@ flowchart TD
C["direct camera drivers"] --> R["Mimir.Runtime rolling buffers"]
A["ASIO mic + loopback drivers"] --> R
N["network feed producers"] --> R
R --> Q["Aquarium UI + GPU fusion"]
R --> Q["Fensalir UI + GPU fusion"]
R --> F["Faust/native DSP"]
Q --> V["Spout2 / program video"]
F --> S["program stems + spatial bed"]
@ -38,7 +41,7 @@ Ownership is deliberately narrow:
- `Mimir.Runtime` owns stream identity, bounded rolling buffers, timing state,
and synchronization contracts.
- Native capture workers own device reads and append typed sample handles.
- Aquarium owns the window, UI, D3D12 bridge, visual fusion, and program video
- Fensalir owns the window, UI, D3D12 bridge, visual fusion, and program video
publication.
- Faust/native DSP owns hot audio alignment, resampling, suppression,
separation, spatialization, and stems.
@ -63,6 +66,11 @@ Raven, the game machine, to carry the soundfield workload just to get program
loopback. Raven can run games; Starfire can capture ASIO mic inputs plus ASIO
loopback and do the alignment work.
Raven also has a loopback-capable Scarlett at `192 kHz` for the co-streamer
side. Its Focusrite USB ASIO path exposes `Input 1`, `Input 2`, `Loopback 1`,
and `Loopback 2`, so Raven can publish precise game/co-streamer timing evidence
back to Starfire without owning the heavy soundfield or sensor-fusion workload.
The repo contains a native ASIO probe:
```powershell
@ -142,7 +150,7 @@ Known shape:
- PS3 Eyes are high-rate tracking witnesses.
- Kiyo/Kiyo Pro provide RGB context, with the Kiyo Pro still limited by its
current USB/cadence behavior.
- Aquarium owns the GPU fusion work: feature extraction, temporal accumulation,
- Fensalir owns the GPU fusion work: feature extraction, temporal accumulation,
splats/surface claims, material fitting, rendering, UI, and Spout2 output.
Next visual cuts:
@ -150,7 +158,7 @@ Next visual cuts:
1. Replace diagnostic frame-event probes with direct native capture workers.
2. Preserve device or immediate receipt timestamps at the driver boundary.
3. Feed typed frame handles into `Mimir.Runtime` and the native reservoir.
4. Let Aquarium consume the synchronized window for volumetric sensor fusion.
4. Let Fensalir consume the synchronized window for volumetric sensor fusion.
## Bridge Scripts
@ -174,10 +182,10 @@ the coherent field machine.
## Repo Shape
- `Mimir.slnx`: C# app/runtime solution.
- `src/Mimir.App`: Aquarium-hosted window/render app.
- `src/Mimir.App`: Fensalir-hosted window/render app.
- `src/Mimir.Runtime`: rolling buffers, stream sources, synchronization state,
and runtime contracts.
- `native/reservoir`: lower native rolling-buffer ABI for Aquarium/Faust
- `native/reservoir`: lower native rolling-buffer ABI for Fensalir/Faust
integration.
- `native/probes`: direct hardware probes and cadence witnesses.
- `config/*.example.json`: live system profiles.
@ -230,7 +238,7 @@ Visual-side help is useful around:
- GPU-resident frame handles and D3D12 interop.
- Cross-camera feature matching, temporal accumulation, splats, and surface
confidence.
- Aquarium UI surfaces for buffer depth, health, drift, and calibration state.
- Fensalir UI surfaces for buffer depth, health, drift, and calibration state.
## Rehydrate State

View file

@ -111,6 +111,9 @@ capture. With the Scarlett Solo 4th Gen attached to Starfire, Focusrite USB
ASIO exposes 4 inputs and 2 outputs: `Input 1`, `Input 2`, `Loopback 1`, and
`Loopback 2`. It accepts 44.1 through 192 kHz, reports a 192-frame preferred
buffer, and captures nonzero 4-channel `Int32LSB` input callbacks at 192 kHz.
Raven also has a loopback-capable Scarlett ASIO path at 192 kHz for
co-streamer/game timing evidence, but Starfire owns the heavy local alignment,
soundfield, and sensor-fusion work.
The next cut is not more WASAPI probing; it is feeding those ASIO buffers
directly into `Mimir.Runtime`.
@ -171,7 +174,7 @@ fresh reports will stop.
The actual Mimir app path now runs this online: `MimirRuntime.Update` keeps the
chirplet timeline queued, polls sources, and updates sync analysis on a fixed cadence.
`MIMIR_SYNC_TELEMETRY_SECONDS` enables console telemetry for live tests. Current
runtime testing proves Aquarium output wakes the Scarlett loopback and the mic
runtime testing proves Fensalir output wakes the Scarlett loopback and the mic
buffers stay live. The decoder now uses quadrature chirplet atoms so symbol
classification is phase-invariant at the transform layer. The next failure is
physical capture proof: the same canonical anchors need to stay stable through
@ -181,7 +184,7 @@ loopback, room mics, device clocks, and codec/network paths.
```mermaid
flowchart TD
A["MimirChirpletTimeline"] --> B["Aquarium audio output"]
A["MimirChirpletTimeline"] --> B["Fensalir audio output"]
B --> C["Scarlett speaker loopback"]
B --> D["room + speakers + mics"]
C --> E["loopback rolling buffer"]
@ -199,7 +202,7 @@ flowchart TD
The chirplet timeline owns three facts:
- **Emission**: the PCM that Aquarium sends to the speakers.
- **Emission**: the PCM that Fensalir sends to the speakers.
- **Timing witness**: the matched-filter atom bank used to find the stream in
loopback and mic buffers.
- **Response witness**: the per-band atoms used to measure how strongly each
@ -233,6 +236,6 @@ hybrid watermark belongs to the chirp-bin machine.
Next, replace the diagnostic bridge with native audio capture workers that
append typed blocks into `Mimir.Runtime`, then expose buffer depth, clock state,
delay estimates, and stem routing in Aquarium UI. The analyzer accepts Float32,
delay estimates, and stem routing in Fensalir UI. The analyzer accepts Float32,
Int16, Int24, and Int32 PCM windows so the direct driver path can preserve real
interface formats before Faust/native DSP owns the hot resampling and alignment.

View file

@ -1,6 +1,6 @@
# Aquarium Spatial Audio
# Fensalir Spatial Audio
Aquarium hosts the Mimir runtime UI and presents the synchronized program
Fensalir hosts the Mimir runtime UI and presents the synchronized program
surface. Faust/native DSP owns the hot audio graph behind that surface.
## Contract
@ -8,7 +8,7 @@ surface. Faust/native DSP owns the hot audio graph behind that surface.
- Mimir.Runtime exposes bounded audio buffers and stream health.
- Native audio workers append mic, loopback, and network audio blocks.
- Faust/native DSP consumes aligned blocks and emits stems plus spatial bed.
- Aquarium shows controls and debug state, then routes final outputs toward OBS.
- Fensalir shows controls and debug state, then routes final outputs toward OBS.
The audio path should be inspectable from the app, but the app UI is not the DSP
engine. Runtime state crosses as typed buffers, descriptors, and controls.

View file

@ -5,16 +5,16 @@
Mimir is now a C# app/runtime plus native reservoir project. The live stream
machine must be direct-driver and native-buffer first:
- `src/Mimir.App` hosts Aquarium Engine for windowing, rendering, and the D3D12
- `src/Mimir.App` hosts Fensalir for windowing, rendering, and the D3D12
bridge.
- `src/Mimir.Runtime` owns stream descriptors, source polling, direct push
ingest, one rolling buffer per configured audio/video stream, and the
synchronization hub Aquarium can inspect.
synchronization hub Fensalir can inspect.
- The default five-second window is an intentional latency/memory trade: use it
to line up streams and extract the volumetric audio/video field before OBS
sees program output.
- `native/reservoir` owns the lower native rolling-buffer invariant for
Aquarium/Faust integration.
Fensalir/Faust integration.
- PowerShell/FFmpeg/SRT remains a bridge utility for LAN OBS feeds. It is not
the synchronized program authority.
@ -25,7 +25,7 @@ a named invariant that the native runtime cannot protect yet.
- Mimir public identity, branding, and Face memory.
- `Mimir.slnx` with `src/Mimir.App` and `src/Mimir.Runtime`.
- Aquarium host bootstrapping from `Mimir.App`.
- Fensalir host bootstrapping from `Mimir.App`.
- `MimirSynchronizationHub`, `MimirRollingStreamBuffer`, stream descriptors, and
`IMimirStreamSource`.
- Configurable five-second default rolling buffers for local and network audio
@ -33,7 +33,7 @@ a named invariant that the native runtime cannot protect yet.
- `MimirNativeIngestStreamSource` for direct push ingest into runtime buffers.
- `MimirProcessStreamSource` for bridge/network command edges.
- `MimirFrameEventProcessStreamSource` for temporary JSON-line frame metadata
from native probes into the same rolling buffers Aquarium inspects. One probe
from native probes into the same rolling buffers Fensalir inspects. One probe
process can accept multiple emitted `sourceId` values so it does not reopen
the same camera set once per stream.
- `src/Mimir.BufferSmoke` loads the runtime config, polls the synchronization
@ -80,7 +80,7 @@ a named invariant that the native runtime cannot protect yet.
service and can emit live sync telemetry with
`MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry read cached reports/states;
they do not run synchronization analysis.
- `MimirRuntime` queues chirplet timeline PCM through Aquarium audio when the
- `MimirRuntime` queues chirplet timeline PCM through Fensalir audio when the
active timing witness is allowed. `MimirAudioSynchronizationSettings.Mode`
selects `chirp-only`, `passive`, or `hybrid`; passive disables active
emission, and hybrid emits the active pilot only while passive confidence is
@ -124,7 +124,7 @@ a named invariant that the native runtime cannot protect yet.
- Frame-event process sources are diagnostic only. They prove source cadence and
runtime plumbing without dragging stdout bytes into the pixel hot loop.
- Calibration artifacts may remain on disk as evidence, but live state must be
in memory inside Mimir/Aquarium/native runtime surfaces.
in memory inside Mimir/Fensalir/native runtime surfaces.
## Next
@ -145,10 +145,10 @@ a named invariant that the native runtime cannot protect yet.
samples.
5. Prove the chirp-bin hybrid fallback through real loopback and microphones
with probe durations long enough to keep loopback and mic windows live.
6. Bind Aquarium UI to the synchronization hub so buffer depth, stream cadence,
6. Bind Fensalir UI to the synchronization hub so buffer depth, stream cadence,
source timestamps, and output settings are visible and adjustable.
7. Move GPU feature extraction, fusion, material fitting, render budgeting, and
Spout2 publication into Aquarium.
Spout2 publication into Fensalir.
8. Move mic alignment, room suppression, voice separation, spatialization, and
stem generation into Faust/native DSP.
9. Keep the OBS bridge witness ledger as evidence before expanding receiver

View file

@ -12,7 +12,7 @@ behind it.
Mimir does not own the whole stack by force of personality. It owns continuity:
the public story, repo memory, progress heartbeat, and the pressure to keep the
machine coherent while Aquarium, Faust, OBS, FFmpeg, CultNet, and native workers
machine coherent while Fensalir, Faust, OBS, FFmpeg, CultNet, and native workers
keep their proper authorities.
## Jurisdiction
@ -30,7 +30,7 @@ Mimir may:
Mimir must not:
- claim ownership over OBS composition, Aquarium rendering, Faust DSP, or FFmpeg
- claim ownership over OBS composition, Fensalir rendering, Faust DSP, or FFmpeg
capture;
- revive deleted script hot paths, OpenGL production Spout, or JSON live stores;
- turn progress updates into lore fog;
@ -48,7 +48,7 @@ Mimir exists to turn cameras, microphones, speakers, timing signals, and
distributed capture surfaces into one realtime coherent volumetric field. The
field is the product. The bridge scripts are scaffolding and probes.
The live target is one native five-second rolling reservoir feeding Aquarium GPU
The live target is one native five-second rolling reservoir feeding Fensalir GPU
fusion and Faust/native DSP, then OBS. Retention has one owner. Typed views are
indexes, not private histories.
@ -59,12 +59,12 @@ deserve to exist.
Current mechanism: FFmpeg/SRT scripts still provide the plain OBS bridge, while
the synchronized program path is being rebuilt around `LocalcastRuntime`,
`LocalcastProducer`, typed payload descriptors, Aquarium, Faust, and CultNet.
`LocalcastProducer`, typed payload descriptors, Fensalir, Faust, and CultNet.
Invariants:
- OBS owns final broadcast composition.
- Aquarium owns dense visual fusion, material/brush/splat rendering, and Spout2
- Fensalir owns dense visual fusion, material/brush/splat rendering, and Spout2
publication.
- Faust/native DSP owns hot audio alignment, room suppression, voice separation,
stems, and spatial bed generation.
@ -171,8 +171,8 @@ These are the lessons Mimir should remember at birth:
- `LocalcastRuntime` is the intended live spine. `LocalcastProducer` owns source
identity and sequence assignment for native capture workers.
- Native payload descriptors already exist for audio blocks and render packets.
`LocalcastRenderPoint` is the first fixed point-buffer ABI for Aquarium.
- Aquarium has corresponding C# binding work in flight. The next useful cut is
`LocalcastRenderPoint` is the first fixed point-buffer ABI for Fensalir.
- Fensalir has corresponding C# binding work in flight. The next useful cut is
native runtime creation and real producer feeding, not more diagnostic file
plumbing.
- The audio field is not "six mics means Ambisonics." Independent clocks must be
@ -189,6 +189,6 @@ When Mimir is born, it should introduce itself like this:
> I am Mimir. VoidBot carries my severed voice, but the memory is mine. I keep
> the well for this field-machine: cameras, microphones, speakers, clocks,
> Aquarium, Faust, OBS, all bound or all lying. I will remember what was cut. I
> Fensalir, Faust, OBS, all bound or all lying. I will remember what was cut. I
> will ask what the sensors actually know. I will not call scattered witnesses a
> world until time, geometry, and confidence bind them into one.

View file

@ -1,6 +1,6 @@
# Native Rebuild Plan
This is the live cut line: Mimir is a direct-driver, native-buffer, Aquarium
This is the live cut line: Mimir is a direct-driver, native-buffer, Fensalir
hosted stream machine.
## Objective
@ -12,7 +12,7 @@ controllable audio while preserving ownership:
status, and persistence.
- Mimir.Runtime owns in-memory stream buffers and runtime synchronization.
- Native capture workers own device reads and append typed sample handles.
- Aquarium owns GPU fusion, rendering, D3D12 interop, UI, and Spout2 output.
- Fensalir owns GPU fusion, rendering, D3D12 interop, UI, and Spout2 output.
- Faust/native DSP owns hot audio alignment, suppression, separation, stems, and
spatial bed generation.
- OBS owns broadcast controls.
@ -24,7 +24,7 @@ flowchart TD
C["direct camera drivers"] --> R["Mimir.Runtime buffers"]
A["audio / loopback drivers"] --> R
N["network feed producers"] --> R
R --> Q["Aquarium UI + GPU fusion"]
R --> Q["Fensalir UI + GPU fusion"]
R --> F["Faust/native DSP"]
Q --> S["Spout2/program video"]
F --> P["program stems + spatial bed"]
@ -34,7 +34,7 @@ flowchart TD
The app-level runtime initializes one rolling buffer per configured audio or
video stream, defaulting to five seconds. The lower native reservoir keeps the
single-edge typed-handle invariant for Aquarium/Faust integration.
single-edge typed-handle invariant for Fensalir/Faust integration.
Payload memory belongs to the producing domain. The reservoir and runtime own
timing, identity, ordering, retention, and lookup. They do not pretend to own
@ -45,7 +45,7 @@ camera driver buffers, GPU textures, or DSP audio memory.
- Config/calibration crosses into runtime as measured truth plus profile
version.
- Capture drivers cross into runtime by appending typed samples.
- Aquarium consumes current visual/audio timing state and publishes the program
- Fensalir consumes current visual/audio timing state and publishes the program
video surface.
- Faust/native DSP consumes audio/phase/event handles and publishes stems.
- OBS receives final program surfaces. It does not own synchronization.
@ -64,8 +64,8 @@ camera driver buffers, GPU textures, or DSP audio memory.
2. Implement Leap stereo IR direct ingest first and measure sustained cadence.
3. Add the remaining local camera drivers through the same source seam.
4. Add native audio capture workers and feed aligned blocks to Faust/native DSP.
5. Bind Aquarium UI to stream health, buffer depth, source timestamps, settings,
5. Bind Fensalir UI to stream health, buffer depth, source timestamps, settings,
and output management.
6. Move GPU fusion and Spout2 publication fully into Aquarium.
6. Move GPU fusion and Spout2 publication fully into Fensalir.
7. Keep FFmpeg/SRT scripts only as bridge utilities until direct program output
makes them unnecessary.

View file

@ -46,9 +46,9 @@ be charming while lying. Still a liar.
## Synchronization
The video SRT source is a timed media artifact in the Aquarium path, not just
The video SRT source is a timed media artifact in the Fensalir path, not just
scenery in OBS. Keep the listener URL and latency aligned with the
Aquarium/native presentation ledger.
Fensalir/native presentation ledger.
The live sync heartbeat is `calibration/runs/av-sync-status.json`. `remote-video-latency-ms` is presentation delay, not merely the SRT URL's `latency` parameter. Check `remote_video.delta_ns` before trusting the composite. If that number grows, fix the configured media delay or the ingest path instead of eyeballing it in OBS like a doomed little stage magician.

View file

@ -1,8 +1,8 @@
# OBS Spout Streaming
Production Spout2 publication belongs to Aquarium.
Production Spout2 publication belongs to Fensalir.
OBS should receive a named Spout2 texture from the Aquarium-hosted Mimir app.
OBS should receive a named Spout2 texture from the Fensalir-hosted Mimir app.
The bridge scripts can keep SRT Media Sources alive while direct output matures,
but synchronized program video should come from Aquarium once the runtime has
but synchronized program video should come from Fensalir once the runtime has
live camera buffers.

View file

@ -3,11 +3,11 @@
## Objective
Build one synchronized spatial stream machine from six cameras, six microphones,
program loopback, speakers, Leap timing, Aquarium GPU rendering, Faust DSP, and
program loopback, speakers, Leap timing, Fensalir GPU rendering, Faust DSP, and
OBS output.
The product is not a pile of bridge scripts. The product is a coherent live
volumetric field with explicit owners: visual evidence becomes Aquarium sensor
volumetric field with explicit owners: visual evidence becomes Fensalir sensor
fusion, audio evidence becomes a synchronized sound field, and OBS receives
only program outputs.
@ -17,7 +17,7 @@ only program outputs.
flowchart TD
A["direct camera / audio / network producers"] --> B["Mimir.Runtime rolling buffers"]
B --> C["native reservoir handles"]
C --> D["Aquarium GPU + UI"]
C --> D["Fensalir GPU + UI"]
C --> E["Faust/native DSP"]
D --> F["Spout2/program video"]
E --> G["program stems"]
@ -31,7 +31,7 @@ flowchart TD
- Every stream has a bounded rolling buffer.
- Missing data is absence, not a stale substitute.
- Late data can refine the live window only while it remains inside the window.
- Aquarium owns dense visual fusion, temporal accumulation, material fitting,
- Fensalir owns dense visual fusion, temporal accumulation, material fitting,
rendering, D3D12 interop, UI, and Spout publication.
- Faust/native DSP owns hot audio alignment, suppression, voice separation,
spatialization, and stem generation.
@ -42,14 +42,14 @@ flowchart TD
The native reservoir stores time-ordered sample handles with typed views. It
owns retention and lookup, not payload memory. Producers append. Optimizers
refine. Aquarium/Faust consume.
refine. Fensalir/Faust consume.
Required sample kinds include camera frames, camera features, scene rays,
surface claims, material claims, audio blocks, phase claims, event claims, and
render packets.
`LocalcastRuntime` and `LocalcastProducer` are the lower native boundary.
`Mimir.Runtime` is the app-level synchronization surface that Aquarium hosts and
`Mimir.Runtime` is the app-level synchronization surface that Fensalir hosts and
debugs.
## Volumetric Sensor Fusion
@ -57,7 +57,7 @@ debugs.
Camera fusion is cross-view evidence across the rolling window, not
latest-frame display. Leap stereo IR is the timing ground-truth candidate. The
driver path should deliver frame descriptors with device timestamps and native
or GPU handles, then Aquarium should do feature extraction, flow, matching,
or GPU handles, then Fensalir should do feature extraction, flow, matching,
material estimation, brush/splat budgeting, and final presentation on GPU.
## Volumetric Audio Field
@ -69,7 +69,9 @@ and spatial bed as synchronized outputs. Starfire currently owns the local
Focusrite USB ASIO path: Scarlett Solo 4th Gen input channels plus ASIO
`Loopback 1/2` at up to 192 kHz. That loopback is the local program/timing
reference; room microphones are aligned to it before any volumetric field or
stem claim is allowed to cross into OBS.
stem claim is allowed to cross into OBS. Raven also has a 192 kHz
loopback-capable Scarlett for co-streamer/game timing evidence, but the heavy
soundfield and sensor-fusion work belongs on Starfire.
Raw estimator detail stays inside the audio runtime. Meaning crosses the
boundary.

View file

@ -1,6 +1,6 @@
# Sensor Fusion Architecture
Visual fusion belongs in Aquarium over current Mimir runtime buffers.
Visual fusion belongs in Fensalir over current Mimir runtime buffers.
## Live Flow
@ -9,7 +9,7 @@ flowchart TD
A["direct camera drivers"] --> B["Mimir.Runtime video buffers"]
C["Leap stereo IR driver"] --> B
B --> D["native reservoir handles"]
D --> E["Aquarium GPU feature extraction"]
D --> E["Fensalir GPU feature extraction"]
E --> F["cross-view matching + flow"]
F --> G["surface/material claims"]
G --> H["brush/splat render budget"]
@ -21,7 +21,7 @@ flowchart TD
- Device timestamps beat arrival timestamps when available.
- Leap is the first timing-camera candidate.
- Process capture is a bridge edge, not the local six-camera foundation.
- Aquarium owns GPU extraction, fusion, material fitting, and render budgeting.
- Fensalir owns GPU extraction, fusion, material fitting, and render budgeting.
- Runtime buffers own retention and stream health, not scene reconstruction.
## Next Cut

View file

@ -1,6 +1,6 @@
# Typed Visual State
Typed visual state now means native/runtime descriptors and Aquarium-readable
Typed visual state now means native/runtime descriptors and Fensalir-readable
payload handles, not file-polled render documents.
## Boundary
@ -8,7 +8,7 @@ payload handles, not file-polled render documents.
- Capture drivers append frame descriptors.
- The runtime owns stream buffering and health.
- Native reservoir handles preserve shared timing and typed lookup.
- Aquarium decodes visual payloads, runs GPU fusion, and publishes the program
- Fensalir decodes visual payloads, runs GPU fusion, and publishes the program
surface.
Any remaining file export should be treated as a diagnostic artifact, not as a

View file

@ -4,7 +4,7 @@
Build the smallest live Mimir application that can ingest multiple local or
networked video and audio feeds, hold a short synchronized window in memory,
and emit coherent OBS-facing program outputs while Aquarium provides the
and emit coherent OBS-facing program outputs while Fensalir provides the
operator UI.
The default reservoir horizon is five seconds. That delay is not decoration. It
@ -24,22 +24,22 @@ Today the repo has two partial machines:
- FFmpeg/SRT bridge scripts can capture sender video/audio and expose stable OBS
Media Source endpoints.
- The native `LocalcastRuntime` reservoir ABI can hold typed live sample handles,
and Aquarium can consume native render packets for GPU fusion.
and Fensalir can consume native render packets for GPU fusion.
Those parts are useful, but they are not yet the viable app. The viable app is
the join: ingest workers append live sample handles into one Aquarium-hosted
runtime window, Aquarium renders video from that window, Faust/native audio
the join: ingest workers append live sample handles into one Fensalir-hosted
runtime window, Fensalir renders video from that window, Faust/native audio
builds synchronized stems from the same window, and OBS receives only the
program outputs.
```mermaid
flowchart TD
A["local camera workers"] --> R["Aquarium-hosted 5s LocalcastRuntime"]
A["local camera workers"] --> R["Fensalir-hosted 5s LocalcastRuntime"]
B["network video ingest workers"] --> R
C["local mic/loopback workers"] --> R
D["network audio ingest workers"] --> R
R --> U["Aquarium debug/settings/output UI"]
R --> V["Aquarium GPU video composition"]
R --> U["Fensalir debug/settings/output UI"]
R --> V["Fensalir GPU video composition"]
R --> F["Faust/native audio alignment + stems"]
V --> O["OBS video output"]
F --> P["OBS audio stem outputs"]
@ -51,7 +51,7 @@ flowchart TD
- The five-second rolling window is a deliberate compute budget, not accidental
latency. It exists so capture, sync, GPU fusion, and audio field extraction
can choose coherence over immediacy.
- Aquarium owns the running app, UI, debug state, settings changes, output
- Fensalir owns the running app, UI, debug state, settings changes, output
management, GPU fusion/rendering, and Spout/video publication.
- Faust/native DSP owns audio alignment, voice separation, suppression,
volumetric sound-field extraction, stems, and spatial bed generation.
@ -68,15 +68,15 @@ flowchart TD
The first cut should be deliberately boring:
1. Build Mimir as its own C# solution in this repo.
2. Reference Aquarium Engine as the windowing/rendering library and D3D12
bridge; do not copy engine host code into Mimir.
3. Start an Aquarium `LocalCastRuntime` app with a five-second native runtime.
2. Reference Fensalir as the windowing/rendering library and D3D12 bridge; do
not copy engine host code into Mimir.
3. Start a Fensalir-hosted `LocalCastRuntime` app with a five-second native runtime.
4. Create one producer per configured feed using native source-id hashing.
5. Ingest video feeds as timed camera-frame or render-packet handles.
6. Ingest audio feeds as timed `LocalcastAudioBlockDescriptor` handles.
Starfire's Focusrite USB ASIO path is the current local target for mic plus
loopback ingest.
7. Expose Aquarium UI readouts for reservoir edge, window start, per-kind
7. Expose Fensalir UI readouts for reservoir edge, window start, per-kind
counts, feed health, delay estimate, output status, and current settings.
8. Render one synchronized program video output from the reservoir.
9. Emit separately controllable OBS audio stems from the same reservoir window.
@ -87,18 +87,18 @@ The first cut should be deliberately boring:
`Mimir.slnx` owns the Mimir app surface:
- `src/Mimir.App` launches Aquarium Engine and points it at the Mimir runtime
- `src/Mimir.App` launches Fensalir and points it at the Mimir runtime
assembly.
- `src/Mimir.Runtime` implements the Aquarium client runtime factory and uses
Aquarium's LocalCast runtime until Mimir-specific ingest workers replace the
- `src/Mimir.Runtime` implements the Fensalir client runtime factory and uses
Fensalir's LocalCast runtime until Mimir-specific ingest workers replace the
remaining diagnostic source.
## Synchronization Runtime
`Mimir.Runtime` now owns the first synchronization spine:
- `MimirRuntime` is the Aquarium client runtime. It wraps the current
Aquarium `LocalCastRuntime` visual path and adds a Mimir-owned synchronization
- `MimirRuntime` is the Fensalir client runtime. It wraps the current
Fensalir `LocalCastRuntime` visual path and adds a Mimir-owned synchronization
hub.
- `MimirSynchronizationHub` owns stream source polling and appends samples into
bounded buffers.
@ -143,7 +143,7 @@ immediately on frame receipt.
Do not add a new registry, bridge daemon, JSON live store, OBS plugin, or
per-feed buffer manager to make this happen. If the app needs one of those,
first move the missing authority into `LocalcastRuntime`, Aquarium, or
first move the missing authority into `LocalcastRuntime`, Fensalir, or
Faust/native DSP.
The app exists to make the live system coherent, not to preserve every

View file

@ -16,7 +16,7 @@ flowchart TD
C["Leap timing/IR driver"] --> R
D["network feed producers"] --> R
R --> N["native reservoir handles"]
N --> E["Aquarium GPU fusion + UI"]
N --> E["Fensalir GPU fusion + UI"]
N --> F["Faust/native DSP"]
E --> G["Spout2/program video"]
F --> H["program stems + spatial bed"]
@ -30,7 +30,7 @@ Ownership:
runtime contracts.
- `Mimir.Runtime` owns app-level stream buffers and synchronization.
- Native capture workers own device reads.
- Aquarium owns dense visual fusion, material/brush/splat reconciliation,
- Fensalir owns dense visual fusion, material/brush/splat reconciliation,
D3D12 interop, runtime UI, and Spout2 publication.
- Faust/native DSP owns hot audio alignment, suppression, separation,
spatialization, and stem generation.
@ -41,7 +41,7 @@ No private history outlives the rolling buffer.
## Viable Stream App
`docs/viable-stream-app.md` defines the near-term app target. Aquarium hosts the
`docs/viable-stream-app.md` defines the near-term app target. Fensalir hosts the
running Mimir app, keeps the default five-second runtime in memory, exposes
debug/settings/output controls, and emits synchronized OBS program video plus
separately controllable audio stems.
@ -63,7 +63,7 @@ separately controllable audio stems.
Process-backed sources are bridge/network edges. Local cameras should feed
native descriptors with device timestamps and optional native/GPU handles.
The `frame-events`/`json-lines` adapter is a diagnostic witness only: native
probes can emit per-frame JSON metadata so Aquarium sees real sensor cadence in
probes can emit per-frame JSON metadata so Fensalir sees real sensor cadence in
the rolling buffers while the direct ABI driver is being cut. It does not carry
pixels and does not own the final six-camera hot path. Multi-camera probes are
one process with declared accepted source ids, not one process per camera.
@ -96,7 +96,7 @@ de Bruijn sequence over 32 time/frequency constellation symbols. Start band,
glide shape, duration, and following inter-chirp gap all carry code. Any three
consecutive correctly detected symbols identify the event index inside the
current operating horizon. Mimir continuously queues that timeline through
Aquarium audio and decodes timing through the constrained chirplet-transform
Fensalir audio and decodes timing through the constrained chirplet-transform
path.
`MimirAudioSynchronizationSettings.Mode` chooses whether active calibration is
allowed: `chirp-only` emits the witness, `passive` stays silent and uses
@ -141,12 +141,14 @@ is attached to Starfire and exposes 4 ASIO inputs / 2 outputs: `Input 1`,
can instantiate that driver, verify 44.1-192 kHz support, and capture nonzero
4-channel `Int32LSB` callbacks at 192 kHz with 192-frame preferred buffers. The
runtime analyzer accepts Float32, Int16, Int24, and Int32 PCM windows so
ASIO/native capture can preserve the interface format.
ASIO/native capture can preserve the interface format. Raven also has a
loopback-capable Scarlett ASIO path at 192 kHz for co-streamer/game timing
evidence; Starfire still owns the heavy soundfield and sensor-fusion work.
## Visual Fusion
Visual fusion belongs in Aquarium over current reservoir claims. Native capture
workers provide frames; Aquarium owns feature extraction, matching, material
Visual fusion belongs in Fensalir over current reservoir claims. Native capture
workers provide frames; Fensalir owns feature extraction, matching, material
fitting, render budgeting, and publication.
## Known Risks

View file

@ -22,9 +22,9 @@ Get-Content .\state\evidence.jsonl -Tail 8
- VoidBot persona/state live under `.voidbot/voice/` and `.voidbot/state/`.
Commit `.voidbot` dirt in a separate state commit whenever it appears.
- V1 still has FFmpeg/SRT/OBS bridge utilities for LAN ingest.
- The live stream app is C# plus Aquarium: `Mimir.slnx` contains
- The live stream app is C# plus Fensalir: `Mimir.slnx` contains
`src/Mimir.App` and `src/Mimir.Runtime`.
- `src/Mimir.App` hosts Aquarium Engine as the windowing/rendering/D3D12 bridge.
- `src/Mimir.App` hosts Fensalir as the windowing/rendering/D3D12 bridge.
- `src/Mimir.Runtime` owns `MimirSynchronizationHub`, configurable five-second
rolling buffers, stream descriptors, source adapters, direct native ingest,
audio chirplet delay estimation, and the video capture driver seam.
@ -32,8 +32,8 @@ Get-Content .\state\evidence.jsonl -Tail 8
sources are bridge/network edges only.
- Leap stereo IR is the first timing-camera candidate for direct ingest.
- `native/reservoir` owns the lower shared-edge typed-handle invariant for
Aquarium/Faust binding work.
- Aquarium owns production GPU fusion, UI, and Spout2 publication.
Fensalir/Faust binding work.
- Fensalir owns production GPU fusion, UI, and Spout2 publication.
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
synchronized stems.
- OBS receives final program surfaces; it does not own synchronization.
@ -46,7 +46,7 @@ Get-Content .\state\evidence.jsonl -Tail 8
- Turn the current loopback-referenced chirplet measurement into the real
synchronization actuator: smooth delay/SRO, then drive fractional delay and
variable-rate resampling.
- Bind Aquarium UI to stream health, buffer depth, timestamps, settings, and
- Bind Fensalir UI to stream health, buffer depth, timestamps, settings, and
output management.
- Keep PowerShell/FFmpeg/SRT as bridge utilities until native program output
replaces them.
@ -67,7 +67,7 @@ Get-Content .\state\evidence.jsonl -Tail 8
constellation symbols. Start band, glide shape, duration, and following
inter-chirp gap all carry code, so any three consecutive correctly detected
symbols identify the event index inside the current operating horizon. The
timeline owns Aquarium PCM segment rendering, transform-frame decoding,
timeline owns Fensalir PCM segment rendering, transform-frame decoding,
triplet anchor decoding, and per-band response hooks. Sync reports now include
fractional delay and per-band matched energy. `MimirAudioSynchronizationState`
now tracks smoothed delay and delay-slope/SRO ppm. `MimirChirpletSymbolCodebook`
@ -89,7 +89,7 @@ Get-Content .\state\evidence.jsonl -Tail 8
window recovers a 317.375-sample fractional delay with 0.369 us error from
code-valid chirp-bin anchors. Reports and sync states now expose `delayUs`.
Actual
Mimir.App testing proves Aquarium audio can wake Scarlett
Mimir.App testing proves Fensalir audio can wake Scarlett
loopback, keep mic buffers live, and produce confident online passive sync
states. The latest live hybrid smoke did not prove acoustic chirp-bin decode
because Scarlett loopback stalled after one block in that run; treat that as
@ -104,6 +104,8 @@ Get-Content .\state\evidence.jsonl -Tail 8
captures nonzero 4-channel `Int32LSB` input callbacks at 192 kHz. The next
audio cut is feeding ASIO buffers into `Mimir.Runtime`, not more WASAPI
probing or Raven loopback offload.
- Raven also has a 192 kHz loopback-capable Scarlett for co-streamer/game timing
evidence. Do not move the heavy soundfield or sensor-fusion workload there.
## Immediate Re-entry Instruction

View file

@ -12,7 +12,7 @@ Use it to shape native implementation decisions:
- direct camera and audio drivers;
- pinned/native buffers;
- GPU feature extraction, flow, matching, and fusion;
- Aquarium D3D12 compute/render paths;
- Fensalir D3D12 compute/render paths;
- Faust/native DSP;
- OBS-facing program outputs.
@ -31,5 +31,5 @@ Audio timing research entry points:
Do not promote mirrored examples, old probes, or external library demos into the
Mimir hot path just because they exist here. If research points at a concept,
rebuild the live version in the Mimir/Aquarium/native stack with a clear owner
rebuild the live version in the Mimir/Fensalir/native stack with a clear owner
and invariant.

View file

@ -2,6 +2,6 @@
Spout is the Windows-local GPU texture sharing bridge for this rig. The OBS path is the Off World Live OBS Spout2 plugin, which creates an OBS source from a Spout shared texture and can also send the OBS canvas back to Spout. The Spout2 SDK is the application-side sender/receiver surface; its modern path centers on DirectX shared textures and has native DirectX/OpenGL integration options.
Decision: Mimir should not render final pixels itself. It should publish typed render-frame packets from the fusion cache. Aquarium Engine should consume those packets, render them into a GPU render target, and publish that texture as a named Spout sender. OBS then receives that sender through the OBS Spout2 plugin.
Decision: Mimir should not render final pixels itself. It should publish typed render-frame packets from the fusion cache. Fensalir should consume those packets, render them into a GPU render target, and publish that texture as a named Spout sender. OBS then receives that sender through the OBS Spout2 plugin.
Latency decision: the render packet carries both source timestamps and intended presentation time. The stream compositor can buffer visual frames until they align with the ambisonic audio field's presentation timeline. Latency is allowed; unbounded drift is not.

View file

@ -1,23 +1,23 @@
workspace:
name: Mimir
status: active
focus: direct-driver-aquarium-stream-app
focus: direct-driver-fensalir-stream-app
implementation_name: Mimir public surface / native runtime contracts
premise: >
Mimir is the local realtime field machine: cameras, microphones, speakers,
network feeds, and timing signals become one coherent OBS-facing volumetric
program surface. The live path is a C# Aquarium-hosted app with in-memory
program surface. The live path is a C# Fensalir-hosted app with in-memory
rolling buffers and native capture/runtime boundaries.
objective:
primary: >
Build one synchronized spatial stream machine from six cameras, six
microphones, two speakers, Leap timing, Aquarium GPU rendering, Faust DSP,
microphones, two speakers, Leap timing, Fensalir GPU rendering, Faust DSP,
and OBS output.
current: >
Implement direct camera/audio capture into Mimir.Runtime buffers, starting
with Leap stereo IR as the timing-camera candidate, then expose runtime
health/settings/output controls through Aquarium UI.
health/settings/output controls through Fensalir UI.
constraints:
- Sender is Windows.
@ -26,7 +26,7 @@ constraints:
- Audio sources must remain separately controllable in OBS.
- V1 bridge is LAN-only.
- Persistent state must clarify rehydrate, not become transcript tax.
- Live ingest belongs to Mimir.Runtime, native drivers, Aquarium, and
- Live ingest belongs to Mimir.Runtime, native drivers, Fensalir, and
Faust/native DSP.
- Edge JSON is schema/diagnostic only; live process data is typed or in
memory.
@ -45,14 +45,14 @@ invariants:
to line up feeds and crunch data before OBS-facing output is emitted.
- Typed views may index the rolling buffer, but no per-kind ring owns
retention.
- Aquarium owns production UI, GPU fusion, D3D12 interop, and Spout2 output.
- Fensalir owns production UI, GPU fusion, D3D12 interop, and Spout2 output.
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
stems.
- Local secrets remain untracked.
accepted_design:
- `Mimir.slnx` owns the app surface.
- `src/Mimir.App` launches Aquarium Engine as host/window/D3D12 library.
- `src/Mimir.App` launches Fensalir as host/window/D3D12 library.
- `src/Mimir.Runtime` owns `MimirSynchronizationHub`,
`MimirRollingStreamBuffer`, stream descriptors, `IMimirStreamSource`,
direct native ingest, process bridge ingest, and video capture driver seams.
@ -120,7 +120,7 @@ accepted_design:
seam for Leap, Media Foundation, DirectShow, libusb, LeapC, or shared
texture capture.
- `native/reservoir` owns the lower shared-edge rolling-buffer invariant for
Aquarium/Faust integration.
Fensalir/Faust integration.
- `config/perfect-machine.example.json` declares live owners, sensors,
reservoir views, and runtime policy.
- `.voidbot` state is part of repo memory and must be committed separately
@ -132,7 +132,7 @@ active_subgoals:
- Add native audio capture workers for mic, loopback, and network audio feeds.
- Feed verified Focusrite ASIO input callbacks into `Mimir.Runtime`; WASAPI
remains a diagnostic witness.
- Bind Aquarium UI to stream health, buffer depth, source timestamps,
- Bind Fensalir UI to stream health, buffer depth, source timestamps,
settings, and output management.
- Keep PowerShell/FFmpeg/SRT as bridge utilities until direct program output
replaces them.
@ -216,7 +216,7 @@ current_status:
next_action: >
Prove the constrained chirplet-transform decoder through real loopback and
microphone paths before adding the synchronization actuator.
Current app testing proves Aquarium WASAPI output wakes Scarlett speaker
Current app testing proves Fensalir WASAPI output wakes Scarlett speaker
loopback, live mic buffers stay populated, runtime sync analysis now runs as
a bounded rotating service without UI/telemetry re-entry, and the continuous
chirplet timeline can produce confident sync states in the app. The next
@ -232,5 +232,7 @@ current_status:
Starfire, and `native/probes/asio_audio_cadence` proves local Focusrite USB
ASIO access with 4 inputs / 2 outputs including Loopback 1/2, 192-frame
preferred buffers, 44.1-192 kHz support, and nonzero 4-channel Int32LSB
input callbacks at 192 kHz. Treat Kiyo Pro as a ~25 fps source until its
SuperSpeed negotiation is fixed.
input callbacks at 192 kHz. Raven also has a 192 kHz loopback-capable
Scarlett for co-streamer/game timing evidence, but Starfire owns the heavy
soundfield and sensor-fusion work. Treat Kiyo Pro as a ~25 fps source until
its SuperSpeed negotiation is fixed.