Map Fensalir rendering rebuild

This commit is contained in:
Meta Vens 2026-05-26 13:29:01 +01:00
commit d53decf276
8 changed files with 398 additions and 0 deletions

View file

@ -50,6 +50,9 @@ Start with:
source-level code map.
- [docs/perfect-machine-domain-index.md](docs/perfect-machine-domain-index.md)
for the indexed problem-domain map and implementation cuts.
- [docs/fensalir-rendering-rebuild-migration.md](docs/fensalir-rendering-rebuild-migration.md)
for the current teardown/rebuild migration plan between Mimir's physical
evidence window and Fensalir's field-evidence renderer.
Naming note: **Fensalir** is the engine/windowing/rendering/D3D12 layer. Older
research notes may still use the previous engine name.

View file

@ -0,0 +1,364 @@
# Fensalir Rendering Rebuild Migration
## Objective
Keep Mimir from becoming a second renderer while Fensalir rebuilds itself into
the field-evidence machine Mimir needs.
Mimir's job is to make the room measurable: cameras, microphones, speakers,
loopbacks, phones, Raven, and Starfire all feed one bounded temporal window.
Fensalir's job is to turn that measured evidence into a coherent visible field
and program surface. OBS should receive the final result, not be asked to
pretend loose feeds are a world.
## Current Mechanism
Mimir currently has the correct broad cut:
```text
drivers / ASIO / network feeds
-> Mimir.Runtime rolling buffers
-> synchronization, calibration, response models
-> Fensalir scene state and debug surfaces
-> Fensalir D3D12 presentation / Spout / program output
-> OBS
```
But the current debug-spectrum path still exposes the pressure leak:
```text
ASIO spectrum history
-> AquariumSplineFrame
-> direct Fensalir spline tube debug surface
```
That path is allowed as a deterministic diagnostic. It is not yet the full
Perfect Machine. The production shape is:
```text
rolling buffer slice + source identity + calibration + surface intent
-> Fensalir field claim
-> selected lowering under budget
-> evidence lanes and temporal guide
-> presentation
```
Mimir must not send "draw these lines" as a production order. It should say
"this rolling buffer slice is a spectrum field with stable axes, identities,
support, and material intent." Fensalir then chooses direct tube SDF, mesh,
field splats, or another lowering.
## Invariants
- Mimir owns raw retention. Fensalir may snapshot or import the five-second
window, but it does not silently become history owner.
- Mimir owns physical provenance: device ids, sample clocks, calibration ids,
path response, confidence, network arrival, and decoded timing evidence.
- Fensalir owns dense visual fusion, temporal field evidence, selected cuts,
residency, D3D12 resources, debug UI, and program video.
- Faust/native DSP owns hot audio movement, resampling, alignment, separation,
spatialization, and stems.
- OBS owns broadcast composition only.
- Render packets are cached lowerings. They are not scene truth.
- TAA may validate pixel history. It does not own source identity.
- Networked phones and Raven may publish local observations and decoded timing
state. They do not become independent clock authorities.
## Authority Map
Owner:
- `Mimir.Runtime` owns stream identity, rolling buffers, synchronization state,
calibration loading, and observation publication.
- Fensalir's future `FieldEvidenceMachine` owns field claims after Mimir
publishes observations and intent.
Inputs:
- local direct-driver video frames;
- local ASIO loopback and mic blocks;
- Raven/phone/network observations;
- calibration receipts, response surfaces, codebooks, path ids, sensor poses;
- user/app policy for what should enter the live field.
Outputs:
- typed observation surfaces for Fensalir;
- timing, response, and calibration constraints;
- audio actuator control for Faust/native DSP;
- health/telemetry state for UI and logs;
- final program output from Fensalir and stems from Faust/native DSP.
Derived state:
- spectrum lines, debug views, screenshots, OBS bridge endpoints, FFmpeg/SRT
helpers, and Eve dashboard feeds are views or diagnostics.
- The direct spline-spectrum dashboard is useful because it reveals buffers; it
is not the architectural owner of field rendering.
Forbidden writers:
- any Mimir-side D3D12 render graph;
- app-specific stable Gaussian/evidence cache;
- JSON/base64 hot path for live media;
- OBS raw source timing as synchronization authority;
- per-debug-view geometry deciding production truth.
Shared paths:
- live cameras, audio timing anchors, acoustic response surfaces, spectrum
diagnostics, calibration events, and future volumetric room claims all publish
typed observations/constraints into the same Mimir-to-Fensalir bridge.
Deletion line:
- If a Mimir feature wants visual output, first define the observation, claim,
or surface intent. A direct draw is allowed only when it is named as fallback
or diagnostic and cannot override field evidence.
## Target Mimir -> Fensalir Contract
### 1. Raw Window
Mimir keeps one retention authority:
```text
RollingStreamWindow {
windowId
duration = 5 seconds by default
streamId
sourceKind
sampleDescriptors
nativeHandle or payload view
deviceTimestamp
canonicalTimestampEstimate
sequenceId
status
}
```
Typed views may index this window. They do not own retention.
### 2. Observation Surface
Mimir publishes physical evidence, not render commands:
```text
MimirObservation {
observationKey
streamId
sensorId
calibrationId
modality: camera | audio | network | timing | response
observedTime
canonicalTimeEstimate
uncertainty
payloadHandle
provenance
confidence
}
```
### 3. Calibration Constraint
Calibration and sync become explicit constraints:
```text
MimirCalibrationConstraint {
constraintKey
pathId
sourceId
receiverId
evidenceKind: loopback | bioacoustic | passive | visual | network
delayEstimate
delayUncertainty
phaseOrGroupDelay
frequencyResponse
usableBandMask
confidence
}
```
### 4. Surface Intent
Debug and production visualization share intent:
```text
MimirSurfaceIntent {
intentKey
sourceObservationKeys
domain
axes
supportPolicy
materialGraph
updateBudget
purpose: debug | calibration | production
}
```
The spectrum dashboard becomes one instance of this: a rolling audio spectrum
field whose frequency axis maps to X, amplitude to Y, age to Z, and stream
identity to stacked lanes.
## Fresh Dataflow
```mermaid
flowchart TD
A["direct camera drivers"] --> B["Mimir rolling window"]
C["ASIO loopback + mics"] --> B
D["Raven / phones / network"] --> B
B --> E["sync + calibration kernel"]
E --> F["typed observations"]
E --> G["calibration constraints"]
B --> H["surface intent"]
F --> I["Fensalir field evidence machine"]
G --> I
H --> I
I --> J["field claims + selected lowerings"]
J --> K["program video + debug UI"]
E --> L["Faust/native DSP controls"]
L --> M["program stems + spatial bed"]
K --> N["OBS"]
M --> N
```
## Migration Plan
### Phase 0: Freeze The Current Debug Surface
Keep the direct `AquariumSplineFrame` dashboard because it exposes live audio
buffers and is useful for sanity checks. Label it as deterministic debug
surface, not production field evidence.
Delete or keep dead any point-splat spectrum path. It already proved the wrong
owner.
### Phase 1: Add Bridge DTOs In Mimir.Runtime
Create pure mapping types for:
- rolling stream windows;
- observations;
- calibration constraints;
- surface intent.
The bridge must not read devices, render, allocate per sample, or own timers.
It maps existing runtime state into Fensalir-facing contracts.
Acceptance:
- one synthetic buffer can produce observations and surface intent in a unit
test;
- the mapping can run repeatedly without growing allocations;
- stream/source ids remain stable across frames.
### Phase 2: Move Spectrum Dashboard To Surface Intent
Represent the current spectrum view as a `MimirSurfaceIntent`:
- stable source lanes;
- stable frequency bins;
- monotonic history sequence ids;
- Z as continuous sample age;
- tube/material intent from amplitude/confidence.
Fensalir may still lower it to direct spline tubes at first. The ownership
changes before the backend does.
Acceptance:
- app code no longer chooses point splats, direct tubes, or mesh ribbons;
- row/band/history identity is the same whether the engine lowers direct tubes
or another representation.
### Phase 3: Camera Observation Bridge
Lower native camera descriptors into observation surfaces:
- source id and calibration id;
- dimensions, format, stride;
- native CPU/GPU handle;
- device timestamp and canonical estimate;
- lens model and confidence/status.
Acceptance:
- at least one local camera stream appears as Fensalir sensor observations;
- missing frames are absence, not zero-valued evidence;
- Fensalir owns any feature extraction result.
### Phase 4: Audio Field Observation Bridge
Lower sync and calibration state into audio field constraints:
- ASIO loopback clock fit;
- mic delay and SRO state;
- per-path frequency response and group delay;
- bioacoustic anchor clusters;
- passive correlation confidence;
- Raven/phone decoded timing observations.
Acceptance:
- Fensalir can visualize acoustic confidence and constraints without receiving
raw PCM as production truth;
- Faust/native DSP remains the actuator for sample movement.
### Phase 5: Sensor Fusion Claims
Let Fensalir fuse Mimir observations into field claims:
- sparse feature tracks first;
- calibrated camera rays and marker/feature candidates;
- acoustic source candidates and confidence volumes;
- surface/material claims only after cross-modal evidence earns them.
Acceptance:
- one synthetic multi-camera marker creates a stable world-space claim;
- one synthetic audio source creates a localizable confidence field;
- claims can be inspected by source observation and calibration id.
### Phase 6: Program Output Receipts
Once field identity is coherent, wire final program output:
- D3D12 shared texture or Spout for OBS/EVE surfaces;
- separately controllable Faust/native DSP stems;
- telemetry proving timing, evidence confidence, dropped/late observations, and
current selected cuts.
Acceptance:
- OBS sees final program surfaces/stems only;
- raw feeds remain debug inputs, not synchronization truth;
- a recorded receipt can explain how each visible surface was derived.
## Verification Matrix
| Check | Expected Result |
| --- | --- |
| Stable buffer identity | Source lanes do not reorder or phase-rotate by render frame. |
| Missing data | No sample means no observation; no zero-fill masquerade. |
| Direct debug path | Can be disabled without changing runtime timing state. |
| Observation bridge | Produces typed evidence without device reads or rendering. |
| Fensalir lowering swap | Direct tube vs future splat/mesh lowering preserves claim ids. |
| TAA guide | Pixel history follows evidence validity, not producer folklore. |
| OBS output | OBS receives final program surfaces, not raw clock chaos. |
## Research And Maps
Primary local references:
- `docs/perfect-machine.md`
- `docs/perfect-machine-domain-index.md`
- `docs/sensor-fusion-architecture.md`
- `research/perfect-machine-study-2026-05-23/fensalir-integration-map.md`
- `research/visual-spatial-map/brushstroke-fusion-rendering.md`
- Fensalir `docs/rendering-teardown-rebuild-protocol.md`
- Fensalir `docs/perfect-machine-architecture.md`
- Fensalir `docs/temporal-spatial-evidence-reservoir.md`
The next implementation pass should cut Phase 1 first. A bridge DTO that owns
nothing but mapping is boring in the best possible way: it makes the rest of
the machine harder to lie about.

View file

@ -34,6 +34,13 @@ surface domains with 2D brushes painting tile payloads, and SDF probes emitting
surface splats only where curvature, projected error, material/brush detail, or
silhouettes justify subdivision.
The current teardown/migration map lives in
[[fensalir-rendering-rebuild-migration|Fensalir Rendering Rebuild Migration]].
Mimir's side of that rebuild is to publish typed physical observations,
calibration constraints, and surface intent; Fensalir's side is to turn those
into field claims, selected lowerings, reusable evidence, temporal guides, and
program output.
The old script stack is gone. Do not add a compatibility edge unless it protects
a named invariant that the native runtime cannot protect yet.

View file

@ -27,6 +27,9 @@ Read this with:
loop candidates.
- [[research/perfect-machine-study-2026-05-23/fensalir-integration-map|Fensalir Integration Map]]
for the Mimir/Fensalir ownership cut.
- [[fensalir-rendering-rebuild-migration|Fensalir Rendering Rebuild Migration]]
for the current teardown/rebuild plan that keeps Mimir on physical evidence
and pushes field rendering authority into Fensalir's evidence machine.
- [[research/perfect-machine-study-2026-05-23/calibration-session-spec|Calibration Session Spec]]
for the planned physical-path calibration command.
- [[perfect-machine-module-library|Perfect Machine Module Library]] for the

View file

@ -316,6 +316,14 @@ lowers those into field claims with travel/depth, metadata, control, and
reservoir-guide lanes. If a path only paints pixels, it is a fallback/debug
draw, not the Perfect Machine surface.
The current teardown/migration map is
`docs/fensalir-rendering-rebuild-migration.md`, paired with Fensalir's
`docs/rendering-teardown-rebuild-protocol.md`. The intended cut is explicit:
Mimir publishes typed physical observations, calibration constraints, and
surface intent; Fensalir owns field domains, field claims, candidate selection,
track/reservoir state, selected lowerings, temporal guide lanes, and
presentation.
The reservoir can resolve pixel-level claims without requiring pixel-sized
contents. Claim support belongs to the represented domain. A smooth flat surface
can be a few huge surface claims; a heightfield terrain belongs in a quadtree of

View file

@ -43,6 +43,12 @@ Get-Content .\state\evidence.jsonl -Tail 8
- `native/reservoir` owns the lower shared-edge typed-handle invariant for
Fensalir/Faust binding work.
- Fensalir owns production GPU fusion, UI, and Spout2 publication.
- The current Mimir/Fensalir rendering teardown map is
[[docs/fensalir-rendering-rebuild-migration|Fensalir Rendering Rebuild
Migration]]. Mimir's production visual output path must publish typed
physical observations, calibration constraints, and surface intent; Fensalir
turns those into field claims, selected lowerings, reusable evidence,
temporal guide lanes, and presentation.
- Fensalir also owns the EVE-facing dashboard pixels through
`Global\MimirFensalirProgramTexture`. `src/Mimir.EveRelay` opens that shared
texture, encodes H.264 Annex-B with NVENC by default, and serves EveCanvas
@ -56,6 +62,9 @@ Get-Content .\state\evidence.jsonl -Tail 8
## Current Pressure
- Implement the first concrete Leap direct capture driver and measure cadence.
- Cut the Phase 1 Mimir-to-Fensalir bridge DTOs for rolling windows,
observations, calibration constraints, and surface intent before adding
another backend-specific visual path.
- Add remaining camera drivers through `IMimirVideoCaptureDriver`.
- Add native audio capture workers for mic, loopback, and network audio feeds.
- Turn the current loopback-referenced chirplet measurement into the real

View file

@ -133,3 +133,4 @@
{"ts": "2026-05-26T11:29:14+01:00", "type": "correction", "status": "ok", "note": "Corrected spectrum spline surface ownership: Fensalir direct splines no longer alpha-blend segment overlap, D3D12SplinePS writes true camera travel into colorTravel.a instead of alpha, coverage/confidence now lives in control/reservoirGuide, and direct blue-noise jitter was removed from the fallback surface. Headless proof artifacts/runtime/mimir-spline-travel-owner-smooth.png shows smooth deterministic synthetic spectrum tubes after the fix. This is still the direct SDF debug path, not the future stochastic reservoir producer."}
{"ts": "2026-05-26T11:39:49+01:00", "type": "correction", "status": "ok", "note": "Added the Fensalir invariant explicitly: Fensalir is not a traditional renderer. Mimir must treat it as a field/evidence machine where contributors emit bounded claims with travel/depth, metadata, control, and reservoir-guide lanes. Pixel-only drawing is fallback/debug, not the Perfect Machine surface. This correction is now in notes/current-system-map.md and docs/implementation-plan.md."}
{"ts": "2026-05-26T11:46:55+01:00", "type": "decision", "status": "ok", "note": "Recorded the support-scale reservoir invariant: Fensalir may resolve at pixel level, but reservoir contents are not pixel-sized by default. Smooth surfaces can be represented by few broad claims; Aquarium-style heightfields should use brush-painted quadtree tiles and emit SDF surface splats adaptively by projected footprint, curvature, brush/material detail, silhouette risk, and temporal uncertainty."}
{"ts": "2026-05-26T13:26:28+01:00", "type": "decision", "status": "ok", "note": "Initiated the full Mimir/Fensalir rendering teardown protocol. Mimir now has docs/fensalir-rendering-rebuild-migration.md as the app-side migration map: Mimir owns rolling physical observations, calibration constraints, and surface intent; Fensalir owns field claims, selected lowerings, evidence reservoirs, temporal guide lanes, and program presentation. Direct spectrum splines remain diagnostic until they are expressed as surface intent."}

View file

@ -48,6 +48,9 @@ invariants:
retention.
- Fensalir owns production UI, GPU fusion, D3D12 interop, Spout2 output, and
the EVE shared D3D12 program-output texture.
- Mimir publishes physical observations, calibration constraints, and surface
intent. Fensalir owns field claims, selected lowerings, evidence reservoirs,
temporal guide lanes, and program presentation.
- EVE fullscreen output must use native decode/composite over Fensalir-owned
program pixels; WebKit/DOM layout is forbidden as pixel authority.
- Faust/native DSP owns hot audio alignment, separation, spatialization, and