Map Fensalir rendering rebuild
This commit is contained in:
parent
530b1af723
commit
d53decf276
8 changed files with 398 additions and 0 deletions
|
|
@ -50,6 +50,9 @@ Start with:
|
|||
source-level code map.
|
||||
- [docs/perfect-machine-domain-index.md](docs/perfect-machine-domain-index.md)
|
||||
for the indexed problem-domain map and implementation cuts.
|
||||
- [docs/fensalir-rendering-rebuild-migration.md](docs/fensalir-rendering-rebuild-migration.md)
|
||||
for the current teardown/rebuild migration plan between Mimir's physical
|
||||
evidence window and Fensalir's field-evidence renderer.
|
||||
|
||||
Naming note: **Fensalir** is the engine/windowing/rendering/D3D12 layer. Older
|
||||
research notes may still use the previous engine name.
|
||||
|
|
|
|||
364
docs/fensalir-rendering-rebuild-migration.md
Normal file
364
docs/fensalir-rendering-rebuild-migration.md
Normal file
|
|
@ -0,0 +1,364 @@
|
|||
# Fensalir Rendering Rebuild Migration
|
||||
|
||||
## Objective
|
||||
|
||||
Keep Mimir from becoming a second renderer while Fensalir rebuilds itself into
|
||||
the field-evidence machine Mimir needs.
|
||||
|
||||
Mimir's job is to make the room measurable: cameras, microphones, speakers,
|
||||
loopbacks, phones, Raven, and Starfire all feed one bounded temporal window.
|
||||
Fensalir's job is to turn that measured evidence into a coherent visible field
|
||||
and program surface. OBS should receive the final result, not be asked to
|
||||
pretend loose feeds are a world.
|
||||
|
||||
## Current Mechanism
|
||||
|
||||
Mimir currently has the correct broad cut:
|
||||
|
||||
```text
|
||||
drivers / ASIO / network feeds
|
||||
-> Mimir.Runtime rolling buffers
|
||||
-> synchronization, calibration, response models
|
||||
-> Fensalir scene state and debug surfaces
|
||||
-> Fensalir D3D12 presentation / Spout / program output
|
||||
-> OBS
|
||||
```
|
||||
|
||||
But the current debug-spectrum path still exposes the pressure leak:
|
||||
|
||||
```text
|
||||
ASIO spectrum history
|
||||
-> AquariumSplineFrame
|
||||
-> direct Fensalir spline tube debug surface
|
||||
```
|
||||
|
||||
That path is allowed as a deterministic diagnostic. It is not yet the full
|
||||
Perfect Machine. The production shape is:
|
||||
|
||||
```text
|
||||
rolling buffer slice + source identity + calibration + surface intent
|
||||
-> Fensalir field claim
|
||||
-> selected lowering under budget
|
||||
-> evidence lanes and temporal guide
|
||||
-> presentation
|
||||
```
|
||||
|
||||
Mimir must not send "draw these lines" as a production order. It should say
|
||||
"this rolling buffer slice is a spectrum field with stable axes, identities,
|
||||
support, and material intent." Fensalir then chooses direct tube SDF, mesh,
|
||||
field splats, or another lowering.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Mimir owns raw retention. Fensalir may snapshot or import the five-second
|
||||
window, but it does not silently become history owner.
|
||||
- Mimir owns physical provenance: device ids, sample clocks, calibration ids,
|
||||
path response, confidence, network arrival, and decoded timing evidence.
|
||||
- Fensalir owns dense visual fusion, temporal field evidence, selected cuts,
|
||||
residency, D3D12 resources, debug UI, and program video.
|
||||
- Faust/native DSP owns hot audio movement, resampling, alignment, separation,
|
||||
spatialization, and stems.
|
||||
- OBS owns broadcast composition only.
|
||||
- Render packets are cached lowerings. They are not scene truth.
|
||||
- TAA may validate pixel history. It does not own source identity.
|
||||
- Networked phones and Raven may publish local observations and decoded timing
|
||||
state. They do not become independent clock authorities.
|
||||
|
||||
## Authority Map
|
||||
|
||||
Owner:
|
||||
|
||||
- `Mimir.Runtime` owns stream identity, rolling buffers, synchronization state,
|
||||
calibration loading, and observation publication.
|
||||
- Fensalir's future `FieldEvidenceMachine` owns field claims after Mimir
|
||||
publishes observations and intent.
|
||||
|
||||
Inputs:
|
||||
|
||||
- local direct-driver video frames;
|
||||
- local ASIO loopback and mic blocks;
|
||||
- Raven/phone/network observations;
|
||||
- calibration receipts, response surfaces, codebooks, path ids, sensor poses;
|
||||
- user/app policy for what should enter the live field.
|
||||
|
||||
Outputs:
|
||||
|
||||
- typed observation surfaces for Fensalir;
|
||||
- timing, response, and calibration constraints;
|
||||
- audio actuator control for Faust/native DSP;
|
||||
- health/telemetry state for UI and logs;
|
||||
- final program output from Fensalir and stems from Faust/native DSP.
|
||||
|
||||
Derived state:
|
||||
|
||||
- spectrum lines, debug views, screenshots, OBS bridge endpoints, FFmpeg/SRT
|
||||
helpers, and Eve dashboard feeds are views or diagnostics.
|
||||
- The direct spline-spectrum dashboard is useful because it reveals buffers; it
|
||||
is not the architectural owner of field rendering.
|
||||
|
||||
Forbidden writers:
|
||||
|
||||
- any Mimir-side D3D12 render graph;
|
||||
- app-specific stable Gaussian/evidence cache;
|
||||
- JSON/base64 hot path for live media;
|
||||
- OBS raw source timing as synchronization authority;
|
||||
- per-debug-view geometry deciding production truth.
|
||||
|
||||
Shared paths:
|
||||
|
||||
- live cameras, audio timing anchors, acoustic response surfaces, spectrum
|
||||
diagnostics, calibration events, and future volumetric room claims all publish
|
||||
typed observations/constraints into the same Mimir-to-Fensalir bridge.
|
||||
|
||||
Deletion line:
|
||||
|
||||
- If a Mimir feature wants visual output, first define the observation, claim,
|
||||
or surface intent. A direct draw is allowed only when it is named as fallback
|
||||
or diagnostic and cannot override field evidence.
|
||||
|
||||
## Target Mimir -> Fensalir Contract
|
||||
|
||||
### 1. Raw Window
|
||||
|
||||
Mimir keeps one retention authority:
|
||||
|
||||
```text
|
||||
RollingStreamWindow {
|
||||
windowId
|
||||
duration = 5 seconds by default
|
||||
streamId
|
||||
sourceKind
|
||||
sampleDescriptors
|
||||
nativeHandle or payload view
|
||||
deviceTimestamp
|
||||
canonicalTimestampEstimate
|
||||
sequenceId
|
||||
status
|
||||
}
|
||||
```
|
||||
|
||||
Typed views may index this window. They do not own retention.
|
||||
|
||||
### 2. Observation Surface
|
||||
|
||||
Mimir publishes physical evidence, not render commands:
|
||||
|
||||
```text
|
||||
MimirObservation {
|
||||
observationKey
|
||||
streamId
|
||||
sensorId
|
||||
calibrationId
|
||||
modality: camera | audio | network | timing | response
|
||||
observedTime
|
||||
canonicalTimeEstimate
|
||||
uncertainty
|
||||
payloadHandle
|
||||
provenance
|
||||
confidence
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Calibration Constraint
|
||||
|
||||
Calibration and sync become explicit constraints:
|
||||
|
||||
```text
|
||||
MimirCalibrationConstraint {
|
||||
constraintKey
|
||||
pathId
|
||||
sourceId
|
||||
receiverId
|
||||
evidenceKind: loopback | bioacoustic | passive | visual | network
|
||||
delayEstimate
|
||||
delayUncertainty
|
||||
phaseOrGroupDelay
|
||||
frequencyResponse
|
||||
usableBandMask
|
||||
confidence
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Surface Intent
|
||||
|
||||
Debug and production visualization share intent:
|
||||
|
||||
```text
|
||||
MimirSurfaceIntent {
|
||||
intentKey
|
||||
sourceObservationKeys
|
||||
domain
|
||||
axes
|
||||
supportPolicy
|
||||
materialGraph
|
||||
updateBudget
|
||||
purpose: debug | calibration | production
|
||||
}
|
||||
```
|
||||
|
||||
The spectrum dashboard becomes one instance of this: a rolling audio spectrum
|
||||
field whose frequency axis maps to X, amplitude to Y, age to Z, and stream
|
||||
identity to stacked lanes.
|
||||
|
||||
## Fresh Dataflow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["direct camera drivers"] --> B["Mimir rolling window"]
|
||||
C["ASIO loopback + mics"] --> B
|
||||
D["Raven / phones / network"] --> B
|
||||
B --> E["sync + calibration kernel"]
|
||||
E --> F["typed observations"]
|
||||
E --> G["calibration constraints"]
|
||||
B --> H["surface intent"]
|
||||
F --> I["Fensalir field evidence machine"]
|
||||
G --> I
|
||||
H --> I
|
||||
I --> J["field claims + selected lowerings"]
|
||||
J --> K["program video + debug UI"]
|
||||
E --> L["Faust/native DSP controls"]
|
||||
L --> M["program stems + spatial bed"]
|
||||
K --> N["OBS"]
|
||||
M --> N
|
||||
```
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 0: Freeze The Current Debug Surface
|
||||
|
||||
Keep the direct `AquariumSplineFrame` dashboard because it exposes live audio
|
||||
buffers and is useful for sanity checks. Label it as deterministic debug
|
||||
surface, not production field evidence.
|
||||
|
||||
Delete or keep dead any point-splat spectrum path. It already proved the wrong
|
||||
owner.
|
||||
|
||||
### Phase 1: Add Bridge DTOs In Mimir.Runtime
|
||||
|
||||
Create pure mapping types for:
|
||||
|
||||
- rolling stream windows;
|
||||
- observations;
|
||||
- calibration constraints;
|
||||
- surface intent.
|
||||
|
||||
The bridge must not read devices, render, allocate per sample, or own timers.
|
||||
It maps existing runtime state into Fensalir-facing contracts.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- one synthetic buffer can produce observations and surface intent in a unit
|
||||
test;
|
||||
- the mapping can run repeatedly without growing allocations;
|
||||
- stream/source ids remain stable across frames.
|
||||
|
||||
### Phase 2: Move Spectrum Dashboard To Surface Intent
|
||||
|
||||
Represent the current spectrum view as a `MimirSurfaceIntent`:
|
||||
|
||||
- stable source lanes;
|
||||
- stable frequency bins;
|
||||
- monotonic history sequence ids;
|
||||
- Z as continuous sample age;
|
||||
- tube/material intent from amplitude/confidence.
|
||||
|
||||
Fensalir may still lower it to direct spline tubes at first. The ownership
|
||||
changes before the backend does.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- app code no longer chooses point splats, direct tubes, or mesh ribbons;
|
||||
- row/band/history identity is the same whether the engine lowers direct tubes
|
||||
or another representation.
|
||||
|
||||
### Phase 3: Camera Observation Bridge
|
||||
|
||||
Lower native camera descriptors into observation surfaces:
|
||||
|
||||
- source id and calibration id;
|
||||
- dimensions, format, stride;
|
||||
- native CPU/GPU handle;
|
||||
- device timestamp and canonical estimate;
|
||||
- lens model and confidence/status.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- at least one local camera stream appears as Fensalir sensor observations;
|
||||
- missing frames are absence, not zero-valued evidence;
|
||||
- Fensalir owns any feature extraction result.
|
||||
|
||||
### Phase 4: Audio Field Observation Bridge
|
||||
|
||||
Lower sync and calibration state into audio field constraints:
|
||||
|
||||
- ASIO loopback clock fit;
|
||||
- mic delay and SRO state;
|
||||
- per-path frequency response and group delay;
|
||||
- bioacoustic anchor clusters;
|
||||
- passive correlation confidence;
|
||||
- Raven/phone decoded timing observations.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Fensalir can visualize acoustic confidence and constraints without receiving
|
||||
raw PCM as production truth;
|
||||
- Faust/native DSP remains the actuator for sample movement.
|
||||
|
||||
### Phase 5: Sensor Fusion Claims
|
||||
|
||||
Let Fensalir fuse Mimir observations into field claims:
|
||||
|
||||
- sparse feature tracks first;
|
||||
- calibrated camera rays and marker/feature candidates;
|
||||
- acoustic source candidates and confidence volumes;
|
||||
- surface/material claims only after cross-modal evidence earns them.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- one synthetic multi-camera marker creates a stable world-space claim;
|
||||
- one synthetic audio source creates a localizable confidence field;
|
||||
- claims can be inspected by source observation and calibration id.
|
||||
|
||||
### Phase 6: Program Output Receipts
|
||||
|
||||
Once field identity is coherent, wire final program output:
|
||||
|
||||
- D3D12 shared texture or Spout for OBS/EVE surfaces;
|
||||
- separately controllable Faust/native DSP stems;
|
||||
- telemetry proving timing, evidence confidence, dropped/late observations, and
|
||||
current selected cuts.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- OBS sees final program surfaces/stems only;
|
||||
- raw feeds remain debug inputs, not synchronization truth;
|
||||
- a recorded receipt can explain how each visible surface was derived.
|
||||
|
||||
## Verification Matrix
|
||||
|
||||
| Check | Expected Result |
|
||||
| --- | --- |
|
||||
| Stable buffer identity | Source lanes do not reorder or phase-rotate by render frame. |
|
||||
| Missing data | No sample means no observation; no zero-fill masquerade. |
|
||||
| Direct debug path | Can be disabled without changing runtime timing state. |
|
||||
| Observation bridge | Produces typed evidence without device reads or rendering. |
|
||||
| Fensalir lowering swap | Direct tube vs future splat/mesh lowering preserves claim ids. |
|
||||
| TAA guide | Pixel history follows evidence validity, not producer folklore. |
|
||||
| OBS output | OBS receives final program surfaces, not raw clock chaos. |
|
||||
|
||||
## Research And Maps
|
||||
|
||||
Primary local references:
|
||||
|
||||
- `docs/perfect-machine.md`
|
||||
- `docs/perfect-machine-domain-index.md`
|
||||
- `docs/sensor-fusion-architecture.md`
|
||||
- `research/perfect-machine-study-2026-05-23/fensalir-integration-map.md`
|
||||
- `research/visual-spatial-map/brushstroke-fusion-rendering.md`
|
||||
- Fensalir `docs/rendering-teardown-rebuild-protocol.md`
|
||||
- Fensalir `docs/perfect-machine-architecture.md`
|
||||
- Fensalir `docs/temporal-spatial-evidence-reservoir.md`
|
||||
|
||||
The next implementation pass should cut Phase 1 first. A bridge DTO that owns
|
||||
nothing but mapping is boring in the best possible way: it makes the rest of
|
||||
the machine harder to lie about.
|
||||
|
|
@ -34,6 +34,13 @@ surface domains with 2D brushes painting tile payloads, and SDF probes emitting
|
|||
surface splats only where curvature, projected error, material/brush detail, or
|
||||
silhouettes justify subdivision.
|
||||
|
||||
The current teardown/migration map lives in
|
||||
[[fensalir-rendering-rebuild-migration|Fensalir Rendering Rebuild Migration]].
|
||||
Mimir's side of that rebuild is to publish typed physical observations,
|
||||
calibration constraints, and surface intent; Fensalir's side is to turn those
|
||||
into field claims, selected lowerings, reusable evidence, temporal guides, and
|
||||
program output.
|
||||
|
||||
The old script stack is gone. Do not add a compatibility edge unless it protects
|
||||
a named invariant that the native runtime cannot protect yet.
|
||||
|
||||
|
|
|
|||
|
|
@ -27,6 +27,9 @@ Read this with:
|
|||
loop candidates.
|
||||
- [[research/perfect-machine-study-2026-05-23/fensalir-integration-map|Fensalir Integration Map]]
|
||||
for the Mimir/Fensalir ownership cut.
|
||||
- [[fensalir-rendering-rebuild-migration|Fensalir Rendering Rebuild Migration]]
|
||||
for the current teardown/rebuild plan that keeps Mimir on physical evidence
|
||||
and pushes field rendering authority into Fensalir's evidence machine.
|
||||
- [[research/perfect-machine-study-2026-05-23/calibration-session-spec|Calibration Session Spec]]
|
||||
for the planned physical-path calibration command.
|
||||
- [[perfect-machine-module-library|Perfect Machine Module Library]] for the
|
||||
|
|
|
|||
|
|
@ -316,6 +316,14 @@ lowers those into field claims with travel/depth, metadata, control, and
|
|||
reservoir-guide lanes. If a path only paints pixels, it is a fallback/debug
|
||||
draw, not the Perfect Machine surface.
|
||||
|
||||
The current teardown/migration map is
|
||||
`docs/fensalir-rendering-rebuild-migration.md`, paired with Fensalir's
|
||||
`docs/rendering-teardown-rebuild-protocol.md`. The intended cut is explicit:
|
||||
Mimir publishes typed physical observations, calibration constraints, and
|
||||
surface intent; Fensalir owns field domains, field claims, candidate selection,
|
||||
track/reservoir state, selected lowerings, temporal guide lanes, and
|
||||
presentation.
|
||||
|
||||
The reservoir can resolve pixel-level claims without requiring pixel-sized
|
||||
contents. Claim support belongs to the represented domain. A smooth flat surface
|
||||
can be a few huge surface claims; a heightfield terrain belongs in a quadtree of
|
||||
|
|
|
|||
|
|
@ -43,6 +43,12 @@ Get-Content .\state\evidence.jsonl -Tail 8
|
|||
- `native/reservoir` owns the lower shared-edge typed-handle invariant for
|
||||
Fensalir/Faust binding work.
|
||||
- Fensalir owns production GPU fusion, UI, and Spout2 publication.
|
||||
- The current Mimir/Fensalir rendering teardown map is
|
||||
[[docs/fensalir-rendering-rebuild-migration|Fensalir Rendering Rebuild
|
||||
Migration]]. Mimir's production visual output path must publish typed
|
||||
physical observations, calibration constraints, and surface intent; Fensalir
|
||||
turns those into field claims, selected lowerings, reusable evidence,
|
||||
temporal guide lanes, and presentation.
|
||||
- Fensalir also owns the EVE-facing dashboard pixels through
|
||||
`Global\MimirFensalirProgramTexture`. `src/Mimir.EveRelay` opens that shared
|
||||
texture, encodes H.264 Annex-B with NVENC by default, and serves EveCanvas
|
||||
|
|
@ -56,6 +62,9 @@ Get-Content .\state\evidence.jsonl -Tail 8
|
|||
## Current Pressure
|
||||
|
||||
- Implement the first concrete Leap direct capture driver and measure cadence.
|
||||
- Cut the Phase 1 Mimir-to-Fensalir bridge DTOs for rolling windows,
|
||||
observations, calibration constraints, and surface intent before adding
|
||||
another backend-specific visual path.
|
||||
- Add remaining camera drivers through `IMimirVideoCaptureDriver`.
|
||||
- Add native audio capture workers for mic, loopback, and network audio feeds.
|
||||
- Turn the current loopback-referenced chirplet measurement into the real
|
||||
|
|
|
|||
|
|
@ -133,3 +133,4 @@
|
|||
{"ts": "2026-05-26T11:29:14+01:00", "type": "correction", "status": "ok", "note": "Corrected spectrum spline surface ownership: Fensalir direct splines no longer alpha-blend segment overlap, D3D12SplinePS writes true camera travel into colorTravel.a instead of alpha, coverage/confidence now lives in control/reservoirGuide, and direct blue-noise jitter was removed from the fallback surface. Headless proof artifacts/runtime/mimir-spline-travel-owner-smooth.png shows smooth deterministic synthetic spectrum tubes after the fix. This is still the direct SDF debug path, not the future stochastic reservoir producer."}
|
||||
{"ts": "2026-05-26T11:39:49+01:00", "type": "correction", "status": "ok", "note": "Added the Fensalir invariant explicitly: Fensalir is not a traditional renderer. Mimir must treat it as a field/evidence machine where contributors emit bounded claims with travel/depth, metadata, control, and reservoir-guide lanes. Pixel-only drawing is fallback/debug, not the Perfect Machine surface. This correction is now in notes/current-system-map.md and docs/implementation-plan.md."}
|
||||
{"ts": "2026-05-26T11:46:55+01:00", "type": "decision", "status": "ok", "note": "Recorded the support-scale reservoir invariant: Fensalir may resolve at pixel level, but reservoir contents are not pixel-sized by default. Smooth surfaces can be represented by few broad claims; Aquarium-style heightfields should use brush-painted quadtree tiles and emit SDF surface splats adaptively by projected footprint, curvature, brush/material detail, silhouette risk, and temporal uncertainty."}
|
||||
{"ts": "2026-05-26T13:26:28+01:00", "type": "decision", "status": "ok", "note": "Initiated the full Mimir/Fensalir rendering teardown protocol. Mimir now has docs/fensalir-rendering-rebuild-migration.md as the app-side migration map: Mimir owns rolling physical observations, calibration constraints, and surface intent; Fensalir owns field claims, selected lowerings, evidence reservoirs, temporal guide lanes, and program presentation. Direct spectrum splines remain diagnostic until they are expressed as surface intent."}
|
||||
|
|
|
|||
|
|
@ -48,6 +48,9 @@ invariants:
|
|||
retention.
|
||||
- Fensalir owns production UI, GPU fusion, D3D12 interop, Spout2 output, and
|
||||
the EVE shared D3D12 program-output texture.
|
||||
- Mimir publishes physical observations, calibration constraints, and surface
|
||||
intent. Fensalir owns field claims, selected lowerings, evidence reservoirs,
|
||||
temporal guide lanes, and program presentation.
|
||||
- EVE fullscreen output must use native decode/composite over Fensalir-owned
|
||||
program pixels; WebKit/DOM layout is forbidden as pixel authority.
|
||||
- Faust/native DSP owns hot audio alignment, separation, spatialization, and
|
||||
|
|
|
|||
Loading…
Reference in a new issue