diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 77320c8..8e1374b 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -159,6 +159,17 @@ a named invariant that the native runtime cannot protect yet. and schedule state. `--bioacoustic-actuator-self-test` estimates a synthetic 317.375-sample bioacoustic delay, applies the fractional correction, and remeasures the residual below printed microsecond precision. +- `src/Mimir.Runtime/Synchronization` now contains the reusable Perfect Machine + module library distilled from the research pass: node profiles, decoder + profiles, language/emission profiles, path-learning sessions, benchmark + panels, audio actuator strategies, native capture profiles, camera ingest + strategies, reservoir strategies, audio/visual field profiles, compute + offload profiles, OBS publication profiles, CultMesh contracts, Fensalir + lowering, and assembly plans. Use + `Mimir.BufferSmoke --perfect-machine-profile-smoke` to prove the catalog + assembles, `--perfect-machine-contract-smoke` to write a CultCache contract + proof, and `--perfect-machine-manifest` to export the module manifest for + tooling/UI/remote witness use. - `MimirVideoFrameDescriptor` for dimensions, pixel format, stride, device timestamp, and native/GPU handle metadata. - `IMimirVideoCaptureDriver` and `MimirVideoCaptureDriverSource` as the live diff --git a/docs/perfect-machine-domain-index.md b/docs/perfect-machine-domain-index.md index ba0bb97..11c86ce 100644 --- a/docs/perfect-machine-domain-index.md +++ b/docs/perfect-machine-domain-index.md @@ -29,6 +29,8 @@ Read this with: for the Mimir/Fensalir ownership cut. - [[research/perfect-machine-study-2026-05-23/calibration-session-spec|Calibration Session Spec]] for the planned physical-path calibration command. +- [[perfect-machine-module-library|Perfect Machine Module Library]] for the + reusable runtime configurations promoted from the research notes. - [[research/perfect-machine-study-2026-05-23/samples/README|Sample Implementation Sketches]] for sample implementation sketches. diff --git a/docs/perfect-machine-module-library.md b/docs/perfect-machine-module-library.md new file mode 100644 index 0000000..c16ce0c --- /dev/null +++ b/docs/perfect-machine-module-library.md @@ -0,0 +1,69 @@ +# Perfect Machine Module Library + +This is the runtime shelf for coherent parts that deserve to be inside Mimir, +not a trophy wall for abandoned attempts. Each module below has one owner and a +job-interview answer for why it belongs in the body of the machine. + +## Runtime Modules + +| Module | Owner | Purpose | Code | +| --- | --- | --- | --- | +| Bioacoustic decoder profiles | `Mimir.Runtime` | Named indexed log-mel/MFCC receiver configurations for clean, compact, robust, and high-band room paths. | `src/Mimir.Runtime/Synchronization/MimirBioacousticDecoderConfiguration.cs` | +| Bioacoustic language profiles | `Mimir.Runtime` | Passive, hybrid, continuous, calibration, and phone-witness vocabulary/emission configurations. | `src/Mimir.Runtime/Synchronization/MimirBioacousticLanguageConfiguration.cs` | +| Bioacoustic clock solver | `Mimir.Runtime` | Fits canonical source time from decoded word anchors for local, Raven, phone, and microcontroller witnesses. | `src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs` | +| Alignment actuator | `Mimir.Runtime` + Faust | Converts sync state into bounded fractional-delay, gain, and sample-rate-offset controls; Faust moves samples. | `src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs` | +| Audio actuator strategies | `Mimir.Runtime` + Faust | Integer baseline, Farrow fractional delay, variable ASRC, and hybrid delay/ASRC strategy profiles. | `src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs` | +| CultMesh contracts | `Mimir.Runtime` | Typed codebook, decoder, acoustic path, and actuator state documents for distributed Mimir nodes. | `src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs` | +| Acoustic path learning | `Mimir.Runtime` | Calibration stages for usable bands, confusion, global delay, group delay, and codebook adaptation. | `src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs` | +| Acoustic localization | `Mimir.Runtime` + Fensalir | Pairwise TDOA, SRP-PHAT grid, sparse source, and visual-constrained localization profiles plus a small SRP/TDOA scorer. | `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs`, `src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs` | +| Benchmark panels | `Mimir.Runtime` | Decoder golf and meatspace acceptance degradation panels with receipt roots and thresholds. | `src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs` | +| Capture profiles | `Mimir.Runtime` + native workers | Hardware intent for Leap, Kiyos, PS Eyes, Starfire Scarlett, and Raven Scarlett. | `src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs` | +| Camera ingest strategies | `Mimir.Runtime` + native workers | Diagnostic metadata, managed native wrapper, native SPSC ring, and shared-GPU-texture ingest shapes. | `src/Mimir.Runtime/Synchronization/MimirCameraIngestConfiguration.cs` | +| Reservoir strategies | `Mimir.Runtime` + native reservoir + Fensalir | Managed rolling buffers, native shared-edge rings, and derived GPU temporal field storage. | `src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs` | +| Distributed witnesses | `Mimir.Runtime` + CultMesh | Raven, phone, microcontroller, and remote camera rig configurations for compact timing/path state. | `src/Mimir.Runtime/Synchronization/MimirDistributedWitnessConfiguration.cs` | +| Audio field configurations | `Mimir.Runtime` | Six-mic, source-based, ambisonic, and hybrid evidence field shapes. | `src/Mimir.Runtime/Synchronization/MimirAudioFieldConfiguration.cs` | +| Visual fusion configurations | `Mimir.Runtime` + Fensalir | Sensor roles and fusion modes from cadence proof through full hybrid evidence. | `src/Mimir.Runtime/Synchronization/MimirVisualFusionConfiguration.cs` | +| Compute offload configurations | `Mimir.Runtime` + CultMesh | Local-heavy, distributed witness, and calibration sweep workload placement. | `src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs` | +| Assembly plans | `Mimir.Runtime` | Named combinations of modules that prove synthetic, Scarlett, meatspace, and distributed cuts. | `src/Mimir.Runtime/Synchronization/MimirMachineAssemblyPlan.cs` | +| OBS publication profiles | Fensalir + Faust + OBS | Spout/shared-texture video and separately mixable Faust stem publication. | `src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs` | +| Fensalir lowering | `Mimir.Runtime` -> Fensalir | Pure mapping from Mimir rolling/video/audio state into Fensalir GPU sensor and acoustic field frames. | `src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs` | +| Module catalog | `Mimir.Runtime` | Indexes the live module set and verification commands. | `src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs` | +| Manifest factory | `Mimir.Runtime` | Produces an exportable JSON manifest for tools, docs, UI, and remote witnesses. | `src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs` | + +## Current Authority Map + +- `Mimir.Runtime` owns device and synchronization truth before it becomes a + renderer/DSP packet. +- Fensalir owns GPU resource lifetime, temporal field rendering, debug UI, and + program video publication. +- Faust owns hot sample movement, alignment, spatialization, separation, and + stem generation. +- CultMesh carries typed state surfaces; remote nodes may observe and decode, + but they do not become independent clock authorities. +- OBS receives final program surfaces and stems. It does not own calibration. + +## Verification Surface + +Use these as quick probes after touching the module shelf: + +```powershell +dotnet build .\Mimir.slnx +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --perfect-machine-manifest +dotnet run --no-build --project .\src\Mimir.BufferSmoke\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test +``` + +## Research Links + +- [[research/perfect-machine-study-2026-05-23/index|Perfect Machine Study Index]] +- [[research/perfect-machine-study-2026-05-23/option-matrix|Option Matrix]] +- [[research/perfect-machine-study-2026-05-23/state-machine-invariants|State Machine Invariants]] +- [[research/perfect-machine-study-2026-05-23/fensalir-integration-map|Fensalir Integration Map]] +- [[research/perfect-machine-study-2026-05-23/optimization-ledger|Optimization Ledger]] + +## Cut Line + +If a future implementation supersedes one of these module shapes, delete or +demote the old module in the same pass. The library is meant to keep usable +organs ready at hand, not preserve every fossil that once twitched. diff --git a/src/Mimir.BufferSmoke/Program.cs b/src/Mimir.BufferSmoke/Program.cs index 07102c5..8d5019b 100644 --- a/src/Mimir.BufferSmoke/Program.cs +++ b/src/Mimir.BufferSmoke/Program.cs @@ -51,6 +51,25 @@ if (args.Any(arg => string.Equals(arg, "--bioacoustic-actuator-self-test", Strin ParseDoubleOption(args, "--delay-samples", 317.375)); } +if (args.Any(arg => string.Equals(arg, "--perfect-machine-profile-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return RunPerfectMachineProfileSmoke(); +} + +if (args.Any(arg => string.Equals(arg, "--perfect-machine-contract-smoke", StringComparison.OrdinalIgnoreCase))) +{ + return await RunPerfectMachineContractSmokeAsync( + ParseStringOption(args, "--output", "artifacts/perfect-machine/contracts.cc")) + .ConfigureAwait(false); +} + +if (args.Any(arg => string.Equals(arg, "--perfect-machine-manifest", StringComparison.OrdinalIgnoreCase))) +{ + return await WritePerfectMachineManifestAsync( + ParseStringOption(args, "--output", "artifacts/perfect-machine/manifest.json")) + .ConfigureAwait(false); +} + if (args.Any(arg => string.Equals(arg, "--standalone-chirp-bin-self-test", StringComparison.OrdinalIgnoreCase))) { return RunStandaloneChirpBinSelfTest( @@ -456,13 +475,7 @@ static int RunBioacousticCepstralSmoke(int sampleRate, double seconds) } var source = samples.ToArray(); - var settings = new[] - { - new CepstralDegradationSetting("clean-roundtrip", 0.0, 0.0, 0), - new CepstralDegradationSetting("blur-light", 0.0, 0.0, 1), - new CepstralDegradationSetting("warp-light", 0.75, 1.25, 0), - new CepstralDegradationSetting("warp-light-blur", 0.75, 1.25, 1), - }; + var settings = CepstralSmokeDegradationSettings(); var failures = 0; foreach (var setting in settings) @@ -495,7 +508,7 @@ static int RunBioacousticCepstralSmoke(int sampleRate, double seconds) if (failures > 0) { - Console.Error.WriteLine($"bioacoustic cepstral smoke failed: {failures}/{settings.Length} degradation settings lost too much word identity"); + Console.Error.WriteLine($"bioacoustic cepstral smoke failed: {failures}/{settings.Count} degradation settings lost too much word identity"); return 1; } @@ -524,14 +537,7 @@ static async Task RunBioacousticTrainingAsync(int sampleRate, double second } var source = samples.ToArray(); - var degradations = new[] - { - new CepstralDegradationSetting("clean-roundtrip", 0.0, 0.0, 0), - new CepstralDegradationSetting("blur-light", 0.0, 0.0, 1), - new CepstralDegradationSetting("warp-light", 0.75, 1.25, 0), - new CepstralDegradationSetting("warp-light-blur", 0.75, 1.25, 1), - new CepstralDegradationSetting("warp-heavy-blur", 1.25, 1.75, 1), - }; + var degradations = CepstralTrainingDegradationSettings(); var hypotheses = BioacousticTrainingHypotheses(); var expectedEvents = timeline.EventsOverlapping(0.0, source.Length / (double)sampleRate) .Select(timelineEvent => timelineEvent.Index) @@ -568,7 +574,7 @@ static async Task RunBioacousticTrainingAsync(int sampleRate, double second var timingMae = MeanCepstralTimingError(observations, timeline, sampleRate); var clock = FitBioacousticClockHypothesis(observations, timeline, sampleRate); var globalTimingMae = clock?.MeanAbsoluteErrorSamples ?? double.PositiveInfinity; - var anchorCoverage = expectedEvents.Count == 0 || clock == null ? 0.0 : clock.AnchorCount / (double)expectedEvents.Count; + var anchorCoverage = clock?.AnchorCoverage ?? 0.0; var realtime = decodeMilliseconds <= 0.0 ? double.PositiveInfinity : seconds * 1000.0 / decodeMilliseconds; var identityScore = precision * 0.45 + recall * 0.45 + confidence * 0.10; var timingScore = clock?.Confidence ?? 0.0; @@ -845,6 +851,151 @@ static int RunBioacousticActuatorSelfTest(int sampleRate, double delaySamples) : 1; } +static int RunPerfectMachineProfileSmoke() +{ + var profiles = MimirPerfectMachineProfiles.All; + var calibrationPlans = MimirCalibrationSessionPlans.BuiltIn; + var audioFields = MimirAudioFieldConfigurations.BuiltIn; + var visualFields = MimirVisualFusionConfigurations.BuiltIn; + var computePlans = MimirComputeOffloadConfigurations.BuiltIn; + var assemblyPlans = MimirMachineAssemblyPlans.BuiltIn; + var captureProfiles = MimirNativeCaptureConfigurations.BuiltIn; + var publications = MimirObsPublicationConfigurations.BuiltIn; + var moduleCatalog = MimirModuleLibrary.Entries; + var languageProfiles = MimirBioacousticLanguageConfigurations.BuiltIn; + var pathLearningProfiles = MimirAcousticPathLearningConfigurations.BuiltIn; + var benchmarkPanels = MimirBenchmarkPanelConfigurations.BuiltIn; + var actuatorStrategies = MimirAudioActuatorConfigurations.BuiltIn; + var cameraIngestStrategies = MimirCameraIngestConfigurations.BuiltIn; + var reservoirStrategies = MimirReservoirConfigurations.BuiltIn; + var localizationProfiles = MimirAcousticLocalizationConfigurations.BuiltIn; + var distributedWitnesses = MimirDistributedWitnessConfigurations.BuiltIn; + var networkTransports = MimirNetworkTransportConfigurations.BuiltIn; + var authorityPolicies = MimirAuthorityPolicyConfigurations.BuiltIn; + var codebook = MimirCultMeshContractFactory.CreateCodebookState( + "mimir-bioacoustic-default", + "deterministic-segment-schedule-v1", + MimirBioacousticTimeline.Default); + var decoder = MimirCultMeshContractFactory.CreateDecoderState( + "baseline-mfcc-index", + codebook.CodebookId, + MimirBioacousticDecoderConfiguration.BaselineMfccIndex); + var controller = new MimirSroPllActuatorController(); + var command = controller.Update( + "scarlett-host-mic", + 0, + delaySamples: 317.375, + confidence: 0.82, + dtSeconds: 0.5); + var actuator = MimirCultMeshContractFactory.CreateActuatorState( + "scarlett-host-mic-actuator", + MimirAlignmentActuatorProfile.SixSourceFaust.Id, + command); + var observations = new[] + { + new MimirBioacousticWordObservation(0, MimirBioacousticTimeline.Default.EventForIndex(0).StartSeconds * MimirBioacousticTimeline.SampleRate + 317.375, 0.95), + new MimirBioacousticWordObservation(1, MimirBioacousticTimeline.Default.EventForIndex(1).StartSeconds * MimirBioacousticTimeline.SampleRate + 317.375, 0.95), + new MimirBioacousticWordObservation(2, MimirBioacousticTimeline.Default.EventForIndex(2).StartSeconds * MimirBioacousticTimeline.SampleRate + 317.375, 0.95), + new MimirBioacousticWordObservation(3, MimirBioacousticTimeline.Default.EventForIndex(3).StartSeconds * MimirBioacousticTimeline.SampleRate + 317.375, 0.95), + }; + var clock = new MimirBioacousticClockSolver().Fit( + observations, + MimirBioacousticTimeline.Default, + MimirBioacousticTimeline.SampleRate, + expectedEventCount: 4); + var videoBuffer = new MimirRollingStreamBuffer( + new MimirStreamDescriptor("kiyo-pro-rgb", MimirStreamKind.Video, MimirStreamOrigin.LocalDevice), + TimeSpan.FromSeconds(5)); + videoBuffer.Append(new MimirStreamSample( + "kiyo-pro-rgb", + MimirStreamKind.Video, + MimirStreamOrigin.LocalDevice, + TimestampNs: 1_000_000_000L, + ArrivalNs: 1_000_000_010L, + Sequence: 1, + PayloadHandle: 0, + VideoFrame: new MimirVideoFrameDescriptor( + 1920, + 1080, + MimirVideoPixelFormat.Bgra8, + 1920 * 4, + 1_000_000_000L, + NativeHandle: 42, + NativeHandleKind: "shared-d3d12-texture"))); + var lowerer = new MimirFensalirFieldLowering(); + var gpuFrame = lowerer.BuildGpuSensorFrame([videoBuffer]); + var acousticFrame = lowerer.BuildAcousticFieldFrame([ + new MimirAudioSynchronizationState( + "loopback-scarlett-speakers", + "scarlett-host-mic", + 192_000, + 61.25, + 61.25, + 0.319, + 0.875, + 0.94, + [], + 1_000_000_000L, + 10, + 10) + ]); + var localizationGrid = MimirSrpPhatGridSolver.BuildGrid( + new Vector3(-0.5f, 0.0f, 1.0f), + new Vector3(0.5f, 0.0f, 1.0f), + spacingMeters: 0.5); + var localization = new MimirSrpPhatGridSolver().FindBestCandidate( + [ + new MimirMicrophonePose("mic-left", new Vector3(-0.5f, 0.0f, 0.0f)), + new MimirMicrophonePose("mic-right", new Vector3(0.5f, 0.0f, 0.0f)) + ], + localizationGrid, + [ + new MimirPairDelayObservation("mic-left", "mic-right", 0.0, 0.95) + ]); + + Console.WriteLine( + $"perfect-machine-profile-smoke profiles={profiles.Count} calibrationPlans={calibrationPlans.Count} audioFields={audioFields.Count} visualFields={visualFields.Count} computePlans={computePlans.Count} assemblyPlans={assemblyPlans.Count} captureProfiles={captureProfiles.Count} publications={publications.Count} languageProfiles={languageProfiles.Count} pathLearning={pathLearningProfiles.Count} localization={localizationProfiles.Count} benchmarkPanels={benchmarkPanels.Count} actuatorStrategies={actuatorStrategies.Count} cameraIngest={cameraIngestStrategies.Count} reservoirs={reservoirStrategies.Count} distributedWitnesses={distributedWitnesses.Count} networkTransports={networkTransports.Count} authorityPolicies={authorityPolicies.Count} modules={moduleCatalog.Count}"); + Console.WriteLine( + $"perfect-machine-codebook id={codebook.CodebookId} motifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} augmentations={decoder.Configuration.TemplateAugmentations.Length}"); + Console.WriteLine( + $"perfect-machine-actuator source={actuator.SourceId} delay={actuator.TargetDelaySamples:0.000} ratio={actuator.ResampleRatio:0.000000000} controls={actuator.FaustControls.Length}"); + Console.WriteLine( + $"perfect-machine-clock anchors={clock?.AnchorCount ?? 0} coverage={clock?.AnchorCoverage ?? 0:0.000} offset={clock?.SourceOffsetSamples ?? double.NaN:0.000} confidence={clock?.Confidence ?? 0:0.000}"); + Console.WriteLine( + $"perfect-machine-fensalir gpuCameras={gpuFrame.Cameras.Count} gpuTextures={gpuFrame.ExternalTextures.Count} acousticConstraints={acousticFrame.Constraints.Count} timingConfidence={acousticFrame.TimingConfidence:0.000}"); + Console.WriteLine( + $"perfect-machine-localization candidates={localizationGrid.Count} best=({localization?.PositionMeters.X ?? float.NaN:0.000},{localization?.PositionMeters.Y ?? float.NaN:0.000},{localization?.PositionMeters.Z ?? float.NaN:0.000}) score={localization?.Score ?? 0.0:0.000}"); + + return profiles.Count >= 6 && + calibrationPlans.Count >= 3 && + audioFields.Count >= 4 && + visualFields.Count >= 4 && + computePlans.Count >= 3 && + assemblyPlans.Count >= 4 && + captureProfiles.Count >= 8 && + publications.Count >= 3 && + languageProfiles.Count >= 5 && + pathLearningProfiles.Count >= 3 && + localizationProfiles.Count >= 4 && + benchmarkPanels.Count >= 2 && + actuatorStrategies.Count >= 4 && + cameraIngestStrategies.Count >= 4 && + reservoirStrategies.Count >= 3 && + distributedWitnesses.Count >= 4 && + networkTransports.Count >= 4 && + authorityPolicies.Count >= 1 && + moduleCatalog.Count >= 17 && + codebook.Motifs.Length == MimirBioacousticTimeline.SymbolCount && + decoder.Configuration.TemplateAugmentations.Length > 0 && + actuator.FaustControls.Length >= 2 && + gpuFrame.HasInput && + acousticFrame.HasInput && + localization is { Score: > 0.90 } && + clock is { AnchorCount: >= 4, Confidence: > 0.70 } + ? 0 + : 1; +} + static MimirAudioSynchronizationReport? EstimateBioacousticDelay(float[] reference, float[] candidate, int sampleRate) { const string referenceSourceId = "actuator-reference"; @@ -937,6 +1088,105 @@ static int RenderChirpBinFloat32( return 0; } +static async Task RunPerfectMachineContractSmokeAsync(string outputPath) +{ + var timeline = MimirBioacousticTimeline.Default; + var createdAt = DateTimeOffset.UtcNow; + var codebook = MimirCultMeshContractFactory.CreateCodebookState( + "mimir-bioacoustic-default", + "mimir-bioacoustic-segmented-birdcall-v1", + timeline, + createdAt); + var decoder = MimirCultMeshContractFactory.CreateDecoderState( + "mimir-decoder-baseline-mfcc-index", + codebook.CodebookId, + MimirBioacousticDecoderConfiguration.BaselineMfccIndex, + createdAt); + var pathState = MimirCultMeshContractFactory.CreatePathState( + new MimirAudioSynchronizationState( + "loopback-scarlett-speakers", + "scarlett-host-mic", + 192_000, + 61.25, + 61.25, + 0.319, + 0.875, + 0.94, + [new MimirChirpletBandResponse(5_600.0, 0.92), new MimirChirpletBandResponse(8_400.0, 0.86), new MimirChirpletBandResponse(12_600.0, 0.71)], + 1_000_000_000L, + 10, + 10), + "loopback-scarlett-speakers", + "bioacoustic-contract-smoke"); + var controller = new MimirSroPllActuatorController(); + var command = controller.Update( + "scarlett-host-mic", + 0, + new MimirAudioSynchronizationState( + "loopback-scarlett-speakers", + "scarlett-host-mic", + 192_000, + 61.25, + 61.25, + 0.319, + 0.875, + 0.94, + [], + 1_000_000_000L, + 10, + 10), + dtSeconds: 0.001); + var actuator = MimirCultMeshContractFactory.CreateActuatorState( + "scarlett-host-mic-alignment", + MimirAlignmentActuatorProfile.SixSourceFaust.Id, + command, + createdAt); + + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); + using var cache = await CultCacheMessagePack.OpenAsync(outputPath, new CultCacheOpenOptions + { + PullOnOpen = File.Exists(outputPath) + }).ConfigureAwait(false); + await cache.UpsertAsync( + codebook, + new CultRecordHandle(new CultRecordKey($"mimir-codebook:{codebook.CodebookId}"))) + .ConfigureAwait(false); + await cache.UpsertAsync( + decoder, + new CultRecordHandle(new CultRecordKey($"mimir-decoder:{decoder.DecoderId}"))) + .ConfigureAwait(false); + await cache.UpsertAsync( + pathState, + new CultRecordHandle(new CultRecordKey($"mimir-path:{pathState.PathId}"))) + .ConfigureAwait(false); + await cache.UpsertAsync( + actuator, + new CultRecordHandle(new CultRecordKey($"mimir-actuator:{actuator.ActuatorId}"))) + .ConfigureAwait(false); + await cache.FlushAsync().ConfigureAwait(false); + + Console.WriteLine( + $"perfect-machine-contract-smoke path={outputPath} codebookMotifs={codebook.Motifs.Length} decoder={decoder.Configuration.Id} pathBands={pathState.BandResponses.Length} actuatorControls={actuator.FaustControls.Length}"); + return 0; +} + +static async Task WritePerfectMachineManifestAsync(string outputPath) +{ + var manifest = MimirPerfectMachineManifestFactory.Create(); + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputPath)) ?? "."); + await File.WriteAllTextAsync( + outputPath, + JsonSerializer.Serialize(manifest, new JsonSerializerOptions { WriteIndented = true })) + .ConfigureAwait(false); + Console.WriteLine( + $"perfect-machine-manifest path={outputPath} schema={manifest.Schema} modules={manifest.Modules.Count} profiles={manifest.NodeProfiles.Count} captures={manifest.CaptureProfiles.Count} publications={manifest.Publications.Count}"); + return manifest.Modules.Count >= 13 && + manifest.CaptureProfiles.Count >= 8 && + manifest.Publications.Count >= 3 + ? 0 + : 1; +} + static int RenderBioacousticFloat32(string outputPath, int sampleRate, double seconds) { var segmentCount = Math.Max(1, (int)Math.Ceiling(seconds / MimirBioacousticTimeline.SegmentSeconds)); @@ -971,74 +1221,28 @@ static int RenderBioacousticFloat32(string outputPath, int sampleRate, double se static IReadOnlyList BioacousticTrainingHypotheses() { - var clean = new CepstralDegradationSetting("template-clean", 0.0, 0.0, 0); - var blur = new CepstralDegradationSetting("template-blur", 0.0, 0.0, 1); - var warp = new CepstralDegradationSetting("template-warp-light", 0.75, 1.25, 0); - var warpBlur = new CepstralDegradationSetting("template-warp-blur", 0.75, 1.25, 1); - return - [ - new( - "baseline-mfcc-index", - "Current smoke baseline: four projection tables, 14-bit hashes, light degradation augmentation.", - CepstralDecoderOptions.Default), - new( - "compact-fast-index", - "Smaller feature and table shape. It should run faster if identity survives; otherwise it dies with a receipt.", - new CepstralDecoderOptions( - FftSize: 1024, - HopSize: 256, - MelBins: 32, - CepstralCoefficients: 10, - MinFrequencyHz: 300, - MaxFrequencyHz: 14_500, - TableCount: 3, - HashBits: 12, - NearHashRadius: 3, - DenseStepSeconds: 0.040, - ProposalBudgetMultiplier: 6.0, - TemplateAugmentations: [clean, blur, warpBlur])), - new( - "robust-wide-index", - "More cepstra and a wider augmentation bank. It pays more CPU for warped-room survival.", - new CepstralDecoderOptions( - FftSize: 1024, - HopSize: 192, - MelBins: 48, - CepstralCoefficients: 18, - MinFrequencyHz: 180, - MaxFrequencyHz: 15_500, - TableCount: 5, - HashBits: 15, - NearHashRadius: 4, - DenseStepSeconds: 0.030, - ProposalBudgetMultiplier: 9.0, - TemplateAugmentations: - [ - clean, - blur, - warp, - warpBlur, - new("template-warp-heavy", 1.25, 1.75, 1) - ])), - new( - "highband-room-index", - "Bias toward the measured useful upper speech/birdcall band and away from low room junk.", - new CepstralDecoderOptions( - FftSize: 1024, - HopSize: 256, - MelBins: 40, - CepstralCoefficients: 14, - MinFrequencyHz: 1_200, - MaxFrequencyHz: 16_000, - TableCount: 4, - HashBits: 14, - NearHashRadius: 4, - DenseStepSeconds: 0.035, - ProposalBudgetMultiplier: 8.0, - TemplateAugmentations: [clean, blur, warp, warpBlur])) - ]; + return MimirBioacousticDecoderConfiguration.BuiltInProfiles + .Select(profile => new BioacousticTrainingHypothesis( + profile.Id, + profile.Description, + CepstralDecoderOptions.FromRuntime(profile))) + .ToArray(); } +static IReadOnlyList CepstralSmokeDegradationSettings() => +[ + new("clean-roundtrip", CepstralDegradationProfiles.Clean.WarpFrames, CepstralDegradationProfiles.Clean.WarpCoefficients, CepstralDegradationProfiles.Clean.BlurPasses), + new("blur-light", CepstralDegradationProfiles.Blur.WarpFrames, CepstralDegradationProfiles.Blur.WarpCoefficients, CepstralDegradationProfiles.Blur.BlurPasses), + new("warp-light", CepstralDegradationProfiles.WarpLight.WarpFrames, CepstralDegradationProfiles.WarpLight.WarpCoefficients, CepstralDegradationProfiles.WarpLight.BlurPasses), + new("warp-light-blur", CepstralDegradationProfiles.WarpBlur.WarpFrames, CepstralDegradationProfiles.WarpBlur.WarpCoefficients, CepstralDegradationProfiles.WarpBlur.BlurPasses) +]; + +static IReadOnlyList CepstralTrainingDegradationSettings() => +[ + .. CepstralSmokeDegradationSettings(), + new("warp-heavy-blur", CepstralDegradationProfiles.WarpHeavy.WarpFrames, CepstralDegradationProfiles.WarpHeavy.WarpCoefficients, CepstralDegradationProfiles.WarpHeavy.BlurPasses) +]; + static MimirChirpBinCalibrationModel? LoadOptionalCalibration(string[] args) { var path = ParseStringOption(args, "--calibration", ""); @@ -1623,150 +1827,20 @@ static double MeanCepstralTimingError( Math.Abs(observation.SampleOffset - timeline.EventForIndex(observation.EventIndex).StartSeconds * sampleRate)); } -static BioacousticClockHypothesis? FitBioacousticClockHypothesis( +static MimirBioacousticClockHypothesis? FitBioacousticClockHypothesis( IReadOnlyList observations, MimirBioacousticTimeline timeline, - int sampleRate) -{ - var anchors = observations - .Select(observation => new BioacousticClockAnchor( - observation.EventIndex, - timeline.EventForIndex(observation.EventIndex).StartSeconds, - observation.SampleOffset, - Math.Clamp(observation.Confidence, 0.001, 1.0))) - .OrderByDescending(anchor => anchor.Confidence) - .Take(24) - .ToArray(); - if (anchors.Length == 0) - { - return null; - } - - var candidates = new List(); - foreach (var anchor in anchors) - { - var offset = anchor.SampleOffset - anchor.TimelineSeconds * sampleRate; - AddClockCandidate(candidates, anchors, sampleRate, offset, sampleRate); - } - - if (anchors.Length >= 3) - { - for (var first = 0; first < anchors.Length; first++) - { - for (var second = first + 1; second < anchors.Length; second++) - { - var dt = anchors[second].TimelineSeconds - anchors[first].TimelineSeconds; - if (Math.Abs(dt) < 0.08) - { - continue; - } - - var rate = (anchors[second].SampleOffset - anchors[first].SampleOffset) / dt; - if (!double.IsFinite(rate) || rate < sampleRate * 0.98 || rate > sampleRate * 1.02) - { - continue; - } - - var offset = anchors[first].SampleOffset - anchors[first].TimelineSeconds * rate; - AddClockCandidate(candidates, anchors, sampleRate, offset, rate); - } - } - } - - return candidates - .OrderByDescending(candidate => candidate.Score) - .ThenBy(candidate => candidate.MeanAbsoluteErrorSamples) - .FirstOrDefault(); -} - -static void AddClockCandidate( - List candidates, - IReadOnlyList anchors, - int nominalSampleRate, - double sourceOffsetSamples, - double effectiveSampleRate) -{ - var tolerance = Math.Max(36.0, nominalSampleRate * 0.012); - var inliers = anchors - .Where(anchor => Math.Abs(anchor.SampleOffset - (sourceOffsetSamples + anchor.TimelineSeconds * effectiveSampleRate)) <= tolerance) - .ToArray(); - if (inliers.Length == 0) - { - return; - } - - var refined = FitBioacousticClockToInliers(inliers, nominalSampleRate); - if (refined != null) - { - candidates.Add(refined); - } -} - -static BioacousticClockHypothesis? FitBioacousticClockToInliers( - IReadOnlyList anchors, - int nominalSampleRate) -{ - if (anchors.Count == 0) - { - return null; - } - - if (anchors.Count == 1) - { - var anchor = anchors[0]; - var offset = anchor.SampleOffset - anchor.TimelineSeconds * nominalSampleRate; - var singleAnchorConfidence = anchor.Confidence * 0.20; - return new BioacousticClockHypothesis( - offset, - nominalSampleRate, - 1, - 0.0, - singleAnchorConfidence, - singleAnchorConfidence, - anchors.ToArray()); - } - - var totalWeight = anchors.Sum(anchor => Math.Max(1.0e-6, anchor.Confidence)); - var meanTimeline = anchors.Sum(anchor => anchor.TimelineSeconds * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; - var meanSample = anchors.Sum(anchor => anchor.SampleOffset * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; - var covariance = 0.0; - var variance = 0.0; - foreach (var anchor in anchors) - { - var weight = Math.Max(1.0e-6, anchor.Confidence); - var dt = anchor.TimelineSeconds - meanTimeline; - covariance += weight * dt * (anchor.SampleOffset - meanSample); - variance += weight * dt * dt; - } - - var effectiveSampleRate = anchors.Count >= 3 && variance > 1.0e-12 ? covariance / variance : nominalSampleRate; - if (!double.IsFinite(effectiveSampleRate) || - effectiveSampleRate < nominalSampleRate * 0.98 || - effectiveSampleRate > nominalSampleRate * 1.02) - { - effectiveSampleRate = nominalSampleRate; - } - - var sourceOffset = meanSample - effectiveSampleRate * meanTimeline; - var weightedResidual = anchors.Sum(anchor => - { - var predicted = sourceOffset + anchor.TimelineSeconds * effectiveSampleRate; - return Math.Abs(anchor.SampleOffset - predicted) * Math.Max(1.0e-6, anchor.Confidence); - }) / totalWeight; - var residualConfidence = 1.0 / (1.0 + weightedResidual / Math.Max(1.0, nominalSampleRate * 0.001)); - var countConfidence = Math.Clamp(anchors.Count / 5.0, 0.0, 1.0); - var anchorConfidence = Math.Clamp(anchors.Average(anchor => anchor.Confidence), 0.0, 1.0); - var confidence = residualConfidence * 0.35 + countConfidence * 0.45 + anchorConfidence * 0.20; - var score = confidence + Math.Min(anchors.Count, 8) * 0.15 - weightedResidual / Math.Max(1.0, nominalSampleRate * 0.010); - return new BioacousticClockHypothesis( - sourceOffset, - effectiveSampleRate, - anchors.Count, - weightedResidual, - confidence, - score, - anchors.ToArray()); -} + int sampleRate) => + new MimirBioacousticClockSolver().Fit( + observations + .Select(observation => new MimirBioacousticWordObservation( + observation.EventIndex, + observation.SampleOffset, + observation.Confidence)) + .ToArray(), + timeline, + sampleRate, + observations.Count); static double CepstralDistance(IReadOnlyList first, IReadOnlyList second) { @@ -2348,7 +2422,11 @@ internal sealed record CepstralDegradationSetting( string Name, double WarpFrames, double WarpCoefficients, - int BlurPasses); + int BlurPasses) +{ + public static CepstralDegradationSetting FromRuntime(MimirCepstralDegradationProfile profile) => + new(profile.Id, profile.WarpFrames, profile.WarpCoefficients, profile.BlurPasses); +} internal sealed record CepstralDecoderOptions( int FftSize, @@ -2364,25 +2442,25 @@ internal sealed record CepstralDecoderOptions( double ProposalBudgetMultiplier, IReadOnlyList TemplateAugmentations) { - public static CepstralDecoderOptions Default { get; } = new( - FftSize: 1024, - HopSize: 256, - MelBins: 40, - CepstralCoefficients: 14, - MinFrequencyHz: 180.0, - MaxFrequencyHz: 15_000.0, - TableCount: 4, - HashBits: 14, - NearHashRadius: 4, - DenseStepSeconds: 0.040, - ProposalBudgetMultiplier: 8.0, - TemplateAugmentations: - [ - new CepstralDegradationSetting("template-clean", 0.0, 0.0, 0), - new CepstralDegradationSetting("template-blur", 0.0, 0.0, 1), - new CepstralDegradationSetting("template-warp-light", 0.75, 1.25, 0), - new CepstralDegradationSetting("template-warp-blur", 0.75, 1.25, 1) - ]); + public static CepstralDecoderOptions Default { get; } = + FromRuntime(MimirBioacousticDecoderConfiguration.BaselineMfccIndex); + + public static CepstralDecoderOptions FromRuntime(MimirBioacousticDecoderConfiguration configuration) => + new( + configuration.FftSize, + configuration.HopSize, + configuration.MelBins, + configuration.CepstralCoefficients, + configuration.MinFrequencyHz, + configuration.MaxFrequencyHz, + configuration.ProjectionTableCount, + configuration.ProjectionHashBits, + configuration.NearHashRadius, + configuration.DenseStepSeconds, + configuration.ProposalBudgetMultiplier, + configuration.TemplateAugmentations + .Select(CepstralDegradationSetting.FromRuntime) + .ToArray()); } internal sealed record BioacousticTrainingHypothesis( @@ -2405,21 +2483,6 @@ internal sealed record CepstralWordObservation( double SampleOffset, double Confidence); -internal sealed record BioacousticClockAnchor( - ulong EventIndex, - double TimelineSeconds, - double SampleOffset, - double Confidence); - -internal sealed record BioacousticClockHypothesis( - double SourceOffsetSamples, - double EffectiveSampleRate, - int AnchorCount, - double MeanAbsoluteErrorSamples, - double Confidence, - double Score, - IReadOnlyList Anchors); - internal sealed record CepstralWordIndex( IReadOnlyList Templates, IReadOnlyDictionary<(int Table, int Hash), List> Buckets); diff --git a/src/Mimir.Runtime/Mimir.Runtime.csproj b/src/Mimir.Runtime/Mimir.Runtime.csproj index 0bd4952..6745c44 100644 --- a/src/Mimir.Runtime/Mimir.Runtime.csproj +++ b/src/Mimir.Runtime/Mimir.Runtime.csproj @@ -12,6 +12,7 @@ + diff --git a/src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs new file mode 100644 index 0000000..49ddbce --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationConfiguration.cs @@ -0,0 +1,82 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirAcousticLocalizationModel +{ + PairwiseTdoa, + SrpPhatGrid, + DelayAndSumBeamformer, + EquivalentSparseSources, + VisualConstrainedHybrid +} + +public sealed record MimirAcousticLocalizationConfiguration( + string Id, + string Description, + MimirAcousticLocalizationModel Model, + string[] RequiredInputs, + string[] OptionalInputs, + double GridSpacingMeters, + double SpeedOfSoundMetersPerSecond, + double MinimumTimingConfidence, + double TargetPositionUncertaintyMeters, + bool EmitsFensalirConstraints, + string Notes); + +public static class MimirAcousticLocalizationConfigurations +{ + public static MimirAcousticLocalizationConfiguration PairwiseTdoa { get; } = new( + "pairwise-tdoa-baseline", + "Smallest position proof from calibrated pair delays.", + MimirAcousticLocalizationModel.PairwiseTdoa, + RequiredInputs: ["mic-geometry", "pairwise-delay-state"], + OptionalInputs: ["speaker-position", "visual-prior"], + GridSpacingMeters: 0.25, + SpeedOfSoundMetersPerSecond: 343.0, + MinimumTimingConfidence: 0.70, + TargetPositionUncertaintyMeters: 0.50, + EmitsFensalirConstraints: true, + "Good first proof after real mic timing works."); + + public static MimirAcousticLocalizationConfiguration SrpPhatGrid { get; } = new( + "srp-phat-grid", + "Grid search over calibrated mic pair delays with PHAT-style robustness.", + MimirAcousticLocalizationModel.SrpPhatGrid, + RequiredInputs: ["mic-geometry", "pairwise-delay-state", "band-response"], + OptionalInputs: ["visual-prior", "room-bounds"], + GridSpacingMeters: 0.10, + SpeedOfSoundMetersPerSecond: 343.0, + MinimumTimingConfidence: 0.75, + TargetPositionUncertaintyMeters: 0.25, + EmitsFensalirConstraints: true, + "Promote when pairwise baseline survives reflection enough to justify a grid."); + + public static MimirAcousticLocalizationConfiguration EquivalentSources { get; } = SrpPhatGrid with + { + Id = "equivalent-sparse-sources", + Description = "Sparse physical source dictionary for volumetric field reconstruction.", + Model = MimirAcousticLocalizationModel.EquivalentSparseSources, + GridSpacingMeters = 0.15, + MinimumTimingConfidence = 0.80, + TargetPositionUncertaintyMeters = 0.20, + Notes = "Use only after calibration gives a credible measurement model; inverse problems lie politely." + }; + + public static MimirAcousticLocalizationConfiguration VisualConstrainedHybrid { get; } = SrpPhatGrid with + { + Id = "visual-constrained-hybrid", + Description = "Likely Mimir field model: acoustic candidates constrained by Fensalir visual tracks.", + Model = MimirAcousticLocalizationModel.VisualConstrainedHybrid, + RequiredInputs = ["mic-geometry", "pairwise-delay-state", "band-response", "fensalir-visual-claims"], + OptionalInputs = ["speaker-position", "room-bounds", "motion-tracks"], + TargetPositionUncertaintyMeters = 0.15, + Notes = "Acoustic evidence becomes weighted constraints in the temporal field, not a solo oracle." + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + PairwiseTdoa, + SrpPhatGrid, + EquivalentSources, + VisualConstrainedHybrid + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs b/src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs new file mode 100644 index 0000000..ff2ff5c --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirAcousticLocalizationSolver.cs @@ -0,0 +1,101 @@ +using System.Numerics; + +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirMicrophonePose( + string SourceId, + Vector3 PositionMeters); + +public sealed record MimirPairDelayObservation( + string MicA, + string MicB, + double DelaySeconds, + double Confidence); + +public sealed record MimirAcousticSourceCandidate( + Vector3 PositionMeters, + double Score, + int SupportingPairs); + +public sealed class MimirSrpPhatGridSolver( + double speedOfSoundMetersPerSecond = 343.0, + double halfScoreErrorMicroseconds = 100.0) +{ + public MimirAcousticSourceCandidate? FindBestCandidate( + IReadOnlyList microphones, + IReadOnlyList candidatePoints, + IReadOnlyList observations) + { + if (microphones.Count < 2 || candidatePoints.Count == 0 || observations.Count == 0) + { + return null; + } + + var micById = microphones.ToDictionary(mic => mic.SourceId, StringComparer.Ordinal); + MimirAcousticSourceCandidate? best = null; + foreach (var point in candidatePoints) + { + var weightedScore = 0.0; + var weight = 0.0; + var supportingPairs = 0; + foreach (var observation in observations) + { + if (observation.Confidence <= 0.0 || + !micById.TryGetValue(observation.MicA, out var micA) || + !micById.TryGetValue(observation.MicB, out var micB)) + { + continue; + } + + var predicted = (DistanceMeters(micB.PositionMeters, point) - DistanceMeters(micA.PositionMeters, point)) / + speedOfSoundMetersPerSecond; + var errorUs = Math.Abs(predicted - observation.DelaySeconds) * 1_000_000.0; + var localScore = 1.0 / (1.0 + errorUs / halfScoreErrorMicroseconds); + weightedScore += localScore * observation.Confidence; + weight += observation.Confidence; + supportingPairs++; + } + + if (weight <= 0.0) + { + continue; + } + + var score = weightedScore / weight; + if (best == null || score > best.Score) + { + best = new MimirAcousticSourceCandidate(point, score, supportingPairs); + } + } + + return best; + } + + public static IReadOnlyList BuildGrid(Vector3 min, Vector3 max, double spacingMeters) + { + if (spacingMeters <= 0.0) + { + throw new ArgumentOutOfRangeException(nameof(spacingMeters), "Grid spacing must be positive."); + } + + var points = new List(); + for (var z = min.Z; z <= max.Z + 1.0e-6f; z += (float)spacingMeters) + { + for (var y = min.Y; y <= max.Y + 1.0e-6f; y += (float)spacingMeters) + { + for (var x = min.X; x <= max.X + 1.0e-6f; x += (float)spacingMeters) + { + points.Add(new Vector3(x, y, z)); + } + } + } + + return points; + } + + private static double DistanceMeters(Vector3 a, Vector3 b) + { + var delta = a - b; + return Math.Sqrt(delta.LengthSquared()); + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs new file mode 100644 index 0000000..db8c629 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirAcousticPathLearningConfiguration.cs @@ -0,0 +1,84 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirPathLearningTarget +{ + MagnitudeResponse, + ConfusionMatrix, + DelayHypothesis, + FrequencyShiftHypothesis, + PhaseGroupDelay, + ReflectionRisk, + CodebookAdaptation +} + +public sealed record MimirPathLearningStage( + string Id, + MimirPathLearningTarget Target, + double DurationSeconds, + double MinimumConfidence, + string[] Outputs, + string[] Measurements); + +public sealed record MimirAcousticPathLearningConfiguration( + string Id, + string Description, + string OutputSourceId, + string[] MicSourceIds, + int SampleRate, + double BufferSeconds, + IReadOnlyList Stages, + string PersistenceDocument, + string Notes); + +public static class MimirAcousticPathLearningConfigurations +{ + public static MimirAcousticPathLearningConfiguration StarfireRoom { get; } = new( + "starfire-room-path-learning", + "Measure local monitor to Starfire mic paths before trusting active acoustic timing.", + "loopback-scarlett-speakers", + ["scarlett-input-1", "scarlett-input-2", "kiyo-pro-mic", "ps3eye-left-mic", "ps3eye-right-mic"], + SampleRate: 192_000, + BufferSeconds: 5.0, + Stages: + [ + new("usable-band", MimirPathLearningTarget.MagnitudeResponse, 8.0, 0.30, ["left", "right"], ["per-bin-energy", "noise-floor"]), + new("confusion", MimirPathLearningTarget.ConfusionMatrix, 12.0, 0.40, ["left", "right"], ["expected-symbol", "observed-energy-vector", "confidence"]), + new("global-delay", MimirPathLearningTarget.DelayHypothesis, 10.0, 0.55, ["left", "right"], ["timeline-offset", "residual", "anchor-coverage"]), + new("phase-delay", MimirPathLearningTarget.PhaseGroupDelay, 10.0, 0.45, ["left", "right"], ["phase-slope", "group-delay", "band-coherence"]), + new("adapt-codebook", MimirPathLearningTarget.CodebookAdaptation, 4.0, 0.50, ["left", "right"], ["reliable-symbols", "recommended-order"]) + ], + "mimir.acoustic_path_state", + "This is the calibration command shape requested for physical paths."); + + public static MimirAcousticPathLearningConfiguration RavenRoundTrip { get; } = StarfireRoom with + { + Id = "raven-roundtrip-path-learning", + Description = "Learn Raven loopback/mic paths and network delay as remote typed evidence.", + OutputSourceId = "raven-loopback", + MicSourceIds = ["raven-scarlett-input-1", "raven-scarlett-input-2"], + PersistenceDocument = "mimir.acoustic_path_state", + Notes = "Raven decodes locally; Starfire receives timing/path documents, not raw authority." + }; + + public static MimirAcousticPathLearningConfiguration PhoneWitness { get; } = StarfireRoom with + { + Id = "phone-witness-path-learning", + Description = "Learn robust reduced-band paths for a phone microphone in the room.", + MicSourceIds = ["phone-mic"], + SampleRate = 48_000, + Stages = + [ + new("usable-band", MimirPathLearningTarget.MagnitudeResponse, 6.0, 0.25, ["left", "right"], ["mel-band-energy", "noise-floor"]), + new("global-delay", MimirPathLearningTarget.DelayHypothesis, 8.0, 0.40, ["left", "right"], ["timeline-offset", "residual", "anchor-coverage"]), + new("adapt-codebook", MimirPathLearningTarget.CodebookAdaptation, 4.0, 0.40, ["left", "right"], ["reliable-words", "recommended-profile"]) + ], + Notes = "A reduced receiver must still place itself on the canonical source timeline." + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + StarfireRoom, + RavenRoundTrip, + PhoneWitness + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs new file mode 100644 index 0000000..cc81672 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirAudioActuatorConfiguration.cs @@ -0,0 +1,78 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirAudioActuatorKind +{ + IntegerDelay, + FarrowFractionalDelay, + VariableAsrc, + FrequencyDomainPhaseCorrection, + HybridDelayAsrc +} + +public sealed record MimirAudioActuatorConfiguration( + string Id, + string Description, + MimirAudioActuatorKind Kind, + string Owner, + double MaxDelaySamples, + double MaxSroPpm, + double TargetResidualMicroseconds, + bool SuitableForHotPath, + string[] RequiredSignals, + string[] ProofCommands); + +public static class MimirAudioActuatorConfigurations +{ + public static MimirAudioActuatorConfiguration IntegerDelayBaseline { get; } = new( + "integer-delay-baseline", + "Coarse diagnostic baseline only.", + MimirAudioActuatorKind.IntegerDelay, + "Mimir.Runtime diagnostic", + MaxDelaySamples: 4096.0, + MaxSroPpm: 0.0, + TargetResidualMicroseconds: 1000.0 / 192.0, + SuitableForHotPath: false, + RequiredSignals: ["smoothed-delay"], + ProofCommands: ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test"]); + + public static MimirAudioActuatorConfiguration FaustFarrowDelay { get; } = new( + "faust-farrow-fractional-delay", + "Production-shaped fractional delay surface; Mimir estimates, Faust moves samples.", + MimirAudioActuatorKind.FarrowFractionalDelay, + "Faust/native DSP", + MaxDelaySamples: 4096.0, + MaxSroPpm: 0.0, + TargetResidualMicroseconds: 1.0, + SuitableForHotPath: true, + RequiredSignals: ["smoothed-delay", "confidence", "band-response"], + ProofCommands: ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test"]); + + public static MimirAudioActuatorConfiguration VariableAsrc { get; } = new( + "variable-asrc-drift-hold", + "Long-horizon sample-rate-offset correction after delay is already bounded.", + MimirAudioActuatorKind.VariableAsrc, + "Faust/native DSP", + MaxDelaySamples: 512.0, + MaxSroPpm: 300.0, + TargetResidualMicroseconds: 1.0, + SuitableForHotPath: true, + RequiredSignals: ["delay-slope", "sro-ppm", "confidence"], + ProofCommands: ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]); + + public static MimirAudioActuatorConfiguration HybridDelayAsrc { get; } = FaustFarrowDelay with + { + Id = "hybrid-delay-asrc", + Description = "Final target: fractional delay for phase, ASRC for drift, both driven by one sync state.", + Kind = MimirAudioActuatorKind.HybridDelayAsrc, + MaxSroPpm = 300.0, + RequiredSignals = ["smoothed-delay", "sro-ppm", "confidence", "band-response", "group-delay-model"] + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + IntegerDelayBaseline, + FaustFarrowDelay, + VariableAsrc, + HybridDelayAsrc + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs b/src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs new file mode 100644 index 0000000..de89a00 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirAudioAlignmentActuator.cs @@ -0,0 +1,88 @@ +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirAlignmentActuatorProfile( + string Id, + string FaustDspPath, + int SourceCount, + double MaxDelaySamples, + string DelayControlFormat, + string GainControlFormat) +{ + public static MimirAlignmentActuatorProfile SixSourceFaust { get; } = new( + "six-source-faust-fractional-delay", + "faust/mimir_alignment_actuator.dsp", + SourceCount: 6, + MaxDelaySamples: 4096.0, + DelayControlFormat: "source{0}/delay_samples", + GainControlFormat: "source{0}/gain"); +} + +public sealed record MimirActuatorCommand( + string SourceId, + double TargetDelaySamples, + double ResampleRatio, + double Confidence, + IReadOnlyDictionary FaustControls); + +public sealed record MimirActuatorControllerOptions( + double MinimumConfidence = 0.05, + double MinimumDtSeconds = 0.001, + double DelayAlphaMin = 0.005, + double DelayAlphaMax = 0.05, + double SroBetaMin = 0.0005, + double SroBetaMax = 0.006, + double MaxSroPpm = 300.0); + +public sealed class MimirSroPllActuatorController( + MimirAlignmentActuatorProfile? profile = null, + MimirActuatorControllerOptions? options = null) +{ + private readonly MimirAlignmentActuatorProfile profile = profile ?? MimirAlignmentActuatorProfile.SixSourceFaust; + private readonly MimirActuatorControllerOptions options = options ?? new(); + private readonly Dictionary states = new(StringComparer.Ordinal); + + public MimirActuatorCommand Update(string sourceId, int sourceIndex, double delaySamples, double confidence, double dtSeconds) + { + if (!states.TryGetValue(sourceId, out var state)) + { + state = new ActuatorState(); + states[sourceId] = state; + } + + if (confidence >= options.MinimumConfidence && dtSeconds >= options.MinimumDtSeconds) + { + var error = delaySamples - state.DelaySamples; + var alpha = Math.Clamp(confidence * 0.08, options.DelayAlphaMin, options.DelayAlphaMax); + var beta = Math.Clamp(confidence * 0.01, options.SroBetaMin, options.SroBetaMax); + state.DelaySamples += alpha * error; + state.SroPpm = Math.Clamp( + state.SroPpm + beta * error / dtSeconds, + -options.MaxSroPpm, + options.MaxSroPpm); + state.Confidence += (confidence - state.Confidence) * 0.05; + } + else + { + state.Confidence *= 0.995; + } + + var targetDelay = Math.Clamp(state.DelaySamples, 0.0, profile.MaxDelaySamples); + var resampleRatio = 1.0 - state.SroPpm * 1.0e-6; + var controls = new Dictionary(StringComparer.Ordinal) + { + [string.Format(profile.DelayControlFormat, Math.Clamp(sourceIndex, 0, profile.SourceCount - 1))] = (float)targetDelay, + [string.Format(profile.GainControlFormat, Math.Clamp(sourceIndex, 0, profile.SourceCount - 1))] = 1.0f + }; + return new MimirActuatorCommand(sourceId, targetDelay, resampleRatio, state.Confidence, controls); + } + + public MimirActuatorCommand Update(string sourceId, int sourceIndex, MimirAudioSynchronizationState state, double dtSeconds) => + Update(sourceId, sourceIndex, state.SmoothedDelaySamples, state.Confidence, dtSeconds); + + private sealed class ActuatorState + { + public double DelaySamples { get; set; } + public double SroPpm { get; set; } + public double Confidence { get; set; } + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirAudioFieldConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirAudioFieldConfiguration.cs new file mode 100644 index 0000000..9e668e3 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirAudioFieldConfiguration.cs @@ -0,0 +1,127 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirSpatialOutputTarget +{ + ObsMonoStem, + ObsStereoStem, + FirstOrderAmbisonic, + HigherOrderAmbisonic, + FensalirDiagnosticField, + CultMeshRemoteObservation +} + +public enum MimirLocalizationModel +{ + None, + TdoaPairwise, + SrpPhatGrid, + VisualConstrainedSrp, + SparseEquivalentSourceGrid, + HybridTemporalEvidence +} + +public sealed record MimirMicrophoneGeometry( + string SourceId, + double X, + double Y, + double Z, + string Role, + double Reliability); + +public sealed record MimirSpeakerGeometry( + string SourceId, + double X, + double Y, + double Z, + string Role); + +public sealed record MimirAudioFieldConfiguration( + string Id, + string Description, + MimirAudioFieldModel FieldModel, + MimirLocalizationModel Localization, + MimirSpatialOutputTarget OutputTarget, + int AmbisonicOrder, + bool RequiresKnownMicGeometry, + bool UsesVisualConstraints, + IReadOnlyList Microphones, + IReadOnlyList Speakers); + +public static class MimirAudioFieldConfigurations +{ + public static MimirAudioFieldConfiguration AlignedStemsSixMic { get; } = new( + "aligned-stems-six-mic", + "Production-first six-mic alignment: aligned host/co-streamer/context stems with no volumetric claim.", + MimirAudioFieldModel.AlignedStems, + MimirLocalizationModel.None, + MimirSpatialOutputTarget.ObsStereoStem, + AmbisonicOrder: 0, + RequiresKnownMicGeometry: false, + UsesVisualConstraints: false, + Microphones: CreateDefaultMicrophones(), + Speakers: CreateDefaultSpeakers()); + + public static MimirAudioFieldConfiguration SourceBasedSpatialBus { get; } = new( + "source-based-spatial-bus", + "Honest first spatial bed: source tracks and known speaker/mic geometry drive a Faust spatial bus.", + MimirAudioFieldModel.SourceBasedSpatialBus, + MimirLocalizationModel.TdoaPairwise, + MimirSpatialOutputTarget.ObsStereoStem, + AmbisonicOrder: 0, + RequiresKnownMicGeometry: true, + UsesVisualConstraints: true, + Microphones: CreateDefaultMicrophones(), + Speakers: CreateDefaultSpeakers()); + + public static MimirAudioFieldConfiguration FirstOrderAmbisonicBed { get; } = new( + "foa-bed", + "Low-order ambisonic bed for diffuse/context audio after synchronization earns trust.", + MimirAudioFieldModel.FirstOrderAmbisonicBed, + MimirLocalizationModel.VisualConstrainedSrp, + MimirSpatialOutputTarget.FirstOrderAmbisonic, + AmbisonicOrder: 1, + RequiresKnownMicGeometry: true, + UsesVisualConstraints: true, + Microphones: CreateDefaultMicrophones(), + Speakers: CreateDefaultSpeakers()); + + public static MimirAudioFieldConfiguration HybridEvidenceField { get; } = new( + "hybrid-evidence-field", + "Perfect Machine target: synchronized acoustic observations become temporal evidence claims for Fensalir and Faust.", + MimirAudioFieldModel.HybridEvidenceField, + MimirLocalizationModel.HybridTemporalEvidence, + MimirSpatialOutputTarget.FensalirDiagnosticField, + AmbisonicOrder: 1, + RequiresKnownMicGeometry: true, + UsesVisualConstraints: true, + Microphones: CreateDefaultMicrophones(), + Speakers: CreateDefaultSpeakers()); + + public static IReadOnlyList BuiltIn { get; } = + [ + AlignedStemsSixMic, + SourceBasedSpatialBus, + FirstOrderAmbisonicBed, + HybridEvidenceField + ]; + + public static IReadOnlyList DefaultMicrophones { get; } = CreateDefaultMicrophones(); + + public static IReadOnlyList DefaultSpeakers { get; } = CreateDefaultSpeakers(); + + private static IReadOnlyList CreateDefaultMicrophones() => + [ + new("scarlett-host-mic", 0.0, 0.0, 1.25, "dialogue-anchor", 1.0), + new("scarlett-raven-mic", 1.5, 0.8, 1.25, "co-streamer-anchor", 0.95), + new("kiyo-basic-mic", -0.6, 0.3, 1.1, "room-context", 0.45), + new("kiyo-pro-mic", 0.7, 0.2, 1.1, "rgb-ground-context", 0.45), + new("ps3-eye-0-mic", -0.9, -0.4, 1.0, "tracking-context", 0.40), + new("ps3-eye-1-mic", 0.9, -0.4, 1.0, "tracking-context", 0.40) + ]; + + private static IReadOnlyList CreateDefaultSpeakers() => + [ + new("scarlett-left-monitor", -0.65, 0.7, 1.15, "left-reference"), + new("scarlett-right-monitor", 0.65, 0.7, 1.15, "right-reference") + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirAuthorityPolicyConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirAuthorityPolicyConfiguration.cs new file mode 100644 index 0000000..8f89804 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirAuthorityPolicyConfiguration.cs @@ -0,0 +1,72 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirAuthorityDecision +{ + ObserveOnly, + CandidateEvidence, + TrustedEvidence, + CanonicalAuthority, + Reject +} + +public sealed record MimirAuthorityRule( + string Id, + string AppliesTo, + MimirAuthorityDecision Decision, + string[] RequiredEvidence, + string[] ForbiddenConditions, + string Notes); + +public sealed record MimirAuthorityPolicyConfiguration( + string Id, + string Description, + IReadOnlyList Rules); + +public static class MimirAuthorityPolicyConfigurations +{ + public static MimirAuthorityPolicyConfiguration DefaultTimingPolicy { get; } = new( + "default-timing-authority-policy", + "One canonical timing authority, many witnesses. Trust is earned by decoded anchors and calibration receipts.", + [ + new( + "starfire-loopback-authority", + "loopback-scarlett-speakers", + MimirAuthorityDecision.CanonicalAuthority, + RequiredEvidence: ["same-asio-clock-domain", "fresh-loopback-buffer"], + ForbiddenConditions: ["device-missing", "stale-buffer"], + "Starfire loopback is the normal local source-time authority."), + new( + "raven-typed-evidence", + "raven-scarlett-witness", + MimirAuthorityDecision.TrustedEvidence, + RequiredEvidence: ["known-node", "matching-codebook", "clock-fit-confidence", "response-profile"], + ForbiddenConditions: ["claims-canonical-authority", "wrong-codebook", "network-only-timestamps"], + "Raven may contribute compact decoded truth but does not own the canonical clock."), + new( + "phone-candidate-evidence", + "phone-mic-witness", + MimirAuthorityDecision.CandidateEvidence, + RequiredEvidence: ["matching-codebook", "anchor-batch", "health-counters"], + ForbiddenConditions: ["unknown-node", "raw-timestamp-only", "agc-unknown-and-low-confidence"], + "Phones start as candidates until repeated calibration earns trust."), + new( + "diagnostic-media-observe-only", + "srt-program-bridge", + MimirAuthorityDecision.ObserveOnly, + RequiredEvidence: ["explicit-diagnostic-mode"], + ForbiddenConditions: ["used-as-clock-authority"], + "Bridge media can be useful without being trusted as time."), + new( + "unknown-node-reject", + "*", + MimirAuthorityDecision.Reject, + RequiredEvidence: [], + ForbiddenConditions: ["unknown-node", "missing-identity"], + "Codebook possession is not permission to steer the machine.") + ]); + + public static IReadOnlyList BuiltIn { get; } = + [ + DefaultTimingPolicy + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs new file mode 100644 index 0000000..270a40f --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirBenchmarkPanelConfiguration.cs @@ -0,0 +1,65 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirBenchmarkDegradationKind +{ + Clean, + CepstralWarp, + CepstralBlur, + WarpAndBlur, + BandLimited, + PacketJitter, + NetworkDelay, + CpuBudgetPressure +} + +public sealed record MimirBenchmarkDegradation( + string Id, + MimirBenchmarkDegradationKind Kind, + double Strength, + string Notes); + +public sealed record MimirBenchmarkPanelConfiguration( + string Id, + string Description, + string[] DecoderProfiles, + IReadOnlyList Degradations, + double MinimumRealtimeFactor, + double MinimumIdentityScore, + double MinimumTimingConfidence, + string ReceiptRoot); + +public static class MimirBenchmarkPanelConfigurations +{ + public static MimirBenchmarkPanelConfiguration BioacousticGolf { get; } = new( + "bioacoustic-golf", + "Performance golf panel for singing/hearing organ tuning.", + MimirBioacousticDecoderConfiguration.BuiltInProfiles.Select(profile => profile.Id).ToArray(), + [ + new("clean", MimirBenchmarkDegradationKind.Clean, 0.0, "Round trip without intentional damage."), + new("warp-light", MimirBenchmarkDegradationKind.CepstralWarp, 0.75, "Simplex-like domain warp target from the harness."), + new("blur-light", MimirBenchmarkDegradationKind.CepstralBlur, 1.0, "Separable five-tap Gaussian blur target."), + new("warp-blur", MimirBenchmarkDegradationKind.WarpAndBlur, 1.0, "Combined lossy-room stress."), + new("bandlimited-phone", MimirBenchmarkDegradationKind.BandLimited, 0.65, "Consumer mic / codec survival proxy."), + new("network-jitter", MimirBenchmarkDegradationKind.PacketJitter, 0.40, "Remote witness observation jitter proxy.") + ], + MinimumRealtimeFactor: 50.0, + MinimumIdentityScore: 0.85, + MinimumTimingConfidence: 0.80, + "artifacts/bioacoustic-training"); + + public static MimirBenchmarkPanelConfiguration MeatspaceAcceptance { get; } = BioacousticGolf with + { + Id = "meatspace-acceptance", + Description = "Physical Scarlett/monitor/mic acceptance panel.", + MinimumRealtimeFactor = 10.0, + MinimumIdentityScore = 0.65, + MinimumTimingConfidence = 0.70, + ReceiptRoot = "artifacts/meatspace-sync" + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + BioacousticGolf, + MeatspaceAcceptance + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs b/src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs new file mode 100644 index 0000000..18ee56e --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirBioacousticClockHypothesis.cs @@ -0,0 +1,196 @@ +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirBioacousticWordObservation( + ulong EventIndex, + double SampleOffset, + double Confidence); + +public sealed record MimirBioacousticClockAnchor( + ulong EventIndex, + double TimelineSeconds, + double SampleOffset, + double Confidence); + +public sealed record MimirBioacousticClockHypothesis( + double SourceOffsetSamples, + double EffectiveSampleRate, + int AnchorCount, + double AnchorCoverage, + double MeanAbsoluteErrorSamples, + double Confidence, + double Score, + IReadOnlyList Anchors) +{ + public double DelayMicroseconds(int sampleRate) => SourceOffsetSamples * 1_000_000.0 / sampleRate; +} + +public sealed record MimirBioacousticClockSolverOptions( + int MaxObservations = 24, + double MinimumPairSpacingSeconds = 0.08, + double InlierToleranceSeconds = 0.012, + double MinimumRateRatio = 0.98, + double MaximumRateRatio = 1.02, + double FullAnchorConfidenceCount = 5.0); + +public sealed class MimirBioacousticClockSolver(MimirBioacousticClockSolverOptions? options = null) +{ + private readonly MimirBioacousticClockSolverOptions options = options ?? new(); + + public MimirBioacousticClockHypothesis? Fit( + IReadOnlyList observations, + MimirBioacousticTimeline timeline, + int sampleRate, + int expectedEventCount) + { + var anchors = observations + .Select(observation => new MimirBioacousticClockAnchor( + observation.EventIndex, + timeline.EventForIndex(observation.EventIndex).StartSeconds, + observation.SampleOffset, + Math.Clamp(observation.Confidence, 0.001, 1.0))) + .OrderByDescending(anchor => anchor.Confidence) + .Take(options.MaxObservations) + .ToArray(); + if (anchors.Length == 0) + { + return null; + } + + var candidates = new List(); + foreach (var anchor in anchors) + { + AddCandidate( + candidates, + anchors, + sampleRate, + expectedEventCount, + anchor.SampleOffset - anchor.TimelineSeconds * sampleRate, + sampleRate); + } + + if (anchors.Length >= 3) + { + for (var first = 0; first < anchors.Length; first++) + { + for (var second = first + 1; second < anchors.Length; second++) + { + var dt = anchors[second].TimelineSeconds - anchors[first].TimelineSeconds; + if (Math.Abs(dt) < options.MinimumPairSpacingSeconds) + { + continue; + } + + var rate = (anchors[second].SampleOffset - anchors[first].SampleOffset) / dt; + if (!double.IsFinite(rate) || + rate < sampleRate * options.MinimumRateRatio || + rate > sampleRate * options.MaximumRateRatio) + { + continue; + } + + AddCandidate( + candidates, + anchors, + sampleRate, + expectedEventCount, + anchors[first].SampleOffset - anchors[first].TimelineSeconds * rate, + rate); + } + } + } + + return candidates + .OrderByDescending(candidate => candidate.Score) + .ThenBy(candidate => candidate.MeanAbsoluteErrorSamples) + .FirstOrDefault(); + } + + private void AddCandidate( + List candidates, + IReadOnlyList anchors, + int nominalSampleRate, + int expectedEventCount, + double sourceOffsetSamples, + double effectiveSampleRate) + { + var tolerance = Math.Max(36.0, nominalSampleRate * options.InlierToleranceSeconds); + var inliers = anchors + .Where(anchor => Math.Abs(anchor.SampleOffset - (sourceOffsetSamples + anchor.TimelineSeconds * effectiveSampleRate)) <= tolerance) + .ToArray(); + var refined = FitInliers(inliers, nominalSampleRate, expectedEventCount); + if (refined != null) + { + candidates.Add(refined); + } + } + + private MimirBioacousticClockHypothesis? FitInliers( + IReadOnlyList anchors, + int nominalSampleRate, + int expectedEventCount) + { + if (anchors.Count == 0) + { + return null; + } + + var anchorCoverage = expectedEventCount <= 0 ? 0.0 : anchors.Count / (double)expectedEventCount; + if (anchors.Count == 1) + { + var anchor = anchors[0]; + var offset = anchor.SampleOffset - anchor.TimelineSeconds * nominalSampleRate; + var singleAnchorConfidence = anchor.Confidence * 0.20; + return new MimirBioacousticClockHypothesis( + offset, + nominalSampleRate, + 1, + anchorCoverage, + 0.0, + singleAnchorConfidence, + singleAnchorConfidence, + anchors.ToArray()); + } + + var totalWeight = anchors.Sum(anchor => Math.Max(1.0e-6, anchor.Confidence)); + var meanTimeline = anchors.Sum(anchor => anchor.TimelineSeconds * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; + var meanSample = anchors.Sum(anchor => anchor.SampleOffset * Math.Max(1.0e-6, anchor.Confidence)) / totalWeight; + var covariance = 0.0; + var variance = 0.0; + foreach (var anchor in anchors) + { + var weight = Math.Max(1.0e-6, anchor.Confidence); + var dt = anchor.TimelineSeconds - meanTimeline; + covariance += weight * dt * (anchor.SampleOffset - meanSample); + variance += weight * dt * dt; + } + + var effectiveSampleRate = anchors.Count >= 3 && variance > 1.0e-12 ? covariance / variance : nominalSampleRate; + if (!double.IsFinite(effectiveSampleRate) || + effectiveSampleRate < nominalSampleRate * options.MinimumRateRatio || + effectiveSampleRate > nominalSampleRate * options.MaximumRateRatio) + { + effectiveSampleRate = nominalSampleRate; + } + + var sourceOffset = meanSample - effectiveSampleRate * meanTimeline; + var weightedResidual = anchors.Sum(anchor => + { + var predicted = sourceOffset + anchor.TimelineSeconds * effectiveSampleRate; + return Math.Abs(anchor.SampleOffset - predicted) * Math.Max(1.0e-6, anchor.Confidence); + }) / totalWeight; + var residualConfidence = 1.0 / (1.0 + weightedResidual / Math.Max(1.0, nominalSampleRate * 0.001)); + var countConfidence = Math.Clamp(anchors.Count / options.FullAnchorConfidenceCount, 0.0, 1.0); + var anchorConfidence = Math.Clamp(anchors.Average(anchor => anchor.Confidence), 0.0, 1.0); + var confidence = residualConfidence * 0.35 + countConfidence * 0.45 + anchorConfidence * 0.20; + var score = confidence + Math.Min(anchors.Count, 8) * 0.15 - weightedResidual / Math.Max(1.0, nominalSampleRate * 0.010); + return new MimirBioacousticClockHypothesis( + sourceOffset, + effectiveSampleRate, + anchors.Count, + anchorCoverage, + weightedResidual, + confidence, + score, + anchors.ToArray()); + } +} diff --git a/src/Mimir.Runtime/Synchronization/MimirBioacousticDecoderConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirBioacousticDecoderConfiguration.cs new file mode 100644 index 0000000..f26e1b2 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirBioacousticDecoderConfiguration.cs @@ -0,0 +1,129 @@ +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirCepstralDegradationProfile( + string Id, + double WarpFrames, + double WarpCoefficients, + int BlurPasses); + +public sealed record MimirBioacousticDecoderConfiguration( + string Id, + string Description, + int FftSize, + int HopSize, + int MelBins, + int CepstralCoefficients, + double MinFrequencyHz, + double MaxFrequencyHz, + int ProjectionTableCount, + int ProjectionHashBits, + int NearHashRadius, + double DenseStepSeconds, + double ProposalBudgetMultiplier, + IReadOnlyList TemplateAugmentations) +{ + public static MimirBioacousticDecoderConfiguration BaselineMfccIndex { get; } = new( + "baseline-mfcc-index", + "Balanced indexed MFCC/log-mel receiver shape from the current bioacoustic smoke path.", + FftSize: 1024, + HopSize: 256, + MelBins: 40, + CepstralCoefficients: 14, + MinFrequencyHz: 180.0, + MaxFrequencyHz: 15_000.0, + ProjectionTableCount: 4, + ProjectionHashBits: 14, + NearHashRadius: 4, + DenseStepSeconds: 0.040, + ProposalBudgetMultiplier: 8.0, + TemplateAugmentations: + [ + CepstralDegradationProfiles.Clean, + CepstralDegradationProfiles.Blur, + CepstralDegradationProfiles.WarpLight, + CepstralDegradationProfiles.WarpBlur + ]); + + public static MimirBioacousticDecoderConfiguration CompactFastIndex { get; } = new( + "compact-fast-index", + "Smaller feature and projection surface for clean-throughput pressure.", + FftSize: 1024, + HopSize: 256, + MelBins: 32, + CepstralCoefficients: 10, + MinFrequencyHz: 300.0, + MaxFrequencyHz: 14_500.0, + ProjectionTableCount: 3, + ProjectionHashBits: 12, + NearHashRadius: 3, + DenseStepSeconds: 0.040, + ProposalBudgetMultiplier: 6.0, + TemplateAugmentations: + [ + CepstralDegradationProfiles.Clean, + CepstralDegradationProfiles.Blur, + CepstralDegradationProfiles.WarpBlur + ]); + + public static MimirBioacousticDecoderConfiguration RobustWideIndex { get; } = new( + "robust-wide-index", + "Wider cepstral and augmentation surface for path-damaged acoustic evidence.", + FftSize: 1024, + HopSize: 192, + MelBins: 48, + CepstralCoefficients: 18, + MinFrequencyHz: 180.0, + MaxFrequencyHz: 15_500.0, + ProjectionTableCount: 5, + ProjectionHashBits: 15, + NearHashRadius: 4, + DenseStepSeconds: 0.030, + ProposalBudgetMultiplier: 9.0, + TemplateAugmentations: + [ + CepstralDegradationProfiles.Clean, + CepstralDegradationProfiles.Blur, + CepstralDegradationProfiles.WarpLight, + CepstralDegradationProfiles.WarpBlur, + CepstralDegradationProfiles.WarpHeavy + ]); + + public static MimirBioacousticDecoderConfiguration HighbandRoomIndex { get; } = new( + "highband-room-index", + "Upper-band-biased receiver to avoid low room junk and test monitor/mic highband survival.", + FftSize: 1024, + HopSize: 256, + MelBins: 40, + CepstralCoefficients: 14, + MinFrequencyHz: 1_200.0, + MaxFrequencyHz: 16_000.0, + ProjectionTableCount: 4, + ProjectionHashBits: 14, + NearHashRadius: 4, + DenseStepSeconds: 0.035, + ProposalBudgetMultiplier: 8.0, + TemplateAugmentations: + [ + CepstralDegradationProfiles.Clean, + CepstralDegradationProfiles.Blur, + CepstralDegradationProfiles.WarpLight, + CepstralDegradationProfiles.WarpBlur + ]); + + public static IReadOnlyList BuiltInProfiles { get; } = + [ + BaselineMfccIndex, + CompactFastIndex, + RobustWideIndex, + HighbandRoomIndex + ]; +} + +public static class CepstralDegradationProfiles +{ + public static MimirCepstralDegradationProfile Clean { get; } = new("template-clean", 0.0, 0.0, 0); + public static MimirCepstralDegradationProfile Blur { get; } = new("template-blur", 0.0, 0.0, 1); + public static MimirCepstralDegradationProfile WarpLight { get; } = new("template-warp-light", 0.75, 1.25, 0); + public static MimirCepstralDegradationProfile WarpBlur { get; } = new("template-warp-blur", 0.75, 1.25, 1); + public static MimirCepstralDegradationProfile WarpHeavy { get; } = new("template-warp-heavy", 1.25, 1.75, 1); +} diff --git a/src/Mimir.Runtime/Synchronization/MimirBioacousticLanguageConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirBioacousticLanguageConfiguration.cs new file mode 100644 index 0000000..6e71814 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirBioacousticLanguageConfiguration.cs @@ -0,0 +1,111 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirBioacousticVocabularyShape +{ + DirectWordIndex, + SpeakerSplitWordIndex, + FormantBirdcall, + RoomLearnedMotifBank, + DeviceSpecificWitness +} + +public enum MimirBioacousticEmissionDuty +{ + SilentPassive, + LowGainContinuousWatermark, + ConfidenceGatedHybrid, + CalibrationSweep, + BenchStress +} + +public sealed record MimirBioacousticLanguageConfiguration( + string Id, + string Description, + MimirBioacousticVocabularyShape VocabularyShape, + MimirBioacousticEmissionDuty EmissionDuty, + int WordCount, + int SpeakerCount, + int SyllablesPerWord, + double SegmentSeconds, + double MinFrequencyHz, + double MaxFrequencyHz, + double TargetRms, + double MaxPeak, + bool SpeakerSpecificWords, + bool SupportsStandaloneReceiver, + string[] DecoderProfiles, + string Notes); + +public static class MimirBioacousticLanguageConfigurations +{ + public static MimirBioacousticLanguageConfiguration RuntimeBirdcallWatermark { get; } = new( + "runtime-birdcall-watermark", + "Current active timing witness: low-gain, self-identifying, speaker-split birdcall words.", + MimirBioacousticVocabularyShape.FormantBirdcall, + MimirBioacousticEmissionDuty.LowGainContinuousWatermark, + MimirBioacousticTimeline.WordCount, + MimirBioacousticTimeline.SpeakerCount, + SyllablesPerWord: 4, + MimirBioacousticTimeline.SegmentSeconds, + MinFrequencyHz: 180.0, + MaxFrequencyHz: 15_000.0, + TargetRms: 0.010, + MaxPeak: 0.060, + SpeakerSpecificWords: true, + SupportsStandaloneReceiver: true, + DecoderProfiles: MimirBioacousticDecoderConfiguration.BuiltInProfiles.Select(profile => profile.Id).ToArray(), + "The receiver must recover source time from codebook and schedule alone."); + + public static MimirBioacousticLanguageConfiguration PassiveOnly { get; } = RuntimeBirdcallWatermark with + { + Id = "passive-only", + Description = "No active emission; program audio is the only timing witness until confidence falls.", + EmissionDuty = MimirBioacousticEmissionDuty.SilentPassive, + TargetRms = 0.0, + MaxPeak = 0.0, + DecoderProfiles = [], + Notes = "Useful for checking that the watermark is not hiding passive timing regressions." + }; + + public static MimirBioacousticLanguageConfiguration HybridFallback { get; } = RuntimeBirdcallWatermark with + { + Id = "hybrid-fallback", + Description = "Passive by default, then low-gain birdcall emission when confidence is weak.", + EmissionDuty = MimirBioacousticEmissionDuty.ConfidenceGatedHybrid, + TargetRms = 0.006, + MaxPeak = 0.040, + Notes = "This is the live comfort target: keep the room quiet unless the timing kernel needs active evidence." + }; + + public static MimirBioacousticLanguageConfiguration CalibrationLoud { get; } = RuntimeBirdcallWatermark with + { + Id = "calibration-loud", + Description = "Known louder emission for per-output/mic response, confusion, delay, and group-delay learning.", + EmissionDuty = MimirBioacousticEmissionDuty.CalibrationSweep, + TargetRms = 0.040, + MaxPeak = 0.180, + Notes = "Not a stream mode. This exists to measure meatspace paths without guessing." + }; + + public static MimirBioacousticLanguageConfiguration PhoneWitness { get; } = RuntimeBirdcallWatermark with + { + Id = "phone-witness", + Description = "Reduced-band word set for consumer phone microphones and lossy room/network paths.", + VocabularyShape = MimirBioacousticVocabularyShape.DeviceSpecificWitness, + MinFrequencyHz = 600.0, + MaxFrequencyHz = 9_500.0, + TargetRms = 0.012, + MaxPeak = 0.070, + DecoderProfiles = [MimirBioacousticDecoderConfiguration.CompactFastIndex.Id, MimirBioacousticDecoderConfiguration.RobustWideIndex.Id], + Notes = "A phone should self-locate on the canonical timeline with only the schedule/codebook state." + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + RuntimeBirdcallWatermark, + PassiveOnly, + HybridFallback, + CalibrationLoud, + PhoneWitness + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirCalibrationSessionPlan.cs b/src/Mimir.Runtime/Synchronization/MimirCalibrationSessionPlan.cs new file mode 100644 index 0000000..0fd9060 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirCalibrationSessionPlan.cs @@ -0,0 +1,86 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirCalibrationStimulusKind +{ + BioacousticMotifSong, + ChirpBinReference, + PassiveProgramAudio, + SilenceFloor, + ImpulseResponse +} + +public sealed record MimirCalibrationStimulus( + string Id, + MimirCalibrationStimulusKind Kind, + double DurationSeconds, + double Gain, + string Purpose); + +public sealed record MimirCalibrationSessionPlan( + string Id, + string Description, + IReadOnlyList Stimuli, + IReadOnlyList DecoderPanel, + IReadOnlyList FieldModels, + string[] RequiredArtifacts) +{ + public double TotalDurationSeconds => Stimuli.Sum(stimulus => stimulus.DurationSeconds); +} + +public static class MimirCalibrationSessionPlans +{ + public static MimirCalibrationSessionPlan QuickSynthetic { get; } = new( + "quick-synthetic-bioacoustic", + "Short synthetic panel for CI/smoke: proves identity, clock fit, and actuator handoff without devices.", + [ + new("bioacoustic-short", MimirCalibrationStimulusKind.BioacousticMotifSong, 0.75, 1.0, "Decode word identity and global clock under synthetic degradation."), + new("actuator-delay-proof", MimirCalibrationStimulusKind.BioacousticMotifSong, 2.0, 1.0, "Measure delay, apply correction, and remeasure residual.") + ], + [ + MimirBioacousticDecoderConfiguration.BaselineMfccIndex, + MimirBioacousticDecoderConfiguration.CompactFastIndex, + MimirBioacousticDecoderConfiguration.HighbandRoomIndex + ], + [MimirAudioFieldModel.AlignedStems], + ["bioacoustic-training.cc", "run-summary.json", "source-pre-warp.wav"]); + + public static MimirCalibrationSessionPlan ScarlettLoopback { get; } = new( + "scarlett-loopback-authority", + "Electrical loopback proof for emitted timeline and interface clock behavior.", + [ + new("silence-floor", MimirCalibrationStimulusKind.SilenceFloor, 2.0, 0.0, "Measure noise floor and stale loopback behavior."), + new("bioacoustic-loopback", MimirCalibrationStimulusKind.BioacousticMotifSong, 8.0, 0.06, "Recover canonical anchors through Scarlett loopback."), + new("chirp-bin-reference", MimirCalibrationStimulusKind.ChirpBinReference, 6.0, 0.04, "Preserve chirp-bin response/confusion reference evidence.") + ], + [ + MimirBioacousticDecoderConfiguration.BaselineMfccIndex, + MimirBioacousticDecoderConfiguration.CompactFastIndex + ], + [MimirAudioFieldModel.AlignedStems], + ["asio-capture.f32", "bioacoustic-training.cc", "path-state.cc"]); + + public static MimirCalibrationSessionPlan MeatspaceRoom { get; } = new( + "meatspace-room-paths", + "Physical monitor/mic calibration: learn usable bands, path delay, group-delay pressure, and anchor density.", + [ + new("silence-floor", MimirCalibrationStimulusKind.SilenceFloor, 3.0, 0.0, "Measure room and device noise before active evidence."), + new("bioacoustic-room-low", MimirCalibrationStimulusKind.BioacousticMotifSong, 12.0, 0.03, "Low-gain continuous motif language for unobtrusive decode."), + new("bioacoustic-room-raised", MimirCalibrationStimulusKind.BioacousticMotifSong, 8.0, 0.08, "Raised-gain pass for weak microphone paths."), + new("chirp-bin-reference", MimirCalibrationStimulusKind.ChirpBinReference, 8.0, 0.05, "Reference response matrix and group-delay pressure."), + new("passive-music", MimirCalibrationStimulusKind.PassiveProgramAudio, 20.0, 1.0, "Compare passive program-audio drift evidence against active anchors.") + ], + MimirBioacousticDecoderConfiguration.BuiltInProfiles, + [ + MimirAudioFieldModel.AlignedStems, + MimirAudioFieldModel.SourceBasedSpatialBus, + MimirAudioFieldModel.HybridEvidenceField + ], + ["room-manifest.json", "bioacoustic-training.cc", "path-state.cc", "actuator-state.cc"]); + + public static IReadOnlyList BuiltIn { get; } = + [ + QuickSynthetic, + ScarlettLoopback, + MeatspaceRoom + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirCameraIngestConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirCameraIngestConfiguration.cs new file mode 100644 index 0000000..ccfb328 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirCameraIngestConfiguration.cs @@ -0,0 +1,74 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirCameraIngestStrategy +{ + JsonCadenceProbe, + ManagedNativeWrapper, + NativeSpscFrameRing, + SharedGpuTexture +} + +public sealed record MimirCameraIngestConfiguration( + string Id, + string Description, + MimirCameraIngestStrategy Strategy, + string Owner, + bool CarriesPixels, + bool HotPathCandidate, + string[] DeviceProfiles, + string[] RequiredProofs, + string Notes); + +public static class MimirCameraIngestConfigurations +{ + public static MimirCameraIngestConfiguration CadenceProbe { get; } = new( + "json-cadence-probe", + "Metadata-only diagnostic bridge for checking driver cadence.", + MimirCameraIngestStrategy.JsonCadenceProbe, + "native probes + Mimir diagnostics", + CarriesPixels: false, + HotPathCandidate: false, + DeviceProfiles: MimirNativeCaptureConfigurations.LocalSixCameraProfiles.Select(profile => profile.Id).ToArray(), + RequiredProofs: ["source cadence enters rolling buffers"], + "Keep this diagnostic-only; it is not a pixel transport."); + + public static MimirCameraIngestConfiguration ManagedWrapper { get; } = new( + "managed-native-wrapper", + "First real driver integration shape through IMimirVideoCaptureDriver.", + MimirCameraIngestStrategy.ManagedNativeWrapper, + "Mimir.Runtime + native worker", + CarriesPixels: true, + HotPathCandidate: true, + DeviceProfiles: ["leap-left-ir", "leap-right-ir"], + RequiredProofs: ["one-camera sustained cadence", "bounded allocation", "device timestamp preserved"], + "Good first cut for Leap because observability matters while the ABI is still moving."); + + public static MimirCameraIngestConfiguration NativeSpscRing { get; } = new( + "native-spsc-frame-ring", + "Production local-camera ingest: native producer rings with typed payload handles.", + MimirCameraIngestStrategy.NativeSpscFrameRing, + "native capture + native/reservoir", + CarriesPixels: true, + HotPathCandidate: true, + DeviceProfiles: MimirNativeCaptureConfigurations.LocalSixCameraProfiles.Select(profile => profile.Id).ToArray(), + RequiredProofs: ["all six cameras sustained", "no stdout transport", "bounded memory", "consumer lag telemetry"], + "This is the six-camera throughput answer."); + + public static MimirCameraIngestConfiguration SharedTexture { get; } = NativeSpscRing with + { + Id = "shared-gpu-texture", + Description = "Final copy-avoidance target: native capture provides importable GPU handles for Fensalir.", + Strategy = MimirCameraIngestStrategy.SharedGpuTexture, + Owner = "native capture + Fensalir", + RequiredProofs = ["Fensalir imports frames", "rendered pixels match source cadence", "handle lifetime is explicit"], + Notes = "Promote only when driver/API friction is paid down; Fensalir owns GPU lifetime." + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + CadenceProbe, + ManagedWrapper, + NativeSpscRing, + SharedTexture + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs new file mode 100644 index 0000000..5b63587 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirComputeOffloadConfiguration.cs @@ -0,0 +1,75 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirWorkloadKind +{ + BioacousticDecode, + PassiveGccPhat, + ClockHypothesis, + PathCalibration, + ActuatorControl, + FaustDsp, + CameraCapture, + VisualFeatureExtraction, + VisualFusion, + ObsPublication +} + +public sealed record MimirWorkloadPlacement( + MimirWorkloadKind Workload, + MimirComputeBackend PreferredBackend, + string Owner, + bool CanRunRemote, + bool RequiresRealtime, + string Invariant); + +public sealed record MimirComputeOffloadConfiguration( + string Id, + string Description, + IReadOnlyList Placements); + +public static class MimirComputeOffloadConfigurations +{ + public static MimirComputeOffloadConfiguration StarfireLocalHeavy { get; } = new( + "starfire-local-heavy", + "Keep canonical timing, DSP, and fusion local to Starfire; remote nodes send typed observations.", + [ + new(MimirWorkloadKind.BioacousticDecode, MimirComputeBackend.ManagedScalar, "local receiver or remote witness", CanRunRemote: true, RequiresRealtime: true, "Decode emits observations, not global truth."), + new(MimirWorkloadKind.PassiveGccPhat, MimirComputeBackend.ManagedScalar, "Starfire", CanRunRemote: false, RequiresRealtime: true, "Loopback remains local timing authority."), + new(MimirWorkloadKind.ClockHypothesis, MimirComputeBackend.ManagedScalar, "Starfire", CanRunRemote: false, RequiresRealtime: true, "Canonical clock fit is centralized."), + new(MimirWorkloadKind.PathCalibration, MimirComputeBackend.ManagedScalar, "Starfire", CanRunRemote: true, RequiresRealtime: false, "Calibration receipts are replayable typed evidence."), + new(MimirWorkloadKind.ActuatorControl, MimirComputeBackend.FaustNativeDsp, "Faust/native DSP", CanRunRemote: false, RequiresRealtime: true, "DSP moves samples; runtime only estimates state."), + new(MimirWorkloadKind.FaustDsp, MimirComputeBackend.FaustNativeDsp, "Faust/native DSP", CanRunRemote: false, RequiresRealtime: true, "Alignment, separation, and spatial bed are audio-rate DSP."), + new(MimirWorkloadKind.CameraCapture, MimirComputeBackend.NativeSimd, "native capture workers", CanRunRemote: true, RequiresRealtime: true, "Drivers own device reads and timestamps."), + new(MimirWorkloadKind.VisualFeatureExtraction, MimirComputeBackend.D3D12Compute, "Fensalir", CanRunRemote: false, RequiresRealtime: true, "GPU owns dense feature work."), + new(MimirWorkloadKind.VisualFusion, MimirComputeBackend.D3D12Compute, "Fensalir", CanRunRemote: false, RequiresRealtime: true, "Fensalir owns temporal evidence field."), + new(MimirWorkloadKind.ObsPublication, MimirComputeBackend.D3D12Compute, "OBS/Fensalir output", CanRunRemote: false, RequiresRealtime: true, "OBS receives program surfaces, not sync authority.") + ]); + + public static MimirComputeOffloadConfiguration DistributedWitnesses { get; } = new( + "distributed-witnesses", + "Phones, Raven, and microcontrollers decode locally and return typed observations over CultMesh.", + [ + new(MimirWorkloadKind.BioacousticDecode, MimirComputeBackend.RemoteCultMeshWorker, "remote witness", CanRunRemote: true, RequiresRealtime: true, "Remote nodes self-locate but do not own canonical time."), + new(MimirWorkloadKind.ClockHypothesis, MimirComputeBackend.ManagedScalar, "Starfire", CanRunRemote: false, RequiresRealtime: true, "Starfire arbitrates clock/path hypotheses."), + new(MimirWorkloadKind.PathCalibration, MimirComputeBackend.RemoteCultMeshWorker, "remote witness", CanRunRemote: true, RequiresRealtime: false, "Remote path evidence comes home as receipts."), + new(MimirWorkloadKind.ActuatorControl, MimirComputeBackend.FaustNativeDsp, "Starfire Faust/native DSP", CanRunRemote: false, RequiresRealtime: true, "Only the program graph moves program samples."), + new(MimirWorkloadKind.CameraCapture, MimirComputeBackend.NativeSimd, "local or remote capture worker", CanRunRemote: true, RequiresRealtime: true, "Each machine reads its own devices directly.") + ]); + + public static MimirComputeOffloadConfiguration CalibrationSweep { get; } = new( + "calibration-sweep", + "Offline-heavy calibration sweep using native/GPU batches where useful, preserving all scores.", + [ + new(MimirWorkloadKind.BioacousticDecode, MimirComputeBackend.NativeSimd, "calibration worker", CanRunRemote: true, RequiresRealtime: false, "Large candidate batches can use native SIMD."), + new(MimirWorkloadKind.PathCalibration, MimirComputeBackend.D3D12Compute, "calibration worker", CanRunRemote: true, RequiresRealtime: false, "GPU is useful when batches are large enough."), + new(MimirWorkloadKind.ClockHypothesis, MimirComputeBackend.ManagedScalar, "calibration worker", CanRunRemote: true, RequiresRealtime: false, "Clock solve stays branchy and inspectable."), + new(MimirWorkloadKind.VisualFeatureExtraction, MimirComputeBackend.D3D12Compute, "Fensalir/calibration worker", CanRunRemote: true, RequiresRealtime: false, "Offline feature batches may run away from the render tick.") + ]); + + public static IReadOnlyList BuiltIn { get; } = + [ + StarfireLocalHeavy, + DistributedWitnesses, + CalibrationSweep + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs new file mode 100644 index 0000000..2298b7b --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirCultMeshContracts.cs @@ -0,0 +1,200 @@ +using GameCult.Caching; +using MessagePack; + +namespace Mimir.Runtime.Synchronization; + +[CultDocument("mimir.bioacoustic_codebook_state", "mimir.bioacoustic_codebook_state.v1")] +[MessagePackObject] +public sealed record MimirBioacousticCodebookState( + [property: Key(0)] + [property: CultName] + string CodebookId, + [property: Key(1)] string CreatedAtUtc, + [property: Key(2)] int WordCount, + [property: Key(3)] int SpeakerCount, + [property: Key(4)] double SegmentSeconds, + [property: Key(5)] string ScheduleId, + [property: Key(6)] MimirBioacousticMotifSnapshot[] Motifs); + +[MessagePackObject] +public sealed record MimirBioacousticMotifSnapshot( + [property: Key(0)] int SymbolId, + [property: Key(1)] string Speaker, + [property: Key(2)] MimirBioacousticSyllableSnapshot[] Syllables); + +[MessagePackObject] +public sealed record MimirBioacousticSyllableSnapshot( + [property: Key(0)] double StartSeconds, + [property: Key(1)] double DurationSeconds, + [property: Key(2)] double StartHz, + [property: Key(3)] double EndHz, + [property: Key(4)] double Weight); + +[CultDocument("mimir.bioacoustic_decoder_state", "mimir.bioacoustic_decoder_state.v1")] +[MessagePackObject] +public sealed record MimirBioacousticDecoderState( + [property: Key(0)] + [property: CultName] + string DecoderId, + [property: Key(1)] string CreatedAtUtc, + [property: Key(2)] string CodebookId, + [property: Key(3)] MimirBioacousticDecoderConfigurationSnapshot Configuration); + +[MessagePackObject] +public sealed record MimirBioacousticDecoderConfigurationSnapshot( + [property: Key(0)] string Id, + [property: Key(1)] string Description, + [property: Key(2)] int FftSize, + [property: Key(3)] int HopSize, + [property: Key(4)] int MelBins, + [property: Key(5)] int CepstralCoefficients, + [property: Key(6)] double MinFrequencyHz, + [property: Key(7)] double MaxFrequencyHz, + [property: Key(8)] int ProjectionTableCount, + [property: Key(9)] int ProjectionHashBits, + [property: Key(10)] int NearHashRadius, + [property: Key(11)] double DenseStepSeconds, + [property: Key(12)] double ProposalBudgetMultiplier, + [property: Key(13)] MimirCepstralDegradationSnapshot[] TemplateAugmentations); + +[MessagePackObject] +public sealed record MimirCepstralDegradationSnapshot( + [property: Key(0)] string Id, + [property: Key(1)] double WarpFrames, + [property: Key(2)] double WarpCoefficients, + [property: Key(3)] int BlurPasses); + +[CultDocument("mimir.acoustic_path_state", "mimir.acoustic_path_state.v1")] +[MessagePackObject] +public sealed record MimirAcousticPathState( + [property: Key(0)] + [property: CultName] + string PathId, + [property: Key(1)] string UpdatedAtUtc, + [property: Key(2)] string OutputSourceId, + [property: Key(3)] string MicSourceId, + [property: Key(4)] int SampleRate, + [property: Key(5)] double DelaySamples, + [property: Key(6)] double SamplingRateOffsetPpm, + [property: Key(7)] double Confidence, + [property: Key(8)] MimirBandResponseSnapshot[] BandResponses, + [property: Key(9)] string EvidenceKind); + +[MessagePackObject] +public sealed record MimirBandResponseSnapshot( + [property: Key(0)] double CenterHz, + [property: Key(1)] double Energy); + +[CultDocument("mimir.actuator_state", "mimir.actuator_state.v1")] +[MessagePackObject] +public sealed record MimirActuatorStateDocument( + [property: Key(0)] + [property: CultName] + string ActuatorId, + [property: Key(1)] string UpdatedAtUtc, + [property: Key(2)] string SourceId, + [property: Key(3)] string ProfileId, + [property: Key(4)] double TargetDelaySamples, + [property: Key(5)] double ResampleRatio, + [property: Key(6)] double Confidence, + [property: Key(7)] MimirFaustControlSnapshot[] FaustControls); + +[MessagePackObject] +public sealed record MimirFaustControlSnapshot( + [property: Key(0)] string Path, + [property: Key(1)] float Value); + +public static class MimirCultMeshContractFactory +{ + public static MimirBioacousticCodebookState CreateCodebookState( + string codebookId, + string scheduleId, + MimirBioacousticTimeline timeline, + DateTimeOffset? createdAt = null) => + new( + codebookId, + (createdAt ?? DateTimeOffset.UtcNow).ToString("O"), + MimirBioacousticTimeline.WordCount, + MimirBioacousticTimeline.SpeakerCount, + MimirBioacousticTimeline.SegmentSeconds, + scheduleId, + timeline.Codebook.Select(motif => new MimirBioacousticMotifSnapshot( + motif.SymbolId, + (motif.SymbolId & 1) == 0 ? "left" : "right", + motif.Syllables + .Select(syllable => new MimirBioacousticSyllableSnapshot( + syllable.StartSeconds, + syllable.DurationSeconds, + syllable.StartHz, + syllable.EndHz, + syllable.Weight)) + .ToArray())) + .ToArray()); + + public static MimirBioacousticDecoderState CreateDecoderState( + string decoderId, + string codebookId, + MimirBioacousticDecoderConfiguration configuration, + DateTimeOffset? createdAt = null) => + new( + decoderId, + (createdAt ?? DateTimeOffset.UtcNow).ToString("O"), + codebookId, + Snapshot(configuration)); + + public static MimirAcousticPathState CreatePathState(MimirAudioSynchronizationState state, string outputSourceId, string evidenceKind) => + new( + $"{outputSourceId}->{state.SourceId}", + DateTimeOffset.UtcNow.ToString("O"), + outputSourceId, + state.SourceId, + state.SampleRate, + state.SmoothedDelaySamples, + state.SamplingRateOffsetPpm, + state.Confidence, + state.BandResponses + .Select(response => new MimirBandResponseSnapshot(response.CenterHz, response.Energy)) + .ToArray(), + evidenceKind); + + public static MimirActuatorStateDocument CreateActuatorState( + string actuatorId, + string profileId, + MimirActuatorCommand command, + DateTimeOffset? updatedAt = null) => + new( + actuatorId, + (updatedAt ?? DateTimeOffset.UtcNow).ToString("O"), + command.SourceId, + profileId, + command.TargetDelaySamples, + command.ResampleRatio, + command.Confidence, + command.FaustControls + .OrderBy(pair => pair.Key, StringComparer.Ordinal) + .Select(pair => new MimirFaustControlSnapshot(pair.Key, pair.Value)) + .ToArray()); + + private static MimirBioacousticDecoderConfigurationSnapshot Snapshot(MimirBioacousticDecoderConfiguration configuration) => + new( + configuration.Id, + configuration.Description, + configuration.FftSize, + configuration.HopSize, + configuration.MelBins, + configuration.CepstralCoefficients, + configuration.MinFrequencyHz, + configuration.MaxFrequencyHz, + configuration.ProjectionTableCount, + configuration.ProjectionHashBits, + configuration.NearHashRadius, + configuration.DenseStepSeconds, + configuration.ProposalBudgetMultiplier, + configuration.TemplateAugmentations + .Select(profile => new MimirCepstralDegradationSnapshot( + profile.Id, + profile.WarpFrames, + profile.WarpCoefficients, + profile.BlurPasses)) + .ToArray()); +} diff --git a/src/Mimir.Runtime/Synchronization/MimirDistributedWitnessConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirDistributedWitnessConfiguration.cs new file mode 100644 index 0000000..51c9dce --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirDistributedWitnessConfiguration.cs @@ -0,0 +1,83 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirDistributedWitnessKind +{ + RavenScarlett, + PhoneMicrophone, + MicrocontrollerListener, + RemoteCameraRig, + BrowserDiagnostics +} + +public sealed record MimirDistributedWitnessConfiguration( + string Id, + string Description, + MimirDistributedWitnessKind Kind, + string NodeRole, + string[] Inputs, + string[] RequiredStateDocuments, + string[] EmittedStateDocuments, + bool MayStreamRawMedia, + bool MayOwnCanonicalClock, + double ExpectedNetworkLatencyMilliseconds, + string Notes); + +public static class MimirDistributedWitnessConfigurations +{ + public static MimirDistributedWitnessConfiguration Raven { get; } = new( + "raven-scarlett-witness", + "Co-streamer/game machine with Scarlett loopback/mic evidence and network latency.", + MimirDistributedWitnessKind.RavenScarlett, + "remote-witness", + Inputs: ["raven-scarlett-input-1", "raven-scarlett-input-2", "raven-loopback", "raven-camera"], + RequiredStateDocuments: ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state"], + EmittedStateDocuments: ["mimir.acoustic_path_state", "mimir.bioacoustic_decoder_state"], + MayStreamRawMedia: true, + MayOwnCanonicalClock: false, + ExpectedNetworkLatencyMilliseconds: 2.0, + "Raven decodes locally and ships compact timing/path truth. Starfire still owns heavy field work."); + + public static MimirDistributedWitnessConfiguration Phone { get; } = new( + "phone-mic-witness", + "Harsh receiver target: OS resampling, AGC, lossy mic path, and mobile CPU.", + MimirDistributedWitnessKind.PhoneMicrophone, + "remote-witness", + Inputs: ["phone-mic"], + RequiredStateDocuments: ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state"], + EmittedStateDocuments: ["mimir.acoustic_path_state"], + MayStreamRawMedia: false, + MayOwnCanonicalClock: false, + ExpectedNetworkLatencyMilliseconds: 20.0, + "If the phone can self-locate from codebook and schedule, the language is earning its keep."); + + public static MimirDistributedWitnessConfiguration Microcontroller { get; } = Phone with + { + Id = "microcontroller-listener", + Description = "Tiny low-power listener for room timing beacons and coarse occupancy evidence.", + Kind = MimirDistributedWitnessKind.MicrocontrollerListener, + Inputs = ["single-channel-adc"], + RequiredStateDocuments = ["mimir.bioacoustic_codebook_state"], + EmittedStateDocuments = ["mimir.acoustic_path_state"], + ExpectedNetworkLatencyMilliseconds = 50.0, + Notes = "Use compact profiles and narrow output: anchors, confidence, health counters." + }; + + public static MimirDistributedWitnessConfiguration RemoteCameraRig { get; } = Raven with + { + Id = "remote-camera-rig", + Description = "Networked camera/audio producer with local timing decode.", + Kind = MimirDistributedWitnessKind.RemoteCameraRig, + Inputs = ["remote-video", "remote-audio", "remote-loopback"], + MayStreamRawMedia = true, + ExpectedNetworkLatencyMilliseconds = 8.0, + Notes = "Raw media may arrive late; decoded anchors tell Mimir where it belongs in the window." + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + Raven, + Phone, + Microcontroller, + RemoteCameraRig + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs b/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs new file mode 100644 index 0000000..4902a2c --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirFensalirFieldLowering.cs @@ -0,0 +1,118 @@ +using System.Numerics; +using Aquarium.Engine.Render; + +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirFensalirLoweringOptions( + double AccumulationWindowSeconds = 5.0, + double PresentationDelaySeconds = 5.0, + double DefaultTimingUncertaintyMicroseconds = 1000.0); + +public sealed class MimirFensalirFieldLowering(MimirFensalirLoweringOptions? options = null) +{ + private readonly MimirFensalirLoweringOptions options = options ?? new(); + + public AquariumGpuSensorFrame BuildGpuSensorFrame(IEnumerable buffers) + { + var textures = new List(); + var cameras = new List(); + foreach (var buffer in buffers) + { + if (buffer.Descriptor.Kind != MimirStreamKind.Video || buffer.Latest?.VideoFrame is not { } frame) + { + continue; + } + + var firstTexture = textures.Count; + if (frame.NativeHandle != 0) + { + textures.Add(new AquariumExternalGpuTexture( + default, + new IntPtr(unchecked((long)frame.NativeHandle)), + frame.Width, + frame.Height, + ToFensalirPixelFormat(frame.PixelFormat), + frame.DeviceTimestampNs, + SharedHandleName: frame.NativeHandleKind)); + } + + cameras.Add(new AquariumGpuSensorCamera( + buffer.Descriptor.SourceId, + ToFensalirSensorKind(buffer.Descriptor.SourceId, frame.PixelFormat), + Matrix4x4.Identity, + Matrix4x4.Identity, + Vector4.Zero, + Vector4.Zero, + Vector4.Zero, + frame.Width, + frame.Height, + firstTexture, + textures.Count - firstTexture, + frame.DeviceTimestampNs)); + } + + return new AquariumGpuSensorFrame + { + Cameras = cameras, + ExternalTextures = textures, + AccumulationWindowSeconds = (float)options.AccumulationWindowSeconds, + PresentationDelaySeconds = (float)options.PresentationDelaySeconds + }; + } + + public AquariumAcousticFieldFrame BuildAcousticFieldFrame(IEnumerable states) + { + var orderedStates = states + .OrderByDescending(state => state.Confidence) + .ThenBy(state => state.SourceId, StringComparer.Ordinal) + .ToArray(); + var constraints = orderedStates + .Where(state => state.Confidence > 0.0) + .Select(state => new AquariumAcousticConstraint( + $"{state.ReferenceSourceId}->{state.SourceId}", + AquariumAcousticConstraintKind.SpeakerProbe, + Vector3.Zero, + Vector3.Zero, + RadiusMeters: 0.10f, + Confidence: (float)Math.Clamp(state.Confidence, 0.0, 1.0), + TimestampNs: state.UpdatedAtNs)) + .ToArray(); + var oracle = orderedStates.FirstOrDefault(); + return new AquariumAcousticFieldFrame + { + Constraints = constraints, + TimingOracleNs = oracle?.UpdatedAtNs ?? 0, + TimingConfidence = (float)Math.Clamp(oracle?.Confidence ?? 0.0, 0.0, 1.0), + TimingUncertaintyMicroseconds = (float)(oracle == null + ? options.DefaultTimingUncertaintyMicroseconds + : Math.Max(0.1, (1.0 - Math.Clamp(oracle.Confidence, 0.0, 1.0)) * options.DefaultTimingUncertaintyMicroseconds)), + AccumulationWindowSeconds = (float)options.AccumulationWindowSeconds, + PresentationDelaySeconds = (float)options.PresentationDelaySeconds + }; + } + + private static AquariumGpuSensorKind ToFensalirSensorKind(string sourceId, MimirVideoPixelFormat pixelFormat) + { + if (pixelFormat == MimirVideoPixelFormat.LeapStereoIr || sourceId.Contains("leap", StringComparison.OrdinalIgnoreCase)) + { + return AquariumGpuSensorKind.LeapPackedMap; + } + + if (sourceId.Contains("eye", StringComparison.OrdinalIgnoreCase)) + { + return AquariumGpuSensorKind.HighRateTracker; + } + + return AquariumGpuSensorKind.RgbCamera; + } + + private static AquariumGpuSensorPixelFormat ToFensalirPixelFormat(MimirVideoPixelFormat pixelFormat) => + pixelFormat switch + { + MimirVideoPixelFormat.Gray8 or MimirVideoPixelFormat.R8 or MimirVideoPixelFormat.Bayer8 => AquariumGpuSensorPixelFormat.R8Unorm, + MimirVideoPixelFormat.Rg8 => AquariumGpuSensorPixelFormat.Rg8Unorm, + MimirVideoPixelFormat.Bgra8 => AquariumGpuSensorPixelFormat.Bgra8Unorm, + MimirVideoPixelFormat.LeapStereoIr => AquariumGpuSensorPixelFormat.LeapPackedMap, + _ => AquariumGpuSensorPixelFormat.Unknown + }; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirMachineAssemblyPlan.cs b/src/Mimir.Runtime/Synchronization/MimirMachineAssemblyPlan.cs new file mode 100644 index 0000000..2d070e2 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirMachineAssemblyPlan.cs @@ -0,0 +1,97 @@ +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirMachineAssemblyPlan( + string Id, + string Description, + MimirPerfectMachineProfile NodeProfile, + MimirCalibrationSessionPlan Calibration, + MimirAudioFieldConfiguration AudioField, + MimirVisualFusionConfiguration VisualFusion, + MimirComputeOffloadConfiguration Compute, + string[] FirstProofCommands, + string[] PromotionGates); + +public static class MimirMachineAssemblyPlans +{ + public static MimirMachineAssemblyPlan SyntheticReceiverAndActuator { get; } = new( + "synthetic-receiver-actuator", + "Smallest coherent proof: bioacoustic identity, global clock fit, CultMesh contract shape, and actuator residual.", + MimirPerfectMachineProfiles.CalibrationBench, + MimirCalibrationSessionPlans.QuickSynthetic, + MimirAudioFieldConfigurations.AlignedStemsSixMic, + MimirVisualFusionConfigurations.CadenceProof, + MimirComputeOffloadConfigurations.CalibrationSweep, + [ + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-train --sample-rate 48000 --seconds 0.75", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test --sample-rate 48000 --delay-samples 317.375" + ], + [ + "profile smoke passes", + "training receipt persists CultCache and audio artifacts", + "actuator residual remeasures below one microsecond" + ]); + + public static MimirMachineAssemblyPlan ScarlettLoopbackAuthority { get; } = new( + "scarlett-loopback-authority", + "Electrical timing authority proof: local Scarlett loopback validates emitted timeline and DSP correction before room acoustics.", + MimirPerfectMachineProfiles.StarfireAuthority, + MimirCalibrationSessionPlans.ScarlettLoopback, + MimirAudioFieldConfigurations.AlignedStemsSixMic, + MimirVisualFusionConfigurations.CadenceProof, + MimirComputeOffloadConfigurations.StarfireLocalHeavy, + [ + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --render-bioacoustic-f32 --seconds 8", + "native\\probes\\asio_audio_cadence\\build\\Release\\asio_audio_cadence.exe --play-f32-mono artifacts\\asio\\bioacoustic-f32.raw --record-f32-interleaved artifacts\\asio\\scarlett-bioacoustic.f32 --sample-rate 192000 --seconds 8", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --analyze-asio-f32 --input artifacts\\asio\\scarlett-bioacoustic.f32 --sample-rate 192000" + ], + [ + "loopback channels decode stable canonical anchors", + "path state CultMesh document can be produced", + "actuator target can be generated from sync state" + ]); + + public static MimirMachineAssemblyPlan MeatspaceFieldCalibration { get; } = new( + "meatspace-field-calibration", + "Physical room proof: monitors, loopback, microphones, response weighting, and visual constraints start forming a field.", + MimirPerfectMachineProfiles.StarfireAuthority, + MimirCalibrationSessionPlans.MeatspaceRoom, + MimirAudioFieldConfigurations.HybridEvidenceField, + MimirVisualFusionConfigurations.AudioConstrainedField, + MimirComputeOffloadConfigurations.StarfireLocalHeavy, + [ + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-train --sample-rate 48000 --seconds 8", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --chirp-only-sync-self-test --sample-rate 192000", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke" + ], + [ + "at least one physical mic decodes stable canonical anchors", + "path response identifies reliable bands", + "field claim remains diagnostic until geometry residuals are bounded" + ]); + + public static MimirMachineAssemblyPlan DistributedCultMeshWitnesses { get; } = new( + "distributed-cultmesh-witnesses", + "Remote receivers: Raven, phones, and tiny listeners decode local evidence and sync typed state through CultMesh.", + MimirPerfectMachineProfiles.RavenRemote, + MimirCalibrationSessionPlans.MeatspaceRoom, + MimirAudioFieldConfigurations.SourceBasedSpatialBus, + MimirVisualFusionConfigurations.AudioConstrainedField, + MimirComputeOffloadConfigurations.DistributedWitnesses, + [ + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke" + ], + [ + "remote codebook and decoder state round-trip through CultMesh", + "remote path observations never claim canonical clock authority", + "Starfire can combine remote anchors with local loopback truth" + ]); + + public static IReadOnlyList BuiltIn { get; } = + [ + SyntheticReceiverAndActuator, + ScarlettLoopbackAuthority, + MeatspaceFieldCalibration, + DistributedCultMeshWitnesses + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs new file mode 100644 index 0000000..d016807 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirModuleLibrary.cs @@ -0,0 +1,120 @@ +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirModuleCatalogEntry( + string Id, + string Owner, + string Purpose, + string[] Configurations, + string[] VerificationCommands); + +public static class MimirModuleLibrary +{ + public static IReadOnlyList Entries { get; } = + [ + new( + "bioacoustic-decoder", + "Mimir.Runtime", + "Reusable indexed log-mel/MFCC receiver configurations for standalone word identity and canonical clock recovery.", + MimirBioacousticDecoderConfiguration.BuiltInProfiles.Select(profile => profile.Id).ToArray(), + [ + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-cepstral-smoke", + "dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-train" + ]), + new( + "bioacoustic-language", + "Mimir.Runtime", + "Emission duty and vocabulary profiles for passive, hybrid, continuous, calibration, and phone-witness modes.", + MimirBioacousticLanguageConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "bioacoustic-clock", + "Mimir.Runtime", + "Global clock hypothesis solver over decoded word anchors; this is the timing truth shape for remote witnesses.", + ["single-anchor-offset", "multi-anchor-rate-fit", "coverage-confidence"], + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --standalone-bioacoustic-self-test"]), + new( + "alignment-actuator", + "Mimir.Runtime + Faust", + "Control surface for fractional delay and SRO correction. Mimir estimates; Faust moves samples.", + [MimirAlignmentActuatorProfile.SixSourceFaust.Id], + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test"]), + new( + "audio-actuator-strategies", + "Mimir.Runtime + Faust", + "Actuator strategy configurations from coarse diagnostic delay through final delay-plus-ASRC drift hold.", + MimirAudioActuatorConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-actuator-self-test"]), + new( + "cultmesh-contracts", + "Mimir.Runtime", + "Typed state documents for codebooks, decoder state, acoustic path state, and actuator state.", + ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state", "mimir.actuator_state"], + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-contract-smoke"]), + new( + "path-learning", + "Mimir.Runtime", + "Calibration session policy for usable bands, confusion matrices, delay hypotheses, group delay, and codebook adaptation.", + MimirAcousticPathLearningConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "acoustic-localization", + "Mimir.Runtime + Fensalir", + "TDOA, SRP-PHAT, sparse-source, and visual-constrained localization configurations for volumetric audio constraints.", + MimirAcousticLocalizationConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "benchmark-panels", + "Mimir.Runtime", + "Named degradation and acceptance panels for decoder golf and physical-path acceptance receipts.", + MimirBenchmarkPanelConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --bioacoustic-train"]), + new( + "native-capture", + "Mimir.Runtime + native workers", + "Device-profile library for six cameras, Scarlett ASIO, Raven witness audio, and direct-driver ownership.", + MimirNativeCaptureConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "camera-ingest-strategies", + "Mimir.Runtime + native/reservoir + Fensalir", + "Ingest configurations for diagnostic metadata, managed native wrappers, native SPSC rings, and shared GPU textures.", + MimirCameraIngestConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "reservoir-strategies", + "Mimir.Runtime + native/reservoir + Fensalir", + "Managed, native, and GPU-resident reservoir configurations with explicit retention owners.", + MimirReservoirConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "distributed-witnesses", + "Mimir.Runtime + CultMesh", + "Remote mini-Mimir configurations for Raven, phones, microcontrollers, and networked camera rigs.", + MimirDistributedWitnessConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "network-transports", + "Mimir.Runtime + CultMesh + diagnostics", + "Transport policy for typed timing state, debug windows, bridge media, and experimental browser feeds.", + MimirNetworkTransportConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "authority-policy", + "Mimir.Runtime", + "Trust rules for loopback authority, Raven evidence, phone candidates, bridge diagnostics, and unknown nodes.", + MimirAuthorityPolicyConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "field-assembly", + "Mimir.Runtime", + "Composable assembly plans for synthetic proof, Scarlett loopback, meatspace calibration, and distributed CultMesh witnesses.", + MimirMachineAssemblyPlans.BuiltIn.Select(plan => plan.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]), + new( + "obs-publication", + "Fensalir + Faust + OBS", + "OBS-facing program surface and separately mixable audio stem configurations.", + MimirObsPublicationConfigurations.BuiltIn.Select(profile => profile.Id).ToArray(), + ["dotnet run --project .\\src\\Mimir.BufferSmoke\\Mimir.BufferSmoke.csproj -- --perfect-machine-profile-smoke"]) + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs new file mode 100644 index 0000000..ba54c21 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirNativeCaptureConfiguration.cs @@ -0,0 +1,151 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirNativeCaptureTransport +{ + VendorSdk, + LibUsbIsochronous, + UvcDirect, + AsioCallback, + NetworkCultMesh, + NetworkCompressedAv +} + +public enum MimirCaptureClockDomain +{ + DeviceHardware, + InterfaceAsio, + HostQpc, + NetworkSender, + CanonicalFitted +} + +public sealed record MimirNativeCaptureDeviceProfile( + string Id, + string DisplayName, + MimirStreamKind StreamKind, + MimirNativeCaptureTransport Transport, + MimirCaptureClockDomain ClockDomain, + int ChannelCount, + int PreferredWidth, + int PreferredHeight, + int PreferredSampleRate, + double PreferredFramesPerSecond, + double RollingBufferSeconds, + bool Required, + string[] OwnedOutputs, + string Notes); + +public static class MimirNativeCaptureConfigurations +{ + public const double DefaultRollingBufferSeconds = 5.0; + + public static MimirNativeCaptureDeviceProfile LeapLeftIr { get; } = new( + "leap-left-ir", + "Ultraleap stereo IR left", + MimirStreamKind.Video, + MimirNativeCaptureTransport.VendorSdk, + MimirCaptureClockDomain.DeviceHardware, + ChannelCount: 1, + PreferredWidth: 640, + PreferredHeight: 480, + PreferredSampleRate: 0, + PreferredFramesPerSecond: 120.0, + DefaultRollingBufferSeconds, + Required: true, + OwnedOutputs: ["near-field-feature-claims", "timing-camera-claims"], + "Ground-truth near-field timing camera. Keep full frame; tracking eats pixels."); + + public static MimirNativeCaptureDeviceProfile LeapRightIr { get; } = LeapLeftIr with + { + Id = "leap-right-ir", + DisplayName = "Ultraleap stereo IR right" + }; + + public static MimirNativeCaptureDeviceProfile KiyoProRgb { get; } = new( + "kiyo-pro-rgb", + "Razer Kiyo Pro RGB ground truth", + MimirStreamKind.Video, + MimirNativeCaptureTransport.UvcDirect, + MimirCaptureClockDomain.DeviceHardware, + ChannelCount: 1, + PreferredWidth: 1920, + PreferredHeight: 1080, + PreferredSampleRate: 0, + PreferredFramesPerSecond: 60.0, + DefaultRollingBufferSeconds, + Required: true, + OwnedOutputs: ["rgb-ground-truth", "skin/material-observations"], + "RGB quality anchor. Prefer direct UVC control; do not put Media Foundation in the hot loop."); + + public static MimirNativeCaptureDeviceProfile KiyoBasicRgb { get; } = KiyoProRgb with + { + Id = "kiyo-basic-rgb", + DisplayName = "Razer Kiyo RGB stereo/context", + OwnedOutputs = ["rgb-context", "stereo-rgb-claims"] + }; + + public static MimirNativeCaptureDeviceProfile Ps3EyeLeft { get; } = new( + "ps3eye-left", + "PlayStation Eye high-rate tracker left", + MimirStreamKind.Video, + MimirNativeCaptureTransport.LibUsbIsochronous, + MimirCaptureClockDomain.DeviceHardware, + ChannelCount: 1, + PreferredWidth: 640, + PreferredHeight: 480, + PreferredSampleRate: 0, + PreferredFramesPerSecond: 60.0, + DefaultRollingBufferSeconds, + Required: true, + OwnedOutputs: ["wide-feature-tracks", "motion-field-claims"], + "Tracking witness. Prefer frame rate and low latency over prettiness."); + + public static MimirNativeCaptureDeviceProfile Ps3EyeRight { get; } = Ps3EyeLeft with + { + Id = "ps3eye-right", + DisplayName = "PlayStation Eye high-rate tracker right" + }; + + public static MimirNativeCaptureDeviceProfile ScarlettStarfire { get; } = new( + "starfire-scarlett-4th-gen", + "Starfire Focusrite Scarlett ASIO", + MimirStreamKind.Audio, + MimirNativeCaptureTransport.AsioCallback, + MimirCaptureClockDomain.InterfaceAsio, + ChannelCount: 4, + PreferredWidth: 0, + PreferredHeight: 0, + PreferredSampleRate: 192_000, + PreferredFramesPerSecond: 0.0, + DefaultRollingBufferSeconds, + Required: true, + OwnedOutputs: ["host-dialogue", "loopback-clock", "room-response"], + "Heavy local audio authority: mic inputs plus loopback in one ASIO clock domain."); + + public static MimirNativeCaptureDeviceProfile ScarlettRaven { get; } = ScarlettStarfire with + { + Id = "raven-scarlett", + DisplayName = "Raven Focusrite Scarlett ASIO", + Transport = MimirNativeCaptureTransport.NetworkCultMesh, + ClockDomain = MimirCaptureClockDomain.NetworkSender, + OwnedOutputs = ["remote-dialogue", "remote-loopback-clock", "network-delay-evidence"], + Notes = "Remote co-streamer audio witness. It decodes locally and ships typed timing/path state back over CultMesh." + }; + + public static IReadOnlyList LocalSixCameraProfiles { get; } = + [ + LeapLeftIr, + LeapRightIr, + KiyoProRgb, + KiyoBasicRgb, + Ps3EyeLeft, + Ps3EyeRight + ]; + + public static IReadOnlyList BuiltIn { get; } = + [ + .. LocalSixCameraProfiles, + ScarlettStarfire, + ScarlettRaven + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirNetworkTransportConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirNetworkTransportConfiguration.cs new file mode 100644 index 0000000..13b2636 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirNetworkTransportConfiguration.cs @@ -0,0 +1,90 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirNetworkPayloadKind +{ + TypedTimingState, + RawAudioDebugWindow, + CompressedAudioProgramFeed, + CompressedVideoProgramFeed, + RawVideoDiagnosticWindow +} + +public enum MimirNetworkTransportKind +{ + CultMeshTypedState, + LanUdpDatagrams, + SrtDiagnosticMedia, + WebRtcExperimental, + LocalPipeDiagnostic +} + +public sealed record MimirNetworkTransportConfiguration( + string Id, + string Description, + MimirNetworkTransportKind TransportKind, + MimirNetworkPayloadKind PayloadKind, + bool MayAffectClock, + bool CarriesRawMedia, + double TargetLatencyMilliseconds, + string[] RequiredDocuments, + string[] RejectionConditions, + string Notes); + +public static class MimirNetworkTransportConfigurations +{ + public static MimirNetworkTransportConfiguration CultMeshTimingState { get; } = new( + "cultmesh-timing-state", + "Primary remote-witness path: compact decoded anchors, clock fits, response profiles, and health.", + MimirNetworkTransportKind.CultMeshTypedState, + MimirNetworkPayloadKind.TypedTimingState, + MayAffectClock: true, + CarriesRawMedia: false, + TargetLatencyMilliseconds: 20.0, + RequiredDocuments: ["mimir.bioacoustic_codebook_state", "mimir.bioacoustic_decoder_state", "mimir.acoustic_path_state"], + RejectionConditions: ["unknown-node", "wrong-codebook", "low-confidence", "clock-authority-conflict"], + "Network timestamps are metadata. Decoded anchors are timing evidence."); + + public static MimirNetworkTransportConfiguration RawAudioDebugWindow { get; } = new( + "raw-audio-debug-window", + "Short raw PCM window for debugging remote witness failures.", + MimirNetworkTransportKind.LanUdpDatagrams, + MimirNetworkPayloadKind.RawAudioDebugWindow, + MayAffectClock: false, + CarriesRawMedia: true, + TargetLatencyMilliseconds: 50.0, + RequiredDocuments: ["mimir.acoustic_path_state"], + RejectionConditions: ["not-requested", "outside-debug-session", "oversized-window"], + "Raw audio can explain failures; it should not become the normal timing authority."); + + public static MimirNetworkTransportConfiguration SrtProgramBridge { get; } = new( + "srt-program-bridge", + "Existing LAN media bridge for OBS diagnostics and remote program feeds.", + MimirNetworkTransportKind.SrtDiagnosticMedia, + MimirNetworkPayloadKind.CompressedVideoProgramFeed, + MayAffectClock: false, + CarriesRawMedia: true, + TargetLatencyMilliseconds: 1500.0, + RequiredDocuments: [], + RejectionConditions: ["used-as-clock-authority", "used-for-local-hot-path"], + "Useful bridge. Not the Perfect Machine's synchronization core."); + + public static MimirNetworkTransportConfiguration WebRtcExperiment { get; } = CultMeshTimingState with + { + Id = "webrtc-experimental-media", + Description = "Experimental phone/browser media path; typed timing state still carries authority.", + TransportKind = MimirNetworkTransportKind.WebRtcExperimental, + PayloadKind = MimirNetworkPayloadKind.CompressedAudioProgramFeed, + CarriesRawMedia = true, + TargetLatencyMilliseconds = 80.0, + RejectionConditions = ["used-without-decoded-anchors", "browser-audio-processing-unknown"], + Notes = "Do not trust browser media timestamps without decoded witness state." + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + CultMeshTimingState, + RawAudioDebugWindow, + SrtProgramBridge, + WebRtcExperiment + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs new file mode 100644 index 0000000..35a8edc --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirObsPublicationConfiguration.cs @@ -0,0 +1,83 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirObsVideoPublicationKind +{ + Spout2, + SharedD3D12Texture, + SrtDiagnosticBridge, + WindowCaptureDiagnostic +} + +public enum MimirObsAudioPublicationKind +{ + FaustStemBus, + AsioOutputPair, + WasapiLoopbackDiagnostic, + SrtDiagnosticBridge +} + +public sealed record MimirObsAudioStem( + string Id, + string DisplayName, + int ChannelCount, + bool UserMixable, + bool CarriesTimingWitness, + string Notes); + +public sealed record MimirObsPublicationConfiguration( + string Id, + string Description, + MimirObsVideoPublicationKind VideoKind, + string VideoSourceName, + MimirObsAudioPublicationKind AudioKind, + IReadOnlyList AudioStems, + double TargetPresentationDelaySeconds, + bool DiagnosticOnly); + +public static class MimirObsPublicationConfigurations +{ + public static IReadOnlyList ProgramStems { get; } = + [ + new("host_voice", "Host voice", 1, UserMixable: true, CarriesTimingWitness: false, "Primary Starfire dialogue stem."), + new("co_streamer_voice", "Co-streamer voice", 1, UserMixable: true, CarriesTimingWitness: false, "Raven dialogue after network clock fit."), + new("ambient", "Ambient room", 2, UserMixable: true, CarriesTimingWitness: false, "Room bed after suppression/separation."), + new("transients", "Transients", 2, UserMixable: true, CarriesTimingWitness: false, "Keyboard, desk, impacts, and short events."), + new("local_loopback", "Local loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Starfire program loopback and watermark witness."), + new("remote_loopback", "Remote loopback", 2, UserMixable: true, CarriesTimingWitness: true, "Raven program loopback and network-delay witness."), + new("spatial_bed", "Spatial bed", 4, UserMixable: true, CarriesTimingWitness: false, "Encoded volumetric ambience or ambisonic fold.") + ]; + + public static MimirObsPublicationConfiguration NativeProgram { get; } = new( + "native-program-spout-faust", + "Production target: Fensalir publishes synchronized video, Faust publishes aligned stems.", + MimirObsVideoPublicationKind.Spout2, + "Mimir Point Cloud", + MimirObsAudioPublicationKind.FaustStemBus, + ProgramStems, + TargetPresentationDelaySeconds: 5.0, + DiagnosticOnly: false); + + public static MimirObsPublicationConfiguration TextureInteropProgram { get; } = NativeProgram with + { + Id = "native-program-d3d12-shared-texture", + VideoKind = MimirObsVideoPublicationKind.SharedD3D12Texture, + VideoSourceName = "Mimir D3D12 Program Texture" + }; + + public static MimirObsPublicationConfiguration DiagnosticSrtBridge { get; } = new( + "diagnostic-srt-bridge", + "OBS-facing fallback for bridge tests only. It does not own synchronized program truth.", + MimirObsVideoPublicationKind.SrtDiagnosticBridge, + "Mimir Diagnostic SRT", + MimirObsAudioPublicationKind.SrtDiagnosticBridge, + ProgramStems.Where(stem => stem.Id is "host_voice" or "local_loopback").ToArray(), + TargetPresentationDelaySeconds: 1.5, + DiagnosticOnly: true); + + public static IReadOnlyList BuiltIn { get; } = + [ + NativeProgram, + TextureInteropProgram, + DiagnosticSrtBridge + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs new file mode 100644 index 0000000..4e20d8f --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineManifest.cs @@ -0,0 +1,53 @@ +namespace Mimir.Runtime.Synchronization; + +public sealed record MimirPerfectMachineManifest( + string Schema, + string GeneratedAtUtc, + IReadOnlyList Modules, + IReadOnlyList NodeProfiles, + IReadOnlyList DecoderProfiles, + IReadOnlyList LanguageProfiles, + IReadOnlyList PathLearningProfiles, + IReadOnlyList AcousticLocalizationProfiles, + IReadOnlyList BenchmarkPanels, + IReadOnlyList AudioActuatorStrategies, + IReadOnlyList CaptureProfiles, + IReadOnlyList CameraIngestStrategies, + IReadOnlyList ReservoirStrategies, + IReadOnlyList DistributedWitnesses, + IReadOnlyList NetworkTransports, + IReadOnlyList AuthorityPolicies, + IReadOnlyList AudioFields, + IReadOnlyList VisualFields, + IReadOnlyList ComputePlans, + IReadOnlyList Publications, + IReadOnlyList AssemblyPlans); + +public static class MimirPerfectMachineManifestFactory +{ + public const string Schema = "gamecult.mimir.perfect_machine_manifest.v1"; + + public static MimirPerfectMachineManifest Create(DateTimeOffset? generatedAt = null) => + new( + Schema, + (generatedAt ?? DateTimeOffset.UtcNow).ToString("O"), + MimirModuleLibrary.Entries, + MimirPerfectMachineProfiles.All, + MimirBioacousticDecoderConfiguration.BuiltInProfiles, + MimirBioacousticLanguageConfigurations.BuiltIn, + MimirAcousticPathLearningConfigurations.BuiltIn, + MimirAcousticLocalizationConfigurations.BuiltIn, + MimirBenchmarkPanelConfigurations.BuiltIn, + MimirAudioActuatorConfigurations.BuiltIn, + MimirNativeCaptureConfigurations.BuiltIn, + MimirCameraIngestConfigurations.BuiltIn, + MimirReservoirConfigurations.BuiltIn, + MimirDistributedWitnessConfigurations.BuiltIn, + MimirNetworkTransportConfigurations.BuiltIn, + MimirAuthorityPolicyConfigurations.BuiltIn, + MimirAudioFieldConfigurations.BuiltIn, + MimirVisualFusionConfigurations.BuiltIn, + MimirComputeOffloadConfigurations.BuiltIn, + MimirObsPublicationConfigurations.BuiltIn, + MimirMachineAssemblyPlans.BuiltIn); +} diff --git a/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs new file mode 100644 index 0000000..9a09926 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirPerfectMachineProfiles.cs @@ -0,0 +1,168 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirPerfectMachineNodeRole +{ + StarfireAuthority, + RavenRemote, + PhoneWitness, + MicrocontrollerWitness, + CalibrationBench, + ObsProgramHost +} + +public enum MimirAudioFieldModel +{ + AlignedStems, + SourceBasedSpatialBus, + FirstOrderAmbisonicBed, + SparseEquivalentSources, + HybridEvidenceField +} + +public enum MimirComputeBackend +{ + ManagedScalar, + NativeSimd, + D3D12Compute, + FaustNativeDsp, + RemoteCultMeshWorker +} + +public sealed record MimirPerfectMachineProfile( + string Id, + string Description, + MimirPerfectMachineNodeRole Role, + MimirBioacousticDecoderConfiguration Decoder, + MimirAlignmentActuatorProfile Actuator, + MimirAudioFieldModel AudioFieldModel, + IReadOnlyList ComputeBackends, + bool EmitsWitness, + bool OwnsCanonicalClock, + bool PublishesObsProgram, + TimeSpan RollingWindow, + string[] RequiredCultMeshDocuments); + +public static class MimirPerfectMachineProfiles +{ + public static MimirPerfectMachineProfile StarfireAuthority { get; } = new( + "starfire-authority", + "Heavy local workstation: ASIO loopback authority, runtime buffers, calibration truth, Fensalir/Faust lowering, OBS-facing program output.", + MimirPerfectMachineNodeRole.StarfireAuthority, + MimirBioacousticDecoderConfiguration.BaselineMfccIndex, + MimirAlignmentActuatorProfile.SixSourceFaust, + MimirAudioFieldModel.HybridEvidenceField, + [MimirComputeBackend.ManagedScalar, MimirComputeBackend.NativeSimd, MimirComputeBackend.D3D12Compute, MimirComputeBackend.FaustNativeDsp], + EmitsWitness: true, + OwnsCanonicalClock: true, + PublishesObsProgram: true, + RollingWindow: TimeSpan.FromSeconds(5), + RequiredCultMeshDocuments: + [ + "mimir.bioacoustic_codebook_state.v1", + "mimir.bioacoustic_decoder_state.v1", + "mimir.acoustic_path_state.v1", + "mimir.actuator_state.v1" + ]); + + public static MimirPerfectMachineProfile RavenRemote { get; } = new( + "raven-remote", + "Co-streamer/game machine: captures local loopback/mic evidence, decodes against shared schedule, ships typed observations back to Starfire.", + MimirPerfectMachineNodeRole.RavenRemote, + MimirBioacousticDecoderConfiguration.CompactFastIndex, + MimirAlignmentActuatorProfile.SixSourceFaust, + MimirAudioFieldModel.AlignedStems, + [MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker], + EmitsWitness: false, + OwnsCanonicalClock: false, + PublishesObsProgram: false, + RollingWindow: TimeSpan.FromSeconds(5), + RequiredCultMeshDocuments: + [ + "mimir.bioacoustic_codebook_state.v1", + "mimir.bioacoustic_decoder_state.v1", + "mimir.acoustic_path_state.v1" + ]); + + public static MimirPerfectMachineProfile PhoneWitness { get; } = new( + "phone-witness", + "Small listener: knows codebook/schedule, decodes local mic anchors, reports clock/path observations without owning global truth.", + MimirPerfectMachineNodeRole.PhoneWitness, + MimirBioacousticDecoderConfiguration.CompactFastIndex, + MimirAlignmentActuatorProfile.SixSourceFaust, + MimirAudioFieldModel.AlignedStems, + [MimirComputeBackend.ManagedScalar, MimirComputeBackend.RemoteCultMeshWorker], + EmitsWitness: false, + OwnsCanonicalClock: false, + PublishesObsProgram: false, + RollingWindow: TimeSpan.FromSeconds(3), + RequiredCultMeshDocuments: + [ + "mimir.bioacoustic_codebook_state.v1", + "mimir.bioacoustic_decoder_state.v1" + ]); + + public static MimirPerfectMachineProfile MicrocontrollerWitness { get; } = new( + "microcontroller-witness", + "Tiny listener profile: narrow codebook/schedule receiver that reports sparse anchors and health, not full audio.", + MimirPerfectMachineNodeRole.MicrocontrollerWitness, + MimirBioacousticDecoderConfiguration.HighbandRoomIndex, + MimirAlignmentActuatorProfile.SixSourceFaust, + MimirAudioFieldModel.AlignedStems, + [MimirComputeBackend.RemoteCultMeshWorker], + EmitsWitness: false, + OwnsCanonicalClock: false, + PublishesObsProgram: false, + RollingWindow: TimeSpan.FromSeconds(2), + RequiredCultMeshDocuments: + [ + "mimir.bioacoustic_codebook_state.v1", + "mimir.bioacoustic_decoder_state.v1" + ]); + + public static MimirPerfectMachineProfile CalibrationBench { get; } = new( + "calibration-bench", + "Offline or live calibration worker: explores decoder/codebook/path variants and preserves receipts.", + MimirPerfectMachineNodeRole.CalibrationBench, + MimirBioacousticDecoderConfiguration.RobustWideIndex, + MimirAlignmentActuatorProfile.SixSourceFaust, + MimirAudioFieldModel.SparseEquivalentSources, + [MimirComputeBackend.ManagedScalar, MimirComputeBackend.NativeSimd, MimirComputeBackend.D3D12Compute], + EmitsWitness: true, + OwnsCanonicalClock: false, + PublishesObsProgram: false, + RollingWindow: TimeSpan.FromSeconds(8), + RequiredCultMeshDocuments: + [ + "mimir.bioacoustic_codebook_state.v1", + "mimir.bioacoustic_decoder_state.v1", + "mimir.acoustic_path_state.v1" + ]); + + public static MimirPerfectMachineProfile ObsProgramHost { get; } = new( + "obs-program-host", + "Program-output profile: consumes aligned stems/spatial bed and publishes OBS-facing surfaces without becoming sync authority.", + MimirPerfectMachineNodeRole.ObsProgramHost, + MimirBioacousticDecoderConfiguration.BaselineMfccIndex, + MimirAlignmentActuatorProfile.SixSourceFaust, + MimirAudioFieldModel.SourceBasedSpatialBus, + [MimirComputeBackend.FaustNativeDsp, MimirComputeBackend.D3D12Compute], + EmitsWitness: false, + OwnsCanonicalClock: false, + PublishesObsProgram: true, + RollingWindow: TimeSpan.FromSeconds(5), + RequiredCultMeshDocuments: + [ + "mimir.acoustic_path_state.v1", + "mimir.actuator_state.v1" + ]); + + public static IReadOnlyList All { get; } = + [ + StarfireAuthority, + RavenRemote, + PhoneWitness, + MicrocontrollerWitness, + CalibrationBench, + ObsProgramHost + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs new file mode 100644 index 0000000..86760d3 --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirReservoirConfiguration.cs @@ -0,0 +1,71 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirReservoirStorageStrategy +{ + ManagedRollingBuffers, + NativeSharedEdgeRing, + GpuResidentTemporalField +} + +public sealed record MimirReservoirViewConfiguration( + string Id, + string PayloadKind, + string Owner, + bool RetainsHistory, + string Notes); + +public sealed record MimirReservoirConfiguration( + string Id, + string Description, + MimirReservoirStorageStrategy Strategy, + double WindowSeconds, + IReadOnlyList Views, + string ExpiryPolicy, + string Notes); + +public static class MimirReservoirConfigurations +{ + public static IReadOnlyList StandardViews { get; } = + [ + new("camera_frame", "video payload/native handle", "Mimir.Runtime/native capture", RetainsHistory: false, "Raw frame evidence inside the rolling window."), + new("camera_feature", "sparse visual feature", "Fensalir", RetainsHistory: false, "Derived visual claims."), + new("audio_block", "PCM/native audio block", "Mimir.Runtime/native audio", RetainsHistory: false, "Timing and field evidence."), + new("phase_claim", "audio phase/group-delay evidence", "Mimir.Runtime", RetainsHistory: false, "Calibration/normalization input."), + new("event_claim", "timeline anchor", "Mimir.Runtime", RetainsHistory: false, "Bioacoustic/passive/calibration events."), + new("surface_claim", "spatial surface observation", "Fensalir", RetainsHistory: false, "Fusion candidate."), + new("render_packet", "OBS-facing packet", "Fensalir/Faust", RetainsHistory: false, "Output-ready view.") + ]; + + public static MimirReservoirConfiguration ManagedRuntime { get; } = new( + "managed-runtime-rolling-buffers", + "Current app-level five-second rolling buffers.", + MimirReservoirStorageStrategy.ManagedRollingBuffers, + WindowSeconds: 5.0, + StandardViews, + "accept-inside-window-expire-outside", + "Best for correctness and UI inspection while native rings mature."); + + public static MimirReservoirConfiguration NativeSharedEdge { get; } = ManagedRuntime with + { + Id = "native-shared-edge-ring", + Description = "Production lower reservoir: native shared-edge retention with typed views.", + Strategy = MimirReservoirStorageStrategy.NativeSharedEdgeRing, + Notes = "Native owns retention mechanics; Mimir owns policy and typed meaning." + }; + + public static MimirReservoirConfiguration FensalirTemporalField { get; } = ManagedRuntime with + { + Id = "fensalir-temporal-field", + Description = "Renderer-owned stable evidence field after Mimir lowers observations.", + Strategy = MimirReservoirStorageStrategy.GpuResidentTemporalField, + ExpiryPolicy = "fensalir-evidence-lifetime", + Notes = "This is derived field memory, not raw capture history." + }; + + public static IReadOnlyList BuiltIn { get; } = + [ + ManagedRuntime, + NativeSharedEdge, + FensalirTemporalField + ]; +} diff --git a/src/Mimir.Runtime/Synchronization/MimirVisualFusionConfiguration.cs b/src/Mimir.Runtime/Synchronization/MimirVisualFusionConfiguration.cs new file mode 100644 index 0000000..8488f6e --- /dev/null +++ b/src/Mimir.Runtime/Synchronization/MimirVisualFusionConfiguration.cs @@ -0,0 +1,98 @@ +namespace Mimir.Runtime.Synchronization; + +public enum MimirVisualSensorRole +{ + LeapStereoIrTiming, + Ps3EyeHighRateTracking, + KiyoRgbGroundTruth, + KiyoProRgbGroundTruth, + RemoteRgbOrTracking, + DiagnosticOnly +} + +public enum MimirVisualFusionModel +{ + TimestampedFramesOnly, + SparseFeatureTracks, + VisualConstrainedAudioField, + TemporalGaussianClaims, + FullHybridEvidenceField +} + +public sealed record MimirVisualSensorConfiguration( + string SourceId, + MimirVisualSensorRole Role, + int Width, + int Height, + double TargetFramesPerSecond, + bool RequiresDirectDriver, + bool PreferredGpuHandle, + string Notes); + +public sealed record MimirVisualFusionConfiguration( + string Id, + string Description, + MimirVisualFusionModel Model, + IReadOnlyList Sensors, + bool RequiresFensalirD3D12, + bool PublishesSpout, + bool UsesAudioConstraints); + +public static class MimirVisualFusionConfigurations +{ + public static MimirVisualFusionConfiguration CadenceProof { get; } = new( + "cadence-proof", + "Typed frame descriptors and sustained cadence only; no dense visual fusion claim.", + MimirVisualFusionModel.TimestampedFramesOnly, + CreateDefaultSensors(), + RequiresFensalirD3D12: false, + PublishesSpout: false, + UsesAudioConstraints: false); + + public static MimirVisualFusionConfiguration TrackingFusion { get; } = new( + "tracking-fusion", + "High-rate PS3 Eye and Leap observations become sparse feature tracks; Kiyos provide RGB context.", + MimirVisualFusionModel.SparseFeatureTracks, + CreateDefaultSensors(), + RequiresFensalirD3D12: true, + PublishesSpout: false, + UsesAudioConstraints: false); + + public static MimirVisualFusionConfiguration AudioConstrainedField { get; } = new( + "audio-constrained-field", + "Visual feature tracks constrain acoustic source hypotheses before Fensalir stabilizes field claims.", + MimirVisualFusionModel.VisualConstrainedAudioField, + CreateDefaultSensors(), + RequiresFensalirD3D12: true, + PublishesSpout: false, + UsesAudioConstraints: true); + + public static MimirVisualFusionConfiguration FullHybridEvidence { get; } = new( + "full-hybrid-evidence", + "Perfect Machine visual target: synchronized camera and acoustic constraints lower into Fensalir temporal evidence.", + MimirVisualFusionModel.FullHybridEvidenceField, + CreateDefaultSensors(), + RequiresFensalirD3D12: true, + PublishesSpout: true, + UsesAudioConstraints: true); + + public static IReadOnlyList BuiltIn { get; } = + [ + CadenceProof, + TrackingFusion, + AudioConstrainedField, + FullHybridEvidence + ]; + + public static IReadOnlyList DefaultSensors { get; } = CreateDefaultSensors(); + + private static IReadOnlyList CreateDefaultSensors() => + [ + new("leap-stereo-ir", MimirVisualSensorRole.LeapStereoIrTiming, 640, 240, 110.0, RequiresDirectDriver: true, PreferredGpuHandle: true, "Packed stereo IR pair; timing/close-range tracking candidate."), + new("ps3-eye-0", MimirVisualSensorRole.Ps3EyeHighRateTracking, 320, 240, 187.0, RequiresDirectDriver: true, PreferredGpuHandle: true, "High-rate tracking witness; quality is secondary."), + new("ps3-eye-1", MimirVisualSensorRole.Ps3EyeHighRateTracking, 320, 240, 187.0, RequiresDirectDriver: true, PreferredGpuHandle: true, "Second high-rate tracking witness for geometry."), + new("kiyo-basic", MimirVisualSensorRole.KiyoRgbGroundTruth, 1920, 1080, 30.0, RequiresDirectDriver: true, PreferredGpuHandle: true, "RGB context/ground truth in known-good cadence modes."), + new("kiyo-pro", MimirVisualSensorRole.KiyoProRgbGroundTruth, 1920, 1080, 25.0, RequiresDirectDriver: true, PreferredGpuHandle: true, "RGB ground truth; currently SuperSpeed/cadence-limited."), + new("raven-camera", MimirVisualSensorRole.RemoteRgbOrTracking, 1280, 720, 30.0, RequiresDirectDriver: true, PreferredGpuHandle: false, "Remote feed enters as network producer, not clock authority.") + ]; +} diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 28223a7..4028114 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -86,3 +86,4 @@ {"ts": "2026-05-24T11:38:08+01:00", "type": "implementation", "status": "ok", "note": "Added bioacoustic indexed-cepstral training harness with CultLib/CultCache typed result receipts and WAV artifact directories. Latest run artifacts/bioacoustic-training/bioacoustic-20260524-103438 scored 20 hypothesis/degradation cases; identity often survives mel-cepstral warp/blur, but timing MAE under warp points next at a global clock/path hypothesis rather than larger brute-force matching."} {"ts": "2026-05-24T16:06:07+01:00", "type": "implementation", "status": "ok", "note": "Added global bioacoustic clock hypothesis scoring to the training harness and the first Faust-owned alignment actuator surface. Corrected receipt artifacts/bioacoustic-training/bioacoustic-20260524-150000 favors baseline-mfcc-index/blur-light with total 0.766 after penalizing low-anchor timing fits. bioacoustic-actuator-self-test estimates 317.375 samples, applies fractional correction, and remeasures 0.000 us residual; faust/mimir_alignment_actuator.dsp validates through Faust C++ generation."} {"ts": "2026-05-24T16:09:30+01:00", "type": "implementation", "status": "ok", "note": "Extended the corrected bioacoustic training run to 0.75s in artifacts/bioacoustic-training/bioacoustic-20260524-150729. Compact indexing wins clean throughput at total 0.752/realtime 6.6x, baseline remains steadier under degradation, highband has warped-path value, and anchor density/time-to-lock should become explicit metrics."} +{"ts": "2026-05-24T17:02:15+01:00", "type": "implementation", "status": "ok", "note": "Promoted Perfect Machine research configurations into Mimir.Runtime module library: decoder/language/path-learning/localization/benchmark/actuator/capture/ingest/reservoir/distributed/network/authority/publication profiles, CultMesh contracts, Fensalir lowering, manifest export, and BufferSmoke profile/contract smoke commands."} diff --git a/state/map.yaml b/state/map.yaml index 57b4edd..65849cb 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -184,6 +184,14 @@ accepted_design: Fensalir/Faust integration. - `config/perfect-machine.example.json` declares live owners, sensors, reservoir views, and runtime policy. + - `src/Mimir.Runtime/Synchronization` now includes the Perfect Machine module + library distilled from the research notes: node profiles, decoder/language + profiles, path learning, benchmark panels, actuator strategies, capture and + ingest strategies, reservoir strategies, field profiles, compute offload, + OBS publication, CultMesh contracts, Fensalir lowering, and exportable + manifest generation. + - `docs/perfect-machine-module-library.md` indexes those reusable runtime + modules and their smoke commands. - `docs/code-algorithm-map.md` is the source-level ownership and algorithm map for the current repo code surface. - `docs/perfect-machine-domain-index.md` indexes Mimir's full problem domain,