diff --git a/docs/code-algorithm-map.md b/docs/code-algorithm-map.md index 9ed7367..bf2b2ab 100644 --- a/docs/code-algorithm-map.md +++ b/docs/code-algorithm-map.md @@ -664,15 +664,16 @@ Owns: Blocks: -- Fields: source list, analyzer, state tracker, latest reports dictionary, +- Fields: source list, ordinary audio analyzer, optional complex-contour + analyzer, state tracker, latest ordinary/contour report dictionaries, optional chirp-bin calibration model, ingest count, round-robin candidate. -- Constructor: creates buffer set, loads optional calibration model, pre-creates - configured stream buffers. +- Constructor: creates buffer set, loads optional calibration and + complex-contour channel models, pre-creates configured stream buffers. - `Settings`, `Buffers`, `IngestedSamples`, `SourceCount`: read-only public state. - `AudioSynchronizationReports`, `AudioSynchronizationStates`, - `AudioSynchronizationDecodeTraces`, `ChirpBinEmissionPlan`: cached analyzer - and calibration outputs. + `ComplexContourSynchronizationReports`, `AudioSynchronizationDecodeTraces`, + `ChirpBinEmissionPlan`: cached analyzer and calibration outputs. - `Summary`: short buffer/source summary. - `AddSource`: registers source and ensures descriptor buffer if exposed. - `PollSources`: drains all currently available samples from all sources into @@ -680,6 +681,10 @@ Blocks: - `AnalyzeAudioSynchronization`: full analyzer pass over audio buffers. - `AnalyzeAudioSynchronizationStep`: bounded rotating analyzer pass, so one candidate can be analyzed per tick instead of torching a frame. +- `AnalyzeComplexContourSynchronizationStep`: heavy matched-filter contour pass + for selected Float32 candidates. Live app and Well cadence paths suppress this + inline by default; worker/job ownership or the diagnostic + `MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE=1` override must own execution. - `TryLoadCalibrationModel`: loads persisted chirp-bin response model. - `StoreAudioSynchronizationReports`: updates report cache and smoothed state, removing contradictory passive-negative-lag states. @@ -687,8 +692,10 @@ Blocks: Invariant: -- The hub owns cached runtime belief. The analyzer is a worker; the hub decides - what current belief the rest of the app can read. +- The hub owns cached runtime belief. Ordinary analyzer steps may run as bounded + cadence work; complex-contour matched filtering is configured capability until + a worker/job path commits reports back. The hub decides what current belief + the rest of the app can read. ### `MimirAudioSynchronizationReport.cs` @@ -904,9 +911,13 @@ Blocks: Invariant: -- The live contour lane only runs when explicitly enabled. When enabled, the - runtime emitter and contour analyzer share the same contestant profile instead - of learning one song and playing another. +- The live contour lane only executes heavy analysis when a worker/job owns the + pass or `MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE=1` is set for diagnostics. + `enableComplexContourRuntime` means the capability/profile/model/report lane is + configured; it does not give render threads or Well cadence authority to run + matched filtering. When executed, the runtime emitter and contour analyzer + share the same contestant profile instead of learning one song and playing + another. ### `MimirComplexContourChannelModelDocument.cs` diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 41a8bff..decaf75 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -625,9 +625,13 @@ a named invariant that the native runtime cannot protect yet. long-running Starfire ingest/publish daemon: it loads `config/mimir-runtime.well.local.json`, creates every configured source it can, polls the `MimirSynchronizationHub` with a bounded daemon cadence, - updates audio sync/probe state, and publishes `mimir.well_snapshot.v1` - records with source buffers, composite controls, publication/stem contract, - and synchronized-frame status. It also publishes bounded + updates ordinary audio sync/probe state, and publishes + `mimir.well_snapshot.v1` records with source buffers, composite controls, + publication/stem contract, and synchronized-frame status. Complex contour can + be configured and advertised, but expensive matched-filter contour analysis is + suppressed in the Well loop unless `MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE=1` + is set for diagnostics; the production path needs a worker-owned report lane. + It also publishes bounded `mimir.well_capture_page.v1` records with synchronized frame-slice timing, configured composite state, and inline source sample bodies when CPU payload bytes exist and fit the configured body limit. The online supervisor diff --git a/notes/current-system-map.md b/notes/current-system-map.md index 71e9105..9bac79e 100644 --- a/notes/current-system-map.md +++ b/notes/current-system-map.md @@ -183,14 +183,17 @@ diagnostic reference for constrained chirplet-transform work. The active receiver is the bioacoustic motif timeline; the analyzer only emits active timing reports from matched canonical anchors. The hub also owns cached reports plus smoothed per-source sync state with delay-slope/SRO in ppm. -`MimirRuntime` runs analysis -online as a bounded rotating service and can print live telemetry with -`MIMIR_SYNC_TELEMETRY_SECONDS`. UI and telemetry are passive readers of cached -sync state; they must not invoke the analyzer. Current app testing proves -loopback wakeup, live mic buffers, and confident online sync states, but the -next proof is stable canonical anchors through real mic streams. Camera mics are -spatial/context witnesses; Focusrite devices are dialogue anchors. Fractional -delay and the hot resampler belong in Faust/native DSP. +`MimirRuntime` runs ordinary audio analysis online as a bounded rotating service +and can print live telemetry with `MIMIR_SYNC_TELEMETRY_SECONDS`. UI and +telemetry are passive readers of cached sync state; they must not invoke the +analyzer. Complex-contour matched-filter analysis is excluded from render-thread +and Well-loop cadence by default even when the contour capability is configured; +`MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE=1` is a diagnostic override, not the live +owner. Current app testing proves loopback wakeup, live mic buffers, and +confident online sync states, but the next proof is stable canonical anchors +through real mic streams. Camera mics are spatial/context witnesses; Focusrite +devices are dialogue anchors. Fractional delay and the hot resampler belong in +Faust/native DSP. The WASAPI cadence probe is now a format/state diagnostic: it can request shared or exclusive sample rates, bit depths, channel counts, and float/PCM @@ -310,9 +313,12 @@ inside BufferSmoke artifact replay. When `enableComplexContourRuntime` is true, extracts Float32 windows from rolling audio buffers to publish `evidence=complex-contour` reports. The synthetic runtime-shaped proof `--complex-contour-runtime-self-test` at 192 kHz recovers a 693.5-sample delay -with about 0.219 us error from rolling buffers. The next real-world proof is to -run that lane through live Scarlett loopback and mics, not only stored ASIO -artifacts. +with about 0.219 us error from rolling buffers. In live app/Well operation this +is a configured and publishable evidence capability, not inline cadence +authority: expensive matched-filter execution belongs to a worker queue that can +commit cached reports back to the hub or CultMesh. The next real-world proof is +to run that worker lane through live Scarlett loopback and mics, not only stored +ASIO artifacts. `MimirCalibratedAudioCompositeBuilder` is the first C# diagnostic owner for turning that acoustic evidence into a corrected composite stream. It consumes sample windows plus `MimirAudioSynchronizationState` delay/confidence and diff --git a/scripts/start-online-verse-daemons.ps1 b/scripts/start-online-verse-daemons.ps1 index 0b4f806..a7df8e1 100644 --- a/scripts/start-online-verse-daemons.ps1 +++ b/scripts/start-online-verse-daemons.ps1 @@ -334,7 +334,7 @@ if (-not $SkipRuntime) { "--sync-ms", "250", "--presentation-delay-ms", "2500", "--max-samples-per-source", "32", - "--visual-calibration", "true", + "--visual-calibration", "false", "--visual-calibration-ms", "250", "--visual-expected-leds", "38", "--visual-minimum-luma", "0.55", diff --git a/src/Mimir.Well/Program.cs b/src/Mimir.Well/Program.cs index 63319f9..36b34c8 100644 --- a/src/Mimir.Well/Program.cs +++ b/src/Mimir.Well/Program.cs @@ -1,4 +1,5 @@ using System.Net.WebSockets; +using System.Collections.Concurrent; using System.Diagnostics; using System.Text; using System.Text.Json; @@ -67,6 +68,8 @@ var exposureController = new MimirCameraExposureController(new MimirCameraExposu options.VisualMinimumLuma, options.VisualSettingSeconds, options.VisualResweepSeconds)); +var inlineComplexContourRuntimeEnabled = MimirWellEnvironment.IsTruthy(Environment.GetEnvironmentVariable("MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE")); +var inlineComplexContourRuntimeSuppressionLogged = false; Console.Error.WriteLine($"Mimir Well publishing to {options.PublishUrl}"); Console.Error.WriteLine($"Mimir Well sources={runtimeConfig.SourceFactories.Count} buffers={hub.Buffers.Buffers.Count}"); @@ -85,10 +88,18 @@ while (!stopping.IsCancellationRequested) runtimeConfig.Settings.Audio.ReferenceSourceId, runtimeConfig.Settings.Audio.Mode, options.SyncCandidatesPerStep); - hub.AnalyzeComplexContourSynchronizationStep( - runtimeConfig.Settings.Audio.ReferenceSourceId, - (now - startedAt).TotalSeconds, - options.SyncCandidatesPerStep); + if (hub.ComplexContourRuntimeEnabled && inlineComplexContourRuntimeEnabled) + { + hub.AnalyzeComplexContourSynchronizationStep( + runtimeConfig.Settings.Audio.ReferenceSourceId, + (now - startedAt).TotalSeconds, + options.SyncCandidatesPerStep); + } + else if (hub.ComplexContourRuntimeEnabled && !inlineComplexContourRuntimeSuppressionLogged) + { + inlineComplexContourRuntimeSuppressionLogged = true; + Console.Error.WriteLine("mimir-well complex-contour inline=false reason=well-loop-budget set MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE=1 to run the heavy matched filter inline"); + } } hub.UpdateBioacousticProbeSchedule(NowNs()); @@ -609,7 +620,15 @@ internal sealed record MimirWellPublishReceipt( int ByteLength, double SerializeMilliseconds, double SendMilliseconds, - double TotalMilliseconds); + double TotalMilliseconds, + bool Dropped = false, + int QueueDepth = 0); + +internal static class MimirWellEnvironment +{ + public static bool IsTruthy(string? value) => + value is "1" or "true" or "TRUE" or "yes" or "YES" or "on" or "ON"; +} internal sealed class MimirWellStreamTelemetry { @@ -620,8 +639,11 @@ internal sealed class MimirWellStreamTelemetry private double totalPollMilliseconds; private long publishedDocuments; private long publishedBytes; + private long droppedDocuments; private double maxPublishMilliseconds; private double totalPublishMilliseconds; + private int lastPublishQueueDepth; + private int maxPublishQueueDepth; private string lastPublishedDocument = ""; private int lastPublishedBytes; private double lastPublishMilliseconds; @@ -642,6 +664,14 @@ internal sealed class MimirWellStreamTelemetry public void ObservePublish(MimirWellPublishReceipt receipt) { + lastPublishQueueDepth = receipt.QueueDepth; + maxPublishQueueDepth = Math.Max(maxPublishQueueDepth, receipt.QueueDepth); + if (receipt.Dropped) + { + droppedDocuments++; + return; + } + publishedDocuments++; publishedBytes += receipt.ByteLength; totalPublishMilliseconds += receipt.TotalMilliseconds; @@ -671,6 +701,9 @@ internal sealed class MimirWellStreamTelemetry bytes = publishedBytes, averageMilliseconds = publishedDocuments == 0 ? 0.0 : totalPublishMilliseconds / publishedDocuments, maxMilliseconds = maxPublishMilliseconds, + lastQueueDepth = lastPublishQueueDepth, + maxQueueDepth = maxPublishQueueDepth, + droppedDocuments, lastDocument = lastPublishedDocument, lastBytes = lastPublishedBytes, lastMilliseconds = lastPublishMilliseconds, @@ -797,13 +830,23 @@ internal sealed record MimirWellOptions( internal sealed class MimirWellPublisher(Uri url) : IAsyncDisposable { private readonly ClientWebSocket socket = new(); + private readonly ConcurrentQueue controlQueue = new(); + private readonly ConcurrentQueue bodyQueue = new(); + private readonly SemaphoreSlim signal = new(0); + private readonly CancellationTokenSource senderStopping = new(); + private Task? senderTask; + private int queueDepth; + + private const int MaxQueuedDocuments = 256; + private const int StreamFrameDropThreshold = 128; public async Task ConnectAsync(CancellationToken stopping) { await socket.ConnectAsync(url, stopping).ConfigureAwait(false); + senderTask = Task.Run(() => SendLoopAsync(senderStopping.Token)); } - public async Task PublishAsync( + public Task PublishAsync( object document, string documentName, CancellationToken stopping) @@ -812,19 +855,92 @@ internal sealed class MimirWellPublisher(Uri url) : IAsyncDisposable var json = JsonSerializer.Serialize(document); var bytes = Encoding.UTF8.GetBytes(json); var serializeMilliseconds = stopwatch.Elapsed.TotalMilliseconds; - stopwatch.Restart(); - await socket.SendAsync(bytes, WebSocketMessageType.Text, endOfMessage: true, stopping).ConfigureAwait(false); stopwatch.Stop(); - return new MimirWellPublishReceipt( + var depth = Volatile.Read(ref queueDepth); + if (depth >= MaxQueuedDocuments || + (depth >= StreamFrameDropThreshold && + string.Equals(documentName, "mimir.cultmesh_stream_frame.v1", StringComparison.Ordinal))) + { + return Task.FromResult(new MimirWellPublishReceipt( + documentName, + bytes.Length, + serializeMilliseconds, + 0.0, + serializeMilliseconds, + Dropped: true, + QueueDepth: depth)); + } + + if (string.Equals(documentName, "mimir.cultmesh_stream_frame.v1", StringComparison.Ordinal)) + { + bodyQueue.Enqueue(new QueuedPublish(documentName, bytes)); + } + else + { + controlQueue.Enqueue(new QueuedPublish(documentName, bytes)); + } + + var queued = Interlocked.Increment(ref queueDepth); + signal.Release(); + return Task.FromResult(new MimirWellPublishReceipt( documentName, bytes.Length, serializeMilliseconds, - stopwatch.Elapsed.TotalMilliseconds, - serializeMilliseconds + stopwatch.Elapsed.TotalMilliseconds); + 0.0, + serializeMilliseconds, + QueueDepth: queued)); } + private async Task SendLoopAsync(CancellationToken stopping) + { + while (!stopping.IsCancellationRequested) + { + try + { + await signal.WaitAsync(stopping).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + break; + } + + while (TryDequeueNext(out var item)) + { + Interlocked.Decrement(ref queueDepth); + try + { + await socket.SendAsync(item.Bytes, WebSocketMessageType.Text, endOfMessage: true, stopping).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + return; + } + catch (WebSocketException ex) + { + Console.Error.WriteLine($"mimir-well publish-warning document={item.Document} {ex.Message}"); + return; + } + } + } + } + + private bool TryDequeueNext(out QueuedPublish item) => + controlQueue.TryDequeue(out item!) || bodyQueue.TryDequeue(out item!); + public async ValueTask DisposeAsync() { + senderStopping.Cancel(); + if (senderTask != null) + { + try + { + await senderTask.WaitAsync(TimeSpan.FromSeconds(1)).ConfigureAwait(false); + } + catch + { + } + } + if (socket.State == WebSocketState.Open) { using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(1)); @@ -838,7 +954,11 @@ internal sealed class MimirWellPublisher(Uri url) : IAsyncDisposable } socket.Dispose(); + signal.Dispose(); + senderStopping.Dispose(); } + + private sealed record QueuedPublish(string Document, byte[] Bytes); } internal static class MimirWellCapturePage @@ -1077,7 +1197,8 @@ internal static class MimirWellCapturePage object slice) { var bodyId = $"{options.NodeId}:{captureSequence}:{sample.SourceId}:{sample.Sequence}"; - var transport = PreferredBodyTransport(sample); + var inlineBody = options.StreamFrameInlineBodies && sample.Kind == MimirStreamKind.Audio; + var transport = PreferredBodyTransport(sample, inlineBody); return new { type = "cultmesh-observation", @@ -1095,7 +1216,7 @@ internal static class MimirWellCapturePage ownerPeerId = options.NodeId, kind = sample.Kind.ToString(), origin = sample.Origin.ToString(), - preferredTransports = StreamTransports(sample), + preferredTransports = StreamTransports(sample, inlineBody), clockDomainId = sample.SourceId, }, frame = new @@ -1115,7 +1236,7 @@ internal static class MimirWellCapturePage }, slice, sample = SampleMetadata(sample), - body = CaptureBody(options, sample, options.StreamFrameInlineBodies), + body = CaptureBody(options, sample, inlineBody), }; } @@ -1155,7 +1276,7 @@ internal static class MimirWellCapturePage }, }; - private static string PreferredBodyTransport(MimirStreamSample sample) + private static string PreferredBodyTransport(MimirStreamSample sample, bool inlineBody) { var kind = NativeHandleKind(sample); if (!string.IsNullOrWhiteSpace(kind)) @@ -1167,18 +1288,19 @@ internal static class MimirWellCapturePage : "native-handle"; } - return sample.Data.IsEmpty ? "metadata-only" : "inline-bytes"; + return sample.Data.IsEmpty ? "metadata-only" : inlineBody ? "inline-bytes" : "mimir-recorder-page"; } - private static string[] StreamTransports(MimirStreamSample sample) + private static string[] StreamTransports(MimirStreamSample sample, bool inlineBody) { - var transport = PreferredBodyTransport(sample); + var transport = PreferredBodyTransport(sample, inlineBody); return transport switch { "shared-d3d12-texture" => ["shared-d3d12-texture", "mimir-recorder-page", "cultcache-page"], "shared-d3d11-texture" => ["shared-d3d11-texture", "mimir-recorder-page", "cultcache-page"], "native-handle" => ["native-handle", "mimir-recorder-page", "cultcache-page"], "inline-bytes" => ["mimir-recorder-page", "inline-bytes", "cultcache-page"], + "mimir-recorder-page" => ["mimir-recorder-page", "cultcache-page"], _ => ["metadata-only"], }; } @@ -1428,6 +1550,7 @@ internal static class MimirWellSnapshot referenceSourceId = runtimeConfig.Settings.Audio.ReferenceSourceId, mode = runtimeConfig.Settings.Audio.Mode.ToString(), complexContour = hub.ComplexContourRuntimeEnabled, + complexContourInline = MimirWellEnvironment.IsTruthy(Environment.GetEnvironmentVariable("MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE")), states = hub.AudioSynchronizationStates.Select(state => new { state.SourceId, diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 43139a1..3ee3aa1 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -368,3 +368,4 @@ {"ts": "2026-06-02T23:49:45+01:00", "type": "decision", "status": "ok", "note": "Mimir Well/Fensalir live repair: rolling-buffer authority now uses canonical arrival UTC timestamps for ASIO/native camera samples while preserving device timestamps as calibration evidence; Fensalir CultCache persistence is best-effort so a Windows File.Replace sharing failure cannot kill the live Eve daemon."} {"ts": "2026-06-03T04:13:02+01:00", "type": "implementation", "status": "ok", "note": "2026-06-03: ASIO readiness repair: Mimir now derives ASIO block times from native device timestamps, preserves local-epoch timestamps in the canonical mapper, sorts planner samples chronologically, keeps frames visible when overlap pinches, and lets the latency controller chase FutureSampleOnly slices downward. Live Well reached repeated 6/6 Ready frames at roughly 75-80 ms presentation delay with ASIO distances reported around 32-36 ms."} {"ts": "2026-06-03T04:40:48+01:00", "type": "implementation", "status": "ok", "note": "2026-06-03: Source health and process-clock repair: FFmpeg/process/frame-event sources now use UTC arrival timestamps instead of Stopwatch-domain timestamps; Well sourceHealth distinguishes live, stalled, waiting-network, and waiting-device-samples. Live verification after restart reached repeated 6/6 Ready frames, while streamPressure exposed the next bottleneck: inline WebSocket body lane published ~66 MB across 53 Well snapshots at ~82 ms average publish cost."} +{"ts": "2026-06-03T04:55:12+01:00", "type": "implementation", "status": "ok", "note": "Well live-loop stall was caused by inline complex-contour matched-filter analysis running inside the drinking/publish cadence. Mimir.Well now follows the app runtime authority: complex contour may be configured/published but only runs inline when MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE is truthy; otherwise it logs suppression and leaves the heavy decoder for a worker-owned path. Live verification in online-verse-daemons-20260602-233914 after restart reached seq 122->140 over ~4.4s with 6/6 Ready, publish avg <0.3 ms, zero dropped documents, and adaptive delay near 40 ms."} diff --git a/state/map.yaml b/state/map.yaml index 1aa0365..aeca856 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -448,9 +448,14 @@ invariants: source payloads without bloating the observation ledger. `Mimir.Well` is the Starfire Well daemon: it loads the broad Well runtime profile, starts every configured source it can, polls `MimirSynchronizationHub` - with a bounded daemon budget, updates audio sync/probe state, and publishes - `mimir.well_snapshot.v1` records containing buffer health, composite - controls, publication/stem contract, and sync status. It also publishes + with a bounded daemon budget, updates ordinary audio sync/probe state, and + publishes `mimir.well_snapshot.v1` records containing buffer health, + composite controls, publication/stem contract, and sync status. Complex + contour may be configured and reported as a capability, but the Well loop + does not run the matched-filter contour analyzer unless + `MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE=1` is set as a diagnostic override; + the live owner for that heavy decoder is a worker/job lane that returns + cached reports without owning drink/publish cadence. It also publishes bounded `mimir.well_capture_page.v1` records for synchronized frame slices: each page carries timing metadata, configured composite state, and inline source sample bodies when a sample has CPU bytes and stays under the @@ -870,8 +875,10 @@ accepted_design: 192 kHz recovers a 693.5-sample delay with about 0.219 us error. A live dump on 2026-06-02 showed the heavy matched-filter analyzer running `MimirComplexContourMatchedFilterBank.ScoreAt` inline on the Fensalir render - thread at 192 kHz, hanging the app. Runtime complex-contour analysis is now - suppressed on the render thread by default; `MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE=1` + thread at 192 kHz, hanging the app; a later Well shakedown showed the same + heavy analyzer could stall the drink/publish loop after initial snapshots. + Runtime complex-contour analysis is now suppressed on both render-thread and + Well-loop inline paths by default; `MIMIR_COMPLEX_CONTOUR_RUNTIME_INLINE=1` is a diagnostic override, and the proper live owner is a bounded worker/job lane that returns reports to the runtime without owning frame cadence. `MimirCalibratedAudioCompositeBuilder` is the first diagnostic composite