diff --git a/docs/audio-doctrine.md b/docs/audio-doctrine.md index 954dd34..4c4832c 100644 --- a/docs/audio-doctrine.md +++ b/docs/audio-doctrine.md @@ -320,11 +320,19 @@ Useful metrics: Song-scene parity targets must publish target evidence before candidate golf: log-mel spectrogram CSV, per-band statistics, first and second derivative -summaries, RMS envelope, and envelope autocorrelation. Those artifacts are not -loose side files only; scoring writes them as typed song challenge evidence -records in the `.cc` CultCache/CultMesh store and references them from each -trial result. Qdrant/Ollama retrieval may derive vectors from those records, but -the `.cc` records remain truth. +summaries, RMS envelope, envelope autocorrelation, and whitened spectral +autocorrelation over band-normalized log-mel flux. Those artifacts are not loose +side files only; scoring writes them as typed song challenge evidence records in +the `.cc` CultCache/CultMesh store and references them from each trial result. +Qdrant/Ollama retrieval may derive vectors from those records, but the `.cc` +records remain truth. + +Song-scene agents should iterate like a tiny studio. Render the candidate, +extract the same features from the Faust output, compare target and candidate +band motion, envelope autocorrelation, whitened spectral autocorrelation, +loudness, centroid, and instrument-role metrics, then move sources, filters, +gates, patterns, and gains. A one-shot patch is sample data; a five-attempt +revision trace is curriculum. Noise is not a scene by itself. Background air, dust, codec grit, tape, vinyl, and room color need a role owner that shapes spectrum and time. Prefer diff --git a/state/evidence.jsonl b/state/evidence.jsonl index 9bd76d6..37cf474 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -257,3 +257,4 @@ {"date":"2026-05-31","area":"song-curriculum-paired-pass","lesson":"A paired song-curriculum pass completed with five agents, two targets per agent, one patch per target, ten scored trials, ten target evidence bundles, sixty Qdrant chunks, and a distilled `.cc` store. Metrics were all `pressure`: best log-mel cosine `0.502128`, mean `0.377955`, worst `0.27853`, mean audio score `0.339432`. Distillation kept all ten and identified clocked dust/hat texture plus rubber/sub bass as reusable roles, while preserving the shaped-noise owner requirement as a transfer rule rather than letting static white hiss back into the curriculum.","evidence":"artifacts/parity/song-snippet-swarms/20260531T074935298/song-trial-results.cc; artifacts/parity/song-snippet-swarms/20260531T074935298/pass-001/distillation-report.md; artifacts/parity/song-snippet-swarms/20260531T074935298/pass-001/vector-index-after.md; artifacts/parity/song-snippet-swarms/20260531T074935298/pass-001/semantic-search-after.md"} {"date":"2026-05-31","area":"song-curriculum-playlists-and-invalid-candidates","lesson":"Song curriculum passes now publish a playable top-candidate playlist per pass and keep invalid agent syntax as negative evidence instead of aborting the swarm. A two-pass paired run produced pass-001 best cosine `0.386098` with 8 playable candidates and 2 `render-failed` syntax casualties, then pass-002 improved to best cosine `0.502435`, mean cosine `0.352907`, 10 playable candidates, and 8 pressure verdicts after reading distilled pass-1 evidence. Playlist reports filter to candidates with actual rendered WAVs and list skipped non-playable failures separately.","evidence":"tools/run-song-snippet-trial-swarm.ps1; tools/IpaTrialWorker/Program.cs; artifacts/parity/song-snippet-swarms/20260531T084455803/pass-001/top-scoring-candidates.m3u; artifacts/parity/song-snippet-swarms/20260531T084455803/pass-002/top-scoring-candidates.m3u; artifacts/parity/song-snippet-swarms/20260531T084455803/pass-002/distillation-report.md"} {"date":"2026-05-31","area":"song-instrument-owner-scoring","lesson":"The song curriculum harness must teach instrument ownership, not only spectrogram proximity. `song-score` now records syrinx-voice, subtractive-drum, additive/PAD, musical-instrument-score, and chip-distress-risk metrics for every candidate; swarm prompts require voice-like leads to prefer syrinx/acoustic sources, percussion to prefer subtractive pitched body plus filtered noise skin, and pads/beds to prefer additive/PAD banks. Distillation ranking now rewards owned roles and demotes chip-distress-risk candidates unless they are explicit contrast evidence.","evidence":"tools/IpaTrialWorker/Program.cs; tools/run-song-snippet-trial-swarm.ps1; docs/audio-doctrine.md; patches/advanced/syrinx-voice.aqua; patches/advanced/subtractive-drums.aqua; patches/advanced/additive-pad.aqua; artifacts/parity/song-instrument-metric-smoke/smoke/summary.csv"} +{"date":"2026-05-31","area":"song-full-iteration-loop","lesson":"Song agents need a studio loop, not a one-shot submission. `song-prepare --duration-seconds 0` freezes full decoded songs; the swarm exposes `-FullSongs`; song challenges and candidate renders now include whitened spectral autocorrelation over band-whitened log-mel flux; and the prompt requires five local score/revise attempts per target before publishing a final patch. Candidate-side analysis artifacts let agents compare their Faust render against the target using the same feature vocabulary instead of guessing from the final verdict.","evidence":"tools/IpaTrialWorker/Program.cs; tools/run-song-snippet-trial-swarm.ps1; artifacts/parity/song-candidate-feature-smoke-2; artifacts/parity/song-snippet-swarms/20260531T120800752"} diff --git a/state/scratch.md b/state/scratch.md index 6452658..d1152e9 100644 --- a/state/scratch.md +++ b/state/scratch.md @@ -1829,3 +1829,29 @@ Next likely slice: `artifacts/parity/song-instrument-metric-smoke/smoke/summary.csv` records a syrinx candidate with `instrument_voice_syrinx=1`, `musical_instrument_score=0.34`, and `chip_distress_risk=0`. + +2026-05-31 full-song iterative harness slice: + +- `song-prepare --duration-seconds 0` now means "freeze the full decoded source + file from sample zero" instead of a random excerpt. `run-song-snippet-trial-swarm.ps1` + has `-FullSongs`, keeps the source folder recursive, and widened local audio + discovery to `.mp3/.wav/.flac/.aiff/.aif/.ogg/.m4a/.mp4`. Current Reinier + parent corpus discovery sees 2421 supported audio files. +- Song challenge analysis now includes `whitened-spectral-autocorr.csv`, derived + from autocorrelation of a band-whitened log-mel spectral-flux series. This is + intended to expose repeating spectral/rhythmic structure after broad tone + color is normalized away. +- `song-score` writes candidate-side render analysis artifacts from Faust output: + `candidate-analysis.md`, `candidate-logmel-spectrogram.csv`, + `candidate-logmel-band-stats.csv`, `candidate-rms-envelope.csv`, + `candidate-rms-envelope-autocorr.csv`, and + `candidate-whitened-spectral-autocorr.csv`. Agents can now inspect their own + render with the same feature vocabulary used for the target. +- The swarm prompt now requires an in-agent studio loop: each worker must run + five local attempts per target, call `song-score` on each attempt, read the + candidate analysis/comparison artifacts, move timings/filters/gains/sources, + and publish only one final `.aqua` per target to the official output folder. +- Verification: `dotnet build tools/IpaTrialWorker/IpaTrialWorker.csproj` + passed; zero-pass full-song swarm syntax smoke passed; candidate feature + smoke wrote the new candidate autocorrelation and analysis artifacts under + `artifacts/parity/song-candidate-feature-smoke-2`. diff --git a/state/spine.yaml b/state/spine.yaml index 1790e48..fd5f205 100644 --- a/state/spine.yaml +++ b/state/spine.yaml @@ -150,6 +150,7 @@ live_invariants: - External IPA exploration now has a worker/tool boundary. `tools/IpaTrialWorker` owns `seed`, `score`, `search`, `show`, and audit-only `dump` commands over the typed `.cc` trial-result store; `score` renders external Codex-authored `.aqua` candidates through `IpaTrialOrchestrator.RunCandidateScriptsAsync`, while `search` is the semantic retrieval organ for hypothesis agents. `IpaTrialResult` records now embed extracted primitive timeline facts from `ProbeTimelineReport` for contact closure/reservoir/release, source load/flow, branch admittance/exchanged flow, radiation boundary/output, path area/delay/energy, and passivity; `search`, `show`, `dump`, and summary CSV output surface release/radiation/passivity witnesses directly from the `.cc` record. `search` follows VoidBot's Qdrant shape: `.cc` records plus timeline facts and referenced local artifacts become evidence chunks, local Ollama `qwen3-embedding:0.6b` produces 1024-dim vectors, Qdrant collection `aquasynth_ipa_trial_results` stores stable SHA-256 UUID points with trial/candidate payload metadata, and the report surfaces vector scores, excerpts, tags, metric facts, timeline facts, contrast candidates, `class_focus`, per-hit `class_match`, and `show` commands. Qdrant is a derived index; CultCache remains truth, and lexical/metric search is fallback when the vector stack is unavailable. Retrieval tuning added phonetic class-affinity scoring and candidate-specific mixed-lane ownership so `mix-p` can support stop seams without every mixed candidate matching stop evidence. `tools/run-ipa-trial-loop.ps1` orchestrates external `codex exec` hypothesis/evaluator workers around that store. Source-edit-enabled rounds now allow agents to golf DSL syntax, parser/model records, Faust lowering, worker evidence surfaces, and tests when evidence points at a repeated owner failure; agents must write `source-change-plan.md`, and the loop runs a focused build before scoring. `hypotheses.md` is the hypothesis worker report of record; `hypothesis-agent.md` is only the external agent's final chat summary. The first loose source-edit round produced no source edits and no aggregate metric win, then exposed a deeper learning-loop fault: family ideas were rotated across unrelated targets instead of aligned to phonetic owners. Production prompts now pass the previous evaluation path into the hypothesizer, require `Previous Round Learning`, require `Target-Class Lane Plan`, forbid round-robin family assignment, and let the evaluator fail class alignment when family/target assignments do not match phonetic owner classes or explicit transfer-test mutations. Candidate reports must include a `Scene Voice Model` naming primary-articulatory, tract-noise, release/transient, background-room, and recording-condition voices, with general AquaSynth synth primitives allowed for helper scene voices but not accepted as articulation evidence unless lower-layer witnesses move. At least one scene/noise or recording-condition family in a source-golf round must implement real auxiliary `.aqua` `voice` declarations instead of satisfying scene modeling with prose. The active trial shape is five target sets of five phoneme lanes each: vowels, nasals/approximants, fricatives, stops, and mixed generalization. Candidate names are exactly `____.aqua`, with lowercase kebab-case family ids. Three dry external Codex smoke passes refined hypothesizer/evaluator prompts into exact 25-lane output, retrieval receipts, claim audits, evidence-quality ledgers/verdicts, and fail-closed evaluator compliance. A later five-pass prompt optimization showed the best science comes from fixed receipt packets plus constrained reasoning; production prompts now require contrast pairs, `show` metric values, primitive timeline facts/excerpts or explicit `missing`, class-consistent reference matrices, scene voice models, `X owns Y so Z remains true` owner sentences, at least five micro-sweep perturbations before a seam is actionable, and a novelty gate rejecting loudness/dressing-only families as loss-landscape novelty. Verification: focused builds/tests pass, old-store show compatibility passes, and a fresh one-candidate smoke under `artifacts/parity/ipa-timeline-fact-smoke/20260531T233217641` proves release/radiation/passivity facts persist and retrieve through Qdrant/Ollama. - Song-snippet parity now treats target analysis as CultMesh database evidence. `song-prepare` writes each 30-second clip's log-mel spectrogram CSV, per-band first/second derivative stats, RMS envelope, envelope autocorrelation, and analysis report. `song-score` upserts those as typed `SongChallengeEvidenceDocument` records in the same `.cc` store as `IpaTrialResult`, adds target metrics and `cultmesh://aquasynth/song-challenge-evidence/...` artifacts to every result, and `search/index` derive Qdrant chunks from the database records. `pattern` and `scale` DSL sugar are live authoring conveniences that lower into normal parameters and hold-loop curves; prompt agents may propose richer Strudel-like syntax, but candidate patches must use implemented syntax. - Song-scene scoring now records instrument-role ownership so the music-generator curriculum does not learn that simple oscillator blips are good instruments by default. `song-score` emits `instrument_voice_syrinx`, `instrument_drum_subtractive`, `instrument_pad_additive`, `musical_instrument_score`, and `chip_distress_risk`; verdict/distillation ranking rewards owned roles and demotes chip-distress-risk candidates unless preserved as contrast. Swarm prompts now require voice-like leads to prefer syrinx/acoustic sources with pressure/opening/radiation motion, percussion to prefer subtractive pitched bodies plus filtered noise skins, and pads/beds to prefer additive/PAD `layer`/`harmonics`/`spectrum` banks. + - Song-scene harness supports full-song curriculum and in-agent revision. `song-prepare --duration-seconds 0` freezes the whole decoded song from sample zero, and `run-song-snippet-trial-swarm.ps1 -FullSongs` samples recursively from the Reinier parent folder including `.m4a/.mp4`. Target and candidate analysis now include whitened spectral autocorrelation over band-whitened log-mel flux. Candidate-side Faust-render artifacts include log-mel spectrogram/stats, RMS envelope/autocorr, whitened spectral autocorr, and an analysis markdown report. Swarm prompts require five local score/revise attempts per target before publishing the final `.aqua`. - TurbulenceJet now includes a release-timed burst-noise component in addition to the deterministic pressure pulse, but a stronger trial barely moved papa; the source is not the main remaining articulation bottleneck. Graph radiation now mixes 30% raw boundary flow with 70% high-passed flow after local admittance weighting, down from the over-bright 15/85 blend. Latest radiation/source artifact under `artifacts/parity/pink-trombone-utterance-logmel/20260527T174236049`: mama/papa/thrombosis cosine 0.6578/0.7692/0.5585, rmsRatio 3.0006/2.6153/2.3488, articulation 0.1437/0.1173/0.1191. Static `closure-release` under `artifacts/parity/pink-trombone-logmel/20260527T174139661` has rmsRatio 0.9900 and speechBandRatio 0.0842. This is still not speaking; next pressure is true tract resonance/passive scattering and pressure storage, not more burst gain. - Same-path two-port area discontinuities now emit `graph_area_energy_in_*` and `graph_area_energy_out_*` probes alongside the three-port branch `graph_connection_energy_*` probes. Use these to inspect passivity and formant-body failures at the scatter sites rather than judging only final output loudness. - `FaustExportOptions.DebugProbeUi` wraps graph probe signals in Faust `vbargraph` UI controls under `/debug/...` without changing audio output arity. Native streaming Faust patches now separate writable controls from read-only probe zones and can read/snapshot probe values after `ProcessBlock`; use this for acoustic graph passivity/radiation inspection, leave it off for normal parity renders. diff --git a/tools/IpaTrialWorker/Program.cs b/tools/IpaTrialWorker/Program.cs index 37454aa..646085c 100644 --- a/tools/IpaTrialWorker/Program.cs +++ b/tools/IpaTrialWorker/Program.cs @@ -109,10 +109,13 @@ static async Task SongPrepareAsync(Dictionary options) Directory.CreateDirectory(artifactRoot); var decoded = DecodeMono(source); var resampled = Resample(decoded.Samples, decoded.SampleRate, sampleRate); - var clipLength = Math.Min(resampled.Length, Math.Max(1, (int)MathF.Round(durationSeconds * sampleRate))); + var fullSong = durationSeconds <= 0; + var clipLength = fullSong + ? resampled.Length + : Math.Min(resampled.Length, Math.Max(1, (int)MathF.Round(durationSeconds * sampleRate))); var maxStart = Math.Max(0, resampled.Length - clipLength); var random = new Random(seed); - var startSample = maxStart == 0 ? 0 : random.Next(0, maxStart + 1); + var startSample = fullSong || maxStart == 0 ? 0 : random.Next(0, maxStart + 1); var clip = new float[clipLength]; Array.Copy(resampled, startSample, clip, 0, clipLength); NormalizePeak(clip, .9f); @@ -127,11 +130,13 @@ static async Task SongPrepareAsync(Dictionary options) var bandStatsPath = Path.Combine(artifactRoot, "logmel-band-stats.csv"); var envelopePath = Path.Combine(artifactRoot, "rms-envelope.csv"); var envelopeAutocorrPath = Path.Combine(artifactRoot, "rms-envelope-autocorr.csv"); + var whitenedSpectralAutocorrPath = Path.Combine(artifactRoot, "whitened-spectral-autocorr.csv"); var analysisReportPath = Path.Combine(artifactRoot, "analysis.md"); await File.WriteAllTextAsync(spectrogramPath, SpectrogramCsv(analysis.LogMelSpectrogram), Encoding.UTF8); await File.WriteAllTextAsync(bandStatsPath, SpectrogramBandStatsCsv(analysis.LogMelSpectrogram), Encoding.UTF8); await File.WriteAllTextAsync(envelopePath, EnvelopeCsv(analysis.RmsEnvelope, sampleRate, analyzer.Config.HopSize), Encoding.UTF8); await File.WriteAllTextAsync(envelopeAutocorrPath, AutocorrelationCsv(analysis.RmsEnvelope, sampleRate / (float)Math.Max(1, analyzer.Config.HopSize)), Encoding.UTF8); + await File.WriteAllTextAsync(whitenedSpectralAutocorrPath, WhitenedSpectralAutocorrelationCsv(analysis.LogMelSpectrogram, sampleRate / (float)Math.Max(1, analyzer.Config.HopSize)), Encoding.UTF8); var features = new SongChallengeFeatures( analysis.Features.DurationSeconds, analysis.Features.Peak, @@ -155,6 +160,7 @@ static async Task SongPrepareAsync(Dictionary options) bandStatsPath, envelopePath, envelopeAutocorrPath, + whitenedSpectralAutocorrPath, analysisReportPath); var challenge = new SongChallenge( challengeId, @@ -263,6 +269,13 @@ static async Task SongScoreAsync(Dictionary options) NormalizePeak(matched, .9f); WriteWav(candidateWav, matched, challenge.SampleRate); artifacts.Add(Artifact("candidate-wav", candidateWav)); + var candidateAnalysisArtifacts = await WriteSongRenderAnalysisAsync( + matched, + challenge.SampleRate, + audioDir, + "candidate", + analyzer); + artifacts.AddRange(candidateAnalysisArtifacts.Select(ArtifactFromAnalysis)); var comparison = analyzer.Compare(referenceSamples, matched); metrics.AddRange(SongComparisonMetrics(comparison)); verdict = SongVerdict(comparison, instrumentProfile); @@ -303,7 +316,9 @@ static async Task SongScoreAsync(Dictionary options) verdict, [ "This is a local-only song clip challenge; reference audio is not redistributed by the repo.", - $"The target is a randomly selected {challenge.DurationSeconds:0.###}-second scene-audio snippet, not IPA articulation truth.", + challenge.StartSeconds <= 0.001f + ? $"The target starts at the source file beginning and lasts {challenge.DurationSeconds:0.###} seconds; in full-song runs this is the whole decoded song, not IPA articulation truth." + : $"The target is a randomly selected {challenge.DurationSeconds:0.###}-second scene-audio snippet, not IPA articulation truth.", "Challenge spectrogram, derivative, envelope, and autocorrelation artifacts are stored as typed CultMesh/CultCache evidence documents in the .cc database.", "Full-patch FM/AM/noise/scene modeling is allowed; primitive timeline facts remain diagnostic when present.", instrumentProfile.ChipDistressRisk >= .6f @@ -1934,6 +1949,10 @@ static IEnumerable SongChallengeArtifacts(SongChallenge ch yield return Artifact("target-logmel-band-stats", challenge.Artifacts.LogMelBandStatsCsv); yield return Artifact("target-rms-envelope", challenge.Artifacts.RmsEnvelopeCsv); yield return Artifact("target-rms-envelope-autocorr", challenge.Artifacts.RmsEnvelopeAutocorrCsv); + if (!string.IsNullOrWhiteSpace(challenge.Artifacts.WhitenedSpectralAutocorrCsv)) + { + yield return Artifact("target-whitened-spectral-autocorr", challenge.Artifacts.WhitenedSpectralAutocorrCsv); + } } static SpeechRenderArtifact SongChallengeEvidenceArtifact(SongChallengeEvidenceDocument document) => @@ -1942,6 +1961,9 @@ static SpeechRenderArtifact SongChallengeEvidenceArtifact(SongChallengeEvidenceD $"cultmesh://aquasynth/song-challenge-evidence/{Uri.EscapeDataString(document.EvidenceId)}", document.ContentHash); +static SpeechRenderArtifact ArtifactFromAnalysis(SongRenderAnalysisArtifacts artifact) => + Artifact($"candidate-{artifact.Kind}", artifact.Path); + static IReadOnlyList SongChallengeEvidenceDocuments(SongChallenge challenge) { var documents = new List(); @@ -1955,6 +1977,7 @@ static IReadOnlyList SongChallengeEvidenceDocumen AddSongChallengeEvidence(documents, challenge, "logmel-band-stats", "text/csv", challenge.Artifacts.LogMelBandStatsCsv); AddSongChallengeEvidence(documents, challenge, "rms-envelope", "text/csv", challenge.Artifacts.RmsEnvelopeCsv); AddSongChallengeEvidence(documents, challenge, "rms-envelope-autocorr", "text/csv", challenge.Artifacts.RmsEnvelopeAutocorrCsv); + AddSongChallengeEvidence(documents, challenge, "whitened-spectral-autocorr", "text/csv", challenge.Artifacts.WhitenedSpectralAutocorrCsv); return documents; } @@ -1981,6 +2004,35 @@ static void AddSongChallengeEvidence( DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture))); } +static async Task> WriteSongRenderAnalysisAsync( + IReadOnlyList samples, + int sampleRate, + string artifactRoot, + string prefix, + AudioAnalyzer analyzer) +{ + Directory.CreateDirectory(artifactRoot); + var analysis = analyzer.Analyze(samples.ToArray()); + var frameRate = sampleRate / (float)Math.Max(1, analyzer.Config.HopSize); + var artifacts = new[] + { + new SongRenderAnalysisArtifacts("analysis-report", Path.Combine(artifactRoot, $"{prefix}-analysis.md")), + new SongRenderAnalysisArtifacts("logmel-spectrogram", Path.Combine(artifactRoot, $"{prefix}-logmel-spectrogram.csv")), + new SongRenderAnalysisArtifacts("logmel-band-stats", Path.Combine(artifactRoot, $"{prefix}-logmel-band-stats.csv")), + new SongRenderAnalysisArtifacts("rms-envelope", Path.Combine(artifactRoot, $"{prefix}-rms-envelope.csv")), + new SongRenderAnalysisArtifacts("rms-envelope-autocorr", Path.Combine(artifactRoot, $"{prefix}-rms-envelope-autocorr.csv")), + new SongRenderAnalysisArtifacts("whitened-spectral-autocorr", Path.Combine(artifactRoot, $"{prefix}-whitened-spectral-autocorr.csv")) + }; + + await File.WriteAllTextAsync(artifacts[0].Path, SongRenderAnalysisReport(prefix, analysis, frameRate), Encoding.UTF8); + await File.WriteAllTextAsync(artifacts[1].Path, SpectrogramCsv(analysis.LogMelSpectrogram), Encoding.UTF8); + await File.WriteAllTextAsync(artifacts[2].Path, SpectrogramBandStatsCsv(analysis.LogMelSpectrogram), Encoding.UTF8); + await File.WriteAllTextAsync(artifacts[3].Path, EnvelopeCsv(analysis.RmsEnvelope, sampleRate, analyzer.Config.HopSize), Encoding.UTF8); + await File.WriteAllTextAsync(artifacts[4].Path, AutocorrelationCsv(analysis.RmsEnvelope, frameRate), Encoding.UTF8); + await File.WriteAllTextAsync(artifacts[5].Path, WhitenedSpectralAutocorrelationCsv(analysis.LogMelSpectrogram, frameRate), Encoding.UTF8); + return artifacts; +} + static AutocorrelationPoint ReadAutocorrPeak(string? path) { if (string.IsNullOrWhiteSpace(path) || !File.Exists(path)) @@ -2571,6 +2623,59 @@ static string AutocorrelationCsv(IReadOnlyList values, float frameRate) return builder.ToString(); } +static string WhitenedSpectralAutocorrelationCsv(Spectrogram spectrogram, float frameRate) +{ + var series = WhitenedSpectralFluxSeries(spectrogram); + var builder = new StringBuilder(); + builder.AppendLine("lag_frames,lag_seconds,correlation"); + foreach (var point in Autocorrelation(series, frameRate, Math.Min(Math.Max(1, series.Length - 1), 512))) + { + builder.Append(point.LagFrames).Append(','); + builder.Append(point.LagSeconds.ToString("0.########", CultureInfo.InvariantCulture)).Append(','); + builder.AppendLine(point.Correlation.ToString("0.########", CultureInfo.InvariantCulture)); + } + + return builder.ToString(); +} + +static float[] WhitenedSpectralFluxSeries(Spectrogram spectrogram) +{ + if (spectrogram.Frames <= 1 || spectrogram.Bands <= 0) + { + return []; + } + + var means = new float[spectrogram.Bands]; + var stddevs = new float[spectrogram.Bands]; + for (var band = 0; band < spectrogram.Bands; band++) + { + var values = new float[spectrogram.Frames]; + for (var frame = 0; frame < spectrogram.Frames; frame++) + { + values[frame] = spectrogram.At(frame, band); + } + + means[band] = Mean(values); + stddevs[band] = Math.Max(1e-4f, StdDev(values)); + } + + var series = new float[spectrogram.Frames - 1]; + for (var frame = 1; frame < spectrogram.Frames; frame++) + { + var sum = 0f; + for (var band = 0; band < spectrogram.Bands; band++) + { + var current = (spectrogram.At(frame, band) - means[band]) / stddevs[band]; + var previous = (spectrogram.At(frame - 1, band) - means[band]) / stddevs[band]; + sum += MathF.Abs(current - previous); + } + + series[frame - 1] = sum / Math.Max(1, spectrogram.Bands); + } + + return series; +} + static IReadOnlyList Autocorrelation(IReadOnlyList values, float frameRate, int maxLags) { if (values.Count < 2) @@ -2645,6 +2750,9 @@ static IReadOnlyList SongTargetMetrics( var autocorrPeak = ReadAutocorrPeak(challenge.Artifacts?.RmsEnvelopeAutocorrCsv); metrics.Add(new SpeechScoreMetric("target_envelope_autocorr_peak", autocorrPeak.Correlation, 0)); metrics.Add(new SpeechScoreMetric("target_envelope_autocorr_peak_lag_seconds", autocorrPeak.LagSeconds, 0)); + var whitenedPeak = ReadAutocorrPeak(challenge.Artifacts?.WhitenedSpectralAutocorrCsv); + metrics.Add(new SpeechScoreMetric("target_whitened_spectral_autocorr_peak", whitenedPeak.Correlation, 0)); + metrics.Add(new SpeechScoreMetric("target_whitened_spectral_autocorr_peak_lag_seconds", whitenedPeak.LagSeconds, 0)); return metrics; } @@ -2798,6 +2906,37 @@ static string SongAnalysisReport(SongChallenge challenge, AudioAnalysis analysis builder.AppendLine($"- log_mel_band_stats_csv: `{challenge.Artifacts?.LogMelBandStatsCsv}`"); builder.AppendLine($"- rms_envelope_csv: `{challenge.Artifacts?.RmsEnvelopeCsv}`"); builder.AppendLine($"- rms_envelope_autocorr_csv: `{challenge.Artifacts?.RmsEnvelopeAutocorrCsv}`"); + builder.AppendLine($"- whitened_spectral_autocorr_csv: `{challenge.Artifacts?.WhitenedSpectralAutocorrCsv}`"); + return builder.ToString(); +} + +static string SongRenderAnalysisReport(string label, AudioAnalysis analysis, float frameRate) +{ + var bands = SpectrogramBandStats(analysis.LogMelSpectrogram); + var brightest = bands.OrderByDescending(band => band.Mean).Take(5).Select(band => band.Band).ToArray(); + var mostMoving = bands.OrderByDescending(band => band.AbsDeltaMean).Take(5).Select(band => band.Band).ToArray(); + var spectralAutocorr = Autocorrelation(WhitenedSpectralFluxSeries(analysis.LogMelSpectrogram), frameRate, maxLags: 64) + .OrderByDescending(point => point.Correlation) + .Take(5) + .ToArray(); + var builder = new StringBuilder(); + builder.AppendLine($"# Song Render Analysis: {label}"); + builder.AppendLine(); + builder.AppendLine($"duration_seconds: `{analysis.Features.DurationSeconds:0.######}`"); + builder.AppendLine($"peak: `{analysis.Features.Peak:0.######}`"); + builder.AppendLine($"rms: `{analysis.Features.Rms:0.######}`"); + builder.AppendLine($"zero_crossing_rate: `{analysis.Features.ZeroCrossingRate:0.######}`"); + builder.AppendLine($"spectral_centroid_hz: `{analysis.Features.SpectralCentroidHz:0.######}`"); + builder.AppendLine($"spectral_rolloff_hz: `{analysis.Features.SpectralRolloffHz:0.######}`"); + builder.AppendLine($"brightest_bands: `{string.Join(",", brightest)}`"); + builder.AppendLine($"highest_delta_bands: `{string.Join(",", mostMoving)}`"); + builder.AppendLine(); + builder.AppendLine("## Whitened Spectral Autocorrelation Peaks"); + foreach (var point in spectralAutocorr) + { + builder.AppendLine(CultureInfo.InvariantCulture, $"- lag_frames `{point.LagFrames}`, lag_seconds `{point.LagSeconds:0.######}`, correlation `{point.Correlation:0.######}`"); + } + return builder.ToString(); } @@ -2837,6 +2976,7 @@ static string SongChallengeReport(SongChallenge challenge) => log_mel_band_stats_csv: `{challenge.Artifacts?.LogMelBandStatsCsv ?? ""}` rms_envelope_csv: `{challenge.Artifacts?.RmsEnvelopeCsv ?? ""}` rms_envelope_autocorr_csv: `{challenge.Artifacts?.RmsEnvelopeAutocorrCsv ?? ""}` + whitened_spectral_autocorr_csv: `{challenge.Artifacts?.WhitenedSpectralAutocorrCsv ?? ""}` """; static string SongComparisonReport(SongChallenge challenge, IpaTrialScriptCandidate candidate, AudioComparison comparison, string verdict, SongInstrumentProfile profile) => @@ -3023,6 +3163,7 @@ static void PrintHelp() --embed-model qwen3-embedding:0.6b --collection aquasynth_ipa_trial_results + Use song-prepare --duration-seconds 0 to freeze the full decoded source file from sample zero. Agent-authored patch files must be named __candidate-name.aqua. Known seed target ids: a, i, u, e, o, m, n, ng, l, r, s, z, f, v, th, p, b, t, d, k, mix-a, mix-m, mix-s, mix-p, mix-u. """); @@ -3059,8 +3200,11 @@ sealed record SongChallengeAnalysisArtifacts( string LogMelBandStatsCsv, string RmsEnvelopeCsv, string RmsEnvelopeAutocorrCsv, + string WhitenedSpectralAutocorrCsv, string AnalysisReportMarkdown); +sealed record SongRenderAnalysisArtifacts(string Kind, string Path); + sealed record SongChallengeFeatures( float DurationSeconds, float Peak, diff --git a/tools/run-song-snippet-trial-swarm.ps1 b/tools/run-song-snippet-trial-swarm.ps1 index 5688a4a..2ab1eaa 100644 --- a/tools/run-song-snippet-trial-swarm.ps1 +++ b/tools/run-song-snippet-trial-swarm.ps1 @@ -5,6 +5,8 @@ param( [string]$Source = "D:\Music\Reinier\Heyoka\Gate Code\Heyoka - Alien Gibberish.mp3", [string]$SourceFolder = "D:\Music\Reinier\Heyoka", [float]$DurationSeconds = 30, + [switch]$FullSongs, + [int]$IterationsPerTarget = 5, [int]$Seed = 0, [switch]$NewSegmentPerPass, [switch]$RandomSourcePerAgent, @@ -144,7 +146,7 @@ function Get-SongSources { if ($SourceFolder.Length -gt 0) { $files = @(Get-ChildItem -LiteralPath $SourceFolder -File -Recurse | - Where-Object { $_.Extension -in @(".mp3", ".wav", ".flac", ".aiff", ".aif", ".ogg") } | + Where-Object { $_.Extension -in @(".mp3", ".wav", ".flac", ".aiff", ".aif", ".ogg", ".m4a", ".mp4") } | Sort-Object FullName) if ($files.Count -eq 0) { throw "No supported song files found in '$SourceFolder'." @@ -246,9 +248,11 @@ Set-Content -LiteralPath (Join-Path $loopRoot "loop-index.md") -Value @" - passes: $Passes - agents_per_pass: $AgentsPerPass - songs_per_agent: $SongsPerAgent +- iterations_per_target: $IterationsPerTarget - playlist_top_count: $PlaylistTopCount - seed: $seedValue - duration_seconds: $DurationSeconds +- full_songs: $FullSongs - new_segment_per_pass: $NewSegmentPerPass - random_source_per_agent: $RandomSourcePerAgent - store: $storePath @@ -317,6 +321,7 @@ for ($pass = 1; $pass -le $Passes; $pass++) { } $agentSource = $songSources[[int]$sourceIndex] $agentSeed = $phaseSeed + ($agent * 104729) + ($song * 8191) + $durationArg = if ($FullSongs) { "0" } else { [string]$DurationSeconds } Invoke-LoggedProcess ` -FilePath "dotnet" ` -ArgumentList @( @@ -330,7 +335,7 @@ for ($pass = 1; $pass -le $Passes; $pass++) { "--artifact-root", $agentChallengeRoot, "--duration-seconds", - ([string]$DurationSeconds), + $durationArg, "--seed", ([string]$agentSeed), "--challenge-id", @@ -350,6 +355,9 @@ for ($pass = 1; $pass -le $Passes; $pass++) { $targetPatchList = ($targetPatchDirs | ForEach-Object { "- target patch output directory: $_" }) -join "`n" $agentSourceList = ($agentSources | ForEach-Object { "- assigned source: $_" }) -join "`n" $agentSeedList = ($agentSeeds | ForEach-Object { "- target seed: $_" }) -join "`n" + $durationLabel = if ($FullSongs) { "the full decoded song duration from each challenge report" } else { "$DurationSeconds seconds" } + $targetKind = if ($FullSongs) { "full-song" } else { "$DurationSeconds second" } + $iterationRoot = Join-Path $agentDir "iterations" $outputPath = Join-Path $agentDir "hypothesis-agent.md" $promptPath = Join-Path $agentDir "hypothesis-agent.prompt.md" $logPath = Join-Path $logRoot "$passId-$agentId-hypothesis-agent.log" @@ -367,21 +375,32 @@ $targetPatchList - candidate filename prefix: $agentId - phase id: $passId - phase seed: $phaseSeed +- private iteration root: $iterationRoot +- required self-iteration attempts per target: $IterationsPerTarget $agentSourceList $agentSeedList -- required duration: $DurationSeconds seconds +- required duration: $durationLabel Goal: -Write exactly one parseable AquaSynth `.aqua` patch for each frozen $DurationSeconds second reference clip assigned to you. This is scene-audio parity, not IPA articulation. You may use ordinary voices, FM, AM/tremolo, filters, formants, noise layers, acoustic vocal/syrinx primitives, additive/PAD layers, curves, and helper voices. +Write exactly one parseable AquaSynth `.aqua` patch for each frozen $targetKind reference assigned to you. This is scene-audio parity, not IPA articulation. You may use ordinary voices, FM, AM/tremolo, filters, formants, noise layers, acoustic vocal/syrinx primitives, additive/PAD layers, curves, and helper voices. Generalization: - You own two target patches, but your report must compare both targets and state what reusable scene/instrument knowledge transfers between them. - The two patches may share instrument roles, texture roles, control idioms, and DSL conventions, but each patch should fit its own target's tempo/register/spectral evidence. - Future agents will receive distilled evidence from this trial when asked to zero-shot an audio production request. Make your `hypotheses.md` useful retrieval context: name what transferred, what did not, and what evidence would change your next patch. +Self-iteration loop: +- For each target, run exactly $IterationsPerTarget local attempts before publishing the final `.aqua` file. +- Each attempt must write a candidate under `$iterationRoot/attempt-NN//patch`, run the scoring worker against that one target, inspect the rendered candidate evidence, and revise the next attempt. +- Scoring command pattern: + `dotnet run --project "$workerProject" -- song-score --patch-root "" --challenge "" --artifact-root "" --batch-id "--attempt-NN" --store "/iteration-results.cc" --hypothesizer "$agentId-iteration"` +- After each score, read the attempt's `evaluator-report.md`, `summary.csv`, `audio/comparison.txt`, `audio/candidate-analysis.md`, `audio/candidate-logmel-band-stats.csv`, `audio/candidate-rms-envelope-autocorr.csv`, and `audio/candidate-whitened-spectral-autocorr.csv`. +- Use those candidate-side facts the same way you use target facts: compare spectral bands, envelope autocorrelation, whitened spectral autocorrelation, centroid/RMS, instrument-role metrics, and chip-distress risk. Move timings, filters, gains, sources, roles, and patterns between attempts. +- Keep intermediate candidates inside `$iterationRoot`; publish only one final `.aqua` per target to the official target patch output directory. + Noise and texture: - Do not model background or recording color as a full-duration raw `voice wave=noise` bed. That produces static white-noise wash and will be treated as broken subtractive synthesis, not scene modeling. -- Use the shaped texture owner for noise roles: `texture name=dust_hat role=dust pattern=x..x step= gain=.08 sustain=$DurationSeconds`, `texture name=air_wash role=air gain=.035 sustain=$DurationSeconds`, `texture name=codec_bed role=codec gain=.04 sustain=$DurationSeconds`. +- Use the shaped texture owner for noise roles: `texture name=dust_hat role=dust pattern=x..x step= gain=.08 sustain=`, `texture name=air_wash role=air gain=.035 sustain=`, `texture name=codec_bed role=codec gain=.04 sustain=`. - `texture` lowers to a gated noise voice with role-specific bandpass/highpass/lowpass/tremolo defaults and a control curve or pattern gate. Raw `voice wave=noise` is only acceptable for short transients with explicit gates and narrow filters. Instrument ownership: @@ -430,7 +449,7 @@ Evidence contract: - Write exactly one `.aqua` file under each target patch output directory named `____.aqua`. - Include at least three scene voices or layers in the patch: a primary voice/body, a rhythmic drum/transient/noise role, and a pad/bed/recording-color role. Prefer syrinx for the primary voice when it is voice-like, subtractive for the drum/transient, and additive/PAD or shaped texture for the bed. - Background/recording-color layers must use `texture` or an equivalently gated and band-shaped explicit voice; a static full-duration white-noise bed is failed evidence. -- Make each patch duration/gates cover its matching $DurationSeconds second target. +- Make each patch duration/gates cover its matching target duration from the challenge report. Return a short final summary naming the patch and report. "@