GameCult/AquaSynth
Watch
9
0
Fork
You've already forked AquaSynth
 0
Code Issues Pull requests 4 Projects Releases Packages Wiki Activity Actions

Let song agents iterate on rendered features

Browse source
This commit is contained in:
Meta Vens 2026-05-31 13:09:57 +01:00
commit 4b40d12816
6 changed files with 213 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

18
docs/audio-doctrine.md
View file

@ -320,11 +320,19 @@ Useful metrics:
Song-scene parity targets must publish target evidence before candidate golf:
log-mel spectrogram CSV, per-band statistics, first and second derivative
summaries, RMS envelope, and envelope autocorrelation. Those artifacts are not
loose side files only; scoring writes them as typed song challenge evidence
records in the `.cc` CultCache/CultMesh store and references them from each
trial result. Qdrant/Ollama retrieval may derive vectors from those records, but
the `.cc` records remain truth.
summaries, RMS envelope, envelope autocorrelation, and whitened spectral
autocorrelation over band-normalized log-mel flux. Those artifacts are not loose
side files only; scoring writes them as typed song challenge evidence records in
the `.cc` CultCache/CultMesh store and references them from each trial result.
Qdrant/Ollama retrieval may derive vectors from those records, but the `.cc`
records remain truth.
Song-scene agents should iterate like a tiny studio. Render the candidate,
extract the same features from the Faust output, compare target and candidate
band motion, envelope autocorrelation, whitened spectral autocorrelation,
loudness, centroid, and instrument-role metrics, then move sources, filters,
gates, patterns, and gains. A one-shot patch is sample data; a five-attempt
revision trace is curriculum.
Noise is not a scene by itself. Background air, dust, codec grit, tape, vinyl,
and room color need a role owner that shapes spectrum and time. Prefer

1
state/evidence.jsonl
View file

@ -257,3 +257,4 @@
{"date":"2026-05-31","area":"song-curriculum-paired-pass","lesson":"A paired song-curriculum pass completed with five agents, two targets per agent, one patch per target, ten scored trials, ten target evidence bundles, sixty Qdrant chunks, and a distilled `.cc` store. Metrics were all `pressure`: best log-mel cosine `0.502128`, mean `0.377955`, worst `0.27853`, mean audio score `0.339432`. Distillation kept all ten and identified clocked dust/hat texture plus rubber/sub bass as reusable roles, while preserving the shaped-noise owner requirement as a transfer rule rather than letting static white hiss back into the curriculum.","evidence":"artifacts/parity/song-snippet-swarms/20260531T074935298/song-trial-results.cc; artifacts/parity/song-snippet-swarms/20260531T074935298/pass-001/distillation-report.md; artifacts/parity/song-snippet-swarms/20260531T074935298/pass-001/vector-index-after.md; artifacts/parity/song-snippet-swarms/20260531T074935298/pass-001/semantic-search-after.md"}
{"date":"2026-05-31","area":"song-curriculum-playlists-and-invalid-candidates","lesson":"Song curriculum passes now publish a playable top-candidate playlist per pass and keep invalid agent syntax as negative evidence instead of aborting the swarm. A two-pass paired run produced pass-001 best cosine `0.386098` with 8 playable candidates and 2 `render-failed` syntax casualties, then pass-002 improved to best cosine `0.502435`, mean cosine `0.352907`, 10 playable candidates, and 8 pressure verdicts after reading distilled pass-1 evidence. Playlist reports filter to candidates with actual rendered WAVs and list skipped non-playable failures separately.","evidence":"tools/run-song-snippet-trial-swarm.ps1; tools/IpaTrialWorker/Program.cs; artifacts/parity/song-snippet-swarms/20260531T084455803/pass-001/top-scoring-candidates.m3u; artifacts/parity/song-snippet-swarms/20260531T084455803/pass-002/top-scoring-candidates.m3u; artifacts/parity/song-snippet-swarms/20260531T084455803/pass-002/distillation-report.md"}
{"date":"2026-05-31","area":"song-instrument-owner-scoring","lesson":"The song curriculum harness must teach instrument ownership, not only spectrogram proximity. `song-score` now records syrinx-voice, subtractive-drum, additive/PAD, musical-instrument-score, and chip-distress-risk metrics for every candidate; swarm prompts require voice-like leads to prefer syrinx/acoustic sources, percussion to prefer subtractive pitched body plus filtered noise skin, and pads/beds to prefer additive/PAD banks. Distillation ranking now rewards owned roles and demotes chip-distress-risk candidates unless they are explicit contrast evidence.","evidence":"tools/IpaTrialWorker/Program.cs; tools/run-song-snippet-trial-swarm.ps1; docs/audio-doctrine.md; patches/advanced/syrinx-voice.aqua; patches/advanced/subtractive-drums.aqua; patches/advanced/additive-pad.aqua; artifacts/parity/song-instrument-metric-smoke/smoke/summary.csv"}
{"date":"2026-05-31","area":"song-full-iteration-loop","lesson":"Song agents need a studio loop, not a one-shot submission. `song-prepare --duration-seconds 0` freezes full decoded songs; the swarm exposes `-FullSongs`; song challenges and candidate renders now include whitened spectral autocorrelation over band-whitened log-mel flux; and the prompt requires five local score/revise attempts per target before publishing a final patch. Candidate-side analysis artifacts let agents compare their Faust render against the target using the same feature vocabulary instead of guessing from the final verdict.","evidence":"tools/IpaTrialWorker/Program.cs; tools/run-song-snippet-trial-swarm.ps1; artifacts/parity/song-candidate-feature-smoke-2; artifacts/parity/song-snippet-swarms/20260531T120800752"}

26
state/scratch.md
View file

@ -1829,3 +1829,29 @@ Next likely slice:
`artifacts/parity/song-instrument-metric-smoke/smoke/summary.csv` records a
syrinx candidate with `instrument_voice_syrinx=1`,
`musical_instrument_score=0.34`, and `chip_distress_risk=0`.
2026-05-31 full-song iterative harness slice:
- `song-prepare --duration-seconds 0` now means "freeze the full decoded source
file from sample zero" instead of a random excerpt. `run-song-snippet-trial-swarm.ps1`
has `-FullSongs`, keeps the source folder recursive, and widened local audio
discovery to `.mp3/.wav/.flac/.aiff/.aif/.ogg/.m4a/.mp4`. Current Reinier
parent corpus discovery sees 2421 supported audio files.
- Song challenge analysis now includes `whitened-spectral-autocorr.csv`, derived
from autocorrelation of a band-whitened log-mel spectral-flux series. This is
intended to expose repeating spectral/rhythmic structure after broad tone
color is normalized away.
- `song-score` writes candidate-side render analysis artifacts from Faust output:
`candidate-analysis.md`, `candidate-logmel-spectrogram.csv`,
`candidate-logmel-band-stats.csv`, `candidate-rms-envelope.csv`,
`candidate-rms-envelope-autocorr.csv`, and
`candidate-whitened-spectral-autocorr.csv`. Agents can now inspect their own
render with the same feature vocabulary used for the target.
- The swarm prompt now requires an in-agent studio loop: each worker must run
five local attempts per target, call `song-score` on each attempt, read the
candidate analysis/comparison artifacts, move timings/filters/gains/sources,
and publish only one final `.aqua` per target to the official output folder.
- Verification: `dotnet build tools/IpaTrialWorker/IpaTrialWorker.csproj`
passed; zero-pass full-song swarm syntax smoke passed; candidate feature
smoke wrote the new candidate autocorrelation and analysis artifacts under
`artifacts/parity/song-candidate-feature-smoke-2`.

1
state/spine.yaml
View file

@ -150,6 +150,7 @@ live_invariants:
- External IPA exploration now has a worker/tool boundary. `tools/IpaTrialWorker` owns `seed`, `score`, `search`, `show`, and audit-only `dump` commands over the typed `.cc` trial-result store; `score` renders external Codex-authored `.aqua` candidates through `IpaTrialOrchestrator.RunCandidateScriptsAsync`, while `search` is the semantic retrieval organ for hypothesis agents. `IpaTrialResult` records now embed extracted primitive timeline facts from `ProbeTimelineReport` for contact closure/reservoir/release, source load/flow, branch admittance/exchanged flow, radiation boundary/output, path area/delay/energy, and passivity; `search`, `show`, `dump`, and summary CSV output surface release/radiation/passivity witnesses directly from the `.cc` record. `search` follows VoidBot's Qdrant shape: `.cc` records plus timeline facts and referenced local artifacts become evidence chunks, local Ollama `qwen3-embedding:0.6b` produces 1024-dim vectors, Qdrant collection `aquasynth_ipa_trial_results` stores stable SHA-256 UUID points with trial/candidate payload metadata, and the report surfaces vector scores, excerpts, tags, metric facts, timeline facts, contrast candidates, `class_focus`, per-hit `class_match`, and `show` commands. Qdrant is a derived index; CultCache remains truth, and lexical/metric search is fallback when the vector stack is unavailable. Retrieval tuning added phonetic class-affinity scoring and candidate-specific mixed-lane ownership so `mix-p` can support stop seams without every mixed candidate matching stop evidence. `tools/run-ipa-trial-loop.ps1` orchestrates external `codex exec` hypothesis/evaluator workers around that store. Source-edit-enabled rounds now allow agents to golf DSL syntax, parser/model records, Faust lowering, worker evidence surfaces, and tests when evidence points at a repeated owner failure; agents must write `source-change-plan.md`, and the loop runs a focused build before scoring. `hypotheses.md` is the hypothesis worker report of record; `hypothesis-agent.md` is only the external agent's final chat summary. The first loose source-edit round produced no source edits and no aggregate metric win, then exposed a deeper learning-loop fault: family ideas were rotated across unrelated targets instead of aligned to phonetic owners. Production prompts now pass the previous evaluation path into the hypothesizer, require `Previous Round Learning`, require `Target-Class Lane Plan`, forbid round-robin family assignment, and let the evaluator fail class alignment when family/target assignments do not match phonetic owner classes or explicit transfer-test mutations. Candidate reports must include a `Scene Voice Model` naming primary-articulatory, tract-noise, release/transient, background-room, and recording-condition voices, with general AquaSynth synth primitives allowed for helper scene voices but not accepted as articulation evidence unless lower-layer witnesses move. At least one scene/noise or recording-condition family in a source-golf round must implement real auxiliary `.aqua` `voice` declarations instead of satisfying scene modeling with prose. The active trial shape is five target sets of five phoneme lanes each: vowels, nasals/approximants, fricatives, stops, and mixed generalization. Candidate names are exactly `<targetId>__<family>__<hypothesis-name>.aqua`, with lowercase kebab-case family ids. Three dry external Codex smoke passes refined hypothesizer/evaluator prompts into exact 25-lane output, retrieval receipts, claim audits, evidence-quality ledgers/verdicts, and fail-closed evaluator compliance. A later five-pass prompt optimization showed the best science comes from fixed receipt packets plus constrained reasoning; production prompts now require contrast pairs, `show` metric values, primitive timeline facts/excerpts or explicit `missing`, class-consistent reference matrices, scene voice models, `X owns Y so Z remains true` owner sentences, at least five micro-sweep perturbations before a seam is actionable, and a novelty gate rejecting loudness/dressing-only families as loss-landscape novelty. Verification: focused builds/tests pass, old-store show compatibility passes, and a fresh one-candidate smoke under `artifacts/parity/ipa-timeline-fact-smoke/20260531T233217641` proves release/radiation/passivity facts persist and retrieve through Qdrant/Ollama.
- Song-snippet parity now treats target analysis as CultMesh database evidence. `song-prepare` writes each 30-second clip's log-mel spectrogram CSV, per-band first/second derivative stats, RMS envelope, envelope autocorrelation, and analysis report. `song-score` upserts those as typed `SongChallengeEvidenceDocument` records in the same `.cc` store as `IpaTrialResult`, adds target metrics and `cultmesh://aquasynth/song-challenge-evidence/...` artifacts to every result, and `search/index` derive Qdrant chunks from the database records. `pattern` and `scale` DSL sugar are live authoring conveniences that lower into normal parameters and hold-loop curves; prompt agents may propose richer Strudel-like syntax, but candidate patches must use implemented syntax.
- Song-scene scoring now records instrument-role ownership so the music-generator curriculum does not learn that simple oscillator blips are good instruments by default. `song-score` emits `instrument_voice_syrinx`, `instrument_drum_subtractive`, `instrument_pad_additive`, `musical_instrument_score`, and `chip_distress_risk`; verdict/distillation ranking rewards owned roles and demotes chip-distress-risk candidates unless preserved as contrast. Swarm prompts now require voice-like leads to prefer syrinx/acoustic sources with pressure/opening/radiation motion, percussion to prefer subtractive pitched bodies plus filtered noise skins, and pads/beds to prefer additive/PAD `layer`/`harmonics`/`spectrum` banks.
- Song-scene harness supports full-song curriculum and in-agent revision. `song-prepare --duration-seconds 0` freezes the whole decoded song from sample zero, and `run-song-snippet-trial-swarm.ps1 -FullSongs` samples recursively from the Reinier parent folder including `.m4a/.mp4`. Target and candidate analysis now include whitened spectral autocorrelation over band-whitened log-mel flux. Candidate-side Faust-render artifacts include log-mel spectrogram/stats, RMS envelope/autocorr, whitened spectral autocorr, and an analysis markdown report. Swarm prompts require five local score/revise attempts per target before publishing the final `.aqua`.
- TurbulenceJet now includes a release-timed burst-noise component in addition to the deterministic pressure pulse, but a stronger trial barely moved papa; the source is not the main remaining articulation bottleneck. Graph radiation now mixes 30% raw boundary flow with 70% high-passed flow after local admittance weighting, down from the over-bright 15/85 blend. Latest radiation/source artifact under `artifacts/parity/pink-trombone-utterance-logmel/20260527T174236049`: mama/papa/thrombosis cosine 0.6578/0.7692/0.5585, rmsRatio 3.0006/2.6153/2.3488, articulation 0.1437/0.1173/0.1191. Static `closure-release` under `artifacts/parity/pink-trombone-logmel/20260527T174139661` has rmsRatio 0.9900 and speechBandRatio 0.0842. This is still not speaking; next pressure is true tract resonance/passive scattering and pressure storage, not more burst gain.
- Same-path two-port area discontinuities now emit `graph_area_energy_in_*` and `graph_area_energy_out_*` probes alongside the three-port branch `graph_connection_energy_*` probes. Use these to inspect passivity and formant-body failures at the scatter sites rather than judging only final output loudness.
- `FaustExportOptions.DebugProbeUi` wraps graph probe signals in Faust `vbargraph` UI controls under `/debug/...` without changing audio output arity. Native streaming Faust patches now separate writable controls from read-only probe zones and can read/snapshot probe values after `ProcessBlock`; use this for acoustic graph passivity/radiation inspection, leave it off for normal parity renders.

150
tools/IpaTrialWorker/Program.cs
View file

@ -109,10 +109,13 @@ static async Task SongPrepareAsync(Dictionary<string, string> options)
Directory.CreateDirectory(artifactRoot);
var decoded = DecodeMono(source);
var resampled = Resample(decoded.Samples, decoded.SampleRate, sampleRate);
var clipLength = Math.Min(resampled.Length, Math.Max(1, (int)MathF.Round(durationSeconds * sampleRate)));
var fullSong = durationSeconds <= 0;
var clipLength = fullSong
? resampled.Length
: Math.Min(resampled.Length, Math.Max(1, (int)MathF.Round(durationSeconds * sampleRate)));
var maxStart = Math.Max(0, resampled.Length - clipLength);
var random = new Random(seed);
var startSample = maxStart == 0 ? 0 : random.Next(0, maxStart + 1);
var startSample = fullSong || maxStart == 0 ? 0 : random.Next(0, maxStart + 1);
var clip = new float[clipLength];
Array.Copy(resampled, startSample, clip, 0, clipLength);
NormalizePeak(clip, .9f);
@ -127,11 +130,13 @@ static async Task SongPrepareAsync(Dictionary<string, string> options)
var bandStatsPath = Path.Combine(artifactRoot, "logmel-band-stats.csv");
var envelopePath = Path.Combine(artifactRoot, "rms-envelope.csv");
var envelopeAutocorrPath = Path.Combine(artifactRoot, "rms-envelope-autocorr.csv");
var whitenedSpectralAutocorrPath = Path.Combine(artifactRoot, "whitened-spectral-autocorr.csv");
var analysisReportPath = Path.Combine(artifactRoot, "analysis.md");
await File.WriteAllTextAsync(spectrogramPath, SpectrogramCsv(analysis.LogMelSpectrogram), Encoding.UTF8);
await File.WriteAllTextAsync(bandStatsPath, SpectrogramBandStatsCsv(analysis.LogMelSpectrogram), Encoding.UTF8);
await File.WriteAllTextAsync(envelopePath, EnvelopeCsv(analysis.RmsEnvelope, sampleRate, analyzer.Config.HopSize), Encoding.UTF8);
await File.WriteAllTextAsync(envelopeAutocorrPath, AutocorrelationCsv(analysis.RmsEnvelope, sampleRate / (float)Math.Max(1, analyzer.Config.HopSize)), Encoding.UTF8);
await File.WriteAllTextAsync(whitenedSpectralAutocorrPath, WhitenedSpectralAutocorrelationCsv(analysis.LogMelSpectrogram, sampleRate / (float)Math.Max(1, analyzer.Config.HopSize)), Encoding.UTF8);
var features = new SongChallengeFeatures(
analysis.Features.DurationSeconds,
analysis.Features.Peak,
@ -155,6 +160,7 @@ static async Task SongPrepareAsync(Dictionary<string, string> options)
bandStatsPath,
envelopePath,
envelopeAutocorrPath,
whitenedSpectralAutocorrPath,
analysisReportPath);
var challenge = new SongChallenge(
challengeId,
@ -263,6 +269,13 @@ static async Task SongScoreAsync(Dictionary<string, string> options)
NormalizePeak(matched, .9f);
WriteWav(candidateWav, matched, challenge.SampleRate);
artifacts.Add(Artifact("candidate-wav", candidateWav));
var candidateAnalysisArtifacts = await WriteSongRenderAnalysisAsync(
matched,
challenge.SampleRate,
audioDir,
"candidate",
analyzer);
artifacts.AddRange(candidateAnalysisArtifacts.Select(ArtifactFromAnalysis));
var comparison = analyzer.Compare(referenceSamples, matched);
metrics.AddRange(SongComparisonMetrics(comparison));
verdict = SongVerdict(comparison, instrumentProfile);
@ -303,7 +316,9 @@ static async Task SongScoreAsync(Dictionary<string, string> options)
verdict,
[
"This is a local-only song clip challenge; reference audio is not redistributed by the repo.",
$"The target is a randomly selected {challenge.DurationSeconds:0.###}-second scene-audio snippet, not IPA articulation truth.",
challenge.StartSeconds <= 0.001f
? $"The target starts at the source file beginning and lasts {challenge.DurationSeconds:0.###} seconds; in full-song runs this is the whole decoded song, not IPA articulation truth."
: $"The target is a randomly selected {challenge.DurationSeconds:0.###}-second scene-audio snippet, not IPA articulation truth.",
"Challenge spectrogram, derivative, envelope, and autocorrelation artifacts are stored as typed CultMesh/CultCache evidence documents in the .cc database.",
"Full-patch FM/AM/noise/scene modeling is allowed; primitive timeline facts remain diagnostic when present.",
instrumentProfile.ChipDistressRisk >= .6f
@ -1934,6 +1949,10 @@ static IEnumerable<SpeechRenderArtifact> SongChallengeArtifacts(SongChallenge ch
yield return Artifact("target-logmel-band-stats", challenge.Artifacts.LogMelBandStatsCsv);
yield return Artifact("target-rms-envelope", challenge.Artifacts.RmsEnvelopeCsv);
yield return Artifact("target-rms-envelope-autocorr", challenge.Artifacts.RmsEnvelopeAutocorrCsv);
if (!string.IsNullOrWhiteSpace(challenge.Artifacts.WhitenedSpectralAutocorrCsv))
{
yield return Artifact("target-whitened-spectral-autocorr", challenge.Artifacts.WhitenedSpectralAutocorrCsv);
}
}
static SpeechRenderArtifact SongChallengeEvidenceArtifact(SongChallengeEvidenceDocument document) =>
@ -1942,6 +1961,9 @@ static SpeechRenderArtifact SongChallengeEvidenceArtifact(SongChallengeEvidenceD
$"cultmesh://aquasynth/song-challenge-evidence/{Uri.EscapeDataString(document.EvidenceId)}",
document.ContentHash);
static SpeechRenderArtifact ArtifactFromAnalysis(SongRenderAnalysisArtifacts artifact) =>
Artifact($"candidate-{artifact.Kind}", artifact.Path);
static IReadOnlyList<SongChallengeEvidenceDocument> SongChallengeEvidenceDocuments(SongChallenge challenge)
{
var documents = new List<SongChallengeEvidenceDocument>();
@ -1955,6 +1977,7 @@ static IReadOnlyList<SongChallengeEvidenceDocument> SongChallengeEvidenceDocumen
AddSongChallengeEvidence(documents, challenge, "logmel-band-stats", "text/csv", challenge.Artifacts.LogMelBandStatsCsv);
AddSongChallengeEvidence(documents, challenge, "rms-envelope", "text/csv", challenge.Artifacts.RmsEnvelopeCsv);
AddSongChallengeEvidence(documents, challenge, "rms-envelope-autocorr", "text/csv", challenge.Artifacts.RmsEnvelopeAutocorrCsv);
AddSongChallengeEvidence(documents, challenge, "whitened-spectral-autocorr", "text/csv", challenge.Artifacts.WhitenedSpectralAutocorrCsv);
return documents;
}
@ -1981,6 +2004,35 @@ static void AddSongChallengeEvidence(
DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture)));
}
static async Task<IReadOnlyList<SongRenderAnalysisArtifacts>> WriteSongRenderAnalysisAsync(
IReadOnlyList<float> samples,
int sampleRate,
string artifactRoot,
string prefix,
AudioAnalyzer analyzer)
{
Directory.CreateDirectory(artifactRoot);
var analysis = analyzer.Analyze(samples.ToArray());
var frameRate = sampleRate / (float)Math.Max(1, analyzer.Config.HopSize);
var artifacts = new[]
{
new SongRenderAnalysisArtifacts("analysis-report", Path.Combine(artifactRoot, $"{prefix}-analysis.md")),
new SongRenderAnalysisArtifacts("logmel-spectrogram", Path.Combine(artifactRoot, $"{prefix}-logmel-spectrogram.csv")),
new SongRenderAnalysisArtifacts("logmel-band-stats", Path.Combine(artifactRoot, $"{prefix}-logmel-band-stats.csv")),
new SongRenderAnalysisArtifacts("rms-envelope", Path.Combine(artifactRoot, $"{prefix}-rms-envelope.csv")),
new SongRenderAnalysisArtifacts("rms-envelope-autocorr", Path.Combine(artifactRoot, $"{prefix}-rms-envelope-autocorr.csv")),
new SongRenderAnalysisArtifacts("whitened-spectral-autocorr", Path.Combine(artifactRoot, $"{prefix}-whitened-spectral-autocorr.csv"))
};
await File.WriteAllTextAsync(artifacts[0].Path, SongRenderAnalysisReport(prefix, analysis, frameRate), Encoding.UTF8);
await File.WriteAllTextAsync(artifacts[1].Path, SpectrogramCsv(analysis.LogMelSpectrogram), Encoding.UTF8);
await File.WriteAllTextAsync(artifacts[2].Path, SpectrogramBandStatsCsv(analysis.LogMelSpectrogram), Encoding.UTF8);
await File.WriteAllTextAsync(artifacts[3].Path, EnvelopeCsv(analysis.RmsEnvelope, sampleRate, analyzer.Config.HopSize), Encoding.UTF8);
await File.WriteAllTextAsync(artifacts[4].Path, AutocorrelationCsv(analysis.RmsEnvelope, frameRate), Encoding.UTF8);
await File.WriteAllTextAsync(artifacts[5].Path, WhitenedSpectralAutocorrelationCsv(analysis.LogMelSpectrogram, frameRate), Encoding.UTF8);
return artifacts;
}
static AutocorrelationPoint ReadAutocorrPeak(string? path)
{
if (string.IsNullOrWhiteSpace(path) || !File.Exists(path))
@ -2571,6 +2623,59 @@ static string AutocorrelationCsv(IReadOnlyList<float> values, float frameRate)
return builder.ToString();
}
static string WhitenedSpectralAutocorrelationCsv(Spectrogram spectrogram, float frameRate)
{
var series = WhitenedSpectralFluxSeries(spectrogram);
var builder = new StringBuilder();
builder.AppendLine("lag_frames,lag_seconds,correlation");
foreach (var point in Autocorrelation(series, frameRate, Math.Min(Math.Max(1, series.Length - 1), 512)))
{
builder.Append(point.LagFrames).Append(',');
builder.Append(point.LagSeconds.ToString("0.########", CultureInfo.InvariantCulture)).Append(',');
builder.AppendLine(point.Correlation.ToString("0.########", CultureInfo.InvariantCulture));
}
return builder.ToString();
}
static float[] WhitenedSpectralFluxSeries(Spectrogram spectrogram)
{
if (spectrogram.Frames <= 1 || spectrogram.Bands <= 0)
{
return [];
}
var means = new float[spectrogram.Bands];
var stddevs = new float[spectrogram.Bands];
for (var band = 0; band < spectrogram.Bands; band++)
{
var values = new float[spectrogram.Frames];
for (var frame = 0; frame < spectrogram.Frames; frame++)
{
values[frame] = spectrogram.At(frame, band);
}
means[band] = Mean(values);
stddevs[band] = Math.Max(1e-4f, StdDev(values));
}
var series = new float[spectrogram.Frames - 1];
for (var frame = 1; frame < spectrogram.Frames; frame++)
{
var sum = 0f;
for (var band = 0; band < spectrogram.Bands; band++)
{
var current = (spectrogram.At(frame, band) - means[band]) / stddevs[band];
var previous = (spectrogram.At(frame - 1, band) - means[band]) / stddevs[band];
sum += MathF.Abs(current - previous);
}
series[frame - 1] = sum / Math.Max(1, spectrogram.Bands);
}
return series;
}
static IReadOnlyList<AutocorrelationPoint> Autocorrelation(IReadOnlyList<float> values, float frameRate, int maxLags)
{
if (values.Count < 2)
@ -2645,6 +2750,9 @@ static IReadOnlyList<SpeechScoreMetric> SongTargetMetrics(
var autocorrPeak = ReadAutocorrPeak(challenge.Artifacts?.RmsEnvelopeAutocorrCsv);
metrics.Add(new SpeechScoreMetric("target_envelope_autocorr_peak", autocorrPeak.Correlation, 0));
metrics.Add(new SpeechScoreMetric("target_envelope_autocorr_peak_lag_seconds", autocorrPeak.LagSeconds, 0));
var whitenedPeak = ReadAutocorrPeak(challenge.Artifacts?.WhitenedSpectralAutocorrCsv);
metrics.Add(new SpeechScoreMetric("target_whitened_spectral_autocorr_peak", whitenedPeak.Correlation, 0));
metrics.Add(new SpeechScoreMetric("target_whitened_spectral_autocorr_peak_lag_seconds", whitenedPeak.LagSeconds, 0));
return metrics;
}
@ -2798,6 +2906,37 @@ static string SongAnalysisReport(SongChallenge challenge, AudioAnalysis analysis
builder.AppendLine($"- log_mel_band_stats_csv: `{challenge.Artifacts?.LogMelBandStatsCsv}`");
builder.AppendLine($"- rms_envelope_csv: `{challenge.Artifacts?.RmsEnvelopeCsv}`");
builder.AppendLine($"- rms_envelope_autocorr_csv: `{challenge.Artifacts?.RmsEnvelopeAutocorrCsv}`");
builder.AppendLine($"- whitened_spectral_autocorr_csv: `{challenge.Artifacts?.WhitenedSpectralAutocorrCsv}`");
return builder.ToString();
}
static string SongRenderAnalysisReport(string label, AudioAnalysis analysis, float frameRate)
{
var bands = SpectrogramBandStats(analysis.LogMelSpectrogram);
var brightest = bands.OrderByDescending(band => band.Mean).Take(5).Select(band => band.Band).ToArray();
var mostMoving = bands.OrderByDescending(band => band.AbsDeltaMean).Take(5).Select(band => band.Band).ToArray();
var spectralAutocorr = Autocorrelation(WhitenedSpectralFluxSeries(analysis.LogMelSpectrogram), frameRate, maxLags: 64)
.OrderByDescending(point => point.Correlation)
.Take(5)
.ToArray();
var builder = new StringBuilder();
builder.AppendLine($"# Song Render Analysis: {label}");
builder.AppendLine();
builder.AppendLine($"duration_seconds: `{analysis.Features.DurationSeconds:0.######}`");
builder.AppendLine($"peak: `{analysis.Features.Peak:0.######}`");
builder.AppendLine($"rms: `{analysis.Features.Rms:0.######}`");
builder.AppendLine($"zero_crossing_rate: `{analysis.Features.ZeroCrossingRate:0.######}`");
builder.AppendLine($"spectral_centroid_hz: `{analysis.Features.SpectralCentroidHz:0.######}`");
builder.AppendLine($"spectral_rolloff_hz: `{analysis.Features.SpectralRolloffHz:0.######}`");
builder.AppendLine($"brightest_bands: `{string.Join(",", brightest)}`");
builder.AppendLine($"highest_delta_bands: `{string.Join(",", mostMoving)}`");
builder.AppendLine();
builder.AppendLine("## Whitened Spectral Autocorrelation Peaks");
foreach (var point in spectralAutocorr)
{
builder.AppendLine(CultureInfo.InvariantCulture, $"- lag_frames `{point.LagFrames}`, lag_seconds `{point.LagSeconds:0.######}`, correlation `{point.Correlation:0.######}`");
}
return builder.ToString();
}
@ -2837,6 +2976,7 @@ static string SongChallengeReport(SongChallenge challenge) =>
log_mel_band_stats_csv: `{challenge.Artifacts?.LogMelBandStatsCsv ?? ""}`
rms_envelope_csv: `{challenge.Artifacts?.RmsEnvelopeCsv ?? ""}`
rms_envelope_autocorr_csv: `{challenge.Artifacts?.RmsEnvelopeAutocorrCsv ?? ""}`
whitened_spectral_autocorr_csv: `{challenge.Artifacts?.WhitenedSpectralAutocorrCsv ?? ""}`
""";
static string SongComparisonReport(SongChallenge challenge, IpaTrialScriptCandidate candidate, AudioComparison comparison, string verdict, SongInstrumentProfile profile) =>
@ -3023,6 +3163,7 @@ static void PrintHelp()
--embed-model qwen3-embedding:0.6b
--collection aquasynth_ipa_trial_results
Use song-prepare --duration-seconds 0 to freeze the full decoded source file from sample zero.
Agent-authored patch files must be named <targetId>__candidate-name.aqua.
Known seed target ids: a, i, u, e, o, m, n, ng, l, r, s, z, f, v, th, p, b, t, d, k, mix-a, mix-m, mix-s, mix-p, mix-u.
""");
@ -3059,8 +3200,11 @@ sealed record SongChallengeAnalysisArtifacts(
string LogMelBandStatsCsv,
string RmsEnvelopeCsv,
string RmsEnvelopeAutocorrCsv,
string WhitenedSpectralAutocorrCsv,
string AnalysisReportMarkdown);
sealed record SongRenderAnalysisArtifacts(string Kind, string Path);
sealed record SongChallengeFeatures(
float DurationSeconds,
float Peak,

31
tools/run-song-snippet-trial-swarm.ps1
View file

@ -5,6 +5,8 @@ param(
[string]$Source = "D:\Music\Reinier\Heyoka\Gate Code\Heyoka - Alien Gibberish.mp3",
[string]$SourceFolder = "D:\Music\Reinier\Heyoka",
[float]$DurationSeconds = 30,
[switch]$FullSongs,
[int]$IterationsPerTarget = 5,
[int]$Seed = 0,
[switch]$NewSegmentPerPass,
[switch]$RandomSourcePerAgent,
@ -144,7 +146,7 @@ function Get-SongSources {
if ($SourceFolder.Length -gt 0) {
$files = @(Get-ChildItem -LiteralPath $SourceFolder -File -Recurse |
Where-Object { $_.Extension -in @(".mp3", ".wav", ".flac", ".aiff", ".aif", ".ogg") } |
Where-Object { $_.Extension -in @(".mp3", ".wav", ".flac", ".aiff", ".aif", ".ogg", ".m4a", ".mp4") } |
Sort-Object FullName)
if ($files.Count -eq 0) {
throw "No supported song files found in '$SourceFolder'."
@ -246,9 +248,11 @@ Set-Content -LiteralPath (Join-Path $loopRoot "loop-index.md") -Value @"
- passes: $Passes
- agents_per_pass: $AgentsPerPass
- songs_per_agent: $SongsPerAgent
- iterations_per_target: $IterationsPerTarget
- playlist_top_count: $PlaylistTopCount
- seed: $seedValue
- duration_seconds: $DurationSeconds
- full_songs: $FullSongs
- new_segment_per_pass: $NewSegmentPerPass
- random_source_per_agent: $RandomSourcePerAgent
- store: $storePath
@ -317,6 +321,7 @@ for ($pass = 1; $pass -le $Passes; $pass++) {
}
$agentSource = $songSources[[int]$sourceIndex]
$agentSeed = $phaseSeed + ($agent * 104729) + ($song * 8191)
$durationArg = if ($FullSongs) { "0" } else { [string]$DurationSeconds }
Invoke-LoggedProcess `
-FilePath "dotnet" `
-ArgumentList @(
@ -330,7 +335,7 @@ for ($pass = 1; $pass -le $Passes; $pass++) {
"--artifact-root",
$agentChallengeRoot,
"--duration-seconds",
([string]$DurationSeconds),
$durationArg,
"--seed",
([string]$agentSeed),
"--challenge-id",
@ -350,6 +355,9 @@ for ($pass = 1; $pass -le $Passes; $pass++) {
$targetPatchList = ($targetPatchDirs | ForEach-Object { "- target patch output directory: $_" }) -join "`n"
$agentSourceList = ($agentSources | ForEach-Object { "- assigned source: $_" }) -join "`n"
$agentSeedList = ($agentSeeds | ForEach-Object { "- target seed: $_" }) -join "`n"
$durationLabel = if ($FullSongs) { "the full decoded song duration from each challenge report" } else { "$DurationSeconds seconds" }
$targetKind = if ($FullSongs) { "full-song" } else { "$DurationSeconds second" }
$iterationRoot = Join-Path $agentDir "iterations"
$outputPath = Join-Path $agentDir "hypothesis-agent.md"
$promptPath = Join-Path $agentDir "hypothesis-agent.prompt.md"
$logPath = Join-Path $logRoot "$passId-$agentId-hypothesis-agent.log"
@ -367,21 +375,32 @@ $targetPatchList
- candidate filename prefix: $agentId
- phase id: $passId
- phase seed: $phaseSeed
- private iteration root: $iterationRoot
- required self-iteration attempts per target: $IterationsPerTarget
$agentSourceList
$agentSeedList
- required duration: $DurationSeconds seconds
- required duration: $durationLabel
Goal:
Write exactly one parseable AquaSynth `.aqua` patch for each frozen $DurationSeconds second reference clip assigned to you. This is scene-audio parity, not IPA articulation. You may use ordinary voices, FM, AM/tremolo, filters, formants, noise layers, acoustic vocal/syrinx primitives, additive/PAD layers, curves, and helper voices.
Write exactly one parseable AquaSynth `.aqua` patch for each frozen $targetKind reference assigned to you. This is scene-audio parity, not IPA articulation. You may use ordinary voices, FM, AM/tremolo, filters, formants, noise layers, acoustic vocal/syrinx primitives, additive/PAD layers, curves, and helper voices.
Generalization:
- You own two target patches, but your report must compare both targets and state what reusable scene/instrument knowledge transfers between them.
- The two patches may share instrument roles, texture roles, control idioms, and DSL conventions, but each patch should fit its own target's tempo/register/spectral evidence.
- Future agents will receive distilled evidence from this trial when asked to zero-shot an audio production request. Make your `hypotheses.md` useful retrieval context: name what transferred, what did not, and what evidence would change your next patch.
Self-iteration loop:
- For each target, run exactly $IterationsPerTarget local attempts before publishing the final `.aqua` file.
- Each attempt must write a candidate under `$iterationRoot/attempt-NN/<target-id>/patch`, run the scoring worker against that one target, inspect the rendered candidate evidence, and revise the next attempt.
- Scoring command pattern:
`dotnet run --project "$workerProject" -- song-score --patch-root "<attempt-patch-dir>" --challenge "<challenge-json>" --artifact-root "<attempt-dir>" --batch-id "<agent-id>-<target-id>-attempt-NN" --store "<attempt-dir>/iteration-results.cc" --hypothesizer "$agentId-iteration"`
- After each score, read the attempt's `evaluator-report.md`, `summary.csv`, `audio/comparison.txt`, `audio/candidate-analysis.md`, `audio/candidate-logmel-band-stats.csv`, `audio/candidate-rms-envelope-autocorr.csv`, and `audio/candidate-whitened-spectral-autocorr.csv`.
- Use those candidate-side facts the same way you use target facts: compare spectral bands, envelope autocorrelation, whitened spectral autocorrelation, centroid/RMS, instrument-role metrics, and chip-distress risk. Move timings, filters, gains, sources, roles, and patterns between attempts.
- Keep intermediate candidates inside `$iterationRoot`; publish only one final `.aqua` per target to the official target patch output directory.
Noise and texture:
- Do not model background or recording color as a full-duration raw `voice wave=noise` bed. That produces static white-noise wash and will be treated as broken subtractive synthesis, not scene modeling.
- Use the shaped texture owner for noise roles: `texture name=dust_hat role=dust pattern=x..x step=<beat_seconds/4> gain=.08 sustain=$DurationSeconds`, `texture name=air_wash role=air gain=.035 sustain=$DurationSeconds`, `texture name=codec_bed role=codec gain=.04 sustain=$DurationSeconds`.
- Use the shaped texture owner for noise roles: `texture name=dust_hat role=dust pattern=x..x step=<beat_seconds/4> gain=.08 sustain=<duration_seconds>`, `texture name=air_wash role=air gain=.035 sustain=<duration_seconds>`, `texture name=codec_bed role=codec gain=.04 sustain=<duration_seconds>`.
- `texture` lowers to a gated noise voice with role-specific bandpass/highpass/lowpass/tremolo defaults and a control curve or pattern gate. Raw `voice wave=noise` is only acceptable for short transients with explicit gates and narrow filters.
Instrument ownership:
@ -430,7 +449,7 @@ Evidence contract:
- Write exactly one `.aqua` file under each target patch output directory named `<agent-id>__<family>__<hypothesis>.aqua`.
- Include at least three scene voices or layers in the patch: a primary voice/body, a rhythmic drum/transient/noise role, and a pad/bed/recording-color role. Prefer syrinx for the primary voice when it is voice-like, subtractive for the drum/transient, and additive/PAD or shaped texture for the bed.
- Background/recording-color layers must use `texture` or an equivalently gated and band-shaped explicit voice; a static full-duration white-noise bed is failed evidence.
- Make each patch duration/gates cover its matching $DurationSeconds second target.
- Make each patch duration/gates cover its matching target duration from the challenge report.
Return a short final summary naming the patch and report.
"@