Add song snippet trial swarm
This commit is contained in:
Meta Vens
parent
06875c1eef
commit
d5975f6726
8 changed files with 1253 additions and 6 deletions
|
|
@ -78,6 +78,31 @@ When a scene voice exists mainly to match background noise or capture color, the
|
|||
report must name that role and score it as context, not as proof that the
|
||||
primary body learned to speak.
|
||||
|
||||
## Song Snippet Challenges
|
||||
|
||||
Song-reference golfing is a scene-audio challenge, not an IPA or anatomy oracle.
|
||||
Local music clips may be used as private parity pressure, but the repo stores
|
||||
only derived challenge metadata and local artifact paths unless a fixture is
|
||||
explicitly licensed for redistribution.
|
||||
|
||||
The worker freezes a random clip into a local `reference.wav`, then exposes
|
||||
features that patch-writing agents can treat as control hints:
|
||||
|
||||
- tempo comes from onset-envelope autocorrelation and is reported as
|
||||
`tempo_bpm`, `beat_seconds`, and `tempo_confidence`;
|
||||
- register comes from dominant-frequency autocorrelation plus spectral fallback
|
||||
and is reported as `dominant_hz`, `register_low_hz`, `register_high_hz`, and
|
||||
`root_note`;
|
||||
- scale hints are deterministic authoring pressure, not musicology truth:
|
||||
`suggested_scale` and `scale_frequencies_hz` give agents a bounded pitch set
|
||||
so candidates do not spray every octave while trying to match one song clip.
|
||||
|
||||
Current Strudel-like sequencing is expressed through existing AquaSynth
|
||||
controls: declare a `param`, drive it with `curve ... interp=hold loop=true`,
|
||||
then bind the parameter into voice fields such as `freq=@/seq/pitch` or
|
||||
`gain=@/seq/kick`. Native `scale`, `note`, and pattern sugar should be added
|
||||
only when the DSL owns the lowering and the emitted Faust remains inspectable.
|
||||
|
||||
## Faust Boundary
|
||||
|
||||
Faust describes pure DSP: inputs to outputs. Architecture files and host code
|
||||
|
|
|
|||
|
|
@ -247,3 +247,4 @@
|
|||
{"date":"2026-05-31","area":"ipa-trial-source-golf-scene-voices","lesson":"IPA hypothesis agents need authority to golf the DSL/Faust machine when repeated evidence points at a missing owner, and every parity patch should model the reference as a scene with named voices. Source-edit-enabled rounds may now change parser/model/lowering/Faust/tests/docs with a source-change plan and pre-score build; candidate reports must separate primary articulation, tract noise, release/transient, background-room, and recording-condition voices. General synth primitives are allowed for scene/background modeling, but helper voices cannot impersonate articulation without lower-layer evidence.","evidence":"tools/run-ipa-trial-loop.ps1; docs/ipa-vocal-tract-roadmap.md; docs/audio-doctrine.md"}
|
||||
{"date":"2026-05-31","area":"ipa-trial-source-golf-smoke","lesson":"The first loose source-edit-enabled IPA round completed without tracked source edits and did not improve the aggregate metrics, which is valid negative evidence rather than a failure of authority. It exposed two harness contracts to tighten: evaluators must judge `hypotheses.md` as the report of record instead of the external agent's final `hypothesis-agent.md` chat summary, and scene/noise families must implement at least one actual helper/background `.aqua` `voice` declaration instead of satisfying scene modeling with prose.","evidence":"artifacts/parity/ipa-trial-loops/20260531T001831690; tools/run-ipa-trial-loop.ps1; docs/ipa-vocal-tract-roadmap.md; state/scratch.md"}
|
||||
{"date":"2026-05-31","area":"ipa-trial-learning-loop","lesson":"The loose source-golf round obeyed the file-count ritual but did not learn: it rotated families across unrelated phonetic targets, assigned stop-release ideas to vowel/approximant lanes, gave `p` an oral-pressure tweak instead of the stop-release owner, and produced aggregate regressions without converting them into the next hypothesis. The trial loop now passes previous evaluation paths into the hypothesizer, requires `Previous Round Learning`, requires a `Target-Class Lane Plan`, forbids round-robin family assignment, and lets the evaluator fail class alignment when families do not match phonetic owner classes or explicit transfer-test mutations.","evidence":"artifacts/parity/ipa-trial-loops/20260531T001831690/round-001/hypotheses.md; artifacts/parity/ipa-trial-loops/20260531T001831690/round-001/evaluation.md; tools/run-ipa-trial-loop.ps1; docs/ipa-vocal-tract-roadmap.md"}
|
||||
{"date":"2026-05-31","area":"song-snippet-trial-swarm","lesson":"Song-scene parity needs musical control evidence in addition to spectrogram loss. The new song challenge lane freezes a seeded local clip, derives tempo from onset-envelope autocorrelation, derives a register/root/scale hint from dominant-frequency autocorrelation plus spectral fallback, and prompts agents to sequence explicit AquaSynth `param` + `curve interp=hold loop=true` controls instead of inventing unimplemented Strudel syntax. Register-bounded scale frequencies are a temporary explicit contract until native scale/note sugar has a real DSL/lowering owner. Song scoring must not require vocal-network primitive timelines; non-vocal scene patches now score with an empty timeline witness instead of failing before render.","evidence":"tools/IpaTrialWorker/Program.cs; tools/run-song-snippet-trial-swarm.ps1; docs/audio-doctrine.md; artifacts/parity/song-snippet-smoke/challenge.md; artifacts/parity/song-snippet-swarms/20260531T012627197/challenge.json; artifacts/parity/song-snippet-score-smoke/smoke-001/evaluator-report.md"}
|
||||
|
|
|
|||
|
|
@ -60,6 +60,7 @@
|
|||
"speech timing/confidence receipts for render requests, results, and checkpoints",
|
||||
"CultMesh speech render/scoring orchestration with payload manifests, peer/lease admission receipts, worker assignment, result collection, and central checkpoint application while the authority trainer owns optimizer steps and checkpoint commits",
|
||||
"frozen IPA gesture experiment rounds and local render trials with candidate scripts, primitive timelines, extracted primitive timeline facts, gesture-layer metrics, manifests, evidence JSONL, metric summaries, candidate clusters, science briefs, WAV artifacts, evaluator reports, CultCache .cc trial-result records, and Qdrant/Ollama-derived semantic retrieval indexes over those records and local artifacts",
|
||||
"local song-snippet scene parity challenges with seeded clip extraction, tempo autocorrelation, register/scale hints, external Codex swarm orchestration, rendered candidate scoring, and song trial-result CultCache records",
|
||||
"script scoring",
|
||||
"preset catalogs",
|
||||
"Faust emission and compiler wrapper"
|
||||
|
|
@ -92,7 +93,7 @@
|
|||
"Surge XT for modern hybrid synth modulation",
|
||||
"Cardinal/VCV-style patches for modular cable/rate semantics"
|
||||
],
|
||||
"rule": "For speech, add schema fields and learned-control lanes only when the parity harness has a loss surface that can exercise them. For reference synths, add DSL abstractions only when rebuilds prove the missing invariant. Parsing alone is inventory, not evidence of expressivity. Pink Trombone should be decomposed into reusable low-level Aqua primitives before audio golf, not copied as a bespoke clone or inflated into PT-shaped modules. After the 2026-05-29 proprioception correction, collapse graph-vocal lowering into a small set of instrumented physical primitives before more utterance golf. Gesture motion belongs in ControlSurface/ControlSpline timeline ownership, not primitive-local smoothing; cut any generated contact/port spline that invents motion not supported by primitive reference timelines. Public clients author gestures through ControlSurfaceCatalog plus ControlSplineTimeline and stream surface control dictionaries; higher-level gesture templates and phoneme_gesture are only spline emitters. IPA descriptors seed which anatomical surfaces move, while local datasets golf numeric spline defaults and patch voice code. Training reports must separate gesture_score, clean_vocal_score, and full_parity_score so full-patch recording-condition helpers cannot impersonate articulation. Frozen IPA gesture rounds own candidate scripts, primitive timelines, gesture metrics, manifests, evidence receipts, metric summaries, candidate clusters, and science briefs; `IpaTrialResult` CultCache records own rendered trial outcomes plus extracted primitive timeline facts for contact release, source flow, branch admittance, radiation output, path energy, and passivity. External Codex IPA exploration must query the `.cc` store through the worker's semantic `search`/`show` organ, backed by Qdrant/Ollama vector retrieval when available; Qdrant is a derived index over CultCache and local artifacts, not a second truth. Search reports expose class focus/match and use phonetic class-affinity plus timeline-fact text so agents do not treat generic weak-articulation language as cross-class evidence. Source-edit-enabled IPA rounds may golf DSL syntax, parser/model records, Faust lowering, worker evidence surfaces, and tests when repeated evidence identifies a missing owner; every source edit needs an owner map, invariant, expected metric movement, verification, and rollback trigger. `hypotheses.md` is the hypothesis worker report of record; `hypothesis-agent.md` is only the external agent's final chat summary. Each new round must learn from the previous evaluation: flat/regressed/failed families are falsified until the report states what changed, and family/target assignment must be phonetic-class aligned instead of round-robin. Stop-release families belong on stop and `mix-p` lanes, nasal-branch on nasal/approximant and `mix-m`, frication on fricative and `mix-s`, and vowel/source-shape on vowel-like lanes unless an explicit transfer-test mutation names class-specific controls. IPA parity candidates should model scene voices explicitly: primary articulation, tract noise, release/transient, background-room, and recording-condition. At least one scene/noise or recording-condition family in a source-golf round must use real auxiliary `.aqua` `voice` declarations when claiming helper/background scene modeling. General synth primitives are allowed for helper scene voices, but they cannot count as articulation without gesture, primitive timeline, and clean-vocal movement. Crude full-store dumps are audit fallback, not agent memory. Statistical science workers and audio parity stages consume those bundles without rewriting gesture evidence. If gesture scores rise but rendered audio is silent, fix the primitive source/tract owner before adding full-patch dressing."
|
||||
"rule": "For speech, add schema fields and learned-control lanes only when the parity harness has a loss surface that can exercise them. For reference synths, add DSL abstractions only when rebuilds prove the missing invariant. Parsing alone is inventory, not evidence of expressivity. Pink Trombone should be decomposed into reusable low-level Aqua primitives before audio golf, not copied as a bespoke clone or inflated into PT-shaped modules. After the 2026-05-29 proprioception correction, collapse graph-vocal lowering into a small set of instrumented physical primitives before more utterance golf. Gesture motion belongs in ControlSurface/ControlSpline timeline ownership, not primitive-local smoothing; cut any generated contact/port spline that invents motion not supported by primitive reference timelines. Public clients author gestures through ControlSurfaceCatalog plus ControlSplineTimeline and stream surface control dictionaries; higher-level gesture templates and phoneme_gesture are only spline emitters. IPA descriptors seed which anatomical surfaces move, while local datasets golf numeric spline defaults and patch voice code. Training reports must separate gesture_score, clean_vocal_score, and full_parity_score so full-patch recording-condition helpers cannot impersonate articulation. Frozen IPA gesture rounds own candidate scripts, primitive timelines, gesture metrics, manifests, evidence receipts, metric summaries, candidate clusters, and science briefs; `IpaTrialResult` CultCache records own rendered trial outcomes plus extracted primitive timeline facts for contact release, source flow, branch admittance, radiation output, path energy, and passivity. External Codex IPA exploration must query the `.cc` store through the worker's semantic `search`/`show` organ, backed by Qdrant/Ollama vector retrieval when available; Qdrant is a derived index over CultCache and local artifacts, not a second truth. Search reports expose class focus/match and use phonetic class-affinity plus timeline-fact text so agents do not treat generic weak-articulation language as cross-class evidence. Source-edit-enabled IPA rounds may golf DSL syntax, parser/model records, Faust lowering, worker evidence surfaces, and tests when repeated evidence identifies a missing owner; every source edit needs an owner map, invariant, expected metric movement, verification, and rollback trigger. `hypotheses.md` is the hypothesis worker report of record; `hypothesis-agent.md` is only the external agent's final chat summary. Each new round must learn from the previous evaluation: flat/regressed/failed families are falsified until the report states what changed, and family/target assignment must be phonetic-class aligned instead of round-robin. Stop-release families belong on stop and `mix-p` lanes, nasal-branch on nasal/approximant and `mix-m`, frication on fricative and `mix-s`, and vowel/source-shape on vowel-like lanes unless an explicit transfer-test mutation names class-specific controls. IPA parity candidates should model scene voices explicitly: primary articulation, tract noise, release/transient, background-room, and recording-condition. At least one scene/noise or recording-condition family in a source-golf round must use real auxiliary `.aqua` `voice` declarations when claiming helper/background scene modeling. Song-snippet parity is scene-audio pressure: derive tempo with onset autocorrelation, derive register/root/scale hints from the clip, and sequence explicit `param` plus `curve interp=hold loop=true` controls until native Strudel-like scale/note sugar has a real owner. General synth primitives are allowed for helper scene voices, but they cannot count as articulation without gesture, primitive timeline, and clean-vocal movement. Crude full-store dumps are audit fallback, not agent memory. Statistical science workers and audio parity stages consume those bundles without rewriting gesture evidence. If gesture scores rise but rendered audio is silent, fix the primitive source/tract owner before adding full-patch dressing."
|
||||
},
|
||||
"ipa_vocal_tract_path": {
|
||||
"status": "This is now the active project center. Roadmap and research notes align the curriculum around structured utterance metadata -> learned utterance embedding -> learned synth automation -> vocal tract/envelope/LFO/filter/spectral controls -> rendered audio loss. The first Core contracts still defend the boundary: PhoneticIntent is the Weksa/language-owned handoff, ArticulatoryPlan is the AquaSynth realization layer, VocalTractMorphology declares anatomy capabilities, ArticulatoryConstraintReport is the host-visible veto surface, and VocalTractPlanResult is the tiny end-to-end proof artifact for host branching. IPA is an authoring input only: tokenization and feature mapping lower into language phonology profiles, timed articulatory gestures, morphology-specific tract/excitation curves, and Faust DSP. The staged support plan starts with tokenization and vowels, then core pulmonic consonants, expanded human IPA, airstream/phonation, and Weksa/alien morphologies. Optional eSpeak NG tests provide the first generated speech workout when ESPEAK_NG or espeak-ng is installed; eSpeak is intelligibility/timing/log-mel pressure, not anatomy truth. AquaSynth owns the learned utterance embedding encoder and synth driver: speech text embeddings plus prosody/emphasis hints plus Ghostlight/Epiphany-shaped character state vectors compress into utterance embeddings, and phonetic features plus that embedding feed a packed-buffer native C# backprop/Adam mapper that predicts tract, AM/FM, LFO, filter, and mel spectral-envelope controls. SpeechBackpropagationPipeline chains the two models: synth-driver target loss returns gradients with respect to its input, slices the utterance-embedding gradient, and updates the utterance encoder from downstream speech loss. The generated ground-truth training fixture renders eSpeak syllables, extracts log-mel evidence, and trains that chained pipeline toward eSpeak-seeded mel/control targets. CultMesh speech training now has payload manifests, peer/lease admission receipts, worker assignment, result collection, and central checkpoint application; workers do not own model updates. Real remote artifact chunk transfer and process supervision remain CultMesh infrastructure surfaces, not AquaSynth workaround territory. IpaGestureExperiment writes frozen descriptor/variant rounds with candidate scripts, primitive timelines, gesture metrics, manifests, and evidence JSONL, then analyzes them into metric summaries, score-band clusters, and a science brief. IpaTrialOrchestrator now writes local five-seed IPA render trials, accepts externally authored candidate scripts through RunCandidateScriptsAsync, writes WAV/report artifacts, evaluator reports, extracted primitive timeline facts, and `ipa-trial-results.cc` CultCache records. `tools/IpaTrialWorker` is the public worker tool for seed/score/search/show/index over the store; search now auto-upserts Qdrant/Ollama evidence chunks including timeline-fact text and returns vector-backed reports, while `tools/run-ipa-trial-loop.ps1` dispatches external Codex hypothesis/evaluator workers around five five-phoneme target sets. The latest seed run improved voiced vowels/nasals after primitive `SourcePort` lowering gained an oscillatory carrier; plosive closure/release remains the next weak owner.",
|
||||
|
|
@ -121,9 +122,9 @@
|
|||
"binding_rule": "Parameter binding uses field-site references such as `lpf=@/macro/brightness` and `/opgraphs/0/operators/2/level`; the field owns meaning and the parameter owns the runtime value. Avoid target-routing semantics unless a new graph authority is deliberately mapped."
|
||||
},
|
||||
"last_verified": {
|
||||
"command": "powershell -NoProfile -ExecutionPolicy Bypass -File tools\\run-ipa-trial-loop.ps1 -Rounds 0 -SkipLocalTrialRun; dotnet build tests\\AquaSynth.Dsl.Tests\\AquaSynth.Dsl.Tests.csproj --no-restore --disable-build-servers -p:UseSharedCompilation=false -p:BuildInParallel=false -v minimal",
|
||||
"result": "IPA trial loop prompts now require previous-round learning, target-class lane plans, phonetic class-aligned family assignment, source-edit owner maps, `hypotheses.md` as the report of record, and at least one real auxiliary `.aqua` helper voice when scene/noise families claim background or recording-condition modeling.",
|
||||
"commit": "pending ipa learning-loop correction"
|
||||
"command": "dotnet build tools\\IpaTrialWorker\\IpaTrialWorker.csproj --no-restore --disable-build-servers -p:UseSharedCompilation=false -p:BuildInParallel=false -v minimal; dotnet run --project tools\\IpaTrialWorker\\IpaTrialWorker.csproj --no-build -- song-prepare --source \"D:\\Music\\Reinier\\Heyoka\\Gate Code\\Heyoka - Alien Gibberish.mp3\" --artifact-root artifacts\\parity\\song-snippet-smoke --duration-seconds 10 --seed 123 --output artifacts\\parity\\song-snippet-smoke\\challenge.json; powershell -NoProfile -ExecutionPolicy Bypass -File tools\\run-song-snippet-trial-swarm.ps1 -Passes 0 -Seed 123 -DurationSeconds 10; dotnet run --project tools\\IpaTrialWorker\\IpaTrialWorker.csproj --no-build -- song-score --patch-root artifacts\\parity\\song-snippet-score-smoke\\candidates --challenge artifacts\\parity\\song-snippet-smoke\\challenge.json --artifact-root artifacts\\parity\\song-snippet-score-smoke --batch-id smoke-001 --store artifacts\\parity\\song-snippet-score-smoke\\song-trial-results.cc --hypothesizer smoke-song-worker",
|
||||
"result": "Song-snippet worker and swarm prep now build, decode a seeded local clip, emit tempo/register/scale challenge facts, prepare a loop skeleton without launching agents, and score an ordinary non-vocal scene patch into song trial-result artifacts.",
|
||||
"commit": "pending song-snippet trial swarm"
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,37 @@
|
|||
|
||||
Current slice:
|
||||
|
||||
- Song-snippet challenge harness is now a real local parity lane, not a
|
||||
hand-waved prompt. `tools/IpaTrialWorker` has `song-prepare` and `song-score`:
|
||||
`song-prepare` decodes a local audio file through NAudio, freezes a seeded
|
||||
ten-second mono 44.1 kHz `reference.wav`, writes `challenge.json` and
|
||||
`challenge.md`, and reports scene features including spectral centroid,
|
||||
rolloff, active duty, spectral flux, onset-autocorrelation tempo
|
||||
(`tempo_bpm`, `beat_seconds`, `tempo_confidence`), dominant pitch/register
|
||||
(`dominant_hz`, `register_low_hz`, `register_high_hz`, `root_note`), and
|
||||
deterministic scale hints (`suggested_scale`, `scale_frequencies_hz`).
|
||||
`song-score` renders agent-authored `.aqua` candidates against that frozen
|
||||
reference, writes candidate WAV/DSP/timeline/comparison artifacts, and stores
|
||||
typed `IpaTrialResult` records in a song `.cc` store.
|
||||
|
||||
- `tools/run-song-snippet-trial-swarm.ps1` prepares the random clip from
|
||||
`D:\Music\Reinier\Heyoka\Gate Code\Heyoka - Alien Gibberish.mp3`, then can
|
||||
dispatch passes of parallel external Codex agents. Agents are prompted to use
|
||||
scene voices, FM/AM/noise/envelopes, rhythmic controls from
|
||||
autocorrelation-derived tempo, and register-bounded explicit scale-frequency
|
||||
curves. Until native Strudel-like `scale`/`note` sugar exists, the live syntax
|
||||
is `param` plus `curve ... interp=hold loop=true` bound into voice fields.
|
||||
Smoke prep with seed `123` produced a clip starting at `305.2523s` with
|
||||
`tempo_bpm=92.30769`, `beat_seconds=0.65`, `dominant_hz=175`, `root_note=F3`,
|
||||
and `minor-pentatonic-plus-tritone` scale frequencies from about `174.614` to
|
||||
`698.457` Hz. Focused worker build and `-Passes 0` swarm prep pass. A
|
||||
one-candidate score smoke at
|
||||
`artifacts/parity/song-snippet-score-smoke/smoke-001` now proves ordinary
|
||||
non-vocal scene patches can score; the first attempt failed because
|
||||
`song-score` assumed every candidate had a `VocalNetwork` named `voice`, so
|
||||
primitive timelines are now optional witnesses for song scenes instead of a
|
||||
required anatomy gate.
|
||||
|
||||
- User correctly called out that the loose agents learned nothing. The actual
|
||||
failure was not just weak metrics; the candidate assignment was incoherent.
|
||||
Families were rotated across targets instead of aligned to phonetic owners:
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
project: AquaSynth
|
||||
spine_version: 1
|
||||
updated: 2026-05-29T15:36:32+01:00
|
||||
updated: 2026-05-31T01:31:00+01:00
|
||||
|
||||
prime_directive: >
|
||||
Build coherent synth machinery. References exist to reveal missing structure,
|
||||
|
|
@ -56,6 +56,7 @@ current_mechanism:
|
|||
live_invariants:
|
||||
- AquaSynth owns its patch graph; external synths are references, not runtime dependencies.
|
||||
- Patch scripts lower into explicit graph records before Faust emission.
|
||||
- Song-snippet parity is scene-audio pressure, not redistributed fixture data. The worker may derive local tempo/register/scale hints from private clips, but agents must express live rhythm and pitch motion through owned AquaSynth controls until native sequencing sugar has an explicit DSL/lowering owner.
|
||||
- Operator graphs own FM topology: operators, modulation edges, carriers, and feedback. Do not hide this authority inside Voice fields.
|
||||
- Readable operator graph authoring uses `operator`, `route`, and `carrier` declarations; compact `ops=`/`edges=` is only scaffolding/interchange.
|
||||
- Notes own frequency, gate duration, and host/MIDI source; envelopes own ADSR level shape. Do not put note duration or SFXR punch back into the general envelope model.
|
||||
|
|
|
|||
|
|
@ -5,6 +5,10 @@
|
|||
<ProjectReference Include="..\..\src\AquaSynth.Faust\AquaSynth.Faust.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="NAudio" Version="2.2.1" />
|
||||
</ItemGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
using System.Globalization;
|
||||
using System.Diagnostics;
|
||||
using System.Net.Http.Json;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
|
|
@ -7,6 +8,7 @@ using System.Text.Json.Serialization;
|
|||
|
||||
using AquaSynth.Dsl;
|
||||
using AquaSynth.Faust;
|
||||
using NAudio.Wave;
|
||||
|
||||
var command = args.FirstOrDefault();
|
||||
if (string.IsNullOrWhiteSpace(command) || command is "-h" or "--help")
|
||||
|
|
@ -26,6 +28,12 @@ try
|
|||
case "score":
|
||||
await ScoreAsync(options);
|
||||
return 0;
|
||||
case "song-prepare":
|
||||
await SongPrepareAsync(options);
|
||||
return 0;
|
||||
case "song-score":
|
||||
await SongScoreAsync(options);
|
||||
return 0;
|
||||
case "dump":
|
||||
await DumpAsync(options);
|
||||
return 0;
|
||||
|
|
@ -80,6 +88,205 @@ static async Task ScoreAsync(Dictionary<string, string> options)
|
|||
Console.WriteLine(store);
|
||||
}
|
||||
|
||||
static async Task SongPrepareAsync(Dictionary<string, string> options)
|
||||
{
|
||||
var source = Required(options, "source");
|
||||
var artifactRoot = Required(options, "artifact-root");
|
||||
var durationSeconds = FloatValue(options, "duration-seconds", 10f);
|
||||
var sampleRate = IntValue(options, "sample-rate", 44100);
|
||||
var seed = IntValue(options, "seed", RandomNumberGenerator.GetInt32(1, int.MaxValue));
|
||||
var challengeId = Value(options, "challenge-id", $"song-snippet-{DateTimeOffset.UtcNow:yyyyMMddTHHmmssfff}");
|
||||
var output = Value(options, "output", Path.Combine(artifactRoot, "challenge.json"));
|
||||
|
||||
if (!File.Exists(source))
|
||||
{
|
||||
throw new FileNotFoundException("Song source file was not found.", source);
|
||||
}
|
||||
|
||||
Directory.CreateDirectory(artifactRoot);
|
||||
var decoded = DecodeMono(source);
|
||||
var resampled = Resample(decoded.Samples, decoded.SampleRate, sampleRate);
|
||||
var clipLength = Math.Min(resampled.Length, Math.Max(1, (int)MathF.Round(durationSeconds * sampleRate)));
|
||||
var maxStart = Math.Max(0, resampled.Length - clipLength);
|
||||
var random = new Random(seed);
|
||||
var startSample = maxStart == 0 ? 0 : random.Next(0, maxStart + 1);
|
||||
var clip = new float[clipLength];
|
||||
Array.Copy(resampled, startSample, clip, 0, clipLength);
|
||||
NormalizePeak(clip, .9f);
|
||||
|
||||
var referenceWav = Path.Combine(artifactRoot, "reference.wav");
|
||||
WriteWav(referenceWav, clip, sampleRate);
|
||||
var analyzer = new AudioAnalyzer(new AudioAnalysisConfig(SampleRate: sampleRate));
|
||||
var analysis = analyzer.Analyze(clip);
|
||||
var tempo = EstimateTempo(clip, sampleRate);
|
||||
var register = EstimateRegister(clip, sampleRate, analysis.Features.SpectralCentroidHz);
|
||||
var features = new SongChallengeFeatures(
|
||||
analysis.Features.DurationSeconds,
|
||||
analysis.Features.Peak,
|
||||
analysis.Features.Rms,
|
||||
analysis.Features.ZeroCrossingRate,
|
||||
analysis.Features.SpectralCentroidHz,
|
||||
analysis.Features.SpectralRolloffHz,
|
||||
ActiveDuty(clip, sampleRate),
|
||||
SpectralFlux(clip, sampleRate),
|
||||
tempo.Bpm,
|
||||
tempo.BeatSeconds,
|
||||
tempo.Confidence,
|
||||
register.DominantHz,
|
||||
register.LowHz,
|
||||
register.HighHz,
|
||||
register.RootNote,
|
||||
register.SuggestedScale,
|
||||
string.Join(",", register.ScaleFrequencies.Select(value => value.ToString("0.###", CultureInfo.InvariantCulture))));
|
||||
var challenge = new SongChallenge(
|
||||
challengeId,
|
||||
source,
|
||||
Path.GetFileName(source),
|
||||
seed,
|
||||
startSample / (float)sampleRate,
|
||||
clipLength / (float)sampleRate,
|
||||
sampleRate,
|
||||
referenceWav,
|
||||
features);
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(output))!);
|
||||
await File.WriteAllTextAsync(output, JsonSerializer.Serialize(challenge, JsonOptions()), Encoding.UTF8);
|
||||
await File.WriteAllTextAsync(Path.Combine(artifactRoot, "challenge.md"), SongChallengeReport(challenge), Encoding.UTF8);
|
||||
Console.WriteLine(output);
|
||||
}
|
||||
|
||||
static async Task SongScoreAsync(Dictionary<string, string> options)
|
||||
{
|
||||
var patchRoot = Required(options, "patch-root");
|
||||
var challengePath = Required(options, "challenge");
|
||||
var artifactRoot = Required(options, "artifact-root");
|
||||
var batchId = Value(options, "batch-id", $"song-round-{DateTimeOffset.UtcNow:yyyyMMddTHHmmssfff}");
|
||||
var store = Value(options, "store", Path.Combine(artifactRoot, "song-trial-results.cc"));
|
||||
var hypothesizer = Value(options, "hypothesizer", "external-codex-song-worker");
|
||||
var challenge = JsonSerializer.Deserialize<SongChallenge>(
|
||||
await File.ReadAllTextAsync(challengePath, Encoding.UTF8),
|
||||
JsonOptions()) ?? throw new InvalidDataException($"Could not read song challenge `{challengePath}`.");
|
||||
var referenceSamples = ReadMonoPcm16Wav(challenge.ReferenceWavPath);
|
||||
var candidates = SongCandidateScripts(patchRoot, hypothesizer);
|
||||
if (candidates.Count == 0)
|
||||
{
|
||||
throw new InvalidDataException($"No .aqua song candidates found under `{patchRoot}`.");
|
||||
}
|
||||
|
||||
var batchDirectory = Path.Combine(artifactRoot, batchId);
|
||||
var candidateRoot = Path.Combine(batchDirectory, "song-candidates");
|
||||
var timelineRoot = Path.Combine(batchDirectory, "song-timelines");
|
||||
Directory.CreateDirectory(candidateRoot);
|
||||
Directory.CreateDirectory(timelineRoot);
|
||||
var analyzer = new AudioAnalyzer(new AudioAnalysisConfig(SampleRate: challenge.SampleRate));
|
||||
var results = new List<IpaTrialResult>();
|
||||
|
||||
foreach (var candidate in candidates)
|
||||
{
|
||||
var startedAt = DateTimeOffset.UtcNow;
|
||||
var startedStamp = Stopwatch.GetTimestamp();
|
||||
var safeCandidateId = SafeName(candidate.CandidateId);
|
||||
var candidateDir = Path.Combine(candidateRoot, safeCandidateId);
|
||||
var audioDir = Path.Combine(candidateDir, "audio");
|
||||
Directory.CreateDirectory(audioDir);
|
||||
var scriptCopy = Path.Combine(candidateDir, "candidate.aqua");
|
||||
File.Copy(candidate.ScriptPath, scriptCopy, overwrite: true);
|
||||
var script = await File.ReadAllTextAsync(scriptCopy, Encoding.UTF8);
|
||||
var patch = PatchScript.Parse(script);
|
||||
var timelineNetwork = patch.VocalNetworks.FirstOrDefault();
|
||||
var timeline = timelineNetwork is null
|
||||
? Array.Empty<ProbeTimelineSample>()
|
||||
: ProbeTimelineReport.Build(patch, timelineNetwork.Name, 12);
|
||||
var timelinePath = Path.Combine(timelineRoot, $"{safeCandidateId}.csv");
|
||||
await File.WriteAllTextAsync(timelinePath, ProbeTimelineReport.ToCsv(timeline), Encoding.UTF8);
|
||||
var source = FaustEmitter.EmitScript(script, new FaustExportOptions(safeCandidateId)).Source;
|
||||
var dspPath = Path.Combine(candidateDir, "candidate.dsp");
|
||||
await File.WriteAllTextAsync(dspPath, source, Encoding.UTF8);
|
||||
|
||||
var candidateWav = Path.Combine(audioDir, "candidate.wav");
|
||||
var metrics = new List<SpeechScoreMetric>();
|
||||
var artifacts = new List<SpeechRenderArtifact>
|
||||
{
|
||||
Artifact("song-challenge", challengePath),
|
||||
Artifact("reference-wav", challenge.ReferenceWavPath),
|
||||
Artifact("candidate-script", scriptCopy),
|
||||
Artifact("candidate-dsp", dspPath),
|
||||
Artifact("primitive-timeline", timelinePath)
|
||||
};
|
||||
var timelineFacts = PrimitiveTimelineFactExtractor.Extract(timeline).ToArray();
|
||||
string verdict;
|
||||
string evaluation;
|
||||
var render = await FaustCompiler.RenderAsync(
|
||||
source,
|
||||
new FaustRenderOptions(challenge.SampleRate, challenge.DurationSeconds));
|
||||
if (render is null || render.Samples.Length == 0)
|
||||
{
|
||||
verdict = "render-failed";
|
||||
evaluation = render is null
|
||||
? "Faust was not available to render this song candidate."
|
||||
: $"Faust render produced no samples. stderr: {render.Stderr}";
|
||||
metrics.Add(new SpeechScoreMetric("render_failed", 1, 1));
|
||||
}
|
||||
else
|
||||
{
|
||||
var matched = MatchLength(render.Samples, referenceSamples.Length);
|
||||
NormalizePeak(matched, .9f);
|
||||
WriteWav(candidateWav, matched, challenge.SampleRate);
|
||||
artifacts.Add(Artifact("candidate-wav", candidateWav));
|
||||
var comparison = analyzer.Compare(referenceSamples, matched);
|
||||
metrics.AddRange(SongComparisonMetrics(comparison));
|
||||
verdict = SongVerdict(comparison);
|
||||
evaluation = SongEvaluationSentence(challenge, comparison, verdict);
|
||||
var comparisonPath = Path.Combine(audioDir, "comparison.txt");
|
||||
await File.WriteAllTextAsync(comparisonPath, SongComparisonReport(challenge, candidate, comparison, verdict), Encoding.UTF8);
|
||||
artifacts.Add(Artifact("comparison-report", comparisonPath));
|
||||
}
|
||||
|
||||
var latency = Stopwatch.GetElapsedTime(startedStamp).TotalMilliseconds;
|
||||
results.Add(new IpaTrialResult(
|
||||
$"{batchId}:song-snippet:{safeCandidateId}",
|
||||
batchId,
|
||||
startedAt.ToString("O", CultureInfo.InvariantCulture),
|
||||
"song-snippet",
|
||||
["song", "snippet", "alien-gibberish"],
|
||||
challenge.ChallengeId,
|
||||
safeCandidateId,
|
||||
candidate.HypothesizerId,
|
||||
candidate.Hypothesis,
|
||||
scriptCopy,
|
||||
challenge.ReferenceWavPath,
|
||||
File.Exists(candidateWav) ? candidateWav : "",
|
||||
timelinePath,
|
||||
metrics.ToArray(),
|
||||
artifacts.ToArray(),
|
||||
"song-snippet-audio-evaluator",
|
||||
evaluation,
|
||||
verdict,
|
||||
[
|
||||
"This is a local-only song clip challenge; reference audio is not redistributed by the repo.",
|
||||
"The target is a randomly selected ten-second scene-audio snippet, not IPA articulation truth.",
|
||||
"Full-patch FM/AM/noise/scene modeling is allowed; primitive timeline facts remain diagnostic when present."
|
||||
],
|
||||
[
|
||||
new SpeechTimingReceipt(
|
||||
"song-snippet-render-score",
|
||||
startedAt.ToString("O", CultureInfo.InvariantCulture),
|
||||
latency,
|
||||
0,
|
||||
SongConfidence(metrics),
|
||||
"Local worker rendered an AquaSynth candidate against a frozen song snippet and wrote CultCache trial result data.")
|
||||
],
|
||||
timelineFacts));
|
||||
}
|
||||
|
||||
await IpaTrialResultCultCacheStore.UpsertResultsAsync(store, results);
|
||||
var summaryPath = Path.Combine(batchDirectory, "summary.csv");
|
||||
await File.WriteAllTextAsync(summaryPath, SongSummaryCsv(results), Encoding.UTF8);
|
||||
var reportPath = Path.Combine(batchDirectory, "evaluator-report.md");
|
||||
await File.WriteAllTextAsync(reportPath, SongEvaluatorReport(challenge, results), Encoding.UTF8);
|
||||
Console.WriteLine(batchDirectory);
|
||||
Console.WriteLine(store);
|
||||
}
|
||||
|
||||
static async Task DumpAsync(Dictionary<string, string> options)
|
||||
{
|
||||
var store = Required(options, "store");
|
||||
|
|
@ -409,6 +616,28 @@ static IReadOnlyList<IpaTrialScriptCandidate> CandidateScripts(string patchRoot,
|
|||
.ToArray();
|
||||
}
|
||||
|
||||
static IReadOnlyList<IpaTrialScriptCandidate> SongCandidateScripts(string patchRoot, string hypothesizer)
|
||||
{
|
||||
if (!Directory.Exists(patchRoot))
|
||||
{
|
||||
throw new DirectoryNotFoundException(patchRoot);
|
||||
}
|
||||
|
||||
return Directory.EnumerateFiles(patchRoot, "*.aqua", SearchOption.AllDirectories)
|
||||
.Order(StringComparer.OrdinalIgnoreCase)
|
||||
.Select(path =>
|
||||
{
|
||||
var stem = Path.GetFileNameWithoutExtension(path);
|
||||
return new IpaTrialScriptCandidate(
|
||||
"song-snippet",
|
||||
stem,
|
||||
path,
|
||||
hypothesizer,
|
||||
$"External Codex-authored song-snippet patch candidate `{stem}`.");
|
||||
})
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
static string TargetIdFromStem(string stem, IReadOnlyList<string> targetIds)
|
||||
{
|
||||
var explicitSplit = stem.Split(["__"], StringSplitOptions.None);
|
||||
|
|
@ -1246,6 +1475,24 @@ static string TimelineFact(IpaTrialResult result, string name) =>
|
|||
.Max() ?? 0)
|
||||
.ToString("0.######", CultureInfo.InvariantCulture);
|
||||
|
||||
static SpeechRenderArtifact Artifact(string kind, string path) =>
|
||||
new(kind, path, File.Exists(path) ? Sha256(path) : "");
|
||||
|
||||
static string Sha256(string path)
|
||||
{
|
||||
using var stream = File.OpenRead(path);
|
||||
var hash = SHA256.HashData(stream);
|
||||
return "sha256:" + Convert.ToHexString(hash).ToLowerInvariant();
|
||||
}
|
||||
|
||||
static string SafeName(string value) =>
|
||||
new(value.Select(character => char.IsLetterOrDigit(character) ? character : '_').ToArray());
|
||||
|
||||
static string Escape(string value) =>
|
||||
value.Contains(',') || value.Contains('"') || value.Contains('\n') || value.Contains('\r')
|
||||
? $"\"{value.Replace("\"", "\"\"", StringComparison.Ordinal)}\""
|
||||
: value;
|
||||
|
||||
static IpaTrialResult? ContrastCandidate(IpaTrialResult result, IReadOnlyList<IpaTrialResult> allResults)
|
||||
{
|
||||
var sameTarget = allResults
|
||||
|
|
@ -1286,9 +1533,550 @@ static string StableUuid(string id)
|
|||
return $"{hex[..8]}-{hex[8..12]}-{hex[12..16]}-{hex[16..20]}-{hex[20..32]}";
|
||||
}
|
||||
|
||||
static DecodedAudio DecodeMono(string path)
|
||||
{
|
||||
using var reader = new AudioFileReader(path);
|
||||
var interleaved = new float[4096 * reader.WaveFormat.Channels];
|
||||
var mono = new List<float>();
|
||||
int read;
|
||||
while ((read = reader.Read(interleaved, 0, interleaved.Length)) > 0)
|
||||
{
|
||||
for (var index = 0; index < read; index += reader.WaveFormat.Channels)
|
||||
{
|
||||
var sum = 0f;
|
||||
for (var channel = 0; channel < reader.WaveFormat.Channels && index + channel < read; channel++)
|
||||
{
|
||||
sum += interleaved[index + channel];
|
||||
}
|
||||
|
||||
mono.Add(sum / reader.WaveFormat.Channels);
|
||||
}
|
||||
}
|
||||
|
||||
return new DecodedAudio(reader.WaveFormat.SampleRate, mono.ToArray());
|
||||
}
|
||||
|
||||
static float[] Resample(IReadOnlyList<float> samples, int sourceRate, int targetRate)
|
||||
{
|
||||
if (sourceRate == targetRate)
|
||||
{
|
||||
return samples.ToArray();
|
||||
}
|
||||
|
||||
var result = new float[Math.Max(1, (int)Math.Round(samples.Count * (double)targetRate / sourceRate))];
|
||||
for (var index = 0; index < result.Length; index++)
|
||||
{
|
||||
var position = index * (sourceRate / (double)targetRate);
|
||||
var left = Math.Clamp((int)Math.Floor(position), 0, samples.Count - 1);
|
||||
var right = Math.Clamp(left + 1, 0, samples.Count - 1);
|
||||
var t = (float)(position - left);
|
||||
result[index] = samples[left] * (1 - t) + samples[right] * t;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
static float[] MatchLength(IReadOnlyList<float> samples, int length)
|
||||
{
|
||||
var result = new float[length];
|
||||
for (var index = 0; index < result.Length && index < samples.Count; index++)
|
||||
{
|
||||
result[index] = samples[index];
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
static void NormalizePeak(IList<float> samples, float peak)
|
||||
{
|
||||
var current = samples.Select(MathF.Abs).DefaultIfEmpty(0).Max();
|
||||
if (current <= 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var gain = peak / current;
|
||||
for (var index = 0; index < samples.Count; index++)
|
||||
{
|
||||
samples[index] *= gain;
|
||||
}
|
||||
}
|
||||
|
||||
static float[] ReadMonoPcm16Wav(string path)
|
||||
{
|
||||
using var stream = File.OpenRead(path);
|
||||
using var reader = new BinaryReader(stream, Encoding.ASCII);
|
||||
if (new string(reader.ReadChars(4)) != "RIFF")
|
||||
{
|
||||
throw new InvalidDataException($"`{path}` is not a RIFF WAV file.");
|
||||
}
|
||||
|
||||
reader.ReadInt32();
|
||||
if (new string(reader.ReadChars(4)) != "WAVE")
|
||||
{
|
||||
throw new InvalidDataException($"`{path}` is not a WAVE file.");
|
||||
}
|
||||
|
||||
short channels = 1;
|
||||
short bitsPerSample = 16;
|
||||
byte[]? data = null;
|
||||
while (stream.Position < stream.Length)
|
||||
{
|
||||
var chunkId = new string(reader.ReadChars(4));
|
||||
var chunkSize = reader.ReadInt32();
|
||||
if (chunkId == "fmt ")
|
||||
{
|
||||
var format = reader.ReadInt16();
|
||||
channels = reader.ReadInt16();
|
||||
reader.ReadInt32();
|
||||
reader.ReadInt32();
|
||||
reader.ReadInt16();
|
||||
bitsPerSample = reader.ReadInt16();
|
||||
if (chunkSize > 16)
|
||||
{
|
||||
reader.ReadBytes(chunkSize - 16);
|
||||
}
|
||||
|
||||
if (format != 1 || bitsPerSample != 16)
|
||||
{
|
||||
throw new InvalidDataException($"`{path}` must be PCM16.");
|
||||
}
|
||||
}
|
||||
else if (chunkId == "data")
|
||||
{
|
||||
data = reader.ReadBytes(chunkSize);
|
||||
}
|
||||
else
|
||||
{
|
||||
reader.ReadBytes(chunkSize);
|
||||
}
|
||||
}
|
||||
|
||||
if (data is null)
|
||||
{
|
||||
throw new InvalidDataException($"`{path}` has no data chunk.");
|
||||
}
|
||||
|
||||
var samples = new List<float>(data.Length / Math.Max(1, channels * sizeof(short)));
|
||||
for (var offset = 0; offset + sizeof(short) * channels <= data.Length; offset += sizeof(short) * channels)
|
||||
{
|
||||
var sum = 0f;
|
||||
for (var channel = 0; channel < channels; channel++)
|
||||
{
|
||||
sum += BitConverter.ToInt16(data, offset + channel * sizeof(short)) / (float)short.MaxValue;
|
||||
}
|
||||
|
||||
samples.Add(sum / channels);
|
||||
}
|
||||
|
||||
return samples.ToArray();
|
||||
}
|
||||
|
||||
static void WriteWav(string path, IReadOnlyList<float> samples, int sampleRate)
|
||||
{
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
|
||||
using var stream = File.Create(path);
|
||||
using var writer = new BinaryWriter(stream, Encoding.ASCII);
|
||||
var dataSize = samples.Count * sizeof(short);
|
||||
writer.Write("RIFF"u8);
|
||||
writer.Write(36 + dataSize);
|
||||
writer.Write("WAVE"u8);
|
||||
writer.Write("fmt "u8);
|
||||
writer.Write(16);
|
||||
writer.Write((short)1);
|
||||
writer.Write((short)1);
|
||||
writer.Write(sampleRate);
|
||||
writer.Write(sampleRate * sizeof(short));
|
||||
writer.Write((short)sizeof(short));
|
||||
writer.Write((short)16);
|
||||
writer.Write("data"u8);
|
||||
writer.Write(dataSize);
|
||||
foreach (var sample in samples)
|
||||
{
|
||||
writer.Write((short)Math.Clamp(MathF.Round(sample * short.MaxValue), short.MinValue, short.MaxValue));
|
||||
}
|
||||
}
|
||||
|
||||
static float ActiveDuty(IReadOnlyList<float> samples, int sampleRate)
|
||||
{
|
||||
var frameSize = Math.Max(64, sampleRate / 100);
|
||||
var peak = samples.Select(MathF.Abs).DefaultIfEmpty(0).Max();
|
||||
var threshold = Math.Max(.01f, peak * .04f);
|
||||
var active = 0;
|
||||
var frames = 0;
|
||||
for (var start = 0; start < samples.Count; start += frameSize)
|
||||
{
|
||||
var end = Math.Min(samples.Count, start + frameSize);
|
||||
var rms = MathF.Sqrt(samples.Skip(start).Take(end - start).Select(sample => sample * sample).DefaultIfEmpty(0).Average());
|
||||
if (rms >= threshold)
|
||||
{
|
||||
active++;
|
||||
}
|
||||
|
||||
frames++;
|
||||
}
|
||||
|
||||
return active / (float)Math.Max(1, frames);
|
||||
}
|
||||
|
||||
static float SpectralFlux(IReadOnlyList<float> samples, int sampleRate)
|
||||
{
|
||||
var frameSize = Math.Max(64, sampleRate / 100);
|
||||
var previous = 0f;
|
||||
var flux = 0f;
|
||||
var frames = 0;
|
||||
for (var start = 0; start < samples.Count; start += frameSize)
|
||||
{
|
||||
var end = Math.Min(samples.Count, start + frameSize);
|
||||
var rms = MathF.Sqrt(samples.Skip(start).Take(end - start).Select(sample => sample * sample).DefaultIfEmpty(0).Average());
|
||||
if (frames > 0)
|
||||
{
|
||||
flux += MathF.Abs(rms - previous);
|
||||
}
|
||||
|
||||
previous = rms;
|
||||
frames++;
|
||||
}
|
||||
|
||||
var peak = samples.Select(MathF.Abs).DefaultIfEmpty(0).Max();
|
||||
return frames <= 1 || peak <= 0 ? 0 : Math.Clamp(flux / (frames * peak), 0, 1);
|
||||
}
|
||||
|
||||
static TempoEstimate EstimateTempo(IReadOnlyList<float> samples, int sampleRate)
|
||||
{
|
||||
var hopSize = Math.Max(64, sampleRate / 100);
|
||||
var frameRate = sampleRate / (float)hopSize;
|
||||
var envelope = RmsEnvelope(samples, hopSize)
|
||||
.Select(value => MathF.Log(1e-7f + value))
|
||||
.ToArray();
|
||||
if (envelope.Length < 4)
|
||||
{
|
||||
return new TempoEstimate(0, 0, 0);
|
||||
}
|
||||
|
||||
var onset = new float[envelope.Length];
|
||||
for (var index = 1; index < envelope.Length; index++)
|
||||
{
|
||||
onset[index] = MathF.Max(0, envelope[index] - envelope[index - 1]);
|
||||
}
|
||||
|
||||
var mean = onset.Average();
|
||||
for (var index = 0; index < onset.Length; index++)
|
||||
{
|
||||
onset[index] = MathF.Max(0, onset[index] - mean);
|
||||
}
|
||||
|
||||
var minBpm = 60f;
|
||||
var maxBpm = 200f;
|
||||
var minLag = Math.Max(1, (int)MathF.Floor(frameRate * 60f / maxBpm));
|
||||
var maxLag = Math.Min(onset.Length - 1, (int)MathF.Ceiling(frameRate * 60f / minBpm));
|
||||
var bestLag = 0;
|
||||
var best = 0f;
|
||||
var energy = onset.Sum(value => value * value);
|
||||
if (energy <= 1e-9f)
|
||||
{
|
||||
return new TempoEstimate(0, 0, 0);
|
||||
}
|
||||
|
||||
for (var lag = minLag; lag <= maxLag; lag++)
|
||||
{
|
||||
var sum = 0f;
|
||||
for (var index = 0; index + lag < onset.Length; index++)
|
||||
{
|
||||
sum += onset[index] * onset[index + lag];
|
||||
}
|
||||
|
||||
var normalized = sum / energy;
|
||||
if (normalized > best)
|
||||
{
|
||||
best = normalized;
|
||||
bestLag = lag;
|
||||
}
|
||||
}
|
||||
|
||||
if (bestLag <= 0)
|
||||
{
|
||||
return new TempoEstimate(0, 0, 0);
|
||||
}
|
||||
|
||||
var beatSeconds = bestLag / frameRate;
|
||||
var bpm = 60f / Math.Max(1e-6f, beatSeconds);
|
||||
return new TempoEstimate(bpm, beatSeconds, Math.Clamp(best, 0, 1));
|
||||
}
|
||||
|
||||
static SongRegister EstimateRegister(IReadOnlyList<float> samples, int sampleRate, float spectralCentroidHz)
|
||||
{
|
||||
var dominantHz = DominantFrequency(samples, sampleRate);
|
||||
if (dominantHz <= 0)
|
||||
{
|
||||
var fallback = Math.Clamp(spectralCentroidHz * .25f, 55f, 880f);
|
||||
dominantHz = float.IsFinite(fallback) ? fallback : 220f;
|
||||
}
|
||||
|
||||
var midi = (int)MathF.Round(69f + 12f * MathF.Log2(dominantHz / 440f));
|
||||
var rootMidi = midi;
|
||||
while (rootMidi > 72)
|
||||
{
|
||||
rootMidi -= 12;
|
||||
}
|
||||
|
||||
while (rootMidi < 36)
|
||||
{
|
||||
rootMidi += 12;
|
||||
}
|
||||
|
||||
var rootHz = MidiToFrequency(rootMidi);
|
||||
var lowHz = rootHz;
|
||||
while (lowHz > 220f)
|
||||
{
|
||||
lowHz *= .5f;
|
||||
}
|
||||
|
||||
while (lowHz < 55f)
|
||||
{
|
||||
lowHz *= 2f;
|
||||
}
|
||||
|
||||
var highHz = Math.Min(4000f, lowHz * 8f);
|
||||
var scaleName = spectralCentroidHz > 1800f ? "minor-pentatonic-plus-tritone" : "minor-pentatonic";
|
||||
var intervals = scaleName == "minor-pentatonic-plus-tritone"
|
||||
? new[] { 0, 3, 5, 6, 7, 10, 12, 15, 17, 18, 19, 22, 24 }
|
||||
: new[] { 0, 3, 5, 7, 10, 12, 15, 17, 19, 22, 24 };
|
||||
var scaleFrequencies = intervals
|
||||
.Select(interval => MidiToFrequency(rootMidi + interval))
|
||||
.Where(value => value >= lowHz * .9f && value <= highHz * 1.1f)
|
||||
.ToArray();
|
||||
|
||||
return new SongRegister(
|
||||
dominantHz,
|
||||
lowHz,
|
||||
highHz,
|
||||
NoteName(rootMidi),
|
||||
scaleName,
|
||||
scaleFrequencies);
|
||||
}
|
||||
|
||||
static float DominantFrequency(IReadOnlyList<float> samples, int sampleRate)
|
||||
{
|
||||
if (samples.Count < 128)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
var windowSize = Math.Min(samples.Count, Math.Max(2048, sampleRate / 5));
|
||||
var hopSize = Math.Max(256, windowSize / 4);
|
||||
var bestStart = 0;
|
||||
var bestEnergy = 0f;
|
||||
for (var start = 0; start + windowSize <= samples.Count; start += hopSize)
|
||||
{
|
||||
var energy = 0f;
|
||||
for (var index = start; index < start + windowSize; index++)
|
||||
{
|
||||
energy += samples[index] * samples[index];
|
||||
}
|
||||
|
||||
if (energy > bestEnergy)
|
||||
{
|
||||
bestEnergy = energy;
|
||||
bestStart = start;
|
||||
}
|
||||
}
|
||||
|
||||
if (bestEnergy <= 1e-9f)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
var minLag = Math.Max(1, sampleRate / 2000);
|
||||
var maxLag = Math.Min(windowSize - 1, sampleRate / 40);
|
||||
var bestLag = 0;
|
||||
var bestScore = 0f;
|
||||
for (var lag = minLag; lag <= maxLag; lag++)
|
||||
{
|
||||
var sum = 0f;
|
||||
var left = 0f;
|
||||
var right = 0f;
|
||||
for (var offset = 0; offset + lag < windowSize; offset++)
|
||||
{
|
||||
var a = samples[bestStart + offset];
|
||||
var b = samples[bestStart + offset + lag];
|
||||
sum += a * b;
|
||||
left += a * a;
|
||||
right += b * b;
|
||||
}
|
||||
|
||||
var normalized = sum / MathF.Sqrt(Math.Max(1e-12f, left * right));
|
||||
if (normalized > bestScore)
|
||||
{
|
||||
bestScore = normalized;
|
||||
bestLag = lag;
|
||||
}
|
||||
}
|
||||
|
||||
return bestLag <= 0 || bestScore < .08f ? 0 : sampleRate / (float)bestLag;
|
||||
}
|
||||
|
||||
static float MidiToFrequency(int midi) =>
|
||||
440f * MathF.Pow(2f, (midi - 69) / 12f);
|
||||
|
||||
static string NoteName(int midi)
|
||||
{
|
||||
ReadOnlySpan<string> names = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"];
|
||||
var octave = midi / 12 - 1;
|
||||
var pitchClass = ((midi % 12) + 12) % 12;
|
||||
return $"{names[pitchClass]}{octave}";
|
||||
}
|
||||
|
||||
static float[] RmsEnvelope(IReadOnlyList<float> samples, int frameSize)
|
||||
{
|
||||
var envelope = new List<float>();
|
||||
for (var start = 0; start < samples.Count; start += frameSize)
|
||||
{
|
||||
var end = Math.Min(samples.Count, start + frameSize);
|
||||
var sum = 0f;
|
||||
for (var index = start; index < end; index++)
|
||||
{
|
||||
sum += samples[index] * samples[index];
|
||||
}
|
||||
|
||||
envelope.Add(MathF.Sqrt(sum / Math.Max(1, end - start)));
|
||||
}
|
||||
|
||||
return envelope.ToArray();
|
||||
}
|
||||
|
||||
static IReadOnlyList<SpeechScoreMetric> SongComparisonMetrics(AudioComparison comparison) =>
|
||||
[
|
||||
new("log_mel_cosine", comparison.LogMelCosineSimilarity, .25f),
|
||||
new("log_mel_distance", comparison.LogMelDistance, .15f),
|
||||
new("audio_score", comparison.Score, .15f),
|
||||
new("envelope_distance", comparison.EnvelopeDistance, .10f),
|
||||
new("rms_ratio", comparison.RmsRatio, .10f),
|
||||
new("centroid_ratio", comparison.CentroidRatio, .10f),
|
||||
new("zero_crossing_ratio", comparison.ZeroCrossingRatio, .05f),
|
||||
new("articulation_score", comparison.Articulation.ArticulationScore, .05f),
|
||||
new("speech_band_ratio", comparison.Articulation.SpeechBandRatio, .05f)
|
||||
];
|
||||
|
||||
static string SongVerdict(AudioComparison comparison)
|
||||
{
|
||||
if (comparison.LogMelCosineSimilarity >= .70f && comparison.Score >= .45f)
|
||||
{
|
||||
return "promising";
|
||||
}
|
||||
|
||||
if (comparison.LogMelCosineSimilarity >= .35f || comparison.Score >= .30f)
|
||||
{
|
||||
return "pressure";
|
||||
}
|
||||
|
||||
return "weak";
|
||||
}
|
||||
|
||||
static float SongConfidence(IReadOnlyList<SpeechScoreMetric> metrics)
|
||||
{
|
||||
var cosine = metrics.FirstOrDefault(metric => metric.Name == "log_mel_cosine")?.Value ?? 0;
|
||||
var score = metrics.FirstOrDefault(metric => metric.Name == "audio_score")?.Value ?? 0;
|
||||
return Math.Clamp((cosine + score) * .5f, 0, 1);
|
||||
}
|
||||
|
||||
static string SongEvaluationSentence(SongChallenge challenge, AudioComparison comparison, string verdict) =>
|
||||
$"Song snippet `{challenge.ChallengeId}` is `{verdict}`: logMelCosine={comparison.LogMelCosineSimilarity:0.0000}, logMelDistance={comparison.LogMelDistance:0.0000}, score={comparison.Score:0.0000}, rmsRatio={comparison.RmsRatio:0.0000}, centroidRatio={comparison.CentroidRatio:0.0000}.";
|
||||
|
||||
static string SongChallengeReport(SongChallenge challenge) =>
|
||||
$"""
|
||||
# Song Snippet Challenge: {challenge.ChallengeId}
|
||||
|
||||
source: `{challenge.SourcePath}`
|
||||
file: `{challenge.SourceFileName}`
|
||||
seed: `{challenge.Seed}`
|
||||
start_seconds: `{challenge.StartSeconds.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
duration_seconds: `{challenge.DurationSeconds.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
sample_rate: `{challenge.SampleRate}`
|
||||
reference_wav: `{challenge.ReferenceWavPath}`
|
||||
|
||||
## Reference Features
|
||||
peak: `{challenge.Features.Peak.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
rms: `{challenge.Features.Rms.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
active_duty: `{challenge.Features.ActiveDuty.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
zero_crossing_rate: `{challenge.Features.ZeroCrossingRate.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
spectral_centroid_hz: `{challenge.Features.SpectralCentroidHz.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
spectral_rolloff_hz: `{challenge.Features.SpectralRolloffHz.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
spectral_flux: `{challenge.Features.SpectralFlux.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
tempo_bpm: `{challenge.Features.TempoBpm.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
beat_seconds: `{challenge.Features.BeatSeconds.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
tempo_confidence: `{challenge.Features.TempoConfidence.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
dominant_hz: `{challenge.Features.DominantHz.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
register_low_hz: `{challenge.Features.RegisterLowHz.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
register_high_hz: `{challenge.Features.RegisterHighHz.ToString("0.######", CultureInfo.InvariantCulture)}`
|
||||
root_note: `{challenge.Features.RootNote}`
|
||||
suggested_scale: `{challenge.Features.SuggestedScale}`
|
||||
scale_frequencies_hz: `{challenge.Features.ScaleFrequenciesHz}`
|
||||
""";
|
||||
|
||||
static string SongComparisonReport(SongChallenge challenge, IpaTrialScriptCandidate candidate, AudioComparison comparison, string verdict) =>
|
||||
$"""
|
||||
candidate={candidate.CandidateId}
|
||||
challenge={challenge.ChallengeId}
|
||||
verdict={verdict}
|
||||
logMelCosine={comparison.LogMelCosineSimilarity:0.######}
|
||||
logMelDistance={comparison.LogMelDistance:0.######}
|
||||
score={comparison.Score:0.######}
|
||||
envelopeDistance={comparison.EnvelopeDistance:0.######}
|
||||
rmsRatio={comparison.RmsRatio:0.######}
|
||||
centroidRatio={comparison.CentroidRatio:0.######}
|
||||
zeroCrossingRatio={comparison.ZeroCrossingRatio:0.######}
|
||||
articulation={comparison.Articulation.ArticulationScore:0.######}
|
||||
speechBandRatio={comparison.Articulation.SpeechBandRatio:0.######}
|
||||
""";
|
||||
|
||||
static string SongSummaryCsv(IReadOnlyList<IpaTrialResult> results)
|
||||
{
|
||||
var builder = new StringBuilder();
|
||||
builder.AppendLine("trial_id,candidate_id,reference_id,verdict,log_mel_cosine,log_mel_distance,audio_score,envelope_distance,rms_ratio,centroid_ratio,zero_crossing_ratio,articulation_score");
|
||||
foreach (var result in results)
|
||||
{
|
||||
builder.Append(Escape(result.TrialId)).Append(',');
|
||||
builder.Append(Escape(result.CandidateId)).Append(',');
|
||||
builder.Append(Escape(result.ReferenceId)).Append(',');
|
||||
builder.Append(Escape(result.Verdict)).Append(',');
|
||||
builder.Append(Metric(result, "log_mel_cosine")).Append(',');
|
||||
builder.Append(Metric(result, "log_mel_distance")).Append(',');
|
||||
builder.Append(Metric(result, "audio_score")).Append(',');
|
||||
builder.Append(Metric(result, "envelope_distance")).Append(',');
|
||||
builder.Append(Metric(result, "rms_ratio")).Append(',');
|
||||
builder.Append(Metric(result, "centroid_ratio")).Append(',');
|
||||
builder.Append(Metric(result, "zero_crossing_ratio")).Append(',');
|
||||
builder.AppendLine(Metric(result, "articulation_score"));
|
||||
}
|
||||
|
||||
return builder.ToString();
|
||||
}
|
||||
|
||||
static string SongEvaluatorReport(SongChallenge challenge, IReadOnlyList<IpaTrialResult> results)
|
||||
{
|
||||
var builder = new StringBuilder();
|
||||
builder.AppendLine($"# Song Snippet Evaluator Report: {challenge.ChallengeId}");
|
||||
builder.AppendLine();
|
||||
builder.AppendLine(SongChallengeReport(challenge));
|
||||
builder.AppendLine();
|
||||
builder.AppendLine("| candidate | verdict | logMelCosine | score | envelopeDistance | rmsRatio | centroidRatio |");
|
||||
builder.AppendLine("| --- | --- | ---: | ---: | ---: | ---: | ---: |");
|
||||
foreach (var result in results.OrderByDescending(result => MetricValue(result, "log_mel_cosine")))
|
||||
{
|
||||
builder.AppendLine(CultureInfo.InvariantCulture, $"| {result.CandidateId} | {result.Verdict} | {MetricValue(result, "log_mel_cosine"):0.######} | {MetricValue(result, "audio_score"):0.######} | {MetricValue(result, "envelope_distance"):0.######} | {MetricValue(result, "rms_ratio"):0.######} | {MetricValue(result, "centroid_ratio"):0.######} |");
|
||||
}
|
||||
|
||||
return builder.ToString();
|
||||
}
|
||||
|
||||
static float MetricValue(IpaTrialResult result, string name) =>
|
||||
result.Metrics.FirstOrDefault(metric => metric.Name.Equals(name, StringComparison.OrdinalIgnoreCase))?.Value ?? 0;
|
||||
|
||||
static JsonSerializerOptions JsonOptions() => new(JsonSerializerDefaults.Web)
|
||||
{
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||
WriteIndented = true
|
||||
};
|
||||
|
||||
static SearchOptions SearchOptionsFrom(Dictionary<string, string> options) => new(
|
||||
|
|
@ -1367,6 +2155,16 @@ static bool BoolValue(Dictionary<string, string> options, string key) =>
|
|||
value.Equals("1", StringComparison.OrdinalIgnoreCase) ||
|
||||
value.Equals("yes", StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
static int IntValue(Dictionary<string, string> options, string key, int fallback) =>
|
||||
int.TryParse(Value(options, key, fallback.ToString(CultureInfo.InvariantCulture)), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed)
|
||||
? parsed
|
||||
: fallback;
|
||||
|
||||
static float FloatValue(Dictionary<string, string> options, string key, float fallback) =>
|
||||
float.TryParse(Value(options, key, fallback.ToString(CultureInfo.InvariantCulture)), NumberStyles.Float, CultureInfo.InvariantCulture, out var parsed)
|
||||
? parsed
|
||||
: fallback;
|
||||
|
||||
static void PrintHelp()
|
||||
{
|
||||
Console.WriteLine("""
|
||||
|
|
@ -1375,6 +2173,8 @@ static void PrintHelp()
|
|||
Commands:
|
||||
seed --artifact-root <dir> [--batch-id five-seed-trials] [--store <trial-results.cc>]
|
||||
score --patch-root <dir> --artifact-root <dir> [--batch-id round-001] [--store <trial-results.cc>] [--hypothesizer id]
|
||||
song-prepare --source <audio-file> --artifact-root <dir> [--duration-seconds 10] [--seed n] [--output challenge.json]
|
||||
song-score --patch-root <dir> --challenge <challenge.json> --artifact-root <dir> [--batch-id round-001] [--store <trial-results.cc>] [--hypothesizer id]
|
||||
dump --store <trial-results.cc> --output <report.md>
|
||||
index --store <trial-results.cc> [--output <report.md>] [--force true]
|
||||
search --store <trial-results.cc> --query <text> --output <report.md> [--limit 12] [--no-vector true] [--require-vector true] [--skip-index true]
|
||||
|
|
@ -1401,6 +2201,48 @@ static string ReadPayloadString(Dictionary<string, JsonElement> payload, string
|
|||
return value.ValueKind == JsonValueKind.String ? value.GetString() ?? "" : value.ToString();
|
||||
}
|
||||
|
||||
sealed record DecodedAudio(int SampleRate, float[] Samples);
|
||||
|
||||
sealed record TempoEstimate(float Bpm, float BeatSeconds, float Confidence);
|
||||
|
||||
sealed record SongChallenge(
|
||||
string ChallengeId,
|
||||
string SourcePath,
|
||||
string SourceFileName,
|
||||
int Seed,
|
||||
float StartSeconds,
|
||||
float DurationSeconds,
|
||||
int SampleRate,
|
||||
string ReferenceWavPath,
|
||||
SongChallengeFeatures Features);
|
||||
|
||||
sealed record SongChallengeFeatures(
|
||||
float DurationSeconds,
|
||||
float Peak,
|
||||
float Rms,
|
||||
float ZeroCrossingRate,
|
||||
float SpectralCentroidHz,
|
||||
float SpectralRolloffHz,
|
||||
float ActiveDuty,
|
||||
float SpectralFlux,
|
||||
float TempoBpm,
|
||||
float BeatSeconds,
|
||||
float TempoConfidence,
|
||||
float DominantHz,
|
||||
float RegisterLowHz,
|
||||
float RegisterHighHz,
|
||||
string RootNote,
|
||||
string SuggestedScale,
|
||||
string ScaleFrequenciesHz);
|
||||
|
||||
sealed record SongRegister(
|
||||
float DominantHz,
|
||||
float LowHz,
|
||||
float HighHz,
|
||||
string RootNote,
|
||||
string SuggestedScale,
|
||||
float[] ScaleFrequencies);
|
||||
|
||||
sealed record SearchOptions(
|
||||
string QdrantUrl,
|
||||
string OllamaUrl,
|
||||
|
|
|
|||
342
tools/run-song-snippet-trial-swarm.ps1
Normal file
342
tools/run-song-snippet-trial-swarm.ps1
Normal file
|
|
@ -0,0 +1,342 @@
|
|||
param(
|
||||
[int]$Passes = 1,
|
||||
[int]$AgentsPerPass = 5,
|
||||
[string]$Source = "D:\Music\Reinier\Heyoka\Gate Code\Heyoka - Alien Gibberish.mp3",
|
||||
[float]$DurationSeconds = 10,
|
||||
[int]$Seed = 0,
|
||||
[string]$CodexCommand = "codex",
|
||||
[string]$CodexModel = "gpt-5.3-codex"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
function Resolve-RepoRoot {
|
||||
$dir = Get-Item -LiteralPath (Get-Location)
|
||||
while ($null -ne $dir) {
|
||||
if (Test-Path -LiteralPath (Join-Path $dir.FullName "AquaSynth.sln")) {
|
||||
return $dir.FullName
|
||||
}
|
||||
|
||||
$dir = $dir.Parent
|
||||
}
|
||||
|
||||
throw "Could not find AquaSynth.sln above the current directory."
|
||||
}
|
||||
|
||||
function New-Timestamp {
|
||||
return [DateTimeOffset]::UtcNow.ToString("yyyyMMddTHHmmssfff")
|
||||
}
|
||||
|
||||
function Invoke-LoggedProcess {
|
||||
param(
|
||||
[string]$FilePath,
|
||||
[string[]]$ArgumentList,
|
||||
[string]$LogPath
|
||||
)
|
||||
|
||||
New-Item -ItemType Directory -Force -Path (Split-Path -Parent $LogPath) | Out-Null
|
||||
$display = "$FilePath $($ArgumentList -join ' ')"
|
||||
Add-Content -LiteralPath $LogPath -Value "[$([DateTimeOffset]::UtcNow.ToString("O"))] $display"
|
||||
& $FilePath @ArgumentList *>&1 | Tee-Object -FilePath $LogPath -Append
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Command failed with exit code $LASTEXITCODE`: $display"
|
||||
}
|
||||
}
|
||||
|
||||
function Start-CodexAgentJob {
|
||||
param(
|
||||
[string]$RepoRoot,
|
||||
[string]$Prompt,
|
||||
[string]$PromptPath,
|
||||
[string]$OutputPath,
|
||||
[string]$LogPath,
|
||||
[string]$CodexCommand,
|
||||
[string]$CodexModel,
|
||||
[string]$JobName
|
||||
)
|
||||
|
||||
New-Item -ItemType Directory -Force -Path (Split-Path -Parent $OutputPath) | Out-Null
|
||||
Set-Content -LiteralPath $PromptPath -Value $Prompt
|
||||
|
||||
$args = @(
|
||||
"exec",
|
||||
"--cd", $RepoRoot,
|
||||
"--full-auto",
|
||||
"--output-last-message", $OutputPath,
|
||||
"--config", 'shell_environment_policy.inherit="all"',
|
||||
"--config", 'sandbox_permissions=["workspace-write"]'
|
||||
)
|
||||
if ($CodexModel.Length -gt 0) {
|
||||
$args += @("--model", $CodexModel)
|
||||
}
|
||||
$args += "-"
|
||||
|
||||
Start-Job -Name $JobName -ArgumentList $CodexCommand, $args, $PromptPath, $LogPath -ScriptBlock {
|
||||
param(
|
||||
[string]$JobCodexCommand,
|
||||
[string[]]$JobArgs,
|
||||
[string]$JobPromptPath,
|
||||
[string]$JobLogPath
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Continue"
|
||||
New-Item -ItemType Directory -Force -Path (Split-Path -Parent $JobLogPath) | Out-Null
|
||||
Add-Content -LiteralPath $JobLogPath -Value "[$([DateTimeOffset]::UtcNow.ToString("O"))] $JobCodexCommand $($JobArgs -join ' ')"
|
||||
Get-Content -LiteralPath $JobPromptPath -Raw | & $JobCodexCommand @JobArgs 2>&1 | Tee-Object -FilePath $JobLogPath -Append
|
||||
[pscustomobject]@{
|
||||
ExitCode = $LASTEXITCODE
|
||||
LogPath = $JobLogPath
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Wait-CodexJobs {
|
||||
param([System.Management.Automation.Job[]]$Jobs)
|
||||
|
||||
foreach ($job in $Jobs) {
|
||||
Wait-Job -Job $job | Out-Null
|
||||
}
|
||||
|
||||
$failures = @()
|
||||
foreach ($job in $Jobs) {
|
||||
$received = Receive-Job -Job $job -Keep
|
||||
$exitRecord = $received | Where-Object { $_.PSObject.Properties.Name -contains "ExitCode" } | Select-Object -Last 1
|
||||
if ($null -eq $exitRecord -or $exitRecord.ExitCode -ne 0) {
|
||||
$failures += $job.Name
|
||||
}
|
||||
}
|
||||
|
||||
Remove-Job -Job $Jobs -Force
|
||||
if ($failures.Count -gt 0) {
|
||||
throw "External Codex job(s) failed: $($failures -join ', ')"
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-SongCandidates {
|
||||
param(
|
||||
[string]$PatchDirectory,
|
||||
[int]$ExpectedCount
|
||||
)
|
||||
|
||||
$files = @(Get-ChildItem -LiteralPath $PatchDirectory -Filter "*.aqua" -File -Recurse)
|
||||
if ($files.Count -ne $ExpectedCount) {
|
||||
throw "Song challenge wrote $($files.Count) .aqua candidates; expected $ExpectedCount."
|
||||
}
|
||||
|
||||
foreach ($file in $files) {
|
||||
$stem = [IO.Path]::GetFileNameWithoutExtension($file.Name)
|
||||
if ($stem -cnotmatch '^[a-z0-9]+(-[a-z0-9]+)*__[a-z0-9]+(-[a-z0-9]+)*__[a-z0-9]+(-[a-z0-9]+)*$') {
|
||||
throw "Candidate '$stem' must be <agent-id>__<family>__<hypothesis>.aqua with lowercase kebab-case segments."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$repoRoot = Resolve-RepoRoot
|
||||
$codex = Get-Command $CodexCommand -ErrorAction Stop
|
||||
$loopId = New-Timestamp
|
||||
$loopRoot = Join-Path $repoRoot "artifacts/parity/song-snippet-swarms/$loopId"
|
||||
$logRoot = Join-Path $loopRoot "logs"
|
||||
$workerProject = Join-Path $repoRoot "tools/IpaTrialWorker/IpaTrialWorker.csproj"
|
||||
$storePath = Join-Path $loopRoot "song-trial-results.cc"
|
||||
$challengePath = Join-Path $loopRoot "challenge.json"
|
||||
$challengeRoot = Join-Path $loopRoot "challenge"
|
||||
$seedValue = if ($Seed -eq 0) { Get-Random -Minimum 1 -Maximum ([int]::MaxValue) } else { $Seed }
|
||||
New-Item -ItemType Directory -Force -Path $loopRoot, $logRoot, $challengeRoot | Out-Null
|
||||
|
||||
Set-Content -LiteralPath (Join-Path $loopRoot "loop-index.md") -Value @"
|
||||
# Song Snippet Trial Swarm $loopId
|
||||
|
||||
- repo: $repoRoot
|
||||
- source: $Source
|
||||
- codex: $($codex.Source)
|
||||
- passes: $Passes
|
||||
- agents_per_pass: $AgentsPerPass
|
||||
- seed: $seedValue
|
||||
- duration_seconds: $DurationSeconds
|
||||
- challenge: $challengePath
|
||||
- store: $storePath
|
||||
- worker: $workerProject
|
||||
|
||||
"@
|
||||
|
||||
Invoke-LoggedProcess `
|
||||
-FilePath "dotnet" `
|
||||
-ArgumentList @(
|
||||
"run",
|
||||
"--project",
|
||||
$workerProject,
|
||||
"--",
|
||||
"song-prepare",
|
||||
"--source",
|
||||
$Source,
|
||||
"--artifact-root",
|
||||
$challengeRoot,
|
||||
"--duration-seconds",
|
||||
([string]$DurationSeconds),
|
||||
"--seed",
|
||||
([string]$seedValue),
|
||||
"--output",
|
||||
$challengePath
|
||||
) `
|
||||
-LogPath (Join-Path $logRoot "challenge-prepare.log")
|
||||
|
||||
$challengeMarkdown = Join-Path $challengeRoot "challenge.md"
|
||||
|
||||
for ($pass = 1; $pass -le $Passes; $pass++) {
|
||||
$passId = "pass-{0:000}" -f $pass
|
||||
$passDir = Join-Path $loopRoot $passId
|
||||
$patchRoot = Join-Path $passDir "proposed-patches"
|
||||
New-Item -ItemType Directory -Force -Path $passDir, $patchRoot | Out-Null
|
||||
|
||||
$preEvidence = Join-Path $passDir "semantic-search-before.md"
|
||||
if (Test-Path -LiteralPath $storePath) {
|
||||
Invoke-LoggedProcess `
|
||||
-FilePath "dotnet" `
|
||||
-ArgumentList @(
|
||||
"run",
|
||||
"--project",
|
||||
$workerProject,
|
||||
"--",
|
||||
"search",
|
||||
"--store",
|
||||
$storePath,
|
||||
"--query",
|
||||
"song snippet alien gibberish rhythm tempo timbre spectral envelope modulation scene voices weak promising",
|
||||
"--limit",
|
||||
"40",
|
||||
"--output",
|
||||
$preEvidence,
|
||||
"--skip-index",
|
||||
"true"
|
||||
) `
|
||||
-LogPath (Join-Path $logRoot "$passId-search-before.log")
|
||||
}
|
||||
else {
|
||||
Set-Content -LiteralPath $preEvidence -Value "# No prior song-snippet evidence yet"
|
||||
}
|
||||
|
||||
$jobs = @()
|
||||
for ($agent = 1; $agent -le $AgentsPerPass; $agent++) {
|
||||
$agentId = "agent-{0:00}" -f $agent
|
||||
$agentDir = Join-Path $passDir $agentId
|
||||
$agentPatchDir = Join-Path $patchRoot $agentId
|
||||
New-Item -ItemType Directory -Force -Path $agentDir, $agentPatchDir | Out-Null
|
||||
$outputPath = Join-Path $agentDir "hypothesis-agent.md"
|
||||
$promptPath = Join-Path $agentDir "hypothesis-agent.prompt.md"
|
||||
$logPath = Join-Path $logRoot "$passId-$agentId-hypothesis-agent.log"
|
||||
$prompt = @"
|
||||
You are an AquaSynth song-snippet parity worker.
|
||||
|
||||
Challenge:
|
||||
- frozen challenge JSON: $challengePath
|
||||
- human-readable challenge report: $challengeMarkdown
|
||||
- shared prior evidence: $preEvidence
|
||||
- patch output directory: $agentPatchDir
|
||||
- candidate filename prefix: $agentId
|
||||
|
||||
Goal:
|
||||
Write exactly one parseable AquaSynth `.aqua` patch that tries to reproduce the frozen 10 second reference clip. This is scene-audio parity, not IPA articulation. You may use ordinary voices, FM, AM/tremolo, filters, formants, noise layers, acoustic vocal/syrinx primitives, curves, and helper voices.
|
||||
|
||||
Rhythm and tempo:
|
||||
- The challenge report includes `tempo_bpm`, `beat_seconds`, and `tempo_confidence`, estimated from spectral/RMS onset autocorrelation.
|
||||
- Use those timing facts to build rhythmic controls.
|
||||
- Current sequencing syntax is Strudel-ish through ordinary AquaSynth parameter curves:
|
||||
- declare a parameter: `param path=/seq/kick default=0 min=0 max=1 step=.001`
|
||||
- step a pattern with hold interpolation: `curve name=kick_seq path=/seq/kick points=0:1,0.12:0,0.48:1,0.60:0 interp=hold loop=true`
|
||||
- use beat-derived times from `beat_seconds`; eighth notes are `beat_seconds/2`, sixteenths are `beat_seconds/4`.
|
||||
- bind parameters into voices or primitives, for example `gain=@/seq/kick`, `tremolo_hz=<tempo_bpm/60>`, `noise=@/seq/hiss`, `freq=@/seq/pitch`.
|
||||
- Do not invent unimplemented syntax like `s("bd sn")` unless you also make a source-edit plan; use `param` + `curve interp=hold loop=true` as the live DSP sequencing surface.
|
||||
|
||||
Register and scale:
|
||||
- The challenge report includes `dominant_hz`, `register_low_hz`, `register_high_hz`, `root_note`, `suggested_scale`, and `scale_frequencies_hz`.
|
||||
- Songs tend to pick a register and stay there. Treat those bounds as the default melodic playground unless your hypothesis explicitly tests octave spread.
|
||||
- Current scale sugar is explicit frequency sequencing:
|
||||
- declare pitch: `param path=/seq/pitch default=<dominant_hz> min=<register_low_hz> max=<register_high_hz> step=.01 unit=Hz`
|
||||
- step the scale: `curve name=lead_scale path=/seq/pitch points=0:<scale0>,0.25:<scale1>,0.5:<scale2>,0.75:<scale3> interp=hold loop=true`
|
||||
- bind it: `freq=@/seq/pitch`
|
||||
- Do not invent native `scale`, `note`, or Strudel pattern syntax in the patch unless you also edit the DSL/lowering source and document the new syntax. For candidate patches, explicit scale frequencies are the contract.
|
||||
|
||||
Evidence contract:
|
||||
- Read $challengeMarkdown and $preEvidence.
|
||||
- Write $agentDir/hypotheses.md with: reference features, tempo/rhythm plan, register/scale plan, scene voices, synthesis owners, expected metric movement, known risks.
|
||||
- Write exactly one `.aqua` file under $agentPatchDir named `<agent-id>__<family>__<hypothesis>.aqua`.
|
||||
- Include at least three scene voices or layers in the patch, such as primary alien-voice/body, rhythmic transient/noise, and background/recording color.
|
||||
- Make the patch duration/gates cover the 10 second target.
|
||||
|
||||
Return a short final summary naming the patch and report.
|
||||
"@
|
||||
$jobs += Start-CodexAgentJob `
|
||||
-RepoRoot $repoRoot `
|
||||
-Prompt $prompt `
|
||||
-PromptPath $promptPath `
|
||||
-OutputPath $outputPath `
|
||||
-LogPath $logPath `
|
||||
-CodexCommand $CodexCommand `
|
||||
-CodexModel $CodexModel `
|
||||
-JobName "$passId-$agentId"
|
||||
}
|
||||
|
||||
Wait-CodexJobs -Jobs $jobs
|
||||
Assert-SongCandidates -PatchDirectory $patchRoot -ExpectedCount $AgentsPerPass
|
||||
|
||||
Invoke-LoggedProcess `
|
||||
-FilePath "dotnet" `
|
||||
-ArgumentList @(
|
||||
"run",
|
||||
"--project",
|
||||
$workerProject,
|
||||
"--",
|
||||
"song-score",
|
||||
"--patch-root",
|
||||
$patchRoot,
|
||||
"--challenge",
|
||||
$challengePath,
|
||||
"--artifact-root",
|
||||
$passDir,
|
||||
"--batch-id",
|
||||
$passId,
|
||||
"--store",
|
||||
$storePath,
|
||||
"--hypothesizer",
|
||||
"external-codex-song-swarm-$passId"
|
||||
) `
|
||||
-LogPath (Join-Path $logRoot "$passId-score-candidates.log")
|
||||
|
||||
Invoke-LoggedProcess `
|
||||
-FilePath "dotnet" `
|
||||
-ArgumentList @(
|
||||
"run",
|
||||
"--project",
|
||||
$workerProject,
|
||||
"--",
|
||||
"index",
|
||||
"--store",
|
||||
$storePath,
|
||||
"--output",
|
||||
(Join-Path $passDir "vector-index-after.md")
|
||||
) `
|
||||
-LogPath (Join-Path $logRoot "$passId-index-after.log")
|
||||
|
||||
Invoke-LoggedProcess `
|
||||
-FilePath "dotnet" `
|
||||
-ArgumentList @(
|
||||
"run",
|
||||
"--project",
|
||||
$workerProject,
|
||||
"--",
|
||||
"search",
|
||||
"--store",
|
||||
$storePath,
|
||||
"--query",
|
||||
"$passId song snippet current candidates promising weak rhythm tempo timbre spectral scene voices",
|
||||
"--limit",
|
||||
"40",
|
||||
"--output",
|
||||
(Join-Path $passDir "semantic-search-after.md"),
|
||||
"--skip-index",
|
||||
"true"
|
||||
) `
|
||||
-LogPath (Join-Path $logRoot "$passId-search-after.log")
|
||||
}
|
||||
|
||||
Write-Host "Song snippet trial swarm complete: $loopRoot"
|
||||
Loading…
Reference in a new issue