Make music knowledge store cumulative

docs/music-production-knowledge/music-intelligence-source-map.md

@@ -0,0 +1,103 @@
+# Music Intelligence Source Map For AquaSynth
+
+kind: source-map
+tier: current-source-doctrine
+topic: external music intelligence signals for song practice targets
+
+## Owner
+
+This source map owns how AquaSynth should use external genre, mood, tempo, key,
+and audio-feature services while training song agents. External metadata is
+context and contrast, not authority over the local reference audio.
+
+## Current Landscape
+
+- Every Noise at Once is reachable as a genre/listening-map snapshot, but it is
+  no longer a maintained live discovery organ. Use it for genre vocabulary,
+  neighborhood clues, and artist/genre relation hypotheses, not current-release
+  truth.
+- Echo Nest is gone. Its public API was shut down years ago after Spotify
+  acquired it; do not design new infrastructure around Echo Nest endpoints.
+- Spotify audio features and audio analysis are deprecated/restricted for new
+  Web API use cases. Spotify also warns that Spotify content may not be used to
+  train or ingest into ML/AI models. Do not feed Spotify content or proprietary
+  feature payloads into CultCache training memory.
+- AcousticBrainz stopped collecting new data in 2022, but its public dataset
+  and API remain useful for historical MusicBrainz-linked acoustic descriptors
+  when a recording MBID is available.
+- Essentia is the strongest local/open path: it provides feature extractors and
+  models for genre/style, mood, instrumentation, tonality, pitch, tempo, source
+  separation, and tagging. Prefer local Essentia-style extraction over remote
+  black-box labels when we have the audio files.
+- Paid or hosted APIs such as Soundcharts, Tunebat, Beatlyze, SoundStat,
+  MetaMagican, Kapiko, and RapidAPI-hosted track-analysis services can provide
+  BPM/key/mood/genre/energy hints. Treat them as optional adapters with license,
+  privacy, quota, and provenance checks.
+
+## Transfer Rules
+
+- Local audio analysis owns primary truth: waveform, log-mel bands, RMS,
+  autocorrelation, key/register estimates, sections, loudness, and rendered
+  comparison metrics.
+- External metadata owns vocabulary and priors: likely genre family, mood tags,
+  related styles, energy/danceability labels, and production reference words.
+- If an external service disagrees with the local audio, preserve both and let
+  the producer brief state the conflict.
+- Never promote a genre label unless it changes a concrete AquaSynth decision:
+  tempo grid, drum pattern, bass role, harmonic language, section plan, timbre,
+  mix priority, or failure check.
+- Every external feature should carry provider, timestamp, source track id or
+  file hash, confidence if available, and licensing/usage note.
+
+## AquaSynth Integration Pattern
+
+Recommended per-target `intelligence.md` shape:
+
+```markdown
+# Target Music Intelligence
+
+- local_file_hash:
+- source_title:
+- local_features: bpm, key/root, register, active duty, spectral centroid,
+  flux, section-energy hints
+- external_labels:
+  - provider:
+    query:
+    result:
+    confidence:
+    license_or_policy_note:
+- production_translation:
+  - genre/mood clue:
+  - AquaSynth consequence:
+  - rejected assumption:
+```
+
+Recommended retrieval query additions:
+
+```text
+genre style mood energy danceability key tempo section instrumentation
+EveryNoise AcousticBrainz Essentia local audio analysis production reference
+```
+
+## Failure Modes
+
+- Treating Spotify/Echo-Nest-derived features as available public training
+  infrastructure in 2026.
+- Copying a genre label into the prompt without translating it into composition
+  or production choices.
+- Letting remote labels override local audio artifacts.
+- Forgetting provenance and later being unable to explain why the agent believed
+  a track was halftime, dark, danceable, acoustic, aggressive, or vocal-heavy.
+- Using black-box API output as curriculum doctrine instead of bounded,
+  provider-labeled evidence.
+
+## Source Anchors
+
+- Every Noise at Once: https://everynoise.com/
+- Spotify Web API changes, 2024-11-27:
+  https://developer.spotify.com/blog/2024-11-27-changes-to-the-web-api
+- Spotify Get Track's Audio Features reference:
+  https://developer.spotify.com/documentation/web-api/reference/get-audio-features
+- AcousticBrainz: https://acousticbrainz.org/
+- Essentia models: https://essentia.upf.edu/models.html
+

state/evidence.jsonl

@@ -266,3 +266,4 @@
 {"timestamp":"2026-05-31T23:15:00+01:00","topic":"producer-apprenticeship-curriculum-gate","claim":"Song curriculum admission must be owned by producer evidence and audible structure, not syntax attendance. `song-score` now records studio document coverage, producer musicianship, composition section score, template-loop risk, noise-percussion risk, and Aqua gap count; verdicts can fail as `weak-producer-missing` or `weak-slop-template`. `music-distill` promotes candidate and role doctrine only from rendered rows that clear the producer gate, while slop/noise/template rows become failure pressure.","evidence":"tools/IpaTrialWorker/Program.cs; tools/run-song-snippet-trial-swarm.ps1; docs/audio-doctrine.md; state/scratch.md"}
 {"timestamp":"2026-06-01T00:12:00+01:00","topic":"qdrant-music-knowledge-and-curator-pass","claim":"Music-production knowledge injection is now Qdrant-backed instead of lexical summary matching. `music-index` embeds full knowledge packets in `aquasynth_music_production_knowledge`; `music-search` defaults to Qdrant/Ollama retrieval and emits full owner, transfer rules, patterns, failure modes, and retrieved chunks. The song swarm now runs a dedicated knowledge-curator agent after each musician phase, and `music-distill` ingests `knowledge-curation.md` as curator-distillation doctrine pressure before indexing the next pass store.","evidence":"tools/IpaTrialWorker/Program.cs; tools/run-song-snippet-trial-swarm.ps1; docs/audio-doctrine.md; artifacts/parity/music-qdrant-smoke/20260601T000755/music-search.md"}
 {"timestamp":"2026-06-01T00:22:00+01:00","topic":"music-production-manual-ingestion","claim":"Fresh music knowledge stores now include repo-native manuals before agent-derived pressure. `docs/music-production-knowledge/aqua-dsp-usage-manual.md` teaches AquaSynth composition/DSP lowering, and `docs/music-production-knowledge/open-music-theory-production-distill.md` distills open music theory and digital production sources into actionable AquaSynth patterns. `music-distill` ingests these as `manual` documents, and Qdrant search retrieves them as full context packets.","evidence":"docs/music-production-knowledge/aqua-dsp-usage-manual.md; docs/music-production-knowledge/open-music-theory-production-distill.md; tools/IpaTrialWorker/Program.cs; artifacts/parity/manual-knowledge-smoke/20260601T001932/music-search.md"}
+{"timestamp":"2026-06-01T00:31:00+01:00","topic":"cumulative-music-knowledge-and-external-signal-map","claim":"The song swarm now owns one cumulative loop-level music-production `.cc`: it seeds manuals once, indexes them before pass one, then appends pass-learned worker/curator evidence with `music-distill --learned-only true --include-manuals false`. Added a current music-intelligence source map: Every Noise is static genre-vocabulary context, Echo Nest is gone, Spotify audio features are restricted/deprecated and not a training substrate, AcousticBrainz is historical, and local Essentia-style extraction should own primary audio signal.","evidence":"tools/run-song-snippet-trial-swarm.ps1; tools/IpaTrialWorker/Program.cs; docs/music-production-knowledge/music-intelligence-source-map.md; artifacts/parity/song-snippet-swarms/20260531T232719334/music-production-knowledge-seed.md; artifacts/parity/manual-knowledge-smoke/20260601T002939/report.md"}

state/scratch.md

@@ -1983,3 +1983,18 @@ Next likely slice:
   produced a 16-document knowledge store with 2 manuals, indexed it into
   Qdrant, and `music-search` retrieved the AquaSynth DSP manual with
   `retrieval: qdrant-ollama`.
+
+2026-06-01 cumulative music knowledge store correction:
+
+- User caught that `music-distill` should not reseed theory/manual documents
+  after every pass. The swarm now creates one loop-level
+  `music-production-knowledge.cc`, seeds it once with manuals, indexes that
+  base store, and keeps using that store for all pass retrieval.
+- After each pass, `music-distill` runs with `--input-store` equal to the
+  loop-level store, `--learned-only true`, and `--include-manuals false`, so it
+  appends/upserts worker/curator/candidate/failure evidence without rewriting
+  the manual spine.
+- Added `music-intelligence-source-map.md` as a third manual: Every Noise is
+  genre-vocabulary context, Echo Nest is dead, Spotify audio features are not a
+  usable training substrate, AcousticBrainz is historical, and local
+  Essentia-style extraction is the preferred signal path.

tools/IpaTrialWorker/Program.cs

@@ -422,19 +422,33 @@ static async Task MusicDistillAsync(Dictionary<string, string> options)
     var artifactRoot = Required(options, "artifact-root");
     var outputStore = Required(options, "output-store");
     var output = Required(options, "output");
+    var inputStore = Value(options, "input-store", "");
+    var includeManuals = !Value(options, "include-manuals", "true").Equals("false", StringComparison.OrdinalIgnoreCase);
+    var manualsOnly = BoolValue(options, "manuals-only");
+    var learnedOnly = BoolValue(options, "learned-only");
     var maxCandidates = int.TryParse(Value(options, "max-candidates", "16"), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed)
         ? Math.Clamp(parsed, 1, 100)
         : 16;
 
-    var documents = BuildMusicKnowledgeDocuments(artifactRoot, maxCandidates).ToArray();
+    var newDocuments = BuildMusicKnowledgeDocuments(artifactRoot, maxCandidates, includeManuals, manualsOnly, learnedOnly).ToArray();
+    var existingDocuments = !string.IsNullOrWhiteSpace(inputStore) && File.Exists(inputStore)
+        ? await IpaTrialResultCultCacheStore.ReadMusicProductionKnowledgeAsync(inputStore)
+        : [];
+    var documents = existingDocuments
+        .Concat(newDocuments)
+        .GroupBy(document => document.KnowledgeId, StringComparer.OrdinalIgnoreCase)
+        .Select(group => group.Last())
+        .ToArray();
     Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outputStore))!);
-    if (File.Exists(outputStore))
+    var sameStore = !string.IsNullOrWhiteSpace(inputStore) &&
+                    Path.GetFullPath(inputStore).Equals(Path.GetFullPath(outputStore), StringComparison.OrdinalIgnoreCase);
+    if (!sameStore && File.Exists(outputStore))
     {
         File.Delete(outputStore);
     }
 
     var recordDirectory = DirectoryMessagePackBackingStore.DefaultRecordDirectoryPath(outputStore);
-    if (Directory.Exists(recordDirectory))
+    if (!sameStore && Directory.Exists(recordDirectory))
     {
         Directory.Delete(recordDirectory, recursive: true);
     }
@@ -1847,7 +1861,12 @@ static string DetailReport(string store, IpaTrialResult result)
     return builder.ToString();
 }
 
-static IEnumerable<MusicProductionKnowledgeDocument> BuildMusicKnowledgeDocuments(string artifactRoot, int maxCandidates)
+static IEnumerable<MusicProductionKnowledgeDocument> BuildMusicKnowledgeDocuments(
+    string artifactRoot,
+    int maxCandidates,
+    bool includeManuals = true,
+    bool manualsOnly = false,
+    bool learnedOnly = false)
 {
     if (!Directory.Exists(artifactRoot))
     {
@@ -1855,6 +1874,16 @@ static IEnumerable<MusicProductionKnowledgeDocument> BuildMusicKnowledgeDocument
     }
 
     var created = DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture);
+    if (manualsOnly)
+    {
+        foreach (var manualDocument in MusicManualDocuments(created))
+        {
+            yield return manualDocument;
+        }
+
+        yield break;
+    }
+
     var rows = Directory.EnumerateFiles(artifactRoot, "summary.csv", SearchOption.AllDirectories)
         .Where(IsOfficialMusicArtifactPath)
         .SelectMany(ReadSongSummaryRows)
@@ -1871,52 +1900,55 @@ static IEnumerable<MusicProductionKnowledgeDocument> BuildMusicKnowledgeDocument
     var slopRejected = rows.Where(row => row.TemplateLoopRisk >= .55f || row.NoisePercussionRisk >= .55f).ToArray();
     var missingProducerEvidence = rows.Where(row => row.RequiredStudioDocsPresent < .5f || row.ProducerMusicianshipScore < .45f).ToArray();
 
-    yield return new MusicProductionKnowledgeDocument(
+    if (!learnedOnly)
-        "music-production-quality-standard-v1",
-        "quality-standard",
-        "future song curriculum admission standard",
-        "master-standard",
-        "The music curriculum store owns distilled production knowledge; raw trials and giant feature dumps are only evidence, not curriculum.",
-        "Fresh agents should retrieve role owners, transfer rules, failure modes, and AquaSynth patterns before writing patches. New runs must add compact, evidence-backed documents, not whole spectrogram dumps or undigested trial chatter.",
-        [
-            "Admit a candidate as reusable knowledge only when it renders, has explicit instrument-role ownership, includes producer/listening/gap evidence, and states what production job it owns.",
-            "Reject one-phrase-plus-texture arrangements as failure pressure; continuity across the clip is part of the production contract.",
-            "Require an explicit composition map: meter, progression or tonal center, instrument lanes, section events, composition-scale automation, and mix movement.",
-            "Prefer compact role abstractions over one-off target mimicry; a reusable owner sentence is more valuable than a small metric bump.",
-            "Reject stock loop skeletons and raw-noise percussion as failure pressure unless the listening journal proves target-specific role behavior.",
-            "Keep failed renders as failure-mode pressure only; do not let syntax wreckage dominate retrieval.",
-            "Store full artifacts on disk and cite them; CultCache curriculum records should carry bounded summaries, metrics, and transfer rules."
-        ],
-        [
-            "voice-like lead -> syrinx/acoustic source ports with pressure, opening, and radiation motion",
-            "drums/transients -> subtractive body plus filtered noise skin plus pattern gate",
-            "pads/beds -> additive/PAD layer, harmonics, and spectrum banks",
-            "recording color -> texture role with band limits and gates, not full-duration raw noise",
-            "composition form -> `meter`, `sequence`, `chords`, `scale`, `pattern`, and `mix` sugar lowering to visible `param`/`curve` owners"
-        ],
-        [
-            "Single-file stores or raw spectrogram payloads are not curriculum; distill to paged CultCache records.",
-            "Metric winners with unclear role ownership should be pressure, not doctrine.",
-            "High cosine with high mode-collapse risk is a trap: it copies a moment, not a song.",
-            "Instrument timbre without phrase, progression, lane entrances, and mix movement is sound design, not composition."
-        ],
-        rows.Select(row => row.TrialId).Take(24).ToArray(),
-        admissible.Select(row => row.CandidateId).Distinct(StringComparer.OrdinalIgnoreCase).Take(24).ToArray(),
-        [
-            new SpeechScoreMetric("input_trial_count", rows.Length, 0),
-            new SpeechScoreMetric("rendered_trial_count", rendered.Length, 0),
-            new SpeechScoreMetric("admissible_curriculum_candidate_count", admissible.Length, 0),
-            new SpeechScoreMetric("render_failed_count", failed.Length, 0),
-            new SpeechScoreMetric("mode_collapsed_count", collapsed.Length, 0),
-            new SpeechScoreMetric("slop_template_rejected_count", slopRejected.Length, 0),
-            new SpeechScoreMetric("missing_producer_evidence_count", missingProducerEvidence.Length, 0)
-        ],
-        [artifactRoot],
-        created);
-
-    foreach (var role in MusicRoleDocuments(admissible, created))
     {
-        yield return role;
+        yield return new MusicProductionKnowledgeDocument(
+            "music-production-quality-standard-v1",
+            "quality-standard",
+            "future song curriculum admission standard",
+            "master-standard",
+            "The music curriculum store owns distilled production knowledge; raw trials and giant feature dumps are only evidence, not curriculum.",
+            "Fresh agents should retrieve role owners, transfer rules, failure modes, and AquaSynth patterns before writing patches. New runs must add compact, evidence-backed documents, not whole spectrogram dumps or undigested trial chatter.",
+            [
+                "Admit a candidate as reusable knowledge only when it renders, has explicit instrument-role ownership, includes producer/listening/gap evidence, and states what production job it owns.",
+                "Reject one-phrase-plus-texture arrangements as failure pressure; continuity across the clip is part of the production contract.",
+                "Require an explicit composition map: meter, progression or tonal center, instrument lanes, section events, composition-scale automation, and mix movement.",
+                "Prefer compact role abstractions over one-off target mimicry; a reusable owner sentence is more valuable than a small metric bump.",
+                "Reject stock loop skeletons and raw-noise percussion as failure pressure unless the listening journal proves target-specific role behavior.",
+                "Keep failed renders as failure-mode pressure only; do not let syntax wreckage dominate retrieval.",
+                "Store full artifacts on disk and cite them; CultCache curriculum records should carry bounded summaries, metrics, and transfer rules."
+            ],
+            [
+                "voice-like lead -> syrinx/acoustic source ports with pressure, opening, and radiation motion",
+                "drums/transients -> subtractive body plus filtered noise skin plus pattern gate",
+                "pads/beds -> additive/PAD layer, harmonics, and spectrum banks",
+                "recording color -> texture role with band limits and gates, not full-duration raw noise",
+                "composition form -> `meter`, `sequence`, `chords`, `scale`, `pattern`, and `mix` sugar lowering to visible `param`/`curve` owners"
+            ],
+            [
+                "Single-file stores or raw spectrogram payloads are not curriculum; distill to paged CultCache records.",
+                "Metric winners with unclear role ownership should be pressure, not doctrine.",
+                "High cosine with high mode-collapse risk is a trap: it copies a moment, not a song.",
+                "Instrument timbre without phrase, progression, lane entrances, and mix movement is sound design, not composition."
+            ],
+            rows.Select(row => row.TrialId).Take(24).ToArray(),
+            admissible.Select(row => row.CandidateId).Distinct(StringComparer.OrdinalIgnoreCase).Take(24).ToArray(),
+            [
+                new SpeechScoreMetric("input_trial_count", rows.Length, 0),
+                new SpeechScoreMetric("rendered_trial_count", rendered.Length, 0),
+                new SpeechScoreMetric("admissible_curriculum_candidate_count", admissible.Length, 0),
+                new SpeechScoreMetric("render_failed_count", failed.Length, 0),
+                new SpeechScoreMetric("mode_collapsed_count", collapsed.Length, 0),
+                new SpeechScoreMetric("slop_template_rejected_count", slopRejected.Length, 0),
+                new SpeechScoreMetric("missing_producer_evidence_count", missingProducerEvidence.Length, 0)
+            ],
+            [artifactRoot],
+            created);
+
+        foreach (var role in MusicRoleDocuments(admissible, created))
+        {
+            yield return role;
+        }
     }
 
     foreach (var document in MusicFailureDocuments(failed, rows, created, artifactRoot))
@@ -1958,9 +1990,12 @@ static IEnumerable<MusicProductionKnowledgeDocument> BuildMusicKnowledgeDocument
         yield return curatorDocument;
     }
 
-    foreach (var manualDocument in MusicManualDocuments(created))
+    if (includeManuals)
     {
-        yield return manualDocument;
+        foreach (var manualDocument in MusicManualDocuments(created))
+        {
+            yield return manualDocument;
+        }
     }
 }
 
@@ -4715,7 +4750,7 @@ static void PrintHelp()
           song-score --patch-root <dir> --challenge <challenge.json> --artifact-root <dir> [--batch-id round-001] [--store <trial-results.cc>] [--hypothesizer id]
           dump  --store <trial-results.cc> --output <report.md>
           distill --store <trial-results.cc> --output-store <distilled.cc> --output <report.md> [--min-cosine .35] [--max-results 40]
-          music-distill --artifact-root <song-swarm-dir> --output-store <music-knowledge.cc> --output <report.md> [--max-candidates 16]
+          music-distill --artifact-root <song-swarm-dir> --output-store <music-knowledge.cc> --output <report.md> [--input-store <music-knowledge.cc>] [--max-candidates 16] [--manuals-only true] [--learned-only true]
           music-index --store <music-knowledge.cc> [--output <report.md>] [--force true]
           music-search --store <music-knowledge.cc> --query <text> --output <report.md> [--limit 12] [--allow-lexical-fallback true]
           music-show --store <music-knowledge.cc> --knowledge-id <id-or-topic> --output <detail.md>

tools/run-song-snippet-trial-swarm.ps1

@@ -267,7 +267,42 @@ Set-Content -LiteralPath (Join-Path $loopRoot "loop-index.md") -Value @"
 "@
 
 $latestDistilledStore = ""
-$latestMusicKnowledgeStore = ""
+$musicKnowledgeStore = Join-Path $loopRoot "music-production-knowledge.cc"
+Invoke-LoggedProcess `
+    -FilePath "dotnet" `
+    -ArgumentList @(
+        "run",
+        "--project",
+        $workerProject,
+        "--",
+        "music-distill",
+        "--artifact-root",
+        $loopRoot,
+        "--output-store",
+        $musicKnowledgeStore,
+        "--output",
+        (Join-Path $loopRoot "music-production-knowledge-seed.md"),
+        "--manuals-only",
+        "true"
+    ) `
+    -LogPath (Join-Path $logRoot "music-knowledge-seed.log")
+Invoke-LoggedProcess `
+    -FilePath "dotnet" `
+    -ArgumentList @(
+        "run",
+        "--project",
+        $workerProject,
+        "--",
+        "music-index",
+        "--store",
+        $musicKnowledgeStore,
+        "--output",
+        (Join-Path $loopRoot "music-vector-index-seed.md"),
+        "--timeout-seconds",
+        "300"
+    ) `
+    -LogPath (Join-Path $logRoot "music-knowledge-seed-index.log")
+$latestMusicKnowledgeStore = $musicKnowledgeStore
 for ($pass = 1; $pass -le $Passes; $pass++) {
     $passId = "pass-{0:000}" -f $pass
     $passDir = Join-Path $loopRoot $passId
@@ -690,7 +725,6 @@ Return a short final note naming knowledge-curation.md.
         -LogPath (Join-Path $logRoot "$passId-distill.log")
     $latestDistilledStore = $distilledStore
 
-    $musicKnowledgeStore = Join-Path $passDir "music-production-knowledge.cc"
     Invoke-LoggedProcess `
         -FilePath "dotnet" `
         -ArgumentList @(
@@ -700,13 +734,19 @@ Return a short final note naming knowledge-curation.md.
             "--",
             "music-distill",
             "--artifact-root",
-            $loopRoot,
+            $passDir,
+            "--input-store",
+            $musicKnowledgeStore,
             "--output-store",
             $musicKnowledgeStore,
             "--output",
             (Join-Path $passDir "music-production-knowledge-report.md"),
             "--max-candidates",
-            "40"
+            "40",
+            "--include-manuals",
+            "false",
+            "--learned-only",
+            "true"
         ) `
         -LogPath (Join-Path $logRoot "$passId-music-distill.log")