AquaSynth/docs/music-agent-orchestration-lessons.md

# Music Agent Orchestration Lessons

## Purpose

This report generalizes what the music agents and the AquaSynth orchestration
loop have learned so far. It is not a victory lap. Most of the useful learning
came from watching agents do weak, plausible things and then changing the
machine so those weak things stopped counting as progress.

The live lesson is simple: a music agent does not learn because it produced a
rendered patch. It learns when the system can say what role the patch attempted,
what evidence moved, what failed, what transferred to another target, and what
must be cut before the next agent repeats the same little ritual with new
filenames.

## The Shape Of The Current Machine

The current song curriculum is an orchestration loop, not a single patch writer.
It prepares local song or snippet challenges, extracts reference evidence,
dispatches external musician agents, scores rendered AquaSynth candidates,
keeps render failures as negative evidence, asks curator agents to distill
novel lessons, writes those lessons into CultCache, and indexes them into
Qdrant/Ollama so later agents retrieve evidence packets instead of stale chat
summaries.

The loop now treats these records as separate authorities:

- Challenge evidence owns the target: frozen audio path, tempo, register, scale
  hints, spectral motion, RMS motion, autocorrelation, and local intelligence
  reports.
- Candidate evidence owns what the agent actually built: `.aqua` script,
  rendered WAV or render failure, analysis artifacts, evaluator report, and
  trial metrics.
- Studio evidence owns the agent's reasoning: producer brief, hypotheses,
  listening journal, Aqua gap ledger, studio lesson, abstraction ledger, and
  instrument conventions.
- Curator evidence owns promotion into doctrine: only novel, actionable,
  evidence-backed lessons should enter the cumulative music knowledge store.
- The cumulative music-production CultCache store owns reusable curriculum
  memory. Qdrant is a derived retrieval index, not a second truth.

That separation matters. Earlier loops let a metric, a prompt, or a summary
pretend to be the whole mind. The current loop is less glamorous and more
honest: every stage has a job, and failures are first-class records.

## What The Music Agents Learned

The agents first learned that timbre tricks are not arrangement. Early winners
could get weak log-mel agreement by writing a bright lead burst and then hiding
the rest of the clip under low-motion noise, pads, or air. That looked like
activity to the scorer and sounded like a loop giving up. The curriculum now
treats full-form continuity as an invariant: distinct musical events or motif
mutations must happen after the opening, with audible activity across the
beginning, middle, and ending.

They learned that roles need owners. A patch that names `kick`, `hat`, `pad`,
or `syrinx` has not earned those names. Drums need pitched body plus filtered
skin. Leads need pitch, gate, contour, and a source/filter/modulation identity.
Pads need harmonic fields and entrance/exit or brightness motion. Texture needs
band limits, gates, and a musical job. A role name without behavior is just a
little costume on a sine wave.

They learned that texture is support, not composition. Background air, dust,
codec grit, room tone, tape hiss, and vinyl-style beds can be useful scene
voices, but full-duration broadband noise is failed evidence unless it is
shaped, gated, quiet, and subordinate to musical structure.

They learned that instrument ownership matters for future memory. Simple
oscillator blips were producing "8-bit distress signal" candidates that could
score just enough to contaminate the curriculum. The scorer now records
syrinx/acoustic voice evidence, subtractive drum evidence, additive/PAD
evidence, musical instrument score, and chip-distress risk so the system can
distinguish a musical sketch from an emergency beep wearing a hat.

They learned that a production attempt needs a studio loop. One-shot patches
are sample data. Useful agents now render their own candidate, read the same
feature vocabulary used by the target, revise several times, and leave a
listening journal explaining what sounded alive, fake, static, missing, or too
generic.

They learned that a future DSL abstraction is not magic accepted syntax. Agents
may propose role sugar, but today's candidate must use implemented AquaSynth
syntax. Proposed abstractions belong in the abstraction ledger with owner,
lowering, expected benefit, and keep/cut verdict. This stopped agents from
hallucinating syntax and calling the parser's refusal a mystery.

They learned that current AquaSynth can support composition only when musical
time is explicit. The small Strudel-ish surface now includes `meter`,
`sequence`, `scale`, `chords`, `mix`, `pattern`, and `texture`, all lowering to
visible parameters, curves, and voices. The lesson is not "add more sugar." The
lesson is "sugar may own brevity and defaults; the lowered graph still owns the
truth."

## What The Orchestrator Learned

The orchestrator learned that metrics can be fooled by cheap continuity. It now
records active coverage, motion coverage, first-second energy share, tail energy
share, and mode-collapse risk. A one-second riff followed by static texture is
not allowed to become a good lesson merely because its spectrum is vaguely in
the neighborhood.

It learned that render failure is evidence. Earlier passes could abort or lose
failed candidates, which taught nothing and hid syntax pressure. Now failed
renders write failure files, appear in evaluator reports, and can be promoted
as failure-mode knowledge when they expose bad prompt instructions, bad manual
examples, missing parser support, or environment problems.

It learned that the worker environment is part of the machine. Renderer builds
must not depend on user-profile NuGet or dotnet config. Temporary render
projects now use local HOME, APPDATA, LOCALAPPDATA, DOTNET_CLI_HOME, and
NUGET_PACKAGES paths so spawned workers do not fail for reasons unrelated to
music.

It learned that hot trial storage cannot be a monolithic archive. Full-song
stores exceeded 2 GB and whole-file MessagePack rewriting became the owner of
failure. CultCache trial-result stores moved to paged directory backing: the
`.cc` file is a manifest, records live as individual MessagePack pages, and hot
writes no longer drag cold history through RAM.

It learned that lexical "semantic" search was mostly theater. Music knowledge
retrieval is now Qdrant/Ollama-backed and returns full packets: owner, summary,
transfer rules, AquaSynth patterns, failure modes, retrieved chunks, and source
paths. Lexical fallback is a debugging path, not the normal agent memory.

It learned that manuals must seed the curriculum, but not be re-copied every
pass. The loop now starts with repo-native manuals once, then appends learned
worker and curator evidence into a cumulative store. Static doctrine remains a
spine; pass-specific lessons remain pass-specific until they earn promotion.

It learned that curation must be a separate job. Musician agents are too close
to their own patches. A curator agent reads the scored phase, extracts only
novel actionable lessons, cites evidence paths, and writes rejected-copy notes.
This keeps raw effort from becoming doctrine by emotional proximity.

It learned that local signal intelligence beats provider fantasy. Current
source maps treat Every Noise as genre vocabulary context, Echo Nest as gone,
Spotify audio features as restricted/deprecated and not a training substrate,
AcousticBrainz as historical, and local extraction as the primary signal path.
The machine stopped waiting for a convenient external oracle and started
measuring the audio it actually has.

## Mistakes We Have Learned To Stop Making

Do not count syntax attendance as musicianship. A patch can use `meter`,
`sequence`, `chords`, and `mix` and still be a dead loop. The evaluator must ask
whether time, roles, and mix motion stayed alive.

Do not let background texture impersonate arrangement. Noise can preserve RMS,
tail energy, and spectral coverage while the music collapses. Texture is a
scene role, not a substitute for phrases, sections, groove, or harmonic motion.

Do not let one metric become the ear. Log-mel cosine, envelope distance, RMS,
centroid, and autocorrelation are evidence. They are not music. The user's ear
remains the higher court, annoyingly analog and therefore expensive to fool.

Do not promote weak winners blindly. A best candidate in a weak pass may be
only the least bad failure. Distillation must preserve "pressure" and
"failure-mode" status instead of laundering every top-ranked row into a pattern
future agents should copy.

Do not make agents solve invisible targets. They need challenge reports,
candidate-side analysis, local music intelligence, and prior knowledge packets.
Without that context they optimize from vibes, and vibes have no stack trace.

Do not ask agents to learn from the previous pass if retrieval cannot actually
surface the previous pass. The store, index, search report, and prompt injection
must form one working memory path.

Do not hide parser or renderer faults behind musical criticism. If the manual
teaches `env=ad attack=... decay=...`, the parser should accept it or the manual
should be corrected. If the renderer cannot run because HOME points somewhere
forbidden, fix the environment owner before blaming the patch.

Do not let full-song scale reuse snippet storage assumptions. Long artifacts
force storage, indexing, and progress-reporting discipline. Whole-store rewrites
are not a cute implementation detail; they are a delayed outage.

Do not treat proposed abstractions as accepted reality. Agents may mine sugar
from repeated pain, but the live candidate must lower through implemented
syntax. Future-syntax proposals belong in ledgers until source work gives them
an owner.

Do not confuse previous-agent prose with durable knowledge. Raw reports are
evidence. Curated, source-linked, scored, retrievable records are memory.

## Reusable Principles

Every musical role should have an owner sentence: "X owns Y so that Z remains
true." Examples:

- `sequence` owns kick gate timing so groove accents stay inspectable.
- A subtractive drum body owns low-frequency impact so filtered noise does not
  pretend to be a kick.
- A texture voice owns band-limited recording color so full-band hiss does not
  become composition.
- `mix` owns section gain motion so roles can enter, leave, and trade attention.
- A curator document owns promotion so raw candidate reports do not become
  doctrine by default.

Every pass should separate four verdicts:

- Did the patch render?
- Did it preserve audible form across time?
- Did it use role owners that transfer?
- Did it teach anything worth retrieving later?

Every negative result should be named at the layer that failed:

- Syntax failure: parser/manual/prompt mismatch.
- Render failure: compiler/runtime/environment/toolchain owner.
- Sound failure: source, filter, envelope, timing, role, or mix owner.
- Composition failure: meter, motif, harmony, section, continuity, or attention
  owner.
- Curriculum failure: distillation, indexing, retrieval, curation, or storage
  owner.

This is the difference between a training loop and a pile of scored artifacts.
The pile has numbers. The loop has memory with teeth.

## Current Doctrine For Future Music Agents

Start with a producer brief. Name the target's meter feel, tonal center or
progression, energy contour, section map, role map, and mix priorities before
touching timbre.

Use target evidence as constraints, not decoration. Tempo/autocorrelation
informs gates. Register/root/scale informs pitch lanes. Band stats inform
filters. Motion and RMS coverage inform density and arrangement.

Build separate role owners. Leads, bass, drums, pads, and texture need separate
lanes so the next agent can adjust them without opening the whole patch like a
badly packed suitcase.

Iterate locally. Render, score, inspect the candidate analysis, revise one
hypothesis at a time, and write down what changed.

Leave useful studio evidence. The next agent needs the failed trick, the kept
trick, the source path, the metrics, the listening observation, and the Aqua gap
more than it needs a paragraph saying the process was promising.

Promote less than you produce. Most candidates are pressure. A few become
patterns. Some become failure doctrine. The rest can stay as artifacts without
being invited into the house.

## Bottom Line

The music agents have not "learned music" in the grand sense. They have learned
several important humiliations, which is better. They have learned that role
names are not roles, texture is not composition, metrics are not ears, syntax is
not musicianship, and memory that cannot retrieve its own evidence is just
archive-shaped fog.

Our orchestration has learned the matching lesson: do not ask an agent to become
a musician while feeding it weak memory, hidden failures, monolithic storage,
and one-number rewards. Give it evidence, roles, iteration, curation,
retrieval, and hard negative pressure. Then let the next patch be wrong in a
newer, more informative way.