Back IPA trial search with Qdrant embeddings
This commit is contained in:
Meta Vens
parent
7e13c72691
commit
e3753ebc98
7 changed files with 877 additions and 52 deletions
|
|
@ -671,12 +671,27 @@ workers to hand-edit evidence:
|
|||
- `score` reads agent-authored `.aqua` candidates, renders and scores them
|
||||
against the PT fixtures, and upserts typed `IpaTrialResult` records.
|
||||
- `search` is the semantic retrieval surface for the `.cc` store. It expands
|
||||
speech-control vocabulary such as vowel/nasal/fricative/stop, ranks records
|
||||
with metric-aware evidence bias, and writes a compact markdown result set.
|
||||
speech-control vocabulary such as vowel/nasal/fricative/stop, indexes trial
|
||||
summaries plus referenced local artifact snippets into Qdrant with Ollama
|
||||
embeddings when the local vector stack is available, ranks records with
|
||||
vector and metric-aware evidence bias, and writes a compact markdown result
|
||||
set. If Qdrant or Ollama is down, it falls back to the lexical/metric ranker
|
||||
and names the fallback in the report.
|
||||
- `show` drills into one trial or candidate with full metrics, artifacts,
|
||||
known contamination, hypothesis, and evaluator summary.
|
||||
- `dump` remains an audit escape hatch, not the normal agent memory path.
|
||||
|
||||
The default local vector organ mirrors VoidBot's retrieval pattern: stable
|
||||
Qdrant point IDs are derived from SHA-256 evidence chunk ids, payload owns the
|
||||
real trial/candidate/source metadata, collection metadata records the corpus and
|
||||
embedder, and Qdrant is treated as a derived index over the `.cc` truth. The
|
||||
default services are `http://127.0.0.1:6333` for Qdrant and
|
||||
`http://127.0.0.1:11434` for Ollama using `qwen3-embedding:0.6b`. `index`
|
||||
can rebuild the collection explicitly, while `search` auto-upserts current
|
||||
store evidence before querying. The outer trial loop runs `index` at round
|
||||
boundaries and passes `--skip-index true` to agent searches so each agent query
|
||||
uses fresh vectors without re-embedding the store.
|
||||
|
||||
`tools/run-ipa-trial-loop.ps1` orchestrates the outer loop with external
|
||||
`codex exec` workers. Each round searches accumulated trial memory, asks a
|
||||
hypothesis worker to write a batch of candidates, scores those candidates
|
||||
|
|
|
|||
|
|
@ -241,3 +241,4 @@
|
|||
{"date":"2026-05-29","area":"ipa-trial-loop-prompts","lesson":"Prompt smoke found that agent prompts need hard acceptance contracts, not just instructions: exact candidate count, canonical filename schema, normalized family ids, retrieval receipts, claim-to-metric audit rows, and fail-closed evaluator compliance prevent report-shaped fog from passing as loss-landscape science."}
|
||||
{"date":"2026-05-29","area":"ipa-trial-evidence-quality","lesson":"The IPA exploration loop should optimize evidence quality, not just prompt compliance. Hypothesizers must publish specificity/comparability/falsifiability/reuse ledgers per family, and evaluators must penalize vague receipts even when candidate metrics look better."}
|
||||
{"date":"2026-05-29","area":"ipa-trial-prompt-optimization","lesson":"Five prompt optimization passes showed that live external-agent retrieval is too slow and noisy for science unless tightly bounded. The sharpest hypothesizer smoke used a fixed six-receipt packet and a reasoning-only boundary, then scored evidence by contrast pairs, concrete show metrics, primitive timeline excerpts, class-consistent references, owner sentences, micro-sweep plans, and novelty gates. Production prompts now require those witnesses before treating a DSL seam as actionable."}
|
||||
{"date":"2026-05-30","area":"ipa-trial-vector-retrieval","lesson":"The IPA trial semantic retrieval organ now uses the local full-ass stack: CultCache `.cc` records remain truth, while `IpaTrialWorker index/search` derives Qdrant points with stable SHA-256 UUIDs and Ollama `qwen3-embedding:0.6b` vectors. Search reports include vector scores, metric facts, evidence tags, contrast candidates, artifact snippets, and show commands, with lexical fallback only when the vector stack is unavailable.","evidence":"tools/IpaTrialWorker/Program.cs; artifacts/parity/ipa-trial-worker-smoke/20260529T2204/qdrant-index-smoke.md; artifacts/parity/ipa-trial-worker-smoke/20260529T2204/qdrant-search-stops.md"}
|
||||
|
|
|
|||
|
|
@ -59,7 +59,7 @@
|
|||
"Faust-generated C# candidate rendering",
|
||||
"speech timing/confidence receipts for render requests, results, and checkpoints",
|
||||
"CultMesh speech render/scoring orchestration with payload manifests, peer/lease admission receipts, worker assignment, result collection, and central checkpoint application while the authority trainer owns optimizer steps and checkpoint commits",
|
||||
"frozen IPA gesture experiment rounds and local render trials with candidate scripts, primitive timelines, gesture-layer metrics, manifests, evidence JSONL, metric summaries, candidate clusters, science briefs, WAV artifacts, evaluator reports, and CultCache .cc trial-result records",
|
||||
"frozen IPA gesture experiment rounds and local render trials with candidate scripts, primitive timelines, gesture-layer metrics, manifests, evidence JSONL, metric summaries, candidate clusters, science briefs, WAV artifacts, evaluator reports, CultCache .cc trial-result records, and Qdrant/Ollama-derived semantic retrieval indexes over those records and local artifacts",
|
||||
"script scoring",
|
||||
"preset catalogs",
|
||||
"Faust emission and compiler wrapper"
|
||||
|
|
@ -92,10 +92,10 @@
|
|||
"Surge XT for modern hybrid synth modulation",
|
||||
"Cardinal/VCV-style patches for modular cable/rate semantics"
|
||||
],
|
||||
"rule": "For speech, add schema fields and learned-control lanes only when the parity harness has a loss surface that can exercise them. For reference synths, add DSL abstractions only when rebuilds prove the missing invariant. Parsing alone is inventory, not evidence of expressivity. Pink Trombone should be decomposed into reusable low-level Aqua primitives before audio golf, not copied as a bespoke clone or inflated into PT-shaped modules. After the 2026-05-29 proprioception correction, collapse graph-vocal lowering into a small set of instrumented physical primitives before more utterance golf. Gesture motion belongs in ControlSurface/ControlSpline timeline ownership, not primitive-local smoothing; cut any generated contact/port spline that invents motion not supported by primitive reference timelines. Public clients author gestures through ControlSurfaceCatalog plus ControlSplineTimeline and stream surface control dictionaries; higher-level gesture templates and phoneme_gesture are only spline emitters. IPA descriptors seed which anatomical surfaces move, while local datasets golf numeric spline defaults and patch voice code. Training reports must separate gesture_score, clean_vocal_score, and full_parity_score so full-patch recording-condition helpers cannot impersonate articulation. Frozen IPA gesture rounds own candidate scripts, primitive timelines, gesture metrics, manifests, evidence receipts, metric summaries, candidate clusters, and science briefs; `IpaTrialResult` CultCache records own rendered trial outcomes. External Codex IPA exploration must query the `.cc` store through the worker's semantic `search`/`show` organ, then write candidate batches across the five five-phoneme target sets; crude full-store dumps are audit fallback, not agent memory. Statistical science workers and audio parity stages consume those bundles without rewriting gesture evidence. If gesture scores rise but rendered audio is silent, fix the primitive source/tract owner before adding full-patch dressing."
|
||||
"rule": "For speech, add schema fields and learned-control lanes only when the parity harness has a loss surface that can exercise them. For reference synths, add DSL abstractions only when rebuilds prove the missing invariant. Parsing alone is inventory, not evidence of expressivity. Pink Trombone should be decomposed into reusable low-level Aqua primitives before audio golf, not copied as a bespoke clone or inflated into PT-shaped modules. After the 2026-05-29 proprioception correction, collapse graph-vocal lowering into a small set of instrumented physical primitives before more utterance golf. Gesture motion belongs in ControlSurface/ControlSpline timeline ownership, not primitive-local smoothing; cut any generated contact/port spline that invents motion not supported by primitive reference timelines. Public clients author gestures through ControlSurfaceCatalog plus ControlSplineTimeline and stream surface control dictionaries; higher-level gesture templates and phoneme_gesture are only spline emitters. IPA descriptors seed which anatomical surfaces move, while local datasets golf numeric spline defaults and patch voice code. Training reports must separate gesture_score, clean_vocal_score, and full_parity_score so full-patch recording-condition helpers cannot impersonate articulation. Frozen IPA gesture rounds own candidate scripts, primitive timelines, gesture metrics, manifests, evidence receipts, metric summaries, candidate clusters, and science briefs; `IpaTrialResult` CultCache records own rendered trial outcomes. External Codex IPA exploration must query the `.cc` store through the worker's semantic `search`/`show` organ, backed by Qdrant/Ollama vector retrieval when available; Qdrant is a derived index over CultCache and local artifacts, not a second truth. Crude full-store dumps are audit fallback, not agent memory. Statistical science workers and audio parity stages consume those bundles without rewriting gesture evidence. If gesture scores rise but rendered audio is silent, fix the primitive source/tract owner before adding full-patch dressing."
|
||||
},
|
||||
"ipa_vocal_tract_path": {
|
||||
"status": "This is now the active project center. Roadmap and research notes align the curriculum around structured utterance metadata -> learned utterance embedding -> learned synth automation -> vocal tract/envelope/LFO/filter/spectral controls -> rendered audio loss. The first Core contracts still defend the boundary: PhoneticIntent is the Weksa/language-owned handoff, ArticulatoryPlan is the AquaSynth realization layer, VocalTractMorphology declares anatomy capabilities, ArticulatoryConstraintReport is the host-visible veto surface, and VocalTractPlanResult is the tiny end-to-end proof artifact for host branching. IPA is an authoring input only: tokenization and feature mapping lower into language phonology profiles, timed articulatory gestures, morphology-specific tract/excitation curves, and Faust DSP. The staged support plan starts with tokenization and vowels, then core pulmonic consonants, expanded human IPA, airstream/phonation, and Weksa/alien morphologies. Optional eSpeak NG tests provide the first generated speech workout when ESPEAK_NG or espeak-ng is installed; eSpeak is intelligibility/timing/log-mel pressure, not anatomy truth. AquaSynth owns the learned utterance embedding encoder and synth driver: speech text embeddings plus prosody/emphasis hints plus Ghostlight/Epiphany-shaped character state vectors compress into utterance embeddings, and phonetic features plus that embedding feed a packed-buffer native C# backprop/Adam mapper that predicts tract, AM/FM, LFO, filter, and mel spectral-envelope controls. SpeechBackpropagationPipeline chains the two models: synth-driver target loss returns gradients with respect to its input, slices the utterance-embedding gradient, and updates the utterance encoder from downstream speech loss. The generated ground-truth training fixture renders eSpeak syllables, extracts log-mel evidence, and trains that chained pipeline toward eSpeak-seeded mel/control targets. CultMesh speech training now has payload manifests, peer/lease admission receipts, worker assignment, result collection, and central checkpoint application; workers do not own model updates. Real remote artifact chunk transfer and process supervision remain CultMesh infrastructure surfaces, not AquaSynth workaround territory. IpaGestureExperiment writes frozen descriptor/variant rounds with candidate scripts, primitive timelines, gesture metrics, manifests, and evidence JSONL, then analyzes them into metric summaries, score-band clusters, and a science brief. IpaTrialOrchestrator now writes local five-seed IPA render trials, accepts externally authored candidate scripts through RunCandidateScriptsAsync, writes WAV/report artifacts, evaluator reports, and `ipa-trial-results.cc` CultCache records. `tools/IpaTrialWorker` is the public worker tool for seed/score/search/show over the store, and `tools/run-ipa-trial-loop.ps1` dispatches external Codex hypothesis/evaluator workers around five five-phoneme target sets. The latest seed run improved voiced vowels/nasals after primitive `SourcePort` lowering gained an oscillatory carrier; plosive closure/release remains the next weak owner.",
|
||||
"status": "This is now the active project center. Roadmap and research notes align the curriculum around structured utterance metadata -> learned utterance embedding -> learned synth automation -> vocal tract/envelope/LFO/filter/spectral controls -> rendered audio loss. The first Core contracts still defend the boundary: PhoneticIntent is the Weksa/language-owned handoff, ArticulatoryPlan is the AquaSynth realization layer, VocalTractMorphology declares anatomy capabilities, ArticulatoryConstraintReport is the host-visible veto surface, and VocalTractPlanResult is the tiny end-to-end proof artifact for host branching. IPA is an authoring input only: tokenization and feature mapping lower into language phonology profiles, timed articulatory gestures, morphology-specific tract/excitation curves, and Faust DSP. The staged support plan starts with tokenization and vowels, then core pulmonic consonants, expanded human IPA, airstream/phonation, and Weksa/alien morphologies. Optional eSpeak NG tests provide the first generated speech workout when ESPEAK_NG or espeak-ng is installed; eSpeak is intelligibility/timing/log-mel pressure, not anatomy truth. AquaSynth owns the learned utterance embedding encoder and synth driver: speech text embeddings plus prosody/emphasis hints plus Ghostlight/Epiphany-shaped character state vectors compress into utterance embeddings, and phonetic features plus that embedding feed a packed-buffer native C# backprop/Adam mapper that predicts tract, AM/FM, LFO, filter, and mel spectral-envelope controls. SpeechBackpropagationPipeline chains the two models: synth-driver target loss returns gradients with respect to its input, slices the utterance-embedding gradient, and updates the utterance encoder from downstream speech loss. The generated ground-truth training fixture renders eSpeak syllables, extracts log-mel evidence, and trains that chained pipeline toward eSpeak-seeded mel/control targets. CultMesh speech training now has payload manifests, peer/lease admission receipts, worker assignment, result collection, and central checkpoint application; workers do not own model updates. Real remote artifact chunk transfer and process supervision remain CultMesh infrastructure surfaces, not AquaSynth workaround territory. IpaGestureExperiment writes frozen descriptor/variant rounds with candidate scripts, primitive timelines, gesture metrics, manifests, and evidence JSONL, then analyzes them into metric summaries, score-band clusters, and a science brief. IpaTrialOrchestrator now writes local five-seed IPA render trials, accepts externally authored candidate scripts through RunCandidateScriptsAsync, writes WAV/report artifacts, evaluator reports, and `ipa-trial-results.cc` CultCache records. `tools/IpaTrialWorker` is the public worker tool for seed/score/search/show/index over the store; search now auto-upserts Qdrant/Ollama evidence chunks and returns vector-backed reports, while `tools/run-ipa-trial-loop.ps1` dispatches external Codex hypothesis/evaluator workers around five five-phoneme target sets. The latest seed run improved voiced vowels/nasals after primitive `SourcePort` lowering gained an oscillatory carrier; plosive closure/release remains the next weak owner.",
|
||||
"lesson": "Do not build a complete IPA parser before the tract can say basic syllables. The durable machine is tokenizer -> feature mapper -> phonology profile -> gesture planner -> morphology -> tract plan -> Faust renderer, with dependency-injected seams and mocks at each boundary. Weksa should use IPA-like phonetic input without letting human IPA semantics own alien anatomy. Anatomy has veto power: if a morphology lacks bilabial closure or another required capability, AquaSynth reports that failure instead of smearing over it.",
|
||||
"next": "Start the next speech gradient descent pass from Weksa's emitted v0.1 seed batch at `E:\\Projects\\weksa\\examples\\speech-training\\tiny-panphon-v0.1\\batch.json`: 1024 bge-m3 semantic speech-text floats, variable-length PanPhon sequence evidence, 256 AquaSynth-owned phonetic-realization embedding floats from `aquasynth.panphon_sequence_encoder.v0.1`, 32 prosody/emphasis floats, 64 projected character-state floats, and 64 AquaSynth utterance-embedding output floats. Keep semantic text embeddings and phonetic realization separate. Next tighten the parity harness: load that batch into AquaSynth training records, render tiny utterance fixtures, sweep controller outputs through compiled Faust controls, score log-mel/timing/envelope loss, record per-stage decision time/latency/budget/confidence receipts, backpropagate into the utterance encoder plus synth driver, and connect Stage 0/1 IPA token spans, vowel feature bundles, and human baseline tract-area targets to that loop. For distributed work, use RunCultMeshTrainingStep as the boring authority path: admitted leased workers return gradients/results, then the trainer applies centrally. Do not attempt federated averaging until this path is exercised against real Faust-rendered speech jobs."
|
||||
},
|
||||
|
|
@ -121,9 +121,9 @@
|
|||
"binding_rule": "Parameter binding uses field-site references such as `lpf=@/macro/brightness` and `/opgraphs/0/operators/2/level`; the field owns meaning and the parameter owns the runtime value. Avoid target-routing semantics unless a new graph authority is deliberately mapped."
|
||||
},
|
||||
"last_verified": {
|
||||
"command": "powershell -NoProfile -ExecutionPolicy Bypass -File tools\\run-ipa-trial-loop.ps1 -Rounds 0 -SkipLocalTrialRun; node -e \"JSON.parse(require('fs').readFileSync('state/memory.json','utf8')); console.log('memory-json-ok')\"; git diff --check",
|
||||
"result": "Five IPA prompt-optimization passes hardened the external trial-loop prompts around evidence quality: contrast pairs, show-backed metrics, primitive timeline excerpts or explicit missing, class-consistent references, owner sentences, five micro-sweep perturbations, and novelty gates. Loop skeleton, memory JSON parse, and whitespace diff check pass.",
|
||||
"commit": "pending ipa prompt evidence quality"
|
||||
"command": "dotnet build tools\\IpaTrialWorker\\IpaTrialWorker.csproj --disable-build-servers -p:UseSharedCompilation=false -p:BuildInParallel=false -v minimal; dotnet run --no-build --no-restore --project tools\\IpaTrialWorker\\IpaTrialWorker.csproj -- index --store artifacts\\parity\\ipa-trial-worker-smoke\\20260529T2204\\ipa-trial-results.cc --output artifacts\\parity\\ipa-trial-worker-smoke\\20260529T2204\\qdrant-index-smoke.md --force true; dotnet run --no-build --no-restore --project tools\\IpaTrialWorker\\IpaTrialWorker.csproj -- search --store artifacts\\parity\\ipa-trial-worker-smoke\\20260529T2204\\ipa-trial-results.cc --query \"stop closure release reservoir weak plosive contrast primitive timeline\" --limit 8 --output artifacts\\parity\\ipa-trial-worker-smoke\\20260529T2204\\qdrant-search-stops.md --require-vector true; powershell -NoProfile -ExecutionPolicy Bypass -File tools\\run-ipa-trial-loop.ps1 -Rounds 0 -SkipLocalTrialRun",
|
||||
"result": "IPA trial retrieval now indexes CultCache trial records and local artifact snippets into Docker Qdrant using local Ollama qwen3-embedding:0.6b vectors. Build passes; index smoke wrote 25 chunks with 1024-dimensional vectors; vector search returned through qdrant-ollama with relevant stop/closure evidence, vector scores, contrast candidates, excerpts, and show commands. The trial loop skeleton still passes after adding index boundaries and skip-index agent search commands.",
|
||||
"commit": "pending ipa qdrant retrieval"
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,24 @@
|
|||
|
||||
Current slice:
|
||||
|
||||
- Upgraded the IPA trial retrieval organ from lexical metric search to a
|
||||
Qdrant/Ollama vector index modeled after VoidBot. `IpaTrialWorker search`
|
||||
now builds evidence chunks from each `IpaTrialResult`, includes metric text,
|
||||
hypotheses, evaluations, known lies, timing receipts, and local artifact
|
||||
snippets such as `.aqua`, timeline CSV, markdown, text, and DSP files, embeds
|
||||
them through local Ollama `qwen3-embedding:0.6b`, stores them in Qdrant
|
||||
collection `aquasynth_ipa_trial_results`, and returns vector-backed markdown
|
||||
search reports with vector scores, evidence tags, metric facts, contrast
|
||||
candidates, excerpts, and `show` commands. Qdrant is derived from the `.cc`
|
||||
store, not a second truth. Verified against the live Docker Qdrant and local
|
||||
Ollama server on
|
||||
`artifacts/parity/ipa-trial-worker-smoke/20260529T2204/ipa-trial-results.cc`:
|
||||
explicit index wrote 25 chunks with 1024-dim vectors, and stop-closure search
|
||||
returned through `qdrant-ollama`. The trial loop now runs `index` before and
|
||||
after each round's scoring boundary, then passes `--skip-index true` to agent
|
||||
searches so the agents query fresh vectors without re-embedding on every
|
||||
receipt.
|
||||
|
||||
- Ran five IPA prompt-optimization passes against the external trial-loop
|
||||
agent prompts. Pass 1 produced useful seam science but treated fallback file
|
||||
reads as evidence after `dotnet run` failed. Pass 2 proved live search/show
|
||||
|
|
|
|||
|
|
@ -145,7 +145,7 @@ live_invariants:
|
|||
- IPA training scoring contract is documented. Reports must keep `gesture_score`, `clean_vocal_score`, and `full_parity_score` separate. Gesture score measures descriptor-to-surface coverage, motion direction, spline contour/timing, primitive timeline consequences, and optional external articulation landmarks. Full parity patches may use FM/AM/modulators/envelopes/filters, auxiliary animated voices, and recording-condition helpers, but those helpers cannot count as articulation evidence unless gesture and clean-vocal scores move coherently.
|
||||
- `IpaGestureExperiment.WriteRound` is the first frozen batch harness for IPA hypothesis work. It writes `manifest.yaml`, candidate `.aqua` scripts, primitive `ProbeTimelineReport` CSVs, gesture-layer `metrics.csv`, and per-candidate `evidence.jsonl` under a caller-provided artifact root. `IpaGestureExperiment.AnalyzeRound` writes `analysis/metric-summary.csv`, `analysis/candidate-clusters.csv`, and `analysis/science-brief.md` for the science-worker handoff. This harness owns frozen descriptor/spline/primitive evidence only; clean vocal identity, full spectrogram parity, optimizer checkpoints, and worker orchestration stay downstream. Focused verification: `PatchScriptTests` pass 93/93.
|
||||
- `IpaTrialOrchestrator.RunAsync` is the first local render/scoring worker for IPA trials. It generates candidate patches through `IpaGestureExperiment`, renders them through Faust, compares against local PT fixture references with `AudioAnalyzer`, writes WAV/report artifacts, and stores typed `IpaTrialResult` records in `ipa-trial-results.cc`. The five-seed trial run at `tests/AquaSynth.Dsl.Tests/bin/Debug/net10.0/artifacts/parity/ipa-trials/20260529T214127955/five-seed-trials` showed voiced vowels/nasals became audible only after primitive `SourcePort` lowering gained a true oscillatory voiced carrier; sibilants are pressure, plosives remain weak. Verification: opt-in IPA trial test passed, and focused `PatchScriptTests|IpaTrialResultsRoundTripThroughCultCache` pass 94/94.
|
||||
- External IPA exploration now has a worker/tool boundary. `tools/IpaTrialWorker` owns `seed`, `score`, `search`, `show`, and audit-only `dump` commands over the typed `.cc` trial-result store; `score` renders external Codex-authored `.aqua` candidates through `IpaTrialOrchestrator.RunCandidateScriptsAsync`, while `search` is the semantic retrieval organ for hypothesis agents. `tools/run-ipa-trial-loop.ps1` orchestrates external `codex exec` hypothesis/evaluator workers around that store. The active trial shape is five target sets of five phoneme lanes each: vowels, nasals/approximants, fricatives, stops, and mixed generalization. Candidate names are exactly `<targetId>__<family>__<hypothesis-name>.aqua`, with lowercase kebab-case family ids. Three dry external Codex smoke passes refined hypothesizer/evaluator prompts into exact 25-lane output, retrieval receipts, claim audits, evidence-quality ledgers/verdicts, and fail-closed evaluator compliance. A later five-pass prompt optimization showed the best science comes from fixed receipt packets plus constrained reasoning; production prompts now require contrast pairs, `show` metric values, primitive timeline excerpts or explicit `missing`, class-consistent reference matrices, `X owns Y so Z remains true` owner sentences, at least five micro-sweep perturbations before a seam is actionable, and a novelty gate rejecting loudness/dressing-only families as loss-landscape novelty. Verification: `IpaTrialWorker` builds, `search` smoke reads the seed store, and the loop skeleton runs with `-Rounds 0 -SkipLocalTrialRun`.
|
||||
- External IPA exploration now has a worker/tool boundary. `tools/IpaTrialWorker` owns `seed`, `score`, `search`, `show`, and audit-only `dump` commands over the typed `.cc` trial-result store; `score` renders external Codex-authored `.aqua` candidates through `IpaTrialOrchestrator.RunCandidateScriptsAsync`, while `search` is the semantic retrieval organ for hypothesis agents. `search` now follows VoidBot's Qdrant shape: `.cc` records plus referenced local artifacts become evidence chunks, local Ollama `qwen3-embedding:0.6b` produces 1024-dim vectors, Qdrant collection `aquasynth_ipa_trial_results` stores stable SHA-256 UUID points with trial/candidate payload metadata, and the report surfaces vector scores, excerpts, tags, metric facts, contrast candidates, and `show` commands. Qdrant is a derived index; CultCache remains truth, and lexical/metric search is fallback when the vector stack is unavailable. `tools/run-ipa-trial-loop.ps1` orchestrates external `codex exec` hypothesis/evaluator workers around that store. The active trial shape is five target sets of five phoneme lanes each: vowels, nasals/approximants, fricatives, stops, and mixed generalization. Candidate names are exactly `<targetId>__<family>__<hypothesis-name>.aqua`, with lowercase kebab-case family ids. Three dry external Codex smoke passes refined hypothesizer/evaluator prompts into exact 25-lane output, retrieval receipts, claim audits, evidence-quality ledgers/verdicts, and fail-closed evaluator compliance. A later five-pass prompt optimization showed the best science comes from fixed receipt packets plus constrained reasoning; production prompts now require contrast pairs, `show` metric values, primitive timeline excerpts or explicit `missing`, class-consistent reference matrices, `X owns Y so Z remains true` owner sentences, at least five micro-sweep perturbations before a seam is actionable, and a novelty gate rejecting loudness/dressing-only families as loss-landscape novelty. Verification: `IpaTrialWorker` builds, Qdrant/Ollama index/search smoke reads the seed store, and the loop skeleton runs with `-Rounds 0 -SkipLocalTrialRun`.
|
||||
- TurbulenceJet now includes a release-timed burst-noise component in addition to the deterministic pressure pulse, but a stronger trial barely moved papa; the source is not the main remaining articulation bottleneck. Graph radiation now mixes 30% raw boundary flow with 70% high-passed flow after local admittance weighting, down from the over-bright 15/85 blend. Latest radiation/source artifact under `artifacts/parity/pink-trombone-utterance-logmel/20260527T174236049`: mama/papa/thrombosis cosine 0.6578/0.7692/0.5585, rmsRatio 3.0006/2.6153/2.3488, articulation 0.1437/0.1173/0.1191. Static `closure-release` under `artifacts/parity/pink-trombone-logmel/20260527T174139661` has rmsRatio 0.9900 and speechBandRatio 0.0842. This is still not speaking; next pressure is true tract resonance/passive scattering and pressure storage, not more burst gain.
|
||||
- Same-path two-port area discontinuities now emit `graph_area_energy_in_*` and `graph_area_energy_out_*` probes alongside the three-port branch `graph_connection_energy_*` probes. Use these to inspect passivity and formant-body failures at the scatter sites rather than judging only final output loudness.
|
||||
- `FaustExportOptions.DebugProbeUi` wraps graph probe signals in Faust `vbargraph` UI controls under `/debug/...` without changing audio output arity. Native streaming Faust patches now separate writable controls from read-only probe zones and can read/snapshot probe values after `ProcessBlock`; use this for acoustic graph passivity/radiation inspection, leave it off for normal parity renders.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,9 @@
|
|||
using System.Globalization;
|
||||
using System.Net.Http.Json;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
using AquaSynth.Dsl;
|
||||
using AquaSynth.Faust;
|
||||
|
|
@ -28,6 +32,9 @@ try
|
|||
case "search":
|
||||
await SearchAsync(options);
|
||||
return 0;
|
||||
case "index":
|
||||
await IndexAsync(options);
|
||||
return 0;
|
||||
case "show":
|
||||
await ShowAsync(options);
|
||||
return 0;
|
||||
|
|
@ -93,10 +100,267 @@ static async Task SearchAsync(Dictionary<string, string> options)
|
|||
: 12;
|
||||
var results = await IpaTrialResultCultCacheStore.ReadResultsAsync(store);
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(output))!);
|
||||
await File.WriteAllTextAsync(output, SearchReport(store, query, Rank(results, query, limit)), Encoding.UTF8);
|
||||
var searchOptions = SearchOptionsFrom(options);
|
||||
var ranked = await SearchRankAsync(results, query, limit, store, searchOptions);
|
||||
await File.WriteAllTextAsync(output, SearchReport(store, query, results, ranked, searchOptions), Encoding.UTF8);
|
||||
Console.WriteLine(output);
|
||||
}
|
||||
|
||||
static async Task IndexAsync(Dictionary<string, string> options)
|
||||
{
|
||||
var store = Required(options, "store");
|
||||
var output = Value(options, "output", "");
|
||||
var results = await IpaTrialResultCultCacheStore.ReadResultsAsync(store);
|
||||
var searchOptions = SearchOptionsFrom(options);
|
||||
var index = await EnsureVectorIndexAsync(results, store, searchOptions, force: BoolValue(options, "force"));
|
||||
var report = new StringBuilder();
|
||||
report.AppendLine("# IPA Trial Vector Index");
|
||||
report.AppendLine();
|
||||
report.AppendLine($"store: `{store}`");
|
||||
report.AppendLine($"collection: `{searchOptions.Collection}`");
|
||||
report.AppendLine($"qdrant: `{searchOptions.QdrantUrl}`");
|
||||
report.AppendLine($"ollama: `{searchOptions.OllamaUrl}`");
|
||||
report.AppendLine($"embedder: `{searchOptions.EmbedModel}`");
|
||||
report.AppendLine($"chunks: {index.ChunkCount}");
|
||||
report.AppendLine($"vector_dimensions: {index.VectorSize}");
|
||||
report.AppendLine($"store_key: `{StoreKey(store)}`");
|
||||
if (!string.IsNullOrWhiteSpace(output))
|
||||
{
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(output))!);
|
||||
await File.WriteAllTextAsync(output, report.ToString(), Encoding.UTF8);
|
||||
Console.WriteLine(output);
|
||||
}
|
||||
else
|
||||
{
|
||||
Console.Write(report.ToString());
|
||||
}
|
||||
}
|
||||
|
||||
static async Task<IReadOnlyList<SearchHit>> SearchRankAsync(
|
||||
IReadOnlyList<IpaTrialResult> results,
|
||||
string query,
|
||||
int limit,
|
||||
string store,
|
||||
SearchOptions options)
|
||||
{
|
||||
if (!options.UseVector)
|
||||
{
|
||||
return Rank(results, query, limit, store, "lexical-disabled");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
if (!options.SkipIndex)
|
||||
{
|
||||
await EnsureVectorIndexAsync(results, store, options, force: false);
|
||||
}
|
||||
|
||||
var vectorHits = await QueryVectorIndexAsync(query, limit * 3, store, options);
|
||||
if (vectorHits.Count > 0)
|
||||
{
|
||||
return MergeVectorAndLexicalHits(results, query, limit, store, vectorHits);
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (!options.RequireVector)
|
||||
{
|
||||
return Rank(results, query, limit, store, $"vector-fallback:{ex.GetType().Name}:{ex.Message}");
|
||||
}
|
||||
|
||||
return Rank(results, query, limit, store, "vector-empty-fallback");
|
||||
}
|
||||
|
||||
static async Task<VectorIndexResult> EnsureVectorIndexAsync(
|
||||
IReadOnlyList<IpaTrialResult> results,
|
||||
string store,
|
||||
SearchOptions options,
|
||||
bool force)
|
||||
{
|
||||
using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(options.TimeoutSeconds) };
|
||||
var chunks = BuildEvidenceChunks(results, store).ToArray();
|
||||
if (chunks.Length == 0)
|
||||
{
|
||||
return new VectorIndexResult(0, 0);
|
||||
}
|
||||
|
||||
if (force)
|
||||
{
|
||||
await DeleteCollectionIfExistsAsync(http, options);
|
||||
}
|
||||
|
||||
var sampleVector = await EmbedOneAsync(http, options, chunks[0].Text);
|
||||
await EnsureQdrantCollectionAsync(http, options, sampleVector.Length);
|
||||
|
||||
const int batchSize = 32;
|
||||
for (var index = 0; index < chunks.Length; index += batchSize)
|
||||
{
|
||||
var batch = chunks.Skip(index).Take(batchSize).ToArray();
|
||||
var vectors = await EmbedManyAsync(http, options, batch.Select(chunk => chunk.Text).ToArray());
|
||||
if (vectors.Length != batch.Length)
|
||||
{
|
||||
throw new InvalidDataException($"Ollama returned {vectors.Length} vectors for {batch.Length} evidence chunks.");
|
||||
}
|
||||
|
||||
var points = batch.Select((chunk, offset) => new QdrantPoint(
|
||||
StableUuid(chunk.Id),
|
||||
vectors[offset],
|
||||
QdrantPayloadFrom(chunk, options.EmbedModel))).ToArray();
|
||||
var upsert = new QdrantUpsertRequest(points);
|
||||
var response = await http.PutAsJsonAsync(
|
||||
$"{options.QdrantUrl.TrimEnd('/')}/collections/{Uri.EscapeDataString(options.Collection)}/points?wait=true",
|
||||
upsert,
|
||||
JsonOptions());
|
||||
response.EnsureSuccessStatusCode();
|
||||
}
|
||||
|
||||
await EnsurePayloadIndexesAsync(http, options);
|
||||
return new VectorIndexResult(chunks.Length, sampleVector.Length);
|
||||
}
|
||||
|
||||
static async Task<IReadOnlyList<VectorSearchHit>> QueryVectorIndexAsync(
|
||||
string query,
|
||||
int limit,
|
||||
string store,
|
||||
SearchOptions options)
|
||||
{
|
||||
using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(options.TimeoutSeconds) };
|
||||
var queryVector = await EmbedOneAsync(http, options, query);
|
||||
var request = new QdrantSearchRequest(
|
||||
queryVector,
|
||||
Math.Clamp(limit, 1, 100),
|
||||
true,
|
||||
false,
|
||||
QdrantFilter.Store(StoreKey(store), options.EmbedModel));
|
||||
var response = await http.PostAsJsonAsync(
|
||||
$"{options.QdrantUrl.TrimEnd('/')}/collections/{Uri.EscapeDataString(options.Collection)}/points/search",
|
||||
request,
|
||||
JsonOptions());
|
||||
response.EnsureSuccessStatusCode();
|
||||
var result = await response.Content.ReadFromJsonAsync<QdrantSearchResponse>(JsonOptions())
|
||||
?? new QdrantSearchResponse([]);
|
||||
return result.Result
|
||||
.Select(VectorSearchHitFrom)
|
||||
.Where(hit => !string.IsNullOrWhiteSpace(hit.TrialId))
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
static IReadOnlyList<SearchHit> MergeVectorAndLexicalHits(
|
||||
IReadOnlyList<IpaTrialResult> results,
|
||||
string query,
|
||||
int limit,
|
||||
string store,
|
||||
IReadOnlyList<VectorSearchHit> vectorHits)
|
||||
{
|
||||
var lexical = Rank(results, query, Math.Max(limit, results.Count), store, "hybrid-lexical");
|
||||
var lexicalByTrial = lexical.ToDictionary(hit => hit.Result.TrialId, StringComparer.OrdinalIgnoreCase);
|
||||
var vectorByTrial = vectorHits
|
||||
.GroupBy(hit => hit.TrialId, StringComparer.OrdinalIgnoreCase)
|
||||
.ToDictionary(group => group.Key, group => group.OrderByDescending(hit => hit.Score).First(), StringComparer.OrdinalIgnoreCase);
|
||||
var resultsByTrial = results.ToDictionary(result => result.TrialId, StringComparer.OrdinalIgnoreCase);
|
||||
return vectorByTrial.Values
|
||||
.Where(hit => resultsByTrial.ContainsKey(hit.TrialId))
|
||||
.Select(hit =>
|
||||
{
|
||||
var result = resultsByTrial[hit.TrialId];
|
||||
var lexicalHit = lexicalByTrial.GetValueOrDefault(result.TrialId);
|
||||
var matchedByField = lexicalHit?.MatchedByField ?? new Dictionary<string, string[]>(StringComparer.OrdinalIgnoreCase);
|
||||
var matchedTerms = lexicalHit?.MatchedTerms ?? [];
|
||||
var tags = EvidenceTags(result, SearchDocument.From(result, DocumentFields(result, store)));
|
||||
var lexicalScore = lexicalHit?.Score ?? 0;
|
||||
var score = hit.Score * 10f + MathF.Min(lexicalScore, 8f) + MetricBias(result, ExpandTerms(Tokenize(query)).ToArray());
|
||||
return new SearchHit(
|
||||
result,
|
||||
score,
|
||||
matchedTerms,
|
||||
matchedByField,
|
||||
tags,
|
||||
"qdrant-ollama",
|
||||
hit.Score,
|
||||
hit.ChunkId,
|
||||
hit.Text);
|
||||
})
|
||||
.OrderByDescending(hit => hit.Score)
|
||||
.ThenByDescending(hit => ScoreSort(hit.Result))
|
||||
.Take(limit)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
static async Task DeleteCollectionIfExistsAsync(HttpClient http, SearchOptions options)
|
||||
{
|
||||
var response = await http.GetAsync($"{options.QdrantUrl.TrimEnd('/')}/collections/{Uri.EscapeDataString(options.Collection)}");
|
||||
if (!response.IsSuccessStatusCode)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var delete = await http.DeleteAsync($"{options.QdrantUrl.TrimEnd('/')}/collections/{Uri.EscapeDataString(options.Collection)}");
|
||||
delete.EnsureSuccessStatusCode();
|
||||
}
|
||||
|
||||
static async Task EnsureQdrantCollectionAsync(HttpClient http, SearchOptions options, int vectorSize)
|
||||
{
|
||||
var url = $"{options.QdrantUrl.TrimEnd('/')}/collections/{Uri.EscapeDataString(options.Collection)}";
|
||||
var existing = await http.GetAsync(url);
|
||||
if (existing.IsSuccessStatusCode)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var create = new QdrantCreateCollectionRequest(
|
||||
new QdrantVectorConfig(vectorSize, "Cosine", true),
|
||||
true,
|
||||
new Dictionary<string, object>
|
||||
{
|
||||
["managedBy"] = "aquasynth",
|
||||
["corpusKind"] = "ipa_trial_result",
|
||||
["embedderId"] = options.EmbedModel,
|
||||
["schemaVersion"] = 1
|
||||
});
|
||||
var response = await http.PutAsJsonAsync(url, create, JsonOptions());
|
||||
response.EnsureSuccessStatusCode();
|
||||
}
|
||||
|
||||
static async Task EnsurePayloadIndexesAsync(HttpClient http, SearchOptions options)
|
||||
{
|
||||
foreach (var field in new[] { "storeKey", "trialId", "candidateId", "targetSetId", "chunkKind", "embedderId" })
|
||||
{
|
||||
var response = await http.PutAsJsonAsync(
|
||||
$"{options.QdrantUrl.TrimEnd('/')}/collections/{Uri.EscapeDataString(options.Collection)}/index?wait=true",
|
||||
new QdrantPayloadIndexRequest(field, "keyword"),
|
||||
JsonOptions());
|
||||
if (!response.IsSuccessStatusCode)
|
||||
{
|
||||
// Existing indexes may be reported as an error by older Qdrant builds.
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static async Task<float[]> EmbedOneAsync(HttpClient http, SearchOptions options, string text) =>
|
||||
(await EmbedManyAsync(http, options, [text])).FirstOrDefault()
|
||||
?? throw new InvalidDataException("Ollama returned no embedding.");
|
||||
|
||||
static async Task<float[][]> EmbedManyAsync(HttpClient http, SearchOptions options, string[] texts)
|
||||
{
|
||||
var response = await http.PostAsJsonAsync(
|
||||
$"{options.OllamaUrl.TrimEnd('/')}/api/embed",
|
||||
new OllamaEmbedRequest(options.EmbedModel, texts),
|
||||
JsonOptions());
|
||||
response.EnsureSuccessStatusCode();
|
||||
var body = await response.Content.ReadFromJsonAsync<OllamaEmbedResponse>(JsonOptions())
|
||||
?? throw new InvalidDataException("Ollama returned an empty embedding response.");
|
||||
if (body.Embeddings is { Length: > 0 })
|
||||
{
|
||||
return body.Embeddings;
|
||||
}
|
||||
|
||||
if (body.Embedding is { Length: > 0 })
|
||||
{
|
||||
return [body.Embedding];
|
||||
}
|
||||
|
||||
throw new InvalidDataException("Ollama returned no embeddings.");
|
||||
}
|
||||
|
||||
static async Task ShowAsync(Dictionary<string, string> options)
|
||||
{
|
||||
var store = Required(options, "store");
|
||||
|
|
@ -162,30 +426,68 @@ static string TargetIdFromStem(string stem, IReadOnlyList<string> targetIds)
|
|||
$"Could not infer IPA target id from `{stem}`. Name files as `<targetId>__candidate-name.aqua`; known target ids: {string.Join(", ", targetIds)}.");
|
||||
}
|
||||
|
||||
static IReadOnlyList<(IpaTrialResult Result, float Score, string[] MatchedTerms)> Rank(
|
||||
static IReadOnlyList<SearchHit> Rank(
|
||||
IReadOnlyList<IpaTrialResult> results,
|
||||
string query,
|
||||
int limit)
|
||||
int limit,
|
||||
string store,
|
||||
string retrievalMode = "lexical")
|
||||
{
|
||||
var terms = ExpandTerms(Tokenize(query)).ToArray();
|
||||
return results
|
||||
.Select(result =>
|
||||
var queryTerms = ExpandTerms(Tokenize(query)).ToArray();
|
||||
var documents = results
|
||||
.Select(result => SearchDocument.From(result, DocumentFields(result, store)))
|
||||
.ToArray();
|
||||
var documentFrequency = new Dictionary<string, int>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var document in documents)
|
||||
{
|
||||
foreach (var term in document.AllTerms.Distinct(StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
var document = Tokenize(DocumentText(result)).ToArray();
|
||||
var termCounts = document.GroupBy(term => term, StringComparer.OrdinalIgnoreCase)
|
||||
.ToDictionary(group => group.Key, group => group.Count(), StringComparer.OrdinalIgnoreCase);
|
||||
var matched = terms.Where(term => termCounts.ContainsKey(term)).Distinct(StringComparer.OrdinalIgnoreCase).ToArray();
|
||||
var lexical = matched.Sum(term => 1f + MathF.Log(1 + termCounts[term]));
|
||||
var metricBias = MetricBias(result, terms);
|
||||
var score = lexical + metricBias;
|
||||
return (result, score, matched);
|
||||
documentFrequency[term] = documentFrequency.TryGetValue(term, out var existingCount) ? existingCount + 1 : 1;
|
||||
}
|
||||
}
|
||||
|
||||
var count = Math.Max(1, documents.Length);
|
||||
return results
|
||||
.Zip(documents)
|
||||
.Select(pair =>
|
||||
{
|
||||
var (result, document) = pair;
|
||||
var matchedByField = new Dictionary<string, string[]>(StringComparer.OrdinalIgnoreCase);
|
||||
var lexical = 0f;
|
||||
foreach (var (field, weightedTerms) in document.Fields)
|
||||
{
|
||||
var termCounts = weightedTerms.Terms.GroupBy(term => term, StringComparer.OrdinalIgnoreCase)
|
||||
.ToDictionary(group => group.Key, group => group.Count(), StringComparer.OrdinalIgnoreCase);
|
||||
var matched = queryTerms.Where(term => termCounts.ContainsKey(term))
|
||||
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||
.ToArray();
|
||||
if (matched.Length == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
matchedByField[field] = matched;
|
||||
foreach (var term in matched)
|
||||
{
|
||||
var df = documentFrequency.TryGetValue(term, out var value) ? value : 1;
|
||||
var idf = MathF.Log(1 + ((count - df + 0.5f) / (df + 0.5f)));
|
||||
lexical += weightedTerms.Weight * (1f + MathF.Log(1 + termCounts[term])) * Math.Max(0.1f, idf);
|
||||
}
|
||||
}
|
||||
|
||||
var matchedTerms = matchedByField.Values.SelectMany(term => term)
|
||||
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||
.ToArray();
|
||||
var metricBias = MetricBias(result, queryTerms);
|
||||
var evidenceBias = EvidenceBias(result, queryTerms, document);
|
||||
var score = lexical + metricBias + evidenceBias;
|
||||
return new SearchHit(result, score, matchedTerms, matchedByField, EvidenceTags(result, document), retrievalMode);
|
||||
})
|
||||
.Where(item => item.score > 0)
|
||||
.OrderByDescending(item => item.score)
|
||||
.ThenByDescending(item => ScoreSort(item.result))
|
||||
.ThenBy(item => item.result.TrialId, StringComparer.Ordinal)
|
||||
.Where(item => item.Score > 0)
|
||||
.OrderByDescending(item => item.Score)
|
||||
.ThenByDescending(item => ScoreSort(item.Result))
|
||||
.ThenBy(item => item.Result.TrialId, StringComparer.Ordinal)
|
||||
.Take(limit)
|
||||
.Select(item => (item.result, item.score, item.matched))
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
|
|
@ -251,34 +553,227 @@ static float MetricBias(IpaTrialResult result, IReadOnlyCollection<string> terms
|
|||
return bias;
|
||||
}
|
||||
|
||||
static string DocumentText(IpaTrialResult result)
|
||||
static float EvidenceBias(IpaTrialResult result, IReadOnlyCollection<string> terms, SearchDocument document)
|
||||
{
|
||||
var bias = 0f;
|
||||
if (terms.Contains("timeline", StringComparer.OrdinalIgnoreCase) &&
|
||||
document.Fields.TryGetValue("artifacts", out var artifacts) &&
|
||||
artifacts.Terms.Contains("timeline", StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
bias += 1.25f;
|
||||
}
|
||||
|
||||
if (terms.Contains("contrast", StringComparer.OrdinalIgnoreCase) ||
|
||||
terms.Contains("pair", StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
bias += result.Phonemes.Length > 1 ? 0.25f : 0.5f;
|
||||
}
|
||||
|
||||
if (terms.Contains("source", StringComparer.OrdinalIgnoreCase) &&
|
||||
result.Hypothesis.Contains("source", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
bias += 0.75f;
|
||||
}
|
||||
|
||||
return bias;
|
||||
}
|
||||
|
||||
static Dictionary<string, WeightedTerms> DocumentFields(IpaTrialResult result, string store)
|
||||
{
|
||||
var fields = new Dictionary<string, WeightedTerms>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["identity"] = new(4.0f, Tokenize(string.Join(' ', [
|
||||
result.TrialId,
|
||||
result.BatchId,
|
||||
result.TargetSetId,
|
||||
result.ReferenceId,
|
||||
result.CandidateId,
|
||||
result.HypothesizerId,
|
||||
result.Verdict
|
||||
]))),
|
||||
["phonetics"] = new(4.0f, Tokenize(string.Join(' ', result.Phonemes))),
|
||||
["hypothesis"] = new(2.5f, Tokenize(result.Hypothesis)),
|
||||
["evaluation"] = new(2.5f, Tokenize(result.EvaluationSummary)),
|
||||
["known_lies"] = new(2.0f, Tokenize(string.Join(' ', result.KnownLies))),
|
||||
["metrics"] = new(2.0f, Tokenize(MetricText(result))),
|
||||
["artifacts"] = new(1.75f, Tokenize(ArtifactText(result, store))),
|
||||
["timing"] = new(1.25f, Tokenize(TimingText(result)))
|
||||
};
|
||||
return fields;
|
||||
}
|
||||
|
||||
static string MetricText(IpaTrialResult result)
|
||||
{
|
||||
var builder = new StringBuilder();
|
||||
builder.Append(result.TrialId).Append(' ');
|
||||
builder.Append(result.BatchId).Append(' ');
|
||||
builder.Append(result.TargetSetId).Append(' ');
|
||||
builder.AppendJoin(' ', result.Phonemes).Append(' ');
|
||||
builder.Append(result.ReferenceId).Append(' ');
|
||||
builder.Append(result.CandidateId).Append(' ');
|
||||
builder.Append(result.HypothesizerId).Append(' ');
|
||||
builder.Append(result.Hypothesis).Append(' ');
|
||||
builder.Append(result.EvaluatorId).Append(' ');
|
||||
builder.Append(result.EvaluationSummary).Append(' ');
|
||||
builder.Append(result.Verdict).Append(' ');
|
||||
builder.AppendJoin(' ', result.KnownLies).Append(' ');
|
||||
foreach (var metric in result.Metrics)
|
||||
{
|
||||
builder.Append(metric.Name).Append(' ');
|
||||
}
|
||||
|
||||
foreach (var artifact in result.Artifacts)
|
||||
{
|
||||
builder.Append(artifact.Kind).Append(' ').Append(artifact.Uri).Append(' ');
|
||||
builder.Append(metric.Value.ToString("0.######", CultureInfo.InvariantCulture)).Append(' ');
|
||||
if (metric.Name.Contains("cosine", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
builder.Append(metric.Value < 0.2f ? "weak low failed " : "promising high ");
|
||||
}
|
||||
if (metric.Name.Contains("articulation", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
builder.Append(metric.Value < 0.3f ? "weak articulation failed " : "promising articulation ");
|
||||
}
|
||||
if (metric.Name.Contains("gesture", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
builder.Append(metric.Value < 0.6f ? "weak gesture " : "promising gesture ");
|
||||
}
|
||||
}
|
||||
|
||||
return builder.ToString();
|
||||
}
|
||||
|
||||
static string TimingText(IpaTrialResult result) =>
|
||||
string.Join(' ', result.TimingReceipts.Select(receipt =>
|
||||
$"{receipt.StageId} {receipt.Notes} latency {receipt.LatencyMilliseconds:0.###} budget {receipt.BudgetMilliseconds:0.###} confidence {receipt.Confidence:0.###}"));
|
||||
|
||||
static string ArtifactText(IpaTrialResult result, string store)
|
||||
{
|
||||
var builder = new StringBuilder();
|
||||
builder.Append(result.CandidatePatchUri).Append(' ');
|
||||
builder.Append(result.ReferenceArtifactUri).Append(' ');
|
||||
builder.Append(result.CandidateArtifactUri).Append(' ');
|
||||
builder.Append(result.PrimitiveTimelineUri).Append(' ');
|
||||
foreach (var artifact in result.Artifacts)
|
||||
{
|
||||
builder.Append(artifact.Kind).Append(' ').Append(artifact.Uri).Append(' ').Append(artifact.ContentHash).Append(' ');
|
||||
builder.Append(ReadArtifactSnippet(artifact.Uri, store)).Append(' ');
|
||||
}
|
||||
|
||||
builder.Append(ReadArtifactSnippet(result.CandidatePatchUri, store)).Append(' ');
|
||||
builder.Append(ReadArtifactSnippet(result.PrimitiveTimelineUri, store)).Append(' ');
|
||||
return builder.ToString();
|
||||
}
|
||||
|
||||
static string ReadArtifactSnippet(string uri, string store)
|
||||
{
|
||||
var path = ResolveArtifactPath(uri, store);
|
||||
if (path is null || !File.Exists(path))
|
||||
{
|
||||
return "";
|
||||
}
|
||||
|
||||
var extension = Path.GetExtension(path);
|
||||
if (!new[] { ".aqua", ".md", ".txt", ".csv", ".jsonl", ".yaml", ".yml", ".dsp" }
|
||||
.Contains(extension, StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
return "";
|
||||
}
|
||||
|
||||
using var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
|
||||
using var reader = new StreamReader(stream, Encoding.UTF8, detectEncodingFromByteOrderMarks: true);
|
||||
var buffer = new char[32768];
|
||||
var read = reader.Read(buffer, 0, buffer.Length);
|
||||
return new string(buffer, 0, read);
|
||||
}
|
||||
|
||||
static string? ResolveArtifactPath(string uri, string store)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(uri))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (File.Exists(uri))
|
||||
{
|
||||
return uri;
|
||||
}
|
||||
|
||||
if (Uri.TryCreate(uri, UriKind.Absolute, out var parsed) && parsed.IsFile)
|
||||
{
|
||||
return parsed.LocalPath;
|
||||
}
|
||||
|
||||
if (uri.StartsWith("file://", StringComparison.OrdinalIgnoreCase) &&
|
||||
Uri.TryCreate(uri, UriKind.Absolute, out var fileUri))
|
||||
{
|
||||
return fileUri.LocalPath;
|
||||
}
|
||||
|
||||
var storeDirectory = Path.GetDirectoryName(Path.GetFullPath(store));
|
||||
if (!string.IsNullOrWhiteSpace(storeDirectory))
|
||||
{
|
||||
var relative = Path.GetFullPath(Path.Combine(storeDirectory, uri));
|
||||
if (File.Exists(relative))
|
||||
{
|
||||
return relative;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
static IEnumerable<EvidenceChunk> BuildEvidenceChunks(IReadOnlyList<IpaTrialResult> results, string store)
|
||||
{
|
||||
foreach (var result in results)
|
||||
{
|
||||
var text = new StringBuilder();
|
||||
text.AppendLine($"trial {result.TrialId}");
|
||||
text.AppendLine($"candidate {result.CandidateId}");
|
||||
text.AppendLine($"target {result.TargetSetId}");
|
||||
text.AppendLine($"phonemes {string.Join(' ', result.Phonemes)}");
|
||||
text.AppendLine($"reference {result.ReferenceId}");
|
||||
text.AppendLine($"verdict {result.Verdict}");
|
||||
text.AppendLine("metrics");
|
||||
foreach (var metric in result.Metrics.OrderBy(metric => metric.Name, StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
text.AppendLine($"{metric.Name} {metric.Value.ToString("0.######", CultureInfo.InvariantCulture)} weight {metric.Weight.ToString("0.######", CultureInfo.InvariantCulture)}");
|
||||
}
|
||||
|
||||
text.AppendLine("hypothesis");
|
||||
text.AppendLine(result.Hypothesis);
|
||||
text.AppendLine("evaluation");
|
||||
text.AppendLine(result.EvaluationSummary);
|
||||
text.AppendLine("known_lies");
|
||||
text.AppendLine(string.Join(Environment.NewLine, result.KnownLies));
|
||||
text.AppendLine("artifacts");
|
||||
text.AppendLine(ArtifactText(result, store));
|
||||
yield return new EvidenceChunk(
|
||||
$"ipa-trial:{StoreKey(store)}:{result.TrialId}:summary",
|
||||
result.TrialId,
|
||||
result.CandidateId,
|
||||
result.TargetSetId,
|
||||
"summary",
|
||||
text.ToString(),
|
||||
store);
|
||||
}
|
||||
}
|
||||
|
||||
static string[] EvidenceTags(IpaTrialResult result, SearchDocument document)
|
||||
{
|
||||
var tags = new List<string>();
|
||||
var cosine = result.Metrics.FirstOrDefault(metric => metric.Name == "log_mel_cosine")?.Value;
|
||||
var articulation = result.Metrics.FirstOrDefault(metric => metric.Name == "articulation_score")?.Value;
|
||||
var gesture = result.Metrics.FirstOrDefault(metric => metric.Name == "gesture_score")?.Value;
|
||||
if (cosine is < 0.2f || articulation is < 0.25f)
|
||||
{
|
||||
tags.Add("weak");
|
||||
}
|
||||
if (cosine is > 0.5f || articulation is > 0.4f)
|
||||
{
|
||||
tags.Add("promising");
|
||||
}
|
||||
if (gesture is > 0.7f && articulation is < 0.3f)
|
||||
{
|
||||
tags.Add("gesture-audio-gap");
|
||||
}
|
||||
if (document.AllTerms.Contains("timeline", StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
tags.Add("timeline");
|
||||
}
|
||||
if (document.AllTerms.Contains("dressing", StringComparer.OrdinalIgnoreCase) ||
|
||||
document.AllTerms.Contains("fm", StringComparer.OrdinalIgnoreCase) ||
|
||||
document.AllTerms.Contains("am", StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
tags.Add("dressing");
|
||||
}
|
||||
|
||||
return tags.Distinct(StringComparer.OrdinalIgnoreCase).ToArray();
|
||||
}
|
||||
|
||||
static string[] Tokenize(string text) =>
|
||||
text.ToLowerInvariant()
|
||||
.Split([' ', '\t', '\r', '\n', ',', '.', ':', ';', '/', '\\', '`', '"', '\'', '[', ']', '(', ')', '{', '}', '<', '>', '|', '_', '-'], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)
|
||||
|
|
@ -288,17 +783,24 @@ static string[] Tokenize(string text) =>
|
|||
static string SearchReport(
|
||||
string store,
|
||||
string query,
|
||||
IReadOnlyList<(IpaTrialResult Result, float Score, string[] MatchedTerms)> ranked)
|
||||
IReadOnlyList<IpaTrialResult> allResults,
|
||||
IReadOnlyList<SearchHit> ranked,
|
||||
SearchOptions options)
|
||||
{
|
||||
var builder = new StringBuilder();
|
||||
builder.AppendLine("# IPA Trial Semantic Search");
|
||||
builder.AppendLine();
|
||||
builder.AppendLine($"store: `{store}`");
|
||||
builder.AppendLine($"query: `{query}`");
|
||||
builder.AppendLine($"retrieval: `{(ranked.Any(hit => hit.RetrievalMode == "qdrant-ollama") ? "qdrant-ollama" : "lexical-fallback")}`");
|
||||
builder.AppendLine($"collection: `{options.Collection}`");
|
||||
builder.AppendLine($"embedder: `{options.EmbedModel}`");
|
||||
builder.AppendLine($"matches: {ranked.Count}");
|
||||
builder.AppendLine();
|
||||
foreach (var (result, score, matched) in ranked)
|
||||
foreach (var hit in ranked)
|
||||
{
|
||||
var result = hit.Result;
|
||||
var contrast = ContrastCandidate(result, allResults);
|
||||
builder.Append("- ");
|
||||
builder.Append(result.TrialId);
|
||||
builder.Append(" / ");
|
||||
|
|
@ -306,9 +808,17 @@ static string SearchReport(
|
|||
builder.Append(" / ");
|
||||
builder.Append(result.Verdict);
|
||||
builder.Append(" / score=");
|
||||
builder.Append(score.ToString("0.###", CultureInfo.InvariantCulture));
|
||||
builder.Append(hit.Score.ToString("0.###", CultureInfo.InvariantCulture));
|
||||
if (hit.VectorScore is not null)
|
||||
{
|
||||
builder.Append(" / vector=");
|
||||
builder.Append(hit.VectorScore.Value.ToString("0.###", CultureInfo.InvariantCulture));
|
||||
}
|
||||
|
||||
builder.Append(" / matched=");
|
||||
builder.AppendLine(string.Join('|', matched.Take(12)));
|
||||
builder.AppendLine(string.Join('|', hit.MatchedTerms.Take(12)));
|
||||
builder.Append(" tags: ");
|
||||
builder.AppendLine(string.Join(", ", hit.Tags));
|
||||
builder.Append(" metrics: gesture=");
|
||||
builder.Append(Metric(result, "gesture_score"));
|
||||
builder.Append(", logMelCosine=");
|
||||
|
|
@ -321,6 +831,27 @@ static string SearchReport(
|
|||
builder.AppendLine(result.Hypothesis);
|
||||
builder.Append(" evaluation: ");
|
||||
builder.AppendLine(result.EvaluationSummary);
|
||||
if (contrast is not null)
|
||||
{
|
||||
builder.Append(" contrast: ");
|
||||
builder.Append(contrast.TrialId);
|
||||
builder.Append(" / ");
|
||||
builder.Append(contrast.CandidateId);
|
||||
builder.Append(" / logMelCosine=");
|
||||
builder.Append(Metric(contrast, "log_mel_cosine"));
|
||||
builder.Append(" / articulation=");
|
||||
builder.AppendLine(Metric(contrast, "articulation_score"));
|
||||
}
|
||||
if (!string.IsNullOrWhiteSpace(hit.VectorChunkId))
|
||||
{
|
||||
builder.Append(" vector_chunk: ");
|
||||
builder.AppendLine(hit.VectorChunkId);
|
||||
}
|
||||
if (!string.IsNullOrWhiteSpace(hit.VectorText))
|
||||
{
|
||||
builder.Append(" vector_excerpt: ");
|
||||
builder.AppendLine(OneLine(hit.VectorText, 280));
|
||||
}
|
||||
builder.Append(" patch: ");
|
||||
builder.AppendLine(result.CandidatePatchUri);
|
||||
builder.Append(" show: dotnet run --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- show --store \"");
|
||||
|
|
@ -441,6 +972,88 @@ static string Metric(IpaTrialResult result, string name) =>
|
|||
(result.Metrics.FirstOrDefault(metric => metric.Name == name)?.Value ?? 0)
|
||||
.ToString("0.######", CultureInfo.InvariantCulture);
|
||||
|
||||
static IpaTrialResult? ContrastCandidate(IpaTrialResult result, IReadOnlyList<IpaTrialResult> allResults)
|
||||
{
|
||||
var sameTarget = allResults
|
||||
.Where(item => !ReferenceEquals(item, result) && item.TargetSetId.Equals(result.TargetSetId, StringComparison.OrdinalIgnoreCase))
|
||||
.OrderByDescending(item => Math.Abs(ScoreSort(item) - ScoreSort(result)))
|
||||
.ThenByDescending(item => item.Metrics.FirstOrDefault(metric => metric.Name == "articulation_score")?.Value ?? 0)
|
||||
.FirstOrDefault();
|
||||
if (sameTarget is not null)
|
||||
{
|
||||
return sameTarget;
|
||||
}
|
||||
|
||||
return allResults
|
||||
.Where(item => !ReferenceEquals(item, result) && item.Phonemes.Intersect(result.Phonemes, StringComparer.OrdinalIgnoreCase).Any())
|
||||
.OrderByDescending(item => Math.Abs(ScoreSort(item) - ScoreSort(result)))
|
||||
.FirstOrDefault();
|
||||
}
|
||||
|
||||
static string OneLine(string text, int maxLength)
|
||||
{
|
||||
var normalized = text.Replace('\r', ' ').Replace('\n', ' ').Trim();
|
||||
return normalized.Length <= maxLength ? normalized : normalized[..maxLength] + "...";
|
||||
}
|
||||
|
||||
static string StoreKey(string store)
|
||||
{
|
||||
var fullPath = Path.GetFullPath(store).ToLowerInvariant();
|
||||
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(fullPath));
|
||||
return Convert.ToHexString(hash)[..16].ToLowerInvariant();
|
||||
}
|
||||
|
||||
static string StableUuid(string id)
|
||||
{
|
||||
var digest = SHA256.HashData(Encoding.UTF8.GetBytes(id));
|
||||
digest[6] = (byte)((digest[6] & 0x0f) | 0x50);
|
||||
digest[8] = (byte)((digest[8] & 0x3f) | 0x80);
|
||||
var hex = Convert.ToHexString(digest[..16]).ToLowerInvariant();
|
||||
return $"{hex[..8]}-{hex[8..12]}-{hex[12..16]}-{hex[16..20]}-{hex[20..32]}";
|
||||
}
|
||||
|
||||
static JsonSerializerOptions JsonOptions() => new(JsonSerializerDefaults.Web)
|
||||
{
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
|
||||
};
|
||||
|
||||
static SearchOptions SearchOptionsFrom(Dictionary<string, string> options) => new(
|
||||
Value(options, "qdrant-url", "http://127.0.0.1:6333"),
|
||||
Value(options, "ollama-url", "http://127.0.0.1:11434"),
|
||||
Value(options, "embed-model", "qwen3-embedding:0.6b"),
|
||||
Value(options, "collection", "aquasynth_ipa_trial_results"),
|
||||
int.TryParse(Value(options, "timeout-seconds", "120"), NumberStyles.Integer, CultureInfo.InvariantCulture, out var timeout)
|
||||
? Math.Clamp(timeout, 5, 600)
|
||||
: 120,
|
||||
!BoolValue(options, "no-vector"),
|
||||
BoolValue(options, "require-vector"),
|
||||
BoolValue(options, "skip-index"));
|
||||
|
||||
static QdrantPayload QdrantPayloadFrom(EvidenceChunk chunk, string embedderId) => new(
|
||||
chunk.Id,
|
||||
chunk.TrialId,
|
||||
"ipa_trial_result",
|
||||
"ipa_trial_result",
|
||||
StoreKey(chunk.Store),
|
||||
chunk.TrialId,
|
||||
chunk.CandidateId,
|
||||
chunk.TargetSetId,
|
||||
chunk.ChunkKind,
|
||||
embedderId,
|
||||
chunk.Text);
|
||||
|
||||
static VectorSearchHit VectorSearchHitFrom(QdrantScoredPoint point)
|
||||
{
|
||||
var payload = point.Payload ?? new Dictionary<string, JsonElement>();
|
||||
return new VectorSearchHit(
|
||||
ReadPayloadString(payload, "chunkId"),
|
||||
ReadPayloadString(payload, "trialId"),
|
||||
ReadPayloadString(payload, "candidateId"),
|
||||
ReadPayloadString(payload, "targetSetId"),
|
||||
ReadPayloadString(payload, "text"),
|
||||
point.Score);
|
||||
}
|
||||
|
||||
static Dictionary<string, string> ParseOptions(IReadOnlyList<string> args)
|
||||
{
|
||||
var options = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
|
@ -474,6 +1087,12 @@ static string Required(Dictionary<string, string> options, string key) =>
|
|||
static string Value(Dictionary<string, string> options, string key, string fallback) =>
|
||||
options.TryGetValue(key, out var value) && !string.IsNullOrWhiteSpace(value) ? value : fallback;
|
||||
|
||||
static bool BoolValue(Dictionary<string, string> options, string key) =>
|
||||
options.TryGetValue(key, out var value) &&
|
||||
(value.Equals("true", StringComparison.OrdinalIgnoreCase) ||
|
||||
value.Equals("1", StringComparison.OrdinalIgnoreCase) ||
|
||||
value.Equals("yes", StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
static void PrintHelp()
|
||||
{
|
||||
Console.WriteLine("""
|
||||
|
|
@ -483,10 +1102,146 @@ static void PrintHelp()
|
|||
seed --artifact-root <dir> [--batch-id five-seed-trials] [--store <trial-results.cc>]
|
||||
score --patch-root <dir> --artifact-root <dir> [--batch-id round-001] [--store <trial-results.cc>] [--hypothesizer id]
|
||||
dump --store <trial-results.cc> --output <report.md>
|
||||
search --store <trial-results.cc> --query <text> --output <report.md> [--limit 12]
|
||||
index --store <trial-results.cc> [--output <report.md>] [--force true]
|
||||
search --store <trial-results.cc> --query <text> --output <report.md> [--limit 12] [--no-vector true] [--require-vector true] [--skip-index true]
|
||||
show --store <trial-results.cc> --trial-id <id-or-candidate> --output <detail.md>
|
||||
|
||||
Vector search defaults:
|
||||
--qdrant-url http://127.0.0.1:6333
|
||||
--ollama-url http://127.0.0.1:11434
|
||||
--embed-model qwen3-embedding:0.6b
|
||||
--collection aquasynth_ipa_trial_results
|
||||
|
||||
Agent-authored patch files must be named <targetId>__candidate-name.aqua.
|
||||
Known seed target ids: a, i, u, e, o, m, n, ng, l, r, s, z, f, v, th, p, b, t, d, k, mix-a, mix-m, mix-s, mix-p, mix-u.
|
||||
""");
|
||||
}
|
||||
|
||||
static string ReadPayloadString(Dictionary<string, JsonElement> payload, string key)
|
||||
{
|
||||
if (!payload.TryGetValue(key, out var value))
|
||||
{
|
||||
return "";
|
||||
}
|
||||
|
||||
return value.ValueKind == JsonValueKind.String ? value.GetString() ?? "" : value.ToString();
|
||||
}
|
||||
|
||||
sealed record SearchOptions(
|
||||
string QdrantUrl,
|
||||
string OllamaUrl,
|
||||
string EmbedModel,
|
||||
string Collection,
|
||||
int TimeoutSeconds,
|
||||
bool UseVector,
|
||||
bool RequireVector,
|
||||
bool SkipIndex);
|
||||
|
||||
sealed record WeightedTerms(float Weight, string[] Terms);
|
||||
|
||||
sealed record SearchDocument(IpaTrialResult Result, Dictionary<string, WeightedTerms> Fields)
|
||||
{
|
||||
public string[] AllTerms { get; } = Fields.Values
|
||||
.SelectMany(field => field.Terms)
|
||||
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||
.ToArray();
|
||||
|
||||
public static SearchDocument From(IpaTrialResult result, Dictionary<string, WeightedTerms> fields) => new(result, fields);
|
||||
}
|
||||
|
||||
sealed record SearchHit(
|
||||
IpaTrialResult Result,
|
||||
float Score,
|
||||
string[] MatchedTerms,
|
||||
IReadOnlyDictionary<string, string[]> MatchedByField,
|
||||
string[] Tags,
|
||||
string RetrievalMode = "lexical",
|
||||
float? VectorScore = null,
|
||||
string? VectorChunkId = null,
|
||||
string? VectorText = null);
|
||||
|
||||
sealed record EvidenceChunk(
|
||||
string Id,
|
||||
string TrialId,
|
||||
string CandidateId,
|
||||
string TargetSetId,
|
||||
string ChunkKind,
|
||||
string Text,
|
||||
string Store);
|
||||
|
||||
sealed record VectorIndexResult(int ChunkCount, int VectorSize);
|
||||
|
||||
sealed record VectorSearchHit(
|
||||
string ChunkId,
|
||||
string TrialId,
|
||||
string CandidateId,
|
||||
string TargetSetId,
|
||||
string Text,
|
||||
float Score);
|
||||
|
||||
sealed record OllamaEmbedRequest(string Model, string[] Input);
|
||||
|
||||
sealed record OllamaEmbedResponse(
|
||||
[property: JsonPropertyName("embedding")] float[]? Embedding,
|
||||
[property: JsonPropertyName("embeddings")] float[][]? Embeddings);
|
||||
|
||||
sealed record QdrantVectorConfig(
|
||||
[property: JsonPropertyName("size")] int Size,
|
||||
[property: JsonPropertyName("distance")] string Distance,
|
||||
[property: JsonPropertyName("on_disk")] bool OnDisk);
|
||||
|
||||
sealed record QdrantCreateCollectionRequest(
|
||||
[property: JsonPropertyName("vectors")] QdrantVectorConfig Vectors,
|
||||
[property: JsonPropertyName("on_disk_payload")] bool OnDiskPayload,
|
||||
[property: JsonPropertyName("metadata")] Dictionary<string, object> Metadata);
|
||||
|
||||
sealed record QdrantPayloadIndexRequest(
|
||||
[property: JsonPropertyName("field_name")] string FieldName,
|
||||
[property: JsonPropertyName("field_schema")] string FieldSchema);
|
||||
|
||||
sealed record QdrantPoint(
|
||||
[property: JsonPropertyName("id")] string Id,
|
||||
[property: JsonPropertyName("vector")] float[] Vector,
|
||||
[property: JsonPropertyName("payload")] QdrantPayload Payload);
|
||||
|
||||
sealed record QdrantUpsertRequest([property: JsonPropertyName("points")] QdrantPoint[] Points);
|
||||
|
||||
sealed record QdrantPayload(
|
||||
[property: JsonPropertyName("chunkId")] string ChunkId,
|
||||
[property: JsonPropertyName("sourceId")] string SourceId,
|
||||
[property: JsonPropertyName("sourceKind")] string SourceKind,
|
||||
[property: JsonPropertyName("corpusKind")] string CorpusKind,
|
||||
[property: JsonPropertyName("storeKey")] string StoreKey,
|
||||
[property: JsonPropertyName("trialId")] string TrialId,
|
||||
[property: JsonPropertyName("candidateId")] string CandidateId,
|
||||
[property: JsonPropertyName("targetSetId")] string TargetSetId,
|
||||
[property: JsonPropertyName("chunkKind")] string ChunkKind,
|
||||
[property: JsonPropertyName("embedderId")] string EmbedderId,
|
||||
[property: JsonPropertyName("text")] string Text);
|
||||
|
||||
sealed record QdrantMatch([property: JsonPropertyName("value")] string Value);
|
||||
|
||||
sealed record QdrantCondition(
|
||||
[property: JsonPropertyName("key")] string Key,
|
||||
[property: JsonPropertyName("match")] QdrantMatch Match);
|
||||
|
||||
sealed record QdrantFilter([property: JsonPropertyName("must")] QdrantCondition[] Must)
|
||||
{
|
||||
public static QdrantFilter Store(string storeKey, string embedderId) => new([
|
||||
new QdrantCondition("storeKey", new QdrantMatch(storeKey)),
|
||||
new QdrantCondition("embedderId", new QdrantMatch(embedderId))
|
||||
]);
|
||||
}
|
||||
|
||||
sealed record QdrantSearchRequest(
|
||||
[property: JsonPropertyName("vector")] float[] Vector,
|
||||
[property: JsonPropertyName("limit")] int Limit,
|
||||
[property: JsonPropertyName("with_payload")] bool WithPayload,
|
||||
[property: JsonPropertyName("with_vector")] bool WithVector,
|
||||
[property: JsonPropertyName("filter")] QdrantFilter Filter);
|
||||
|
||||
sealed record QdrantSearchResponse([property: JsonPropertyName("result")] QdrantScoredPoint[] Result);
|
||||
|
||||
sealed record QdrantScoredPoint(
|
||||
[property: JsonPropertyName("score")] float Score,
|
||||
[property: JsonPropertyName("payload")] Dictionary<string, JsonElement>? Payload);
|
||||
|
|
|
|||
|
|
@ -205,6 +205,22 @@ for ($round = 1; $round -le $Rounds; $round++) {
|
|||
New-Item -ItemType Directory -Force -Path $patchDir | Out-Null
|
||||
Add-Content -LiteralPath $indexPath -Value "- ${roundId}: $roundDir"
|
||||
|
||||
$indexBeforeLog = Join-Path $logRoot "$roundId-index-before.log"
|
||||
Invoke-LoggedProcess `
|
||||
-FilePath "dotnet" `
|
||||
-ArgumentList @(
|
||||
"run",
|
||||
"--project",
|
||||
$workerProject,
|
||||
"--",
|
||||
"index",
|
||||
"--store",
|
||||
$storePath,
|
||||
"--output",
|
||||
(Join-Path $roundDir "vector-index-before.md")
|
||||
) `
|
||||
-LogPath $indexBeforeLog
|
||||
|
||||
$preEvidence = Join-Path $roundDir "semantic-search-before.md"
|
||||
$preDumpLog = Join-Path $logRoot "$roundId-search-before.log"
|
||||
Invoke-LoggedProcess `
|
||||
|
|
@ -222,7 +238,9 @@ for ($round = 1; $round -le $Rounds; $round++) {
|
|||
"--limit",
|
||||
"40",
|
||||
"--output",
|
||||
$preEvidence
|
||||
$preEvidence,
|
||||
"--skip-index",
|
||||
"true"
|
||||
) `
|
||||
-LogPath $preDumpLog
|
||||
|
||||
|
|
@ -238,7 +256,7 @@ The backing store is:
|
|||
$storePath
|
||||
|
||||
Use the worker as your search organ. Do not ask for crude full dumps unless retrieval fails:
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- search --store "$storePath" --query "<your evidence question>" --limit 20 --output "$roundDir/search-<topic>.md"
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- search --store "$storePath" --query "<your evidence question>" --limit 20 --output "$roundDir/search-<topic>.md" --skip-index true
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- show --store "$storePath" --trial-id "<trial id from search>" --output "$roundDir/detail-<candidate>.md"
|
||||
|
||||
Retrieval failure policy:
|
||||
|
|
@ -357,6 +375,22 @@ Return a short final summary naming the files you wrote.
|
|||
) `
|
||||
-LogPath $scoreLog
|
||||
|
||||
$indexAfterLog = Join-Path $logRoot "$roundId-index-after.log"
|
||||
Invoke-LoggedProcess `
|
||||
-FilePath "dotnet" `
|
||||
-ArgumentList @(
|
||||
"run",
|
||||
"--project",
|
||||
$workerProject,
|
||||
"--",
|
||||
"index",
|
||||
"--store",
|
||||
$storePath,
|
||||
"--output",
|
||||
(Join-Path $roundDir "vector-index-after.md")
|
||||
) `
|
||||
-LogPath $indexAfterLog
|
||||
|
||||
$postEvidence = Join-Path $roundDir "semantic-search-after.md"
|
||||
$postDumpLog = Join-Path $logRoot "$roundId-search-after.log"
|
||||
Invoke-LoggedProcess `
|
||||
|
|
@ -374,7 +408,9 @@ Return a short final summary naming the files you wrote.
|
|||
"--limit",
|
||||
"60",
|
||||
"--output",
|
||||
$postEvidence
|
||||
$postEvidence,
|
||||
"--skip-index",
|
||||
"true"
|
||||
) `
|
||||
-LogPath $postDumpLog
|
||||
|
||||
|
|
@ -401,9 +437,9 @@ Before judging:
|
|||
3. If the hypothesis worker skipped the required 25-lane matrix or retrieval receipts, mark orchestration/prompt compliance as failed even if some audio metrics improved.
|
||||
|
||||
Use the worker as your search organ:
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- search --store "$storePath" --query "$roundId external-codex-$roundId current round candidates" --limit 30 --output "$roundDir/eval-search-current-round.md"
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- search --store "$storePath" --query "weak regression failed articulation stop plosive source tract radiation" --limit 30 --output "$roundDir/eval-search-weak.md"
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- search --store "$storePath" --query "promising transfer generalization vowel nasal fricative articulation" --limit 30 --output "$roundDir/eval-search-promising.md"
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- search --store "$storePath" --query "$roundId external-codex-$roundId current round candidates" --limit 30 --output "$roundDir/eval-search-current-round.md" --skip-index true
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- search --store "$storePath" --query "weak regression failed articulation stop plosive source tract radiation" --limit 30 --output "$roundDir/eval-search-weak.md" --skip-index true
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- search --store "$storePath" --query "promising transfer generalization vowel nasal fricative articulation" --limit 30 --output "$roundDir/eval-search-promising.md" --skip-index true
|
||||
`$env:DOTNET_CLI_HOME="$roundDir/dotnet-home"; `$env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE="1"; `$env:DOTNET_CLI_TELEMETRY_OPTOUT="1"; dotnet run --no-build --no-restore --project tools/IpaTrialWorker/IpaTrialWorker.csproj -- show --store "$storePath" --trial-id "<trial id from current round search>" --output "$roundDir/eval-detail-<candidate>.md"
|
||||
|
||||
Retrieval failure policy:
|
||||
|
|
|
|||
Loading…
Reference in a new issue