diff --git a/README.md b/README.md index bfa07f2..34f83c0 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,8 @@ culture, and constraint, and the planned lowerer that turns an agent's meaning packet into the reader's display language. **What is valuable here.** The interlingua standard and the lowering design: a -meaning packet owns what is meant, speaker and scene state (planned; not yet live) own register, +meaning packet owns what is meant, the scene state (live: `weksa scene put|dump`; speaker +documents are planned) owns the speaker's relation to the addressee, which selects register, address and politeness, and a lowerer owns surface text and leaves a trace. **The problem it solves.** An agent's outward speech today is one English @@ -84,9 +85,9 @@ target-specific English receipt. The language ontology is `data/ontology/ontology.cc` (read it with `weksa ontology dump`): per language, the lexicon, grammar and discourse choices a lowerer -reads for a stance and discourse packet. No entry fixes an address or politeness level; those -belong to the speaker and scene state. Its en-US and ja-JP entries are -model-drafted seeds, never Test evidence. Nothing yet grows them from accepted +reads for a stance, a discourse and a scene relation. Only an entry with a Scene condition names an +address or politeness level; the relation, from `data/original/speaker-state.cc`, picks it. Its +en-US and ja-JP entries and the keeper scenes are model-drafted or model-annotated, never Test evidence. Nothing yet grows them from accepted rounds; until that mint exists they change only by a new `weksa ontology admit`. For the first AquaSynth speech-training handoff, see diff --git a/crates/weksa/examples/show_working.rs b/crates/weksa/examples/show_working.rs index 8c23fb2..6c39bdb 100644 --- a/crates/weksa/examples/show_working.rs +++ b/crates/weksa/examples/show_working.rs @@ -259,6 +259,7 @@ fn lower_error(e: &LowerError) -> Value { LowerError::ConsentMissing { predication } => ("ConsentMissing", format!("{predication} is a request the packet does not carry the addressee's consent to")), LowerError::AskerNotAddressee { predication } => ("AskerNotAddressee", format!("the agent of {predication} is not the addressee")), LowerError::EntityRead { entity, .. } => ("EntityRead", format!("the entity store failed reading {entity}")), + LowerError::SceneUnread => ("SceneUnread", "the view's scene was never read".to_string()), }; json!({ "variant": variant, "reason": reason }) } diff --git a/crates/weksa/src/context.rs b/crates/weksa/src/context.rs index 2b88a08..3dec960 100644 --- a/crates/weksa/src/context.rs +++ b/crates/weksa/src/context.rs @@ -37,7 +37,7 @@ use crate::packet::{ use crate::run::LADDER_STEP; use crate::split::{Checkpoint, Split, SplitScheme, fnv1a64, hex16}; use crate::store::{CorpusStore, normalized}; -use crate::validate::LoweringInput; +use crate::validate::{LoweringInput, SceneRead}; /// The instructions a mint prompt starts with: the editor's whole brief. Fixed /// per run by `PROMPT_SOURCE_DIGEST`. @@ -71,7 +71,7 @@ A feature is {feature, value}; value is {\"Grade\": 0 to 1000}, {\"Option\": lab /// Fixed per run by `PROMPT_SOURCE_DIGEST`. pub const LOWER_INSTRUCTIONS: &str = "\ Say the packet below in the language on the language line, as one utterance of its speaker: all of its meaning and nothing it does not carry. -Every id is the inventory's listed below and every feature value is as listed; a grade is permille, 0 to 1000. The stance is the speaker's felt affect and restraint: it shapes how the utterance is said, not what it says. The ontology lines are choices this language makes for this speaker and stance: realize each where the packet allows; they add no meaning. +Every id is the inventory's listed below and every feature value is as listed; a grade is permille, 0 to 1000. The stance is the speaker's felt affect and restraint: it shapes how the utterance is said, not what it says. The ontology lines are choices this language makes for this speaker, stance and scene: realize each where the packet allows; they add no meaning. A referent with an entity is named by one of that entity's forms on the entities line. Answer with one JSON object and nothing else: {\"text\": \"...\"}. No code fence, no commentary."; @@ -89,7 +89,7 @@ Answer with one JSON object and nothing else: {\"text\": \"...\"}, the continuat /// ```no_run /// # let dir = tempfile::tempdir().unwrap(); /// # let p = |n: &str| dir.path().join(n); -/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc") }; +/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc"), speaker_state: &p("s.cc") }; /// # let panel = ["en-US".to_string()]; /// # let run = weksa::run::RunStores::create(&p("run"), "t", paths, "m", "e", weksa::run::LowererReach::ToolLessRelay, weksa::corpus::ModelFamily::Unrecorded, &panel).unwrap(); /// let t = run.train().unwrap(); @@ -104,7 +104,7 @@ Answer with one JSON object and nothing else: {\"text\": \"...\"}, the continuat /// # let dir = tempfile::tempdir().unwrap(); /// # let p = |n: &str| dir.path().join(n); /// # for n in ["c.cc", "i.cc", "e.cc", "o.cc"] { std::fs::write(p(n), [0x90]).unwrap(); } -/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc") }; +/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc"), speaker_state: &p("s.cc") }; /// # let run = weksa::run::RunStores::create(&p("run"), "t", paths, "m", "e", weksa::run::LowererReach::ToolLessRelay, weksa::corpus::ModelFamily::Unrecorded, &["en-US".to_string()]).unwrap(); /// let t = run.train().unwrap(); /// let _u = t.clone(); @@ -115,7 +115,7 @@ Answer with one JSON object and nothing else: {\"text\": \"...\"}, the continuat /// # let dir = tempfile::tempdir().unwrap(); /// # let p = |n: &str| dir.path().join(n); /// # for n in ["c.cc", "i.cc", "e.cc", "o.cc"] { std::fs::write(p(n), [0x90]).unwrap(); } -/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc") }; +/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc"), speaker_state: &p("s.cc") }; /// # let run = weksa::run::RunStores::create(&p("run"), "t", paths, "m", "e", weksa::run::LowererReach::ToolLessRelay, weksa::corpus::ModelFamily::Unrecorded, &["en-US".to_string()]).unwrap(); /// let t = run.train().unwrap(); /// s(&t); @@ -337,7 +337,7 @@ impl EditFeedback { /// ```no_run /// # let dir = tempfile::tempdir().unwrap(); /// # let p = |n: &str| dir.path().join(n); -/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc") }; +/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc"), speaker_state: &p("s.cc") }; /// # let panel = ["en-US".to_string()]; /// # let run = weksa::run::RunStores::create(&p("run"), "t", paths, "m", "e", weksa::run::LowererReach::ToolLessRelay, weksa::corpus::ModelFamily::Unrecorded, &panel).unwrap(); /// let t = run.train().unwrap(); @@ -356,7 +356,7 @@ impl EditFeedback { /// # let dir = tempfile::tempdir().unwrap(); /// # let p = |n: &str| dir.path().join(n); /// # for n in ["c.cc", "i.cc", "e.cc", "o.cc"] { std::fs::write(p(n), [0x90]).unwrap(); } -/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc") }; +/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc"), speaker_state: &p("s.cc") }; /// # let run = weksa::run::RunStores::create(&p("run"), "t", paths, "m", "e", weksa::run::LowererReach::ToolLessRelay, weksa::corpus::ModelFamily::Unrecorded, &["en-US".to_string()]).unwrap(); /// let t = run.train().unwrap(); /// let inv = run.inventory().inventory().unwrap(); @@ -369,7 +369,7 @@ impl EditFeedback { /// # let dir = tempfile::tempdir().unwrap(); /// # let p = |n: &str| dir.path().join(n); /// # for n in ["c.cc", "i.cc", "e.cc", "o.cc"] { std::fs::write(p(n), [0x90]).unwrap(); } -/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc") }; +/// # let paths = weksa::run::RunPaths { corpus: &p("c.cc"), inventory: &p("i.cc"), entities: &p("e.cc"), ontology: &p("o.cc"), speaker_state: &p("s.cc") }; /// # let run = weksa::run::RunStores::create(&p("run"), "t", paths, "m", "e", weksa::run::LowererReach::ToolLessRelay, weksa::corpus::ModelFamily::Unrecorded, &["en-US".to_string()]).unwrap(); /// let t = run.train().unwrap(); /// let inv = run.inventory().inventory().unwrap(); @@ -528,9 +528,11 @@ impl Prompt { /// `input` (no kind, constraints, scene, speaker or group), the forms in /// `language` of every entity its referents cite, the `ontology:` section /// and the entries of `inventory` live at the view's revision. The section - /// holds the entries `ontology::select` chooses from the view's stance and - /// discourse features and `language`, never the caller; the view's speaker - /// and group are still never read. `item` + /// holds the entries `ontology::select` chooses from the view's stance, + /// discourse and the scene relation `with_scene` read, and `language`, + /// never the caller; the scene's values are never printed, only the entries + /// they select. A view whose scene was never read is refused + /// ([`LowerError::SceneUnread`]). `item` /// goes only to [`Prompt::item_keys`], never into the text. A speaker's /// entity id appears /// only when the encoder cites that entity on a referent, which is naming a @@ -545,6 +547,9 @@ impl Prompt { entities: &dyn EntityForms, ontology: &Ontology, ) -> Result { + if input.scene_read() == &SceneRead::Unread { + return Err(LowerError::SceneUnread); + } let mut forms: BTreeMap<&str, Vec> = BTreeMap::new(); for id in input.referents().iter().filter_map(|r| r.entity.as_ref()) { let found = entity(entities, id)?; diff --git a/crates/weksa/src/inventory.rs b/crates/weksa/src/inventory.rs index ec59b13..4820ea5 100644 --- a/crates/weksa/src/inventory.rs +++ b/crates/weksa/src/inventory.rs @@ -541,7 +541,7 @@ pub fn proof_revision() -> (InventoryRevision, Vec) { ) } -/// Revision 3: the speaker-state vocabulary. First the Scene features, the +/// Revision `n`: the speaker-state vocabulary. First the Scene features, the /// speaker's relation to one party of a scene (slot `Scene`): `relative_status` /// is the speaker's standing against the addressee's, not a register word, what /// it selects is each language's ontology's business. Then the Speaker features @@ -551,7 +551,7 @@ pub fn proof_revision() -> (InventoryRevision, Vec) { /// arrive as community ids with cut `language-ontology`. Hand-seeded like /// [`proof_revision`] until the loop's minting round owns the vocabulary; the /// features take the ids after the seed's. -pub fn speaker_scene_revision() -> (InventoryRevision, Vec) { +pub fn speaker_scene_revision(n: u32) -> (InventoryRevision, Vec) { let taken = seed() .1 .iter() @@ -593,15 +593,15 @@ pub fn speaker_scene_revision() -> (InventoryRevision, Vec) { gloss: gloss.to_string(), value_kind: Some(kind), applies_to: vec![slot], - introduced_rev: 3, + introduced_rev: n, retired_rev: None, }); } let changes = entries.iter().map(|e| Change::Add(e.id)).collect(); ( InventoryRevision { - n: 3, - parent: Some(2), + n, + parent: Some(n - 1), changes, frozen: false, }, diff --git a/crates/weksa/src/lower.rs b/crates/weksa/src/lower.rs index 76f3158..28f4ca2 100644 --- a/crates/weksa/src/lower.rs +++ b/crates/weksa/src/lower.rs @@ -182,6 +182,8 @@ pub enum LowerError { AskerNotAddressee { predication: NodeId }, /// The entity port failed to read. EntityRead { entity: EntityId, why: String }, + /// A lowering prompt was asked of a view whose scene was never read. + SceneUnread, } impl fmt::Display for LowerError { @@ -203,6 +205,7 @@ impl fmt::Display for LowerError { LowerError::ConsentMissing { .. } => "request lacks the addressee's consent", LowerError::AskerNotAddressee { .. } => "request agent is not the addressee", LowerError::EntityRead { .. } => "entity port failed to read", + LowerError::SceneUnread => "the view's scene was never read", }) } } diff --git a/crates/weksa/src/main.rs b/crates/weksa/src/main.rs index b9b094e..c4d7be6 100644 --- a/crates/weksa/src/main.rs +++ b/crates/weksa/src/main.rs @@ -10,7 +10,8 @@ use unicode_normalization::UnicodeNormalization; use weksa::corpus::{CorpusItem, ModelFamily, NameSpan, Source, Turn}; use weksa::encode::OntologyDraft; use weksa::entity::{Entity, EntityOrigin, LanguageForms, Pronunciation}; -use weksa::inventory::{proof_revision, seed}; +use weksa::inventory::{proof_revision, seed, speaker_scene_revision}; +use weksa::packet::{FeatValue, FeatureValue, Grade}; use weksa::packet::proof_packet; use weksa::model::RecordedAnswers; use weksa::run::{ @@ -18,8 +19,8 @@ use weksa::run::{ }; use weksa::score::ScoreInput; use weksa::{ - Annotator, AnswerFiles, CorpusStore, EntityId, EntityStore, InventoryStore, OllamaEmbedder, OntologyStore, PacketStore, - Scorer, Split, SplitScheme, StoreError, lower, pronounce, validate, + Annotator, AnswerFiles, CorpusStore, EntityId, EntityStore, Family, InventoryStore, OllamaEmbedder, OntologyStore, OptionLabel, + PacketStore, PairRelation, Party, Scene, Slot, SpeakerStateStore, ValueKind, Scorer, Split, SplitScheme, StoreError, lower, pronounce, validate, }; #[derive(Parser)] @@ -40,6 +41,9 @@ struct Cli { /// The language ontology store. #[arg(long, global = true, default_value = "data/ontology/ontology.cc")] ontology: PathBuf, + /// The speaker-state store (scenes). + #[arg(long, global = true, default_value = "data/corpus/speaker-state.cc")] + speaker_state: PathBuf, #[command(subcommand)] command: Command, } @@ -57,6 +61,8 @@ enum Command { #[command(subcommand)] Ontology(OntologyCommand), #[command(subcommand)] + Scene(SceneCommand), + #[command(subcommand)] Packet(PacketCommand), /// Lower a stored packet to English text. Lower { @@ -87,7 +93,7 @@ enum Command { #[derive(Subcommand)] enum LoopCommand { - /// Pin the global --corpus, --inventory, --entities and --ontology as a run at ROOT. + /// Pin the global --corpus, --inventory, --entities, --ontology and --speaker-state as a run at ROOT. Open { root: PathBuf, #[arg(long)] @@ -191,6 +197,38 @@ enum InventoryCommand { Proof, /// Admit revision 3 of the Dao De Jing test bed stores (after seed and proof). Dao, + /// Admit the speaker-state revision (Scene and Speaker features) as the + /// inventory's latest plus one. + SpeakerScene, +} + +#[derive(Subcommand)] +enum SceneCommand { + /// Append a scene revision for one corpus group: its one live speaker's + /// relation to the addressee, as Scene-slot feature values. + Put { + #[arg(long)] + group: String, + /// `entity:` of a named addressee, or `unnamed`. + #[arg(long)] + to: String, + /// The inventory revision the glosses resolve at. + #[arg(long)] + inventory_rev: u32, + /// `gloss=value`, repeated: a grade 0-1000, an option label or true/false. + #[arg(long = "set", value_name = "GLOSS=VALUE")] + set: Vec, + #[arg(long, requires = "generator", conflicts_with = "operator")] + model_family: Option, + /// The attestation string naming the annotating model. + #[arg(long, requires = "model_family")] + generator: Option, + /// Annotated by the operator, not a model. + #[arg(long)] + operator: bool, + }, + /// Text projection of the scenes for review; not a store. + Dump, } #[derive(Subcommand)] @@ -734,6 +772,143 @@ fn main() -> Result<()> { .map_err(refused)?; println!("admitted rev:2 with {count} entries"); } + Command::Inventory(InventoryCommand::SpeakerScene) => { + let store = InventoryStore::open(&cli.inventory)?; + let latest = store + .inventory()? + .latest() + .map(|r| r.n) + .context("the inventory holds no revision")?; + let (rev, entries) = speaker_scene_revision(latest + 1); + let (n, count) = (rev.n, entries.len()); + let mut store = store; + store.admit_revision(rev, entries).map_err(refused)?; + println!("admitted rev:{n} with {count} entries"); + } + Command::Scene(SceneCommand::Put { + group, + to, + inventory_rev, + set, + model_family, + generator, + operator, + }) => { + let annotator = match (model_family, generator, operator) { + (Some(family), Some(generator), false) => Annotator::Model { + family: family.into(), + generator, + }, + (None, None, true) => Annotator::Operator, + _ => bail!("name the annotator: --model-family with --generator, or --operator"), + }; + let inventory_store = InventoryStore::open(&cli.inventory)?; + let inventory = inventory_store.inventory()?; + let corpus = CorpusStore::open(&cli.corpus)?; + let live: Vec = corpus + .all()? + .into_iter() + .filter(|i| i.group == group && !i.withdrawn) + .collect(); + let mut speakers: Vec = live.iter().map(|i| i.speaker.clone()).collect(); + speakers.sort(); + speakers.dedup(); + let [from] = speakers.as_slice() else { + bail!("a scene group holds one live speaker; multi-party scenes wait"); + }; + let to = match to.as_str() { + "unnamed" => Party::Unnamed, + named => Party::Named( + EntityId::parse(named).map_err(|_| anyhow::anyhow!("the addressee is unnamed or an entity id"))?, + ), + }; + let mut features = Vec::new(); + for pair in &set { + let bad = || anyhow::anyhow!("a --set is gloss=value of a live Scene feature"); + let (gloss, value) = pair.split_once('=').ok_or_else(bad)?; + let entry = inventory + .entries() + .find(|e| { + e.family() == Family::Feature + && e.gloss == gloss + && e.live_at(inventory_rev) + && e.applies_to.contains(&Slot::Scene) + }) + .ok_or_else(bad)?; + let value = match entry.value_kind.as_ref().ok_or_else(bad)? { + ValueKind::Grade => FeatValue::Grade( + Grade::new(value.parse().map_err(|_| bad())?).map_err(|_| bad())?, + ), + ValueKind::Options(_) => { + FeatValue::Option(OptionLabel::parse(value).map_err(|_| bad())?) + } + ValueKind::Flag => FeatValue::Flag(value.parse().map_err(|_| bad())?), + }; + features.push(FeatureValue { feature: entry.id, value }); + } + let mut participants = vec![from.clone()]; + if let Party::Named(a) = &to + && a != from + { + participants.push(a.clone()); + } + let mut state = SpeakerStateStore::open(&cli.speaker_state)?; + let latest = state.scene(&group)?; + let mut scene = Scene { + group, + n: latest.as_ref().map_or(1, |l| l.n + 1), + inventory_rev, + participants, + relations: vec![PairRelation { from: from.clone(), to, features }], + annotator, + }; + if let Some(l) = &latest { + let same = Scene { n: l.n, ..scene.clone() }; + if &same == l { + println!("unchanged"); + return Ok(()); + } + } + scene.n = latest.map_or(1, |l| l.n + 1); + let key = weksa::speaker::scene_key(&scene.group, scene.n); + let outcome = state + .put_scene(scene, &inventory_store, &corpus, &EntityStore::open(&cli.entities)?) + .map_err(refused)?; + println!("{outcome:?} {key}"); + } + Command::Scene(SceneCommand::Dump) => { + let inventory = InventoryStore::open(&cli.inventory)?.inventory()?; + let gloss = |f: &FeatureValue| { + let name = inventory.entry(&f.feature).map_or("?", |e| e.gloss.as_str()); + match &f.value { + FeatValue::Grade(g) => format!("{name}={}", g.get()), + FeatValue::Option(l) => format!("{name}={l}"), + FeatValue::Flag(b) => format!("{name}={b}"), + } + }; + for s in SpeakerStateStore::open(&cli.speaker_state)?.all_scenes()? { + let who = match &s.annotator { + Annotator::Operator => "Operator".to_string(), + Annotator::Model { family, generator } => format!("Model {family:?} {generator}"), + }; + let people: Vec<&str> = s.participants.iter().map(|p| p.as_str()).collect(); + println!( + "scene:{}@{} rev:{} {who} [{}]", + s.group, + s.n, + s.inventory_rev, + people.join(",") + ); + for r in &s.relations { + let to = match &r.to { + Party::Named(e) => e.as_str().to_string(), + Party::Unnamed => "unnamed".to_string(), + }; + let feats: Vec = r.features.iter().map(&gloss).collect(); + println!(" {}->{to} {}", r.from, feats.join(" ")); + } + } + } Command::Inventory(InventoryCommand::Dao) => { let (rev, entries) = weksa::daodejing::revision(); let count = entries.len(); @@ -850,6 +1025,7 @@ fn main() -> Result<()> { inventory: &cli.inventory, entities: &cli.entities, ontology: &cli.ontology, + speaker_state: &cli.speaker_state, }, &answerer, &embed_model, diff --git a/crates/weksa/src/ontology.rs b/crates/weksa/src/ontology.rs index 30bc5dd..5919588 100644 --- a/crates/weksa/src/ontology.rs +++ b/crates/weksa/src/ontology.rs @@ -1,7 +1,7 @@ //! The language ontology: per target language, the choices its culture makes //! about word, grammar, discourse, sound and script for a speaker in a stance //! (ruling `persona-drives-ontology`). An entry says when it applies (feature -//! conditions over the lowering view's stance and discourse) and what it +//! conditions over the lowering view's stance, discourse and scene relation) and what it //! chooses. `select` is the only reader that decides which entries a lowering //! prompt carries, and it runs only inside `Prompt::lower`; //! `OntologyStore::admit` is the only writer. Entries are immutable. @@ -51,8 +51,8 @@ pub enum Test { Is(FeatValue), } -/// One feature of one slot of the view, tested. Only `Stance` and `Discourse` -/// reach a lowering view today. +/// One feature of one slot of the view, tested. `Stance`, `Discourse` and +/// `Scene` reach a lowering view; `Speaker` does not yet. #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)] pub struct Condition { pub slot: Slot, @@ -100,7 +100,9 @@ pub struct OntologyEntry { /// The entry's number within its language, from 1. #[cultcache(key = 1)] pub n: u32, - /// The inventory revision the conditions read against. + /// The revision, of the inventory the entry was admitted through, at which the + /// store read its conditions: provenance only; a run reads the conditions' + /// features at its own head. #[cultcache(key = 2)] pub inventory_rev: u32, #[cultcache(key = 3)] @@ -149,8 +151,9 @@ impl Ontology { } /// The entries of exactly `language` whose every condition holds against the -/// view's stance and discourse features, in `n` order. An absent feature meets -/// no condition. Nothing else of the view is read. +/// view's stance, discourse and scene-relation features (the scene's only when +/// `with_scene` selected a relation), in `n` order. An absent feature meets no +/// condition. Nothing else of the view is read. pub fn select<'a>( o: &'a Ontology, language: &str, @@ -163,6 +166,7 @@ pub fn select<'a>( e.when.iter().all(|c| match c.slot { Slot::Stance => c.holds(&input.stance().features), Slot::Discourse => c.holds(&input.discourse().features), + Slot::Scene => c.holds(input.scene()), _ => false, }) }) diff --git a/crates/weksa/src/run.rs b/crates/weksa/src/run.rs index 770d9b1..3d0da2a 100644 --- a/crates/weksa/src/run.rs +++ b/crates/weksa/src/run.rs @@ -43,15 +43,16 @@ use crate::corpus::{CorpusItem, CorpusItemId, ModelFamily, Source}; use crate::encode::{EditDraft, LoweringDraft, PacketDraft, encoder_id}; use crate::form::{is_attestation, is_language_tag, is_slug}; use crate::model::{ModelPort, PortError}; -use crate::ontology::OntologyEntry; use crate::packet::{Packet, Stance}; use crate::score::{CHRF_VERSION, chrf, Embedder, ItemScore, ScoreInput, Scorer, SplitMean, aggregate}; use crate::split::{Split, SplitScheme, fnv1a64, fnv1a64_of, hex16}; use crate::store::{ - CorpusStore, EntityStore, InventoryStore, OntologyStore, PacketStore, StoreError, + CorpusStore, EntityStore, InventoryStore, OntologyStore, PacketStore, SpeakerStateStore, + StoreError, check_ontology_entry, open_cache_over, }; -use crate::validate::LoweringInput; +use crate::inventory::Slot; +use crate::validate::{LoweringInput, feature_refusals}; /// The digest of the prompt builder's source: `context.rs`, then `split.rs`, /// whose scheme and digest function decide which lines the prompt holds as @@ -137,6 +138,13 @@ pub struct RunRecord { /// 16 hex of the ontology file's bytes. #[cultcache(key = 16)] pub ontology_digest: String, + /// Canonical absolute path of the speaker-state store (scenes): read-only + /// for the run, pinned by `speaker_state_digest`. + #[cultcache(key = 17)] + pub speaker_state: String, + /// 16 hex of the speaker-state file's bytes. + #[cultcache(key = 18)] + pub speaker_state_digest: String, } /// How a run's lowering prompts reach the model that answers them (rulings @@ -442,6 +450,7 @@ pub enum DriftOf { Corpus, Entities, Ontology, + SpeakerState, Prompts, Model, Embedder, @@ -480,6 +489,8 @@ pub enum RunError { /// An ontology entry of a panel language does not read against the run's /// head inventory at its revision. Ontology, + /// A scene does not read against the run's inventory. + SpeakerState, } impl fmt::Display for RunError { @@ -504,6 +515,9 @@ impl fmt::Display for RunError { RunError::Ontology => f.write_str( "an ontology entry of a panel language does not read against the run's inventory", ), + RunError::SpeakerState => { + f.write_str("a scene does not read against the run's inventory") + } RunError::Panel => f.write_str( "the panel is empty, repeats or misspells a language, or names one with no live Train or Validation item", ), @@ -519,13 +533,14 @@ impl fmt::Debug for RunError { impl std::error::Error for RunError {} -/// The four stores a run pins. +/// The five stores a run pins. #[derive(Clone, Copy)] pub struct RunPaths<'a> { pub corpus: &'a Path, pub inventory: &'a Path, pub entities: &'a Path, pub ontology: &'a Path, + pub speaker_state: &'a Path, } /// A run's open stores, reopened only from its record. The one public producer @@ -538,6 +553,7 @@ pub struct RunStores { inventory: InventoryStore, entities: EntityStore, ontology: OntologyStore, + speaker_state: SpeakerStateStore, } /// The run's own copy of the inventory, made at `create`. @@ -554,6 +570,9 @@ fn check_pins(cache: &CultCache, root: &Path, record: &RunRecord) -> Result<(), if file_digest(&record.ontology).as_ref() != Some(&record.ontology_digest) { return Err(RunError::Drift(DriftOf::Ontology)); } + if file_digest(&record.speaker_state).as_ref() != Some(&record.speaker_state_digest) { + return Err(RunError::Drift(DriftOf::SpeakerState)); + } let (path, digest, _) = head_of(cache, root, record)?; if file_digest_at(&path).as_ref() != Some(&digest) { return Err(RunError::Drift(DriftOf::Inventory)); @@ -692,6 +711,8 @@ impl RunStores { entities: canonical(paths.entities)?, ontology: canonical(paths.ontology)?, ontology_digest: String::new(), + speaker_state: canonical(paths.speaker_state)?, + speaker_state_digest: String::new(), corpus_digest: String::new(), entity_digest: String::new(), prompt_digest: hex16(PROMPT_SOURCE_DIGEST), @@ -714,12 +735,14 @@ impl RunStores { entity_digest: file_digest(&record.entities).ok_or(RunError::Store)?, inventory_digest: file_digest_at(&head).ok_or(RunError::Store)?, ontology_digest: file_digest(&record.ontology).ok_or(RunError::Store)?, + speaker_state_digest: file_digest(&record.speaker_state).ok_or(RunError::Store)?, ..record }; between(); let mut run = Self::with(root, cache, record)?; run.check_panel()?; run.check_ontology()?; + run.check_scenes()?; // The stores were read between the digests: unless they still match // the pins, what was opened is not what was pinned. Nothing is // recorded for a run that did not pin. @@ -771,6 +794,8 @@ impl RunStores { inventory: InventoryStore::open(&head).map_err(store)?, entities: EntityStore::open(Path::new(&record.entities)).map_err(store)?, ontology: OntologyStore::open(Path::new(&record.ontology)).map_err(store)?, + speaker_state: SpeakerStateStore::open(Path::new(&record.speaker_state)) + .map_err(store)?, cache, record, }) @@ -781,28 +806,50 @@ impl RunStores { } /// Every ontology entry of a panel language reads against the run's head - /// inventory at its own revision: the check the store applied at admission, - /// applied again to this inventory. + /// inventory: every condition's feature is live at the head in the + /// condition's slot and fits its test. An entry's `inventory_rev` names a + /// revision of the inventory it was admitted through, and beds number + /// revisions independently, so a run never reads it. fn check_ontology(&self) -> Result<(), RunError> { let inventory = self.inventory.inventory().map_err(|_| RunError::Store)?; let entries = self.ontology.all().map_err(|_| RunError::Store)?; - let head = inventory.latest().map(|r| r.n); entries .iter() .filter(|e| self.record.panel.contains(&e.language)) .try_for_each(|e| { - // At its own revision, and again at the head: an entry on a - // feature the head retired could never select. - let at_head = OntologyEntry { - inventory_rev: head.ok_or(RunError::Ontology)?, - ..e.clone() - }; - check_ontology_entry(e, &inventory) - .and_then(|()| check_ontology_entry(&at_head, &inventory)) - .map_err(|_| RunError::Ontology) + let head = inventory.latest().map(|r| r.n).ok_or(RunError::Ontology)?; + check_ontology_entry(e, &inventory, head).map_err(|_| RunError::Ontology) }) } + /// Every scene's relation features read against the run's head inventory + /// in slot `Scene`, as [`check_ontology`](Self::check_ontology) reads + /// entries: the scene's `inventory_rev` is admission provenance a run never reads. + fn check_scenes(&self) -> Result<(), RunError> { + let inventory = self.inventory.inventory().map_err(|_| RunError::Store)?; + let scenes = self.speaker_state.all_scenes().map_err(|_| RunError::Store)?; + if scenes.is_empty() { + return Ok(()); + } + let head = inventory.latest().map(|r| r.n).ok_or(RunError::SpeakerState)?; + for r in scenes.iter().flat_map(|s| &s.relations) { + if !feature_refusals(&inventory, head, &r.features, Slot::Scene).is_empty() { + return Err(RunError::SpeakerState); + } + } + Ok(()) + } + + /// The one door by which a loop view gets its scene: `with_scene` over the + /// run's pinned speaker-state and corpus stores. + fn assembled( + &self, + view: LoweringInput, + inventory: &InventoryStore, + ) -> Result { + view.with_scene(&inventory.inventory()?, &self.speaker_state, &self.corpus) + } + /// Every panel language holds a live Train and a live Validation item. fn check_panel(&self) -> Result<(), RunError> { let items = self.corpus.all().map_err(|_| RunError::Store)?; @@ -1300,6 +1347,9 @@ impl RunStores { let view = stores[slot] .admit(packet, &self.inventory, &self.corpus, &self.entities) .map_err(|_| RunError::Store)?; + let view = self + .assembled(view, &self.inventory) + .map_err(|_| RunError::Store)?; Prompt::lower( &view, &packet_item, @@ -1586,19 +1636,23 @@ fn score_round( // Phase 2: one lowering prompt per admitted view, answered through the // port. Nothing is kept unless every one is answered. let mut unanswered = Vec::new(); - for (at, id, packet_key, view) in &admitted { + for (at, id, packet_key, view) in admitted { // Each item is lowered in its own language: the only reference the // scorer has is the item's own text. - let language = &items[*at].language; - let Ok(prompt) = Prompt::lower(view, id, language, &inventory, &run.entities, &ontology) else { - fates[*at] = Fate::Refused(Stage::Lower, Some(packet_key.clone())); + let language = &items[at].language; + let Ok(view) = run.assembled(view, inventory_store) else { + fates[at] = Fate::Refused(Stage::Lower, Some(packet_key.clone())); + continue; + }; + let Ok(prompt) = Prompt::lower(&view, &id, language, &inventory, &run.entities, &ontology) else { + fates[at] = Fate::Refused(Stage::Lower, Some(packet_key.clone())); continue; }; calls.push(Call { prompt: prompt.digest(), items: prompt.item_keys().to_vec(), }); - fates[*at] = match model.answer(&prompt) { + fates[at] = match model.answer(&prompt) { Ok(answer) => match LoweringDraft::parse(&answer) { Ok(draft) if draft.usable() => { Fate::Lowered(draft.text, packet_key.clone()) @@ -2793,6 +2847,8 @@ mod tests { panel: vec!["en-US".into()], ontology: String::new(), ontology_digest: String::new(), + speaker_state: String::new(), + speaker_state_digest: String::new(), } } @@ -2884,7 +2940,7 @@ mod tests { } /// Answers the Dao encode prompts from `answers` by item, the candidate - /// inventory's (rev 4) from the good drafts, and the editor with one concept. + /// inventory's (rev 5) from the good drafts, and the editor with one concept. struct Fake { degraded: HashMap, good: HashMap, @@ -2911,7 +2967,7 @@ mod tests { }; return Ok(format!(r#"{{"text":"{text}"}}"#)); } - let answers = if prompt.text().contains("\n\ninventory rev:4\n") { + let answers = if prompt.text().contains("\n\ninventory rev:5\n") { &self.good } else { &self.degraded @@ -2985,6 +3041,8 @@ mod tests { ); let ontology = dir.path().join("o.cc"); fs::write(&ontology, [0x90]).unwrap(); + let speaker_state = dir.path().join("s.cc"); + fs::write(&speaker_state, [0x90]).unwrap(); let root = dir.path().join("run"); let mut run = RunStores::create( &root, @@ -2994,6 +3052,7 @@ mod tests { inventory: &inventory, entities: &entities, ontology: &ontology, + speaker_state: &speaker_state, }, "fake-model", "bag", diff --git a/crates/weksa/src/speaker.rs b/crates/weksa/src/speaker.rs index c2e0e58..0f44d13 100644 --- a/crates/weksa/src/speaker.rs +++ b/crates/weksa/src/speaker.rs @@ -71,6 +71,8 @@ pub struct Scene { /// The revision, from 1. #[cultcache(key = 1)] pub n: u32, + /// The revision, of the inventory the scene was admitted through, at which + /// `put_scene` read its features: provenance only; a run never reads it. #[cultcache(key = 2)] pub inventory_rev: u32, #[cultcache(key = 3)] diff --git a/crates/weksa/src/store.rs b/crates/weksa/src/store.rs index 5afb025..7408dce 100644 --- a/crates/weksa/src/store.rs +++ b/crates/weksa/src/store.rs @@ -1273,7 +1273,7 @@ impl OntologyStore { return Err(malformed(&key, "a language holds at most 64 entries")); } counted.insert(e.language.as_str(), e.n); - check_ontology_entry(e, &inv)?; + check_ontology_entry(e, &inv, e.inventory_rev)?; } let mut staged = Vec::new(); for e in &batch { @@ -1286,13 +1286,14 @@ impl OntologyStore { /// The rules of one ontology entry against an inventory: the annotator, the /// bounded and normalized text of the choice and examples, the condition count -/// and each condition's feature (slot `Stance` or `Discourse`, a grade feature -/// for `AtLeast` and `AtMost`, an option or flag value for `Is`) read at the -/// entry's inventory revision. The store applies it on admission and a run -/// applies it again against its own head inventory. +/// and each condition's feature (slot `Stance`, `Discourse` or `Scene`, a grade +/// feature for `AtLeast` and `AtMost`, an option or flag value for `Is`) read at +/// `rev`. The store passes the entry's own `inventory_rev` on admission; a run +/// passes its inventory's head, because beds number revisions independently. pub(crate) fn check_ontology_entry( e: &OntologyEntry, inventory: &Inventory, + rev: u32, ) -> Result<(), StoreError> { let key = e.key(); check_annotator(&key, &e.annotator)?; @@ -1320,21 +1321,24 @@ pub(crate) fn check_ontology_entry( } let mut out = Vec::new(); for c in &e.when { - if !matches!(c.slot, Slot::Stance | Slot::Discourse) { - return Err(malformed(&key, "a condition reads slot Stance or Discourse")); + if !matches!(c.slot, Slot::Stance | Slot::Discourse | Slot::Scene) { + return Err(malformed( + &key, + "a condition reads slot Stance, Discourse or Scene", + )); } if matches!(c.test, Test::Is(FeatValue::Grade(_))) { return Err(malformed(&key, "Is names an option or a flag")); } out.extend(feature_refusals( inventory, - e.inventory_rev, + rev, &[c.as_value()], c.slot, )); } - if e.when.is_empty() && !inventory.has_revision(e.inventory_rev) { - out.extend(feature_refusals(inventory, e.inventory_rev, &[], Slot::Stance)); + if e.when.is_empty() && !inventory.has_revision(rev) { + out.extend(feature_refusals(inventory, rev, &[], Slot::Stance)); } if !out.is_empty() { return Err(StoreError::Invalid(out)); diff --git a/crates/weksa/tests/cli.rs b/crates/weksa/tests/cli.rs index d46db9f..65563ea 100644 --- a/crates/weksa/tests/cli.rs +++ b/crates/weksa/tests/cli.rs @@ -14,6 +14,8 @@ fn weksa(dir: &std::path::Path, args: &[&str]) -> Output { "./e.cc", "--ontology", "./o.cc", + "--speaker-state", + "./s.cc", ]) .args(args) .output() @@ -1341,6 +1343,7 @@ fn dao_stores(dir: &std::path::Path) { std::fs::copy(src.join(from), dir.join(to)).unwrap(); } std::fs::write(dir.join("o.cc"), [0x90]).unwrap(); + std::fs::write(dir.join("s.cc"), [0x90]).unwrap(); } #[test] @@ -1413,7 +1416,7 @@ fn loop_open_and_unanswered_round() { assert!(second.status.success(), "{}", out(&second)); assert_eq!( out(&second), - "round 1 rev 3 en-US train n=1 scored=0 chrf=0.00 meaning=0.0000 refused=1 validation n=2 scored=0 chrf=0.00 meaning=0.0000 refused parse=2 admit=0 lower=0 parse-reasons=syntaxx3 lowering=tool-less-relay\n" + "round 1 rev 4 en-US train n=1 scored=0 chrf=0.00 meaning=0.0000 refused=1 validation n=2 scored=0 chrf=0.00 meaning=0.0000 refused parse=2 admit=0 lower=0 parse-reasons=syntaxx3 lowering=tool-less-relay\n" ); // The global store flags are not read by loop round: a drifted store is refused. @@ -1546,7 +1549,7 @@ fn loop_cycle_asks_then_decides() { assert!(done.status.success(), "{}", out(&done)); assert_eq!( out(&done), - "cycle 1 rev 3->none en-US base=0.0000 candidate=none floor=0.0000 threshold=0.0050 rejected edit-parse lowering=tool-less-relay\n" + "cycle 1 rev 4->none en-US base=0.0000 candidate=none floor=0.0000 threshold=0.0050 rejected edit-parse lowering=tool-less-relay\n" ); // The next cycle asks afresh in its own directory. @@ -1811,3 +1814,64 @@ fn ontology_admit_and_dump() { assert!(!out(&o).contains("LEAKYCHOICE"), "{}", out(&o)); assert_eq!(out(&weksa(d.path(), &["ontology", "dump"])), before); } + +#[test] +fn scene_put_and_dump() { + let d = tempfile::tempdir().unwrap(); + cast_entities(d.path(), &["nibu", "brokkr"]); + for step in ["seed", "proof", "speaker-scene"] { + let o = weksa(d.path(), &["inventory", step]); + assert!(o.status.success(), "{}", out(&o)); + } + let mut add = ADD.to_vec(); + add.extend(["--persona", "nibu", "--fixture", "f.yaml", "--author", "Ada Writer"]); + let o = weksa(d.path(), &add); + assert!(o.status.success(), "{}", out(&o)); + let put = |group: &str, to: &str, sets: &[&str]| { + let mut args = vec![ + "scene", "put", "--group", group, "--to", to, "--inventory-rev", "3", + "--model-family", "anthropic", "--generator", "a scene annotator", + ]; + for s in sets { + args.extend(["--set", s]); + } + weksa(d.path(), &args) + }; + let sets = ["relative_status=above", "intimacy=200", "in_group=false"]; + let o = put("scene-1", "unnamed", &sets); + assert!(o.status.success(), "{}", out(&o)); + let dump = out(&weksa(d.path(), &["scene", "dump"])); + for want in ["scene:scene-1@1", "rev:3", "Anthropic", "a scene annotator", "entity:nibu->unnamed", "relative_status=above", "intimacy=200", "in_group=false"] { + assert!(dump.contains(want), "{want}: {dump}"); + } + // A repeat is unchanged and writes nothing. + let o = put("scene-1", "unnamed", &sets); + assert!(out(&o).contains("unchanged"), "{}", out(&o)); + assert_eq!(out(&weksa(d.path(), &["scene", "dump"])), dump); + // A changed value is the next revision. + let o = put("scene-1", "unnamed", &["relative_status=level", "intimacy=200", "in_group=false"]); + assert!(o.status.success(), "{}", out(&o)); + assert!(out(&weksa(d.path(), &["scene", "dump"])).contains("scene:scene-1@2")); + // An unknown gloss and an unknown group are refused and not echoed. + for (group, set) in [("scene-1", "canarygloss=1"), ("canarygroup", "intimacy=1")] { + let before = out(&weksa(d.path(), &["scene", "dump"])); + let o = put(group, "unnamed", &[set]); + assert!(!o.status.success()); + assert!(!out(&o).contains("canary"), "{}", out(&o)); + assert_eq!(out(&weksa(d.path(), &["scene", "dump"])), before); + } + // A group with two live speakers is refused: multi-party scenes wait. + let mut two = add.clone(); + let at = two.iter().position(|a| *a == "entity:nibu").unwrap(); + two[at] = "entity:brokkr"; + let at = two.iter().position(|a| *a == "nibu").unwrap(); + two[at] = "brokkr"; + two.retain(|a| *a != "Mind the hatch."); + let at = two.iter().position(|a| *a == "--text").unwrap() + 1; + two.insert(at, "Brokkr speaks."); + let o = weksa(d.path(), &two); + assert!(o.status.success(), "{}", out(&o)); + let o = put("scene-1", "unnamed", &sets); + assert!(!o.status.success()); + assert!(!out(&o).contains("scene-1"), "{}", out(&o)); +} diff --git a/crates/weksa/tests/common/mod.rs b/crates/weksa/tests/common/mod.rs index 8350428..fa2deb9 100644 --- a/crates/weksa/tests/common/mod.rs +++ b/crates/weksa/tests/common/mod.rs @@ -123,6 +123,35 @@ pub fn ontology_in(dir: &std::path::Path) -> std::path::PathBuf { path } +/// A new inventory store at `path` holding the Dao inventory as it stood at +/// rev 3 (seed, proof, the Dao revision): the loop tests' fixed head, whose +/// round and candidate revisions they pin. The committed +/// `data/daodejing/inventory.cc` is rev 4 and is read by the corpus tests. +pub fn dao_rev3_at(path: &std::path::Path) { + let mut store = weksa::InventoryStore::open(path).unwrap(); + for (rev, entries) in [ + weksa::inventory::seed(), + weksa::inventory::proof_revision(), + weksa::daodejing::revision(), + ] { + store.admit_revision(rev, entries).unwrap(); + } +} + +/// An empty speaker-state store beside `ontology` (same directory), made if absent. +pub fn speaker_state_beside(ontology: &std::path::Path) -> std::path::PathBuf { + let path = ontology.with_file_name("speaker-state.cc"); + if !path.exists() { + std::fs::write(&path, [0x90]).unwrap(); + } + path +} + +/// An empty speaker-state store at `/speaker-state.cc`, made if absent. +pub fn speaker_state_in(dir: &std::path::Path) -> std::path::PathBuf { + speaker_state_beside(&dir.join("ontology.cc")) +} + /// A run over the stores `world()` and `corpus()` keep in `dir`: `c.cc`, /// `i.cc` and `entities.cc`. Read at the call, so create it after the items. /// The run's en-US panel needs a live Train and a live Validation item: a @@ -158,6 +187,7 @@ pub fn run_over(dir: &std::path::Path) -> weksa::run::RunStores { inventory: &dir.join("i.cc"), entities: &dir.join("entities.cc"), ontology: &dir.join("ontology.cc"), + speaker_state: &speaker_state_beside(&dir.join("ontology.cc")), }, "fake-model", "bag", diff --git a/crates/weksa/tests/context.rs b/crates/weksa/tests/context.rs index 5d44f01..82cf5b9 100644 --- a/crates/weksa/tests/context.rs +++ b/crates/weksa/tests/context.rs @@ -547,7 +547,8 @@ mod lowering { Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data").join(sub) } - fn admitted( + /// The admitted view, its scene never read. + fn admitted_unread( packet: Packet, inv: &InventoryStore, corpus: &str, @@ -561,6 +562,37 @@ mod lowering { .unwrap() } + /// The admitted view with its scene read over an empty speaker-state store. + fn admitted( + packet: Packet, + inv: &InventoryStore, + corpus: &str, + entities: &EntityStore, + ) -> LoweringInput { + let view = admitted_unread(packet, inv, corpus, entities); + let dir = tempfile::tempdir().unwrap(); + let state = weksa::SpeakerStateStore::open(&dir.path().join("s.cc")).unwrap(); + let corpus = CorpusStore::open(&data(corpus)).unwrap(); + view.with_scene(&inv.inventory().unwrap(), &state, &corpus).unwrap() + } + + #[test] + fn lowering_prompt_refuses_an_unread_scene() { + let cast = InventoryStore::open(&data("interlingua/inventory.cc")).unwrap(); + let entities = EntityStore::open(&data("entity/entity.cc")).unwrap(); + let inv = cast.inventory().unwrap(); + let item = CorpusItemId::parse("item:nibu:1").unwrap(); + let held = Held(vec![nibu(&[("en-US", &["Nibu"])])]); + let unread = admitted_unread(proof_packet(), &cast, "corpus/corpus.cc", &entities); + let empty = weksa::Ontology::empty(); + assert_eq!( + Prompt::lower(&unread, &item, "en-US", &inv, &held, &empty).err(), + Some(LowerError::SceneUnread) + ); + let read = admitted(proof_packet(), &cast, "corpus/corpus.cc", &entities); + assert!(Prompt::lower(&read, &item, "en-US", &inv, &held, &empty).is_ok()); + } + fn dao(index: usize) -> (LoweringInput, Inventory, EntityStore, CorpusItemId) { let inv = InventoryStore::open(&data("daodejing/inventory.cc")).unwrap(); let entities = EntityStore::open(&data("daodejing/entity.cc")).unwrap(); diff --git a/crates/weksa/tests/golf_loop.rs b/crates/weksa/tests/golf_loop.rs index 0b22f4c..3ce5e62 100644 --- a/crates/weksa/tests/golf_loop.rs +++ b/crates/weksa/tests/golf_loop.rs @@ -24,12 +24,13 @@ fn rewrite_run(root: &Path, change: impl FnOnce(&mut RunRecord)) { cache.put(format!("run:{}", record.label), &record).unwrap(); } -fn paths(dir: &Path) -> (PathBuf, PathBuf, PathBuf, PathBuf) { +fn paths(dir: &Path) -> (PathBuf, PathBuf, PathBuf, PathBuf, PathBuf) { ( dir.join("c.cc"), dir.join("i.cc"), dir.join("entities.cc"), common::ontology_in(dir), + common::speaker_state_in(dir), ) } @@ -42,7 +43,7 @@ fn a_run_pins_its_stores_and_refuses_drift() { let mut w = world(); let dir = w.dir.path().to_path_buf(); let root = dir.join("run"); - let (c, i, e, o) = paths(&dir); + let (c, i, e, o, s) = paths(&dir); let at = |label: &str, model: &str| { RunStores::create( &root, @@ -52,6 +53,7 @@ fn a_run_pins_its_stores_and_refuses_drift() { inventory: &i, entities: &e, ontology: &o, + speaker_state: &s, }, model, "bag", @@ -147,7 +149,7 @@ fn a_run_works_on_its_own_copy_of_the_inventory() { let w = world(); let dir = w.dir.path().to_path_buf(); let root = dir.join("run"); - let (c, i, e, o) = paths(&dir); + let (c, i, e, o, s) = paths(&dir); let source_bytes = std::fs::read(&i).unwrap(); let mut run = RunStores::create( &root, @@ -157,6 +159,7 @@ fn a_run_works_on_its_own_copy_of_the_inventory() { inventory: &i, entities: &e, ontology: &o, + speaker_state: &s, }, "fake-model", "bag", @@ -236,6 +239,7 @@ fn create_refuses_a_root_that_holds_an_inventory_and_never_truncates_it() { inventory, entities: &dao.entities, ontology: &dao.ontology, + speaker_state: &common::speaker_state_beside(&dao.ontology), }, "fake-model", "bag", @@ -285,6 +289,7 @@ fn a_create_that_fails_leaves_no_inventory_and_the_retry_succeeds() { inventory, entities, ontology: &dao.ontology, + speaker_state: &common::speaker_state_beside(&dao.ontology), }, "fake-model", "bag", @@ -398,11 +403,13 @@ fn a_store_changed_between_its_digest_and_its_open_is_drift() { } let (c, i, e) = (dir.join("c.cc"), dir.join("i.cc"), dir.join("entities.cc")); let o = dir.join("ontology.cc"); + let s = common::speaker_state_beside(&o); let paths = || weksa::run::RunPaths { corpus: &c, inventory: &i, entities: &e, ontology: &o, + speaker_state: &s, }; assert_eq!( RunStores::create_between( @@ -457,6 +464,7 @@ fn window( inventory: &i, entities: &e, ontology: &o, + speaker_state: &common::speaker_state_beside(&o), }; let created = RunStores::create_between( &root, @@ -888,9 +896,11 @@ fn dao_build( std::fs::copy(src.join(from), dir.path().join(to)).unwrap(); dir.path().join(to) }; + // The Dao inventory at rev 3: the pins below are written against it. + common::dao_rev3_at(&dir.path().join("i.cc")); let (corpus, inventory, entities) = ( copy("corpus.cc", "c.cc"), - copy("inventory.cc", "i.cc"), + dir.path().join("i.cc"), copy("entity.cc", "e.cc"), ); edit(&corpus, &entities); @@ -905,6 +915,7 @@ fn dao_build( inventory: &inventory, entities: &entities, ontology: &ontology, + speaker_state: &common::speaker_state_beside(&ontology), }, "fake-model", "bag", @@ -1087,6 +1098,8 @@ fn lowering_prompts(dao: &Dao, inventory: &InventoryStore) -> Vec<(String, Promp let view = store .admit(packet, inventory, &corpus, dao.run.entities()) .unwrap(); + let state = weksa::SpeakerStateStore::open(&dir.path().join("s.cc")).unwrap(); + let view = view.with_scene(&inv, &state, &corpus).unwrap(); let prompt = Prompt::lower(&view, &key, "en-US", &inv, dao.run.entities(), &weksa::Ontology::empty()).unwrap(); (key.to_string(), prompt) }) @@ -1560,6 +1573,7 @@ fn errors_echo_no_input() { inventory: &missing, entities: &missing, ontology: &missing, + speaker_state: &common::speaker_state_beside(&missing), }; let e = RunStores::create( &root, @@ -1629,6 +1643,7 @@ fn errors_echo_no_input() { inventory: &dao.inventory, entities: &dao.entities, ontology: &dao.ontology, + speaker_state: &common::speaker_state_beside(&dao.ontology), }, "fake-model", "bag", @@ -2551,6 +2566,7 @@ fn a_changed_head_is_drift() { inventory: &dao.inventory, entities: &dao.entities, ontology: &dao.ontology, + speaker_state: &common::speaker_state_beside(&dao.ontology), }, "fake-model", "bag", @@ -2891,6 +2907,7 @@ fn a_summary_line_says_how_lowerings_were_answered() { inventory: &dao.inventory, entities: &dao.entities, ontology: &dao.ontology, + speaker_state: &common::speaker_state_beside(&dao.ontology), }; RunStores::create( &root, @@ -3615,6 +3632,7 @@ fn mixed_family_round(family: ModelFamily) -> u32 { inventory: &dir.join("i.cc"), entities: &dir.join("entities.cc"), ontology: &dir.join("ontology.cc"), + speaker_state: &common::speaker_state_beside(&dir.join("ontology.cc")), }, "not-a-draft", "bag", @@ -4402,7 +4420,7 @@ fn open_refuses_a_bad_panel() { let created = RunStores::create( &root, "p", - RunPaths { corpus, inventory, entities, ontology: &dao.ontology }, + RunPaths { corpus, inventory, entities, ontology: &dao.ontology, speaker_state: &common::speaker_state_beside(&dao.ontology) }, "fake-model", "bag", weksa::run::LowererReach::ToolLessRelay, @@ -4762,6 +4780,7 @@ fn create_refuses_an_entry_its_inventory_cannot_read() { inventory: &dir.join("i.cc"), entities: &dir.join("e.cc"), ontology, + speaker_state: &common::speaker_state_beside(ontology), }, "fake-model", "bag", @@ -4817,3 +4836,281 @@ fn round_lowering_prompts_carry_selected_entries() { } assert_eq!(seen.len(), 6, "three items in each language"); } + +/// Admits the speaker-state revision (rev 4) onto the rev-3 Dao inventory at +/// `inventory`, and the scene for the group of `item:daodejing:3`: Laozi's +/// relation to an unnamed party, above, intimacy 200. +fn put_dao_scene(speaker_state: &Path, inventory: &Path, corpus: &Path, entities: &Path) { + use weksa::packet::{FeatValue, FeatureValue}; + let mut inv_store = InventoryStore::open(inventory).unwrap(); + let (rev, entries) = weksa::inventory::speaker_scene_revision(4); + inv_store.admit_revision(rev, entries).unwrap(); + let inv = inv_store.inventory().unwrap(); + let id = |gloss: &str| inv.entries().find(|e| e.gloss == gloss).unwrap().id; + let corpus = CorpusStore::open(corpus).unwrap(); + let item = corpus.get("item:daodejing:3").unwrap().unwrap(); + let scene = weksa::Scene { + group: item.group.clone(), + n: 1, + inventory_rev: 4, + participants: vec![item.speaker.clone()], + relations: vec![weksa::PairRelation { + from: item.speaker.clone(), + to: weksa::Party::Unnamed, + features: vec![ + FeatureValue { + feature: id("relative_status"), + value: FeatValue::Option(common::lab("above")), + }, + FeatureValue { + feature: id("intimacy"), + value: FeatValue::Grade(common::grade(200)), + }, + ], + }], + annotator: weksa::Annotator::Operator, + }; + weksa::SpeakerStateStore::open(speaker_state) + .unwrap() + .put_scene(scene, &inv_store, &corpus, &weksa::EntityStore::open(entities).unwrap()) + .unwrap(); +} + +/// A Dao run whose store holds a scene for item 3's group and whose ontology +/// holds en-US:1, selected when the speaker is above the addressee. +fn scene_dao(extra: impl FnOnce(&Path, &Path)) -> Dao { + dao_build( + |_, _| {}, + &["en-US"], + |ontology, inventory| { + use weksa::{Condition, Layer, OntologyEntry, OntologyStore, Slot, Test}; + let dir = ontology.parent().unwrap(); + put_dao_scene( + &common::speaker_state_beside(ontology), + inventory, + &dir.join("c.cc"), + &dir.join("e.cc"), + ); + let inv_store = InventoryStore::open(inventory).unwrap(); + let status = inv_store + .inventory() + .unwrap() + .entries() + .find(|e| e.gloss == "relative_status") + .unwrap() + .id; + OntologyStore::open(ontology) + .unwrap() + .admit( + vec![OntologyEntry { + language: "en-US".into(), + n: 1, + inventory_rev: 4, + layer: Layer::Discourse, + when: vec![Condition { + slot: Slot::Scene, + feature: status, + test: Test::Is(weksa::packet::FeatValue::Option(common::lab("above"))), + }], + choice: "ENABOVE".into(), + examples: vec![], + annotator: weksa::Annotator::Operator, + }], + &inv_store, + ) + .unwrap(); + extra(&dir.join("c.cc"), &dir.join("e.cc")); + }, + ) +} + +/// Answers for items 1 to 3 where item 3 names referent n2 (no entity) as its +/// addressee, or nobody. +fn addressing(addressee: Option<&str>) -> Vec<(usize, String)> { + (1..=3) + .map(|n| { + let mut v = serde_json::to_value(draft_of(n)).unwrap(); + if n == 3 { + v["discourse"]["addressee"] = serde_json::json!(addressee); + } + (n, v.to_string()) + }) + .collect() +} + +fn lowering_of(model: &FakeModel, marker: &str) -> Vec { + lowering_texts(model) + .into_iter() + .filter(|t| t.contains(marker)) + .map(String::from) + .collect() +} + +#[test] +fn round_prompts_carry_scene_selected_entries() { + // Above an unnamed addressee: item 3's prompt holds the entry, the other + // two (no scene for their groups) read NoScene and print without it. + let mut dao = scene_dao(|_, _| {}); + let mut model = fake(&addressing(Some("n2"))); + recorded(round(&mut dao.run, &mut model, &Scorer::new(Bag)).unwrap()); + assert_eq!(lowering_texts(&model).len(), 3); + let with = lowering_of(&model, "\nonto:en-US:1 [discourse] ENABOVE"); + assert_eq!(with.len(), 1, "only item 3 has a scene and an addressee"); + assert_eq!( + lowering_texts(&model).iter().filter(|t| t.contains("ontology: none")).count(), + 2 + ); + // The same item with no addressee selects no Scene entry. + let mut dao = scene_dao(|_, _| {}); + let mut model = fake(&addressing(None)); + recorded(round(&mut dao.run, &mut model, &Scorer::new(Bag)).unwrap()); + assert!(lowering_of(&model, "ENABOVE").is_empty()); + assert_eq!(lowering_texts(&model).len(), 3); +} + +#[test] +fn a_scene_read_failure_is_a_lower_refusal() { + // A live item of item 3's group, in a language outside the panel, spoken by + // someone the scene does not cover: V16 for that group. + let mut dao = scene_dao(|corpus, entities| { + let mut entity_store = weksa::EntityStore::open(entities).unwrap(); + entity_store.put(ent("stranger", "en-US", &["Stranger"])).unwrap(); + let mut store = CorpusStore::open(corpus).unwrap(); + let mut twin = store.get("item:daodejing:3").unwrap().unwrap(); + twin.id = "item:daodejing:93".into(); + twin.language = "pt-BR".into(); + twin.text = "PORTUGUESEMARK 93 o sabio".into(); + twin.speaker = common::eid("stranger"); + twin.names = vec![]; + twin.prior_turns = vec![]; + store.add(twin, &entity_store).unwrap(); + }); + let mut model = fake(&addressing(Some("n2"))); + let r = recorded(round(&mut dao.run, &mut model, &Scorer::new(Bag)).unwrap()); + // Item 3 never reached the model; items 1 and 2 did, and the round recorded. + assert_eq!(lowering_texts(&model).len(), 2); + assert_eq!(r.n(), 1); +} + +#[test] +fn a_run_pins_its_speaker_state_and_refuses_drift() { + let dao = scene_dao(|_, _| {}); + let r = dao.run.record(); + let path = dao.ontology.with_file_name("speaker-state.cc"); + assert_eq!(r.speaker_state, std::fs::canonicalize(&path).unwrap().to_str().unwrap()); + assert_eq!(r.speaker_state_digest, hex(&path)); + let root = dao.root.clone(); + assert_eq!(RunStores::open(&root).unwrap().record(), dao.run.record()); + let mut bytes = std::fs::read(&path).unwrap(); + bytes.push(0xc0); + std::fs::write(&path, bytes).unwrap(); + assert_eq!( + RunStores::open(&root).err(), + Some(RunError::Drift(DriftOf::SpeakerState)) + ); +} + +#[test] +fn create_refuses_a_scene_its_inventory_cannot_read() { + // The scene is admitted against a copy that gained rev 4; the run's head + // inventory stops at rev 3. + let src = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/daodejing"); + let dir = tempfile::tempdir().unwrap(); + for (from, to) in [("corpus.cc", "c.cc"), ("entity.cc", "e.cc")] { + std::fs::copy(src.join(from), dir.path().join(to)).unwrap(); + } + common::dao_rev3_at(&dir.path().join("i.cc")); + common::dao_rev3_at(&dir.path().join("rich.cc")); + let state = common::speaker_state_in(dir.path()); + put_dao_scene(&state, &dir.path().join("rich.cc"), &dir.path().join("c.cc"), &dir.path().join("e.cc")); + let ontology = common::ontology_in(dir.path()); + let root = dir.path().join("run"); + let create = || { + RunStores::create( + &root, + "x", + RunPaths { + corpus: &dir.path().join("c.cc"), + inventory: &dir.path().join("i.cc"), + entities: &dir.path().join("e.cc"), + ontology: &ontology, + speaker_state: &state, + }, + "fake-model", + "bag", + weksa::run::LowererReach::ToolLessRelay, + ModelFamily::Unrecorded, + &["en-US".to_string()], + ) + }; + assert_eq!(create().err(), Some(RunError::SpeakerState)); + assert!(!root.join("run.cc").exists()); + assert!(!root.join("inventory.cc").exists()); +} + +#[test] +fn a_run_reads_entries_and_scenes_at_its_own_head() { + use weksa::{Condition, Layer, OntologyEntry, OntologyStore, Slot, Test}; + // An entry and a scene admitted at rev 4 through a Dao-shaped inventory open + // a run whose own inventory is seed, proof and the scene revision as rev 3: + // beds number revisions independently, so a run reads features at its head. + let src = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/daodejing"); + let dir = tempfile::tempdir().unwrap(); + for (from, to) in [("corpus.cc", "c.cc"), ("entity.cc", "e.cc")] { + std::fs::copy(src.join(from), dir.path().join(to)).unwrap(); + } + let rich = dir.path().join("rich.cc"); + common::dao_rev3_at(&rich); + let state = common::speaker_state_in(dir.path()); + put_dao_scene(&state, &rich, &dir.path().join("c.cc"), &dir.path().join("e.cc")); + let rich_store = InventoryStore::open(&rich).unwrap(); + let status = rich_store.inventory().unwrap().entries().find(|e| e.gloss == "relative_status").unwrap().id; + let ontology = common::ontology_in(dir.path()); + OntologyStore::open(&ontology) + .unwrap() + .admit( + vec![OntologyEntry { + language: "en-US".into(), + n: 1, + inventory_rev: 4, + layer: Layer::Discourse, + when: vec![Condition { + slot: Slot::Scene, + feature: status, + test: Test::Is(weksa::packet::FeatValue::Option(common::lab("above"))), + }], + choice: "ENABOVE".into(), + examples: vec![], + annotator: weksa::Annotator::Operator, + }], + &rich_store, + ) + .unwrap(); + let head = dir.path().join("i.cc"); + let mut own = InventoryStore::open(&head).unwrap(); + for (rev, entries) in [ + weksa::inventory::seed(), + weksa::inventory::proof_revision(), + weksa::inventory::speaker_scene_revision(3), + ] { + own.admit_revision(rev, entries).unwrap(); + } + drop(own); + let run = RunStores::create( + &dir.path().join("run"), + "x", + RunPaths { + corpus: &dir.path().join("c.cc"), + inventory: &head, + entities: &dir.path().join("e.cc"), + ontology: &ontology, + speaker_state: &state, + }, + "fake-model", + "bag", + weksa::run::LowererReach::ToolLessRelay, + ModelFamily::Unrecorded, + &["en-US".to_string()], + ); + assert!(run.is_ok(), "{:?}", run.err()); +} diff --git a/crates/weksa/tests/original_corpus.rs b/crates/weksa/tests/original_corpus.rs index 2a48196..6a651e7 100644 --- a/crates/weksa/tests/original_corpus.rs +++ b/crates/weksa/tests/original_corpus.rs @@ -18,7 +18,7 @@ use weksa::{CorpusStore, EntityStore, InventoryStore}; fn stores() -> (CorpusStore, EntityStore) { let dir = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/original"); let tmp = tempfile::tempdir().unwrap(); - for f in ["corpus.cc", "entity.cc", "inventory.cc"] { + for f in ["corpus.cc", "entity.cc", "inventory.cc", "speaker-state.cc"] { std::fs::copy(dir.join(f), tmp.path().join(f)).unwrap(); } let keep = tmp.keep(); @@ -241,14 +241,22 @@ fn the_first_six_original_items_are_unchanged() { } #[test] -fn the_original_inventory_is_the_dao_revision_three() { +fn the_original_inventory_is_the_dao_inventory() { let dao = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/daodejing/inventory.cc"); let own = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/original/inventory.cc"); assert_eq!(std::fs::read(dao).unwrap(), std::fs::read(&own).unwrap()); let tmp = tempfile::tempdir().unwrap(); let copy = tmp.path().join("inventory.cc"); std::fs::copy(&own, ©).unwrap(); - assert!(InventoryStore::open(©).is_ok()); + let inventory = InventoryStore::open(©).unwrap().inventory().unwrap(); + // Revision 4 is the speaker-state vocabulary: the same six features the + // generator admits, ids and all. + assert_eq!(inventory.latest().unwrap().n, 4); + let (rev, entries) = weksa::inventory::speaker_scene_revision(4); + assert_eq!(inventory.latest().unwrap().changes, rev.changes); + for e in &entries { + assert_eq!(inventory.entry(&e.id), Some(e), "{}", e.gloss); + } } /// The committed seed ontology: every entry model-drafted, exactly the two @@ -270,8 +278,13 @@ fn committed_ontology_is_model_drafted_and_reads_every_inventory() { match &e.annotator { Annotator::Model { family, generator } => { assert_eq!(*family, ModelFamily::Anthropic, "{}", e.key()); - assert_eq!( - generator, "claude-sonnet-5-5 (Weksa Hands, model-drafted seed ontology)", + assert!( + [ + "claude-sonnet-5-5 (Weksa Hands, model-drafted seed ontology)", + "claude-sonnet-5-5 (Weksa Hands, model-drafted scene ontology)", + "claude-sonnet-5-5 (Weksa Hands, reworded seed entry)", + ] + .contains(&generator.as_str()), "{}", e.key() ); @@ -283,6 +296,15 @@ fn committed_ontology_is_model_drafted_and_reads_every_inventory() { langs.sort(); langs.dedup(); assert_eq!(langs, PANEL); + // Every entry from number 10 on is conditioned on the scene relation, and + // only an entry with a Scene condition may name a politeness level. + for e in entries.iter().filter(|e| e.n >= 10) { + assert!( + e.when.iter().any(|c| c.slot == weksa::inventory::Slot::Scene), + "{} has no Scene condition", + e.key() + ); + } for lang in PANEL { let stance = |c: &Condition| c.slot == weksa::inventory::Slot::Stance; assert!( @@ -292,21 +314,35 @@ fn committed_ontology_is_model_drafted_and_reads_every_inventory() { "{lang} has no Stance-conditioned entry" ); } - for (corpus, panel) in [("original", &PANEL[..]), ("daodejing", &PANEL[..1])] { + // A run opens over every committed inventory bed with its panel. The default + // bed (data/interlingua) numbers the scene revision 3 and holds no Dao + // revision; the Dao beds number it 4. Entries and scenes admitted at 4 open + // over both because a run reads features at its own head. The default bed is + // read with the original corpus, entities and scenes, as `loop open` with its + // defaults and `--corpus data/original` does. + for (corpus, bed, panel) in [ + ("original", "original", &PANEL[..]), + ("daodejing", "daodejing", &PANEL[..1]), + ("original", "interlingua", &PANEL[..]), + ] { let dir = data.join(corpus); - let copy = |f: &str| { - std::fs::copy(dir.join(f), tmp.path().join(format!("{corpus}-{f}"))).unwrap(); - tmp.path().join(format!("{corpus}-{f}")) + let name = format!("{corpus}-{bed}"); + let copy = |from: &Path, f: &str| { + let to = tmp.path().join(format!("{name}-{f}")); + std::fs::copy(from.join(f), &to).unwrap(); + to }; - let (c, i, e) = (copy("corpus.cc"), copy("inventory.cc"), copy("entity.cc")); + let (c, i, e) = (copy(&dir, "corpus.cc"), copy(&data.join(bed), "inventory.cc"), copy(&dir, "entity.cc")); + let s = copy(&dir, "speaker-state.cc"); RunStores::create( - &tmp.path().join(format!("run-{corpus}")), + &tmp.path().join(format!("run-{name}")), corpus, RunPaths { corpus: &c, inventory: &i, entities: &e, ontology: &onto, + speaker_state: &s, }, "m", "bag", @@ -314,7 +350,34 @@ fn committed_ontology_is_model_drafted_and_reads_every_inventory() { ModelFamily::Unrecorded, &panel.iter().map(|l| l.to_string()).collect::>(), ) - .unwrap_or_else(|e| panic!("{corpus}: {e:?}")); +.unwrap_or_else(|e| panic!("{name}: {e:?}")); + } + // Revision numbers are local to a bed, so agreement is proven on the beds: + // every feature an entry condition or a committed scene relation names means + // the same in all three (gloss, value kind, slots, retirement). + let mut named: std::collections::BTreeSet = entries + .iter() + .flat_map(|e| e.when.iter().map(|c| c.feature)) + .collect(); + let scenes = weksa::SpeakerStateStore::open(&data.join("original/speaker-state.cc")).unwrap(); + for s in scenes.all_scenes().unwrap() { + named.extend(s.relations.iter().flat_map(|r| r.features.iter().map(|f| f.feature))); + } + let beds: Vec<_> = ["interlingua", "original", "daodejing"] + .iter() + .map(|b| { + let copy = tmp.path().join(format!("bed-{b}.cc")); + std::fs::copy(data.join(b).join("inventory.cc"), ©).unwrap(); + InventoryStore::open(©).unwrap().inventory().unwrap() + }) + .collect(); + for id in named { + let shape = |inv: &weksa::Inventory| { + let e = inv.entry(&id).unwrap_or_else(|| panic!("{id} is missing from a bed")); + (e.gloss.clone(), e.value_kind.clone(), e.applies_to.clone(), e.retired_rev) + }; + assert_eq!(shape(&beds[0]), shape(&beds[1]), "{id}"); + assert_eq!(shape(&beds[1]), shape(&beds[2]), "{id}"); } } @@ -351,6 +414,7 @@ fn the_recall_probe_covers_every_live_original_item_and_names_no_work() { inventory: &i, entities: &e, ontology: &o, + speaker_state: &common::speaker_state_beside(&o), }, "m", "bag", @@ -417,7 +481,7 @@ fn original_round(family: ModelFamily) -> (weksa::run::RunStores, weksa::run::Ro use weksa::run::{LowererReach, RoundOutcome, RunPaths, RunStores, round}; let src = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/original"); let tmp = tempfile::tempdir().unwrap().keep(); - for f in ["corpus.cc", "entity.cc", "inventory.cc"] { + for f in ["corpus.cc", "entity.cc", "inventory.cc", "speaker-state.cc"] { std::fs::copy(src.join(f), tmp.join(f)).unwrap(); } std::fs::write(tmp.join("ontology.cc"), [0x90]).unwrap(); @@ -429,6 +493,7 @@ fn original_round(family: ModelFamily) -> (weksa::run::RunStores, weksa::run::Ro inventory: &tmp.join("inventory.cc"), entities: &tmp.join("entity.cc"), ontology: &tmp.join("ontology.cc"), + speaker_state: &common::speaker_state_beside(&tmp.join("ontology.cc")), }, "nope", "bag", @@ -575,7 +640,7 @@ fn no_test_item_is_offered_to_a_round_or_recall_or_recorded() { use weksa::run::{LowererReach, RoundOutcome, RunPaths, RunStores, round}; let src = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/original"); let tmp = tempfile::tempdir().unwrap(); - for f in ["corpus.cc", "entity.cc", "inventory.cc"] { + for f in ["corpus.cc", "entity.cc", "inventory.cc", "speaker-state.cc"] { std::fs::copy(src.join(f), tmp.path().join(f)).unwrap(); } std::fs::write(tmp.path().join("ontology.cc"), [0x90]).unwrap(); @@ -588,6 +653,7 @@ fn no_test_item_is_offered_to_a_round_or_recall_or_recorded() { inventory: &p("inventory.cc"), entities: &p("entity.cc"), ontology: &p("ontology.cc"), + speaker_state: &common::speaker_state_beside(&p("ontology.cc")), }, "nope", "bag", @@ -951,3 +1017,95 @@ fn every_original_passage_rests_on_a_live_inventory_concept() { assert!(hit.len() >= 12, "{lang}: {} distinct concepts", hit.len()); } } + +/// The committed scenes: one per live Train or Validation group of +/// `data/original`, none for any other group, every one model-annotated and +/// never Test evidence (a Test group gets no scene and reads `NoScene`). +#[test] +fn committed_scenes_cover_original_groups() { + use std::collections::BTreeSet; + use weksa::speaker::Annotator; + use weksa::{CorpusStore, Split, SplitScheme, SpeakerStateStore}; + let data = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/original"); + let tmp = tempfile::tempdir().unwrap(); + let state = tmp.path().join("speaker-state.cc"); + std::fs::copy(data.join("speaker-state.cc"), &state).unwrap(); + let scenes = SpeakerStateStore::open(&state).unwrap().all_scenes().unwrap(); + let live: BTreeSet = CorpusStore::open(&data.join("corpus.cc")) + .unwrap() + .all() + .unwrap() + .into_iter() + .filter(|i| matches!(SplitScheme::V1.live(i), Some(Split::Train | Split::Validation))) + .map(|i| i.group) + .collect(); + assert_eq!(live.len(), 56); + let held: BTreeSet = scenes.iter().map(|s| s.group.clone()).collect(); + assert_eq!(held, live, "a scene for each live Train or Validation group, none other"); + // Revisions append: a group's later revisions re-derived its relation. + assert!(scenes.iter().all(|s| s.n <= 2)); + for s in &scenes { + match &s.annotator { + Annotator::Model { family, generator } => { + assert_eq!(*family, ModelFamily::Anthropic, "{}", s.group); + assert_eq!( + generator, "claude-sonnet-5-5 (Weksa Hands, model-annotated original scenes)", + "{}", + s.group + ); + } + other => panic!("{} is {other:?}", s.group), + } + } +} + +/// The committed ontology's Scene-conditioned entries, over every scene relation +/// (relative_status x in_group x intimacy in steps of 50): two entries of one +/// language and layer are never selected by the same relation, so the ontology +/// never gives two instructions for one relation in one layer. Whether an entry's +/// text keeps a politeness level on the scene is a model reading owed per change, +/// not a test. +#[test] +fn scene_entries_never_co_select_in_a_layer() { + use weksa::inventory::Slot; + use weksa::ontology::{Condition, Test}; + use weksa::packet::FeatValue; + use weksa::OntologyStore; + let data = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data"); + let tmp = tempfile::tempdir().unwrap(); + let onto = tmp.path().join("ontology.cc"); + std::fs::copy(data.join("ontology/ontology.cc"), &onto).unwrap(); + let inv = InventoryStore::open(&data.join("original/inventory.cc")).unwrap().inventory().unwrap(); + let id = |gloss: &str| inv.entries().find(|e| e.gloss == gloss).unwrap().id; + let (status, intimacy, in_group) = (id("relative_status"), id("intimacy"), id("in_group")); + let entries = OntologyStore::open(&onto).unwrap().all().unwrap(); + let holds = |c: &Condition, rel: (&str, u16, bool)| match (c.feature, &c.test) { + (f, Test::Is(FeatValue::Option(l))) if f == status => l.as_str() == rel.0, + (f, Test::AtLeast(g)) if f == intimacy => rel.1 >= g.get(), + (f, Test::AtMost(g)) if f == intimacy => rel.1 <= g.get(), + (f, Test::Is(FeatValue::Flag(b))) if f == in_group => *b == rel.2, + _ => panic!("a Scene condition this test does not model"), + }; + let selects = |e: &weksa::OntologyEntry, rel| { + e.when.iter().all(|c| c.slot != Slot::Scene || holds(c, rel)) + && e.when.iter().all(|c| c.slot == Slot::Scene) + }; + for st in ["below", "level", "above"] { + for ig in [false, true] { + for it in (0..=1000).step_by(50) { + let rel = (st, it, ig); + let mut seen = std::collections::BTreeSet::new(); + for e in entries.iter().filter(|e| e.when.iter().any(|c| c.slot == Slot::Scene)) { + if selects(e, rel) { + assert!( + seen.insert((e.language.clone(), e.layer.name())), + "{} co-selects with another {} entry at {rel:?}", + e.key(), + e.layer.name() + ); + } + } + } + } + } +} diff --git a/crates/weksa/tests/scene.rs b/crates/weksa/tests/scene.rs index a85b2e0..5974883 100644 --- a/crates/weksa/tests/scene.rs +++ b/crates/weksa/tests/scene.rs @@ -27,7 +27,7 @@ struct W { fn world() -> W { let dir = tempfile::tempdir().unwrap(); let mut inventory = InventoryStore::open(&dir.path().join("i.cc")).unwrap(); - for (rev, entries) in [seed(), proof_revision(), speaker_scene_revision()] { + for (rev, entries) in [seed(), proof_revision(), speaker_scene_revision(3)] { inventory.admit_revision(rev, entries).unwrap(); } let entities = entities_in(dir.path()); @@ -60,7 +60,7 @@ fn feat(gloss: &str, value: FeatValue) -> FeatureValue { let feature = seed() .1 .into_iter() - .chain(speaker_scene_revision().1) + .chain(speaker_scene_revision(3).1) .find(|e| e.gloss == gloss) .unwrap_or_else(|| panic!("no feature {gloss}")) .id; @@ -408,3 +408,60 @@ fn only_live_speakers_of_the_views_group_and_the_views_own_speaker_must_be_cover ["V16"] ); } + +mod select { + use super::*; + use weksa::speaker::Annotator; + use weksa::{Condition, Layer, OntologyEntry, OntologyStore, Slot, Test, select}; + + fn scene_feature(gloss: &str) -> weksa::InvId { + speaker_scene_revision(3).1.into_iter().find(|e| e.gloss == gloss).unwrap().id + } + + fn entry(n: u32, slot: Slot, gloss: &str, test: Test) -> OntologyEntry { + OntologyEntry { + language: "ja-JP".into(), + n, + inventory_rev: 3, + layer: Layer::Grammar, + when: vec![Condition { slot, feature: scene_feature(gloss), test }], + choice: "a choice".into(), + examples: vec![], + annotator: Annotator::Operator, + } + } + + #[test] + fn select_reads_the_scene_relation() { + let mut w = world(); + put(&mut w, standard("g", 1)); + let mut store = OntologyStore::open(&w.dir.path().join("o.cc")).unwrap(); + store + .admit( + vec![ + entry(1, Slot::Scene, "relative_status", Test::Is(FeatValue::Option(lab("above")))), + entry(2, Slot::Scene, "intimacy", Test::AtLeast(grade(150))), + ], + &w.inventory, + ) + .unwrap(); + let ontology = store.ontology().unwrap(); + let keys = |read: SceneRead, w: &W| { + let view = view_of(w, 3, Some("n3")).with_scene(&inv(w), &w.state, &w.corpus).unwrap(); + assert_eq!(view.scene_read(), &read); + select(&ontology, "ja-JP", &view).iter().map(|e| e.n).collect::>() + }; + // Brokkr is above-status at intimacy 200: both Scene entries select. + assert_eq!(keys(selected(1, status()), &w), [1, 2]); + // No scene for the group: neither selects. + let empty = world(); + assert_eq!(keys(SceneRead::NoScene, &empty), Vec::::new()); + // No addressee, and no relation to the addressee: neither selects. + let view = view_of(&w, 3, None).with_scene(&inv(&w), &w.state, &w.corpus).unwrap(); + assert_eq!(view.scene_read(), &SceneRead::NoAddressee { rev: 1 }); + assert!(select(&ontology, "ja-JP", &view).is_empty()); + let mut lone = world(); + put(&mut lone, scene("g", 1, &["nibu", "brokkr"], vec![relation("brokkr", Party::Named(eid("nibu")), intimacy(99))])); + assert_eq!(keys(SceneRead::NoRelation { rev: 1 }, &lone), Vec::::new()); + } +} diff --git a/crates/weksa/tests/speaker_state.rs b/crates/weksa/tests/speaker_state.rs index dfb4c59..d79b9dd 100644 --- a/crates/weksa/tests/speaker_state.rs +++ b/crates/weksa/tests/speaker_state.rs @@ -26,7 +26,7 @@ struct W { fn world() -> W { let dir = tempfile::tempdir().unwrap(); let mut inventory = InventoryStore::open(&dir.path().join("i.cc")).unwrap(); - for (rev, entries) in [seed(), proof_revision(), speaker_scene_revision()] { + for (rev, entries) in [seed(), proof_revision(), speaker_scene_revision(3)] { inventory.admit_revision(rev, entries).unwrap(); } let entities = entities_in(dir.path()); @@ -60,7 +60,7 @@ fn feat(gloss: &str, value: FeatValue) -> FeatureValue { let feature = seed() .1 .into_iter() - .chain(speaker_scene_revision().1) + .chain(speaker_scene_revision(3).1) .find(|e| e.gloss == gloss) .unwrap_or_else(|| panic!("no feature {gloss}")) .id; diff --git a/crates/weksa/tests/stores.rs b/crates/weksa/tests/stores.rs index f027611..706adb9 100644 --- a/crates/weksa/tests/stores.rs +++ b/crates/weksa/tests/stores.rs @@ -1584,7 +1584,9 @@ fn seed_reseeded() { .unwrap(); assert_eq!(items.len(), 3); // The committed inventory is the seed, reseeded (stance features in, the - // displayed-affect features out), then the nibu-proof revision. + // displayed-affect features out), the nibu-proof revision and the + // speaker-scene revision as rev 3: never the Dao revision (ruling + // `dao-round-seed-inventory`). let committed = InventoryStore::open(&root.join("data/interlingua/inventory.cc")) .unwrap() .inventory() @@ -1593,6 +1595,8 @@ fn seed_reseeded() { let (proof, proof_entries) = weksa::inventory::proof_revision(); assert!(committed.has_revision(rev.n) && committed.has_revision(proof.n)); entries.extend(proof_entries); + entries.extend(weksa::inventory::speaker_scene_revision(3).1); + assert_eq!(committed.latest().unwrap().n, 3); let mut have: Vec<_> = committed.entries().cloned().collect(); let mut want = entries; have.sort_by_key(|e| e.id); @@ -1740,9 +1744,9 @@ mod ontology { "five conditions", vec![OntologyEntry { when: (0..5).map(|_| restraint()).collect(), ..entry("ja-JP", 1) }], )); - for slot in [Slot::Speaker, Slot::Scene, Slot::Predication] { + for slot in [Slot::Speaker, Slot::Predication] { cases.push(( - "slot not Stance or Discourse", + "slot not Stance, Discourse or Scene", vec![OntologyEntry { when: vec![when(slot, "restraint", Test::AtLeast(Grade::new(1).unwrap()))], ..entry("ja-JP", 1) @@ -1818,6 +1822,43 @@ mod ontology { assert_eq!(s.all().unwrap().len(), 64); } + #[test] + fn ontology_admit_takes_scene_refuses_speaker() { + use weksa::inventory::{proof_revision, speaker_scene_revision}; + let d = dir(); + let mut inventory = InventoryStore::open(&d.path().join("i.cc")).unwrap(); + for (rev, entries) in [seed(), proof_revision(), speaker_scene_revision(3)] { + inventory.admit_revision(rev, entries).unwrap(); + } + let scene_feature = |gloss: &str| { + speaker_scene_revision(3).1.into_iter().find(|e| e.gloss == gloss).unwrap().id + }; + let at = |slot, feature, test| OntologyEntry { + inventory_rev: 3, + when: vec![Condition { slot, feature, test }], + ..entry("ja-JP", 1) + }; + let above = Test::Is(FeatValue::Option(lab("above"))); + let mut s = store(&d); + // A Scene condition on a Scene feature admits. + assert_eq!( + s.admit(vec![at(Slot::Scene, scene_feature("relative_status"), above.clone())], &inventory) + .unwrap(), + AddOutcome::Added + ); + // Speaker and Predication slots refuse, and a Scene condition naming a + // Stance feature refuses. + for bad in [ + at(Slot::Speaker, scene_feature("education"), Test::AtLeast(Grade::new(1).unwrap())), + at(Slot::Predication, feature("modality"), Test::Is(FeatValue::Option(lab("actual")))), + at(Slot::Scene, feature("restraint"), Test::AtLeast(Grade::new(1).unwrap())), + ] { + let bad = OntologyEntry { n: 2, ..bad }; + assert!(s.admit(vec![bad], &inventory).is_err()); + } + assert_eq!(s.all().unwrap().len(), 1); + } + #[test] fn ontology_admit_is_all_or_nothing() { let d = dir(); diff --git a/data/corpus/speaker-state.cc b/data/corpus/speaker-state.cc new file mode 100644 index 0000000..9e99dd5 --- /dev/null +++ b/data/corpus/speaker-state.cc @@ -0,0 +1 @@ +� \ No newline at end of file diff --git a/data/daodejing/inventory.cc b/data/daodejing/inventory.cc index 373eb3c..3904b80 100644 Binary files a/data/daodejing/inventory.cc and b/data/daodejing/inventory.cc differ diff --git a/data/daodejing/speaker-state.cc b/data/daodejing/speaker-state.cc new file mode 100644 index 0000000..9e99dd5 --- /dev/null +++ b/data/daodejing/speaker-state.cc @@ -0,0 +1 @@ +� \ No newline at end of file diff --git a/data/interlingua/inventory.cc b/data/interlingua/inventory.cc index cf353db..de8593f 100644 Binary files a/data/interlingua/inventory.cc and b/data/interlingua/inventory.cc differ diff --git a/data/ontology/ontology.cc b/data/ontology/ontology.cc index 2f9cdf7..e3feabb 100644 Binary files a/data/ontology/ontology.cc and b/data/ontology/ontology.cc differ diff --git a/data/original/inventory.cc b/data/original/inventory.cc index 373eb3c..3904b80 100644 Binary files a/data/original/inventory.cc and b/data/original/inventory.cc differ diff --git a/data/original/speaker-state.cc b/data/original/speaker-state.cc new file mode 100644 index 0000000..c4f563d Binary files /dev/null and b/data/original/speaker-state.cc differ diff --git a/docs/targets/interlingua-lowering-map.md b/docs/targets/interlingua-lowering-map.md index 22db5cd..b32c83c 100644 --- a/docs/targets/interlingua-lowering-map.md +++ b/docs/targets/interlingua-lowering-map.md @@ -33,11 +33,11 @@ means no store holds it and nothing writes it. | Schema version | `weksa.interlingua_packet.v` constant in `crates/weksa`; published JSON Schema at `schemas/cultnet/` | Bumped by a commit that changes the Rust types; the published schema is regenerated in the same commit and a test fails on drift | A cut (Hands, under a spec); never the loop | | Inventory entry | Key `:`, `n` canonical decimal >= 1 (families: concept, frame, role, relation, speech_act, feature, referent_kind, extension); type `weksa.inventory_entry`, with a bounded gloss, value kind (option labels bounded), the slots a feature may sit in (`Discourse`, `Referent`, `Predication`, `Relation`; `Stance` from cut `speaker-state`, `Speaker` and `Scene` from cut `speaker-scene`), `introduced_rev` and optional `retired_rev`. Speaker axes, scene relations and affects are features of those slots, so the loop grows them like any closed-class family and convergence covers them | Added or retired by an inventory revision; never edited in place (a changed meaning is a retire plus an add); a retired id still resolves for packets of earlier revisions | The loop's accepted rounds, whose minting call the context builder assembles from Train items only; Self on merge to main. Admission checks form, not text | | Inventory revision | Key `rev:`, type `weksa.inventory_revision`, with parent, change list and `frozen` | Appended per accepted round; exactly one revision is ever marked frozen, by the convergence test | The loop's accept rule appends; the convergence test freezes; Self records the freeze as a ruling | -| Target profile | Locale or language-project id (`en-US`, `ja-JP`, a conlang's slug). Until entries grow in rounds there is no profile revision document: a run names the profile it lowered with by the ontology store's path and digest in its `RunRecord` (cut `language-ontology`); `profile::` revision documents arrive with follow-up `language-ontology-growth` | The ontology store is revised by commit; a run pins it and refuses drift | Profile authors and accepted rounds; never the lowerer's defaults (`persona-drives-register`) | -| Language ontology entry (cut `language-ontology`) | Key `onto::`, `n` from 1 per language; type `weksa.ontology_entry` in `data/ontology/ontology.cc`: the language, the inventory revision its conditions are read at, a layer (lexicon, grammar, discourse, phonology, orthography), at most four conditions (a feature id of slot `Stance` or `Discourse` with `AtLeast`/`AtMost` a grade or `Is` an option or flag; slots `Speaker` and `Scene` and community ids join with follow-up `language-ontology-view-assembly`), a choice of 1 to 200 bytes, at most three examples of 1 to 120 bytes, and an `Annotator` | Admitted in all-or-nothing batches through `OntologyStore::admit`, immutable, never edited; retire arrives with `language-ontology-growth`. The run pins the store like the entity store (path and digest, `Drift(Ontology)`), never copies it, because no loop step writes it yet. Cut `language-ontology-seed` converts by drafting, not by parse: a fresh model drafts each language's seed from that language's YAML and the inventory's feature lines with no corpus text, then deletes `data/target-language-ontology/` and `.weksa/cultural-ontology/` in the same cut | Seed: a labelled model (`Annotator::Model`), no operator gate, never Test evidence (`ja-ontology-owner-answer`); attestation (operator, native reader) appends later. Accepted rounds later, through a mint call from Train items only. Selection is `ontology::select(ontology, language, view)`, called only inside `Prompt::lower`, so no caller picks entries; the prompt prints each selected entry's id, layer, choice and examples. Its text is minted text for the leak diagnostic, which today has no runtime caller (follow-up `leak-diagnostic-in-round`) | +| Target profile | Locale or language-project id (`en-US`, `ja-JP`, a conlang's slug). Until entries grow in rounds there is no profile revision document: a run names the profile it lowered with by the ontology store's path and digest in its `RunRecord` (cut `language-ontology`); `profile::` revision documents arrive with follow-up `language-ontology-growth` | The ontology store is revised by commit; a run pins it and refuses drift; it likewise pins the speaker-state store (`RunRecord` keys 17 and 18, `Drift(SpeakerState)`, cut `language-ontology-scene`) | Profile authors and accepted rounds; never the lowerer's defaults (`persona-drives-register`) | +| Language ontology entry (cut `language-ontology`) | Key `onto::`, `n` from 1 per language; type `weksa.ontology_entry` in `data/ontology/ontology.cc`: the language, the revision of the inventory it was admitted through (provenance: beds number revisions independently and a run reads each condition's feature at its own head), a layer (lexicon, grammar, discourse, phonology, orthography), at most four conditions (a feature id of slot `Stance`, `Discourse` or `Scene` with `AtLeast`/`AtMost` a grade or `Is` an option or flag; a `Scene` condition reads the relation of the item's speaker to its addressee that `with_scene` selected; slot `Speaker` and community ids wait on follow-up `language-ontology-communities`), a choice of 1 to 200 bytes, at most three examples of 1 to 120 bytes, and an `Annotator`. Whether an entry's text keeps politeness on the scene is a model reader's judgment, owed per change (a mint owes it under `language-ontology-growth`), not a test | Admitted in all-or-nothing batches through `OntologyStore::admit`, immutable, never edited; retire arrives with `language-ontology-growth`. The run pins the store like the entity store (path and digest, `Drift(Ontology)`), never copies it, because no loop step writes it yet. Cut `language-ontology-seed` converts by drafting, not by parse: a fresh model drafts each language's seed from that language's YAML and the inventory's feature lines with no corpus text, then deletes `data/target-language-ontology/` and `.weksa/cultural-ontology/` in the same cut | Seed: a labelled model (`Annotator::Model`), no operator gate, never Test evidence (`ja-ontology-owner-answer`); attestation (operator, native reader) appends later. Accepted rounds later, through a mint call from Train items only. Selection is `ontology::select(ontology, language, view)`, called only inside `Prompt::lower`, so no caller picks entries; the prompt prints each selected entry's id, layer, choice and examples. Its text is minted text for the leak diagnostic, which today has no runtime caller (follow-up `leak-diagnostic-in-round`) | | Community (cut `language-ontology`, follow-up `language-ontology-communities`) | Key `community:`; type `weksa.community`, converted from `data/cultural-ontology/cultures/` and `referents/` (YAML deleted in the same cut) | Revised by commit; a speaker names the communities it belongs to by id | The cultural ontology's author (owner-attested evidence only; Discord-only content stays deleted, `discord-only-content-deleted`) | | Speaker (cut `speaker-scene`) | Key `speaker:@`; type `weksa.speaker`: stable axes as feature values of slot `Speaker` (origin, socioeconomic standing, age cohort, education and whatever the loop adds), at an inventory revision, and an `Annotator` | A new revision appends, never edits; a corpus item and every prior turn name a speaker slug, and a round records the speaker revisions it read | The annotator it names: `PersonaSnapshot` (a projection of a `gamecult.persona_state.v0` document, naming its source document and `updatedAt`), `Operator`, `WorkMetadata` (a public-domain work's own cast list and narration), or `Model { family, generator }` (the 2b `ModelFamily`) | -| Scene (cut `speaker-scene`) | Key `scene:@`, the item's split group; type `weksa.scene`: participants (speaker slugs) and, per ordered pair, relation features of slot `Scene` (relative status, intimacy, in-group, audience and whatever the loop adds), at an inventory revision, and an `Annotator` | As Speaker. Keyed by the group, so a scene never straddles the split | As Speaker | +| Scene (cut `speaker-scene`) | Key `scene:@`, the item's split group; type `weksa.scene`: participants (speaker slugs) and, per ordered pair, relation features of slot `Scene` (relative status, intimacy, in-group, audience and whatever the loop adds), at an inventory revision (provenance; a run reads the features at its own head), and an `Annotator` | As Speaker. Keyed by the group, so a scene never straddles the split. `data/original/speaker-state.cc` holds Model-annotated keeper scenes, one per Train or Validation group of both languages (`weksa scene put|dump`); a Test group has none and reads `NoScene` | As Speaker; a model-annotated scene is never Test evidence | | Corpus item | Key `item::`, `n` canonical decimal >= 1 (no sign, no leading zero, from cut 2b); type `weksa.corpus_item` with text, language, speaker, split group, prior turns, source. Source `ModelWritten` carries a `ModelFamily` enum, the generator string as the fixture wrote it, and a prompt digest (cut 2b). The item carries no speaker state of its own: its speaker slug and group name the Speaker and Scene documents | Admitted once; the whole item is immutable (re-adding an identical item is a no-op, any difference is refused; a correction is a new item that `supersedes` the old); text already held under another key is refused unless the new item supersedes that key (cut 2b); withdrawn, never deleted, so old scores stay explainable | Importers and `weksa corpus add`, under `corpus-cast-plus-public-domain`; the source enum has no variant for Discord or any training-forbidden source (`corpus-clean`) | | Split assignment | Derived: `split(scheme, item.group)` | Fixed for a scheme; a new scheme is a new constant, never a reassignment. A model-written item whose group derives test is in no split; a superseding item keeps its predecessor's group | The hash function. Nobody assigns by hand, so there is no writer to forbid | | Loop round | Key `run: