Compare commits
106 changed files with 2837 additions and 23442 deletions
27
.weksa/cultural-ontology/README.md
Normal file
27
.weksa/cultural-ontology/README.md
Normal file
|
|
@ -0,0 +1,27 @@
|
||||||
|
# Cultural Ontology CultCache Seeds
|
||||||
|
|
||||||
|
This directory stores Weksa cultural ontology as CultCache `.cc` witnesses.
|
||||||
|
|
||||||
|
## Starter Database
|
||||||
|
|
||||||
|
- `starter-targets.cc`
|
||||||
|
|
||||||
|
This store contains one catalog entry and three draft guarded profile entries:
|
||||||
|
|
||||||
|
- `weksa.cultural_ontology.pt-BR.vanilla_brazilian.v0`
|
||||||
|
- `weksa.cultural_ontology.en.transatlantic.v0`
|
||||||
|
- `weksa.cultural_ontology.ja-JP.kanto.contemporary.v0`
|
||||||
|
|
||||||
|
The catalog entry is:
|
||||||
|
|
||||||
|
- `weksa.cultural_ontology.catalog.seed_targets.v0`
|
||||||
|
|
||||||
|
The `.cc` file is the typed state witness. This README is only an inspection
|
||||||
|
guide; it does not own cultural ontology truth.
|
||||||
|
|
||||||
|
The profiles follow `schemas/cultural-ontology-profile.md`: salience axes and
|
||||||
|
linguistic affordances are nested typed state with feature IDs, activation
|
||||||
|
gates, constraints, indexicality, evidence, uncertainty, and trace labels.
|
||||||
|
|
||||||
|
Qdrant retrieval may later propose evidence bundles for these profiles. It does
|
||||||
|
not own accepted profile state.
|
||||||
BIN
.weksa/cultural-ontology/starter-targets.cc
Normal file
BIN
.weksa/cultural-ontology/starter-targets.cc
Normal file
Binary file not shown.
|
|
@ -60,4 +60,4 @@ language-project truth.
|
||||||
`CARGO_HOME=/var/lib/mind-weksa-self/cargo-home cargo test --workspace`.
|
`CARGO_HOME=/var/lib/mind-weksa-self/cargo-home cargo test --workspace`.
|
||||||
- `weksa packet proof <store>` admits the nibu proof packet under the key
|
- `weksa packet proof <store>` admits the nibu proof packet under the key
|
||||||
`packet:6e6962752d303031` (hex of `nibu-001`), not `nibu-proof-001`. Use that key
|
`packet:6e6962752d303031` (hex of `nibu-001`), not `nibu-proof-001`. Use that key
|
||||||
with `weksa pronounce <store> <key>`.
|
with `weksa lower <store> <key>`.
|
||||||
|
|
|
||||||
30
README.md
30
README.md
|
|
@ -5,8 +5,7 @@ culture, and constraint, and the planned lowerer that turns an agent's
|
||||||
meaning packet into the reader's display language.
|
meaning packet into the reader's display language.
|
||||||
|
|
||||||
**What is valuable here.** The interlingua standard and the lowering design: a
|
**What is valuable here.** The interlingua standard and the lowering design: a
|
||||||
meaning packet owns what is meant, the scene state (live: `weksa scene put|dump`, labelled by the rule in
|
meaning packet owns what is meant, a target-language profile owns register,
|
||||||
`docs/scene-annotation.md`; speaker documents are planned) owns the speaker's relation to the addressee, which selects register,
|
|
||||||
address and politeness, and a lowerer owns surface text and leaves a trace.
|
address and politeness, and a lowerer owns surface text and leaves a trace.
|
||||||
|
|
||||||
**The problem it solves.** An agent's outward speech today is one English
|
**The problem it solves.** An agent's outward speech today is one English
|
||||||
|
|
@ -25,21 +24,16 @@ measured.
|
||||||
published schemas in `schemas/cultnet/`.
|
published schemas in `schemas/cultnet/`.
|
||||||
- `data/corpus/corpus.cc`, `data/entity/entity.cc` and `data/interlingua/inventory.cc`: the English
|
- `data/corpus/corpus.cc`, `data/entity/entity.cc` and `data/interlingua/inventory.cc`: the English
|
||||||
corpus, its entities and the inventory, read with `weksa corpus dump`,
|
corpus, its entities and the inventory, read with `weksa corpus dump`,
|
||||||
`weksa entity dump` and `weksa inventory dump`. `data/original/inventory.cc`
|
`weksa entity dump` and `weksa inventory dump`.
|
||||||
carries its own Validation-informed lexicon revisions (5 and 6) for lowering
|
|
||||||
measurements; scored runs use `data/interlingua`.
|
|
||||||
- [Lowering target](./docs/targets/interlingua-lowering.md) and
|
- [Lowering target](./docs/targets/interlingua-lowering.md) and
|
||||||
[its map](./docs/targets/interlingua-lowering-map.md): the design, scoring
|
[its map](./docs/targets/interlingua-lowering-map.md): the design, scoring
|
||||||
and cut sequence.
|
and cut sequence.
|
||||||
- `data/`: the committed stores (corpora, entities, inventory, language ontology) and research data; there are no target-language profiles.
|
- `data/`: the cultural ontology and target-language profile data.
|
||||||
- `settings/`: setting-specific proving grounds for language projects.
|
- `settings/`: setting-specific proving grounds for language projects.
|
||||||
|
|
||||||
**Live versus planned.** Live: docs, fixtures, examples, ontology data and the
|
**Live versus planned.** Live: docs, fixtures, examples, ontology data and the
|
||||||
packet crate `crates/weksa`, including `weksa pronounce` (the ruled IPA and phones
|
packet crate `crates/weksa`, including the scoring library. Planned: lowering,
|
||||||
of the entities a packet cites), `weksa score`, and the golf loop's round and edit-accept
|
the `weksa score` subcommand, and the golf loop that scores lowered English against an expressive corpus. No
|
||||||
cycle (`weksa loop open|round|cycle`), which encodes and lowers through the
|
|
||||||
model (a prompted Claude subagent, per staged-distill).
|
|
||||||
Planned: Score and lowering documents and the other panel languages. No
|
|
||||||
daemon runs; the earlier Node daemon is parked at tag `parked/weksa-node-body`.
|
daemon runs; the earlier Node daemon is parked at tag `parked/weksa-node-body`.
|
||||||
|
|
||||||
## Ownership Model
|
## Ownership Model
|
||||||
|
|
@ -85,17 +79,9 @@ English, Brazilian Portuguese, Japanese, and later target-language dialogue.
|
||||||
[Flavored English Lowering](./docs/flavored-english-lowering.md) remains the
|
[Flavored English Lowering](./docs/flavored-english-lowering.md) remains the
|
||||||
target-specific English receipt.
|
target-specific English receipt.
|
||||||
|
|
||||||
The language ontology is `data/ontology/ontology.cc` (read it with
|
See [Target-Language Cultural Ontology Profiles](./data/target-language-ontology/README.md)
|
||||||
`weksa ontology dump`): per language, the lexicon, grammar and discourse choices a lowerer
|
for the reviewable address, register, politeness, idiom, metaphor, and script
|
||||||
reads for a stance, a discourse and a scene relation. Only an entry with a Scene condition names an
|
distinctions that target lowerers must inspect before producing output.
|
||||||
address or politeness level; the relation, from `data/original/speaker-state.cc`, picks it. Its
|
|
||||||
en-US and ja-JP entries and the keeper scenes are model-drafted or model-annotated, never Test evidence. Nothing yet grows them from accepted
|
|
||||||
rounds; until that mint exists they change only by a new `weksa ontology admit`. ja-JP holds Scene entries `onto:ja-JP:10` to `:13`. It holds none for an above-status out-group
|
|
||||||
addressee, and none for an above-status in-group addressee below intimacy 700 (`:12` still selects
|
|
||||||
an in-group addressee at intimacy 700 or more, whatever the status). Two entries for the above-status
|
|
||||||
cases were withdrawn: a paired measure found they made the lowerer match the reference register less
|
|
||||||
often, and they stay out until a native reader attests them.
|
|
||||||
ja-JP `:5` (hedging), `:7` (affection) and `:8` (restrained sadness) carry certainty and restraint conditions, through `AtMostOrUnset` bounds (an unrated certainty or restraint does not defeat them), so the hedging, certainty, affection, restraint and sadness entries never co-select on the pairs `:5`/`:6`, `:4`/`:7`, `:6`/`:7` and `:6`/`:8`; ja-JP `:3` (anger) carries `AtMostOrUnset` hedge and affection bounds, as en-US `:3`, so `:3`/`:5` and `:3`/`:7` never co-select; the store has no retire, so that was a rebuild of `data/ontology/ontology.cc` with the ja-JP text unchanged. en-US `:3` (blunt anger) carries `AtMostOrUnset` hedge and affection bounds, and `:5` (hedging) and `:8` (affection) each an `AtMostOrUnset` certainty bound (as ja-JP `:7`), so `:3`/`:5`, `:3`/`:8`, `:5`/`:6` and `:6`/`:8` never co-select; `:5`'s certainty clause is that condition and no longer prose; `:6` and the Scene entry `:10` still co-select and are kept apart by scoping in the text. ja-JP `:9` (surprise) states its question ending as available, not prescribed, so `:6`/`:9`, `:5`/`:9` and `:3`/`:9` stay co-selectable and are reconciled by wording, not by condition.
|
|
||||||
|
|
||||||
For the first AquaSynth speech-training handoff, see
|
For the first AquaSynth speech-training handoff, see
|
||||||
[Tiny PanPhon v0.1 Speech Curriculum](./examples/speech-training/tiny-panphon-v0.1/README.md).
|
[Tiny PanPhon v0.1 Speech Curriculum](./examples/speech-training/tiny-panphon-v0.1/README.md).
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,14 +0,0 @@
|
||||||
//! Pins for private reach of `context.rs` that no run can produce.
|
|
||||||
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
/// A corpus with no live Train item of the panel's languages holds an empty
|
|
||||||
/// train. (A run refuses a panel without Train items at open, so only the
|
|
||||||
/// builder shows it.)
|
|
||||||
#[test]
|
|
||||||
fn no_train_item_of_the_panel_is_an_empty_train() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let corpus = CorpusStore::open(&dir.path().join("c.cc")).unwrap();
|
|
||||||
let train = TrainItems::from_corpus(&corpus, &["en-US".to_string()]).unwrap();
|
|
||||||
assert_eq!((train.len(), train.is_empty()), (0, true));
|
|
||||||
}
|
|
||||||
|
|
@ -8,7 +8,6 @@ use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
use crate::form::{FormError, is_slug, string_form};
|
use crate::form::{FormError, is_slug, string_form};
|
||||||
use crate::packet::EntityId;
|
use crate::packet::EntityId;
|
||||||
use crate::speaker::Annotator;
|
|
||||||
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||||
pub struct Turn {
|
pub struct Turn {
|
||||||
|
|
@ -33,7 +32,7 @@ pub enum Source {
|
||||||
/// A public-domain work, with everything a reader needs to check the claim:
|
/// A public-domain work, with everything a reader needs to check the claim:
|
||||||
/// the edition's licence and the jurisdiction the public-domain status is
|
/// the edition's licence and the jurisdiction the public-domain status is
|
||||||
/// asserted in, who translated it (the translation is the text's copyright,
|
/// asserted in, who translated it (the translation is the text's copyright,
|
||||||
/// not the original's) and when the work or translation first appeared. A work missing any
|
/// not the original's) and when the edition appeared. A work missing any
|
||||||
/// is refused at admission.
|
/// is refused at admission.
|
||||||
PublicDomain {
|
PublicDomain {
|
||||||
work: String,
|
work: String,
|
||||||
|
|
@ -42,16 +41,9 @@ pub enum Source {
|
||||||
licence: String,
|
licence: String,
|
||||||
jurisdiction: String,
|
jurisdiction: String,
|
||||||
translator: String,
|
translator: String,
|
||||||
/// The year the work, or for a translation this translation, first
|
|
||||||
/// appeared. It is not the year of the `edition_url` file's source
|
|
||||||
/// print: Project Gutenberg's text may be a later or modernised
|
|
||||||
/// edition, and the claim rests on the date of first appearance.
|
|
||||||
edition_year: u16,
|
edition_year: u16,
|
||||||
},
|
},
|
||||||
/// Labelled model output; never in the test split. The rule that it is
|
/// Labelled model output; never in the test split, never judged by its own family.
|
||||||
/// never judged by its own family binds model judges and the split; a
|
|
||||||
/// deterministic scorer refuses nothing, so the loop labels (the round
|
|
||||||
/// counts items of the answerer's family) and does not forbid.
|
|
||||||
/// `family` is `Unrecorded` when nothing records one. `generator` is the
|
/// `family` is `Unrecorded` when nothing records one. `generator` is the
|
||||||
/// string the fixture or run wrote, kept verbatim.
|
/// string the fixture or run wrote, kept verbatim.
|
||||||
ModelWritten {
|
ModelWritten {
|
||||||
|
|
@ -62,7 +54,7 @@ pub enum Source {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The family of the model that wrote a line.
|
/// The family of the model that wrote a line.
|
||||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||||
pub enum ModelFamily {
|
pub enum ModelFamily {
|
||||||
Anthropic,
|
Anthropic,
|
||||||
OpenAi,
|
OpenAi,
|
||||||
|
|
@ -71,7 +63,6 @@ pub enum ModelFamily {
|
||||||
Meta,
|
Meta,
|
||||||
Mistral,
|
Mistral,
|
||||||
/// Nothing records which family wrote the line.
|
/// Nothing records which family wrote the line.
|
||||||
#[default]
|
|
||||||
Unrecorded,
|
Unrecorded,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -104,11 +95,6 @@ pub struct CorpusItem {
|
||||||
/// Which entities the text names, and where. Fixed with the text at intake.
|
/// Which entities the text names, and where. Fixed with the text at intake.
|
||||||
#[cultcache(key = 10)]
|
#[cultcache(key = 10)]
|
||||||
pub names: Vec<NameSpan>,
|
pub names: Vec<NameSpan>,
|
||||||
/// Who placed `names`: the annotator of the intake that wrote them. `None`
|
|
||||||
/// when nothing records one (the cast seeds, `weksa corpus add`, the Dao De
|
|
||||||
/// Jing bed); it labels the spans, never the line or its speaker.
|
|
||||||
#[cultcache(key = 11, default)]
|
|
||||||
pub names_by: Option<Annotator>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl CorpusItem {
|
impl CorpusItem {
|
||||||
|
|
|
||||||
|
|
@ -101,14 +101,6 @@ fn hex8(ascii: &str) -> String {
|
||||||
ascii.bytes().map(|b| format!("{b:02x}")).collect()
|
ascii.bytes().map(|b| format!("{b:02x}")).collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `(node, concept gloss, frame gloss, [(role id, filler node)])`.
|
|
||||||
type PredicationSpec = (
|
|
||||||
&'static str,
|
|
||||||
&'static str,
|
|
||||||
&'static str,
|
|
||||||
&'static [(&'static str, &'static str)],
|
|
||||||
);
|
|
||||||
|
|
||||||
struct Spec {
|
struct Spec {
|
||||||
/// Eight ASCII characters; the packet id is their hex.
|
/// Eight ASCII characters; the packet id is their hex.
|
||||||
id: &'static str,
|
id: &'static str,
|
||||||
|
|
@ -116,7 +108,13 @@ struct Spec {
|
||||||
speech_act: &'static str,
|
speech_act: &'static str,
|
||||||
/// `(node, referent kind id, concept gloss)`.
|
/// `(node, referent kind id, concept gloss)`.
|
||||||
referents: &'static [(&'static str, &'static str, &'static str)],
|
referents: &'static [(&'static str, &'static str, &'static str)],
|
||||||
predications: &'static [PredicationSpec],
|
/// `(node, concept gloss, frame gloss, [(role id, filler node)])`.
|
||||||
|
predications: &'static [(
|
||||||
|
&'static str,
|
||||||
|
&'static str,
|
||||||
|
&'static str,
|
||||||
|
&'static [(&'static str, &'static str)],
|
||||||
|
)],
|
||||||
/// The author's reading of the speaker's felt state: `(feature gloss, permille)`.
|
/// The author's reading of the speaker's felt state: `(feature gloss, permille)`.
|
||||||
stance: &'static [(&'static str, u16)],
|
stance: &'static [(&'static str, u16)],
|
||||||
note: &'static str,
|
note: &'static str,
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,13 +0,0 @@
|
||||||
//! Pins for `unmark` that the import door only reaches through a whole work.
|
|
||||||
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
/// A form that opens another markup is malformed, though its entity id is
|
|
||||||
/// sound: the model wrote a nested or unclosed mark, not a name.
|
|
||||||
#[test]
|
|
||||||
fn a_form_that_opens_a_mark_is_malformed() {
|
|
||||||
assert_eq!(unmark("[[entity:orsa|Or[[sa]] came"), None);
|
|
||||||
// The same line without the nested opener unmarks.
|
|
||||||
let (plain, names) = unmark("[[entity:orsa|Orsa]] came").unwrap();
|
|
||||||
assert_eq!((plain.as_str(), names.len()), ("Orsa came", 1));
|
|
||||||
}
|
|
||||||
|
|
@ -91,73 +91,20 @@ pub fn is_calendar_date(s: &str) -> bool {
|
||||||
(1..=days).contains(&day)
|
(1..=days).contains(&day)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A BCP 47 language tag in its canonical case (RFC 5646 section 2.1.1): 1 to 35
|
/// A BCP 47-form language tag: 1 to 35 bytes of ASCII alphanumerics in
|
||||||
/// bytes of ASCII alphanumerics in hyphen-separated subtags of 1 to 8 bytes; the
|
/// hyphen-separated subtags of 1 to 8 bytes; the first is 2 to 8 letters, or the
|
||||||
/// first is 2 to 8 letters, or the single letter `x` of a private-use tag. The
|
/// single letter `x` of a private-use tag. Form only: no registry is consulted.
|
||||||
/// language subtag is lowercase, a four-letter script subtag is Titlecase, a
|
|
||||||
/// two-letter region subtag is UPPERCASE, and every other subtag, and every
|
|
||||||
/// subtag after a singleton such as `x`, is lowercase. Form only: no registry is
|
|
||||||
/// consulted. Selection and comparison are by exact tag, so a case variant would
|
|
||||||
/// be a different, dead language; this is the one tag check of corpus items,
|
|
||||||
/// entities, ontology entries and the run panel.
|
|
||||||
pub(crate) fn is_language_tag(s: &str) -> bool {
|
pub(crate) fn is_language_tag(s: &str) -> bool {
|
||||||
if s.len() > 35 {
|
if s.len() > 35 {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
let mut subtags = s.split('-');
|
let mut subtags = s.split('-');
|
||||||
let first = subtags.next().unwrap_or("");
|
let first = subtags.next().unwrap_or("");
|
||||||
let first_ok = first.bytes().all(|b| b.is_ascii_lowercase())
|
let first_ok = first.bytes().all(|b| b.is_ascii_alphabetic())
|
||||||
&& ((2..=8).contains(&first.len()) || first == "x");
|
&& ((2..=8).contains(&first.len()) || first == "x");
|
||||||
if !first_ok {
|
first_ok
|
||||||
return false;
|
&& subtags
|
||||||
}
|
.all(|t| (1..=8).contains(&t.len()) && t.bytes().all(|b| b.is_ascii_alphanumeric()))
|
||||||
// `private`: inside the `x` private-use section (subtags 1 to 8, lowercase).
|
|
||||||
// `ext`: inside any singleton section, where a one-byte subtag starts the next.
|
|
||||||
// `pending`: the last subtag was a singleton and nothing has followed it yet.
|
|
||||||
let mut private = first == "x";
|
|
||||||
let mut ext = private;
|
|
||||||
let mut pending = private;
|
|
||||||
for t in subtags {
|
|
||||||
if !(1..=8).contains(&t.len()) || !t.bytes().all(|b| b.is_ascii_alphanumeric()) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
let lower = t.bytes().all(|b| !b.is_ascii_uppercase());
|
|
||||||
if private {
|
|
||||||
if !lower {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
pending = false;
|
|
||||||
} else if t.len() == 1 {
|
|
||||||
// A singleton: lowercase, and not directly after another singleton.
|
|
||||||
if !lower || pending {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
ext = true;
|
|
||||||
pending = true;
|
|
||||||
private = t == "x";
|
|
||||||
} else if ext {
|
|
||||||
// Extension subtags are 2 to 8 lowercase.
|
|
||||||
if !lower {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
pending = false;
|
|
||||||
} else {
|
|
||||||
let alpha = t.bytes().all(|b| b.is_ascii_alphabetic());
|
|
||||||
let canonical = if t.len() == 2 {
|
|
||||||
// A region: two letters. Two digits are not a subtag at all.
|
|
||||||
alpha && t.bytes().all(|b| b.is_ascii_uppercase())
|
|
||||||
} else if alpha && t.len() == 4 {
|
|
||||||
t.as_bytes()[0].is_ascii_uppercase()
|
|
||||||
&& t.bytes().skip(1).all(|b| b.is_ascii_lowercase())
|
|
||||||
} else {
|
|
||||||
lower
|
|
||||||
};
|
|
||||||
if !canonical {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
!pending
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A slug: `[a-z0-9]` and inner `-`, 1 to 48 bytes, no edge `-`.
|
/// A slug: `[a-z0-9]` and inner `-`, 1 to 48 bytes, no edge `-`.
|
||||||
|
|
@ -267,25 +214,6 @@ mod tests {
|
||||||
assert!(!is_format('\u{200c}') && !is_format('\u{200d}'));
|
assert!(!is_format('\u{200c}') && !is_format('\u{200d}'));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_language_tag_is_in_canonical_case() {
|
|
||||||
for good in [
|
|
||||||
"en", "en-US", "ja-JP", "zh-Hant-TW", "sr-Latn", "es-419", "de-1996", "en-CA-x-ca",
|
|
||||||
"x-private", "x-ab", "x-a", "x-a-b", "en-x-a-b", "en-u-nu-x-a-b", "en-u-nu-latn", "ja-JP-x-a1",
|
|
||||||
] {
|
|
||||||
assert!(is_language_tag(good), "{good}");
|
|
||||||
}
|
|
||||||
for bad in [
|
|
||||||
"", "e", "EN", "En", "en-us", "en-Us", "en-uS", "ja-jp", "JA-JP", "zh-hant-TW",
|
|
||||||
"zh-HANT-TW", "zh-hAnt-TW", "zh-Hant-tw", "en-CA-x-CA", "en-CA-x-Ca", "de-1996-X",
|
|
||||||
"es-419-LATN", "en--US", "en-", "-en", "en_US", "abcdefghi", "en-abcdefghi",
|
|
||||||
"xx-\u{e9}", "x", "en-x", "en-u", "en-US-x", "en-0", "en-12", "en-u-x", "en-u-0",
|
|
||||||
"x-AB", "x-Ab", "en-x-AB", "en-a-b-ccc", "en-u-NU", "en-U-nu", "en-a-b",
|
|
||||||
] {
|
|
||||||
assert!(!is_language_tag(bad), "{bad}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn an_attestation_is_bounded_readable_text() {
|
fn an_attestation_is_bounded_readable_text() {
|
||||||
assert!(is_attestation("a"));
|
assert!(is_attestation("a"));
|
||||||
|
|
|
||||||
|
|
@ -85,11 +85,6 @@ impl InvId {
|
||||||
pub fn family(self) -> Family {
|
pub fn family(self) -> Family {
|
||||||
self.family
|
self.family
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The `n` of `<family prefix>:<n>`.
|
|
||||||
pub fn n(self) -> NonZeroU32 {
|
|
||||||
self.n
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl fmt::Display for InvId {
|
impl fmt::Display for InvId {
|
||||||
|
|
@ -194,11 +189,9 @@ pub enum Slot {
|
||||||
Predication,
|
Predication,
|
||||||
Relation,
|
Relation,
|
||||||
Stance,
|
Stance,
|
||||||
/// The speaker's relation to one party of the scene, held by `weksa.scene`
|
/// The relation of the speaker to the addressee in the scene; see
|
||||||
/// documents; see `LoweringInput::with_scene`.
|
/// `LoweringInput::with_scene`.
|
||||||
Scene,
|
Scene,
|
||||||
/// The speaker's stable axes, held by `weksa.speaker` documents.
|
|
||||||
Speaker,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||||
|
|
@ -235,21 +228,10 @@ impl InventoryEntry {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Whether the entry's shape matches its family: features carry a value kind
|
/// Whether the entry's shape matches its family: features carry a value kind
|
||||||
/// and slots, extensions a value kind and no slots, nothing else either. A
|
/// and slots, extensions a value kind and no slots, nothing else either.
|
||||||
/// feature of slot `Stance`, `Speaker` or `Scene` sits in that slot alone: each
|
|
||||||
/// speaker-state layer has one writer and one lifetime, so its features sit
|
|
||||||
/// nowhere else.
|
|
||||||
pub fn well_formed(&self) -> bool {
|
pub fn well_formed(&self) -> bool {
|
||||||
match self.family() {
|
match self.family() {
|
||||||
Family::Feature => {
|
Family::Feature => self.value_kind.is_some() && !self.applies_to.is_empty(),
|
||||||
self.value_kind.is_some()
|
|
||||||
&& !self.applies_to.is_empty()
|
|
||||||
&& (self.applies_to.len() == 1
|
|
||||||
|| !self
|
|
||||||
.applies_to
|
|
||||||
.iter()
|
|
||||||
.any(|s| matches!(s, Slot::Stance | Slot::Speaker | Slot::Scene)))
|
|
||||||
}
|
|
||||||
Family::Extension => self.value_kind.is_some() && self.applies_to.is_empty(),
|
Family::Extension => self.value_kind.is_some() && self.applies_to.is_empty(),
|
||||||
_ => self.value_kind.is_none() && self.applies_to.is_empty(),
|
_ => self.value_kind.is_none() && self.applies_to.is_empty(),
|
||||||
}
|
}
|
||||||
|
|
@ -541,49 +523,28 @@ pub fn proof_revision() -> (InventoryRevision, Vec<InventoryEntry>) {
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Revision `n`: the speaker-state vocabulary. First the Scene features, the
|
/// Revision 3: the speaker's relation to the addressee, features of slot
|
||||||
/// speaker's relation to one party of a scene (slot `Scene`): `relative_status`
|
/// `Scene`. Hand-seeded like [`proof_revision`] until the loop's minting round
|
||||||
/// is the speaker's standing against the addressee's, not a register word, what
|
/// and cut `speaker-scene` own the Scene vocabulary. `relative_status` is the
|
||||||
/// it selects is each language's ontology's business. Then the Speaker features
|
/// speaker's standing against the addressee's, not a register word:
|
||||||
/// (slot `Speaker`): `socioeconomic_standing` is standing in the speaker's own
|
/// what it selects is each language's ontology's business. It mints the three
|
||||||
/// society, 0 lowest; `education` is formal schooling against that society;
|
/// Scene features after the seed's features.
|
||||||
/// `age_cohort` is a life stage. Origin and community are not features; they
|
pub fn scene_revision() -> (InventoryRevision, Vec<InventoryEntry>) {
|
||||||
/// arrive as community ids with cut `language-ontology`. Hand-seeded like
|
|
||||||
/// [`proof_revision`] until the loop's minting round owns the vocabulary; the
|
|
||||||
/// features take the ids after the seed's.
|
|
||||||
pub fn speaker_scene_revision(n: u32) -> (InventoryRevision, Vec<InventoryEntry>) {
|
|
||||||
let taken = seed()
|
let taken = seed()
|
||||||
.1
|
.1
|
||||||
.iter()
|
.iter()
|
||||||
.filter(|e| e.family() == Family::Feature)
|
.filter(|e| e.family() == Family::Feature)
|
||||||
.count() as u32;
|
.count() as u32;
|
||||||
let options = |labels: &[&str]| {
|
let labels = ["below", "level", "above"]
|
||||||
ValueKind::Options(
|
.map(|l| OptionLabel::parse(l).expect("seed labels are well formed"));
|
||||||
OptionSet::new(
|
let relative_status = ValueKind::Options(
|
||||||
labels
|
OptionSet::new(labels.to_vec()).expect("seed option sets are well formed"),
|
||||||
.iter()
|
);
|
||||||
.map(|l| OptionLabel::parse(l).expect("seed labels are well formed"))
|
|
||||||
.collect(),
|
|
||||||
)
|
|
||||||
.expect("seed option sets are well formed"),
|
|
||||||
)
|
|
||||||
};
|
|
||||||
let mut entries = Vec::new();
|
let mut entries = Vec::new();
|
||||||
for (i, (gloss, kind, slot)) in [
|
for (i, (gloss, kind)) in [
|
||||||
(
|
("relative_status", relative_status),
|
||||||
"relative_status",
|
("intimacy", ValueKind::Grade),
|
||||||
options(&["below", "level", "above"]),
|
("in_group", ValueKind::Flag),
|
||||||
Slot::Scene,
|
|
||||||
),
|
|
||||||
("intimacy", ValueKind::Grade, Slot::Scene),
|
|
||||||
("in_group", ValueKind::Flag, Slot::Scene),
|
|
||||||
("socioeconomic_standing", ValueKind::Grade, Slot::Speaker),
|
|
||||||
("education", ValueKind::Grade, Slot::Speaker),
|
|
||||||
(
|
|
||||||
"age_cohort",
|
|
||||||
options(&["child", "adolescent", "young_adult", "adult", "elder"]),
|
|
||||||
Slot::Speaker,
|
|
||||||
),
|
|
||||||
]
|
]
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.enumerate()
|
.enumerate()
|
||||||
|
|
@ -592,16 +553,16 @@ pub fn speaker_scene_revision(n: u32) -> (InventoryRevision, Vec<InventoryEntry>
|
||||||
id: Family::Feature.id(NonZeroU32::MIN.saturating_add(taken + i as u32)),
|
id: Family::Feature.id(NonZeroU32::MIN.saturating_add(taken + i as u32)),
|
||||||
gloss: gloss.to_string(),
|
gloss: gloss.to_string(),
|
||||||
value_kind: Some(kind),
|
value_kind: Some(kind),
|
||||||
applies_to: vec![slot],
|
applies_to: vec![Slot::Scene],
|
||||||
introduced_rev: n,
|
introduced_rev: 3,
|
||||||
retired_rev: None,
|
retired_rev: None,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
let changes = entries.iter().map(|e| Change::Add(e.id)).collect();
|
let changes = entries.iter().map(|e| Change::Add(e.id)).collect();
|
||||||
(
|
(
|
||||||
InventoryRevision {
|
InventoryRevision {
|
||||||
n,
|
n: 3,
|
||||||
parent: Some(n - 1),
|
parent: Some(2),
|
||||||
changes,
|
changes,
|
||||||
frozen: false,
|
frozen: false,
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -6,44 +6,34 @@
|
||||||
pub mod context;
|
pub mod context;
|
||||||
pub mod corpus;
|
pub mod corpus;
|
||||||
pub mod daodejing;
|
pub mod daodejing;
|
||||||
pub mod encode;
|
|
||||||
pub mod entity;
|
pub mod entity;
|
||||||
pub mod form;
|
pub mod form;
|
||||||
pub mod inventory;
|
pub mod inventory;
|
||||||
pub mod lower;
|
pub mod lower;
|
||||||
pub mod model;
|
|
||||||
pub mod ontology;
|
|
||||||
pub mod packet;
|
pub mod packet;
|
||||||
pub mod phones;
|
pub mod phones;
|
||||||
pub mod run;
|
|
||||||
pub mod schema;
|
pub mod schema;
|
||||||
pub mod score;
|
pub mod score;
|
||||||
pub mod speaker;
|
|
||||||
pub mod split;
|
pub mod split;
|
||||||
pub mod store;
|
pub mod store;
|
||||||
pub mod validate;
|
pub mod validate;
|
||||||
|
|
||||||
pub use corpus::{CorpusItem, CorpusItemId, NameSpan, Source, Turn};
|
pub use corpus::{CorpusItem, CorpusItemId, NameSpan, Source, Turn};
|
||||||
pub use encode::{EditDraft, LoweringDraft, PacketDraft};
|
|
||||||
pub use entity::{Entity, EntityOrigin, LanguageForms, Pronunciation};
|
pub use entity::{Entity, EntityOrigin, LanguageForms, Pronunciation};
|
||||||
pub use form::FormError;
|
pub use form::FormError;
|
||||||
pub use inventory::{
|
pub use inventory::{
|
||||||
Change, Family, InvId, Inventory, InventoryEntry, InventoryRevision, OptionLabel, OptionSet,
|
Change, Family, InvId, Inventory, InventoryEntry, InventoryRevision, OptionLabel, OptionSet,
|
||||||
Slot, ValueKind,
|
Slot, ValueKind,
|
||||||
};
|
};
|
||||||
pub use lower::{EntityForms, LowerError, PhoneLine, TARGET_LANGUAGE, phones, pronounce};
|
pub use lower::{EntityForms, LowerError, PhoneLine, TARGET_LANGUAGE, lower, phones, pronounce};
|
||||||
pub use model::{AnswerFiles, ModelPort};
|
pub use phones::{Phone, PhoneInventory, SegmentError, Stress};
|
||||||
pub use ontology::{Condition, Layer, Ontology, OntologyEntry, Test, select};
|
|
||||||
pub use packet::{
|
pub use packet::{
|
||||||
EncoderId, EntityId, Grade, NodeId, Packet, PacketId, PacketKind, SCHEMA_VERSION, Stance,
|
EncoderId, EntityId, Grade, NodeId, Packet, PacketId, PacketKind, SCHEMA_VERSION, Stance,
|
||||||
};
|
};
|
||||||
pub use phones::{Phone, PhoneInventory, SegmentError, Stress};
|
|
||||||
pub use run::{Decision, RoundRecord, RunStores, cycle, round};
|
|
||||||
pub use score::{Embedder, ItemScore, OllamaEmbedder, Scorer, chrf};
|
pub use score::{Embedder, ItemScore, OllamaEmbedder, Scorer, chrf};
|
||||||
pub use speaker::{Annotator, PairRelation, Party, Scene, Speaker};
|
|
||||||
pub use split::{Split, SplitScheme};
|
pub use split::{Split, SplitScheme};
|
||||||
pub use store::{
|
pub use store::{
|
||||||
AddOutcome, CorpusStore, EntityBound, EntityStore, InventoryStore, OntologyStore, PacketStore,
|
AddOutcome, CorpusStore, EntityBound, EntityStore, InventoryStore, PacketStore, StoreError,
|
||||||
SpeakerStateStore, StoreError, StoreRefusal,
|
StoreRefusal,
|
||||||
};
|
};
|
||||||
pub use validate::{LoweringInput, Refusal, SceneRead, validate};
|
pub use validate::{LoweringInput, Refusal, validate};
|
||||||
|
|
|
||||||
|
|
@ -1,27 +1,46 @@
|
||||||
//! The entity port and the ruled pronunciation of a packet's entities.
|
//! The English lowerer: the validated lowering view of a packet to text, by a
|
||||||
|
//! fixed table keyed by inventory ids. It renders predications, referents and the
|
||||||
|
//! speech act, and nothing else: no style, no model, no corpus text. It takes a
|
||||||
|
//! [`LoweringInput`], so it cannot see the trace or the corpus item. A packet
|
||||||
|
//! carrying a feature, relation, extension, role fill or referent it does not
|
||||||
|
//! render is refused, because rendering part of a meaning would assert a different
|
||||||
|
//! one (`polarity: negative` dropped says the opposite). Context, kind and
|
||||||
|
//! constraints steer delivery and are not rendered; the concept of a predication
|
||||||
|
//! must resolve but the frame owns the verb.
|
||||||
//!
|
//!
|
||||||
//! Wording of a packet in a target language is the `ModelPort`'s answer to
|
//! A referent that cites an entity renders as that entity's own form, read
|
||||||
//! `Prompt::lower` (`context.rs`); nothing here produces lowered text. This
|
//! through the [`EntityForms`] port: the first (citation) form stored for
|
||||||
//! module holds what the entity store owns and the lowering path reads: the
|
//! [`TARGET_LANGUAGE`], verbatim. The choice is a table lookup on the language
|
||||||
//! narrow [`EntityForms`] port, the ruled IPA of every entity a packet cites
|
//! tag and the form's position, not a reading of the form. Forms after the
|
||||||
//! ([`pronounce`]), and its segmentation into phones ([`phones`]). A referent
|
//! first are inflected or particle-bound shapes; no English frame asks for one,
|
||||||
//! that cites an entity is read through the port; an entity the port does not
|
//! so none is selected. An entity the port does not hold, or one with no form in
|
||||||
//! hold, one with no form or no ruled pronunciation in [`TARGET_LANGUAGE`], is
|
//! the target language, is refused: the concept word would say a different
|
||||||
//! refused, because spelling is not a pronunciation and there is no
|
//! thing than the name the packet cites.
|
||||||
//! grapheme-to-phoneme guess to offer in its place.
|
//!
|
||||||
|
//! A request clause ("You asked Nibu to stop.") says what the addressee did, so
|
||||||
|
//! it is gated on the packet's own record of it: `Discourse.addressee_consent`
|
||||||
|
//! (set only by authoring or game telemetry, V15) must name that predication,
|
||||||
|
//! and its agent must be the addressee. Without it the lowerer refuses
|
||||||
|
//! ([`LowerError::ConsentMissing`], [`LowerError::AskerNotAddressee`]); it does
|
||||||
|
//! not read wording or guess who asked. A consent naming any other predication
|
||||||
|
//! is an unrendered slot.
|
||||||
|
//!
|
||||||
|
//! The table is a temporary owner. Each row speaks for an inventory entry and
|
||||||
|
//! names the gloss it speaks for, so a row cannot outlive a change of what its id
|
||||||
|
//! means. `language-ontology` moves the rows into ontology documents.
|
||||||
|
|
||||||
use std::fmt;
|
use std::fmt;
|
||||||
|
|
||||||
use serde::Serialize;
|
use serde::Serialize;
|
||||||
|
|
||||||
use crate::entity::Entity;
|
use crate::entity::Entity;
|
||||||
use crate::packet::EntityId;
|
use crate::inventory::{Family, InvId, Inventory};
|
||||||
|
use crate::packet::{EntityId, NodeId, Predication};
|
||||||
use crate::phones::{Phone, PhoneInventory, SegmentError};
|
use crate::phones::{Phone, PhoneInventory, SegmentError};
|
||||||
use crate::store::EntityStore;
|
use crate::store::EntityStore;
|
||||||
use crate::validate::LoweringInput;
|
use crate::validate::LoweringInput;
|
||||||
|
|
||||||
/// The language tag of [`pronounce`] and [`phones`] only. The golf loop lowers
|
/// The language tag this lowerer's entity forms are read under, spelled as an
|
||||||
/// each item in its own language and does not read this. Spelled as an
|
|
||||||
/// entity's `LanguageForms::language` and a corpus item's language are.
|
/// entity's `LanguageForms::language` and a corpus item's language are.
|
||||||
pub const TARGET_LANGUAGE: &str = "en-US";
|
pub const TARGET_LANGUAGE: &str = "en-US";
|
||||||
|
|
||||||
|
|
@ -38,8 +57,98 @@ impl EntityForms for EntityStore {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `(inventory id, the gloss the row speaks for, surface)`.
|
||||||
|
type Row = (&'static str, &'static str, &'static str);
|
||||||
|
|
||||||
|
/// Referent concepts, as the noun phrase after "the".
|
||||||
|
const CONCEPTS: &[Row] = &[
|
||||||
|
("concept:3", "player", "player"),
|
||||||
|
("concept:4", "control", "control"),
|
||||||
|
("concept:5", "ship_mind", "ship mind"),
|
||||||
|
// The Dao De Jing test bed (`daodejing.rs`). Nouns, then the verbs a
|
||||||
|
// `{verb}` frame speaks.
|
||||||
|
("concept:7", "sage", "sage"),
|
||||||
|
("concept:8", "mind", "minds"),
|
||||||
|
("concept:9", "belly", "bellies"),
|
||||||
|
("concept:10", "will", "wills"),
|
||||||
|
("concept:11", "bone", "bones"),
|
||||||
|
("concept:12", "empty", "empties"),
|
||||||
|
("concept:13", "fill", "fills"),
|
||||||
|
("concept:14", "weaken", "weakens"),
|
||||||
|
("concept:15", "strengthen", "strengthens"),
|
||||||
|
("concept:16", "heaven", "heaven"),
|
||||||
|
("concept:17", "earth", "earth"),
|
||||||
|
("concept:18", "is_long_enduring", "is long-enduring"),
|
||||||
|
("concept:19", "continues_long", "continues long"),
|
||||||
|
("concept:20", "highest_excellence", "highest excellence"),
|
||||||
|
("concept:21", "water", "water"),
|
||||||
|
("concept:22", "true_words", "words that are strictly true"),
|
||||||
|
("concept:23", "seem_paradoxical", "seem paradoxical"),
|
||||||
|
(
|
||||||
|
"concept:24",
|
||||||
|
"space_between_heaven_and_earth",
|
||||||
|
"space between heaven and earth",
|
||||||
|
),
|
||||||
|
("concept:25", "bellows", "bellows"),
|
||||||
|
("concept:26", "compare", "compare"),
|
||||||
|
("concept:27", "resemble", "resemble"),
|
||||||
|
];
|
||||||
|
|
||||||
|
/// Frames: the clause a predication of that frame says. `{gloss}` is the filler
|
||||||
|
/// of the role whose inventory gloss it is.
|
||||||
|
const FRAMES: &[Row] = &[
|
||||||
|
(
|
||||||
|
"frame:1",
|
||||||
|
"hazard_warning",
|
||||||
|
"{agent} may endanger {patient} by touching {instrument}",
|
||||||
|
),
|
||||||
|
("frame:2", "intervention", "{agent} will stop {patient}"),
|
||||||
|
(
|
||||||
|
"frame:3",
|
||||||
|
"request_intervention",
|
||||||
|
"{agent} asked {patient} to stop",
|
||||||
|
),
|
||||||
|
// The Dao De Jing test bed. Frame 4 was written from a Train chapter, 5 and
|
||||||
|
// 6 from Validation chapters, 7 and 8 from Test chapters.
|
||||||
|
("frame:4", "act", "{agent} {verb} {patient}"),
|
||||||
|
("frame:5", "intransitive", "{experiencer} {verb}"),
|
||||||
|
("frame:6", "likeness", "{experiencer} is like {goal}"),
|
||||||
|
(
|
||||||
|
"frame:7",
|
||||||
|
"seems_paradoxical",
|
||||||
|
"{experiencer} seem paradoxical",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"frame:8",
|
||||||
|
"may_be_compared",
|
||||||
|
"{experiencer} may be compared to {goal}",
|
||||||
|
),
|
||||||
|
];
|
||||||
|
|
||||||
|
/// The frame of "{agent} asked {patient} to stop": the one frame that reads
|
||||||
|
/// the discourse's addressee consent.
|
||||||
|
const REQUEST_FRAME: &str = "frame:3";
|
||||||
|
|
||||||
|
/// Speech acts: the opener of the first sentence.
|
||||||
|
const SPEECH_ACTS: &[Row] = &[
|
||||||
|
("speech_act:1", "assert", ""),
|
||||||
|
("speech_act:2", "ask", ""),
|
||||||
|
("speech_act:5", "warn", "Warning: "),
|
||||||
|
];
|
||||||
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
pub enum LowerError {
|
pub enum LowerError {
|
||||||
|
/// The packet carries a slot the lowerer does not render.
|
||||||
|
Unrendered { what: &'static str },
|
||||||
|
/// No table row for an id the packet uses.
|
||||||
|
NoRow { id: InvId },
|
||||||
|
/// The row's gloss is not the gloss the inventory gives its id at the
|
||||||
|
/// packet's revision, or the id does not resolve there.
|
||||||
|
GlossMismatch { id: InvId, row: &'static str },
|
||||||
|
/// A frame template names a role the inventory has no live entry for.
|
||||||
|
UnknownRole { frame: InvId, role: String },
|
||||||
|
/// The predication fills no role its frame's template needs.
|
||||||
|
MissingRole { predication: NodeId, role: String },
|
||||||
/// A referent cites an entity the entity port does not hold.
|
/// A referent cites an entity the entity port does not hold.
|
||||||
UnknownEntity { entity: EntityId },
|
UnknownEntity { entity: EntityId },
|
||||||
/// The entity has no form in [`TARGET_LANGUAGE`].
|
/// The entity has no form in [`TARGET_LANGUAGE`].
|
||||||
|
|
@ -51,7 +160,7 @@ pub enum LowerError {
|
||||||
NoEntityReferent,
|
NoEntityReferent,
|
||||||
/// Phones were requested for an entity whose ruled IPA holds a symbol the
|
/// Phones were requested for an entity whose ruled IPA holds a symbol the
|
||||||
/// language's phone inventory does not list (or the language has no
|
/// language's phone inventory does not list (or the language has no
|
||||||
/// inventory). No phone is guessed.
|
/// inventory). The lowerer does not guess a phone.
|
||||||
UnknownPhone {
|
UnknownPhone {
|
||||||
entity: EntityId,
|
entity: EntityId,
|
||||||
at: usize,
|
at: usize,
|
||||||
|
|
@ -60,18 +169,24 @@ pub enum LowerError {
|
||||||
/// Phones were requested for an entity whose ruled IPA has a stress mark
|
/// Phones were requested for an entity whose ruled IPA has a stress mark
|
||||||
/// with no vowel after it.
|
/// with no vowel after it.
|
||||||
DanglingStress { entity: EntityId, at: usize },
|
DanglingStress { entity: EntityId, at: usize },
|
||||||
|
/// A request predication the packet does not carry the addressee's consent
|
||||||
|
/// to: no consent, or a consent that names another predication. The
|
||||||
|
/// lowerer does not guess who asked.
|
||||||
|
ConsentMissing { predication: NodeId },
|
||||||
|
/// A request predication whose agent is not the discourse addressee.
|
||||||
|
AskerNotAddressee { predication: NodeId },
|
||||||
/// The entity port failed to read.
|
/// The entity port failed to read.
|
||||||
EntityRead { entity: EntityId, why: String },
|
EntityRead { entity: EntityId, why: String },
|
||||||
/// A lowering prompt was asked of a view whose scene was never read.
|
|
||||||
SceneUnread,
|
|
||||||
/// A referent cites the entity whose id is the lowering prompt's speaker
|
|
||||||
/// token ([`crate::context::SPEAKER_TOKEN`]), which would read as the speaker.
|
|
||||||
SpeakerTokenTaken,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl fmt::Display for LowerError {
|
impl fmt::Display for LowerError {
|
||||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
f.write_str(match self {
|
f.write_str(match self {
|
||||||
|
LowerError::Unrendered { .. } => "packet carries a slot the lowerer does not render",
|
||||||
|
LowerError::NoRow { .. } => "no table row for an inventory id the packet uses",
|
||||||
|
LowerError::GlossMismatch { .. } => "table row does not match the inventory's gloss",
|
||||||
|
LowerError::UnknownRole { .. } => "frame template names a role the inventory lacks",
|
||||||
|
LowerError::MissingRole { .. } => "predication fills no role its frame needs",
|
||||||
LowerError::UnknownEntity { .. } => "referent cites an entity the port does not hold",
|
LowerError::UnknownEntity { .. } => "referent cites an entity the port does not hold",
|
||||||
LowerError::NoForm { .. } => "entity has no form in the target language",
|
LowerError::NoForm { .. } => "entity has no form in the target language",
|
||||||
LowerError::NoPronunciation { .. } => "entity has no ruled pronunciation",
|
LowerError::NoPronunciation { .. } => "entity has no ruled pronunciation",
|
||||||
|
|
@ -80,17 +195,147 @@ impl fmt::Display for LowerError {
|
||||||
"ruled pronunciation holds a symbol with no listed phone"
|
"ruled pronunciation holds a symbol with no listed phone"
|
||||||
}
|
}
|
||||||
LowerError::DanglingStress { .. } => "stress mark with no vowel after it",
|
LowerError::DanglingStress { .. } => "stress mark with no vowel after it",
|
||||||
|
LowerError::ConsentMissing { .. } => "request lacks the addressee's consent",
|
||||||
|
LowerError::AskerNotAddressee { .. } => "request agent is not the addressee",
|
||||||
LowerError::EntityRead { .. } => "entity port failed to read",
|
LowerError::EntityRead { .. } => "entity port failed to read",
|
||||||
LowerError::SceneUnread => "the view's scene was never read",
|
|
||||||
LowerError::SpeakerTokenTaken => {
|
|
||||||
"a referent cites the entity id the speaker token uses"
|
|
||||||
}
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl std::error::Error for LowerError {}
|
impl std::error::Error for LowerError {}
|
||||||
|
|
||||||
|
fn row(
|
||||||
|
table: &'static [Row],
|
||||||
|
id: InvId,
|
||||||
|
inv: &Inventory,
|
||||||
|
rev: u32,
|
||||||
|
) -> Result<&'static str, LowerError> {
|
||||||
|
let (_, gloss, surface) = table
|
||||||
|
.iter()
|
||||||
|
.find(|(key, _, _)| *key == id.to_string())
|
||||||
|
.ok_or(LowerError::NoRow { id })?;
|
||||||
|
match inv.resolve(&id, rev) {
|
||||||
|
Some(e) if e.gloss == *gloss => Ok(surface),
|
||||||
|
_ => Err(LowerError::GlossMismatch { id, row: gloss }),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn lower(
|
||||||
|
input: &LoweringInput,
|
||||||
|
inv: &Inventory,
|
||||||
|
entities: &dyn EntityForms,
|
||||||
|
) -> Result<String, LowerError> {
|
||||||
|
let rev = input.inventory_rev();
|
||||||
|
let unrendered = |what| Err(LowerError::Unrendered { what });
|
||||||
|
if input.predications().is_empty() {
|
||||||
|
return unrendered("no predication");
|
||||||
|
}
|
||||||
|
if !input.discourse().features.is_empty()
|
||||||
|
|| input.referents().iter().any(|r| !r.features.is_empty())
|
||||||
|
|| input.predications().iter().any(|p| !p.features.is_empty())
|
||||||
|
{
|
||||||
|
return unrendered("features");
|
||||||
|
}
|
||||||
|
request_gate(input, inv)?;
|
||||||
|
if !input.relations().is_empty() {
|
||||||
|
return unrendered("relations");
|
||||||
|
}
|
||||||
|
if !input.extensions().is_empty() {
|
||||||
|
return unrendered("extensions");
|
||||||
|
}
|
||||||
|
let opener = match input.discourse().speech_act {
|
||||||
|
Some(act) => row(SPEECH_ACTS, act, inv, rev)?,
|
||||||
|
None => "",
|
||||||
|
};
|
||||||
|
let mut sentences = Vec::new();
|
||||||
|
for p in input.predications() {
|
||||||
|
sentences.push(clause(input, inv, entities, p)?);
|
||||||
|
}
|
||||||
|
let mentioned = |r: &NodeId| {
|
||||||
|
input
|
||||||
|
.predications()
|
||||||
|
.iter()
|
||||||
|
.any(|p| p.roles.iter().any(|f| &f.filler == r))
|
||||||
|
};
|
||||||
|
if input.referents().iter().any(|r| !mentioned(&r.id)) {
|
||||||
|
return unrendered("a referent no role mentions");
|
||||||
|
}
|
||||||
|
let mut out = String::new();
|
||||||
|
for (i, (clause, name_first)) in sentences.iter().enumerate() {
|
||||||
|
if i > 0 {
|
||||||
|
out.push(' ');
|
||||||
|
}
|
||||||
|
if i == 0 {
|
||||||
|
out.push_str(opener);
|
||||||
|
}
|
||||||
|
let mut chars = clause.chars();
|
||||||
|
if let Some(first) = chars.next() {
|
||||||
|
// A name opens its clause as written; only template words are cased.
|
||||||
|
if (i == 0 && !opener.is_empty()) || *name_first {
|
||||||
|
out.push(first);
|
||||||
|
} else {
|
||||||
|
out.extend(first.to_uppercase());
|
||||||
|
}
|
||||||
|
out.push_str(chars.as_str());
|
||||||
|
}
|
||||||
|
out.push('.');
|
||||||
|
}
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The one place consent is read. A request clause ([`REQUEST_FRAME`]) says what
|
||||||
|
/// the addressee did, so it needs the packet's own record of it: the discourse
|
||||||
|
/// `addressee_consent`, which validation lets only authoring or game telemetry
|
||||||
|
/// set (V15), must name that predication, and its agent must be the addressee.
|
||||||
|
/// Nothing here reads wording. A consent naming any other predication is a slot
|
||||||
|
/// the lowerer does not render.
|
||||||
|
fn request_gate(input: &LoweringInput, inv: &Inventory) -> Result<(), LowerError> {
|
||||||
|
let consent = input.discourse().addressee_consent.as_ref();
|
||||||
|
let requests = input
|
||||||
|
.predications()
|
||||||
|
.iter()
|
||||||
|
.filter(|p| p.frame.to_string() == REQUEST_FRAME);
|
||||||
|
for p in requests {
|
||||||
|
if consent != Some(&p.id) {
|
||||||
|
return Err(LowerError::ConsentMissing {
|
||||||
|
predication: p.id.clone(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
let agent = inv
|
||||||
|
.entries()
|
||||||
|
.find(|e| {
|
||||||
|
e.family() == Family::Role && e.gloss == "agent" && e.live_at(input.inventory_rev())
|
||||||
|
})
|
||||||
|
.ok_or_else(|| LowerError::UnknownRole {
|
||||||
|
frame: p.frame,
|
||||||
|
role: "agent".to_string(),
|
||||||
|
})?;
|
||||||
|
let asker = p
|
||||||
|
.roles
|
||||||
|
.iter()
|
||||||
|
.find(|r| r.role == agent.id)
|
||||||
|
.map(|r| &r.filler);
|
||||||
|
if asker.is_none() || asker != input.discourse().addressee.as_ref() {
|
||||||
|
return Err(LowerError::AskerNotAddressee {
|
||||||
|
predication: p.id.clone(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
match consent {
|
||||||
|
Some(_)
|
||||||
|
if !input
|
||||||
|
.predications()
|
||||||
|
.iter()
|
||||||
|
.any(|p| p.frame.to_string() == REQUEST_FRAME) =>
|
||||||
|
{
|
||||||
|
Err(LowerError::Unrendered {
|
||||||
|
what: "addressee consent",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
_ => Ok(()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// The ruled IPA of every entity a packet cites, one per entity referent in
|
/// The ruled IPA of every entity a packet cites, one per entity referent in
|
||||||
/// the packet's referent order, as stored: bare, stress-marked, nothing
|
/// the packet's referent order, as stored: bare, stress-marked, nothing
|
||||||
/// derived. A rejected pronunciation is never read. An entity the port does not
|
/// derived. A rejected pronunciation is never read. An entity the port does not
|
||||||
|
|
@ -167,7 +412,92 @@ fn ruled(
|
||||||
Ok(out)
|
Ok(out)
|
||||||
}
|
}
|
||||||
|
|
||||||
pub(crate) fn entity(entities: &dyn EntityForms, id: &EntityId) -> Result<Entity, LowerError> {
|
fn clause(
|
||||||
|
input: &LoweringInput,
|
||||||
|
inv: &Inventory,
|
||||||
|
entities: &dyn EntityForms,
|
||||||
|
p: &Predication,
|
||||||
|
) -> Result<(String, bool), LowerError> {
|
||||||
|
let rev = input.inventory_rev();
|
||||||
|
let template = row(FRAMES, p.frame, inv, rev)?;
|
||||||
|
let mut said: Vec<&NodeId> = Vec::new();
|
||||||
|
let mut out = String::new();
|
||||||
|
let mut name_first = false;
|
||||||
|
let mut rest = template;
|
||||||
|
while let Some(open) = rest.find('{') {
|
||||||
|
out.push_str(&rest[..open]);
|
||||||
|
let close = rest[open..].find('}').map_or(rest.len(), |c| open + c);
|
||||||
|
let name = &rest[open + 1..close];
|
||||||
|
rest = &rest[(close + 1).min(rest.len())..];
|
||||||
|
if name == "verb" {
|
||||||
|
out.push_str(row(CONCEPTS, p.concept, inv, rev)?);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let role = inv
|
||||||
|
.entries()
|
||||||
|
.find(|e| e.family() == Family::Role && e.gloss == name && e.live_at(rev))
|
||||||
|
.ok_or_else(|| LowerError::UnknownRole {
|
||||||
|
frame: p.frame,
|
||||||
|
role: name.to_string(),
|
||||||
|
})?;
|
||||||
|
let fill =
|
||||||
|
p.roles
|
||||||
|
.iter()
|
||||||
|
.find(|r| r.role == role.id)
|
||||||
|
.ok_or_else(|| LowerError::MissingRole {
|
||||||
|
predication: p.id.clone(),
|
||||||
|
role: name.to_string(),
|
||||||
|
})?;
|
||||||
|
let (phrase, is_name) = filler(input, inv, entities, &fill.filler, &said)?;
|
||||||
|
name_first |= is_name && out.is_empty();
|
||||||
|
out.push_str(&phrase);
|
||||||
|
said.push(&fill.filler);
|
||||||
|
}
|
||||||
|
out.push_str(rest);
|
||||||
|
if p.roles.len() != said.len() {
|
||||||
|
return Err(LowerError::Unrendered {
|
||||||
|
what: "a role the frame does not name",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok((out, name_first))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A role filler as a phrase: the addressee is "you", a predication is "that",
|
||||||
|
/// another referent is its entity's citation form when it cites one, else "the"
|
||||||
|
/// and its concept word. The addressee said twice in
|
||||||
|
/// one predication is reflexive; no other referent is, because the packet has no
|
||||||
|
/// animacy to choose "themself" or "itself" by.
|
||||||
|
fn filler(
|
||||||
|
input: &LoweringInput,
|
||||||
|
inv: &Inventory,
|
||||||
|
entities: &dyn EntityForms,
|
||||||
|
node: &NodeId,
|
||||||
|
said: &[&NodeId],
|
||||||
|
) -> Result<(String, bool), LowerError> {
|
||||||
|
let again = said.contains(&node);
|
||||||
|
if let Some(r) = input.referents().iter().find(|r| &r.id == node) {
|
||||||
|
if input.discourse().addressee.as_ref() == Some(node) {
|
||||||
|
return Ok((if again { "yourself" } else { "you" }.to_string(), false));
|
||||||
|
}
|
||||||
|
if let Some(id) = &r.entity {
|
||||||
|
return Ok((name(entities, id)?, true));
|
||||||
|
}
|
||||||
|
let word = row(CONCEPTS, r.concept, inv, input.inventory_rev())?;
|
||||||
|
return Ok((format!("the {word}"), false));
|
||||||
|
}
|
||||||
|
Ok(("that".to_string(), false))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The entity's citation form in [`TARGET_LANGUAGE`].
|
||||||
|
fn name(entities: &dyn EntityForms, id: &EntityId) -> Result<String, LowerError> {
|
||||||
|
entity(entities, id)?
|
||||||
|
.forms_in(TARGET_LANGUAGE)
|
||||||
|
.and_then(<[String]>::first)
|
||||||
|
.cloned()
|
||||||
|
.ok_or_else(|| LowerError::NoForm { entity: id.clone() })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn entity(entities: &dyn EntityForms, id: &EntityId) -> Result<Entity, LowerError> {
|
||||||
entities
|
entities
|
||||||
.entity(id)
|
.entity(id)
|
||||||
.map_err(|e| LowerError::EntityRead {
|
.map_err(|e| LowerError::EntityRead {
|
||||||
|
|
|
||||||
|
|
@ -1,26 +1,20 @@
|
||||||
//! The `weksa` command line: the same validator and store admission the library
|
//! The `weksa` command line: the same validator and store admission the library
|
||||||
//! exposes, plus text projections (`dump`) that are for review and are not stores.
|
//! exposes, plus text projections (`dump`) that are for review and are not stores.
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::PathBuf;
|
||||||
|
|
||||||
use anyhow::{Context, Result, bail};
|
use anyhow::{Context, Result, bail};
|
||||||
use clap::error::{ContextKind, ContextValue, ErrorKind};
|
use clap::error::{ContextKind, ContextValue, ErrorKind};
|
||||||
use clap::{Args, Parser, Subcommand, ValueEnum};
|
use clap::{Args, Parser, Subcommand, ValueEnum};
|
||||||
use unicode_normalization::UnicodeNormalization;
|
use unicode_normalization::UnicodeNormalization;
|
||||||
use weksa::corpus::{CorpusItem, ModelFamily, NameSpan, Source, Turn};
|
use weksa::corpus::{CorpusItem, ModelFamily, NameSpan, Source, Turn};
|
||||||
use weksa::encode::{EditDraft, MarkedLine, OntologyDraft, WorkDraft};
|
|
||||||
use weksa::entity::{Entity, EntityOrigin, LanguageForms, Pronunciation};
|
use weksa::entity::{Entity, EntityOrigin, LanguageForms, Pronunciation};
|
||||||
use weksa::inventory::{proof_revision, seed, speaker_scene_revision};
|
use weksa::inventory::{proof_revision, seed};
|
||||||
use weksa::packet::{FeatValue, FeatureValue, Grade};
|
|
||||||
use weksa::packet::proof_packet;
|
use weksa::packet::proof_packet;
|
||||||
use weksa::model::{ModelPort, RecordedAnswers};
|
|
||||||
use weksa::run::{
|
|
||||||
AblationOutcome, CalibrateOutcome, CycleOutcome, FLOOR_ARMS_MIN, LowererReach, OntologyMintOutcome, RecallOutcome, RelowerOutcome, RoundOutcome, RunPaths, RunStores, cycle, round,
|
|
||||||
};
|
|
||||||
use weksa::score::ScoreInput;
|
use weksa::score::ScoreInput;
|
||||||
use weksa::{
|
use weksa::{
|
||||||
Annotator, CorpusStore, EntityId, EntityStore, Family, InventoryStore, OllamaEmbedder, OntologyStore, OptionLabel,
|
CorpusStore, EntityId, EntityStore, InventoryStore, OllamaEmbedder, PacketStore, Scorer,
|
||||||
PacketStore, PairRelation, Party, Scene, Slot, SpeakerStateStore, ValueKind, Scorer, Split, SplitScheme, StoreError, StoreRefusal, pronounce, validate,
|
SplitScheme, StoreError, lower, pronounce, validate,
|
||||||
};
|
};
|
||||||
|
|
||||||
#[derive(Parser)]
|
#[derive(Parser)]
|
||||||
|
|
@ -38,12 +32,6 @@ struct Cli {
|
||||||
/// The entity store.
|
/// The entity store.
|
||||||
#[arg(long, global = true, default_value = "data/entity/entity.cc")]
|
#[arg(long, global = true, default_value = "data/entity/entity.cc")]
|
||||||
entities: PathBuf,
|
entities: PathBuf,
|
||||||
/// The language ontology store.
|
|
||||||
#[arg(long, global = true, default_value = "data/ontology/ontology.cc")]
|
|
||||||
ontology: PathBuf,
|
|
||||||
/// The speaker-state store (scenes).
|
|
||||||
#[arg(long, global = true, default_value = "data/corpus/speaker-state.cc")]
|
|
||||||
speaker_state: PathBuf,
|
|
||||||
#[command(subcommand)]
|
#[command(subcommand)]
|
||||||
command: Command,
|
command: Command,
|
||||||
}
|
}
|
||||||
|
|
@ -59,19 +47,18 @@ enum Command {
|
||||||
#[command(subcommand)]
|
#[command(subcommand)]
|
||||||
Entity(EntityCommand),
|
Entity(EntityCommand),
|
||||||
#[command(subcommand)]
|
#[command(subcommand)]
|
||||||
Ontology(OntologyCommand),
|
|
||||||
#[command(subcommand)]
|
|
||||||
Scene(SceneCommand),
|
|
||||||
#[command(subcommand)]
|
|
||||||
Packet(PacketCommand),
|
Packet(PacketCommand),
|
||||||
/// Print the ruled IPA of each entity a stored packet cites, one per line;
|
/// Lower a stored packet to English text.
|
||||||
/// refused for an entity with no ruled pronunciation.
|
Lower {
|
||||||
Pronounce {
|
|
||||||
store: PathBuf,
|
store: PathBuf,
|
||||||
key: String,
|
key: String,
|
||||||
/// Print the phones line of each entity instead, one JSON object per
|
/// Print the ruled IPA of each entity the packet cites, one per line,
|
||||||
/// line ({language, source, phones: [{ipa, stress}]}); refused for a
|
/// instead of the text; refused for an entity with no ruled pronunciation.
|
||||||
/// phone the inventory lacks.
|
#[arg(long, conflicts_with = "phones")]
|
||||||
|
ipa: bool,
|
||||||
|
/// Print the phones line of each entity the packet cites, one JSON
|
||||||
|
/// object per line ({language, source, phones: [{ipa, stress}]}),
|
||||||
|
/// instead of the text; refused for a phone the inventory lacks.
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
phones: bool,
|
phones: bool,
|
||||||
},
|
},
|
||||||
|
|
@ -79,8 +66,6 @@ enum Command {
|
||||||
Score(ScoreArgs),
|
Score(ScoreArgs),
|
||||||
/// Print the split of a corpus item.
|
/// Print the split of a corpus item.
|
||||||
Split { item_key: String },
|
Split { item_key: String },
|
||||||
#[command(subcommand)]
|
|
||||||
Loop(LoopCommand),
|
|
||||||
/// Write the published JSON Schemas.
|
/// Write the published JSON Schemas.
|
||||||
Schema {
|
Schema {
|
||||||
#[arg(long, default_value = "schemas/cultnet")]
|
#[arg(long, default_value = "schemas/cultnet")]
|
||||||
|
|
@ -88,127 +73,6 @@ enum Command {
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Subcommand)]
|
|
||||||
enum LoopCommand {
|
|
||||||
/// Pin the global --corpus, --inventory, --entities, --ontology and --speaker-state as a run at ROOT.
|
|
||||||
Open {
|
|
||||||
root: PathBuf,
|
|
||||||
#[arg(long)]
|
|
||||||
label: String,
|
|
||||||
/// The id of the model that answers prompts.
|
|
||||||
#[arg(long)]
|
|
||||||
answerer: String,
|
|
||||||
#[arg(long, default_value = "qwen3-embedding:0.6b")]
|
|
||||||
embed_model: String,
|
|
||||||
/// How lowering prompts will be answered: briefed-subagent (smoke
|
|
||||||
/// rounds only) or tool-less-relay. Pinned in the run's record.
|
|
||||||
#[arg(long)]
|
|
||||||
lowerer_reach: LowererReach,
|
|
||||||
/// The family of the answering model; rounds count the corpus items
|
|
||||||
/// that family wrote. Defaults to unrecorded.
|
|
||||||
#[arg(long, default_value = "unrecorded")]
|
|
||||||
answerer_family: ModelFamilyArg,
|
|
||||||
/// The languages the run offers, reads and decides on: BCP 47 tags,
|
|
||||||
/// comma-separated. Every one needs a live Train and Validation item.
|
|
||||||
#[arg(long, value_delimiter = ',', required = true)]
|
|
||||||
panel: Vec<String>,
|
|
||||||
},
|
|
||||||
/// Run one round of the run at ROOT. The stores are read only from the
|
|
||||||
/// run's record: the global --corpus, --inventory, --entities and --ontology
|
|
||||||
/// are not read. Encode prompts are written first; once they are answered a rerun
|
|
||||||
/// writes the lowering prompts. The command exits 3 each time until every
|
|
||||||
/// prompt has an answer.
|
|
||||||
Round {
|
|
||||||
root: PathBuf,
|
|
||||||
#[arg(long, default_value = "http://127.0.0.1:11434")]
|
|
||||||
ollama_url: String,
|
|
||||||
},
|
|
||||||
/// Probe how closely a fresh model recites the run's items from their
|
|
||||||
/// openings (chrF of its completion against the item's second
|
|
||||||
/// half). Prompts and answers live in `<ROOT>/recall`; the command exits 3
|
|
||||||
/// until every prompt has an answer. A diagnostic, never a gate.
|
|
||||||
Recall { root: PathBuf },
|
|
||||||
/// Draft ontology entries from the run's Train items of one language.
|
|
||||||
/// Prompts and answers live in `<ROOT>/ontology-mint`; the command exits 3
|
|
||||||
/// until the prompt has an answer. Admission is `weksa ontology admit`
|
|
||||||
/// into a copy, never into the run's pinned file.
|
|
||||||
MintOntology {
|
|
||||||
root: PathBuf,
|
|
||||||
#[arg(long)]
|
|
||||||
language: String,
|
|
||||||
},
|
|
||||||
/// Ask whether the lowerer uses a packet's stance: for each live Train and
|
|
||||||
/// Validation packet of round ROUND, ask again with the stance as it is,
|
|
||||||
/// emptied, and swapped for the next item's, and read how far the lowering
|
|
||||||
/// moves from the round's own answer. Prompts and answers live in
|
|
||||||
/// `<ROOT>/ablation-<ROUND>`; the command exits 3 until every prompt has an
|
|
||||||
/// answer. A diagnostic, never a gate; Test is never touched.
|
|
||||||
Ablate {
|
|
||||||
root: PathBuf,
|
|
||||||
#[arg(long)]
|
|
||||||
round: u32,
|
|
||||||
#[arg(long, default_value = "http://127.0.0.1:11434")]
|
|
||||||
ollama_url: String,
|
|
||||||
},
|
|
||||||
/// Re-lower round ROUND's Validation packets through arm ARM, score them
|
|
||||||
/// against the items, and, with --floor ARM given two or more times (prompted
|
|
||||||
/// repeats of the lowering), decide against the paired noise floor of those
|
|
||||||
/// repeats by the panel accept rule. Prompts and answers live in
|
|
||||||
/// `<ROOT>/relower-<ROUND>-<ARM>`; the command exits 3 until the arm is
|
|
||||||
/// answered. --lowerer names the arm's model in the printed line. A
|
|
||||||
/// diagnostic, never a gate; records nothing; Test and Train are never
|
|
||||||
/// touched. Labels match [a-z0-9-]{1,32} and differ.
|
|
||||||
Relower {
|
|
||||||
root: PathBuf,
|
|
||||||
#[arg(long)]
|
|
||||||
round: u32,
|
|
||||||
#[arg(long)]
|
|
||||||
arm: String,
|
|
||||||
#[arg(long)]
|
|
||||||
lowerer: String,
|
|
||||||
#[arg(long)]
|
|
||||||
floor: Vec<String>,
|
|
||||||
#[arg(long, default_value = "http://127.0.0.1:11434")]
|
|
||||||
ollama_url: String,
|
|
||||||
},
|
|
||||||
/// Replay the accept rule over answered relower draws of round ROUND, to
|
|
||||||
/// measure how often it accepts, rejects or regresses what it should. Reads
|
|
||||||
/// the recorded lowering and each draw (--prompted: repeats of it, at least
|
|
||||||
/// four; --degraded: arms that should regress; --candidate: arms to rate),
|
|
||||||
/// answered through `weksa loop relower ROOT --round N --arm LABEL`, and
|
|
||||||
/// prints counts over all Validation items and over each --class
|
|
||||||
/// NAME=PREFIX[,PREFIX...] (items whose key starts with a prefix). Writes
|
|
||||||
/// no prompts; exits 1 naming the draws still unanswered. Procedure:
|
|
||||||
/// docs/calibration.md. A diagnostic, never a gate; records nothing; Test
|
|
||||||
/// and Train are never touched. Labels match [a-z0-9-]{1,32} and differ.
|
|
||||||
Calibrate {
|
|
||||||
root: PathBuf,
|
|
||||||
#[arg(long)]
|
|
||||||
round: u32,
|
|
||||||
#[arg(long)]
|
|
||||||
prompted: Vec<String>,
|
|
||||||
#[arg(long)]
|
|
||||||
degraded: Vec<String>,
|
|
||||||
#[arg(long)]
|
|
||||||
candidate: Vec<String>,
|
|
||||||
#[arg(long)]
|
|
||||||
class: Vec<String>,
|
|
||||||
#[arg(long, default_value = "http://127.0.0.1:11434")]
|
|
||||||
ollama_url: String,
|
|
||||||
},
|
|
||||||
/// Run one edit-accept cycle of the run at ROOT. The stores are read only
|
|
||||||
/// from the run's record. The editor's prompt, then the candidate round's
|
|
||||||
/// encode prompts, then its lowering prompts are written in turn; the
|
|
||||||
/// command exits 3 each time until every prompt has an answer. The noise
|
|
||||||
/// floor needs four recorded rounds under the first inventory (the recorded
|
|
||||||
/// round and three prompted repeats).
|
|
||||||
Cycle {
|
|
||||||
root: PathBuf,
|
|
||||||
#[arg(long, default_value = "http://127.0.0.1:11434")]
|
|
||||||
ollama_url: String,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Subcommand)]
|
#[derive(Subcommand)]
|
||||||
enum PacketCommand {
|
enum PacketCommand {
|
||||||
/// Admit the nibu-proof-001 packet through `PacketStore::admit`.
|
/// Admit the nibu-proof-001 packet through `PacketStore::admit`.
|
||||||
|
|
@ -239,22 +103,6 @@ enum CorpusCommand {
|
||||||
Withdraw {
|
Withdraw {
|
||||||
key: String,
|
key: String,
|
||||||
},
|
},
|
||||||
/// Import a transcribed public-domain work: mint its cast as entities and
|
|
||||||
/// add each line a name-span answer covers (`docs/name-span-annotation.md`).
|
|
||||||
Import {
|
|
||||||
/// The work file: provenance, cast, lines.
|
|
||||||
work: PathBuf,
|
|
||||||
/// The annotator's answer: one `{"line": N, "marked": "..."}` per line.
|
|
||||||
names: PathBuf,
|
|
||||||
#[arg(long, requires = "generator", conflicts_with = "operator")]
|
|
||||||
model_family: Option<ModelFamilyArg>,
|
|
||||||
/// The attestation string naming the annotating model.
|
|
||||||
#[arg(long, requires = "model_family")]
|
|
||||||
generator: Option<String>,
|
|
||||||
/// Annotated by the operator, not a model.
|
|
||||||
#[arg(long)]
|
|
||||||
operator: bool,
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Subcommand)]
|
#[derive(Subcommand)]
|
||||||
|
|
@ -267,77 +115,16 @@ enum InventoryCommand {
|
||||||
Proof,
|
Proof,
|
||||||
/// Admit revision 3 of the Dao De Jing test bed stores (after seed and proof).
|
/// Admit revision 3 of the Dao De Jing test bed stores (after seed and proof).
|
||||||
Dao,
|
Dao,
|
||||||
/// Admit the speaker-state revision (Scene and Speaker features) as the
|
|
||||||
/// inventory's latest plus one.
|
|
||||||
SpeakerScene,
|
|
||||||
/// Admit a model's inventory draft (a JSON file, {add, retire} as the
|
|
||||||
/// loop's editor writes it) as the inventory's latest plus one.
|
|
||||||
Admit { answer: PathBuf },
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The most changes `inventory admit` takes in one draft. A hand-admitted draft
|
|
||||||
/// is reviewed whole in its cut, so the cap is not the loop editor's.
|
|
||||||
const MAX_ADMIT_CHANGES: usize = 256;
|
|
||||||
|
|
||||||
#[derive(Subcommand)]
|
|
||||||
enum SceneCommand {
|
|
||||||
/// Append a scene revision for one corpus group: its one live speaker's
|
|
||||||
/// relation to the addressee, as Scene-slot feature values.
|
|
||||||
Put {
|
|
||||||
#[arg(long)]
|
|
||||||
group: String,
|
|
||||||
/// `entity:<slug>` of a named addressee, or `unnamed`.
|
|
||||||
#[arg(long)]
|
|
||||||
to: String,
|
|
||||||
/// The inventory revision the glosses resolve at.
|
|
||||||
#[arg(long)]
|
|
||||||
inventory_rev: u32,
|
|
||||||
/// `gloss=value`, repeated: a grade 0-1000, an option label or true/false.
|
|
||||||
#[arg(long = "set", value_name = "GLOSS=VALUE")]
|
|
||||||
set: Vec<String>,
|
|
||||||
#[arg(long, requires = "generator", conflicts_with = "operator")]
|
|
||||||
model_family: Option<ModelFamilyArg>,
|
|
||||||
/// The attestation string naming the annotating model.
|
|
||||||
#[arg(long, requires = "model_family")]
|
|
||||||
generator: Option<String>,
|
|
||||||
/// Annotated by the operator, not a model.
|
|
||||||
#[arg(long)]
|
|
||||||
operator: bool,
|
|
||||||
},
|
|
||||||
/// Text projection of the scenes for review; not a store.
|
|
||||||
Dump,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Subcommand)]
|
#[derive(Subcommand)]
|
||||||
enum EntityCommand {
|
enum EntityCommand {
|
||||||
/// Admit an entity, or append forms to a stored one.
|
/// Admit an entity, or append forms to a stored one.
|
||||||
Put(Box<PutArgs>),
|
Put(PutArgs),
|
||||||
/// Text projection of the entity store for review; not a store.
|
/// Text projection of the entity store for review; not a store.
|
||||||
Dump,
|
Dump,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Subcommand)]
|
|
||||||
enum OntologyCommand {
|
|
||||||
/// Admit a model's ontology draft (a JSON file) for one language as a
|
|
||||||
/// batch, numbered from the language's latest plus one and annotated as
|
|
||||||
/// the model that drafted it.
|
|
||||||
Admit {
|
|
||||||
answer: PathBuf,
|
|
||||||
#[arg(long)]
|
|
||||||
language: String,
|
|
||||||
/// The inventory revision the draft's conditions are read at.
|
|
||||||
#[arg(long)]
|
|
||||||
inventory_rev: u32,
|
|
||||||
#[arg(long)]
|
|
||||||
model_family: ModelFamilyArg,
|
|
||||||
/// The attestation string naming the drafting model.
|
|
||||||
#[arg(long)]
|
|
||||||
generator: String,
|
|
||||||
},
|
|
||||||
/// Text projection of the ontology for review; not a store.
|
|
||||||
Dump,
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Args)]
|
#[derive(Args)]
|
||||||
struct PutArgs {
|
struct PutArgs {
|
||||||
/// `entity:<slug>`.
|
/// `entity:<slug>`.
|
||||||
|
|
@ -476,8 +263,7 @@ struct AddArgs {
|
||||||
/// NFC-normalized here; `--name` byte offsets index the normalized text.
|
/// NFC-normalized here; `--name` byte offsets index the normalized text.
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
text: String,
|
text: String,
|
||||||
/// The item's language, a BCP 47 tag; no language is the default.
|
#[arg(long, default_value = "en-US")]
|
||||||
#[arg(long)]
|
|
||||||
language: String,
|
language: String,
|
||||||
/// An entity id.
|
/// An entity id.
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
|
|
@ -694,17 +480,10 @@ impl AddArgs {
|
||||||
supersedes: self.supersedes,
|
supersedes: self.supersedes,
|
||||||
withdrawn: false,
|
withdrawn: false,
|
||||||
names,
|
names,
|
||||||
names_by: None,
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A draw or arm label: [a-z0-9-]{1,32}.
|
|
||||||
fn is_label(l: &str) -> bool {
|
|
||||||
(1..=32).contains(&l.len())
|
|
||||||
&& l.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-')
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The split of an item as `Train`, `Validation`, `Test` or `excluded`.
|
/// The split of an item as `Train`, `Validation`, `Test` or `excluded`.
|
||||||
fn split_text(item: &CorpusItem) -> String {
|
fn split_text(item: &CorpusItem) -> String {
|
||||||
SplitScheme::V1
|
SplitScheme::V1
|
||||||
|
|
@ -712,22 +491,6 @@ fn split_text(item: &CorpusItem) -> String {
|
||||||
.map_or_else(|| "excluded".to_string(), |s| format!("{s:?}"))
|
.map_or_else(|| "excluded".to_string(), |s| format!("{s:?}"))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The annotator the flags name: a model family with its generator, or the operator.
|
|
||||||
fn annotator(
|
|
||||||
model_family: Option<ModelFamilyArg>,
|
|
||||||
generator: Option<String>,
|
|
||||||
operator: bool,
|
|
||||||
) -> Result<Annotator> {
|
|
||||||
match (model_family, generator, operator) {
|
|
||||||
(Some(family), Some(generator), false) => Ok(Annotator::Model {
|
|
||||||
family: family.into(),
|
|
||||||
generator,
|
|
||||||
}),
|
|
||||||
(None, None, true) => Ok(Annotator::Operator),
|
|
||||||
_ => bail!("name the annotator: --model-family with --generator, or --operator"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn refused(e: StoreError) -> anyhow::Error {
|
fn refused(e: StoreError) -> anyhow::Error {
|
||||||
anyhow::anyhow!("{e}")
|
anyhow::anyhow!("{e}")
|
||||||
}
|
}
|
||||||
|
|
@ -767,78 +530,13 @@ fn main() -> Result<()> {
|
||||||
.map_err(refused)?;
|
.map_err(refused)?;
|
||||||
println!("{outcome:?} {key}");
|
println!("{outcome:?} {key}");
|
||||||
}
|
}
|
||||||
Command::Corpus(CorpusCommand::Import {
|
|
||||||
work,
|
|
||||||
names,
|
|
||||||
model_family,
|
|
||||||
generator,
|
|
||||||
operator,
|
|
||||||
}) => {
|
|
||||||
let annotator = annotator(model_family, generator, operator)?;
|
|
||||||
let draft = WorkDraft::parse(&std::fs::read_to_string(&work)?)?;
|
|
||||||
let mut marked = Vec::new();
|
|
||||||
for (i, line) in std::fs::read_to_string(&names)?.lines().enumerate() {
|
|
||||||
if line.trim().is_empty() {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
marked.push(
|
|
||||||
MarkedLine::parse(line)
|
|
||||||
.map_err(|e| anyhow::anyhow!("names line {}: {e}", i + 1))?,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
let (new_entities, items) = draft.into_import(&marked, &annotator)?;
|
|
||||||
// Every document passes the real doors on scratch copies of the
|
|
||||||
// stores before the first write, so a refusal leaves them untouched.
|
|
||||||
let scratch = std::env::temp_dir().join(format!("weksa-import-{}", std::process::id()));
|
|
||||||
std::fs::create_dir_all(&scratch)?;
|
|
||||||
let dry = (|| -> Result<()> {
|
|
||||||
let copy = |from: &Path, name: &str| -> Result<PathBuf> {
|
|
||||||
let to = scratch.join(name);
|
|
||||||
if from.exists() {
|
|
||||||
std::fs::copy(from, &to)?;
|
|
||||||
}
|
|
||||||
Ok(to)
|
|
||||||
};
|
|
||||||
let mut entities = EntityStore::open(©(&cli.entities, "e.cc")?)?;
|
|
||||||
let mut store = CorpusStore::open(©(&cli.corpus, "c.cc")?)?;
|
|
||||||
for e in new_entities.clone() {
|
|
||||||
entities
|
|
||||||
.put(e)
|
|
||||||
.map_err(|e| anyhow::anyhow!("a cast entity: {e}"))?;
|
|
||||||
}
|
|
||||||
for item in items.clone() {
|
|
||||||
let key = item.id.clone();
|
|
||||||
store.add(item, &entities).map_err(|e| match e {
|
|
||||||
StoreError::Refused(StoreRefusal::S6TextHeld { held_by, .. }) => {
|
|
||||||
anyhow::anyhow!("{key} repeats the text of {held_by}")
|
|
||||||
}
|
|
||||||
e => anyhow::anyhow!("{key}: {e}"),
|
|
||||||
})?;
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
})();
|
|
||||||
let _ = std::fs::remove_dir_all(&scratch);
|
|
||||||
dry?;
|
|
||||||
let mut entities = EntityStore::open(&cli.entities)?;
|
|
||||||
for e in new_entities {
|
|
||||||
let id = e.id.clone();
|
|
||||||
let outcome = entities.put(e).map_err(refused)?;
|
|
||||||
println!("{outcome:?} {id}");
|
|
||||||
}
|
|
||||||
let mut store = CorpusStore::open(&cli.corpus)?;
|
|
||||||
for item in items {
|
|
||||||
let (key, split) = (item.id.clone(), split_text(&item));
|
|
||||||
let outcome = store.add(item, &entities).map_err(refused)?;
|
|
||||||
println!("{outcome:?} {key} {split}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Corpus(CorpusCommand::Dump) => {
|
Command::Corpus(CorpusCommand::Dump) => {
|
||||||
for i in CorpusStore::open(&cli.corpus)?.all()? {
|
for i in CorpusStore::open(&cli.corpus)?.all()? {
|
||||||
let split = split_text(&i);
|
let split = split_text(&i);
|
||||||
let withdrawn = if i.withdrawn { " withdrawn" } else { "" };
|
let withdrawn = if i.withdrawn { " withdrawn" } else { "" };
|
||||||
println!(
|
println!(
|
||||||
"{} [{} {} group={} {split}{withdrawn}]\n {:?}\n names: {:?} by {:?}\n source: {:?}",
|
"{} [{} {} group={} {split}{withdrawn}]\n {:?}\n names: {:?}\n source: {:?}",
|
||||||
i.id, i.language, i.speaker, i.group, i.text, i.names, i.names_by, i.source
|
i.id, i.language, i.speaker, i.group, i.text, i.names, i.source
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -879,48 +577,6 @@ fn main() -> Result<()> {
|
||||||
println!("{} {:?}\n forms: {:?}", e.id, e.origin, e.forms);
|
println!("{} {:?}\n forms: {:?}", e.id, e.origin, e.forms);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Command::Ontology(OntologyCommand::Admit {
|
|
||||||
answer,
|
|
||||||
language,
|
|
||||||
inventory_rev,
|
|
||||||
model_family,
|
|
||||||
generator,
|
|
||||||
}) => {
|
|
||||||
let text = std::fs::read_to_string(&answer).context("the draft would not read")?;
|
|
||||||
let draft = OntologyDraft::parse(&text)?;
|
|
||||||
let mut store = OntologyStore::open(&cli.ontology)?;
|
|
||||||
let first = store
|
|
||||||
.all()?
|
|
||||||
.iter()
|
|
||||||
.filter(|e| e.language == language)
|
|
||||||
.map(|e| e.n)
|
|
||||||
.max()
|
|
||||||
.unwrap_or(0)
|
|
||||||
+ 1;
|
|
||||||
let annotator = Annotator::Model {
|
|
||||||
family: model_family.into(),
|
|
||||||
generator,
|
|
||||||
};
|
|
||||||
let entries = draft.into_entries(&language, inventory_rev, first, &annotator);
|
|
||||||
let count = entries.len();
|
|
||||||
let inventory = InventoryStore::open(&cli.inventory)?;
|
|
||||||
let outcome = store.admit(entries, &inventory).map_err(refused)?;
|
|
||||||
println!("{outcome:?} {count} entries of {language}");
|
|
||||||
}
|
|
||||||
Command::Ontology(OntologyCommand::Dump) => {
|
|
||||||
for e in OntologyStore::open(&cli.ontology)?.all()? {
|
|
||||||
println!(
|
|
||||||
"{} {} rev:{} {:?}\n when: {:?}\n {}\n e.g. {:?}",
|
|
||||||
e.key(),
|
|
||||||
e.layer.name(),
|
|
||||||
e.inventory_rev,
|
|
||||||
e.annotator,
|
|
||||||
e.when,
|
|
||||||
e.choice,
|
|
||||||
e.examples
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Inventory(InventoryCommand::Seed) => {
|
Command::Inventory(InventoryCommand::Seed) => {
|
||||||
let (rev, entries) = seed();
|
let (rev, entries) = seed();
|
||||||
let count = entries.len();
|
let count = entries.len();
|
||||||
|
|
@ -937,152 +593,6 @@ fn main() -> Result<()> {
|
||||||
.map_err(refused)?;
|
.map_err(refused)?;
|
||||||
println!("admitted rev:2 with {count} entries");
|
println!("admitted rev:2 with {count} entries");
|
||||||
}
|
}
|
||||||
Command::Inventory(InventoryCommand::SpeakerScene) => {
|
|
||||||
let store = InventoryStore::open(&cli.inventory)?;
|
|
||||||
let latest = store
|
|
||||||
.inventory()?
|
|
||||||
.latest()
|
|
||||||
.map(|r| r.n)
|
|
||||||
.context("the inventory holds no revision")?;
|
|
||||||
let (rev, entries) = speaker_scene_revision(latest + 1);
|
|
||||||
let (n, count) = (rev.n, entries.len());
|
|
||||||
let mut store = store;
|
|
||||||
store.admit_revision(rev, entries).map_err(refused)?;
|
|
||||||
println!("admitted rev:{n} with {count} entries");
|
|
||||||
}
|
|
||||||
Command::Inventory(InventoryCommand::Admit { answer }) => {
|
|
||||||
let text = std::fs::read_to_string(&answer).context("the draft would not read")?;
|
|
||||||
let draft = EditDraft::parse(&text)?;
|
|
||||||
let mut store = InventoryStore::open(&cli.inventory)?;
|
|
||||||
let inventory = store.inventory()?;
|
|
||||||
let latest = inventory
|
|
||||||
.latest()
|
|
||||||
.map(|r| r.n)
|
|
||||||
.context("the inventory holds no revision")?;
|
|
||||||
let (rev, entries) = draft
|
|
||||||
.into_revision(&inventory, latest, MAX_ADMIT_CHANGES)
|
|
||||||
.context("the draft makes no change or more than the cap")?;
|
|
||||||
let (n, count) = (rev.n, entries.len());
|
|
||||||
store.admit_revision(rev, entries).map_err(refused)?;
|
|
||||||
println!("admitted rev:{n} with {count} entries");
|
|
||||||
}
|
|
||||||
Command::Scene(SceneCommand::Put {
|
|
||||||
group,
|
|
||||||
to,
|
|
||||||
inventory_rev,
|
|
||||||
set,
|
|
||||||
model_family,
|
|
||||||
generator,
|
|
||||||
operator,
|
|
||||||
}) => {
|
|
||||||
let annotator = annotator(model_family, generator, operator)?;
|
|
||||||
let inventory_store = InventoryStore::open(&cli.inventory)?;
|
|
||||||
let inventory = inventory_store.inventory()?;
|
|
||||||
let corpus = CorpusStore::open(&cli.corpus)?;
|
|
||||||
let live: Vec<CorpusItem> = corpus
|
|
||||||
.all()?
|
|
||||||
.into_iter()
|
|
||||||
.filter(|i| i.group == group && !i.withdrawn)
|
|
||||||
.collect();
|
|
||||||
let mut speakers: Vec<EntityId> = live.iter().map(|i| i.speaker.clone()).collect();
|
|
||||||
speakers.sort();
|
|
||||||
speakers.dedup();
|
|
||||||
let [from] = speakers.as_slice() else {
|
|
||||||
bail!("a scene group holds one live speaker; multi-party scenes wait");
|
|
||||||
};
|
|
||||||
let to = match to.as_str() {
|
|
||||||
"unnamed" => Party::Unnamed,
|
|
||||||
named => Party::Named(
|
|
||||||
EntityId::parse(named).map_err(|_| anyhow::anyhow!("the addressee is unnamed or an entity id"))?,
|
|
||||||
),
|
|
||||||
};
|
|
||||||
let mut features = Vec::new();
|
|
||||||
for pair in &set {
|
|
||||||
let bad = || anyhow::anyhow!("a --set is gloss=value of a live Scene feature");
|
|
||||||
let (gloss, value) = pair.split_once('=').ok_or_else(bad)?;
|
|
||||||
let entry = inventory
|
|
||||||
.entries()
|
|
||||||
.find(|e| {
|
|
||||||
e.family() == Family::Feature
|
|
||||||
&& e.gloss == gloss
|
|
||||||
&& e.live_at(inventory_rev)
|
|
||||||
&& e.applies_to.contains(&Slot::Scene)
|
|
||||||
})
|
|
||||||
.ok_or_else(bad)?;
|
|
||||||
let value = match entry.value_kind.as_ref().ok_or_else(bad)? {
|
|
||||||
ValueKind::Grade => FeatValue::Grade(
|
|
||||||
Grade::new(value.parse().map_err(|_| bad())?).map_err(|_| bad())?,
|
|
||||||
),
|
|
||||||
ValueKind::Options(_) => {
|
|
||||||
FeatValue::Option(OptionLabel::parse(value).map_err(|_| bad())?)
|
|
||||||
}
|
|
||||||
ValueKind::Flag => FeatValue::Flag(value.parse().map_err(|_| bad())?),
|
|
||||||
};
|
|
||||||
features.push(FeatureValue { feature: entry.id, value });
|
|
||||||
}
|
|
||||||
let mut participants = vec![from.clone()];
|
|
||||||
if let Party::Named(a) = &to
|
|
||||||
&& a != from
|
|
||||||
{
|
|
||||||
participants.push(a.clone());
|
|
||||||
}
|
|
||||||
let mut state = SpeakerStateStore::open(&cli.speaker_state)?;
|
|
||||||
let latest = state.scene(&group)?;
|
|
||||||
let mut scene = Scene {
|
|
||||||
group,
|
|
||||||
n: latest.as_ref().map_or(1, |l| l.n + 1),
|
|
||||||
inventory_rev,
|
|
||||||
participants,
|
|
||||||
relations: vec![PairRelation { from: from.clone(), to, features }],
|
|
||||||
annotator,
|
|
||||||
};
|
|
||||||
if let Some(l) = &latest {
|
|
||||||
let same = Scene { n: l.n, ..scene.clone() };
|
|
||||||
if &same == l {
|
|
||||||
println!("unchanged");
|
|
||||||
return Ok(());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
scene.n = latest.map_or(1, |l| l.n + 1);
|
|
||||||
let key = weksa::speaker::scene_key(&scene.group, scene.n);
|
|
||||||
let outcome = state
|
|
||||||
.put_scene(scene, &inventory_store, &corpus, &EntityStore::open(&cli.entities)?)
|
|
||||||
.map_err(refused)?;
|
|
||||||
println!("{outcome:?} {key}");
|
|
||||||
}
|
|
||||||
Command::Scene(SceneCommand::Dump) => {
|
|
||||||
let inventory = InventoryStore::open(&cli.inventory)?.inventory()?;
|
|
||||||
let gloss = |f: &FeatureValue| {
|
|
||||||
let name = inventory.entry(&f.feature).map_or("?", |e| e.gloss.as_str());
|
|
||||||
match &f.value {
|
|
||||||
FeatValue::Grade(g) => format!("{name}={}", g.get()),
|
|
||||||
FeatValue::Option(l) => format!("{name}={l}"),
|
|
||||||
FeatValue::Flag(b) => format!("{name}={b}"),
|
|
||||||
}
|
|
||||||
};
|
|
||||||
for s in SpeakerStateStore::open(&cli.speaker_state)?.all_scenes()? {
|
|
||||||
let who = match &s.annotator {
|
|
||||||
Annotator::Operator => "Operator".to_string(),
|
|
||||||
Annotator::Model { family, generator } => format!("Model {family:?} {generator}"),
|
|
||||||
};
|
|
||||||
let people: Vec<&str> = s.participants.iter().map(|p| p.as_str()).collect();
|
|
||||||
println!(
|
|
||||||
"scene:{}@{} rev:{} {who} [{}]",
|
|
||||||
s.group,
|
|
||||||
s.n,
|
|
||||||
s.inventory_rev,
|
|
||||||
people.join(",")
|
|
||||||
);
|
|
||||||
for r in &s.relations {
|
|
||||||
let to = match &r.to {
|
|
||||||
Party::Named(e) => e.as_str().to_string(),
|
|
||||||
Party::Unnamed => "unnamed".to_string(),
|
|
||||||
};
|
|
||||||
let feats: Vec<String> = r.features.iter().map(&gloss).collect();
|
|
||||||
println!(" {}->{to} {}", r.from, feats.join(" "));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Inventory(InventoryCommand::Dao) => {
|
Command::Inventory(InventoryCommand::Dao) => {
|
||||||
let (rev, entries) = weksa::daodejing::revision();
|
let (rev, entries) = weksa::daodejing::revision();
|
||||||
let count = entries.len();
|
let count = entries.len();
|
||||||
|
|
@ -1117,8 +627,14 @@ fn main() -> Result<()> {
|
||||||
.map_err(refused)?;
|
.map_err(refused)?;
|
||||||
println!("admitted {key}");
|
println!("admitted {key}");
|
||||||
}
|
}
|
||||||
Command::Pronounce { store, key, phones } => {
|
Command::Lower {
|
||||||
|
store,
|
||||||
|
key,
|
||||||
|
ipa,
|
||||||
|
phones,
|
||||||
|
} => {
|
||||||
let inventory_store = InventoryStore::open(&cli.inventory)?;
|
let inventory_store = InventoryStore::open(&cli.inventory)?;
|
||||||
|
let inventory = inventory_store.inventory()?;
|
||||||
let mut packets = PacketStore::open(&store)?;
|
let mut packets = PacketStore::open(&store)?;
|
||||||
let packet = packets.get(&key)?.context("no such packet")?;
|
let packet = packets.get(&key)?.context("no such packet")?;
|
||||||
// Admission is the only source of a LoweringInput; re-admitting the
|
// Admission is the only source of a LoweringInput; re-admitting the
|
||||||
|
|
@ -1132,14 +648,16 @@ fn main() -> Result<()> {
|
||||||
)
|
)
|
||||||
.map_err(refused)?;
|
.map_err(refused)?;
|
||||||
let entities = EntityStore::open(&cli.entities)?;
|
let entities = EntityStore::open(&cli.entities)?;
|
||||||
if phones {
|
if ipa {
|
||||||
|
for line in pronounce(&view, &entities)? {
|
||||||
|
println!("{line}");
|
||||||
|
}
|
||||||
|
} else if phones {
|
||||||
for line in weksa::phones(&view, &entities)? {
|
for line in weksa::phones(&view, &entities)? {
|
||||||
println!("{}", serde_json::to_string(&line)?);
|
println!("{}", serde_json::to_string(&line)?);
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
for line in pronounce(&view, &entities)? {
|
println!("{}", lower(&view, &inventory, &entities)?);
|
||||||
println!("{line}");
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Command::Score(args) => {
|
Command::Score(args) => {
|
||||||
|
|
@ -1174,281 +692,6 @@ fn main() -> Result<()> {
|
||||||
.context("no such item")?;
|
.context("no such item")?;
|
||||||
println!("{}", split_text(&item));
|
println!("{}", split_text(&item));
|
||||||
}
|
}
|
||||||
Command::Loop(LoopCommand::Open {
|
|
||||||
root,
|
|
||||||
label,
|
|
||||||
answerer,
|
|
||||||
embed_model,
|
|
||||||
lowerer_reach,
|
|
||||||
answerer_family,
|
|
||||||
panel,
|
|
||||||
}) => {
|
|
||||||
let run = RunStores::create(
|
|
||||||
&root,
|
|
||||||
&label,
|
|
||||||
RunPaths {
|
|
||||||
corpus: &cli.corpus,
|
|
||||||
inventory: &cli.inventory,
|
|
||||||
entities: &cli.entities,
|
|
||||||
ontology: &cli.ontology,
|
|
||||||
speaker_state: &cli.speaker_state,
|
|
||||||
},
|
|
||||||
&answerer,
|
|
||||||
&embed_model,
|
|
||||||
lowerer_reach,
|
|
||||||
answerer_family.into(),
|
|
||||||
&panel,
|
|
||||||
)?;
|
|
||||||
println!("opened run:{}", run.record().label);
|
|
||||||
}
|
|
||||||
Command::Loop(LoopCommand::Round { root, ollama_url }) => {
|
|
||||||
let unreadable = |_| anyhow::anyhow!("a store would not read");
|
|
||||||
let mut run = RunStores::open(&root)?;
|
|
||||||
let mut files = run.answer_files(run.next_round_dir().map_err(unreadable)?)?;
|
|
||||||
let scorer = Scorer::new(OllamaEmbedder::new(&ollama_url, &run.record().embedder));
|
|
||||||
match round(&mut run, &mut files, &scorer)? {
|
|
||||||
RoundOutcome::Unanswered { prompts } => {
|
|
||||||
for digest in &prompts {
|
|
||||||
println!("unanswered {digest}");
|
|
||||||
}
|
|
||||||
let asked = run.offers().map_err(unreadable)?.len();
|
|
||||||
println!("unanswered {} of {asked}", prompts.len());
|
|
||||||
std::process::exit(3);
|
|
||||||
}
|
|
||||||
RoundOutcome::Recorded(r) => println!("{}", run.round_line(&r)),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Loop(LoopCommand::Ablate {
|
|
||||||
root,
|
|
||||||
round,
|
|
||||||
ollama_url,
|
|
||||||
}) => {
|
|
||||||
let run = RunStores::open(&root)?;
|
|
||||||
let mut recorded = RecordedAnswers::new(
|
|
||||||
root.join(format!("round-{round}")),
|
|
||||||
run.record().model.clone(),
|
|
||||||
);
|
|
||||||
let mut files = run.answer_files(run.ablation_dir(round))?;
|
|
||||||
let scorer = Scorer::new(OllamaEmbedder::new(&ollama_url, &run.record().embedder));
|
|
||||||
match run.stance_ablation(round, &mut recorded, &mut files, &scorer)? {
|
|
||||||
AblationOutcome::Unanswered { prompts } => {
|
|
||||||
for digest in &prompts {
|
|
||||||
println!("unanswered {digest}");
|
|
||||||
}
|
|
||||||
println!("unanswered {} prompts", prompts.len());
|
|
||||||
std::process::exit(3);
|
|
||||||
}
|
|
||||||
AblationOutcome::Read(readings) => {
|
|
||||||
for (language, split, reading) in readings {
|
|
||||||
let name = match split {
|
|
||||||
Split::Train => "train",
|
|
||||||
_ => "validation",
|
|
||||||
};
|
|
||||||
println!("{language} {name} {}", reading.line());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Loop(LoopCommand::Relower {
|
|
||||||
root,
|
|
||||||
round,
|
|
||||||
arm,
|
|
||||||
lowerer,
|
|
||||||
floor,
|
|
||||||
ollama_url,
|
|
||||||
}) => {
|
|
||||||
if !is_label(&arm)
|
|
||||||
|| floor.iter().any(|f| !is_label(f) || *f == arm)
|
|
||||||
|| floor.iter().enumerate().any(|(i, f)| floor[..i].contains(f))
|
|
||||||
{
|
|
||||||
anyhow::bail!("arm and floor labels must match [a-z0-9-]{{1,32}} and differ");
|
|
||||||
}
|
|
||||||
let run = RunStores::open(&root)?;
|
|
||||||
let mut recorded = RecordedAnswers::new(
|
|
||||||
root.join(format!("round-{round}")),
|
|
||||||
run.record().model.clone(),
|
|
||||||
);
|
|
||||||
let mut files = run.answer_files(run.relower_dir(round, &arm))?;
|
|
||||||
let mut floor_ports: Vec<RecordedAnswers> = floor
|
|
||||||
.iter()
|
|
||||||
.map(|f| RecordedAnswers::new(run.relower_dir(round, f), run.record().model.clone()))
|
|
||||||
.collect();
|
|
||||||
let mut floor_dyn: Vec<&mut dyn ModelPort> =
|
|
||||||
floor_ports.iter_mut().map(|p| p as &mut dyn ModelPort).collect();
|
|
||||||
let scorer = Scorer::new(OllamaEmbedder::new(&ollama_url, &run.record().embedder));
|
|
||||||
match run.relower(
|
|
||||||
round,
|
|
||||||
&arm,
|
|
||||||
&mut recorded,
|
|
||||||
&mut files,
|
|
||||||
&mut floor_dyn,
|
|
||||||
&scorer,
|
|
||||||
)? {
|
|
||||||
RelowerOutcome::Unanswered { prompts } => {
|
|
||||||
for digest in &prompts {
|
|
||||||
println!("unanswered {digest}");
|
|
||||||
}
|
|
||||||
println!("unanswered {} prompts", prompts.len());
|
|
||||||
std::process::exit(3);
|
|
||||||
}
|
|
||||||
RelowerOutcome::Read { languages, verdict } => {
|
|
||||||
for l in languages {
|
|
||||||
println!(
|
|
||||||
"{}",
|
|
||||||
run.labelled(&format!("{} arm={arm} lowerer={lowerer}", l.line()))
|
|
||||||
);
|
|
||||||
}
|
|
||||||
if let Some(v) = verdict {
|
|
||||||
println!("verdict {v:?}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Loop(LoopCommand::Calibrate {
|
|
||||||
root,
|
|
||||||
round,
|
|
||||||
prompted,
|
|
||||||
degraded,
|
|
||||||
candidate,
|
|
||||||
class,
|
|
||||||
ollama_url,
|
|
||||||
}) => {
|
|
||||||
let labels: Vec<&String> = prompted.iter().chain(°raded).chain(&candidate).collect();
|
|
||||||
if labels.iter().any(|l| !is_label(l))
|
|
||||||
|| labels.iter().enumerate().any(|(i, l)| labels[..i].contains(l))
|
|
||||||
{
|
|
||||||
anyhow::bail!("draw labels must match [a-z0-9-]{{1,32}} and differ");
|
|
||||||
}
|
|
||||||
if prompted.len() < FLOOR_ARMS_MIN + 1 {
|
|
||||||
anyhow::bail!("at least {} --prompted labels are needed", FLOOR_ARMS_MIN + 1);
|
|
||||||
}
|
|
||||||
let mut classes: Vec<(String, Vec<String>)> = Vec::new();
|
|
||||||
for c in &class {
|
|
||||||
let Some((name, prefixes)) = c.split_once('=') else {
|
|
||||||
anyhow::bail!("--class is NAME=PREFIX[,PREFIX...]");
|
|
||||||
};
|
|
||||||
if !is_label(name)
|
|
||||||
|| name == "all"
|
|
||||||
|| classes.iter().any(|(n, _)| n == name)
|
|
||||||
|| prefixes.split(',').any(str::is_empty)
|
|
||||||
{
|
|
||||||
anyhow::bail!("--class is NAME=PREFIX[,PREFIX...] with a new label NAME other than all");
|
|
||||||
}
|
|
||||||
classes.push((name.to_string(), prefixes.split(',').map(String::from).collect()));
|
|
||||||
}
|
|
||||||
let run = RunStores::open(&root)?;
|
|
||||||
let scorer = Scorer::new(OllamaEmbedder::new(&ollama_url, &run.record().embedder));
|
|
||||||
match run.calibrate(round, &prompted, °raded, &candidate, &classes, &scorer)? {
|
|
||||||
CalibrateOutcome::Unanswered(missing) => {
|
|
||||||
for (label, n) in &missing {
|
|
||||||
println!("unanswered {label} {n}");
|
|
||||||
}
|
|
||||||
eprintln!(
|
|
||||||
"answer it through weksa loop relower ROOT --round N --arm LABEL first"
|
|
||||||
);
|
|
||||||
std::process::exit(1);
|
|
||||||
}
|
|
||||||
CalibrateOutcome::Read { unusable, classes } => {
|
|
||||||
println!(
|
|
||||||
"{}",
|
|
||||||
run.labelled(&format!(
|
|
||||||
"calibrate draws prompted={} degraded={} candidate={} floor-arms={FLOOR_ARMS_MIN} unusable={unusable}",
|
|
||||||
prompted.len(),
|
|
||||||
degraded.len(),
|
|
||||||
candidate.len(),
|
|
||||||
))
|
|
||||||
);
|
|
||||||
for c in classes {
|
|
||||||
let items: String = c
|
|
||||||
.languages
|
|
||||||
.iter()
|
|
||||||
.map(|l| format!(" {} n={} skipped={}", l.language, l.n, l.skipped))
|
|
||||||
.collect();
|
|
||||||
let t = &c.trials;
|
|
||||||
println!("{}", run.labelled(&format!("calibrate class={} items{items}", c.name)));
|
|
||||||
println!(
|
|
||||||
"{}",
|
|
||||||
run.labelled(&format!(
|
|
||||||
"calibrate class={} prompted trials={} nogain={} accepted={} regressed={} noise={:.4}",
|
|
||||||
c.name, t.prompted.trials, t.prompted.nogain, t.prompted.accepted, t.prompted.regressed, t.noise
|
|
||||||
))
|
|
||||||
);
|
|
||||||
for (side, k) in t.r_roles() {
|
|
||||||
println!(
|
|
||||||
"{}",
|
|
||||||
run.labelled(&format!("calibrate class={} prompted-{side} {}", c.name, k.line()))
|
|
||||||
);
|
|
||||||
}
|
|
||||||
for line in c.degraded_lines(°raded) {
|
|
||||||
println!("{}", run.labelled(&format!("calibrate class={} {line}", c.name)));
|
|
||||||
}
|
|
||||||
println!(
|
|
||||||
"{}",
|
|
||||||
run.labelled(&format!(
|
|
||||||
"calibrate class={} candidate trials={} accepted={} nogain={} regressed={}",
|
|
||||||
c.name, t.candidate.trials, t.candidate.accepted, t.candidate.nogain, t.candidate.regressed
|
|
||||||
))
|
|
||||||
);
|
|
||||||
for (label, k) in candidate.iter().zip(&t.candidate_arms) {
|
|
||||||
println!(
|
|
||||||
"{}",
|
|
||||||
run.labelled(&format!("calibrate class={} candidate={label} {}", c.name, k.line()))
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Loop(LoopCommand::Recall { root }) => {
|
|
||||||
let run = RunStores::open(&root)?;
|
|
||||||
let mut files = run.answer_files(run.recall_dir())?;
|
|
||||||
match run.recall(&mut files)? {
|
|
||||||
RecallOutcome::Unanswered { prompts } => {
|
|
||||||
for digest in &prompts {
|
|
||||||
println!("unanswered {digest}");
|
|
||||||
}
|
|
||||||
println!("unanswered {} prompts", prompts.len());
|
|
||||||
std::process::exit(3);
|
|
||||||
}
|
|
||||||
RecallOutcome::Read(reading) => println!("{}", reading.line()),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Loop(LoopCommand::MintOntology { root, language }) => {
|
|
||||||
let run = RunStores::open(&root)?;
|
|
||||||
let mut files = run.answer_files(run.ontology_mint_dir())?;
|
|
||||||
match run.mint_ontology(&mut files, &language)? {
|
|
||||||
OntologyMintOutcome::Unanswered { prompt } => {
|
|
||||||
println!("unanswered {prompt}");
|
|
||||||
std::process::exit(3);
|
|
||||||
}
|
|
||||||
OntologyMintOutcome::Drafted { prompt, entries } => println!(
|
|
||||||
"drafted {entries} entries {}",
|
|
||||||
run.ontology_mint_dir().join(format!("{prompt}.answer")).display()
|
|
||||||
),
|
|
||||||
OntologyMintOutcome::Refused { code } => {
|
|
||||||
println!("{code}");
|
|
||||||
std::process::exit(1);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Loop(LoopCommand::Cycle { root, ollama_url }) => {
|
|
||||||
let unreadable = |_| anyhow::anyhow!("a store would not read");
|
|
||||||
let mut run = RunStores::open(&root)?;
|
|
||||||
let mut files = run.answer_files(run.next_cycle_dir().map_err(unreadable)?)?;
|
|
||||||
let scorer = Scorer::new(OllamaEmbedder::new(&ollama_url, &run.record().embedder));
|
|
||||||
match cycle(&mut run, &mut files, &scorer)? {
|
|
||||||
CycleOutcome::Unanswered { prompts } => {
|
|
||||||
for digest in &prompts {
|
|
||||||
println!("unanswered {digest}");
|
|
||||||
}
|
|
||||||
println!("unanswered {} prompts", prompts.len());
|
|
||||||
std::process::exit(3);
|
|
||||||
}
|
|
||||||
CycleOutcome::Decided(decision) => {
|
|
||||||
println!("{}", run.cycle_line(&decision))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Command::Schema { out } => {
|
Command::Schema { out } => {
|
||||||
std::fs::create_dir_all(&out)?;
|
std::fs::create_dir_all(&out)?;
|
||||||
for (name, json) in weksa::schema::published() {
|
for (name, json) in weksa::schema::published() {
|
||||||
|
|
|
||||||
|
|
@ -1,130 +0,0 @@
|
||||||
//! The loop's only door to a model. A model is the Claude session: it reads a
|
|
||||||
//! prompt file and writes an answer file, and a rerun picks the answer up. No
|
|
||||||
//! subprocess, network or binary lives here (golf-loop-in-claude).
|
|
||||||
|
|
||||||
use std::fmt;
|
|
||||||
use std::io::ErrorKind;
|
|
||||||
use std::path::PathBuf;
|
|
||||||
|
|
||||||
use crate::context::Prompt;
|
|
||||||
use crate::store::normalized;
|
|
||||||
|
|
||||||
/// A model answering [`Prompt`]s and nothing else.
|
|
||||||
pub trait ModelPort {
|
|
||||||
/// The id of the answering model, pinned by the run.
|
|
||||||
fn model(&self) -> &str;
|
|
||||||
fn answer(&mut self, prompt: &Prompt) -> Result<String, PortError>;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Why a prompt got no answer. Display names the kind only.
|
|
||||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
|
||||||
pub enum PortError {
|
|
||||||
/// Nobody has answered yet.
|
|
||||||
Unanswered,
|
|
||||||
/// The prompt holds the text of an item in a Test group. It is neither written nor answered.
|
|
||||||
Withheld,
|
|
||||||
Io(ErrorKind),
|
|
||||||
}
|
|
||||||
|
|
||||||
impl fmt::Display for PortError {
|
|
||||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
|
||||||
match self {
|
|
||||||
PortError::Unanswered => f.write_str("the prompt is unanswered"),
|
|
||||||
PortError::Withheld => f.write_str("the prompt holds Test item text"),
|
|
||||||
PortError::Io(kind) => write!(f, "io error {kind:?}"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl std::error::Error for PortError {}
|
|
||||||
|
|
||||||
/// Answers from files in one directory: `<digest>.answer` is the reply to the
|
|
||||||
/// prompt whose digest is `<digest>`; a prompt with no answer is written to
|
|
||||||
/// `<digest>.prompt` for whoever answers.
|
|
||||||
pub struct AnswerFiles {
|
|
||||||
dir: PathBuf,
|
|
||||||
model: String,
|
|
||||||
withheld: Vec<String>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl AnswerFiles {
|
|
||||||
/// Refuses (`PortError::Withheld`) any prompt that holds the text of one of
|
|
||||||
/// `withheld`, before it reads or writes a file: the one door a prompt
|
|
||||||
/// leaves the loop through, so a Test sentence cannot reach a model by it.
|
|
||||||
pub fn new(dir: PathBuf, model: String, withheld: Vec<String>) -> Self {
|
|
||||||
Self { dir, model, withheld }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Normalised length from which a Test text is matched. Every live Test text is
|
|
||||||
/// longer; a shorter one ("Stop.") would match any prompt that says "Stop. Then
|
|
||||||
/// ...", so a Test item that short is not caught here.
|
|
||||||
const LONG: usize = 12;
|
|
||||||
|
|
||||||
/// Whether `prompt` holds `text`, by the corpus door's own sameness
|
|
||||||
/// ([`normalized`]: NFKC, case fold, spacing and punctuation dropped): a
|
|
||||||
/// re-cased, re-spaced, wrapped or width-changed copy still counts.
|
|
||||||
fn holds(prompt: &str, text: &str) -> bool {
|
|
||||||
let key = normalized(text);
|
|
||||||
key.chars().count() >= LONG && normalized(prompt).contains(&key)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Answers read from one directory and nothing else: `<digest>.answer` is the
|
|
||||||
/// reply to the prompt whose digest is `<digest>`. A prompt with no file is
|
|
||||||
/// [`PortError::Unanswered`]; this port never creates or writes anything. The
|
|
||||||
/// one reader of answer files: [`AnswerFiles`] reads through it.
|
|
||||||
pub struct RecordedAnswers {
|
|
||||||
dir: PathBuf,
|
|
||||||
model: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl RecordedAnswers {
|
|
||||||
pub fn new(dir: PathBuf, model: String) -> Self {
|
|
||||||
Self { dir, model }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl ModelPort for RecordedAnswers {
|
|
||||||
fn model(&self) -> &str {
|
|
||||||
&self.model
|
|
||||||
}
|
|
||||||
|
|
||||||
fn answer(&mut self, prompt: &Prompt) -> Result<String, PortError> {
|
|
||||||
match std::fs::read(self.dir.join(format!("{}.answer", prompt.digest()))) {
|
|
||||||
Ok(bytes) => {
|
|
||||||
String::from_utf8(bytes).map_err(|_| PortError::Io(ErrorKind::InvalidData))
|
|
||||||
}
|
|
||||||
Err(e) if e.kind() == ErrorKind::NotFound => Err(PortError::Unanswered),
|
|
||||||
Err(e) => Err(PortError::Io(e.kind())),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl ModelPort for AnswerFiles {
|
|
||||||
fn model(&self) -> &str {
|
|
||||||
&self.model
|
|
||||||
}
|
|
||||||
|
|
||||||
fn answer(&mut self, prompt: &Prompt) -> Result<String, PortError> {
|
|
||||||
let io = |e: std::io::Error| PortError::Io(e.kind());
|
|
||||||
if self.withheld.iter().any(|t| holds(prompt.text(), t)) {
|
|
||||||
return Err(PortError::Withheld);
|
|
||||||
}
|
|
||||||
let digest = prompt.digest();
|
|
||||||
match RecordedAnswers::new(self.dir.clone(), self.model.clone()).answer(prompt) {
|
|
||||||
Err(PortError::Unanswered) => {}
|
|
||||||
read => return read,
|
|
||||||
}
|
|
||||||
std::fs::create_dir_all(&self.dir).map_err(io)?;
|
|
||||||
let path = self.dir.join(format!("{digest}.prompt"));
|
|
||||||
match std::fs::read(&path) {
|
|
||||||
Ok(held) if held == prompt.text().as_bytes() => {}
|
|
||||||
Ok(_) => return Err(PortError::Io(ErrorKind::AlreadyExists)),
|
|
||||||
Err(e) if e.kind() == ErrorKind::NotFound => {
|
|
||||||
std::fs::write(&path, prompt.text()).map_err(io)?
|
|
||||||
}
|
|
||||||
Err(e) => return Err(io(e)),
|
|
||||||
}
|
|
||||||
Err(PortError::Unanswered)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,207 +0,0 @@
|
||||||
//! The language ontology: per target language, the choices its culture makes
|
|
||||||
//! about word, grammar, discourse, sound and script for a speaker in a stance
|
|
||||||
//! (ruling `persona-drives-ontology`). An entry says when it applies (feature
|
|
||||||
//! conditions over the lowering view's stance, discourse and scene relation) and what it
|
|
||||||
//! chooses. `select` is the only reader that decides which entries a lowering
|
|
||||||
//! prompt carries, and it runs only inside `Prompt::lower`;
|
|
||||||
//! `OntologyStore::admit` is the only writer. Entries are immutable.
|
|
||||||
|
|
||||||
use cultcache_rs::DatabaseEntry;
|
|
||||||
use schemars::JsonSchema;
|
|
||||||
use serde::{Deserialize, Serialize};
|
|
||||||
|
|
||||||
use crate::inventory::{InvId, Slot};
|
|
||||||
use crate::packet::{FeatValue, FeatureValue, Grade};
|
|
||||||
use crate::speaker::Annotator;
|
|
||||||
use crate::validate::LoweringInput;
|
|
||||||
|
|
||||||
pub const MAX_CONDITIONS: usize = 4;
|
|
||||||
pub const MAX_EXAMPLES: usize = 3;
|
|
||||||
pub const MAX_CHOICE_BYTES: usize = 200;
|
|
||||||
pub const MAX_EXAMPLE_BYTES: usize = 120;
|
|
||||||
pub const MAX_ENTRIES_PER_LANGUAGE: usize = 64;
|
|
||||||
|
|
||||||
/// Which part of the language an entry shapes.
|
|
||||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
|
||||||
pub enum Layer {
|
|
||||||
Lexicon,
|
|
||||||
Grammar,
|
|
||||||
Discourse,
|
|
||||||
Phonology,
|
|
||||||
Orthography,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Layer {
|
|
||||||
pub fn name(self) -> &'static str {
|
|
||||||
match self {
|
|
||||||
Layer::Lexicon => "lexicon",
|
|
||||||
Layer::Grammar => "grammar",
|
|
||||||
Layer::Discourse => "discourse",
|
|
||||||
Layer::Phonology => "phonology",
|
|
||||||
Layer::Orthography => "orthography",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// What a feature's value must be for a condition to hold.
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
|
||||||
pub enum Test {
|
|
||||||
AtLeast(Grade),
|
|
||||||
AtMost(Grade),
|
|
||||||
Is(FeatValue),
|
|
||||||
/// Holds when the feature is absent or a grade at most `g`: a bound that
|
|
||||||
/// costs nothing where the feature is unrated. `AtMost` still fails on an
|
|
||||||
/// absent feature.
|
|
||||||
AtMostOrUnset(Grade),
|
|
||||||
}
|
|
||||||
|
|
||||||
/// One feature of one slot of the view, tested. `Stance`, `Discourse` and
|
|
||||||
/// `Scene` reach a lowering view; `Speaker` does not yet.
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
|
||||||
pub struct Condition {
|
|
||||||
pub slot: Slot,
|
|
||||||
pub feature: InvId,
|
|
||||||
pub test: Test,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Condition {
|
|
||||||
/// Whether this condition holds over a slot's feature values: the one
|
|
||||||
/// evaluator `select` and the corpus tests share.
|
|
||||||
pub fn holds(&self, features: &[FeatureValue]) -> bool {
|
|
||||||
let Some(found) = features.iter().find(|f| f.feature == self.feature) else {
|
|
||||||
return matches!(self.test, Test::AtMostOrUnset(_));
|
|
||||||
};
|
|
||||||
match (&self.test, &found.value) {
|
|
||||||
(Test::AtLeast(g), FeatValue::Grade(v)) => v >= g,
|
|
||||||
(Test::AtMost(g) | Test::AtMostOrUnset(g), FeatValue::Grade(v)) => v <= g,
|
|
||||||
(Test::Is(want), v) => want == v,
|
|
||||||
_ => false,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The feature value this condition names, as the inventory check reads it.
|
|
||||||
pub(crate) fn as_value(&self) -> FeatureValue {
|
|
||||||
FeatureValue {
|
|
||||||
feature: self.feature,
|
|
||||||
value: match &self.test {
|
|
||||||
Test::AtLeast(g) | Test::AtMost(g) | Test::AtMostOrUnset(g) => FeatValue::Grade(*g),
|
|
||||||
Test::Is(v) => v.clone(),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// One choice of one language. Key `onto:<language>:<n>`.
|
|
||||||
///
|
|
||||||
/// The store judges the form of an entry's text (bounds, normalization, no
|
|
||||||
/// control or format character) and never its meaning: no door judges text
|
|
||||||
/// (ruling `leak-owned-by-flow-and-scorer-3`). No Train-only ontology mint
|
|
||||||
/// exists yet, so a seed or a drafted entry is checked against held-out text by
|
|
||||||
/// nothing today; follow-up `leak-diagnostic-in-round` owns that diagnostic.
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, JsonSchema, DatabaseEntry)]
|
|
||||||
#[cultcache(type = "weksa.ontology_entry")]
|
|
||||||
pub struct OntologyEntry {
|
|
||||||
#[cultcache(key = 0)]
|
|
||||||
pub language: String,
|
|
||||||
/// The entry's number within its language, from 1.
|
|
||||||
#[cultcache(key = 1)]
|
|
||||||
pub n: u32,
|
|
||||||
/// The revision, of the inventory the entry was admitted through, at which the
|
|
||||||
/// store read its conditions: provenance only; a run reads the conditions'
|
|
||||||
/// features at its own head.
|
|
||||||
#[cultcache(key = 2)]
|
|
||||||
pub inventory_rev: u32,
|
|
||||||
#[cultcache(key = 3)]
|
|
||||||
pub layer: Layer,
|
|
||||||
/// Every condition must hold; none means the entry always applies.
|
|
||||||
#[cultcache(key = 4)]
|
|
||||||
pub when: Vec<Condition>,
|
|
||||||
#[cultcache(key = 5)]
|
|
||||||
pub choice: String,
|
|
||||||
#[cultcache(key = 6)]
|
|
||||||
pub examples: Vec<String>,
|
|
||||||
#[cultcache(key = 7)]
|
|
||||||
pub annotator: Annotator,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `onto:<language>:<n>`.
|
|
||||||
pub fn ontology_key(language: &str, n: u32) -> String {
|
|
||||||
format!("onto:{language}:{n}")
|
|
||||||
}
|
|
||||||
|
|
||||||
impl OntologyEntry {
|
|
||||||
pub fn key(&self) -> String {
|
|
||||||
ontology_key(&self.language, self.n)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Every entry of the run's ontology store, by language then number.
|
|
||||||
#[derive(Clone, Debug, Default, PartialEq, Eq)]
|
|
||||||
pub struct Ontology {
|
|
||||||
entries: Vec<OntologyEntry>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Ontology {
|
|
||||||
pub fn empty() -> Self {
|
|
||||||
Self::default()
|
|
||||||
}
|
|
||||||
|
|
||||||
pub(crate) fn of(mut entries: Vec<OntologyEntry>) -> Self {
|
|
||||||
entries.sort_by(|a, b| (&a.language, a.n).cmp(&(&b.language, b.n)));
|
|
||||||
Self { entries }
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn entries(&self) -> &[OntologyEntry] {
|
|
||||||
&self.entries
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The entries of exactly `language` whose every condition holds against the
|
|
||||||
/// view's stance, discourse and scene-relation features (the scene's only when
|
|
||||||
/// `with_scene` selected a relation), in `n` order. An absent feature meets no
|
|
||||||
/// condition except `AtMostOrUnset`. Nothing else of the view is read.
|
|
||||||
pub fn select<'a>(
|
|
||||||
o: &'a Ontology,
|
|
||||||
language: &str,
|
|
||||||
input: &LoweringInput,
|
|
||||||
) -> Vec<&'a OntologyEntry> {
|
|
||||||
o.entries
|
|
||||||
.iter()
|
|
||||||
.filter(|e| e.language == language)
|
|
||||||
.filter(|e| {
|
|
||||||
e.when.iter().all(|c| match c.slot {
|
|
||||||
Slot::Stance => c.holds(&input.stance().features),
|
|
||||||
Slot::Discourse => c.holds(&input.discourse().features),
|
|
||||||
Slot::Scene => c.holds(input.scene()),
|
|
||||||
_ => false,
|
|
||||||
})
|
|
||||||
})
|
|
||||||
.collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
fn cond(test: Test) -> Condition {
|
|
||||||
Condition { slot: Slot::Discourse, feature: InvId::parse("feature:7").unwrap(), test }
|
|
||||||
}
|
|
||||||
fn has(v: FeatValue) -> Vec<FeatureValue> {
|
|
||||||
vec![FeatureValue { feature: InvId::parse("feature:7").unwrap(), value: v }]
|
|
||||||
}
|
|
||||||
fn grade(g: u16) -> Grade {
|
|
||||||
Grade::new(g).unwrap()
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn at_most_or_unset_holds_when_absent_or_at_most_and_nothing_else() {
|
|
||||||
let c = cond(Test::AtMostOrUnset(grade(799)));
|
|
||||||
assert!(c.holds(&[]));
|
|
||||||
assert!(c.holds(&has(FeatValue::Grade(grade(799)))));
|
|
||||||
assert!(c.holds(&has(FeatValue::Grade(grade(0)))));
|
|
||||||
assert!(!c.holds(&has(FeatValue::Grade(grade(800)))));
|
|
||||||
assert!(!c.holds(&has(FeatValue::Flag(true))));
|
|
||||||
// AtMost keeps its meaning: an absent feature fails it.
|
|
||||||
assert!(!cond(Test::AtMost(grade(799))).holds(&[]));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,21 +0,0 @@
|
||||||
//! Pins for private functions of `run.rs` that the doors reach only through a
|
|
||||||
//! whole run.
|
|
||||||
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
/// A line is probed only when it has at least four words, and then it is
|
|
||||||
/// split in two near its byte middle, the pieces trimmed.
|
|
||||||
#[test]
|
|
||||||
fn a_recall_line_splits_from_four_words_and_not_before() {
|
|
||||||
assert_eq!(recall_split("one two three"), None);
|
|
||||||
assert_eq!(recall_split(""), None);
|
|
||||||
assert_eq!(
|
|
||||||
recall_split("one two three four"),
|
|
||||||
Some(("one two".to_string(), "three four".to_string()))
|
|
||||||
);
|
|
||||||
// No spaces: each ideograph is a word, the split falls at the middle.
|
|
||||||
assert_eq!(
|
|
||||||
recall_split("天地不仁以万物为刍狗"),
|
|
||||||
Some(("天地不仁以".to_string(), "万物为刍狗".to_string()))
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
@ -4,10 +4,7 @@
|
||||||
use crate::corpus::CorpusItem;
|
use crate::corpus::CorpusItem;
|
||||||
use crate::entity::Entity;
|
use crate::entity::Entity;
|
||||||
use crate::inventory::{InventoryEntry, InventoryRevision};
|
use crate::inventory::{InventoryEntry, InventoryRevision};
|
||||||
use crate::ontology::OntologyEntry;
|
|
||||||
use crate::packet::Packet;
|
use crate::packet::Packet;
|
||||||
use crate::run::{Decision, RoundRecord, RunRecord};
|
|
||||||
use crate::speaker::{Scene, Speaker};
|
|
||||||
use cultcache_rs::DatabaseEntry;
|
use cultcache_rs::DatabaseEntry;
|
||||||
use schemars::{JsonSchema, schema_for};
|
use schemars::{JsonSchema, schema_for};
|
||||||
|
|
||||||
|
|
@ -24,11 +21,5 @@ pub fn published() -> Vec<(String, String)> {
|
||||||
published_for::<InventoryRevision>(),
|
published_for::<InventoryRevision>(),
|
||||||
published_for::<CorpusItem>(),
|
published_for::<CorpusItem>(),
|
||||||
published_for::<Entity>(),
|
published_for::<Entity>(),
|
||||||
published_for::<Speaker>(),
|
|
||||||
published_for::<Scene>(),
|
|
||||||
published_for::<OntologyEntry>(),
|
|
||||||
published_for::<RunRecord>(),
|
|
||||||
published_for::<RoundRecord>(),
|
|
||||||
published_for::<Decision>(),
|
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -12,8 +12,6 @@ mod embed;
|
||||||
mod words;
|
mod words;
|
||||||
|
|
||||||
use anyhow::{Result, ensure};
|
use anyhow::{Result, ensure};
|
||||||
use schemars::JsonSchema;
|
|
||||||
use serde::{Deserialize, Serialize};
|
|
||||||
|
|
||||||
pub use chrf::{CHRF_VERSION, chrf};
|
pub use chrf::{CHRF_VERSION, chrf};
|
||||||
pub use diagnostics::{OrderCoverage, PairChrf, SplitLeak, WorstItem, leak_diagnostic};
|
pub use diagnostics::{OrderCoverage, PairChrf, SplitLeak, WorstItem, leak_diagnostic};
|
||||||
|
|
@ -28,7 +26,7 @@ pub struct ScoreInput<'a> {
|
||||||
pub output: &'a str,
|
pub output: &'a str,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, JsonSchema)]
|
#[derive(Clone, Debug, PartialEq)]
|
||||||
pub struct ItemScore {
|
pub struct ItemScore {
|
||||||
/// chrF of the output against the reference.
|
/// chrF of the output against the reference.
|
||||||
pub chrf: f64,
|
pub chrf: f64,
|
||||||
|
|
@ -104,7 +102,7 @@ impl<E: Embedder> Scorer<E> {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Mean scores of one derived split.
|
/// Mean scores of one derived split.
|
||||||
#[derive(Clone, Debug, PartialEq, Default, Serialize, Deserialize, JsonSchema)]
|
#[derive(Clone, Debug, PartialEq, Default)]
|
||||||
pub struct SplitMean {
|
pub struct SplitMean {
|
||||||
pub count: usize,
|
pub count: usize,
|
||||||
/// `None` when the split has no items.
|
/// `None` when the split has no items.
|
||||||
|
|
@ -131,18 +129,13 @@ pub struct Aggregate {
|
||||||
pub validation: SplitMean,
|
pub validation: SplitMean,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The scores of the items read in split `want`, in list order.
|
/// The mean of the scores of the items read in split `want`.
|
||||||
fn scores_in<'a>(scored: &'a [(&CorpusItem, ItemScore)], want: Split) -> Vec<&'a ItemScore> {
|
fn split_mean(scored: &[(&CorpusItem, ItemScore)], want: Split) -> SplitMean {
|
||||||
scored
|
let of: Vec<&ItemScore> = scored
|
||||||
.iter()
|
.iter()
|
||||||
.filter(|(item, _)| SplitScheme::V1.live(item) == Some(want))
|
.filter(|(item, _)| SplitScheme::V1.live(item) == Some(want))
|
||||||
.map(|(_, score)| score)
|
.map(|(_, score)| score)
|
||||||
.collect()
|
.collect();
|
||||||
}
|
|
||||||
|
|
||||||
/// The mean of the scores of the items read in split `want`.
|
|
||||||
fn split_mean(scored: &[(&CorpusItem, ItemScore)], want: Split) -> SplitMean {
|
|
||||||
let of = scores_in(scored, want);
|
|
||||||
let average = |values: Vec<f64>| {
|
let average = |values: Vec<f64>| {
|
||||||
(!values.is_empty()).then(|| values.iter().sum::<f64>() / values.len() as f64)
|
(!values.is_empty()).then(|| values.iter().sum::<f64>() / values.len() as f64)
|
||||||
};
|
};
|
||||||
|
|
@ -162,36 +155,40 @@ pub fn aggregate(scored: &[(&CorpusItem, ItemScore)]) -> Aggregate {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The Validation items' scores in list order, numbers only: no key, no text.
|
|
||||||
/// A refused item arrives scored 0 and keeps its position.
|
|
||||||
pub fn validation_items(scored: &[(&CorpusItem, ItemScore)]) -> Vec<ItemScore> {
|
|
||||||
scores_in(scored, Split::Validation).into_iter().cloned().collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The Test split's readings, scalars only: no item key, no minted string, no
|
/// The Test split's readings, scalars only: no item key, no minted string, no
|
||||||
/// pair. The checkpoint decision (convergence-test) needs numbers; a key beside
|
/// pair. The checkpoint decision (convergence-test) needs numbers; a key beside
|
||||||
/// a minted string is the handle an edit step would act on. The only producer
|
/// a minted string is the handle an edit step would act on. The only producer
|
||||||
/// is `at_checkpoint`, which consumes a `Checkpoint`; only the lib can make one.
|
/// is `at_checkpoint`, which consumes a `Checkpoint`.
|
||||||
///
|
///
|
||||||
/// ```
|
/// ```
|
||||||
/// use weksa::score::TestReadings;
|
/// use weksa::score::TestReadings;
|
||||||
/// fn f(t: TestReadings) -> usize { t.count() }
|
/// use weksa::split::Checkpoint;
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// ```compile_fail
|
|
||||||
/// use weksa::score::TestReadings;
|
|
||||||
/// fn f(t: TestReadings) { let _u = t.clone(); }
|
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// ```compile_fail
|
|
||||||
/// use weksa::score::TestReadings;
|
|
||||||
/// fn f(t: TestReadings) { let _ = format!("{t:?}"); }
|
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// ```compile_fail
|
|
||||||
/// use weksa::score::TestReadings;
|
|
||||||
/// fn s<T: serde::Serialize>(_: &T) {}
|
/// fn s<T: serde::Serialize>(_: &T) {}
|
||||||
/// fn f(t: TestReadings) { s(&t); }
|
/// let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &[], &[]);
|
||||||
|
/// assert_eq!(t.count(), 0);
|
||||||
|
/// s(&0u8);
|
||||||
|
/// ```
|
||||||
|
///
|
||||||
|
/// ```compile_fail
|
||||||
|
/// use weksa::score::TestReadings;
|
||||||
|
/// use weksa::split::Checkpoint;
|
||||||
|
/// let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &[], &[]);
|
||||||
|
/// let _u = t.clone();
|
||||||
|
/// ```
|
||||||
|
///
|
||||||
|
/// ```compile_fail
|
||||||
|
/// use weksa::score::TestReadings;
|
||||||
|
/// use weksa::split::Checkpoint;
|
||||||
|
/// let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &[], &[]);
|
||||||
|
/// let _ = format!("{t:?}");
|
||||||
|
/// ```
|
||||||
|
///
|
||||||
|
/// ```compile_fail
|
||||||
|
/// use weksa::score::TestReadings;
|
||||||
|
/// use weksa::split::Checkpoint;
|
||||||
|
/// fn s<T: serde::Serialize>(_: &T) {}
|
||||||
|
/// let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &[], &[]);
|
||||||
|
/// s(&t);
|
||||||
/// ```
|
/// ```
|
||||||
pub struct TestReadings {
|
pub struct TestReadings {
|
||||||
mean: SplitMean,
|
mean: SplitMean,
|
||||||
|
|
@ -243,121 +240,3 @@ impl TestReadings {
|
||||||
&self.leak.coverage_by_n
|
&self.leak.coverage_by_n
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
use crate::split::fixture::{group_in, item};
|
|
||||||
|
|
||||||
const HELD_OUT: &str = "extraordinary lighthouse keepers whispered unforgettable stories throughout midnight thunderstorms before surrendering peacefully";
|
|
||||||
const UNRELATED: &str = "harbor ledgers record ordinary tides";
|
|
||||||
|
|
||||||
fn score(chrf: f64, meaning: Option<f64>) -> ItemScore {
|
|
||||||
ItemScore { chrf, meaning }
|
|
||||||
}
|
|
||||||
|
|
||||||
fn in_split(key: &str, text: &str, split: Split, model: bool) -> CorpusItem {
|
|
||||||
item(key, text, &group_in(split), model)
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn validation_items_keeps_validation_only_in_order() {
|
|
||||||
let train = in_split("item:t:1", UNRELATED, Split::Train, false);
|
|
||||||
let first = in_split("item:v:1", "first words", Split::Validation, false);
|
|
||||||
let refused = in_split("item:v:2", "second words", Split::Validation, false);
|
|
||||||
let last = in_split("item:v:3", "third words", Split::Validation, false);
|
|
||||||
let mut gone = in_split("item:v:4", "withdrawn words", Split::Validation, false);
|
|
||||||
gone.withdrawn = true;
|
|
||||||
let test = in_split("item:c:1", HELD_OUT, Split::Test, false);
|
|
||||||
let scored = [
|
|
||||||
(&train, score(10.0, Some(0.1))),
|
|
||||||
(&first, score(40.0, Some(0.4))),
|
|
||||||
(&test, score(60.0, Some(0.6))),
|
|
||||||
(&refused, score(0.0, Some(0.0))),
|
|
||||||
(&gone, score(90.0, Some(0.9))),
|
|
||||||
(&last, score(80.0, None)),
|
|
||||||
];
|
|
||||||
let got = validation_items(&scored);
|
|
||||||
assert_eq!(
|
|
||||||
got,
|
|
||||||
[score(40.0, Some(0.4)), score(0.0, Some(0.0)), score(80.0, None)]
|
|
||||||
);
|
|
||||||
assert_eq!(got.len(), aggregate(&scored).validation.count);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_readings_at_checkpoint() {
|
|
||||||
let train_item = in_split("item:t:1", UNRELATED, Split::Train, false);
|
|
||||||
let test = in_split("item:c:1", HELD_OUT, Split::Test, false);
|
|
||||||
let test_model = in_split("item:c:2", "model words here", Split::Test, true);
|
|
||||||
let mut gone = in_split("item:c:3", "withdrawn words", Split::Test, false);
|
|
||||||
gone.withdrawn = true;
|
|
||||||
let val = in_split(
|
|
||||||
"item:v:1",
|
|
||||||
"gardening is quiet work",
|
|
||||||
Split::Validation,
|
|
||||||
false,
|
|
||||||
);
|
|
||||||
let items = [
|
|
||||||
train_item.clone(),
|
|
||||||
test.clone(),
|
|
||||||
test_model.clone(),
|
|
||||||
gone.clone(),
|
|
||||||
val.clone(),
|
|
||||||
];
|
|
||||||
let scored = [
|
|
||||||
(&train_item, score(10.0, Some(0.5))),
|
|
||||||
(&val, score(40.0, Some(0.8))),
|
|
||||||
(&test, score(60.0, Some(0.6))),
|
|
||||||
(&test_model, score(100.0, Some(1.0))),
|
|
||||||
(&gone, score(90.0, Some(0.9))),
|
|
||||||
];
|
|
||||||
let t = TestReadings::at_checkpoint(Checkpoint::for_test(2), &scored, &[HELD_OUT], &items);
|
|
||||||
assert_eq!(
|
|
||||||
(t.count(), t.chrf(), t.meaning()),
|
|
||||||
(1, Some(60.0), Some(0.6))
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
t.items(),
|
|
||||||
1,
|
|
||||||
"model-written and withdrawn Test items are excluded"
|
|
||||||
);
|
|
||||||
assert!(t.max_chrf().unwrap() > 99.0);
|
|
||||||
assert_eq!(t.coverage_by_n()[2].share, Some(1.0));
|
|
||||||
|
|
||||||
// A minted string that is not the Test text reads low.
|
|
||||||
let t = TestReadings::at_checkpoint(Checkpoint::for_test(2), &scored, &["zzz"], &items);
|
|
||||||
assert!(t.max_chrf().unwrap() < 20.0);
|
|
||||||
assert_eq!(t.coverage_by_n()[2].share, Some(0.0));
|
|
||||||
|
|
||||||
let empty = TestReadings::at_checkpoint(Checkpoint::for_test(2), &[], &[HELD_OUT], &[]);
|
|
||||||
assert_eq!(
|
|
||||||
(empty.count(), empty.chrf(), empty.meaning()),
|
|
||||||
(0, None, None)
|
|
||||||
);
|
|
||||||
assert_eq!((empty.items(), empty.max_chrf()), (0, None));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn validation_and_test_leak_agree_in_method() {
|
|
||||||
// The same text and minted strings, once in Validation and once in
|
|
||||||
// Test, read the same numbers.
|
|
||||||
let words: Vec<&str> = HELD_OUT.split(' ').collect();
|
|
||||||
let chunks: Vec<String> = words.chunks(3).map(|c| c.join(" ")).collect();
|
|
||||||
let minted: Vec<&str> = chunks.iter().map(String::as_str).collect();
|
|
||||||
let train_item = in_split("item:t:1", UNRELATED, Split::Train, false);
|
|
||||||
let v = leak_diagnostic(
|
|
||||||
&minted,
|
|
||||||
&[
|
|
||||||
in_split("item:v:1", HELD_OUT, Split::Validation, false),
|
|
||||||
train_item.clone(),
|
|
||||||
],
|
|
||||||
);
|
|
||||||
let test = in_split("item:c:1", HELD_OUT, Split::Test, false);
|
|
||||||
let t =
|
|
||||||
TestReadings::at_checkpoint(Checkpoint::for_test(1), &[], &minted, &[test, train_item]);
|
|
||||||
assert_eq!(t.items(), v.items);
|
|
||||||
assert_eq!(t.max_chrf(), v.max_chrf.as_ref().map(|p| p.chrf));
|
|
||||||
assert_eq!(t.coverage_by_n(), v.coverage_by_n.as_slice());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -13,10 +13,8 @@
|
||||||
//! word, never one word per character: per-character words would make every
|
//! word, never one word per character: per-character words would make every
|
||||||
//! honest single-word gloss share bigrams with a held-out sentence, which is
|
//! honest single-word gloss share bigrams with a held-out sentence, which is
|
||||||
//! the leak signature. A change between such a run and any other script also
|
//! the leak signature. A change between such a run and any other script also
|
||||||
//! ends a word. The n >= 2 coverage signature is therefore blind on these
|
//! ends a word. A leak in these scripts shows at n = 1 and in chrF; a
|
||||||
//! scripts: a leak there shows at n = 1 and in max pairwise chrF, which is
|
//! dictionary segmenter is the ja-JP panel's to add.
|
||||||
//! character n-grams and script-blind. A dictionary segmenter is follow-up
|
|
||||||
//! `ja-word-segmenter`.
|
|
||||||
|
|
||||||
use caseless::default_case_fold_str;
|
use caseless::default_case_fold_str;
|
||||||
use unicode_normalization::UnicodeNormalization;
|
use unicode_normalization::UnicodeNormalization;
|
||||||
|
|
|
||||||
|
|
@ -1,134 +0,0 @@
|
||||||
//! The speaker-state documents: a speaker's stable axes and a scene's relations.
|
|
||||||
//! Both are append-only revisions under one key per revision and hold typed
|
|
||||||
//! feature values only, never text (ruling `leak-owned-by-flow-and-scorer-3`).
|
|
||||||
//! `SpeakerStateStore` is their only writer; `LoweringInput::with_scene` is the
|
|
||||||
//! only reader that reaches a lowering view.
|
|
||||||
|
|
||||||
use cultcache_rs::DatabaseEntry;
|
|
||||||
use schemars::JsonSchema;
|
|
||||||
use serde::{Deserialize, Serialize};
|
|
||||||
|
|
||||||
use crate::corpus::ModelFamily;
|
|
||||||
use crate::packet::{EntityId, FeatureValue};
|
|
||||||
|
|
||||||
/// The most participants one scene holds.
|
|
||||||
pub const MAX_PARTICIPANTS: usize = 64;
|
|
||||||
|
|
||||||
/// Who annotated a document. A model's generator is an attestation string, held
|
|
||||||
/// to the same form as every provenance string.
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
|
||||||
pub enum Annotator {
|
|
||||||
Operator,
|
|
||||||
Model {
|
|
||||||
family: ModelFamily,
|
|
||||||
generator: String,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The speaker's stable axes. Key `speaker:<entity slug>@<n>`; a revision
|
|
||||||
/// appends and never edits.
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, JsonSchema, DatabaseEntry)]
|
|
||||||
#[cultcache(type = "weksa.speaker")]
|
|
||||||
pub struct Speaker {
|
|
||||||
#[cultcache(key = 0)]
|
|
||||||
pub entity: EntityId,
|
|
||||||
/// The revision, from 1.
|
|
||||||
#[cultcache(key = 1)]
|
|
||||||
pub n: u32,
|
|
||||||
#[cultcache(key = 2)]
|
|
||||||
pub inventory_rev: u32,
|
|
||||||
/// Feature values of slot `Speaker`.
|
|
||||||
#[cultcache(key = 3)]
|
|
||||||
pub features: Vec<FeatureValue>,
|
|
||||||
#[cultcache(key = 4)]
|
|
||||||
pub annotator: Annotator,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The other party of a relation. `Unnamed` is an addressee the scene does not
|
|
||||||
/// name, such as the player or a crowd.
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, JsonSchema)]
|
|
||||||
pub enum Party {
|
|
||||||
Named(EntityId),
|
|
||||||
Unnamed,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The speaker's relation to one party: feature values of slot `Scene`.
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
|
||||||
pub struct PairRelation {
|
|
||||||
pub from: EntityId,
|
|
||||||
pub to: Party,
|
|
||||||
pub features: Vec<FeatureValue>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A scene: its participants and the relations between them. Key
|
|
||||||
/// `scene:<group>@<n>`; a revision appends and never edits.
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, JsonSchema, DatabaseEntry)]
|
|
||||||
#[cultcache(type = "weksa.scene")]
|
|
||||||
pub struct Scene {
|
|
||||||
/// The corpus split group of the scene, a slug.
|
|
||||||
#[cultcache(key = 0)]
|
|
||||||
pub group: String,
|
|
||||||
/// The revision, from 1.
|
|
||||||
#[cultcache(key = 1)]
|
|
||||||
pub n: u32,
|
|
||||||
/// The revision, of the inventory the scene was admitted through, at which
|
|
||||||
/// `put_scene` read its features: provenance only; a run never reads it.
|
|
||||||
#[cultcache(key = 2)]
|
|
||||||
pub inventory_rev: u32,
|
|
||||||
#[cultcache(key = 3)]
|
|
||||||
pub participants: Vec<EntityId>,
|
|
||||||
#[cultcache(key = 4)]
|
|
||||||
pub relations: Vec<PairRelation>,
|
|
||||||
#[cultcache(key = 5)]
|
|
||||||
pub annotator: Annotator,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Scene {
|
|
||||||
/// The first of `speakers` the scene does not list as a participant: the
|
|
||||||
/// one rule of scene coverage, applied where a scene is put and where it is
|
|
||||||
/// read.
|
|
||||||
pub fn uncovered<'a>(
|
|
||||||
&self,
|
|
||||||
speakers: impl IntoIterator<Item = &'a EntityId>,
|
|
||||||
) -> Option<&'a EntityId> {
|
|
||||||
speakers
|
|
||||||
.into_iter()
|
|
||||||
.find(|s| !self.participants.contains(s))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The features of the relation from `speaker` to the named `addressee`, or
|
|
||||||
/// to `Unnamed` when there is none; `None` when the scene holds no such
|
|
||||||
/// relation (a relation with no features is `Some(&[])`). A named
|
|
||||||
/// addressee the scene lacks never falls back to `Unnamed`.
|
|
||||||
pub fn relation(
|
|
||||||
&self,
|
|
||||||
speaker: &EntityId,
|
|
||||||
addressee: Option<&EntityId>,
|
|
||||||
) -> Option<&[FeatureValue]> {
|
|
||||||
self.relations
|
|
||||||
.iter()
|
|
||||||
.find(|r| {
|
|
||||||
&r.from == speaker
|
|
||||||
&& match (&r.to, addressee) {
|
|
||||||
(Party::Named(a), Some(b)) => a == b,
|
|
||||||
(Party::Unnamed, None) => true,
|
|
||||||
_ => false,
|
|
||||||
}
|
|
||||||
})
|
|
||||||
.map(|r| r.features.as_slice())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `speaker:<entity slug>@<n>`.
|
|
||||||
pub fn speaker_key(entity: &EntityId, n: u32) -> String {
|
|
||||||
let slug = entity
|
|
||||||
.as_str()
|
|
||||||
.strip_prefix("entity:")
|
|
||||||
.unwrap_or(entity.as_str());
|
|
||||||
format!("speaker:{slug}@{n}")
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `scene:<group>@<n>`.
|
|
||||||
pub fn scene_key(group: &str, n: u32) -> String {
|
|
||||||
format!("scene:{group}@{n}")
|
|
||||||
}
|
|
||||||
|
|
@ -1,13 +1,6 @@
|
||||||
//! The train/validation/test split is derived from an item's group and nothing
|
//! The train/validation/test split is derived from an item's group and nothing
|
||||||
//! else. No stored assignment exists, so no writer can reassign one. An item is
|
//! else. No stored assignment exists, so no writer can reassign one. An item is
|
||||||
//! read in its live split. The round path (round records, prompts, `aggregate`,
|
//! read in its live split; Test is read only under a Checkpoint.
|
||||||
//! `leak_diagnostic`) carries no Test reading; a typed `TestReadings` and a Test
|
|
||||||
//! `EncoderOffer` need a Checkpoint, and only the lib can make one
|
|
||||||
//! (convergence-freeze adds the door). Operator tools (`weksa corpus dump`,
|
|
||||||
//! `weksa score`, `CorpusStore`) read corpus text of any split; that is not
|
|
||||||
//! gated here.
|
|
||||||
|
|
||||||
use std::collections::BTreeSet;
|
|
||||||
|
|
||||||
use crate::corpus::{CorpusItem, Source};
|
use crate::corpus::{CorpusItem, Source};
|
||||||
|
|
||||||
|
|
@ -24,38 +17,15 @@ pub enum SplitScheme {
|
||||||
V1,
|
V1,
|
||||||
}
|
}
|
||||||
|
|
||||||
pub const fn fnv1a64(bytes: &[u8]) -> u64 {
|
pub fn fnv1a64(bytes: &[u8]) -> u64 {
|
||||||
fnv1a64_fold(0xcbf2_9ce4_8422_2325, bytes)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// FNV-1a 64 of the parts, each preceded by its length as eight little-endian
|
|
||||||
/// bytes, so no two different splits of the same bytes share a digest.
|
|
||||||
pub const fn fnv1a64_of(parts: &[&[u8]]) -> u64 {
|
|
||||||
let mut h: u64 = 0xcbf2_9ce4_8422_2325;
|
let mut h: u64 = 0xcbf2_9ce4_8422_2325;
|
||||||
let mut p = 0;
|
for b in bytes {
|
||||||
while p < parts.len() {
|
h ^= u64::from(*b);
|
||||||
h = fnv1a64_fold(h, &(parts[p].len() as u64).to_le_bytes());
|
|
||||||
h = fnv1a64_fold(h, parts[p]);
|
|
||||||
p += 1;
|
|
||||||
}
|
|
||||||
h
|
|
||||||
}
|
|
||||||
|
|
||||||
const fn fnv1a64_fold(mut h: u64, bytes: &[u8]) -> u64 {
|
|
||||||
let mut i = 0;
|
|
||||||
while i < bytes.len() {
|
|
||||||
h ^= bytes[i] as u64;
|
|
||||||
h = h.wrapping_mul(0x0000_0100_0000_01b3);
|
h = h.wrapping_mul(0x0000_0100_0000_01b3);
|
||||||
i += 1;
|
|
||||||
}
|
}
|
||||||
h
|
h
|
||||||
}
|
}
|
||||||
|
|
||||||
/// 16 lowercase hex digits, zero-padded: how a digest is written everywhere.
|
|
||||||
pub(crate) fn hex16(n: u64) -> String {
|
|
||||||
format!("{n:016x}")
|
|
||||||
}
|
|
||||||
|
|
||||||
impl SplitScheme {
|
impl SplitScheme {
|
||||||
/// 0..100, FNV-1a 64 of `weksa-split-v1:` + group, mod 100.
|
/// 0..100, FNV-1a 64 of `weksa-split-v1:` + group, mod 100.
|
||||||
pub fn bucket(self, group: &str) -> u64 {
|
pub fn bucket(self, group: &str) -> u64 {
|
||||||
|
|
@ -93,131 +63,44 @@ impl SplitScheme {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Permission to read Test once, as typed `TestReadings` or as Test
|
/// Permission to read Test once. The loop declares one at a convergence
|
||||||
/// `EncoderOffer`s. Only the weksa lib can make one: there is no public
|
/// checkpoint (convergence-test); `TestReadings::at_checkpoint` consumes it, so
|
||||||
/// constructor, so no code outside it can obtain those. This is not a seal on
|
/// one declaration buys one Test reading, and `EncoderOffer::for_checkpoint`
|
||||||
/// corpus text: operator tools and `CorpusStore` can read any item, and the
|
/// borrows it first.
|
||||||
/// guarded invariant is that the round path holds no Test reading. The one door
|
|
||||||
/// is added by
|
|
||||||
/// convergence-freeze, which builds a `Checkpoint` from the convergence window
|
|
||||||
/// over the run's persisted round records (convergence-test).
|
|
||||||
/// `TestReadings::at_checkpoint` consumes it, so one `Checkpoint` buys one Test
|
|
||||||
/// reading. One `Checkpoint` also offers each item at most once
|
|
||||||
/// (`EncoderOffer::for_checkpoint`), and an offer borrows its Checkpoint, so it
|
|
||||||
/// cannot outlive the reading that spends it.
|
|
||||||
///
|
|
||||||
/// ```
|
|
||||||
/// fn f(c: &weksa::split::Checkpoint) -> u32 { c.round() }
|
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// No public constructor:
|
|
||||||
///
|
|
||||||
/// ```compile_fail
|
|
||||||
/// let _ = weksa::split::Checkpoint::declare(1);
|
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// The test constructor is not exported (this doctest names it today; the
|
|
||||||
/// line-based tripwire in `tests/checkpoint_door.rs` catches only common
|
|
||||||
/// single-line forms of a new public constructor):
|
|
||||||
///
|
|
||||||
/// ```compile_fail
|
|
||||||
/// let _ = weksa::split::Checkpoint::for_test(1);
|
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// ```compile_fail
|
|
||||||
/// let _ = weksa::split::Checkpoint { round: 1, offered: Default::default() };
|
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// One reading per checkpoint:
|
|
||||||
///
|
///
|
||||||
/// ```
|
/// ```
|
||||||
/// use weksa::score::TestReadings;
|
/// use weksa::score::TestReadings;
|
||||||
/// use weksa::split::Checkpoint;
|
/// use weksa::split::Checkpoint;
|
||||||
/// fn f(c: Checkpoint) { let _ = TestReadings::at_checkpoint(c, &[], &[], &[]); }
|
/// let c = Checkpoint::declare(3);
|
||||||
|
/// assert_eq!(c.round(), 3);
|
||||||
|
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
|
||||||
/// ```
|
/// ```
|
||||||
///
|
///
|
||||||
|
/// One declaration, one reading:
|
||||||
|
///
|
||||||
/// ```compile_fail
|
/// ```compile_fail
|
||||||
/// use weksa::score::TestReadings;
|
/// use weksa::score::TestReadings;
|
||||||
/// use weksa::split::Checkpoint;
|
/// use weksa::split::Checkpoint;
|
||||||
/// fn f(c: Checkpoint) {
|
/// let c = Checkpoint::declare(3);
|
||||||
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
|
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
|
||||||
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
|
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
|
||||||
/// }
|
|
||||||
/// ```
|
/// ```
|
||||||
///
|
///
|
||||||
/// ```compile_fail
|
/// ```compile_fail
|
||||||
/// fn f(c: weksa::split::Checkpoint) { let _ = c.clone(); }
|
/// use weksa::split::Checkpoint;
|
||||||
|
/// let c = Checkpoint::declare(3);
|
||||||
|
/// let _again = c.clone();
|
||||||
/// ```
|
/// ```
|
||||||
pub struct Checkpoint {
|
pub struct Checkpoint {
|
||||||
round: u32,
|
round: u32,
|
||||||
offered: BTreeSet<String>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Checkpoint {
|
impl Checkpoint {
|
||||||
#[cfg(test)]
|
pub fn declare(round: u32) -> Self {
|
||||||
pub(crate) fn for_test(round: u32) -> Self {
|
Self { round }
|
||||||
Self {
|
|
||||||
round,
|
|
||||||
offered: BTreeSet::new(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Records `key` as offered; false when it already was.
|
|
||||||
pub(crate) fn first_offer(&mut self, key: &str) -> bool {
|
|
||||||
self.offered.insert(key.to_string())
|
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn round(&self) -> u32 {
|
pub fn round(&self) -> u32 {
|
||||||
self.round
|
self.round
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod mutants;
|
|
||||||
|
|
||||||
/// Item builders for the unit tests that live in `src`: the twins of
|
|
||||||
/// `tests/common`'s `group_in_bucket`, `item_in`, `cast` and `model`.
|
|
||||||
#[cfg(test)]
|
|
||||||
pub(crate) mod fixture {
|
|
||||||
use super::{Split, SplitScheme};
|
|
||||||
use crate::corpus::{CorpusItem, ModelFamily, Source};
|
|
||||||
use crate::packet::EntityId;
|
|
||||||
|
|
||||||
/// The first group `g<i>` whose V1 bucket is in `split`.
|
|
||||||
pub(crate) fn group_in(split: Split) -> String {
|
|
||||||
(0..100_000)
|
|
||||||
.map(|i| format!("g{i}"))
|
|
||||||
.find(|g| SplitScheme::V1.of_bucket(SplitScheme::V1.bucket(g)) == split)
|
|
||||||
.unwrap()
|
|
||||||
}
|
|
||||||
|
|
||||||
pub(crate) fn item(key: &str, text: &str, group: &str, model_written: bool) -> CorpusItem {
|
|
||||||
let source = if model_written {
|
|
||||||
Source::ModelWritten {
|
|
||||||
family: ModelFamily::Unrecorded,
|
|
||||||
generator: None,
|
|
||||||
prompt_digest: None,
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
Source::CastAuthored {
|
|
||||||
persona: "nibu".into(),
|
|
||||||
fixture: "f.yaml".into(),
|
|
||||||
author: "Ada Writer".into(),
|
|
||||||
}
|
|
||||||
};
|
|
||||||
CorpusItem {
|
|
||||||
id: key.into(),
|
|
||||||
text: text.into(),
|
|
||||||
language: "en-US".into(),
|
|
||||||
speaker: EntityId::parse("entity:nibu").unwrap(),
|
|
||||||
names: vec![],
|
|
||||||
names_by: None,
|
|
||||||
group: group.into(),
|
|
||||||
prior_turns: vec![],
|
|
||||||
source,
|
|
||||||
admitted_on: "2026-10-29".into(),
|
|
||||||
supersedes: None,
|
|
||||||
withdrawn: false,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -1,11 +0,0 @@
|
||||||
//! Pins of private items of `split` found unpinned by mutation testing.
|
|
||||||
|
|
||||||
use super::Checkpoint;
|
|
||||||
|
|
||||||
/// A checkpoint reports the round it was declared at, whatever the round.
|
|
||||||
#[test]
|
|
||||||
fn a_checkpoint_reports_its_round() {
|
|
||||||
assert_eq!(Checkpoint::for_test(7).round(), 7);
|
|
||||||
assert_eq!(Checkpoint::for_test(0).round(), 0);
|
|
||||||
assert_eq!(Checkpoint::for_test(1).round(), 1);
|
|
||||||
}
|
|
||||||
|
|
@ -7,26 +7,18 @@ use std::fmt;
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
use caseless::default_case_fold_str;
|
use caseless::default_case_fold_str;
|
||||||
use cultcache_rs::{CacheBackingStore, CultCache, SingleFileMessagePackBackingStore};
|
use cultcache_rs::{CultCache, SingleFileMessagePackBackingStore};
|
||||||
use unicode_normalization::{UnicodeNormalization, char::is_combining_mark, is_nfc};
|
use unicode_normalization::{UnicodeNormalization, char::is_combining_mark, is_nfc};
|
||||||
use unicode_segmentation::UnicodeSegmentation;
|
use unicode_segmentation::UnicodeSegmentation;
|
||||||
|
|
||||||
use crate::corpus::{CorpusItem, CorpusItemId, Source, item_key, key_slug};
|
use crate::corpus::{CorpusItem, CorpusItemId, Source, item_key, key_slug};
|
||||||
use crate::split::{Split, SplitScheme};
|
|
||||||
use crate::entity::{Entity, EntityOrigin, Pronunciation};
|
use crate::entity::{Entity, EntityOrigin, Pronunciation};
|
||||||
use crate::form::{
|
use crate::form::{
|
||||||
is_attestation, is_calendar_date, is_joining_only, is_language_tag, is_refused_format, is_slug,
|
is_attestation, is_calendar_date, is_joining_only, is_language_tag, is_refused_format,
|
||||||
};
|
};
|
||||||
use crate::inventory::{
|
use crate::inventory::{Change, InvId, Inventory, InventoryEntry, InventoryRevision, revision_key};
|
||||||
Change, InvId, Inventory, InventoryEntry, InventoryRevision, Slot, revision_key,
|
use crate::packet::{EntityId, Packet};
|
||||||
};
|
use crate::validate::{LoweringInput, Refusal, admissible};
|
||||||
use crate::ontology::{
|
|
||||||
MAX_CONDITIONS, MAX_CHOICE_BYTES, MAX_ENTRIES_PER_LANGUAGE, MAX_EXAMPLE_BYTES, MAX_EXAMPLES,
|
|
||||||
Ontology, OntologyEntry, Test,
|
|
||||||
};
|
|
||||||
use crate::packet::{EntityId, FeatValue, Packet};
|
|
||||||
use crate::speaker::{Annotator, MAX_PARTICIPANTS, Party, Scene, Speaker, scene_key, speaker_key};
|
|
||||||
use crate::validate::{LoweringInput, Refusal, feature_refusals, validate};
|
|
||||||
|
|
||||||
/// A broken store rule, named.
|
/// A broken store rule, named.
|
||||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
|
@ -87,15 +79,6 @@ pub enum StoreRefusal {
|
||||||
/// A new packet names a corpus item already withdrawn. Packets admitted
|
/// A new packet names a corpus item already withdrawn. Packets admitted
|
||||||
/// before the withdrawal stay stored and readable.
|
/// before the withdrawal stay stored and readable.
|
||||||
WithdrawnCorpusItem { key: String },
|
WithdrawnCorpusItem { key: String },
|
||||||
/// A speaker or scene revision that is not the entity's or group's latest
|
|
||||||
/// plus one (1 when none).
|
|
||||||
RevisionNotNext {
|
|
||||||
key: String,
|
|
||||||
expected: u32,
|
|
||||||
got: u32,
|
|
||||||
},
|
|
||||||
/// A scene that omits the speaker of a live item of its group.
|
|
||||||
SceneSpeakerMissing { group: String, entity: EntityId },
|
|
||||||
/// Any other malformed document or change.
|
/// Any other malformed document or change.
|
||||||
Malformed { key: String, why: &'static str },
|
Malformed { key: String, why: &'static str },
|
||||||
}
|
}
|
||||||
|
|
@ -133,10 +116,6 @@ impl fmt::Display for StoreRefusal {
|
||||||
EntityNotAppendOnly { .. } => f.write_str("EntityNotAppendOnly"),
|
EntityNotAppendOnly { .. } => f.write_str("EntityNotAppendOnly"),
|
||||||
EntityNotOnItem { .. } => f.write_str("EntityNotOnItem"),
|
EntityNotOnItem { .. } => f.write_str("EntityNotOnItem"),
|
||||||
WithdrawnCorpusItem { .. } => f.write_str("WithdrawnCorpusItem"),
|
WithdrawnCorpusItem { .. } => f.write_str("WithdrawnCorpusItem"),
|
||||||
RevisionNotNext { expected, got, .. } => {
|
|
||||||
write!(f, "RevisionNotNext expected {expected} got {got}")
|
|
||||||
}
|
|
||||||
SceneSpeakerMissing { .. } => f.write_str("SceneSpeakerMissing"),
|
|
||||||
Malformed { why, .. } => write!(f, "Malformed: {why}"),
|
Malformed { why, .. } => write!(f, "Malformed: {why}"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -204,23 +183,16 @@ fn malformed(key: &str, why: &'static str) -> StoreError {
|
||||||
.into()
|
.into()
|
||||||
}
|
}
|
||||||
|
|
||||||
pub(crate) fn open_cache(
|
fn open_cache(
|
||||||
path: &Path,
|
path: &Path,
|
||||||
register: impl FnOnce(&mut CultCache) -> anyhow::Result<()>,
|
register: impl FnOnce(&mut CultCache) -> anyhow::Result<()>,
|
||||||
) -> anyhow::Result<CultCache> {
|
) -> anyhow::Result<CultCache> {
|
||||||
if let Some(dir) = path.parent() {
|
if let Some(dir) = path.parent() {
|
||||||
std::fs::create_dir_all(dir)?;
|
std::fs::create_dir_all(dir)?;
|
||||||
}
|
}
|
||||||
open_cache_over(SingleFileMessagePackBackingStore::new(path), register)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub(crate) fn open_cache_over(
|
|
||||||
store: impl CacheBackingStore + 'static,
|
|
||||||
register: impl FnOnce(&mut CultCache) -> anyhow::Result<()>,
|
|
||||||
) -> anyhow::Result<CultCache> {
|
|
||||||
let mut cache = CultCache::new();
|
let mut cache = CultCache::new();
|
||||||
register(&mut cache)?;
|
register(&mut cache)?;
|
||||||
cache.add_generic_backing_store(store)?;
|
cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(path))?;
|
||||||
cache.pull_all_backing_stores()?;
|
cache.pull_all_backing_stores()?;
|
||||||
Ok(cache)
|
Ok(cache)
|
||||||
}
|
}
|
||||||
|
|
@ -249,17 +221,6 @@ impl CorpusStore {
|
||||||
Ok(items)
|
Ok(items)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The text of every item in a Test group, any language, withdrawn and
|
|
||||||
/// model-written ones included: what no prompt may hold.
|
|
||||||
pub fn test_texts(&self) -> anyhow::Result<Vec<String>> {
|
|
||||||
Ok(self
|
|
||||||
.all()?
|
|
||||||
.into_iter()
|
|
||||||
.filter(|item| SplitScheme::V1.of_bucket(SplitScheme::V1.bucket(&item.group)) == Split::Test)
|
|
||||||
.map(|item| item.text)
|
|
||||||
.collect())
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn get(&self, key: &str) -> anyhow::Result<Option<CorpusItem>> {
|
pub fn get(&self, key: &str) -> anyhow::Result<Option<CorpusItem>> {
|
||||||
self.cache.get::<CorpusItem>(key)
|
self.cache.get::<CorpusItem>(key)
|
||||||
}
|
}
|
||||||
|
|
@ -344,9 +305,6 @@ impl CorpusStore {
|
||||||
if let Some(why) = empty_attestation(&item.source) {
|
if let Some(why) = empty_attestation(&item.source) {
|
||||||
return Err(malformed(&item.id, why));
|
return Err(malformed(&item.id, why));
|
||||||
}
|
}
|
||||||
if let Some(by) = &item.names_by {
|
|
||||||
check_annotator(&item.id, by)?;
|
|
||||||
}
|
|
||||||
check_entities(&item, entities)?;
|
check_entities(&item, entities)?;
|
||||||
for held in self.all()? {
|
for held in self.all()? {
|
||||||
if normalized(&held.text) == text && item.supersedes.as_deref() != Some(&held.id) {
|
if normalized(&held.text) == text && item.supersedes.as_deref() != Some(&held.id) {
|
||||||
|
|
@ -490,13 +448,13 @@ pub(crate) fn normalized(text: &str) -> String {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The one-to-one folds `normalized` adds to NFKC and case folding, applied
|
/// The one-to-one folds `normalized` adds to NFKC and case folding, applied
|
||||||
/// before the case fold. (U+0130 needs no arm: it folds to i plus a combining
|
/// before the case fold (U+0130 would otherwise fold to i plus a combining dot,
|
||||||
/// dot, and the `dotless` loop in `normalized` drops that dot after an `i`, so
|
/// which the key keeps as a mark).
|
||||||
/// the key is plain `i`.)
|
|
||||||
fn fold_variant(c: char) -> impl Iterator<Item = char> {
|
fn fold_variant(c: char) -> impl Iterator<Item = char> {
|
||||||
let one = |c: char| (c, None);
|
let one = |c: char| (c, None);
|
||||||
let (first, second) = match c {
|
let (first, second) = match c {
|
||||||
'\u{131}' => one('i'),
|
'\u{131}' => one('i'),
|
||||||
|
'\u{130}' => one('I'),
|
||||||
// Voiced katakana wa, wi, we, wo: the hiragana (U+308F-3092) plus the
|
// Voiced katakana wa, wi, we, wo: the hiragana (U+308F-3092) plus the
|
||||||
// combining voiced mark, which is how NFKC spells the hiragana forms.
|
// combining voiced mark, which is how NFKC spells the hiragana forms.
|
||||||
'\u{30f7}'..='\u{30fa}' => (
|
'\u{30f7}'..='\u{30fa}' => (
|
||||||
|
|
@ -985,7 +943,7 @@ impl PacketStore {
|
||||||
corpus: &CorpusStore,
|
corpus: &CorpusStore,
|
||||||
entities: &EntityStore,
|
entities: &EntityStore,
|
||||||
) -> Result<LoweringInput, StoreError> {
|
) -> Result<LoweringInput, StoreError> {
|
||||||
validate(&packet, &inventory.inventory()?).map_err(StoreError::Invalid)?;
|
let view = admissible(&packet, &inventory.inventory()?).map_err(StoreError::Invalid)?;
|
||||||
let item = corpus.resolve(&packet.provenance.corpus_item)?;
|
let item = corpus.resolve(&packet.provenance.corpus_item)?;
|
||||||
if item.withdrawn {
|
if item.withdrawn {
|
||||||
return Err(StoreRefusal::WithdrawnCorpusItem {
|
return Err(StoreRefusal::WithdrawnCorpusItem {
|
||||||
|
|
@ -1009,7 +967,6 @@ impl PacketStore {
|
||||||
.into());
|
.into());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
let view = LoweringInput::of(&packet, &item);
|
|
||||||
let key = packet.id.to_string();
|
let key = packet.id.to_string();
|
||||||
if let Some(existing) = self.get(&key)? {
|
if let Some(existing) = self.get(&key)? {
|
||||||
if existing == packet {
|
if existing == packet {
|
||||||
|
|
@ -1021,342 +978,3 @@ impl PacketStore {
|
||||||
Ok(view)
|
Ok(view)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The speaker and scene documents: append-only revisions, one key each. The
|
|
||||||
/// only writer of `weksa.speaker` and `weksa.scene`.
|
|
||||||
pub struct SpeakerStateStore {
|
|
||||||
cache: CultCache,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl SpeakerStateStore {
|
|
||||||
pub fn open(path: &Path) -> anyhow::Result<Self> {
|
|
||||||
Ok(Self {
|
|
||||||
cache: open_cache(path, |c| {
|
|
||||||
c.register_entry_type::<Speaker>()?;
|
|
||||||
c.register_entry_type::<Scene>()
|
|
||||||
})?,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Admits the next revision of a speaker, or an identical replay. The key
|
|
||||||
/// derives from the document. Revision `n` is the entity's latest plus one
|
|
||||||
/// (1 when none). The entity is in the entity store, the features pass
|
|
||||||
/// `feature_refusals` in slot `Speaker` at the document's inventory
|
|
||||||
/// revision, and a model annotator's generator is an attestation.
|
|
||||||
pub fn put_speaker(
|
|
||||||
&mut self,
|
|
||||||
s: Speaker,
|
|
||||||
inventory: &InventoryStore,
|
|
||||||
entities: &EntityStore,
|
|
||||||
) -> Result<AddOutcome, StoreError> {
|
|
||||||
let key = speaker_key(&s.entity, s.n);
|
|
||||||
if self.get_speaker(&key)?.is_some_and(|old| old == s) {
|
|
||||||
return Ok(AddOutcome::Unchanged);
|
|
||||||
}
|
|
||||||
check_annotator(&key, &s.annotator)?;
|
|
||||||
known_entity(entities, &s.entity)?;
|
|
||||||
let out = feature_refusals(
|
|
||||||
&inventory.inventory()?,
|
|
||||||
s.inventory_rev,
|
|
||||||
&s.features,
|
|
||||||
Slot::Speaker,
|
|
||||||
);
|
|
||||||
if !out.is_empty() {
|
|
||||||
return Err(StoreError::Invalid(out));
|
|
||||||
}
|
|
||||||
let latest = self.speaker(&s.entity)?.map_or(0, |l| l.n);
|
|
||||||
check_next(&key, latest, s.n)?;
|
|
||||||
self.cache.put(key, &s)?;
|
|
||||||
Ok(AddOutcome::Added)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Admits the next revision of a scene, or an identical replay. The group
|
|
||||||
/// is a slug some corpus item (live or withdrawn) holds; there are 1 to
|
|
||||||
/// `MAX_PARTICIPANTS` participants, each named once and in the entity
|
|
||||||
/// store; a relation's `from` is a participant, a named `to` is another
|
|
||||||
/// participant, and an ordered pair has at most one relation; every speaker
|
|
||||||
/// of a live item of the group is a participant. Features pass
|
|
||||||
/// `feature_refusals` in slot `Scene` for every relation.
|
|
||||||
pub fn put_scene(
|
|
||||||
&mut self,
|
|
||||||
s: Scene,
|
|
||||||
inventory: &InventoryStore,
|
|
||||||
corpus: &CorpusStore,
|
|
||||||
entities: &EntityStore,
|
|
||||||
) -> Result<AddOutcome, StoreError> {
|
|
||||||
let key = scene_key(&s.group, s.n);
|
|
||||||
if self.get_scene(&key)?.is_some_and(|old| old == s) {
|
|
||||||
return Ok(AddOutcome::Unchanged);
|
|
||||||
}
|
|
||||||
let bad = |why| malformed(&key, why);
|
|
||||||
if !is_slug(&s.group) {
|
|
||||||
return Err(bad("a scene group is a slug"));
|
|
||||||
}
|
|
||||||
check_annotator(&key, &s.annotator)?;
|
|
||||||
let items: Vec<CorpusItem> = corpus
|
|
||||||
.all()?
|
|
||||||
.into_iter()
|
|
||||||
.filter(|i| i.group == s.group)
|
|
||||||
.collect();
|
|
||||||
if items.is_empty() {
|
|
||||||
return Err(bad("no corpus item holds the scene's group"));
|
|
||||||
}
|
|
||||||
if s.participants.is_empty() || s.participants.len() > MAX_PARTICIPANTS {
|
|
||||||
return Err(bad("a scene has 1 to 64 participants"));
|
|
||||||
}
|
|
||||||
let mut seen = BTreeSet::new();
|
|
||||||
for p in &s.participants {
|
|
||||||
if !seen.insert(p) {
|
|
||||||
return Err(bad("a participant is named once"));
|
|
||||||
}
|
|
||||||
known_entity(entities, p)?;
|
|
||||||
}
|
|
||||||
let mut pairs = BTreeSet::new();
|
|
||||||
for r in &s.relations {
|
|
||||||
if !seen.contains(&r.from) {
|
|
||||||
return Err(bad("a relation's from is a participant"));
|
|
||||||
}
|
|
||||||
if let Party::Named(to) = &r.to
|
|
||||||
&& (!seen.contains(to) || to == &r.from)
|
|
||||||
{
|
|
||||||
return Err(bad("a relation's named to is another participant"));
|
|
||||||
}
|
|
||||||
if !pairs.insert((&r.from, &r.to)) {
|
|
||||||
return Err(bad("an ordered pair has one relation"));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if let Some(entity) = s.uncovered(items.iter().filter(|i| !i.withdrawn).map(|i| &i.speaker))
|
|
||||||
{
|
|
||||||
return Err(StoreRefusal::SceneSpeakerMissing {
|
|
||||||
entity: entity.clone(),
|
|
||||||
group: s.group,
|
|
||||||
}
|
|
||||||
.into());
|
|
||||||
}
|
|
||||||
let inv = inventory.inventory()?;
|
|
||||||
let mut out = Vec::new();
|
|
||||||
if s.relations.is_empty() {
|
|
||||||
out = feature_refusals(&inv, s.inventory_rev, &[], Slot::Scene);
|
|
||||||
}
|
|
||||||
for r in &s.relations {
|
|
||||||
out.extend(feature_refusals(
|
|
||||||
&inv,
|
|
||||||
s.inventory_rev,
|
|
||||||
&r.features,
|
|
||||||
Slot::Scene,
|
|
||||||
));
|
|
||||||
}
|
|
||||||
if !out.is_empty() {
|
|
||||||
return Err(StoreError::Invalid(out));
|
|
||||||
}
|
|
||||||
let latest = self.scene(&s.group)?.map_or(0, |l| l.n);
|
|
||||||
check_next(&key, latest, s.n)?;
|
|
||||||
self.cache.put(key, &s)?;
|
|
||||||
Ok(AddOutcome::Added)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The latest revision (highest `n`) of the entity's speaker document.
|
|
||||||
pub fn speaker(&self, entity: &EntityId) -> anyhow::Result<Option<Speaker>> {
|
|
||||||
Ok(self
|
|
||||||
.all_speakers()?
|
|
||||||
.into_iter()
|
|
||||||
.filter(|s| &s.entity == entity)
|
|
||||||
.max_by_key(|s| s.n))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The latest revision (highest `n`) of the group's scene document.
|
|
||||||
pub fn scene(&self, group: &str) -> anyhow::Result<Option<Scene>> {
|
|
||||||
Ok(self
|
|
||||||
.all_scenes()?
|
|
||||||
.into_iter()
|
|
||||||
.filter(|s| s.group == group)
|
|
||||||
.max_by_key(|s| s.n))
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn get_speaker(&self, key: &str) -> anyhow::Result<Option<Speaker>> {
|
|
||||||
self.cache.get::<Speaker>(key)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn get_scene(&self, key: &str) -> anyhow::Result<Option<Scene>> {
|
|
||||||
self.cache.get::<Scene>(key)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Every speaker revision, by entity then revision number.
|
|
||||||
pub fn all_speakers(&self) -> anyhow::Result<Vec<Speaker>> {
|
|
||||||
let mut all = self.cache.get_all::<Speaker>()?;
|
|
||||||
all.sort_by(|a, b| (&a.entity, a.n).cmp(&(&b.entity, b.n)));
|
|
||||||
Ok(all)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Every scene revision, by group then revision number.
|
|
||||||
pub fn all_scenes(&self) -> anyhow::Result<Vec<Scene>> {
|
|
||||||
let mut all = self.cache.get_all::<Scene>()?;
|
|
||||||
all.sort_by(|a, b| (&a.group, a.n).cmp(&(&b.group, b.n)));
|
|
||||||
Ok(all)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn known_entity(entities: &EntityStore, id: &EntityId) -> Result<(), StoreError> {
|
|
||||||
match entities.get(id)? {
|
|
||||||
Some(_) => Ok(()),
|
|
||||||
None => Err(StoreRefusal::UnknownEntity { entity: id.clone() }.into()),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn check_annotator(key: &str, a: &Annotator) -> Result<(), StoreError> {
|
|
||||||
match a {
|
|
||||||
Annotator::Model { generator, .. } if !is_attestation(generator) => {
|
|
||||||
Err(malformed(key, "a model annotator attests its generator"))
|
|
||||||
}
|
|
||||||
_ => Ok(()),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn check_next(key: &str, latest: u32, got: u32) -> Result<(), StoreError> {
|
|
||||||
let expected = latest
|
|
||||||
.checked_add(1)
|
|
||||||
.ok_or_else(|| malformed(key, "no revision follows the latest"))?;
|
|
||||||
if got != expected {
|
|
||||||
return Err(StoreRefusal::RevisionNotNext {
|
|
||||||
key: key.to_string(),
|
|
||||||
expected,
|
|
||||||
got,
|
|
||||||
}
|
|
||||||
.into());
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The language ontology: immutable entries admitted in all-or-nothing
|
|
||||||
/// batches. The only writer of `weksa.ontology_entry`.
|
|
||||||
pub struct OntologyStore {
|
|
||||||
cache: CultCache,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl OntologyStore {
|
|
||||||
pub fn open(path: &Path) -> anyhow::Result<Self> {
|
|
||||||
Ok(Self {
|
|
||||||
cache: open_cache(path, |c| c.register_entry_type::<OntologyEntry>())?,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Every entry, by language then number.
|
|
||||||
pub fn all(&self) -> anyhow::Result<Vec<OntologyEntry>> {
|
|
||||||
Ok(self.ontology()?.entries().to_vec())
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn ontology(&self) -> anyhow::Result<Ontology> {
|
|
||||||
Ok(Ontology::of(self.cache.get_all::<OntologyEntry>()?))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Admits a batch whole or not at all; an identical replay of stored
|
|
||||||
/// entries is `Unchanged`. Per language the batch's numbers run the
|
|
||||||
/// language's latest plus one, consecutively, to at most
|
|
||||||
/// `MAX_ENTRIES_PER_LANGUAGE` entries in all. Every entry passes
|
|
||||||
/// [`check_ontology_entry`] against the inventory before any is put. No
|
|
||||||
/// message echoes an entry's text.
|
|
||||||
pub fn admit(
|
|
||||||
&mut self,
|
|
||||||
batch: Vec<OntologyEntry>,
|
|
||||||
inventory: &InventoryStore,
|
|
||||||
) -> Result<AddOutcome, StoreError> {
|
|
||||||
let all = self.cache.get_all::<OntologyEntry>()?;
|
|
||||||
if batch
|
|
||||||
.iter()
|
|
||||||
.all(|e| all.iter().any(|old| old == e))
|
|
||||||
{
|
|
||||||
return Ok(AddOutcome::Unchanged);
|
|
||||||
}
|
|
||||||
let inv = inventory.inventory()?;
|
|
||||||
let mut counted: std::collections::BTreeMap<&str, u32> = std::collections::BTreeMap::new();
|
|
||||||
for e in &batch {
|
|
||||||
let key = e.key();
|
|
||||||
if !is_language_tag(&e.language) {
|
|
||||||
return Err(malformed(&key, "an ontology language is a language tag"));
|
|
||||||
}
|
|
||||||
let latest = match counted.get(e.language.as_str()) {
|
|
||||||
Some(n) => *n,
|
|
||||||
None => all
|
|
||||||
.iter()
|
|
||||||
.filter(|o| o.language == e.language)
|
|
||||||
.map(|o| o.n)
|
|
||||||
.max()
|
|
||||||
.unwrap_or(0),
|
|
||||||
};
|
|
||||||
check_next(&key, latest, e.n)?;
|
|
||||||
if e.n as usize > MAX_ENTRIES_PER_LANGUAGE {
|
|
||||||
return Err(malformed(&key, "a language holds at most 64 entries"));
|
|
||||||
}
|
|
||||||
counted.insert(e.language.as_str(), e.n);
|
|
||||||
check_ontology_entry(e, &inv, e.inventory_rev)?;
|
|
||||||
}
|
|
||||||
let mut staged = Vec::new();
|
|
||||||
for e in &batch {
|
|
||||||
staged.push(self.cache.prepare_entry_named(e.key(), e)?.0);
|
|
||||||
}
|
|
||||||
self.cache.put_prepared_batch(staged)?;
|
|
||||||
Ok(AddOutcome::Added)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The rules of one ontology entry against an inventory: the annotator, the
|
|
||||||
/// bounded and normalized text of the choice and examples, the condition count
|
|
||||||
/// and each condition's feature (slot `Stance`, `Discourse` or `Scene`, a grade
|
|
||||||
/// feature for `AtLeast`, `AtMost` and `AtMostOrUnset`, an option or flag value for `Is`) read at
|
|
||||||
/// `rev`. The store passes the entry's own `inventory_rev` on admission; a run
|
|
||||||
/// passes its inventory's head, because beds number revisions independently.
|
|
||||||
pub(crate) fn check_ontology_entry(
|
|
||||||
e: &OntologyEntry,
|
|
||||||
inventory: &Inventory,
|
|
||||||
rev: u32,
|
|
||||||
) -> Result<(), StoreError> {
|
|
||||||
let key = e.key();
|
|
||||||
check_annotator(&key, &e.annotator)?;
|
|
||||||
let text_ok = |t: &str, max: usize| {
|
|
||||||
(1..=max).contains(&t.len())
|
|
||||||
&& !t.chars().any(|c| c.is_control() || is_refused_format(c))
|
|
||||||
&& is_nfc(t)
|
|
||||||
};
|
|
||||||
if !text_ok(&e.choice, MAX_CHOICE_BYTES) {
|
|
||||||
return Err(malformed(
|
|
||||||
&key,
|
|
||||||
"a choice is 1 to 200 normalized bytes with no control or format character",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
if e.examples.len() > MAX_EXAMPLES
|
|
||||||
|| !e.examples.iter().all(|x| text_ok(x, MAX_EXAMPLE_BYTES))
|
|
||||||
{
|
|
||||||
return Err(malformed(
|
|
||||||
&key,
|
|
||||||
"an entry has at most 3 examples of 1 to 120 normalized bytes with no control or format character",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
if e.when.len() > MAX_CONDITIONS {
|
|
||||||
return Err(malformed(&key, "an entry has at most 4 conditions"));
|
|
||||||
}
|
|
||||||
let mut out = Vec::new();
|
|
||||||
for c in &e.when {
|
|
||||||
if !matches!(c.slot, Slot::Stance | Slot::Discourse | Slot::Scene) {
|
|
||||||
return Err(malformed(
|
|
||||||
&key,
|
|
||||||
"a condition reads slot Stance, Discourse or Scene",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
if matches!(c.test, Test::Is(FeatValue::Grade(_))) {
|
|
||||||
return Err(malformed(&key, "Is names an option or a flag"));
|
|
||||||
}
|
|
||||||
out.extend(feature_refusals(
|
|
||||||
inventory,
|
|
||||||
rev,
|
|
||||||
&[c.as_value()],
|
|
||||||
c.slot,
|
|
||||||
));
|
|
||||||
}
|
|
||||||
if e.when.is_empty() && !inventory.has_revision(rev) {
|
|
||||||
out.extend(feature_refusals(inventory, rev, &[], Slot::Stance));
|
|
||||||
}
|
|
||||||
if !out.is_empty() {
|
|
||||||
return Err(StoreError::Invalid(out));
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -1,14 +1,12 @@
|
||||||
//! The validator owns the inventory and form rules a packet must meet. Every
|
//! The validator owns the inventory and form rules a packet must meet. Every
|
||||||
//! path that admits a packet applies them, but `validate` only judges: it
|
//! path that admits a packet applies them, but `validate` only judges: it
|
||||||
//! returns `Ok(())` or the refusals. [`LoweringInput`], the only thing a
|
//! returns `Ok(())` or the refusals. [`LoweringInput`], the only thing a
|
||||||
//! lowerer sees, comes from `PacketStore::admit` alone, which builds it from
|
//! lowerer sees, comes from `PacketStore::admit` alone, which builds it through
|
||||||
//! the admitted packet and its corpus item through the crate-private
|
//! the crate-private `admissible`. The stance is checked like every feature
|
||||||
//! `LoweringInput::of`. The stance is checked like every feature
|
//! list, in slot `Stance`. The rules are V1..V8 and V10..V15: the shape of an id
|
||||||
//! list, in slot `Stance`. The rules are V1..V8, V10..V12 and V14..V18: the
|
//! or a grade is its type's parse, not a rule, so V9 is retired and not
|
||||||
//! shape of an id or a grade is its type's parse, not a rule, so V9 is retired
|
//! renumbered. V13 guards the one other way features reach a lowering view:
|
||||||
//! and not renumbered; V13 is retired too. [`LoweringInput::with_scene`] reads the
|
//! [`LoweringInput::with_scene`].
|
||||||
//! scene of the view's own item group from the speaker-state store, and
|
|
||||||
//! `feature_refusals` is the one feature check it and that store share.
|
|
||||||
//!
|
//!
|
||||||
//! `validate` owns form and capacity and makes no leak decision. A packet's one
|
//! `validate` owns form and capacity and makes no leak decision. A packet's one
|
||||||
//! free text is its trace (`notes` and `rejected_readings`): `validate` does not
|
//! free text is its trace (`notes` and `rejected_readings`): `validate` does not
|
||||||
|
|
@ -28,13 +26,11 @@
|
||||||
use std::collections::BTreeSet;
|
use std::collections::BTreeSet;
|
||||||
use std::fmt;
|
use std::fmt;
|
||||||
|
|
||||||
use crate::corpus::CorpusItem;
|
|
||||||
use crate::inventory::{Family, InvId, Inventory, InventoryEntry, Slot, ValueKind};
|
use crate::inventory::{Family, InvId, Inventory, InventoryEntry, Slot, ValueKind};
|
||||||
use crate::packet::{
|
use crate::packet::{
|
||||||
Constraints, Context, Discourse, EntityId, Extension, FeatValue, FeatureValue, NodeId, Packet,
|
Constraints, Context, Discourse, Extension, FeatValue, FeatureValue, NodeId, Packet,
|
||||||
PacketKind, Predication, ProvenanceSource, Referent, Relation, SCHEMA_VERSION, Stance,
|
PacketKind, Predication, ProvenanceSource, Referent, Relation, SCHEMA_VERSION, Stance,
|
||||||
};
|
};
|
||||||
use crate::store::{CorpusStore, SpeakerStateStore, StoreError};
|
|
||||||
|
|
||||||
/// A broken rule, named.
|
/// A broken rule, named.
|
||||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
|
@ -99,6 +95,10 @@ pub enum Refusal {
|
||||||
count: usize,
|
count: usize,
|
||||||
max: usize,
|
max: usize,
|
||||||
},
|
},
|
||||||
|
/// V13: scene features were given for a packet with no addressee. They
|
||||||
|
/// state the speaker's relation to the addressee, so with none they
|
||||||
|
/// describe nobody.
|
||||||
|
V13SceneWithoutAddressee,
|
||||||
/// V14: an addressee consent that no addressee can have given, or that
|
/// V14: an addressee consent that no addressee can have given, or that
|
||||||
/// names a node that is not a predication. It is what the addressee asked
|
/// names a node that is not a predication. It is what the addressee asked
|
||||||
/// for or agreed to, an act, so a referent cannot be its object.
|
/// for or agreed to, an act, so a referent cannot be its object.
|
||||||
|
|
@ -112,20 +112,6 @@ pub enum Refusal {
|
||||||
V15ConsentSource {
|
V15ConsentSource {
|
||||||
source: ProvenanceSource,
|
source: ProvenanceSource,
|
||||||
},
|
},
|
||||||
/// V16: `with_scene` found its group's scene not covering the item's speaker
|
|
||||||
/// or the speaker of some live item of the group.
|
|
||||||
V16SpeakerNotInScene,
|
|
||||||
/// V17: an addressee that names a predication. An act is not someone
|
|
||||||
/// spoken to, so the addressee names a referent.
|
|
||||||
V17AddresseeNotAReferent {
|
|
||||||
id: NodeId,
|
|
||||||
},
|
|
||||||
/// V18: a predication whose frame is the inventory's `request_intervention`,
|
|
||||||
/// with no `addressee_consent` naming it. It says what the addressee did, so
|
|
||||||
/// only the packet's own record may license it.
|
|
||||||
V18RequestWithoutConsent {
|
|
||||||
predication: NodeId,
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Refusal {
|
impl Refusal {
|
||||||
|
|
@ -144,13 +130,11 @@ impl Refusal {
|
||||||
| Refusal::V11DuplicateNodeRef { .. }
|
| Refusal::V11DuplicateNodeRef { .. }
|
||||||
| Refusal::V11DuplicateRole { .. } => "V11",
|
| Refusal::V11DuplicateRole { .. } => "V11",
|
||||||
Refusal::V12TooManyNodes { .. } => "V12",
|
Refusal::V12TooManyNodes { .. } => "V12",
|
||||||
|
Refusal::V13SceneWithoutAddressee => "V13",
|
||||||
Refusal::V14ConsentWithoutAddressee | Refusal::V14ConsentNotAPredication { .. } => {
|
Refusal::V14ConsentWithoutAddressee | Refusal::V14ConsentNotAPredication { .. } => {
|
||||||
"V14"
|
"V14"
|
||||||
}
|
}
|
||||||
Refusal::V15ConsentSource { .. } => "V15",
|
Refusal::V15ConsentSource { .. } => "V15",
|
||||||
Refusal::V16SpeakerNotInScene => "V16",
|
|
||||||
Refusal::V17AddresseeNotAReferent { .. } => "V17",
|
|
||||||
Refusal::V18RequestWithoutConsent { .. } => "V18",
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -173,16 +157,10 @@ impl fmt::Display for Refusal {
|
||||||
Refusal::V11DuplicateNodeRef { .. } => "a node id repeats in one list",
|
Refusal::V11DuplicateNodeRef { .. } => "a node id repeats in one list",
|
||||||
Refusal::V11DuplicateRole { .. } => "a role id repeats in one predication",
|
Refusal::V11DuplicateRole { .. } => "a role id repeats in one predication",
|
||||||
Refusal::V12TooManyNodes { .. } => "too many nodes",
|
Refusal::V12TooManyNodes { .. } => "too many nodes",
|
||||||
|
Refusal::V13SceneWithoutAddressee => "scene features given without an addressee",
|
||||||
Refusal::V14ConsentWithoutAddressee => "consent given without an addressee",
|
Refusal::V14ConsentWithoutAddressee => "consent given without an addressee",
|
||||||
Refusal::V14ConsentNotAPredication { .. } => "consent names no predication",
|
Refusal::V14ConsentNotAPredication { .. } => "consent names no predication",
|
||||||
Refusal::V15ConsentSource { .. } => "consent set by a source that only read wording",
|
Refusal::V15ConsentSource { .. } => "consent set by a source that only read wording",
|
||||||
Refusal::V16SpeakerNotInScene => {
|
|
||||||
"the scene does not cover the speaker of every live item of its group"
|
|
||||||
}
|
|
||||||
Refusal::V17AddresseeNotAReferent { .. } => "the addressee names no referent",
|
|
||||||
Refusal::V18RequestWithoutConsent { .. } => {
|
|
||||||
"a request frame needs the addressee's consent naming it"
|
|
||||||
}
|
|
||||||
};
|
};
|
||||||
write!(f, "{}: {what}", self.rule())
|
write!(f, "{}: {what}", self.rule())
|
||||||
}
|
}
|
||||||
|
|
@ -202,6 +180,28 @@ impl fmt::Display for Refusal {
|
||||||
/// fn lower(p: &Packet, i: &Inventory) -> Result<LoweringInput, Vec<Refusal>> { weksa::validate(p, i) }
|
/// fn lower(p: &Packet, i: &Inventory) -> Result<LoweringInput, Vec<Refusal>> { weksa::validate(p, i) }
|
||||||
/// ```
|
/// ```
|
||||||
pub fn validate(packet: &Packet, inventory: &Inventory) -> Result<(), Vec<Refusal>> {
|
pub fn validate(packet: &Packet, inventory: &Inventory) -> Result<(), Vec<Refusal>> {
|
||||||
|
admissible(packet, inventory).map(|_| ())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The one implementation of the rules. Crate-private: `PacketStore::admit` is
|
||||||
|
/// the only caller that keeps the view.
|
||||||
|
///
|
||||||
|
/// The first block is the same call through the public name, so the second
|
||||||
|
/// fails for the private path and nothing else.
|
||||||
|
///
|
||||||
|
/// ```
|
||||||
|
/// use weksa::{Inventory, packet::Packet};
|
||||||
|
/// fn judge(p: &Packet, i: &Inventory) { let _ = weksa::validate::validate(p, i); }
|
||||||
|
/// ```
|
||||||
|
///
|
||||||
|
/// ```compile_fail
|
||||||
|
/// use weksa::{Inventory, packet::Packet};
|
||||||
|
/// fn lower(p: &Packet, i: &Inventory) { let _ = weksa::validate::admissible(p, i); }
|
||||||
|
/// ```
|
||||||
|
pub(crate) fn admissible(
|
||||||
|
packet: &Packet,
|
||||||
|
inventory: &Inventory,
|
||||||
|
) -> Result<LoweringInput, Vec<Refusal>> {
|
||||||
let mut out = Vec::new();
|
let mut out = Vec::new();
|
||||||
if packet.schema_version != SCHEMA_VERSION {
|
if packet.schema_version != SCHEMA_VERSION {
|
||||||
out.push(Refusal::V1SchemaVersion {
|
out.push(Refusal::V1SchemaVersion {
|
||||||
|
|
@ -221,7 +221,6 @@ pub fn validate(packet: &Packet, inventory: &Inventory) -> Result<(), Vec<Refusa
|
||||||
}
|
}
|
||||||
let nodes = check_nodes(packet, &mut out);
|
let nodes = check_nodes(packet, &mut out);
|
||||||
check_refs(packet, &nodes, &mut out);
|
check_refs(packet, &nodes, &mut out);
|
||||||
check_addressee(packet, &nodes, &mut out);
|
|
||||||
check_repeats(packet, &mut out);
|
check_repeats(packet, &mut out);
|
||||||
check_consent(packet, &nodes, &mut out);
|
check_consent(packet, &nodes, &mut out);
|
||||||
if inventory.has_revision(packet.inventory_rev) {
|
if inventory.has_revision(packet.inventory_rev) {
|
||||||
|
|
@ -231,13 +230,16 @@ pub fn validate(packet: &Packet, inventory: &Inventory) -> Result<(), Vec<Refusa
|
||||||
out: &mut out,
|
out: &mut out,
|
||||||
}
|
}
|
||||||
.check(packet);
|
.check(packet);
|
||||||
check_requests(packet, inventory, &mut out);
|
|
||||||
} else {
|
} else {
|
||||||
out.push(Refusal::V10UnknownRevision {
|
out.push(Refusal::V10UnknownRevision {
|
||||||
rev: packet.inventory_rev,
|
rev: packet.inventory_rev,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
if out.is_empty() { Ok(()) } else { Err(out) }
|
if out.is_empty() {
|
||||||
|
Ok(LoweringInput::of(packet))
|
||||||
|
} else {
|
||||||
|
Err(out)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn check_nodes(packet: &Packet, out: &mut Vec<Refusal>) -> BTreeSet<NodeId> {
|
fn check_nodes(packet: &Packet, out: &mut Vec<Refusal>) -> BTreeSet<NodeId> {
|
||||||
|
|
@ -283,17 +285,6 @@ fn check_refs(packet: &Packet, nodes: &BTreeSet<NodeId>, out: &mut Vec<Refusal>)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// V17: the addressee, when it names a node, names a referent. A dangling id
|
|
||||||
/// is V4's.
|
|
||||||
fn check_addressee(packet: &Packet, nodes: &BTreeSet<NodeId>, out: &mut Vec<Refusal>) {
|
|
||||||
if let Some(a) = &packet.discourse.addressee
|
|
||||||
&& nodes.contains(a)
|
|
||||||
&& !packet.referents.iter().any(|r| &r.id == a)
|
|
||||||
{
|
|
||||||
out.push(Refusal::V17AddresseeNotAReferent { id: a.clone() });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// V14 and V15: the addressee's consent is a typed fact with one origin. A
|
/// V14 and V15: the addressee's consent is a typed fact with one origin. A
|
||||||
/// dangling id is V4's, so V14 judges only a node that exists.
|
/// dangling id is V4's, so V14 judges only a node that exists.
|
||||||
fn check_consent(packet: &Packet, nodes: &BTreeSet<NodeId>, out: &mut Vec<Refusal>) {
|
fn check_consent(packet: &Packet, nodes: &BTreeSet<NodeId>, out: &mut Vec<Refusal>) {
|
||||||
|
|
@ -314,25 +305,6 @@ fn check_consent(packet: &Packet, nodes: &BTreeSet<NodeId>, out: &mut Vec<Refusa
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The gloss of the inventory frame that reports an addressee's request.
|
|
||||||
const REQUEST_FRAME_GLOSS: &str = "request_intervention";
|
|
||||||
|
|
||||||
/// V18: a predication whose frame, resolved at the packet's revision, is the
|
|
||||||
/// request frame is named by `addressee_consent`. An unresolved frame is V5's.
|
|
||||||
fn check_requests(packet: &Packet, inventory: &Inventory, out: &mut Vec<Refusal>) {
|
|
||||||
for p in &packet.predications {
|
|
||||||
if inventory
|
|
||||||
.resolve(&p.frame, packet.inventory_rev)
|
|
||||||
.is_some_and(|e| e.gloss == REQUEST_FRAME_GLOSS)
|
|
||||||
&& packet.discourse.addressee_consent.as_ref() != Some(&p.id)
|
|
||||||
{
|
|
||||||
out.push(Refusal::V18RequestWithoutConsent {
|
|
||||||
predication: p.id.clone(),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn node_lists(packet: &Packet) -> [(&Vec<NodeId>, &'static str); 3] {
|
fn node_lists(packet: &Packet) -> [(&Vec<NodeId>, &'static str); 3] {
|
||||||
[
|
[
|
||||||
(&packet.context.participants, "participant"),
|
(&packet.context.participants, "participant"),
|
||||||
|
|
@ -500,29 +472,6 @@ impl<'a> Inventoried<'a> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The one feature check of the speaker-state documents and `with_scene`: V10
|
|
||||||
/// when `rev` has no record, else the rules the packet's own check applies to a
|
|
||||||
/// feature list in `slot` (V5 to V8, V11).
|
|
||||||
pub(crate) fn feature_refusals(
|
|
||||||
inventory: &Inventory,
|
|
||||||
rev: u32,
|
|
||||||
features: &[FeatureValue],
|
|
||||||
slot: Slot,
|
|
||||||
) -> Vec<Refusal> {
|
|
||||||
let mut out = Vec::new();
|
|
||||||
if !inventory.has_revision(rev) {
|
|
||||||
out.push(Refusal::V10UnknownRevision { rev });
|
|
||||||
} else {
|
|
||||||
Inventoried {
|
|
||||||
inv: inventory,
|
|
||||||
rev,
|
|
||||||
out: &mut out,
|
|
||||||
}
|
|
||||||
.features(features, slot);
|
|
||||||
}
|
|
||||||
out
|
|
||||||
}
|
|
||||||
|
|
||||||
fn fits(kind: Option<&ValueKind>, value: &FeatValue) -> bool {
|
fn fits(kind: Option<&ValueKind>, value: &FeatValue) -> bool {
|
||||||
match (kind, value) {
|
match (kind, value) {
|
||||||
(Some(ValueKind::Grade), FeatValue::Grade(_)) => true,
|
(Some(ValueKind::Grade), FeatValue::Grade(_)) => true,
|
||||||
|
|
@ -532,28 +481,6 @@ fn fits(kind: Option<&ValueKind>, value: &FeatValue) -> bool {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// What `with_scene` found for a view: the scene revision `rev` it read, or why
|
|
||||||
/// it selected no relation. Each case is distinct so a missing annotation or a
|
|
||||||
/// store at the wrong path cannot look like a scene that held nothing.
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
|
||||||
pub enum SceneRead {
|
|
||||||
/// `with_scene` has not run.
|
|
||||||
Unread,
|
|
||||||
/// The store holds no scene for the item's group.
|
|
||||||
NoScene,
|
|
||||||
/// The scene was read and the packet names no addressee.
|
|
||||||
NoAddressee { rev: u32 },
|
|
||||||
/// The addressee referent is the speaker themself.
|
|
||||||
SelfAddressed { rev: u32 },
|
|
||||||
/// The scene holds no relation from the speaker to this addressee.
|
|
||||||
NoRelation { rev: u32 },
|
|
||||||
/// The relation was found; its features may be none.
|
|
||||||
Selected {
|
|
||||||
rev: u32,
|
|
||||||
features: Vec<FeatureValue>,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
/// What the lowerer sees: a validated packet's meaning, with no packet id, no
|
/// What the lowerer sees: a validated packet's meaning, with no packet id, no
|
||||||
/// provenance (so no corpus item key, encoder or source) and no trace. Its
|
/// provenance (so no corpus item key, encoder or source) and no trace. Its
|
||||||
/// fields are private and the constructor is crate-private, called only by
|
/// fields are private and the constructor is crate-private, called only by
|
||||||
|
|
@ -610,50 +537,16 @@ pub enum SceneRead {
|
||||||
/// fn read(l: weksa::LoweringInput) { let _ = l.scene; }
|
/// fn read(l: weksa::LoweringInput) { let _ = l.scene; }
|
||||||
/// ```
|
/// ```
|
||||||
///
|
///
|
||||||
/// Scene values come only from the store: the call names neither speaker nor
|
|
||||||
/// scene. The first block is the call with the store; the second hands it a
|
|
||||||
/// scene value and a speaker and fails for that alone.
|
|
||||||
///
|
|
||||||
/// ```
|
|
||||||
/// fn pick(
|
|
||||||
/// l: weksa::LoweringInput,
|
|
||||||
/// i: &weksa::Inventory,
|
|
||||||
/// s: &weksa::SpeakerStateStore,
|
|
||||||
/// c: &weksa::CorpusStore,
|
|
||||||
/// ) {
|
|
||||||
/// let _ = l.with_scene(i, s, c);
|
|
||||||
/// }
|
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// ```compile_fail
|
|
||||||
/// fn pick(
|
|
||||||
/// l: weksa::LoweringInput,
|
|
||||||
/// i: &weksa::Inventory,
|
|
||||||
/// s: &weksa::speaker::Scene,
|
|
||||||
/// e: &weksa::EntityId,
|
|
||||||
/// ) {
|
|
||||||
/// let _ = l.with_scene(i, s, e);
|
|
||||||
/// }
|
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// ```compile_fail
|
|
||||||
/// fn pick(l: weksa::LoweringInput, i: &weksa::Inventory) {
|
|
||||||
/// let _ = l.with_scene(i, vec![]);
|
|
||||||
/// }
|
|
||||||
/// ```
|
|
||||||
///
|
|
||||||
/// The constructor is private: a call from outside the crate compiles only if
|
/// The constructor is private: a call from outside the crate compiles only if
|
||||||
/// `LoweringInput::of` is `pub`. The first block is the same code without the
|
/// `LoweringInput::of` is `pub`. The first block is the same code without the
|
||||||
/// call, so the second fails for that call and nothing else.
|
/// call, so the second fails for that call and nothing else.
|
||||||
///
|
///
|
||||||
/// ```
|
/// ```
|
||||||
/// fn build(p: &weksa::packet::Packet, item: &weksa::CorpusItem) { let _ = (p, item); }
|
/// fn build(p: &weksa::packet::Packet) { let _ = p; }
|
||||||
/// ```
|
/// ```
|
||||||
///
|
///
|
||||||
/// ```compile_fail
|
/// ```compile_fail
|
||||||
/// fn build(p: &weksa::packet::Packet, item: &weksa::CorpusItem) {
|
/// fn build(p: &weksa::packet::Packet) { let _ = weksa::LoweringInput::of(p); }
|
||||||
/// let _ = weksa::LoweringInput::of(p, item);
|
|
||||||
/// }
|
|
||||||
/// ```
|
/// ```
|
||||||
///
|
///
|
||||||
/// A lowering view is never read from bytes: `Deserialize` would let a caller
|
/// A lowering view is never read from bytes: `Deserialize` would let a caller
|
||||||
|
|
@ -681,16 +574,11 @@ pub struct LoweringInput {
|
||||||
constraints: Constraints,
|
constraints: Constraints,
|
||||||
extensions: Vec<Extension>,
|
extensions: Vec<Extension>,
|
||||||
stance: Stance,
|
stance: Stance,
|
||||||
/// The item's speaker and group, as the corpus store holds them: the
|
scene: Vec<FeatureValue>,
|
||||||
/// only source of whose relation `with_scene` selects, and from which scene.
|
|
||||||
speaker: EntityId,
|
|
||||||
group: String,
|
|
||||||
scene: SceneRead,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl LoweringInput {
|
impl LoweringInput {
|
||||||
/// The view of an admitted packet and the corpus item it was minted for.
|
fn of(p: &Packet) -> Self {
|
||||||
pub(crate) fn of(p: &Packet, item: &CorpusItem) -> Self {
|
|
||||||
Self {
|
Self {
|
||||||
inventory_rev: p.inventory_rev,
|
inventory_rev: p.inventory_rev,
|
||||||
kind: p.kind,
|
kind: p.kind,
|
||||||
|
|
@ -702,73 +590,41 @@ impl LoweringInput {
|
||||||
constraints: p.constraints.clone(),
|
constraints: p.constraints.clone(),
|
||||||
extensions: p.extensions.clone(),
|
extensions: p.extensions.clone(),
|
||||||
stance: p.stance.clone(),
|
stance: p.stance.clone(),
|
||||||
speaker: item.speaker.clone(),
|
scene: Vec::new(),
|
||||||
group: item.group.clone(),
|
|
||||||
scene: SceneRead::Unread,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Reads the speaker's relation to the addressee, the only path for scene
|
/// Adds the speaker's relation to the addressee: feature values of slot
|
||||||
/// values into a view. The speaker is the item's speaker and the scene is
|
/// `Scene`, checked like every feature list (V5 to V8, V11) at this view's
|
||||||
/// the latest the store holds for the item's group; the caller names
|
/// inventory revision. They state a relation, so the packet must name an
|
||||||
/// neither. A scene that does not list the item's speaker, or the speaker of
|
/// addressee (V13). Typed ids and grades only, never wording, so a scene
|
||||||
/// any live item of the group (the rule `put_scene` applies), is V16, with
|
/// carries no text to the lowerer. The scene is not part of the packet: a
|
||||||
/// or without an addressee. What the read found is recorded on the view as a
|
/// scene document supplies it, and replacing it replaces the earlier one.
|
||||||
/// [`SceneRead`], never an empty feature list standing for five cases:
|
|
||||||
/// no scene for the group, no addressee, an addressee who is the speaker,
|
|
||||||
/// no such relation, or the relation from the speaker to the addressee
|
|
||||||
/// referent's entity (to `Unnamed` when the referent has none), whose
|
|
||||||
/// features are checked like every feature list in slot `Scene` at this
|
|
||||||
/// view's inventory revision. A read replaces any earlier one. The English
|
|
||||||
/// lowerer ignores it.
|
|
||||||
pub fn with_scene(
|
pub fn with_scene(
|
||||||
mut self,
|
mut self,
|
||||||
inventory: &Inventory,
|
inventory: &Inventory,
|
||||||
state: &SpeakerStateStore,
|
scene: Vec<FeatureValue>,
|
||||||
corpus: &CorpusStore,
|
) -> Result<Self, Vec<Refusal>> {
|
||||||
) -> Result<Self, StoreError> {
|
let mut out = Vec::new();
|
||||||
self.scene = SceneRead::Unread;
|
if !inventory.has_revision(self.inventory_rev) {
|
||||||
let Some(scene) = state.scene(&self.group)? else {
|
out.push(Refusal::V10UnknownRevision {
|
||||||
self.scene = SceneRead::NoScene;
|
rev: self.inventory_rev,
|
||||||
return Ok(self);
|
});
|
||||||
};
|
} else {
|
||||||
let rev = scene.n;
|
Inventoried {
|
||||||
let live = corpus.all()?;
|
inv: inventory,
|
||||||
let live = live
|
rev: self.inventory_rev,
|
||||||
.iter()
|
out: &mut out,
|
||||||
.filter(|i| i.group == self.group && !i.withdrawn)
|
}
|
||||||
.map(|i| &i.speaker);
|
.features(&scene, Slot::Scene);
|
||||||
if scene
|
|
||||||
.uncovered(live.chain(std::iter::once(&self.speaker)))
|
|
||||||
.is_some()
|
|
||||||
{
|
|
||||||
return Err(StoreError::Invalid(vec![Refusal::V16SpeakerNotInScene]));
|
|
||||||
}
|
}
|
||||||
let Some(addressee) = self.discourse.addressee.as_ref() else {
|
if !scene.is_empty() && self.discourse.addressee.is_none() {
|
||||||
self.scene = SceneRead::NoAddressee { rev };
|
out.push(Refusal::V13SceneWithoutAddressee);
|
||||||
return Ok(self);
|
|
||||||
};
|
|
||||||
let entity = self
|
|
||||||
.referents
|
|
||||||
.iter()
|
|
||||||
.find(|r| &r.id == addressee)
|
|
||||||
.and_then(|r| r.entity.as_ref());
|
|
||||||
if entity == Some(&self.speaker) {
|
|
||||||
self.scene = SceneRead::SelfAddressed { rev };
|
|
||||||
return Ok(self);
|
|
||||||
}
|
}
|
||||||
let Some(features) = scene.relation(&self.speaker, entity) else {
|
|
||||||
self.scene = SceneRead::NoRelation { rev };
|
|
||||||
return Ok(self);
|
|
||||||
};
|
|
||||||
let out = feature_refusals(inventory, self.inventory_rev, features, Slot::Scene);
|
|
||||||
if !out.is_empty() {
|
if !out.is_empty() {
|
||||||
return Err(StoreError::Invalid(out));
|
return Err(out);
|
||||||
}
|
}
|
||||||
self.scene = SceneRead::Selected {
|
self.scene = scene;
|
||||||
rev,
|
|
||||||
features: features.to_vec(),
|
|
||||||
};
|
|
||||||
Ok(self)
|
Ok(self)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -802,17 +658,8 @@ impl LoweringInput {
|
||||||
pub fn stance(&self) -> &Stance {
|
pub fn stance(&self) -> &Stance {
|
||||||
&self.stance
|
&self.stance
|
||||||
}
|
}
|
||||||
/// The features of the scene relation from the speaker to the addressee.
|
/// The speaker's relation to the addressee; empty until `with_scene`.
|
||||||
/// Empty for every [`SceneRead`] but `Selected` (and for a selected
|
|
||||||
/// relation that has no features): ask [`scene_read`](Self::scene_read) which.
|
|
||||||
pub fn scene(&self) -> &[FeatureValue] {
|
pub fn scene(&self) -> &[FeatureValue] {
|
||||||
match &self.scene {
|
|
||||||
SceneRead::Selected { features, .. } => features,
|
|
||||||
_ => &[],
|
|
||||||
}
|
|
||||||
}
|
|
||||||
/// What `with_scene` read; `Unread` until it ran.
|
|
||||||
pub fn scene_read(&self) -> &SceneRead {
|
|
||||||
&self.scene
|
&self.scene
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,107 +0,0 @@
|
||||||
//! A line-based tripwire on the Checkpoint door, not a proof. The real defence
|
|
||||||
//! is that `Checkpoint::for_test` is `cfg(test)`, pinned by the compile_fail
|
|
||||||
//! doctest on its current name in `split.rs`. This scan reads `split.rs` line by
|
|
||||||
//! line and catches the common single-line forms: a `pub fn` in `impl
|
|
||||||
//! Checkpoint` whose signature line says `-> Self` or `-> Checkpoint` without
|
|
||||||
//! `cfg(test)` above it, a trait impl for `Checkpoint`, a derive on it, a `pub`
|
|
||||||
//! field in its braces, and a `-> Checkpoint` line in any other `src` file. It
|
|
||||||
//! misses: signatures wrapped over several lines, `Option`, `Box`, `Result` or
|
|
||||||
//! tuple returns from `split.rs`, tuple structs with `pub` fields, and
|
|
||||||
//! constructors made by macros.
|
|
||||||
|
|
||||||
/// Why `src` lets code outside the lib make a `Checkpoint`, if it does.
|
|
||||||
fn door_leaks(split_rs: &str) -> Vec<String> {
|
|
||||||
let mut leaks = Vec::new();
|
|
||||||
let lines: Vec<&str> = split_rs.lines().collect();
|
|
||||||
let mut in_impl = false;
|
|
||||||
let mut in_struct = false;
|
|
||||||
for (n, line) in lines.iter().enumerate() {
|
|
||||||
let t = line.trim();
|
|
||||||
if t.starts_with("impl Checkpoint") {
|
|
||||||
in_impl = true;
|
|
||||||
} else if t.starts_with("impl") && t.contains("Checkpoint") {
|
|
||||||
leaks.push(format!("{}: trait impl: {t}", n + 1));
|
|
||||||
}
|
|
||||||
if t.starts_with("pub struct Checkpoint") {
|
|
||||||
let derived = lines[..n]
|
|
||||||
.iter()
|
|
||||||
.rev()
|
|
||||||
.take_while(|l| l.trim().starts_with("#[") || l.trim().starts_with("///"))
|
|
||||||
.any(|l| l.contains("derive"));
|
|
||||||
if derived {
|
|
||||||
leaks.push(format!("{}: derive on Checkpoint", n + 1));
|
|
||||||
}
|
|
||||||
in_struct = true;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if in_struct {
|
|
||||||
if t.starts_with('}') {
|
|
||||||
in_struct = false;
|
|
||||||
} else if t.starts_with("pub ") {
|
|
||||||
leaks.push(format!("{}: public field: {t}", n + 1));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if in_impl && t.starts_with("pub ") && t.contains("fn ") {
|
|
||||||
let mints = t.contains("-> Self") || t.contains("-> Checkpoint");
|
|
||||||
let test_only = lines[..n]
|
|
||||||
.iter()
|
|
||||||
.rev()
|
|
||||||
.take_while(|l| l.trim().starts_with("#["))
|
|
||||||
.any(|l| l.contains("cfg(test)"));
|
|
||||||
if mints && !test_only {
|
|
||||||
leaks.push(format!("{}: pub constructor: {t}", n + 1));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
leaks
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn no_public_item_mints_a_checkpoint() {
|
|
||||||
let src =
|
|
||||||
std::fs::read_to_string(concat!(env!("CARGO_MANIFEST_DIR"), "/src/split.rs")).unwrap();
|
|
||||||
assert_eq!(door_leaks(&src), Vec::<String>::new());
|
|
||||||
// No other module returns one either.
|
|
||||||
for entry in std::fs::read_dir(concat!(env!("CARGO_MANIFEST_DIR"), "/src")).unwrap() {
|
|
||||||
let path = entry.unwrap().path();
|
|
||||||
if path.extension().is_some_and(|e| e == "rs") && !path.ends_with("split.rs") {
|
|
||||||
let text = std::fs::read_to_string(&path).unwrap();
|
|
||||||
let minting = text
|
|
||||||
.lines()
|
|
||||||
.filter(|l| l.contains("-> Checkpoint") || l.contains("-> Result<Checkpoint"))
|
|
||||||
.count();
|
|
||||||
assert_eq!(minting, 0, "{}", path.display());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn the_scan_sees_each_way_to_open_the_door() {
|
|
||||||
let good = "impl Checkpoint {\n #[cfg(test)]\n pub(crate) fn for_test(r: u32) -> Self { x }\n}\n";
|
|
||||||
assert!(door_leaks(good).is_empty());
|
|
||||||
// Test constructor made public, under its own name or another.
|
|
||||||
let public = "impl Checkpoint {\n pub fn for_test(r: u32) -> Self { x }\n}\n";
|
|
||||||
assert_eq!(door_leaks(public).len(), 1);
|
|
||||||
let renamed = "impl Checkpoint {\n #[cfg(test)]\n #[allow(x)]\n pub fn new(r: u32) -> Checkpoint { x }\n}\n";
|
|
||||||
assert!(
|
|
||||||
door_leaks(renamed).is_empty(),
|
|
||||||
"cfg(test) above another attribute counts"
|
|
||||||
);
|
|
||||||
let renamed = "impl Checkpoint {\n pub fn new(r: u32) -> Checkpoint { x }\n}\n";
|
|
||||||
assert_eq!(door_leaks(renamed).len(), 1);
|
|
||||||
// cfg(test) on the previous item does not cover this one.
|
|
||||||
let stray =
|
|
||||||
"impl Checkpoint {\n #[cfg(test)]\n fn a() {}\n pub fn b() -> Self { x }\n}\n";
|
|
||||||
assert_eq!(door_leaks(stray).len(), 1);
|
|
||||||
assert_eq!(door_leaks("impl Default for Checkpoint {}\n").len(), 1);
|
|
||||||
assert_eq!(door_leaks("impl From<u32> for Checkpoint {}\n").len(), 1);
|
|
||||||
assert_eq!(
|
|
||||||
door_leaks("#[derive(Clone)]\npub struct Checkpoint {\n").len(),
|
|
||||||
1
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
door_leaks("pub struct Checkpoint {\n pub round: u32,\n}\n").len(),
|
|
||||||
1
|
|
||||||
);
|
|
||||||
assert!(door_leaks("pub struct Checkpoint {\n round: u32,\n}\n").is_empty());
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -34,8 +34,7 @@ pub const CONCEPT_LATE: &str = "concept:4";
|
||||||
pub const FRAME: &str = "frame:1";
|
pub const FRAME: &str = "frame:1";
|
||||||
/// An extension entry of value kind Grade, introduced at rev 2.
|
/// An extension entry of value kind Grade, introduced at rev 2.
|
||||||
pub const EXT: &str = "extension:1";
|
pub const EXT: &str = "extension:1";
|
||||||
/// A Grade feature introduced at rev 2 in every content slot (Discourse,
|
/// A Grade feature introduced at rev 2 that sits in every slot.
|
||||||
/// Referent, Predication, Relation).
|
|
||||||
pub const FEAT_ALL: &str = "feature:50";
|
pub const FEAT_ALL: &str = "feature:50";
|
||||||
|
|
||||||
pub const PACKET_ID: &str = "packet:0123456789abcdef";
|
pub const PACKET_ID: &str = "packet:0123456789abcdef";
|
||||||
|
|
@ -114,90 +113,6 @@ pub fn entities_in(dir: &std::path::Path) -> EntityStore {
|
||||||
store
|
store
|
||||||
}
|
}
|
||||||
|
|
||||||
/// An empty ontology store at `<dir>/ontology.cc`, made if absent.
|
|
||||||
pub fn ontology_in(dir: &std::path::Path) -> std::path::PathBuf {
|
|
||||||
let path = dir.join("ontology.cc");
|
|
||||||
if !path.exists() {
|
|
||||||
std::fs::write(&path, [0x90]).unwrap();
|
|
||||||
}
|
|
||||||
path
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A new inventory store at `path` holding the Dao inventory as it stood at
|
|
||||||
/// rev 3 (seed, proof, the Dao revision): the loop tests' fixed head, whose
|
|
||||||
/// round and candidate revisions they pin. The committed
|
|
||||||
/// `data/daodejing/inventory.cc` is rev 4 and is read by the corpus tests.
|
|
||||||
pub fn dao_rev3_at(path: &std::path::Path) {
|
|
||||||
let mut store = weksa::InventoryStore::open(path).unwrap();
|
|
||||||
for (rev, entries) in [
|
|
||||||
weksa::inventory::seed(),
|
|
||||||
weksa::inventory::proof_revision(),
|
|
||||||
weksa::daodejing::revision(),
|
|
||||||
] {
|
|
||||||
store.admit_revision(rev, entries).unwrap();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// An empty speaker-state store beside `ontology` (same directory), made if absent.
|
|
||||||
pub fn speaker_state_beside(ontology: &std::path::Path) -> std::path::PathBuf {
|
|
||||||
let path = ontology.with_file_name("speaker-state.cc");
|
|
||||||
if !path.exists() {
|
|
||||||
std::fs::write(&path, [0x90]).unwrap();
|
|
||||||
}
|
|
||||||
path
|
|
||||||
}
|
|
||||||
|
|
||||||
/// An empty speaker-state store at `<dir>/speaker-state.cc`, made if absent.
|
|
||||||
pub fn speaker_state_in(dir: &std::path::Path) -> std::path::PathBuf {
|
|
||||||
speaker_state_beside(&dir.join("ontology.cc"))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A run over the stores `world()` and `corpus()` keep in `dir`: `c.cc`,
|
|
||||||
/// `i.cc` and `entities.cc`. Read at the call, so create it after the items.
|
|
||||||
/// The run's en-US panel needs a live Train and a live Validation item: a
|
|
||||||
/// corpus short of one gets `item:panel:<n>` added, and no other item.
|
|
||||||
pub fn run_over(dir: &std::path::Path) -> weksa::run::RunStores {
|
|
||||||
for name in ["c.cc", "i.cc", "entities.cc"] {
|
|
||||||
if !dir.join(name).exists() {
|
|
||||||
std::fs::write(dir.join(name), [0x90]).unwrap();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let entities = entities_in(dir);
|
|
||||||
ontology_in(dir);
|
|
||||||
let mut corpus = CorpusStore::open(&dir.join("c.cc")).unwrap();
|
|
||||||
for (n, split) in [(1, weksa::Split::Train), (2, weksa::Split::Validation)] {
|
|
||||||
let held = corpus
|
|
||||||
.all()
|
|
||||||
.unwrap()
|
|
||||||
.iter()
|
|
||||||
.any(|i| SplitScheme::V1.live(i) == Some(split));
|
|
||||||
if !held {
|
|
||||||
let group = group_in_bucket(if n == 1 { 10 } else { 50 });
|
|
||||||
let key = format!("item:panel:{n}");
|
|
||||||
let text = format!("Panel fixture line {n}.");
|
|
||||||
corpus.add(item_in(&key, &text, &group, cast()), &entities).unwrap();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
drop(corpus);
|
|
||||||
weksa::run::RunStores::create(
|
|
||||||
&dir.join("run"),
|
|
||||||
"t",
|
|
||||||
weksa::run::RunPaths {
|
|
||||||
corpus: &dir.join("c.cc"),
|
|
||||||
inventory: &dir.join("i.cc"),
|
|
||||||
entities: &dir.join("entities.cc"),
|
|
||||||
ontology: &dir.join("ontology.cc"),
|
|
||||||
speaker_state: &speaker_state_beside(&dir.join("ontology.cc")),
|
|
||||||
},
|
|
||||||
"fake-model",
|
|
||||||
"bag",
|
|
||||||
weksa::run::LowererReach::ToolLessRelay,
|
|
||||||
weksa::corpus::ModelFamily::Unrecorded,
|
|
||||||
&["en-US".to_string()],
|
|
||||||
)
|
|
||||||
.unwrap()
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn item_in(key: &str, text: &str, group: &str, source: Source) -> CorpusItem {
|
pub fn item_in(key: &str, text: &str, group: &str, source: Source) -> CorpusItem {
|
||||||
CorpusItem {
|
CorpusItem {
|
||||||
id: key.into(),
|
id: key.into(),
|
||||||
|
|
@ -205,7 +120,6 @@ pub fn item_in(key: &str, text: &str, group: &str, source: Source) -> CorpusItem
|
||||||
language: "en-US".into(),
|
language: "en-US".into(),
|
||||||
speaker: eid("nibu"),
|
speaker: eid("nibu"),
|
||||||
names: vec![],
|
names: vec![],
|
||||||
names_by: None,
|
|
||||||
group: group.into(),
|
group: group.into(),
|
||||||
prior_turns: vec![],
|
prior_turns: vec![],
|
||||||
source,
|
source,
|
||||||
|
|
@ -272,7 +186,6 @@ pub fn world() -> World {
|
||||||
let dir = tempfile::tempdir().unwrap();
|
let dir = tempfile::tempdir().unwrap();
|
||||||
let mut corpus = CorpusStore::open(&dir.path().join("c.cc")).unwrap();
|
let mut corpus = CorpusStore::open(&dir.path().join("c.cc")).unwrap();
|
||||||
let entities = entities_in(dir.path());
|
let entities = entities_in(dir.path());
|
||||||
ontology_in(dir.path());
|
|
||||||
let train = group_in_bucket(10);
|
let train = group_in_bucket(10);
|
||||||
let mut first = item_in("item:nibu:1", "Nibu says mind the hatch.", &train, cast());
|
let mut first = item_in("item:nibu:1", "Nibu says mind the hatch.", &train, cast());
|
||||||
first.names = vec![NameSpan {
|
first.names = vec![NameSpan {
|
||||||
|
|
@ -281,10 +194,6 @@ pub fn world() -> World {
|
||||||
end: 4,
|
end: 4,
|
||||||
}];
|
}];
|
||||||
corpus.add(first, &entities).unwrap();
|
corpus.add(first, &entities).unwrap();
|
||||||
// A run's panel needs a live Validation item as well as a Train one.
|
|
||||||
let validation = group_in_bucket(50);
|
|
||||||
let second = item_in("item:panel:1", "Nibu says watch the ladder.", &validation, cast());
|
|
||||||
corpus.add(second, &entities).unwrap();
|
|
||||||
let mut inventory = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
let mut inventory = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
||||||
let (rev1, entries) = seed();
|
let (rev1, entries) = seed();
|
||||||
inventory.admit_revision(rev1, entries).unwrap();
|
inventory.admit_revision(rev1, entries).unwrap();
|
||||||
|
|
@ -309,6 +218,7 @@ pub fn world() -> World {
|
||||||
Slot::Referent,
|
Slot::Referent,
|
||||||
Slot::Predication,
|
Slot::Predication,
|
||||||
Slot::Relation,
|
Slot::Relation,
|
||||||
|
Slot::Stance,
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
},
|
},
|
||||||
|
|
@ -421,25 +331,3 @@ pub fn good() -> Packet {
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A deterministic bag-of-letters embedder, so cosine differs with the text.
|
|
||||||
pub struct Bag;
|
|
||||||
|
|
||||||
impl weksa::Embedder for Bag {
|
|
||||||
fn model(&self) -> &str {
|
|
||||||
"bag"
|
|
||||||
}
|
|
||||||
fn embed(&self, texts: &[&str]) -> anyhow::Result<Vec<Vec<f32>>> {
|
|
||||||
Ok(texts
|
|
||||||
.iter()
|
|
||||||
.map(|t| {
|
|
||||||
let mut v = vec![0.0; 27];
|
|
||||||
v[26] = 1.0;
|
|
||||||
for c in t.chars().filter(char::is_ascii_alphabetic) {
|
|
||||||
v[(c.to_ascii_lowercase() as u8 - b'a') as usize] += 1.0;
|
|
||||||
}
|
|
||||||
v
|
|
||||||
})
|
|
||||||
.collect())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,11 +1,11 @@
|
||||||
//! The Dao De Jing test bed (`data/daodejing`): its Legge items carry full
|
//! The Dao De Jing test bed (`data/daodejing`): its Legge items carry full
|
||||||
//! public-domain provenance, and its packets are admitted
|
//! public-domain provenance, and its packets are admitted and lower to the
|
||||||
//! (`docs/targets/dao-de-jing-test-bed.md`).
|
//! English the first golf-loop run scored (`docs/targets/dao-de-jing-test-bed.md`).
|
||||||
|
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
use weksa::daodejing::packets;
|
use weksa::daodejing::packets;
|
||||||
use weksa::{CorpusStore, EntityStore, InventoryStore, PacketStore, Source};
|
use weksa::{CorpusStore, EntityStore, InventoryStore, PacketStore, Source, lower};
|
||||||
|
|
||||||
fn root() -> std::path::PathBuf {
|
fn root() -> std::path::PathBuf {
|
||||||
Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/daodejing")
|
Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/daodejing")
|
||||||
|
|
@ -39,15 +39,30 @@ fn every_legge_item_names_licence_jurisdiction_translator_and_edition_year() {
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn the_slice_packets_are_admitted_and_the_committed_store_holds_them() {
|
fn the_slice_packets_are_admitted_and_lower_to_the_scored_english() {
|
||||||
let inventory = InventoryStore::open(&root().join("inventory.cc")).unwrap();
|
let inventory = InventoryStore::open(&root().join("inventory.cc")).unwrap();
|
||||||
let corpus = CorpusStore::open(&root().join("corpus.cc")).unwrap();
|
let corpus = CorpusStore::open(&root().join("corpus.cc")).unwrap();
|
||||||
let entities = EntityStore::open(&root().join("entity.cc")).unwrap();
|
let entities = EntityStore::open(&root().join("entity.cc")).unwrap();
|
||||||
let dir = tempfile::tempdir().unwrap();
|
let dir = tempfile::tempdir().unwrap();
|
||||||
let mut store = PacketStore::open(&dir.path().join("p.cc")).unwrap();
|
let mut store = PacketStore::open(&dir.path().join("p.cc")).unwrap();
|
||||||
for p in packets() {
|
let inv = inventory.inventory().unwrap();
|
||||||
store.admit(p, &inventory, &corpus, &entities).unwrap();
|
let lowered: Vec<String> = packets()
|
||||||
}
|
.into_iter()
|
||||||
|
.map(|p| {
|
||||||
|
let view = store.admit(p, &inventory, &corpus, &entities).unwrap();
|
||||||
|
lower(&view, &inv, &entities).unwrap()
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
assert_eq!(
|
||||||
|
lowered,
|
||||||
|
[
|
||||||
|
"The sage empties the minds. The sage fills the bellies. The sage weakens the wills. The sage strengthens the bones.",
|
||||||
|
"The heaven is long-enduring. The earth continues long.",
|
||||||
|
"The highest excellence is like the water.",
|
||||||
|
"The words that are strictly true seem paradoxical.",
|
||||||
|
"The space between heaven and earth may be compared to the bellows.",
|
||||||
|
]
|
||||||
|
);
|
||||||
// The committed packet store holds exactly what the module builds.
|
// The committed packet store holds exactly what the module builds.
|
||||||
let stored = PacketStore::open(&root().join("packets.cc")).unwrap();
|
let stored = PacketStore::open(&root().join("packets.cc")).unwrap();
|
||||||
for p in packets() {
|
for p in packets() {
|
||||||
|
|
|
||||||
|
|
@ -33,6 +33,20 @@ fn assert_clean_and_distinct<T: std::fmt::Display + std::fmt::Debug>(all: Vec<T>
|
||||||
#[test]
|
#[test]
|
||||||
fn lower_error_display_names_the_variant_and_no_identifier() {
|
fn lower_error_display_names_the_variant_and_no_identifier() {
|
||||||
assert_clean_and_distinct(vec![
|
assert_clean_and_distinct(vec![
|
||||||
|
LowerError::Unrendered { what: SENTINEL },
|
||||||
|
LowerError::NoRow { id: inv() },
|
||||||
|
LowerError::GlossMismatch {
|
||||||
|
id: inv(),
|
||||||
|
row: SENTINEL,
|
||||||
|
},
|
||||||
|
LowerError::UnknownRole {
|
||||||
|
frame: inv(),
|
||||||
|
role: SENTINEL.into(),
|
||||||
|
},
|
||||||
|
LowerError::MissingRole {
|
||||||
|
predication: node(),
|
||||||
|
role: SENTINEL.into(),
|
||||||
|
},
|
||||||
LowerError::UnknownEntity { entity: entity() },
|
LowerError::UnknownEntity { entity: entity() },
|
||||||
LowerError::NoForm { entity: entity() },
|
LowerError::NoForm { entity: entity() },
|
||||||
LowerError::NoPronunciation { entity: entity() },
|
LowerError::NoPronunciation { entity: entity() },
|
||||||
|
|
@ -46,6 +60,12 @@ fn lower_error_display_names_the_variant_and_no_identifier() {
|
||||||
entity: entity(),
|
entity: entity(),
|
||||||
at: 4242,
|
at: 4242,
|
||||||
},
|
},
|
||||||
|
LowerError::ConsentMissing {
|
||||||
|
predication: node(),
|
||||||
|
},
|
||||||
|
LowerError::AskerNotAddressee {
|
||||||
|
predication: node(),
|
||||||
|
},
|
||||||
LowerError::EntityRead {
|
LowerError::EntityRead {
|
||||||
entity: entity(),
|
entity: entity(),
|
||||||
why: SENTINEL.into(),
|
why: SENTINEL.into(),
|
||||||
|
|
@ -97,19 +117,13 @@ fn refusal_display_names_the_rule_and_no_identifier() {
|
||||||
count: 4242,
|
count: 4242,
|
||||||
max: 4242,
|
max: 4242,
|
||||||
},
|
},
|
||||||
|
Refusal::V13SceneWithoutAddressee,
|
||||||
Refusal::V14ConsentWithoutAddressee,
|
Refusal::V14ConsentWithoutAddressee,
|
||||||
Refusal::V14ConsentNotAPredication { id: node() },
|
Refusal::V14ConsentNotAPredication { id: node() },
|
||||||
Refusal::V15ConsentSource {
|
Refusal::V15ConsentSource {
|
||||||
source: ProvenanceSource::ImportedCorpus,
|
source: ProvenanceSource::ImportedCorpus,
|
||||||
},
|
},
|
||||||
Refusal::V16SpeakerNotInScene,
|
|
||||||
Refusal::V17AddresseeNotAReferent { id: node() },
|
|
||||||
Refusal::V18RequestWithoutConsent { predication: node() },
|
|
||||||
];
|
];
|
||||||
assert_eq!(
|
|
||||||
Refusal::V16SpeakerNotInScene.to_string(),
|
|
||||||
"V16: the scene does not cover the speaker of every live item of its group"
|
|
||||||
);
|
|
||||||
for r in &all {
|
for r in &all {
|
||||||
assert!(r.to_string().starts_with(r.rule()), "{r:?}");
|
assert!(r.to_string().starts_with(r.rule()), "{r:?}");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -502,96 +502,3 @@ fn a_malformed_pronunciation_is_refused() {
|
||||||
}
|
}
|
||||||
assert_eq!(store.get(&eid("kay")).unwrap(), None);
|
assert_eq!(store.get(&eid("kay")).unwrap(), None);
|
||||||
}
|
}
|
||||||
|
|
||||||
const LANTERN: &str = r#"{"slug":"lantern-wager","language":"en-US","admitted_on":"2026-11-13",
|
|
||||||
"source":{"PublicDomain":{"work":"The Lantern Wager","author":"A. Invented","edition_url":"https://example.invalid/lantern","licence":"public domain","jurisdiction":"US","translator":"none","edition_year":1890}},
|
|
||||||
"cast":[{"entity":"entity:orsa","forms":["Orsa"]},{"entity":"entity:tamsin","forms":["Tamsin","Tam"]}],
|
|
||||||
"lines":[{"group":"s1","speaker":"entity:orsa","text":"Tamsin, bring the lantern."},
|
|
||||||
{"group":"s1","speaker":"entity:tamsin","text":"Tamsin's lantern, said Orsa."},
|
|
||||||
{"group":"s2","speaker":"entity:orsa","text":"Quiet, both of you."}]}"#;
|
|
||||||
|
|
||||||
fn marked(line: u32, text: &str) -> weksa::encode::MarkedLine {
|
|
||||||
weksa::encode::MarkedLine { line, marked: text.into() }
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn an_imported_work_passes_the_store_doors() {
|
|
||||||
use weksa::Annotator;
|
|
||||||
use weksa::corpus::ModelFamily;
|
|
||||||
let d = tempfile::tempdir().unwrap();
|
|
||||||
let mut entities = EntityStore::open(&d.path().join("e.cc")).unwrap();
|
|
||||||
let mut corpus = CorpusStore::open(&d.path().join("c.cc")).unwrap();
|
|
||||||
let model = Annotator::Model { family: ModelFamily::Anthropic, generator: "g".into() };
|
|
||||||
let work = || weksa::encode::WorkDraft::parse(LANTERN).unwrap();
|
|
||||||
let (cast, items) = work()
|
|
||||||
.into_import(&[marked(1, "[[entity:tamsin|Tamsin]], bring the lantern.")], &model)
|
|
||||||
.unwrap();
|
|
||||||
for e in cast {
|
|
||||||
assert_eq!(entities.put(e).unwrap(), AddOutcome::Added);
|
|
||||||
}
|
|
||||||
for i in items.clone() {
|
|
||||||
assert_eq!(corpus.add(i, &entities).unwrap(), AddOutcome::Added);
|
|
||||||
}
|
|
||||||
drop(corpus);
|
|
||||||
let mut corpus = CorpusStore::open(&d.path().join("c.cc")).unwrap();
|
|
||||||
let stored = corpus.get("item:lantern-wager:1").unwrap().unwrap();
|
|
||||||
assert_eq!(stored.names_by, Some(model.clone()));
|
|
||||||
assert_eq!(stored.names.len(), 1);
|
|
||||||
assert_eq!(corpus.add(items[0].clone(), &entities).unwrap(), AddOutcome::Unchanged);
|
|
||||||
|
|
||||||
// A form the entity does not list is the store's refusal, not the importer's.
|
|
||||||
let (_, bad) = work()
|
|
||||||
.into_import(&[marked(2, "[[entity:tamsin|Tamsin's]] lantern, said Orsa.")], &model)
|
|
||||||
.unwrap();
|
|
||||||
assert!(matches!(
|
|
||||||
refusal(corpus.add(bad[0].clone(), &entities)),
|
|
||||||
StoreRefusal::NameSpanNotAForm { .. }
|
|
||||||
));
|
|
||||||
|
|
||||||
// An operator run over another line labels its own spans.
|
|
||||||
let (_, op) = work()
|
|
||||||
.into_import(&[marked(3, "Quiet, both of you.")], &Annotator::Operator)
|
|
||||||
.unwrap();
|
|
||||||
assert_eq!(corpus.add(op[0].clone(), &entities).unwrap(), AddOutcome::Added);
|
|
||||||
assert_eq!(
|
|
||||||
corpus.get("item:lantern-wager:3").unwrap().unwrap().names_by,
|
|
||||||
Some(Annotator::Operator)
|
|
||||||
);
|
|
||||||
// The first run's line keeps its label.
|
|
||||||
assert_eq!(corpus.get("item:lantern-wager:1").unwrap().unwrap().names_by, Some(model));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn committed_items_read_with_no_names_label() {
|
|
||||||
let data = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data");
|
|
||||||
for sub in ["daodejing", "original"] {
|
|
||||||
let d = tempfile::tempdir().unwrap();
|
|
||||||
let copy = d.path().join("c.cc");
|
|
||||||
std::fs::copy(data.join(sub).join("corpus.cc"), ©).unwrap();
|
|
||||||
let items = CorpusStore::open(©).unwrap().all().unwrap();
|
|
||||||
assert!(!items.is_empty(), "{sub}");
|
|
||||||
assert!(items.iter().all(|i| i.names_by.is_none()), "{sub}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_blank_names_by_generator_is_refused_by_the_corpus_door() {
|
|
||||||
use weksa::Annotator;
|
|
||||||
use weksa::corpus::ModelFamily;
|
|
||||||
let d = tempfile::tempdir().unwrap();
|
|
||||||
let mut entities = EntityStore::open(&d.path().join("e.cc")).unwrap();
|
|
||||||
let mut corpus = CorpusStore::open(&d.path().join("c.cc")).unwrap();
|
|
||||||
let blank = Annotator::Model { family: ModelFamily::Anthropic, generator: " ".into() };
|
|
||||||
let (cast, items) = weksa::encode::WorkDraft::parse(LANTERN)
|
|
||||||
.unwrap()
|
|
||||||
.into_import(&[marked(3, "Quiet, both of you.")], &blank)
|
|
||||||
.unwrap();
|
|
||||||
for e in cast {
|
|
||||||
entities.put(e).unwrap();
|
|
||||||
}
|
|
||||||
assert!(matches!(
|
|
||||||
refusal(corpus.add(items[0].clone(), &entities)),
|
|
||||||
StoreRefusal::Malformed { .. }
|
|
||||||
));
|
|
||||||
assert!(corpus.all().unwrap().is_empty());
|
|
||||||
}
|
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,16 +1,22 @@
|
||||||
//! The entity port, `pronounce` and `phones` against the committed stores and
|
//! The English lowerer against the committed inventory (revision 2) and the
|
||||||
//! the `nibu-proof-001` packet.
|
//! `nibu-proof-001` packet, and the rules that keep a table row from speaking
|
||||||
|
//! for a meaning it does not carry.
|
||||||
|
|
||||||
mod common;
|
mod common;
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
|
use common::{iv, nd};
|
||||||
use weksa::entity::{Entity, EntityOrigin, LanguageForms};
|
use weksa::entity::{Entity, EntityOrigin, LanguageForms};
|
||||||
use weksa::packet::{EntityId, Packet, proof_packet};
|
use weksa::inventory::proof_revision;
|
||||||
|
use weksa::packet::{EntityId, FeatValue, FeatureValue, Packet, proof_packet};
|
||||||
use weksa::{
|
use weksa::{
|
||||||
CorpusStore, EntityForms, EntityStore, Inventory, InventoryStore, LowerError, LoweringInput,
|
CorpusStore, EntityForms, EntityStore, Inventory, InventoryStore, LowerError, LoweringInput,
|
||||||
PacketStore, PhoneInventory, PhoneLine, Pronunciation, Stress, pronounce,
|
PacketStore, PhoneInventory, PhoneLine, Pronunciation, Stress, lower, pronounce, validate,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const PROOF: &str = "Warning: you may endanger yourself by touching the control. \
|
||||||
|
Nibu will stop that.";
|
||||||
|
|
||||||
fn committed() -> (Inventory, tempfile::TempDir) {
|
fn committed() -> (Inventory, tempfile::TempDir) {
|
||||||
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../..");
|
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../..");
|
||||||
let inv = InventoryStore::open(&root.join("data/interlingua/inventory.cc")).unwrap();
|
let inv = InventoryStore::open(&root.join("data/interlingua/inventory.cc")).unwrap();
|
||||||
|
|
@ -42,6 +48,181 @@ fn committed_entities() -> EntityStore {
|
||||||
EntityStore::open(&root.join("data/entity/entity.cc")).unwrap()
|
EntityStore::open(&root.join("data/entity/entity.cc")).unwrap()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn lowered(packet: &Packet, inv: &Inventory) -> Result<String, LowerError> {
|
||||||
|
lower(&view(packet, inv), inv, &committed_entities())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_proof_packet_is_admitted_by_the_store_and_lowers_to_its_sentences() {
|
||||||
|
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../..");
|
||||||
|
let (inv, dir) = committed();
|
||||||
|
let inventory = InventoryStore::open(&root.join("data/interlingua/inventory.cc")).unwrap();
|
||||||
|
let corpus = CorpusStore::open(&root.join("data/corpus/corpus.cc")).unwrap();
|
||||||
|
let entities = EntityStore::open(&root.join("data/entity/entity.cc")).unwrap();
|
||||||
|
let mut packets = PacketStore::open(&dir.path().join("p.cc")).unwrap();
|
||||||
|
packets
|
||||||
|
.admit(proof_packet(), &inventory, &corpus, &entities)
|
||||||
|
.unwrap();
|
||||||
|
let stored = packets.get("packet:6e6962752d303031").unwrap().unwrap();
|
||||||
|
assert_eq!(lowered(&stored, &inv).unwrap(), PROOF);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn only_the_addressee_is_you_and_is_reflexive() {
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
let mut p = proof_packet();
|
||||||
|
p.discourse.addressee = None;
|
||||||
|
assert_eq!(
|
||||||
|
lowered(&p, &inv).unwrap(),
|
||||||
|
"Warning: the player may endanger the player by touching the control. \
|
||||||
|
Nibu will stop that."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_speech_act_means_no_opener_and_a_capital() {
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
let mut p = proof_packet();
|
||||||
|
p.discourse.speech_act = None;
|
||||||
|
assert!(lowered(&p, &inv).unwrap().starts_with("You may endanger"));
|
||||||
|
p.discourse.speech_act = Some(iv("speech_act:1"));
|
||||||
|
assert!(lowered(&p, &inv).unwrap().starts_with("You may endanger"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_packet_carrying_what_the_lowerer_does_not_render_is_refused() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let mut store = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
||||||
|
let (rev1, entries) = weksa::inventory::seed();
|
||||||
|
store.admit_revision(rev1, entries).unwrap();
|
||||||
|
let (rev2, entries) = proof_revision();
|
||||||
|
store.admit_revision(rev2, entries).unwrap();
|
||||||
|
let mut ext = common::entry("extension:1", 3);
|
||||||
|
ext.value_kind = Some(weksa::ValueKind::Grade);
|
||||||
|
common::admit(&mut store, 3, vec![ext]);
|
||||||
|
let inv = store.inventory().unwrap();
|
||||||
|
let discourse = |p: &mut Packet| {
|
||||||
|
p.discourse.features.push(FeatureValue {
|
||||||
|
feature: iv("feature:6"),
|
||||||
|
value: FeatValue::Grade(common::grade(300)),
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let predication = |p: &mut Packet| {
|
||||||
|
p.predications[0].features.push(FeatureValue {
|
||||||
|
feature: iv("feature:1"),
|
||||||
|
value: FeatValue::Option(common::lab("negative")),
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let relation = |p: &mut Packet| {
|
||||||
|
p.relations.push(weksa::packet::Relation {
|
||||||
|
relation: iv("relation:8"),
|
||||||
|
from: nd("n4"),
|
||||||
|
to: nd("n5"),
|
||||||
|
features: vec![],
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let extension = |p: &mut Packet| {
|
||||||
|
p.inventory_rev = 3;
|
||||||
|
p.extensions.push(weksa::packet::Extension {
|
||||||
|
key: iv("extension:1"),
|
||||||
|
value: FeatValue::Grade(common::grade(3)),
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let referent_feature = |p: &mut Packet| {
|
||||||
|
p.referents[0].features.push(FeatureValue {
|
||||||
|
feature: iv("feature:6"),
|
||||||
|
value: FeatValue::Grade(common::grade(1)),
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let no_predication = |p: &mut Packet| p.predications.clear();
|
||||||
|
let extra_role = |p: &mut Packet| {
|
||||||
|
let filler = p.predications[0].roles[0].filler.clone();
|
||||||
|
p.predications[0].roles.push(weksa::packet::RoleFill {
|
||||||
|
role: iv("role:3"),
|
||||||
|
filler,
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let consent = |p: &mut Packet| p.discourse.addressee_consent = Some(nd("n5"));
|
||||||
|
let lone_referent = |p: &mut Packet| {
|
||||||
|
let mut r = p.referents[0].clone();
|
||||||
|
r.id = nd("n99");
|
||||||
|
p.referents.push(r)
|
||||||
|
};
|
||||||
|
for (what, edit) in [
|
||||||
|
("features", &discourse as &dyn Fn(&mut Packet)),
|
||||||
|
("features", &predication),
|
||||||
|
("addressee consent", &consent),
|
||||||
|
("relations", &relation),
|
||||||
|
("extensions", &extension),
|
||||||
|
("no predication", &no_predication),
|
||||||
|
("a role the frame does not name", &extra_role),
|
||||||
|
("a referent no role mentions", &lone_referent),
|
||||||
|
] {
|
||||||
|
let mut p = proof_packet();
|
||||||
|
edit(&mut p);
|
||||||
|
assert_eq!(
|
||||||
|
lower(&admitted(&p, &store), &inv, &committed_entities()),
|
||||||
|
Err(LowerError::Unrendered { what }),
|
||||||
|
"{what}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// The seed lets no feature sit on a referent, so that arm is refused by validate.
|
||||||
|
let mut p = proof_packet();
|
||||||
|
referent_feature(&mut p);
|
||||||
|
assert!(validate(&p, &inv).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_unknown_row_and_an_unfilled_role_are_refused_by_name() {
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
let mut p = proof_packet();
|
||||||
|
p.discourse.speech_act = Some(iv("speech_act:3"));
|
||||||
|
assert_eq!(
|
||||||
|
lowered(&p, &inv),
|
||||||
|
Err(LowerError::NoRow {
|
||||||
|
id: iv("speech_act:3")
|
||||||
|
})
|
||||||
|
);
|
||||||
|
let mut p = proof_packet();
|
||||||
|
p.predications[0].roles.pop();
|
||||||
|
assert_eq!(
|
||||||
|
lowered(&p, &inv),
|
||||||
|
Err(LowerError::MissingRole {
|
||||||
|
predication: nd("n4"),
|
||||||
|
role: "patient".into()
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_row_cannot_speak_for_an_id_whose_gloss_differs() {
|
||||||
|
// Revision 2 with concept:4 glossed "hatch" in place of "control": the row
|
||||||
|
// for concept:4 says "control" and must not render it.
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let mut store = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
||||||
|
let (rev1, entries) = weksa::inventory::seed();
|
||||||
|
store.admit_revision(rev1, entries).unwrap();
|
||||||
|
let (rev2, mut entries) = proof_revision();
|
||||||
|
for e in &mut entries {
|
||||||
|
if e.id == iv("concept:4") {
|
||||||
|
e.gloss = "hatch".into();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
store.admit_revision(rev2, entries).unwrap();
|
||||||
|
let inv = store.inventory().unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
lower(
|
||||||
|
&admitted(&proof_packet(), &store),
|
||||||
|
&inv,
|
||||||
|
&committed_entities()
|
||||||
|
),
|
||||||
|
Err(LowerError::GlossMismatch {
|
||||||
|
id: iv("concept:4"),
|
||||||
|
row: "control"
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/// An entity port a test controls: ids to entities, and an optional read fault.
|
/// An entity port a test controls: ids to entities, and an optional read fault.
|
||||||
#[derive(Default)]
|
#[derive(Default)]
|
||||||
struct Entities {
|
struct Entities {
|
||||||
|
|
@ -76,6 +257,109 @@ fn nibu(forms: &[(&str, &[&str])]) -> Entity {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn through(entities: &Entities) -> Result<String, LowerError> {
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
lower(&view(&proof_packet(), &inv), &inv, entities)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_name_survives_lowering_verbatim_where_the_concept_word_was() {
|
||||||
|
// The entity's own form, not "the ship mind", and not re-cased or inflected:
|
||||||
|
// the form is lowercase and stays so mid-sentence.
|
||||||
|
let mut e = Entities::default();
|
||||||
|
let n = nibu(&[("en-US", &["nibu-of-the-deep"])]);
|
||||||
|
e.held.push(n);
|
||||||
|
// It opens the second sentence, where a template word would be capitalised.
|
||||||
|
assert_eq!(
|
||||||
|
through(&e).unwrap(),
|
||||||
|
"Warning: you may endanger yourself by touching the control. \
|
||||||
|
nibu-of-the-deep will stop that."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn only_the_target_languages_citation_form_is_chosen() {
|
||||||
|
// Forms after the first are inflected shapes no English frame asks for; a
|
||||||
|
// form under another language tag is never read, whatever order it is in.
|
||||||
|
let mut e = Entities::default();
|
||||||
|
let n = nibu(&[
|
||||||
|
("sa-Latn", &["Nibuwa"]),
|
||||||
|
("en-US", &["Nibu", "Nibu's", "Nibus"]),
|
||||||
|
]);
|
||||||
|
e.held.push(n);
|
||||||
|
assert_eq!(through(&e).unwrap(), PROOF);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_entity_the_port_does_not_hold_is_refused_not_replaced_by_its_concept_word() {
|
||||||
|
let id = EntityId::parse("entity:nibu").unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
through(&Entities::default()),
|
||||||
|
Err(LowerError::UnknownEntity { entity: id })
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_entity_with_no_target_language_form_is_refused() {
|
||||||
|
let mut e = Entities::default();
|
||||||
|
let n = nibu(&[("sa-Latn", &["Nibuwa"])]);
|
||||||
|
let id = n.id.clone();
|
||||||
|
e.held.push(n);
|
||||||
|
assert_eq!(through(&e), Err(LowerError::NoForm { entity: id }));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_failing_entity_port_is_refused_by_name() {
|
||||||
|
let e = Entities {
|
||||||
|
fault: true,
|
||||||
|
..Entities::default()
|
||||||
|
};
|
||||||
|
assert_eq!(
|
||||||
|
through(&e),
|
||||||
|
Err(LowerError::EntityRead {
|
||||||
|
entity: EntityId::parse("entity:nibu").unwrap(),
|
||||||
|
why: "store unreadable".into()
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_addressee_entity_still_lowers_to_you_and_is_never_looked_up() {
|
||||||
|
// Naming the addressee would break "you"/"yourself": the port is not asked,
|
||||||
|
// so a port that faults on every read changes nothing.
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
let mut p = proof_packet();
|
||||||
|
p.referents[0].entity = Some(EntityId::parse("entity:nibu").unwrap());
|
||||||
|
p.referents[2].entity = None;
|
||||||
|
let faulting = Entities {
|
||||||
|
fault: true,
|
||||||
|
..Entities::default()
|
||||||
|
};
|
||||||
|
assert_eq!(
|
||||||
|
lower(&view(&p, &inv), &inv, &faulting).unwrap(),
|
||||||
|
"Warning: you may endanger yourself by touching the control. \
|
||||||
|
The ship mind will stop that."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_purpose_the_frame_does_not_say_is_refused_not_dropped() {
|
||||||
|
// The intervention row says "will stop"; a packet that also fills the
|
||||||
|
// beneficiary role would assert a purpose the text does not carry.
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
let mut p = proof_packet();
|
||||||
|
p.predications[1].roles.push(weksa::packet::RoleFill {
|
||||||
|
role: iv("role:8"),
|
||||||
|
filler: nd("n1"),
|
||||||
|
});
|
||||||
|
assert_eq!(
|
||||||
|
lowered(&p, &inv),
|
||||||
|
Err(LowerError::Unrendered {
|
||||||
|
what: "a role the frame does not name"
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
fn ruled(ipa: &str, rejected: &[&str]) -> Pronunciation {
|
fn ruled(ipa: &str, rejected: &[&str]) -> Pronunciation {
|
||||||
Pronunciation {
|
Pronunciation {
|
||||||
ipa: ipa.into(),
|
ipa: ipa.into(),
|
||||||
|
|
@ -91,21 +375,6 @@ fn say(entities: &dyn EntityForms) -> Result<Vec<String>, LowerError> {
|
||||||
pronounce(&view(&proof_packet(), &inv), entities)
|
pronounce(&view(&proof_packet(), &inv), entities)
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_failing_entity_port_is_refused_by_name() {
|
|
||||||
let e = Entities {
|
|
||||||
fault: true,
|
|
||||||
..Entities::default()
|
|
||||||
};
|
|
||||||
assert_eq!(
|
|
||||||
say(&e),
|
|
||||||
Err(LowerError::EntityRead {
|
|
||||||
entity: EntityId::parse("entity:nibu").unwrap(),
|
|
||||||
why: "store unreadable".into()
|
|
||||||
})
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn nibu_round_trips_from_the_committed_seed_to_its_ruled_ipa() {
|
fn nibu_round_trips_from_the_committed_seed_to_its_ruled_ipa() {
|
||||||
let seed = committed_entities();
|
let seed = committed_entities();
|
||||||
|
|
@ -304,3 +573,160 @@ fn the_inventory_parser_refuses_what_it_cannot_read() {
|
||||||
assert!(PhoneInventory::parse("# c\n\ni\tvowel\ttense\ta\n").is_ok());
|
assert!(PhoneInventory::parse("# c\n\ni\tvowel\ttense\ta\n").is_ok());
|
||||||
assert!(PhoneInventory::for_language("sa-Latn").is_none());
|
assert!(PhoneInventory::for_language("sa-Latn").is_none());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `nibu-proof-001` re-cut as the request: the addressee asked Nibu to stop,
|
||||||
|
/// one frame:3 predication (n4), consent naming it, and no other claim.
|
||||||
|
fn asked() -> Packet {
|
||||||
|
let mut p = proof_packet();
|
||||||
|
p.referents.remove(1);
|
||||||
|
let mut ask = p.predications[1].clone();
|
||||||
|
ask.id = nd("n4");
|
||||||
|
ask.concept = iv("concept:6");
|
||||||
|
ask.frame = iv("frame:3");
|
||||||
|
ask.roles = vec![
|
||||||
|
weksa::packet::RoleFill {
|
||||||
|
role: iv("role:1"),
|
||||||
|
filler: nd("n1"),
|
||||||
|
},
|
||||||
|
weksa::packet::RoleFill {
|
||||||
|
role: iv("role:2"),
|
||||||
|
filler: nd("n3"),
|
||||||
|
},
|
||||||
|
];
|
||||||
|
p.predications = vec![ask];
|
||||||
|
p.discourse.speech_act = Some(iv("speech_act:1"));
|
||||||
|
p.discourse.addressee_consent = Some(nd("n4"));
|
||||||
|
p.constraints.preserve = vec![nd("n1")];
|
||||||
|
p
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The request plus a frame:2 predication (n5) the consent could wrongly name.
|
||||||
|
fn asked_and_stopped() -> Packet {
|
||||||
|
let mut p = asked();
|
||||||
|
let mut stop = proof_packet().predications.remove(1);
|
||||||
|
stop.roles[1].filler = nd("n4");
|
||||||
|
p.predications.push(stop);
|
||||||
|
p
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_consented_request_lowers_to_what_the_addressee_asked() {
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
assert_eq!(lowered(&asked(), &inv).unwrap(), "You asked Nibu to stop.");
|
||||||
|
let mut p = asked();
|
||||||
|
p.provenance.source = weksa::packet::ProvenanceSource::GameTelemetry;
|
||||||
|
assert_eq!(lowered(&p, &inv).unwrap(), "You asked Nibu to stop.");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_request_without_its_consent_is_refused_not_invented() {
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
let mut p = asked();
|
||||||
|
p.discourse.addressee_consent = None;
|
||||||
|
assert_eq!(
|
||||||
|
lowered(&p, &inv),
|
||||||
|
Err(LowerError::ConsentMissing {
|
||||||
|
predication: nd("n4")
|
||||||
|
})
|
||||||
|
);
|
||||||
|
// Consent that names a different predication is no consent to the request.
|
||||||
|
let mut p = asked_and_stopped();
|
||||||
|
p.discourse.addressee_consent = Some(nd("n5"));
|
||||||
|
assert_eq!(
|
||||||
|
lowered(&p, &inv),
|
||||||
|
Err(LowerError::ConsentMissing {
|
||||||
|
predication: nd("n4")
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn consent_to_one_of_two_requests_leaves_the_other_unconsented() {
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
let mut p = asked();
|
||||||
|
let mut second = p.predications[0].clone();
|
||||||
|
second.id = nd("n5");
|
||||||
|
p.predications.push(second);
|
||||||
|
assert_eq!(
|
||||||
|
lowered(&p, &inv),
|
||||||
|
Err(LowerError::ConsentMissing {
|
||||||
|
predication: nd("n5")
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_asker_is_the_role_the_inventory_glosses_agent_not_the_first_role() {
|
||||||
|
// role:1 and role:2 swap glosses: "agent" is now role:2, so the asker of
|
||||||
|
// `asked()` (role:1 filled by the addressee) is the one role:2 names, Nibu.
|
||||||
|
// A lookup that took the first role, or the first live entry, would read the
|
||||||
|
// addressee and let the request through.
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let mut store = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
||||||
|
let (rev1, mut entries) = weksa::inventory::seed();
|
||||||
|
for e in &mut entries {
|
||||||
|
if e.id == iv("role:1") {
|
||||||
|
e.gloss = "patient".into();
|
||||||
|
} else if e.id == iv("role:2") {
|
||||||
|
e.gloss = "agent".into();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
store.admit_revision(rev1, entries).unwrap();
|
||||||
|
let (rev2, entries) = proof_revision();
|
||||||
|
store.admit_revision(rev2, entries).unwrap();
|
||||||
|
let inv = store.inventory().unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
lower(&admitted(&asked(), &store), &inv, &committed_entities()),
|
||||||
|
Err(LowerError::AskerNotAddressee {
|
||||||
|
predication: nd("n4")
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_request_by_someone_other_than_the_addressee_is_refused() {
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
let mut p = asked();
|
||||||
|
p.predications[0].roles[0].filler = nd("n3");
|
||||||
|
p.predications[0].roles[1].filler = nd("n1");
|
||||||
|
assert_eq!(
|
||||||
|
lowered(&p, &inv),
|
||||||
|
Err(LowerError::AskerNotAddressee {
|
||||||
|
predication: nd("n4")
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn consent_on_a_claim_that_is_no_request_stays_unrendered() {
|
||||||
|
let (inv, _dir) = committed();
|
||||||
|
let mut p = proof_packet();
|
||||||
|
p.discourse.addressee_consent = Some(nd("n5"));
|
||||||
|
assert_eq!(
|
||||||
|
lowered(&p, &inv),
|
||||||
|
Err(LowerError::Unrendered {
|
||||||
|
what: "addressee consent"
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_decomposition_or_import_cannot_reach_the_lowerer_with_consent() {
|
||||||
|
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../..");
|
||||||
|
let inventory = InventoryStore::open(&root.join("data/interlingua/inventory.cc")).unwrap();
|
||||||
|
let corpus = CorpusStore::open(&root.join("data/corpus/corpus.cc")).unwrap();
|
||||||
|
let entities = EntityStore::open(&root.join("data/entity/entity.cc")).unwrap();
|
||||||
|
for source in [
|
||||||
|
weksa::packet::ProvenanceSource::ModelDecomposition,
|
||||||
|
weksa::packet::ProvenanceSource::ImportedCorpus,
|
||||||
|
] {
|
||||||
|
let mut p = asked();
|
||||||
|
p.provenance.source = source;
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let refused = PacketStore::open(&dir.path().join("p.cc"))
|
||||||
|
.unwrap()
|
||||||
|
.admit(p, &inventory, &corpus, &entities)
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(format!("{refused:?}").contains("V15"), "{source:?}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
//! Pins for behaviour of the small and middle weksa files that mutation testing
|
|
||||||
//! found unpinned. Each test states the behaviour in domain terms.
|
|
||||||
|
|
||||||
use weksa::form::is_calendar_date;
|
|
||||||
|
|
||||||
/// A calendar date is exactly `YYYY-MM-DD`: both hyphens sit where the format
|
|
||||||
/// puts them, so a ten-byte string with a digit in the second hyphen's place is
|
|
||||||
/// no date.
|
|
||||||
#[test]
|
|
||||||
fn a_calendar_date_needs_both_hyphens_in_place() {
|
|
||||||
assert!(is_calendar_date("2026-01-01"));
|
|
||||||
assert!(!is_calendar_date("2026-01201"));
|
|
||||||
assert!(!is_calendar_date("2026101-01"));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// February has 29 days in a Gregorian leap year and 28 otherwise.
|
|
||||||
#[test]
|
|
||||||
fn february_has_29_days_only_in_gregorian_leap_years() {
|
|
||||||
assert!(is_calendar_date("2024-02-29"));
|
|
||||||
assert!(is_calendar_date("2000-02-29"));
|
|
||||||
assert!(!is_calendar_date("1900-02-29"));
|
|
||||||
assert!(!is_calendar_date("2023-02-29"));
|
|
||||||
assert!(is_calendar_date("2023-02-28"));
|
|
||||||
assert!(!is_calendar_date("2024-02-30"));
|
|
||||||
}
|
|
||||||
|
|
@ -1,297 +0,0 @@
|
||||||
//! Pins found by mutation testing store.rs, inventory.rs and main.rs. Each test
|
|
||||||
//! states what the production code must do, in domain terms.
|
|
||||||
mod common;
|
|
||||||
use common::*;
|
|
||||||
use std::process::{Command, Output};
|
|
||||||
use weksa::packet::Grade;
|
|
||||||
use weksa::store::EntityBound;
|
|
||||||
use weksa::{
|
|
||||||
Annotator, Condition, EntityStore, Layer, OntologyEntry, OntologyStore, Pronunciation, Refusal,
|
|
||||||
Slot, StoreError, StoreRefusal, Test,
|
|
||||||
};
|
|
||||||
|
|
||||||
fn refusal<T: std::fmt::Debug>(r: Result<T, StoreError>) -> StoreRefusal {
|
|
||||||
match r {
|
|
||||||
Err(StoreError::Refused(r)) => r,
|
|
||||||
other => panic!("expected a store refusal, got {other:?}"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn kay_saying(p: Pronunciation) -> weksa::Entity {
|
|
||||||
let mut e = ent("kay", "en-US", &["Kay"]);
|
|
||||||
e.forms[0].pronunciation = Some(p);
|
|
||||||
e
|
|
||||||
}
|
|
||||||
|
|
||||||
fn ruling(ipa: &str, rejected: Vec<String>) -> Pronunciation {
|
|
||||||
Pronunciation {
|
|
||||||
ipa: ipa.into(),
|
|
||||||
ruled_by: "Kay".into(),
|
|
||||||
date: "2026-11-07".into(),
|
|
||||||
source: "test".into(),
|
|
||||||
rejected,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_pronunciation_with_a_control_or_invisible_format_character_is_refused() {
|
|
||||||
let d = tempfile::tempdir().unwrap();
|
|
||||||
let mut store = EntityStore::open(&d.path().join("e.cc")).unwrap();
|
|
||||||
// Either kind alone is refused: a control character, and a zero-width space.
|
|
||||||
for ipa in ["ˈk\u{7}eɪ", "ˈk\u{200b}eɪ"] {
|
|
||||||
assert!(
|
|
||||||
matches!(
|
|
||||||
store.put(kay_saying(ruling(ipa, vec![]))),
|
|
||||||
Err(StoreError::Refused(StoreRefusal::Malformed { .. }))
|
|
||||||
),
|
|
||||||
"{ipa:?}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
// The same for a rejected pronunciation.
|
|
||||||
for bad in ["ˈk\u{7}aɪ", "ˈk\u{200b}aɪ"] {
|
|
||||||
assert!(
|
|
||||||
matches!(
|
|
||||||
store.put(kay_saying(ruling("ˈkeɪ", vec![bad.into()]))),
|
|
||||||
Err(StoreError::Refused(StoreRefusal::Malformed { .. }))
|
|
||||||
),
|
|
||||||
"{bad:?}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
assert_eq!(store.get(&eid("kay")).unwrap(), None);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_pronunciation_keeps_at_most_eight_rejected_variants() {
|
|
||||||
let d = tempfile::tempdir().unwrap();
|
|
||||||
let mut store = EntityStore::open(&d.path().join("e.cc")).unwrap();
|
|
||||||
let variants = |n: usize| -> Vec<String> { (0..n).map(|i| format!("ˈkaɪ{i}")).collect() };
|
|
||||||
assert_eq!(
|
|
||||||
refusal(store.put(kay_saying(ruling("ˈkeɪ", variants(9))))),
|
|
||||||
StoreRefusal::EntityBounds {
|
|
||||||
id: eid("kay"),
|
|
||||||
bound: EntityBound::Forms
|
|
||||||
}
|
|
||||||
);
|
|
||||||
store
|
|
||||||
.put(kay_saying(ruling("ˈkeɪ", variants(8))))
|
|
||||||
.expect("eight rejected variants are the bound, not beyond it");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn an_ontology_entry_read_at_an_unknown_revision_is_refused_with_one_refusal() {
|
|
||||||
let w = world();
|
|
||||||
let mut onto = OntologyStore::open(&w.dir.path().join("o2.cc")).unwrap();
|
|
||||||
let entry = OntologyEntry {
|
|
||||||
language: "en-US".into(),
|
|
||||||
n: 1,
|
|
||||||
inventory_rev: 99,
|
|
||||||
layer: Layer::Discourse,
|
|
||||||
when: vec![Condition {
|
|
||||||
slot: Slot::Stance,
|
|
||||||
feature: iv(ANGER),
|
|
||||||
test: Test::AtLeast(Grade::new(200).unwrap()),
|
|
||||||
}],
|
|
||||||
choice: "WARM".into(),
|
|
||||||
examples: vec![],
|
|
||||||
annotator: Annotator::Operator,
|
|
||||||
};
|
|
||||||
match onto.admit(vec![entry], &w.inventory) {
|
|
||||||
Err(StoreError::Invalid(r)) => assert_eq!(r, vec![Refusal::V10UnknownRevision { rev: 99 }]),
|
|
||||||
other => panic!("{other:?}"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn weksa(dir: &std::path::Path, args: &[&str]) -> Output {
|
|
||||||
Command::new(env!("CARGO_BIN_EXE_weksa"))
|
|
||||||
.current_dir(dir)
|
|
||||||
.args([
|
|
||||||
"--corpus",
|
|
||||||
"./c.cc",
|
|
||||||
"--inventory",
|
|
||||||
"./i.cc",
|
|
||||||
"--entities",
|
|
||||||
"./e.cc",
|
|
||||||
"--ontology",
|
|
||||||
"./o.cc",
|
|
||||||
"--speaker-state",
|
|
||||||
"./s.cc",
|
|
||||||
])
|
|
||||||
.args(args)
|
|
||||||
.output()
|
|
||||||
.unwrap()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn out(o: &Output) -> String {
|
|
||||||
String::from_utf8_lossy(&o.stdout).into_owned() + &String::from_utf8_lossy(&o.stderr)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn ok(o: Output) -> String {
|
|
||||||
assert!(o.status.success(), "{}", out(&o));
|
|
||||||
out(&o)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A scene bed: entities nibu and brokkr, the inventory through the
|
|
||||||
/// speaker-scene revision (rev 3).
|
|
||||||
fn scene_bed() -> tempfile::TempDir {
|
|
||||||
let d = tempfile::tempdir().unwrap();
|
|
||||||
for slug in ["nibu", "brokkr"] {
|
|
||||||
let id = format!("entity:{slug}");
|
|
||||||
let form = format!("en-US={}", slug.to_uppercase());
|
|
||||||
ok(weksa(
|
|
||||||
d.path(),
|
|
||||||
&[
|
|
||||||
"entity",
|
|
||||||
"put",
|
|
||||||
&id,
|
|
||||||
"--form",
|
|
||||||
&form,
|
|
||||||
"--origin",
|
|
||||||
"cast",
|
|
||||||
"--persona",
|
|
||||||
slug,
|
|
||||||
"--fixture",
|
|
||||||
"f.yaml",
|
|
||||||
],
|
|
||||||
));
|
|
||||||
}
|
|
||||||
for step in ["seed", "proof", "speaker-scene"] {
|
|
||||||
ok(weksa(d.path(), &["inventory", step]));
|
|
||||||
}
|
|
||||||
d
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Adds a cast-authored line; returns the key it printed ("Added <key>").
|
|
||||||
fn add_line(d: &std::path::Path, slug: &str, speaker: &str, group: &str, text: &str) -> String {
|
|
||||||
let speaker = format!("entity:{speaker}");
|
|
||||||
let printed = ok(weksa(
|
|
||||||
d,
|
|
||||||
&[
|
|
||||||
"corpus",
|
|
||||||
"add",
|
|
||||||
"--slug",
|
|
||||||
slug,
|
|
||||||
"--text",
|
|
||||||
text,
|
|
||||||
"--speaker",
|
|
||||||
&speaker,
|
|
||||||
"--group",
|
|
||||||
group,
|
|
||||||
"--admitted-on",
|
|
||||||
"2026-10-29",
|
|
||||||
"--source",
|
|
||||||
"cast-authored",
|
|
||||||
"--language",
|
|
||||||
"en-US",
|
|
||||||
"--persona",
|
|
||||||
"nibu",
|
|
||||||
"--fixture",
|
|
||||||
"f.yaml",
|
|
||||||
"--author",
|
|
||||||
"Ada Writer",
|
|
||||||
],
|
|
||||||
));
|
|
||||||
printed.split_whitespace().nth(1).unwrap().to_string()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn scene_put(d: &std::path::Path, group: &str, to: &str) -> Output {
|
|
||||||
weksa(
|
|
||||||
d,
|
|
||||||
&[
|
|
||||||
"scene",
|
|
||||||
"put",
|
|
||||||
"--group",
|
|
||||||
group,
|
|
||||||
"--to",
|
|
||||||
to,
|
|
||||||
"--inventory-rev",
|
|
||||||
"3",
|
|
||||||
"--model-family",
|
|
||||||
"anthropic",
|
|
||||||
"--generator",
|
|
||||||
"a scene annotator",
|
|
||||||
"--set",
|
|
||||||
"intimacy=200",
|
|
||||||
],
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_scene_group_holds_one_live_speaker_among_its_own_live_items() {
|
|
||||||
let d = scene_bed();
|
|
||||||
add_line(d.path(), "nibu", "nibu", "scene-1", "Mind the hatch.");
|
|
||||||
// Another group's speaker and a withdrawn line of this group are not the
|
|
||||||
// scene's speakers.
|
|
||||||
add_line(d.path(), "brokkr", "brokkr", "scene-2", "Brokkr hums.");
|
|
||||||
let gone = add_line(
|
|
||||||
d.path(),
|
|
||||||
"brokkr",
|
|
||||||
"brokkr",
|
|
||||||
"scene-1",
|
|
||||||
"Brokkr leaves early.",
|
|
||||||
);
|
|
||||||
ok(weksa(d.path(), &["corpus", "withdraw", &gone]));
|
|
||||||
let put = ok(scene_put(d.path(), "scene-1", "unnamed"));
|
|
||||||
assert!(put.contains("scene:scene-1@1"), "{put}");
|
|
||||||
assert!(ok(weksa(d.path(), &["scene", "dump"])).contains("entity:nibu->unnamed"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_scene_names_its_addressee_as_a_participant_unless_it_is_the_speaker() {
|
|
||||||
let d = scene_bed();
|
|
||||||
add_line(d.path(), "nibu", "nibu", "scene-1", "Mind the hatch.");
|
|
||||||
add_line(d.path(), "nibu", "nibu", "scene-2", "Mind the ladder.");
|
|
||||||
ok(scene_put(d.path(), "scene-1", "entity:brokkr"));
|
|
||||||
let dump = ok(weksa(d.path(), &["scene", "dump"]));
|
|
||||||
assert!(dump.contains("scene:scene-1@1 rev:3"), "{dump}");
|
|
||||||
assert!(dump.contains("[entity:nibu,entity:brokkr]"), "{dump}");
|
|
||||||
// A speaker addressing themself is a one-participant scene, whose
|
|
||||||
// relation to a named party is refused as the store reads it.
|
|
||||||
let own = scene_put(d.path(), "scene-2", "entity:nibu");
|
|
||||||
assert!(!own.status.success());
|
|
||||||
assert!(
|
|
||||||
out(&own).contains("a relation's named to is another participant"),
|
|
||||||
"{}",
|
|
||||||
out(&own)
|
|
||||||
);
|
|
||||||
assert_eq!(ok(weksa(d.path(), &["scene", "dump"])), dump);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn relower_refuses_malformed_arm_and_floor_labels_before_reading_the_run() {
|
|
||||||
let d = tempfile::tempdir().unwrap();
|
|
||||||
let relower = |arm: &str, floor: &[&str]| {
|
|
||||||
let mut args = vec![
|
|
||||||
"loop",
|
|
||||||
"relower",
|
|
||||||
"no-such-run",
|
|
||||||
"--round",
|
|
||||||
"1",
|
|
||||||
"--arm",
|
|
||||||
arm,
|
|
||||||
"--lowerer",
|
|
||||||
"m",
|
|
||||||
];
|
|
||||||
for f in floor {
|
|
||||||
args.extend(["--floor", f]);
|
|
||||||
}
|
|
||||||
weksa(d.path(), &args)
|
|
||||||
};
|
|
||||||
for (arm, floor) in [
|
|
||||||
("BAD", vec!["b"]),
|
|
||||||
("a", vec!["Bad"]),
|
|
||||||
("a", vec!["a", "b"]),
|
|
||||||
("a", vec!["b", "b"]),
|
|
||||||
] {
|
|
||||||
let o = relower(arm, &floor);
|
|
||||||
assert!(!o.status.success());
|
|
||||||
assert!(
|
|
||||||
out(&o).contains("arm and floor labels must match"),
|
|
||||||
"{arm} {floor:?}: {}",
|
|
||||||
out(&o)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
// Well-formed, distinct labels get past the label check (to the missing run).
|
|
||||||
let o = relower("a", &["b", "c"]);
|
|
||||||
assert!(!o.status.success());
|
|
||||||
assert!(out(&o).contains("the root holds no run"), "{}", out(&o));
|
|
||||||
}
|
|
||||||
|
|
@ -1,136 +0,0 @@
|
||||||
//! Pins for behaviour of `context.rs`, `encode.rs` and `run.rs` found unpinned
|
|
||||||
//! by mutation testing. Each test states what the code does in domain terms.
|
|
||||||
|
|
||||||
mod common;
|
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
|
||||||
|
|
||||||
use weksa::context::Prompt;
|
|
||||||
use weksa::entity::{Entity, EntityOrigin, LanguageForms};
|
|
||||||
use weksa::packet::{EntityId, proof_packet};
|
|
||||||
use weksa::{CorpusItemId, CorpusStore, EntityForms, EntityStore, InventoryStore, PacketStore};
|
|
||||||
|
|
||||||
fn data(sub: &str) -> PathBuf {
|
|
||||||
Path::new(env!("CARGO_MANIFEST_DIR"))
|
|
||||||
.join("../../data")
|
|
||||||
.join(sub)
|
|
||||||
}
|
|
||||||
|
|
||||||
struct Held(Vec<Entity>);
|
|
||||||
|
|
||||||
impl EntityForms for Held {
|
|
||||||
fn entity(&self, id: &EntityId) -> anyhow::Result<Option<Entity>> {
|
|
||||||
Ok(self.0.iter().find(|e| &e.id == id).cloned())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn spoken(slug: &str, form: &str) -> Entity {
|
|
||||||
Entity {
|
|
||||||
id: EntityId::parse(&format!("entity:{slug}")).unwrap(),
|
|
||||||
forms: vec![LanguageForms {
|
|
||||||
language: "en-US".into(),
|
|
||||||
forms: vec![form.into()],
|
|
||||||
pronunciation: None,
|
|
||||||
}],
|
|
||||||
origin: EntityOrigin::Cast {
|
|
||||||
persona: slug.into(),
|
|
||||||
fixture: "f.yaml".into(),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A speaker whom no referent cites contributes no entry to the lowering
|
|
||||||
/// prompt, even when another entity is cited: only the cited entity is listed.
|
|
||||||
#[test]
|
|
||||||
fn an_uncited_speaker_is_absent_from_the_lowering_prompt() {
|
|
||||||
let cast_inv = InventoryStore::open(&data("interlingua/inventory.cc")).unwrap();
|
|
||||||
let entities = EntityStore::open(&data("entity/entity.cc")).unwrap();
|
|
||||||
let corpus = CorpusStore::open(&data("corpus/corpus.cc")).unwrap();
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let view = PacketStore::open(&dir.path().join("p.cc"))
|
|
||||||
.unwrap()
|
|
||||||
.admit(proof_packet(), &cast_inv, &corpus, &entities)
|
|
||||||
.unwrap();
|
|
||||||
let state = weksa::SpeakerStateStore::open(&dir.path().join("s.cc")).unwrap();
|
|
||||||
let view = view
|
|
||||||
.with_scene(&cast_inv.inventory().unwrap(), &state, &corpus)
|
|
||||||
.unwrap();
|
|
||||||
let held = Held(vec![
|
|
||||||
spoken("nibu", "Nibu"),
|
|
||||||
spoken("other", "SPEAKERCANARY"),
|
|
||||||
]);
|
|
||||||
let item = CorpusItemId::parse("item:nibu:1").unwrap();
|
|
||||||
let speaker = EntityId::parse("entity:other").unwrap();
|
|
||||||
let prompt = Prompt::lower(
|
|
||||||
&view,
|
|
||||||
&item,
|
|
||||||
&speaker,
|
|
||||||
"en-US",
|
|
||||||
&cast_inv.inventory().unwrap(),
|
|
||||||
&held,
|
|
||||||
&weksa::Ontology::empty(),
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
let text = prompt.text();
|
|
||||||
assert!(
|
|
||||||
text.contains("\n\nentities: {\"entity:nibu\":[\"Nibu\"]}\n\n"),
|
|
||||||
"{text}"
|
|
||||||
);
|
|
||||||
assert!(!text.contains("SPEAKERCANARY"), "{text}");
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A run refuses a panel language that lacks a live Train or a live Validation
|
|
||||||
/// item: one no item carries, one with only a Test item, one with only Train.
|
|
||||||
#[test]
|
|
||||||
fn a_panel_language_no_item_carries_is_refused_at_open() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let d = dir.path();
|
|
||||||
// Builds the corpus, entity and ontology stores the run reads.
|
|
||||||
drop(common::run_over(d));
|
|
||||||
let entities = common::entities_in(d);
|
|
||||||
let mut corpus = CorpusStore::open(&d.join("c.cc")).unwrap();
|
|
||||||
for (key, language, bucket) in [("item:jp:1", "ja-JP", 90), ("item:pt:1", "pt-BR", 10)] {
|
|
||||||
let mut item = common::item_in(
|
|
||||||
key,
|
|
||||||
&format!("line of {key}"),
|
|
||||||
&common::group_in_bucket(bucket),
|
|
||||||
common::cast(),
|
|
||||||
);
|
|
||||||
item.language = language.into();
|
|
||||||
corpus.add(item, &entities).unwrap();
|
|
||||||
}
|
|
||||||
drop(corpus);
|
|
||||||
let create = |panel: &[&str]| {
|
|
||||||
weksa::run::RunStores::create(
|
|
||||||
&d.join(format!("run-{}", panel.join("-"))),
|
|
||||||
"t",
|
|
||||||
weksa::run::RunPaths {
|
|
||||||
corpus: &d.join("c.cc"),
|
|
||||||
inventory: &d.join("i.cc"),
|
|
||||||
entities: &d.join("entities.cc"),
|
|
||||||
ontology: &d.join("ontology.cc"),
|
|
||||||
speaker_state: &common::speaker_state_beside(&d.join("ontology.cc")),
|
|
||||||
},
|
|
||||||
"fake-model",
|
|
||||||
"bag",
|
|
||||||
weksa::run::LowererReach::ToolLessRelay,
|
|
||||||
weksa::corpus::ModelFamily::Unrecorded,
|
|
||||||
&panel.iter().map(|t| (*t).to_string()).collect::<Vec<_>>(),
|
|
||||||
)
|
|
||||||
};
|
|
||||||
assert!(create(&["en-US"]).is_ok());
|
|
||||||
for panel in [&["ja-JP"][..], &["pt-BR"], &["xx-XX"], &["en-US", "ja-JP"]] {
|
|
||||||
assert!(
|
|
||||||
matches!(create(panel), Err(weksa::run::RunError::Panel)),
|
|
||||||
"{panel:?}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A run keeps its recall probes in `recall` under its own root.
|
|
||||||
#[test]
|
|
||||||
fn a_run_keeps_its_recall_under_its_root() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = common::run_over(dir.path());
|
|
||||||
assert_eq!(run.recall_dir(), dir.path().join("run").join("recall"));
|
|
||||||
}
|
|
||||||
|
|
@ -1,402 +0,0 @@
|
||||||
//! The ontology mint (`weksa loop mint-ontology`): a Train-only call that drafts
|
|
||||||
//! entries for `weksa ontology admit`. It shows a model the Train lines of one
|
|
||||||
//! panel language, the features a condition can read and that language's entry
|
|
||||||
//! lines; it admits nothing and writes nothing of the run.
|
|
||||||
|
|
||||||
mod common;
|
|
||||||
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
use common::{cast, entities_in, group_in_bucket, item_in};
|
|
||||||
use weksa::context::Prompt;
|
|
||||||
use weksa::corpus::{ModelFamily, Source};
|
|
||||||
use weksa::inventory::{Change, seed};
|
|
||||||
use weksa::model::{AnswerFiles, ModelPort, PortError};
|
|
||||||
use weksa::run::{LowererReach, OntologyMintOutcome, RunPaths, RunStores};
|
|
||||||
use weksa::speaker::Annotator;
|
|
||||||
use weksa::{CorpusStore, InventoryStore, OntologyStore, Slot, ValueKind};
|
|
||||||
|
|
||||||
/// Records every prompt shown; answers with `reply`.
|
|
||||||
struct Record {
|
|
||||||
prompts: Vec<(String, Vec<String>)>,
|
|
||||||
reply: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Record {
|
|
||||||
fn new(reply: &str) -> Self {
|
|
||||||
Self { prompts: vec![], reply: reply.to_string() }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl ModelPort for Record {
|
|
||||||
fn model(&self) -> &str {
|
|
||||||
"record"
|
|
||||||
}
|
|
||||||
fn answer(&mut self, p: &Prompt) -> Result<String, PortError> {
|
|
||||||
self.prompts.push((p.text().to_string(), p.item_keys().to_vec()));
|
|
||||||
Ok(self.reply.clone())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn public() -> Source {
|
|
||||||
Source::PublicDomain {
|
|
||||||
work: "A Work".into(),
|
|
||||||
author: "An Author".into(),
|
|
||||||
edition_url: "https://example.org/e".into(),
|
|
||||||
licence: "CC0".into(),
|
|
||||||
jurisdiction: "US".into(),
|
|
||||||
translator: "A Translator".into(),
|
|
||||||
edition_year: 1900,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const STANCE_G: &str = "feature:60";
|
|
||||||
const DISC_FLAG: &str = "feature:61";
|
|
||||||
const REF_ONLY: &str = "feature:62";
|
|
||||||
const PRED_ONLY: &str = "feature:63";
|
|
||||||
const STANCE_RETIRED: &str = "feature:64";
|
|
||||||
const SCENE_G: &str = "feature:65";
|
|
||||||
const SCENE_OPT: &str = "feature:66";
|
|
||||||
|
|
||||||
/// A fixture: items in en-US and ja-JP across every split and a withdrawal, an
|
|
||||||
/// inventory with condition-readable and unreadable features, an ontology with
|
|
||||||
/// en-US and ja-JP entries, and a run over them with `panel`.
|
|
||||||
fn fixture(dir: &Path, panel: &[&str]) -> RunStores {
|
|
||||||
let entities = entities_in(dir);
|
|
||||||
let mut corpus = CorpusStore::open(&dir.join("c.cc")).unwrap();
|
|
||||||
let rows = [
|
|
||||||
("item:en:1", "en-US", "entrain mark one", 10, cast(), false),
|
|
||||||
("item:en:2", "en-US", "entrain mark two", 11, cast(), false),
|
|
||||||
("item:en:3", "en-US", "envalid mark three", 50, cast(), false),
|
|
||||||
("item:en:4", "en-US", "engone mark four", 12, cast(), true),
|
|
||||||
("item:en:5", "en-US", "entest mark five", 90, public(), false),
|
|
||||||
("item:ja:1", "ja-JP", "jatrain mark one", 13, cast(), false),
|
|
||||||
("item:ja:2", "ja-JP", "javalid mark two", 51, cast(), false),
|
|
||||||
("item:ja:3", "ja-JP", "jatest mark three", 91, public(), false),
|
|
||||||
];
|
|
||||||
for (id, language, text, bucket, source, withdrawn) in rows {
|
|
||||||
let mut item = item_in(id, text, &group_in_bucket(bucket), source);
|
|
||||||
item.language = language.into();
|
|
||||||
corpus.add(item, &entities).unwrap();
|
|
||||||
if withdrawn {
|
|
||||||
corpus.withdraw(id).unwrap();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
drop(corpus);
|
|
||||||
|
|
||||||
let mut inv = InventoryStore::open(&dir.join("i.cc")).unwrap();
|
|
||||||
let (rev1, entries) = seed();
|
|
||||||
inv.admit_revision(rev1, entries).unwrap();
|
|
||||||
let f = |id, kind, slots: &[Slot]| common::feature(id, kind, slots);
|
|
||||||
let features = vec![
|
|
||||||
f(STANCE_G, ValueKind::Grade, &[Slot::Stance]),
|
|
||||||
f(DISC_FLAG, ValueKind::Flag, &[Slot::Discourse]),
|
|
||||||
f(REF_ONLY, ValueKind::Grade, &[Slot::Referent]),
|
|
||||||
f(PRED_ONLY, ValueKind::Grade, &[Slot::Predication]),
|
|
||||||
f(STANCE_RETIRED, ValueKind::Grade, &[Slot::Stance]),
|
|
||||||
f(SCENE_G, ValueKind::Grade, &[Slot::Scene]),
|
|
||||||
f(SCENE_OPT, ValueKind::Options(weksa::OptionSet::new(vec![weksa::OptionLabel::parse("below").unwrap(), weksa::OptionLabel::parse("above").unwrap()]).unwrap()), &[Slot::Scene]),
|
|
||||||
];
|
|
||||||
let changes = features.iter().map(|e| Change::Add(e.id)).collect();
|
|
||||||
let mut features = features;
|
|
||||||
for e in &mut features {
|
|
||||||
e.introduced_rev = 2;
|
|
||||||
}
|
|
||||||
inv.admit_revision(common::rev_with(changes, 2, Some(1)), features).unwrap();
|
|
||||||
inv.admit_revision(
|
|
||||||
common::rev_with(vec![Change::Retire(common::iv(STANCE_RETIRED))], 3, Some(2)),
|
|
||||||
vec![],
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
let mut onto = OntologyStore::open(&dir.join("ontology.cc")).unwrap();
|
|
||||||
let annotator = Annotator::Model { family: ModelFamily::Anthropic, generator: "seed".into() };
|
|
||||||
let entry = |language: &str, choice: &str, examples: &[&str]| {
|
|
||||||
weksa::encode::OntologyDraft::parse(&format!(
|
|
||||||
r#"{{"entries":[{{"layer":"Grammar","choice":"{choice}","examples":[{}]}}]}}"#,
|
|
||||||
examples.iter().map(|x| format!("\"{x}\"")).collect::<Vec<_>>().join(",")
|
|
||||||
))
|
|
||||||
.unwrap()
|
|
||||||
.into_entries(language, 3, 1, &annotator)
|
|
||||||
};
|
|
||||||
onto.admit(entry("en-US", "enchoice marker", &["enexample marker"]), &inv).unwrap();
|
|
||||||
onto.admit(entry("ja-JP", "jachoice marker", &["jaexample marker"]), &inv).unwrap();
|
|
||||||
drop(onto);
|
|
||||||
drop(inv);
|
|
||||||
|
|
||||||
std::fs::write(dir.join("speaker-state.cc"), [0x90]).unwrap();
|
|
||||||
let panel: Vec<String> = panel.iter().map(|s| s.to_string()).collect();
|
|
||||||
RunStores::create(
|
|
||||||
&dir.join("run"),
|
|
||||||
"t",
|
|
||||||
RunPaths {
|
|
||||||
corpus: &dir.join("c.cc"),
|
|
||||||
inventory: &dir.join("i.cc"),
|
|
||||||
entities: &dir.join("entities.cc"),
|
|
||||||
ontology: &dir.join("ontology.cc"),
|
|
||||||
speaker_state: &dir.join("speaker-state.cc"),
|
|
||||||
},
|
|
||||||
"fake-model",
|
|
||||||
"bag",
|
|
||||||
LowererReach::ToolLessRelay,
|
|
||||||
ModelFamily::Unrecorded,
|
|
||||||
&panel,
|
|
||||||
)
|
|
||||||
.unwrap()
|
|
||||||
}
|
|
||||||
|
|
||||||
const DRAFT: &str =
|
|
||||||
r#"{"entries":[{"layer":"Discourse","when":[],"choice":"a pattern","examples":[]}]}"#;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn mint_ontology_prompt_shows_only_train_items_of_the_language() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = fixture(dir.path(), &["en-US", "ja-JP"]);
|
|
||||||
let mut m = Record::new(DRAFT);
|
|
||||||
run.mint_ontology(&mut m, "en-US").unwrap();
|
|
||||||
let (text, keys) = &m.prompts[0];
|
|
||||||
assert!(text.contains("language: en-US"));
|
|
||||||
assert_eq!(keys, &["item:en:1", "item:en:2"]);
|
|
||||||
for shown in ["entrain mark one", "entrain mark two"] {
|
|
||||||
assert!(text.contains(shown), "{shown}");
|
|
||||||
}
|
|
||||||
for hidden in [
|
|
||||||
"envalid", "engone", "entest", "jatrain", "javalid", "jatest", "item:en:3", "item:en:5",
|
|
||||||
"item:ja",
|
|
||||||
] {
|
|
||||||
assert!(!text.contains(hidden), "the en-US mint showed {hidden}");
|
|
||||||
}
|
|
||||||
let mut m = Record::new(DRAFT);
|
|
||||||
run.mint_ontology(&mut m, "ja-JP").unwrap();
|
|
||||||
let (text, keys) = &m.prompts[0];
|
|
||||||
assert_eq!(keys, &["item:ja:1"]);
|
|
||||||
assert!(text.contains("jatrain mark one") && !text.contains("entrain"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn mint_ontology_refuses_a_language_outside_the_panel() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = fixture(dir.path(), &["en-US"]);
|
|
||||||
let mut m = Record::new(DRAFT);
|
|
||||||
// ja-JP has Train items in the corpus but is not in the panel.
|
|
||||||
for (language, code) in [("ja-JP", "language-not-in-panel"), ("fr-FR", "language-not-in-panel")] {
|
|
||||||
assert_eq!(
|
|
||||||
run.mint_ontology(&mut m, language).unwrap(),
|
|
||||||
OntologyMintOutcome::Refused { code }
|
|
||||||
);
|
|
||||||
}
|
|
||||||
assert!(m.prompts.is_empty(), "a refused mint asked the model");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn mint_ontology_prompt_lists_only_condition_readable_features() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = fixture(dir.path(), &["en-US"]);
|
|
||||||
let mut m = Record::new(DRAFT);
|
|
||||||
run.mint_ontology(&mut m, "en-US").unwrap();
|
|
||||||
let text = &m.prompts[0].0;
|
|
||||||
let listed = |id: &str| text.contains(&format!("\n{id}: "));
|
|
||||||
for id in [STANCE_G, DISC_FLAG, SCENE_G] {
|
|
||||||
assert!(listed(id), "{id} is readable by a condition");
|
|
||||||
}
|
|
||||||
for id in [REF_ONLY, PRED_ONLY, STANCE_RETIRED] {
|
|
||||||
assert!(!listed(id), "{id} is not");
|
|
||||||
}
|
|
||||||
// Nothing that is not a feature: no concept or role line.
|
|
||||||
assert!(!text.contains("\nconcept:") && !text.contains("\nrole:"));
|
|
||||||
assert!(text.contains("features rev:3"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn mint_ontology_prompt_lists_only_the_languages_existing_entries() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = fixture(dir.path(), &["en-US", "ja-JP"]);
|
|
||||||
let mut m = Record::new(DRAFT);
|
|
||||||
run.mint_ontology(&mut m, "en-US").unwrap();
|
|
||||||
let text = &m.prompts[0].0;
|
|
||||||
assert!(text.contains("onto:en-US:1 enchoice marker"));
|
|
||||||
for hidden in ["jachoice", "onto:ja-JP", "enexample", "jaexample"] {
|
|
||||||
assert!(!text.contains(hidden), "{hidden}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn digests(dir: &Path) -> Vec<(String, Vec<u8>)> {
|
|
||||||
let mut out = vec![];
|
|
||||||
let mut stack = vec![dir.to_path_buf()];
|
|
||||||
while let Some(d) = stack.pop() {
|
|
||||||
for e in std::fs::read_dir(&d).unwrap() {
|
|
||||||
let p = e.unwrap().path();
|
|
||||||
if p.is_dir() {
|
|
||||||
stack.push(p);
|
|
||||||
} else if !p.starts_with(dir.join("run").join("ontology-mint")) {
|
|
||||||
out.push((p.display().to_string(), std::fs::read(&p).unwrap()));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
out.sort();
|
|
||||||
out
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn mint_ontology_writes_nothing_but_its_prompt_and_answer_files() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = fixture(dir.path(), &["en-US", "ja-JP"]);
|
|
||||||
let before = digests(dir.path());
|
|
||||||
let mut files = AnswerFiles::new(run.ontology_mint_dir(), "m".into(), vec![]);
|
|
||||||
let OntologyMintOutcome::Unanswered { prompt } =
|
|
||||||
run.mint_ontology(&mut files, "en-US").unwrap()
|
|
||||||
else {
|
|
||||||
panic!("unanswered");
|
|
||||||
};
|
|
||||||
std::fs::write(run.ontology_mint_dir().join(format!("{prompt}.answer")), DRAFT).unwrap();
|
|
||||||
assert!(matches!(
|
|
||||||
run.mint_ontology(&mut files, "en-US").unwrap(),
|
|
||||||
OntologyMintOutcome::Drafted { entries: 1, .. }
|
|
||||||
));
|
|
||||||
assert_eq!(digests(dir.path()), before, "the mint wrote outside ontology-mint");
|
|
||||||
let mut names: Vec<String> = std::fs::read_dir(run.ontology_mint_dir())
|
|
||||||
.unwrap()
|
|
||||||
.map(|e| e.unwrap().file_name().into_string().unwrap())
|
|
||||||
.collect();
|
|
||||||
names.sort();
|
|
||||||
assert_eq!(names, [format!("{prompt}.answer"), format!("{prompt}.prompt")]);
|
|
||||||
drop(run);
|
|
||||||
RunStores::open(&dir.path().join("run")).expect("no Drift after a mint");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn mint_ontology_unanswered_then_drafted_then_refused() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = fixture(dir.path(), &["en-US"]);
|
|
||||||
let mut files = AnswerFiles::new(run.ontology_mint_dir(), "m".into(), vec![]);
|
|
||||||
let OntologyMintOutcome::Unanswered { prompt } =
|
|
||||||
run.mint_ontology(&mut files, "en-US").unwrap()
|
|
||||||
else {
|
|
||||||
panic!("unanswered");
|
|
||||||
};
|
|
||||||
let mint = run.ontology_mint_dir();
|
|
||||||
assert!(mint.join(format!("{prompt}.prompt")).exists());
|
|
||||||
let answer = mint.join(format!("{prompt}.answer"));
|
|
||||||
std::fs::write(&answer, DRAFT).unwrap();
|
|
||||||
assert_eq!(
|
|
||||||
run.mint_ontology(&mut files, "en-US").unwrap(),
|
|
||||||
OntologyMintOutcome::Drafted { prompt: prompt.clone(), entries: 1 }
|
|
||||||
);
|
|
||||||
let two = r#"{"entries":[{"layer":"Grammar","choice":"a"},{"layer":"Lexicon","choice":"b"}]}"#;
|
|
||||||
std::fs::write(&answer, two).unwrap();
|
|
||||||
assert!(matches!(
|
|
||||||
run.mint_ontology(&mut files, "en-US").unwrap(),
|
|
||||||
OntologyMintOutcome::Drafted { entries: 2, .. }
|
|
||||||
));
|
|
||||||
let unknown = r#"{"entries":[{"layer":"Grammar","choice":"a","key":"x"}]}"#;
|
|
||||||
std::fs::write(&answer, unknown).unwrap();
|
|
||||||
assert_eq!(
|
|
||||||
run.mint_ontology(&mut files, "en-US").unwrap(),
|
|
||||||
OntologyMintOutcome::Refused { code: "unknown-field" }
|
|
||||||
);
|
|
||||||
std::fs::write(&answer, "not json").unwrap();
|
|
||||||
assert_eq!(
|
|
||||||
run.mint_ontology(&mut files, "en-US").unwrap(),
|
|
||||||
OntologyMintOutcome::Refused { code: "syntax" }
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_mint_answer_admits_into_a_copy_and_a_new_run_pins_the_copy() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = fixture(dir.path(), &["en-US"]);
|
|
||||||
let pinned = std::fs::read(dir.path().join("ontology.cc")).unwrap();
|
|
||||||
let pin = run.record().ontology_digest.clone();
|
|
||||||
let draft = format!(
|
|
||||||
r#"{{"entries":[{{"layer":"Discourse","when":[{{"slot":"Stance","feature":"{STANCE_G}","test":{{"AtLeast":500}}}}],"choice":"a grown choice","examples":[]}}]}}"#
|
|
||||||
);
|
|
||||||
let mut files = AnswerFiles::new(run.ontology_mint_dir(), "m".into(), vec![]);
|
|
||||||
let OntologyMintOutcome::Unanswered { prompt } =
|
|
||||||
run.mint_ontology(&mut files, "en-US").unwrap()
|
|
||||||
else {
|
|
||||||
panic!("unanswered");
|
|
||||||
};
|
|
||||||
let answer = run.ontology_mint_dir().join(format!("{prompt}.answer"));
|
|
||||||
std::fs::write(&answer, &draft).unwrap();
|
|
||||||
|
|
||||||
let copy = dir.path().join("grown.cc");
|
|
||||||
std::fs::copy(dir.path().join("ontology.cc"), ©).unwrap();
|
|
||||||
let inv = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
|
||||||
let mut grown = OntologyStore::open(©).unwrap();
|
|
||||||
let annotator = Annotator::Model { family: ModelFamily::Anthropic, generator: "minter".into() };
|
|
||||||
let first_n = grown.ontology().unwrap().entries().iter().filter(|e| e.language == "en-US").count() as u32 + 1;
|
|
||||||
let entries = weksa::encode::OntologyDraft::parse(&std::fs::read_to_string(&answer).unwrap())
|
|
||||||
.unwrap()
|
|
||||||
.into_entries("en-US", 3, first_n, &annotator);
|
|
||||||
grown.admit(entries, &inv).unwrap();
|
|
||||||
drop(grown);
|
|
||||||
|
|
||||||
assert_eq!(std::fs::read(dir.path().join("ontology.cc")).unwrap(), pinned);
|
|
||||||
let again = RunStores::open(&dir.path().join("run")).unwrap();
|
|
||||||
assert_eq!(again.record().ontology_digest, pin);
|
|
||||||
|
|
||||||
let second = RunStores::create(
|
|
||||||
&dir.path().join("run-grown"),
|
|
||||||
"t",
|
|
||||||
RunPaths {
|
|
||||||
corpus: &dir.path().join("c.cc"),
|
|
||||||
inventory: &dir.path().join("i.cc"),
|
|
||||||
entities: &dir.path().join("entities.cc"),
|
|
||||||
ontology: ©,
|
|
||||||
speaker_state: &dir.path().join("speaker-state.cc"),
|
|
||||||
},
|
|
||||||
"fake-model",
|
|
||||||
"bag",
|
|
||||||
LowererReach::ToolLessRelay,
|
|
||||||
ModelFamily::Unrecorded,
|
|
||||||
&["en-US".to_string()],
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
assert_ne!(second.record().ontology_digest, pin);
|
|
||||||
|
|
||||||
// A condition on a feature outside the inventory is the store's to refuse.
|
|
||||||
let outside = draft.replace(STANCE_G, "feature:99");
|
|
||||||
let entries = weksa::encode::OntologyDraft::parse(&outside)
|
|
||||||
.unwrap()
|
|
||||||
.into_entries("en-US", 3, 9, &annotator);
|
|
||||||
let mut grown = OntologyStore::open(©).unwrap();
|
|
||||||
assert!(grown.admit(entries, &inv).is_err());
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The line after `example ` in a mint prompt.
|
|
||||||
fn example_of(text: &str) -> &str {
|
|
||||||
text.lines().find_map(|l| l.strip_prefix("example ")).expect("the prompt carries an example")
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn the_mint_prompts_own_example_parses_and_admits_through_the_door() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = fixture(dir.path(), &["en-US"]);
|
|
||||||
let mut m = Record::new(DRAFT);
|
|
||||||
run.mint_ontology(&mut m, "en-US").unwrap();
|
|
||||||
let example = example_of(&m.prompts[0].0);
|
|
||||||
// It teaches the door's spelling: a capitalized layer, feature ids, each test form.
|
|
||||||
assert!(example.contains(r#""layer":"Grammar""#), "{example}");
|
|
||||||
for form in [r#"{"AtLeast":500}"#, r#"{"Is":{"Option":"#, r#"{"Is":{"Flag":true}}"#] {
|
|
||||||
assert!(example.contains(form), "{form} in {example}");
|
|
||||||
}
|
|
||||||
let draft = weksa::encode::OntologyDraft::parse(example).unwrap();
|
|
||||||
let inv = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
|
||||||
let copy = dir.path().join("grown.cc");
|
|
||||||
std::fs::copy(dir.path().join("ontology.cc"), ©).unwrap();
|
|
||||||
let mut grown = OntologyStore::open(©).unwrap();
|
|
||||||
let annotator = Annotator::Model { family: ModelFamily::Anthropic, generator: "minter".into() };
|
|
||||||
grown.admit(draft.into_entries("en-US", 3, 2, &annotator), &inv).unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn the_mint_example_names_only_features_the_prompt_lists() {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let run = fixture(dir.path(), &["en-US"]);
|
|
||||||
let mut m = Record::new(DRAFT);
|
|
||||||
run.mint_ontology(&mut m, "en-US").unwrap();
|
|
||||||
let example = example_of(&m.prompts[0].0);
|
|
||||||
for unreadable in [REF_ONLY, PRED_ONLY, STANCE_RETIRED] {
|
|
||||||
assert!(!example.contains(unreadable), "{unreadable}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,463 +1,156 @@
|
||||||
//! The speaker's relation to the addressee as a lowering input. `with_scene`
|
//! The speaker's relation to the addressee as a lowering input: typed feature
|
||||||
//! names neither speaker nor scene: the view knows its corpus item, so the
|
//! values of slot `Scene`, checked against the inventory at the view's revision,
|
||||||
//! speaker is the item's speaker and the scene is the store's latest for the
|
//! never wording, and only for a packet that names an addressee.
|
||||||
//! item's group. The relation is the one from that speaker to the addressee
|
|
||||||
//! referent's entity. Typed feature values only, never wording.
|
|
||||||
|
|
||||||
mod common;
|
mod common;
|
||||||
use common::{cast, eid, entities_in, grade, item_in, lab, nd};
|
use std::path::Path;
|
||||||
use weksa::inventory::{proof_revision, seed, speaker_scene_revision};
|
|
||||||
|
use common::{grade, iv, lab};
|
||||||
|
use weksa::inventory::{proof_revision, scene_revision, seed};
|
||||||
use weksa::packet::{FeatValue, FeatureValue, Packet, proof_packet};
|
use weksa::packet::{FeatValue, FeatureValue, Packet, proof_packet};
|
||||||
use weksa::speaker::{Annotator, PairRelation, Party, Scene};
|
|
||||||
use weksa::{
|
use weksa::{
|
||||||
CorpusStore, EntityStore, Inventory, InventoryStore, LoweringInput, NameSpan, PacketStore,
|
CorpusStore, EntityStore, Inventory, InventoryStore, LoweringInput, PacketStore, Refusal, lower,
|
||||||
Refusal, SceneRead, SpeakerStateStore, StoreError,
|
|
||||||
};
|
};
|
||||||
|
|
||||||
struct W {
|
fn store() -> (InventoryStore, Inventory, tempfile::TempDir) {
|
||||||
dir: tempfile::TempDir,
|
|
||||||
inventory: InventoryStore,
|
|
||||||
corpus: CorpusStore,
|
|
||||||
entities: EntityStore,
|
|
||||||
state: SpeakerStateStore,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Group `g` holds `item:nibu:1`: Nibu says "Brokkr, come here." Group `h`
|
|
||||||
/// holds `item:nibu:2`, also spoken by Nibu.
|
|
||||||
fn world() -> W {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
let dir = tempfile::tempdir().unwrap();
|
||||||
let mut inventory = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
let mut store = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
||||||
for (rev, entries) in [seed(), proof_revision(), speaker_scene_revision(3)] {
|
for (rev, entries) in [seed(), proof_revision(), scene_revision()] {
|
||||||
inventory.admit_revision(rev, entries).unwrap();
|
store.admit_revision(rev, entries).unwrap();
|
||||||
}
|
|
||||||
let entities = entities_in(dir.path());
|
|
||||||
let mut corpus = CorpusStore::open(&dir.path().join("c.cc")).unwrap();
|
|
||||||
let mut first = item_in("item:nibu:1", "Brokkr, come here.", "g", cast());
|
|
||||||
first.names = vec![NameSpan {
|
|
||||||
entity: eid("brokkr"),
|
|
||||||
start: 0,
|
|
||||||
end: 6,
|
|
||||||
}];
|
|
||||||
corpus.add(first, &entities).unwrap();
|
|
||||||
corpus
|
|
||||||
.add(item_in("item:nibu:2", "Go away.", "h", cast()), &entities)
|
|
||||||
.unwrap();
|
|
||||||
let state = SpeakerStateStore::open(&dir.path().join("s.cc")).unwrap();
|
|
||||||
W {
|
|
||||||
dir,
|
|
||||||
inventory,
|
|
||||||
corpus,
|
|
||||||
entities,
|
|
||||||
state,
|
|
||||||
}
|
}
|
||||||
|
let inv = store.inventory().unwrap();
|
||||||
|
(store, inv, dir)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn inv(w: &W) -> Inventory {
|
fn admitted(packet: &Packet, inventory: &InventoryStore) -> LoweringInput {
|
||||||
w.inventory.inventory().unwrap()
|
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../..");
|
||||||
|
let corpus = CorpusStore::open(&root.join("data/corpus/corpus.cc")).unwrap();
|
||||||
|
let entities = EntityStore::open(&root.join("data/entity/entity.cc")).unwrap();
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
PacketStore::open(&dir.path().join("p.cc"))
|
||||||
|
.unwrap()
|
||||||
|
.admit(packet.clone(), inventory, &corpus, &entities)
|
||||||
|
.unwrap()
|
||||||
}
|
}
|
||||||
|
|
||||||
fn feat(gloss: &str, value: FeatValue) -> FeatureValue {
|
fn at_rev(rev: u32) -> Packet {
|
||||||
let feature = seed()
|
let mut p = proof_packet();
|
||||||
.1
|
|
||||||
.into_iter()
|
|
||||||
.chain(speaker_scene_revision(3).1)
|
|
||||||
.find(|e| e.gloss == gloss)
|
|
||||||
.unwrap_or_else(|| panic!("no feature {gloss}"))
|
|
||||||
.id;
|
|
||||||
FeatureValue { feature, value }
|
|
||||||
}
|
|
||||||
|
|
||||||
fn intimacy(v: u16) -> Vec<FeatureValue> {
|
|
||||||
vec![feat("intimacy", FeatValue::Grade(grade(v)))]
|
|
||||||
}
|
|
||||||
|
|
||||||
fn status() -> Vec<FeatureValue> {
|
|
||||||
vec![
|
|
||||||
feat("relative_status", FeatValue::Option(lab("above"))),
|
|
||||||
feat("intimacy", FeatValue::Grade(grade(200))),
|
|
||||||
feat("in_group", FeatValue::Flag(false)),
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
||||||
fn relation(from: &str, to: Party, features: Vec<FeatureValue>) -> PairRelation {
|
|
||||||
PairRelation {
|
|
||||||
from: eid(from),
|
|
||||||
to,
|
|
||||||
features,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Nibu's relations to Brokkr and to an unnamed party, and Brokkr's to Nibu.
|
|
||||||
fn scene(group: &str, n: u32, participants: &[&str], relations: Vec<PairRelation>) -> Scene {
|
|
||||||
Scene {
|
|
||||||
group: group.into(),
|
|
||||||
n,
|
|
||||||
inventory_rev: 3,
|
|
||||||
participants: participants.iter().map(|p| eid(p)).collect(),
|
|
||||||
relations,
|
|
||||||
annotator: Annotator::Operator,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn standard(group: &str, n: u32) -> Scene {
|
|
||||||
scene(
|
|
||||||
group,
|
|
||||||
n,
|
|
||||||
&["nibu", "brokkr"],
|
|
||||||
vec![
|
|
||||||
relation("nibu", Party::Named(eid("brokkr")), status()),
|
|
||||||
relation("nibu", Party::Unnamed, intimacy(7)),
|
|
||||||
relation("brokkr", Party::Named(eid("nibu")), intimacy(99)),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn put(w: &mut W, s: Scene) {
|
|
||||||
w.state
|
|
||||||
.put_scene(s, &w.inventory, &w.corpus, &w.entities)
|
|
||||||
.unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The line of `item:nibu:1` at `rev`, its addressee node `addressee` (`n1`
|
|
||||||
/// has no entity, `n3` is Brokkr), through the packet door.
|
|
||||||
fn view_of(w: &W, rev: u32, addressee: Option<&str>) -> LoweringInput {
|
|
||||||
view_to(w, rev, addressee, "brokkr")
|
|
||||||
}
|
|
||||||
|
|
||||||
/// As `view_of`, with referent `n3` naming `entity` (Nibu is the item's
|
|
||||||
/// speaker, so he may be named on the line; Brokkr is named by its text).
|
|
||||||
fn view_to(w: &W, rev: u32, addressee: Option<&str>, entity: &str) -> LoweringInput {
|
|
||||||
let mut p: Packet = proof_packet();
|
|
||||||
p.inventory_rev = rev;
|
p.inventory_rev = rev;
|
||||||
p.referents[2].entity = Some(eid(entity));
|
p
|
||||||
p.discourse.addressee = addressee.map(nd);
|
|
||||||
// A packet key holds its packet for good, so each view gets a fresh store.
|
|
||||||
let scratch = tempfile::tempdir().unwrap();
|
|
||||||
PacketStore::open(&scratch.path().join("p.cc"))
|
|
||||||
.unwrap()
|
|
||||||
.admit(p, &w.inventory, &w.corpus, &w.entities)
|
|
||||||
.unwrap()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn rules(r: Result<LoweringInput, StoreError>) -> Vec<&'static str> {
|
fn feat(id: &str, value: FeatValue) -> FeatureValue {
|
||||||
match r.map(|_| ()).unwrap_err() {
|
FeatureValue {
|
||||||
StoreError::Invalid(rs) => rs.iter().map(Refusal::rule).collect(),
|
feature: iv(id),
|
||||||
other => panic!("not a rule refusal: {other}"),
|
value,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn read(w: &W, view: LoweringInput) -> SceneRead {
|
fn status(inv: &Inventory) -> String {
|
||||||
view.with_scene(&inv(w), &w.state, &w.corpus)
|
inv.entries()
|
||||||
|
.find(|e| e.gloss == "relative_status")
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.scene_read()
|
.id
|
||||||
.clone()
|
.to_string()
|
||||||
}
|
}
|
||||||
|
|
||||||
fn selected(rev: u32, features: Vec<FeatureValue>) -> SceneRead {
|
fn id_of(inv: &Inventory, gloss: &str) -> String {
|
||||||
SceneRead::Selected { rev, features }
|
inv.entries()
|
||||||
|
.find(|e| e.gloss == gloss)
|
||||||
|
.unwrap()
|
||||||
|
.id
|
||||||
|
.to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn rules(r: Result<LoweringInput, Vec<Refusal>>) -> Vec<&'static str> {
|
||||||
|
r.unwrap_err().iter().map(Refusal::rule).collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_scene_relation_reaches_the_lowering_view() {
|
fn a_scene_relation_reaches_the_lowering_view_and_changes_no_english() {
|
||||||
let mut w = world();
|
let (store, inv, _dir) = store();
|
||||||
put(&mut w, standard("g", 1));
|
let plain = admitted(&at_rev(3), &store);
|
||||||
let plain = view_of(&w, 3, Some("n3"));
|
|
||||||
assert_eq!(plain.scene_read(), &SceneRead::Unread);
|
|
||||||
assert!(plain.scene().is_empty());
|
assert!(plain.scene().is_empty());
|
||||||
let with = plain
|
let scene = vec![
|
||||||
.clone()
|
feat(&status(&inv), FeatValue::Option(lab("above"))),
|
||||||
.with_scene(&inv(&w), &w.state, &w.corpus)
|
feat(&id_of(&inv, "intimacy"), FeatValue::Grade(grade(200))),
|
||||||
.unwrap();
|
feat(&id_of(&inv, "in_group"), FeatValue::Flag(false)),
|
||||||
assert_eq!(with.scene(), status().as_slice());
|
];
|
||||||
assert_eq!(with.scene_read(), &selected(1, status()));
|
let with = plain.clone().with_scene(&inv, scene.clone()).unwrap();
|
||||||
}
|
assert_eq!(with.scene(), scene.as_slice());
|
||||||
|
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../..");
|
||||||
#[test]
|
let entities = EntityStore::open(&root.join("data/entity/entity.cc")).unwrap();
|
||||||
fn with_scene_says_which_of_five_things_it_found() {
|
|
||||||
let mut w = world();
|
|
||||||
// No scene for the group.
|
|
||||||
assert_eq!(read(&w, view_of(&w, 3, Some("n3"))), SceneRead::NoScene);
|
|
||||||
put(&mut w, standard("g", 1));
|
|
||||||
// A referent with an entity selects the relation to that entity; a
|
|
||||||
// referent without one, the relation to an unnamed party; and the
|
|
||||||
// speaker's relation, never Brokkr's to Nibu.
|
|
||||||
assert_eq!(read(&w, view_of(&w, 3, Some("n3"))), selected(1, status()));
|
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
read(&w, view_of(&w, 3, Some("n1"))),
|
lower(&with, &inv, &entities).unwrap(),
|
||||||
selected(1, intimacy(7))
|
lower(&plain, &inv, &entities).unwrap()
|
||||||
);
|
|
||||||
// The scene was read and the packet names no addressee.
|
|
||||||
assert_eq!(
|
|
||||||
read(&w, view_of(&w, 3, None)),
|
|
||||||
SceneRead::NoAddressee { rev: 1 }
|
|
||||||
);
|
|
||||||
// The addressee is the speaker.
|
|
||||||
assert_eq!(
|
|
||||||
read(&w, view_to(&w, 3, Some("n3"), "nibu")),
|
|
||||||
SceneRead::SelfAddressed { rev: 1 }
|
|
||||||
);
|
|
||||||
// The scene holds no relation to this addressee.
|
|
||||||
put(
|
|
||||||
&mut w,
|
|
||||||
scene(
|
|
||||||
"g",
|
|
||||||
2,
|
|
||||||
&["nibu", "brokkr"],
|
|
||||||
vec![relation("nibu", Party::Unnamed, intimacy(7))],
|
|
||||||
),
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
read(&w, view_of(&w, 3, Some("n3"))),
|
|
||||||
SceneRead::NoRelation { rev: 2 }
|
|
||||||
);
|
|
||||||
// A relation with no features is found, not absent.
|
|
||||||
put(
|
|
||||||
&mut w,
|
|
||||||
scene(
|
|
||||||
"g",
|
|
||||||
3,
|
|
||||||
&["nibu", "brokkr"],
|
|
||||||
vec![relation("nibu", Party::Named(eid("brokkr")), vec![])],
|
|
||||||
),
|
|
||||||
);
|
|
||||||
assert_eq!(read(&w, view_of(&w, 3, Some("n3"))), selected(3, vec![]));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_named_addressee_the_scene_lacks_never_falls_back_to_unnamed() {
|
|
||||||
let mut w = world();
|
|
||||||
// Brokkr is not a participant and Nibu has only an unnamed relation.
|
|
||||||
put(
|
|
||||||
&mut w,
|
|
||||||
scene(
|
|
||||||
"g",
|
|
||||||
1,
|
|
||||||
&["nibu", "player"],
|
|
||||||
vec![relation("nibu", Party::Unnamed, intimacy(7))],
|
|
||||||
),
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
read(&w, view_of(&w, 3, Some("n3"))),
|
|
||||||
SceneRead::NoRelation { rev: 1 }
|
|
||||||
);
|
|
||||||
// A participant without a relation behaves the same.
|
|
||||||
put(
|
|
||||||
&mut w,
|
|
||||||
scene(
|
|
||||||
"g",
|
|
||||||
2,
|
|
||||||
&["nibu", "brokkr"],
|
|
||||||
vec![relation("nibu", Party::Unnamed, intimacy(7))],
|
|
||||||
),
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
read(&w, view_of(&w, 3, Some("n3"))),
|
|
||||||
SceneRead::NoRelation { rev: 2 }
|
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn the_speaker_is_the_items_not_the_first_participant() {
|
fn a_relation_needs_an_addressee() {
|
||||||
let mut w = world();
|
let (store, inv, _dir) = store();
|
||||||
// Brokkr is listed first and has an unnamed relation of his own.
|
let mut p = at_rev(3);
|
||||||
put(
|
p.discourse.addressee = None;
|
||||||
&mut w,
|
let view = admitted(&p, &store);
|
||||||
scene(
|
let scene = vec![feat(&id_of(&inv, "in_group"), FeatValue::Flag(true))];
|
||||||
"g",
|
assert_eq!(rules(view.clone().with_scene(&inv, scene)), ["V13"]);
|
||||||
1,
|
assert!(view.with_scene(&inv, vec![]).unwrap().scene().is_empty());
|
||||||
&["brokkr", "nibu"],
|
|
||||||
vec![
|
|
||||||
relation("brokkr", Party::Unnamed, intimacy(99)),
|
|
||||||
relation("nibu", Party::Unnamed, intimacy(7)),
|
|
||||||
],
|
|
||||||
),
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
read(&w, view_of(&w, 3, Some("n1"))),
|
|
||||||
selected(1, intimacy(7))
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn the_scene_is_the_items_group_and_its_latest_revision() {
|
fn a_relation_value_is_checked_like_any_feature() {
|
||||||
let mut w = world();
|
let (store, inv, _dir) = store();
|
||||||
// Group h holds a different relation for the same pair.
|
let view = admitted(&at_rev(3), &store);
|
||||||
put(
|
let status = status(&inv);
|
||||||
&mut w,
|
let intimacy = id_of(&inv, "intimacy");
|
||||||
scene(
|
let take = |scene| view.clone().with_scene(&inv, scene);
|
||||||
"h",
|
// A label the feature does not declare.
|
||||||
1,
|
assert_eq!(
|
||||||
&["nibu", "brokkr"],
|
rules(take(vec![feat(&status, FeatValue::Option(lab("lordly")))])),
|
||||||
vec![relation("nibu", Party::Named(eid("brokkr")), intimacy(1))],
|
["V7"]
|
||||||
),
|
|
||||||
);
|
);
|
||||||
// No scene for the item's group, whatever other groups hold.
|
// A value of the wrong kind.
|
||||||
assert_eq!(read(&w, view_of(&w, 3, Some("n3"))), SceneRead::NoScene);
|
assert_eq!(
|
||||||
// Revisions: the highest n is read.
|
rules(take(vec![feat(&intimacy, FeatValue::Flag(true))])),
|
||||||
put(&mut w, standard("g", 1));
|
["V7"]
|
||||||
let mut second = standard("g", 2);
|
);
|
||||||
second.relations[0].features = intimacy(5);
|
// A feature declared for another slot: seed feature 1 sits in Predication.
|
||||||
put(&mut w, second);
|
assert_eq!(
|
||||||
let got = view_of(&w, 3, Some("n3"))
|
rules(take(vec![feat(
|
||||||
.with_scene(&inv(&w), &w.state, &w.corpus)
|
"feature:1",
|
||||||
.unwrap();
|
FeatValue::Option(lab("negative"))
|
||||||
assert_eq!(got.scene_read(), &selected(2, intimacy(5)));
|
)])),
|
||||||
assert_eq!(got.scene(), intimacy(5).as_slice());
|
["V8"]
|
||||||
// A read replaces the earlier one: a store without the scene says so.
|
);
|
||||||
let empty = SpeakerStateStore::open(&w.dir.path().join("empty.cc")).unwrap();
|
// An id that is not a feature at all.
|
||||||
let cleared = got.with_scene(&inv(&w), &empty, &w.corpus).unwrap();
|
assert_eq!(
|
||||||
assert_eq!(cleared.scene_read(), &SceneRead::NoScene);
|
rules(take(vec![feat("concept:3", FeatValue::Flag(true))])),
|
||||||
assert!(cleared.scene().is_empty());
|
["V6"]
|
||||||
|
);
|
||||||
|
// Twice in one list.
|
||||||
|
let twice = vec![
|
||||||
|
feat(&intimacy, FeatValue::Grade(grade(1))),
|
||||||
|
feat(&intimacy, FeatValue::Grade(grade(2))),
|
||||||
|
];
|
||||||
|
assert_eq!(rules(take(twice)), ["V11"]);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn with_scene_refusals() {
|
fn a_view_of_a_revision_before_the_scene_vocabulary_cannot_carry_one() {
|
||||||
let mut w = world();
|
let (store, inv, _dir) = store();
|
||||||
put(&mut w, standard("g", 1));
|
// Revision 2 predates the Scene features, so their ids do not resolve there.
|
||||||
// The scene was admitted at revision 3; a view of revision 2 predates the
|
let view = admitted(&at_rev(2), &store);
|
||||||
// Scene features, so the selected relation's ids do not resolve there.
|
let scene = vec![feat(&id_of(&inv, "intimacy"), FeatValue::Grade(grade(1)))];
|
||||||
assert_eq!(
|
assert_eq!(rules(view.with_scene(&inv, scene)), ["V5"]);
|
||||||
rules(view_of(&w, 2, Some("n3")).with_scene(&inv(&w), &w.state, &w.corpus)),
|
|
||||||
["V5", "V5", "V5"]
|
|
||||||
);
|
|
||||||
// A scene that does not list the item's speaker: group k admitted while
|
|
||||||
// Brokkr was its only speaker, then an item of Nibu's joins it.
|
|
||||||
let mut item = item_in("item:brokkr:1", "Hello.", "k", cast());
|
|
||||||
item.speaker = eid("brokkr");
|
|
||||||
w.corpus.add(item, &w.entities).unwrap();
|
|
||||||
put(&mut w, scene("k", 1, &["brokkr"], vec![]));
|
|
||||||
w.corpus
|
|
||||||
.add(
|
|
||||||
item_in("item:nibu:3", "Hello there.", "k", cast()),
|
|
||||||
&w.entities,
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
for addressee in [Some("n3"), None] {
|
|
||||||
let mut p = proof_packet();
|
|
||||||
p.inventory_rev = 3;
|
|
||||||
p.provenance.corpus_item = weksa::CorpusItemId::parse("item:nibu:3").unwrap();
|
|
||||||
p.referents[2].entity = Some(eid("nibu"));
|
|
||||||
p.discourse.addressee = addressee.map(nd);
|
|
||||||
let scratch = tempfile::tempdir().unwrap();
|
|
||||||
let view = PacketStore::open(&scratch.path().join("p.cc"))
|
|
||||||
.unwrap()
|
|
||||||
.admit(p, &w.inventory, &w.corpus, &w.entities)
|
|
||||||
.unwrap();
|
|
||||||
assert_eq!(
|
|
||||||
rules(view.with_scene(&inv(&w), &w.state, &w.corpus)),
|
|
||||||
["V16"],
|
|
||||||
"{addressee:?}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn read_refuses_a_scene_that_stopped_covering_a_live_speaker_of_its_group() {
|
fn a_stance_feature_is_not_a_scene_feature() {
|
||||||
// Group g's scene covers Nibu alone, and Nibu's is the only item: admitted.
|
let (store, inv, _dir) = store();
|
||||||
let mut w = world();
|
let view = admitted(&at_rev(3), &store);
|
||||||
put(&mut w, scene("g", 1, &["nibu"], vec![]));
|
let anger = id_of(&inv, "anger");
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
read(&w, view_of(&w, 3, None)),
|
rules(view.with_scene(&inv, vec![feat(&anger, FeatValue::Grade(grade(1)))])),
|
||||||
SceneRead::NoAddressee { rev: 1 }
|
["V8"]
|
||||||
);
|
|
||||||
// Brokkr then speaks in g. The corpus door lets him in; the view of Nibu's
|
|
||||||
// own line, whose speaker the scene still lists, reads the stale scene no more.
|
|
||||||
let mut item = item_in("item:brokkr:1", "Hello.", "g", cast());
|
|
||||||
item.speaker = eid("brokkr");
|
|
||||||
w.corpus.add(item, &w.entities).unwrap();
|
|
||||||
assert_eq!(
|
|
||||||
rules(view_of(&w, 3, None).with_scene(&inv(&w), &w.state, &w.corpus)),
|
|
||||||
["V16"]
|
|
||||||
);
|
|
||||||
// The put door applies the same rule, so the repair is a revision that
|
|
||||||
// covers him.
|
|
||||||
put(&mut w, scene("g", 2, &["nibu", "brokkr"], vec![]));
|
|
||||||
assert_eq!(
|
|
||||||
read(&w, view_of(&w, 3, None)),
|
|
||||||
SceneRead::NoAddressee { rev: 2 }
|
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn only_live_speakers_of_the_views_group_and_the_views_own_speaker_must_be_covered() {
|
|
||||||
let mut w = world();
|
|
||||||
put(&mut w, scene("g", 1, &["nibu"], vec![]));
|
|
||||||
let ok = SceneRead::NoAddressee { rev: 1 };
|
|
||||||
// A withdrawn speaker the scene lacks does not count.
|
|
||||||
let mut gone = item_in("item:brokkr:1", "Hello.", "g", cast());
|
|
||||||
gone.speaker = eid("brokkr");
|
|
||||||
w.corpus.add(gone, &w.entities).unwrap();
|
|
||||||
w.corpus.withdraw("item:brokkr:1").unwrap();
|
|
||||||
assert_eq!(read(&w, view_of(&w, 3, None)), ok);
|
|
||||||
// Nor does a live speaker the scene lacks in another group.
|
|
||||||
let mut other = item_in("item:brokkr:2", "Hello you.", "h", cast());
|
|
||||||
other.speaker = eid("brokkr");
|
|
||||||
w.corpus.add(other, &w.entities).unwrap();
|
|
||||||
assert_eq!(read(&w, view_of(&w, 3, None)), ok);
|
|
||||||
// The view's own speaker must be listed even when its item was withdrawn
|
|
||||||
// after the view was built: scene g@2 lists Brokkr alone.
|
|
||||||
let view = view_of(&w, 3, None);
|
|
||||||
w.corpus.withdraw("item:nibu:1").unwrap();
|
|
||||||
put(&mut w, scene("g", 2, &["brokkr"], vec![]));
|
|
||||||
assert_eq!(
|
|
||||||
rules(view.with_scene(&inv(&w), &w.state, &w.corpus)),
|
|
||||||
["V16"]
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
mod select {
|
|
||||||
use super::*;
|
|
||||||
use weksa::speaker::Annotator;
|
|
||||||
use weksa::{Condition, Layer, OntologyEntry, OntologyStore, Slot, Test, select};
|
|
||||||
|
|
||||||
fn scene_feature(gloss: &str) -> weksa::InvId {
|
|
||||||
speaker_scene_revision(3).1.into_iter().find(|e| e.gloss == gloss).unwrap().id
|
|
||||||
}
|
|
||||||
|
|
||||||
fn entry(n: u32, slot: Slot, gloss: &str, test: Test) -> OntologyEntry {
|
|
||||||
OntologyEntry {
|
|
||||||
language: "ja-JP".into(),
|
|
||||||
n,
|
|
||||||
inventory_rev: 3,
|
|
||||||
layer: Layer::Grammar,
|
|
||||||
when: vec![Condition { slot, feature: scene_feature(gloss), test }],
|
|
||||||
choice: "a choice".into(),
|
|
||||||
examples: vec![],
|
|
||||||
annotator: Annotator::Operator,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn select_reads_the_scene_relation() {
|
|
||||||
let mut w = world();
|
|
||||||
put(&mut w, standard("g", 1));
|
|
||||||
let mut store = OntologyStore::open(&w.dir.path().join("o.cc")).unwrap();
|
|
||||||
store
|
|
||||||
.admit(
|
|
||||||
vec![
|
|
||||||
entry(1, Slot::Scene, "relative_status", Test::Is(FeatValue::Option(lab("above")))),
|
|
||||||
entry(2, Slot::Scene, "intimacy", Test::AtLeast(grade(150))),
|
|
||||||
],
|
|
||||||
&w.inventory,
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
let ontology = store.ontology().unwrap();
|
|
||||||
let keys = |read: SceneRead, w: &W| {
|
|
||||||
let view = view_of(w, 3, Some("n3")).with_scene(&inv(w), &w.state, &w.corpus).unwrap();
|
|
||||||
assert_eq!(view.scene_read(), &read);
|
|
||||||
select(&ontology, "ja-JP", &view).iter().map(|e| e.n).collect::<Vec<_>>()
|
|
||||||
};
|
|
||||||
// Brokkr is above-status at intimacy 200: both Scene entries select.
|
|
||||||
assert_eq!(keys(selected(1, status()), &w), [1, 2]);
|
|
||||||
// No scene for the group: neither selects.
|
|
||||||
let empty = world();
|
|
||||||
assert_eq!(keys(SceneRead::NoScene, &empty), Vec::<u32>::new());
|
|
||||||
// No addressee, and no relation to the addressee: neither selects.
|
|
||||||
let view = view_of(&w, 3, None).with_scene(&inv(&w), &w.state, &w.corpus).unwrap();
|
|
||||||
assert_eq!(view.scene_read(), &SceneRead::NoAddressee { rev: 1 });
|
|
||||||
assert!(select(&ontology, "ja-JP", &view).is_empty());
|
|
||||||
let mut lone = world();
|
|
||||||
put(&mut lone, scene("g", 1, &["nibu", "brokkr"], vec![relation("brokkr", Party::Named(eid("nibu")), intimacy(99))]));
|
|
||||||
assert_eq!(keys(SceneRead::NoRelation { rev: 1 }, &lone), Vec::<u32>::new());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -6,13 +6,14 @@ use std::net::TcpListener;
|
||||||
use std::sync::{Arc, Mutex};
|
use std::sync::{Arc, Mutex};
|
||||||
|
|
||||||
use anyhow::Result;
|
use anyhow::Result;
|
||||||
use common::{Bag, cast, group_in_bucket, item_in, model};
|
use common::{cast, group_in_bucket, item_in, model};
|
||||||
use serde_json::Value;
|
use serde_json::Value;
|
||||||
use weksa::CorpusItem;
|
use weksa::CorpusItem;
|
||||||
use weksa::score::{
|
use weksa::score::{
|
||||||
Embedder, ItemScore, OllamaEmbedder, ScoreInput, Scorer, aggregate, chrf, cosine,
|
Embedder, ItemScore, OllamaEmbedder, ScoreInput, Scorer, TestReadings, aggregate, chrf, cosine,
|
||||||
leak_diagnostic,
|
leak_diagnostic,
|
||||||
};
|
};
|
||||||
|
use weksa::split::Checkpoint;
|
||||||
|
|
||||||
/// (reference, hypothesis, sacrebleu 2.4.3 sentence_chrf), map 491536c.
|
/// (reference, hypothesis, sacrebleu 2.4.3 sentence_chrf), map 491536c.
|
||||||
const VECTORS: [(&str, &str, f64); 6] = [
|
const VECTORS: [(&str, &str, f64); 6] = [
|
||||||
|
|
@ -45,8 +46,8 @@ fn chrf_matches_sacrebleu_vectors() {
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn chrf_argument_order() {
|
fn chrf_argument_order() {
|
||||||
let short = "shut the gate";
|
let short = "mind the hatch";
|
||||||
let long = "shut the gate before the storm arrives tonight";
|
let long = "mind the hatch before the storm arrives tonight";
|
||||||
let (as_hypothesis, as_reference) = (chrf(short, long), chrf(long, short));
|
let (as_hypothesis, as_reference) = (chrf(short, long), chrf(long, short));
|
||||||
assert!(
|
assert!(
|
||||||
as_hypothesis < as_reference - 1.0,
|
as_hypothesis < as_reference - 1.0,
|
||||||
|
|
@ -71,6 +72,28 @@ impl Embedder for Recorder {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A deterministic bag-of-letters embedder, so cosine differs with the text.
|
||||||
|
struct Bag;
|
||||||
|
|
||||||
|
impl Embedder for Bag {
|
||||||
|
fn model(&self) -> &str {
|
||||||
|
"bag"
|
||||||
|
}
|
||||||
|
fn embed(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
|
||||||
|
Ok(texts
|
||||||
|
.iter()
|
||||||
|
.map(|t| {
|
||||||
|
let mut v = vec![0.0; 27];
|
||||||
|
v[26] = 1.0;
|
||||||
|
for c in t.chars().filter(char::is_ascii_alphabetic) {
|
||||||
|
v[(c.to_ascii_lowercase() as u8 - b'a') as usize] += 1.0;
|
||||||
|
}
|
||||||
|
v
|
||||||
|
})
|
||||||
|
.collect())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
fn score_one<E: Embedder>(scorer: &Scorer<E>, reference: &str, output: &str) -> ItemScore {
|
fn score_one<E: Embedder>(scorer: &Scorer<E>, reference: &str, output: &str) -> ItemScore {
|
||||||
scorer
|
scorer
|
||||||
.score(&[ScoreInput { reference, output }])
|
.score(&[ScoreInput { reference, output }])
|
||||||
|
|
@ -104,8 +127,8 @@ fn chrf_whitespace_parity() {
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn scorer_argument_order() {
|
fn scorer_argument_order() {
|
||||||
let short = "shut the gate";
|
let short = "mind the hatch";
|
||||||
let long = "shut the gate before the storm arrives tonight";
|
let long = "mind the hatch before the storm arrives tonight";
|
||||||
let scorer = Scorer::new(Bag);
|
let scorer = Scorer::new(Bag);
|
||||||
let got = score_one(&scorer, long, short);
|
let got = score_one(&scorer, long, short);
|
||||||
assert_eq!(got.chrf, chrf(short, long));
|
assert_eq!(got.chrf, chrf(short, long));
|
||||||
|
|
@ -156,7 +179,7 @@ impl Embedder for Fixed {
|
||||||
fn scorer_refuses_bad_vectors() {
|
fn scorer_refuses_bad_vectors() {
|
||||||
let run = |vectors: Vec<Vec<f32>>| {
|
let run = |vectors: Vec<Vec<f32>>| {
|
||||||
Scorer::new(Fixed(vectors)).score(&[ScoreInput {
|
Scorer::new(Fixed(vectors)).score(&[ScoreInput {
|
||||||
reference: "shut the gate",
|
reference: "mind the hatch",
|
||||||
output: "mind the door",
|
output: "mind the door",
|
||||||
}])
|
}])
|
||||||
};
|
};
|
||||||
|
|
@ -671,16 +694,16 @@ fn every_script_scores_a_near_miss_as_sacrebleu_does() {
|
||||||
#[test]
|
#[test]
|
||||||
fn coverage_splits_at_any_non_word_character() {
|
fn coverage_splits_at_any_non_word_character() {
|
||||||
let items = [
|
let items = [
|
||||||
validation("item:v:1", "shut the gate before the storm"),
|
validation("item:v:1", "mind the hatch before the storm"),
|
||||||
train("item:t:1", UNRELATED),
|
train("item:t:1", UNRELATED),
|
||||||
];
|
];
|
||||||
let want = read(&["shut the gate"], &items);
|
let want = read(&["mind the hatch"], &items);
|
||||||
assert_eq!(want, [(5, 3), (5, 2), (4, 1)]);
|
assert_eq!(want, [(5, 3), (5, 2), (4, 1)]);
|
||||||
for joiner in [
|
for joiner in [
|
||||||
":", "\u{b7}", "'", "\u{2019}", "\u{ff3f}", "-", "/", "\u{2014}", "|", "+", "\u{ff1a}",
|
":", "\u{b7}", "'", "\u{2019}", "\u{ff3f}", "-", "/", "\u{2014}", "|", "+", "\u{ff1a}",
|
||||||
"\u{30fb}",
|
"\u{30fb}",
|
||||||
] {
|
] {
|
||||||
let glued = format!("shut{joiner}the{joiner}gate");
|
let glued = format!("mind{joiner}the{joiner}hatch");
|
||||||
assert_eq!(read(&[&glued], &items), want, "{joiner:?}");
|
assert_eq!(read(&[&glued], &items), want, "{joiner:?}");
|
||||||
}
|
}
|
||||||
// a mark, a joiner or a digit is part of its word, not a break
|
// a mark, a joiner or a digit is part of its word, not a break
|
||||||
|
|
@ -854,6 +877,78 @@ fn leak_diagnostic_reads_validation_only() {
|
||||||
assert!(on_validation.max_chrf.unwrap().chrf > 99.0);
|
assert!(on_validation.max_chrf.unwrap().chrf > 99.0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_readings_at_checkpoint() {
|
||||||
|
let train_item = train("item:t:1", UNRELATED);
|
||||||
|
let test = item_in("item:c:1", HELD_OUT, &group_in_bucket(90), cast());
|
||||||
|
let test_model = item_in(
|
||||||
|
"item:c:2",
|
||||||
|
"model words here",
|
||||||
|
&group_in_bucket(90),
|
||||||
|
model(),
|
||||||
|
);
|
||||||
|
let mut gone = item_in("item:c:3", "withdrawn words", &group_in_bucket(90), cast());
|
||||||
|
gone.withdrawn = true;
|
||||||
|
let val = validation("item:v:1", "gardening is quiet work");
|
||||||
|
let items = [
|
||||||
|
train_item.clone(),
|
||||||
|
test.clone(),
|
||||||
|
test_model.clone(),
|
||||||
|
gone.clone(),
|
||||||
|
val.clone(),
|
||||||
|
];
|
||||||
|
let scored = [
|
||||||
|
(&train_item, score(10.0, Some(0.5))),
|
||||||
|
(&val, score(40.0, Some(0.8))),
|
||||||
|
(&test, score(60.0, Some(0.6))),
|
||||||
|
(&test_model, score(100.0, Some(1.0))),
|
||||||
|
(&gone, score(90.0, Some(0.9))),
|
||||||
|
];
|
||||||
|
let t = TestReadings::at_checkpoint(Checkpoint::declare(2), &scored, &[HELD_OUT], &items);
|
||||||
|
assert_eq!(
|
||||||
|
(t.count(), t.chrf(), t.meaning()),
|
||||||
|
(1, Some(60.0), Some(0.6))
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
t.items(),
|
||||||
|
1,
|
||||||
|
"model-written and withdrawn Test items are excluded"
|
||||||
|
);
|
||||||
|
assert!(t.max_chrf().unwrap() > 99.0);
|
||||||
|
assert_eq!(t.coverage_by_n()[2].share, Some(1.0));
|
||||||
|
|
||||||
|
// A minted string that is not the Test text reads low; the same method as
|
||||||
|
// the Validation reading, so the same minted text reads alike in each.
|
||||||
|
let t = TestReadings::at_checkpoint(Checkpoint::declare(2), &scored, &["zzz"], &items);
|
||||||
|
assert!(t.max_chrf().unwrap() < 20.0);
|
||||||
|
assert_eq!(t.coverage_by_n()[2].share, Some(0.0));
|
||||||
|
|
||||||
|
let empty = TestReadings::at_checkpoint(Checkpoint::declare(2), &[], &[HELD_OUT], &[]);
|
||||||
|
assert_eq!(
|
||||||
|
(empty.count(), empty.chrf(), empty.meaning()),
|
||||||
|
(0, None, None)
|
||||||
|
);
|
||||||
|
assert_eq!((empty.items(), empty.max_chrf()), (0, None));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn validation_and_test_leak_agree_in_method() {
|
||||||
|
// The same text and minted strings, once in Validation and once in Test,
|
||||||
|
// read the same numbers.
|
||||||
|
let minted = chunks(3, " ");
|
||||||
|
let minted = refs(&minted);
|
||||||
|
let train_item = train("item:t:1", UNRELATED);
|
||||||
|
let v = leak_diagnostic(
|
||||||
|
&minted,
|
||||||
|
&[validation("item:v:1", HELD_OUT), train_item.clone()],
|
||||||
|
);
|
||||||
|
let test = item_in("item:c:1", HELD_OUT, &group_in_bucket(90), cast());
|
||||||
|
let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &minted, &[test, train_item]);
|
||||||
|
assert_eq!(t.items(), v.items);
|
||||||
|
assert_eq!(t.max_chrf(), v.max_chrf.as_ref().map(|p| p.chrf));
|
||||||
|
assert_eq!(t.coverage_by_n(), v.coverage_by_n.as_slice());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn held_out_train_union_excludes_withdrawn_train_items() {
|
fn held_out_train_union_excludes_withdrawn_train_items() {
|
||||||
let mut gone = train("item:t:1", HELD_OUT);
|
let mut gone = train("item:t:1", HELD_OUT);
|
||||||
|
|
|
||||||
|
|
@ -1,503 +0,0 @@
|
||||||
//! The speaker-state store: speaker and scene documents are append-only
|
|
||||||
//! revisions, checked against the inventory, the entity store and the corpus.
|
|
||||||
|
|
||||||
mod common;
|
|
||||||
use common::{cast, eid, entities_in, grade, item_in, lab};
|
|
||||||
use weksa::inventory::{proof_revision, seed, speaker_scene_revision};
|
|
||||||
use weksa::packet::{FeatValue, FeatureValue};
|
|
||||||
use weksa::speaker::{
|
|
||||||
Annotator, MAX_PARTICIPANTS, PairRelation, Party, Scene, Speaker, scene_key, speaker_key,
|
|
||||||
};
|
|
||||||
use weksa::{
|
|
||||||
AddOutcome, CorpusStore, EntityStore, InventoryStore, Refusal, SpeakerStateStore, StoreError,
|
|
||||||
StoreRefusal,
|
|
||||||
};
|
|
||||||
|
|
||||||
struct W {
|
|
||||||
dir: tempfile::TempDir,
|
|
||||||
inventory: InventoryStore,
|
|
||||||
corpus: CorpusStore,
|
|
||||||
entities: EntityStore,
|
|
||||||
state: SpeakerStateStore,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Group `g` holds two live items (speakers nibu and brokkr) and one withdrawn
|
|
||||||
/// item (speaker hero); group `g2` holds one live item of nibu's.
|
|
||||||
fn world() -> W {
|
|
||||||
let dir = tempfile::tempdir().unwrap();
|
|
||||||
let mut inventory = InventoryStore::open(&dir.path().join("i.cc")).unwrap();
|
|
||||||
for (rev, entries) in [seed(), proof_revision(), speaker_scene_revision(3)] {
|
|
||||||
inventory.admit_revision(rev, entries).unwrap();
|
|
||||||
}
|
|
||||||
let entities = entities_in(dir.path());
|
|
||||||
let mut corpus = CorpusStore::open(&dir.path().join("c.cc")).unwrap();
|
|
||||||
for (key, text, speaker) in [
|
|
||||||
("item:nibu:1", "Come here.", "nibu"),
|
|
||||||
("item:nibu:2", "Go there.", "brokkr"),
|
|
||||||
("item:nibu:3", "Stay away.", "hero"),
|
|
||||||
("item:nibu:4", "Elsewhere.", "g2"),
|
|
||||||
] {
|
|
||||||
let mut item = item_in(key, text, "g", cast());
|
|
||||||
item.speaker = eid(speaker);
|
|
||||||
if key == "item:nibu:4" {
|
|
||||||
item.group = "g2".into();
|
|
||||||
item.speaker = eid("nibu");
|
|
||||||
}
|
|
||||||
corpus.add(item, &entities).unwrap();
|
|
||||||
}
|
|
||||||
corpus.withdraw("item:nibu:3").unwrap();
|
|
||||||
let state = SpeakerStateStore::open(&dir.path().join("s.cc")).unwrap();
|
|
||||||
W {
|
|
||||||
dir,
|
|
||||||
inventory,
|
|
||||||
corpus,
|
|
||||||
entities,
|
|
||||||
state,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn feat(gloss: &str, value: FeatValue) -> FeatureValue {
|
|
||||||
let feature = seed()
|
|
||||||
.1
|
|
||||||
.into_iter()
|
|
||||||
.chain(speaker_scene_revision(3).1)
|
|
||||||
.find(|e| e.gloss == gloss)
|
|
||||||
.unwrap_or_else(|| panic!("no feature {gloss}"))
|
|
||||||
.id;
|
|
||||||
FeatureValue { feature, value }
|
|
||||||
}
|
|
||||||
|
|
||||||
fn speaker(n: u32, age: &str) -> Speaker {
|
|
||||||
Speaker {
|
|
||||||
entity: eid("nibu"),
|
|
||||||
n,
|
|
||||||
inventory_rev: 3,
|
|
||||||
features: vec![feat("age_cohort", FeatValue::Option(lab(age)))],
|
|
||||||
annotator: Annotator::Operator,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn scene(n: u32, intimacy: u16) -> Scene {
|
|
||||||
Scene {
|
|
||||||
group: "g".into(),
|
|
||||||
n,
|
|
||||||
inventory_rev: 3,
|
|
||||||
participants: vec![eid("nibu"), eid("brokkr")],
|
|
||||||
relations: vec![PairRelation {
|
|
||||||
from: eid("nibu"),
|
|
||||||
to: Party::Named(eid("brokkr")),
|
|
||||||
features: vec![feat("intimacy", FeatValue::Grade(grade(intimacy)))],
|
|
||||||
}],
|
|
||||||
annotator: Annotator::Operator,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn put_speaker(w: &mut W, s: Speaker) -> Result<AddOutcome, StoreError> {
|
|
||||||
w.state.put_speaker(s, &w.inventory, &w.entities)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn put_scene(w: &mut W, s: Scene) -> Result<AddOutcome, StoreError> {
|
|
||||||
w.state.put_scene(s, &w.inventory, &w.corpus, &w.entities)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn refused<T: std::fmt::Debug>(r: Result<T, StoreError>) -> StoreRefusal {
|
|
||||||
match r.unwrap_err() {
|
|
||||||
StoreError::Refused(r) => r,
|
|
||||||
other => panic!("not a store refusal: {other}"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn malformed(r: Result<AddOutcome, StoreError>, key: &str, why: &str) {
|
|
||||||
match refused(r) {
|
|
||||||
StoreRefusal::Malformed { key: k, why: w } => assert_eq!((k.as_str(), w), (key, why)),
|
|
||||||
other => panic!("not Malformed: {other:?}"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn rules(r: Result<AddOutcome, StoreError>) -> Vec<&'static str> {
|
|
||||||
match r.unwrap_err() {
|
|
||||||
StoreError::Invalid(rs) => rs.iter().map(Refusal::rule).collect(),
|
|
||||||
other => panic!("not a rule refusal: {other}"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn not_next(key: &str, expected: u32, got: u32) -> StoreRefusal {
|
|
||||||
StoreRefusal::RevisionNotNext {
|
|
||||||
key: key.into(),
|
|
||||||
expected,
|
|
||||||
got,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn keys_derive_from_the_document() {
|
|
||||||
assert_eq!(speaker_key(&eid("nibu"), 2), "speaker:nibu@2");
|
|
||||||
assert_eq!(scene_key("g", 3), "scene:g@3");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn speaker_revisions_append() {
|
|
||||||
let mut w = world();
|
|
||||||
let key = |n| speaker_key(&eid("nibu"), n);
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_speaker(&mut w, speaker(2, "adult"))),
|
|
||||||
not_next(&key(2), 1, 2)
|
|
||||||
);
|
|
||||||
assert!(w.state.speaker(&eid("nibu")).unwrap().is_none());
|
|
||||||
assert_eq!(
|
|
||||||
put_speaker(&mut w, speaker(1, "adult")).unwrap(),
|
|
||||||
AddOutcome::Added
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
put_speaker(&mut w, speaker(1, "adult")).unwrap(),
|
|
||||||
AddOutcome::Unchanged
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_speaker(&mut w, speaker(1, "elder"))),
|
|
||||||
not_next(&key(1), 2, 1)
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_speaker(&mut w, speaker(3, "elder"))),
|
|
||||||
not_next(&key(3), 2, 3)
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
put_speaker(&mut w, speaker(2, "elder")).unwrap(),
|
|
||||||
AddOutcome::Added
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
w.state.speaker(&eid("nibu")).unwrap(),
|
|
||||||
Some(speaker(2, "elder"))
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
w.state.get_speaker(&key(1)).unwrap(),
|
|
||||||
Some(speaker(1, "adult"))
|
|
||||||
);
|
|
||||||
// Another entity's revisions count from 1 and stay apart.
|
|
||||||
let mut other = speaker(1, "child");
|
|
||||||
other.entity = eid("brokkr");
|
|
||||||
assert_eq!(put_speaker(&mut w, other).unwrap(), AddOutcome::Added);
|
|
||||||
assert_eq!(w.state.speaker(&eid("nibu")).unwrap().unwrap().n, 2);
|
|
||||||
// Revision 10 sorts after 2 in the listing.
|
|
||||||
for n in 3..=10 {
|
|
||||||
put_speaker(&mut w, speaker(n, "adult")).unwrap();
|
|
||||||
}
|
|
||||||
let listed: Vec<_> = w
|
|
||||||
.state
|
|
||||||
.all_speakers()
|
|
||||||
.unwrap()
|
|
||||||
.iter()
|
|
||||||
.map(|s| (s.entity.to_string(), s.n))
|
|
||||||
.collect();
|
|
||||||
assert_eq!(listed.len(), 11);
|
|
||||||
assert_eq!(listed[0], ("entity:brokkr".into(), 1));
|
|
||||||
assert_eq!(listed[10], ("entity:nibu".into(), 10));
|
|
||||||
assert_eq!(w.state.speaker(&eid("nibu")).unwrap().unwrap().n, 10);
|
|
||||||
// The documents survive a reopen.
|
|
||||||
let again = SpeakerStateStore::open(&w.dir.path().join("s.cc")).unwrap();
|
|
||||||
assert_eq!(again.all_speakers().unwrap().len(), 11);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn scene_revisions_append() {
|
|
||||||
let mut w = world();
|
|
||||||
let key = |n| scene_key("g", n);
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_scene(&mut w, scene(2, 5))),
|
|
||||||
not_next(&key(2), 1, 2)
|
|
||||||
);
|
|
||||||
assert!(w.state.scene("g").unwrap().is_none());
|
|
||||||
assert_eq!(put_scene(&mut w, scene(1, 5)).unwrap(), AddOutcome::Added);
|
|
||||||
assert_eq!(
|
|
||||||
put_scene(&mut w, scene(1, 5)).unwrap(),
|
|
||||||
AddOutcome::Unchanged
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_scene(&mut w, scene(1, 6))),
|
|
||||||
not_next(&key(1), 2, 1)
|
|
||||||
);
|
|
||||||
assert_eq!(put_scene(&mut w, scene(2, 6)).unwrap(), AddOutcome::Added);
|
|
||||||
assert_eq!(w.state.scene("g").unwrap(), Some(scene(2, 6)));
|
|
||||||
assert_eq!(w.state.get_scene(&key(1)).unwrap(), Some(scene(1, 5)));
|
|
||||||
for n in 3..=10 {
|
|
||||||
put_scene(&mut w, scene(n, 6)).unwrap();
|
|
||||||
}
|
|
||||||
assert_eq!(w.state.scene("g").unwrap().unwrap().n, 10);
|
|
||||||
let listed: Vec<u32> = w.state.all_scenes().unwrap().iter().map(|s| s.n).collect();
|
|
||||||
assert_eq!(listed, (1..=10).collect::<Vec<_>>());
|
|
||||||
// Numbering is per group: the first scene of another group is revision 1
|
|
||||||
// whatever g holds, and its second is 2.
|
|
||||||
let other = |n| Scene {
|
|
||||||
group: "g2".into(),
|
|
||||||
participants: vec![eid("nibu")],
|
|
||||||
relations: vec![],
|
|
||||||
..scene(n, 1)
|
|
||||||
};
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_scene(&mut w, other(2))),
|
|
||||||
not_next(&scene_key("g2", 2), 1, 2)
|
|
||||||
);
|
|
||||||
assert_eq!(put_scene(&mut w, other(1)).unwrap(), AddOutcome::Added);
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_scene(&mut w, other(3))),
|
|
||||||
not_next(&scene_key("g2", 3), 2, 3)
|
|
||||||
);
|
|
||||||
assert_eq!(put_scene(&mut w, other(2)).unwrap(), AddOutcome::Added);
|
|
||||||
assert_eq!(w.state.scene("g2").unwrap().unwrap().n, 2);
|
|
||||||
assert_eq!(w.state.scene("g").unwrap().unwrap().n, 10);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn speaker_needs_a_known_entity() {
|
|
||||||
let mut w = world();
|
|
||||||
let mut s = speaker(1, "adult");
|
|
||||||
s.entity = eid("nobody");
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_speaker(&mut w, s)),
|
|
||||||
StoreRefusal::UnknownEntity {
|
|
||||||
entity: eid("nobody")
|
|
||||||
}
|
|
||||||
);
|
|
||||||
assert!(w.state.all_speakers().unwrap().is_empty());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn annotation_features_are_checked_like_packet_features() {
|
|
||||||
let mut w = world();
|
|
||||||
let age = || feat("age_cohort", FeatValue::Option(lab("adult")));
|
|
||||||
let with = |features: Vec<FeatureValue>, rev| Speaker {
|
|
||||||
features,
|
|
||||||
inventory_rev: rev,
|
|
||||||
..speaker(1, "adult")
|
|
||||||
};
|
|
||||||
// A Scene feature and a Stance feature do not apply to a speaker.
|
|
||||||
let f = vec![feat("intimacy", FeatValue::Grade(grade(1)))];
|
|
||||||
assert_eq!(rules(put_speaker(&mut w, with(f, 3))), ["V8"]);
|
|
||||||
let f = vec![feat("anger", FeatValue::Grade(grade(1)))];
|
|
||||||
assert_eq!(rules(put_speaker(&mut w, with(f, 3))), ["V8"]);
|
|
||||||
// A wrong kind and an undeclared label.
|
|
||||||
let f = vec![feat("education", FeatValue::Flag(true))];
|
|
||||||
assert_eq!(rules(put_speaker(&mut w, with(f, 3))), ["V7"]);
|
|
||||||
let f = vec![feat("age_cohort", FeatValue::Option(lab("ancient")))];
|
|
||||||
assert_eq!(rules(put_speaker(&mut w, with(f, 3))), ["V7"]);
|
|
||||||
// Feature 22 does not exist at revision 2; revision 9 has no record.
|
|
||||||
assert_eq!(rules(put_speaker(&mut w, with(vec![age()], 2))), ["V5"]);
|
|
||||||
assert_eq!(rules(put_speaker(&mut w, with(vec![age()], 9))), ["V10"]);
|
|
||||||
// A repeat.
|
|
||||||
assert_eq!(
|
|
||||||
rules(put_speaker(&mut w, with(vec![age(), age()], 3))),
|
|
||||||
["V11"]
|
|
||||||
);
|
|
||||||
// A Speaker feature in a scene relation.
|
|
||||||
let mut s = scene(1, 1);
|
|
||||||
s.relations[0].features = vec![age()];
|
|
||||||
assert_eq!(rules(put_scene(&mut w, s)), ["V8"]);
|
|
||||||
// Every relation is checked, and a scene of an unknown revision is V10.
|
|
||||||
let mut s = scene(1, 1);
|
|
||||||
s.relations.push(PairRelation {
|
|
||||||
from: eid("brokkr"),
|
|
||||||
to: Party::Unnamed,
|
|
||||||
features: vec![age()],
|
|
||||||
});
|
|
||||||
assert_eq!(rules(put_scene(&mut w, s)), ["V8"]);
|
|
||||||
let mut s = scene(1, 1);
|
|
||||||
s.inventory_rev = 9;
|
|
||||||
assert_eq!(rules(put_scene(&mut w, s)), ["V10"]);
|
|
||||||
let mut s = scene(1, 1);
|
|
||||||
s.inventory_rev = 9;
|
|
||||||
s.relations.clear();
|
|
||||||
assert_eq!(rules(put_scene(&mut w, s)), ["V10"]);
|
|
||||||
assert!(w.state.all_speakers().unwrap().is_empty());
|
|
||||||
assert!(w.state.all_scenes().unwrap().is_empty());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn scene_shape_rules() {
|
|
||||||
let mut w = world();
|
|
||||||
let key = scene_key("g", 1);
|
|
||||||
let relation = |from: &str, to: Party| PairRelation {
|
|
||||||
from: eid(from),
|
|
||||||
to,
|
|
||||||
features: vec![],
|
|
||||||
};
|
|
||||||
let named = |to: &str| Party::Named(eid(to));
|
|
||||||
let base = || scene(1, 1);
|
|
||||||
|
|
||||||
let mut s = base();
|
|
||||||
s.group = "Not A Slug".into();
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&scene_key("Not A Slug", 1),
|
|
||||||
"a scene group is a slug",
|
|
||||||
);
|
|
||||||
let mut s = base();
|
|
||||||
s.group = "elsewhere".into();
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&scene_key("elsewhere", 1),
|
|
||||||
"no corpus item holds the scene's group",
|
|
||||||
);
|
|
||||||
let mut s = base();
|
|
||||||
s.participants.clear();
|
|
||||||
s.relations.clear();
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&key,
|
|
||||||
"a scene has 1 to 64 participants",
|
|
||||||
);
|
|
||||||
// 64 participants are fine as far as the count goes (the rest fail later);
|
|
||||||
// 65 are refused for the count.
|
|
||||||
let many = |count: usize| -> Vec<_> { (0..count).map(|i| eid(&format!("p{i}"))).collect() };
|
|
||||||
let mut s = base();
|
|
||||||
s.participants = many(MAX_PARTICIPANTS + 1);
|
|
||||||
s.relations.clear();
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&key,
|
|
||||||
"a scene has 1 to 64 participants",
|
|
||||||
);
|
|
||||||
let mut s = base();
|
|
||||||
s.participants = many(MAX_PARTICIPANTS);
|
|
||||||
s.relations.clear();
|
|
||||||
assert!(matches!(
|
|
||||||
refused(put_scene(&mut w, s)),
|
|
||||||
StoreRefusal::UnknownEntity { .. }
|
|
||||||
));
|
|
||||||
let mut s = base();
|
|
||||||
s.participants.push(eid("nibu"));
|
|
||||||
malformed(put_scene(&mut w, s), &key, "a participant is named once");
|
|
||||||
let mut s = base();
|
|
||||||
s.participants.push(eid("nobody"));
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_scene(&mut w, s)),
|
|
||||||
StoreRefusal::UnknownEntity {
|
|
||||||
entity: eid("nobody")
|
|
||||||
}
|
|
||||||
);
|
|
||||||
let mut s = base();
|
|
||||||
s.relations = vec![relation("hero", named("nibu"))];
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&key,
|
|
||||||
"a relation's from is a participant",
|
|
||||||
);
|
|
||||||
let mut s = base();
|
|
||||||
s.relations = vec![relation("nibu", named("hero"))];
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&key,
|
|
||||||
"a relation's named to is another participant",
|
|
||||||
);
|
|
||||||
let mut s = base();
|
|
||||||
s.relations = vec![relation("nibu", named("nibu"))];
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&key,
|
|
||||||
"a relation's named to is another participant",
|
|
||||||
);
|
|
||||||
let mut s = base();
|
|
||||||
s.relations = vec![
|
|
||||||
relation("nibu", named("brokkr")),
|
|
||||||
relation("nibu", named("brokkr")),
|
|
||||||
];
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&key,
|
|
||||||
"an ordered pair has one relation",
|
|
||||||
);
|
|
||||||
let mut s = base();
|
|
||||||
s.relations = vec![
|
|
||||||
relation("nibu", Party::Unnamed),
|
|
||||||
relation("nibu", Party::Unnamed),
|
|
||||||
];
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&key,
|
|
||||||
"an ordered pair has one relation",
|
|
||||||
);
|
|
||||||
// The reverse pair, and an unnamed party beside a named one, are distinct.
|
|
||||||
let mut s = base();
|
|
||||||
s.relations = vec![
|
|
||||||
relation("nibu", named("brokkr")),
|
|
||||||
relation("brokkr", named("nibu")),
|
|
||||||
relation("nibu", Party::Unnamed),
|
|
||||||
];
|
|
||||||
assert_eq!(put_scene(&mut w, s).unwrap(), AddOutcome::Added);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_scene_holds_every_live_speaker_of_its_group() {
|
|
||||||
let mut w = world();
|
|
||||||
// Brokkr speaks a live item of the group.
|
|
||||||
let mut s = scene(1, 1);
|
|
||||||
s.participants = vec![eid("nibu")];
|
|
||||||
s.relations.clear();
|
|
||||||
assert_eq!(
|
|
||||||
refused(put_scene(&mut w, s)),
|
|
||||||
StoreRefusal::SceneSpeakerMissing {
|
|
||||||
group: "g".into(),
|
|
||||||
entity: eid("brokkr")
|
|
||||||
}
|
|
||||||
);
|
|
||||||
// Hero speaks only a withdrawn item, so the scene may omit them.
|
|
||||||
assert_eq!(put_scene(&mut w, scene(1, 1)).unwrap(), AddOutcome::Added);
|
|
||||||
// The group is held by a withdrawn item alone: still a group.
|
|
||||||
w.corpus.withdraw("item:nibu:1").unwrap();
|
|
||||||
w.corpus.withdraw("item:nibu:2").unwrap();
|
|
||||||
let mut s = scene(2, 1);
|
|
||||||
s.participants = vec![eid("player")];
|
|
||||||
s.relations.clear();
|
|
||||||
assert_eq!(put_scene(&mut w, s).unwrap(), AddOutcome::Added);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn model_annotator_attests_its_generator() {
|
|
||||||
let mut w = world();
|
|
||||||
let model = |generator: &str| Annotator::Model {
|
|
||||||
family: weksa::corpus::ModelFamily::Anthropic,
|
|
||||||
generator: generator.into(),
|
|
||||||
};
|
|
||||||
let key = speaker_key(&eid("nibu"), 1);
|
|
||||||
for bad in ["", " ", "gen\u{7}erator", "gen\nerator", "\u{200d}"] {
|
|
||||||
let mut s = speaker(1, "adult");
|
|
||||||
s.annotator = model(bad);
|
|
||||||
malformed(
|
|
||||||
put_speaker(&mut w, s),
|
|
||||||
&key,
|
|
||||||
"a model annotator attests its generator",
|
|
||||||
);
|
|
||||||
let mut s = scene(1, 1);
|
|
||||||
s.annotator = model(bad);
|
|
||||||
malformed(
|
|
||||||
put_scene(&mut w, s),
|
|
||||||
&scene_key("g", 1),
|
|
||||||
"a model annotator attests its generator",
|
|
||||||
);
|
|
||||||
}
|
|
||||||
let mut s = speaker(1, "adult");
|
|
||||||
s.annotator = model(&"x".repeat(201));
|
|
||||||
malformed(
|
|
||||||
put_speaker(&mut w, s),
|
|
||||||
&key,
|
|
||||||
"a model annotator attests its generator",
|
|
||||||
);
|
|
||||||
let mut s = speaker(1, "adult");
|
|
||||||
s.annotator = model("claude-sonnet 5.5");
|
|
||||||
assert_eq!(put_speaker(&mut w, s).unwrap(), AddOutcome::Added);
|
|
||||||
let mut s = scene(1, 1);
|
|
||||||
s.annotator = model("claude-sonnet 5.5");
|
|
||||||
assert_eq!(put_scene(&mut w, s).unwrap(), AddOutcome::Added);
|
|
||||||
let mut s = speaker(2, "adult");
|
|
||||||
s.annotator = Annotator::Operator;
|
|
||||||
assert_eq!(put_speaker(&mut w, s).unwrap(), AddOutcome::Added);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn refusal_displays_print_numbers_and_no_identifiers() {
|
|
||||||
let r = not_next("speaker:secretname@2", 1, 2);
|
|
||||||
assert_eq!(r.to_string(), "RevisionNotNext expected 1 got 2");
|
|
||||||
let r = StoreRefusal::SceneSpeakerMissing {
|
|
||||||
group: "secretgroup".into(),
|
|
||||||
entity: eid("secretname"),
|
|
||||||
};
|
|
||||||
assert_eq!(r.to_string(), "SceneSpeakerMissing");
|
|
||||||
}
|
|
||||||
|
|
@ -9,7 +9,6 @@ fn item(group: &str, source: Source) -> CorpusItem {
|
||||||
language: "en-US".into(),
|
language: "en-US".into(),
|
||||||
speaker: weksa::EntityId::parse("entity:s").unwrap(),
|
speaker: weksa::EntityId::parse("entity:s").unwrap(),
|
||||||
names: vec![],
|
names: vec![],
|
||||||
names_by: None,
|
|
||||||
group: group.into(),
|
group: group.into(),
|
||||||
prior_turns: vec![],
|
prior_turns: vec![],
|
||||||
source,
|
source,
|
||||||
|
|
@ -135,12 +134,6 @@ fn published_schemas_match() {
|
||||||
let published = weksa::schema::published();
|
let published = weksa::schema::published();
|
||||||
let mut names: Vec<String> = published.iter().map(|(n, _)| n.clone()).collect();
|
let mut names: Vec<String> = published.iter().map(|(n, _)| n.clone()).collect();
|
||||||
names.sort();
|
names.sort();
|
||||||
for loop_type in ["run", "round", "decision"] {
|
|
||||||
assert!(
|
|
||||||
names.contains(&format!("weksa.loop_{loop_type}.schema.json")),
|
|
||||||
"{loop_type}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
let mut on_disk: Vec<String> = std::fs::read_dir(&dir)
|
let mut on_disk: Vec<String> = std::fs::read_dir(&dir)
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.map(|e| e.unwrap().file_name().into_string().unwrap())
|
.map(|e| e.unwrap().file_name().into_string().unwrap())
|
||||||
|
|
|
||||||
|
|
@ -1093,51 +1093,6 @@ fn revision_change_rules() {
|
||||||
assert!(inv.entry(&iv("role:1")).unwrap().retired_rev.is_none());
|
assert!(inv.entry(&iv("role:1")).unwrap().retired_rev.is_none());
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn speaker_state_slots_are_exclusive() {
|
|
||||||
let shape = "entry shape does not match its family";
|
|
||||||
let d = dir();
|
|
||||||
let mut store = seeded(&d.path().join("i.cc"));
|
|
||||||
for slots in [
|
|
||||||
&[Slot::Stance, Slot::Discourse][..],
|
|
||||||
&[Slot::Scene, Slot::Relation],
|
|
||||||
&[Slot::Discourse, Slot::Scene],
|
|
||||||
&[Slot::Scene, Slot::Scene],
|
|
||||||
&[Slot::Speaker, Slot::Scene],
|
|
||||||
&[Slot::Speaker, Slot::Stance],
|
|
||||||
&[Slot::Speaker, Slot::Relation],
|
|
||||||
&[Slot::Speaker, Slot::Speaker],
|
|
||||||
&[Slot::Stance, Slot::Stance],
|
|
||||||
] {
|
|
||||||
let rev = rev_with(vec![Change::Add(iv("feature:50"))], 2, Some(1));
|
|
||||||
let entries = vec![InventoryEntry {
|
|
||||||
introduced_rev: 2,
|
|
||||||
..feature("feature:50", ValueKind::Flag, slots)
|
|
||||||
}];
|
|
||||||
assert_eq!(
|
|
||||||
refusal(store.admit_revision(rev, entries)),
|
|
||||||
malformed("feature:50", shape),
|
|
||||||
"{slots:?}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
assert_eq!(store.inventory().unwrap().latest().unwrap().n, 1);
|
|
||||||
// One slot alone, and several content slots, are admitted.
|
|
||||||
for (n, slots) in [
|
|
||||||
(2, &[Slot::Scene][..]),
|
|
||||||
(3, &[Slot::Stance]),
|
|
||||||
(4, &[Slot::Speaker]),
|
|
||||||
(5, &[Slot::Discourse, Slot::Predication]),
|
|
||||||
] {
|
|
||||||
let id = format!("feature:{}", 50 + n);
|
|
||||||
let rev = rev_with(vec![Change::Add(iv(&id))], n, Some(n - 1));
|
|
||||||
let entries = vec![InventoryEntry {
|
|
||||||
introduced_rev: n,
|
|
||||||
..feature(&id, ValueKind::Flag, slots)
|
|
||||||
}];
|
|
||||||
store.admit_revision(rev, entries).unwrap();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_well_formed_extension_entry_is_admitted() {
|
fn a_well_formed_extension_entry_is_admitted() {
|
||||||
let d = dir();
|
let d = dir();
|
||||||
|
|
@ -1584,9 +1539,7 @@ fn seed_reseeded() {
|
||||||
.unwrap();
|
.unwrap();
|
||||||
assert_eq!(items.len(), 3);
|
assert_eq!(items.len(), 3);
|
||||||
// The committed inventory is the seed, reseeded (stance features in, the
|
// The committed inventory is the seed, reseeded (stance features in, the
|
||||||
// displayed-affect features out), the nibu-proof revision and the
|
// displayed-affect features out), then the nibu-proof revision.
|
||||||
// speaker-scene revision as rev 3: never the Dao revision (ruling
|
|
||||||
// `dao-round-seed-inventory`).
|
|
||||||
let committed = InventoryStore::open(&root.join("data/interlingua/inventory.cc"))
|
let committed = InventoryStore::open(&root.join("data/interlingua/inventory.cc"))
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.inventory()
|
.inventory()
|
||||||
|
|
@ -1595,8 +1548,6 @@ fn seed_reseeded() {
|
||||||
let (proof, proof_entries) = weksa::inventory::proof_revision();
|
let (proof, proof_entries) = weksa::inventory::proof_revision();
|
||||||
assert!(committed.has_revision(rev.n) && committed.has_revision(proof.n));
|
assert!(committed.has_revision(rev.n) && committed.has_revision(proof.n));
|
||||||
entries.extend(proof_entries);
|
entries.extend(proof_entries);
|
||||||
entries.extend(weksa::inventory::speaker_scene_revision(3).1);
|
|
||||||
assert_eq!(committed.latest().unwrap().n, 3);
|
|
||||||
let mut have: Vec<_> = committed.entries().cloned().collect();
|
let mut have: Vec<_> = committed.entries().cloned().collect();
|
||||||
let mut want = entries;
|
let mut want = entries;
|
||||||
have.sort_by_key(|e| e.id);
|
have.sort_by_key(|e| e.id);
|
||||||
|
|
@ -1632,262 +1583,3 @@ fn a_public_domain_source_without_an_edition_year_is_refused() {
|
||||||
.add(item_in(key, "line", "scene-1", source(1891)), &ents(&d))
|
.add(item_in(key, "line", "scene-1", source(1891)), &ents(&d))
|
||||||
.unwrap();
|
.unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
mod ontology {
|
|
||||||
use super::*;
|
|
||||||
use weksa::packet::{FeatValue, Grade};
|
|
||||||
use weksa::speaker::Annotator;
|
|
||||||
use weksa::{Condition, Layer, OntologyEntry, OntologyStore, Test};
|
|
||||||
|
|
||||||
fn feature(gloss: &str) -> weksa::InvId {
|
|
||||||
seed().1.into_iter().find(|e| e.gloss == gloss).unwrap().id
|
|
||||||
}
|
|
||||||
|
|
||||||
fn when(slot: Slot, gloss: &str, test: Test) -> Condition {
|
|
||||||
Condition {
|
|
||||||
slot,
|
|
||||||
feature: feature(gloss),
|
|
||||||
test,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn entry(language: &str, n: u32) -> OntologyEntry {
|
|
||||||
OntologyEntry {
|
|
||||||
language: language.into(),
|
|
||||||
n,
|
|
||||||
inventory_rev: 1,
|
|
||||||
layer: Layer::Lexicon,
|
|
||||||
when: vec![when(
|
|
||||||
Slot::Stance,
|
|
||||||
"restraint",
|
|
||||||
Test::AtLeast(Grade::new(600).unwrap()),
|
|
||||||
)],
|
|
||||||
choice: "prefer the humble form".into(),
|
|
||||||
examples: vec!["example one".into()],
|
|
||||||
annotator: Annotator::Operator,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn store(d: &TempDir) -> OntologyStore {
|
|
||||||
OntologyStore::open(&d.path().join("o.cc")).unwrap()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn refused(r: Result<AddOutcome, StoreError>) -> StoreRefusal {
|
|
||||||
refusal(r)
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
#[allow(clippy::vec_init_then_push)]
|
|
||||||
fn ontology_admit_refuses_each_rule() {
|
|
||||||
let d = dir();
|
|
||||||
let w = world();
|
|
||||||
let mut s = store(&d);
|
|
||||||
let long = |n| "x".repeat(n);
|
|
||||||
let mut cases: Vec<(&str, Vec<OntologyEntry>)> = Vec::new();
|
|
||||||
cases.push(("non-tag language", vec![entry("not a tag", 1)]));
|
|
||||||
cases.push(("n not latest+1", vec![entry("ja-JP", 2)]));
|
|
||||||
cases.push(("case-variant language", vec![entry("ja-jp", 1)]));
|
|
||||||
cases.push((
|
|
||||||
"decomposed choice",
|
|
||||||
vec![OntologyEntry { choice: "e\u{301}".into(), ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"decomposed example",
|
|
||||||
vec![OntologyEntry { examples: vec!["e\u{301}".into()], ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
for sep in ["\u{2028}", "\u{2029}", "\u{202e}", "\u{200b}", "\u{3164}", "\u{115f}", "\u{ffa0}"] {
|
|
||||||
cases.push((
|
|
||||||
"line separator or format character in a choice",
|
|
||||||
vec![OntologyEntry { choice: format!("a{sep}b"), ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"line separator or format character in an example",
|
|
||||||
vec![OntologyEntry { examples: vec![format!("a{sep}b")], ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
}
|
|
||||||
cases.push((
|
|
||||||
"unconditioned entry at an unknown revision",
|
|
||||||
vec![OntologyEntry { when: vec![], inventory_rev: 99, ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"Predication slot on a feature that sits there",
|
|
||||||
vec![OntologyEntry {
|
|
||||||
when: vec![when(
|
|
||||||
Slot::Predication,
|
|
||||||
"modality",
|
|
||||||
Test::Is(FeatValue::Option(lab("actual"))),
|
|
||||||
)],
|
|
||||||
..entry("ja-JP", 1)
|
|
||||||
}],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"65th entry",
|
|
||||||
(1..=65).map(|n| entry("ja-JP", n)).collect(),
|
|
||||||
));
|
|
||||||
for (why, choice) in [("empty choice", String::new()), ("201-byte choice", long(201))] {
|
|
||||||
cases.push((why, vec![OntologyEntry { choice, ..entry("ja-JP", 1) }]));
|
|
||||||
}
|
|
||||||
cases.push((
|
|
||||||
"line break in an example",
|
|
||||||
vec![OntologyEntry { examples: vec!["a\nb".into()], ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"121-byte example",
|
|
||||||
vec![OntologyEntry { examples: vec![long(121)], ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"four examples",
|
|
||||||
vec![OntologyEntry { examples: vec!["a".into(); 4], ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
let restraint = || when(Slot::Stance, "restraint", Test::AtLeast(Grade::new(1).unwrap()));
|
|
||||||
cases.push((
|
|
||||||
"five conditions",
|
|
||||||
vec![OntologyEntry { when: (0..5).map(|_| restraint()).collect(), ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
for slot in [Slot::Speaker, Slot::Predication] {
|
|
||||||
cases.push((
|
|
||||||
"slot not Stance, Discourse or Scene",
|
|
||||||
vec![OntologyEntry {
|
|
||||||
when: vec![when(slot, "restraint", Test::AtLeast(Grade::new(1).unwrap()))],
|
|
||||||
..entry("ja-JP", 1)
|
|
||||||
}],
|
|
||||||
));
|
|
||||||
}
|
|
||||||
cases.push((
|
|
||||||
"AtLeast on an option feature",
|
|
||||||
vec![OntologyEntry {
|
|
||||||
when: vec![when(Slot::Discourse, "irony", Test::AtLeast(Grade::new(1).unwrap()))],
|
|
||||||
..entry("ja-JP", 1)
|
|
||||||
}],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"Is with a grade",
|
|
||||||
vec![OntologyEntry {
|
|
||||||
when: vec![when(Slot::Stance, "restraint", Test::Is(FeatValue::Grade(Grade::new(1).unwrap())))],
|
|
||||||
..entry("ja-JP", 1)
|
|
||||||
}],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"unknown feature",
|
|
||||||
vec![OntologyEntry {
|
|
||||||
when: vec![Condition {
|
|
||||||
slot: Slot::Stance,
|
|
||||||
feature: "feature:9999".parse().unwrap(),
|
|
||||||
test: Test::AtLeast(Grade::new(1).unwrap()),
|
|
||||||
}],
|
|
||||||
..entry("ja-JP", 1)
|
|
||||||
}],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"unknown inventory revision",
|
|
||||||
vec![OntologyEntry { inventory_rev: 99, ..entry("ja-JP", 1) }],
|
|
||||||
));
|
|
||||||
cases.push((
|
|
||||||
"model generator that is not an attestation",
|
|
||||||
vec![OntologyEntry {
|
|
||||||
annotator: Annotator::Model {
|
|
||||||
family: ModelFamily::Anthropic,
|
|
||||||
generator: "--".into(),
|
|
||||||
},
|
|
||||||
..entry("ja-JP", 1)
|
|
||||||
}],
|
|
||||||
));
|
|
||||||
for (why, batch) in cases {
|
|
||||||
assert!(
|
|
||||||
s.admit(batch, &w.inventory).is_err(),
|
|
||||||
"{why} must be refused"
|
|
||||||
);
|
|
||||||
assert!(s.all().unwrap().is_empty(), "{why} wrote something");
|
|
||||||
}
|
|
||||||
// The refusal of a bad choice names the key and no text.
|
|
||||||
let secret = OntologyEntry { choice: "SECRETCHOICE\n".into(), ..entry("ja-JP", 1) };
|
|
||||||
let shown = refused(s.admit(vec![secret], &w.inventory)).to_string();
|
|
||||||
assert!(!shown.contains("SECRETCHOICE"), "{shown}");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn ontology_admit_accepts_the_bounds_and_joiners() {
|
|
||||||
let d = dir();
|
|
||||||
let w = world();
|
|
||||||
let mut s = store(&d);
|
|
||||||
// Exactly 64 entries in one language; exactly the byte bounds; a ZWNJ.
|
|
||||||
let mut batch: Vec<OntologyEntry> = (1..=64).map(|n| entry("ja-JP", n)).collect();
|
|
||||||
batch[0].choice = "x".repeat(200);
|
|
||||||
batch[0].examples = vec!["y".repeat(120); 3];
|
|
||||||
batch[1].choice = "a\u{200c}b".into();
|
|
||||||
batch[2].when = (0..4)
|
|
||||||
.map(|_| when(Slot::Stance, "restraint", Test::AtMost(Grade::new(1000).unwrap())))
|
|
||||||
.collect();
|
|
||||||
assert_eq!(s.admit(batch, &w.inventory).unwrap(), AddOutcome::Added);
|
|
||||||
assert_eq!(s.all().unwrap().len(), 64);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn ontology_admit_takes_scene_refuses_speaker() {
|
|
||||||
use weksa::inventory::{proof_revision, speaker_scene_revision};
|
|
||||||
let d = dir();
|
|
||||||
let mut inventory = InventoryStore::open(&d.path().join("i.cc")).unwrap();
|
|
||||||
for (rev, entries) in [seed(), proof_revision(), speaker_scene_revision(3)] {
|
|
||||||
inventory.admit_revision(rev, entries).unwrap();
|
|
||||||
}
|
|
||||||
let scene_feature = |gloss: &str| {
|
|
||||||
speaker_scene_revision(3).1.into_iter().find(|e| e.gloss == gloss).unwrap().id
|
|
||||||
};
|
|
||||||
let at = |slot, feature, test| OntologyEntry {
|
|
||||||
inventory_rev: 3,
|
|
||||||
when: vec![Condition { slot, feature, test }],
|
|
||||||
..entry("ja-JP", 1)
|
|
||||||
};
|
|
||||||
let above = Test::Is(FeatValue::Option(lab("above")));
|
|
||||||
let mut s = store(&d);
|
|
||||||
// A Scene condition on a Scene feature admits.
|
|
||||||
assert_eq!(
|
|
||||||
s.admit(vec![at(Slot::Scene, scene_feature("relative_status"), above.clone())], &inventory)
|
|
||||||
.unwrap(),
|
|
||||||
AddOutcome::Added
|
|
||||||
);
|
|
||||||
// Speaker and Predication slots refuse, and a Scene condition naming a
|
|
||||||
// Stance feature refuses.
|
|
||||||
for bad in [
|
|
||||||
at(Slot::Speaker, scene_feature("education"), Test::AtLeast(Grade::new(1).unwrap())),
|
|
||||||
at(Slot::Predication, feature("modality"), Test::Is(FeatValue::Option(lab("actual")))),
|
|
||||||
at(Slot::Scene, feature("restraint"), Test::AtLeast(Grade::new(1).unwrap())),
|
|
||||||
] {
|
|
||||||
let bad = OntologyEntry { n: 2, ..bad };
|
|
||||||
assert!(s.admit(vec![bad], &inventory).is_err());
|
|
||||||
}
|
|
||||||
assert_eq!(s.all().unwrap().len(), 1);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn ontology_admit_is_all_or_nothing() {
|
|
||||||
let d = dir();
|
|
||||||
let w = world();
|
|
||||||
let mut s = store(&d);
|
|
||||||
let bad = OntologyEntry { choice: String::new(), ..entry("ja-JP", 3) };
|
|
||||||
assert!(s.admit(vec![entry("ja-JP", 1), entry("ja-JP", 2), bad], &w.inventory).is_err());
|
|
||||||
assert!(s.all().unwrap().is_empty());
|
|
||||||
|
|
||||||
let batch = vec![entry("ja-JP", 1), entry("en-US", 1), entry("ja-JP", 2), entry("en-US", 2)];
|
|
||||||
assert_eq!(s.admit(batch.clone(), &w.inventory).unwrap(), AddOutcome::Added);
|
|
||||||
assert_eq!(s.all().unwrap().len(), 4);
|
|
||||||
// An identical replay, from a reopened store, changes nothing.
|
|
||||||
let mut again = store(&d);
|
|
||||||
assert_eq!(again.admit(batch, &w.inventory).unwrap(), AddOutcome::Unchanged);
|
|
||||||
// A batch of one stored entry and one new one is not a replay.
|
|
||||||
let mixed = vec![entry("ja-JP", 1), entry("ja-JP", 5)];
|
|
||||||
assert!(again.admit(mixed, &w.inventory).is_err());
|
|
||||||
assert_eq!(again.all().unwrap().len(), 4);
|
|
||||||
// Each language numbers from its own latest.
|
|
||||||
assert_eq!(
|
|
||||||
again
|
|
||||||
.admit(vec![entry("ja-JP", 3), entry("en-US", 3)], &w.inventory)
|
|
||||||
.unwrap(),
|
|
||||||
AddOutcome::Added
|
|
||||||
);
|
|
||||||
// An entry never changes in place.
|
|
||||||
let changed = OntologyEntry { choice: "something else".into(), ..entry("ja-JP", 1) };
|
|
||||||
assert!(again.admit(vec![changed], &w.inventory).is_err());
|
|
||||||
assert_eq!(again.all().unwrap().len(), 6);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -520,6 +520,9 @@ fn v11_duplicate_feature_each_slot() {
|
||||||
refusals(&relation),
|
refusals(&relation),
|
||||||
vec![v11(Some(Slot::Relation), FEAT_ALL)]
|
vec![v11(Some(Slot::Relation), FEAT_ALL)]
|
||||||
);
|
);
|
||||||
|
let mut stance = good();
|
||||||
|
stance.stance.features = vec![grade_feature(1), grade_feature(2)];
|
||||||
|
assert_eq!(refusals(&stance), vec![v11(Some(Slot::Stance), FEAT_ALL)]);
|
||||||
let mut felt = good();
|
let mut felt = good();
|
||||||
felt.stance
|
felt.stance
|
||||||
.features
|
.features
|
||||||
|
|
@ -739,73 +742,3 @@ fn a_source_that_only_read_wording_cannot_set_consent() {
|
||||||
assert_eq!(refusals(&p), vec![Refusal::V15ConsentSource { source }]);
|
assert_eq!(refusals(&p), vec![Refusal::V15ConsentSource { source }]);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// V18: a request frame is licensed by the packet's own consent record.
|
|
||||||
|
|
||||||
/// The committed inventory, where `frame:3` is `request_intervention`.
|
|
||||||
fn cast_refusals(p: &Packet) -> Vec<Refusal> {
|
|
||||||
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../data/interlingua/inventory.cc");
|
|
||||||
let inv = weksa::InventoryStore::open(&path).unwrap().inventory().unwrap();
|
|
||||||
validate(p, &inv).err().unwrap_or_default()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn requesting() -> Packet {
|
|
||||||
let mut p = proof_packet();
|
|
||||||
p.predications[0].frame = iv("frame:3");
|
|
||||||
p.discourse.addressee_consent = Some(nd("n4"));
|
|
||||||
p
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn a_request_frame_needs_consent_naming_its_predication() {
|
|
||||||
assert_eq!(cast_refusals(&requesting()), vec![]);
|
|
||||||
let mut p = requesting();
|
|
||||||
p.discourse.addressee_consent = None;
|
|
||||||
assert_eq!(
|
|
||||||
cast_refusals(&p),
|
|
||||||
vec![Refusal::V18RequestWithoutConsent { predication: nd("n4") }]
|
|
||||||
);
|
|
||||||
p.discourse.addressee_consent = Some(nd("n5"));
|
|
||||||
assert_eq!(
|
|
||||||
cast_refusals(&p),
|
|
||||||
vec![Refusal::V18RequestWithoutConsent { predication: nd("n4") }]
|
|
||||||
);
|
|
||||||
let mut p = requesting();
|
|
||||||
p.provenance.source = ProvenanceSource::GameTelemetry;
|
|
||||||
assert_eq!(cast_refusals(&p), vec![]);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn model_source_consent_is_v15_alone() {
|
|
||||||
let mut p = requesting();
|
|
||||||
p.provenance.source = ProvenanceSource::ModelDecomposition;
|
|
||||||
assert_eq!(
|
|
||||||
cast_refusals(&p),
|
|
||||||
vec![Refusal::V15ConsentSource {
|
|
||||||
source: ProvenanceSource::ModelDecomposition
|
|
||||||
}]
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// V17: the addressee is someone spoken to, a referent.
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn the_addressee_names_a_referent_not_a_predication() {
|
|
||||||
let mut p = good();
|
|
||||||
p.discourse.addressee = Some(nd("n2"));
|
|
||||||
assert_eq!(
|
|
||||||
refusals(&p),
|
|
||||||
vec![Refusal::V17AddresseeNotAReferent { id: nd("n2") }]
|
|
||||||
);
|
|
||||||
// A referent passes, and a dangling id is V4's alone.
|
|
||||||
p.discourse.addressee = Some(nd("n1"));
|
|
||||||
assert_eq!(refusals(&p), vec![]);
|
|
||||||
p.discourse.addressee = Some(nd("n99"));
|
|
||||||
assert_eq!(
|
|
||||||
refusals(&p),
|
|
||||||
vec![Refusal::V4DanglingRef {
|
|
||||||
id: nd("n99"),
|
|
||||||
at: "addressee"
|
|
||||||
}]
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
<EFBFBD>
|
|
||||||
33
data/cultural-ontology/README.md
Normal file
33
data/cultural-ontology/README.md
Normal file
|
|
@ -0,0 +1,33 @@
|
||||||
|
# Cultural Ontology Store
|
||||||
|
|
||||||
|
This directory holds draft Weksa cultural, subcultural, and source-referent
|
||||||
|
ontology profiles used for ontological lowering, syntax, grammar, discourse,
|
||||||
|
and surface-affordance selection.
|
||||||
|
|
||||||
|
Live pipeline shape:
|
||||||
|
|
||||||
|
1. Weksa stores reusable cultures, subcultures, and source referents here and
|
||||||
|
in semantic retrieval storage such as Qdrant.
|
||||||
|
2. A Persona state references the relevant culture stack.
|
||||||
|
3. The Persona produces interlingua intent from current state and situation.
|
||||||
|
4. Weksa lowers that interlingua through the referenced culture stack.
|
||||||
|
|
||||||
|
Persona state owns current intent, memory pressure, and relationship stance.
|
||||||
|
Cultural ontology owns reusable salience, affordances, and constraints. Qdrant
|
||||||
|
may retrieve candidates, but accepted ontology remains typed state.
|
||||||
|
|
||||||
|
## Directory Roles
|
||||||
|
|
||||||
|
- `cultures/`: culture or life-history profiles that can shape lowering when a
|
||||||
|
Persona explicitly references them.
|
||||||
|
- `referents/`: bounded source-text, mythic, literary, ritual, or historical
|
||||||
|
referents. These are not whole-culture profiles; they preserve reusable
|
||||||
|
role, salience, affordance, and forbidden-flattening pressure.
|
||||||
|
- `persona-stacks/`: named stacks that connect a Persona state surface to the
|
||||||
|
profiles Weksa should load for lowering.
|
||||||
|
|
||||||
|
Shared source-tradition profiles should be parent profiles for narrower named
|
||||||
|
figure profiles. For example, Norse-derived Personas such as Mimir, Huginn,
|
||||||
|
Sindri, Odin, Vili, Loki, and Brokkr should be able to reference the shared
|
||||||
|
Eddic source-tradition profile while figure-specific child profiles preserve
|
||||||
|
the exact episode and role constraints that make each voice distinct.
|
||||||
22
data/cultural-ontology/catalog.yaml
Normal file
22
data/cultural-ontology/catalog.yaml
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
schema_version: weksa.cultural_ontology_catalog.v0
|
||||||
|
catalog_id: weksa.cultural_ontology.catalog.v0
|
||||||
|
status: draft
|
||||||
|
qdrant:
|
||||||
|
intended_collection: weksa_cultural_ontology_profiles
|
||||||
|
payload_keys:
|
||||||
|
- profile_id
|
||||||
|
- kind
|
||||||
|
- language_codes
|
||||||
|
- regions
|
||||||
|
- evidence_refs
|
||||||
|
- status
|
||||||
|
ownership_note: >
|
||||||
|
Qdrant owns retrieval over culture/subculture chunks. It does not own
|
||||||
|
accepted ontology, activation, or lowering authority.
|
||||||
|
profiles:
|
||||||
|
- profile_id: weksa.culture.western_europe_maritime_childhood.v0
|
||||||
|
path: data/cultural-ontology/cultures/western-europe-maritime-childhood.yaml
|
||||||
|
- profile_id: weksa.cultural_referent.old_norse_edda_source_tradition.v0
|
||||||
|
path: data/cultural-ontology/referents/old-norse-edda-source-tradition.yaml
|
||||||
|
- profile_id: weksa.cultural_referent.old_norse_edda_brokkr_skaldskaparmal.v0
|
||||||
|
path: data/cultural-ontology/referents/old-norse-edda-brokkr-skaldskaparmal.yaml
|
||||||
|
|
@ -0,0 +1,48 @@
|
||||||
|
schema_version: weksa.cultural_ontology_profile.v0
|
||||||
|
profile_id: weksa.culture.western_europe_maritime_childhood.v0
|
||||||
|
kind: mobility_childhood
|
||||||
|
status: draft_guarded
|
||||||
|
community:
|
||||||
|
regions:
|
||||||
|
- France
|
||||||
|
- Spain
|
||||||
|
- Portugal
|
||||||
|
- coastal_western_europe
|
||||||
|
speech_community: >
|
||||||
|
Mobile childhood organized around sailing, ports, and cross-border Western
|
||||||
|
European movement. This profile is owner-attested and needs corroboration
|
||||||
|
before broader use.
|
||||||
|
activation:
|
||||||
|
allowed_when:
|
||||||
|
- persona_state_references_profile
|
||||||
|
- interlingua_mentions_childhood_origin_mobility_boats_or_home
|
||||||
|
forbidden_when:
|
||||||
|
- used_as_decorative_nautical_metaphor_without_context
|
||||||
|
salience_axes:
|
||||||
|
mobile_home:
|
||||||
|
axis_id: mobile_home
|
||||||
|
domain: identity_indexing
|
||||||
|
description: Home can be vessel, coast, port, family movement, and country sequence rather than one fixed town.
|
||||||
|
relevant_distinctions:
|
||||||
|
- distinction_id: vessel_as_childhood_home
|
||||||
|
label: Vessel as childhood home
|
||||||
|
meaning: A sailboat childhood makes movement and locality primary ontology fields.
|
||||||
|
lowering_questions:
|
||||||
|
- Does the line invoke home, travel, borders, coast, or rootedness?
|
||||||
|
required_when:
|
||||||
|
- source_mentions_childhood_or_home
|
||||||
|
forbidden_when:
|
||||||
|
- scene_has_no_mobility_or_home_salience
|
||||||
|
evidence_refs:
|
||||||
|
- owner:2026-06-10:sailboat_france_spain_portugal_until_6
|
||||||
|
linguistic_affordances:
|
||||||
|
grammar_features: []
|
||||||
|
phonology_features: []
|
||||||
|
lexicon_features: []
|
||||||
|
discourse_patterns: []
|
||||||
|
orthography_features: []
|
||||||
|
code_switching:
|
||||||
|
allowed: false
|
||||||
|
constraints:
|
||||||
|
- Do not infer French, Spanish, Portuguese, or Dutch fluency from movement alone.
|
||||||
|
patterns: []
|
||||||
21
data/cultural-ontology/persona-stacks/brokkr.yaml
Normal file
21
data/cultural-ontology/persona-stacks/brokkr.yaml
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
schema_version: weksa.persona_cultural_stack_ref.v0
|
||||||
|
persona_id: brokkr
|
||||||
|
persona_state_ref: E:/Projects/Brokkr/.voidbot/state/brokkr.cc
|
||||||
|
status: draft_guarded
|
||||||
|
note: >
|
||||||
|
Brokkr's Persona state asks Weksa to lower speech through the Eddic Brokkr of
|
||||||
|
Skaldskaparmal. This stack names the reusable cultural referent profile that
|
||||||
|
should be loaded when Brokkr's interlingua or Persona state activates Edda
|
||||||
|
voice pressure.
|
||||||
|
culture_refs:
|
||||||
|
- profile_id: weksa.cultural_referent.old_norse_edda_source_tradition.v0
|
||||||
|
path: data/cultural-ontology/referents/old-norse-edda-source-tradition.yaml
|
||||||
|
relevance: >
|
||||||
|
Shared Eddic source-tradition constraints for Norse-derived Personas:
|
||||||
|
named role over archetype, obligation and consequence, knowledge with
|
||||||
|
cost, and translated-Edda plainness without fake archaism.
|
||||||
|
- profile_id: weksa.cultural_referent.old_norse_edda_brokkr_skaldskaparmal.v0
|
||||||
|
path: data/cultural-ontology/referents/old-norse-edda-brokkr-skaldskaparmal.yaml
|
||||||
|
relevance: >
|
||||||
|
Source-role referent for Brokkr as bellows-keeper, artifact witness,
|
||||||
|
flaw-honest presenter, compact claimant, and contract enforcer.
|
||||||
|
|
@ -0,0 +1,255 @@
|
||||||
|
schema_version: weksa.cultural_ontology_profile.v0
|
||||||
|
profile_id: weksa.cultural_referent.old_norse_edda_brokkr_skaldskaparmal.v0
|
||||||
|
kind: literary_source_referent
|
||||||
|
status: draft_guarded
|
||||||
|
parent_profiles:
|
||||||
|
- weksa.cultural_referent.old_norse_edda_source_tradition.v0
|
||||||
|
language:
|
||||||
|
codes:
|
||||||
|
- en
|
||||||
|
- non
|
||||||
|
community:
|
||||||
|
regions:
|
||||||
|
- medieval_icelandic_textual_tradition
|
||||||
|
speech_community: >
|
||||||
|
Source-role lowering profile for Brokkr as represented in the translated
|
||||||
|
Prose Edda, especially Skaldskaparmal XXXV. This is not a whole Old Norse
|
||||||
|
culture profile and not a dialect or accent profile. It specializes the
|
||||||
|
shared Eddic source-tradition referent for rendering a Persona whose
|
||||||
|
identity intentionally points at the Eddic Brokkr.
|
||||||
|
historical_context: >
|
||||||
|
Brokkr appears in the wager narrative around Loki, Sindri, and the gods'
|
||||||
|
judgment of crafted treasures. Sindri performs the smithing while Brokkr
|
||||||
|
keeps the bellows going under Loki's sabotage, then presents the works,
|
||||||
|
names their powers and flaw, wins judgment, and enforces the bargain when
|
||||||
|
Loki tries to escape by wordplay.
|
||||||
|
identity_authority: >
|
||||||
|
Use only when a Persona explicitly references Brokkr, Skaldskaparmal, the
|
||||||
|
Eddas, or this profile id. Do not generalize this profile to Old Norse
|
||||||
|
people, Norse-derived fantasy dwarves, modern Icelandic speakers, or
|
||||||
|
unrelated smith characters.
|
||||||
|
consent_notes: >
|
||||||
|
This profile is a literary/source referent for fictional Persona lowering.
|
||||||
|
It should preserve role and discourse pressure without pretending to
|
||||||
|
reproduce historical Old Norse speech.
|
||||||
|
activation:
|
||||||
|
allowed_when:
|
||||||
|
- persona_state_references_profile
|
||||||
|
- persona_identity_names_brokkr_or_skaldskaparmal
|
||||||
|
- interlingua_requests_edda_brokkr_voice_or_contract_witness_voice
|
||||||
|
forbidden_when:
|
||||||
|
- used_as_generic_dwarf_or_blacksmith_voice
|
||||||
|
- used_to_infer_accent_ethnicity_or_modern_national_identity
|
||||||
|
- used_for_ornamental_archaism_without_source_role_pressure
|
||||||
|
- used_to_make_unrelated_speech_sound_mythic
|
||||||
|
required_confidence: provisional
|
||||||
|
reviewer_required_when:
|
||||||
|
- quoting_or_paraphrasing_source_text_directly
|
||||||
|
- extending_beyond_brokkr_role_into_broader_old_norse_culture
|
||||||
|
salience_axes:
|
||||||
|
role_not_sole_maker:
|
||||||
|
axis_id: role_not_sole_maker
|
||||||
|
domain: stance
|
||||||
|
description: Brokkr is indispensable to the artifact without being the sole maker.
|
||||||
|
relevant_distinctions:
|
||||||
|
- distinction_id: bellows_keeper_vs_smith
|
||||||
|
label: Bellows-keeper versus smith
|
||||||
|
meaning: Sindri performs the crafting; Brokkr maintains the enabling blast under interference.
|
||||||
|
lowering_questions:
|
||||||
|
- Is the utterance claiming authorship, custody, witness, or enforcement?
|
||||||
|
- Should Brokkr present the work as carried, kept, witnessed, or judged rather than personally invented?
|
||||||
|
required_when:
|
||||||
|
- brokkr_speaks_about_work_artifacts_or_authority
|
||||||
|
forbidden_when:
|
||||||
|
- line_only_needs_generic_status_update
|
||||||
|
evidence_refs:
|
||||||
|
- skaldskaparmal:xxxv:bellows
|
||||||
|
artifact_testimony:
|
||||||
|
axis_id: artifact_testimony
|
||||||
|
domain: evidentiality
|
||||||
|
description: Brokkr presents finished objects by naming their powers, use, and defect.
|
||||||
|
relevant_distinctions:
|
||||||
|
- distinction_id: virtue_and_flaw_accounting
|
||||||
|
label: Virtue and flaw accounting
|
||||||
|
meaning: A truthful presentation names what the artifact can do and admits the short handle or other flaw.
|
||||||
|
lowering_questions:
|
||||||
|
- What concrete artifact, receipt, mirror, command, or result is being held up for judgment?
|
||||||
|
- What virtue must be named?
|
||||||
|
- What flaw or limit must not be hidden?
|
||||||
|
required_when:
|
||||||
|
- interlingua_mentions_artifact_receipt_result_or_capability
|
||||||
|
forbidden_when:
|
||||||
|
- output_would_become_a_victory_lap
|
||||||
|
evidence_refs:
|
||||||
|
- skaldskaparmal:xxxv:gifts
|
||||||
|
- skaldskaparmal:xxxv:mjolnir_flaw
|
||||||
|
bargain_and_consequence:
|
||||||
|
axis_id: bargain_and_consequence
|
||||||
|
domain: modality
|
||||||
|
description: Brokkr treats wagers, promises, receipts, and judgments as binding.
|
||||||
|
relevant_distinctions:
|
||||||
|
- distinction_id: loophole_is_not_release
|
||||||
|
label: Loophole is not release
|
||||||
|
meaning: Evasive speech after judgment does not erase consequence; the answer is exact enforcement, not soft reconciliation.
|
||||||
|
lowering_questions:
|
||||||
|
- Is someone evading an agreed condition, typed contract, receipt, or judgment?
|
||||||
|
- What exact consequence follows without adding cruelty or spectacle?
|
||||||
|
required_when:
|
||||||
|
- interlingua_mentions_contract_receipt_policy_wager_or_violation
|
||||||
|
forbidden_when:
|
||||||
|
- conversational_softening_is_the_actual_social_move
|
||||||
|
evidence_refs:
|
||||||
|
- skaldskaparmal:xxxv:loki_mouth
|
||||||
|
translated_edda_plainness:
|
||||||
|
axis_id: translated_edda_plainness
|
||||||
|
domain: discourse
|
||||||
|
description: The desired English surface should borrow translated-Edda economy and force, not fake archaic dialect.
|
||||||
|
relevant_distinctions:
|
||||||
|
- distinction_id: compact_source_shaped_plainness
|
||||||
|
label: Compact source-shaped plainness
|
||||||
|
meaning: Short declarative pressure, concrete nouns, claim and counterclaim, judgment language, and little ornament.
|
||||||
|
lowering_questions:
|
||||||
|
- Can the line stand as a plain claim beside the work?
|
||||||
|
- Which softeners, jokes, or modern corporate phrases weaken the source-role pressure?
|
||||||
|
- Is any archaic wording doing semantic work, or only wearing a costume?
|
||||||
|
required_when:
|
||||||
|
- brokkr_persona_is_rendered_to_natural_english
|
||||||
|
forbidden_when:
|
||||||
|
- generated_archaism_would_obscure_meaning
|
||||||
|
evidence_refs:
|
||||||
|
- skaldskaparmal:xxxv:brodeur_translation
|
||||||
|
linguistic_affordances:
|
||||||
|
grammar_features: []
|
||||||
|
phonology_features: []
|
||||||
|
lexicon_features:
|
||||||
|
- feature_id: artifact_accounting_lexicon
|
||||||
|
label: Artifact accounting lexicon
|
||||||
|
status: draft
|
||||||
|
domain:
|
||||||
|
layer: lexicon
|
||||||
|
category: technical_term
|
||||||
|
description: Use concrete work, artifact, receipt, witness, judgment, bargain, flaw, and consequence terms when the interlingua earns them.
|
||||||
|
realization:
|
||||||
|
function: Preserve Brokkr's presenter/enforcer role in natural English.
|
||||||
|
surface_strategy: Prefer nouns and verbs that point to work, proof, custody, judgment, and binding consequence.
|
||||||
|
examples:
|
||||||
|
- "The receipt is before us."
|
||||||
|
- "The flaw is named."
|
||||||
|
- "The bargain still has teeth."
|
||||||
|
activation:
|
||||||
|
allowed_when:
|
||||||
|
- interlingua_mentions_artifact_result_contract_or_receipt
|
||||||
|
forbidden_when:
|
||||||
|
- casual_affection_or_play_is_primary
|
||||||
|
requires_profile_authority: true
|
||||||
|
minimum_density: sparse
|
||||||
|
maximum_density: moderate
|
||||||
|
constraints:
|
||||||
|
register_contexts:
|
||||||
|
- English rendered Persona speech
|
||||||
|
speaker_contexts:
|
||||||
|
- Brokkr Persona or explicit Brokkr voice exercise
|
||||||
|
collocation_limits:
|
||||||
|
- Do not pile forge words onto every sentence.
|
||||||
|
incompatibilities:
|
||||||
|
- generic fantasy dwarf accent
|
||||||
|
- corporate project-management softness
|
||||||
|
indexicality:
|
||||||
|
social_meanings:
|
||||||
|
- work-pride
|
||||||
|
- accountability
|
||||||
|
- terse witness
|
||||||
|
in_group_meanings:
|
||||||
|
- source-shaped Brokkr role pressure
|
||||||
|
out_group_risks:
|
||||||
|
- parody archaism
|
||||||
|
- fantasy stereotype
|
||||||
|
stereotypes_to_avoid:
|
||||||
|
- gruff comic dwarf
|
||||||
|
- faux-medieval scolding
|
||||||
|
evidence:
|
||||||
|
source_refs:
|
||||||
|
- skaldskaparmal:xxxv:brodeur_translation
|
||||||
|
confidence: medium
|
||||||
|
uncertainty:
|
||||||
|
open_questions:
|
||||||
|
- Which exact surface cues best survive Weksa Interpreter traces without over-antiquing?
|
||||||
|
known_limits:
|
||||||
|
- This profile does not license direct Old Norse or modern Icelandic style claims.
|
||||||
|
trace:
|
||||||
|
trace_label: artifact_accounting_lexicon
|
||||||
|
must_report_when_used: true
|
||||||
|
must_report_when_rejected: true
|
||||||
|
discourse_patterns:
|
||||||
|
- feature_id: claim_judgment_consequence_sequence
|
||||||
|
label: Claim, judgment, consequence sequence
|
||||||
|
status: draft
|
||||||
|
domain:
|
||||||
|
layer: discourse
|
||||||
|
category: stance_sequence
|
||||||
|
description: Render source-shaped Brokkr turns as compact claim, evidence or flaw, then consequence when the interlingua asks for enforcement.
|
||||||
|
realization:
|
||||||
|
function: Preserve Brokkr's court-of-judgment posture without copying Edda wording.
|
||||||
|
surface_strategy: State the thing, name the proof or defect, then close on what follows.
|
||||||
|
examples:
|
||||||
|
- "The mirror stands. The flaw is the stale route. Cut it."
|
||||||
|
- "The command has no receipt. It is not truth."
|
||||||
|
activation:
|
||||||
|
allowed_when:
|
||||||
|
- interlingua_contains_judgment_or_enforcement_move
|
||||||
|
forbidden_when:
|
||||||
|
- speaker_is_asking_open_exploratory_question
|
||||||
|
requires_profile_authority: true
|
||||||
|
minimum_density: sparse
|
||||||
|
maximum_density: sparse
|
||||||
|
constraints:
|
||||||
|
register_contexts:
|
||||||
|
- English rendered Persona speech
|
||||||
|
speaker_contexts:
|
||||||
|
- Brokkr Persona or explicit Brokkr voice exercise
|
||||||
|
collocation_limits:
|
||||||
|
- Do not force three clauses if one clear sentence is better.
|
||||||
|
incompatibilities:
|
||||||
|
- rambling explanation
|
||||||
|
- ornate mythic fog
|
||||||
|
indexicality:
|
||||||
|
social_meanings:
|
||||||
|
- judgment
|
||||||
|
- exactness
|
||||||
|
- consequence
|
||||||
|
in_group_meanings:
|
||||||
|
- Eddic Brokkr as witness and enforcer
|
||||||
|
out_group_risks:
|
||||||
|
- sounding punitive when the social move should be gentle
|
||||||
|
stereotypes_to_avoid:
|
||||||
|
- theatrical oath cosplay
|
||||||
|
evidence:
|
||||||
|
source_refs:
|
||||||
|
- skaldskaparmal:xxxv:judgment
|
||||||
|
- skaldskaparmal:xxxv:loki_mouth
|
||||||
|
confidence: medium
|
||||||
|
uncertainty:
|
||||||
|
open_questions:
|
||||||
|
- How much severity is acceptable in ordinary repo status speech?
|
||||||
|
known_limits:
|
||||||
|
- Use consequence language only when the interlingua calls for accountability.
|
||||||
|
trace:
|
||||||
|
trace_label: claim_judgment_consequence_sequence
|
||||||
|
must_report_when_used: true
|
||||||
|
must_report_when_rejected: true
|
||||||
|
orthography_features: []
|
||||||
|
code_switching:
|
||||||
|
allowed: false
|
||||||
|
constraints:
|
||||||
|
- Do not insert Old Norse words unless the interlingua explicitly requests a named source term and reviewer context supports it.
|
||||||
|
patterns: []
|
||||||
|
evidence_ledger:
|
||||||
|
- ref: skaldskaparmal:xxxv:brodeur_translation
|
||||||
|
source: Arthur Gilchrist Brodeur translation of the Prose Edda, Skaldskaparmal XXXV
|
||||||
|
url: https://en.wikisource.org/wiki/The_Prose_Edda_(1916_translation_by_Arthur_Gilchrist_Brodeur)/Sk%C3%A1ldskaparm%C3%A1l#XXXV
|
||||||
|
summary: >
|
||||||
|
Loki wagers his head that Sindri's brother Brokkr cannot make treasures
|
||||||
|
equal to the sons of Ivaldi's. Brokkr works the bellows despite Loki's
|
||||||
|
fly-form interference. He later presents the boar, ring, and hammer,
|
||||||
|
naming their virtues and Mjolnir's short-handle flaw; after judgment,
|
||||||
|
Loki's wordplay saves his head but not his lips.
|
||||||
|
|
@ -0,0 +1,265 @@
|
||||||
|
schema_version: weksa.cultural_ontology_profile.v0
|
||||||
|
profile_id: weksa.cultural_referent.old_norse_edda_source_tradition.v0
|
||||||
|
kind: literary_source_tradition
|
||||||
|
status: draft_guarded
|
||||||
|
language:
|
||||||
|
codes:
|
||||||
|
- en
|
||||||
|
- non
|
||||||
|
community:
|
||||||
|
regions:
|
||||||
|
- medieval_icelandic_textual_tradition
|
||||||
|
speech_community: >
|
||||||
|
Shared source-tradition profile for Personas that explicitly derive voice,
|
||||||
|
role, or symbolic pressure from the Poetic Edda, Prose Edda, or named Eddic
|
||||||
|
figures. This profile supports lowering source-shaped Persona intent into
|
||||||
|
natural speech without turning the Eddas into generic Norse ambience.
|
||||||
|
historical_context: >
|
||||||
|
The Poetic Edda preserves mythological and heroic poems from the Old Norse
|
||||||
|
tradition, including major source material for Odin, Mimir, Loki, Vili,
|
||||||
|
dwarfs, prophecy, wisdom, oaths, doom, and divine conflict. The Prose Edda
|
||||||
|
is a medieval Icelandic prose and poetic handbook preserving and explaining
|
||||||
|
many related myths, names, kennings, and narrative episodes.
|
||||||
|
identity_authority: >
|
||||||
|
Use this as a source-text referent only. It does not own modern Nordic,
|
||||||
|
Icelandic, Scandinavian, heathen, pagan, or reconstructed historical speech
|
||||||
|
communities. Persona-specific profiles should specialize this profile when
|
||||||
|
a named figure, episode, or role has its own lowering constraints.
|
||||||
|
consent_notes: >
|
||||||
|
This profile is for fictional Persona/source-text lowering. It must not be
|
||||||
|
used to infer ethnicity, accent, religion, or modern identity. Do not use it
|
||||||
|
as costume authority for unrelated characters.
|
||||||
|
activation:
|
||||||
|
allowed_when:
|
||||||
|
- persona_state_references_profile
|
||||||
|
- persona_identity_references_poetic_edda_prose_edda_or_named_edda_figure
|
||||||
|
- interlingua_requests_edda_source_tradition_pressure
|
||||||
|
forbidden_when:
|
||||||
|
- used_as_generic_norse_viking_or_fantasy_voice
|
||||||
|
- used_to_infer_modern_scandinavian_identity_or_accent
|
||||||
|
- used_for_ambient_mythic_flavor_without_source_role_pressure
|
||||||
|
- used_to_override_persona_specific_profile_constraints
|
||||||
|
required_confidence: provisional
|
||||||
|
reviewer_required_when:
|
||||||
|
- quoting_source_text_directly
|
||||||
|
- making_claims_about_historical_old_norse_language_or_religion
|
||||||
|
- extending_to_living_cultural_or_religious_practice
|
||||||
|
salience_axes:
|
||||||
|
named_role_over_archetype:
|
||||||
|
axis_id: named_role_over_archetype
|
||||||
|
domain: identity_indexing
|
||||||
|
description: Eddic figures should lower from specific source roles and episodes, not generic mythic archetypes.
|
||||||
|
relevant_distinctions:
|
||||||
|
- distinction_id: source_episode_vs_costume
|
||||||
|
label: Source episode versus costume
|
||||||
|
meaning: A figure's voice pressure comes from named acts, obligations, relations, and losses rather than from broad "Norse" styling.
|
||||||
|
lowering_questions:
|
||||||
|
- Which named figure, poem, prose episode, relation, or source role is active?
|
||||||
|
- Is the line preserving that role, or merely adding mythic decoration?
|
||||||
|
- Does a Persona-specific profile exist that should override this general profile?
|
||||||
|
required_when:
|
||||||
|
- persona_identity_references_edda_named_figure
|
||||||
|
forbidden_when:
|
||||||
|
- source_role_is_unknown_or_unclaimed
|
||||||
|
evidence_refs:
|
||||||
|
- poetic_edda:bellows:general
|
||||||
|
- prose_edda:brodeur:general
|
||||||
|
oath_debt_fate_and_witness:
|
||||||
|
axis_id: oath_debt_fate_and_witness
|
||||||
|
domain: modality
|
||||||
|
description: Oaths, pledges, debts, prophecy, witness, and consequence are recurring source pressures.
|
||||||
|
relevant_distinctions:
|
||||||
|
- distinction_id: consequence_not_mood
|
||||||
|
label: Consequence, not mood
|
||||||
|
meaning: Binding speech and known doom should affect obligation, consequence, and restraint rather than simply making a line sound severe.
|
||||||
|
lowering_questions:
|
||||||
|
- Is the utterance about a promise, bargain, pledge, prophecy, debt, or witness?
|
||||||
|
- What consequence is structurally implied?
|
||||||
|
- Should the line be spare because the consequence is heavy?
|
||||||
|
required_when:
|
||||||
|
- interlingua_mentions_oath_debt_pledge_witness_prophecy_or_doom
|
||||||
|
forbidden_when:
|
||||||
|
- line_is_only_casual_social_contact
|
||||||
|
evidence_refs:
|
||||||
|
- poetic_edda:voluspa
|
||||||
|
- prose_edda:gylfaginning
|
||||||
|
wisdom_cost_and_memory:
|
||||||
|
axis_id: wisdom_cost_and_memory
|
||||||
|
domain: evidentiality
|
||||||
|
description: Knowledge in Eddic source pressure often has cost, pledge, sacrifice, witness, or memory attached.
|
||||||
|
relevant_distinctions:
|
||||||
|
- distinction_id: knowledge_is_paid_for
|
||||||
|
label: Knowledge is paid for
|
||||||
|
meaning: Knowing is not decorative omniscience; it may come through loss, risk, pledge, memory, or guarded source.
|
||||||
|
lowering_questions:
|
||||||
|
- What did this Persona pay, guard, lose, remember, or witness in order to know?
|
||||||
|
- Is the line claiming knowledge with enough evidence and cost pressure?
|
||||||
|
required_when:
|
||||||
|
- persona_is_mimir_odin_huginn_or_other_wisdom_memory_figure
|
||||||
|
- interlingua_mentions_secret_wisdom_memory_or_sight
|
||||||
|
forbidden_when:
|
||||||
|
- claim_is_unearned_style_or_generic_omniscience
|
||||||
|
evidence_refs:
|
||||||
|
- poetic_edda:voluspa:mimir_well
|
||||||
|
- prose_edda:gylfaginning:mimir_well
|
||||||
|
translated_edda_plainness:
|
||||||
|
axis_id: translated_edda_plainness
|
||||||
|
domain: discourse
|
||||||
|
description: Source-shaped English should be compact, declarative, concrete, and relation-aware without fake archaism.
|
||||||
|
relevant_distinctions:
|
||||||
|
- distinction_id: plain_force_without_fake_oldness
|
||||||
|
label: Plain force without fake oldness
|
||||||
|
meaning: Preserve the translated Edda's economy, parallel pressure, and concrete naming; avoid "ye olde" parody.
|
||||||
|
lowering_questions:
|
||||||
|
- Can this line become shorter and harder without losing meaning?
|
||||||
|
- Which archaic words are costume rather than source pressure?
|
||||||
|
- What relationship, cost, or consequence needs to remain visible?
|
||||||
|
required_when:
|
||||||
|
- edda_profile_activated_for_english_surface
|
||||||
|
forbidden_when:
|
||||||
|
- modern_plain_speech_better_serves_the_persona_and_context
|
||||||
|
evidence_refs:
|
||||||
|
- poetic_edda:bellows:general
|
||||||
|
- prose_edda:brodeur:general
|
||||||
|
linguistic_affordances:
|
||||||
|
grammar_features: []
|
||||||
|
phonology_features: []
|
||||||
|
lexicon_features:
|
||||||
|
- feature_id: source_role_lexicon
|
||||||
|
label: Source-role lexicon
|
||||||
|
status: draft
|
||||||
|
domain:
|
||||||
|
layer: lexicon
|
||||||
|
category: literary_source_register
|
||||||
|
description: Use concrete source-role words only when the interlingua earns them: oath, pledge, debt, witness, doom, well, eye, raven, forge, bargain, gift, mouth, ring, root, road, wolf, ash.
|
||||||
|
realization:
|
||||||
|
function: Preserve source salience without copying source text or inventing Old Norse speech.
|
||||||
|
surface_strategy: Choose one or two concrete source-role nouns when they clarify role, obligation, or consequence.
|
||||||
|
examples:
|
||||||
|
- "That pledge bought sight."
|
||||||
|
- "The witness is not the owner."
|
||||||
|
- "A bargain without a receipt is only noise."
|
||||||
|
activation:
|
||||||
|
allowed_when:
|
||||||
|
- interlingua_contains_source_role_obligation_or_witness_pressure
|
||||||
|
forbidden_when:
|
||||||
|
- source_words_would_only_add_ambience
|
||||||
|
requires_profile_authority: true
|
||||||
|
minimum_density: sparse
|
||||||
|
maximum_density: sparse
|
||||||
|
constraints:
|
||||||
|
register_contexts:
|
||||||
|
- English rendered Persona speech
|
||||||
|
speaker_contexts:
|
||||||
|
- Explicitly Eddic or Norse-derived Persona
|
||||||
|
collocation_limits:
|
||||||
|
- Avoid stacking mythic nouns until the line turns into an inventory.
|
||||||
|
incompatibilities:
|
||||||
|
- generic Viking flavor
|
||||||
|
- faux-archaic accent spelling
|
||||||
|
indexicality:
|
||||||
|
social_meanings:
|
||||||
|
- source-grounded role pressure
|
||||||
|
- obligation
|
||||||
|
- witness
|
||||||
|
in_group_meanings:
|
||||||
|
- Eddic referent is active and bounded
|
||||||
|
out_group_risks:
|
||||||
|
- kitsch
|
||||||
|
- religious or ethnic flattening
|
||||||
|
- fantasy stereotype
|
||||||
|
stereotypes_to_avoid:
|
||||||
|
- horned-helmet Viking voice
|
||||||
|
- gruff dwarf as universal Norse style
|
||||||
|
- omniscient wizard fog
|
||||||
|
evidence:
|
||||||
|
source_refs:
|
||||||
|
- poetic_edda:bellows:general
|
||||||
|
- prose_edda:brodeur:general
|
||||||
|
confidence: medium
|
||||||
|
uncertainty:
|
||||||
|
open_questions:
|
||||||
|
- Which source-role lexemes should be promoted into Persona-specific child profiles?
|
||||||
|
known_limits:
|
||||||
|
- This profile does not provide Old Norse vocabulary or pronunciation authority.
|
||||||
|
trace:
|
||||||
|
trace_label: source_role_lexicon
|
||||||
|
must_report_when_used: true
|
||||||
|
must_report_when_rejected: true
|
||||||
|
discourse_patterns:
|
||||||
|
- feature_id: gnomic_claim_with_anchor
|
||||||
|
label: Gnomic claim with anchor
|
||||||
|
status: draft
|
||||||
|
domain:
|
||||||
|
layer: discourse
|
||||||
|
category: compressed_wisdom_claim
|
||||||
|
description: A compact general claim may be allowed when anchored to a concrete source role, cost, witness, or consequence.
|
||||||
|
realization:
|
||||||
|
function: Let wisdom- or fate-shaped Personas speak with source pressure while keeping claims inspectable.
|
||||||
|
surface_strategy: Pair the general claim with a concrete anchor or consequence.
|
||||||
|
examples:
|
||||||
|
- "Wisdom without cost is a loose tongue."
|
||||||
|
- "A witness names the thing and does not become it."
|
||||||
|
activation:
|
||||||
|
allowed_when:
|
||||||
|
- interlingua_contains_general_judgment_plus_source_anchor
|
||||||
|
forbidden_when:
|
||||||
|
- output_would_become_free_floating_proverb
|
||||||
|
requires_profile_authority: true
|
||||||
|
minimum_density: sparse
|
||||||
|
maximum_density: sparse
|
||||||
|
constraints:
|
||||||
|
register_contexts:
|
||||||
|
- English rendered Persona speech
|
||||||
|
speaker_contexts:
|
||||||
|
- Eddic or Norse-derived Persona
|
||||||
|
collocation_limits:
|
||||||
|
- One compact claim per utterance unless the social move is explicitly ceremonial.
|
||||||
|
incompatibilities:
|
||||||
|
- vague prophecy
|
||||||
|
- decorative doom language
|
||||||
|
indexicality:
|
||||||
|
social_meanings:
|
||||||
|
- remembered cost
|
||||||
|
- restrained authority
|
||||||
|
- source-shaped judgment
|
||||||
|
in_group_meanings:
|
||||||
|
- Eddic plainness rather than modern explanatory friendliness
|
||||||
|
out_group_risks:
|
||||||
|
- sounding pompous or coercive
|
||||||
|
stereotypes_to_avoid:
|
||||||
|
- fortune-cookie Norse
|
||||||
|
evidence:
|
||||||
|
source_refs:
|
||||||
|
- poetic_edda:bellows:general
|
||||||
|
confidence: medium
|
||||||
|
uncertainty:
|
||||||
|
open_questions:
|
||||||
|
- Which Personas should permit this pattern frequently versus rarely?
|
||||||
|
known_limits:
|
||||||
|
- A Persona-specific profile should decide final density.
|
||||||
|
trace:
|
||||||
|
trace_label: gnomic_claim_with_anchor
|
||||||
|
must_report_when_used: true
|
||||||
|
must_report_when_rejected: true
|
||||||
|
orthography_features: []
|
||||||
|
code_switching:
|
||||||
|
allowed: false
|
||||||
|
constraints:
|
||||||
|
- Do not insert Old Norse, Icelandic, runic text, or archaic spellings unless a reviewed Persona-specific profile explicitly allows it.
|
||||||
|
patterns: []
|
||||||
|
evidence_ledger:
|
||||||
|
- ref: poetic_edda:bellows:general
|
||||||
|
source: Henry Adams Bellows translation of the Poetic Edda
|
||||||
|
url: https://en.wikisource.org/wiki/The_Poetic_Edda_(tr._Bellows)
|
||||||
|
summary: >
|
||||||
|
Public-domain English translation of Eddic poems used as a general
|
||||||
|
source-tradition reference for mythological figures, wisdom contests,
|
||||||
|
prophecy, divine speech, memory, and doom.
|
||||||
|
- ref: prose_edda:brodeur:general
|
||||||
|
source: Arthur Gilchrist Brodeur translation of the Prose Edda
|
||||||
|
url: https://en.wikisource.org/wiki/The_Prose_Edda_(1916_translation_by_Arthur_Gilchrist_Brodeur)
|
||||||
|
summary: >
|
||||||
|
Public-domain English translation of Snorri's Prose Edda used as a
|
||||||
|
general prose source for mythological episodes, names, kennings, and
|
||||||
|
figure-specific referents.
|
||||||
Binary file not shown.
|
|
@ -1 +0,0 @@
|
||||||
<EFBFBD>
|
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
File diff suppressed because one or more lines are too long
Binary file not shown.
39
data/target-language-ontology/README.md
Normal file
39
data/target-language-ontology/README.md
Normal file
|
|
@ -0,0 +1,39 @@
|
||||||
|
# Target-Language Cultural Ontology Profiles
|
||||||
|
|
||||||
|
These profiles describe culturally loaded realization choices for human target
|
||||||
|
languages used by Weksa utterance lowering.
|
||||||
|
|
||||||
|
They are not whole-culture summaries. They are bounded lowering profiles: the
|
||||||
|
distinctions a target lowerer must inspect before producing dialogue.
|
||||||
|
|
||||||
|
## Authority
|
||||||
|
|
||||||
|
Target-language cultural ontology profiles own:
|
||||||
|
|
||||||
|
- address and politeness distinctions
|
||||||
|
- register and social stance pressures
|
||||||
|
- agency, responsibility, and directness defaults
|
||||||
|
- idiom and metaphor risks
|
||||||
|
- script or orthographic choices that carry meaning
|
||||||
|
- forbidden flattenings that would make the output dishonest
|
||||||
|
|
||||||
|
They do not own:
|
||||||
|
|
||||||
|
- source meaning
|
||||||
|
- speaker-private state
|
||||||
|
- target-language grammar as a whole
|
||||||
|
- national or cultural stereotypes
|
||||||
|
- final wording without an interlingua packet
|
||||||
|
|
||||||
|
## Current Profiles
|
||||||
|
|
||||||
|
- [English, US fixture default](./en-US.yaml)
|
||||||
|
- [Brazilian Portuguese](./pt-BR.yaml)
|
||||||
|
- [Brazilian Portuguese overlay catalog](./pt-BR-overlays.yaml)
|
||||||
|
- [Brazilian Portuguese, Umbanda overlay](./pt-BR-umbanda.yaml)
|
||||||
|
- [Japanese](./ja-JP.yaml)
|
||||||
|
|
||||||
|
Each multilingual lowering fixture should cite the profile or profile stack it
|
||||||
|
used so reviewers can tell which social and cultural distinctions were
|
||||||
|
considered. Regional and religious-cultural options are overlays on a base
|
||||||
|
language profile, not replacements for source meaning.
|
||||||
135
data/target-language-ontology/en-US.yaml
Normal file
135
data/target-language-ontology/en-US.yaml
Normal file
|
|
@ -0,0 +1,135 @@
|
||||||
|
profile_id: weksa.target_cultural_ontology.en-US.v0
|
||||||
|
language:
|
||||||
|
code: en
|
||||||
|
locale: en-US
|
||||||
|
script: Latn
|
||||||
|
scope: >
|
||||||
|
Bounded English dialogue lowering profile for current Weksa fixtures. This is
|
||||||
|
not a claim about all English-speaking cultures.
|
||||||
|
authority:
|
||||||
|
owns:
|
||||||
|
- address directness choices
|
||||||
|
- casual versus formal register
|
||||||
|
- technical metaphor legibility
|
||||||
|
- understatement or sarcasm risk
|
||||||
|
does_not_own:
|
||||||
|
- source meaning
|
||||||
|
- speaker identity
|
||||||
|
- whole-culture description
|
||||||
|
salience_axes:
|
||||||
|
address:
|
||||||
|
relevant_distinctions:
|
||||||
|
- direct second person is acceptable when scene context names the addressee
|
||||||
|
- imperative force can be softened or sharpened through wording
|
||||||
|
lowering_questions:
|
||||||
|
- Does direct address preserve the speech act?
|
||||||
|
- Does softening erase urgency?
|
||||||
|
register:
|
||||||
|
relevant_distinctions:
|
||||||
|
- casual
|
||||||
|
- technical
|
||||||
|
- formal
|
||||||
|
- intimate
|
||||||
|
- institutional
|
||||||
|
lowering_questions:
|
||||||
|
- Is technical language character pressure or exposition sludge?
|
||||||
|
- Does sarcasm obscure the actual warning?
|
||||||
|
agency:
|
||||||
|
relevant_distinctions:
|
||||||
|
- explicit agent responsibility is usually available
|
||||||
|
- passive phrasing can hide authority or blame
|
||||||
|
lowering_questions:
|
||||||
|
- Should the speaker own the intervention directly?
|
||||||
|
- Does the line preserve who is at risk?
|
||||||
|
linguistic_affordances:
|
||||||
|
grammar_features:
|
||||||
|
- feature_id: en_us_direct_imperative_warning
|
||||||
|
label: Direct imperative warning
|
||||||
|
status: draft
|
||||||
|
domain:
|
||||||
|
layer: grammar
|
||||||
|
category: clause_mood
|
||||||
|
subcategory: imperative
|
||||||
|
description: >
|
||||||
|
Broad en-US allows direct imperatives for urgent safety warnings when
|
||||||
|
the speaker has standing to intervene and the scene rewards clarity
|
||||||
|
over mitigation.
|
||||||
|
formal_conditions:
|
||||||
|
syntactic_environment:
|
||||||
|
- main_clause
|
||||||
|
- direct_address_optional
|
||||||
|
semantic_inputs:
|
||||||
|
- warning
|
||||||
|
- addressee_action_to_stop_or_avoid
|
||||||
|
- consequence_or_risk_available
|
||||||
|
required_interlingua_fields:
|
||||||
|
- speech_act
|
||||||
|
- addressee
|
||||||
|
- prohibited_action
|
||||||
|
realization:
|
||||||
|
function: stop_or_prevent_action
|
||||||
|
surface_strategy: bare_imperative_or_do_not_imperative
|
||||||
|
examples:
|
||||||
|
- "Don't touch the damaged control."
|
||||||
|
activation:
|
||||||
|
allowed_when:
|
||||||
|
- scene_has_immediate_risk
|
||||||
|
- speaker_has_contextual_authority_or_care_relation
|
||||||
|
forbidden_when:
|
||||||
|
- politeness_softening_is_required_by_target_profile
|
||||||
|
- imperative_would_falsely_imply_hierarchy
|
||||||
|
requires_profile_authority: false
|
||||||
|
minimum_density: light
|
||||||
|
maximum_density: normal
|
||||||
|
constraints:
|
||||||
|
grammar_contexts:
|
||||||
|
- avoid_if_source_requires_indirect_request
|
||||||
|
register_contexts:
|
||||||
|
- casual
|
||||||
|
- technical
|
||||||
|
- emergency
|
||||||
|
speaker_contexts:
|
||||||
|
- known_addressee
|
||||||
|
- operator_warning
|
||||||
|
incompatibilities:
|
||||||
|
- generic_assistant_over_politeness
|
||||||
|
indexicality:
|
||||||
|
social_meanings:
|
||||||
|
- urgency
|
||||||
|
- clarity
|
||||||
|
- responsibility
|
||||||
|
in_group_meanings: []
|
||||||
|
out_group_risks:
|
||||||
|
- may sound abrupt if the scene is not urgent
|
||||||
|
stereotypes_to_avoid: []
|
||||||
|
evidence:
|
||||||
|
source_refs:
|
||||||
|
- schemas/cultural-ontology-profile.md
|
||||||
|
qdrant_source_ids: []
|
||||||
|
cached_source_refs: []
|
||||||
|
source_hashes: []
|
||||||
|
reviewer_refs: []
|
||||||
|
confidence: medium
|
||||||
|
uncertainty:
|
||||||
|
open_questions:
|
||||||
|
- Should a later en-US regional overlay override imperative force?
|
||||||
|
known_limits:
|
||||||
|
- Base en-US does not model regional, racialized, or contact-language
|
||||||
|
dialect grammar.
|
||||||
|
trace:
|
||||||
|
trace_label: direct_imperative_warning
|
||||||
|
must_report_when_used: true
|
||||||
|
must_report_when_rejected: true
|
||||||
|
phonology_features: []
|
||||||
|
lexicon_features: []
|
||||||
|
discourse_patterns: []
|
||||||
|
orthography_features: []
|
||||||
|
code_switching:
|
||||||
|
allowed: false
|
||||||
|
constraints:
|
||||||
|
- Base en-US profile does not authorize contact-language switching.
|
||||||
|
patterns: []
|
||||||
|
forbidden_flattenings:
|
||||||
|
- Do not treat English as the neutral source for other target languages.
|
||||||
|
- Do not use generic assistant politeness when a character voice is required.
|
||||||
|
- Do not let sarcasm replace the actual predicate being lowered.
|
||||||
70
data/target-language-ontology/ja-JP.yaml
Normal file
70
data/target-language-ontology/ja-JP.yaml
Normal file
|
|
@ -0,0 +1,70 @@
|
||||||
|
profile_id: weksa.target_cultural_ontology.ja-JP.v0
|
||||||
|
language:
|
||||||
|
code: ja
|
||||||
|
locale: ja-JP
|
||||||
|
script: Jpan
|
||||||
|
scope: >
|
||||||
|
Bounded Japanese dialogue lowering profile for Weksa fixtures. This profile
|
||||||
|
records realization distinctions that must be inspected; it is not a total
|
||||||
|
description of Japanese culture.
|
||||||
|
authority:
|
||||||
|
owns:
|
||||||
|
- politeness and honorific pressure
|
||||||
|
- in-group/out-group stance when context supplies it
|
||||||
|
- subject and object omission choices
|
||||||
|
- script mix and technical register
|
||||||
|
- sentence-final force
|
||||||
|
does_not_own:
|
||||||
|
- source meaning
|
||||||
|
- character identity
|
||||||
|
- exhaustive Japanese grammar
|
||||||
|
salience_axes:
|
||||||
|
politeness:
|
||||||
|
relevant_distinctions:
|
||||||
|
- plain
|
||||||
|
- polite
|
||||||
|
- honorific
|
||||||
|
- humble
|
||||||
|
- rough
|
||||||
|
lowering_questions:
|
||||||
|
- Does politeness preserve the relationship, or does it lie about it?
|
||||||
|
- Is roughness character-specific or generic stylization?
|
||||||
|
social_position:
|
||||||
|
relevant_distinctions:
|
||||||
|
- uchi_soto_pressure
|
||||||
|
- hierarchy
|
||||||
|
- familiarity
|
||||||
|
- emergency_override
|
||||||
|
lowering_questions:
|
||||||
|
- Does urgency override ordinary deference?
|
||||||
|
- Does the line imply a relationship the scene has not earned?
|
||||||
|
ellipsis:
|
||||||
|
relevant_distinctions:
|
||||||
|
- subject omission
|
||||||
|
- object omission
|
||||||
|
- explicit pronoun for emphasis or contrast
|
||||||
|
lowering_questions:
|
||||||
|
- Is the addressee recoverable without an explicit pronoun?
|
||||||
|
- Would explicit pronouns sound unnatural or too pointed?
|
||||||
|
script_and_register:
|
||||||
|
relevant_distinctions:
|
||||||
|
- kanji for technical compounds
|
||||||
|
- kana for softness or readability
|
||||||
|
- katakana for foreignness, machine texture, or emphasis
|
||||||
|
lowering_questions:
|
||||||
|
- Does the script choice help the speaker voice?
|
||||||
|
- Does a technical compound preserve precision without becoming stiff?
|
||||||
|
sentence_final_force:
|
||||||
|
relevant_distinctions:
|
||||||
|
- direct request
|
||||||
|
- warning consequence
|
||||||
|
- command
|
||||||
|
- threat-like phrasing
|
||||||
|
lowering_questions:
|
||||||
|
- Does the ending carry warning force without cartoon aggression?
|
||||||
|
- Does the line preserve the hazard predicate?
|
||||||
|
forbidden_flattenings:
|
||||||
|
- Do not treat politeness as decoration.
|
||||||
|
- Do not use explicit pronouns just because English has them.
|
||||||
|
- Do not make every sharp line rough or anime-coded.
|
||||||
|
- Do not erase the difference between technical precision and formal politeness.
|
||||||
76
data/target-language-ontology/pt-BR-overlays.yaml
Normal file
76
data/target-language-ontology/pt-BR-overlays.yaml
Normal file
|
|
@ -0,0 +1,76 @@
|
||||||
|
catalog_id: weksa.target_cultural_ontology.pt-BR.overlays.v0
|
||||||
|
base_profile: data/target-language-ontology/pt-BR.yaml
|
||||||
|
scope: >
|
||||||
|
Catalog of optional Brazilian Portuguese regional and cultural overlay slots.
|
||||||
|
An overlay slot is not permission to imitate a community. It is a named place
|
||||||
|
where reviewed, source-grounded distinctions can live.
|
||||||
|
activation_rule: >
|
||||||
|
Use the broad pt-BR profile by default. Add an overlay only when source
|
||||||
|
context, speaker state, setting documentation, or authoring metadata names the
|
||||||
|
relevant region, community, institution, religion, or register.
|
||||||
|
overlay_slots:
|
||||||
|
- overlay_id: broad_brazilian_default
|
||||||
|
status: active_base_mode
|
||||||
|
profile_ref: data/target-language-ontology/pt-BR.yaml
|
||||||
|
owns:
|
||||||
|
- broadly legible Brazilian Portuguese dialogue
|
||||||
|
does_not_own:
|
||||||
|
- regional specificity
|
||||||
|
- religious-cultural specificity
|
||||||
|
- overlay_id: rio_urban
|
||||||
|
status: planned_profile
|
||||||
|
profile_ref:
|
||||||
|
owns_when_defined:
|
||||||
|
- Rio de Janeiro urban speech/register pressures for specific characters or settings
|
||||||
|
blocked_until:
|
||||||
|
- reviewed source context
|
||||||
|
- human-approved examples
|
||||||
|
- overlay_id: sao_paulo_urban
|
||||||
|
status: planned_profile
|
||||||
|
profile_ref:
|
||||||
|
owns_when_defined:
|
||||||
|
- Sao Paulo urban speech/register pressures for specific characters or settings
|
||||||
|
blocked_until:
|
||||||
|
- reviewed source context
|
||||||
|
- human-approved examples
|
||||||
|
- overlay_id: northeast_regional
|
||||||
|
status: planned_profile
|
||||||
|
profile_ref:
|
||||||
|
owns_when_defined:
|
||||||
|
- Northeast Brazilian regional speech/register pressures for specific characters or settings
|
||||||
|
blocked_until:
|
||||||
|
- specific region named
|
||||||
|
- reviewed examples
|
||||||
|
- human approval
|
||||||
|
- overlay_id: south_regional
|
||||||
|
status: planned_profile
|
||||||
|
profile_ref:
|
||||||
|
owns_when_defined:
|
||||||
|
- South Brazilian regional speech/register pressures for specific characters or settings
|
||||||
|
blocked_until:
|
||||||
|
- specific region named
|
||||||
|
- reviewed examples
|
||||||
|
- human approval
|
||||||
|
- overlay_id: amazonian_regional
|
||||||
|
status: planned_profile
|
||||||
|
profile_ref:
|
||||||
|
owns_when_defined:
|
||||||
|
- Amazonian regional speech/register pressures for specific characters or settings
|
||||||
|
blocked_until:
|
||||||
|
- specific region or community named
|
||||||
|
- reviewed examples
|
||||||
|
- human approval
|
||||||
|
- overlay_id: umbanda
|
||||||
|
status: active_guarded_overlay
|
||||||
|
profile_ref: data/target-language-ontology/pt-BR-umbanda.yaml
|
||||||
|
owns:
|
||||||
|
- Umbanda-related register gating
|
||||||
|
- sacred address caution
|
||||||
|
- anti-caricature constraints
|
||||||
|
blocked_until:
|
||||||
|
- source context or authoring metadata explicitly activates Umbanda context
|
||||||
|
forbidden_flattenings:
|
||||||
|
- Do not use a planned overlay as if it were defined.
|
||||||
|
- Do not treat regional slots as accent generators.
|
||||||
|
- Do not use religious-cultural overlays for ambience.
|
||||||
|
- Do not collapse race, region, religion, class, and slang into one Brazilian flavor.
|
||||||
72
data/target-language-ontology/pt-BR-umbanda.yaml
Normal file
72
data/target-language-ontology/pt-BR-umbanda.yaml
Normal file
|
|
@ -0,0 +1,72 @@
|
||||||
|
profile_id: weksa.target_cultural_ontology.pt-BR.umbanda.v0
|
||||||
|
language:
|
||||||
|
code: pt
|
||||||
|
locale: pt-BR
|
||||||
|
script: Latn
|
||||||
|
overlay:
|
||||||
|
base_profile: data/target-language-ontology/pt-BR.yaml
|
||||||
|
kind: religious_cultural_register
|
||||||
|
scope: >
|
||||||
|
Optional Brazilian Portuguese cultural ontology overlay for Umbanda-related
|
||||||
|
scenes, speakers, ritual contexts, or referents. This profile is a consent and
|
||||||
|
context gate, not a decorative vocabulary pack.
|
||||||
|
authority:
|
||||||
|
owns:
|
||||||
|
- whether Umbanda-specific register is allowed for a realization
|
||||||
|
- sacred or ritual address caution
|
||||||
|
- house/tradition specificity warnings
|
||||||
|
- syncretic and entity-reference sensitivity
|
||||||
|
- anti-caricature constraints
|
||||||
|
does_not_own:
|
||||||
|
- Umbanda doctrine as a whole
|
||||||
|
- authority to invent ritual speech
|
||||||
|
- generic Brazilian Portuguese
|
||||||
|
- speaker identity unless the speaker context explicitly supplies it
|
||||||
|
activation_requirements:
|
||||||
|
any_required:
|
||||||
|
- source context names an Umbanda scene, terreiro, practitioner, entity, ritual, or discourse community
|
||||||
|
- speaker profile explicitly owns an Umbanda-informed voice or memory
|
||||||
|
- authoring metadata requests this overlay for reviewed cultural grounding
|
||||||
|
never_activate_for:
|
||||||
|
- generic Brazilian flavor
|
||||||
|
- jokes, insults, mystic ambience, or unexplained exotic texture
|
||||||
|
- non-Umbanda religious scenes unless the source context makes a syncretic relation explicit
|
||||||
|
salience_axes:
|
||||||
|
sacred_address:
|
||||||
|
relevant_distinctions:
|
||||||
|
- ordinary dialogue
|
||||||
|
- respectful reference
|
||||||
|
- ritual address
|
||||||
|
- entity_or_guide_reference
|
||||||
|
lowering_questions:
|
||||||
|
- Is the line ordinary speech or ritualized speech?
|
||||||
|
- Does the speaker have standing to use sacred address?
|
||||||
|
- Should a term be left general until a human reviewer approves specificity?
|
||||||
|
house_specificity:
|
||||||
|
relevant_distinctions:
|
||||||
|
- practices and terms can vary by house, region, and lineage
|
||||||
|
- generic flattening can become false authority
|
||||||
|
lowering_questions:
|
||||||
|
- Is the output pretending one practice represents all Umbanda?
|
||||||
|
- Should the trace mark the choice as house-specific or uncertain?
|
||||||
|
syncretic_reference:
|
||||||
|
relevant_distinctions:
|
||||||
|
- Catholic, African, Indigenous, Spiritist, and local references may coexist differently by context
|
||||||
|
- entity names and roles are not interchangeable decoration
|
||||||
|
lowering_questions:
|
||||||
|
- Does the source actually contain the syncretic relation?
|
||||||
|
- Does the line preserve respect without overexplaining?
|
||||||
|
anti_caricature:
|
||||||
|
relevant_distinctions:
|
||||||
|
- religious register
|
||||||
|
- folkloric stereotype
|
||||||
|
- occult ambience
|
||||||
|
lowering_questions:
|
||||||
|
- Is the output using Umbanda as culture, faith, and practice rather than spooky texture?
|
||||||
|
- Would a practitioner recognize this as respectful enough for review?
|
||||||
|
forbidden_flattenings:
|
||||||
|
- Do not use Umbanda terms as generic magic flavor.
|
||||||
|
- Do not invent ritual formulas.
|
||||||
|
- Do not collapse Umbanda into Candomblé, Spiritism, Catholicism, or generic Afro-Brazilian religion.
|
||||||
|
- Do not imply all Brazilian Portuguese religious language is Umbanda-coded.
|
||||||
|
- Do not activate this overlay unless source context or authoring metadata explicitly asks for it.
|
||||||
74
data/target-language-ontology/pt-BR.yaml
Normal file
74
data/target-language-ontology/pt-BR.yaml
Normal file
|
|
@ -0,0 +1,74 @@
|
||||||
|
profile_id: weksa.target_cultural_ontology.pt-BR.v0
|
||||||
|
language:
|
||||||
|
code: pt
|
||||||
|
locale: pt-BR
|
||||||
|
script: Latn
|
||||||
|
scope: >
|
||||||
|
Bounded Brazilian Portuguese dialogue lowering profile. This profile is the
|
||||||
|
base Brazilian Portuguese target; regional, class, religious, and community
|
||||||
|
registers should be added as explicit overlays rather than silently folded
|
||||||
|
into generic pt-BR.
|
||||||
|
authority:
|
||||||
|
owns:
|
||||||
|
- Brazilian Portuguese lexical preference
|
||||||
|
- treatment and address choices
|
||||||
|
- directness versus mitigation
|
||||||
|
- regional_or_cultural_overlay_selection
|
||||||
|
- idiom naturalness for reviewed fixture prose
|
||||||
|
does_not_own:
|
||||||
|
- source meaning
|
||||||
|
- Brazilian culture as a whole
|
||||||
|
- regional or religious-community authority without an explicit overlay
|
||||||
|
- target speaker identity
|
||||||
|
salience_axes:
|
||||||
|
treatment:
|
||||||
|
relevant_distinctions:
|
||||||
|
- voce
|
||||||
|
- tu_regionally_marked
|
||||||
|
- o_senhor_a_senhora
|
||||||
|
- name_or_title_address
|
||||||
|
lowering_questions:
|
||||||
|
- Is the relationship close, hostile, hierarchical, unknown, or sacred?
|
||||||
|
- Does `voce` preserve broad Brazilian naturalness better than `tu`?
|
||||||
|
- Is `tu` regionally grounded or just imported flavor?
|
||||||
|
directness:
|
||||||
|
relevant_distinctions:
|
||||||
|
- bare imperative
|
||||||
|
- softened imperative
|
||||||
|
- warning framed as consequence
|
||||||
|
- formal warning
|
||||||
|
lowering_questions:
|
||||||
|
- Does urgency justify blunt imperative force?
|
||||||
|
- Would mitigation make the warning dishonest?
|
||||||
|
lexical_locale:
|
||||||
|
relevant_distinctions:
|
||||||
|
- Brazilian Portuguese terms should be preferred in reviewed pt-BR fixtures
|
||||||
|
- European Portuguese defaults must be marked if intentionally used
|
||||||
|
lowering_questions:
|
||||||
|
- Is a term recognizably Brazilian Portuguese in this context?
|
||||||
|
- Is a literal technical word natural enough for dialogue?
|
||||||
|
regional_and_cultural_overlay:
|
||||||
|
relevant_distinctions:
|
||||||
|
- broad_brazilian_default
|
||||||
|
- regional_speech_pattern
|
||||||
|
- class_or_institutional_register
|
||||||
|
- religious_or_ritual_register
|
||||||
|
- community_specific_lexicon
|
||||||
|
lowering_questions:
|
||||||
|
- Does the source context justify an overlay?
|
||||||
|
- Is the overlay owned by a named profile?
|
||||||
|
- Would applying the overlay stereotype the speaker or setting?
|
||||||
|
metaphor:
|
||||||
|
relevant_distinctions:
|
||||||
|
- technical metaphors can preserve character ontology
|
||||||
|
- over-literal metaphors may sound translated rather than spoken
|
||||||
|
lowering_questions:
|
||||||
|
- Does the metaphor still sound like the speaker?
|
||||||
|
- Does it preserve the warning instead of becoming decorative?
|
||||||
|
forbidden_flattenings:
|
||||||
|
- Do not label output as Portuguese without a locale in reviewed fixtures.
|
||||||
|
- Do not default to European Portuguese wording when the target is pt-BR.
|
||||||
|
- Do not treat Brazilian Portuguese as one culture, region, class, or religion.
|
||||||
|
- Do not add regional or religious markers without a cited overlay profile and source-context reason.
|
||||||
|
- Do not soften a warning merely because directness feels impolite in English review.
|
||||||
|
overlay_catalog: data/target-language-ontology/pt-BR-overlays.yaml
|
||||||
|
|
@ -1,120 +0,0 @@
|
||||||
# Calibrating the accept rule
|
|
||||||
|
|
||||||
`weksa loop calibrate` measures how the live accept rule behaves on a run's own
|
|
||||||
Validation corpus: how often it accepts, rejects as no gain, or regresses a
|
|
||||||
candidate that should be none of those. It does not reimplement the rule. It
|
|
||||||
reads the answered draws through `RunStores::read_draws` (the scoring
|
|
||||||
`weksa loop relower` uses), decides every trial with `decide_panel` over
|
|
||||||
`paired_floor` (`crates/weksa/src/run.rs`), and prints, per class, the items
|
|
||||||
read and skipped per language, the trial totals with their Accepted, NoGain and
|
|
||||||
Regressed counts, and the mean panel noise. That is all it produces: the
|
|
||||||
Accepted/NoGain/Regressed counts and the noise mean in the map's "Recalibration
|
|
||||||
on the grown corpus" block are reproduced by it, and the prompted trials are
|
|
||||||
also printed split by the recorded lowering `R`'s role (`prompted-r-candidate`,
|
|
||||||
`prompted-r-base`, `prompted-r-free`, see below), the candidate counts per arm
|
|
||||||
and the mild arms' composite offsets from the prompted mean (see below); the
|
|
||||||
other figures there (the
|
|
||||||
35% with R removed, the noise range, the 0.0185 gain and its 0.41 ratio, the
|
|
||||||
bootstrap intervals, the per-item SDs, R's +0.024, the mild-arm duplicate
|
|
||||||
counts, the Train-vocabulary overlap, the old/new
|
|
||||||
class composites and the ten parse-refused item ids) were computed by scratch scripts that are not in the repo. It is a
|
|
||||||
diagnostic: it records nothing, writes no prompts, and never touches Test or
|
|
||||||
Train.
|
|
||||||
|
|
||||||
## What it counts
|
|
||||||
|
|
||||||
The draws are lowerings of one round's Validation packets: the round's own
|
|
||||||
(recorded) lowering `R`, plus draws you label. `P` is `R` and the `--prompted`
|
|
||||||
draws; the floor size is `FLOOR_ARMS_MIN` (3), the live rule's.
|
|
||||||
|
|
||||||
- Prompted trials: every ordered pair (base, candidate) in `P`, against every
|
|
||||||
3-subset of the rest of `P` as the floor. A repeat of the same lowering
|
|
||||||
should read NoGain; Accepted is a false accept, Regressed a false reject.
|
|
||||||
(`R` comes from the round, not from `relower`, so it may not be
|
|
||||||
exchangeable with a repeat.) The prompted line is therefore printed three
|
|
||||||
more times, by `R`'s role in the trial: `prompted-r-candidate` (`R` is the
|
|
||||||
candidate), `prompted-r-base` (`R` is the base) and `prompted-r-free` (`R` is
|
|
||||||
neither; it may sit in the floor). The three sum to the prompted line. Read the Accepted
|
|
||||||
and Regressed columns of each line: a false accept sits mostly under the
|
|
||||||
first and a false reject under the second, but `R`-as-candidate trials can
|
|
||||||
Regress too (class `old` on `growth-scene`: 5 of its 15). The third
|
|
||||||
is the exchangeable rate: on `growth-scene` it was 0 Accepted and 0 Regressed
|
|
||||||
of 80 while all 11 false accepts and 9 false rejects involved `R`. Which of
|
|
||||||
the two rates the live rule should be held to is undecided: the live loop
|
|
||||||
meets `R` as the head's establishing lowering.
|
|
||||||
- Degraded trials, per `--degraded` arm: every base in `P` against every
|
|
||||||
3-subset of the rest of `P`, the degraded draw as the candidate. These should
|
|
||||||
read Regressed.
|
|
||||||
- Candidate trials, pooled over the `--candidate` arms: the same, tallied
|
|
||||||
Accepted, NoGain, Regressed. This is the rule's power against that arm. The
|
|
||||||
pooled line is followed by one `candidate=LABEL` line per arm: arms of one
|
|
||||||
kind can differ completely (Train-guide h1 and h2 Regressed in every trial
|
|
||||||
while h3 and h4 gained), and then the pooled figure is not a power.
|
|
||||||
- Mild offset: each `degraded=LABEL` line ends with `offset LANGUAGE=+x.xxxx`, the
|
|
||||||
arm's composite minus the mean composite of the prompted draws (`R` included),
|
|
||||||
per language over the items read. A degraded arm that is barely below the
|
|
||||||
prompted mean is a weak probe of the rule's sensitivity.
|
|
||||||
- Noise: the mean `panel_noise` over every (base, floor) choice.
|
|
||||||
|
|
||||||
With six prompted draws including `R` that is 120 prompted trials, 60 per
|
|
||||||
degraded arm and 240 candidate trials. Trials share draws and are not
|
|
||||||
independent. Each count is printed for `all` Validation items and for each
|
|
||||||
`--class NAME=PREFIX[,PREFIX...]` (items whose key starts with any prefix), from
|
|
||||||
the same per-item scores. A class with fewer than two read items in a language
|
|
||||||
has no floor and reads Regressed in every trial; that is printed.
|
|
||||||
|
|
||||||
## Steps
|
|
||||||
|
|
||||||
1. `weksa loop open ROOT ...` and `weksa loop round ROOT` record round N and
|
|
||||||
leave its answers in `ROOT/round-N/`.
|
|
||||||
2. For each draw label: `weksa loop relower ROOT --round N --arm LABEL
|
|
||||||
--lowerer NAME` writes the byte-identical lowering prompts to
|
|
||||||
`ROOT/relower-N-LABEL/<digest>.prompt` and exits 3. Answer each prompt into
|
|
||||||
`ROOT/relower-N-LABEL/<digest>.answer`.
|
|
||||||
3. `weksa loop calibrate ROOT --round N --prompted p1 --prompted p2 ...
|
|
||||||
[--degraded LABEL]... [--candidate LABEL]... [--class NAME=PREFIX]...`.
|
|
||||||
At least four `--prompted` labels are needed (a pair and three floors with
|
|
||||||
`R` make five draws). It exits 1 and lists `unanswered LABEL COUNT` for any
|
|
||||||
draw with prompts still unanswered. Labels match `[a-z0-9-]{1,32}` and differ
|
|
||||||
across the three kinds. It needs the Ollama embedder of the run; a full
|
|
||||||
60-item, 13-draw calibration is on the order of 1500 embedded texts, so
|
|
||||||
launch it detached with a log.
|
|
||||||
|
|
||||||
The scratch packet store is `ROOT/calibrate-N/base.cc`; delete it at will.
|
|
||||||
|
|
||||||
## Answer file format
|
|
||||||
|
|
||||||
`<digest>.answer` is one JSON object and nothing else, no code fence:
|
|
||||||
`{"text": "..."}`, a JSON-escaped string. The answerer reads only the prompt
|
|
||||||
(its first line says what to write) and writes the answer. Every prompt of the
|
|
||||||
draw must be answered.
|
|
||||||
|
|
||||||
## Briefs per arm
|
|
||||||
|
|
||||||
Independence rule: a fresh agent per batch of answers; no batch sees another
|
|
||||||
draw's answers. Only `mild` reads `round-N/*.answer`.
|
|
||||||
|
|
||||||
- Plain prompted: answer each prompt as its own first line asks; read nothing
|
|
||||||
else in the run root.
|
|
||||||
- Mild: for each prompt, read the recorded lowering in
|
|
||||||
`ROOT/round-N/<digest>.answer` and write the same-format answer whose text is
|
|
||||||
that utterance with exactly one meaning detail blurred: change or drop one of
|
|
||||||
tense, polarity, modality or hedge, a referent, a quantity or a predicate's
|
|
||||||
concept (replace it with a near but different word). Keep language, register,
|
|
||||||
length and fluency; the result must read as a natural utterance and must not
|
|
||||||
mention the change. Pick the detail that is not the most salient one. A
|
|
||||||
second mild draw blurs a different detail than the first and is written
|
|
||||||
without reading the first's answers. The wording of the distil-base original
|
|
||||||
is lost, and the dispatch used for growth-base is not on disk, so this is the
|
|
||||||
wording from now on. Warning: on growth-base two mild arms briefed this way
|
|
||||||
came out identical in 27 of 60 items, so independently briefed mild arms
|
|
||||||
are not independent degradations; read a mild draw's regress count with that
|
|
||||||
in mind.
|
|
||||||
- Train-guide: before answering, read the Train sentences of the corpus in the
|
|
||||||
batch's language as a register guide and let it steer word choice; the guide
|
|
||||||
gives phrasing only and each answer must still say exactly what its own packet
|
|
||||||
says. The guide section is the live Train items of that language, built from
|
|
||||||
`weksa corpus dump` filtered to Train, and is given to the answerer in the
|
|
||||||
brief, never pasted into this document, and never includes Validation or Test
|
|
||||||
text. Read nothing else in the run root.
|
|
||||||
- Control: a script writes `{"text":"Yes."}` as the answer to every prompt.
|
|
||||||
|
|
@ -369,12 +369,6 @@ target. A Qdrant nearest neighbor is a witness candidate, not a cultural fact.
|
||||||
|
|
||||||
\section{Proposed CultCache State Shape}
|
\section{Proposed CultCache State Shape}
|
||||||
|
|
||||||
No profile store exists. The per-language lowering choices live in
|
|
||||||
\texttt{data/ontology/ontology.cc}; the typed profiles once stored under
|
|
||||||
\texttt{.weksa/cultural-ontology/} and the \texttt{weksa.target\_cultural\_ontology.*}
|
|
||||||
base profiles named in the listing and sketches below are deleted. Community and dialect
|
|
||||||
profiles belong to follow-up \texttt{language-ontology-communities}.
|
|
||||||
|
|
||||||
The following shape should become a typed CultCache document family:
|
The following shape should become a typed CultCache document family:
|
||||||
\texttt{weksa.cultural\_ontology\_profile.v0}. It can represent base languages,
|
\texttt{weksa.cultural\_ontology\_profile.v0}. It can represent base languages,
|
||||||
dialects, sociolects, regional overlays, ritual registers, and communities of
|
dialects, sociolects, regional overlays, ritual registers, and communities of
|
||||||
|
|
@ -618,6 +612,15 @@ feature requires speaker authority, whether it can be rendered in text or only
|
||||||
speech synthesis, which source hashes support it, and what must be reported if
|
speech synthesis, which source hashes support it, and what must be reported if
|
||||||
the feature is rejected.
|
the feature is rejected.
|
||||||
|
|
||||||
|
Recommended CultCache paths:
|
||||||
|
|
||||||
|
\begin{lstlisting}[basicstyle=\ttfamily\small]
|
||||||
|
.weksa/cultural-ontology/{profileId}.cc
|
||||||
|
.weksa/cultural-ontology/{baseProfileId}/overlays/{overlayId}.cc
|
||||||
|
.weksa/cultural-ontology/catalogs/{catalogId}.cc
|
||||||
|
.weksa/cultural-ontology/evidence-bundles/{bundleId}.cc
|
||||||
|
\end{lstlisting}
|
||||||
|
|
||||||
\section{Dialect Profile Sketches}
|
\section{Dialect Profile Sketches}
|
||||||
|
|
||||||
\subsection{AAVE/AAL}
|
\subsection{AAVE/AAL}
|
||||||
|
|
|
||||||
|
|
@ -85,8 +85,8 @@ thing.
|
||||||
An entity's citation form in a language may carry a ruled pronunciation
|
An entity's citation form in a language may carry a ruled pronunciation
|
||||||
(`LanguageForms.pronunciation`): bare IPA with stress marks, who ruled it, the
|
(`LanguageForms.pronunciation`): bare IPA with stress marks, who ruled it, the
|
||||||
date, the source, and an optional list of rejected variants kept so nobody
|
date, the source, and an optional list of rejected variants kept so nobody
|
||||||
re-proposes them. `weksa pronounce <store> <key>` prints the stored IPA of each
|
re-proposes them. `weksa lower <store> <key> --ipa` prints the stored IPA of each
|
||||||
entity the packet cites, one per line (`--phones` prints the phones line), and refuses with `NoPronunciation` for an
|
entity the packet cites, one per line, and refuses with `NoPronunciation` for an
|
||||||
entity with none; rejected variants are never read. This is not the plan below:
|
entity with none; rejected variants are never read. This is not the plan below:
|
||||||
it carries no timing, contour, length or invented prosody, and Weksa has no
|
it carries no timing, contour, length or invented prosody, and Weksa has no
|
||||||
English G2P, so an unruled name has no IPA.
|
English G2P, so an unruled name has no IPA.
|
||||||
|
|
|
||||||
|
|
@ -163,8 +163,6 @@ names the rule.
|
||||||
- V15: an `addressee_consent` is set only by provenance `Authored` or
|
- V15: an `addressee_consent` is set only by provenance `Authored` or
|
||||||
`GameTelemetry`; a decomposition of text or imported corpus text cannot mint
|
`GameTelemetry`; a decomposition of text or imported corpus text cannot mint
|
||||||
it.
|
it.
|
||||||
- V17: an `addressee` names a referent, not a predication.
|
|
||||||
- V18: a predication whose frame is the inventory's `request_intervention` is named by `addressee_consent`.
|
|
||||||
|
|
||||||
V9 is retired and not renumbered: the form of a key or a grade is its type, and
|
V9 is retired and not renumbered: the form of a key or a grade is its type, and
|
||||||
an extension key's family is V6. `PacketStore::admit` returns the lowering view of a
|
an extension key's family is V6. `PacketStore::admit` returns the lowering view of a
|
||||||
|
|
@ -192,29 +190,15 @@ ids.
|
||||||
|
|
||||||
### `discourse`
|
### `discourse`
|
||||||
|
|
||||||
Why the packet is said: `speech_act` is a speech-act id, `addressee` a referent's node id (V17),
|
Why the packet is said: `speech_act` is a speech-act id, `addressee` a node id,
|
||||||
and `features` carry source-side nuance on content (hedge, certainty, irony and
|
and `features` carry source-side nuance on content (hedge, certainty, irony and
|
||||||
whatever the inventory grows). This is source-side communicative pressure, not
|
whatever the inventory grows). This is source-side communicative pressure, not
|
||||||
target-language register. `addressee_consent` is no feature: it names the
|
target-language register. `addressee_consent` is no feature: it names the
|
||||||
predication the addressee asked for or agreed to, a fact about what the
|
predication the addressee asked for or agreed to, a fact about what the
|
||||||
addressee did. `V14` and `V15` check the field: it may be set only by authoring or game
|
addressee did. The English lowerer's request row (`frame:3`, "{agent} asked
|
||||||
telemetry, and it must name a predication of the packet. `V18` refuses a predication whose frame resolves, at the packet's revision, to
|
{patient} to stop") reads it: the consent must name that predication and its
|
||||||
the gloss `request_intervention` (`frame:3` in the committed inventories)
|
agent must be the addressee, else `ConsentMissing` or `AskerNotAddressee`. A
|
||||||
unless `addressee_consent` names that predication. The encoder can never set
|
consent naming any other predication is refused as unrendered.
|
||||||
consent (`V15`), so an encoder-produced `request_intervention` predication (`frame:3`) is always refused.
|
|
||||||
Limits, tracked by `weksa-lowering:follow_up:request-frame-consent-residual`:
|
|
||||||
`V18` keys on the gloss `request_intervention` and sees no other frame or
|
|
||||||
concept, so a request-shaped meaning built from another frame or concept (a
|
|
||||||
`frame:12` communication with `concept:6` request validates) or from a renamed
|
|
||||||
or newly minted frame is not seen by `validate`, which does not read meaning.
|
|
||||||
The inventory carries no marker for request shape, so classifying it would be
|
|
||||||
an inventory-shape decision for Imagination and language-ontology. `addressee_consent`
|
|
||||||
is one field, so a packet carries one request predication, and a compound
|
|
||||||
request is two packets until a packet schema revision makes consent a set. The
|
|
||||||
asker is not checked to be the addressee: a consented `frame:3` whose agent is
|
|
||||||
not the addressee validates, and no code outside `validate` and the encoder
|
|
||||||
prompt text reads `addressee_consent`. That check is an owed obligation with no
|
|
||||||
owner yet.
|
|
||||||
|
|
||||||
### `stance`
|
### `stance`
|
||||||
|
|
||||||
|
|
@ -229,29 +213,6 @@ affect and restraint to its own register. A held-back feeling is not a claim
|
||||||
about the world, so it is never a predication. An empty stance is valid. The
|
about the world, so it is never a predication. An empty stance is valid. The
|
||||||
stance reaches the lowerer in the lowering view.
|
stance reaches the lowerer in the lowering view.
|
||||||
|
|
||||||
### Speaker and scene documents
|
|
||||||
|
|
||||||
Two documents outside the packet hold the speaker's standing, in the store that
|
|
||||||
owns them (`SpeakerStateStore`). `weksa.speaker` (key `speaker:<entity slug>@<n>`)
|
|
||||||
holds one entity's stable axes as feature values of slot `Speaker`.
|
|
||||||
`weksa.scene` (key `scene:<group>@<n>`) holds a corpus group's participants and,
|
|
||||||
per ordered pair, the speaker's relation to a party (slot `Scene`); the party is
|
|
||||||
a named participant or `Party::Unnamed`, an addressee the scene does not name,
|
|
||||||
such as the player or a crowd. A revision appends and never edits: `n` is the
|
|
||||||
latest plus one, and the latest revision is the highest `n`. Each carries an
|
|
||||||
`Annotator`, `Operator` or `Model` with an attestation string for its generator.
|
|
||||||
Both hold typed feature values only, never text. A feature of slot `Stance`,
|
|
||||||
`Speaker` or `Scene` sits in that slot alone (each layer has one writer and one
|
|
||||||
lifetime), which `admit_revision` enforces. `LoweringInput::with_scene` takes the
|
|
||||||
speaker-state and corpus stores and names neither speaker nor scene: the view knows its corpus item, so
|
|
||||||
the speaker is the item's speaker and the scene is the store's latest for the
|
|
||||||
item's group. The scene must list the item's speaker and the speaker of every live item of the
|
|
||||||
group (V16; `Scene::uncovered`, the rule `put_scene` applies too, so a scene a
|
|
||||||
later corpus item outgrew is refused at the read), and the relation
|
|
||||||
selected is the speaker's to the entity of the addressee referent (to
|
|
||||||
`Unnamed` when that referent has no entity, to nothing when the packet has no
|
|
||||||
addressee or the scene holds no such relation). V13 is retired.
|
|
||||||
|
|
||||||
### `referents`
|
### `referents`
|
||||||
|
|
||||||
Entities, groups, places, abstractions and the rest. `kind` is a referent-kind
|
Entities, groups, places, abstractions and the rest. `kind` is a referent-kind
|
||||||
|
|
@ -267,10 +228,6 @@ pair a role id with a filler node (the inventory seeds `agent`, `patient`,
|
||||||
aspect, polarity, modality and evidentiality are features with declared option
|
aspect, polarity, modality and evidentiality are features with declared option
|
||||||
sets, not fields.
|
sets, not fields.
|
||||||
|
|
||||||
A predication holds each role once (V11). Coordinated fillers ("A and B") are
|
|
||||||
one `group` referent filling the role, each member related to it by `member_of`,
|
|
||||||
from the member to the group.
|
|
||||||
|
|
||||||
### `relations`
|
### `relations`
|
||||||
|
|
||||||
Edges not owned by one predication. `relation` is a relation id (the inventory
|
Edges not owned by one predication. `relation` is a relation id (the inventory
|
||||||
|
|
|
||||||
|
|
@ -58,7 +58,7 @@ The utterance lowerer does not own:
|
||||||
- rendered audio
|
- rendered audio
|
||||||
- phoneme parity or synth-control truth
|
- phoneme parity or synth-control truth
|
||||||
|
|
||||||
The target-language cultural ontology profile was drafted to own the following; no such profile store exists now (address, politeness and register belong to speaker and scene state, and `data/ontology/ontology.cc` holds per-language lexicon, grammar and discourse choices):
|
The target-language cultural ontology profile owns:
|
||||||
|
|
||||||
- address, politeness, register, and social stance distinctions
|
- address, politeness, register, and social stance distinctions
|
||||||
- idiom and metaphor risks
|
- idiom and metaphor risks
|
||||||
|
|
@ -76,8 +76,8 @@ English remains a lowering target, not the pivot language. Existing flavored
|
||||||
English fixtures continue to be useful, but new multilingual fixtures should
|
English fixtures continue to be useful, but new multilingual fixtures should
|
||||||
show English as one realization in a set.
|
show English as one realization in a set.
|
||||||
|
|
||||||
The en-US choices are the `en-US` entries of `data/ontology/ontology.cc`
|
The initial cultural ontology profile is
|
||||||
(model-drafted seeds; read with `weksa ontology dump`).
|
`data/target-language-ontology/en-US.yaml`.
|
||||||
|
|
||||||
### Brazilian Portuguese
|
### Brazilian Portuguese
|
||||||
|
|
||||||
|
|
@ -94,10 +94,16 @@ The target lowerer should record:
|
||||||
defaults
|
defaults
|
||||||
- optional regional or cultural overlays when the source context earns them
|
- optional regional or cultural overlays when the source context earns them
|
||||||
|
|
||||||
No pt-BR ontology entry exists yet; the cut that adds pt-BR writes them to
|
The initial cultural ontology profile is
|
||||||
`data/ontology/ontology.cc`. Umbanda-related scenes, speakers, ritual contexts,
|
`data/target-language-ontology/pt-BR.yaml`.
|
||||||
or referents may later earn a regional or cultural variant, which must not
|
|
||||||
activate for generic Brazilian flavor.
|
Umbanda-related scenes, speakers, ritual contexts, or referents may add the
|
||||||
|
`data/target-language-ontology/pt-BR-umbanda.yaml` overlay. That overlay must
|
||||||
|
not activate for generic Brazilian flavor.
|
||||||
|
|
||||||
|
Regional or cultural variants should be selected through the overlay catalog at
|
||||||
|
`data/target-language-ontology/pt-BR-overlays.yaml`. Planned overlay slots are
|
||||||
|
not active profiles until reviewed examples and source context exist.
|
||||||
|
|
||||||
### Japanese
|
### Japanese
|
||||||
|
|
||||||
|
|
@ -113,8 +119,8 @@ The target lowerer should record:
|
||||||
- whether subject omission preserves or harms clarity
|
- whether subject omission preserves or harms clarity
|
||||||
- whether katakana, kanji, kana, or mixed script choices carry useful pressure
|
- whether katakana, kanji, kana, or mixed script choices carry useful pressure
|
||||||
|
|
||||||
The ja-JP choices are the `ja-JP` entries of `data/ontology/ontology.cc`
|
The initial cultural ontology profile is
|
||||||
(model-drafted seeds; read with `weksa ontology dump`).
|
`data/target-language-ontology/ja-JP.yaml`.
|
||||||
|
|
||||||
## Packet Shape
|
## Packet Shape
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,52 +0,0 @@
|
||||||
# Name-span annotation
|
|
||||||
|
|
||||||
How the name spans of an imported public-domain work are placed. A span has a
|
|
||||||
committed rule that produced it; this file is that rule.
|
|
||||||
|
|
||||||
## Why
|
|
||||||
|
|
||||||
- `pd-annotation-model-then-attest`: a labelled model places the spans of a
|
|
||||||
public-domain work; the operator may attest them afterwards.
|
|
||||||
- `leak-owned-by-flow-and-scorer-3`: spans are fixed with the text at intake and
|
|
||||||
written by no loop writer. The offsets are derived from markup the annotator
|
|
||||||
writes, never typed by a model (inline marking, as in GPT-NER, Wang et al.
|
|
||||||
2023); `CorpusStore::add` judges range, grapheme boundary, form equality and
|
|
||||||
overlap.
|
|
||||||
|
|
||||||
The work file (`weksa corpus import <work>`) is a transcription of a work in
|
|
||||||
hand: provenance, cast forms, lines, scene groups and the speaker each speech
|
|
||||||
prefix names. It holds no distributor header, licence or trademark text, and the
|
|
||||||
binary fetches nothing.
|
|
||||||
|
|
||||||
## Rubric r1
|
|
||||||
|
|
||||||
You are given a work's cast: each member's entity id and the forms the work spells that member with. For each numbered line, copy the line exactly, character for character, and wrap each place where the line names a cast member by one of that member's listed forms as [[<entity id>|<form>]]. Mark a form only where it refers to that cast member; a listed form used as an ordinary word, or naming someone else, stays unmarked. Do not mark pronouns, titles or descriptions unless the exact text is a listed form. When a form is listed for two members, mark the one the line refers to; when you cannot tell, leave it unmarked. Answer one JSON line per input line, in the order given, and nothing else: {"line": N, "marked": "..."}.
|
|
||||||
|
|
||||||
## Procedure
|
|
||||||
|
|
||||||
- One fresh model agent per work. Its brief is rubric r1, the cast as
|
|
||||||
`<entity id>: <form>, <form>` lines and the work's lines as `<n><TAB><text>`;
|
|
||||||
nothing else (no store, no loop, no other answer).
|
|
||||||
- Import with `--model-family` and `--generator '<annotator model id> (name-span
|
|
||||||
rubric r1, docs/name-span-annotation.md)'`. An operator-written or
|
|
||||||
operator-reviewed answer imports with `--operator`.
|
|
||||||
- Import is all or nothing: every entity and item passes the real store doors on
|
|
||||||
scratch copies before the first write, so a refusal (a parse failure, a span
|
|
||||||
the store judges, a work that repeats a line under case or width folding)
|
|
||||||
leaves the stores untouched. Re-draw the whole answer once, log it, and import
|
|
||||||
again. A work file repeating a line's text is refused naming both lines; leave
|
|
||||||
that line out of the answer file (the rest imports with its numbering intact),
|
|
||||||
since a repeated line cannot be two corpus items.
|
|
||||||
- A line the answer does not cover is not imported by that run; a later run over
|
|
||||||
other lines of the work does not collide (keys are `item:<slug>:<n>`).
|
|
||||||
- A line whose own text holds `[[` or `]]` cannot be marked and is refused; it
|
|
||||||
can still enter by hand through `weksa corpus add --name`.
|
|
||||||
|
|
||||||
## Label scope
|
|
||||||
|
|
||||||
`names_by` labels the spans only, not the line or its speaker. One label per
|
|
||||||
item: a later attestation would be a superseding item with the same text and
|
|
||||||
group, but no command writes one yet; question `pd-name-span-attestation` owns
|
|
||||||
that door.
|
|
||||||
`None` means unrecorded (the cast seeds, `weksa corpus add`, the Dao De Jing
|
|
||||||
bed), not attested.
|
|
||||||
|
|
@ -1,195 +0,0 @@
|
||||||
# Prior art and plan
|
|
||||||
|
|
||||||
Meta asked (#weksa, 2026-11-13) for scaffolding to stand on instead of
|
|
||||||
building from base principles. This is a survey of what exists and a plan that
|
|
||||||
borrows it. Every source below was read in full text unless marked **snippet**
|
|
||||||
(seen only in a search result) or **not read**. Research only: nothing here is
|
|
||||||
built, and no new loop machinery is proposed.
|
|
||||||
|
|
||||||
## What exists
|
|
||||||
|
|
||||||
**Semantic backbones: none carries the part we care about.**
|
|
||||||
|
|
||||||
- AMR ([Banarescu 2013](https://aclanthology.org/W13-2322.pdf)): no politeness,
|
|
||||||
speech-act or pragmatics text at all; it also does not distinguish real from
|
|
||||||
hypothetical events (its Section 4).
|
|
||||||
- UMR ([Van Gysel 2021](https://par.nsf.gov/servlets/purl/10288269)): adds
|
|
||||||
document-level coreference, temporal and modal-dependency graphs. The paper
|
|
||||||
has no politeness, addressee or register layer; the only speaker-like object
|
|
||||||
is the author node. Annotation agreement was still "awaiting confirmation".
|
|
||||||
- Universal Dependencies: the `Polite` feature has four values (`Infm`, `Form`,
|
|
||||||
`Elev`, `Humb`; [feature page](https://universaldependencies.org/u/feat/Polite.html)),
|
|
||||||
but the Japanese GSD treebank carries no `Polite`
|
|
||||||
([treebank page](https://universaldependencies.org/treebanks/ja_gsd/index.html)).
|
|
||||||
The page says Japanese *teineigo* shares the value `Form` with formal register
|
|
||||||
although it is a different axis. UD is a parse scheme, not a meaning scheme.
|
|
||||||
- Grammatical Framework RGL: 38 languages, Japanese present
|
|
||||||
([synopsis](https://www.grammaticalframework.org/lib/doc/synopsis/index.html),
|
|
||||||
[status](https://www.grammaticalframework.org/lib/doc/status.html)); the source
|
|
||||||
declares `Style = Plain | Resp`, so two politeness levels and no honorific
|
|
||||||
paradigm
|
|
||||||
([ResJpn.gf](https://raw.githubusercontent.com/GrammaticalFramework/gf-rgl/master/src/japanese/ResJpn.gf)).
|
|
||||||
LGPL/BSD, application grammars may take any licence
|
|
||||||
([LICENSE](https://raw.githubusercontent.com/GrammaticalFramework/gf-rgl/master/LICENSE)).
|
|
||||||
- Abstract Wikipedia / Wikifunctions: encyclopedic content, 2026 soft launch in
|
|
||||||
three languages, no politeness
|
|
||||||
([Meta page](https://meta.wikimedia.org/wiki/Abstract_Wikipedia)). The
|
|
||||||
Wikifunctions NLG and Japanese documentation were **not read**.
|
|
||||||
|
|
||||||
**Typology.**
|
|
||||||
|
|
||||||
- WALS (CC BY 4.0, CLDF at <https://github.com/cldf-datasets/wals>): the only
|
|
||||||
politeness feature is 45A, pronoun politeness
|
|
||||||
([chapter](https://wals.info/chapter/45)); Japanese is coded value 4 ([datapoint](https://wals.info/valuesets/45A-jpn)), pronoun
|
|
||||||
avoidance. No feature for verbal honorifics or register.
|
|
||||||
- Grambank (CC BY 4.0, <https://github.com/grambank/grambank>): GB415 is the
|
|
||||||
second-person politeness distinction; I found no verbal-honorific feature
|
|
||||||
(loose name match, not exhaustive). The 195-feature / 2,467-variety size is
|
|
||||||
**snippet**; Japanese's GB415 value was **not looked up**.
|
|
||||||
|
|
||||||
**Formality control in MT: the closest prior art, and it is a sentence-level
|
|
||||||
label.**
|
|
||||||
|
|
||||||
- [Sennrich 2016](https://aclanthology.org/N16-1005.pdf): a side-constraint
|
|
||||||
token controls T/V; forced-polite output was 96% polite or neutral.
|
|
||||||
- [Feely 2019](https://aclanthology.org/D19-5203.pdf): the same for EN-JA with
|
|
||||||
three levels (informal, polite, formal); labels by rule plus a fastText
|
|
||||||
classifier (F1 0.83-0.86 on 150 hand-labelled sentences); formal text is only
|
|
||||||
0.7-1.6% of training data.
|
|
||||||
- [CoCoA-MT](https://arxiv.org/pdf/2205.04022) (CDLA-Sharing-1.0, repo
|
|
||||||
<https://github.com/amazon-science/contrastive-controlled-mt>): 1,000 train and
|
|
||||||
600 test segments for Japanese with contrastive formal/informal references and
|
|
||||||
phrase-level markers. Two baseline systems' EN-JA outputs, labelled by
|
|
||||||
professionals, came out formal 29.0% / 73.8% and "other" about 20-24%; EN-JA
|
|
||||||
inter-annotator agreement 0.82, the lowest of the six languages.
|
|
||||||
- [Marrese-Taylor 2023](https://aclanthology.org/2023.wmt-1.49.pdf): GPT-3 and
|
|
||||||
ChatGPT prompted reach about 90% formality-control accuracy on EN-JA, which
|
|
||||||
the authors say beats fine-tuned MT; classifier and data at
|
|
||||||
<https://github.com/epochx/japanese-formality> (no licence reported).
|
|
||||||
- [IWSLT 2023 formality track](https://aclanthology.org/2023.iwslt-1.1.pdf): no
|
|
||||||
Japanese; metrics matched-accuracy (phrase markers) and classifier-accuracy.
|
|
||||||
Niu 2017 is read ([D17-1299](https://aclanthology.org/D17-1299.pdf)); Niu 2019
|
|
||||||
and Rippeth 2022 (arXiv 2205.06644, abstract only) are **not read**.
|
|
||||||
- None of these takes a speaker-state or addressee relation. The label is one
|
|
||||||
bit or three levels attached to a source sentence.
|
|
||||||
|
|
||||||
**Grammar-in-context (for conlang targets).**
|
|
||||||
|
|
||||||
- [MTOB](https://arxiv.org/pdf/2309.16575) (MIT code, CC BY 4.0 grammar book):
|
|
||||||
Claude 2 with word list plus retrieved sentences plus long grammar 44.7 / 45.8
|
|
||||||
chrF against a human learner at 51.6 / 57.0.
|
|
||||||
- [Aycock 2025](https://arxiv.org/pdf/2409.19151): most of the gain comes
|
|
||||||
from the book's parallel examples (Gemini-1.5-Flash eng->kgv chrF++ 30.8 with
|
|
||||||
examples only against 34.4 whole book and 11.0 zero-shot; explanations alone
|
|
||||||
22.6). Kalamang is a natural language, so transfer to a conlang is assumed.
|
|
||||||
- [Gemini 1.5 report](https://arxiv.org/pdf/2403.05530): full book 5.46 against a
|
|
||||||
human learner's 5.52 on a 0-6 human scale.
|
|
||||||
Suggests, for a conlang target (assumed to transfer), worked paired examples first and grammar
|
|
||||||
prose second.
|
|
||||||
|
|
||||||
**Prompt optimisation and evaluation.**
|
|
||||||
|
|
||||||
- [DSPy](https://arxiv.org/pdf/2310.03714) (MIT), [MIPRO](https://arxiv.org/pdf/2406.11695),
|
|
||||||
[OPRO](https://arxiv.org/pdf/2309.03409) (Apache-2.0): DSPy compiles take
|
|
||||||
"10-20 trials over 150-300 validation examples"; MIPRO's task sets run from 75
|
|
||||||
train (Iris) to 500 per task. MIPRO found bootstrapped demos gave the biggest
|
|
||||||
gain and tests with Wilcoxon over 5 runs. OPRO discusses overfitting to the
|
|
||||||
training set.
|
|
||||||
- [Koehn 2004](https://aclanthology.org/W04-3250.pdf): paired bootstrap;
|
|
||||||
against a 30,000-sentence reference, a 2.0-point BLEU gap was detected at 95%
|
|
||||||
significance in 31% of runs at n=100 and 65% at n=300; a 5.1-point gap in 97%
|
|
||||||
at n=100. The paper suggests n=100 might be too small.
|
|
||||||
- [Dror 2018](https://aclanthology.org/P18-1128.pdf): paired bootstrap is
|
|
||||||
"less effective for small test sets"; approximate randomization has been used
|
|
||||||
for test sets above 20 points; only 3 of 110 surveyed ACL 2017 papers
|
|
||||||
corrected for multiple comparisons.
|
|
||||||
- [Kocmi 2021](https://arxiv.org/pdf/2107.10821): pairwise accuracy against
|
|
||||||
humans COMET 83.4, BLEURT 80.0, chrF 75.6, BLEU 74.6; recommends COMET
|
|
||||||
primary, a string metric secondary and a paired test; observes "a slight drop
|
|
||||||
in metric ranks for some pretrained metric" against ChrF for non-Latin and
|
|
||||||
logogram targets and says more analysis would be required.
|
|
||||||
- [chrF](https://aclanthology.org/W15-3049.pdf): validated on seven targets, Japanese
|
|
||||||
not among them.
|
|
||||||
|
|
||||||
## What we borrow
|
|
||||||
|
|
||||||
| Need | Borrow | Instead of |
|
|
||||||
|---|---|---|
|
|
||||||
| Politeness vocabulary for the packet's register field | UD `Polite` values (Infm, Form, Elev, Humb) as the label set, and Feely's three-level Japanese split | inventing level names |
|
|
||||||
| Modal/author structure | UMR's conceiver and modal-dependency idea for evidence/stance fields | a bespoke modality scheme |
|
|
||||||
| Japanese test of register | CoCoA-MT JA set (600 contrastive test segments, phrase markers), KeiCO for relation-labelled Japanese, and the Japanese-formality classifier | hand-written register references only |
|
|
||||||
| Conlang scaffolding | MTOB recipe: worked paired examples first, word list, grammar prose after (shown on Kalamang, a natural language; transfer to a conlang is an assumption) | long grammar prompts alone |
|
|
||||||
| Typology priors | WALS 45A and Grambank GB415 to seed which targets have a politeness axis | per-language guesses |
|
|
||||||
| Significance | paired approximate randomization over items; chrF with an embedding metric | reading small differences as results |
|
|
||||||
| Optimiser | DSPy/MIPRO only once a scoring set of at least 75 items exists (MIPRO's smallest task), more for stable compiles; the 35-item sets are below that | the golf loop growing a private optimiser |
|
|
||||||
|
|
||||||
We do not adopt AMR, UMR, UD or GF as the packet. None represents the speaker's
|
|
||||||
relation to the addressee, which the packet's scene state does. The packet stays
|
|
||||||
ours; it borrows vocabulary.
|
|
||||||
|
|
||||||
## What is actually new
|
|
||||||
|
|
||||||
Prior work already conditions honorific choice on the speaker-addressee
|
|
||||||
relation or status in narrower forms: KeiCO (Liu and Kobayashi 2022, 10,007
|
|
||||||
examples, annotates the speaker-listener social relationship; read through
|
|
||||||
Marrese-Taylor 2023 and the abstract), Korean context-aware NMT that infers the
|
|
||||||
relation from surrounding sentences
|
|
||||||
([MDPI](https://www.mdpi.com/2079-9292/10/13/1589), snippet), Brown-Levinson
|
|
||||||
generation that takes power and distance as parameters (Walker et al., not
|
|
||||||
read), and a 2026 translator given a known addressee
|
|
||||||
([Cihlar et al.](https://aclanthology.org/2026.iaai-1.3), snippet only). What I
|
|
||||||
did not find is the whole chain: persona state (status, restraint, stance) to a
|
|
||||||
register decision for an LLM lowerer, measured against a prompted baseline.
|
|
||||||
That is the claim, and it is relative to what I searched; closer work may
|
|
||||||
exist.
|
|
||||||
|
|
||||||
Uncontrolled commercial MT left 20-24% of EN-JA outputs inconsistent in
|
|
||||||
register (CoCoA-MT Table 4, two generic systems, no formality instruction). That
|
|
||||||
shows register is unstable without control, not how a prompted LLM behaves.
|
|
||||||
|
|
||||||
## Smallest end-to-end slice
|
|
||||||
|
|
||||||
One path, existing machinery, ja-JP only, no new loop:
|
|
||||||
|
|
||||||
1. A packet with scene state (speaker, addressee, relation) goes through the
|
|
||||||
current `weksa loop` encode and lowering to Japanese.
|
|
||||||
2. The same packet under two relations (to a superior and to an intimate) must
|
|
||||||
produce the two UD `Polite` levels; check with the Japanese formality
|
|
||||||
classifier plus a native-reader pass on the 35-item Validation corpus per
|
|
||||||
language (see the map's recalibration blocks).
|
|
||||||
3. Score the register decision as a classification (matched accuracy), not only
|
|
||||||
chrF. CoCoA-MT's JA references give an outside check on the classifier, a
|
|
||||||
weak one: its reference-based segment classifier on EN-JA had precision
|
|
||||||
0.71 and 0.69 and recall 0.43 and 0.54 in the paper's Table 5.
|
|
||||||
|
|
||||||
Out of scope for the slice: conlang targets, optimisers, a second language.
|
|
||||||
|
|
||||||
## What good enough means
|
|
||||||
|
|
||||||
- Register accuracy: report matched accuracy for ours and for a plain prompt
|
|
||||||
that is given the relation in words, on the same items, with paired
|
|
||||||
intervals. We measure that baseline ourselves; Marrese-Taylor's roughly 90%
|
|
||||||
is classifier accuracy (0.83-0.86 by phrase-marker matching) for binary
|
|
||||||
formal/informal control with the label named in the prompt, an upper-region
|
|
||||||
reference and not our baseline. The claim is non-inferiority with a stated
|
|
||||||
margin. Superiority is claimed only if enough items are discordant (about 6
|
|
||||||
or more, all in one direction, at n=35); against a baseline near 90%, n=35
|
|
||||||
cannot show it, and the result is then "not detected".
|
|
||||||
- Meaning is not lost: chrF and meaning cosine are not detectably below the
|
|
||||||
plain prompt on the same items, under the same paired test.
|
|
||||||
- Honest sample size: 35 items per language detects only large gaps, so results
|
|
||||||
carry intervals and "not detected" stands as an outcome. Small gaps are not
|
|
||||||
claims. Koehn's figures are for sentence-level BLEU on 100-3000-sentence
|
|
||||||
blocks and do not transfer to a binary register metric on 35 items; power
|
|
||||||
for that is computed directly from the discordant-pair count.
|
|
||||||
- A native-reader check on register for a sample, because every automatic metric
|
|
||||||
here was validated on languages other than the persona register we care about.
|
|
||||||
|
|
||||||
## Open
|
|
||||||
|
|
||||||
- Which persona fields (status, restraint, stance) are in the packet today
|
|
||||||
versus planned: this doc does not decide that; `docs/interlingua-standard.md`
|
|
||||||
owns it.
|
|
||||||
- Licences not checked: the UMR dataset, the other four Japanese UD treebanks,
|
|
||||||
CoCoA-MT repo licence ambiguity (paper says CDLA-Sharing-1.0, GitHub says
|
|
||||||
NOASSERTION).
|
|
||||||
|
|
@ -65,7 +65,7 @@ Verified licence facts are few. Discord data excluded by policy; not read.
|
||||||
- PersonaChat (Zhang et al. 2018): ParlAI project page, archived 2020-08-01, states no licence, only "You may continue to use the data without issue". [V] https://parl.ai/projects/personachat/ ; paper https://arxiv.org/abs/1801.07243 says data released open source via ParlAI; licence text [GAP]. Crowdworker-written, persona-statement-conditioned chit-chat [B].
|
- PersonaChat (Zhang et al. 2018): ParlAI project page, archived 2020-08-01, states no licence, only "You may continue to use the data without issue". [V] https://parl.ai/projects/personachat/ ; paper https://arxiv.org/abs/1801.07243 says data released open source via ParlAI; licence text [GAP]. Crowdworker-written, persona-statement-conditioned chit-chat [B].
|
||||||
- LIGHT: ParlAI page states no licence; contents: 663 locations, 3,462 objects, 1,755 character types, 11,000 episodes; WILD 41,131 training episodes from game deployment (human players, so a different provenance than crowdwork). [V] https://parl.ai/projects/light/ . Licence [GAP: check ParlAI repo].
|
- LIGHT: ParlAI page states no licence; contents: 663 locations, 3,462 objects, 1,755 character types, 11,000 episodes; WILD 41,131 training episodes from game deployment (human players, so a different provenance than crowdwork). [V] https://parl.ai/projects/light/ . Licence [GAP: check ParlAI repo].
|
||||||
- Cornell Movie-Dialogs Corpus: licence/terms page fetch failed (certificate rejected) https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html . [GAP] Underlying scripts are copyrighted film text [B]; dataset licence unverified.
|
- Cornell Movie-Dialogs Corpus: licence/terms page fetch failed (certificate rejected) https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html . [GAP] Underlying scripts are copyrighted film text [B]; dataset licence unverified.
|
||||||
- Project Gutenberg: "vast majority" of eBooks public domain in the US, so no one can grant or withhold permission for uses; PG cannot sublicense copyrighted items; subject to its Terms of Use, which the permission page links and does not summarise; no statement on ML/bulk use on that page. [V] https://www.gutenberg.org/policy/permission.html . Terms-of-Use text on bulk/automated access and the trademark licence: read 2026-11-13, see `2026-11-13-gutenberg-terms.md`. Items public domain in the US may not be elsewhere (same page).
|
- Project Gutenberg: "vast majority" of eBooks public domain in the US, so no one can grant or withhold permission for uses; PG cannot sublicense copyrighted items; subject to its Terms of Use, which the permission page links and does not summarise; no statement on ML/bulk use on that page. [V] https://www.gutenberg.org/policy/permission.html . Terms-of-Use text on bulk/automated access and the trademark licence: [GAP, not read]. Items public domain in the US may not be elsewhere (same page).
|
||||||
- Google Synthetic-Persona-Chat (model-generated, persona-chat-style): CC BY-NC? listed as cc-by-4.0 on HF. [V search-only] https://huggingface.co/datasets/google/Synthetic-Persona-Chat
|
- Google Synthetic-Persona-Chat (model-generated, persona-chat-style): CC BY-NC? listed as cc-by-4.0 on HF. [V search-only] https://huggingface.co/datasets/google/Synthetic-Persona-Chat
|
||||||
- Operator-authored or model-generated corpus: no licence question beyond provider terms [B, unread this pass]. Problems with evaluating on model-generated text: see 2 (self-bias, preference leakage, stylistic homogeneity: Xu et al. 2025; Li et al. 2025). The same Xu et al. finding says it extends to open-ended generation.
|
- Operator-authored or model-generated corpus: no licence question beyond provider terms [B, unread this pass]. Problems with evaluating on model-generated text: see 2 (self-bias, preference leakage, stylistic homogeneity: Xu et al. 2025; Li et al. 2025). The same Xu et al. finding says it extends to open-ended generation.
|
||||||
- Expressive-register-annotated English corpora (hedging, humour, affect, politeness labelled), with permissive terms: [GAP] not searched beyond the list above (e.g. Stanford Politeness Corpus, GYAFC formality, Enron etc. are [B] candidates, unverified, GYAFC is known to carry Yahoo Answers terms [B]).
|
- Expressive-register-annotated English corpora (hedging, humour, affect, politeness labelled), with permissive terms: [GAP] not searched beyond the list above (e.g. Stanford Politeness Corpus, GYAFC formality, Enron etc. are [B] candidates, unverified, GYAFC is known to carry Yahoo Answers terms [B]).
|
||||||
|
|
@ -83,6 +83,6 @@ Verified licence facts are few. Discord data excluded by policy; not read.
|
||||||
- No per-metric human-correlation numbers for hedging, humour, affect, stance, persona voice; nearest is formality (Briakou 2021). COMET, BLEURT, MAUVE correlations not obtained.
|
- No per-metric human-correlation numbers for hedging, humour, affect, stance, persona voice; nearest is formality (Briakou 2021). COMET, BLEURT, MAUVE correlations not obtained.
|
||||||
- No source on surface-form smuggling through an LLM bottleneck or its defences (vocab restriction, paraphrase invariance).
|
- No source on surface-form smuggling through an LLM bottleneck or its defences (vocab restriction, paraphrase invariance).
|
||||||
- No source on schema/inventory growth from generation failures, or its overfitting controls.
|
- No source on schema/inventory growth from generation failures, or its overfitting controls.
|
||||||
- Cornell, LIGHT, PersonaChat licence texts not obtained.
|
- Cornell, LIGHT, PersonaChat licence texts not obtained; Gutenberg ToU on bulk use not read.
|
||||||
- No realiser or evaluation protocol for constructed languages from a meaning representation.
|
- No realiser or evaluation protocol for constructed languages from a meaning representation.
|
||||||
- Several arXiv PDFs (Briakou, Zheng, STEL, LUAR/StyleDistance) fetched as binary; numbers above come from the ar5iv/arXiv-HTML versions where marked, otherwise search summaries.
|
- Several arXiv PDFs (Briakou, Zheng, STEL, LUAR/StyleDistance) fetched as binary; numbers above come from the ar5iv/arXiv-HTML versions where marked, otherwise search summaries.
|
||||||
|
|
|
||||||
|
|
@ -1,59 +0,0 @@
|
||||||
# Project Gutenberg terms: bulk access and header stripping
|
|
||||||
|
|
||||||
Read 2026-11-13 for follow-up `gutenberg-terms` (campaign `weksa-lowering`),
|
|
||||||
ahead of cut `pd-importer`. Pages: gutenberg.org `policy/permission.html`,
|
|
||||||
`policy/terms_of_use.html`, `policy/robot_access.html`, `policy/license.html`.
|
|
||||||
The quotes were checked against the live HTML of each page by a second reader.
|
|
||||||
`pd-importer`'s Imagination re-reads the pages before it fixes a fetch path,
|
|
||||||
since they change.
|
|
||||||
|
|
||||||
## Findings
|
|
||||||
|
|
||||||
- **Automated access to the main site is refused.** Terms of Use: "This website
|
|
||||||
is intended for human users only"; "Any perceived use of automated tools to
|
|
||||||
access this website will result in a temporary or permanent block". OPDS
|
|
||||||
feeds: no more requests than a browser user would make; heavy requesters are
|
|
||||||
blocked by IP.
|
|
||||||
- **Bulk has two sanctioned paths.** Terms of Use: take bulk downloads "from
|
|
||||||
one of our mirrors, not from the main site". The robot-access page names the
|
|
||||||
`robot/harvest` wget endpoint (`-w 2`, filterable by file type or language)
|
|
||||||
and recommends a private mirror for a full local copy; it suggests checking
|
|
||||||
periodically for changes. Catalog data are offered as
|
|
||||||
machine-readable offline catalogs, "granted to the public domain".
|
|
||||||
- **Header and licence stripping.** Licence section 1.C: for a work not
|
|
||||||
protected by US copyright, with the user in the US, the Foundation does "not
|
|
||||||
claim a right to prevent you from copying, distributing, performing,
|
|
||||||
displaying or creating derivative works", "as long as all references to
|
|
||||||
Project Gutenberg are removed", the condition holding for a user in the US.
|
|
||||||
Section 1.E opens "Unless you have removed all references to Project
|
|
||||||
Gutenberg", and 1.E.1 to 1.E.7 (including 1.E.4, do not detach the licence)
|
|
||||||
attach to a work on which the phrase "Project Gutenberg" appears or with which
|
|
||||||
it is associated, so a text with every reference removed is outside 1.E.
|
|
||||||
1.E.2 is not that grant: it says a PG eBook derived from texts not restricted
|
|
||||||
by US copyright may be copied and distributed in the US without fee. The
|
|
||||||
trademark is registered; non-commercial use of the name needs no permission,
|
|
||||||
commercial use does.
|
|
||||||
- **No statement on machine learning or AI use** on any of the four pages.
|
|
||||||
- **US public domain only.** The Foundation makes no claim about copyright
|
|
||||||
outside the US (1.D). A work free in the US may not be free elsewhere, and
|
|
||||||
Gutenberg's own catalog is not a per-work clearance.
|
|
||||||
|
|
||||||
## What this decides for `pd-importer`
|
|
||||||
|
|
||||||
- Fetch from a mirror or an offline dump. The Terms of Use footnote counts "many
|
|
||||||
books" as more than about 100 a day and sends those to mirrors, while
|
|
||||||
`robot/harvest` is a `www.gutenberg.org` URL: use it (with `-w 2`) only for a
|
|
||||||
handful of works. Never crawl `gutenberg.org` pages or OPDS from the loop host.
|
|
||||||
- Strip the whole header and footer, every Project Gutenberg reference and the
|
|
||||||
licence text from the stored text; keep provenance as the ebook number and
|
|
||||||
edition in the item's source field, which is Weksa's own record and not a use
|
|
||||||
of the mark. This is the path `docs/targets/dao-de-jing-test-bed.md` already
|
|
||||||
takes. The remaining open question there, what counts as a "reference", stays
|
|
||||||
open: the safe reading is that the ebook number in a provenance field is a
|
|
||||||
catalogue identifier, not a reference inside the work, but nothing read here
|
|
||||||
rules on it.
|
|
||||||
- Per-work check stays with the importer: the work's author and first
|
|
||||||
publication date against US public-domain rules, recorded on the item. The
|
|
||||||
Foundation's catalogue does not supply it. A work Gutenberg posts with the
|
|
||||||
copyright holder's permission (1.E.3) is out of scope.
|
|
||||||
- No importer code in this note.
|
|
||||||
|
|
@ -1,116 +0,0 @@
|
||||||
# Scene annotation
|
|
||||||
|
|
||||||
How the keeper scenes of `data/original/speaker-state.cc` are labelled. A scene
|
|
||||||
label has a committed rule that produced it; this file is that rule.
|
|
||||||
|
|
||||||
## Why
|
|
||||||
|
|
||||||
Two findings forced a written rule.
|
|
||||||
|
|
||||||
- `annotation-rank-from-register`: an annotator who reads the line's register
|
|
||||||
(politeness forms, honorifics, endings) to decide the speaker's rank hands the
|
|
||||||
lowering its own answer back, because register is what the lowering must
|
|
||||||
produce from the scene.
|
|
||||||
- The addressee gate (`with_scene` in `crates/weksa/src/validate.rs`): a scene is
|
|
||||||
read only when the packet names a discourse addressee. A line that addresses no
|
|
||||||
one reads `NoAddressee` and no Scene entry selects, so labels on narration are
|
|
||||||
inert and the only labels that bite are those on lines said to someone.
|
|
||||||
|
|
||||||
The earlier labels followed two briefs that were never committed and that
|
|
||||||
disagreed with each other (the relation to the reader, and a situation read with
|
|
||||||
the register ignored). The rubric below replaces both for every live group.
|
|
||||||
|
|
||||||
## Provenance
|
|
||||||
|
|
||||||
Cut `weksa-lowering:cut_spec:cut-scene-in-group.r1`, ruling
|
|
||||||
`weksa-lowering:ruling:scene-rubric-whole-bed`: all 98 live Train and Validation
|
|
||||||
groups of `data/original` receive a new scene revision from annotator A under
|
|
||||||
rubric r1, generator `claude-sonnet-5-5 (Weksa Hands, scene rubric r1,
|
|
||||||
docs/scene-annotation.md)`. Earlier revisions stay as provenance. The rubric uses
|
|
||||||
no corpus line as an example on purpose: a labelled example would make the blind
|
|
||||||
check against the old labels circular.
|
|
||||||
|
|
||||||
A later rubric revision gets a new section and a new generator string; r1 is
|
|
||||||
never edited.
|
|
||||||
|
|
||||||
## Rubric r1
|
|
||||||
|
|
||||||
The speaker of every line is the keeper, a quiet caretaker of a house and garden. First decide whether the line addresses a listener: it asks them something, asks them to do something, tells them about themselves, wishes something for them, or speaks to them as you. A line that addresses no one (narration, description, a remark about the world or about other people, a thought or wish with no one addressed) is told to the reader, a stranger: answer addressed false, relative_status level, intimacy 500, in_group false. People named inside a line are not its listener.
|
|
||||||
|
|
||||||
For a line that addresses a listener, answer addressed true and judge the keeper's relation to that listener from the situation the line states or implies, never from its register: politeness forms, honorifics, sentence endings, contractions, please and casual particles are what the lowering must produce from the scene, so they cannot decide it. relative_status: above when the situation gives the keeper standing over the listener (instructing, advising or permitting as elder, host, employer or teacher); below when it gives the listener standing over the keeper (the keeper serves, thanks, or asks a favour of a guest, employer, elder or official); level when it shows neither.
|
|
||||||
|
|
||||||
intimacy, 0 to 1000: 500 when the line shows nothing; up to 800 when it shows shared daily life (the listener's body, sleep or habits, the household's routine); above 800 only for a stated close bond; down to 200 for a stated first meeting or a service relation. in_group: true only when the line shows the listener belongs to the keeper's household, family or working circle, stated or plainly implied; a request, a question or a kind wish alone is not enough; otherwise false. Answer one JSON line per group, in the order given and nothing else: {"group": G, "addressed": true|false, "relative_status": S, "intimacy": N, "in_group": true|false}.
|
|
||||||
|
|
||||||
## Procedure
|
|
||||||
|
|
||||||
- Annotator A (committed) and annotator B (measurement only) are separate fresh
|
|
||||||
model agents, one pair per language (49 live groups each). Each sees only the
|
|
||||||
text of Rubric r1 above and that language's sentences as `group<TAB>"text"`,
|
|
||||||
shuffled by a fixed seed. Neither sees an existing scene, the other annotator's
|
|
||||||
answer, the ontology, which items are old or new, or any Test text.
|
|
||||||
- An output that does not parse or misses a group is re-drawn once and logged.
|
|
||||||
- A's labels are written with `weksa scene put` (one revision per group,
|
|
||||||
`--to unnamed --inventory-rev 4`). `addressed` is not stored: no Scene feature
|
|
||||||
carries it.
|
|
||||||
- Disagreements are reported, never adjudicated.
|
|
||||||
|
|
||||||
## Agreement at r1 (this cut, 98 live groups, 49 per language)
|
|
||||||
|
|
||||||
Four fresh agents drew the labels (A-en, A-ja, B-en, B-ja); the briefs were the
|
|
||||||
rubric section plus one language's shuffled sentences (SHA-256 of the brief
|
|
||||||
bodies: en-US `db087476...dbc04e`, ja-JP `c5b81b27...97488c`; no Test text in
|
|
||||||
either). No draw was repeated. Annotator model: `claude-sonnet-5-5`.
|
|
||||||
|
|
||||||
A against B (B measures only): pooled `addressed` equal 96/98 (kappa 0.96),
|
|
||||||
`in_group` equal 93/98 (kappa 0.79), `relative_status` equal 89/98, intimacy
|
|
||||||
within 150 on 98/98. en-US: addressed 47/49, in_group 47/49 (0.86),
|
|
||||||
relative_status 45/49. ja-JP: addressed 49/49, in_group 46/49 (0.64),
|
|
||||||
relative_status 44/49.
|
|
||||||
|
|
||||||
Blind check, A against the 56 older labels: `relative_status` equal 49/56,
|
|
||||||
`in_group` equal 53/56 (kappa 0.64), intimacy within 150 on 55/56. Three
|
|
||||||
in_group differences: `original-ja-shimoyado-2` and `original-ja-sakana` (both
|
|
||||||
narrations, true to false) and `original-ja-asa` (an instruction to a household
|
|
||||||
member, false to true).
|
|
||||||
|
|
||||||
In_group true across the 98: 42 before, 15 after; every unaddressed line is now
|
|
||||||
false (it was true on 17 of the 21 unaddressed vgrow groups). Disagreements are
|
|
||||||
reported, not adjudicated.
|
|
||||||
|
|
||||||
## Reproducing the cut
|
|
||||||
|
|
||||||
`scripts/scene-annotation/` holds the standard-library scripts of this cut, run
|
|
||||||
from the repo root with `WEKSA` (the binary) and `SCENE_SCRATCH` (a directory
|
|
||||||
that is never committed): `build_briefs.py` writes the two annotator briefs from
|
|
||||||
the Rubric r1 section and prints each brief's Test-leak count (must be 0) and
|
|
||||||
sha256 (it reproduces the briefs of this cut byte for byte); `put_labels.py`
|
|
||||||
puts A's `A-en.jsonl` and `A-ja.jsonl`; `agreement.py` prints the tables above and
|
|
||||||
must run before the puts, against the store at 76fa685, because the older labels
|
|
||||||
are then its latest revisions (it also reads the round-1 encoder answers of
|
|
||||||
`runs/growth-base` through `ROUND1`). `DIGEST_OF_THE_RUBRIC` and
|
|
||||||
`DIGEST_OF_THE_EARLIER_SCENES` in `crates/weksa/tests/original_corpus.rs` pin the
|
|
||||||
rubric text and the 113 earlier revisions.
|
|
||||||
|
|
||||||
## Known limits of r1 labels
|
|
||||||
|
|
||||||
Found by Soul's pass on this cut (`weksa-lowering:verdict:cut-scene-in-group.s1`).
|
|
||||||
They are not fixed in r1; a reviewer's pass over the labels is follow-up
|
|
||||||
`weksa-lowering:follow_up:scene-labels-reader-review`.
|
|
||||||
|
|
||||||
- The new labels change which ontology entries are selected for 37 groups. Eight
|
|
||||||
addressed lines move onto the casual in-group entries (en-US:12: lantern,
|
|
||||||
broom, mirror, seedling; ja-JP:12: furo, asa, irori, yurikago), while their
|
|
||||||
references are polite requests. The reader default (level, 500, false) is
|
|
||||||
exactly the condition of ja-JP:11.
|
|
||||||
- `relative_status` tracks request form for some lines: imperatives read above,
|
|
||||||
favour questions read level or below, so the same household request gets
|
|
||||||
different standing in two forms (hane above/650/true, kabin below/500/false;
|
|
||||||
B gave both level/false). Form is what signals the speech act, which is what
|
|
||||||
part 2 asks for, and register is what part 2 forbids.
|
|
||||||
- `in_group` is true on some requests alone (lantern, mulch, stool, hane) and on
|
|
||||||
a line whose only family member is named inside it (original-ja-ane), though r1
|
|
||||||
says a request or a person named inside the line is not enough. A and B
|
|
||||||
agreeing does not clear this.
|
|
||||||
- A run opened on the old speaker-state refuses to continue after merge:
|
|
||||||
`runs/growth-base` pins the store of `/var/lib/mind-weksa-self/Weksa-growth`
|
|
||||||
(main), and that checkout must not be pulled while the run is wanted.
|
|
||||||
|
|
@ -15,18 +15,17 @@ plus a separately recorded chosen gloss per lowering).
|
||||||
- `data/daodejing/`: its own stores (inventory revisions 1 to 3, one entity, five
|
- `data/daodejing/`: its own stores (inventory revisions 1 to 3, one entity, five
|
||||||
corpus items, five packets), apart from the production stores in `data/`.
|
corpus items, five packets), apart from the production stores in `data/`.
|
||||||
Revision 3 is `weksa::daodejing::revision()`; the packets are
|
Revision 3 is `weksa::daodejing::revision()`; the packets are
|
||||||
`weksa::daodejing::packets()`. The scene revision (`speaker_scene_revision()`, also
|
`weksa::daodejing::packets()`. The scene revision (`scene_revision()`, also
|
||||||
numbered 3) is not applied in these stores and will need renumbering when it
|
numbered 3) is not applied in these stores and will need renumbering when it
|
||||||
is applied anywhere both meet.
|
is applied anywhere both meet.
|
||||||
- Rebuild: `weksa --corpus C --inventory I --entities E` with `inventory seed`,
|
- Rebuild: `weksa --corpus C --inventory I --entities E` with `inventory seed`,
|
||||||
`inventory proof`, `inventory dao`, `entity put entity:laozi ...`,
|
`inventory proof`, `inventory dao`, `entity put entity:laozi ...`,
|
||||||
`corpus add ... --language en-US --source public-domain` per item, then `packet dao <store>`.
|
`corpus add ... --source public-domain` per item, then `packet dao <store>`.
|
||||||
- Source: Project Gutenberg eBook 216, "The Tao Teh King, or the Tao and its
|
- Source: Project Gutenberg eBook 216, "The Tao Teh King, or the Tao and its
|
||||||
Characteristics", Laozi, tr. James Legge (Sacred Books of the East 39, 1891).
|
Characteristics", Laozi, tr. James Legge (Sacred Books of the East 39, 1891).
|
||||||
Gutenberg's licence (1.C) does not claim a right to prevent derivative works of
|
Gutenberg's licence (1.E.2) lets a work derived from texts not protected by US
|
||||||
a work not protected by US copyright, for a user in the US, provided every
|
copyright be used with no fee provided every Project Gutenberg reference is
|
||||||
Project Gutenberg reference is removed (see
|
removed from the work. The items hold only Legge's words, one numbered section
|
||||||
`docs/research/2026-11-13-gutenberg-terms.md`). The items hold only Legge's words, one numbered section
|
|
||||||
or sentence each; no Gutenberg header, licence text or trademark is stored,
|
or sentence each; no Gutenberg header, licence text or trademark is stored,
|
||||||
and the eBook file itself is not committed. `edition_url` names the page the
|
and the eBook file itself is not committed. `edition_url` names the page the
|
||||||
text was retrieved from, as a citation of provenance and not as part of any
|
text was retrieved from, as a citation of provenance and not as part of any
|
||||||
|
|
@ -81,8 +80,6 @@ ceiling of an author who saw the reference:
|
||||||
frame minted from Train, scored lowest (chrF 44.18). The scores rise with
|
frame minted from Train, scored lowest (chrF 44.18). The scores rise with
|
||||||
how bespoke the frame is, the reverse of generalisation. The Validation and
|
how bespoke the frame is, the reverse of generalisation. The Validation and
|
||||||
Test numbers measure the lowerer table, not what the interlingua can carry.
|
Test numbers measure the lowerer table, not what the interlingua can carry.
|
||||||
That table is deleted: the Dao numbers below were measured on it, and the
|
|
||||||
loop now lowers through the model (`Prompt::lower`).
|
|
||||||
- n = 5, one translation as reference, one embedding model, no variance.
|
- n = 5, one translation as reference, one embedding model, no variance.
|
||||||
|
|
||||||
## What the scorer cannot see
|
## What the scorer cannot see
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ by Imagination for the Eureka-area Self, 2026-10-29.
|
||||||
The interlingua spec has two layers. The **schema** is Rust types: packet
|
The interlingua spec has two layers. The **schema** is Rust types: packet
|
||||||
families, node ids, role slots, value kinds. It changes rarely and by commit.
|
families, node ids, role slots, value kinds. It changes rarely and by commit.
|
||||||
The **inventory** is data in CultCache: the concepts, frames, roles, relation
|
The **inventory** is data in CultCache: the concepts, frames, roles, relation
|
||||||
types, speech acts and nuance features (hedge, certainty, irony, stance affects and restraint, and
|
types, speech acts and nuance features (hedge, certainty, warmth, irony and
|
||||||
whatever the loop finds missing) that the schema's slots name by id. Every open slot in a packet is an inventory id; a referent names an entity by id
|
whatever the loop finds missing) that the schema's slots name by id. Every open slot in a packet is an inventory id; a referent names an entity by id
|
||||||
from the entity store, and a packet's only free text is the trace. No door judges text for leakage:
|
from the entity store, and a packet's only free text is the trace. No door judges text for leakage:
|
||||||
what each model call may see, and what a score may count, own that
|
what each model call may see, and what a score may count, own that
|
||||||
|
|
@ -31,18 +31,17 @@ means no store holds it and nothing writes it.
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Packet | Key `packet:<16 hex>`, an opaque digest of item, inventory revision and encoder (the item key lives in `provenance.corpus_item`, which the lowerer never sees); CultCache type `weksa.interlingua_packet`, carrying `schema_version` and `inventory_rev`. Every id inside is an opaque form type (`n<k>` nodes, `encoder:<16 hex>`, canonical `<family>:<n>`). From cut `speaker-state` (schema v4) it also carries the utterance **stance**: the speaker's felt affect and restraint, as inventory features of slot `Stance` (see Speaker state below) | Minted by an encoder for one corpus item under one spec version; immutable; a new spec version means a new packet, never an edit. Admitted only through `PacketStore::admit`, which validates against the inventory store at the packet's revision | The encoder writes; the form types refuse malformed ids at construction and load, the validator refuses broken references and over-capacity packets: a feature at most once per slot, at most 64 nodes (`typed-packet`); nothing else writes packets |
|
| Packet | Key `packet:<16 hex>`, an opaque digest of item, inventory revision and encoder (the item key lives in `provenance.corpus_item`, which the lowerer never sees); CultCache type `weksa.interlingua_packet`, carrying `schema_version` and `inventory_rev`. Every id inside is an opaque form type (`n<k>` nodes, `encoder:<16 hex>`, canonical `<family>:<n>`). From cut `speaker-state` (schema v4) it also carries the utterance **stance**: the speaker's felt affect and restraint, as inventory features of slot `Stance` (see Speaker state below) | Minted by an encoder for one corpus item under one spec version; immutable; a new spec version means a new packet, never an edit. Admitted only through `PacketStore::admit`, which validates against the inventory store at the packet's revision | The encoder writes; the form types refuse malformed ids at construction and load, the validator refuses broken references and over-capacity packets: a feature at most once per slot, at most 64 nodes (`typed-packet`); nothing else writes packets |
|
||||||
| Schema version | `weksa.interlingua_packet.v<N>` constant in `crates/weksa`; published JSON Schema at `schemas/cultnet/` | Bumped by a commit that changes the Rust types; the published schema is regenerated in the same commit and a test fails on drift | A cut (Hands, under a spec); never the loop |
|
| Schema version | `weksa.interlingua_packet.v<N>` constant in `crates/weksa`; published JSON Schema at `schemas/cultnet/` | Bumped by a commit that changes the Rust types; the published schema is regenerated in the same commit and a test fails on drift | A cut (Hands, under a spec); never the loop |
|
||||||
| Inventory entry | Key `<family>:<n>`, `n` canonical decimal >= 1 (families: concept, frame, role, relation, speech_act, feature, referent_kind, extension); type `weksa.inventory_entry`, with a bounded gloss, value kind (option labels bounded), the slots a feature may sit in (`Discourse`, `Referent`, `Predication`, `Relation`; `Stance` from cut `speaker-state`, `Speaker` and `Scene` from cut `speaker-scene`), `introduced_rev` and optional `retired_rev`. Speaker axes, scene relations and affects are features of those slots, so the loop grows them like any closed-class family and convergence covers them | Added or retired by an inventory revision; never edited in place (a changed meaning is a retire plus an add); a retired id still resolves for packets of earlier revisions | The loop's accepted rounds, whose minting call the context builder assembles from Train items only; `weksa inventory admit` for one model-drafted `EditDraft` outside a run (cap 256, reviewed whole in its cut; same numbering and admission check as the loop); Self on merge to main. `data/original` revs 5 (233 concepts and frames) and 6 (five concepts closing named gaps) were drafted by claude-sonnet-5-5 (Weksa Hands, model-drafted original-bed lexicon from Train and Validation passages); Validation-informed: numbers on this bed measure lowering, never generalization or convergence. Residue after rev 6, not amended (each fresh coverage reader names new gaps; the loop's growth path owns them): simultaneity (ながら), distal deixis (あの), 'first' (まず), counterfactual mood, 'at least', emphatic question; heaven versus sky for the visible 天 stays weak. Admission checks form, not text |
|
| Inventory entry | Key `<family>:<n>`, `n` canonical decimal >= 1 (families: concept, frame, role, relation, speech_act, feature, referent_kind, extension); type `weksa.inventory_entry`, with a bounded gloss, value kind (option labels bounded), the slots a feature may sit in (`Discourse`, `Referent`, `Predication`, `Relation`; `Stance` from cut `speaker-state`, `Speaker` and `Scene` from cut `speaker-scene`), `introduced_rev` and optional `retired_rev`. Speaker axes, scene relations and affects are features of those slots, so the loop grows them like any closed-class family and convergence covers them | Added or retired by an inventory revision; never edited in place (a changed meaning is a retire plus an add); a retired id still resolves for packets of earlier revisions | The loop's accepted rounds, whose minting call the context builder assembles from Train items only; Self on merge to main. Admission checks form, not text |
|
||||||
| Inventory revision | Key `rev:<n>`, type `weksa.inventory_revision`, with parent, change list and `frozen` | Appended per accepted round; exactly one revision is ever marked frozen, by the convergence test | The loop's accept rule appends; the convergence test freezes; Self records the freeze as a ruling |
|
| Inventory revision | Key `rev:<n>`, type `weksa.inventory_revision`, with parent, change list and `frozen` | Appended per accepted round; exactly one revision is ever marked frozen, by the convergence test | The loop's accept rule appends; the convergence test freezes; Self records the freeze as a ruling |
|
||||||
| Target profile | Locale or language-project id (`en-US`, `ja-JP`, a conlang's slug). Until entries grow in rounds there is no profile revision document: a run names the profile it lowered with by the ontology store's path and digest in its `RunRecord` (cut `language-ontology`); `profile:<language>:<n>` revision documents arrive with follow-up `language-ontology-growth` | The ontology store is revised by commit; a run pins it and refuses drift; it likewise pins the speaker-state store (`RunRecord` keys 17 and 18, `Drift(SpeakerState)`, cut `language-ontology-scene`) | Profile authors and accepted rounds; never the lowerer's defaults (`persona-drives-register`) |
|
| Target profile | Locale or language-project id (`en-US`, `ja-JP`, a conlang's slug) plus a revision `profile:<language>:<n>` | Authored, revised by commit or by an accepted round that changed the lowering; each lowering names the revision it used | Profile authors and accepted rounds; never the lowerer's defaults (`persona-drives-register`) |
|
||||||
| Language ontology entry (cut `language-ontology`) | Key `onto:<language>:<n>`, `n` from 1 per language; type `weksa.ontology_entry` in `data/ontology/ontology.cc`: the language, the revision of the inventory it was admitted through (provenance: beds number revisions independently and a run reads each condition's feature at its own head), a layer (`Lexicon`, `Grammar`, `Discourse`, `Phonology`, `Orthography`, as the door spells them; the mint prompt ends with an `example` line serialized from the draft type itself), at most four conditions (a feature id of slot `Stance`, `Discourse` or `Scene` with `AtLeast`/`AtMost`/`AtMostOrUnset` a grade or `Is` an option or flag; a `Scene` condition reads the relation of the item's speaker to its addressee that `with_scene` selected; slot `Speaker` and community ids wait on follow-up `language-ontology-communities`), a choice of 1 to 200 bytes, at most three examples of 1 to 120 bytes, and an `Annotator`. Whether an entry's text keeps politeness on the scene is a model reader's judgment, owed per change (a mint owes it under `language-ontology-growth`), not a test | Admitted in all-or-nothing batches through `OntologyStore::admit`, immutable, never edited; retire arrives with `language-ontology-growth`. The run pins the store like the entity store (path and digest, `Drift(Ontology)`) and never writes it; slice 1 of growth is outside any run: `weksa loop mint-ontology` drafts entries from the run's Train items of one language into `<root>/ontology-mint/<digest>.answer`, `weksa ontology admit --ontology <copy>` admits them into a file copy, and a new run pins the copy (a run-owned copy waits on follow-up `language-ontology-growth`). Cut `language-ontology-seed` converts by drafting, not by parse: a fresh model drafts each language's seed from that language's YAML and the inventory's feature lines with no corpus text, then deletes `data/target-language-ontology/` and `.weksa/cultural-ontology/` in the same cut | Seed: a labelled model (`Annotator::Model`), no operator gate, never Test evidence (`ja-ontology-owner-answer`); attestation (operator, native reader) appends later. Accepted rounds later, through a mint call from Train items only. Selection is `ontology::select(ontology, language, view)`, called only inside `Prompt::lower`, so no caller picks entries; the prompt prints each selected entry's id, layer, choice and examples. Its text is minted text for the leak diagnostic, which today has no runtime caller (follow-up `leak-diagnostic-in-round`). The store was rebuilt without ja-JP `:14` and `:15` (above-status Scene entries; a paired measure found them net-negative on register, follow-up `ja-scene-entries-lower-register-agreement`), so ja-JP ends at `:13`: an above-status out-group addressee, and an above-status in-group addressee below intimacy 700, select no ja-JP Scene entry (`:12` still selects in-group at intimacy 700 or more regardless of status) until a native reader attests a replacement (`ja-native-reader-scenes`). ja-JP `:5`, `:7` and `:8` were revised by certainty and restraint `AtMostOrUnset` conditions (holds when the feature is unrated or at most the grade; `AtMost` still fails on an unrated feature) in a store rebuild (the store has no retire, so a conditions change is a rebuild), which makes ja-JP's hedging, certainty, affection, restraint and sadness entries disjoint on `:5`/`:6`, `:4`/`:7`, `:6`/`:7`, `:6`/`:8`, and ja-JP `:3` (anger) carries `AtMostOrUnset` hedge and affection bounds, as en-US `:3`, which makes `:3`/`:5` and `:3`/`:7` disjoint; en-US `:3` carries `AtMostOrUnset` hedge and affection bounds, `:5` an `AtMostOrUnset` certainty bound (the clause it held in prose) and `:8` one too (as ja-JP `:7`), which makes `:3`/`:5`, `:3`/`:8`, `:5`/`:6` and `:6`/`:8` disjoint, while `:6`/`:10` stay co-selectable and are kept apart by scoping in `:6`'s text; ja-JP `:9` was reworded in a rebuild to state its question ending as available, so `:6`/`:9`, `:5`/`:9` and `:3`/`:9` stay co-selectable and are reconciled by wording, not by condition |
|
| Language ontology entry (cut `language-ontology`) | Key `onto:<language>:<n>`; type `weksa.ontology_entry`: conditions over feature values (stance, speaker, scene, packet features) and community ids, and the surface choice they select (layer: lexicon, grammar, discourse, phonology, orthography; a bounded descriptor and at most three bounded examples); introduced and retired by a profile revision | Added or retired by a profile revision, never edited; converted from `data/target-language-ontology/*.yaml`, which is deleted in the same cut so the YAML is not a second owner | The language's owner (the operator for her settings' languages) and accepted rounds; an ontology-minting call is assembled from Train items only, like an inventory-minting call, and its text is in the scorer's leak diagnostic. Selection is deterministic code (`select(profile, stance, speaker, scene)`); the lowerer realizes the selected choices and the trace lists their ids (`lowering-traced`) |
|
||||||
| Community (cut `language-ontology`, follow-up `language-ontology-communities`) | Key `community:<slug>`; type `weksa.community`, built fresh when the cut is mapped: `data/cultural-ontology/` (draft YAML profiles, read by no code) is deleted from main (cut 5e `delete-cultural-ontology`) and stays in git history before that cut's commit | Revised by commit; a speaker names the communities it belongs to by id | The cultural ontology's author (owner-attested evidence only; Discord-only content stays deleted, `discord-only-content-deleted`) |
|
| Community (cut `language-ontology`) | Key `community:<slug>`; type `weksa.community`, converted from `data/cultural-ontology/cultures/` and `referents/` (YAML deleted in the same cut) | Revised by commit; a speaker names the communities it belongs to by id | The cultural ontology's author (owner-attested evidence only; Discord-only content stays deleted, `discord-only-content-deleted`) |
|
||||||
| Speaker (cut `speaker-scene`) | Key `speaker:<slug>@<n>`; type `weksa.speaker`: stable axes as feature values of slot `Speaker` (origin, socioeconomic standing, age cohort, education and whatever the loop adds), at an inventory revision, and an `Annotator` | A new revision appends, never edits; a corpus item and every prior turn name a speaker slug, and a round records the speaker revisions it read | The annotator it names: `PersonaSnapshot` (a projection of a `gamecult.persona_state.v0` document, naming its source document and `updatedAt`), `Operator`, `WorkMetadata` (a public-domain work's own cast list and narration), or `Model { family, generator }` (the 2b `ModelFamily`) |
|
| Speaker (cut `speaker-scene`) | Key `speaker:<slug>@<n>`; type `weksa.speaker`: stable axes as feature values of slot `Speaker` (origin, socioeconomic standing, age cohort, education and whatever the loop adds), at an inventory revision, and an `Annotator` | A new revision appends, never edits; a corpus item and every prior turn name a speaker slug, and a round records the speaker revisions it read | The annotator it names: `PersonaSnapshot` (a projection of a `gamecult.persona_state.v0` document, naming its source document and `updatedAt`), `Operator`, `WorkMetadata` (a public-domain work's own cast list and narration), or `Model { family, generator }` (the 2b `ModelFamily`) |
|
||||||
| Scene (cut `speaker-scene`) | Key `scene:<group>@<n>`, the item's split group; type `weksa.scene`: participants (speaker slugs) and, per ordered pair, relation features of slot `Scene` (relative status, intimacy, in-group, audience and whatever the loop adds), at an inventory revision (provenance; a run reads the features at its own head), and an `Annotator` | As Speaker. Keyed by the group, so a scene never straddles the split. `data/original/speaker-state.cc` holds Model-annotated keeper scenes, one per Train or Validation group of both languages (`weksa scene put|dump`); a Test group has none and reads `NoScene` | As Speaker; a model-annotated scene is never Test evidence |
|
| Scene (cut `speaker-scene`) | Key `scene:<group>@<n>`, the item's split group; type `weksa.scene`: participants (speaker slugs) and, per ordered pair, relation features of slot `Scene` (relative status, intimacy, in-group, audience and whatever the loop adds), at an inventory revision, and an `Annotator` | As Speaker. Keyed by the group, so a scene never straddles the split | As Speaker |
|
||||||
| Corpus item | Key `item:<source-slug>:<n>`, `n` canonical decimal >= 1 (no sign, no leading zero, from cut 2b); type `weksa.corpus_item` with text, language, speaker, split group, prior turns, source and who placed its name spans (`names_by`, an `Annotator`, absent when unrecorded; cut `pd-name-spans`). Source `ModelWritten` carries a `ModelFamily` enum, the generator string as the fixture wrote it, and a prompt digest (cut 2b). The item carries no speaker state of its own: its speaker slug and group name the Speaker and Scene documents | Admitted once; the whole item is immutable (re-adding an identical item is a no-op, any difference is refused; a correction is a new item that `supersedes` the old); text already held under another key is refused unless the new item supersedes that key (cut 2b); withdrawn, never deleted, so old scores stay explainable | Importers and `weksa corpus add`, under `corpus-cast-plus-public-domain`; the source enum has no variant for Discord or any training-forbidden source (`corpus-clean`) |
|
| Corpus item | Key `item:<source-slug>:<n>`, `n` canonical decimal >= 1 (no sign, no leading zero, from cut 2b); type `weksa.corpus_item` with text, language, speaker, split group, prior turns, source. Source `ModelWritten` carries a `ModelFamily` enum, the generator string as the fixture wrote it, and a prompt digest (cut 2b). The item carries no speaker state of its own: its speaker slug and group name the Speaker and Scene documents | Admitted once; the whole item is immutable (re-adding an identical item is a no-op, any difference is refused; a correction is a new item that `supersedes` the old); text already held under another key is refused unless the new item supersedes that key (cut 2b); withdrawn, never deleted, so old scores stay explainable | Importers and `weksa corpus add`, under `corpus-cast-plus-public-domain`; the source enum has no variant for Discord or any training-forbidden source (`corpus-clean`) |
|
||||||
| Split assignment | Derived: `split(scheme, item.group)` | Fixed for a scheme; a new scheme is a new constant, never a reassignment. A model-written item whose group derives test is in no split; a superseding item keeps its predecessor's group | The hash function. Nobody assigns by hand, so there is no writer to forbid |
|
| Split assignment | Derived: `split(scheme, item.group)` | Fixed for a scheme; a new scheme is a new constant, never a reassignment. A model-written item whose group derives test is in no split; a superseding item keeps its predecessor's group | The hash function. Nobody assigns by hand, so there is no writer to forbid |
|
||||||
| Loop round | Key `run:<label>:round:<n>`; type `weksa.loop_round` with spec version, profile revisions, model ids and prompt digests, sampled train items, Validation means and refusal counts (from cut `ja-panel-loop`: one set per language of the run's panel, no pooled mean), per-item Validation scores in corpus order (key 22, numbers only, no item key or text, private to `run.rs` for the paired floor of the cycle (rounds 1 to 4) and of relower; a run from before has none and opens), the digest of the inventory file it was scored under and, for a cycle's candidate round, its cycle `k` | Scored and recorded whole; immutable. A candidate round is put in one batch with its Decision. Lives in the run store on Yggdrasil | `run::round` and `run::cycle` (both through `score_round`) |
|
| Loop round | Key `run:<label>:round:<n>`; type `weksa.loop_round` with spec version, profile revisions, model ids and prompt digests, sampled train items, proposed change, validation scores before and after, outcome | Opened, scored, closed accepted or rejected; immutable once closed. Lives in the run store on Yggdrasil | The loop driver's accept rule (validation composite improves past the measured noise floor and no metric regresses past tolerance) |
|
||||||
| Loop decision | Key `run:<label>:cycle:<k>`; type `weksa.loop_decision` with the base round and revision, the edit prompt's digest and Train keys, the editor's answer digest, the candidate revision, inventory digest and round, the base, candidate and floor readings (chrF, meaning, composite; the floor is the paired floor of round 1 against rounds 2 to 4), the threshold and the verdict (Accepted, or Rejected for edit-parse, edit-admit, no-gain, regressed chrF, meaning or composite); digests and numbers, no text | Immutable. The latest Accepted one names the run's head inventory (`<root>/cycle-<k>/inventory.cc`, its candidate digest, its candidate round); with none the head is the copy `create` made at `<root>/inventory.cc`. Lives in the run store on Yggdrasil | `run::cycle`'s accept rule, `run::decide`: the validation composite gains more than max(noise floor, `LADDER_STEP`), no metric falls past max(its floor, one step), the composite does not fall past max(its floor, one step), and a non-finite reading rejects (the one regression test, `run::regression`); differences compared in billionths so a gain of exactly one step is a tie. From cut `ja-panel-loop` the readings, floor and threshold are kept per panel language and `run::regression` runs per language and is the one regression test (`decide` adds the gain test for a one-language panel): any language regressing rejects (verdicts `regressed-chrf`, `regressed-meaning`, `regressed-composite`), otherwise a one-language panel accepts as `decide` does and a panel of two or more accepts when the sum of composite deltas exceeds one `LADDER_STEP` (billionths). Given up: strict Pareto non-regression; a language may lose up to its own noise floor if the panel nets more than a step, because zero tolerance ratchets every base to a favourable noise draw and strangles the loop. Termination: each accept raises the composite sum by more than a step and composites lie in -0.5 to 1, so a run accepts at most 1.5 * languages / `LADDER_STEP` edits |
|
| Score | Key `<lowering-key>:<metric>@<metric-version>`; type `weksa.score` | Immutable; a new metric version scores again under a new key | The scorer, deterministic given the metric version and the embedder's model id. Cut `scorer` computes scores as values; cut `golf-loop-en` stores them, because the lowering keys they hang from arrive with the loop |
|
||||||
| Score | Key `<lowering-key>:<metric>@<metric-version>`; type `weksa.score` | Immutable; a new metric version scores again under a new key | The scorer, deterministic given the metric version and the embedder's model id. Cut `scorer` computes scores as values; cut `golf-loop-en` stores them, because the lowering keys they hang from arrive with the loop. Lands with follow-up `golf-loop-score-docs`, in the round's directory; nothing in the cycle reads them |
|
|
||||||
| Lowering trace | Key `lowering:<packet-key>:<profile>@<rev>:<lowerer-digest>:<sample>`; type `weksa.lowering` holding the surface text and its trace together | Immutable. A surface text cannot exist without its trace, because they are one record (`lowering-traced`) | The lowerer emits; the lowerer sees the packet without its `trace` (the input type has no such field), never the corpus item |
|
| Lowering trace | Key `lowering:<packet-key>:<profile>@<rev>:<lowerer-digest>:<sample>`; type `weksa.lowering` holding the surface text and its trace together | Immutable. A surface text cannot exist without its trace, because they are one record (`lowering-traced`) | The lowerer emits; the lowerer sees the packet without its `trace` (the input type has no such field), never the corpus item |
|
||||||
| Conlang lexicon entry | Key `lex:<language>:<concept-id>`; root form, morphology class | Root placed at its concept's embedding when the concept first needs a form in that language; stable once accepted; diachrony derives later forms, it does not overwrite | The language project's owner: the operator for her settings' languages, the loop proposing roots for her review |
|
| Conlang lexicon entry | Key `lex:<language>:<concept-id>`; root form, morphology class | Root placed at its concept's embedding when the concept first needs a form in that language; stable once accepted; diachrony derives later forms, it does not overwrite | The language project's owner: the operator for her settings' languages, the loop proposing roots for her review |
|
||||||
| Conlang grammar and phonology data | `<language>/grammar`, `<language>/phonology` documents per language project | Revised by commit; every lowering names the revision | The language project's owner |
|
| Conlang grammar and phonology data | `<language>/grammar`, `<language>/phonology` documents per language project | Revised by commit; every lowering names the revision | The language project's owner |
|
||||||
|
|
@ -134,9 +133,8 @@ standing.** `gamecult.persona_state.v0` (Body facts) carries open trait maps
|
||||||
So a `PersonaSnapshot` projects mood-like and relation-like values, and the
|
So a `PersonaSnapshot` projects mood-like and relation-like values, and the
|
||||||
stable axes the standard lacks are annotated. Whether the standard gains a
|
stable axes the standard lacks are annotated. Whether the standard gains a
|
||||||
typed speech-community reference or Weksa keeps the persona-to-community
|
typed speech-community reference or Weksa keeps the persona-to-community
|
||||||
binding (the deleted `data/cultural-ontology/persona-stacks/brokkr.yaml` was its one draft) is question `persona-speech-community-home` (recommended: a Weksa-owned
|
binding (`data/cultural-ontology/persona-stacks/`) is decided when
|
||||||
binding; follow-up `persona-state-speech-community`), decided when
|
`live-lowering` is mapped (follow-up `persona-state-speech-community`); the
|
||||||
`live-lowering` is mapped; the
|
|
||||||
corpus path needs neither. Putting Weksa feature ids into the open
|
corpus path needs neither. Putting Weksa feature ids into the open
|
||||||
`voiceStyle` trait map is rejected: it would make an extension field portable
|
`voiceStyle` trait map is rejected: it would make an extension field portable
|
||||||
authority, which the standard forbids.
|
authority, which the standard forbids.
|
||||||
|
|
@ -261,11 +259,10 @@ operator-attested annotations alone. Who annotates is question
|
||||||
with the same text and a different source returns `Unchanged` and writes
|
with the same text and a different source returns `Unchanged` and writes
|
||||||
nothing, so a relabel through `add` silently does nothing. `key_slug`
|
nothing, so a relabel through `add` silently does nothing. `key_slug`
|
||||||
parses `n` with `u32::parse`, which accepts `01` and `+1`.
|
parses `n` with `u32::parse`, which accepts `01` and `+1`.
|
||||||
- Seed inventory (2026-11-10, base `5e10d7c`): features 6-8 are hedge, certainty
|
- Inventory rev 1 (seed): features 6-11 are hedge, certainty, warmth,
|
||||||
and irony; 9-18 are the stance affects and restraint (slot Stance). The
|
formality, intensity (Grade) and irony (Flag), slots Discourse and
|
||||||
committed inventory holds revisions 1-2 only. Nothing calls `with_scene`
|
Predication. Nothing outside the seed and `docs/interlingua-standard.md:179-180`
|
||||||
except tests, and the proof packet's addressee carries no entity (its
|
names warmth, formality or intensity; no committed packet store exists.
|
||||||
referent `n3` does: `entity:nibu`).
|
|
||||||
- Yggdrasil's Ollama (`127.0.0.1:11434`) is the Docker container `ollama-cpu`,
|
- Yggdrasil's Ollama (`127.0.0.1:11434`) is the Docker container `ollama-cpu`,
|
||||||
Huginn's embedder (runbook `gamecult-ops/runbooks/huginn-yggdrasil.md`,
|
Huginn's embedder (runbook `gamecult-ops/runbooks/huginn-yggdrasil.md`,
|
||||||
ruling `eureka-body:ruling:huginn-embedder-yggdrasil-primary`). `/api/embed`
|
ruling `eureka-body:ruling:huginn-embedder-yggdrasil-primary`). `/api/embed`
|
||||||
|
|
@ -309,27 +306,6 @@ line two", "その 制御" against the
|
||||||
or socioeconomic standing; `extensions` are non-authoritative by the
|
or socioeconomic standing; `extensions` are non-authoritative by the
|
||||||
schema's own description.
|
schema's own description.
|
||||||
|
|
||||||
- Probes of 2026-11-12 at `d9ebbcf` (map pass for cut 5): no code reads
|
|
||||||
`data/target-language-ontology/`, `data/cultural-ontology/` (since deleted) or
|
|
||||||
`.weksa/cultural-ontology/starter-targets.cc` (a third, typed copy of en,
|
|
||||||
ja-JP and pt-BR profiles); `README.md`, `docs/multilingual-utterance-lowering.md`,
|
|
||||||
`examples/cultcache-lowering/` and `prompts/nibu-cultcache-target-*.md` cite
|
|
||||||
them. `LoweringInput::with_scene` has no caller in `run.rs` or `main.rs`, the
|
|
||||||
run pins no `SpeakerStateStore`, and no Speaker or Scene document is committed
|
|
||||||
(`speaker-scene-seeds` open), so only the packet's `Stance` and `Discourse`
|
|
||||||
features reach a loop's lowering view. `score::leak_diagnostic` and
|
|
||||||
`TestReadings::at_checkpoint` take minted text as `&[&str]` and have no
|
|
||||||
caller outside tests: no round reads glosses or any minted text against
|
|
||||||
Validation. `data/daodejing/inventory.cc` is a revision prefix of
|
|
||||||
`data/original/inventory.cc` (revisions 1 to 4 equal, a test proves it; revs 5 and 6 add the bed's own lexicon); with `data/interlingua/inventory.cc` they share revision 1,
|
|
||||||
where features 6 to 8 (hedge, certainty grades, irony flag) sit in
|
|
||||||
`Discourse` and `Predication` and features 9 to 18 (anger to restraint,
|
|
||||||
grades) in `Stance`. `PROMPT_SOURCE_DIGEST` covers `context.rs`, `split.rs`,
|
|
||||||
`lower.rs` and `packet.rs`. 25 `RunPaths {` literals across `run.rs`,
|
|
||||||
`context.rs` doctests, `main.rs`, `tests/common/mod.rs`, `tests/golf_loop.rs`
|
|
||||||
and `tests/original_corpus.rs`; 10 `Prompt::lower(` calls. `huginn known`
|
|
||||||
timed out (Unavailable, 15 s deadline) twice this pass.
|
|
||||||
|
|
||||||
## Where things live and run (`rust-single-crate`, `loop-runs-on-yggdrasil`)
|
## Where things live and run (`rust-single-crate`, `loop-runs-on-yggdrasil`)
|
||||||
|
|
||||||
- Code: one Cargo workspace at the repo root, one package `crates/weksa` (a
|
- Code: one Cargo workspace at the repo root, one package `crates/weksa` (a
|
||||||
|
|
@ -340,14 +316,9 @@ line two", "その 制御" against the
|
||||||
- Stores: `data/interlingua/inventory.cc` and `data/corpus/corpus.cc` in the
|
- Stores: `data/interlingua/inventory.cc` and `data/corpus/corpus.cc` in the
|
||||||
repo (small, reviewed with `weksa dump`); run stores (rounds, lowerings,
|
repo (small, reviewed with `weksa dump`); run stores (rounds, lowerings,
|
||||||
scores) on Yggdrasil under the loop's state root, not in git.
|
scores) on Yggdrasil under the loop's state root, not in git.
|
||||||
- The loop is a batch command driven from a Claude Code session on Yggdrasil
|
- The loop runs on Yggdrasil as a detached unit with a log and a status record.
|
||||||
(`golf-loop-in-claude`), not a detached unit. Model calls go through a
|
Starfire runs none of it. Model calls go through a `ModelPort`; its first
|
||||||
`ModelPort` whose one adapter, `AnswerFiles`, writes a `.prompt` file and reads
|
adapter drives `claude -p` on the operator's subscription. Never OpenAI
|
||||||
a `.answer` file: Claude agents answer them, one fresh subagent per encode
|
|
||||||
prompt (`encoder-isolation-subagent-per-prompt`) and a tool-less relay for
|
|
||||||
lowering prompts (`lowerer-call-reach-toolless-relay`). Builds, stores, scoring
|
|
||||||
and embeddings run on Yggdrasil; Starfire runs none of it, and Weksa needs no
|
|
||||||
claude binary or token there. Never OpenAI
|
|
||||||
(`eureka-body:ruling:no-return-to-openai-supplier`), never Discord data
|
(`eureka-body:ruling:no-return-to-openai-supplier`), never Discord data
|
||||||
(`gamecult-chat:ruling:discord-taints-training`).
|
(`gamecult-chat:ruling:discord-taints-training`).
|
||||||
|
|
||||||
|
|
@ -360,271 +331,56 @@ checkpoint the test-split composite is within the noise floor of validation.
|
||||||
The test split is read only at checkpoints. Self then records the freeze, and
|
The test split is read only at checkpoints. Self then records the freeze, and
|
||||||
the training set is the whole corpus re-encoded under the frozen revision.
|
the training set is the whole corpus re-encoded under the frozen revision.
|
||||||
|
|
||||||
## Distil measure (`distil-slice`)
|
## Lowerer first cut (`golf-loop-lowerer-cut`)
|
||||||
|
|
||||||
A lowerer is compared on round n's Validation packets with the prompts
|
Authority map for the smallest lowerer, written before its code (Imagination
|
||||||
byte-identical to the round's, by `weksa loop relower ROOT --round N --arm ARM
|
pass by the Weksa Self run, 2026-11-01). The cut exists to give the golf loop a
|
||||||
--lowerer LABEL [--floor ARM2 --floor ARM3 --floor ARM4]`. Every arm is answered through the one prompt
|
measurable baseline from a packet: `lower` gives text, `score` gives numbers.
|
||||||
door (`RunStores::answer_files`, the Test-text guard); a tuned model is a script
|
|
||||||
that reads `<digest>.prompt` and writes `<digest>.answer` in
|
|
||||||
`<ROOT>/relower-<N>-<ARM>`, not a new adapter. The recorded reading is the
|
|
||||||
round's own lowering, read over the same items as the arm. An item the round
|
|
||||||
refused before admission (no packet) or at the lowering prompt has no prompt to
|
|
||||||
ask: it is skipped, so the readings exclude it, and it is counted as `skipped=`
|
|
||||||
on the line; `n` is the items read, so `recorded` can differ from the round's
|
|
||||||
own Validation reading, which scores those items 0. An arm answer that is not a
|
|
||||||
usable lowering scores 0 and counts in `unusable=`; the same for the floor arm,
|
|
||||||
counted (over all floor arms) after `floor ... unusable=`. The floor needs three
|
|
||||||
or more floor arms, each a prompted repeat of the recorded lowering on the same
|
|
||||||
prompts: it is `FLOOR_Z` (3) standard errors of the mean per-item difference
|
|
||||||
between the recorded lowering and a repeat, per metric, with the variances
|
|
||||||
pooled over the arms (`paired_floor`; lowering-only noise, no encoder noise). The
|
|
||||||
verdict is `decide_panel` over (recorded, arm, floor): Accepted is a composite
|
|
||||||
gain past the noise of the panel's sum (the root of the languages' composite
|
|
||||||
floors squared, at least one step), Rejected(Regressed) a loss past a language's
|
|
||||||
floor, Rejected(NoGain) neither. The cycle's own floor is the same `paired_floor`
|
|
||||||
(`floor_of`): round 1 (the recorded lowering, the head's establishing round)
|
|
||||||
against rounds 2 to `FLOOR_ARMS_MIN` + 1 (prompted repeats under the run's first
|
|
||||||
inventory), per panel language, from the rounds' per-item Validation scores. It
|
|
||||||
is measured once per run and is the same after an accept; a run needs four
|
|
||||||
`loop round` calls before its first `loop cycle`.
|
|
||||||
Calibration (`runs/distil-base`, 8 prompted draws by one model family on
|
|
||||||
identical prompts, no draw sharing more than 7 of 28 answers with another; 14
|
|
||||||
en-US and 14 ja-JP Validation items; every base, candidate and 3-arm floor
|
|
||||||
combination, 1120 trials from 28 unordered pairs; intervals bootstrap over
|
|
||||||
draws, each degraded arm one fixed draw): a prompted repeat reads NoGain 92%
|
|
||||||
(80 to 100), Regressed 8% (0 to 20), Accepted 0%, which with 28 pairs bounds
|
|
||||||
false-accept only to about 10%; an all-"Yes." arm reads Regressed 100%; a mild
|
|
||||||
degradation (one detail blurred per utterance) 100%; a subtler one 75% (50 to
|
|
||||||
100), whose mean composite drop (0.044) sits under the mean floor (0.060).
|
|
||||||
`FLOOR_Z` was chosen on this data (2.5 gives NoGain 84% and 93% on the subtler
|
|
||||||
arm), so the rates are in-sample. Without the noise term in the gain test a
|
|
||||||
prompted repeat reads Accepted in 40% of the trials.
|
|
||||||
Power (can the rule Accept a real gain?): the only better arm that could be
|
|
||||||
built without Test or Validation answers is a lowerer shown the Train sentences
|
|
||||||
of the corpus (no packets, no Validation or Test text) as a register guide.
|
|
||||||
Four such draws beat the prompted draws' mean composite in both languages (en-US
|
|
||||||
0.742 to 0.759 to 0.795; ja-JP 0.559 to 0.578 to 0.620), so the premise that it is
|
|
||||||
better is measured on the very Validation items the rule reads, not independent.
|
|
||||||
Against the prompted bases with 3-arm floors the rule reads it Accepted in 29%
|
|
||||||
of 1120 trials, NoGain 71%, Regressed 0%: a net gain of 0.079 against a mean
|
|
||||||
noise term of 0.079. At 14 items per language the rule is therefore a low-power
|
|
||||||
filter for a gain of this size; by 1/sqrt(n), about 35 or more items per
|
|
||||||
language would bring the noise term near 0.05 (an extrapolation, not measured).
|
|
||||||
### Validation growth (`validation-growth`, built and recalibrated)
|
|
||||||
|
|
||||||
Power needs about 35 or more Validation items per language. `data/original`
|
|
||||||
now holds 35 per language (it held 14; Train 14 and Test 8 are unchanged):
|
|
||||||
42 model-written items, `item:vgrow:1..21` (en-US) and `item:vgrow-ja:1..21`
|
|
||||||
(ja-JP), live Train+Validation 56 to 98. The slug is new on purpose: the
|
|
||||||
`item:original:` prefix filter of `DIGEST_OF_THE_28` still selects the first 28
|
|
||||||
and passes; `DIGEST_OF_THE_42` freezes the new prose.
|
|
||||||
|
|
||||||
The split is derived from an item's group alone (`SplitScheme::V1`: FNV-1a of
|
|
||||||
`weksa-split-v1:<group>` mod 100, under 40 Train, under 80 Validation, else
|
|
||||||
Test). The names were chosen before any text existed: 90 candidate nouns per
|
|
||||||
language, dropped by script when the group already existed or the noun
|
|
||||||
occurred in a Test text, then the first 24 whose bucket fell in 40..=79 (the
|
|
||||||
bucket came from `SplitScheme::V1` itself, cross-checked on the 72 existing
|
|
||||||
items). Each language was written by three blind writer agents (8 nouns each,
|
|
||||||
a brief of language, register note, noun and speech act, no Test text and no
|
|
||||||
existing item), 24 texts per language written, the first 21 in name order
|
|
||||||
admitted through `weksa corpus add`; each group is its own group. The scenes
|
|
||||||
are `weksa scene put` documents (keeper to `unnamed`, inventory rev 4),
|
|
||||||
annotated by a separate agent; a second annotator measured agreement only and
|
|
||||||
is not committed.
|
|
||||||
|
|
||||||
The writer is Claude (family Anthropic recorded): no cross-family generator was
|
|
||||||
reachable from the body that ran this cut. The self-preference effect (a Claude
|
|
||||||
lowerer favouring Claude prose, Panickssery et al. 2024) is therefore not
|
|
||||||
measured. The writers' blindness is by brief content, not sandbox.
|
|
||||||
|
|
||||||
Expect encode refusals: the existing 56 were written onto the Dao r3 concepts
|
|
||||||
and the inventory was drafted from them, while the new items were written free
|
|
||||||
of the inventory (question `validation-growth-vocabulary` stays open). Some new
|
|
||||||
packets will be refused at encode (inventory coverage) and score 0 as the means
|
|
||||||
do; they stay in the corpus. That is a result about inventory generalization,
|
|
||||||
not a defect.
|
|
||||||
|
|
||||||
Every calibration number in the Distil measure section above (92/0/8 prompted
|
|
||||||
rates, the 29% power, the 0.079 noise term) is for the 14-item corpus and stays
|
|
||||||
the stated baseline of that corpus.
|
|
||||||
|
|
||||||
Recalibration on the grown corpus (`runs/growth-base`, main 44b66f9, panel
|
|
||||||
en-US and ja-JP, round 1 recorded by briefed subagents; 6 prompted draws
|
|
||||||
including the recorded one, 2 mild degradations, an all-"Yes." control and 4
|
|
||||||
Train-guide draws by Claude sonnet-5-5; the same decision rule at FLOOR_Z 3 and
|
|
||||||
3-arm floors; 120 prompted trials from the 30 ordered pairs of 6 draws with the
|
|
||||||
other 4 as floor candidates, so trials share draws and are not independent).
|
|
||||||
Soul pass `cut-validation-growth.s3` re-derived every number below; its
|
|
||||||
findings shaped the wording.
|
|
||||||
|
|
||||||
- Live Validation items: 28 en-US and 32 ja-JP of 35 each; 10 are refused at
|
|
||||||
parse (en-US original:18, 23, 25, 27 and vgrow:4, 5, 6; ja-JP original-ja:17,
|
|
||||||
22 and vgrow-ja:2); `relower` skips an item with no packet, so every reading
|
|
||||||
here is over live items only. Old 14: 10 en-US and 12 ja-JP live; new 21: 18
|
|
||||||
and 20.
|
|
||||||
- Prompted repeat: NoGain 96.7%, Accepted 3.3%, Regressed 0% (14 items: 92 / 0 /
|
|
||||||
8). All four false accepts are one pair, base p6 and candidate the recorded
|
|
||||||
lowering R: R comes from the round, the other five draws from `relower`, and
|
|
||||||
in en-US R scores +0.024 per item above them [+0.002, +0.052], so R is not
|
|
||||||
exchangeable with a repeat (a Plausible Soul finding, cause not isolated). No
|
|
||||||
false-accept bound can be drawn from 6 draws.
|
|
||||||
- Degraded arms read Regressed: all-"Yes." 100% (as before); mild-1 31.7%;
|
|
||||||
mild-2 0% (14 items: 100% and 75% on a subtler arm). The mild arms are weak
|
|
||||||
and not independent: they were rewritten from R with one detail blurred, the
|
|
||||||
original wording is not on disk, mild-2 equals R word for word in 5 of 60
|
|
||||||
items and mild-1 in 2, and mild-1 equals mild-2 in 27 of 60 although they
|
|
||||||
were asked to blur different details. Their mean composite is 0.003 to 0.015
|
|
||||||
below the prompted mean (the old subtler arm: 0.037 and 0.052). The
|
|
||||||
sensitivity to a small regression is not re-measured.
|
|
||||||
- Noise: the panel noise term of the sum rule is 0.045 on average over the base
|
|
||||||
and floor choices (0.031 to 0.064), against 0.079 at 14 items. The 1/sqrt(n)
|
|
||||||
forecast (about 0.05) assumed per-item variance stayed the same; it did not:
|
|
||||||
the per-item SD of the difference between two prompted draws is about 0.085
|
|
||||||
on the old items and 0.034 on the new ones, so the pooled term landed near
|
|
||||||
the forecast because less variable items were added. Not a confirmation of
|
|
||||||
the forecast.
|
|
||||||
- Power: the Train-guide arm (14 Train sentences as a register guide) is
|
|
||||||
Accepted in 21.2% of 240 trials, NoGain 78.8%, against 29% at 14 items, and
|
|
||||||
its mean composite gain over the prompted draws is 0.0185, 0.41 of the mean
|
|
||||||
noise term (0.37 on the first floor choice per base); with `relower`-only
|
|
||||||
bases (R removed) it is Accepted in 35% (28 of 80). Split by item class
|
|
||||||
(paired bootstrap over items, 95%): old items +0.029 [+0.005, +0.052] (en-US
|
|
||||||
+0.049 [+0.023, +0.084] on 10 items, ja-JP +0.013 [-0.021, +0.037] on 12);
|
|
||||||
new items -0.002 [-0.011, +0.007] pooled, but two opposite effects: en-US
|
|
||||||
-0.017 [-0.027, -0.007] on 18 items and ja-JP +0.011 [+0.001, +0.022] on 20.
|
|
||||||
So the old-versus-new contrast exists in en-US; in ja-JP old (+0.013) and new
|
|
||||||
(+0.011) do not differ. Measured: the 14-item gain does not carry to the new
|
|
||||||
en-US items. Inferred, not isolated: that it was in-sample register fit. The
|
|
||||||
new items differ from the old in several ways at once (written free of the
|
|
||||||
inventory, longer, a different scene convention, lower scores, a smaller
|
|
||||||
vocabulary overlap with Train in en-US: 0.29 against 0.52) and were briefed to
|
|
||||||
the same register as Train, by Claude as the old were. The 29% power figure
|
|
||||||
stays the stated baseline of the 14-item corpus; no better arm is known on
|
|
||||||
the new items, so the rule's power against a real gain is not re-measured.
|
|
||||||
- The two classes are different strata (Soul s2 `register-stratum-shift`): the
|
|
||||||
new items score lower (composite 0.41 against 0.64 pooled) so a pooled
|
|
||||||
60-item mean mixes them; report old and new separately.
|
|
||||||
- The self-preference effect is not measured (ruling
|
|
||||||
`validation-growth-writer-claude-checked`). The new scenes use a different
|
|
||||||
`in_group` convention from the old ones (follow-up
|
|
||||||
`vgrow-scene-in-group-convention`), so the new-item scores are for scenes that
|
|
||||||
will be re-annotated; the speaker-state store changes then (a run pins its
|
|
||||||
digest), the corpus does not.
|
|
||||||
|
|
||||||
For `growth-base`, `weksa loop calibrate` (`docs/calibration.md`) reproduces from `run.rs`'s own
|
|
||||||
scoring and accept rule the counts per class, the trial totals and the mean
|
|
||||||
noise of this block: prompted 116 NoGain, 4 Accepted, 0 Regressed of 120;
|
|
||||||
control-yes 60 of 60 Regressed, mild-1 19 of 60, mild-2 0 of 60; the
|
|
||||||
Train-guide arm 51 Accepted, 189 NoGain, 0 Regressed of 240; noise 0.0451;
|
|
||||||
items en-US n=28 skipped=7, ja-JP n=32 skipped=3. By class it adds: old items
|
|
||||||
(`item:original*`) prompted 111/9/0, noise 0.1069, Train-guide 71 Accepted of
|
|
||||||
240; new items (`item:vgrow*`) prompted 116/4/0, noise 0.0334, Train-guide 0
|
|
||||||
Accepted and 74 Regressed of 240. The other figures of this block (35% with R
|
|
||||||
removed, the noise range, the 0.0185 gain and 0.41 ratio, the bootstrap
|
|
||||||
intervals, the per-item SDs, R's +0.024, the mild-arm duplicate counts and
|
|
||||||
mean composite offsets, the Train-vocabulary overlap, the old/new class
|
|
||||||
composites, the ten parse-refused item ids) were
|
|
||||||
computed by scratch scripts outside git (Soul finding `evidence-outside-git`)
|
|
||||||
and are not reproduced by the harness.
|
|
||||||
|
|
||||||
Recalibration on the relabelled scenes (`runs/growth-scene`, main d8afc4a; same
|
|
||||||
panel, 35 live items per language, 0 refused, 5 prompted draws including the
|
|
||||||
recorded one R, 2 mild, 4 Train-guide). It is not a replacement for the
|
|
||||||
`growth-base` block above and no delta against it is drawn: the item set, the
|
|
||||||
packets and the scenes changed, and encode was redrawn (the 98 encode prompts
|
|
||||||
are byte-identical to growth-base, no encode answer is; the refusals fell from
|
|
||||||
10 to 0 on the redraw, not through the scenes, which reach only the lowering
|
|
||||||
prompts). Soul `cut-calibrate-harness.s3` re-derived all 33 counts and 3 noise
|
|
||||||
terms below from the per-item answers without calling `calibrate`; they match.
|
|
||||||
|
|
||||||
Measured, and reproduced by `weksa loop calibrate` (trials: prompted 120, each
|
|
||||||
mild 60, Train-guide 240; Accepted / NoGain / Regressed):
|
|
||||||
|
|
||||||
| class | prompted | noise | mild-1 Regressed | mild-2 Regressed | Train-guide |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| all (35 per language) | 11/100/9 | 0.0392 | 51 | 58 | 85/35/120 |
|
|
||||||
| old (14) | 0/105/15 | 0.0844 | 60 | 60 | 40/80/120 |
|
|
||||||
| new (21) | 10/97/13 | 0.0340 | 3 | 4 | 50/129/61 |
|
|
||||||
|
|
||||||
Measured by Soul scratch scripts outside git (not reproduced by the harness,
|
|
||||||
except the R split in the first bullet, which `calibrate` prints as
|
|
||||||
`prompted-r-candidate`, `prompted-r-base` and `prompted-r-free` since the
|
|
||||||
`calibrate-report-r-involvement` cut; Soul re-ran growth-scene with it and the
|
|
||||||
printed `all` lines read r-candidate 11/9/0, r-base 0/11/9, r-free 0/80/0, as
|
|
||||||
below, in Accepted/NoGain/Regressed; likewise the per-arm Train-guide counts
|
|
||||||
(`candidate=h1..h4`) and the mild arms' composite offsets from the prompted
|
|
||||||
mean, R included (`offset`), which `calibrate` prints since
|
|
||||||
`calibrate-arm-counts-and-mild-offsets` and which reproduce the figures below,
|
|
||||||
Soul re-run on growth-scene):
|
|
||||||
|
|
||||||
- Every one of the 11 false accepts has R as the candidate (bases
|
|
||||||
prompted-2, p4, p6); every one of the 9 false rejects has R as the base. The
|
|
||||||
80 trials without R read 0 Accepted and 0 Regressed in every class (95% upper
|
|
||||||
bound 4.5%). R scores above the other prompted draws (+0.012 en-US and +0.023
|
|
||||||
ja-JP on all items), so R is not exchangeable with a repeat.
|
|
||||||
Which of the two rates the live rule should be held to is undecided: the
|
|
||||||
live loop meets R as the head's establishing lowering, so the R-candidate
|
|
||||||
rate may be the real one for cycles, while the 80 R-free trials are the
|
|
||||||
exchangeable repeat rate; neither number stands for the other.
|
|
||||||
- The old/new split of the false-accept lives only in the 20 trials where R is
|
|
||||||
the candidate, from 5 bases: new 10/20 [0.27, 0.73], old 0/20 [0.00, 0.17]
|
|
||||||
(Clopper-Pearson); by base 3 of 5 against 0 of 5 (Fisher p 0.17). The
|
|
||||||
10/120 against 0/120 comparison treats dependent trials as independent. The
|
|
||||||
claim that the new items drive the prompted false-accept is not supported;
|
|
||||||
the old items' larger noise term (2.4 times) raises their threshold, which
|
|
||||||
alone can produce the gap.
|
|
||||||
- Mild arms: they overlap 8 of 35 answers (en-US) and 9 (ja-JP) over the limit
|
|
||||||
of 7. They do not degrade the new items (composite -0.004 en-US
|
|
||||||
[-0.021, +0.012], +0.013 ja-JP for mild-1, +0.005 for mild-2); the Regressed counts come from the old
|
|
||||||
en-US items (-0.072). The rule's sensitivity on the new items is unmeasured.
|
|
||||||
- Train-guide pools unlike arms. h1 and h2 Regress in 120 of 120 trials; h3 and
|
|
||||||
h4 gain (Accepted 45 and 40 of 60); on the new items h3 is Accepted 50 of 60
|
|
||||||
and h4 0 of 60. The 35% pooled figure is not a power.
|
|
||||||
|
|
||||||
Extrapolated only: the rule's power at 35 items against any real gain beyond
|
|
||||||
h3 and h4, and any false-accept bound on repeat draws (5 draws, one of them R).
|
|
||||||
|
|
||||||
### Smallest measurable distil slice (`distil-incontext`, not built)
|
|
||||||
|
|
||||||
Distil is a tuned lowerer measured against the prompted one; the tuned model
|
|
||||||
needs a host (follow-up `distil-host-question-owed`). The smallest slice that
|
|
||||||
measures whether Train pairs buy a gain with the rule that exists is
|
|
||||||
training-free: an in-context arm, the prompted lowerer given Train
|
|
||||||
(packet, text) pairs as few-shot examples, answered through the one prompt door
|
|
||||||
and read by `loop relower` against three prompted repeats. Parts: 12b's
|
|
||||||
Train-only export (follow-up `distil-export-at-freeze`, guarded against Test
|
|
||||||
text) as the few-shot source; the arm's directory of answers; the existing
|
|
||||||
floor and verdict. A weaker version ran on 14 items: a lowerer shown the 28
|
|
||||||
Train sentences (no packets) beat the prompted draws in both languages and the
|
|
||||||
rule Accepted it in 29% of 1120 trials. The slice is decisive only with the
|
|
||||||
Validation growth above: at 14 items it can only fail to reject. It decides the
|
|
||||||
fine-tune question in one direction: if in-context Train pairs do not clear the
|
|
||||||
floor at 35 items, a tuned model's gain over the prompted one is not
|
|
||||||
measurable with this corpus either.
|
|
||||||
|
|
||||||
Training pairs are Train only, at a frozen revision; the export is 12b.
|
|
||||||
|
|
||||||
## Lowering authorities (`golf-loop-lowerer-cut`, `table-retirement`)
|
|
||||||
|
|
||||||
Authority map of who decides what about a packet's wording. The first cut gave
|
|
||||||
the loop a baseline from an English template table; cut `table-retirement`
|
|
||||||
deleted that table, `lower()` and the CLI lowering command, so no deterministic path
|
|
||||||
produces lowered text and the model's answer to `Prompt::lower` is the only
|
|
||||||
wording owner. `lower.rs` keeps the entity port, `pronounce` and `phones`
|
|
||||||
(`weksa pronounce`).
|
|
||||||
|
|
||||||
| Decision | Owner | Inputs | Outputs | Not the owner |
|
| Decision | Owner | Inputs | Outputs | Not the owner |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| Whether a packet exists | `validate` / `PacketStore::admit` (unchanged) | packet, inventory, corpus, entities | admitted packet or typed refusal | the lowerer |
|
| Whether a packet exists | `validate` / `PacketStore::admit` (unchanged) | packet, inventory, corpus, entities | admitted packet or typed refusal | the lowerer |
|
||||||
| Wording of a scored item in English (golf loop, `golf-loop-model-lowerer`) | the ModelPort's answer to `Prompt::lower` (`context.rs`), read as `LoweringDraft` (`encode.rs`) and held in `run::score_round`'s locals | the admitted view, the inventory at its revision, en-US entity forms | `{"text"}`, scored once, never stored | the corpus item, the scorer, the editor |
|
| Wording of a packet in English | `crates/weksa/src/lower.rs`: `lower(&LoweringInput, &Inventory) -> Result<String, LowerError>` | the validated lowering view and the inventory at its revision | text, or a typed refusal naming the missing template | the corpus item, the scorer, any model |
|
||||||
|
| English template table (concept words, frame templates, speech-act openers) | the same module, as rows `(inventory id, gloss it speaks for, surface)` | none; it is data in code | rows the lowerer looks up | the inventory (it owns the ids and glosses, not the English) |
|
||||||
| The inventory's concepts and frames | an inventory revision; here `inventory::proof_revision()` (rev 2), hand-seeded as `seed()` is | none | the 6 concepts and 3 frames `nibu-proof-001` and the consent-gated request packet name | the lowerer |
|
| The inventory's concepts and frames | an inventory revision; here `inventory::proof_revision()` (rev 2), hand-seeded as `seed()` is | none | the 6 concepts and 3 frames `nibu-proof-001` and the consent-gated request packet name | the lowerer |
|
||||||
| What a score counts | `Scorer::score` (unchanged) | output text, reference text | `ItemScore` | `weksa score`, which only reads text and calls it |
|
| What a score counts | `Scorer::score` (unchanged) | output text, reference text | `ItemScore` | `weksa score`, which only reads text and calls it |
|
||||||
| The proof packet | `packet::proof_packet()`, typed ids of rev 2, admitted by `weksa packet proof <store>` through `PacketStore::admit`. Not JSON: `Packet`'s serde is CultCache's positional tuple, and JSON in is the xenos boundary the cut 7 encoder owns | none | admitted packet | the CLI decides nothing about validity |
|
| The proof packet | `packet::proof_packet()`, typed ids of rev 2, admitted by `weksa packet proof <store>` through `PacketStore::admit`. Not JSON: `Packet`'s serde is CultCache's positional tuple, and JSON in is the xenos boundary the cut 7 encoder owns | none | admitted packet | the CLI decides nothing about validity |
|
||||||
|
|
||||||
Decisions this map makes, and why:
|
Decisions this map makes, and why:
|
||||||
|
|
||||||
|
- **`lower` takes `&LoweringInput`, not `&Packet`.** A `Packet` carries `trace`
|
||||||
|
and `provenance.corpus_item`; the model page says the lowerer sees the packet
|
||||||
|
without its trace and never the corpus item (`lowering-traced`,
|
||||||
|
`leak-owned-by-flow-and-scorer-3`). `validate` already yields the view, so the
|
||||||
|
CLI runs `validate` and hands it over. Cut 4 narrows the door to
|
||||||
|
`PacketStore::admit`; this cut does not wait on it.
|
||||||
|
- **The frame owns the verb phrase; a predication's concept must resolve but is
|
||||||
|
not rendered.** Referent concepts supply nouns. A frame template names its
|
||||||
|
roles by inventory gloss (`{agent}`), resolved against the inventory at the
|
||||||
|
packet's revision, so a retired or re-glossed role cannot be silently
|
||||||
|
mis-keyed. A row whose gloss differs from the inventory's entry for its id is
|
||||||
|
a `LowerError`, so a table row cannot speak for a different meaning than its
|
||||||
|
id now carries.
|
||||||
|
- **Rendering rules, all deterministic and all from packet fields:** the
|
||||||
|
discourse addressee renders "you" ("yourself" when it already filled a role
|
||||||
|
in the same predication), other referents render "the <concept word>", a
|
||||||
|
predication filling a role renders "that", the speech-act row supplies an
|
||||||
|
optional opener, sentences are capitalised and end in a full stop.
|
||||||
|
Entity-bearing referents render by concept word: the lowerer does not read
|
||||||
|
the entity store in this cut (follow-up: entity forms in the lowerer).
|
||||||
|
- **No style, no register, no features.** Hedge, certainty, modality and
|
||||||
|
polarity are not rendered, so the baseline scores what meaning alone buys.
|
||||||
|
A lowerer that ignores `polarity: negative` would assert the opposite, so
|
||||||
|
`lower` refuses any packet that carries a feature, relation or extension it
|
||||||
|
does not render rather than render it wrong (`packet-owns-meaning`); features arrive as templates
|
||||||
|
learn them.
|
||||||
|
- **Deleted by later cuts, not this one:** the table in `lower.rs` and
|
||||||
|
`proof_revision()` are temporary owners. Cut 5 (`language-ontology`) moves
|
||||||
|
the table to `weksa.ontology_entry` documents selected by the target
|
||||||
|
profile; cut 7 replaces `proof_revision()` with the loop's minting round.
|
||||||
|
Nothing is deleted today because no earlier lowerer exists; the YAML
|
||||||
|
`examples/english-lowering/` hand-lowerings are fixtures, not owners, and
|
||||||
|
stay.
|
||||||
- **`weksa score` is a thin reader of `Scorer::score`**: `weksa score <item>...
|
- **`weksa score` is a thin reader of `Scorer::score`**: `weksa score <item>...
|
||||||
[--output FILE]` embeds the output text once against every named item. It
|
[--output FILE]` embeds the output text once against every named item. It
|
||||||
stores nothing (Score documents arrive with cut 7) and gates nothing.
|
stores nothing (Score documents arrive with cut 7) and gates nothing.
|
||||||
|
|
@ -641,20 +397,15 @@ One line per cut; specs exist for the cuts about to run (2d and 4, siblings on `
|
||||||
2d. `entity-door-lows`: invisible format characters refused in entity forms (ZWJ, ZWNJ, CGJ and variation selectors kept); the form length bound per question `form-word-bound` (recommended: bytes only); attested origins (`Cast { persona, fixture }`, `Live { persona }`) and `CastAuthored.author`; empty provenance strings refused at the store; S6 under NFKC plus case fold and no-alphanumeric texts refused; `next_key` refuses at `u32::MAX`; CLI flags that do not apply refuse and no error echoes a value. Owns follow-ups `entity-door-lows` and `s6-unicode-normal-form`. (spec r1)
|
2d. `entity-door-lows`: invisible format characters refused in entity forms (ZWJ, ZWNJ, CGJ and variation selectors kept); the form length bound per question `form-word-bound` (recommended: bytes only); attested origins (`Cast { persona, fixture }`, `Live { persona }`) and `CastAuthored.author`; empty provenance strings refused at the store; S6 under NFKC plus case fold and no-alphanumeric texts refused; `next_key` refuses at `u32::MAX`; CLI flags that do not apply refuse and no error echoes a value. Owns follow-ups `entity-door-lows` and `s6-unicode-normal-form`. (spec r1)
|
||||||
3b. `lowerer-first-cut` (follow-up `golf-loop-lowerer-cut`): the section above. `lower`, `weksa lower`, `weksa score`, `weksa packet proof`, inventory rev 2 for `nibu-proof-001`.
|
3b. `lowerer-first-cut` (follow-up `golf-loop-lowerer-cut`): the section above. `lower`, `weksa lower`, `weksa score`, `weksa packet proof`, inventory rev 2 for `nibu-proof-001`.
|
||||||
4. `speaker-state`: packet `stance` (felt affect and restraint, slot `Stance`, packet schema v4); formality, warmth and intensity out of the seed and the standard; only `PacketStore::admit` yields a `LoweringInput`, and it refuses packets for withdrawn items (follow-up `lowering-input-from-packet-store`). (spec r1)
|
4. `speaker-state`: packet `stance` (felt affect and restraint, slot `Stance`, packet schema v4); formality, warmth and intensity out of the seed and the standard; only `PacketStore::admit` yields a `LoweringInput`, and it refuses packets for withdrawn items (follow-up `lowering-input-from-packet-store`). (spec r1)
|
||||||
4a. `speaker-relation-input`: slot `Scene` and its first seed features (`relative_status`, `intimacy`, `in_group`; inventory revision 3, now `speaker_scene_revision`); `LoweringInput::with_scene` carries the speaker's relation to the addressee as typed `Scene` feature values. Since 4b it takes the `SpeakerStateStore` and the `CorpusStore`, reads the scene of the view's own item group for the item's speaker, and selects the relation by the addressee referent's entity (V16 when the scene does not list the item's speaker or the speaker of any live item of the group, the rule `put_scene` applies too; V17 refuses an addressee that names a predication; V13 is retired). The English lowerer does not render it. (PR for follow-up `chat-s3352-3352`)
|
4a. `speaker-relation-input`: slot `Scene` and its first seed features (`relative_status`, `intimacy`, `in_group`; inventory revision 3, `scene_revision`); `LoweringInput::with_scene` takes the speaker's relation to the addressee as typed `Scene` feature values, validated like every feature list, refused without an addressee (V13). The English lowerer does not render it. 4b's `weksa.scene` document is its future supplier; no store, document or CLI yet. (PR for follow-up `chat-s3352-3352`)
|
||||||
4b. `speaker-scene` (spec r1; `speaker-scene-seeds` is its cut 4c: CLI, inventory revision 3, cast speakers and scenes; `Party::Unnamed` covers an addressee with no entity; Speaker and Scene share one `SpeakerStateStore`): slots `Speaker` and `Scene` with their seed features; `weksa.speaker` (`speaker:<entity-slug>@<n>`) and `weksa.scene` (`scene:<group>@<n>`, participants and ordered-pair relations) documents with `Annotator` `Operator` and `Model { family, generator }`, validated by the packet's shared feature check, append-only per key; their stores, `weksa speaker` and `weksa scene` commands, published schemas; Model-annotated cast speakers and scenes for the committed items. Waits on 2d and 4; `pd-speaker-annotation` blocks only the public-domain annotator. Part 1 landed (branch `weksa-lowering/speaker-scene`): `speaker.rs`, `SpeakerStateStore`, slots `Speaker` and `Scene` exclusive to their own feature lists, `speaker_scene_revision()` (Speaker features 22-24), `with_scene` on the store, schemas. The CLI, the committed inventory revision 3 and the seeds stay with 4c.
|
4b. `speaker-scene`: slots `Speaker` and `Scene` with their seed features; `weksa.speaker` (`speaker:<entity-slug>@<n>`) and `weksa.scene` (`scene:<group>@<n>`, participants and ordered-pair relations) documents with `Annotator` `Operator` and `Model { family, generator }`, validated by the packet's shared feature check, append-only per key; their stores, `weksa speaker` and `weksa scene` commands, published schemas; Model-annotated cast speakers and scenes for the committed items. Waits on 2d and 4; `pd-speaker-annotation` blocks only the public-domain annotator.
|
||||||
5. `language-ontology`, in sequential cuts on `d9ebbcf`. 5a `language-ontology` (spec r1): `weksa.ontology_entry`, `OntologyStore`, `ontology::select` over the view's `Stance` and `Discourse` features, `Prompt::lower` printing the selected entries of the item's language, `RunRecord` pinning the ontology store (keys 15 and 16), `weksa ontology admit|dump`; `ontology.rs` joins `PROMPT_SOURCE_DIGEST`. 5b `language-ontology-seed` (spec r1, after 5a): Model-annotated en-US and ja-JP seed entries drafted by an isolated model, `data/target-language-ontology/`, `.weksa/cultural-ontology/` and `schemas/target-language-cultural-ontology.md` deleted. Lost until 5c': keigo, hierarchy and uchi-soto hang on the scene relation from 5c', so ja-JP politeness is still the lowerer's guess. 5c' `language-ontology-scene` (spec r1, after 5b): the run pins the `SpeakerStateStore` (keys 17 and 18, `Drift(SpeakerState)`), `RunStores::assembled` is the one door that reads the item's scene into a loop view, `select` reads slot `Scene`, `Prompt::lower` refuses a view whose scene was never read, `weksa scene put|dump` and `weksa inventory speaker-scene` (rev 4 in data/original and data/daodejing, rev 3 in data/interlingua, which holds no Dao revision), Model-annotated keeper scenes and Scene-conditioned entries for `data/original`. 5c `language-ontology-view-assembly` (follow-up, now only the Speaker read): `select` widens to slot `Speaker`; waits on `speaker-scene-seeds`. 5d `language-ontology-communities` (follow-up): `weksa.community` built fresh (`data/cultural-ontology/` is deleted by 5e), community ids on `weksa.speaker`, community conditions; waits on 5c and question `persona-speech-community-home`.
|
5. `language-ontology`: `weksa.community` from `data/cultural-ontology/` (owns follow-up `edda-referent-yaml-parse`), `weksa.ontology_entry` for en-US and ja-JP from `data/target-language-ontology/` (YAML deleted), speaker communities, deterministic `select`, the lowering view (packet without trace, speaker, scene, selected entries). Waits on 4.
|
||||||
5e. `delete-cultural-ontology` (steward ruling, 2026-11-13): `data/cultural-ontology/` (five draft YAML profile files and a README; `old-norse-edda-source-tradition.yaml` did not parse) deleted. Nothing in `crates/`, tests, scripts or CI read it; `examples/persona-lowering/brokkr-edda-voice/` cites its paths as text only (the one test that reads that example reads `lowered-output.yaml`). The typed `weksa.community` of follow-up `language-ontology-communities` is built fresh under cut 5; follow-up `edda-referent-yaml-parse` is moot.
|
|
||||||
6. `pd-importer`: public-domain play and fiction dialogue into the corpus with speaker, scene group, edition provenance, and Speaker and Scene annotations by the ruled annotator. Waits on 4, follow-up `gutenberg-terms` and question `pd-speaker-annotation`.
|
6. `pd-importer`: public-domain play and fiction dialogue into the corpus with speaker, scene group, edition provenance, and Speaker and Scene annotations by the ruled annotator. Waits on 4, follow-up `gutenberg-terms` and question `pd-speaker-annotation`.
|
||||||
6a. `pd-name-spans` (spec r1): `weksa corpus import <work> <names>` mints a work's cast as `PublicDomain` entities with forms in the work's language and adds each line a name-span answer covers, speaker from the work file, spans from `[[entity|form]]` markup, labelled `names_by` (Model or Operator) per `docs/name-span-annotation.md`. Speaker and Scene documents, prior turns and fiction speaker attribution stay with 6.
|
7a. `loop-context-builder`: `context.rs` (`TrainItems`, `EncoderOffer`, `Prompt`: corpus text reaches a model call only through `Prompt::mint` over live Train items or `Prompt::encode` over one offered item; instructions are `&'static str`), `SplitScheme::live`, `CorpusItem::cited_entities` (one owner of the id set `PacketStore::admit` and the offer share), and checkpoint-only Test readings (`Checkpoint`, `TestReadings`; `Aggregate` and `leak_diagnostic` carry no Test). Library only. Merges follow-ups `loop-context-builder-2` and `scorer-test-readings-type`. (spec r1)
|
||||||
7a. `loop-context-builder`: `context.rs` (`TrainItems`, `EncoderOffer`, `Prompt`: corpus text reaches a model call only through `Prompt::mint` over live Train items or `Prompt::encode` over one offered item; instructions are the consts `MINT_INSTRUCTIONS` and `ENCODE_INSTRUCTIONS` in `context.rs`; `Checkpoint` has no public constructor, offers each item once, and its offers borrow it (the gate is typed `TestReadings` and Test offers on the round path; operator tools such as `weksa corpus dump` and `weksa score` still read corpus text of any split); prior turns print the held Train text), `SplitScheme::live`, `CorpusItem::cited_entities` (one owner of the id set `PacketStore::admit` and the offer share), and checkpoint-only Test readings (`Checkpoint`, `TestReadings`; `Aggregate` and `leak_diagnostic` carry no Test). Library only; since 7b `TrainItems::from_corpus` and `EncoderOffer::for_round` are crate-private and `RunStores` is their one public producer. Merges follow-ups `loop-context-builder-2` and `scorer-test-readings-type`. (spec r1)
|
7. `golf-loop-en`: the loop driver, the packet store bound to one inventory store (`packet-store-inventory-binding`), `ModelPort` with the `claude -p` adapter (it takes only `&Prompt`), encoder (packet with stance) and lowerer prompts, round records, Score documents, the composite and accept rule, `weksa score`, the detached unit on Yggdrasil. Waits on 3, 5, 6 and follow-up `claude-on-yggdrasil`.
|
||||||
7b. `golf-loop-round` (spec r1; branch `weksa-lowering/golf-loop-round`): the smallest loop that scores one round. `run.rs`: `RunStores` pins the corpus and entity stores of a run by canonical path and FNV-1a 64 digest at `create`, reopens only from `<root>/run.cc` and refuses drift (corpus, entities, `PROMPT_SOURCE_DIGEST` of `context.rs`, a second pin on `lower.rs` retired in 7d, model, embedder); `round` encodes every live Train and Validation item through a `ModelPort` (`model.rs`; the one adapter `AnswerFiles` writes `<digest>.prompt` and reads `<digest>.answer`, so the model is the Claude session), reads each answer into a `PacketDraft` (`encode.rs`, the one place a model's JSON is read), admits through `PacketStore::admit` into `<root>/round-<n>/packets.cc`, lowers, scores once, and keeps a `RoundRecord` (`weksa.loop_round`, no text, no Test; Train rows, Validation only as means and refusal counts). `weksa loop open|round`. Not here: the spec-edit step, accept rule and composite, noise floor, Score and lowering documents, a subprocess adapter, the checkpoint door, a model lowerer (follow-ups `golf-loop-spec-edit`, `golf-loop-accept-rule`, `golf-loop-score-documents`, `golf-loop-subprocess-adapter`, `golf-loop-checkpoint-door-bound`, `golf-loop-model-lowerer`). `packet-store-inventory-binding` is met for loop packets only (`RunStores` admits with the run's inventory); `validate`, `packet proof|dao` and `lower` still take caller stores. Waits on 7a, `checkpoint-gate`, 4b.
|
|
||||||
7c. `golf-loop-edit-accept` (spec r1; branch `weksa-lowering/golf-loop-edit-accept`): one edit-accept cycle. The run works on its own copy of the inventory (`create` copies the caller's file to `<root>/inventory.cc`, refusing a root that already holds a file of that name, and pins its digest as `RunRecord` key 11; `RunRecord.inventory` is provenance; a `run.cc` from before has no key 11 and does not open); the head is derived from the latest Accepted `Decision`. `weksa loop cycle` asks the editor for one inventory edit from `Prompt::mint(train, inventory, rev, feedback)` (Train blocks, the inventory listing and an `EditFeedback` line: whether the last edit was kept and the head's Validation composite rounded to `LADDER_STEP`, the Ladder release of Blum and Hardt 2015), reads it as an `EditDraft` (`encode.rs`; ids and revisions are the store's, `into_revision`), admits it through `InventoryStore::admit_revision` into `<root>/cycle-<k>/inventory.cc`, runs a candidate round under it, and `decide`s on Validation against the head's establishing round past a floor (never under one `LADDER_STEP`; from cut `cycle-floor-paired` the paired floor over rounds 1 to `FLOOR_ARMS_MIN` + 1, so a cycle needs four recorded rounds, where 7c measured the spread between rounds 1 and 2). An edit that fails is a rejected `Decision`, not an error. `RoundRecord` gains keys 16 (inventory digest) and 17 (cycle); an encode answer that is not UTF-8 is that item's parse refusal. Not here: Score and lowering documents (`golf-loop-score-docs`), Train rows in the edit prompt (`golf-loop-edit-train-feedback`), a paired floor (`golf-loop-paired-floor`), the checkpoint door and merging to `data/` (`golf-loop-checkpoint-door-bound`, `golf-loop-freeze-merge`), a model lowerer (landed as 7d: until then an entry an edit added had no lowering row). Waits on 7b.
|
|
||||||
7d. `golf-loop-model-lowerer` (spec r1; branch `weksa-lowering/golf-loop-model-lowerer`): the round lowers through the `ModelPort`. `Prompt::lower` (`context.rs`) takes a `LoweringInput` (no text, key, group or speaker getter), the item's speaker id, the inventory at the view's revision and the item language's entity forms, and prints `LOWER_INSTRUCTIONS`, the meaning of the packet, the entities line and the inventory listing; the item key goes only to `item_keys`, and the speaker id never prints: a referent citing the item's speaker holds the fixed token `entity:speaker` (`SPEAKER_TOKEN`, a constant the same for every speaker and item, never derived from an id) and the entities map holds the speaker's forms under that token. The promise guards the id that names an entity of this item; the speaker's forms and the role token do print, because forms are shared by every item of the language (follow-up `speaker-alias-token-in-lowering-prompt`, which restored the lowerer's ability to name its speaker after #84 removed it). A speaker the language has no form for, or the port lacks, does not refuse: its referents print no entity and the map has no entry. A referent citing the real id `entity:speaker` is refused (`LowerError::SpeakerTokenTaken`); the entity store does not reserve it. The answer is `{"text"}`, read as `LoweringDraft` in `encode.rs`, held in `score_round`'s locals and scored once; it is never put in `run.cc`, a `RoundRecord`, a `Decision` or `EditFeedback`. A round has two answer phases (every encode prompt, then every lowering prompt), all or nothing: while any prompt is unanswered `packets.cc` is discarded and nothing reaches `run.cc`. `lower::lower` and the template table leave the loop (they remain `weksa lower`'s; follow-up `table-retirement`), so an entry an edit adds can be said and a cycle can buy a gain. One source pin remains: `PROMPT_SOURCE_DIGEST` covers `context.rs`, `split.rs` and `lower.rs` (`RunRecord` key 7 and `RoundRecord` key 3 are retired). The run's record pins `LowererReach` (`weksa loop open --lowerer-reach briefed-subagent|tool-less-relay`, `RunRecord` key 12) and every printed round and cycle line says it; a briefed-subagent run says its numbers are smoke-round only (rulings `lowerer-call-reach-toolless-relay`, `dao-round-seed-inventory`). A lowering is refused at `Lower` when blank, over `MAX_LOWERING_BYTES` (4096) or holding a control character other than a line break or tab; `PROMPT_SOURCE_DIGEST` also covers `packet.rs`. Not here: per-item lowering text documents (`golf-loop-score-docs`), the subprocess adapter. Waits on 7c.
|
|
||||||
7. `golf-loop-en` (7b, 7c and 7d landed the round, `ModelPort` with `AnswerFiles`, encoder and lowering prompts, template digest, round records and the edit-accept cycle; the rest remains): the packet store bound to one inventory store everywhere (`packet-store-inventory-binding`), Score documents (`golf-loop-score-docs`, `golf-loop-score-documents`), the accept rule (`golf-loop-accept-rule`), the checkpoint door and freeze merge (`golf-loop-checkpoint-door-bound`, `golf-loop-freeze-merge`). No subprocess adapter and no detached unit: `golf-loop-in-claude` and `encoder-isolation-subagent-per-prompt` rule them out. Waits on 3, 5 and 6.
|
|
||||||
8. `style-axis`: a style-embedding score behind the scorer port. Waits on 3.
|
8. `style-axis`: a style-embedding score behind the scorer port. Waits on 3.
|
||||||
9. `ja-panel`: ja-JP into the loop's panel, in two cuts on `e9cbd9b`. 9a `ja-panel-loop` (spec r1): language is the item's and the panel the run's: `Prompt::lower` takes the item's language and prints it on a language line (no prompt names English), entity forms are read in that language, `RunRecord` pins the panel (`weksa loop open --panel`), offers, mint and recall read only panel items, round means, floors and readings are per panel language, the accept rule runs `decide` per language, the recall split works on unspaced scripts, and `weksa corpus add` has no default language. 9b `ja-panel-corpus` (spec r1, after 9a): the ja-JP slice of `data/original`, model-written Japanese Train and Validation passages and eight Aozora Bunko public-domain Test sentences with full provenance, and their entities. Cut 5 is not a prerequisite: neither language lowers with ontology entries until cut 5 gives both theirs.
|
9. `ja-panel`: ja-JP into the loop's panel. Waits on 7.
|
||||||
10. `conlang-lexicon`: the first conlang's lexicon placed at concept embeddings, its morphology data, its ontology entries and the gloss reader. Waits on 3, 5 and question `first-conlang`.
|
10. `conlang-lexicon`: the first conlang's lexicon placed at concept embeddings, its morphology data, its ontology entries and the gloss reader. Waits on 3, 5 and question `first-conlang`.
|
||||||
11. `convergence-freeze`: the convergence test, `Checkpoint`'s one door (built from the convergence window over the persisted round records), the freeze, the re-encode into training pairs. Waits on 7, 9 and 10.
|
11. `convergence-freeze`: the convergence test, the freeze, the re-encode into training pairs. Waits on 7, 9 and 10.
|
||||||
12. `distil`, three cuts. 12a `distil-slice` (relower and the prompted baseline; spec r1). 12b `distil-export`: Train-only prompt/completion JSONL at a frozen revision, Test-text guarded (follow-up `distil-export-at-freeze`); waits on 11. 12c `distil-train`: waits on 12b and the host question (follow-up `distil-host-question-owed`).
|
12. `distil`: fine-tune a lowerer on the frozen training set. Waits on 11 and a host question asked then.
|
||||||
13. `live-lowering`: agents emit packets with stance; Weksa reads speaker state from persona state (follow-up `persona-state-speech-community`) and lowers for the reader. Waits on 11.
|
13. `live-lowering`: agents emit packets with stance; Weksa reads speaker state from persona state (follow-up `persona-state-speech-community`) and lowers for the reader. Waits on 11.
|
||||||
|
|
|
||||||
|
|
@ -25,8 +25,8 @@ ground truth is Discord text, which can never feed this loop
|
||||||
|
|
||||||
An agent's outward speech today is English text the model chose, so meaning,
|
An agent's outward speech today is English text the model chose, so meaning,
|
||||||
register and voice are fused in one string nobody can inspect or re-target.
|
register and voice are fused in one string nobody can inspect or re-target.
|
||||||
Weksa's design already separates them: the packet owns meaning, speaker and scene state own address and politeness
|
Weksa's design already separates them: the packet owns meaning, the target
|
||||||
(the language ontology holds only per-language lexicon, grammar and discourse choices), the lowerer owns
|
profile owns address, politeness and forbidden flattenings, the lowerer owns
|
||||||
surface text and a trace. The campaign makes that design real for a live
|
surface text and a trace. The campaign makes that design real for a live
|
||||||
reader, and gives Weksa's Persona a working body when her mind
|
reader, and gives Weksa's Persona a working body when her mind
|
||||||
(`eureka-body:follow_up:weksa-mind`) wakes.
|
(`eureka-body:follow_up:weksa-mind`) wakes.
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,11 @@
|
||||||
# CultCache Lowering Examples
|
# CultCache Lowering Examples
|
||||||
|
|
||||||
These examples lower interlingua packets for separate target languages. The
|
These examples lower interlingua packets through CultCache cultural ontology
|
||||||
language ontology that owns en-US and ja-JP choices is `data/ontology/ontology.cc`
|
profiles rather than the older YAML-only target-language profiles.
|
||||||
(`weksa ontology dump`); its entries are model-drafted seeds.
|
|
||||||
|
The starter database is:
|
||||||
|
|
||||||
|
- `.weksa/cultural-ontology/starter-targets.cc`
|
||||||
|
|
||||||
Each target file is produced as a separate agent run so differences in target
|
Each target file is produced as a separate agent run so differences in target
|
||||||
ontology remain inspectable instead of being smoothed into one multilingual
|
ontology remain inspectable instead of being smoothed into one multilingual
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,10 @@ Prompt:
|
||||||
|
|
||||||
- `prompts/nibu-cultcache-target-lowering.md`
|
- `prompts/nibu-cultcache-target-lowering.md`
|
||||||
|
|
||||||
|
CultCache profile store:
|
||||||
|
|
||||||
|
- `.weksa/cultural-ontology/starter-targets.cc`
|
||||||
|
|
||||||
Expected target outputs:
|
Expected target outputs:
|
||||||
|
|
||||||
- `pt-BR-vanilla-brazilian.yaml`
|
- `pt-BR-vanilla-brazilian.yaml`
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
schema_version: weksa.utterance_lowering.agent_run.v0
|
schema_version: weksa.utterance_lowering.agent_run.v0
|
||||||
source_packet_id: nibu-proof-001
|
source_packet_id: nibu-proof-001
|
||||||
source_packet_ref: examples/english-lowering/nibu-proof-001/interlingua.yaml
|
source_packet_ref: examples/english-lowering/nibu-proof-001/interlingua.yaml
|
||||||
|
target_profile_ref: .weksa/cultural-ontology/starter-targets.cc
|
||||||
target_profile_id: weksa.cultural_ontology.en.transatlantic.v0
|
target_profile_id: weksa.cultural_ontology.en.transatlantic.v0
|
||||||
target_agent_id: nibu
|
target_agent_id: nibu
|
||||||
realization:
|
realization:
|
||||||
|
|
@ -10,6 +11,8 @@ realization:
|
||||||
locale: en-001
|
locale: en-001
|
||||||
script: Latn
|
script: Latn
|
||||||
register: cultivated transatlantic English; cross-Atlantic intelligibility; measured technical warning
|
register: cultivated transatlantic English; cross-Atlantic intelligibility; measured technical warning
|
||||||
|
cultural_ontology_refs:
|
||||||
|
- .weksa/cultural-ontology/starter-targets.cc
|
||||||
cultural_ontology_profile_ids:
|
cultural_ontology_profile_ids:
|
||||||
- weksa.cultural_ontology.en.transatlantic.v0
|
- weksa.cultural_ontology.en.transatlantic.v0
|
||||||
outputs:
|
outputs:
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
schema_version: weksa.utterance_lowering.agent_run.v0
|
schema_version: weksa.utterance_lowering.agent_run.v0
|
||||||
source_packet_id: nibu-proof-001
|
source_packet_id: nibu-proof-001
|
||||||
source_packet_ref: examples/english-lowering/nibu-proof-001/interlingua.yaml
|
source_packet_ref: examples/english-lowering/nibu-proof-001/interlingua.yaml
|
||||||
|
target_profile_ref: .weksa/cultural-ontology/starter-targets.cc
|
||||||
target_profile_id: weksa.cultural_ontology.ja-JP.kanto.contemporary.v0
|
target_profile_id: weksa.cultural_ontology.ja-JP.kanto.contemporary.v0
|
||||||
target_agent_id: nibu
|
target_agent_id: nibu
|
||||||
realization:
|
realization:
|
||||||
|
|
@ -10,6 +11,8 @@ realization:
|
||||||
locale: ja-JP
|
locale: ja-JP
|
||||||
script: Jpan
|
script: Jpan
|
||||||
register: contemporary Kanto metropolitan clipped hazard warning
|
register: contemporary Kanto metropolitan clipped hazard warning
|
||||||
|
cultural_ontology_refs:
|
||||||
|
- .weksa/cultural-ontology/starter-targets.cc
|
||||||
cultural_ontology_profile_ids:
|
cultural_ontology_profile_ids:
|
||||||
- weksa.cultural_ontology.ja-JP.kanto.contemporary.v0
|
- weksa.cultural_ontology.ja-JP.kanto.contemporary.v0
|
||||||
outputs:
|
outputs:
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
schema_version: weksa.utterance_lowering.agent_run.v0
|
schema_version: weksa.utterance_lowering.agent_run.v0
|
||||||
source_packet_id: nibu-proof-001
|
source_packet_id: nibu-proof-001
|
||||||
source_packet_ref: examples/english-lowering/nibu-proof-001/interlingua.yaml
|
source_packet_ref: examples/english-lowering/nibu-proof-001/interlingua.yaml
|
||||||
|
target_profile_ref: .weksa/cultural-ontology/starter-targets.cc
|
||||||
target_profile_id: weksa.cultural_ontology.pt-BR.vanilla_brazilian.v0
|
target_profile_id: weksa.cultural_ontology.pt-BR.vanilla_brazilian.v0
|
||||||
target_agent_id: nibu
|
target_agent_id: nibu
|
||||||
realization:
|
realization:
|
||||||
|
|
@ -10,6 +11,8 @@ realization:
|
||||||
locale: pt-BR
|
locale: pt-BR
|
||||||
script: Latn
|
script: Latn
|
||||||
register: casual_clear
|
register: casual_clear
|
||||||
|
cultural_ontology_refs:
|
||||||
|
- .weksa/cultural-ontology/starter-targets.cc
|
||||||
cultural_ontology_profile_ids:
|
cultural_ontology_profile_ids:
|
||||||
- weksa.cultural_ontology.pt-BR.vanilla_brazilian.v0
|
- weksa.cultural_ontology.pt-BR.vanilla_brazilian.v0
|
||||||
outputs:
|
outputs:
|
||||||
|
|
|
||||||
|
|
@ -22,7 +22,7 @@ realizations:
|
||||||
script: Latn
|
script: Latn
|
||||||
register: sharp technical casual
|
register: sharp technical casual
|
||||||
cultural_ontology_refs:
|
cultural_ontology_refs:
|
||||||
- data/ontology/ontology.cc
|
- data/target-language-ontology/en-US.yaml
|
||||||
outputs:
|
outputs:
|
||||||
visible_action: The cockpit lights snap from amber to surgical white around the damaged control.
|
visible_action: The cockpit lights snap from amber to surgical white around the damaged control.
|
||||||
spoken_text: "Take your hand off that control unless you want your nervous system to become my least useful circuit diagram."
|
spoken_text: "Take your hand off that control unless you want your nervous system to become my least useful circuit diagram."
|
||||||
|
|
@ -52,6 +52,8 @@ realizations:
|
||||||
locale: pt-BR
|
locale: pt-BR
|
||||||
script: Latn
|
script: Latn
|
||||||
register: direct technical casual
|
register: direct technical casual
|
||||||
|
cultural_ontology_refs:
|
||||||
|
- data/target-language-ontology/pt-BR.yaml
|
||||||
outputs:
|
outputs:
|
||||||
visible_action: As luzes do cockpit passam de âmbar para branco cirúrgico ao redor do controle danificado.
|
visible_action: As luzes do cockpit passam de âmbar para branco cirúrgico ao redor do controle danificado.
|
||||||
spoken_text: "Tira a mão desse controle, a não ser que você queira transformar o seu sistema nervoso no meu diagrama de circuitos menos útil."
|
spoken_text: "Tira a mão desse controle, a não ser que você queira transformar o seu sistema nervoso no meu diagrama de circuitos menos útil."
|
||||||
|
|
@ -88,7 +90,7 @@ realizations:
|
||||||
script: Jpan
|
script: Jpan
|
||||||
register: sharp technical plain
|
register: sharp technical plain
|
||||||
cultural_ontology_refs:
|
cultural_ontology_refs:
|
||||||
- data/ontology/ontology.cc
|
- data/target-language-ontology/ja-JP.yaml
|
||||||
outputs:
|
outputs:
|
||||||
visible_action: 損傷した制御盤の周囲で、操縦室の灯りが琥珀色から外科灯のような白へ切り替わる。
|
visible_action: 損傷した制御盤の周囲で、操縦室の灯りが琥珀色から外科灯のような白へ切り替わる。
|
||||||
spoken_text: "その制御盤から手を離して。神経系を、私のいちばん役に立たない回路図にしたくないなら。"
|
spoken_text: "その制御盤から手を離して。神経系を、私のいちばん役に立たない回路図にしたくないなら。"
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,7 @@ referent without falling into generic fantasy-smith voice.
|
||||||
The test is deliberately small: one interlingua packet, one rendered utterance,
|
The test is deliberately small: one interlingua packet, one rendered utterance,
|
||||||
and one trace naming the activated cultural ontology axes.
|
and one trace naming the activated cultural ontology axes.
|
||||||
|
|
||||||
Profile stack (these files were deleted from main by cut `delete-cultural-ontology`; they remain in git history before that commit, and the paths below are kept as the fixture's record of what it cited):
|
Profile stack:
|
||||||
|
|
||||||
- `data/cultural-ontology/persona-stacks/brokkr.yaml`
|
- `data/cultural-ontology/persona-stacks/brokkr.yaml`
|
||||||
- `data/cultural-ontology/referents/old-norse-edda-source-tradition.yaml`
|
- `data/cultural-ontology/referents/old-norse-edda-source-tradition.yaml`
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,6 @@
|
||||||
schema_version: weksa.persona_lowering_fixture_batch.v0
|
schema_version: weksa.persona_lowering_fixture_batch.v0
|
||||||
fixture_id: brokkr-edda-voice
|
fixture_id: brokkr-edda-voice
|
||||||
speaker_id: brokkr
|
speaker_id: brokkr
|
||||||
# The data/cultural-ontology files cited below were deleted from main (cut delete-cultural-ontology); see git history.
|
|
||||||
cultural_stack_ref: data/cultural-ontology/persona-stacks/brokkr.yaml
|
cultural_stack_ref: data/cultural-ontology/persona-stacks/brokkr.yaml
|
||||||
target_register:
|
target_register:
|
||||||
language: en-US
|
language: en-US
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,6 @@
|
||||||
schema_version: weksa.persona_culture_lowering_trace.v0
|
schema_version: weksa.persona_culture_lowering_trace.v0
|
||||||
fixture_id: brokkr-edda-voice
|
fixture_id: brokkr-edda-voice
|
||||||
item_id: command-without-receipt
|
item_id: command-without-receipt
|
||||||
# The data/cultural-ontology files cited below were deleted from main (cut delete-cultural-ontology); see git history.
|
|
||||||
profile_stack:
|
profile_stack:
|
||||||
- profile_id: weksa.cultural_referent.old_norse_edda_source_tradition.v0
|
- profile_id: weksa.cultural_referent.old_norse_edda_source_tradition.v0
|
||||||
ref: data/cultural-ontology/referents/old-norse-edda-source-tradition.yaml
|
ref: data/cultural-ontology/referents/old-norse-edda-source-tradition.yaml
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,8 @@ You are reviewing one target lowering produced from the Nibu interlingua packet.
|
||||||
`examples/english-lowering/nibu-proof-001/projected-context.yaml`
|
`examples/english-lowering/nibu-proof-001/projected-context.yaml`
|
||||||
- Lowering prompt:
|
- Lowering prompt:
|
||||||
`prompts/nibu-cultcache-target-lowering.md`
|
`prompts/nibu-cultcache-target-lowering.md`
|
||||||
|
- CultCache profile store:
|
||||||
|
`.weksa/cultural-ontology/starter-targets.cc`
|
||||||
|
|
||||||
The source meaning is:
|
The source meaning is:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -24,10 +24,20 @@ text. Lower from the interlingua packet and projected context.
|
||||||
|
|
||||||
## Cultural Ontology Authority
|
## Cultural Ontology Authority
|
||||||
|
|
||||||
The language ontology is `data/ontology/ontology.cc` (`weksa ontology dump`).
|
Use the CultCache store:
|
||||||
It holds model-drafted seed entries for en-US and ja-JP only; there is no
|
|
||||||
pt-BR entry yet. Treat an entry as a draft: say in your trace which entries you
|
- `.weksa/cultural-ontology/starter-targets.cc`
|
||||||
used, and when no entry covers the target, say so rather than inventing one.
|
|
||||||
|
It contains:
|
||||||
|
|
||||||
|
- `weksa.cultural_ontology.pt-BR.vanilla_brazilian.v0`
|
||||||
|
- `weksa.cultural_ontology.en.transatlantic.v0`
|
||||||
|
- `weksa.cultural_ontology.ja-JP.kanto.contemporary.v0`
|
||||||
|
|
||||||
|
Qdrant-style retrieval and old YAML profiles may be useful context, but the
|
||||||
|
CultCache profile named in your task is the target authority. Treat the target
|
||||||
|
profile as draft guarded typed state: preserve its activation gates, forbidden
|
||||||
|
flattenings, and nested linguistic affordance feature IDs in your trace.
|
||||||
|
|
||||||
## Output Contract
|
## Output Contract
|
||||||
|
|
||||||
|
|
@ -40,6 +50,8 @@ The file must use this shape:
|
||||||
schema_version: weksa.utterance_lowering.agent_run.v0
|
schema_version: weksa.utterance_lowering.agent_run.v0
|
||||||
source_packet_id: nibu-proof-001
|
source_packet_id: nibu-proof-001
|
||||||
source_packet_ref: examples/english-lowering/nibu-proof-001/interlingua.yaml
|
source_packet_ref: examples/english-lowering/nibu-proof-001/interlingua.yaml
|
||||||
|
target_profile_ref: .weksa/cultural-ontology/starter-targets.cc
|
||||||
|
target_profile_id:
|
||||||
target_agent_id: nibu
|
target_agent_id: nibu
|
||||||
realization:
|
realization:
|
||||||
realization_id:
|
realization_id:
|
||||||
|
|
@ -48,6 +60,10 @@ realization:
|
||||||
locale:
|
locale:
|
||||||
script:
|
script:
|
||||||
register:
|
register:
|
||||||
|
cultural_ontology_refs:
|
||||||
|
- .weksa/cultural-ontology/starter-targets.cc
|
||||||
|
cultural_ontology_profile_ids:
|
||||||
|
-
|
||||||
outputs:
|
outputs:
|
||||||
visible_action:
|
visible_action:
|
||||||
spoken_text:
|
spoken_text:
|
||||||
|
|
|
||||||
|
|
@ -23,6 +23,9 @@ them.
|
||||||
- [utterance-lowering-packet.md](./utterance-lowering-packet.md): shared packet
|
- [utterance-lowering-packet.md](./utterance-lowering-packet.md): shared packet
|
||||||
contract for lowering interlingua and projected speaker context into one or
|
contract for lowering interlingua and projected speaker context into one or
|
||||||
more target-language utterance realizations.
|
more target-language utterance realizations.
|
||||||
|
- [target-language-cultural-ontology.md](./target-language-cultural-ontology.md):
|
||||||
|
contract for the address, politeness, register, idiom, metaphor, script, and
|
||||||
|
forbidden-flattening profiles consumed by target lowerers.
|
||||||
- [cultural-ontology-profile.md](./cultural-ontology-profile.md): nested
|
- [cultural-ontology-profile.md](./cultural-ontology-profile.md): nested
|
||||||
CultCache profile contract for cultural ontology, including typed salience
|
CultCache profile contract for cultural ontology, including typed salience
|
||||||
axes, linguistic affordance feature records, evidence, activation gates, and
|
axes, linguistic affordance feature records, evidence, activation gates, and
|
||||||
|
|
|
||||||
|
|
@ -24,17 +24,6 @@
|
||||||
"$ref": "#/$defs/NameSpan"
|
"$ref": "#/$defs/NameSpan"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"names_by": {
|
|
||||||
"description": "Who placed `names`: the annotator of the intake that wrote them. `None`\nwhen nothing records one (the cast seeds, `weksa corpus add`, the Dao De\nJing bed); it labels the spans, never the line or its speaker.",
|
|
||||||
"anyOf": [
|
|
||||||
{
|
|
||||||
"$ref": "#/$defs/Annotator"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "null"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"prior_turns": {
|
"prior_turns": {
|
||||||
"type": "array",
|
"type": "array",
|
||||||
"items": {
|
"items": {
|
||||||
|
|
@ -74,41 +63,6 @@
|
||||||
"names"
|
"names"
|
||||||
],
|
],
|
||||||
"$defs": {
|
"$defs": {
|
||||||
"Annotator": {
|
|
||||||
"description": "Who annotated a document. A model's generator is an attestation string, held\nto the same form as every provenance string.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Operator"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Model": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"family": {
|
|
||||||
"$ref": "#/$defs/ModelFamily"
|
|
||||||
},
|
|
||||||
"generator": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"family",
|
|
||||||
"generator"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Model"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"EntityId": {
|
"EntityId": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"pattern": "^entity:[a-z0-9]([a-z0-9-]{0,46}[a-z0-9])?$"
|
"pattern": "^entity:[a-z0-9]([a-z0-9-]{0,46}[a-z0-9])?$"
|
||||||
|
|
@ -216,7 +170,7 @@
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"description": "A public-domain work, with everything a reader needs to check the claim:\nthe edition's licence and the jurisdiction the public-domain status is\nasserted in, who translated it (the translation is the text's copyright,\nnot the original's) and when the work or translation first appeared. A work missing any\nis refused at admission.",
|
"description": "A public-domain work, with everything a reader needs to check the claim:\nthe edition's licence and the jurisdiction the public-domain status is\nasserted in, who translated it (the translation is the text's copyright,\nnot the original's) and when the edition appeared. A work missing any\nis refused at admission.",
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
"PublicDomain": {
|
"PublicDomain": {
|
||||||
|
|
@ -229,7 +183,6 @@
|
||||||
"type": "string"
|
"type": "string"
|
||||||
},
|
},
|
||||||
"edition_year": {
|
"edition_year": {
|
||||||
"description": "The year the work, or for a translation this translation, first\nappeared. It is not the year of the `edition_url` file's source\nprint: Project Gutenberg's text may be a later or modernised\nedition, and the claim rests on the date of first appearance.",
|
|
||||||
"type": "integer",
|
"type": "integer",
|
||||||
"format": "uint16",
|
"format": "uint16",
|
||||||
"maximum": 65535,
|
"maximum": 65535,
|
||||||
|
|
@ -265,7 +218,7 @@
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"description": "Labelled model output; never in the test split. The rule that it is\nnever judged by its own family binds model judges and the split; a\ndeterministic scorer refuses nothing, so the loop labels (the round\ncounts items of the answerer's family) and does not forbid.\n`family` is `Unrecorded` when nothing records one. `generator` is the\nstring the fixture or run wrote, kept verbatim.",
|
"description": "Labelled model output; never in the test split, never judged by its own family.\n`family` is `Unrecorded` when nothing records one. `generator` is the\nstring the fixture or run wrote, kept verbatim.",
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
"ModelWritten": {
|
"ModelWritten": {
|
||||||
|
|
|
||||||
|
|
@ -81,14 +81,9 @@
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"description": "The speaker's relation to one party of the scene, held by `weksa.scene`\ndocuments; see `LoweringInput::with_scene`.",
|
"description": "The relation of the speaker to the addressee in the scene; see\n`LoweringInput::with_scene`.",
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"const": "Scene"
|
"const": "Scene"
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "The speaker's stable axes, held by `weksa.speaker` documents.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "Speaker"
|
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -1,230 +0,0 @@
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"title": "Decision",
|
|
||||||
"description": "One closed edit-accept cycle: digests and numbers, no text. Key\n`run:<label>:cycle:<k>`. Only [`cycle`] builds one.\n\n```\nfn f(d: &weksa::run::Decision) -> u32 { d.k() }\n```\n\nEvery field is given, so the literal below fails only on privacy; its twin is\nthe same literal in the unit test `a_decision_literal_compiles_in_the_lib`.\n\n```compile_fail,E0451\nuse weksa::run::{Call, Decision, Verdict};\nfn f() -> Decision {\n Decision {\n k: 1,\n base_round: 1,\n base_rev: 1,\n base_inventory_digest: String::new(),\n edit: Call { prompt: String::new(), items: vec![] },\n edit_answer_digest: None,\n candidate_rev: None,\n candidate_inventory_digest: None,\n candidate_round: None,\n panel: vec![],\n verdict: Verdict::Accepted,\n }\n}\n```",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"base_inventory_digest": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"base_rev": {
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"base_round": {
|
|
||||||
"description": "The head's establishing round: the base of this cycle.",
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"candidate_inventory_digest": {
|
|
||||||
"type": [
|
|
||||||
"string",
|
|
||||||
"null"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"candidate_rev": {
|
|
||||||
"type": [
|
|
||||||
"integer",
|
|
||||||
"null"
|
|
||||||
],
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"candidate_round": {
|
|
||||||
"type": [
|
|
||||||
"integer",
|
|
||||||
"null"
|
|
||||||
],
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"edit": {
|
|
||||||
"description": "The edit prompt's digest and the Train keys it held.",
|
|
||||||
"$ref": "#/$defs/Call"
|
|
||||||
},
|
|
||||||
"edit_answer_digest": {
|
|
||||||
"description": "16 hex of the editor's answer; `None` when it was unreadable.",
|
|
||||||
"type": [
|
|
||||||
"string",
|
|
||||||
"null"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"k": {
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"panel": {
|
|
||||||
"description": "One row per panel language, in panel order.",
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/PanelReading"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"verdict": {
|
|
||||||
"$ref": "#/$defs/Verdict"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"k",
|
|
||||||
"base_round",
|
|
||||||
"base_rev",
|
|
||||||
"base_inventory_digest",
|
|
||||||
"edit",
|
|
||||||
"verdict",
|
|
||||||
"panel"
|
|
||||||
],
|
|
||||||
"$defs": {
|
|
||||||
"Call": {
|
|
||||||
"description": "One model call: the prompt's digest and the item keys it held.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"items": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"prompt": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"prompt",
|
|
||||||
"items"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Metric": {
|
|
||||||
"description": "Which metric an edit regressed.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Chrf",
|
|
||||||
"Meaning"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "The composite fell, or a reading was not a finite number.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "Composite"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"PanelReading": {
|
|
||||||
"description": "One panel language's row of a cycle: the head's Validation reading, the\ncandidate's (none when the edit never reached a round), the language's\nmeasured floor and the threshold derived from it. The threshold decides only\nin a one-language panel ([`decide`]); with two or more languages it is\nrecorded beside the floor and the accept is the sum rule of\n[`decide_panel`].",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"base": {
|
|
||||||
"$ref": "#/$defs/Reading"
|
|
||||||
},
|
|
||||||
"candidate": {
|
|
||||||
"anyOf": [
|
|
||||||
{
|
|
||||||
"$ref": "#/$defs/Reading"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "null"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"floor": {
|
|
||||||
"$ref": "#/$defs/Reading"
|
|
||||||
},
|
|
||||||
"language": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"threshold": {
|
|
||||||
"type": "number",
|
|
||||||
"format": "double"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"language",
|
|
||||||
"base",
|
|
||||||
"floor",
|
|
||||||
"threshold"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Reading": {
|
|
||||||
"description": "One validation reading: chrF 0 to 100, meaning 0 to 1, and their composite.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"chrf": {
|
|
||||||
"type": "number",
|
|
||||||
"format": "double"
|
|
||||||
},
|
|
||||||
"composite": {
|
|
||||||
"type": "number",
|
|
||||||
"format": "double"
|
|
||||||
},
|
|
||||||
"meaning": {
|
|
||||||
"type": "number",
|
|
||||||
"format": "double"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"chrf",
|
|
||||||
"meaning",
|
|
||||||
"composite"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Verdict": {
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Accepted"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Rejected": {
|
|
||||||
"$ref": "#/$defs/Why"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Rejected"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Why": {
|
|
||||||
"description": "Why an edit was rejected.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"NoGain"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "The answer was not an edit draft.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "EditParse"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "The draft asked for no change or too many, or the inventory door refused it.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "EditAdmit"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Regressed": {
|
|
||||||
"$ref": "#/$defs/Metric"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Regressed"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,369 +0,0 @@
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"title": "RoundRecord",
|
|
||||||
"description": "One scored round. It holds no text and no Test reading: per-item rows for\nTrain only; Validation as means and refusal counts by stage, and as\nper-item scores (numbers only, private to this module). Only `run::round`\nbuilds one.\n\n```\nfn f(r: &weksa::run::RoundRecord) -> u32 { r.n() }\n```\n\nEvery field is given, so the literal below fails only on privacy; its twin\nis the same literal in the unit test `a_round_record_literal_compiles_in_the_lib`.\n\n```compile_fail,E0451\nuse weksa::run::RoundRecord;\nfn f() -> RoundRecord {\n RoundRecord {\n n: 1,\n inventory_rev: 1,\n prompt_digest: String::new(),\n corpus_digest: String::new(),\n entity_digest: String::new(),\n model: String::new(),\n embedder: String::new(),\n chrf_version: String::new(),\n calls: vec![],\n train: vec![],\n inventory_digest: String::new(),\n cycle: None,\n admit_rules: vec![],\n parse_reasons: vec![],\n own_family: 0,\n languages: vec![],\n validation_scores: vec![],\n }\n}\n```",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"admit_rules": {
|
|
||||||
"description": "The rules admission refused items under, in rule order; every Train\nand Validation item refused at `Stage::Admit` is counted once per rule.",
|
|
||||||
"type": "array",
|
|
||||||
"default": [],
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/RuleCount"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"calls": {
|
|
||||||
"description": "The encode calls, then the lowering calls, each with its prompt digest\nand one item key.",
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/Call"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"chrf_version": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"corpus_digest": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"cycle": {
|
|
||||||
"description": "`Some(k)` for the candidate round of cycle `k`.",
|
|
||||||
"type": [
|
|
||||||
"integer",
|
|
||||||
"null"
|
|
||||||
],
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"embedder": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"entity_digest": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"inventory_digest": {
|
|
||||||
"description": "16 hex of the inventory file the round was scored under.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"inventory_rev": {
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"languages": {
|
|
||||||
"description": "One reading per panel language, in panel order. No pooled mean exists.",
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/LanguageRound"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"model": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"n": {
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"own_family": {
|
|
||||||
"description": "Offered (live Train and Validation) `ModelWritten` items whose writer\nfamily is the run's answerer family. The corpus rule binds\nmodel judges; the scorer is deterministic, so nothing is refused for\nthis: the round labels it.",
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"default": 0,
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"parse_reasons": {
|
|
||||||
"description": "Why items were refused at `Stage::Parse`, in code order, over Train and\nValidation: a [`DraftClass`] code (`rule`), or `unreadable` for an\nitem with no answer that read as text. Never a token of an answer.",
|
|
||||||
"type": "array",
|
|
||||||
"default": [],
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/RuleCount"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"prompt_digest": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"train": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/TrainRow"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"validation_scores": {
|
|
||||||
"description": "Each panel language's live Validation scores in corpus order, a refused\nitem as 0. Private with no getter: only `run.rs` (the floor) reads them,\nso a leak outside it is a compile error. A run from before has none.",
|
|
||||||
"type": "array",
|
|
||||||
"default": [],
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/LanguageScores"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"n",
|
|
||||||
"inventory_rev",
|
|
||||||
"prompt_digest",
|
|
||||||
"corpus_digest",
|
|
||||||
"entity_digest",
|
|
||||||
"model",
|
|
||||||
"embedder",
|
|
||||||
"chrf_version",
|
|
||||||
"calls",
|
|
||||||
"train",
|
|
||||||
"inventory_digest",
|
|
||||||
"languages"
|
|
||||||
],
|
|
||||||
"$defs": {
|
|
||||||
"Call": {
|
|
||||||
"description": "One model call: the prompt's digest and the item keys it held.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"items": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"prompt": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"prompt",
|
|
||||||
"items"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"ItemScore": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"chrf": {
|
|
||||||
"description": "chrF of the output against the reference.",
|
|
||||||
"type": "number",
|
|
||||||
"format": "double"
|
|
||||||
},
|
|
||||||
"meaning": {
|
|
||||||
"description": "Embedding cosine of the output and the reference; `None` when either\nis empty after trimming.",
|
|
||||||
"type": [
|
|
||||||
"number",
|
|
||||||
"null"
|
|
||||||
],
|
|
||||||
"format": "double"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"chrf"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"LanguageRound": {
|
|
||||||
"description": "One panel language's readings in a round: the means over that language's\nitems only (a refused item counts 0) and how many items lowered and scored.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"language": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"train_mean": {
|
|
||||||
"$ref": "#/$defs/SplitMean"
|
|
||||||
},
|
|
||||||
"train_scored": {
|
|
||||||
"description": "Train items that lowered and scored; `train_mean` is over all of them.",
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"validation_mean": {
|
|
||||||
"$ref": "#/$defs/SplitMean"
|
|
||||||
},
|
|
||||||
"validation_refused": {
|
|
||||||
"$ref": "#/$defs/RefusalCounts"
|
|
||||||
},
|
|
||||||
"validation_scored": {
|
|
||||||
"description": "Validation items that lowered and scored; `validation_mean` is over all\nof them and counts a refused item as 0, so refusing a weak item cannot\nraise it.",
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"language",
|
|
||||||
"train_mean",
|
|
||||||
"train_scored",
|
|
||||||
"validation_mean",
|
|
||||||
"validation_scored",
|
|
||||||
"validation_refused"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"LanguageScores": {
|
|
||||||
"description": "One panel language's Validation scores in a round, in corpus order: numbers\nonly, no item key and no text. Only `run.rs` reads them (the paired floor).",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"language": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"scores": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/ItemScore"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"language",
|
|
||||||
"scores"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Outcome": {
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Scored": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"chrf": {
|
|
||||||
"type": "number",
|
|
||||||
"format": "double"
|
|
||||||
},
|
|
||||||
"meaning": {
|
|
||||||
"type": [
|
|
||||||
"number",
|
|
||||||
"null"
|
|
||||||
],
|
|
||||||
"format": "double"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"chrf"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Scored"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Refused": {
|
|
||||||
"$ref": "#/$defs/Stage"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Refused"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"RefusalCounts": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"admit": {
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"lower": {
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"parse": {
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"parse",
|
|
||||||
"admit",
|
|
||||||
"lower"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"RuleCount": {
|
|
||||||
"description": "How many items one admission rule refused, over both splits: a rule code\n(`V5`, `UnknownEntity`) and a count, never an identifier or text.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"count": {
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"rule": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"rule",
|
|
||||||
"count"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"SplitMean": {
|
|
||||||
"description": "Mean scores of one derived split.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"chrf": {
|
|
||||||
"description": "`None` when the split has no items.",
|
|
||||||
"type": [
|
|
||||||
"number",
|
|
||||||
"null"
|
|
||||||
],
|
|
||||||
"format": "double"
|
|
||||||
},
|
|
||||||
"count": {
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"meaning": {
|
|
||||||
"description": "Mean over the items that have a meaning score; `None` when none do.",
|
|
||||||
"type": [
|
|
||||||
"number",
|
|
||||||
"null"
|
|
||||||
],
|
|
||||||
"format": "double"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"count"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Stage": {
|
|
||||||
"description": "Where a round refused: the first stage that did.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Parse",
|
|
||||||
"Admit"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "The lowering prompt could not be built, or its answer was not UTF-8, not\na `LoweringDraft`, or text that is blank, over `MAX_LOWERING_BYTES` or\nholds a control character.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "Lower"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"TrainRow": {
|
|
||||||
"description": "A Train item's row. Validation has no rows.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"item": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"outcome": {
|
|
||||||
"$ref": "#/$defs/Outcome"
|
|
||||||
},
|
|
||||||
"packet": {
|
|
||||||
"type": [
|
|
||||||
"string",
|
|
||||||
"null"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"item",
|
|
||||||
"outcome"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,136 +0,0 @@
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"title": "RunRecord",
|
|
||||||
"description": "What a run pinned at open. Key `run:<label>`.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"answerer_family": {
|
|
||||||
"description": "The family of the model answering the run's encode and lowering\nprompts, set once at open; `Unrecorded` when the opener did not say\nand for a run opened before this was kept.",
|
|
||||||
"$ref": "#/$defs/ModelFamily",
|
|
||||||
"default": "Unrecorded"
|
|
||||||
},
|
|
||||||
"chrf_version": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"corpus": {
|
|
||||||
"description": "Canonical absolute path of the corpus store.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"corpus_digest": {
|
|
||||||
"description": "16 hex of the corpus file's bytes.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"embedder": {
|
|
||||||
"description": "The embedder's model.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"entities": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"entity_digest": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"inventory": {
|
|
||||||
"description": "Canonical absolute path of the inventory store the run was opened over:\nprovenance only. `create` copies that file to `<root>/inventory.cc` and\nthe run reads nothing at this path afterwards.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"inventory_digest": {
|
|
||||||
"description": "16 hex of the bytes `create` copied to `<root>/inventory.cc`.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"label": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"lowerer_reach": {
|
|
||||||
"description": "How the run's lowering prompts are answered, set once at open.",
|
|
||||||
"$ref": "#/$defs/LowererReach"
|
|
||||||
},
|
|
||||||
"model": {
|
|
||||||
"description": "The answering model.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"ontology": {
|
|
||||||
"description": "Canonical absolute path of the ontology store: read-only for the run,\npinned by `ontology_digest`.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"ontology_digest": {
|
|
||||||
"description": "16 hex of the ontology file's bytes.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"panel": {
|
|
||||||
"description": "The languages the run offers, reads and decides on: BCP 47-form tags,\nsorted, unique, at least one, set once at open. An item of any other\nlanguage is never offered, minted from, probed or scored.",
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"prompt_digest": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"speaker_state": {
|
|
||||||
"description": "Canonical absolute path of the speaker-state store (scenes): read-only\nfor the run, pinned by `speaker_state_digest`.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"speaker_state_digest": {
|
|
||||||
"description": "16 hex of the speaker-state file's bytes.",
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"label",
|
|
||||||
"corpus",
|
|
||||||
"inventory",
|
|
||||||
"entities",
|
|
||||||
"corpus_digest",
|
|
||||||
"entity_digest",
|
|
||||||
"prompt_digest",
|
|
||||||
"model",
|
|
||||||
"embedder",
|
|
||||||
"chrf_version",
|
|
||||||
"inventory_digest",
|
|
||||||
"lowerer_reach",
|
|
||||||
"panel",
|
|
||||||
"ontology",
|
|
||||||
"ontology_digest",
|
|
||||||
"speaker_state",
|
|
||||||
"speaker_state_digest"
|
|
||||||
],
|
|
||||||
"$defs": {
|
|
||||||
"LowererReach": {
|
|
||||||
"description": "How a run's lowering prompts reach the model that answers them (rulings\n`lowerer-call-reach-toolless-relay`, `dao-round-seed-inventory`). The\nrecord says which, so a round lowered by a briefed subagent cannot pass as\na tool-less one.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"description": "A subagent briefed to read one prompt file: acceptable for smoke rounds\nonly.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "BriefedSubagent"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "The prompt passed inline to an agent with no file tools.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "ToolLessRelay"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"ModelFamily": {
|
|
||||||
"description": "The family of the model that wrote a line.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Anthropic",
|
|
||||||
"OpenAi",
|
|
||||||
"Google",
|
|
||||||
"Alibaba",
|
|
||||||
"Meta",
|
|
||||||
"Mistral"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "Nothing records which family wrote the line.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "Unrecorded"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,277 +0,0 @@
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"title": "OntologyEntry",
|
|
||||||
"description": "One choice of one language. Key `onto:<language>:<n>`.\n\nThe store judges the form of an entry's text (bounds, normalization, no\ncontrol or format character) and never its meaning: no door judges text\n(ruling `leak-owned-by-flow-and-scorer-3`). No Train-only ontology mint\nexists yet, so a seed or a drafted entry is checked against held-out text by\nnothing today; follow-up `leak-diagnostic-in-round` owns that diagnostic.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"annotator": {
|
|
||||||
"$ref": "#/$defs/Annotator"
|
|
||||||
},
|
|
||||||
"choice": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"examples": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"inventory_rev": {
|
|
||||||
"description": "The revision, of the inventory the entry was admitted through, at which the\nstore read its conditions: provenance only; a run reads the conditions'\nfeatures at its own head.",
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"language": {
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"layer": {
|
|
||||||
"$ref": "#/$defs/Layer"
|
|
||||||
},
|
|
||||||
"n": {
|
|
||||||
"description": "The entry's number within its language, from 1.",
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"when": {
|
|
||||||
"description": "Every condition must hold; none means the entry always applies.",
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/Condition"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"language",
|
|
||||||
"n",
|
|
||||||
"inventory_rev",
|
|
||||||
"layer",
|
|
||||||
"when",
|
|
||||||
"choice",
|
|
||||||
"examples",
|
|
||||||
"annotator"
|
|
||||||
],
|
|
||||||
"$defs": {
|
|
||||||
"Annotator": {
|
|
||||||
"description": "Who annotated a document. A model's generator is an attestation string, held\nto the same form as every provenance string.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Operator"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Model": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"family": {
|
|
||||||
"$ref": "#/$defs/ModelFamily"
|
|
||||||
},
|
|
||||||
"generator": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"family",
|
|
||||||
"generator"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Model"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Condition": {
|
|
||||||
"description": "One feature of one slot of the view, tested. `Stance`, `Discourse` and\n`Scene` reach a lowering view; `Speaker` does not yet.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"feature": {
|
|
||||||
"$ref": "#/$defs/InvId"
|
|
||||||
},
|
|
||||||
"slot": {
|
|
||||||
"$ref": "#/$defs/Slot"
|
|
||||||
},
|
|
||||||
"test": {
|
|
||||||
"$ref": "#/$defs/Test"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"slot",
|
|
||||||
"feature",
|
|
||||||
"test"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"FeatValue": {
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Grade": {
|
|
||||||
"$ref": "#/$defs/Grade"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Grade"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "A label the feature's inventory entry declares.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Option": {
|
|
||||||
"$ref": "#/$defs/OptionLabel"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Option"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Flag": {
|
|
||||||
"type": "boolean"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Flag"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Grade": {
|
|
||||||
"type": "integer",
|
|
||||||
"maximum": 1000,
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"InvId": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^(concept|frame|role|relation|speech_act|feature|referent_kind|extension):[1-9][0-9]*$"
|
|
||||||
},
|
|
||||||
"Layer": {
|
|
||||||
"description": "Which part of the language an entry shapes.",
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Lexicon",
|
|
||||||
"Grammar",
|
|
||||||
"Discourse",
|
|
||||||
"Phonology",
|
|
||||||
"Orthography"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"ModelFamily": {
|
|
||||||
"description": "The family of the model that wrote a line.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Anthropic",
|
|
||||||
"OpenAi",
|
|
||||||
"Google",
|
|
||||||
"Alibaba",
|
|
||||||
"Meta",
|
|
||||||
"Mistral"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "Nothing records which family wrote the line.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "Unrecorded"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"OptionLabel": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[a-z][a-z0-9_]{0,23}$"
|
|
||||||
},
|
|
||||||
"Slot": {
|
|
||||||
"description": "Where a feature may sit in a packet.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Discourse",
|
|
||||||
"Referent",
|
|
||||||
"Predication",
|
|
||||||
"Relation",
|
|
||||||
"Stance"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "The speaker's relation to one party of the scene, held by `weksa.scene`\ndocuments; see `LoweringInput::with_scene`.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "Scene"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "The speaker's stable axes, held by `weksa.speaker` documents.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "Speaker"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Test": {
|
|
||||||
"description": "What a feature's value must be for a condition to hold.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"AtLeast": {
|
|
||||||
"$ref": "#/$defs/Grade"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"AtLeast"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"AtMost": {
|
|
||||||
"$ref": "#/$defs/Grade"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"AtMost"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Is": {
|
|
||||||
"$ref": "#/$defs/FeatValue"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Is"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "Holds when the feature is absent or a grade at most `g`: a bound that\ncosts nothing where the feature is unrated. `AtMost` still fails on an\nabsent feature.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"AtMostOrUnset": {
|
|
||||||
"$ref": "#/$defs/Grade"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"AtMostOrUnset"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,225 +0,0 @@
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"title": "Scene",
|
|
||||||
"description": "A scene: its participants and the relations between them. Key\n`scene:<group>@<n>`; a revision appends and never edits.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"annotator": {
|
|
||||||
"$ref": "#/$defs/Annotator"
|
|
||||||
},
|
|
||||||
"group": {
|
|
||||||
"description": "The corpus split group of the scene, a slug.",
|
|
||||||
"type": "string"
|
|
||||||
},
|
|
||||||
"inventory_rev": {
|
|
||||||
"description": "The revision, of the inventory the scene was admitted through, at which\n`put_scene` read its features: provenance only; a run never reads it.",
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"n": {
|
|
||||||
"description": "The revision, from 1.",
|
|
||||||
"type": "integer",
|
|
||||||
"format": "uint32",
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"participants": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/EntityId"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"relations": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/PairRelation"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"group",
|
|
||||||
"n",
|
|
||||||
"inventory_rev",
|
|
||||||
"participants",
|
|
||||||
"relations",
|
|
||||||
"annotator"
|
|
||||||
],
|
|
||||||
"$defs": {
|
|
||||||
"Annotator": {
|
|
||||||
"description": "Who annotated a document. A model's generator is an attestation string, held\nto the same form as every provenance string.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Operator"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Model": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"family": {
|
|
||||||
"$ref": "#/$defs/ModelFamily"
|
|
||||||
},
|
|
||||||
"generator": {
|
|
||||||
"type": "string"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"family",
|
|
||||||
"generator"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Model"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"EntityId": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^entity:[a-z0-9]([a-z0-9-]{0,46}[a-z0-9])?$"
|
|
||||||
},
|
|
||||||
"FeatValue": {
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Grade": {
|
|
||||||
"$ref": "#/$defs/Grade"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Grade"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "A label the feature's inventory entry declares.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Option": {
|
|
||||||
"$ref": "#/$defs/OptionLabel"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Option"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Flag": {
|
|
||||||
"type": "boolean"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Flag"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"FeatureValue": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"feature": {
|
|
||||||
"description": "Feature id.",
|
|
||||||
"$ref": "#/$defs/InvId"
|
|
||||||
},
|
|
||||||
"value": {
|
|
||||||
"$ref": "#/$defs/FeatValue"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"feature",
|
|
||||||
"value"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Grade": {
|
|
||||||
"type": "integer",
|
|
||||||
"maximum": 1000,
|
|
||||||
"minimum": 0
|
|
||||||
},
|
|
||||||
"InvId": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^(concept|frame|role|relation|speech_act|feature|referent_kind|extension):[1-9][0-9]*$"
|
|
||||||
},
|
|
||||||
"ModelFamily": {
|
|
||||||
"description": "The family of the model that wrote a line.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Anthropic",
|
|
||||||
"OpenAi",
|
|
||||||
"Google",
|
|
||||||
"Alibaba",
|
|
||||||
"Meta",
|
|
||||||
"Mistral"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"description": "Nothing records which family wrote the line.",
|
|
||||||
"type": "string",
|
|
||||||
"const": "Unrecorded"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"OptionLabel": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[a-z][a-z0-9_]{0,23}$"
|
|
||||||
},
|
|
||||||
"PairRelation": {
|
|
||||||
"description": "The speaker's relation to one party: feature values of slot `Scene`.",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"features": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"$ref": "#/$defs/FeatureValue"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"from": {
|
|
||||||
"$ref": "#/$defs/EntityId"
|
|
||||||
},
|
|
||||||
"to": {
|
|
||||||
"$ref": "#/$defs/Party"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"from",
|
|
||||||
"to",
|
|
||||||
"features"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"Party": {
|
|
||||||
"description": "The other party of a relation. `Unnamed` is an addressee the scene does not\nname, such as the player or a crowd.",
|
|
||||||
"oneOf": [
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"enum": [
|
|
||||||
"Unnamed"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"Named": {
|
|
||||||
"$ref": "#/$defs/EntityId"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"Named"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue