GameCult/Weksa
Watch
9
0
Fork
You've already forked Weksa
 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

Checkpoint gate: Test readings need a Checkpoint nobody outside tests can declare; fixed instructions; context lows #27

Merged
weksa merged 6 commits from weksa-lowering/checkpoint-gate into main 2026-11-10 03:05:14 +00:00
Conversation 0 Commits 6 Files changed 7 +549 -175
7 changed files with 549 additions and 175 deletions
Download patch file Download diff file Expand all files Collapse all files

171
crates/weksa/src/context.rs
View file

@ -2,15 +2,24 @@
//! model call only as a [`Prompt`], and a `Prompt` is made only by minting from
//! [`TrainItems`] or by encoding one item an [`EncoderOffer`] holds: Train-only
//! minting and per-item encoding (leak-owned-by-flow-and-scorer-3,
//! loop-context-builder-2). Instruction wording is not written here; a caller
//! passes `&'static str`, so the templates are fixed at compile time and hence
//! within a run.
//! loop-context-builder-2). Instruction text is [`MINT_INSTRUCTIONS`] and
//! [`ENCODE_INSTRUCTIONS`], consts in this module, and nothing else: `mint` and
//! `encode` take none, so no run-time text can become instructions.
use std::marker::PhantomData;
use crate::corpus::CorpusItem;
use crate::packet::EntityId;
use crate::split::{Checkpoint, Split, SplitScheme};
use crate::store::{CorpusStore, normalized};
/// The instructions a mint prompt starts with. golf-loop-en edits the wording
/// here.
pub const MINT_INSTRUCTIONS: &str = "Mint glosses from the lines below.";
/// The instructions an encode prompt starts with.
pub const ENCODE_INSTRUCTIONS: &str = "Encode the line below.";
/// The live Train items of a corpus, in key order. The only input minting
/// reads; it cannot be built from a slice, and no accessor returns text or an
/// item.
@ -62,13 +71,14 @@ impl TrainItems {
self.items.is_empty()
}
/// Whether a live Train item of `group` holds `text`, compared by the
/// store's own text key.
fn holds(&self, group: &str, text: &str) -> bool {
/// The text of the live Train item of `group` whose store text key equals
/// `text`'s (store S6 makes it unique).
fn held(&self, group: &str, text: &str) -> Option<&str> {
let key = normalized(text);
self.items
.iter()
.any(|item| item.group == group && normalized(&item.text) == key)
.find(|item| item.group == group && normalized(&item.text) == key)
.map(|item| item.text.as_str())
}
pub fn keys(&self) -> impl Iterator<Item = &str> {
@ -84,30 +94,65 @@ pub enum OfferRefusal {
NoSplit,
/// A Test item is offered only under a [`Checkpoint`].
TestBeforeCheckpoint,
/// An item is offered at most once under one [`Checkpoint`].
OfferedAtCheckpoint,
}
/// One item the encoder may be shown, with the entities a packet for it may
/// cite. Built per item, so a call sees one item and nothing of its group.
pub struct EncoderOffer {
///
/// An offer made under a [`Checkpoint`] borrows it, so it is dead before
/// `TestReadings::at_checkpoint` can spend the checkpoint, and a Test offer is
/// never encoded after (or beside) the reading. A round offer borrows nothing.
///
/// ```
/// use weksa::context::{EncoderOffer, TrainItems};
/// use weksa::split::Checkpoint;
/// use weksa::{CorpusItem, score::TestReadings};
/// fn f(mut c: Checkpoint, i: &CorpusItem, t: &TrainItems) {
/// if let Ok(o) = EncoderOffer::for_checkpoint(i, &mut c) {
/// let _ = weksa::context::Prompt::encode(&o, t);
/// }
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
/// }
/// ```
///
/// ```compile_fail
/// use weksa::context::{EncoderOffer, TrainItems};
/// use weksa::split::Checkpoint;
/// use weksa::{CorpusItem, score::TestReadings};
/// fn f(mut c: Checkpoint, i: &CorpusItem, t: &TrainItems) {
/// let o = EncoderOffer::for_checkpoint(i, &mut c).unwrap();
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
/// let _ = weksa::context::Prompt::encode(&o, t);
/// }
/// ```
pub struct EncoderOffer<'cp> {
item: CorpusItem,
entities: Vec<EntityId>,
_checkpoint: PhantomData<&'cp mut Checkpoint>,
}
impl EncoderOffer {
impl<'cp> EncoderOffer<'cp> {
/// A round offers Train and Validation items.
pub fn for_round(item: &CorpusItem) -> Result<Self, OfferRefusal> {
pub fn for_round(item: &CorpusItem) -> Result<EncoderOffer<'static>, OfferRefusal> {
match Self::gate(item)? {
Split::Test => Err(OfferRefusal::TestBeforeCheckpoint),
_ => Ok(Self::of(item)),
}
}
/// A convergence checkpoint offers any live item, Test included.
/// A convergence checkpoint offers any live item, Test included, each item
/// at most once under one `Checkpoint`. The gate runs first, so a refused
/// item records no key.
pub fn for_checkpoint(
item: &CorpusItem,
_checkpoint: &Checkpoint,
checkpoint: &'cp mut Checkpoint,
) -> Result<Self, OfferRefusal> {
Self::gate(item)?;
if !checkpoint.first_offer(&item.id) {
return Err(OfferRefusal::OfferedAtCheckpoint);
}
Ok(Self::of(item))
}
@ -118,13 +163,14 @@ impl EncoderOffer {
SplitScheme::V1.live(item).ok_or(OfferRefusal::NoSplit)
}
fn of(item: &CorpusItem) -> Self {
fn of<'a>(item: &CorpusItem) -> EncoderOffer<'a> {
let mut entities: Vec<EntityId> = item.cited_entities().cloned().collect();
entities.sort();
entities.dedup();
Self {
EncoderOffer {
item: item.clone(),
entities,
_checkpoint: PhantomData,
}
}
@ -145,17 +191,17 @@ impl EncoderOffer {
/// let dir = tempfile::tempdir().unwrap();
/// let store = weksa::CorpusStore::open(&dir.path().join("c.cc")).unwrap();
/// let t = weksa::context::TrainItems::from_corpus(&store).unwrap();
/// let p = weksa::context::Prompt::mint("Mint.", &t);
/// assert_eq!(p.text(), "Mint.");
/// let p = weksa::context::Prompt::mint(&t);
/// assert_eq!(p.text(), weksa::context::MINT_INSTRUCTIONS);
/// ```
///
/// Instructions are compile-time text:
/// Instructions are not a parameter:
///
/// ```compile_fail
/// let dir = tempfile::tempdir().unwrap();
/// let store = weksa::CorpusStore::open(&dir.path().join("c.cc")).unwrap();
/// let t = weksa::context::TrainItems::from_corpus(&store).unwrap();
/// let p = weksa::context::Prompt::mint(&String::from("x"), &t);
/// let p = weksa::context::Prompt::mint("x", &t);
/// ```
///
/// ```compile_fail
@ -163,7 +209,7 @@ impl EncoderOffer {
/// let dir = tempfile::tempdir().unwrap();
/// let store = weksa::CorpusStore::open(&dir.path().join("c.cc")).unwrap();
/// let t = weksa::context::TrainItems::from_corpus(&store).unwrap();
/// let p = weksa::context::Prompt::mint("x", &t);
/// let p = weksa::context::Prompt::mint(&t);
/// s(&p);
/// ```
pub struct Prompt {
@ -173,20 +219,19 @@ pub struct Prompt {
/// What a prompt prints of an item: its line, and its prior turns only as the
/// one owner allows. A prior turn is a copy of text, not a reference, so it can
/// outlive a withdrawal or quote a held-out line; its text is printed only when
/// a live Train item of the item's group holds it, and the speaker alone is
/// printed otherwise. Mint prints no prior turns (`prior: None`).
/// outlive a withdrawal or quote a held-out line; what is printed is the text of
/// the live Train item of the item's group that holds it (never the turn's own
/// bytes), and the speaker alone otherwise. Mint prints no prior turns (`prior: None`).
fn block(item: &CorpusItem, prior: Option<&TrainItems>) -> String {
let mut lines: Vec<String> = prior
.into_iter()
.flat_map(|train| {
item.prior_turns.iter().map(move |t| {
if train.holds(&item.group, &t.text) {
format!("{}: {}", t.speaker, t.text)
} else {
format!("{}: {ELIDED}", t.speaker)
}
})
item.prior_turns
.iter()
.map(move |t| match train.held(&item.group, &t.text) {
Some(text) => format!("{}: {text}", t.speaker),
None => format!("{}: {ELIDED}", t.speaker),
})
})
.collect();
lines.push(format!("[{}] {}: {}", item.id, item.speaker, item.text));
@ -197,9 +242,9 @@ fn block(item: &CorpusItem, prior: Option<&TrainItems>) -> String {
const ELIDED: &str = "[elided]";
impl Prompt {
/// Instructions, then every Train item in key order.
pub fn mint(instructions: &'static str, train: &TrainItems) -> Prompt {
let mut parts = vec![instructions.to_string()];
/// [`MINT_INSTRUCTIONS`], then every Train item in key order.
pub fn mint(train: &TrainItems) -> Prompt {
let mut parts = vec![MINT_INSTRUCTIONS.to_string()];
parts.extend(train.items.iter().map(|item| block(item, None)));
Prompt {
text: parts.join("\n\n"),
@ -207,13 +252,13 @@ impl Prompt {
}
}
/// Instructions, the one offered item with its prior turns resolved
/// [`ENCODE_INSTRUCTIONS`], the one offered item with its prior turns resolved
/// against `train`, and the entities a packet for it may cite.
pub fn encode(instructions: &'static str, offer: &EncoderOffer, train: &TrainItems) -> Prompt {
pub fn encode(offer: &EncoderOffer, train: &TrainItems) -> Prompt {
let ids: Vec<&str> = offer.entities.iter().map(EntityId::as_str).collect();
Prompt {
text: format!(
"{instructions}\n\n{}\n\nentities: {}",
"{ENCODE_INSTRUCTIONS}\n\n{}\n\nentities: {}",
block(&offer.item, Some(train)),
ids.join(", ")
),
@ -230,3 +275,59 @@ impl Prompt {
&self.items
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::split::fixture::{group_in, item};
#[test]
fn checkpoint_offer_gate() {
// A fresh checkpoint per call: this test is the gate, not the once rule.
let check =
|i: &CorpusItem| EncoderOffer::for_checkpoint(i, &mut Checkpoint::for_test(1)).err();
let train = item("item:a:1", "t", &group_in(Split::Train), false);
let validation = item("item:b:1", "v", &group_in(Split::Validation), false);
let test = item("item:c:1", "x", &group_in(Split::Test), false);
let no_split = item("item:c:2", "m", &group_in(Split::Test), true);
let mut withdrawn = item("item:a:2", "w", &group_in(Split::Train), false);
withdrawn.withdrawn = true;
assert_eq!(check(&train), None);
assert_eq!(check(&validation), None);
assert_eq!(check(&test), None);
assert_eq!(check(&no_split), Some(OfferRefusal::NoSplit));
assert_eq!(check(&withdrawn), Some(OfferRefusal::Withdrawn));
// Withdrawn wins over NoSplit.
let mut both = no_split.clone();
both.withdrawn = true;
assert_eq!(check(&both), Some(OfferRefusal::Withdrawn));
}
#[test]
fn checkpoint_offers_each_item_once() {
let mut cp = Checkpoint::for_test(1);
let offer = |cp: &mut Checkpoint, i: &CorpusItem| EncoderOffer::for_checkpoint(i, cp).err();
let a = item("item:c:1", "a", &group_in(Split::Test), false);
let b = item("item:c:3", "b", &group_in(Split::Test), false);
let v = item("item:b:1", "v", &group_in(Split::Validation), false);
assert_eq!(offer(&mut cp, &a), None);
assert_eq!(offer(&mut cp, &a), Some(OfferRefusal::OfferedAtCheckpoint));
assert_eq!(offer(&mut cp, &b), None);
assert_eq!(offer(&mut cp, &v), None);
assert_eq!(offer(&mut cp, &v), Some(OfferRefusal::OfferedAtCheckpoint));
// A refused item records no key: its refusal stays the gate's.
let mut gone = item("item:c:2", "w", &group_in(Split::Test), false);
gone.withdrawn = true;
assert_eq!(offer(&mut cp, &gone), Some(OfferRefusal::Withdrawn));
assert_eq!(offer(&mut cp, &gone), Some(OfferRefusal::Withdrawn));
let mut live = gone.clone();
live.withdrawn = false;
assert_eq!(offer(&mut cp, &live), None, "the refusal recorded no key");
// The checkpoint is spent by its reading; a new one offers afresh.
let _ = crate::score::TestReadings::at_checkpoint(cp, &[], &[], &[]);
let mut fresh = Checkpoint::for_test(2);
assert_eq!(offer(&mut fresh, &a), None);
}
}

137
crates/weksa/src/score/mod.rs
View file

@ -158,37 +158,27 @@ pub fn aggregate(scored: &[(&CorpusItem, ItemScore)]) -> Aggregate {
/// The Test split's readings, scalars only: no item key, no minted string, no
/// pair. The checkpoint decision (convergence-test) needs numbers; a key beside
/// a minted string is the handle an edit step would act on. The only producer
/// is `at_checkpoint`, which consumes a `Checkpoint`.
/// is `at_checkpoint`, which consumes a `Checkpoint`; only the lib can make one.
///
/// ```
/// use weksa::score::TestReadings;
/// use weksa::split::Checkpoint;
/// fn f(t: TestReadings) -> usize { t.count() }
/// ```
///
/// ```compile_fail
/// use weksa::score::TestReadings;
/// fn f(t: TestReadings) { let _u = t.clone(); }
/// ```
///
/// ```compile_fail
/// use weksa::score::TestReadings;
/// fn f(t: TestReadings) { let _ = format!("{t:?}"); }
/// ```
///
/// ```compile_fail
/// use weksa::score::TestReadings;
/// fn s<T: serde::Serialize>(_: &T) {}
/// let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &[], &[]);
/// assert_eq!(t.count(), 0);
/// s(&0u8);
/// ```
///
/// ```compile_fail
/// use weksa::score::TestReadings;
/// use weksa::split::Checkpoint;
/// let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &[], &[]);
/// let _u = t.clone();
/// ```
///
/// ```compile_fail
/// use weksa::score::TestReadings;
/// use weksa::split::Checkpoint;
/// let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &[], &[]);
/// let _ = format!("{t:?}");
/// ```
///
/// ```compile_fail
/// use weksa::score::TestReadings;
/// use weksa::split::Checkpoint;
/// fn s<T: serde::Serialize>(_: &T) {}
/// let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &[], &[]);
/// s(&t);
/// fn f(t: TestReadings) { s(&t); }
/// ```
pub struct TestReadings {
mean: SplitMean,
@ -240,3 +230,96 @@ impl TestReadings {
&self.leak.coverage_by_n
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::split::fixture::{group_in, item};
const HELD_OUT: &str = "extraordinary lighthouse keepers whispered unforgettable stories throughout midnight thunderstorms before surrendering peacefully";
const UNRELATED: &str = "harbor ledgers record ordinary tides";
fn score(chrf: f64, meaning: Option<f64>) -> ItemScore {
ItemScore { chrf, meaning }
}
fn in_split(key: &str, text: &str, split: Split, model: bool) -> CorpusItem {
item(key, text, &group_in(split), model)
}
#[test]
fn test_readings_at_checkpoint() {
let train_item = in_split("item:t:1", UNRELATED, Split::Train, false);
let test = in_split("item:c:1", HELD_OUT, Split::Test, false);
let test_model = in_split("item:c:2", "model words here", Split::Test, true);
let mut gone = in_split("item:c:3", "withdrawn words", Split::Test, false);
gone.withdrawn = true;
let val = in_split(
"item:v:1",
"gardening is quiet work",
Split::Validation,
false,
);
let items = [
train_item.clone(),
test.clone(),
test_model.clone(),
gone.clone(),
val.clone(),
];
let scored = [
(&train_item, score(10.0, Some(0.5))),
(&val, score(40.0, Some(0.8))),
(&test, score(60.0, Some(0.6))),
(&test_model, score(100.0, Some(1.0))),
(&gone, score(90.0, Some(0.9))),
];
let t = TestReadings::at_checkpoint(Checkpoint::for_test(2), &scored, &[HELD_OUT], &items);
assert_eq!(
(t.count(), t.chrf(), t.meaning()),
(1, Some(60.0), Some(0.6))
);
assert_eq!(
t.items(),
1,
"model-written and withdrawn Test items are excluded"
);
assert!(t.max_chrf().unwrap() > 99.0);
assert_eq!(t.coverage_by_n()[2].share, Some(1.0));
// A minted string that is not the Test text reads low.
let t = TestReadings::at_checkpoint(Checkpoint::for_test(2), &scored, &["zzz"], &items);
assert!(t.max_chrf().unwrap() < 20.0);
assert_eq!(t.coverage_by_n()[2].share, Some(0.0));
let empty = TestReadings::at_checkpoint(Checkpoint::for_test(2), &[], &[HELD_OUT], &[]);
assert_eq!(
(empty.count(), empty.chrf(), empty.meaning()),
(0, None, None)
);
assert_eq!((empty.items(), empty.max_chrf()), (0, None));
}
#[test]
fn validation_and_test_leak_agree_in_method() {
// The same text and minted strings, once in Validation and once in
// Test, read the same numbers.
let words: Vec<&str> = HELD_OUT.split(' ').collect();
let chunks: Vec<String> = words.chunks(3).map(|c| c.join(" ")).collect();
let minted: Vec<&str> = chunks.iter().map(String::as_str).collect();
let train_item = in_split("item:t:1", UNRELATED, Split::Train, false);
let v = leak_diagnostic(
&minted,
&[
in_split("item:v:1", HELD_OUT, Split::Validation, false),
train_item.clone(),
],
);
let test = in_split("item:c:1", HELD_OUT, Split::Test, false);
let t =
TestReadings::at_checkpoint(Checkpoint::for_test(1), &[], &minted, &[test, train_item]);
assert_eq!(t.items(), v.items);
assert_eq!(t.max_chrf(), v.max_chrf.as_ref().map(|p| p.chrf));
assert_eq!(t.coverage_by_n(), v.coverage_by_n.as_slice());
}
}

126
crates/weksa/src/split.rs
View file

@ -1,6 +1,13 @@
//! The train/validation/test split is derived from an item's group and nothing
//! else. No stored assignment exists, so no writer can reassign one. An item is
//! read in its live split; Test is read only under a Checkpoint.
//! read in its live split. The round path (round records, prompts, `aggregate`,
//! `leak_diagnostic`) carries no Test reading; a typed `TestReadings` and a Test
//! `EncoderOffer` need a Checkpoint, and only the lib can make one
//! (convergence-freeze adds the door). Operator tools (`weksa corpus dump`,
//! `weksa score`, `CorpusStore`) read corpus text of any split; that is not
//! gated here.
use std::collections::BTreeSet;
use crate::corpus::{CorpusItem, Source};
@ -63,44 +70,127 @@ impl SplitScheme {
}
}
/// Permission to read Test once. The loop declares one at a convergence
/// checkpoint (convergence-test); `TestReadings::at_checkpoint` consumes it, so
/// one declaration buys one Test reading, and `EncoderOffer::for_checkpoint`
/// borrows it first.
/// Permission to read Test once, as typed `TestReadings` or as Test
/// `EncoderOffer`s. Only the weksa lib can make one: there is no public
/// constructor, so no code outside it can obtain those. This is not a seal on
/// corpus text: operator tools and `CorpusStore` can read any item, and the
/// guarded invariant is that the round path holds no Test reading. The one door
/// is added by
/// convergence-freeze, which builds a `Checkpoint` from the convergence window
/// over the run's persisted round records (convergence-test).
/// `TestReadings::at_checkpoint` consumes it, so one `Checkpoint` buys one Test
/// reading. One `Checkpoint` also offers each item at most once
/// (`EncoderOffer::for_checkpoint`), and an offer borrows its Checkpoint, so it
/// cannot outlive the reading that spends it.
///
/// ```
/// fn f(c: &weksa::split::Checkpoint) -> u32 { c.round() }
/// ```
///
/// No public constructor:
///
/// ```compile_fail
/// let _ = weksa::split::Checkpoint::declare(1);
/// ```
///
/// The test constructor is not exported (this doctest names it today; the
/// line-based tripwire in `tests/checkpoint_door.rs` catches only common
/// single-line forms of a new public constructor):
///
/// ```compile_fail
/// let _ = weksa::split::Checkpoint::for_test(1);
/// ```
///
/// ```compile_fail
/// let _ = weksa::split::Checkpoint { round: 1, offered: Default::default() };
/// ```
///
/// One reading per checkpoint:
///
/// ```
/// use weksa::score::TestReadings;
/// use weksa::split::Checkpoint;
/// let c = Checkpoint::declare(3);
/// assert_eq!(c.round(), 3);
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
/// fn f(c: Checkpoint) { let _ = TestReadings::at_checkpoint(c, &[], &[], &[]); }
/// ```
///
/// One declaration, one reading:
///
/// ```compile_fail
/// use weksa::score::TestReadings;
/// use weksa::split::Checkpoint;
/// let c = Checkpoint::declare(3);
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
/// fn f(c: Checkpoint) {
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
/// let _ = TestReadings::at_checkpoint(c, &[], &[], &[]);
/// }
/// ```
///
/// ```compile_fail
/// use weksa::split::Checkpoint;
/// let c = Checkpoint::declare(3);
/// let _again = c.clone();
/// fn f(c: weksa::split::Checkpoint) { let _ = c.clone(); }
/// ```
pub struct Checkpoint {
round: u32,
offered: BTreeSet<String>,
}
impl Checkpoint {
pub fn declare(round: u32) -> Self {
Self { round }
#[cfg(test)]
pub(crate) fn for_test(round: u32) -> Self {
Self {
round,
offered: BTreeSet::new(),
}
}
/// Records `key` as offered; false when it already was.
pub(crate) fn first_offer(&mut self, key: &str) -> bool {
self.offered.insert(key.to_string())
}
pub fn round(&self) -> u32 {
self.round
}
}
/// Item builders for the unit tests that live in `src`: the twins of
/// `tests/common`'s `group_in_bucket`, `item_in`, `cast` and `model`.
#[cfg(test)]
pub(crate) mod fixture {
use super::{Split, SplitScheme};
use crate::corpus::{CorpusItem, ModelFamily, Source};
use crate::packet::EntityId;
/// The first group `g<i>` whose V1 bucket is in `split`.
pub(crate) fn group_in(split: Split) -> String {
(0..100_000)
.map(|i| format!("g{i}"))
.find(|g| SplitScheme::V1.of_bucket(SplitScheme::V1.bucket(g)) == split)
.unwrap()
}
pub(crate) fn item(key: &str, text: &str, group: &str, model_written: bool) -> CorpusItem {
let source = if model_written {
Source::ModelWritten {
family: ModelFamily::Unrecorded,
generator: None,
prompt_digest: None,
}
} else {
Source::CastAuthored {
persona: "nibu".into(),
fixture: "f.yaml".into(),
author: "Ada Writer".into(),
}
};
CorpusItem {
id: key.into(),
text: text.into(),
language: "en-US".into(),
speaker: EntityId::parse("entity:nibu").unwrap(),
names: vec![],
group: group.into(),
prior_turns: vec![],
source,
admitted_on: "2026-10-29".into(),
supersedes: None,
withdrawn: false,
}
}
}

107
crates/weksa/tests/checkpoint_door.rs Normal file
View file

@ -0,0 +1,107 @@
//! A line-based tripwire on the Checkpoint door, not a proof. The real defence
//! is that `Checkpoint::for_test` is `cfg(test)`, pinned by the compile_fail
//! doctest on its current name in `split.rs`. This scan reads `split.rs` line by
//! line and catches the common single-line forms: a `pub fn` in `impl
//! Checkpoint` whose signature line says `-> Self` or `-> Checkpoint` without
//! `cfg(test)` above it, a trait impl for `Checkpoint`, a derive on it, a `pub`
//! field in its braces, and a `-> Checkpoint` line in any other `src` file. It
//! misses: signatures wrapped over several lines, `Option`, `Box`, `Result` or
//! tuple returns from `split.rs`, tuple structs with `pub` fields, and
//! constructors made by macros.
/// Why `src` lets code outside the lib make a `Checkpoint`, if it does.
fn door_leaks(split_rs: &str) -> Vec<String> {
let mut leaks = Vec::new();
let lines: Vec<&str> = split_rs.lines().collect();
let mut in_impl = false;
let mut in_struct = false;
for (n, line) in lines.iter().enumerate() {
let t = line.trim();
if t.starts_with("impl Checkpoint") {
in_impl = true;
} else if t.starts_with("impl") && t.contains("Checkpoint") {
leaks.push(format!("{}: trait impl: {t}", n + 1));
}
if t.starts_with("pub struct Checkpoint") {
let derived = lines[..n]
.iter()
.rev()
.take_while(|l| l.trim().starts_with("#[") || l.trim().starts_with("///"))
.any(|l| l.contains("derive"));
if derived {
leaks.push(format!("{}: derive on Checkpoint", n + 1));
}
in_struct = true;
continue;
}
if in_struct {
if t.starts_with('}') {
in_struct = false;
} else if t.starts_with("pub ") {
leaks.push(format!("{}: public field: {t}", n + 1));
}
}
if in_impl && t.starts_with("pub ") && t.contains("fn ") {
let mints = t.contains("-> Self") || t.contains("-> Checkpoint");
let test_only = lines[..n]
.iter()
.rev()
.take_while(|l| l.trim().starts_with("#["))
.any(|l| l.contains("cfg(test)"));
if mints && !test_only {
leaks.push(format!("{}: pub constructor: {t}", n + 1));
}
}
}
leaks
}
#[test]
fn no_public_item_mints_a_checkpoint() {
let src =
std::fs::read_to_string(concat!(env!("CARGO_MANIFEST_DIR"), "/src/split.rs")).unwrap();
assert_eq!(door_leaks(&src), Vec::<String>::new());
// No other module returns one either.
for entry in std::fs::read_dir(concat!(env!("CARGO_MANIFEST_DIR"), "/src")).unwrap() {
let path = entry.unwrap().path();
if path.extension().is_some_and(|e| e == "rs") && !path.ends_with("split.rs") {
let text = std::fs::read_to_string(&path).unwrap();
let minting = text
.lines()
.filter(|l| l.contains("-> Checkpoint") || l.contains("-> Result<Checkpoint"))
.count();
assert_eq!(minting, 0, "{}", path.display());
}
}
}
#[test]
fn the_scan_sees_each_way_to_open_the_door() {
let good = "impl Checkpoint {\n #[cfg(test)]\n pub(crate) fn for_test(r: u32) -> Self { x }\n}\n";
assert!(door_leaks(good).is_empty());
// Test constructor made public, under its own name or another.
let public = "impl Checkpoint {\n pub fn for_test(r: u32) -> Self { x }\n}\n";
assert_eq!(door_leaks(public).len(), 1);
let renamed = "impl Checkpoint {\n #[cfg(test)]\n #[allow(x)]\n pub fn new(r: u32) -> Checkpoint { x }\n}\n";
assert!(
door_leaks(renamed).is_empty(),
"cfg(test) above another attribute counts"
);
let renamed = "impl Checkpoint {\n pub fn new(r: u32) -> Checkpoint { x }\n}\n";
assert_eq!(door_leaks(renamed).len(), 1);
// cfg(test) on the previous item does not cover this one.
let stray =
"impl Checkpoint {\n #[cfg(test)]\n fn a() {}\n pub fn b() -> Self { x }\n}\n";
assert_eq!(door_leaks(stray).len(), 1);
assert_eq!(door_leaks("impl Default for Checkpoint {}\n").len(), 1);
assert_eq!(door_leaks("impl From<u32> for Checkpoint {}\n").len(), 1);
assert_eq!(
door_leaks("#[derive(Clone)]\npub struct Checkpoint {\n").len(),
1
);
assert_eq!(
door_leaks("pub struct Checkpoint {\n pub round: u32,\n}\n").len(),
1
);
assert!(door_leaks("pub struct Checkpoint {\n round: u32,\n}\n").is_empty());
}

126
crates/weksa/tests/context.rs
View file

@ -1,16 +1,15 @@
mod common;
use common::{cast, ent, entities_in, group_in_bucket, item_in, model, world};
use weksa::context::{EncoderOffer, OfferRefusal, Prompt, TrainItems};
use weksa::split::Checkpoint;
use weksa::context::{
ENCODE_INSTRUCTIONS as ENCODE, EncoderOffer, MINT_INSTRUCTIONS as MINT, OfferRefusal, Prompt,
TrainItems,
};
use weksa::{
CorpusItem, CorpusItemId, CorpusStore, NameSpan, PacketId, PacketStore, StoreError,
StoreRefusal, Turn,
};
const MINT: &str = "Mint glosses from the lines below.";
const ENCODE: &str = "Encode the line below.";
struct Corpus {
_dir: tempfile::TempDir,
store: CorpusStore,
@ -109,15 +108,15 @@ fn train_items_are_in_key_order() {
let none = CorpusStore::open(&dir.path().join("c.cc")).unwrap();
let none = TrainItems::from_corpus(&none).unwrap();
assert_eq!((none.len(), none.is_empty()), (0, true));
assert_eq!(Prompt::mint(MINT, &none).text(), MINT);
assert!(Prompt::mint(MINT, &none).item_keys().is_empty());
assert_eq!(Prompt::mint(&none).text(), MINT);
assert!(Prompt::mint(&none).item_keys().is_empty());
}
#[test]
fn mint_prompt_holds_train_text_only() {
let c = corpus();
let t = TrainItems::from_corpus(&c.store).unwrap();
let p = Prompt::mint(MINT, &t);
let p = Prompt::mint(&t);
assert!(p.text().starts_with(MINT));
assert!(p.text().contains("alphamarker"));
for other in ["betamarker", "gammamarker", "deltamarker", "epsilonmarker"] {
@ -145,25 +144,24 @@ fn prompts_render_deterministically() {
.unwrap();
let t = TrainItems::from_corpus(&store).unwrap();
assert_eq!(
Prompt::mint(MINT, &t).text(),
Prompt::mint(&t).text(),
format!(
"{MINT}\n\n[item:a:1] entity:nibu: Hello there.\n\n[item:a:2] entity:nibu: Second line."
)
);
assert_eq!(Prompt::mint(MINT, &t).item_keys(), ["item:a:1", "item:a:2"]);
assert_eq!(Prompt::mint(&t).item_keys(), ["item:a:1", "item:a:2"]);
let offer = EncoderOffer::for_round(&store.get("item:a:2").unwrap().unwrap()).unwrap();
assert_eq!(
Prompt::encode(ENCODE, &offer, &t).text(),
Prompt::encode(&offer, &t).text(),
format!("{ENCODE}\n\n[item:a:2] entity:nibu: Second line.\n\nentities: entity:nibu")
);
assert_eq!(Prompt::encode(ENCODE, &offer, &t).item_keys(), ["item:a:2"]);
assert_eq!(Prompt::encode(&offer, &t).item_keys(), ["item:a:2"]);
}
#[test]
fn encoder_offer_split_gate() {
let c = corpus();
let cp = Checkpoint::declare(1);
let train = get(&c, "item:a:1");
let validation = get(&c, "item:b:1");
let test = get(&c, "item:c:1");
@ -171,7 +169,6 @@ fn encoder_offer_split_gate() {
let withdrawn = get(&c, "item:a:2");
assert!(withdrawn.withdrawn);
let round = |i: &CorpusItem| EncoderOffer::for_round(i).err();
let check = |i: &CorpusItem| EncoderOffer::for_checkpoint(i, &cp).err();
assert_eq!(round(&train), None);
assert_eq!(round(&validation), None);
@ -179,17 +176,10 @@ fn encoder_offer_split_gate() {
assert_eq!(round(&no_split), Some(OfferRefusal::NoSplit));
assert_eq!(round(&withdrawn), Some(OfferRefusal::Withdrawn));
assert_eq!(check(&train), None);
assert_eq!(check(&validation), None);
assert_eq!(check(&test), None);
assert_eq!(check(&no_split), Some(OfferRefusal::NoSplit));
assert_eq!(check(&withdrawn), Some(OfferRefusal::Withdrawn));
// Withdrawn wins over NoSplit.
let mut both = no_split.clone();
both.withdrawn = true;
assert_eq!(round(&both), Some(OfferRefusal::Withdrawn));
assert_eq!(check(&both), Some(OfferRefusal::Withdrawn));
// A withdrawn Test item is Withdrawn, not TestBeforeCheckpoint.
let mut test_gone = test.clone();
test_gone.withdrawn = true;
@ -256,7 +246,7 @@ fn encoder_offer_entities_are_the_items_own() {
"a sibling's entity is not the item's"
);
assert_eq!(offer.key(), "item:a:1");
let p = Prompt::encode(ENCODE, &offer, &t);
let p = Prompt::encode(&offer, &t);
assert!(
p.text()
.ends_with("entities: entity:brokkr, entity:hero, entity:kay")
@ -359,7 +349,7 @@ fn prior_turns_print_only_text_a_live_train_item_holds() {
store.withdraw("item:a:1").unwrap();
let t = TrainItems::from_corpus(&store).unwrap();
let mint = Prompt::mint(MINT, &t);
let mint = Prompt::mint(&t);
for secret in ["withdrawnsecret", "testsecret", "validation reply"] {
assert!(!mint.text().contains(secret), "mint printed {secret}");
}
@ -370,7 +360,7 @@ fn prior_turns_print_only_text_a_live_train_item_holds() {
let encode = |key: &str| {
let item = store.get(key).unwrap().unwrap();
Prompt::encode(ENCODE, &EncoderOffer::for_round(&item).unwrap(), &t)
Prompt::encode(&EncoderOffer::for_round(&item).unwrap(), &t)
.text()
.to_string()
};
@ -391,22 +381,98 @@ fn prior_turns_print_only_text_a_live_train_item_holds() {
);
}
#[test]
fn prior_turn_match_uses_the_stores_text_key() {
/// A scratch Train store holding `train_text`, and an encode prompt for a
/// sibling whose one prior turn is `prior_text`.
fn encode_with_prior(train_text: &str, prior_text: &str) -> (String, Result<(), StoreError>) {
let dir = tempfile::tempdir().unwrap();
let entities = entities_in(dir.path());
let mut store = CorpusStore::open(&dir.path().join("c.cc")).unwrap();
let g = group_in_bucket(10);
store
.add(item_in("item:a:1", "Context Line", &g, cast()), &entities)
.add(item_in("item:a:1", train_text, &g, cast()), &entities)
.unwrap();
let mut reply = item_in("item:a:2", "reply", &g, cast());
reply.prior_turns = vec![Turn {
speaker: common::eid("brokkr"),
text: "context line".into(),
text: prior_text.into(),
}];
store.add(reply.clone(), &entities).unwrap();
let t = TrainItems::from_corpus(&store).unwrap();
let p = Prompt::encode(ENCODE, &EncoderOffer::for_round(&reply).unwrap(), &t);
assert!(p.text().contains("entity:brokkr: context line"));
let p = Prompt::encode(&EncoderOffer::for_round(&reply).unwrap(), &t);
let text = p.text().to_string();
// The store's own rule, as the oracle for key equality; after the prompt is
// built, because an admitted probe would itself become a held line.
let probe = store.add(item_in("item:a:9", prior_text, &g, cast()), &entities);
(text, probe.map(|_| ()))
}
#[test]
fn prior_turn_prints_the_held_train_text() {
let (text, _) = encode_with_prior("Context Line", "context-line!! :)");
assert!(
text.contains("entity:brokkr: Context Line\n[item:a:2]"),
"{text}"
);
for turn_bytes in ["!!", ":)", "context-line"] {
assert!(!text.contains(turn_bytes), "{turn_bytes} in {text}");
}
}
#[test]
fn prior_turn_match_uses_the_stores_text_key() {
let (text, _) = encode_with_prior("Context Line", "context line");
assert!(text.contains("entity:brokkr: Context Line\n"));
assert!(!text.contains("context line"));
}
#[test]
fn prior_turn_fold_matches_the_store() {
let rows = [
("case", "Context Line", "context line"),
(
"punctuation and spacing",
"Context Line",
"context, line!",
),
(
"NFKC fullwidth letters",
"Context",
"\u{ff23}\u{ff4f}\u{ff4e}\u{ff54}\u{ff45}\u{ff58}\u{ff54}",
),
("NFKC fullwidth digits", "room 12", "room \u{ff11}\u{ff12}"),
("dotless i", "Ilik", "\u{131}l\u{131}k"),
("dotted capital I", "Istanbul", "\u{130}stanbul"),
(
"katakana vs hiragana",
"\u{3072}\u{3089}\u{304c}\u{306a}",
"\u{30d2}\u{30e9}\u{30ac}\u{30ca}",
),
("voiced katakana wa", "\u{308f}\u{3099}", "\u{30f7}"),
("Arabic-Indic digits", "room 12", "room \u{661}\u{662}"),
];
for (fold, train_text, prior_text) in rows {
let (text, probe) = encode_with_prior(train_text, prior_text);
assert!(
matches!(
probe,
Err(StoreError::Refused(StoreRefusal::S6TextHeld { .. }))
),
"{fold}: the store does not hold these as one text: {probe:?}"
);
assert!(
text.contains(&format!("entity:brokkr: {train_text}\n[item:a:2]")),
"{fold}: {text}"
);
assert!(!text.contains(ELIDED_MARK), "{fold}: {text}");
}
// A text differing by a letter is a different key: the store admits it and
// the prior turn is elided.
let (text, probe) = encode_with_prior("Context Line", "Context Lane");
assert!(probe.is_ok(), "{probe:?}");
assert!(
text.contains("entity:brokkr: [elided]\n[item:a:2]"),
"{text}"
);
}
const ELIDED_MARK: &str = "[elided]";

75
crates/weksa/tests/score.rs
View file

@ -10,10 +10,9 @@ use common::{cast, group_in_bucket, item_in, model};
use serde_json::Value;
use weksa::CorpusItem;
use weksa::score::{
Embedder, ItemScore, OllamaEmbedder, ScoreInput, Scorer, TestReadings, aggregate, chrf, cosine,
Embedder, ItemScore, OllamaEmbedder, ScoreInput, Scorer, aggregate, chrf, cosine,
leak_diagnostic,
};
use weksa::split::Checkpoint;
/// (reference, hypothesis, sacrebleu 2.4.3 sentence_chrf), map 491536c.
const VECTORS: [(&str, &str, f64); 6] = [
@ -877,78 +876,6 @@ fn leak_diagnostic_reads_validation_only() {
assert!(on_validation.max_chrf.unwrap().chrf > 99.0);
}
#[test]
fn test_readings_at_checkpoint() {
let train_item = train("item:t:1", UNRELATED);
let test = item_in("item:c:1", HELD_OUT, &group_in_bucket(90), cast());
let test_model = item_in(
"item:c:2",
"model words here",
&group_in_bucket(90),
model(),
);
let mut gone = item_in("item:c:3", "withdrawn words", &group_in_bucket(90), cast());
gone.withdrawn = true;
let val = validation("item:v:1", "gardening is quiet work");
let items = [
train_item.clone(),
test.clone(),
test_model.clone(),
gone.clone(),
val.clone(),
];
let scored = [
(&train_item, score(10.0, Some(0.5))),
(&val, score(40.0, Some(0.8))),
(&test, score(60.0, Some(0.6))),
(&test_model, score(100.0, Some(1.0))),
(&gone, score(90.0, Some(0.9))),
];
let t = TestReadings::at_checkpoint(Checkpoint::declare(2), &scored, &[HELD_OUT], &items);
assert_eq!(
(t.count(), t.chrf(), t.meaning()),
(1, Some(60.0), Some(0.6))
);
assert_eq!(
t.items(),
1,
"model-written and withdrawn Test items are excluded"
);
assert!(t.max_chrf().unwrap() > 99.0);
assert_eq!(t.coverage_by_n()[2].share, Some(1.0));
// A minted string that is not the Test text reads low; the same method as
// the Validation reading, so the same minted text reads alike in each.
let t = TestReadings::at_checkpoint(Checkpoint::declare(2), &scored, &["zzz"], &items);
assert!(t.max_chrf().unwrap() < 20.0);
assert_eq!(t.coverage_by_n()[2].share, Some(0.0));
let empty = TestReadings::at_checkpoint(Checkpoint::declare(2), &[], &[HELD_OUT], &[]);
assert_eq!(
(empty.count(), empty.chrf(), empty.meaning()),
(0, None, None)
);
assert_eq!((empty.items(), empty.max_chrf()), (0, None));
}
#[test]
fn validation_and_test_leak_agree_in_method() {
// The same text and minted strings, once in Validation and once in Test,
// read the same numbers.
let minted = chunks(3, " ");
let minted = refs(&minted);
let train_item = train("item:t:1", UNRELATED);
let v = leak_diagnostic(
&minted,
&[validation("item:v:1", HELD_OUT), train_item.clone()],
);
let test = item_in("item:c:1", HELD_OUT, &group_in_bucket(90), cast());
let t = TestReadings::at_checkpoint(Checkpoint::declare(1), &[], &minted, &[test, train_item]);
assert_eq!(t.items(), v.items);
assert_eq!(t.max_chrf(), v.max_chrf.as_ref().map(|p| p.chrf));
assert_eq!(t.coverage_by_n(), v.coverage_by_n.as_slice());
}
#[test]
fn held_out_train_union_excludes_withdrawn_train_items() {
let mut gone = train("item:t:1", HELD_OUT);

6
docs/targets/interlingua-lowering-map.md
View file

@ -402,11 +402,11 @@ One line per cut; specs exist for the cuts about to run (2d and 4, siblings on `
4b. `speaker-scene` (spec r1; `speaker-scene-seeds` is its cut 4c: CLI, inventory revision 3, cast speakers and scenes; `Party::Unnamed` covers an addressee with no entity; Speaker and Scene share one `SpeakerStateStore`): slots `Speaker` and `Scene` with their seed features; `weksa.speaker` (`speaker:<entity-slug>@<n>`) and `weksa.scene` (`scene:<group>@<n>`, participants and ordered-pair relations) documents with `Annotator` `Operator` and `Model { family, generator }`, validated by the packet's shared feature check, append-only per key; their stores, `weksa speaker` and `weksa scene` commands, published schemas; Model-annotated cast speakers and scenes for the committed items. Waits on 2d and 4; `pd-speaker-annotation` blocks only the public-domain annotator.
5. `language-ontology`: `weksa.community` from `data/cultural-ontology/` (owns follow-up `edda-referent-yaml-parse`), `weksa.ontology_entry` for en-US and ja-JP from `data/target-language-ontology/` (YAML deleted), speaker communities, deterministic `select`, the lowering view (packet without trace, speaker, scene, selected entries). Waits on 4.
6. `pd-importer`: public-domain play and fiction dialogue into the corpus with speaker, scene group, edition provenance, and Speaker and Scene annotations by the ruled annotator. Waits on 4, follow-up `gutenberg-terms` and question `pd-speaker-annotation`.
7a. `loop-context-builder`: `context.rs` (`TrainItems`, `EncoderOffer`, `Prompt`: corpus text reaches a model call only through `Prompt::mint` over live Train items or `Prompt::encode` over one offered item; instructions are `&'static str`), `SplitScheme::live`, `CorpusItem::cited_entities` (one owner of the id set `PacketStore::admit` and the offer share), and checkpoint-only Test readings (`Checkpoint`, `TestReadings`; `Aggregate` and `leak_diagnostic` carry no Test). Library only. Merges follow-ups `loop-context-builder-2` and `scorer-test-readings-type`. (spec r1)
7. `golf-loop-en`: the loop driver, the packet store bound to one inventory store (`packet-store-inventory-binding`), `ModelPort` with the `claude -p` adapter (it takes only `&Prompt`), encoder (packet with stance) and lowerer prompts, round records, Score documents, the composite and accept rule, `weksa score`, the detached unit on Yggdrasil. Waits on 3, 5, 6 and follow-up `claude-on-yggdrasil`.
7a. `loop-context-builder`: `context.rs` (`TrainItems`, `EncoderOffer`, `Prompt`: corpus text reaches a model call only through `Prompt::mint` over live Train items or `Prompt::encode` over one offered item; instructions are the consts `MINT_INSTRUCTIONS` and `ENCODE_INSTRUCTIONS` in `context.rs`; `Checkpoint` has no public constructor, offers each item once, and its offers borrow it (the gate is typed `TestReadings` and Test offers on the round path; operator tools such as `weksa corpus dump` and `weksa score` still read corpus text of any split); prior turns print the held Train text), `SplitScheme::live`, `CorpusItem::cited_entities` (one owner of the id set `PacketStore::admit` and the offer share), and checkpoint-only Test readings (`Checkpoint`, `TestReadings`; `Aggregate` and `leak_diagnostic` carry no Test). Library only. Merges follow-ups `loop-context-builder-2` and `scorer-test-readings-type`. (spec r1)
7. `golf-loop-en`: the loop driver, the template wording in `context.rs` with its digest pinned per run (follow-up `template-digest-in-round-record`), the packet store bound to one inventory store (`packet-store-inventory-binding`), `ModelPort` with the `claude -p` adapter (it takes only `&Prompt`), encoder (packet with stance) and lowerer prompts, round records, Score documents, the composite and accept rule, `weksa score`, the detached unit on Yggdrasil. Waits on 3, 5, 6 and follow-up `claude-on-yggdrasil`.
8. `style-axis`: a style-embedding score behind the scorer port. Waits on 3.
9. `ja-panel`: ja-JP into the loop's panel. Waits on 7.
10. `conlang-lexicon`: the first conlang's lexicon placed at concept embeddings, its morphology data, its ontology entries and the gloss reader. Waits on 3, 5 and question `first-conlang`.
11. `convergence-freeze`: the convergence test, the freeze, the re-encode into training pairs. Waits on 7, 9 and 10.
11. `convergence-freeze`: the convergence test, `Checkpoint`'s one door (built from the convergence window over the persisted round records), the freeze, the re-encode into training pairs. Waits on 7, 9 and 10.
12. `distil`: fine-tune a lowerer on the frozen training set. Waits on 11 and a host question asked then.
13. `live-lowering`: agents emit packets with stance; Weksa reads speaker state from persona state (follow-up `persona-state-speech-community`) and lowers for the reader. Waits on 11.