Weksa/docs/scene-annotation.md

Scene annotation

How the keeper scenes of data/original/speaker-state.cc are labelled. A scene label has a committed rule that produced it; this file is that rule.

Why

Two findings forced a written rule.

• annotation-rank-from-register: an annotator who reads the line's register (politeness forms, honorifics, endings) to decide the speaker's rank hands the lowering its own answer back, because register is what the lowering must produce from the scene.
• The addressee gate (with_scene in crates/weksa/src/validate.rs): a scene is read only when the packet names a discourse addressee. A line that addresses no one reads NoAddressee and no Scene entry selects, so labels on narration are inert and the only labels that bite are those on lines said to someone.

The earlier labels followed two briefs that were never committed and that disagreed with each other (the relation to the reader, and a situation read with the register ignored). The rubric below replaces both for every live group.

Provenance

Cut weksa-lowering:cut_spec:cut-scene-in-group.r1, ruling weksa-lowering:ruling:scene-rubric-whole-bed: all 98 live Train and Validation groups of data/original receive a new scene revision from annotator A under rubric r1, generator claude-sonnet-5-5 (Weksa Hands, scene rubric r1, docs/scene-annotation.md). Earlier revisions stay as provenance. The rubric uses no corpus line as an example on purpose: a labelled example would make the blind check against the old labels circular.

A later rubric revision gets a new section and a new generator string; r1 is never edited.

Rubric r1

The speaker of every line is the keeper, a quiet caretaker of a house and garden. First decide whether the line addresses a listener: it asks them something, asks them to do something, tells them about themselves, wishes something for them, or speaks to them as you. A line that addresses no one (narration, description, a remark about the world or about other people, a thought or wish with no one addressed) is told to the reader, a stranger: answer addressed false, relative_status level, intimacy 500, in_group false. People named inside a line are not its listener.

For a line that addresses a listener, answer addressed true and judge the keeper's relation to that listener from the situation the line states or implies, never from its register: politeness forms, honorifics, sentence endings, contractions, please and casual particles are what the lowering must produce from the scene, so they cannot decide it. relative_status: above when the situation gives the keeper standing over the listener (instructing, advising or permitting as elder, host, employer or teacher); below when it gives the listener standing over the keeper (the keeper serves, thanks, or asks a favour of a guest, employer, elder or official); level when it shows neither.

intimacy, 0 to 1000: 500 when the line shows nothing; up to 800 when it shows shared daily life (the listener's body, sleep or habits, the household's routine); above 800 only for a stated close bond; down to 200 for a stated first meeting or a service relation. in_group: true only when the line shows the listener belongs to the keeper's household, family or working circle, stated or plainly implied; a request, a question or a kind wish alone is not enough; otherwise false. Answer one JSON line per group, in the order given and nothing else: {"group": G, "addressed": true|false, "relative_status": S, "intimacy": N, "in_group": true|false}.

Procedure

• Annotator A (committed) and annotator B (measurement only) are separate fresh model agents, one pair per language (49 live groups each). Each sees only the text of Rubric r1 above and that language's sentences as group<TAB>"text", shuffled by a fixed seed. Neither sees an existing scene, the other annotator's answer, the ontology, which items are old or new, or any Test text.
• An output that does not parse or misses a group is re-drawn once and logged.
• A's labels are written with weksa scene put (one revision per group, --to unnamed --inventory-rev 4). addressed is not stored: no Scene feature carries it.
• Disagreements are reported, never adjudicated.

Agreement at r1 (this cut, 98 live groups, 49 per language)

Four fresh agents drew the labels (A-en, A-ja, B-en, B-ja); the briefs were the rubric section plus one language's shuffled sentences (SHA-256 of the brief bodies: en-US db087476...dbc04e, ja-JP c5b81b27...97488c; no Test text in either). No draw was repeated. Annotator model: claude-sonnet-5-5.

A against B (B measures only): pooled addressed equal 96/98 (kappa 0.96), in_group equal 93/98 (kappa 0.79), relative_status equal 89/98, intimacy within 150 on 98/98. en-US: addressed 47/49, in_group 47/49 (0.86), relative_status 45/49. ja-JP: addressed 49/49, in_group 46/49 (0.64), relative_status 44/49.

Blind check, A against the 56 older labels: relative_status equal 49/56, in_group equal 53/56 (kappa 0.64), intimacy within 150 on 55/56. Three in_group differences: original-ja-shimoyado-2 and original-ja-sakana (both narrations, true to false) and original-ja-asa (an instruction to a household member, false to true).

In_group true across the 98: 42 before, 15 after; every unaddressed line is now false (it was true on 17 of the 21 unaddressed vgrow groups). Disagreements are reported, not adjudicated.

Reproducing the cut

scripts/scene-annotation/ holds the standard-library scripts of this cut, run from the repo root with WEKSA (the binary) and SCENE_SCRATCH (a directory that is never committed): build_briefs.py writes the two annotator briefs from the Rubric r1 section and prints each brief's Test-leak count (must be 0) and sha256 (it reproduces the briefs of this cut byte for byte); put_labels.py puts A's A-en.jsonl and A-ja.jsonl; agreement.py prints the tables above and must run before the puts, against the store at 76fa685, because the older labels are then its latest revisions (it also reads the round-1 encoder answers of runs/growth-base through ROUND1). DIGEST_OF_THE_RUBRIC and DIGEST_OF_THE_EARLIER_SCENES in crates/weksa/tests/original_corpus.rs pin the rubric text and the 113 earlier revisions.

Known limits of r1 labels

Found by Soul's pass on this cut (weksa-lowering:verdict:cut-scene-in-group.s1). They are not fixed in r1; a reviewer's pass over the labels is follow-up weksa-lowering:follow_up:scene-labels-reader-review.

• The new labels change which ontology entries are selected for 37 groups. Eight addressed lines move onto the casual in-group entries (en-US:12: lantern, broom, mirror, seedling; ja-JP:12: furo, asa, irori, yurikago), while their references are polite requests. The reader default (level, 500, false) is exactly the condition of ja-JP:11.
• relative_status tracks request form for some lines: imperatives read above, favour questions read level or below, so the same household request gets different standing in two forms (hane above/650/true, kabin below/500/false; B gave both level/false). Form is what signals the speech act, which is what part 2 asks for, and register is what part 2 forbids.
• in_group is true on some requests alone (lantern, mulch, stool, hane) and on a line whose only family member is named inside it (original-ja-ane), though r1 says a request or a person named inside the line is not enough. A and B agreeing does not clear this.
• A run opened on the old speaker-state refuses to continue after merge: runs/growth-base pins the store of /var/lib/mind-weksa-self/Weksa-growth (main), and that checkout must not be pulled while the run is wanted.