docs/research/2026-10-29-golf-loop-prior-art.md

@@ -65,7 +65,7 @@ Verified licence facts are few. Discord data excluded by policy; not read.
 - PersonaChat (Zhang et al. 2018): ParlAI project page, archived 2020-08-01, states no licence, only "You may continue to use the data without issue". [V] https://parl.ai/projects/personachat/ ; paper https://arxiv.org/abs/1801.07243 says data released open source via ParlAI; licence text [GAP]. Crowdworker-written, persona-statement-conditioned chit-chat [B].
 - LIGHT: ParlAI page states no licence; contents: 663 locations, 3,462 objects, 1,755 character types, 11,000 episodes; WILD 41,131 training episodes from game deployment (human players, so a different provenance than crowdwork). [V] https://parl.ai/projects/light/ . Licence [GAP: check ParlAI repo].
 - Cornell Movie-Dialogs Corpus: licence/terms page fetch failed (certificate rejected) https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html . [GAP] Underlying scripts are copyrighted film text [B]; dataset licence unverified.
-- Project Gutenberg: "vast majority" of eBooks public domain in the US, so no one can grant or withhold permission for uses; PG cannot sublicense copyrighted items; subject to its Terms of Use, which the permission page links and does not summarise; no statement on ML/bulk use on that page. [V] https://www.gutenberg.org/policy/permission.html . Terms-of-Use text on bulk/automated access and the trademark licence: [GAP, not read]. Items public domain in the US may not be elsewhere (same page).
+- Project Gutenberg: "vast majority" of eBooks public domain in the US, so no one can grant or withhold permission for uses; PG cannot sublicense copyrighted items; subject to its Terms of Use, which the permission page links and does not summarise; no statement on ML/bulk use on that page. [V] https://www.gutenberg.org/policy/permission.html . Terms-of-Use text on bulk/automated access and the trademark licence: read 2026-11-13, see `2026-11-13-gutenberg-terms.md`. Items public domain in the US may not be elsewhere (same page).
 - Google Synthetic-Persona-Chat (model-generated, persona-chat-style): CC BY-NC? listed as cc-by-4.0 on HF. [V search-only] https://huggingface.co/datasets/google/Synthetic-Persona-Chat
 - Operator-authored or model-generated corpus: no licence question beyond provider terms [B, unread this pass]. Problems with evaluating on model-generated text: see 2 (self-bias, preference leakage, stylistic homogeneity: Xu et al. 2025; Li et al. 2025). The same Xu et al. finding says it extends to open-ended generation.
 - Expressive-register-annotated English corpora (hedging, humour, affect, politeness labelled), with permissive terms: [GAP] not searched beyond the list above (e.g. Stanford Politeness Corpus, GYAFC formality, Enron etc. are [B] candidates, unverified, GYAFC is known to carry Yahoo Answers terms [B]).
@@ -83,6 +83,6 @@ Verified licence facts are few. Discord data excluded by policy; not read.
 - No per-metric human-correlation numbers for hedging, humour, affect, stance, persona voice; nearest is formality (Briakou 2021). COMET, BLEURT, MAUVE correlations not obtained.
 - No source on surface-form smuggling through an LLM bottleneck or its defences (vocab restriction, paraphrase invariance).
 - No source on schema/inventory growth from generation failures, or its overfitting controls.
-- Cornell, LIGHT, PersonaChat licence texts not obtained; Gutenberg ToU on bulk use not read.
+- Cornell, LIGHT, PersonaChat licence texts not obtained.
 - No realiser or evaluation protocol for constructed languages from a meaning representation.
 - Several arXiv PDFs (Briakou, Zheng, STEL, LUAR/StyleDistance) fetched as binary; numbers above come from the ar5iv/arXiv-HTML versions where marked, otherwise search summaries.

docs/research/2026-11-13-gutenberg-terms.md

@@ -0,0 +1,59 @@
+# Project Gutenberg terms: bulk access and header stripping
+
+Read 2026-11-13 for follow-up `gutenberg-terms` (campaign `weksa-lowering`),
+ahead of cut `pd-importer`. Pages: gutenberg.org `policy/permission.html`,
+`policy/terms_of_use.html`, `policy/robot_access.html`, `policy/license.html`.
+The quotes were checked against the live HTML of each page by a second reader.
+`pd-importer`'s Imagination re-reads the pages before it fixes a fetch path,
+since they change.
+
+## Findings
+
+- **Automated access to the main site is refused.** Terms of Use: "This website
+  is intended for human users only"; "Any perceived use of automated tools to
+  access this website will result in a temporary or permanent block". OPDS
+  feeds: no more requests than a browser user would make; heavy requesters are
+  blocked by IP.
+- **Bulk has two sanctioned paths.** Terms of Use: take bulk downloads "from
+  one of our mirrors, not from the main site". The robot-access page names the
+  `robot/harvest` wget endpoint (`-w 2`, filterable by file type or language)
+  and recommends a private mirror for a full local copy; it suggests checking
+  periodically for changes. Catalog data are offered as
+  machine-readable offline catalogs, "granted to the public domain".
+- **Header and licence stripping.** Licence section 1.C: for a work not
+  protected by US copyright, with the user in the US, the Foundation does "not
+  claim a right to prevent you from copying, distributing, performing,
+  displaying or creating derivative works", "as long as all references to
+  Project Gutenberg are removed", the condition holding for a user in the US.
+  Section 1.E opens "Unless you have removed all references to Project
+  Gutenberg", and 1.E.1 to 1.E.7 (including 1.E.4, do not detach the licence)
+  attach to a work on which the phrase "Project Gutenberg" appears or with which
+  it is associated, so a text with every reference removed is outside 1.E.
+  1.E.2 is not that grant: it says a PG eBook derived from texts not restricted
+  by US copyright may be copied and distributed in the US without fee. The
+  trademark is registered; non-commercial use of the name needs no permission,
+  commercial use does.
+- **No statement on machine learning or AI use** on any of the four pages.
+- **US public domain only.** The Foundation makes no claim about copyright
+  outside the US (1.D). A work free in the US may not be free elsewhere, and
+  Gutenberg's own catalog is not a per-work clearance.
+
+## What this decides for `pd-importer`
+
+- Fetch from a mirror or an offline dump. The Terms of Use footnote counts "many
+  books" as more than about 100 a day and sends those to mirrors, while
+  `robot/harvest` is a `www.gutenberg.org` URL: use it (with `-w 2`) only for a
+  handful of works. Never crawl `gutenberg.org` pages or OPDS from the loop host.
+- Strip the whole header and footer, every Project Gutenberg reference and the
+  licence text from the stored text; keep provenance as the ebook number and
+  edition in the item's source field, which is Weksa's own record and not a use
+  of the mark. This is the path `docs/targets/dao-de-jing-test-bed.md` already
+  takes. The remaining open question there, what counts as a "reference", stays
+  open: the safe reading is that the ebook number in a provenance field is a
+  catalogue identifier, not a reference inside the work, but nothing read here
+  rules on it.
+- Per-work check stays with the importer: the work's author and first
+  publication date against US public-domain rules, recorded on the item. The
+  Foundation's catalogue does not supply it. A work Gutenberg posts with the
+  copyright holder's permission (1.E.3) is out of scope.
+- No importer code in this note.

docs/targets/dao-de-jing-test-bed.md

@@ -23,9 +23,10 @@ plus a separately recorded chosen gloss per lowering).
   `corpus add ... --language en-US --source public-domain` per item, then `packet dao <store>`.
 - Source: Project Gutenberg eBook 216, "The Tao Teh King, or the Tao and its
   Characteristics", Laozi, tr. James Legge (Sacred Books of the East 39, 1891).
-  Gutenberg's licence (1.E.2) lets a work derived from texts not protected by US
-  copyright be used with no fee provided every Project Gutenberg reference is
-  removed from the work. The items hold only Legge's words, one numbered section
+  Gutenberg's licence (1.C) does not claim a right to prevent derivative works of
+  a work not protected by US copyright, for a user in the US, provided every
+  Project Gutenberg reference is removed (see
+  `docs/research/2026-11-13-gutenberg-terms.md`). The items hold only Legge's words, one numbered section
   or sentence each; no Gutenberg header, licence text or trademark is stored,
   and the eBook file itself is not committed. `edition_url` names the page the
   text was retrieved from, as a citation of provenance and not as part of any

docs/targets/interlingua-lowering-map.md

@@ -340,9 +340,14 @@ line two", "その　制御" against the
 - Stores: `data/interlingua/inventory.cc` and `data/corpus/corpus.cc` in the
   repo (small, reviewed with `weksa dump`); run stores (rounds, lowerings,
   scores) on Yggdrasil under the loop's state root, not in git.
-- The loop runs on Yggdrasil as a detached unit with a log and a status record.
-  Starfire runs none of it. Model calls go through a `ModelPort`; its first
-  adapter drives `claude -p` on the operator's subscription. Never OpenAI
+- The loop is a batch command driven from a Claude Code session on Yggdrasil
+  (`golf-loop-in-claude`), not a detached unit. Model calls go through a
+  `ModelPort` whose one adapter, `AnswerFiles`, writes a `.prompt` file and reads
+  a `.answer` file: Claude agents answer them, one fresh subagent per encode
+  prompt (`encoder-isolation-subagent-per-prompt`) and a tool-less relay for
+  lowering prompts (`lowerer-call-reach-toolless-relay`). Builds, stores, scoring
+  and embeddings run on Yggdrasil; Starfire runs none of it, and Weksa needs no
+  claude binary or token there. Never OpenAI
   (`eureka-body:ruling:no-return-to-openai-supplier`), never Discord data
   (`gamecult-chat:ruling:discord-taints-training`).
 
@@ -675,9 +680,9 @@ One line per cut; specs exist for the cuts about to run (2d and 4, siblings on `
 6. `pd-importer`: public-domain play and fiction dialogue into the corpus with speaker, scene group, edition provenance, and Speaker and Scene annotations by the ruled annotator. Waits on 4, follow-up `gutenberg-terms` and question `pd-speaker-annotation`.
 7a. `loop-context-builder`: `context.rs` (`TrainItems`, `EncoderOffer`, `Prompt`: corpus text reaches a model call only through `Prompt::mint` over live Train items or `Prompt::encode` over one offered item; instructions are the consts `MINT_INSTRUCTIONS` and `ENCODE_INSTRUCTIONS` in `context.rs`; `Checkpoint` has no public constructor, offers each item once, and its offers borrow it (the gate is typed `TestReadings` and Test offers on the round path; operator tools such as `weksa corpus dump` and `weksa score` still read corpus text of any split); prior turns print the held Train text), `SplitScheme::live`, `CorpusItem::cited_entities` (one owner of the id set `PacketStore::admit` and the offer share), and checkpoint-only Test readings (`Checkpoint`, `TestReadings`; `Aggregate` and `leak_diagnostic` carry no Test). Library only; since 7b `TrainItems::from_corpus` and `EncoderOffer::for_round` are crate-private and `RunStores` is their one public producer. Merges follow-ups `loop-context-builder-2` and `scorer-test-readings-type`. (spec r1)
 7b. `golf-loop-round` (spec r1; branch `weksa-lowering/golf-loop-round`): the smallest loop that scores one round. `run.rs`: `RunStores` pins the corpus and entity stores of a run by canonical path and FNV-1a 64 digest at `create`, reopens only from `<root>/run.cc` and refuses drift (corpus, entities, `PROMPT_SOURCE_DIGEST` of `context.rs`, a second pin on `lower.rs` retired in 7d, model, embedder); `round` encodes every live Train and Validation item through a `ModelPort` (`model.rs`; the one adapter `AnswerFiles` writes `<digest>.prompt` and reads `<digest>.answer`, so the model is the Claude session), reads each answer into a `PacketDraft` (`encode.rs`, the one place a model's JSON is read), admits through `PacketStore::admit` into `<root>/round-<n>/packets.cc`, lowers, scores once, and keeps a `RoundRecord` (`weksa.loop_round`, no text, no Test; Train rows, Validation only as means and refusal counts). `weksa loop open|round`. Not here: the spec-edit step, accept rule and composite, noise floor, Score and lowering documents, a subprocess adapter, the checkpoint door, a model lowerer (follow-ups `golf-loop-spec-edit`, `golf-loop-accept-rule`, `golf-loop-score-documents`, `golf-loop-subprocess-adapter`, `golf-loop-checkpoint-door-bound`, `golf-loop-model-lowerer`). `packet-store-inventory-binding` is met for loop packets only (`RunStores` admits with the run's inventory); `validate`, `packet proof|dao` and `lower` still take caller stores. Waits on 7a, `checkpoint-gate`, 4b.
-7c. `golf-loop-edit-accept` (spec r1; branch `weksa-lowering/golf-loop-edit-accept`): one edit-accept cycle. The run works on its own copy of the inventory (`create` copies the caller's file to `<root>/inventory.cc`, refusing a root that already holds a file of that name, and pins its digest as `RunRecord` key 11; `RunRecord.inventory` is provenance; a `run.cc` from before has no key 11 and does not open); the head is derived from the latest Accepted `Decision`. `weksa loop cycle` asks the editor for one inventory edit from `Prompt::mint(train, inventory, rev, feedback)` (Train blocks, the inventory listing and an `EditFeedback` line: whether the last edit was kept and the head's Validation composite rounded to `LADDER_STEP`, the Ladder release of Blum and Hardt 2015), reads it as an `EditDraft` (`encode.rs`; ids and revisions are the store's, `into_revision`), admits it through `InventoryStore::admit_revision` into `<root>/cycle-<k>/inventory.cc`, runs a candidate round under it, and `decide`s on Validation against the head's establishing round past a floor (never under one `LADDER_STEP`; from cut `cycle-floor-paired` the paired floor over rounds 1 to `FLOOR_ARMS_MIN` + 1, so a cycle needs four recorded rounds, where 7c measured the spread between rounds 1 and 2). An edit that fails is a rejected `Decision`, not an error. `RoundRecord` gains keys 16 (inventory digest) and 17 (cycle); an encode answer that is not UTF-8 is that item's parse refusal. Not here: Score and lowering documents (`golf-loop-score-docs`), Train rows in the edit prompt (`golf-loop-edit-train-feedback`), a paired floor (`golf-loop-paired-floor`), the checkpoint door and merging to `data/` (`golf-loop-checkpoint-door-bound`, `golf-loop-freeze-merge`), a model lowerer (`golf-loop-model-lowerer-next`; until it lands an entry an edit adds has no lowering row, so a cycle gains only by retiring entries or adding ones the encoder does not cite). Waits on 7b.
+7c. `golf-loop-edit-accept` (spec r1; branch `weksa-lowering/golf-loop-edit-accept`): one edit-accept cycle. The run works on its own copy of the inventory (`create` copies the caller's file to `<root>/inventory.cc`, refusing a root that already holds a file of that name, and pins its digest as `RunRecord` key 11; `RunRecord.inventory` is provenance; a `run.cc` from before has no key 11 and does not open); the head is derived from the latest Accepted `Decision`. `weksa loop cycle` asks the editor for one inventory edit from `Prompt::mint(train, inventory, rev, feedback)` (Train blocks, the inventory listing and an `EditFeedback` line: whether the last edit was kept and the head's Validation composite rounded to `LADDER_STEP`, the Ladder release of Blum and Hardt 2015), reads it as an `EditDraft` (`encode.rs`; ids and revisions are the store's, `into_revision`), admits it through `InventoryStore::admit_revision` into `<root>/cycle-<k>/inventory.cc`, runs a candidate round under it, and `decide`s on Validation against the head's establishing round past a floor (never under one `LADDER_STEP`; from cut `cycle-floor-paired` the paired floor over rounds 1 to `FLOOR_ARMS_MIN` + 1, so a cycle needs four recorded rounds, where 7c measured the spread between rounds 1 and 2). An edit that fails is a rejected `Decision`, not an error. `RoundRecord` gains keys 16 (inventory digest) and 17 (cycle); an encode answer that is not UTF-8 is that item's parse refusal. Not here: Score and lowering documents (`golf-loop-score-docs`), Train rows in the edit prompt (`golf-loop-edit-train-feedback`), a paired floor (`golf-loop-paired-floor`), the checkpoint door and merging to `data/` (`golf-loop-checkpoint-door-bound`, `golf-loop-freeze-merge`), a model lowerer (landed as 7d: until then an entry an edit added had no lowering row). Waits on 7b.
 7d. `golf-loop-model-lowerer` (spec r1; branch `weksa-lowering/golf-loop-model-lowerer`): the round lowers through the `ModelPort`. `Prompt::lower` (`context.rs`) takes a `LoweringInput` (no text, key, group or speaker getter), the item's speaker id, the inventory at the view's revision and the item language's entity forms, and prints `LOWER_INSTRUCTIONS`, the meaning of the packet, the entities line and the inventory listing; the item key goes only to `item_keys`, and the speaker id never prints: a referent citing the item's speaker holds the fixed token `entity:speaker` (`SPEAKER_TOKEN`, a constant the same for every speaker and item, never derived from an id) and the entities map holds the speaker's forms under that token. The promise guards the id that names an entity of this item; the speaker's forms and the role token do print, because forms are shared by every item of the language (follow-up `speaker-alias-token-in-lowering-prompt`, which restored the lowerer's ability to name its speaker after #84 removed it). A speaker the language has no form for, or the port lacks, does not refuse: its referents print no entity and the map has no entry. A referent citing the real id `entity:speaker` is refused (`LowerError::SpeakerTokenTaken`); the entity store does not reserve it. The answer is `{"text"}`, read as `LoweringDraft` in `encode.rs`, held in `score_round`'s locals and scored once; it is never put in `run.cc`, a `RoundRecord`, a `Decision` or `EditFeedback`. A round has two answer phases (every encode prompt, then every lowering prompt), all or nothing: while any prompt is unanswered `packets.cc` is discarded and nothing reaches `run.cc`. `lower::lower` and the template table leave the loop (they remain `weksa lower`'s; follow-up `table-retirement`), so an entry an edit adds can be said and a cycle can buy a gain. One source pin remains: `PROMPT_SOURCE_DIGEST` covers `context.rs`, `split.rs` and `lower.rs` (`RunRecord` key 7 and `RoundRecord` key 3 are retired). The run's record pins `LowererReach` (`weksa loop open --lowerer-reach briefed-subagent|tool-less-relay`, `RunRecord` key 12) and every printed round and cycle line says it; a briefed-subagent run says its numbers are smoke-round only (rulings `lowerer-call-reach-toolless-relay`, `dao-round-seed-inventory`). A lowering is refused at `Lower` when blank, over `MAX_LOWERING_BYTES` (4096) or holding a control character other than a line break or tab; `PROMPT_SOURCE_DIGEST` also covers `packet.rs`. Not here: per-item lowering text documents (`golf-loop-score-docs`), the subprocess adapter. Waits on 7c.
-7. `golf-loop-en` (7b and 7c landed the round, `ModelPort`, encoder prompt, template digest, round records and the edit-accept cycle; the rest remains): the packet store bound to one inventory store everywhere (`packet-store-inventory-binding`), a subprocess adapter, Score documents, `weksa score` in the loop, the detached unit on Yggdrasil. Waits on 3, 5, 6 and follow-up `claude-on-yggdrasil`.
+7. `golf-loop-en` (7b, 7c and 7d landed the round, `ModelPort` with `AnswerFiles`, encoder and lowering prompts, template digest, round records and the edit-accept cycle; the rest remains): the packet store bound to one inventory store everywhere (`packet-store-inventory-binding`), Score documents (`golf-loop-score-docs`, `golf-loop-score-documents`), the accept rule (`golf-loop-accept-rule`), the checkpoint door and freeze merge (`golf-loop-checkpoint-door-bound`, `golf-loop-freeze-merge`). No subprocess adapter and no detached unit: `golf-loop-in-claude` and `encoder-isolation-subagent-per-prompt` rule them out. Waits on 3, 5 and 6.
 8. `style-axis`: a style-embedding score behind the scorer port. Waits on 3.
 9. `ja-panel`: ja-JP into the loop's panel, in two cuts on `e9cbd9b`. 9a `ja-panel-loop` (spec r1): language is the item's and the panel the run's: `Prompt::lower` takes the item's language and prints it on a language line (no prompt names English), entity forms are read in that language, `RunRecord` pins the panel (`weksa loop open --panel`), offers, mint and recall read only panel items, round means, floors and readings are per panel language, the accept rule runs `decide` per language, the recall split works on unspaced scripts, and `weksa corpus add` has no default language. 9b `ja-panel-corpus` (spec r1, after 9a): the ja-JP slice of `data/original`, model-written Japanese Train and Validation passages and eight Aozora Bunko public-domain Test sentences with full provenance, and their entities. Cut 5 is not a prerequisite: neither language lowers with ontology entries until cut 5 gives both theirs.
 10. `conlang-lexicon`: the first conlang's lexicon placed at concept embeddings, its morphology data, its ontology entries and the gloss reader. Waits on 3, 5 and question `first-conlang`.