From 6e43aa2bd8fd0baf9b2cdae209945990d2a988de Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sat, 2 May 2026 10:08:58 +0100 Subject: [PATCH] Clarify training corpus scale tiers --- docs/architecture/training-plan.md | 81 +++++++++++++++++++++----- notes/fresh-workspace-handoff.md | 7 ++- notes/ghostlight-current-system-map.md | 6 +- state/evidence.jsonl | 1 + state/map.yaml | 3 +- 5 files changed, 82 insertions(+), 16 deletions(-) diff --git a/docs/architecture/training-plan.md b/docs/architecture/training-plan.md index d1f1221..7025f76 100644 --- a/docs/architecture/training-plan.md +++ b/docs/architecture/training-plan.md @@ -89,6 +89,57 @@ Some stages are trained as generative models. Some are trained as classifiers or rankers. Some may remain manual longer than the rest because bad automation here would create narrative sewage with confidence. +## Corpus Scale Tiers + +The per-stage counts below are pilot gates, not robust training targets. + +Pilot data exists to shake out schema mistakes, artifact shape problems, +leakage boundaries, reviewer workflow, and validator gaps. It is allowed to be +sacrificial. If a pilot proves that the schema is wrong, update the schema and +accept that some pilot artifacts may become obsolete. Better a small controlled +burn than a cathedral built from cursed bricks. + +Use three corpus tiers: + +- `pilot_schema_shakedown` + - proves that artifacts can be produced, reviewed, validated, and loaded + - exposes missing fields, wrong boundaries, bad labels, and impossible + assumptions + - may be invalidated by schema changes +- `review_assistant_target` + - enough examples for a model to draft useful candidates under human or + frontier-model review + - still not trusted for unattended runtime mutation or simulation authority + - expected to include broad negative examples and repair labels +- `runtime_target` + - enough diversity, sequence coverage, negative coverage, and adversarial + checks to trust a model inside automated loops with deterministic gates + - requires held-out evaluation sets, regression fixtures, and failure audits + - still cannot bypass deterministic legality, visibility, source, or mutation + gates + +Approximate target scale: + +| Stage | Pilot Schema Shakedown | Review Assistant Target | Runtime Target | +| --- | ---: | ---: | ---: | +| Lore grounding compiler | 25 accepted / 25 rejected | 500+ digests | 2,000+ digests | +| Coordinator/story runtime | 50 accepted / 25 rejected | 1,000+ turns | 5,000+ turns | +| Memory/lore retriever | 500 labeled pairs | 5,000+ pairs | 50,000+ pairs | +| Projector | 100 accepted / 50 rejected | 1,000+ examples | 5,000+ examples | +| Character agent/responder | 200 accepted / 100 rejected | 2,000-5,000 turns | 10,000+ turns | +| Event resolver normalization | 100 reviewed examples | 1,000+ examples | 5,000+ examples | +| Participant appraiser | 300 appraisals | 3,000+ appraisals | 10,000+ appraisals | +| State mutator | 500 accepted / 200 rejected | 5,000+ patches | 10,000+ patches | +| Relationship/perception updater | 500 updates | 5,000+ updates | 20,000+ sequence updates | +| Guardrail/evaluator classifiers | 300 labeled failures | 3,000+ labeled outputs | 20,000+ labeled outputs | +| Institution/faction/consumer decisions | 500 decisions | 5,000+ decisions | 20,000+ decisions | +| Technology/item manifests | 100 records / 50 rejected | 1,000+ manifests | 5,000+ manifests | + +The exact numbers will move once we know which organs are narrow classifiers, +which are generative, which can lean on retrieval, and which are mostly +deterministic code wearing a little judgment hat. The important rule is that the +first gate proves the schema and workflow. It does not prove robustness. + ## Model Families Use these as the default architecture assumptions until evidence says otherwise. @@ -216,7 +267,7 @@ Training artifacts: - source violation labels - corrections tied to AetheriaLore paths -First training gate: +Pilot schema shakedown gate: - 25 accepted digests across distinct Aetheria eras or institutions - 25 rejected/source-violation examples @@ -290,7 +341,7 @@ Training artifacts: - reviewer labels for pacing, continuity, legibility, game usefulness, and lore grounding -First training gate: +Pilot schema shakedown gate: - 50 accepted coordinator turns across at least 5 scenes - 25 rejected coordinator turns with labeled failures @@ -347,7 +398,7 @@ Training artifacts: - reviewer relevance labels - missed-critical-context labels -First training gate: +Pilot schema shakedown gate: - 500 labeled retrieval pairs before tuning - evaluation set includes false friends: similar vibe, wrong faction or era @@ -402,7 +453,7 @@ Training artifacts: - source grounding failures - reviewer corrections -First training gate: +Pilot schema shakedown gate: - 100 accepted projection examples - 50 rejected examples with clear failure labels @@ -482,7 +533,7 @@ Training artifacts: - future-branch responses exercising post-Rupture concepts as branch-local canon without claiming branch facts apply to sibling histories -First training gate: +Pilot schema shakedown gate: - 200 accepted character turns across eras, factions, species, and social roles - 100 rejected turns with labeled failures @@ -541,7 +592,7 @@ Training artifacts: - normalized action labels - mechanical delta records -First training gate: +Pilot schema shakedown gate: - do not train until action/event schemas stabilize - collect 100 reviewed action-to-event examples before considering normalization @@ -592,7 +643,7 @@ Training artifacts: - rejected interpretations - reviewer rationale -First training gate: +Pilot schema shakedown gate: - 300 participant appraisal examples - balanced set of correct reads, misreads, ambiguous reads, and biased reads @@ -647,7 +698,7 @@ Training artifacts: - branch-local patches separated from promoted state patches - branch-local patches separated from cross-branch source constraints -First training gate: +Pilot schema shakedown gate: - 500 accepted mutation examples - 200 rejected/deferred mutation examples @@ -695,7 +746,7 @@ Training artifacts: - confidence driver labels - rejected updates -First training gate: +Pilot schema shakedown gate: - 500 directional relationship updates - repeated-contact sequences, not only isolated events @@ -741,7 +792,7 @@ Training artifacts: - false positive/false negative review notes - failure taxonomy examples -First training gate: +Pilot schema shakedown gate: - 300 labeled failures across prompt leakage, omniscience, wrong body, wrong lore, invalid mutation, schema drift, and flattened character agency @@ -803,7 +854,7 @@ Training artifacts: - item availability and component dependency decisions - faction access or restriction decisions -First training gate: +Pilot schema shakedown gate: - do not train until scene-local machinery produces reliable decision examples - collect at least 500 reviewed decision examples across consumer and @@ -895,7 +946,7 @@ Training artifacts: - instance-provenance records for manufactured artifacts when a scene or branch needs realized supply-chain history, not just a blueprint -First training gate: +Pilot schema shakedown gate: - item manifest schema and validator exist - validator checks target Aetheria-Economy blueprint class, source-backed @@ -933,7 +984,11 @@ Recommended order: 11. Train institution/faction/consumer models after scene-local decisions provide enough grounded examples. -## Minimum Corpus Before First Fine-Tune +## Minimum Corpus Before First Experimental Fine-Tune + +These are still pilot numbers. Use them for controlled experiments, schema +pressure tests, and early behavior probes. Do not treat them as enough for +robust runtime use. A first useful Aetheria responder fine-tune wants: diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 8b450ba..dfa2c3b 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -145,7 +145,12 @@ reviewed output, hidden-context refs, leakage audit, isolation method, and coordinator intervention labels. Coordinator repairs are allowed, but they must be labeled as repairs rather than treated as raw responder behavior. The concrete training plan now enumerates the trainable stages, their inputs, -outputs, likely model families, artifact families, and first corpus gates. +outputs, likely model families, artifact families, and pilot corpus gates. The +listed pilot counts are schema shakedown thresholds, not robust training +targets. They exist to expose missing fields, bad boundaries, leakage risks, and +validator gaps before large data generation. Robust organs will need roughly an +order of magnitude more examples, with separate review-assistant and runtime +targets. It now uses three timeline lanes: `historical_grounded`, `transition_grounded`, and `future_branch`. Future branches are required training/evaluation material for post-Rupture Elysium concepts, but generated diff --git a/notes/ghostlight-current-system-map.md b/notes/ghostlight-current-system-map.md index d3b2d49..c2b9359 100644 --- a/notes/ghostlight-current-system-map.md +++ b/notes/ghostlight-current-system-map.md @@ -306,7 +306,11 @@ eleven trainable stages and their likely model families: generative decoder LLMs for coordinator, responder, and structured soft outputs; classifiers or cross-encoders for appraisal, evaluation, and relationship movement; embedding/retrieval models for memory and lore selection; -and deterministic code for gates. The coordinator is currently the authoring +and deterministic code for gates. The listed per-stage counts are pilot +schema-shakedown gates, not robust training targets; review-assistant and runtime +targets are larger corpus tiers meant to avoid declaring victory after the first +small pile of sacrificial data merely shows where the schema is wrong. The +coordinator is currently the authoring agent: it glues scenes together, carries world-state refs and unresolved hooks, decides which machinery runs, and emits glue prose. That output should stay game-engine-shaped even while the prose lives in model imagination. The diff --git a/state/evidence.jsonl b/state/evidence.jsonl index b2ebc68..aae2762 100644 --- a/state/evidence.jsonl +++ b/state/evidence.jsonl @@ -58,3 +58,4 @@ {"ts": "2026-05-02T08:42:58+00:00", "type": "research", "status": "ok", "note": "Aetheria-Economy already provides the concrete item schema target: CultCache-backed ItemData subclasses are technological blueprints (SimpleCommodityData, CompoundCommodityData, ConsumableItemData, GearData/EquippableItemData), while ItemInstance subclasses represent realized runtime artifacts carrying quality/provenance/branding/world state. Ghostlight item manifests should map onto that model, with assembly trees, recipes, compatibility, tooling, facilities, process requirements, and supply-chain dependencies recorded as blueprint metadata or engine-schema gaps until the economy code exposes explicit storage."} {"ts": "2026-05-02T08:51:57+00:00", "type": "architecture", "status": "ok", "note": "Documented the player-facing gear fantasy and content-refinery framing: ultimate items should be provenance hunts through blueprint, assembly quality, manufacturer practice, supply-chain lineage, repair history, counterfeit substitution, and operating conditions; Ghostlight should convert lore-grounded ideas into reviewed database-shaped Aetheria-Economy candidates rather than another manual spreadsheet bottleneck."} {"ts": "2026-05-02T09:04:24+00:00", "type": "architecture", "status": "ok", "note": "Accepted sandboxed responder-data doctrine: coordinator may be omniscient, but gold responder examples must be generated from only the exact projected local packet, visible event, allowed actions, and explicitly included source excerpts. Artifacts must preserve raw output, reviewed output, hidden-context refs, leakage audit, isolation method, and coordinator intervention labels so repaired prose is not laundered as raw responder behavior."} +{"ts": "2026-05-02T09:08:43+00:00", "type": "architecture", "status": "ok", "note": "Clarified training corpus scale: the existing per-stage sample counts are pilot schema-shakedown gates for exposing schema, boundary, leakage, validator, and workflow problems, not robust training targets. Review-assistant and runtime targets require much larger corpora, held-out evaluation, negative coverage, and failure audits; pilot data may be invalidated by schema fixes."} diff --git a/state/map.yaml b/state/map.yaml index a07546c..775dd55 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -151,6 +151,7 @@ accepted_design: - An Aetheria-tuned responder should learn lore, tone, factional preconceptions, species affordances, and institutional pressure from strong source-grounded responses spread throughout the timeline, but it still acts only inside Ghostlight's character-local context, deterministic gates, and mutation authority. - The coordinator is a trainable artifact: it glues scenes together, chooses next beats and acting agents, invokes specialist machinery, carries unresolved hooks, emits reviewed glue prose, and proposes world-state changes without bypassing deterministic gates. - The concrete training plan enumerates the trainable stages and their model families: generative decoder LLMs for coordinator/responder/soft structured outputs, classifiers or cross-encoders for appraisal/evaluation/relationship movement, retrieval models for memory and lore selection, and deterministic code for gates. + - Training-plan sample counts are pilot schema-shakedown gates, not robust corpus targets; pilot data may be invalidated by schema fixes, while review-assistant and runtime targets are expected to be roughly an order of magnitude larger with held-out evaluation and failure audits. - The timeline training plan now requires future-branch Elysium examples so the responder, coordinator, appraiser, and decision models learn post-Rupture concepts such as Aether, pseudospace, temporal nonlinearity, spirits, necrotech, mutable bodies, altered substrates, and weird institutions under branch-lineage labels. - Elysium-born factions and institutions can emerge as pressure-born attractors, authored/fated events, quest injections, or branch-local developments; technology discovery order and quest prerequisites are runtime constraints, not flavor text. - Future exploration should discover the Elysium technology landscape by simulating capable major-player decisions, while emitting manifests for branch-local innovations and pre-Elysium starting tech asymmetries across factions. @@ -161,5 +162,5 @@ accepted_design: - Ghostlight's technology pipeline should function as a content refinery: ground ideas in lore and mechanics, decompose them into parts and supply chains, map them to Aetheria-Economy blueprint classes, define performance and instance variation, preserve review labels, and produce database-shaped candidates instead of spreadsheet sludge. current_status: - summary: Ghostlight now has the Epiphany-style persistence spine, the first architecture payload, the v0 canonical agent-state schema seam, prompt projection docs, immediate schema fixes from evaluation, first reviewed Cat/Oz projection examples, a revised Aetheria lore-grounding architecture, a lore grounding digest schema, the first draft Cold Wake historical grounding digest, a Cold Wake story-lab fixture, projection controls, validated Qwen capture receipts, a complete Cold Wake short story assembled from accepted projection outputs, a hand-authored Ink-backed Cold Wake branching scene, the first Qwen-generated Ink draft materialized from local awareness with reviewed training annotations, the first reviewed sequential branch mutation that updates both Maer and Sella after a sanctuary intake beat, a v2 sequential Qwen capture proving the symmetrical turn shape produces better action behavior, a v5 sequential Qwen capture proving local thinking-plus-tools invocation works for the scene runner, the first projected-local-context seam that routes sequential prompts through character-local operating prose instead of raw state variables, tightened receipt handling for projector-routed Qwen failures, a default no-thinking policy for strict sequential tool calls, a v10 no-thinking projector-routed capture accepted as draft, the first materialized projector-routed sequential Ink scene with reviewed mutation training data, a source-checked embodiment/interface projection surface for Navigator scenes, a formal source-first Aetheria generation doctrine, v11-v17 source-checked retry receipts showing improved Maer embodiment plus remaining Sella response/prompt-leak blockers, a soft-model training artifact doctrine, a concrete training plan, sandboxed responder-data requirements, an explicit future-branch lane, Elysium branch event constraints, and a technology/item manifest plan for turning nebulous named tech into blueprint-grade components, assemblies, faction tech bases, supply chains, upgrade paths, gameplay hooks, provenance-hunting player fantasy, and content-refinery outputs mapped onto Aetheria-Economy's CultCache blueprint and instance model. + summary: Ghostlight now has the Epiphany-style persistence spine, the first architecture payload, the v0 canonical agent-state schema seam, prompt projection docs, immediate schema fixes from evaluation, first reviewed Cat/Oz projection examples, a revised Aetheria lore-grounding architecture, a lore grounding digest schema, the first draft Cold Wake historical grounding digest, a Cold Wake story-lab fixture, projection controls, validated Qwen capture receipts, a complete Cold Wake short story assembled from accepted projection outputs, a hand-authored Ink-backed Cold Wake branching scene, the first Qwen-generated Ink draft materialized from local awareness with reviewed training annotations, the first reviewed sequential branch mutation that updates both Maer and Sella after a sanctuary intake beat, a v2 sequential Qwen capture proving the symmetrical turn shape produces better action behavior, a v5 sequential Qwen capture proving local thinking-plus-tools invocation works for the scene runner, the first projected-local-context seam that routes sequential prompts through character-local operating prose instead of raw state variables, tightened receipt handling for projector-routed Qwen failures, a default no-thinking policy for strict sequential tool calls, a v10 no-thinking projector-routed capture accepted as draft, the first materialized projector-routed sequential Ink scene with reviewed mutation training data, a source-checked embodiment/interface projection surface for Navigator scenes, a formal source-first Aetheria generation doctrine, v11-v17 source-checked retry receipts showing improved Maer embodiment plus remaining Sella response/prompt-leak blockers, a soft-model training artifact doctrine, a concrete training plan with pilot schema-shakedown gates separated from robust corpus targets, sandboxed responder-data requirements, an explicit future-branch lane, Elysium branch event constraints, and a technology/item manifest plan for turning nebulous named tech into blueprint-grade components, assemblies, faction tech bases, supply chains, upgrade paths, gameplay hooks, provenance-hunting player fantasy, and content-refinery outputs mapped onto Aetheria-Economy's CultCache blueprint and instance model. next_action: Define the coordinator artifact schema so future story passes capture next-beat choice, glue prose, world-state refs, branch flags, unresolved hooks, and machinery invocations as trainable data, then return to the Sella prompt/rendering fix.