Run elaboration sessions concurrently over a per-sweep claim set (L1 Cut 4)

Rulings L1-Q4 (revised) and L1-Q9 (A).

Library (elaboration.rs):
- The sequential per-jurisdiction loop and step(jurisdiction) are deleted.
  ElaborationOutcome loses Clean and Inactive, which nothing produces now.
- sweep(permits) takes one snapshot and returns Ok unless the world is
  Active. It reads the store's in-flight listing once, builds the claim set,
  and spawns one step_session per entry into a JoinSet. Each session holds a
  permit from the caller's pool for its whole step.
- Every spawned session finishes before sweep returns. A quarantine-class
  error is returned after that, and no session starts after one has returned
  such an error. The sequential loop's early return also started no further
  session. Any other error is logged at debug.
- demand_entries is the claim set. For each swept jurisdiction it takes
  every covered boundary in derivation order, then the deficit if nonzero.
  Each entry is matched against the listing by world, jurisdiction and
  answer before anything is drawn. A match adopts the row's session and
  command id; if several match, the oldest is adopted and the rest are
  logged as orphans. An unmatched entry derives its id and draws its lens
  (fresh_session). An answer is claimed once, and so is a command id.
- step_session(command_id, session, snapshot) is the old step body. Its
  lookup and Cut 3's adoption rule are unchanged.
- step_in(jurisdiction), test-only, steps the first entry of one
  jurisdiction's claim set, so the Cut 3 tests keep reading one session's
  outcome.

Store: the listing now returns only ElaboratorInFlight rows. A committed
session ends in ReadyToSubmit, so listing that stage adopted a committed
deficit row, and every later sweep resubmitted it (AlreadyApplied) instead
of opening the next deficit session. That departs from the map's listing
contract.

Dungeon: elaborate_world and the ignored road test pass
state.elaboration_permits to sweep. The driver doc states that sweeps never
overlap.

Tests: 12 library tests.
- Concurrency: distinct_entries_run_at_once_under_the_ceiling,
  the_sweep_never_exceeds_its_ceiling, a_sweep_holds_a_permit_while_it_infers,
  one_answer_is_one_session_per_sweep.
- Claim set: a_boundary_under_nested_roots_is_claimed_once,
  a_deficit_entry_is_one_per_jurisdiction_per_ancestry.
- Errors: a_retryable_fault_ends_only_its_own_session,
  a_quarantine_error_is_returned_after_every_session_finishes.
- Rediscovery: a_stranded_deficit_session_is_adopted_with_its_refusal_intact,
  a_boundary_session_is_repaired_after_a_commit_and_a_tick,
  the_store_gains_no_orphan_per_commit, a_row_for_another_answer_is_not_adopted.

Dungeon adds saturating_one_pool_never_delays_the_other. The dead-end
fixture can now declare its commons (or commons and a ward) as a
jurisdiction root, with a subject that holds a goal, because a sweep visits
only jurisdictions that have a deficit row. The Cut 3 tests call step_in.

Docs: ghostlight-world-ontology.md states the concurrent sweep as the
shipped rule.

Counts: library 456 + 1 ignored, Dungeon 53 + 1 ignored. This commit
builds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Meta Vens 2026-09-16 17:44:04 +02:00
commit b3963f911d
4 changed files with 1388 additions and 135 deletions

View file

@ -1829,8 +1829,9 @@ const ELABORATION_SWEEP_INTERVAL: Duration = Duration::from_secs(300);
/// The authoring lane's only driver. It runs when the cognition organ opened at
/// all, which is the config gate the runtime already has: no mode flag joins it.
/// One sequential sweep per wake, because a boundary binds to its own digest and
/// the loops are logically independent without being separate tasks.
/// Each sweep runs its sessions under the elaboration pool and never touches the
/// simulation pool. Sweeps never overlap, because this loop awaits one before it
/// ticks the next, so one sweep's claim set is the only set of sessions alive.
async fn elaborate_world(state: AppState) {
let mut interval = tokio::time::interval(ELABORATION_SWEEP_INTERVAL);
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
@ -1845,7 +1846,7 @@ async fn elaborate_world(state: AppState) {
else {
continue;
};
if let Err(error) = runner.sweep().await {
if let Err(error) = runner.sweep(state.elaboration_permits.clone()).await {
tracing::debug!(%error, "elaboration sweep did not complete");
}
}
@ -3338,6 +3339,80 @@ mod tests {
);
}
/// Ruling L1-Q4: neither lane starves the other. With every elaboration
/// permit held by the test, a cover tick still runs both of its cells; with
/// every simulation permit held, an elaboration sweep still reaches the
/// provider. The timeouts only turn a lane blocked on the other lane's pool
/// into a failure instead of a hang; no ordering rests on them.
#[tokio::test]
async fn saturating_one_pool_never_delays_the_other() {
let fixture = fixture().await;
two_cell_world(
&fixture.state,
&fixture.cookie,
BTreeMap::from([(SubjectKind::Person, 4)]),
vec![CreateJurisdictionIntent {
handle: "sere".into(),
label: "The Low Sere".into(),
permille: 1000,
}],
true,
)
.await;
let port = Arc::new(CountingInferencePort::new());
let mut state = fixture.state.clone();
state.controllers = Some(Arc::new(
ControllerRunner::open(
state.world.clone(),
port.clone(),
Arc::new(AlwaysFreshWorkStore),
test_controller_models(),
)
.expect("the fixture ports open"),
));
let snapshot = state.world.snapshot().await.unwrap();
assert_eq!(snapshot.phase, WorldPhase::Active);
assert!(
snapshot.scale_deficit.iter().any(|row| row.deficit > 0),
"the fixture gives the elaborator nothing to answer"
);
let elaboration = state
.elaboration_permits
.clone()
.try_acquire_many_owned(u32::try_from(TEST_ELABORATION_CEILING).expect("a small pool"))
.expect("the elaboration pool is free");
tokio::time::timeout(Duration::from_secs(30), run_cover_tick(&state))
.await
.expect("the cover tick waited on the elaboration pool");
assert!(
port.calls.load(Ordering::SeqCst) >= 2,
"both singleton cells should have reached the port"
);
drop(elaboration);
let before = port.calls.load(Ordering::SeqCst);
let simulation = state
.controller_permits
.clone()
.try_acquire_many_owned(u32::try_from(TEST_CONTROLLER_CONCURRENCY).expect("a small pool"))
.expect("the simulation pool is free");
let runner = state.controllers.clone().expect("the runner");
tokio::time::timeout(
Duration::from_secs(30),
runner.elaborator().sweep(state.elaboration_permits.clone()),
)
.await
.expect("the sweep waited on the simulation pool")
.expect("the sweep completed");
assert!(
port.calls.load(Ordering::SeqCst) > before,
"no elaboration inference completed"
);
drop(simulation);
}
/// Raises `ControllerError::requires_quarantine` on every call. Used to
/// prove the tick driver's quarantine edge rather than any cognition
/// outcome.
@ -3972,7 +4047,10 @@ mod tests {
)
.await;
let cover = state.cover.lock().await.clone();
let elaboration = runner.elaborator().sweep().await;
let elaboration = runner
.elaborator()
.sweep(state.elaboration_permits.clone())
.await;
let after = state.world.snapshot().await.unwrap();
line(format!(
"tick {tick} took={:?} revision {before}->{} now={:?} cover={cover:?} quarantined={} elaboration={elaboration:?} boundaries={} deficit_rows={} subjects={}",

File diff suppressed because it is too large Load diff

View file

@ -35,7 +35,10 @@ use serde_json::Value;
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, BTreeSet};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use thiserror::Error;
use tokio::sync::Semaphore;
use tokio::task::JoinSet;
#[cfg(test)]
use uuid::Uuid;
@ -346,15 +349,9 @@ pub(super) enum ElaborationLoopEvaluation {
Complete { capture: ElaborationCapture },
}
/// What one `step` did, for the driver.
/// What one session's step did.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(super) enum ElaborationOutcome {
/// No boundary and no deficit in this jurisdiction: the terminating
/// condition.
Clean,
/// The world is not Active, so there is nothing to elaborate: seed
/// admission is the owner's lane.
Inactive,
Committed,
/// The kernel returned a complete mismatch set; the next round repairs it.
Rejected,
@ -364,8 +361,9 @@ pub(super) enum ElaborationOutcome {
NoPatch,
}
/// One jurisdiction's authoring loop. Holds no state between steps; every field
/// is a port or an identity.
/// The authoring lane's runner. Holds no state between sweeps; every field is a
/// port or an identity, so a clone is the same runner.
#[derive(Clone)]
pub struct ElaborationRunner {
mailbox: ElaborationPort,
inference: Arc<dyn InferencePort>,
@ -391,26 +389,22 @@ impl ElaborationRunner {
}
}
/// One answer, start to finish. Session identity is derived, so a crashed
/// loop finds its row again: a boundary that survives a repair keeps its
/// digest and therefore its command id, and a boundary whose digest moved
/// is a different answer with a different id.
pub(super) async fn step(
/// One entry of a sweep's claim set, start to finish, under the sweep's
/// snapshot. The entry is either a fresh session under its derived id or
/// an in-flight row the sweep adopted, under that row's id. The store row
/// found by that id still decides the session: a boundary that survives a
/// repair keeps its digest and therefore its command id, and a boundary
/// whose digest moved is a different answer with a different id.
pub(super) async fn step_session(
&self,
jurisdiction: JurisdictionKey,
command_id: CommandId,
fresh: ElaboratorSession,
snapshot: &WorldSnapshot,
) -> Result<ElaborationOutcome, ControllerError> {
let snapshot = self.mailbox.snapshot().await.map_err(snapshot_error)?;
if snapshot.phase != WorldPhase::Active {
return Ok(ElaborationOutcome::Inactive);
}
let Some((command_id, fresh)) = select_answer(&snapshot, jurisdiction)? else {
return Ok(ElaborationOutcome::Clean);
};
let receipts = self
.evidence
.retrieve(&EvidenceQuery {
referents: answer_referents(&snapshot, &fresh.answer),
referents: answer_referents(snapshot, &fresh.answer),
})
.await
.map_err(|error| ControllerError::WorkPersistence(error.to_string()))?;
@ -455,7 +449,7 @@ impl ElaborationRunner {
},
) => (agent_prompt, refusals, completed),
None => (
build_prompt(&snapshot, &session, &receipts),
build_prompt(snapshot, &session, &receipts),
Vec::new(),
Vec::new(),
),
@ -634,35 +628,92 @@ impl ElaborationRunner {
.await
}
/// One sweep over every jurisdiction the world's scale intent names, then
/// the uncovered residual. Sequential: a boundary binds to its own digest,
/// so the loops' logical independence is preserved without eight tasks
/// against one connector and a capacity-32 mailbox.
pub async fn sweep(&self) -> Result<(), ControllerError> {
/// One sweep: one snapshot, one read of the store's in-flight sessions, one
/// claim set, and one session per entry. One permit per session for its
/// whole step, from the ceiling the caller owns. The library holds no
/// concurrency number of its own.
///
/// The claim set is the only set of sessions alive only while one sweep
/// runs at a time. The caller's loop awaits each sweep; nothing here guards
/// against overlap.
///
/// Every spawned session finishes before this returns. A quarantine-class
/// error is returned after that, and no session starts after one has
/// returned such an error; any other error ends only its own session.
pub async fn sweep(&self, permits: Arc<Semaphore>) -> Result<(), ControllerError> {
let snapshot = self.mailbox.snapshot().await.map_err(snapshot_error)?;
let mut jurisdictions: Vec<JurisdictionKey> = snapshot
.scale_deficit
.iter()
.map(|row| row.jurisdiction)
.collect();
jurisdictions.dedup();
if !jurisdictions.contains(&JurisdictionKey::Uncovered) {
jurisdictions.push(JurisdictionKey::Uncovered);
if snapshot.phase != WorldPhase::Active {
return Ok(());
}
for jurisdiction in jurisdictions {
// The stop condition is "no open boundary and no deficit", plus the
// only other fixed point: a pass over this answer that admits
// nothing. Without it a boundary the model cannot answer becomes a
// hot spin against a paid endpoint.
match self.step(jurisdiction).await {
Ok(_) => {}
Err(error) if error.requires_quarantine() => return Err(error),
Err(error) => {
let in_flight = self.work.elaboration_in_flight().await?;
let entries = demand_entries(&snapshot, &sweep_jurisdictions(&snapshot), &in_flight)?;
let snapshot = Arc::new(snapshot);
// Raised by a session before it releases its permit, so the sweep sees
// it before it starts the session that permit admits.
let stopped = Arc::new(AtomicBool::new(false));
let mut running = JoinSet::new();
for (command_id, session) in entries {
if stopped.load(Ordering::SeqCst) {
break;
}
let Ok(permit) = permits.clone().acquire_owned().await else {
break;
};
if stopped.load(Ordering::SeqCst) {
break;
}
let runner = self.clone();
let snapshot = snapshot.clone();
let stopped = stopped.clone();
running.spawn(async move {
let outcome = runner.step_session(command_id, session, &snapshot).await;
if outcome
.as_ref()
.is_err_and(ControllerError::requires_quarantine)
{
stopped.store(true, Ordering::SeqCst);
}
drop(permit);
outcome
});
}
let mut quarantine = None;
while let Some(joined) = running.join_next().await {
match joined {
Ok(Ok(_)) => {}
Ok(Err(error)) if error.requires_quarantine() => {
quarantine.get_or_insert(error);
}
Ok(Err(error)) => {
tracing::debug!(%error, "elaboration step did not admit a patch");
}
Err(error) => std::panic::resume_unwind(error.into_panic()),
}
}
Ok(())
quarantine.map_or(Ok(()), Err)
}
/// The first entry of one jurisdiction's claim set, stepped: what a sweep
/// does for that entry, without the pool. Tests read the outcome here.
#[cfg(test)]
pub(super) async fn step_in(
&self,
jurisdiction: JurisdictionKey,
) -> Result<Option<ElaborationOutcome>, ControllerError> {
let snapshot = self.mailbox.snapshot().await.map_err(snapshot_error)?;
if snapshot.phase != WorldPhase::Active {
return Ok(None);
}
let in_flight = self.work.elaboration_in_flight().await?;
let Some((command_id, session)) = demand_entries(&snapshot, &[jurisdiction], &in_flight)?
.into_iter()
.next()
else {
return Ok(None);
};
self.step_session(command_id, session, &snapshot)
.await
.map(Some)
}
}
@ -670,60 +721,147 @@ fn snapshot_error(error: MailboxError) -> ControllerError {
ControllerError::Snapshot(error)
}
/// The oldest derived boundary in this jurisdiction, else its first nonzero
/// deficit row, else nothing. `snapshot.boundaries` is already ordered by the
/// kernel's derivation, so "oldest" is "first in that order".
/// Every jurisdiction the world's scale deficit names, then the uncovered
/// residual.
fn sweep_jurisdictions(snapshot: &WorldSnapshot) -> Vec<JurisdictionKey> {
let mut jurisdictions: Vec<JurisdictionKey> = snapshot
.scale_deficit
.iter()
.map(|row| row.jurisdiction)
.collect();
jurisdictions.dedup();
if !jurisdictions.contains(&JurisdictionKey::Uncovered) {
jurisdictions.push(JurisdictionKey::Uncovered);
}
jurisdictions
}
/// The claim set: for each jurisdiction in order, every derived boundary it
/// covers in the kernel's derivation order, then its deficit when the row is
/// nonzero. Scheduling state, rebuilt each sweep and never persisted.
///
/// Returns the session a new run would start, with its command id. The id is
/// derived from the answer alone; the lens is drawn from that id and the
/// world's current weights, and the lens's text is authored here. When the
/// store already holds this answer's row, `step` discards this session and
/// reads the recorded one.
fn select_answer(
/// Rediscovery reads state. Before anything is drawn, each entry is matched
/// against the store's in-flight sessions by world, jurisdiction and answer;
/// ancestry, lens and text are not compared. A match is the entry: its session
/// and its command id are adopted and nothing is drawn. Only an unmatched
/// entry derives an id and draws a lens.
///
/// A boundary under nested roots is covered by each of them, so an answer
/// already claimed is not claimed again, and no command id is claimed twice.
/// The stop condition is "no open boundary and no deficit", plus the one other
/// fixed point: a `NoPatch` session, which the store returns by id and which
/// runs no inference.
fn demand_entries(
snapshot: &WorldSnapshot,
jurisdictions: &[JurisdictionKey],
in_flight: &[(CommandId, ElaboratorSession)],
) -> Result<Vec<(CommandId, ElaboratorSession)>, ControllerError> {
let ancestry = snapshot.last_commit_digest.clone().unwrap_or_default();
let mut claimed_ids = BTreeSet::new();
let mut entries: Vec<(CommandId, ElaboratorSession)> = Vec::new();
for &jurisdiction in jurisdictions {
for (answer, answer_digest) in answers_in(snapshot, jurisdiction, &ancestry)? {
if entries.iter().any(|(_, claimed)| claimed.answer == answer) {
continue;
}
let entry = match adopt(snapshot.world_id, jurisdiction, &answer, in_flight) {
Some(row) => row,
None => fresh_session(snapshot, jurisdiction, answer, answer_digest, &ancestry)?,
};
if claimed_ids.insert(entry.0) {
entries.push(entry);
}
}
}
Ok(entries)
}
/// The in-flight row that answers this entry, if any. The listing is least
/// recently stored first, so the first match is the oldest; any other is an
/// orphan, left as it is and named.
fn adopt(
world_id: WorldId,
jurisdiction: JurisdictionKey,
answer: &PatchAnswer,
in_flight: &[(CommandId, ElaboratorSession)],
) -> Option<(CommandId, ElaboratorSession)> {
let mut matches = in_flight.iter().filter(|(_, session)| {
session.world_id == world_id
&& session.jurisdiction == jurisdiction
&& session.answer == *answer
});
let adopted = matches.next()?.clone();
for (orphan, _) in matches {
tracing::warn!(
adopted = ?adopted.0,
orphan = ?orphan,
"an in-flight elaboration row answers an entry another row already answers"
);
}
Some(adopted)
}
/// The answers one jurisdiction holds, in the kernel's derivation order, each
/// with its digest.
fn answers_in(
snapshot: &WorldSnapshot,
jurisdiction: JurisdictionKey,
) -> Result<Option<(CommandId, ElaboratorSession)>, ControllerError> {
let ancestry = snapshot.last_commit_digest.clone().unwrap_or_default();
let (answer, answer_digest) = if let Some(boundary) = snapshot
ancestry: &str,
) -> Result<Vec<(PatchAnswer, BoundaryDigest)>, ControllerError> {
let mut answers: Vec<(PatchAnswer, BoundaryDigest)> = snapshot
.boundaries
.iter()
.find(|boundary| boundary_in(snapshot, jurisdiction, boundary))
{
(
PatchAnswer::Boundary(boundary.clone()),
boundary_digest(boundary),
)
} else {
let deficit = snapshot
.scale_deficit
.iter()
.any(|row| row.jurisdiction == jurisdiction && row.deficit > 0);
if !deficit {
return Ok(None);
}
.filter(|boundary| boundary_in(snapshot, jurisdiction, boundary))
.map(|boundary| {
(
PatchAnswer::Boundary(boundary.clone()),
boundary_digest(boundary),
)
})
.collect();
let deficit = snapshot
.scale_deficit
.iter()
.any(|row| row.jurisdiction == jurisdiction && row.deficit > 0);
if deficit {
// A deficit mixes the ancestry, so each admitted commit opens a fresh
// session and the loop advances instead of resubmitting one id forever.
let digest = digest_of(&(snapshot.world_id, jurisdiction, "deficit", &ancestry))?;
(
let digest = digest_of(&(snapshot.world_id, jurisdiction, "deficit", ancestry))?;
answers.push((
PatchAnswer::Deficit(jurisdiction),
BoundaryDigest::from_digest(digest),
)
};
));
}
Ok(answers)
}
/// The session a new run would start, with its command id. The id is derived
/// from the answer alone; the lens is drawn from that id and the world's
/// current weights, and the lens's text is authored here. When the store
/// already holds a row under this id, `step_session` discards this session and
/// reads the recorded one.
fn fresh_session(
snapshot: &WorldSnapshot,
jurisdiction: JurisdictionKey,
answer: PatchAnswer,
answer_digest: BoundaryDigest,
ancestry: &str,
) -> Result<(CommandId, ElaboratorSession), ControllerError> {
let command_id = session_command_id(snapshot.world_id, jurisdiction, &answer_digest)?;
let lens = lens::draw(&snapshot.lens_weights, snapshot.world_id, command_id)
.map_err(|error| ControllerError::Serialization(error.to_string()))?;
Ok(Some((
Ok((
command_id,
ElaboratorSession {
world_id: snapshot.world_id,
jurisdiction,
answer,
answer_digest,
ancestry,
ancestry: ancestry.to_owned(),
lens,
instructions: lens.instructions(),
},
)))
))
}
/// Selection reads the same covering the kernel's authority check does: a

View file

@ -664,10 +664,13 @@ An elaborator is one `OperationalAgent` loop. Two separate things shape a
session. Its jurisdiction is authority: the place subtree it may write in,
carried as `SystemCapability::Elaborator { jurisdiction }`. Its lens decides
what kind of structure the session reaches for. A lens never enters the caller
identity, never narrows or widens admission, and never fills a quota. Target:
sessions draw a lens by the world's lens weights (below, "Elaboration lenses
and detail rules"). Shipped: the sweep runs one sequential unlensed session per
jurisdiction. The tool catalog is
identity, never narrows or widens admission, and never fills a quota.
Sessions run concurrently over distinct demand entries (every open boundary
and one deficit per jurisdiction), each drawing its lens by the world's weights
(below, "Elaboration lenses and detail rules") and recording it, each holding
one permit of the elaboration ceiling. A sweep first reads the store's
in-flight sessions and resumes any that answer a current demand entry, and
sweeps run one at a time. The tool catalog is
generated from this document's operation set and declaration kinds, plus
`record_gap` and `submit`. It cannot contain an operation the reducer does not
own, because it is derived from the reducer's vocabulary.