diff --git a/notes/fresh-workspace-handoff.md b/notes/fresh-workspace-handoff.md index 3039b304..b2d73cf1 100644 --- a/notes/fresh-workspace-handoff.md +++ b/notes/fresh-workspace-handoff.md @@ -52,6 +52,7 @@ remembers doctrine; the branch remembers the blade. - Thread resume/fork orchestration now lives in `codex_message_processor/thread_resume_routes.rs`; it owns resume from history/rollout, running-thread resume response replay, persisted resume metadata hydration, fork response construction, and the shared thread-title attachment helper. The root processor is about 4,336 lines. - Thread start route handling now lives in `codex_message_processor/thread_start_routes.rs`; the route shell delegates to the root lifecycle `thread_start_task`. The root processor is about 4,250 lines. - Turn control routes now live in `codex_message_processor/turn_routes.rs`: turn start, thread item injection, app-server client info setting, turn steer, and turn interrupt. The root processor is about 3,803 lines. +- Listener/subscription lifecycle now lives in `codex_message_processor/listener_lifecycle.rs`: connection attach/detach, listener startup, idle unload, unsubscribe, teardown, and archive preparation. The root processor is about 3,275 lines. - The May 2026 foundation migration in `notes/epiphany-ideal-architecture-rebuild-plan.md` is complete enough to defend. Read `notes/epiphany-architectural-teardown.md` before touching Epiphany control-plane code, but do not treat it as an open-ended excuse to keep shaving the altar. The rebuild moved scene projection, freshness derivation, pressure policy, targeted context shards, bounded graph traversal, job/progress view derivation, planning view derivation, reorientation resume/regather verdict policy, pure CRRC recommendation policy, fixed-lane coordinator decision policy, role-board projection policy, role/reorient result interpretation, role self-persistence review policy, and role/reorient acceptance bundle policy into `epiphany-core`; typed acceptance receipts replaced summary-string identity for live accept paths, runtime links are dual-written on launch, and result read-back now prefers runtime links. - Read `notes/epiphany-ideal-architecture-rebuild-plan.md` immediately after the teardown. It defines the smallest coherent replacement architecture: durable `EpiphanyState`, runtime-spine-owned `RuntimeState`, separate role memory/heartbeat state, `epiphany-core` as policy owner, app-server as adapter, Aquarium as reflector, typed acceptance receipts, and derived view lenses instead of a protocol verb zoo. - Do not continue Aquarium UI, bridge, Face, or dogfood expansion until the teardown has a source-grounded cleanup slice plan. Epiphany is the foundation; patches on patches are not a purification rite, they are how the altar becomes load-bearing garbage. diff --git a/state/ledgers.msgpack b/state/ledgers.msgpack index d0c8962f..0d53fc7c 100644 Binary files a/state/ledgers.msgpack and b/state/ledgers.msgpack differ diff --git a/state/map.yaml b/state/map.yaml index bb3aecbc..0d28fedd 100644 --- a/state/map.yaml +++ b/state/map.yaml @@ -286,9 +286,10 @@ rejected_paths: current_status: summary: The Codex starvation pass has five typing receipts, fifty-two processor/foundation cuts, a real physical extraction in `epiphany-codex-bridge`, a root `epiphany-state-model` crate for durable Epiphany state, a source-grounded Codex auth spine inventory, the native `epiphany-openai-adapter` document boundary, a buildable Codex auth/model wrapper in `epiphany-openai-codex-spine`, the first native runtime caller in `epiphany-openai-runtime`, native heartbeat launch planning/job opening in `epiphany-core`, native state-update contract/validation/application in `epiphany-core`, native exact retrieval path matching in `epiphany-core`, and the route-facing mutation/view/coordinator projection services in `epiphany-codex-bridge`. `codex-protocol` no longer defines the durable `Epiphany*` state structs; it re-exports the native model for legacy rollout/API compatibility while `epiphany-core` imports the native model directly. `epiphany-core` now has no direct vendored Codex dependency: pressure derivation accepts native `EpiphanyTokenUsageSnapshot`, Codex rollout replay moved to `codex-core::epiphany_rollout`, and exact retrieval search uses Epiphany-owned workspace walking/path scoring instead of `codex-file-search`. The bridge owns Codex-facing Epiphany adapters for scene, jobs, context/planning/graph query, retrieval/indexing projection, result projection, runtime-spine result adaptation, launch/prompt doctrine, mutation document mechanics, mutation service update/promote/role-accept/reorient-accept/role-launch/reorient-launch/job-launch/job-interrupt application, live-state projection, pressure/pre-compaction rendering, freshness/reorientation projection, CRRC/coordinator/role-board projection, watcher invalidation adaptation, full `thread/epiphany/view` response construction, freshness/context/graph-query response construction, role/reorient result response construction, distill/propose patch-response construction, coordinator status/view construction, and acceptance/evidence signal helpers. Vendored app-server no longer retains root Epiphany modules; `codex_message_processor.rs` still contains route dispatch, JSON-RPC response shaping, watcher registration/snapshotting calls, and notification emission, but its root no longer imports the evacuated Epiphany policy/protocol registry or owns the watcher invalidation manager. Codex product surfaces for apps, skills, plugins, marketplace, and external agent config import are cut from the live app-server/protocol path: list routes return empty lists only where compatibility shells still exist, mutation/detail routes report disabled where they survive, `externalAgentConfig/*` routes and import-completed notification are gone, helper/import modules and stale implementation-shape tests were deleted, runtime feature enablement no longer supports `apps` or `plugins`, config writes no longer emit plugin-toggle analytics, plugin/skill cache invalidation on config writes is gone, plugin startup warmups and app-list refreshes were removed, and app-server no longer directly depends on `codex-plugin`, `codex-chatgpt`, or `codex-core-plugins`. `Config::to_mcp_config` now keeps user-declared MCP servers but no longer accepts `PluginsManager`, folds plugin MCP servers into the runtime, enables Codex apps, enables skill MCP dependency install, or advertises plugin capability summaries; `TurnContext::apps_enabled()` now returns false so model turns cannot awaken Codex Apps through ChatGPT auth or feature flags; session startup and per-turn context no longer load skills, turns receive an empty `SkillLoadOutcome`, internal `Op::ListSkills` is gone from `codex-protocol` / `codex-core` / TUI command routing, the Codex skill watcher has been collapsed to a no-op marker with no file watcher, skill-root registration, cache-clear task, or `SkillsUpdateAvailable` session listener, model turns no longer run skill mention collection, skill env-var prompts, skill MCP dependency installation, skill instruction injection, or implicit skill invocation analytics from shell tools, and the turn prompt/router path no longer loads plugin/app capability summaries, injects available-skill/available-plugin/plugin instruction fragments, parses plugin/app mentions, records app/plugin invocation telemetry, or filters MCP tools through Codex app connectors. Skill mutation/list/notification is gone from the app-server edge too: `skills/list`, `skills/config/write`, `skills/changed`, `Op::ListSkills`, `EventMsg::ListSkillsResponse`, `EventMsg::SkillsUpdateAvailable`, the app-server handlers, TUI slash/startup/list/manage-skill surfaces, generated schema entries, core skill-list suite, and fake live-reload/developer-message tests were deleted. Skill mention plumbing is also gone from the live TUI/protocol edge: no `UserInput::Skill` variant, app-server schema branch, TUI skill popup, `skills_all` cache, composer `set_skill_mentions`, skill mention insertion, `skill://` / `SKILL.md` history decoding, or stale skill popup snapshot remains. The `codex-core-skills` crate, core `skills` re-export shim, no-op `skills_watcher`, session `SkillsManager` service, and per-turn `TurnSkillsContext` are deleted; core/protocol no longer keep legacy `` or `` prompt compatibility, memory usage classification, snapshot normalization, or cleanup paths. Remaining product-shaped skill residue is also cut from memory templates, app-server docs/tests/warnings, TUI tooltips/placeholders/warnings/comments, app-server v1 docs, agent-tool schema text, stale `UserInput::Skill` test arms, and obsolete feature flags/schema entries. App-instruction prompt compatibility is cut too: no `APPS_INSTRUCTIONS` protocol constants, no `` snapshot canonicalization, and MCP fixture tests now use neutral `demo_mcp` / `calendar_mcp` names instead of `codex_apps` / `/api/codex/apps` branding. The TUI app connector product path is also deleted: no `/apps` slash command, app-link view, connector AppEvent variants, app connector cache/popup/refresh loop, app mention insertion, `app://` mention codec support, stale app-popup tests, or TUI browser-opening shim remains. The `app/list` route and `app/list/updated` notification are gone from app-server protocol, generated schemas, app-server dispatch, test helpers, and docs; `AppInfo`/app-list metadata schema structs are deleted; the orphaned `codex-connectors` crate and `core::connectors` shim are deleted from the workspace; Codex app analytics no longer tracks `codex_app_mentioned` / `codex_app_used` facts, events, reducer arms, client APIs, or dedupe cache state; `[apps.*]`, `Feature::Apps`, `include_apps_instructions`, app config/read schemas, app managed requirements, app prompt tests, and app file-MCP tests are gone; `Feature::Plugins`, `Feature::RemotePlugin`, `plugin://` structured mentions, plugin instruction tags, and stale plugin popup snapshots are gone too; `[plugins]` / `[marketplaces]`, `PluginConfig`, `MarketplaceConfig`, marketplace edit helpers, stale plugin/external-agent README route claims, and generated schema entries are removed from config; and the bundled `codex-skills` sample-skill crate plus embedded imagegen/openai-docs/plugin-creator/skill-creator/skill-installer payloads are deleted from the workspace. The vendored `vendor/codex/sdk` subtree is deleted as non-spine app-server client/release scaffolding, including stale generated Python SDK route types for removed product surfaces; the root pnpm workspace no longer lists `sdk/typescript`. Live MCP tool approval is named `McpToolApproval`, not `AppToolApproval`, and remains under MCP server config; live structured user-input mentions are MCP-shaped only. `codex_message_processor.rs` is about 9,482 lines; root `message_processor.rs` is about 1,292 lines after the external-agent/config-product cuts; `epiphany_read_routes.rs` is about 579 lines after the full view and read-response extractions; `epiphany_mutation_routes.rs` is about 694 lines after route-load/notification helper cuts and product-surface starvation; the old vendored `epiphany_state_helpers.rs` rollout-state loader is deleted and its remaining function lives in `epiphany-codex-bridge::state`; and `codex_thread.rs` is about 1,425 lines after state-update law moved native. `epiphany-openai-adapter` defines CultCache `DatabaseEntry` request/event/receipt/auth-status documents with no Codex app-server or generic JSON dependency. `epiphany-openai-codex-spine` owns the native OpenAI model transport path: it exposes `default_codex_home` and `auth_manager`, projects `epiphany-openai-auth-spine` `AuthManager` / `CodexAuth` into typed `EpiphanyOpenAiAdapterStatus`, builds a local serializable OpenAI Responses request body instead of Codex protocol request/item/config types, opens HTTP Responses streams through `codex-client`, parses Responses SSE frames into typed stream events and receipts, and no longer directly or transitively depends on Codex's Responses client, model-provider stack, `codex-protocol`, `codex-api`, `codex-model-provider`, or `codex-model-provider-info`; it intentionally reaches retained vendored `codex-login` only through `epiphany-openai-auth-spine`. Auth identity, credential storage, and ChatGPT token refresh remain owned by the Codex-compatible auth organ; external bearer command auth and agent identity remain retained Codex auth capabilities to expose only if a typed Epiphany surface needs them. `epiphany-openai-runtime` no longer depends directly on `codex-login`; it writes OpenAI status/request/stream-event/receipt records plus runtime sessions/jobs/results/events into `epiphany-core::runtime_spine_cache`, can emit the assistant text artifact for legacy result consumers, is now the default executor for startup-only repo birth specialists, and now has `run-worker` to load a durable `EpiphanyRuntimeWorkerLaunchRequest`, call the typed OpenAI runtime through the spine, persist typed `EpiphanyRuntimeRoleWorkerResult` / `EpiphanyRuntimeReorientWorkerResult` documents, and complete the original heartbeat/specialist runtime job without Codex worker execution; `codex exec` survives only as explicit fallback in repo-birth startup. `epiphany-mvp-coordinator` now invokes `epiphany-openai-runtime run-worker` for launched runtime job ids and no longer creates a second coordinator-owned shadow runtime job/result as a compensator. `epiphany-core::plan_runtime_spine_heartbeat_launch` validates launch contracts/conflicts and projects durable job bindings/runtime links; `epiphany-core::open_runtime_spine_heartbeat_job` owns the initialize-runtime/ensure-session/create-job sequence and now persists `EpiphanyRuntimeWorkerLaunchRequest` CultCache records keyed by runtime job id, with indexed kind/contract fields plus MessagePack-encoded typed launch documents instead of discarding work intent after queue receipt creation; `roleResult` and `reorientResult` now require typed worker result documents; generic job summaries are lifecycle receipts, not reviewable findings; `epiphany-core::EpiphanyStateUpdate`, `epiphany_state_update_validation_errors`, and `apply_epiphany_state_update` own state-update document shape and mutation law while `codex-core` calls them for compatibility. Vendored `CodexThread::epiphany_launch_job` is now mostly revision check, persistence validation, launch request handoff, and rollout/session writeback for the compatibility route. Vendored `codex-core` no longer has `epiphany_distillation`, `epiphany_promotion`, `epiphany_proposal`, or `epiphany_retrieval` husk modules; it re-exports native surfaces from `epiphany-core` directly, and now owns only the real Codex rollout adapter because it is a host seam. `epiphany-runtime-spine` advertises OpenAI adapter status/model-request/stream-event/receipt CultNet schemas and mutation contracts, so the native caller now has a real storage route instead of only JSON-RPC. MCP itself is correctly JSON at its protocol edge; the target is typed Epiphany intent/result/receipt wrappers around MCP, not replacing MCP JSON. The current OpenAI request path no longer pulls `codex-api`; `codex-login` remains deliberately sealed behind `epiphany-openai-auth-spine`, and the next spine decision is another Epiphany-in-vendor evacuation surface unless a concrete typed auth-status/runtime need requires exposing external bearer or agent-identity auth. latest_cut: > - Turn control routes moved out of root `codex_message_processor.rs` into - `codex_message_processor/turn_routes.rs`. Turn start, thread item - injection, app-server client info setting, turn steer, and turn interrupt - are split from listener/load/rollout lifecycle; the root processor is about - 3,803 lines after the cut. + Listener/subscription lifecycle moved out of root + `codex_message_processor.rs` into + `codex_message_processor/listener_lifecycle.rs`. Connection attach/detach, + listener startup, idle unload, unsubscribe, teardown, and archive + preparation are split from dispatch/load/rollout helpers; the root + processor is about 3,275 lines after the cut. next_action: Continue the Codex starvation campaign at the next Epiphany-in-vendor evacuation surface; keep vendored Codex code as host routing/lifecycle glue only, and do not cut the retained `codex-login` auth organ unless replacing it with an explicitly permitted OpenAI path rather than a lookalike. diff --git a/vendor/codex/codex-rs/app-server/src/codex_message_processor.rs b/vendor/codex/codex-rs/app-server/src/codex_message_processor.rs index 6d35aa60..a4a64e5f 100644 --- a/vendor/codex/codex-rs/app-server/src/codex_message_processor.rs +++ b/vendor/codex/codex-rs/app-server/src/codex_message_processor.rs @@ -4,6 +4,7 @@ mod command_routes; mod epiphany_automation; mod epiphany_mutation_routes; mod epiphany_read_routes; +mod listener_lifecycle; mod mcp_routes; mod realtime_routes; mod review_routes; @@ -1533,67 +1534,6 @@ impl CodexMessageProcessor { } } - pub(crate) fn thread_created_receiver(&self) -> broadcast::Receiver { - self.thread_manager.subscribe_thread_created() - } - - pub(crate) async fn connection_initialized(&self, connection_id: ConnectionId) { - self.thread_state_manager - .connection_initialized(connection_id) - .await; - } - - pub(crate) async fn connection_closed(&self, connection_id: ConnectionId) { - self.command_exec_manager - .connection_closed(connection_id) - .await; - let thread_ids = self - .thread_state_manager - .remove_connection(connection_id) - .await; - - for thread_id in thread_ids { - if self.thread_manager.get_thread(thread_id).await.is_err() { - // Reconcile stale app-server bookkeeping when the thread has already been - // removed from the core manager. - self.finalize_thread_teardown(thread_id).await; - } - } - } - - pub(crate) fn subscribe_running_assistant_turn_count(&self) -> watch::Receiver { - self.thread_watch_manager.subscribe_running_turn_count() - } - - /// Best-effort: ensure initialized connections are subscribed to this thread. - pub(crate) async fn try_attach_thread_listener( - &self, - thread_id: ThreadId, - connection_ids: Vec, - ) { - if let Ok(thread) = self.thread_manager.get_thread(thread_id).await { - let config_snapshot = thread.config_snapshot().await; - let loaded_thread = - build_thread_from_snapshot(thread_id, &config_snapshot, thread.rollout_path()); - self.thread_watch_manager.upsert_thread(loaded_thread).await; - } - - for connection_id in connection_ids { - Self::log_listener_attach_result( - self.ensure_conversation_listener( - thread_id, - connection_id, - /*raw_events_enabled*/ false, - ApiVersion::V2, - ) - .await, - thread_id, - connection_id, - "thread", - ); - } - } - async fn get_thread_summary( &self, request_id: ConnectionRequestId, @@ -1804,474 +1744,6 @@ impl CodexMessageProcessor { self.outgoing.send_error(request_id, error).await; } - async fn wait_for_thread_shutdown(thread: &Arc) -> ThreadShutdownResult { - match tokio::time::timeout(Duration::from_secs(10), thread.shutdown_and_wait()).await { - Ok(Ok(())) => ThreadShutdownResult::Complete, - Ok(Err(_)) => ThreadShutdownResult::SubmitFailed, - Err(_) => ThreadShutdownResult::TimedOut, - } - } - - async fn finalize_thread_teardown(&self, thread_id: ThreadId) { - self.pending_thread_unloads.lock().await.remove(&thread_id); - self.outgoing - .cancel_requests_for_thread(thread_id, /*error*/ None) - .await; - self.thread_state_manager - .remove_thread_state(thread_id) - .await; - self.epiphany_invalidation_manager - .remove_thread(&thread_id.to_string()) - .await; - self.thread_watch_manager - .remove_thread(&thread_id.to_string()) - .await; - } - - async fn unload_thread_without_subscribers( - thread_manager: Arc, - outgoing: Arc, - pending_thread_unloads: Arc>>, - thread_state_manager: ThreadStateManager, - thread_watch_manager: ThreadWatchManager, - epiphany_invalidation_manager: EpiphanyInvalidationManager, - thread_id: ThreadId, - thread: Arc, - ) { - info!("thread {thread_id} has no subscribers and is idle; shutting down"); - - // Any pending app-server -> client requests for this thread can no longer be - // answered; cancel their callbacks before shutdown/unload. - outgoing - .cancel_requests_for_thread(thread_id, /*error*/ None) - .await; - thread_state_manager.remove_thread_state(thread_id).await; - epiphany_invalidation_manager - .remove_thread(&thread_id.to_string()) - .await; - - tokio::spawn(async move { - match Self::wait_for_thread_shutdown(&thread).await { - ThreadShutdownResult::Complete => { - if thread_manager.remove_thread(&thread_id).await.is_none() { - info!("thread {thread_id} was already removed before teardown finalized"); - epiphany_invalidation_manager - .remove_thread(&thread_id.to_string()) - .await; - thread_watch_manager - .remove_thread(&thread_id.to_string()) - .await; - pending_thread_unloads.lock().await.remove(&thread_id); - return; - } - epiphany_invalidation_manager - .remove_thread(&thread_id.to_string()) - .await; - thread_watch_manager - .remove_thread(&thread_id.to_string()) - .await; - let notification = ThreadClosedNotification { - thread_id: thread_id.to_string(), - }; - outgoing - .send_server_notification(ServerNotification::ThreadClosed(notification)) - .await; - pending_thread_unloads.lock().await.remove(&thread_id); - } - ThreadShutdownResult::SubmitFailed => { - pending_thread_unloads.lock().await.remove(&thread_id); - warn!("failed to submit Shutdown to thread {thread_id}"); - } - ThreadShutdownResult::TimedOut => { - pending_thread_unloads.lock().await.remove(&thread_id); - warn!("thread {thread_id} shutdown timed out; leaving thread loaded"); - } - } - }); - } - - async fn thread_unsubscribe( - &self, - request_id: ConnectionRequestId, - params: ThreadUnsubscribeParams, - ) { - let thread_id = match ThreadId::from_string(¶ms.thread_id) { - Ok(id) => id, - Err(err) => { - self.send_invalid_request_error(request_id, format!("invalid thread id: {err}")) - .await; - return; - } - }; - - if self.thread_manager.get_thread(thread_id).await.is_err() { - // Reconcile stale app-server bookkeeping when the thread has already been - // removed from the core manager. This keeps loaded-status/subscription state - // consistent with the source of truth before reporting NotLoaded. - self.finalize_thread_teardown(thread_id).await; - self.outgoing - .send_response( - request_id, - ThreadUnsubscribeResponse { - status: ThreadUnsubscribeStatus::NotLoaded, - }, - ) - .await; - return; - }; - - let was_subscribed = self - .thread_state_manager - .unsubscribe_connection_from_thread(thread_id, request_id.connection_id) - .await; - - let status = if was_subscribed { - ThreadUnsubscribeStatus::Unsubscribed - } else { - ThreadUnsubscribeStatus::NotSubscribed - }; - self.outgoing - .send_response(request_id, ThreadUnsubscribeResponse { status }) - .await; - } - - async fn prepare_thread_for_archive(&self, thread_id: ThreadId) { - // If the thread is active, request shutdown and wait briefly. - let removed_conversation = self.thread_manager.remove_thread(&thread_id).await; - if let Some(conversation) = removed_conversation { - info!("thread {thread_id} was active; shutting down"); - match Self::wait_for_thread_shutdown(&conversation).await { - ThreadShutdownResult::Complete => {} - ThreadShutdownResult::SubmitFailed => { - error!( - "failed to submit Shutdown to thread {thread_id}; proceeding with archive" - ); - } - ThreadShutdownResult::TimedOut => { - warn!("thread {thread_id} shutdown timed out; proceeding with archive"); - } - } - } - self.finalize_thread_teardown(thread_id).await; - } - - async fn ensure_conversation_listener( - &self, - conversation_id: ThreadId, - connection_id: ConnectionId, - raw_events_enabled: bool, - api_version: ApiVersion, - ) -> Result { - Self::ensure_conversation_listener_task( - ListenerTaskContext { - thread_manager: Arc::clone(&self.thread_manager), - thread_state_manager: self.thread_state_manager.clone(), - outgoing: Arc::clone(&self.outgoing), - pending_thread_unloads: Arc::clone(&self.pending_thread_unloads), - analytics_events_client: self.analytics_events_client.clone(), - general_analytics_enabled: self.config.features.enabled(Feature::GeneralAnalytics), - thread_watch_manager: self.thread_watch_manager.clone(), - epiphany_invalidation_manager: self.epiphany_invalidation_manager.clone(), - fallback_model_provider: self.config.model_provider_id.clone(), - codex_home: self.config.codex_home.to_path_buf(), - }, - conversation_id, - connection_id, - raw_events_enabled, - api_version, - ) - .await - } - - #[expect( - clippy::await_holding_invalid_type, - reason = "listener subscription must be serialized against pending thread unloads" - )] - async fn ensure_conversation_listener_task( - listener_task_context: ListenerTaskContext, - conversation_id: ThreadId, - connection_id: ConnectionId, - raw_events_enabled: bool, - api_version: ApiVersion, - ) -> Result { - let conversation = match listener_task_context - .thread_manager - .get_thread(conversation_id) - .await - { - Ok(conv) => conv, - Err(_) => { - return Err(JSONRPCErrorError { - code: INVALID_REQUEST_ERROR_CODE, - message: format!("thread not found: {conversation_id}"), - data: None, - }); - } - }; - let thread_state = { - let pending_thread_unloads = listener_task_context.pending_thread_unloads.lock().await; - if pending_thread_unloads.contains(&conversation_id) { - return Err(JSONRPCErrorError { - code: INVALID_REQUEST_ERROR_CODE, - message: format!( - "thread {conversation_id} is closing; retry after the thread is closed" - ), - data: None, - }); - } - let Some(thread_state) = listener_task_context - .thread_state_manager - .try_ensure_connection_subscribed( - conversation_id, - connection_id, - raw_events_enabled, - ) - .await - else { - return Ok(EnsureConversationListenerResult::ConnectionClosed); - }; - thread_state - }; - if let Err(error) = Self::ensure_listener_task_running_task( - listener_task_context.clone(), - conversation_id, - conversation, - thread_state, - api_version, - ) - .await - { - let _ = listener_task_context - .thread_state_manager - .unsubscribe_connection_from_thread(conversation_id, connection_id) - .await; - return Err(error); - } - Ok(EnsureConversationListenerResult::Attached) - } - - fn log_listener_attach_result( - result: Result, - thread_id: ThreadId, - connection_id: ConnectionId, - thread_kind: &'static str, - ) { - match result { - Ok(EnsureConversationListenerResult::Attached) => {} - Ok(EnsureConversationListenerResult::ConnectionClosed) => { - tracing::debug!( - thread_id = %thread_id, - connection_id = ?connection_id, - "skipping auto-attach for closed connection" - ); - } - Err(err) => { - tracing::warn!( - "failed to attach listener for {thread_kind} {thread_id}: {message}", - message = err.message - ); - } - } - } - - async fn ensure_listener_task_running( - &self, - conversation_id: ThreadId, - conversation: Arc, - thread_state: Arc>, - api_version: ApiVersion, - ) -> Result<(), JSONRPCErrorError> { - Self::ensure_listener_task_running_task( - ListenerTaskContext { - thread_manager: Arc::clone(&self.thread_manager), - thread_state_manager: self.thread_state_manager.clone(), - outgoing: Arc::clone(&self.outgoing), - pending_thread_unloads: Arc::clone(&self.pending_thread_unloads), - analytics_events_client: self.analytics_events_client.clone(), - general_analytics_enabled: self.config.features.enabled(Feature::GeneralAnalytics), - thread_watch_manager: self.thread_watch_manager.clone(), - epiphany_invalidation_manager: self.epiphany_invalidation_manager.clone(), - fallback_model_provider: self.config.model_provider_id.clone(), - codex_home: self.config.codex_home.to_path_buf(), - }, - conversation_id, - conversation, - thread_state, - api_version, - ) - .await - } - - async fn ensure_listener_task_running_task( - listener_task_context: ListenerTaskContext, - conversation_id: ThreadId, - conversation: Arc, - thread_state: Arc>, - api_version: ApiVersion, - ) -> Result<(), JSONRPCErrorError> { - let (cancel_tx, mut cancel_rx) = oneshot::channel(); - let Some(mut unloading_state) = UnloadingState::new( - &listener_task_context, - conversation_id, - THREAD_UNLOADING_DELAY, - ) - .await - else { - return Err(JSONRPCErrorError { - code: INVALID_REQUEST_ERROR_CODE, - message: format!( - "thread {conversation_id} is closing; retry after the thread is closed" - ), - data: None, - }); - }; - let (mut listener_command_rx, listener_generation) = { - let mut thread_state = thread_state.lock().await; - if thread_state.listener_matches(&conversation) { - return Ok(()); - } - thread_state.set_listener(cancel_tx, &conversation) - }; - let ListenerTaskContext { - outgoing, - thread_manager, - thread_state_manager, - pending_thread_unloads, - analytics_events_client: _, - general_analytics_enabled: _, - thread_watch_manager, - epiphany_invalidation_manager, - fallback_model_provider, - codex_home, - } = listener_task_context; - let outgoing_for_task = Arc::clone(&outgoing); - tokio::spawn(async move { - loop { - tokio::select! { - biased; - _ = &mut cancel_rx => { - // Listener was superseded or the thread is being torn down. - break; - } - listener_command = listener_command_rx.recv() => { - let Some(listener_command) = listener_command else { - break; - }; - handle_thread_listener_command( - conversation_id, - &conversation, - codex_home.as_path(), - &thread_state_manager, - &thread_state, - &thread_watch_manager, - &outgoing_for_task, - &pending_thread_unloads, - listener_command, - ) - .await; - } - event = conversation.next_event() => { - let event = match event { - Ok(event) => event, - Err(err) => { - tracing::warn!("thread.next_event() failed with: {err}"); - break; - } - }; - - // Track the event before emitting any typed - // translations so thread-local state such as raw event - // opt-in stays synchronized with the conversation. - let raw_events_enabled = { - let mut thread_state = thread_state.lock().await; - thread_state.track_current_turn_event(&event.msg); - thread_state.experimental_raw_events - }; - let subscribed_connection_ids = thread_state_manager - .subscribed_connection_ids(conversation_id) - .await; - let thread_outgoing = ThreadScopedOutgoingMessageSender::new( - outgoing_for_task.clone(), - subscribed_connection_ids, - conversation_id, - ); - - if let EventMsg::RawResponseItem(raw_response_item_event) = &event.msg - && !raw_events_enabled - { - maybe_emit_hook_prompt_item_completed( - api_version, - conversation_id, - &event.id, - &raw_response_item_event.item, - &thread_outgoing, - ) - .await; - continue; - } - - apply_bespoke_event_handling( - event.clone(), - conversation_id, - conversation.clone(), - thread_manager.clone(), - listener_task_context - .general_analytics_enabled - .then(|| listener_task_context.analytics_events_client.clone()), - thread_outgoing, - thread_state.clone(), - thread_watch_manager.clone(), - epiphany_invalidation_manager.clone(), - api_version, - fallback_model_provider.clone(), - codex_home.as_path(), - ) - .await; - } - unloading_watchers_open = unloading_state.wait_for_unloading_trigger() => { - if !unloading_watchers_open { - break; - } - if !unloading_state.should_unload_now() { - continue; - } - if matches!(conversation.agent_status().await, AgentStatus::Running) { - unloading_state.note_thread_activity_observed(); - continue; - } - { - let mut pending_thread_unloads = pending_thread_unloads.lock().await; - if pending_thread_unloads.contains(&conversation_id) { - continue; - } - if !unloading_state.should_unload_now() { - continue; - } - pending_thread_unloads.insert(conversation_id); - } - Self::unload_thread_without_subscribers( - thread_manager.clone(), - outgoing_for_task.clone(), - pending_thread_unloads.clone(), - thread_state_manager.clone(), - thread_watch_manager.clone(), - epiphany_invalidation_manager.clone(), - conversation_id, - conversation.clone(), - ) - .await; - break; - } - } - } - - let mut thread_state = thread_state.lock().await; - if thread_state.listener_generation == listener_generation { - thread_state.clear_listener(); - } - }); - Ok(()) - } async fn resolve_rollout_path( &self, conversation_id: ThreadId, diff --git a/vendor/codex/codex-rs/app-server/src/codex_message_processor/listener_lifecycle.rs b/vendor/codex/codex-rs/app-server/src/codex_message_processor/listener_lifecycle.rs new file mode 100644 index 00000000..29dea56d --- /dev/null +++ b/vendor/codex/codex-rs/app-server/src/codex_message_processor/listener_lifecycle.rs @@ -0,0 +1,535 @@ +use super::*; + +impl CodexMessageProcessor { + pub(crate) fn thread_created_receiver(&self) -> broadcast::Receiver { + self.thread_manager.subscribe_thread_created() + } + + pub(crate) async fn connection_initialized(&self, connection_id: ConnectionId) { + self.thread_state_manager + .connection_initialized(connection_id) + .await; + } + + pub(crate) async fn connection_closed(&self, connection_id: ConnectionId) { + self.command_exec_manager + .connection_closed(connection_id) + .await; + let thread_ids = self + .thread_state_manager + .remove_connection(connection_id) + .await; + + for thread_id in thread_ids { + if self.thread_manager.get_thread(thread_id).await.is_err() { + // Reconcile stale app-server bookkeeping when the thread has already been + // removed from the core manager. + self.finalize_thread_teardown(thread_id).await; + } + } + } + + pub(crate) fn subscribe_running_assistant_turn_count(&self) -> watch::Receiver { + self.thread_watch_manager.subscribe_running_turn_count() + } + + /// Best-effort: ensure initialized connections are subscribed to this thread. + pub(crate) async fn try_attach_thread_listener( + &self, + thread_id: ThreadId, + connection_ids: Vec, + ) { + if let Ok(thread) = self.thread_manager.get_thread(thread_id).await { + let config_snapshot = thread.config_snapshot().await; + let loaded_thread = + build_thread_from_snapshot(thread_id, &config_snapshot, thread.rollout_path()); + self.thread_watch_manager.upsert_thread(loaded_thread).await; + } + + for connection_id in connection_ids { + Self::log_listener_attach_result( + self.ensure_conversation_listener( + thread_id, + connection_id, + /*raw_events_enabled*/ false, + ApiVersion::V2, + ) + .await, + thread_id, + connection_id, + "thread", + ); + } + } + + pub(super) async fn wait_for_thread_shutdown( + thread: &Arc, + ) -> ThreadShutdownResult { + match tokio::time::timeout(Duration::from_secs(10), thread.shutdown_and_wait()).await { + Ok(Ok(())) => ThreadShutdownResult::Complete, + Ok(Err(_)) => ThreadShutdownResult::SubmitFailed, + Err(_) => ThreadShutdownResult::TimedOut, + } + } + + pub(super) async fn finalize_thread_teardown(&self, thread_id: ThreadId) { + self.pending_thread_unloads.lock().await.remove(&thread_id); + self.outgoing + .cancel_requests_for_thread(thread_id, /*error*/ None) + .await; + self.thread_state_manager + .remove_thread_state(thread_id) + .await; + self.epiphany_invalidation_manager + .remove_thread(&thread_id.to_string()) + .await; + self.thread_watch_manager + .remove_thread(&thread_id.to_string()) + .await; + } + + pub(super) async fn unload_thread_without_subscribers( + thread_manager: Arc, + outgoing: Arc, + pending_thread_unloads: Arc>>, + thread_state_manager: ThreadStateManager, + thread_watch_manager: ThreadWatchManager, + epiphany_invalidation_manager: EpiphanyInvalidationManager, + thread_id: ThreadId, + thread: Arc, + ) { + info!("thread {thread_id} has no subscribers and is idle; shutting down"); + + // Any pending app-server -> client requests for this thread can no longer be + // answered; cancel their callbacks before shutdown/unload. + outgoing + .cancel_requests_for_thread(thread_id, /*error*/ None) + .await; + thread_state_manager.remove_thread_state(thread_id).await; + epiphany_invalidation_manager + .remove_thread(&thread_id.to_string()) + .await; + + tokio::spawn(async move { + match Self::wait_for_thread_shutdown(&thread).await { + ThreadShutdownResult::Complete => { + if thread_manager.remove_thread(&thread_id).await.is_none() { + info!("thread {thread_id} was already removed before teardown finalized"); + epiphany_invalidation_manager + .remove_thread(&thread_id.to_string()) + .await; + thread_watch_manager + .remove_thread(&thread_id.to_string()) + .await; + pending_thread_unloads.lock().await.remove(&thread_id); + return; + } + epiphany_invalidation_manager + .remove_thread(&thread_id.to_string()) + .await; + thread_watch_manager + .remove_thread(&thread_id.to_string()) + .await; + let notification = ThreadClosedNotification { + thread_id: thread_id.to_string(), + }; + outgoing + .send_server_notification(ServerNotification::ThreadClosed(notification)) + .await; + pending_thread_unloads.lock().await.remove(&thread_id); + } + ThreadShutdownResult::SubmitFailed => { + pending_thread_unloads.lock().await.remove(&thread_id); + warn!("failed to submit Shutdown to thread {thread_id}"); + } + ThreadShutdownResult::TimedOut => { + pending_thread_unloads.lock().await.remove(&thread_id); + warn!("thread {thread_id} shutdown timed out; leaving thread loaded"); + } + } + }); + } + + pub(super) async fn thread_unsubscribe( + &self, + request_id: ConnectionRequestId, + params: ThreadUnsubscribeParams, + ) { + let thread_id = match ThreadId::from_string(¶ms.thread_id) { + Ok(id) => id, + Err(err) => { + self.send_invalid_request_error(request_id, format!("invalid thread id: {err}")) + .await; + return; + } + }; + + if self.thread_manager.get_thread(thread_id).await.is_err() { + // Reconcile stale app-server bookkeeping when the thread has already been + // removed from the core manager. This keeps loaded-status/subscription state + // consistent with the source of truth before reporting NotLoaded. + self.finalize_thread_teardown(thread_id).await; + self.outgoing + .send_response( + request_id, + ThreadUnsubscribeResponse { + status: ThreadUnsubscribeStatus::NotLoaded, + }, + ) + .await; + return; + }; + + let was_subscribed = self + .thread_state_manager + .unsubscribe_connection_from_thread(thread_id, request_id.connection_id) + .await; + + let status = if was_subscribed { + ThreadUnsubscribeStatus::Unsubscribed + } else { + ThreadUnsubscribeStatus::NotSubscribed + }; + self.outgoing + .send_response(request_id, ThreadUnsubscribeResponse { status }) + .await; + } + + pub(super) async fn prepare_thread_for_archive(&self, thread_id: ThreadId) { + // If the thread is active, request shutdown and wait briefly. + let removed_conversation = self.thread_manager.remove_thread(&thread_id).await; + if let Some(conversation) = removed_conversation { + info!("thread {thread_id} was active; shutting down"); + match Self::wait_for_thread_shutdown(&conversation).await { + ThreadShutdownResult::Complete => {} + ThreadShutdownResult::SubmitFailed => { + error!( + "failed to submit Shutdown to thread {thread_id}; proceeding with archive" + ); + } + ThreadShutdownResult::TimedOut => { + warn!("thread {thread_id} shutdown timed out; proceeding with archive"); + } + } + } + self.finalize_thread_teardown(thread_id).await; + } + + pub(super) async fn ensure_conversation_listener( + &self, + conversation_id: ThreadId, + connection_id: ConnectionId, + raw_events_enabled: bool, + api_version: ApiVersion, + ) -> Result { + Self::ensure_conversation_listener_task( + ListenerTaskContext { + thread_manager: Arc::clone(&self.thread_manager), + thread_state_manager: self.thread_state_manager.clone(), + outgoing: Arc::clone(&self.outgoing), + pending_thread_unloads: Arc::clone(&self.pending_thread_unloads), + analytics_events_client: self.analytics_events_client.clone(), + general_analytics_enabled: self.config.features.enabled(Feature::GeneralAnalytics), + thread_watch_manager: self.thread_watch_manager.clone(), + epiphany_invalidation_manager: self.epiphany_invalidation_manager.clone(), + fallback_model_provider: self.config.model_provider_id.clone(), + codex_home: self.config.codex_home.to_path_buf(), + }, + conversation_id, + connection_id, + raw_events_enabled, + api_version, + ) + .await + } + + #[expect( + clippy::await_holding_invalid_type, + reason = "listener subscription must be serialized against pending thread unloads" + )] + pub(super) async fn ensure_conversation_listener_task( + listener_task_context: ListenerTaskContext, + conversation_id: ThreadId, + connection_id: ConnectionId, + raw_events_enabled: bool, + api_version: ApiVersion, + ) -> Result { + let conversation = match listener_task_context + .thread_manager + .get_thread(conversation_id) + .await + { + Ok(conv) => conv, + Err(_) => { + return Err(JSONRPCErrorError { + code: INVALID_REQUEST_ERROR_CODE, + message: format!("thread not found: {conversation_id}"), + data: None, + }); + } + }; + let thread_state = { + let pending_thread_unloads = listener_task_context.pending_thread_unloads.lock().await; + if pending_thread_unloads.contains(&conversation_id) { + return Err(JSONRPCErrorError { + code: INVALID_REQUEST_ERROR_CODE, + message: format!( + "thread {conversation_id} is closing; retry after the thread is closed" + ), + data: None, + }); + } + let Some(thread_state) = listener_task_context + .thread_state_manager + .try_ensure_connection_subscribed( + conversation_id, + connection_id, + raw_events_enabled, + ) + .await + else { + return Ok(EnsureConversationListenerResult::ConnectionClosed); + }; + thread_state + }; + if let Err(error) = Self::ensure_listener_task_running_task( + listener_task_context.clone(), + conversation_id, + conversation, + thread_state, + api_version, + ) + .await + { + let _ = listener_task_context + .thread_state_manager + .unsubscribe_connection_from_thread(conversation_id, connection_id) + .await; + return Err(error); + } + Ok(EnsureConversationListenerResult::Attached) + } + + pub(super) fn log_listener_attach_result( + result: Result, + thread_id: ThreadId, + connection_id: ConnectionId, + thread_kind: &'static str, + ) { + match result { + Ok(EnsureConversationListenerResult::Attached) => {} + Ok(EnsureConversationListenerResult::ConnectionClosed) => { + tracing::debug!( + thread_id = %thread_id, + connection_id = ?connection_id, + "skipping auto-attach for closed connection" + ); + } + Err(err) => { + tracing::warn!( + "failed to attach listener for {thread_kind} {thread_id}: {message}", + message = err.message + ); + } + } + } + + pub(super) async fn ensure_listener_task_running( + &self, + conversation_id: ThreadId, + conversation: Arc, + thread_state: Arc>, + api_version: ApiVersion, + ) -> Result<(), JSONRPCErrorError> { + Self::ensure_listener_task_running_task( + ListenerTaskContext { + thread_manager: Arc::clone(&self.thread_manager), + thread_state_manager: self.thread_state_manager.clone(), + outgoing: Arc::clone(&self.outgoing), + pending_thread_unloads: Arc::clone(&self.pending_thread_unloads), + analytics_events_client: self.analytics_events_client.clone(), + general_analytics_enabled: self.config.features.enabled(Feature::GeneralAnalytics), + thread_watch_manager: self.thread_watch_manager.clone(), + epiphany_invalidation_manager: self.epiphany_invalidation_manager.clone(), + fallback_model_provider: self.config.model_provider_id.clone(), + codex_home: self.config.codex_home.to_path_buf(), + }, + conversation_id, + conversation, + thread_state, + api_version, + ) + .await + } + + pub(super) async fn ensure_listener_task_running_task( + listener_task_context: ListenerTaskContext, + conversation_id: ThreadId, + conversation: Arc, + thread_state: Arc>, + api_version: ApiVersion, + ) -> Result<(), JSONRPCErrorError> { + let (cancel_tx, mut cancel_rx) = oneshot::channel(); + let Some(mut unloading_state) = UnloadingState::new( + &listener_task_context, + conversation_id, + THREAD_UNLOADING_DELAY, + ) + .await + else { + return Err(JSONRPCErrorError { + code: INVALID_REQUEST_ERROR_CODE, + message: format!( + "thread {conversation_id} is closing; retry after the thread is closed" + ), + data: None, + }); + }; + let (mut listener_command_rx, listener_generation) = { + let mut thread_state = thread_state.lock().await; + if thread_state.listener_matches(&conversation) { + return Ok(()); + } + thread_state.set_listener(cancel_tx, &conversation) + }; + let ListenerTaskContext { + outgoing, + thread_manager, + thread_state_manager, + pending_thread_unloads, + analytics_events_client: _, + general_analytics_enabled: _, + thread_watch_manager, + epiphany_invalidation_manager, + fallback_model_provider, + codex_home, + } = listener_task_context; + let outgoing_for_task = Arc::clone(&outgoing); + tokio::spawn(async move { + loop { + tokio::select! { + biased; + _ = &mut cancel_rx => { + // Listener was superseded or the thread is being torn down. + break; + } + listener_command = listener_command_rx.recv() => { + let Some(listener_command) = listener_command else { + break; + }; + handle_thread_listener_command( + conversation_id, + &conversation, + codex_home.as_path(), + &thread_state_manager, + &thread_state, + &thread_watch_manager, + &outgoing_for_task, + &pending_thread_unloads, + listener_command, + ) + .await; + } + event = conversation.next_event() => { + let event = match event { + Ok(event) => event, + Err(err) => { + tracing::warn!("thread.next_event() failed with: {err}"); + break; + } + }; + + // Track the event before emitting any typed + // translations so thread-local state such as raw event + // opt-in stays synchronized with the conversation. + let raw_events_enabled = { + let mut thread_state = thread_state.lock().await; + thread_state.track_current_turn_event(&event.msg); + thread_state.experimental_raw_events + }; + let subscribed_connection_ids = thread_state_manager + .subscribed_connection_ids(conversation_id) + .await; + let thread_outgoing = ThreadScopedOutgoingMessageSender::new( + outgoing_for_task.clone(), + subscribed_connection_ids, + conversation_id, + ); + + if let EventMsg::RawResponseItem(raw_response_item_event) = &event.msg + && !raw_events_enabled + { + maybe_emit_hook_prompt_item_completed( + api_version, + conversation_id, + &event.id, + &raw_response_item_event.item, + &thread_outgoing, + ) + .await; + continue; + } + + apply_bespoke_event_handling( + event.clone(), + conversation_id, + conversation.clone(), + thread_manager.clone(), + listener_task_context + .general_analytics_enabled + .then(|| listener_task_context.analytics_events_client.clone()), + thread_outgoing, + thread_state.clone(), + thread_watch_manager.clone(), + epiphany_invalidation_manager.clone(), + api_version, + fallback_model_provider.clone(), + codex_home.as_path(), + ) + .await; + } + unloading_watchers_open = unloading_state.wait_for_unloading_trigger() => { + if !unloading_watchers_open { + break; + } + if !unloading_state.should_unload_now() { + continue; + } + if matches!(conversation.agent_status().await, AgentStatus::Running) { + unloading_state.note_thread_activity_observed(); + continue; + } + { + let mut pending_thread_unloads = pending_thread_unloads.lock().await; + if pending_thread_unloads.contains(&conversation_id) { + continue; + } + if !unloading_state.should_unload_now() { + continue; + } + pending_thread_unloads.insert(conversation_id); + } + Self::unload_thread_without_subscribers( + thread_manager.clone(), + outgoing_for_task.clone(), + pending_thread_unloads.clone(), + thread_state_manager.clone(), + thread_watch_manager.clone(), + epiphany_invalidation_manager.clone(), + conversation_id, + conversation.clone(), + ) + .await; + break; + } + } + } + + let mut thread_state = thread_state.lock().await; + if thread_state.listener_generation == listener_generation { + thread_state.clear_listener(); + } + }); + Ok(()) + } +}