Fix app-server stack pressure in tests

This commit is contained in:
Meta Vens 2026-04-24 20:01:54 +01:00
commit fbd4dc28ff
8 changed files with 25 additions and 0 deletions

View file

@ -366,6 +366,10 @@ If a future session wakes up from compaction and starts bluffing, this is the pa
- move heavy Epiphany-owned implementation into the sibling crate when that can be done without giving up first-class typed integration
- Important Windows verification footgun still stands:
- use `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex` for `codex-core` work on this machine
- App-server stack-pressure footgun from the richer promotion slice:
- without a larger test-thread stack, unrelated app-server tests can overflow after the expanded request/promotion machinery
- `vendor/codex/codex-rs/.cargo/config.toml` now sets `RUST_MIN_STACK=67108864`
- do not remove that and then act shocked when Windows eats the test harness again
- Snapshot hygiene note:
- the new Epiphany prompt snapshot normalizes temp skill-root paths in the test harness so it stays stable across runs
@ -417,6 +421,7 @@ Verified:
- `cargo test -p codex-app-server-protocol --lib thread_epiphany_retrieve`
- `cargo test -p codex-app-server --lib map_epiphany_retrieve_response_preserves_summary_and_results`
- full `cargo test -p codex-app-server-protocol` passed after restoring stable schema fixtures
- after tracing the stack overflow from the richer promotion slice, full `cargo test -p codex-app-server --lib` now passes 232/232 with `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex` and repo-local Cargo config supplying `RUST_MIN_STACK=67108864`
Important Windows footgun:

View file

@ -92,3 +92,4 @@
{"ts": "2026-04-24T18:05:25+00:00", "type": "implementation", "status": "ok", "note": "Landed first richer promotion safety layer: state replacement promotions now require explicit observation/evidence records and validate subgoal/invariant/graph/frontier/checkpoint/churn structure in epiphany-core before app-server forwards accepted patches to the durable update path."}
{"ts": "2026-04-24T18:06:30+00:00", "type": "verification", "status": "ok", "note": "Verified richer promotion safety layer with epiphany-core full tests, codex-core epiphany tests, and codex-app-server epiphany tests. Full codex-app-server --lib run compiled but hit an unrelated stack overflow in in_process_start_uses_requested_session_source_for_thread_start; Epiphany-scoped app-server tests passed."}
{"ts": "2026-04-24T18:12:04+00:00", "type": "verification", "status": "warning", "note": "Rechecked codex-app-server stack overflow from full --lib verification: the exact test in_process::tests::in_process_start_uses_requested_session_source_for_thread_start reproduces when run alone. The failing test exercises in-process thread/start session-source mapping in app-server/src/in_process.rs, not the Epiphany promotion handler or epiphany-core policy path touched by bb557d1. Treat it as outside the touched Epiphany path, but not proven pre-existing without checking an older commit."}
{"ts": "2026-04-24T19:00:15+00:00", "type": "verification", "status": "ok", "note": "Traced the codex-app-server stack overflow after bb557d1. The original failing in_process thread/start test overflowed after the processor completed thread/start and before response routing; RUST_MIN_STACK=64MB made exact failures pass. Additional exact failures in guardian and thread-start tracing tests confirmed app-server test-thread stack pressure on Windows rather than one semantic Epiphany path failure. Added Cargo config RUST_MIN_STACK=67108864 and boxed large JSON/in-process request futures plus thread-start/promote futures; full cargo test -p codex-app-server --lib now passes 232/232 without setting RUST_MIN_STACK in the shell."}

View file

@ -94,6 +94,7 @@ accepted_design:
- land Epiphany's first typed distillation/proposal surface as `thread/epiphany/distill`: loaded-thread-only, read-only, deterministic observation/evidence patch generation with a concrete `expectedRevision`, and explicit promotion through `thread/epiphany/update`
- land Epiphany's first explicit promotion gate as `thread/epiphany/promote`: loaded-thread-only, verifier-backed, rejects failed candidates without mutation, and applies accepted candidates through the existing durable update path
- land the first richer map/churn promotion safety layer: state replacement promotions must include explicit observations and patch evidence, and subgoal/invariant/graph/frontier/checkpoint/churn replacements get structural validation in `epiphany-core` before reaching the durable update path
- keep Windows app-server verification explicit about stack pressure: Cargo config sets `RUST_MIN_STACK=67108864`, and large JSON/in-process request futures are boxed so the test binary does not depend on shell folklore or tiny default test-thread stacks
rejected_paths:
- name: pinned_map_only

View file

@ -136,6 +136,13 @@ This file is intentionally disposable.
- `cargo test -p codex-core --lib epiphany`
- `cargo test -p codex-app-server --lib epiphany`
- attempted full `cargo test -p codex-app-server --lib` compiled but hit a pre-existing-looking stack overflow in `in_process_start_uses_requested_session_source_for_thread_start`; Epiphany-scoped app-server tests passed
- Stack-overflow trace/fix on 2026-04-24:
- the original `in_process_start_uses_requested_session_source_for_thread_start` failure reproduced without `RUST_MIN_STACK`
- breadcrumbs showed `thread/start` completed in the processor and the overflow happened before response routing, pointing at app-server request/future stack pressure rather than Epiphany promotion semantics
- `RUST_MIN_STACK=67108864` made exact failures pass; later exact failures in guardian lifecycle and thread-start tracing tests confirmed the app-server test binary had crossed a Windows stack threshold
- `vendor/codex/codex-rs/.cargo/config.toml` now sets `RUST_MIN_STACK=67108864` so Cargo-driven tests get the required stack without shell folklore
- large app-server JSON/in-process request futures, `thread_start_task`, and the expanded `thread/epiphany/promote` arm are boxed to reduce dispatcher stack pressure
- full `cargo test -p codex-app-server --lib` now passes 232/232 with no shell `RUST_MIN_STACK` set
## Open Questions

View file

@ -1,3 +1,6 @@
[env]
RUST_MIN_STACK = "67108864"
[target.'cfg(all(windows, target_env = "msvc"))']
rustflags = ["-C", "link-arg=/STACK:8388608"]

View file

@ -384,6 +384,7 @@ use codex_thread_store::ThreadStoreError;
use codex_thread_store::UpdateThreadMetadataParams as StoreUpdateThreadMetadataParams;
use codex_utils_absolute_path::AbsolutePathBuf;
use codex_utils_pty::DEFAULT_OUTPUT_BYTES_CAP;
use futures::FutureExt;
use std::collections::HashMap;
use std::collections::HashSet;
use std::io::Error as IoError;
@ -988,6 +989,7 @@ impl CodexMessageProcessor {
}
ClientRequest::ThreadEpiphanyPromote { request_id, params } => {
self.thread_epiphany_promote(to_connection_request_id(request_id), params)
.boxed()
.await;
}
ClientRequest::ThreadEpiphanyUpdate { request_id, params } => {
@ -2463,8 +2465,10 @@ impl CodexMessageProcessor {
experimental_raw_events,
request_trace,
)
.boxed()
.await;
};
let thread_start_task = thread_start_task.boxed();
self.background_tasks
.spawn(thread_start_task.instrument(request_context.span()));
}

View file

@ -86,6 +86,7 @@ use codex_feedback::CodexFeedback;
use codex_login::AuthManager;
use codex_protocol::protocol::SessionSource;
pub use codex_state::log_db::LogDbLayer;
use futures::FutureExt;
use tokio::sync::mpsc;
use tokio::sync::oneshot;
use tokio::time::timeout;
@ -433,6 +434,7 @@ fn start_uninitialized(args: InProcessStartArgs) -> InProcessClientHandle {
Arc::clone(&session),
&outbound_initialized,
)
.boxed()
.await;
let opted_out_notification_methods_snapshot =
session.opted_out_notification_methods();

View file

@ -422,6 +422,7 @@ impl MessageProcessor {
/*outbound_initialized*/ None,
request_context.clone(),
)
.boxed()
.await;
},
)
@ -466,6 +467,7 @@ impl MessageProcessor {
Some(outbound_initialized),
request_context.clone(),
)
.boxed()
.await;
},
)