Extract Epiphany core and add licensing guardrails
This commit is contained in:
Meta Vens
parent
e425be0d78
commit
80c29e0934
26 changed files with 4140 additions and 1642 deletions
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -1,2 +1,4 @@
|
|||
__pycache__/
|
||||
*.pyc
|
||||
epiphany-core/target/
|
||||
epiphany-core/Cargo.lock
|
||||
|
|
|
|||
36
CONTRIBUTING.md
Normal file
36
CONTRIBUTING.md
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
Contributing To EpiphanyAgent
|
||||
=============================
|
||||
|
||||
EpiphanyAgent is source-available and dual-licensed. That only works if the
|
||||
project keeps clean rights to contributions.
|
||||
|
||||
Before sending anything non-trivial, read:
|
||||
|
||||
- `LICENSE`
|
||||
- `CONTRIBUTOR_LICENSE_AGREEMENT.md`
|
||||
|
||||
By submitting a pull request, patch, commit, issue attachment, or other
|
||||
contribution for inclusion in this project, you agree that your contribution is
|
||||
submitted under the EpiphanyAgent Contributor License Agreement version 1.0,
|
||||
unless the maintainers have accepted a different written agreement from you.
|
||||
|
||||
For small fixes, include this line in the pull request or patch submission:
|
||||
|
||||
```text
|
||||
I agree that my Contributions to EpiphanyAgent are submitted under the
|
||||
EpiphanyAgent Contributor License Agreement version 1.0.
|
||||
```
|
||||
|
||||
For employer-owned, client-owned, organization-owned, generated, copied,
|
||||
third-party, or otherwise legally complicated material, get explicit written
|
||||
permission before submitting it. If you cannot grant the project broad
|
||||
relicensing rights, do not submit the contribution.
|
||||
|
||||
Before accepting outside contributions at scale, the maintainers should record
|
||||
the current Project Steward by legal name or formed entity in the repository or
|
||||
signature records. The CLA already allows successor entities and assignments,
|
||||
but boring paperwork now prevents expensive archaeology later.
|
||||
|
||||
The maintainers may reject or remove contributions when the rights are unclear.
|
||||
That is not hostility. That is how the project avoids waking up years later
|
||||
with one old patch holding the whole licensing plan hostage.
|
||||
180
CONTRIBUTOR_LICENSE_AGREEMENT.md
Normal file
180
CONTRIBUTOR_LICENSE_AGREEMENT.md
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
EpiphanyAgent Contributor License Agreement
|
||||
===========================================
|
||||
|
||||
Version 1.0
|
||||
|
||||
This Contributor License Agreement ("Agreement") governs Contributions
|
||||
submitted to EpiphanyAgent.
|
||||
|
||||
This Agreement is designed as a broad contributor license, not a copyright
|
||||
assignment. You keep ownership of Your Contributions, but You give the Project
|
||||
Steward enough rights to use, sublicense, and relicense them without returning
|
||||
to You for later permission.
|
||||
|
||||
This structure follows the same general family of contributor-license
|
||||
agreements used by projects and organizations such as the Apache Software
|
||||
Foundation, Project Harmony, and Canonical: contributors retain ownership, while
|
||||
the project receives a broad inbound license.
|
||||
|
||||
1. Definitions
|
||||
--------------
|
||||
|
||||
"You" or "Your" means the individual or legal entity making a Contribution.
|
||||
For legal entities, "You" includes any entity that controls, is controlled by,
|
||||
or is under common control with the contributing entity.
|
||||
|
||||
"Project" means EpiphanyAgent and any successor project or distribution that
|
||||
continues, forks, renames, reorganizes, or commercializes substantially the
|
||||
same Project-Authored Material.
|
||||
|
||||
"Project Steward" means the copyright holder or holders who maintain
|
||||
EpiphanyAgent, any legal entity later formed or designated by them to own,
|
||||
license, or commercialize Project-Authored Material, and any successor or
|
||||
assignee authorized by them in writing.
|
||||
|
||||
"Contribution" means any original work of authorship, invention, improvement,
|
||||
modification, patch, documentation, design, media, text, code, configuration,
|
||||
test, or other material intentionally submitted by You to the Project for
|
||||
inclusion in, or use with, the Project.
|
||||
|
||||
"Submitted" means communicated to the Project or its maintainers by any
|
||||
reasonable channel used for Project development, including GitHub pull
|
||||
requests, patches, commits, issues, discussions, email, chat, or other written
|
||||
communication. A submission is not a Contribution if You clearly mark it
|
||||
"Not a Contribution."
|
||||
|
||||
2. Copyright License
|
||||
--------------------
|
||||
|
||||
Subject to this Agreement, You grant the Project Steward a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable, transferable,
|
||||
and sublicensable copyright license to use, reproduce, modify, prepare
|
||||
derivative works of, publicly display, publicly perform, distribute,
|
||||
sublicense, relicense, make available, sell, offer for sale, import, and
|
||||
otherwise exploit Your Contributions, in source or object form, for any
|
||||
purpose.
|
||||
|
||||
This includes the right to license or relicense Your Contributions, and
|
||||
derivative works of them, under any license or terms the Project Steward
|
||||
chooses, including open-source, source-available, noncommercial, commercial,
|
||||
proprietary, dual-license, or future license terms.
|
||||
|
||||
3. Patent License
|
||||
-----------------
|
||||
|
||||
Subject to this Agreement, You grant the Project Steward and recipients of
|
||||
Project distributions a perpetual, worldwide, non-exclusive, no-charge,
|
||||
royalty-free, irrevocable patent license to make, have made, use, offer to
|
||||
sell, sell, import, and otherwise transfer the Project and Your Contributions.
|
||||
|
||||
This patent license covers patent claims that You can license and that would
|
||||
necessarily be infringed by Your Contributions alone or by combination of Your
|
||||
Contributions with the Project to which they were submitted.
|
||||
|
||||
If any entity brings a written patent infringement claim alleging that Your
|
||||
Contribution or the Project infringes a patent, any patent license granted to
|
||||
that entity under this Agreement terminates as of the date that claim is filed.
|
||||
|
||||
4. Ownership
|
||||
------------
|
||||
|
||||
You retain ownership of Your Contributions. This Agreement does not prevent You
|
||||
from using, licensing, selling, or contributing Your Contributions elsewhere,
|
||||
so long as doing so does not violate another agreement or law.
|
||||
|
||||
5. Representations
|
||||
------------------
|
||||
|
||||
You represent that:
|
||||
|
||||
- You are legally entitled to grant the rights in this Agreement.
|
||||
- Each Contribution is Your original creation, or You have sufficient rights to
|
||||
submit it under this Agreement.
|
||||
- If Your employer, client, school, sponsor, or another organization may own or
|
||||
control rights in Your Contribution, You have permission to submit it under
|
||||
this Agreement or that organization has provided a separate written
|
||||
authorization acceptable to the Project Steward.
|
||||
- Your Contribution does not knowingly violate any agreement, license, trade
|
||||
secret obligation, confidentiality obligation, or third-party right.
|
||||
- You will identify any third-party code, data, model output, assets, license
|
||||
restrictions, patent restrictions, trademark restrictions, or other legal
|
||||
constraints of which You are aware and that are associated with Your
|
||||
Contribution.
|
||||
|
||||
6. Third-Party Material
|
||||
-----------------------
|
||||
|
||||
If You submit material that is not entirely Your original creation, You must
|
||||
clearly identify its source and applicable license or restriction.
|
||||
|
||||
The Project Steward may reject, remove, rewrite, or separately track that
|
||||
material if it is incompatible with the Project's licensing or commercial
|
||||
plans.
|
||||
|
||||
7. Moral Rights
|
||||
---------------
|
||||
|
||||
To the maximum extent permitted by law, You waive and agree not to assert any
|
||||
moral rights, droit moral, rights of integrity, rights of attribution, or
|
||||
similar rights that would interfere with the Project Steward's exercise of the
|
||||
rights granted in this Agreement.
|
||||
|
||||
Where those rights cannot be waived, You grant the Project Steward a
|
||||
perpetual, worldwide, royalty-free license to exercise them and agree not to
|
||||
assert them against the Project Steward, its sublicensees, distributors,
|
||||
customers, or users.
|
||||
|
||||
8. No Obligation To Use
|
||||
-----------------------
|
||||
|
||||
The Project Steward is not required to accept, use, preserve, publish, or
|
||||
maintain any Contribution.
|
||||
|
||||
9. No Support Or Warranty
|
||||
-------------------------
|
||||
|
||||
You are not required to provide support for Your Contributions.
|
||||
|
||||
Unless required by law or separately agreed in writing, You provide Your
|
||||
Contributions on an "AS IS" basis, without warranties or conditions of any
|
||||
kind, including warranties or conditions of title, non-infringement,
|
||||
merchantability, or fitness for a particular purpose.
|
||||
|
||||
10. Acceptance
|
||||
--------------
|
||||
|
||||
You accept this Agreement by signing it, by submitting a Contribution after
|
||||
being presented with this Agreement, or by including the following statement in
|
||||
a pull request, issue, patch, email, or other written submission:
|
||||
|
||||
I agree that my Contributions to EpiphanyAgent are submitted under the
|
||||
EpiphanyAgent Contributor License Agreement version 1.0.
|
||||
|
||||
The Project Steward may require a separate signed copy or an organization-level
|
||||
agreement before accepting non-trivial, employer-owned, client-owned, or
|
||||
commercially significant Contributions.
|
||||
|
||||
11. Entire Agreement For Contributions
|
||||
--------------------------------------
|
||||
|
||||
This Agreement governs the rights You grant to the Project Steward for Your
|
||||
Contributions unless You and the Project Steward sign a different written
|
||||
agreement.
|
||||
|
||||
If any part of this Agreement is unenforceable, the remaining parts remain in
|
||||
effect to the maximum extent permitted by law.
|
||||
|
||||
12. Signature Record
|
||||
--------------------
|
||||
|
||||
Contributor name:
|
||||
|
||||
Contributor GitHub username or other account:
|
||||
|
||||
Contributor email:
|
||||
|
||||
If signing for an organization, organization name and authority to sign:
|
||||
|
||||
Signature:
|
||||
|
||||
Date:
|
||||
261
LICENSE
Normal file
261
LICENSE
Normal file
|
|
@ -0,0 +1,261 @@
|
|||
EpiphanyAgent Repository License Notice
|
||||
=======================================
|
||||
|
||||
This repository is a multi-license distribution. Different parts of the
|
||||
repository are available under different terms.
|
||||
|
||||
Nothing in this file narrows, revokes, or relabels rights already granted by
|
||||
upstream licenses. In particular, nothing in this file should be read as
|
||||
withdrawing any permission granted by Apache License 2.0 for Apache-covered
|
||||
material.
|
||||
|
||||
1. Scope
|
||||
--------
|
||||
|
||||
For this file:
|
||||
|
||||
- `Project-Authored Material` means original software, documentation, and other
|
||||
repository content authored for EpiphanyAgent, except where a file or
|
||||
directory states different terms.
|
||||
- `Upstream Material` means material copied or adapted from third parties and
|
||||
distributed under their own licenses.
|
||||
- `Mixed Apache Files` means files under `vendor/codex/**` that contain
|
||||
project-authored changes on top of upstream Apache-licensed material.
|
||||
|
||||
2. Upstream And Third-Party Material
|
||||
------------------------------------
|
||||
|
||||
- All material under `vendor/codex/**` remains subject to its upstream licenses
|
||||
and notices, including Apache License 2.0 at `vendor/codex/LICENSE` and any
|
||||
applicable `NOTICE` or file-level notices.
|
||||
- Other third-party material distributed with the repository keeps its own
|
||||
license terms where provided.
|
||||
- You must continue to comply with those upstream terms when using,
|
||||
redistributing, or modifying upstream material.
|
||||
|
||||
3. Public License For Project-Authored Material
|
||||
-----------------------------------------------
|
||||
|
||||
Except where a file or directory says otherwise, Project-Authored Material
|
||||
outside `vendor/codex/**` is licensed to the public under:
|
||||
|
||||
- PolyForm Noncommercial License 1.0.0
|
||||
|
||||
The full text of that license appears in Appendix A of this file.
|
||||
|
||||
Required Notice: Copyright 2026 EpiphanyAgent authors and contributors.
|
||||
|
||||
4. Commercial License Option
|
||||
----------------------------
|
||||
|
||||
If you want to use Project-Authored Material for any purpose not permitted by
|
||||
PolyForm Noncommercial License 1.0.0, you must obtain a separate written
|
||||
commercial license from the applicable copyright holder or another party
|
||||
expressly authorized in writing to grant that license.
|
||||
|
||||
Commercial rights are not granted by:
|
||||
|
||||
- source availability
|
||||
- public access to this repository
|
||||
- silence
|
||||
- estoppel by implication
|
||||
- the fact that the licensors are not yet organized as a corporation or LLC
|
||||
|
||||
A natural person can own and license copyright. Until a separate business
|
||||
entity is formed and designated in writing, any commercial license for
|
||||
Project-Authored Material may be offered directly by the individual copyright
|
||||
holder or holders.
|
||||
|
||||
5. Mixed Apache Files Under `vendor/codex/**`
|
||||
---------------------------------------------
|
||||
|
||||
Some files under `vendor/codex/**` contain project-authored changes layered on
|
||||
top of upstream Apache-licensed code.
|
||||
|
||||
For those files:
|
||||
|
||||
- Apache License 2.0 and the applicable upstream notices still apply.
|
||||
- You must not read this repository notice as converting those files into a
|
||||
PolyForm-only codebase.
|
||||
- To the extent the project owns copyright in new additions to those files, the
|
||||
project may also offer separate commercial terms for its own contributions.
|
||||
- This repository notice does not claim to erase or retract Apache-2.0
|
||||
permissions already granted for Apache-covered material.
|
||||
|
||||
The strongest commercial boundary in this repository therefore applies to
|
||||
clearly separable Project-Authored Material outside `vendor/codex/**`,
|
||||
including `epiphany-core/**` and other non-vendored project files.
|
||||
|
||||
6. Open Source Status
|
||||
---------------------
|
||||
|
||||
Because the public license path for Project-Authored Material is PolyForm
|
||||
Noncommercial License 1.0.0 rather than an OSI-approved license, the
|
||||
repository as a whole is source-available and dual-licensed. It is not an OSI
|
||||
Open Source distribution as a whole.
|
||||
|
||||
7. Reference Structures
|
||||
-----------------------
|
||||
|
||||
This licensing structure intentionally borrows the general dual-license shape
|
||||
used by established projects such as Qt and MySQL:
|
||||
|
||||
- one public-facing license path
|
||||
- one separate commercial license path
|
||||
|
||||
This repository does not claim that Qt or MySQL use the same exact public
|
||||
license terms used here. Those projects are referenced only as structural
|
||||
examples of dual licensing, not as the operative terms of this repository.
|
||||
|
||||
8. Contributions
|
||||
----------------
|
||||
|
||||
External contributions are accepted only under the EpiphanyAgent Contributor
|
||||
License Agreement in `CONTRIBUTOR_LICENSE_AGREEMENT.md`, or under a separate
|
||||
written agreement accepted by the Project Steward.
|
||||
|
||||
The contributor agreement is intentionally broad. Contributors keep ownership
|
||||
of their contributions, but grant the Project Steward the right to sublicense
|
||||
and relicense those contributions under any license or terms the Project
|
||||
Steward chooses, including commercial or proprietary terms.
|
||||
|
||||
The project should not accept non-trivial outside contributions unless the
|
||||
contributor has agreed to that contributor agreement or an equivalent written
|
||||
agreement.
|
||||
|
||||
Appendix A: PolyForm Noncommercial License 1.0.0
|
||||
------------------------------------------------
|
||||
|
||||
The following text is included as the operative public license for
|
||||
Project-Authored Material described in Section 3.
|
||||
|
||||
# PolyForm Noncommercial License 1.0.0
|
||||
|
||||
<https://polyformproject.org/licenses/noncommercial/1.0.0>
|
||||
|
||||
## Acceptance
|
||||
|
||||
In order to get any license under these terms, you must agree
|
||||
to them as both strict obligations and conditions to all
|
||||
your licenses.
|
||||
|
||||
## Copyright License
|
||||
|
||||
The licensor grants you a copyright license for the
|
||||
software to do everything you might do with the software
|
||||
that would otherwise infringe the licensor's copyright
|
||||
in it for any permitted purpose. However, you may
|
||||
only distribute the software according to Distribution
|
||||
License and make changes or new works based on the software
|
||||
according to Changes and New Works License.
|
||||
|
||||
## Distribution License
|
||||
|
||||
The licensor grants you an additional copyright license
|
||||
to distribute copies of the software. Your license
|
||||
to distribute covers distributing the software with
|
||||
changes and new works permitted by Changes and New Works
|
||||
License.
|
||||
|
||||
## Notices
|
||||
|
||||
You must ensure that anyone who gets a copy of any part of
|
||||
the software from you also gets a copy of these terms or the
|
||||
URL for them above, as well as copies of any plain-text lines
|
||||
beginning with `Required Notice:` that the licensor provided
|
||||
with the software. For example:
|
||||
|
||||
> Required Notice: Copyright Yoyodyne, Inc. (http://example.com)
|
||||
|
||||
## Changes and New Works License
|
||||
|
||||
The licensor grants you an additional copyright license to
|
||||
make changes and new works based on the software for any
|
||||
permitted purpose.
|
||||
|
||||
## Patent License
|
||||
|
||||
The licensor grants you a patent license for the software that
|
||||
covers patent claims the licensor can license, or becomes able
|
||||
to license, that you would infringe by using the software.
|
||||
|
||||
## Noncommercial Purposes
|
||||
|
||||
Any noncommercial purpose is a permitted purpose.
|
||||
|
||||
## Personal Uses
|
||||
|
||||
Personal use for research, experiment, and testing for
|
||||
the benefit of public knowledge, personal study, private
|
||||
entertainment, hobby projects, amateur pursuits, or religious
|
||||
observance, without any anticipated commercial application,
|
||||
is use for a permitted purpose.
|
||||
|
||||
## Noncommercial Organizations
|
||||
|
||||
Use by any charitable organization, educational institution,
|
||||
public research organization, public safety or health
|
||||
organization, environmental protection organization,
|
||||
or government institution is use for a permitted purpose
|
||||
regardless of the source of funding or obligations resulting
|
||||
from the funding.
|
||||
|
||||
## Fair Use
|
||||
|
||||
You may have "fair use" rights for the software under the
|
||||
law. These terms do not limit them.
|
||||
|
||||
## No Other Rights
|
||||
|
||||
These terms do not allow you to sublicense or transfer any of
|
||||
your licenses to anyone else, or prevent the licensor from
|
||||
granting licenses to anyone else. These terms do not imply
|
||||
any other licenses.
|
||||
|
||||
## Patent Defense
|
||||
|
||||
If you make any written claim that the software infringes or
|
||||
contributes to infringement of any patent, your patent license
|
||||
for the software granted under these terms ends immediately. If
|
||||
your company makes such a claim, your patent license ends
|
||||
immediately for work on behalf of your company.
|
||||
|
||||
## Violations
|
||||
|
||||
The first time you are notified in writing that you have
|
||||
violated any of these terms, or done anything with the software
|
||||
not covered by your licenses, your licenses can nonetheless
|
||||
continue if you come into full compliance with these terms,
|
||||
and take practical steps to correct past violations, within
|
||||
32 days of receiving notice. Otherwise, all your licenses
|
||||
end immediately.
|
||||
|
||||
## No Liability
|
||||
|
||||
As far as the law allows, the software comes as is, without
|
||||
any warranty or condition, and the licensor will not be liable
|
||||
to you for any damages arising out of these terms or the use
|
||||
or nature of the software, under any kind of legal claim.
|
||||
|
||||
## Definitions
|
||||
|
||||
The licensor is the individual or entity offering these
|
||||
terms, and the software is the software the licensor makes
|
||||
available under these terms.
|
||||
|
||||
You refers to the individual or entity agreeing to these
|
||||
terms.
|
||||
|
||||
Your company is any legal entity, sole proprietorship,
|
||||
or other kind of organization that you work for, plus all
|
||||
organizations that have control over, are under the control of,
|
||||
or are under common control with that organization. Control
|
||||
means ownership of substantially all the assets of an entity,
|
||||
or the power to direct its management and policies by vote,
|
||||
contract, or otherwise. Control can be direct or indirect.
|
||||
|
||||
Your licenses are all the licenses granted to you for the
|
||||
software under these terms.
|
||||
|
||||
Use means anything you do with the software requiring one
|
||||
of your licenses.
|
||||
16
README.md
16
README.md
|
|
@ -120,3 +120,19 @@ python .\tools\epiphany_state.py add-evidence --type verify --status ok --note "
|
|||
python .\tools\epiphany_state.py add-branch --id b1 --hypothesis "Smaller verifier beats elaborate verifier"
|
||||
python .\tools\epiphany_state.py close-branch --id b1 --status rejected --note "More ceremony, no better signal"
|
||||
```
|
||||
|
||||
## Licensing
|
||||
|
||||
The root `LICENSE` file is now an operative repository license notice, not just
|
||||
a throat-clearing draft. In short: `vendor/codex/**` and other third-party
|
||||
material keep their upstream licenses, while Project-Authored Material outside
|
||||
`vendor/codex/**` is publicly available under PolyForm Noncommercial 1.0.0 and
|
||||
is also intended to be available under separate commercial terms by written
|
||||
agreement. That makes the repo source-available and dual-licensed, not OSI open
|
||||
source as a whole.
|
||||
|
||||
External contributions require the EpiphanyAgent Contributor License Agreement
|
||||
in `CONTRIBUTOR_LICENSE_AGREEMENT.md`, or a separate written agreement accepted
|
||||
by the Project Steward. The point is simple: contributors keep ownership, but
|
||||
the project gets the right to sublicense and relicense contributions without
|
||||
future archaeology through old pull requests.
|
||||
|
|
|
|||
30
epiphany-core/Cargo.toml
Normal file
30
epiphany-core/Cargo.toml
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
[package]
|
||||
name = "epiphany-core"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
publish = false
|
||||
license-file = "../LICENSE"
|
||||
|
||||
[lib]
|
||||
name = "epiphany_core"
|
||||
path = "src/lib.rs"
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
bm25 = "2.3.2"
|
||||
chrono = "0.4.44"
|
||||
codex-file-search = { path = "../vendor/codex/codex-rs/file-search" }
|
||||
codex-protocol = { path = "../vendor/codex/codex-rs/protocol" }
|
||||
ignore = "0.4.23"
|
||||
reqwest = { version = "0.12", features = ["blocking", "json"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
sha1 = "0.10.6"
|
||||
tracing = "0.1.44"
|
||||
uuid = { version = "1", features = ["v4", "v5"] }
|
||||
|
||||
[dev-dependencies]
|
||||
pretty_assertions = "1.4.1"
|
||||
tempfile = "3"
|
||||
tokio = { version = "1", features = ["rt", "rt-multi-thread"] }
|
||||
wiremock = "0.6"
|
||||
15
epiphany-core/src/lib.rs
Normal file
15
epiphany-core/src/lib.rs
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
mod prompt;
|
||||
mod retrieval;
|
||||
mod rollout;
|
||||
|
||||
pub use prompt::render_epiphany_state;
|
||||
pub use retrieval::EPIPHANY_RETRIEVAL_DEFAULT_LIMIT;
|
||||
pub use retrieval::EPIPHANY_RETRIEVAL_MAX_LIMIT;
|
||||
pub use retrieval::EpiphanyRetrieveQuery;
|
||||
pub use retrieval::EpiphanyRetrieveResponse;
|
||||
pub use retrieval::EpiphanyRetrieveResult;
|
||||
pub use retrieval::EpiphanyRetrieveResultKind;
|
||||
pub use retrieval::index_workspace;
|
||||
pub use retrieval::retrieval_state_for_workspace;
|
||||
pub use retrieval::retrieve_workspace;
|
||||
pub use rollout::latest_epiphany_state_from_rollout_items;
|
||||
773
epiphany-core/src/prompt.rs
Normal file
773
epiphany-core/src/prompt.rs
Normal file
|
|
@ -0,0 +1,773 @@
|
|||
use codex_protocol::protocol::EpiphanyChurnState;
|
||||
use codex_protocol::protocol::EpiphanyCodeRef;
|
||||
use codex_protocol::protocol::EpiphanyEvidenceRecord;
|
||||
use codex_protocol::protocol::EpiphanyGraph;
|
||||
use codex_protocol::protocol::EpiphanyGraphCheckpoint;
|
||||
use codex_protocol::protocol::EpiphanyGraphEdge;
|
||||
use codex_protocol::protocol::EpiphanyGraphFrontier;
|
||||
use codex_protocol::protocol::EpiphanyGraphLink;
|
||||
use codex_protocol::protocol::EpiphanyGraphNode;
|
||||
use codex_protocol::protocol::EpiphanyModeState;
|
||||
use codex_protocol::protocol::EpiphanyObservation;
|
||||
use codex_protocol::protocol::EpiphanyScratchPad;
|
||||
use codex_protocol::protocol::EpiphanyThreadState;
|
||||
use std::collections::HashSet;
|
||||
use std::path::Path;
|
||||
|
||||
const SUBGOAL_LIMIT: usize = 4;
|
||||
const INVARIANT_LIMIT: usize = 4;
|
||||
const NODE_LIMIT: usize = 3;
|
||||
const EDGE_LIMIT: usize = 2;
|
||||
const LINK_LIMIT: usize = 3;
|
||||
const OBSERVATION_LIMIT: usize = 3;
|
||||
const EVIDENCE_LIMIT: usize = 3;
|
||||
const CODE_REF_LIMIT: usize = 2;
|
||||
const DIRTY_PATH_LIMIT: usize = 4;
|
||||
|
||||
pub fn render_epiphany_state(state: &EpiphanyThreadState) -> String {
|
||||
let mut sections = vec![concat!(
|
||||
"\n## Epiphany State\n",
|
||||
"Treat this as the current structured thread understanding. Use it to stay oriented during this turn. ",
|
||||
"If your planned edits conflict with it or expose a gap, surface that before broad edits. ",
|
||||
"Do not silently rewrite this state in prose."
|
||||
)
|
||||
.to_string()];
|
||||
|
||||
sections.push(render_overview(state));
|
||||
|
||||
if let Some(subgoals) = render_subgoals(state) {
|
||||
sections.push(subgoals);
|
||||
}
|
||||
if let Some(invariants) = render_invariants(state) {
|
||||
sections.push(invariants);
|
||||
}
|
||||
if let Some(graphs) = render_graphs(state) {
|
||||
sections.push(graphs);
|
||||
}
|
||||
if let Some(scratch) = render_scratch(state.scratch.as_ref()) {
|
||||
sections.push(scratch);
|
||||
}
|
||||
if let Some(observations) = render_observations(&state.observations) {
|
||||
sections.push(observations);
|
||||
}
|
||||
if let Some(evidence) = render_evidence(&state.recent_evidence) {
|
||||
sections.push(evidence);
|
||||
}
|
||||
if let Some(churn) = render_churn(state.churn.as_ref()) {
|
||||
sections.push(churn);
|
||||
}
|
||||
if let Some(mode) = render_mode(state.mode.as_ref()) {
|
||||
sections.push(mode);
|
||||
}
|
||||
|
||||
format!("{}\n", sections.join("\n\n"))
|
||||
}
|
||||
|
||||
fn render_overview(state: &EpiphanyThreadState) -> String {
|
||||
let mut lines = vec![format!("- Revision: {}", state.revision)];
|
||||
if let Some(objective) = state.objective.as_deref() {
|
||||
lines.push(format!("- Objective: {}", compact_text(objective)));
|
||||
}
|
||||
if let Some(active_subgoal_id) = state.active_subgoal_id.as_deref() {
|
||||
lines.push(format!("- Active subgoal: `{active_subgoal_id}`"));
|
||||
}
|
||||
if let Some(last_updated_turn_id) = state.last_updated_turn_id.as_deref() {
|
||||
lines.push(format!("- Last updated turn: `{last_updated_turn_id}`"));
|
||||
}
|
||||
if let Some(frontier) = state.graph_frontier.as_ref() {
|
||||
let focus_ids = collect_focus_node_ids(Some(frontier), state.graph_checkpoint.as_ref());
|
||||
if !focus_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Frontier focus: {}",
|
||||
render_inline_ids(&focus_ids)
|
||||
));
|
||||
}
|
||||
}
|
||||
format!("## Overview\n{}", lines.join("\n"))
|
||||
}
|
||||
|
||||
fn render_subgoals(state: &EpiphanyThreadState) -> Option<String> {
|
||||
if state.subgoals.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
for subgoal in state.subgoals.iter().take(SUBGOAL_LIMIT) {
|
||||
let active_suffix = state
|
||||
.active_subgoal_id
|
||||
.as_deref()
|
||||
.filter(|active_id| *active_id == subgoal.id)
|
||||
.map(|_| "; active")
|
||||
.unwrap_or("");
|
||||
lines.push(format!(
|
||||
"- `{}` [{}{}]: {}",
|
||||
subgoal.id,
|
||||
subgoal.status,
|
||||
active_suffix,
|
||||
compact_text(&subgoal.title)
|
||||
));
|
||||
if let Some(summary) = subgoal.summary.as_deref() {
|
||||
lines.push(format!(" summary: {}", compact_text(summary)));
|
||||
}
|
||||
}
|
||||
push_omitted_count(&mut lines, state.subgoals.len(), SUBGOAL_LIMIT, "subgoals");
|
||||
|
||||
Some(format!("## Subgoals\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_invariants(state: &EpiphanyThreadState) -> Option<String> {
|
||||
if state.invariants.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
for invariant in state.invariants.iter().take(INVARIANT_LIMIT) {
|
||||
lines.push(format!(
|
||||
"- `{}` [{}]: {}",
|
||||
invariant.id,
|
||||
invariant.status,
|
||||
compact_text(&invariant.description)
|
||||
));
|
||||
if let Some(rationale) = invariant.rationale.as_deref() {
|
||||
lines.push(format!(" rationale: {}", compact_text(rationale)));
|
||||
}
|
||||
}
|
||||
push_omitted_count(
|
||||
&mut lines,
|
||||
state.invariants.len(),
|
||||
INVARIANT_LIMIT,
|
||||
"invariants",
|
||||
);
|
||||
|
||||
Some(format!("## Invariants\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_graphs(state: &EpiphanyThreadState) -> Option<String> {
|
||||
if state.graphs.is_empty() && state.graph_frontier.is_none() && state.graph_checkpoint.is_none()
|
||||
{
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
let focus_node_ids = collect_focus_node_ids(
|
||||
state.graph_frontier.as_ref(),
|
||||
state.graph_checkpoint.as_ref(),
|
||||
);
|
||||
|
||||
if let Some(frontier) = state.graph_frontier.as_ref() {
|
||||
append_frontier_lines(&mut lines, frontier);
|
||||
}
|
||||
if let Some(checkpoint) = state.graph_checkpoint.as_ref() {
|
||||
append_checkpoint_lines(&mut lines, checkpoint);
|
||||
}
|
||||
|
||||
append_graph_lines(
|
||||
&mut lines,
|
||||
"Architecture graph",
|
||||
&state.graphs.architecture,
|
||||
&focus_node_ids,
|
||||
state.graph_frontier.as_ref(),
|
||||
);
|
||||
append_graph_lines(
|
||||
&mut lines,
|
||||
"Dataflow graph",
|
||||
&state.graphs.dataflow,
|
||||
&focus_node_ids,
|
||||
state.graph_frontier.as_ref(),
|
||||
);
|
||||
|
||||
if !state.graphs.links.is_empty() {
|
||||
for link in select_links(&state.graphs.links, &focus_node_ids)
|
||||
.into_iter()
|
||||
.take(LINK_LIMIT)
|
||||
{
|
||||
let relationship = link
|
||||
.relationship
|
||||
.as_deref()
|
||||
.map(|value| format!(" ({})", compact_text(value)))
|
||||
.unwrap_or_default();
|
||||
let code_refs = render_code_refs(&link.code_refs);
|
||||
lines.push(format!(
|
||||
"- Link: `{}` -> `{}`{}{}",
|
||||
link.dataflow_node_id,
|
||||
link.architecture_node_id,
|
||||
relationship,
|
||||
render_ref_suffix(&code_refs)
|
||||
));
|
||||
}
|
||||
push_omitted_count(
|
||||
&mut lines,
|
||||
state.graphs.links.len(),
|
||||
LINK_LIMIT,
|
||||
"graph links",
|
||||
);
|
||||
}
|
||||
|
||||
Some(format!("## Graphs\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn append_frontier_lines(lines: &mut Vec<String>, frontier: &EpiphanyGraphFrontier) {
|
||||
if !frontier.active_node_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Frontier active nodes: {}",
|
||||
render_inline_ids(&frontier.active_node_ids)
|
||||
));
|
||||
}
|
||||
if !frontier.active_edge_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Frontier active edges: {}",
|
||||
render_inline_ids(&frontier.active_edge_ids)
|
||||
));
|
||||
}
|
||||
if !frontier.open_question_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Open questions: {}",
|
||||
render_inline_ids(&frontier.open_question_ids)
|
||||
));
|
||||
}
|
||||
if !frontier.open_gap_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Open gaps: {}",
|
||||
render_inline_ids(&frontier.open_gap_ids)
|
||||
));
|
||||
}
|
||||
if !frontier.dirty_paths.is_empty() {
|
||||
let dirty_paths = frontier
|
||||
.dirty_paths
|
||||
.iter()
|
||||
.take(DIRTY_PATH_LIMIT)
|
||||
.map(|path| format!("`{}`", normalize_path(path)))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ");
|
||||
let omitted_suffix =
|
||||
omitted_suffix(frontier.dirty_paths.len(), DIRTY_PATH_LIMIT, "dirty paths");
|
||||
lines.push(format!("- Dirty paths: {dirty_paths}{omitted_suffix}"));
|
||||
}
|
||||
}
|
||||
|
||||
fn append_checkpoint_lines(lines: &mut Vec<String>, checkpoint: &EpiphanyGraphCheckpoint) {
|
||||
lines.push(format!(
|
||||
"- Checkpoint: `{}` (graph revision {})",
|
||||
checkpoint.checkpoint_id, checkpoint.graph_revision
|
||||
));
|
||||
if let Some(summary) = checkpoint.summary.as_deref() {
|
||||
lines.push(format!("- Checkpoint summary: {}", compact_text(summary)));
|
||||
}
|
||||
if !checkpoint.frontier_node_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Checkpoint frontier nodes: {}",
|
||||
render_inline_ids(&checkpoint.frontier_node_ids)
|
||||
));
|
||||
}
|
||||
if !checkpoint.open_question_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Checkpoint open questions: {}",
|
||||
render_inline_ids(&checkpoint.open_question_ids)
|
||||
));
|
||||
}
|
||||
if !checkpoint.open_gap_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Checkpoint open gaps: {}",
|
||||
render_inline_ids(&checkpoint.open_gap_ids)
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
fn append_graph_lines(
|
||||
lines: &mut Vec<String>,
|
||||
label: &str,
|
||||
graph: &EpiphanyGraph,
|
||||
focus_node_ids: &[String],
|
||||
frontier: Option<&EpiphanyGraphFrontier>,
|
||||
) {
|
||||
if graph.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
let selected_nodes = select_nodes(graph, focus_node_ids);
|
||||
let selected_node_ids = selected_nodes
|
||||
.iter()
|
||||
.map(|node| node.id.as_str())
|
||||
.collect::<HashSet<_>>();
|
||||
let active_edge_ids = frontier
|
||||
.map(|graph_frontier| {
|
||||
graph_frontier
|
||||
.active_edge_ids
|
||||
.iter()
|
||||
.map(String::as_str)
|
||||
.collect::<HashSet<_>>()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
|
||||
lines.push(format!(
|
||||
"- {label}: {} nodes, {} edges",
|
||||
graph.nodes.len(),
|
||||
graph.edges.len()
|
||||
));
|
||||
|
||||
for node in &selected_nodes {
|
||||
let status = node
|
||||
.status
|
||||
.as_deref()
|
||||
.map(|value| format!(" [{}]", compact_text(value)))
|
||||
.unwrap_or_default();
|
||||
lines.push(format!(
|
||||
" - Focus node `{}`{}: {}",
|
||||
node.id,
|
||||
status,
|
||||
compact_text(&node.title)
|
||||
));
|
||||
lines.push(format!(" purpose: {}", compact_text(&node.purpose)));
|
||||
if let Some(mechanism) = node.mechanism.as_deref() {
|
||||
lines.push(format!(" mechanism: {}", compact_text(mechanism)));
|
||||
}
|
||||
if let Some(metaphor) = node.metaphor.as_deref() {
|
||||
lines.push(format!(" metaphor: {}", compact_text(metaphor)));
|
||||
}
|
||||
let code_refs = render_code_refs(&node.code_refs);
|
||||
if !code_refs.is_empty() {
|
||||
lines.push(format!(" code refs: {}", code_refs.join(", ")));
|
||||
}
|
||||
}
|
||||
push_omitted_count(lines, graph.nodes.len(), NODE_LIMIT, "graph nodes");
|
||||
|
||||
let selected_edges = select_edges(graph, &selected_node_ids, &active_edge_ids);
|
||||
for edge in selected_edges.iter().take(EDGE_LIMIT) {
|
||||
let edge_id = edge
|
||||
.id
|
||||
.as_deref()
|
||||
.map(|value| format!(" `{value}`"))
|
||||
.unwrap_or_default();
|
||||
let label_suffix = edge
|
||||
.label
|
||||
.as_deref()
|
||||
.map(|value| format!(" [{}]", compact_text(value)))
|
||||
.unwrap_or_default();
|
||||
let mechanism_suffix = edge
|
||||
.mechanism
|
||||
.as_deref()
|
||||
.map(|value| format!("; {}", compact_text(value)))
|
||||
.unwrap_or_default();
|
||||
let code_refs = render_code_refs(&edge.code_refs);
|
||||
lines.push(format!(
|
||||
" - Edge{edge_id}{label_suffix}: `{}` -> `{}` ({}){}{}",
|
||||
edge.source_id,
|
||||
edge.target_id,
|
||||
edge.kind,
|
||||
mechanism_suffix,
|
||||
render_ref_suffix(&code_refs)
|
||||
));
|
||||
}
|
||||
push_omitted_count(
|
||||
lines,
|
||||
selected_edges.len(),
|
||||
EDGE_LIMIT,
|
||||
"focused graph edges",
|
||||
);
|
||||
}
|
||||
|
||||
fn render_scratch(scratch: Option<&EpiphanyScratchPad>) -> Option<String> {
|
||||
let scratch = scratch?;
|
||||
let mut lines = Vec::new();
|
||||
if let Some(summary) = scratch.summary.as_deref() {
|
||||
lines.push(format!("- Summary: {}", compact_text(summary)));
|
||||
}
|
||||
if let Some(hypothesis) = scratch.hypothesis.as_deref() {
|
||||
lines.push(format!("- Hypothesis: {}", compact_text(hypothesis)));
|
||||
}
|
||||
if let Some(next_probe) = scratch.next_probe.as_deref() {
|
||||
lines.push(format!("- Next probe: {}", compact_text(next_probe)));
|
||||
}
|
||||
|
||||
if lines.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(format!("## Scratch\n{}", lines.join("\n")))
|
||||
}
|
||||
}
|
||||
|
||||
fn render_observations(observations: &[EpiphanyObservation]) -> Option<String> {
|
||||
if observations.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
for observation in observations.iter().take(OBSERVATION_LIMIT) {
|
||||
lines.push(format!(
|
||||
"- `{}` [{} via {}]: {}",
|
||||
observation.id,
|
||||
observation.status,
|
||||
observation.source_kind,
|
||||
compact_text(&observation.summary)
|
||||
));
|
||||
if !observation.evidence_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
" evidence: {}",
|
||||
render_inline_ids(&observation.evidence_ids)
|
||||
));
|
||||
}
|
||||
let code_refs = render_code_refs(&observation.code_refs);
|
||||
if !code_refs.is_empty() {
|
||||
lines.push(format!(" code refs: {}", code_refs.join(", ")));
|
||||
}
|
||||
}
|
||||
push_omitted_count(
|
||||
&mut lines,
|
||||
observations.len(),
|
||||
OBSERVATION_LIMIT,
|
||||
"observations",
|
||||
);
|
||||
|
||||
Some(format!("## Recent Observations\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_evidence(evidence: &[EpiphanyEvidenceRecord]) -> Option<String> {
|
||||
if evidence.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
for record in evidence.iter().take(EVIDENCE_LIMIT) {
|
||||
let code_refs = render_code_refs(&record.code_refs);
|
||||
lines.push(format!(
|
||||
"- `{}` [{} / {}]: {}{}",
|
||||
record.id,
|
||||
record.kind,
|
||||
record.status,
|
||||
compact_text(&record.summary),
|
||||
render_ref_suffix(&code_refs)
|
||||
));
|
||||
}
|
||||
push_omitted_count(
|
||||
&mut lines,
|
||||
evidence.len(),
|
||||
EVIDENCE_LIMIT,
|
||||
"evidence records",
|
||||
);
|
||||
|
||||
Some(format!("## Recent Evidence\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_churn(churn: Option<&EpiphanyChurnState>) -> Option<String> {
|
||||
let churn = churn?;
|
||||
let mut lines = vec![
|
||||
format!(
|
||||
"- Understanding status: {}",
|
||||
compact_text(&churn.understanding_status)
|
||||
),
|
||||
format!("- Diff pressure: {}", compact_text(&churn.diff_pressure)),
|
||||
];
|
||||
if let Some(graph_freshness) = churn.graph_freshness.as_deref() {
|
||||
lines.push(format!(
|
||||
"- Graph freshness: {}",
|
||||
compact_text(graph_freshness)
|
||||
));
|
||||
}
|
||||
if let Some(warning) = churn.warning.as_deref() {
|
||||
lines.push(format!("- Warning: {}", compact_text(warning)));
|
||||
}
|
||||
if let Some(unexplained_writes) = churn.unexplained_writes {
|
||||
lines.push(format!("- Unexplained writes: {unexplained_writes}"));
|
||||
}
|
||||
|
||||
Some(format!("## Churn\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_mode(mode: Option<&EpiphanyModeState>) -> Option<String> {
|
||||
let mode = mode?;
|
||||
let mut line = format!("- Name: {}", compact_text(&mode.name));
|
||||
if let Some(kind) = mode.kind {
|
||||
line.push_str(&format!(" ({kind:?})"));
|
||||
}
|
||||
Some(format!("## Mode\n{line}"))
|
||||
}
|
||||
|
||||
fn collect_focus_node_ids(
|
||||
frontier: Option<&EpiphanyGraphFrontier>,
|
||||
checkpoint: Option<&EpiphanyGraphCheckpoint>,
|
||||
) -> Vec<String> {
|
||||
let mut focus_node_ids = Vec::new();
|
||||
let mut seen = HashSet::new();
|
||||
for node_id in frontier
|
||||
.into_iter()
|
||||
.flat_map(|value| value.active_node_ids.iter())
|
||||
.chain(
|
||||
checkpoint
|
||||
.into_iter()
|
||||
.flat_map(|value| value.frontier_node_ids.iter()),
|
||||
)
|
||||
{
|
||||
if seen.insert(node_id.as_str()) {
|
||||
focus_node_ids.push(node_id.clone());
|
||||
}
|
||||
}
|
||||
focus_node_ids
|
||||
}
|
||||
|
||||
fn select_nodes<'a>(
|
||||
graph: &'a EpiphanyGraph,
|
||||
focus_node_ids: &[String],
|
||||
) -> Vec<&'a EpiphanyGraphNode> {
|
||||
let mut selected = Vec::new();
|
||||
let mut seen = HashSet::new();
|
||||
let focus_set = focus_node_ids
|
||||
.iter()
|
||||
.map(String::as_str)
|
||||
.collect::<HashSet<_>>();
|
||||
|
||||
for node in &graph.nodes {
|
||||
if focus_set.contains(node.id.as_str()) && seen.insert(node.id.as_str()) {
|
||||
selected.push(node);
|
||||
if selected.len() == NODE_LIMIT {
|
||||
return selected;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for node in &graph.nodes {
|
||||
if seen.insert(node.id.as_str()) {
|
||||
selected.push(node);
|
||||
if selected.len() == NODE_LIMIT {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
selected
|
||||
}
|
||||
|
||||
fn select_edges<'a>(
|
||||
graph: &'a EpiphanyGraph,
|
||||
selected_node_ids: &HashSet<&'a str>,
|
||||
active_edge_ids: &HashSet<&str>,
|
||||
) -> Vec<&'a EpiphanyGraphEdge> {
|
||||
graph
|
||||
.edges
|
||||
.iter()
|
||||
.filter(|edge| {
|
||||
edge.id
|
||||
.as_deref()
|
||||
.is_some_and(|edge_id| active_edge_ids.contains(edge_id))
|
||||
|| selected_node_ids.contains(edge.source_id.as_str())
|
||||
|| selected_node_ids.contains(edge.target_id.as_str())
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn select_links<'a>(
|
||||
links: &'a [EpiphanyGraphLink],
|
||||
focus_node_ids: &[String],
|
||||
) -> Vec<&'a EpiphanyGraphLink> {
|
||||
let focus_set = focus_node_ids
|
||||
.iter()
|
||||
.map(String::as_str)
|
||||
.collect::<HashSet<_>>();
|
||||
let mut focused = Vec::new();
|
||||
let mut fallback = Vec::new();
|
||||
for link in links {
|
||||
if focus_set.contains(link.dataflow_node_id.as_str())
|
||||
|| focus_set.contains(link.architecture_node_id.as_str())
|
||||
{
|
||||
focused.push(link);
|
||||
} else {
|
||||
fallback.push(link);
|
||||
}
|
||||
}
|
||||
focused.extend(fallback);
|
||||
focused
|
||||
}
|
||||
|
||||
fn render_code_refs(code_refs: &[EpiphanyCodeRef]) -> Vec<String> {
|
||||
let mut rendered = code_refs
|
||||
.iter()
|
||||
.take(CODE_REF_LIMIT)
|
||||
.map(render_code_ref)
|
||||
.collect::<Vec<_>>();
|
||||
if code_refs.len() > CODE_REF_LIMIT {
|
||||
rendered.push(format!(
|
||||
"... {} more refs",
|
||||
code_refs.len() - CODE_REF_LIMIT
|
||||
));
|
||||
}
|
||||
rendered
|
||||
}
|
||||
|
||||
fn render_code_ref(code_ref: &EpiphanyCodeRef) -> String {
|
||||
let mut location = normalize_path(&code_ref.path);
|
||||
match (code_ref.start_line, code_ref.end_line) {
|
||||
(Some(start_line), Some(end_line)) if end_line != start_line => {
|
||||
location.push_str(&format!(":{start_line}-{end_line}"));
|
||||
}
|
||||
(Some(start_line), _) => {
|
||||
location.push_str(&format!(":{start_line}"));
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
|
||||
let mut metadata = Vec::new();
|
||||
if let Some(symbol) = code_ref.symbol.as_deref() {
|
||||
metadata.push(format!("symbol: {}", compact_text(symbol)));
|
||||
}
|
||||
if let Some(note) = code_ref.note.as_deref() {
|
||||
metadata.push(compact_text(note));
|
||||
}
|
||||
|
||||
if metadata.is_empty() {
|
||||
format!("`{location}`")
|
||||
} else {
|
||||
format!("`{location}` ({})", metadata.join("; "))
|
||||
}
|
||||
}
|
||||
|
||||
fn render_ref_suffix(code_refs: &[String]) -> String {
|
||||
if code_refs.is_empty() {
|
||||
String::new()
|
||||
} else {
|
||||
format!("; refs: {}", code_refs.join(", "))
|
||||
}
|
||||
}
|
||||
|
||||
fn render_inline_ids(ids: &[String]) -> String {
|
||||
ids.iter()
|
||||
.map(|id| format!("`{id}`"))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ")
|
||||
}
|
||||
|
||||
fn normalize_path(path: &Path) -> String {
|
||||
path.to_string_lossy().replace('\\', "/")
|
||||
}
|
||||
|
||||
fn compact_text(value: &str) -> String {
|
||||
value.split_whitespace().collect::<Vec<_>>().join(" ")
|
||||
}
|
||||
|
||||
fn omitted_suffix(total: usize, shown: usize, label: &str) -> String {
|
||||
if total > shown {
|
||||
format!("; ... {} more {label} not shown", total - shown)
|
||||
} else {
|
||||
String::new()
|
||||
}
|
||||
}
|
||||
|
||||
fn push_omitted_count(lines: &mut Vec<String>, total: usize, shown: usize, label: &str) {
|
||||
if total > shown {
|
||||
lines.push(format!("- ... {} more {label} not shown", total - shown));
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::render_epiphany_state;
|
||||
use codex_protocol::config_types::ModeKind;
|
||||
use codex_protocol::protocol::EpiphanyChurnState;
|
||||
use codex_protocol::protocol::EpiphanyCodeRef;
|
||||
use codex_protocol::protocol::EpiphanyGraph;
|
||||
use codex_protocol::protocol::EpiphanyGraphCheckpoint;
|
||||
use codex_protocol::protocol::EpiphanyGraphEdge;
|
||||
use codex_protocol::protocol::EpiphanyGraphFrontier;
|
||||
use codex_protocol::protocol::EpiphanyGraphNode;
|
||||
use codex_protocol::protocol::EpiphanyGraphs;
|
||||
use codex_protocol::protocol::EpiphanyModeState;
|
||||
use codex_protocol::protocol::EpiphanyThreadState;
|
||||
use std::path::PathBuf;
|
||||
|
||||
#[test]
|
||||
fn render_prefers_frontier_focus_and_omits_empty_sections() {
|
||||
let state = EpiphanyThreadState {
|
||||
revision: 7,
|
||||
objective: Some("Map the actual turn loop.".to_string()),
|
||||
graphs: EpiphanyGraphs {
|
||||
architecture: EpiphanyGraph {
|
||||
nodes: vec![
|
||||
EpiphanyGraphNode {
|
||||
id: "arch-overview".to_string(),
|
||||
title: "Overview".to_string(),
|
||||
purpose: "Loose context nobody is touching.".to_string(),
|
||||
..Default::default()
|
||||
},
|
||||
EpiphanyGraphNode {
|
||||
id: "arch-session".to_string(),
|
||||
title: "Session".to_string(),
|
||||
purpose: "Owns prompt assembly.".to_string(),
|
||||
mechanism: Some("Builds the initial context bundle.".to_string()),
|
||||
metaphor: Some("The foreman keeping the queue honest.".to_string()),
|
||||
code_refs: vec![EpiphanyCodeRef {
|
||||
path: PathBuf::from("core/src/session/mod.rs"),
|
||||
start_line: Some(2433),
|
||||
end_line: Some(2617),
|
||||
symbol: Some("build_initial_context".to_string()),
|
||||
note: None,
|
||||
}],
|
||||
..Default::default()
|
||||
},
|
||||
EpiphanyGraphNode {
|
||||
id: "arch-rollout".to_string(),
|
||||
title: "Rollout".to_string(),
|
||||
purpose: "Persists thread state.".to_string(),
|
||||
..Default::default()
|
||||
},
|
||||
EpiphanyGraphNode {
|
||||
id: "arch-extra".to_string(),
|
||||
title: "Extra".to_string(),
|
||||
purpose: "Should be trimmed.".to_string(),
|
||||
..Default::default()
|
||||
},
|
||||
],
|
||||
edges: vec![EpiphanyGraphEdge {
|
||||
source_id: "arch-session".to_string(),
|
||||
target_id: "arch-rollout".to_string(),
|
||||
kind: "writes".to_string(),
|
||||
id: Some("edge-session-rollout".to_string()),
|
||||
label: Some("persists".to_string()),
|
||||
mechanism: Some("Writes baseline state after a turn.".to_string()),
|
||||
..Default::default()
|
||||
}],
|
||||
},
|
||||
dataflow: EpiphanyGraph {
|
||||
nodes: vec![EpiphanyGraphNode {
|
||||
id: "flow-build-context".to_string(),
|
||||
title: "Build context".to_string(),
|
||||
purpose: "Turns state into prompt context.".to_string(),
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
},
|
||||
graph_frontier: Some(EpiphanyGraphFrontier {
|
||||
active_node_ids: vec!["arch-session".to_string(), "flow-build-context".to_string()],
|
||||
active_edge_ids: vec!["edge-session-rollout".to_string()],
|
||||
dirty_paths: vec![PathBuf::from("core/src/session/mod.rs")],
|
||||
..Default::default()
|
||||
}),
|
||||
graph_checkpoint: Some(EpiphanyGraphCheckpoint {
|
||||
checkpoint_id: "cp-7".to_string(),
|
||||
graph_revision: 7,
|
||||
frontier_node_ids: vec!["arch-session".to_string()],
|
||||
..Default::default()
|
||||
}),
|
||||
churn: Some(EpiphanyChurnState {
|
||||
understanding_status: "stable".to_string(),
|
||||
diff_pressure: "low".to_string(),
|
||||
graph_freshness: Some("fresh".to_string()),
|
||||
warning: None,
|
||||
unexplained_writes: Some(0),
|
||||
}),
|
||||
mode: Some(EpiphanyModeState {
|
||||
name: "epiphany".to_string(),
|
||||
kind: Some(ModeKind::Default),
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let rendered = render_epiphany_state(&state);
|
||||
|
||||
assert!(rendered.contains("`arch-session`"));
|
||||
assert!(rendered.contains("`flow-build-context`"));
|
||||
assert!(!rendered.contains("## Scratch"));
|
||||
assert!(!rendered.contains("## Invariants"));
|
||||
assert!(rendered.contains("`core/src/session/mod.rs:2433-2617`"));
|
||||
assert!(rendered.contains("... 1 more graph nodes not shown"));
|
||||
}
|
||||
}
|
||||
2023
epiphany-core/src/retrieval.rs
Normal file
2023
epiphany-core/src/retrieval.rs
Normal file
File diff suppressed because it is too large
Load diff
332
epiphany-core/src/rollout.rs
Normal file
332
epiphany-core/src/rollout.rs
Normal file
|
|
@ -0,0 +1,332 @@
|
|||
use codex_protocol::models::ResponseItem;
|
||||
use codex_protocol::protocol::EpiphanyThreadState;
|
||||
use codex_protocol::protocol::EventMsg;
|
||||
use codex_protocol::protocol::RolloutItem;
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct ActiveEpiphanyReplaySegment {
|
||||
turn_id: Option<String>,
|
||||
counts_as_user_turn: bool,
|
||||
epiphany_state: Option<EpiphanyThreadState>,
|
||||
}
|
||||
|
||||
fn turn_ids_are_compatible(active_turn_id: Option<&str>, item_turn_id: Option<&str>) -> bool {
|
||||
active_turn_id
|
||||
.is_none_or(|turn_id| item_turn_id.is_none_or(|item_turn_id| item_turn_id == turn_id))
|
||||
}
|
||||
|
||||
fn finalize_active_segment(
|
||||
active_segment: ActiveEpiphanyReplaySegment,
|
||||
epiphany_state: &mut Option<EpiphanyThreadState>,
|
||||
pending_rollback_turns: &mut usize,
|
||||
) {
|
||||
if *pending_rollback_turns > 0 {
|
||||
if active_segment.counts_as_user_turn {
|
||||
*pending_rollback_turns -= 1;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if epiphany_state.is_none() && active_segment.counts_as_user_turn {
|
||||
*epiphany_state = active_segment.epiphany_state;
|
||||
}
|
||||
}
|
||||
|
||||
pub fn latest_epiphany_state_from_rollout_items<F>(
|
||||
rollout_items: &[RolloutItem],
|
||||
is_user_turn_boundary: F,
|
||||
) -> Option<EpiphanyThreadState>
|
||||
where
|
||||
F: Fn(&ResponseItem) -> bool,
|
||||
{
|
||||
let mut epiphany_state = None;
|
||||
let mut pending_rollback_turns = 0usize;
|
||||
let mut active_segment: Option<ActiveEpiphanyReplaySegment> = None;
|
||||
|
||||
for item in rollout_items.iter().rev() {
|
||||
match item {
|
||||
RolloutItem::EventMsg(EventMsg::ThreadRolledBack(rollback)) => {
|
||||
pending_rollback_turns = pending_rollback_turns
|
||||
.saturating_add(usize::try_from(rollback.num_turns).unwrap_or(usize::MAX));
|
||||
}
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(event)) => {
|
||||
let active_segment =
|
||||
active_segment.get_or_insert_with(ActiveEpiphanyReplaySegment::default);
|
||||
if active_segment.turn_id.is_none() {
|
||||
active_segment.turn_id = Some(event.turn_id.clone());
|
||||
}
|
||||
}
|
||||
RolloutItem::EventMsg(EventMsg::TurnAborted(event)) => {
|
||||
if let Some(active_segment) = active_segment.as_mut() {
|
||||
if active_segment.turn_id.is_none()
|
||||
&& let Some(turn_id) = &event.turn_id
|
||||
{
|
||||
active_segment.turn_id = Some(turn_id.clone());
|
||||
}
|
||||
} else if let Some(turn_id) = &event.turn_id {
|
||||
active_segment = Some(ActiveEpiphanyReplaySegment {
|
||||
turn_id: Some(turn_id.clone()),
|
||||
..Default::default()
|
||||
});
|
||||
}
|
||||
}
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(_)) => {
|
||||
let active_segment =
|
||||
active_segment.get_or_insert_with(ActiveEpiphanyReplaySegment::default);
|
||||
active_segment.counts_as_user_turn = true;
|
||||
}
|
||||
RolloutItem::ResponseItem(response_item) => {
|
||||
let active_segment =
|
||||
active_segment.get_or_insert_with(ActiveEpiphanyReplaySegment::default);
|
||||
active_segment.counts_as_user_turn |= is_user_turn_boundary(response_item);
|
||||
}
|
||||
RolloutItem::EpiphanyState(item) => {
|
||||
let active_segment =
|
||||
active_segment.get_or_insert_with(ActiveEpiphanyReplaySegment::default);
|
||||
if active_segment.turn_id.is_none() {
|
||||
active_segment.turn_id = item.turn_id.clone();
|
||||
}
|
||||
if turn_ids_are_compatible(
|
||||
active_segment.turn_id.as_deref(),
|
||||
item.turn_id.as_deref(),
|
||||
) && active_segment.epiphany_state.is_none()
|
||||
{
|
||||
active_segment.epiphany_state = Some(item.state.clone());
|
||||
}
|
||||
}
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(event)) => {
|
||||
if active_segment.as_ref().is_some_and(|active_segment| {
|
||||
turn_ids_are_compatible(
|
||||
active_segment.turn_id.as_deref(),
|
||||
Some(event.turn_id.as_str()),
|
||||
)
|
||||
}) && let Some(active_segment) = active_segment.take()
|
||||
{
|
||||
finalize_active_segment(
|
||||
active_segment,
|
||||
&mut epiphany_state,
|
||||
&mut pending_rollback_turns,
|
||||
);
|
||||
}
|
||||
}
|
||||
RolloutItem::EventMsg(_)
|
||||
| RolloutItem::Compacted(_)
|
||||
| RolloutItem::TurnContext(_)
|
||||
| RolloutItem::SessionMeta(_) => {}
|
||||
}
|
||||
|
||||
if epiphany_state.is_some() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(active_segment) = active_segment.take() {
|
||||
finalize_active_segment(
|
||||
active_segment,
|
||||
&mut epiphany_state,
|
||||
&mut pending_rollback_turns,
|
||||
);
|
||||
}
|
||||
|
||||
epiphany_state
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::latest_epiphany_state_from_rollout_items;
|
||||
use codex_protocol::models::ContentItem;
|
||||
use codex_protocol::models::ResponseItem;
|
||||
use codex_protocol::protocol::CompactedItem;
|
||||
use codex_protocol::protocol::EpiphanyStateItem;
|
||||
use codex_protocol::protocol::EpiphanyThreadState;
|
||||
use codex_protocol::protocol::EventMsg;
|
||||
use codex_protocol::protocol::RolloutItem;
|
||||
use codex_protocol::protocol::ThreadRolledBackEvent;
|
||||
use codex_protocol::protocol::TurnCompleteEvent;
|
||||
use codex_protocol::protocol::TurnStartedEvent;
|
||||
use codex_protocol::protocol::UserMessageEvent;
|
||||
|
||||
fn simple_is_user_turn_boundary(item: &ResponseItem) -> bool {
|
||||
matches!(
|
||||
item,
|
||||
ResponseItem::Message { role, .. } if role == "user"
|
||||
)
|
||||
}
|
||||
|
||||
fn user_response_message(text: &str) -> ResponseItem {
|
||||
ResponseItem::Message {
|
||||
id: None,
|
||||
role: "user".to_string(),
|
||||
content: vec![ContentItem::InputText {
|
||||
text: text.to_string(),
|
||||
}],
|
||||
end_turn: None,
|
||||
phase: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn sample_epiphany_state(turn_id: &str) -> EpiphanyThreadState {
|
||||
EpiphanyThreadState {
|
||||
revision: 1,
|
||||
objective: Some(format!("Objective for {turn_id}")),
|
||||
active_subgoal_id: Some(format!("subgoal-{turn_id}")),
|
||||
last_updated_turn_id: Some(turn_id.to_string()),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn latest_epiphany_state_from_rollout_items_returns_latest_surviving_snapshot() {
|
||||
let first = sample_epiphany_state("turn-1");
|
||||
let second = sample_epiphany_state("turn-2");
|
||||
let rollout_items = vec![
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-1".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent {
|
||||
message: "first".to_string(),
|
||||
images: None,
|
||||
text_elements: Vec::new(),
|
||||
local_images: Vec::new(),
|
||||
})),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-1".to_string()),
|
||||
state: first,
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-1".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-2".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::ResponseItem(user_response_message("second")),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-2".to_string()),
|
||||
state: second.clone(),
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-2".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
];
|
||||
|
||||
assert_eq!(
|
||||
latest_epiphany_state_from_rollout_items(&rollout_items, simple_is_user_turn_boundary),
|
||||
Some(second)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn latest_epiphany_state_from_rollout_items_skips_rolled_back_turns() {
|
||||
let first = sample_epiphany_state("turn-1");
|
||||
let second = sample_epiphany_state("turn-2");
|
||||
let rollout_items = vec![
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-1".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent {
|
||||
message: "first".to_string(),
|
||||
images: None,
|
||||
text_elements: Vec::new(),
|
||||
local_images: Vec::new(),
|
||||
})),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-1".to_string()),
|
||||
state: first.clone(),
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-1".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-2".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent {
|
||||
message: "second".to_string(),
|
||||
images: None,
|
||||
text_elements: Vec::new(),
|
||||
local_images: Vec::new(),
|
||||
})),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-2".to_string()),
|
||||
state: second,
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-2".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::ThreadRolledBack(ThreadRolledBackEvent {
|
||||
num_turns: 1,
|
||||
})),
|
||||
];
|
||||
|
||||
assert_eq!(
|
||||
latest_epiphany_state_from_rollout_items(&rollout_items, simple_is_user_turn_boundary),
|
||||
Some(first)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn latest_epiphany_state_from_rollout_items_survives_compaction() {
|
||||
let state = sample_epiphany_state("turn-before-compaction");
|
||||
let rollout_items = vec![
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-before-compaction".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent {
|
||||
message: "before compaction".to_string(),
|
||||
images: None,
|
||||
text_elements: Vec::new(),
|
||||
local_images: Vec::new(),
|
||||
})),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-before-compaction".to_string()),
|
||||
state: state.clone(),
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-before-compaction".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
RolloutItem::Compacted(CompactedItem {
|
||||
message: "compact".to_string(),
|
||||
replacement_history: None,
|
||||
}),
|
||||
];
|
||||
|
||||
assert_eq!(
|
||||
latest_epiphany_state_from_rollout_items(&rollout_items, simple_is_user_turn_boundary),
|
||||
Some(state)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -8,8 +8,10 @@ Updated on 2026-04-24 after landing the first bounded Phase 4 slice on `main`:
|
|||
- Phase 2 prompt integration
|
||||
- a minimal Phase 3 typed client read surface via `Thread.epiphanyState`
|
||||
- a verified Phase 4 slice 1 query-time hybrid retriever across protocol, core, app-server protocol, and app-server
|
||||
- a verified Phase 4 slice 2 working-tree follow-up that adds explicit persistent semantic indexing through `thread/epiphany/index` while keeping `thread/epiphany/retrieve` read-only
|
||||
- the heavy Epiphany-owned prompt/replay/retrieval implementation has now been extracted into the repo-owned `epiphany-core` crate, leaving vendored Codex with thin host adapters plus the typed integration seam
|
||||
|
||||
The next job is no longer to prove the retriever exists. That part is done. The next job is to keep the landing zone honest: do not casually turn `thread/epiphany/retrieve` into a durable Epiphany-state writer just because the state object has a `retrieval` field, and do not regress into another blob-shaped persistent semantic store when the time comes to add one.
|
||||
The next job is no longer to prove the retriever exists, and it is no longer to sketch the persistent follow-up in prose. That part is done in the working tree. The next job is to keep the landing zone honest: do not casually turn `thread/epiphany/retrieve` into a durable Epiphany-state writer just because the state object has a `retrieval` field, and do not widen the new explicit indexing path into watcher-driven or GUI-shaped machinery before it earns it.
|
||||
|
||||
## Summary
|
||||
|
||||
|
|
@ -72,8 +74,14 @@ Preflight note:
|
|||
Current landed implementation note:
|
||||
|
||||
- the protocol scaffold still exists in `vendor/codex/codex-rs/protocol/src/protocol.rs`
|
||||
- it is now wired through the working tree across:
|
||||
- the heavier Epiphany-owned implementation now lives in:
|
||||
- `E:/Projects/EpiphanyAgent/epiphany-core/src/prompt.rs`
|
||||
- `E:/Projects/EpiphanyAgent/epiphany-core/src/rollout.rs`
|
||||
- `E:/Projects/EpiphanyAgent/epiphany-core/src/retrieval.rs`
|
||||
- the vendored host seam is now wired through the working tree across:
|
||||
- `core/src/epiphany_retrieval.rs`
|
||||
- `core/src/epiphany_rollout.rs`
|
||||
- `core/src/context/epiphany_state_instructions.rs`
|
||||
- `core/src/codex_thread.rs`
|
||||
- `core/src/session/tests.rs`
|
||||
- `app-server-protocol/src/protocol/common.rs`
|
||||
|
|
@ -112,18 +120,15 @@ Keep this metadata summary-focused. Do not dump raw embeddings, giant posting li
|
|||
|
||||
Touch:
|
||||
|
||||
- `vendor/codex/codex-rs/core/src/`
|
||||
|
||||
Add a focused retrieval module, tentatively:
|
||||
|
||||
- `epiphany_retrieval.rs`
|
||||
- `E:/Projects/EpiphanyAgent/epiphany-core/src/retrieval.rs`
|
||||
- `vendor/codex/codex-rs/core/src/epiphany_retrieval.rs`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- accept a loaded thread plus query params
|
||||
- resolve the relevant workspace root
|
||||
- run exact retrieval and semantic retrieval behind one interface
|
||||
- return typed ranked results with file paths, line anchors, excerpts, and retrieval mode metadata
|
||||
- the repo-owned crate should accept a loaded-thread-derived workspace root plus query params
|
||||
- it should run exact retrieval and semantic retrieval behind one interface
|
||||
- it should return typed ranked results with file paths, line anchors, excerpts, and retrieval mode metadata
|
||||
- the vendored Codex file should stay a thin adapter or re-export layer, not the place where the heavy organ grows back out of spite
|
||||
|
||||
Result types should be explicit and bounded:
|
||||
|
||||
|
|
@ -173,7 +178,21 @@ Suggested response fields:
|
|||
- `indexSummary`
|
||||
- `results`
|
||||
|
||||
Keep it additive and read-only. No write/update/index-control methods yet unless they turn out to be mechanically unavoidable.
|
||||
Keep it additive and read-only. That part is now landed.
|
||||
|
||||
Follow-up addition that is now verified in the working tree:
|
||||
|
||||
- `thread/epiphany/index`
|
||||
|
||||
Current bounded shape of that follow-up:
|
||||
|
||||
- it is the explicit write path for persistent semantic indexing
|
||||
- it persists manifest metadata under `codex_home`
|
||||
- it targets Qdrant as the preferred persistent semantic backend
|
||||
- it uses local Ollama embeddings by default
|
||||
- it leaves exact/path lookup first-class
|
||||
- it keeps BM25 alive as the fallback/control path when the persistent backend is stale, missing, or unavailable
|
||||
- it now runs through the sibling `epiphany-core` crate so modified Codex alone is not quietly the whole product
|
||||
|
||||
### 5. Reuse existing exact-search substrate where practical
|
||||
|
||||
|
|
@ -200,14 +219,15 @@ Add tests for:
|
|||
|
||||
## Immediate Next Step After This Plan
|
||||
|
||||
Treat the current retrieval baseline as real and only grow it when the next bounded follow-up is ready:
|
||||
Treat the current retrieval baseline and the new explicit indexing follow-up as real and only grow them where the current machine is still visibly missing an organ:
|
||||
|
||||
1. treat the current verified-and-landed query-time hybrid retriever as the Phase 4 slice 1 baseline
|
||||
2. do not add durable retrieval-summary writes from `thread/epiphany/retrieve` without a clean out-of-band rollout/update semantic
|
||||
3. if a later follow-up is needed, keep it tight:
|
||||
- retrieval freshness/persistence semantics
|
||||
- probably a dedicated Epiphany update path
|
||||
- Qdrant-backed persistent semantic indexing as the preferred backend, with BM25 fallback for no-Qdrant or degraded mode
|
||||
2. treat the verified working-tree `thread/epiphany/index` slice as the bounded persistent-semantic follow-up
|
||||
3. keep the new `epiphany-core` boundary honest instead of letting vendored Codex re-accumulate the heavy implementation
|
||||
4. do not add durable retrieval-summary writes from `thread/epiphany/retrieve` without a clean out-of-band rollout/update semantic
|
||||
5. if a later follow-up is needed, keep it tight:
|
||||
- live smoke/operational polish for the explicit indexing path
|
||||
- maybe a cleaner config surface for backend settings if env-only starts feeling too feral
|
||||
- not GUI work
|
||||
- not watcher-driven invalidation
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ Scope: current state after Phase 1, Phase 2, the minimal Phase 3 typed read surf
|
|||
|
||||
## Mental model in one sentence
|
||||
|
||||
Current Epiphany is a thin but real overlay on Codex's thread harness: Codex now carries structured thread state through persistence, replay, prompt assembly, hydrated client thread payloads, and one typed repo-local hybrid retrieval surface.
|
||||
Current Epiphany is a thin but real overlay on Codex's thread harness: Codex now carries structured thread state through persistence, replay, prompt assembly, hydrated client thread payloads, and typed repo-local retrieval/indexing surfaces, while the heavier Epiphany prompt, replay, and retrieval organs live in a sibling repo-owned crate instead of entirely inside vendored Codex.
|
||||
|
||||
## Divergence summary
|
||||
|
||||
|
|
@ -26,8 +26,8 @@ Today Epiphany differs from plain Codex in five currently active places:
|
|||
4. **Replay semantics respect rollback and compaction**
|
||||
- the "latest Epiphany state" is not "last one in the file." It is reconstructed using turn/rollback boundaries.
|
||||
|
||||
5. **Repo-local hybrid retrieval now exists behind one typed surface**
|
||||
- loaded threads can answer structured repo queries through exact/path lookup plus BM25 chunk retrieval.
|
||||
5. **Repo-local retrieval/indexing now exists behind typed surfaces**
|
||||
- loaded threads can answer structured repo queries through exact/path lookup plus BM25/Qdrant-backed retrieval, and they can explicitly build the persistent semantic side through a separate indexing path.
|
||||
|
||||
That is already a meaningful divergence from stock Codex, even if the more glamorous organs are still missing.
|
||||
|
||||
|
|
@ -38,15 +38,16 @@ flowchart TD
|
|||
A["SessionState.epiphany_state"] --> B["Persist once per real user turn as RolloutItem::EpiphanyState"]
|
||||
B --> C["Rollout replay / resume / rollback / compaction reconstruction"]
|
||||
C --> D["Live SessionState restored"]
|
||||
D --> E["build_initial_context injects EpiphanyStateInstructions"]
|
||||
D --> E["thin Codex prompt adapter calls epiphany-core renderer"]
|
||||
D --> F["CodexThread exposes epiphany_state()"]
|
||||
F --> G["app-server hydrates Thread.epiphanyState"]
|
||||
B --> H["app-server stored-thread reads reconstruct latest surviving Epiphany state from rollout"]
|
||||
G --> I["Client can read typed state"]
|
||||
H --> I
|
||||
D --> J["CodexThread exposes epiphany_retrieve() and epiphany_retrieval_state()"]
|
||||
D --> J["CodexThread host methods call epiphany-core retrieval/indexing engine"]
|
||||
J --> K["app-server routes thread/epiphany/retrieve"]
|
||||
K --> L["Hybrid exact/path + BM25 chunk results"]
|
||||
J --> M["app-server routes thread/epiphany/index"]
|
||||
K --> L["Hybrid exact/path + BM25/Qdrant-backed retrieval results"]
|
||||
```
|
||||
|
||||
The important thing is that the same state is now flowing through three layers:
|
||||
|
|
@ -97,6 +98,7 @@ This is the first place where Epiphany stops being "remember this in prose" and
|
|||
|
||||
Session persistence and replay now carry Epiphany state through the thread lifecycle:
|
||||
|
||||
- [E:/Projects/EpiphanyAgent/epiphany-core/src/rollout.rs](E:/Projects/EpiphanyAgent/epiphany-core/src/rollout.rs)
|
||||
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/session/mod.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/session/mod.rs)
|
||||
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/epiphany_rollout.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/epiphany_rollout.rs)
|
||||
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/codex_thread.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/codex_thread.rs)
|
||||
|
|
@ -108,6 +110,7 @@ Current behavior:
|
|||
- resume restores the latest surviving Epiphany snapshot
|
||||
- rollback skips rolled-back Epiphany snapshots
|
||||
- compaction does not erase the latest surviving snapshot
|
||||
- the vendored rollout wrapper now delegates the replay scan into `epiphany-core` and only supplies Codex's idea of what counts as a user-turn boundary
|
||||
|
||||
The key helper is:
|
||||
|
||||
|
|
@ -142,6 +145,7 @@ Not the whole future laboratory, just the notebook.
|
|||
|
||||
Turn construction now reads Epiphany state and injects a bounded developer fragment:
|
||||
|
||||
- [E:/Projects/EpiphanyAgent/epiphany-core/src/prompt.rs](E:/Projects/EpiphanyAgent/epiphany-core/src/prompt.rs)
|
||||
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/context/epiphany_state_instructions.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/context/epiphany_state_instructions.rs)
|
||||
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/session/mod.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/session/mod.rs)
|
||||
|
||||
|
|
@ -149,6 +153,7 @@ Current behavior:
|
|||
|
||||
- if `SessionState.epiphany_state` is present
|
||||
- `Session::build_initial_context(...)` adds `EpiphanyStateInstructions`
|
||||
- the vendored wrapper now calls `epiphany_core::render_epiphany_state(...)` instead of owning the heavy rendering logic itself
|
||||
- the fragment is wrapped in `<epiphany_state> ... </epiphany_state>`
|
||||
- the renderer is bounded and selective
|
||||
|
||||
|
|
@ -218,13 +223,14 @@ We do **not** yet have:
|
|||
|
||||
So the client can now see the state, but cannot yet steer or subscribe to it as a first-class live surface.
|
||||
|
||||
## Divergence 5: repo-local hybrid retrieval
|
||||
## Divergence 5: repo-local retrieval/indexing
|
||||
|
||||
### What changed
|
||||
|
||||
Loaded threads now expose a typed retrieval surface that combines exact/path-ish lookup with semantic chunk retrieval:
|
||||
Loaded threads now expose typed retrieval/indexing surfaces backed by a repo-owned implementation crate and a thin vendored host seam:
|
||||
|
||||
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/protocol/src/protocol.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/protocol/src/protocol.rs)
|
||||
- [E:/Projects/EpiphanyAgent/epiphany-core/src/retrieval.rs](E:/Projects/EpiphanyAgent/epiphany-core/src/retrieval.rs)
|
||||
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/epiphany_retrieval.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/epiphany_retrieval.rs)
|
||||
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/codex_thread.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/core/src/codex_thread.rs)
|
||||
- [E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/app-server-protocol/src/protocol/common.rs](E:/Projects/EpiphanyAgent/vendor/codex/codex-rs/app-server-protocol/src/protocol/common.rs)
|
||||
|
|
@ -234,13 +240,18 @@ Loaded threads now expose a typed retrieval surface that combines exact/path-ish
|
|||
Current behavior:
|
||||
|
||||
- `EpiphanyThreadState` now has retrieval metadata via `retrieval`
|
||||
- `epiphany-core` now owns the heavy retrieval/indexing engine and manifest/Qdrant/Ollama/BM25 logic
|
||||
- `vendor/codex/codex-rs/core/src/epiphany_retrieval.rs` is now a thin re-export layer instead of the whole engine
|
||||
- `CodexThread` exposes:
|
||||
- `epiphany_retrieval_state()`
|
||||
- `epiphany_retrieve(...)`
|
||||
- `epiphany_index(...)`
|
||||
- app-server protocol exposes experimental `thread/epiphany/retrieve`
|
||||
- app-server protocol also exposes experimental `thread/epiphany/index`
|
||||
- the retrieval machine is hybrid from day one:
|
||||
- exact/path-ish hits via existing `codex_file_search`
|
||||
- semantic hits via workspace-local BM25 chunk search over a gitignore-respecting corpus
|
||||
- persistent semantic retrieval from Qdrant when the explicit index is fresh
|
||||
- live `thread.epiphanyState` hydration backfills a lightweight retrieval summary when the thread has Epiphany state but no persisted retrieval metadata yet
|
||||
|
||||
### How this differs from plain Codex
|
||||
|
|
@ -258,6 +269,8 @@ What it does not have is one Epiphany-owned retrieval organ that says:
|
|||
- exact and semantic lookup are the same machine
|
||||
- results come back in a typed shape instead of as transcript confetti
|
||||
|
||||
What changed now is that the heavier part of that organ no longer lives entirely inside vendored Codex. The host seam is still patched into Codex, but the actual engine mostly lives in `epiphany-core`, which makes the ownership boundary less muddy and makes modified Codex alone a less complete rebuild kit.
|
||||
|
||||
### Why it matters
|
||||
|
||||
This is the first time Epiphany can ask a structured repo question without falling back to file-by-file terminal spelunking.
|
||||
|
|
@ -273,8 +286,8 @@ This slice is intentionally small and honest:
|
|||
- read-only
|
||||
- local BM25 rather than embeddings
|
||||
- no watcher-driven invalidation
|
||||
- no persistent vector store
|
||||
- no explicit retrieval update/write path yet
|
||||
- explicit indexing rather than hidden retrieval mutation
|
||||
- env-driven backend config for now
|
||||
|
||||
## What has *not* diverged yet
|
||||
|
||||
|
|
@ -312,7 +325,9 @@ input
|
|||
-> rollout + epiphany_state snapshot
|
||||
-> replay-aware epiphany restoration
|
||||
-> hydrated client thread payload with epiphanyState
|
||||
-> thin Codex adapters into repo-owned epiphany-core organs
|
||||
-> optional typed repo retrieval through thread/epiphany/retrieve
|
||||
-> optional explicit semantic indexing through thread/epiphany/index
|
||||
```
|
||||
|
||||
That is the real delta.
|
||||
|
|
@ -321,7 +336,7 @@ The transcript is no longer the only thing with continuity.
|
|||
|
||||
## Where the divergence is already meaningful
|
||||
|
||||
Even without invalidation or GUI, the divergence is already structurally important in four ways:
|
||||
Even without invalidation or GUI, the divergence is already structurally important in five ways:
|
||||
|
||||
1. **Understanding has a durable object now**
|
||||
- not just prose and replayed vibes
|
||||
|
|
@ -332,19 +347,22 @@ Even without invalidation or GUI, the divergence is already structurally importa
|
|||
3. **Clients can read that understanding directly**
|
||||
- not by scraping the prompt or reverse-engineering transcript fragments
|
||||
|
||||
4. **The harness has one typed repo retrieval organ now**
|
||||
4. **The harness has typed repo retrieval/indexing organs now**
|
||||
- not just shell archaeology plus vibes
|
||||
|
||||
5. **Most of the heavy Epiphany-owned implementation now lives outside vendored Codex**
|
||||
- not because plugins are fashionable, but because keeping the thicker organ repo-owned makes the boundary less muddy while preserving first-class typed integration
|
||||
|
||||
That is not the full Epiphany vision yet, but it is no longer cosmetic either.
|
||||
|
||||
## Recommended next companion note
|
||||
|
||||
If the next bounded slice lands, the useful follow-on section for this file is probably not "more retrieval exists."
|
||||
The useful follow-on section for this file is probably not "more retrieval exists."
|
||||
|
||||
It is more likely one of these:
|
||||
|
||||
- retrieval freshness/persistence semantics
|
||||
- structured observation promotion from retrieval/tool output into durable Epiphany state
|
||||
- Qdrant-backed persistent semantic indexing as a preferred backend, with BM25 fallback instead of a hard dependency
|
||||
- live health/config semantics for the explicit indexing path
|
||||
- eventual watcher-driven invalidation only if the current explicit path proves too manual
|
||||
|
||||
Those would be the next places where the machine stops being a better notebook and starts being a more opinionated harness.
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ Important consequence:
|
|||
|
||||
- `vendor/codex` is now tracked directly in the parent repo as ordinary files
|
||||
- it is **not** a submodule anymore
|
||||
- `epiphany-core/` is now a sibling repo-owned crate that holds the bulk of the Epiphany prompt, rollout replay, and retrieval/indexing logic instead of leaving that whole organ smeared through vendored Codex
|
||||
|
||||
Recent anchor commits:
|
||||
|
||||
|
|
@ -36,6 +37,22 @@ Canonical project state still lives in:
|
|||
- `state/scratch.md`
|
||||
- `state/evidence.jsonl`
|
||||
|
||||
## Licensing State
|
||||
|
||||
- the repo now has a top-level `LICENSE` file that acts as an operative
|
||||
repository license notice rather than a policy draft
|
||||
- `vendor/codex/**` remains governed by its upstream Apache-2.0 and related
|
||||
third-party notices
|
||||
- Project-Authored Material outside `vendor/codex/**` is publicly licensed
|
||||
under PolyForm Noncommercial 1.0.0, with separate commercial licensing
|
||||
intended by written agreement
|
||||
- external contributions require `CONTRIBUTOR_LICENSE_AGREEMENT.md` or a
|
||||
separate written agreement accepted by the Project Steward; contributors keep
|
||||
ownership but grant broad sublicensing/relicensing rights so future commercial
|
||||
licensing is not blocked by old contribution archaeology
|
||||
- do not describe the repository as a whole as OSI Open Source unless and until
|
||||
the project-authored material is also published under an OSI-approved license
|
||||
|
||||
Current implementation plan note:
|
||||
|
||||
- `notes/codex-epiphany-mode-plan.md`
|
||||
|
|
@ -50,10 +67,26 @@ Current architectural/spec notes:
|
|||
|
||||
Phase 4 repo-local hybrid retrieval is now landed on `main` at `360dfea` and pushed to `origin/main`. It is no longer a vague scaffold, a maybe, or a working-tree-only event.
|
||||
|
||||
Current landed implementation spans:
|
||||
Current verified working-tree follow-up after that landed baseline:
|
||||
|
||||
- explicit `thread/epiphany/index`
|
||||
- manifest-backed Qdrant semantic persistence under `codex_home`
|
||||
- local Ollama embeddings by default
|
||||
- stale/fresh retrieval summaries
|
||||
- read-only `thread/epiphany/retrieve` with BM25 fallback still intact
|
||||
- repo-owned `epiphany-core` extraction for the heavy prompt/replay/retrieval organ, leaving vendored Codex with thin adapters plus the typed host seam
|
||||
|
||||
Current verified working-tree implementation spans:
|
||||
|
||||
- `epiphany-core/Cargo.toml`
|
||||
- `epiphany-core/src/prompt.rs`
|
||||
- `epiphany-core/src/rollout.rs`
|
||||
- `epiphany-core/src/retrieval.rs`
|
||||
- `epiphany-core/src/lib.rs`
|
||||
- `vendor/codex/codex-rs/protocol/src/protocol.rs`
|
||||
- `vendor/codex/codex-rs/core/src/epiphany_retrieval.rs`
|
||||
- `vendor/codex/codex-rs/core/src/epiphany_rollout.rs`
|
||||
- `vendor/codex/codex-rs/core/src/context/epiphany_state_instructions.rs`
|
||||
- `vendor/codex/codex-rs/core/src/codex_thread.rs`
|
||||
- `vendor/codex/codex-rs/core/src/lib.rs`
|
||||
- `vendor/codex/codex-rs/core/Cargo.toml`
|
||||
|
|
@ -68,17 +101,32 @@ Current landed implementation spans:
|
|||
What is wired right now:
|
||||
|
||||
- `EpiphanyThreadState` already has retrieval metadata fields and types in protocol
|
||||
- `codex-core` now has a new `epiphany_retrieval.rs` module
|
||||
- the core retriever is query-time and hybrid:
|
||||
- repo-owned `epiphany-core` now owns:
|
||||
- the prompt-state renderer used by Phase 2 prompt injection
|
||||
- the rollout replay helper used by Phase 3 stored-thread hydration and rollback/compaction-aware reads
|
||||
- the heavy retrieval/indexing engine used by the current Phase 4 work
|
||||
- vendored `codex-core` now keeps thin adapters at:
|
||||
- `core/src/epiphany_retrieval.rs`
|
||||
- `core/src/epiphany_rollout.rs`
|
||||
- `core/src/context/epiphany_state_instructions.rs`
|
||||
- the retrieval machine is still hybrid:
|
||||
- exact/path-ish lookup via `codex_file_search`
|
||||
- semantic lookup via BM25 chunk search over a local gitignore-respecting corpus
|
||||
- `CodexThread` now exposes:
|
||||
- `epiphany_retrieval_state()`
|
||||
- `epiphany_retrieve(...)`
|
||||
- `epiphany_index(...)`
|
||||
- app-server protocol now declares experimental `thread/epiphany/retrieve`
|
||||
- app-server message handling now routes that method to the core retriever
|
||||
- app-server protocol now also declares experimental `thread/epiphany/index`
|
||||
- app-server message handling now routes those methods through the core host seam
|
||||
- live hydrated `thread.epiphanyState` reads now backfill a lightweight retrieval summary when the thread already has Epiphany state but no persisted retrieval summary yet
|
||||
- basic protocol/core/app-server unit tests were added for the new types and mapping layer
|
||||
- the current verified working tree additionally wires:
|
||||
- Qdrant-backed semantic indexing behind an explicit write path instead of hidden retrieval mutation
|
||||
- manifest-backed freshness checking under `codex_home`
|
||||
- local Ollama embeddings by default, with support for adjacent VoidBot-style env names as fallbacks
|
||||
- persistent semantic retrieval preference when the index is fresh, and BM25 fallback when it is stale, missing, or unavailable
|
||||
- modified Codex alone is no longer the whole Epiphany organ; the heavier implementation now lives in the sibling crate and vendored Codex mostly hosts typed seams and adapters
|
||||
|
||||
Important recovery facts:
|
||||
|
||||
|
|
@ -107,12 +155,14 @@ Updated follow-up direction:
|
|||
|
||||
What got verified:
|
||||
|
||||
- `cargo fmt --manifest-path E:\Projects\EpiphanyAgent\epiphany-core\Cargo.toml` passed
|
||||
- `cargo test --manifest-path E:\Projects\EpiphanyAgent\epiphany-core\Cargo.toml` passed
|
||||
- `cargo fmt --all` passed
|
||||
- `cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run` passed with `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex`
|
||||
- targeted Phase 4 tests passed:
|
||||
- `cargo test -p codex-core --lib retrieve_workspace_`
|
||||
- `cargo test -p codex-app-server-protocol --lib thread_epiphany_retrieve`
|
||||
- `cargo test -p codex-app-server --lib map_epiphany_retrieve_response_preserves_summary_and_results`
|
||||
- `cargo test -p codex-core --lib epiphany`
|
||||
- `cargo test -p codex-app-server-protocol --lib thread_epiphany_`
|
||||
- `cargo test -p codex-app-server --lib map_epiphany_`
|
||||
- full `cargo test -p codex-app-server-protocol` passed after regenerating stable schema fixtures
|
||||
|
||||
Notable verification fixes:
|
||||
|
|
@ -156,7 +206,7 @@ Main touched files:
|
|||
|
||||
Phase 2 is done too.
|
||||
|
||||
The Phase 2 prompt-integration slice was implemented inside vendored Codex:
|
||||
The Phase 2 prompt-integration slice was originally implemented inside vendored Codex and now renders through the repo-owned prompt crate with a thin wrapper in `codex-core`:
|
||||
|
||||
- `EpiphanyStateInstructions` renders a bounded developer fragment from `EpiphanyThreadState`
|
||||
- the fragment is wrapped in `<epiphany_state> ... </epiphany_state>`
|
||||
|
|
@ -166,6 +216,7 @@ The Phase 2 prompt-integration slice was implemented inside vendored Codex:
|
|||
|
||||
Main touched files for Phase 2:
|
||||
|
||||
- `epiphany-core/src/prompt.rs`
|
||||
- `vendor/codex/codex-rs/core/src/context/mod.rs`
|
||||
- `vendor/codex/codex-rs/core/src/context/epiphany_state_instructions.rs`
|
||||
- `vendor/codex/codex-rs/core/src/session/mod.rs`
|
||||
|
|
@ -185,6 +236,7 @@ What landed:
|
|||
|
||||
Main touched files for Phase 3:
|
||||
|
||||
- `epiphany-core/src/rollout.rs`
|
||||
- `vendor/codex/codex-rs/core/src/epiphany_rollout.rs`
|
||||
- `vendor/codex/codex-rs/core/src/lib.rs`
|
||||
- `vendor/codex/codex-rs/core/src/codex_thread.rs`
|
||||
|
|
@ -221,6 +273,9 @@ If a future session wakes up from compaction and starts bluffing, this is the pa
|
|||
- The current landed phase is **Phase 4 slice 1 repo-local hybrid retrieval**:
|
||||
- Epiphany now has a real typed repo retrieval subsystem instead of repeated file-by-file shell archaeology
|
||||
- the next bounded follow-up is persistent semantic indexing, not proving the retriever exists again
|
||||
- The current working tree also extracted most Epiphany-owned implementation into `epiphany-core`:
|
||||
- prompt rendering, rollout replay, and retrieval/indexing logic are now repo-owned
|
||||
- vendored Codex keeps the typed protocol/thread/app-server seam plus thin adapters
|
||||
- Phase 4 slice 1 is now landed on `main` and was verified before landing:
|
||||
- protocol/core/app-server wiring exists and passed targeted verification
|
||||
- stable app-server protocol schema fixtures were regenerated
|
||||
|
|
@ -230,6 +285,9 @@ If a future session wakes up from compaction and starts bluffing, this is the pa
|
|||
- semantic lookup via local BM25 chunk search
|
||||
- query-time, read-only, loaded-thread-first behavior
|
||||
- no embeddings or persistent vector store yet
|
||||
- The recovered boundary choice for the current follow-up is:
|
||||
- keep the protocol/thread/app-server seam inside vendored Codex
|
||||
- move heavy Epiphany-owned implementation into the sibling crate when that can be done without giving up first-class typed integration
|
||||
- Important Windows verification footgun still stands:
|
||||
- use `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex` for `codex-core` work on this machine
|
||||
- Snapshot hygiene note:
|
||||
|
|
@ -301,13 +359,18 @@ The durable state seam exists, the turn loop reads it, clients can load typed Ep
|
|||
Current next implementation move:
|
||||
|
||||
1. keep the current verified-and-landed query-time hybrid retriever as the Phase 4 slice 1 baseline
|
||||
2. after reboot and Qdrant availability, keep the next follow-up narrow:
|
||||
- retrieval persistence/freshness semantics
|
||||
- probably via a dedicated Epiphany update path
|
||||
- Qdrant-backed persistent semantic indexing as the preferred backend, with BM25 fallback for no-Qdrant or degraded mode
|
||||
- not GUI work
|
||||
- not watcher-driven invalidation
|
||||
3. do not casually add durable retrieval-summary writes to `thread/epiphany/retrieve`
|
||||
2. keep the current verified working-tree `thread/epiphany/index` follow-up bounded:
|
||||
- explicit write path
|
||||
- Qdrant-backed semantic persistence
|
||||
- local Ollama embeddings by default
|
||||
- BM25 fallback still alive
|
||||
3. keep the new `epiphany-core` boundary honest:
|
||||
- heavy Epiphany-owned logic in the sibling crate
|
||||
- thin adapters and typed seams in vendored Codex
|
||||
4. next practical move:
|
||||
- smoke the explicit indexing path against a live loaded thread and the real local services
|
||||
- then decide whether env-scoped backend config is good enough for now or deserves a cleaner surface
|
||||
5. do not casually add durable retrieval-summary writes to `thread/epiphany/retrieve`
|
||||
|
||||
## What Not To Do Next
|
||||
|
||||
|
|
@ -321,7 +384,7 @@ Not yet:
|
|||
|
||||
The next slice should stay small and mean:
|
||||
|
||||
- add persistent semantic retrieval/indexing scaffolding
|
||||
- harden the explicit persistent semantic retrieval/indexing path
|
||||
- keep GUI, mutation, and specialist-agent logic dark
|
||||
|
||||
## Verification Commands
|
||||
|
|
|
|||
|
|
@ -54,3 +54,16 @@
|
|||
{"ts": "2026-04-24T02:11:38+01:00", "type": "decision", "status": "ok", "note": "Adopted Qdrant as the intended first persistent semantic retrieval backend after the verified BM25 baseline. The rationale is practical rather than fashionable: the user is already moving away from a blob-thrashing Postgres/JSON embedding store in adjacent infrastructure, so Epiphany should target a real vector DB for persistent semantic indexing while keeping exact/path search first-class and retaining BM25 as bootstrap/fallback/control rather than going vector-only."}
|
||||
{"ts": "2026-04-24T02:43:41+01:00", "type": "decision", "status": "ok", "note": "Resolved the remaining backend policy fork before shipping Phase 4 slice 1: Qdrant is the preferred later persistent semantic backend, but not a hard runtime requirement for basic Epiphany use. The verified BM25 retriever remains the no-hassle bootstrap/fallback/control path for users who do not want Qdrant or when Qdrant is unavailable."}
|
||||
{"ts": "2026-04-24T03:21:18+01:00", "type": "verification", "status": "ok", "note": "Collapsed the temporary phase branch after push so the landed Phase 4 slice now lives directly on main at 360dfea with a clean worktree. Synced map, scratch, plan, and handoff state so the persisted next action is the post-reboot Qdrant-backed persistent-semantic follow-up rather than stale push prep."}
|
||||
{"ts": "2026-04-24T07:48:38+01:00", "type": "implementation", "status": "ok", "note": "Built the next bounded Phase 4 persistent-semantic follow-up in the working tree. Retrieval stays read-only, a new experimental thread/epiphany/index method now drives explicit semantic indexing, codex-core persists Qdrant index manifests under codex_home, local Ollama embeddings default to qwen3-embedding:0.6b, and retrieval now prefers a fresh Qdrant index while preserving exact/path search and BM25 fallback."}
|
||||
{"ts": "2026-04-24T07:48:38+01:00", "type": "verification", "status": "ok", "note": "Verified the explicit Qdrant follow-up with the Windows CARGO_TARGET_DIR workaround. cargo fmt --all passed; cargo test -p codex-core --lib epiphany_retrieval passed; cargo test -p codex-app-server-protocol --lib thread_epiphany_ passed; cargo test -p codex-app-server --lib map_epiphany_ passed; full cargo test -p codex-app-server-protocol passed; and cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run passed on the final working tree."}
|
||||
{"ts": "2026-04-24T07:48:38+01:00", "type": "decision", "status": "ok", "note": "Kept the persistent-semantic follow-up mechanically honest: no durable EpiphanyState writes were bolted onto thread/epiphany/retrieve, the new Qdrant path only mutates the world through explicit thread/epiphany/index, persistent metadata lives under codex_home instead of workspace sludge, and backend config currently stays env-scoped with support for adjacent VoidBot-style env names as fallbacks."}
|
||||
{"ts": "2026-04-24T12:35:59+01:00", "type": "decision", "status": "ok", "note": "Drafted a top-level repository licensing policy instead of pretending the vendored Apache surface and the planned Epiphany dual-license model would sort themselves out by osmosis. The new LICENSE file says upstream Codex and other third-party material keep their original licenses, original Epiphany-authored material is intended for a personal-use / commercial dual-license model, mixed upstream files do not lose Apache rights, and the repository as a whole should not be described as OSI Open Source unless the project-authored material is later released under an OSI-approved license too."}
|
||||
{"ts": "2026-04-24T12:37:21+01:00", "type": "research", "status": "ok", "note": "Grounded the dual-license draft against published models instead of improvising in a vacuum. The LICENSE file now explicitly cites Qt's open-source/commercial dual-licensing overview and Oracle's MySQL GPL/commercial dual-license model as structural references, while also stating honestly that Epiphany's planned personal-use / commercial split is an adaptation of that broader pattern rather than a claim that those projects use the same exact terms."}
|
||||
{"ts": "2026-04-24T13:20:06+01:00", "type": "implementation", "status": "ok", "note": "Extracted the heavy Epiphany-owned prompt and retrieval/indexing logic into a repo-owned sibling crate at epiphany-core. epiphany-core/src/prompt.rs now renders Epiphany thread state, epiphany-core/src/retrieval.rs owns the hybrid BM25/Qdrant/Ollama retrieval and indexing engine, and vendored codex-core keeps thin adapters plus the typed protocol/thread/app-server seams."}
|
||||
{"ts": "2026-04-24T13:20:06+01:00", "type": "verification", "status": "ok", "note": "Verified the new extraction boundary cleanly. cargo fmt --manifest-path .\\\\epiphany-core\\\\Cargo.toml passed; cargo test --manifest-path .\\\\epiphany-core\\\\Cargo.toml passed; cargo fmt --all passed in vendor/codex/codex-rs; CARGO_TARGET_DIR=C:\\\\Users\\\\Meta\\\\.cargo-target-codex cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run passed; and targeted codex-core/app-server-protocol/app-server epiphany tests still passed after swapping vendored logic for thin wrappers and a path dependency."}
|
||||
{"ts": "2026-04-24T13:44:09+01:00", "type": "implementation", "status": "ok", "note": "Pulled the Phase 3 rollout replay helper into epiphany-core too. epiphany-core/src/rollout.rs now owns latest_epiphany_state_from_rollout_items as generic Epiphany bookkeeping, while vendored codex-core/src/epiphany_rollout.rs is reduced to a thin wrapper that provides Codex's user-turn-boundary predicate."}
|
||||
{"ts": "2026-04-24T13:44:09+01:00", "type": "verification", "status": "ok", "note": "Re-verified after moving the rollout helper. cargo fmt --manifest-path .\\\\epiphany-core\\\\Cargo.toml passed; cargo fmt --all passed in vendor/codex/codex-rs; cargo test --manifest-path .\\\\epiphany-core\\\\Cargo.toml passed with 9 tests; CARGO_TARGET_DIR=C:\\\\Users\\\\Meta\\\\.cargo-target-codex cargo test -p codex-core --lib epiphany passed; CARGO_TARGET_DIR=C:\\\\Users\\\\Meta\\\\.cargo-target-codex cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run passed; targeted app-server and app-server-protocol Epiphany tests passed too."}
|
||||
{"ts": "2026-04-24T13:57:09+01:00", "type": "research", "status": "ok", "note": "Grounded the tightened repository license notice against primary sources instead of inventing another bespoke pseudo-license. The rewrite uses PolyForm Noncommercial 1.0.0 as the operative public license for Project-Authored Material outside vendor/codex, preserves Apache-2.0 boundaries for vendored Codex per ASF guidance on modified Apache distributions, references Qt and MySQL only as structural dual-license examples, and records the Copyright Office / 17 U.S.C. 411 rule that copyright exists automatically but registration is still required before a U.S. infringement suit on a U.S. work."}
|
||||
{"ts": "2026-04-24T13:57:09+01:00", "type": "implementation", "status": "ok", "note": "Replaced the root LICENSE draft with an operative repository license notice. The new text clearly separates upstream material from Project-Authored Material, pins the public path for non-vendored project-owned material to PolyForm Noncommercial 1.0.0 with the full text included in the file, states the separate written commercial-license path, makes clear that lack of a corporate entity does not forfeit copyright ownership or licensing ability, and warns that mixed Apache files under vendor/codex are not magically converted into a PolyForm-only codebase."}
|
||||
{"ts": "2026-04-24T14:10:00+01:00", "type": "research", "status": "ok", "note": "Checked established CLA structures before adding Epiphany's contributor agreement. Apache ICLA, Project Harmony, and Canonical all use the same broad pattern that fits this project: contributors retain ownership while granting the project wide rights to distribute, sublicense, and build on contributions; Harmony explicitly documents outbound-license options including broad relicensing, and Canonical documents the rationale that contributors keep copyright while granting the project permission to use contributions."}
|
||||
{"ts": "2026-04-24T14:10:00+01:00", "type": "implementation", "status": "ok", "note": "Added CONTRIBUTOR_LICENSE_AGREEMENT.md and CONTRIBUTING.md, then updated LICENSE, README, and handoff notes. The CLA keeps contributor ownership but grants the Project Steward perpetual, worldwide, no-charge, royalty-free, irrevocable, transferable, and sublicensable copyright and patent rights, including the right to relicense contributions under open-source, source-available, noncommercial, commercial, proprietary, dual-license, or future terms. Contributions are now documented as unacceptable unless submitted under the CLA or an equivalent written agreement."}
|
||||
|
|
|
|||
|
|
@ -45,9 +45,10 @@ data_flow:
|
|||
- commit or backtrack
|
||||
|
||||
active_subgoals:
|
||||
- Keep the landed Phase 4 slice as the retrieval baseline and keep the repo memory aligned with reality.
|
||||
- After reboot and Qdrant availability, design the next bounded persistent semantic follow-up without widening the current machine casually.
|
||||
- Preserve exact/path lookup and BM25 fallback while treating Qdrant as the preferred persistent semantic backend.
|
||||
- Keep the extracted Phase 4 persistent-semantic follow-up aligned with reality in both code and repo memory.
|
||||
- Keep vendored Codex as a thin typed host seam while Epiphany-owned prompt and retrieval/indexing logic live in the repo-owned `epiphany-core` crate.
|
||||
- Preserve exact/path lookup and BM25 fallback while the Qdrant path stays explicit and bounded.
|
||||
- Smoke the new explicit indexing path against a live loaded thread and decide whether backend config should stay env-scoped or grow a cleaner first-class surface later.
|
||||
|
||||
accepted_design:
|
||||
- external typed state files
|
||||
|
|
@ -80,6 +81,9 @@ accepted_design:
|
|||
- repo-local AGENTS instructions should carry the session bootstrap and re-entry protocol explicitly, including running epiphany_state.py status on fresh load and after compaction
|
||||
- implement the first Phase 4 retrieval slice as a query-time hybrid retriever: reuse existing codex_file_search substrate for exact/path lookup and use workspace-local BM25 chunk search as the first semantic backend so the retrieval seam is proven before adding persistence
|
||||
- for the first persistent semantic retrieval store after the verified BM25 baseline, prefer a real vector database such as Qdrant over JSON/blob/postgres-thrash designs; treat Qdrant as the preferred persistent semantic backend rather than a hard requirement, keep exact/path lookup as a first-class organ, and keep BM25 available as a bootstrap/fallback/control path when Qdrant is unavailable or not configured instead of turning retrieval into vector-only soup
|
||||
- land the first persistent-semantic follow-up as an explicit indexing/update path rather than a hidden side effect in retrieval: keep `thread/epiphany/retrieve` read-only, add additive `thread/epiphany/index`, persist Qdrant index metadata under `codex_home`, use local Ollama embeddings by default, and fall back to BM25 whenever the persistent backend is stale, missing, or unavailable
|
||||
- extract Epiphany-owned implementation into a sibling repo-owned crate where possible, leaving vendored Codex with thin host adapters plus the typed protocol/thread/app-server seams that make the integration first-class
|
||||
- keep the licensing path compatible with future commercial use: Project-Authored Material outside `vendor/codex/**` is public under PolyForm Noncommercial 1.0.0 plus separate commercial terms, and outside contributions require a CLA granting broad sublicensing/relicensing rights
|
||||
|
||||
rejected_paths:
|
||||
- name: pinned_map_only
|
||||
|
|
@ -96,5 +100,5 @@ rejected_paths:
|
|||
reason: audited candidates were either terminal-only, stale/aspirational, or incoherent wrappers; the comparison clones were removed from vendor to avoid building on untrusted drift
|
||||
|
||||
current_status:
|
||||
summary: Workspace initialized with protocol files, typed state, a tiny CLI, a git repo, fresh-workspace handoff notes, architecture rationale, a Codex algorithmic map note, an Epiphany core-harness surface spec, and a rolling implementation plan against vendored Codex. Phase 1 durable Epiphany state, Phase 2 prompt integration, and a minimal Phase 3 typed client read surface are all implemented and verified with a real Rust/MSVC toolchain; vendored Codex is tracked directly in the parent repo; app-server Thread payloads can now carry typed Epiphany state without scraping prompt text; pre-compaction persistence is now an explicit part of the intended Epiphany workflow; and the first bounded Phase 4 repo-local hybrid retrieval slice is now landed on `main` at `360dfea`, including stable schema fixture regeneration and the Windows `CARGO_TARGET_DIR` verification path.
|
||||
next_action: After reboot and Qdrant availability, prototype the next bounded persistent semantic retrieval follow-up as a Qdrant-backed backend with explicit freshness/indexing semantics while keeping the landed query-time BM25 retriever as fallback and exact/path lookup first-class. Do not bolt durable EpiphanyState writes onto `thread/epiphany/retrieve` without a clean out-of-band rollout semantic.
|
||||
summary: Workspace initialized with protocol files, typed state, a tiny CLI, a git repo, fresh-workspace handoff notes, architecture rationale, a Codex algorithmic map note, an Epiphany core-harness surface spec, and a rolling implementation plan against vendored Codex. Phase 1 durable Epiphany state, Phase 2 prompt integration, a minimal Phase 3 typed client read surface, and the landed Phase 4 slice 1 query-time hybrid retriever on `main` at `360dfea` are all intact. The current verified working tree now carries the next bounded Phase 4 follow-up too: explicit `thread/epiphany/index`, manifest-backed Qdrant semantic persistence under `codex_home`, local Ollama embeddings by default, stale/fresh retrieval metadata, and read-only retrieval that still preserves exact/path search plus BM25 fallback instead of turning the machine into vector-only soup. The heavy Epiphany-owned prompt renderer, rollout replay helper, and retrieval/indexing engine now live in the repo-owned `epiphany-core` crate, while vendored Codex carries thin adapters and the typed host seam. The repo also now has an operative source-available/commercial licensing notice and a CLA gate for external contributions.
|
||||
next_action: Exercise the explicit indexing path against a live loaded thread with the real local Qdrant and Ollama services now that the heavy prompt/retrieval organs live in `epiphany-core`. Keep vendored Codex thin, keep `thread/epiphany/retrieve` read-only, and only revisit backend configuration if the current env-scoped surface proves too feral during the live smoke.
|
||||
|
|
|
|||
|
|
@ -4,67 +4,54 @@ This file is intentionally disposable.
|
|||
|
||||
## Current Subgoal
|
||||
|
||||
- Rebase the repo memory onto the landed Phase 4 baseline and make the next Qdrant follow-up explicit.
|
||||
- Keep the new `epiphany-core` extraction honest in repo memory, then smoke the explicit indexing path against live Qdrant/Ollama without widening the machine again.
|
||||
|
||||
## Working Notes
|
||||
|
||||
- The first bounded Phase 4 slice is landed on `main` at `360dfea`. The landed retrieval wiring spans:
|
||||
- `vendor/codex/codex-rs/protocol/src/protocol.rs`
|
||||
- `vendor/codex/codex-rs/core/src/epiphany_retrieval.rs`
|
||||
- `vendor/codex/codex-rs/core/src/codex_thread.rs`
|
||||
- `vendor/codex/codex-rs/core/src/lib.rs`
|
||||
- `vendor/codex/codex-rs/core/Cargo.toml`
|
||||
- `vendor/codex/codex-rs/core/src/session/tests.rs`
|
||||
- `vendor/codex/codex-rs/app-server-protocol/src/protocol/common.rs`
|
||||
- `vendor/codex/codex-rs/app-server-protocol/src/protocol/v2.rs`
|
||||
- `vendor/codex/codex-rs/app-server-protocol/src/export.rs`
|
||||
- `vendor/codex/codex-rs/app-server/src/codex_message_processor.rs`
|
||||
- `vendor/codex/codex-rs/app-server/README.md`
|
||||
- regenerated stable fixtures under `vendor/codex/codex-rs/app-server-protocol/schema/`
|
||||
- The current implementation shape is the bounded one we wanted:
|
||||
- The landed Phase 4 slice 1 baseline is still `360dfea` on `main`. The current working tree still carries the explicit `thread/epiphany/index` Qdrant follow-up on top of that.
|
||||
- The extraction boundary is now the important truth:
|
||||
- `epiphany-core/src/retrieval.rs` owns the heavy hybrid retrieval/indexing engine
|
||||
- `epiphany-core/src/prompt.rs` owns the Epiphany prompt-state renderer
|
||||
- `epiphany-core/src/rollout.rs` owns the latest-Epiphany-state replay helper used for Phase 3 stored-thread hydration
|
||||
- `epiphany-core/src/lib.rs` re-exports the stable surface used by vendored Codex
|
||||
- `vendor/codex/codex-rs/core/src/epiphany_retrieval.rs` is now a thin re-export wrapper
|
||||
- `vendor/codex/codex-rs/core/src/epiphany_rollout.rs` is now a thin wrapper that supplies the Codex-specific user-turn-boundary predicate
|
||||
- `vendor/codex/codex-rs/core/src/context/epiphany_state_instructions.rs` is now a thin `ContextualUserFragment` adapter around `epiphany_core::render_epiphany_state(...)`
|
||||
- `vendor/codex/codex-rs/core/Cargo.toml` now depends on the sibling crate by path
|
||||
- What still must live in vendored Codex because that is where the typed host seam exists:
|
||||
- protocol types in `vendor/codex/codex-rs/protocol/src/protocol.rs`
|
||||
- `CodexThread` bridge methods in `vendor/codex/codex-rs/core/src/codex_thread.rs`
|
||||
- the user-turn-boundary rule in `vendor/codex/codex-rs/core/src/context_manager/history.rs`
|
||||
- app-server protocol surfaces in `vendor/codex/codex-rs/app-server-protocol/src/protocol/common.rs` and `vendor/codex/codex-rs/app-server-protocol/src/protocol/v2.rs`
|
||||
- request routing in `vendor/codex/codex-rs/app-server/src/codex_message_processor.rs`
|
||||
- Why this shape is worth keeping:
|
||||
- it shrinks the Apache-mixed surface
|
||||
- it makes modified Codex alone less useful as a standalone Epiphany rebuild kit
|
||||
- it preserves first-class typed integration instead of hiding the machine behind an opaque plugin blob
|
||||
- Retrieval/indexing behavior is still the bounded machine we wanted:
|
||||
- exact/path-ish lookup via `codex_file_search`
|
||||
- semantic lookup via query-time BM25 chunk search in the new core retrieval module
|
||||
- additive experimental app-server method `thread/epiphany/retrieve`
|
||||
- lightweight retrieval-summary backfill for live `thread.epiphanyState` when the thread already has Epiphany state but no persisted retrieval summary yet
|
||||
- `codex-core` now depends on both `bm25` and `codex-file-search`; `ignore` is used to walk a gitignore-respecting local corpus for semantic chunking.
|
||||
- The semantic side is intentionally small:
|
||||
- text-only files
|
||||
- size cap per file
|
||||
- fixed line-window chunking with overlap
|
||||
- no embeddings, no persistent blob, no watcher invalidation
|
||||
- Corpus preflight for the tracked `vendor/codex/codex-rs` source at `vendor/codex` HEAD `d45ab10`: `3642` tracked files, `31.09 MB`.
|
||||
- Do not use raw working-tree size as the retrieval denominator here; build artifacts and other debris inflate it badly and will lie to the design.
|
||||
- Verification is no longer theoretical:
|
||||
- `cargo fmt --all` passed
|
||||
- `cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run` passed with `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex`
|
||||
- targeted Phase 4 tests passed in core, app-server protocol, and app-server
|
||||
- full `cargo test -p codex-app-server-protocol` passed after regenerating stable schema fixtures
|
||||
- the slice is now committed and pushed on `main` as `360dfea`
|
||||
- Small fallout that had to be patched:
|
||||
- `codex_message_processor.rs` had an unnecessary `live_thread.as_ref()`
|
||||
- `core/src/session/tests.rs` needed `retrieval: None` in the prompt fixture Epiphany state
|
||||
- the exact-hit retrieval test fixture needed an actually matchable path (`session_checkpoint.rs`)
|
||||
- `EpiphanyThreadState` needed an explicit exemption in the protocol export test because it is a sparse durable state object, not a params bag
|
||||
- Important schema footgun:
|
||||
- `cargo run -p codex-app-server-protocol --bin write_schema_fixtures -- --experimental` rewrites the same checked-in stable schema tree
|
||||
- leaving the repo in that state makes stable protocol tests fail
|
||||
- the checked-in repo state must stay on the stable fixture generation path
|
||||
- despite the giant `git status` scream wall, the real schema content diff is small and expected:
|
||||
- `git diff --numstat` shows actual content changes in `15` generated schema files
|
||||
- plus the newly tracked Epiphany TypeScript schema files
|
||||
- the broader wall is mostly line-ending/worktree noise, not extra logical surface area
|
||||
- After inspecting the live code path, do not widen `thread/epiphany/retrieve` into a durable Epiphany-state write right now:
|
||||
- durable `EpiphanyState` snapshots are currently persisted on real user-turn boundaries
|
||||
- making retrieval calls append durable snapshots would require a new out-of-band rollout semantic or a fake turn boundary
|
||||
- that is a bigger machine than this slice deserves
|
||||
- New infrastructure direction:
|
||||
- for the first persistent semantic backend after this verified BM25 slice, use Qdrant instead of inventing another monolithic JSON/blob/postgres-embedding store
|
||||
- Qdrant should be the preferred persistent semantic backend later, not a hard runtime requirement for basic Epiphany use
|
||||
- exact/path lookup stays first-class
|
||||
- BM25 should stay available as a bootstrap/fallback/control path for users who do not want Qdrant or when Qdrant is unavailable
|
||||
- semantic lookup via BM25 chunk search when persistent semantic state is unavailable or stale
|
||||
- explicit `thread/epiphany/index` for Qdrant-backed semantic persistence
|
||||
- read-only `thread/epiphany/retrieve`
|
||||
- manifest metadata under `codex_home`
|
||||
- local Ollama embeddings defaulting to `qwen3-embedding:0.6b`
|
||||
- Corpus preflight for the tracked `vendor/codex/codex-rs` source at `vendor/codex` HEAD `d45ab10`: `3642` tracked files / `31.09 MB`.
|
||||
- Do not use raw working-tree size as the retrieval denominator here; build artifacts and other sludge lie.
|
||||
- Verification is now split cleanly across the repo-owned crate and the vendored host:
|
||||
- `cargo fmt --manifest-path .\\epiphany-core\\Cargo.toml`
|
||||
- `cargo test --manifest-path .\\epiphany-core\\Cargo.toml`
|
||||
- `cargo fmt --all`
|
||||
- `cargo test -p codex-core --lib epiphany`
|
||||
- `cargo test -p codex-app-server-protocol --lib thread_epiphany_`
|
||||
- `cargo test -p codex-app-server --lib map_epiphany_`
|
||||
- `cargo test -p codex-core -p codex-app-server-protocol -p codex-app-server --lib --no-run` with `CARGO_TARGET_DIR=C:\Users\Meta\.cargo-target-codex`
|
||||
- Mechanical honesty still matters:
|
||||
- do not widen `thread/epiphany/retrieve` into a durable Epiphany-state write without a clean out-of-band rollout/update semantic
|
||||
- do not let `epiphany-core` sprawl into GUI or watcher machinery just because it now owns the bigger organ
|
||||
|
||||
## Open Questions
|
||||
|
||||
- When retrieval freshness/persistence semantics come later, should they ride a dedicated Epiphany update path instead of piggybacking on a read/query method?
|
||||
- After the explicit indexing path has been live-smoked, should backend config stay env-scoped for a while or earn a cleaner first-class config surface?
|
||||
- How much more can move into `epiphany-core` without sacrificing the typed Codex host seam that makes the integration first-class?
|
||||
|
||||
Do not promote anything from here into the map unless it survives verification or repeated reuse without contradiction.
|
||||
|
|
|
|||
29
vendor/codex/codex-rs/Cargo.lock
generated
vendored
29
vendor/codex/codex-rs/Cargo.lock
generated
vendored
|
|
@ -1614,9 +1614,9 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "chrono"
|
||||
version = "0.4.43"
|
||||
version = "0.4.44"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "fac4744fb15ae8337dc853fee7fb3f4e48c0fbaa23d0afe49c447b4fab126118"
|
||||
checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0"
|
||||
dependencies = [
|
||||
"iana-time-zone",
|
||||
"js-sys",
|
||||
|
|
@ -2394,6 +2394,7 @@ dependencies = [
|
|||
"dirs",
|
||||
"dunce",
|
||||
"env-flags",
|
||||
"epiphany-core",
|
||||
"eventsource-stream",
|
||||
"futures",
|
||||
"http 1.4.0",
|
||||
|
|
@ -5076,6 +5077,24 @@ dependencies = [
|
|||
"log",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "epiphany-core"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"bm25",
|
||||
"chrono",
|
||||
"codex-file-search",
|
||||
"codex-protocol",
|
||||
"ignore",
|
||||
"reqwest",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha1",
|
||||
"tracing",
|
||||
"uuid",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "equivalent"
|
||||
version = "1.0.2"
|
||||
|
|
@ -8646,7 +8665,7 @@ version = "5.0.0"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d"
|
||||
dependencies = [
|
||||
"base64 0.21.7",
|
||||
"base64 0.22.1",
|
||||
"chrono",
|
||||
"getrandom 0.2.17",
|
||||
"http 1.4.0",
|
||||
|
|
@ -9114,7 +9133,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||
checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"windows-sys 0.45.0",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -13748,7 +13767,7 @@ version = "0.1.11"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
|
||||
dependencies = [
|
||||
"windows-sys 0.48.0",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
|
|||
|
|
@ -336,6 +336,11 @@ client_request_definitions! {
|
|||
params: v2::ThreadReadParams,
|
||||
response: v2::ThreadReadResponse,
|
||||
},
|
||||
#[experimental("thread/epiphany/index")]
|
||||
ThreadEpiphanyIndex => "thread/epiphany/index" {
|
||||
params: v2::ThreadEpiphanyIndexParams,
|
||||
response: v2::ThreadEpiphanyIndexResponse,
|
||||
},
|
||||
#[experimental("thread/epiphany/retrieve")]
|
||||
ThreadEpiphanyRetrieve => "thread/epiphany/retrieve" {
|
||||
params: v2::ThreadEpiphanyRetrieveParams,
|
||||
|
|
@ -1634,6 +1639,68 @@ mod tests {
|
|||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serialize_thread_epiphany_index_response() -> Result<()> {
|
||||
let response = ClientResponse::ThreadEpiphanyIndex {
|
||||
request_id: RequestId::Integer(9),
|
||||
response: v2::ThreadEpiphanyIndexResponse {
|
||||
index_summary: v2::ThreadEpiphanyRetrieveIndexSummary {
|
||||
workspace_root: absolute_path("/workspace"),
|
||||
index_revision: Some("qdrant-ollama-v1:qwen3-embedding:0.6b".to_string()),
|
||||
status: codex_protocol::protocol::EpiphanyRetrievalStatus::Ready,
|
||||
semantic_available: true,
|
||||
last_indexed_at_unix_seconds: Some(1_744_500_100),
|
||||
indexed_file_count: Some(12),
|
||||
indexed_chunk_count: Some(34),
|
||||
shards: vec![v2::ThreadEpiphanyRetrieveShardSummary {
|
||||
shard_id: "workspace".to_string(),
|
||||
path_prefix: PathBuf::from("."),
|
||||
indexed_file_count: Some(12),
|
||||
indexed_chunk_count: Some(34),
|
||||
status: codex_protocol::protocol::EpiphanyRetrievalStatus::Ready,
|
||||
exact_available: true,
|
||||
semantic_available: true,
|
||||
}],
|
||||
dirty_paths: Vec::new(),
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
assert_eq!(response.id(), &RequestId::Integer(9));
|
||||
assert_eq!(response.method(), "thread/epiphany/index");
|
||||
assert_eq!(
|
||||
json!({
|
||||
"method": "thread/epiphany/index",
|
||||
"id": 9,
|
||||
"response": {
|
||||
"indexSummary": {
|
||||
"workspaceRoot": absolute_path_string("workspace"),
|
||||
"indexRevision": "qdrant-ollama-v1:qwen3-embedding:0.6b",
|
||||
"status": "ready",
|
||||
"semanticAvailable": true,
|
||||
"lastIndexedAtUnixSeconds": 1744500100,
|
||||
"indexedFileCount": 12,
|
||||
"indexedChunkCount": 34,
|
||||
"shards": [
|
||||
{
|
||||
"shardId": "workspace",
|
||||
"pathPrefix": ".",
|
||||
"indexedFileCount": 12,
|
||||
"indexedChunkCount": 34,
|
||||
"status": "ready",
|
||||
"exactAvailable": true,
|
||||
"semanticAvailable": true
|
||||
}
|
||||
],
|
||||
"dirtyPaths": []
|
||||
}
|
||||
}
|
||||
}),
|
||||
serde_json::to_value(&response)?,
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serialize_config_requirements_read() -> Result<()> {
|
||||
let request = ClientRequest::ConfigRequirementsRead {
|
||||
|
|
@ -2145,6 +2212,19 @@ mod tests {
|
|||
assert_eq!(reason, Some("thread/epiphany/retrieve"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn thread_epiphany_index_is_marked_experimental() {
|
||||
let request = ClientRequest::ThreadEpiphanyIndex {
|
||||
request_id: RequestId::Integer(1),
|
||||
params: v2::ThreadEpiphanyIndexParams {
|
||||
thread_id: "thr_123".to_string(),
|
||||
force_full_rebuild: true,
|
||||
},
|
||||
};
|
||||
let reason = crate::experimental_api::ExperimentalApi::experimental_reason(&request);
|
||||
assert_eq!(reason, Some("thread/epiphany/index"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn thread_realtime_start_is_marked_experimental() {
|
||||
let request = ClientRequest::ThreadRealtimeStart {
|
||||
|
|
|
|||
|
|
@ -3904,6 +3904,22 @@ pub struct ThreadReadResponse {
|
|||
pub thread: Thread,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
#[ts(export_to = "v2/")]
|
||||
pub struct ThreadEpiphanyIndexParams {
|
||||
pub thread_id: String,
|
||||
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
|
||||
pub force_full_rebuild: bool,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
#[ts(export_to = "v2/")]
|
||||
pub struct ThreadEpiphanyIndexResponse {
|
||||
pub index_summary: ThreadEpiphanyRetrieveIndexSummary,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, JsonSchema, TS)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
#[ts(export_to = "v2/")]
|
||||
|
|
|
|||
1
vendor/codex/codex-rs/app-server/README.md
vendored
1
vendor/codex/codex-rs/app-server/README.md
vendored
|
|
@ -140,6 +140,7 @@ Hydrated thread responses such as `thread/start`, `thread/resume`, `thread/fork`
|
|||
|
||||
Experimental Epiphany retrieval:
|
||||
|
||||
- `thread/epiphany/index` runs an explicit loaded-thread-only semantic indexing pass that writes a Qdrant-backed workspace index under Codex home metadata, using local Ollama embeddings by default. It is the write path for persistent semantic retrieval and keeps `thread/epiphany/retrieve` read-only.
|
||||
- `thread/epiphany/retrieve` runs a loaded-thread-only hybrid repo query that combines `codex_file_search` path hits with BM25 chunk retrieval and returns typed result metadata plus an Epiphany retrieval summary.
|
||||
|
||||
- `thread/start` — create a new thread; emits `thread/started` (including the current `thread.status`) and auto-subscribes you to turn/item events for that thread. When the request includes a `cwd` and the resolved sandbox is `workspace-write` or full access, app-server also marks that project as trusted in the user `config.toml`. Pass `sessionStartSource: "clear"` when starting a replacement thread after clearing the current session so `SessionStart` hooks receive `source: "clear"` instead of the default `"startup"`. For permissions, prefer `permissionProfile`; the legacy `sandbox` shorthand is still accepted but cannot be combined with `permissionProfile`.
|
||||
|
|
|
|||
|
|
@ -146,6 +146,8 @@ use codex_app_server_protocol::ThreadCompactStartParams;
|
|||
use codex_app_server_protocol::ThreadCompactStartResponse;
|
||||
use codex_app_server_protocol::ThreadDecrementElicitationParams;
|
||||
use codex_app_server_protocol::ThreadDecrementElicitationResponse;
|
||||
use codex_app_server_protocol::ThreadEpiphanyIndexParams;
|
||||
use codex_app_server_protocol::ThreadEpiphanyIndexResponse;
|
||||
use codex_app_server_protocol::ThreadEpiphanyRetrieveIndexSummary;
|
||||
use codex_app_server_protocol::ThreadEpiphanyRetrieveParams;
|
||||
use codex_app_server_protocol::ThreadEpiphanyRetrieveResponse;
|
||||
|
|
@ -964,6 +966,10 @@ impl CodexMessageProcessor {
|
|||
self.thread_read(to_connection_request_id(request_id), params)
|
||||
.await;
|
||||
}
|
||||
ClientRequest::ThreadEpiphanyIndex { request_id, params } => {
|
||||
self.thread_epiphany_index(to_connection_request_id(request_id), params)
|
||||
.await;
|
||||
}
|
||||
ClientRequest::ThreadEpiphanyRetrieve { request_id, params } => {
|
||||
self.thread_epiphany_retrieve(to_connection_request_id(request_id), params)
|
||||
.await;
|
||||
|
|
@ -4082,6 +4088,57 @@ impl CodexMessageProcessor {
|
|||
self.outgoing.send_response(request_id, response).await;
|
||||
}
|
||||
|
||||
async fn thread_epiphany_index(
|
||||
&self,
|
||||
request_id: ConnectionRequestId,
|
||||
params: ThreadEpiphanyIndexParams,
|
||||
) {
|
||||
let ThreadEpiphanyIndexParams {
|
||||
thread_id,
|
||||
force_full_rebuild,
|
||||
} = params;
|
||||
|
||||
let thread_uuid = match ThreadId::from_string(&thread_id) {
|
||||
Ok(id) => id,
|
||||
Err(err) => {
|
||||
self.send_invalid_request_error(request_id, format!("invalid thread id: {err}"))
|
||||
.await;
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let thread = match self.thread_manager.get_thread(thread_uuid).await {
|
||||
Ok(thread) => thread,
|
||||
Err(_) => {
|
||||
self.send_invalid_request_error(
|
||||
request_id,
|
||||
format!("thread not loaded: {thread_uuid}"),
|
||||
)
|
||||
.await;
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let response = match thread
|
||||
.epiphany_index(force_full_rebuild)
|
||||
.await
|
||||
.and_then(map_epiphany_retrieve_index_summary)
|
||||
.map(|index_summary| ThreadEpiphanyIndexResponse { index_summary })
|
||||
{
|
||||
Ok(response) => response,
|
||||
Err(err) => {
|
||||
self.send_internal_error(
|
||||
request_id,
|
||||
format!("failed to index Epiphany retrieval state for {thread_uuid}: {err}"),
|
||||
)
|
||||
.await;
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
self.outgoing.send_response(request_id, response).await;
|
||||
}
|
||||
|
||||
/// Builds the API view for `thread/read` from persisted metadata plus optional live state.
|
||||
async fn read_thread_view(
|
||||
&self,
|
||||
|
|
@ -11287,6 +11344,39 @@ mod tests {
|
|||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn map_epiphany_retrieve_index_summary_preserves_ready_qdrant_summary() -> Result<()> {
|
||||
let summary = map_epiphany_retrieve_index_summary(
|
||||
codex_protocol::protocol::EpiphanyRetrievalState {
|
||||
workspace_root: test_path_buf("/repo"),
|
||||
index_revision: Some("qdrant-ollama-v1:qwen3-embedding:0.6b".to_string()),
|
||||
status: codex_protocol::protocol::EpiphanyRetrievalStatus::Ready,
|
||||
semantic_available: true,
|
||||
last_indexed_at_unix_seconds: Some(1_744_500_100),
|
||||
indexed_file_count: Some(12),
|
||||
indexed_chunk_count: Some(34),
|
||||
shards: vec![codex_protocol::protocol::EpiphanyRetrievalShardSummary {
|
||||
shard_id: "workspace".to_string(),
|
||||
path_prefix: PathBuf::from("."),
|
||||
indexed_file_count: Some(12),
|
||||
indexed_chunk_count: Some(34),
|
||||
status: codex_protocol::protocol::EpiphanyRetrievalStatus::Ready,
|
||||
exact_available: true,
|
||||
semantic_available: true,
|
||||
}],
|
||||
dirty_paths: Vec::new(),
|
||||
},
|
||||
)?;
|
||||
|
||||
assert_eq!(
|
||||
summary.index_revision.as_deref(),
|
||||
Some("qdrant-ollama-v1:qwen3-embedding:0.6b")
|
||||
);
|
||||
assert_eq!(summary.indexed_file_count, Some(12));
|
||||
assert!(summary.dirty_paths.is_empty());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn aborting_pending_request_clears_pending_state() -> Result<()> {
|
||||
let thread_id = ThreadId::from_string("bfd12a78-5900-467b-9bc5-d3d35df08191")?;
|
||||
|
|
|
|||
3
vendor/codex/codex-rs/core/Cargo.toml
vendored
3
vendor/codex/codex-rs/core/Cargo.toml
vendored
|
|
@ -75,6 +75,7 @@ codex-utils-string = { workspace = true }
|
|||
codex-utils-stream-parser = { workspace = true }
|
||||
codex-utils-template = { workspace = true }
|
||||
codex-windows-sandbox = { package = "codex-windows-sandbox", path = "../windows-sandbox-rs" }
|
||||
epiphany-core = { path = "../../../../epiphany-core" }
|
||||
csv = { workspace = true }
|
||||
dirs = { workspace = true }
|
||||
dunce = { workspace = true }
|
||||
|
|
@ -91,7 +92,7 @@ notify = { workspace = true }
|
|||
once_cell = { workspace = true }
|
||||
rand = { workspace = true }
|
||||
regex-lite = { workspace = true }
|
||||
reqwest = { workspace = true, features = ["json", "stream"] }
|
||||
reqwest = { workspace = true, features = ["blocking", "json", "stream"] }
|
||||
rmcp = { workspace = true, default-features = false, features = [
|
||||
"base64",
|
||||
"macros",
|
||||
|
|
|
|||
36
vendor/codex/codex-rs/core/src/codex_thread.rs
vendored
36
vendor/codex/codex-rs/core/src/codex_thread.rs
vendored
|
|
@ -326,7 +326,38 @@ impl CodexThread {
|
|||
|
||||
pub async fn epiphany_retrieval_state(&self) -> EpiphanyRetrievalState {
|
||||
let config = self.codex.thread_config_snapshot().await;
|
||||
epiphany_retrieval::retrieval_state_for_workspace(config.cwd.as_path())
|
||||
let workspace_root = config.cwd.to_path_buf();
|
||||
let codex_home = self.codex.session.codex_home().await;
|
||||
let fallback_workspace_root = workspace_root.clone();
|
||||
let fallback_codex_home = codex_home.clone();
|
||||
tokio::task::spawn_blocking(move || {
|
||||
epiphany_retrieval::retrieval_state_for_workspace(&workspace_root, codex_home.as_path())
|
||||
})
|
||||
.await
|
||||
.unwrap_or_else(|_| {
|
||||
epiphany_retrieval::retrieval_state_for_workspace(
|
||||
&fallback_workspace_root,
|
||||
fallback_codex_home.as_path(),
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
pub async fn epiphany_index(
|
||||
&self,
|
||||
force_full_rebuild: bool,
|
||||
) -> anyhow::Result<EpiphanyRetrievalState> {
|
||||
let config = self.codex.thread_config_snapshot().await;
|
||||
let workspace_root = config.cwd.to_path_buf();
|
||||
let codex_home = self.codex.session.codex_home().await;
|
||||
tokio::task::spawn_blocking(move || {
|
||||
epiphany_retrieval::index_workspace(
|
||||
&workspace_root,
|
||||
codex_home.as_path(),
|
||||
force_full_rebuild,
|
||||
)
|
||||
})
|
||||
.await
|
||||
.map_err(|err| anyhow::anyhow!("epiphany index worker failed: {err}"))?
|
||||
}
|
||||
|
||||
pub async fn epiphany_retrieve(
|
||||
|
|
@ -335,8 +366,9 @@ impl CodexThread {
|
|||
) -> anyhow::Result<EpiphanyRetrieveResponse> {
|
||||
let config = self.codex.thread_config_snapshot().await;
|
||||
let workspace_root = config.cwd.to_path_buf();
|
||||
let codex_home = self.codex.session.codex_home().await;
|
||||
tokio::task::spawn_blocking(move || {
|
||||
epiphany_retrieval::retrieve_workspace(&workspace_root, query)
|
||||
epiphany_retrieval::retrieve_workspace(&workspace_root, codex_home.as_path(), query)
|
||||
})
|
||||
.await
|
||||
.map_err(|err| anyhow::anyhow!("epiphany retrieval worker failed: {err}"))?
|
||||
|
|
|
|||
|
|
@ -1,31 +1,7 @@
|
|||
use super::ContextualUserFragment;
|
||||
use codex_protocol::protocol::EPIPHANY_STATE_CLOSE_TAG;
|
||||
use codex_protocol::protocol::EPIPHANY_STATE_OPEN_TAG;
|
||||
use codex_protocol::protocol::EpiphanyChurnState;
|
||||
use codex_protocol::protocol::EpiphanyCodeRef;
|
||||
use codex_protocol::protocol::EpiphanyEvidenceRecord;
|
||||
use codex_protocol::protocol::EpiphanyGraph;
|
||||
use codex_protocol::protocol::EpiphanyGraphCheckpoint;
|
||||
use codex_protocol::protocol::EpiphanyGraphEdge;
|
||||
use codex_protocol::protocol::EpiphanyGraphFrontier;
|
||||
use codex_protocol::protocol::EpiphanyGraphLink;
|
||||
use codex_protocol::protocol::EpiphanyGraphNode;
|
||||
use codex_protocol::protocol::EpiphanyModeState;
|
||||
use codex_protocol::protocol::EpiphanyObservation;
|
||||
use codex_protocol::protocol::EpiphanyScratchPad;
|
||||
use codex_protocol::protocol::EpiphanyThreadState;
|
||||
use std::collections::HashSet;
|
||||
use std::path::Path;
|
||||
|
||||
const SUBGOAL_LIMIT: usize = 4;
|
||||
const INVARIANT_LIMIT: usize = 4;
|
||||
const NODE_LIMIT: usize = 3;
|
||||
const EDGE_LIMIT: usize = 2;
|
||||
const LINK_LIMIT: usize = 3;
|
||||
const OBSERVATION_LIMIT: usize = 3;
|
||||
const EVIDENCE_LIMIT: usize = 3;
|
||||
const CODE_REF_LIMIT: usize = 2;
|
||||
const DIRTY_PATH_LIMIT: usize = 4;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub(crate) struct EpiphanyStateInstructions {
|
||||
|
|
@ -35,7 +11,7 @@ pub(crate) struct EpiphanyStateInstructions {
|
|||
impl EpiphanyStateInstructions {
|
||||
pub(crate) fn from_state(state: &EpiphanyThreadState) -> Self {
|
||||
Self {
|
||||
text: render_epiphany_state(state),
|
||||
text: epiphany_core::render_epiphany_state(state),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -50,638 +26,6 @@ impl ContextualUserFragment for EpiphanyStateInstructions {
|
|||
}
|
||||
}
|
||||
|
||||
fn render_epiphany_state(state: &EpiphanyThreadState) -> String {
|
||||
let mut sections = vec![concat!(
|
||||
"\n## Epiphany State\n",
|
||||
"Treat this as the current structured thread understanding. Use it to stay oriented during this turn. ",
|
||||
"If your planned edits conflict with it or expose a gap, surface that before broad edits. ",
|
||||
"Do not silently rewrite this state in prose."
|
||||
)
|
||||
.to_string()];
|
||||
|
||||
sections.push(render_overview(state));
|
||||
|
||||
if let Some(subgoals) = render_subgoals(state) {
|
||||
sections.push(subgoals);
|
||||
}
|
||||
if let Some(invariants) = render_invariants(state) {
|
||||
sections.push(invariants);
|
||||
}
|
||||
if let Some(graphs) = render_graphs(state) {
|
||||
sections.push(graphs);
|
||||
}
|
||||
if let Some(scratch) = render_scratch(state.scratch.as_ref()) {
|
||||
sections.push(scratch);
|
||||
}
|
||||
if let Some(observations) = render_observations(&state.observations) {
|
||||
sections.push(observations);
|
||||
}
|
||||
if let Some(evidence) = render_evidence(&state.recent_evidence) {
|
||||
sections.push(evidence);
|
||||
}
|
||||
if let Some(churn) = render_churn(state.churn.as_ref()) {
|
||||
sections.push(churn);
|
||||
}
|
||||
if let Some(mode) = render_mode(state.mode.as_ref()) {
|
||||
sections.push(mode);
|
||||
}
|
||||
|
||||
format!("{}\n", sections.join("\n\n"))
|
||||
}
|
||||
|
||||
fn render_overview(state: &EpiphanyThreadState) -> String {
|
||||
let mut lines = vec![format!("- Revision: {}", state.revision)];
|
||||
if let Some(objective) = state.objective.as_deref() {
|
||||
lines.push(format!("- Objective: {}", compact_text(objective)));
|
||||
}
|
||||
if let Some(active_subgoal_id) = state.active_subgoal_id.as_deref() {
|
||||
lines.push(format!("- Active subgoal: `{active_subgoal_id}`"));
|
||||
}
|
||||
if let Some(last_updated_turn_id) = state.last_updated_turn_id.as_deref() {
|
||||
lines.push(format!("- Last updated turn: `{last_updated_turn_id}`"));
|
||||
}
|
||||
if let Some(frontier) = state.graph_frontier.as_ref() {
|
||||
let focus_ids = collect_focus_node_ids(Some(frontier), state.graph_checkpoint.as_ref());
|
||||
if !focus_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Frontier focus: {}",
|
||||
render_inline_ids(&focus_ids)
|
||||
));
|
||||
}
|
||||
}
|
||||
format!("## Overview\n{}", lines.join("\n"))
|
||||
}
|
||||
|
||||
fn render_subgoals(state: &EpiphanyThreadState) -> Option<String> {
|
||||
if state.subgoals.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
for subgoal in state.subgoals.iter().take(SUBGOAL_LIMIT) {
|
||||
let active_suffix = state
|
||||
.active_subgoal_id
|
||||
.as_deref()
|
||||
.filter(|active_id| *active_id == subgoal.id)
|
||||
.map(|_| "; active")
|
||||
.unwrap_or("");
|
||||
lines.push(format!(
|
||||
"- `{}` [{}{}]: {}",
|
||||
subgoal.id,
|
||||
subgoal.status,
|
||||
active_suffix,
|
||||
compact_text(&subgoal.title)
|
||||
));
|
||||
if let Some(summary) = subgoal.summary.as_deref() {
|
||||
lines.push(format!(" summary: {}", compact_text(summary)));
|
||||
}
|
||||
}
|
||||
push_omitted_count(&mut lines, state.subgoals.len(), SUBGOAL_LIMIT, "subgoals");
|
||||
|
||||
Some(format!("## Subgoals\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_invariants(state: &EpiphanyThreadState) -> Option<String> {
|
||||
if state.invariants.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
for invariant in state.invariants.iter().take(INVARIANT_LIMIT) {
|
||||
lines.push(format!(
|
||||
"- `{}` [{}]: {}",
|
||||
invariant.id,
|
||||
invariant.status,
|
||||
compact_text(&invariant.description)
|
||||
));
|
||||
if let Some(rationale) = invariant.rationale.as_deref() {
|
||||
lines.push(format!(" rationale: {}", compact_text(rationale)));
|
||||
}
|
||||
}
|
||||
push_omitted_count(
|
||||
&mut lines,
|
||||
state.invariants.len(),
|
||||
INVARIANT_LIMIT,
|
||||
"invariants",
|
||||
);
|
||||
|
||||
Some(format!("## Invariants\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_graphs(state: &EpiphanyThreadState) -> Option<String> {
|
||||
if state.graphs.is_empty() && state.graph_frontier.is_none() && state.graph_checkpoint.is_none()
|
||||
{
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
let focus_node_ids = collect_focus_node_ids(
|
||||
state.graph_frontier.as_ref(),
|
||||
state.graph_checkpoint.as_ref(),
|
||||
);
|
||||
|
||||
if let Some(frontier) = state.graph_frontier.as_ref() {
|
||||
append_frontier_lines(&mut lines, frontier);
|
||||
}
|
||||
if let Some(checkpoint) = state.graph_checkpoint.as_ref() {
|
||||
append_checkpoint_lines(&mut lines, checkpoint);
|
||||
}
|
||||
|
||||
append_graph_lines(
|
||||
&mut lines,
|
||||
"Architecture graph",
|
||||
&state.graphs.architecture,
|
||||
&focus_node_ids,
|
||||
state.graph_frontier.as_ref(),
|
||||
);
|
||||
append_graph_lines(
|
||||
&mut lines,
|
||||
"Dataflow graph",
|
||||
&state.graphs.dataflow,
|
||||
&focus_node_ids,
|
||||
state.graph_frontier.as_ref(),
|
||||
);
|
||||
|
||||
if !state.graphs.links.is_empty() {
|
||||
for link in select_links(&state.graphs.links, &focus_node_ids)
|
||||
.into_iter()
|
||||
.take(LINK_LIMIT)
|
||||
{
|
||||
let relationship = link
|
||||
.relationship
|
||||
.as_deref()
|
||||
.map(|value| format!(" ({})", compact_text(value)))
|
||||
.unwrap_or_default();
|
||||
let code_refs = render_code_refs(&link.code_refs);
|
||||
lines.push(format!(
|
||||
"- Link: `{}` -> `{}`{}{}",
|
||||
link.dataflow_node_id,
|
||||
link.architecture_node_id,
|
||||
relationship,
|
||||
render_ref_suffix(&code_refs)
|
||||
));
|
||||
}
|
||||
push_omitted_count(
|
||||
&mut lines,
|
||||
state.graphs.links.len(),
|
||||
LINK_LIMIT,
|
||||
"graph links",
|
||||
);
|
||||
}
|
||||
|
||||
Some(format!("## Graphs\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn append_frontier_lines(lines: &mut Vec<String>, frontier: &EpiphanyGraphFrontier) {
|
||||
if !frontier.active_node_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Frontier active nodes: {}",
|
||||
render_inline_ids(&frontier.active_node_ids)
|
||||
));
|
||||
}
|
||||
if !frontier.active_edge_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Frontier active edges: {}",
|
||||
render_inline_ids(&frontier.active_edge_ids)
|
||||
));
|
||||
}
|
||||
if !frontier.open_question_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Open questions: {}",
|
||||
render_inline_ids(&frontier.open_question_ids)
|
||||
));
|
||||
}
|
||||
if !frontier.open_gap_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Open gaps: {}",
|
||||
render_inline_ids(&frontier.open_gap_ids)
|
||||
));
|
||||
}
|
||||
if !frontier.dirty_paths.is_empty() {
|
||||
let dirty_paths = frontier
|
||||
.dirty_paths
|
||||
.iter()
|
||||
.take(DIRTY_PATH_LIMIT)
|
||||
.map(|path| format!("`{}`", normalize_path(path)))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ");
|
||||
let omitted_suffix =
|
||||
omitted_suffix(frontier.dirty_paths.len(), DIRTY_PATH_LIMIT, "dirty paths");
|
||||
lines.push(format!("- Dirty paths: {dirty_paths}{omitted_suffix}"));
|
||||
}
|
||||
}
|
||||
|
||||
fn append_checkpoint_lines(lines: &mut Vec<String>, checkpoint: &EpiphanyGraphCheckpoint) {
|
||||
lines.push(format!(
|
||||
"- Checkpoint: `{}` (graph revision {})",
|
||||
checkpoint.checkpoint_id, checkpoint.graph_revision
|
||||
));
|
||||
if let Some(summary) = checkpoint.summary.as_deref() {
|
||||
lines.push(format!("- Checkpoint summary: {}", compact_text(summary)));
|
||||
}
|
||||
if !checkpoint.frontier_node_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Checkpoint frontier nodes: {}",
|
||||
render_inline_ids(&checkpoint.frontier_node_ids)
|
||||
));
|
||||
}
|
||||
if !checkpoint.open_question_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Checkpoint open questions: {}",
|
||||
render_inline_ids(&checkpoint.open_question_ids)
|
||||
));
|
||||
}
|
||||
if !checkpoint.open_gap_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
"- Checkpoint open gaps: {}",
|
||||
render_inline_ids(&checkpoint.open_gap_ids)
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
fn append_graph_lines(
|
||||
lines: &mut Vec<String>,
|
||||
label: &str,
|
||||
graph: &EpiphanyGraph,
|
||||
focus_node_ids: &[String],
|
||||
frontier: Option<&EpiphanyGraphFrontier>,
|
||||
) {
|
||||
if graph.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
let selected_nodes = select_nodes(graph, focus_node_ids);
|
||||
let selected_node_ids = selected_nodes
|
||||
.iter()
|
||||
.map(|node| node.id.as_str())
|
||||
.collect::<HashSet<_>>();
|
||||
let active_edge_ids = frontier
|
||||
.map(|graph_frontier| {
|
||||
graph_frontier
|
||||
.active_edge_ids
|
||||
.iter()
|
||||
.map(String::as_str)
|
||||
.collect::<HashSet<_>>()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
|
||||
lines.push(format!(
|
||||
"- {label}: {} nodes, {} edges",
|
||||
graph.nodes.len(),
|
||||
graph.edges.len()
|
||||
));
|
||||
|
||||
for node in &selected_nodes {
|
||||
let status = node
|
||||
.status
|
||||
.as_deref()
|
||||
.map(|value| format!(" [{}]", compact_text(value)))
|
||||
.unwrap_or_default();
|
||||
lines.push(format!(
|
||||
" - Focus node `{}`{}: {}",
|
||||
node.id,
|
||||
status,
|
||||
compact_text(&node.title)
|
||||
));
|
||||
lines.push(format!(" purpose: {}", compact_text(&node.purpose)));
|
||||
if let Some(mechanism) = node.mechanism.as_deref() {
|
||||
lines.push(format!(" mechanism: {}", compact_text(mechanism)));
|
||||
}
|
||||
if let Some(metaphor) = node.metaphor.as_deref() {
|
||||
lines.push(format!(" metaphor: {}", compact_text(metaphor)));
|
||||
}
|
||||
let code_refs = render_code_refs(&node.code_refs);
|
||||
if !code_refs.is_empty() {
|
||||
lines.push(format!(" code refs: {}", code_refs.join(", ")));
|
||||
}
|
||||
}
|
||||
push_omitted_count(lines, graph.nodes.len(), NODE_LIMIT, "graph nodes");
|
||||
|
||||
let selected_edges = select_edges(graph, &selected_node_ids, &active_edge_ids);
|
||||
for edge in selected_edges.iter().take(EDGE_LIMIT) {
|
||||
let edge_id = edge
|
||||
.id
|
||||
.as_deref()
|
||||
.map(|value| format!(" `{value}`"))
|
||||
.unwrap_or_default();
|
||||
let label_suffix = edge
|
||||
.label
|
||||
.as_deref()
|
||||
.map(|value| format!(" [{}]", compact_text(value)))
|
||||
.unwrap_or_default();
|
||||
let mechanism_suffix = edge
|
||||
.mechanism
|
||||
.as_deref()
|
||||
.map(|value| format!("; {}", compact_text(value)))
|
||||
.unwrap_or_default();
|
||||
let code_refs = render_code_refs(&edge.code_refs);
|
||||
lines.push(format!(
|
||||
" - Edge{edge_id}{label_suffix}: `{}` -> `{}` ({}){}{}",
|
||||
edge.source_id,
|
||||
edge.target_id,
|
||||
edge.kind,
|
||||
mechanism_suffix,
|
||||
render_ref_suffix(&code_refs)
|
||||
));
|
||||
}
|
||||
push_omitted_count(
|
||||
lines,
|
||||
selected_edges.len(),
|
||||
EDGE_LIMIT,
|
||||
"focused graph edges",
|
||||
);
|
||||
}
|
||||
|
||||
fn render_scratch(scratch: Option<&EpiphanyScratchPad>) -> Option<String> {
|
||||
let scratch = scratch?;
|
||||
let mut lines = Vec::new();
|
||||
if let Some(summary) = scratch.summary.as_deref() {
|
||||
lines.push(format!("- Summary: {}", compact_text(summary)));
|
||||
}
|
||||
if let Some(hypothesis) = scratch.hypothesis.as_deref() {
|
||||
lines.push(format!("- Hypothesis: {}", compact_text(hypothesis)));
|
||||
}
|
||||
if let Some(next_probe) = scratch.next_probe.as_deref() {
|
||||
lines.push(format!("- Next probe: {}", compact_text(next_probe)));
|
||||
}
|
||||
|
||||
if lines.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(format!("## Scratch\n{}", lines.join("\n")))
|
||||
}
|
||||
}
|
||||
|
||||
fn render_observations(observations: &[EpiphanyObservation]) -> Option<String> {
|
||||
if observations.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
for observation in observations.iter().take(OBSERVATION_LIMIT) {
|
||||
lines.push(format!(
|
||||
"- `{}` [{} via {}]: {}",
|
||||
observation.id,
|
||||
observation.status,
|
||||
observation.source_kind,
|
||||
compact_text(&observation.summary)
|
||||
));
|
||||
if !observation.evidence_ids.is_empty() {
|
||||
lines.push(format!(
|
||||
" evidence: {}",
|
||||
render_inline_ids(&observation.evidence_ids)
|
||||
));
|
||||
}
|
||||
let code_refs = render_code_refs(&observation.code_refs);
|
||||
if !code_refs.is_empty() {
|
||||
lines.push(format!(" code refs: {}", code_refs.join(", ")));
|
||||
}
|
||||
}
|
||||
push_omitted_count(
|
||||
&mut lines,
|
||||
observations.len(),
|
||||
OBSERVATION_LIMIT,
|
||||
"observations",
|
||||
);
|
||||
|
||||
Some(format!("## Recent Observations\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_evidence(evidence: &[EpiphanyEvidenceRecord]) -> Option<String> {
|
||||
if evidence.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut lines = Vec::new();
|
||||
for record in evidence.iter().take(EVIDENCE_LIMIT) {
|
||||
let code_refs = render_code_refs(&record.code_refs);
|
||||
lines.push(format!(
|
||||
"- `{}` [{} / {}]: {}{}",
|
||||
record.id,
|
||||
record.kind,
|
||||
record.status,
|
||||
compact_text(&record.summary),
|
||||
render_ref_suffix(&code_refs)
|
||||
));
|
||||
}
|
||||
push_omitted_count(
|
||||
&mut lines,
|
||||
evidence.len(),
|
||||
EVIDENCE_LIMIT,
|
||||
"evidence records",
|
||||
);
|
||||
|
||||
Some(format!("## Recent Evidence\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_churn(churn: Option<&EpiphanyChurnState>) -> Option<String> {
|
||||
let churn = churn?;
|
||||
let mut lines = vec![
|
||||
format!(
|
||||
"- Understanding status: {}",
|
||||
compact_text(&churn.understanding_status)
|
||||
),
|
||||
format!("- Diff pressure: {}", compact_text(&churn.diff_pressure)),
|
||||
];
|
||||
if let Some(graph_freshness) = churn.graph_freshness.as_deref() {
|
||||
lines.push(format!(
|
||||
"- Graph freshness: {}",
|
||||
compact_text(graph_freshness)
|
||||
));
|
||||
}
|
||||
if let Some(warning) = churn.warning.as_deref() {
|
||||
lines.push(format!("- Warning: {}", compact_text(warning)));
|
||||
}
|
||||
if let Some(unexplained_writes) = churn.unexplained_writes {
|
||||
lines.push(format!("- Unexplained writes: {unexplained_writes}"));
|
||||
}
|
||||
|
||||
Some(format!("## Churn\n{}", lines.join("\n")))
|
||||
}
|
||||
|
||||
fn render_mode(mode: Option<&EpiphanyModeState>) -> Option<String> {
|
||||
let mode = mode?;
|
||||
let mut line = format!("- Name: {}", compact_text(&mode.name));
|
||||
if let Some(kind) = mode.kind {
|
||||
line.push_str(&format!(" ({kind:?})"));
|
||||
}
|
||||
Some(format!("## Mode\n{line}"))
|
||||
}
|
||||
|
||||
fn collect_focus_node_ids(
|
||||
frontier: Option<&EpiphanyGraphFrontier>,
|
||||
checkpoint: Option<&EpiphanyGraphCheckpoint>,
|
||||
) -> Vec<String> {
|
||||
let mut focus_node_ids = Vec::new();
|
||||
let mut seen = HashSet::new();
|
||||
for node_id in frontier
|
||||
.into_iter()
|
||||
.flat_map(|value| value.active_node_ids.iter())
|
||||
.chain(
|
||||
checkpoint
|
||||
.into_iter()
|
||||
.flat_map(|value| value.frontier_node_ids.iter()),
|
||||
)
|
||||
{
|
||||
if seen.insert(node_id.as_str()) {
|
||||
focus_node_ids.push(node_id.clone());
|
||||
}
|
||||
}
|
||||
focus_node_ids
|
||||
}
|
||||
|
||||
fn select_nodes<'a>(
|
||||
graph: &'a EpiphanyGraph,
|
||||
focus_node_ids: &[String],
|
||||
) -> Vec<&'a EpiphanyGraphNode> {
|
||||
let mut selected = Vec::new();
|
||||
let mut seen = HashSet::new();
|
||||
let focus_set = focus_node_ids
|
||||
.iter()
|
||||
.map(String::as_str)
|
||||
.collect::<HashSet<_>>();
|
||||
|
||||
for node in &graph.nodes {
|
||||
if focus_set.contains(node.id.as_str()) && seen.insert(node.id.as_str()) {
|
||||
selected.push(node);
|
||||
if selected.len() == NODE_LIMIT {
|
||||
return selected;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for node in &graph.nodes {
|
||||
if seen.insert(node.id.as_str()) {
|
||||
selected.push(node);
|
||||
if selected.len() == NODE_LIMIT {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
selected
|
||||
}
|
||||
|
||||
fn select_edges<'a>(
|
||||
graph: &'a EpiphanyGraph,
|
||||
selected_node_ids: &HashSet<&'a str>,
|
||||
active_edge_ids: &HashSet<&str>,
|
||||
) -> Vec<&'a EpiphanyGraphEdge> {
|
||||
graph
|
||||
.edges
|
||||
.iter()
|
||||
.filter(|edge| {
|
||||
edge.id
|
||||
.as_deref()
|
||||
.is_some_and(|edge_id| active_edge_ids.contains(edge_id))
|
||||
|| selected_node_ids.contains(edge.source_id.as_str())
|
||||
|| selected_node_ids.contains(edge.target_id.as_str())
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn select_links<'a>(
|
||||
links: &'a [EpiphanyGraphLink],
|
||||
focus_node_ids: &[String],
|
||||
) -> Vec<&'a EpiphanyGraphLink> {
|
||||
let focus_set = focus_node_ids
|
||||
.iter()
|
||||
.map(String::as_str)
|
||||
.collect::<HashSet<_>>();
|
||||
let mut focused = Vec::new();
|
||||
let mut fallback = Vec::new();
|
||||
for link in links {
|
||||
if focus_set.contains(link.dataflow_node_id.as_str())
|
||||
|| focus_set.contains(link.architecture_node_id.as_str())
|
||||
{
|
||||
focused.push(link);
|
||||
} else {
|
||||
fallback.push(link);
|
||||
}
|
||||
}
|
||||
focused.extend(fallback);
|
||||
focused
|
||||
}
|
||||
|
||||
fn render_code_refs(code_refs: &[EpiphanyCodeRef]) -> Vec<String> {
|
||||
let mut rendered = code_refs
|
||||
.iter()
|
||||
.take(CODE_REF_LIMIT)
|
||||
.map(render_code_ref)
|
||||
.collect::<Vec<_>>();
|
||||
if code_refs.len() > CODE_REF_LIMIT {
|
||||
rendered.push(format!(
|
||||
"... {} more refs",
|
||||
code_refs.len() - CODE_REF_LIMIT
|
||||
));
|
||||
}
|
||||
rendered
|
||||
}
|
||||
|
||||
fn render_code_ref(code_ref: &EpiphanyCodeRef) -> String {
|
||||
let mut location = normalize_path(&code_ref.path);
|
||||
match (code_ref.start_line, code_ref.end_line) {
|
||||
(Some(start_line), Some(end_line)) if end_line != start_line => {
|
||||
location.push_str(&format!(":{start_line}-{end_line}"));
|
||||
}
|
||||
(Some(start_line), _) => {
|
||||
location.push_str(&format!(":{start_line}"));
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
|
||||
let mut metadata = Vec::new();
|
||||
if let Some(symbol) = code_ref.symbol.as_deref() {
|
||||
metadata.push(format!("symbol: {}", compact_text(symbol)));
|
||||
}
|
||||
if let Some(note) = code_ref.note.as_deref() {
|
||||
metadata.push(compact_text(note));
|
||||
}
|
||||
|
||||
if metadata.is_empty() {
|
||||
format!("`{location}`")
|
||||
} else {
|
||||
format!("`{location}` ({})", metadata.join("; "))
|
||||
}
|
||||
}
|
||||
|
||||
fn render_ref_suffix(code_refs: &[String]) -> String {
|
||||
if code_refs.is_empty() {
|
||||
String::new()
|
||||
} else {
|
||||
format!("; refs: {}", code_refs.join(", "))
|
||||
}
|
||||
}
|
||||
|
||||
fn render_inline_ids(ids: &[String]) -> String {
|
||||
ids.iter()
|
||||
.map(|id| format!("`{id}`"))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ")
|
||||
}
|
||||
|
||||
fn normalize_path(path: &Path) -> String {
|
||||
path.to_string_lossy().replace('\\', "/")
|
||||
}
|
||||
|
||||
fn compact_text(value: &str) -> String {
|
||||
value.split_whitespace().collect::<Vec<_>>().join(" ")
|
||||
}
|
||||
|
||||
fn omitted_suffix(total: usize, shown: usize, label: &str) -> String {
|
||||
if total > shown {
|
||||
format!("; ... {} more {label} not shown", total - shown)
|
||||
} else {
|
||||
String::new()
|
||||
}
|
||||
}
|
||||
|
||||
fn push_omitted_count(lines: &mut Vec<String>, total: usize, shown: usize, label: &str) {
|
||||
if total > shown {
|
||||
lines.push(format!("- ... {} more {label} not shown", total - shown));
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::EpiphanyStateInstructions;
|
||||
|
|
|
|||
567
vendor/codex/codex-rs/core/src/epiphany_retrieval.rs
vendored
567
vendor/codex/codex-rs/core/src/epiphany_retrieval.rs
vendored
|
|
@ -1,558 +1,9 @@
|
|||
use anyhow::Context;
|
||||
use anyhow::Result;
|
||||
use bm25::Document;
|
||||
use bm25::Language;
|
||||
use bm25::SearchEngineBuilder;
|
||||
use codex_file_search as file_search;
|
||||
use codex_protocol::protocol::EpiphanyRetrievalShardSummary;
|
||||
use codex_protocol::protocol::EpiphanyRetrievalState;
|
||||
use codex_protocol::protocol::EpiphanyRetrievalStatus;
|
||||
use ignore::WalkBuilder;
|
||||
use std::cmp::Ordering;
|
||||
use std::fs;
|
||||
use std::num::NonZero;
|
||||
use std::path::Component;
|
||||
use std::path::Path;
|
||||
use std::path::PathBuf;
|
||||
|
||||
pub const EPIPHANY_RETRIEVAL_DEFAULT_LIMIT: usize = 10;
|
||||
pub const EPIPHANY_RETRIEVAL_MAX_LIMIT: usize = 50;
|
||||
|
||||
const EXACT_SEARCH_CANDIDATE_MULTIPLIER: usize = 5;
|
||||
const EXACT_SEARCH_MAX_CANDIDATES: usize = 250;
|
||||
const FILE_SEARCH_MAX_THREADS: usize = 12;
|
||||
const QUERY_TIME_INDEX_REVISION: &str = "query-time-bm25-v1";
|
||||
const MAX_TEXT_FILE_BYTES: u64 = 256 * 1024;
|
||||
const CHUNK_LINE_COUNT: usize = 24;
|
||||
const CHUNK_LINE_OVERLAP: usize = 8;
|
||||
const WORKSPACE_SHARD_ID: &str = "workspace";
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct EpiphanyRetrieveQuery {
|
||||
pub query: String,
|
||||
pub limit: usize,
|
||||
pub path_prefixes: Vec<PathBuf>,
|
||||
}
|
||||
|
||||
impl EpiphanyRetrieveQuery {
|
||||
pub fn new(query: String) -> Self {
|
||||
Self {
|
||||
query,
|
||||
limit: EPIPHANY_RETRIEVAL_DEFAULT_LIMIT,
|
||||
path_prefixes: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum EpiphanyRetrieveResultKind {
|
||||
ExactFile,
|
||||
ExactDirectory,
|
||||
SemanticChunk,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct EpiphanyRetrieveResult {
|
||||
pub kind: EpiphanyRetrieveResultKind,
|
||||
pub path: PathBuf,
|
||||
pub score: f32,
|
||||
pub line_start: Option<u32>,
|
||||
pub line_end: Option<u32>,
|
||||
pub excerpt: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct EpiphanyRetrieveResponse {
|
||||
pub query: String,
|
||||
pub index_summary: EpiphanyRetrievalState,
|
||||
pub results: Vec<EpiphanyRetrieveResult>,
|
||||
}
|
||||
|
||||
pub fn retrieval_state_for_workspace(workspace_root: &Path) -> EpiphanyRetrievalState {
|
||||
let available = workspace_root.is_dir();
|
||||
let status = if available {
|
||||
EpiphanyRetrievalStatus::Ready
|
||||
} else {
|
||||
EpiphanyRetrievalStatus::Unavailable
|
||||
};
|
||||
|
||||
EpiphanyRetrievalState {
|
||||
workspace_root: workspace_root.to_path_buf(),
|
||||
index_revision: Some(QUERY_TIME_INDEX_REVISION.to_string()),
|
||||
status,
|
||||
semantic_available: available,
|
||||
last_indexed_at_unix_seconds: None,
|
||||
indexed_file_count: None,
|
||||
indexed_chunk_count: None,
|
||||
shards: vec![EpiphanyRetrievalShardSummary {
|
||||
shard_id: WORKSPACE_SHARD_ID.to_string(),
|
||||
path_prefix: PathBuf::from("."),
|
||||
indexed_file_count: None,
|
||||
indexed_chunk_count: None,
|
||||
status,
|
||||
exact_available: available,
|
||||
semantic_available: available,
|
||||
}],
|
||||
dirty_paths: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn retrieve_workspace(
|
||||
workspace_root: &Path,
|
||||
query: EpiphanyRetrieveQuery,
|
||||
) -> Result<EpiphanyRetrieveResponse> {
|
||||
if !workspace_root.is_dir() {
|
||||
anyhow::bail!(
|
||||
"workspace root does not exist: {}",
|
||||
workspace_root.display()
|
||||
);
|
||||
}
|
||||
|
||||
let query_text = query.query.trim();
|
||||
if query_text.is_empty() {
|
||||
anyhow::bail!("query must not be empty");
|
||||
}
|
||||
|
||||
let limit = query.limit.clamp(1, EPIPHANY_RETRIEVAL_MAX_LIMIT);
|
||||
let path_prefixes = normalize_path_prefixes(workspace_root, &query.path_prefixes);
|
||||
|
||||
let exact_results = run_exact_search(workspace_root, query_text, limit, &path_prefixes)
|
||||
.context("exact retrieval failed")?;
|
||||
let semantic_corpus = collect_semantic_corpus(workspace_root, &path_prefixes)
|
||||
.context("semantic corpus collection failed")?;
|
||||
let semantic_results = search_semantic_chunks(query_text, limit, &semantic_corpus);
|
||||
|
||||
let mut results = Vec::with_capacity(exact_results.len() + semantic_results.len());
|
||||
results.extend(exact_results);
|
||||
results.extend(semantic_results);
|
||||
sort_results(&mut results);
|
||||
results.truncate(limit);
|
||||
|
||||
Ok(EpiphanyRetrieveResponse {
|
||||
query: query_text.to_string(),
|
||||
index_summary: query_time_retrieval_state(
|
||||
workspace_root,
|
||||
semantic_corpus.searchable_file_count,
|
||||
semantic_corpus.chunks.len(),
|
||||
),
|
||||
results,
|
||||
})
|
||||
}
|
||||
|
||||
fn run_exact_search(
|
||||
workspace_root: &Path,
|
||||
query: &str,
|
||||
limit: usize,
|
||||
path_prefixes: &[PathBuf],
|
||||
) -> Result<Vec<EpiphanyRetrieveResult>> {
|
||||
let candidate_limit = if path_prefixes.is_empty() {
|
||||
limit
|
||||
} else {
|
||||
limit
|
||||
.saturating_mul(EXACT_SEARCH_CANDIDATE_MULTIPLIER)
|
||||
.min(EXACT_SEARCH_MAX_CANDIDATES)
|
||||
};
|
||||
#[expect(clippy::expect_used)]
|
||||
let candidate_limit =
|
||||
NonZero::new(candidate_limit.max(1)).expect("candidate limit should be non-zero");
|
||||
|
||||
let search_results = file_search::run(
|
||||
query,
|
||||
vec![workspace_root.to_path_buf()],
|
||||
file_search::FileSearchOptions {
|
||||
limit: candidate_limit,
|
||||
threads: search_threads(),
|
||||
compute_indices: false,
|
||||
..Default::default()
|
||||
},
|
||||
/*cancel_flag*/ None,
|
||||
)?;
|
||||
|
||||
let mut results = search_results
|
||||
.matches
|
||||
.into_iter()
|
||||
.filter_map(|file_match| {
|
||||
let relative_path = file_match
|
||||
.full_path()
|
||||
.strip_prefix(workspace_root)
|
||||
.ok()?
|
||||
.to_path_buf();
|
||||
if !matches_path_prefixes(&relative_path, path_prefixes) {
|
||||
return None;
|
||||
}
|
||||
|
||||
Some(EpiphanyRetrieveResult {
|
||||
kind: match file_match.match_type {
|
||||
file_search::MatchType::File => EpiphanyRetrieveResultKind::ExactFile,
|
||||
file_search::MatchType::Directory => EpiphanyRetrieveResultKind::ExactDirectory,
|
||||
},
|
||||
path: relative_path,
|
||||
score: file_match.score as f32,
|
||||
line_start: None,
|
||||
line_end: None,
|
||||
excerpt: None,
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
results.truncate(limit);
|
||||
Ok(results)
|
||||
}
|
||||
|
||||
fn search_threads() -> NonZero<usize> {
|
||||
let cores = std::thread::available_parallelism()
|
||||
.map(NonZero::get)
|
||||
.unwrap_or(1);
|
||||
#[expect(clippy::expect_used)]
|
||||
NonZero::new(cores.min(FILE_SEARCH_MAX_THREADS).max(1))
|
||||
.expect("file-search threads should be non-zero")
|
||||
}
|
||||
|
||||
struct SemanticCorpus {
|
||||
searchable_file_count: usize,
|
||||
chunks: Vec<SemanticChunk>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
struct SemanticChunk {
|
||||
path: PathBuf,
|
||||
line_start: u32,
|
||||
line_end: u32,
|
||||
excerpt: String,
|
||||
search_text: String,
|
||||
}
|
||||
|
||||
fn collect_semantic_corpus(
|
||||
workspace_root: &Path,
|
||||
path_prefixes: &[PathBuf],
|
||||
) -> Result<SemanticCorpus> {
|
||||
let mut builder = WalkBuilder::new(workspace_root);
|
||||
builder.hidden(false);
|
||||
|
||||
let mut searchable_file_count = 0usize;
|
||||
let mut chunks = Vec::new();
|
||||
for entry in builder.build() {
|
||||
let Ok(entry) = entry else {
|
||||
continue;
|
||||
};
|
||||
if !entry
|
||||
.file_type()
|
||||
.is_some_and(|file_type| file_type.is_file())
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
let path = entry.path();
|
||||
let Ok(relative_path) = path.strip_prefix(workspace_root) else {
|
||||
continue;
|
||||
};
|
||||
if should_skip_relative_path(relative_path) {
|
||||
continue;
|
||||
}
|
||||
if !matches_path_prefixes(relative_path, path_prefixes) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let Ok(metadata) = fs::metadata(path) else {
|
||||
continue;
|
||||
};
|
||||
if metadata.len() > MAX_TEXT_FILE_BYTES {
|
||||
continue;
|
||||
}
|
||||
|
||||
let Ok(contents) = fs::read_to_string(path) else {
|
||||
continue;
|
||||
};
|
||||
let file_chunks = build_semantic_chunks(relative_path, &contents);
|
||||
if file_chunks.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
searchable_file_count += 1;
|
||||
chunks.extend(file_chunks);
|
||||
}
|
||||
|
||||
Ok(SemanticCorpus {
|
||||
searchable_file_count,
|
||||
chunks,
|
||||
})
|
||||
}
|
||||
|
||||
fn build_semantic_chunks(relative_path: &Path, contents: &str) -> Vec<SemanticChunk> {
|
||||
let lines = contents.lines().collect::<Vec<_>>();
|
||||
if lines.is_empty() {
|
||||
return Vec::new();
|
||||
}
|
||||
|
||||
let step = CHUNK_LINE_COUNT.saturating_sub(CHUNK_LINE_OVERLAP).max(1);
|
||||
let mut chunks = Vec::new();
|
||||
let mut start = 0usize;
|
||||
while start < lines.len() {
|
||||
let end = (start + CHUNK_LINE_COUNT).min(lines.len());
|
||||
let excerpt = lines[start..end].join("\n").trim().to_string();
|
||||
if !excerpt.is_empty() {
|
||||
chunks.push(SemanticChunk {
|
||||
path: relative_path.to_path_buf(),
|
||||
line_start: (start + 1) as u32,
|
||||
line_end: end as u32,
|
||||
search_text: format!("{}\n{}", relative_path.display(), excerpt),
|
||||
excerpt,
|
||||
});
|
||||
}
|
||||
|
||||
if end == lines.len() {
|
||||
break;
|
||||
}
|
||||
start += step;
|
||||
}
|
||||
|
||||
chunks
|
||||
}
|
||||
|
||||
fn search_semantic_chunks(
|
||||
query: &str,
|
||||
limit: usize,
|
||||
semantic_corpus: &SemanticCorpus,
|
||||
) -> Vec<EpiphanyRetrieveResult> {
|
||||
if semantic_corpus.chunks.is_empty() {
|
||||
return Vec::new();
|
||||
}
|
||||
|
||||
let documents = semantic_corpus
|
||||
.chunks
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, chunk)| Document::new(index, chunk.search_text.clone()))
|
||||
.collect::<Vec<_>>();
|
||||
let search_engine =
|
||||
SearchEngineBuilder::<usize>::with_documents(Language::English, documents).build();
|
||||
|
||||
search_engine
|
||||
.search(query, limit)
|
||||
.into_iter()
|
||||
.filter_map(|result| {
|
||||
semantic_corpus
|
||||
.chunks
|
||||
.get(result.document.id)
|
||||
.map(|chunk| EpiphanyRetrieveResult {
|
||||
kind: EpiphanyRetrieveResultKind::SemanticChunk,
|
||||
path: chunk.path.clone(),
|
||||
score: result.score,
|
||||
line_start: Some(chunk.line_start),
|
||||
line_end: Some(chunk.line_end),
|
||||
excerpt: Some(chunk.excerpt.clone()),
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn query_time_retrieval_state(
|
||||
workspace_root: &Path,
|
||||
searchable_file_count: usize,
|
||||
chunk_count: usize,
|
||||
) -> EpiphanyRetrievalState {
|
||||
let indexed_file_count = u32::try_from(searchable_file_count).ok();
|
||||
let indexed_chunk_count = u32::try_from(chunk_count).ok();
|
||||
|
||||
EpiphanyRetrievalState {
|
||||
workspace_root: workspace_root.to_path_buf(),
|
||||
index_revision: Some(QUERY_TIME_INDEX_REVISION.to_string()),
|
||||
status: EpiphanyRetrievalStatus::Ready,
|
||||
semantic_available: chunk_count > 0,
|
||||
last_indexed_at_unix_seconds: Some(chrono::Utc::now().timestamp()),
|
||||
indexed_file_count,
|
||||
indexed_chunk_count,
|
||||
shards: vec![EpiphanyRetrievalShardSummary {
|
||||
shard_id: WORKSPACE_SHARD_ID.to_string(),
|
||||
path_prefix: PathBuf::from("."),
|
||||
indexed_file_count,
|
||||
indexed_chunk_count,
|
||||
status: EpiphanyRetrievalStatus::Ready,
|
||||
exact_available: true,
|
||||
semantic_available: chunk_count > 0,
|
||||
}],
|
||||
dirty_paths: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
fn sort_results(results: &mut [EpiphanyRetrieveResult]) {
|
||||
results.sort_by(|left, right| {
|
||||
result_priority(&right.kind)
|
||||
.cmp(&result_priority(&left.kind))
|
||||
.then_with(|| {
|
||||
right
|
||||
.score
|
||||
.partial_cmp(&left.score)
|
||||
.unwrap_or(Ordering::Equal)
|
||||
})
|
||||
.then_with(|| left.path.cmp(&right.path))
|
||||
.then_with(|| {
|
||||
left.line_start
|
||||
.unwrap_or(u32::MAX)
|
||||
.cmp(&right.line_start.unwrap_or(u32::MAX))
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
fn result_priority(kind: &EpiphanyRetrieveResultKind) -> u8 {
|
||||
match kind {
|
||||
EpiphanyRetrieveResultKind::ExactFile => 3,
|
||||
EpiphanyRetrieveResultKind::SemanticChunk => 2,
|
||||
EpiphanyRetrieveResultKind::ExactDirectory => 1,
|
||||
}
|
||||
}
|
||||
|
||||
fn normalize_path_prefixes(workspace_root: &Path, path_prefixes: &[PathBuf]) -> Vec<PathBuf> {
|
||||
let mut normalized = path_prefixes
|
||||
.iter()
|
||||
.filter_map(|prefix| normalize_path_prefix(workspace_root, prefix.as_path()))
|
||||
.collect::<Vec<_>>();
|
||||
normalized.sort();
|
||||
normalized.dedup();
|
||||
normalized
|
||||
}
|
||||
|
||||
fn normalize_path_prefix(workspace_root: &Path, prefix: &Path) -> Option<PathBuf> {
|
||||
if prefix.as_os_str().is_empty() || prefix == Path::new(".") {
|
||||
return None;
|
||||
}
|
||||
|
||||
let relative = if prefix.is_absolute() {
|
||||
prefix.strip_prefix(workspace_root).ok()?
|
||||
} else {
|
||||
prefix
|
||||
};
|
||||
|
||||
let mut normalized = PathBuf::new();
|
||||
for component in relative.components() {
|
||||
match component {
|
||||
Component::Normal(value) => normalized.push(value),
|
||||
Component::CurDir => {}
|
||||
Component::ParentDir | Component::Prefix(_) | Component::RootDir => return None,
|
||||
}
|
||||
}
|
||||
|
||||
if normalized.as_os_str().is_empty() || normalized == Path::new(".") {
|
||||
None
|
||||
} else {
|
||||
Some(normalized)
|
||||
}
|
||||
}
|
||||
|
||||
fn matches_path_prefixes(path: &Path, path_prefixes: &[PathBuf]) -> bool {
|
||||
path_prefixes.is_empty() || path_prefixes.iter().any(|prefix| path.starts_with(prefix))
|
||||
}
|
||||
|
||||
fn should_skip_relative_path(relative_path: &Path) -> bool {
|
||||
relative_path
|
||||
.components()
|
||||
.any(|component| matches!(component, Component::Normal(value) if value == ".git"))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use anyhow::Result;
|
||||
use pretty_assertions::assert_eq;
|
||||
use tempfile::TempDir;
|
||||
|
||||
#[test]
|
||||
fn retrieve_workspace_prioritizes_exact_hits_without_losing_semantic_chunks() -> Result<()> {
|
||||
let temp_dir = TempDir::new()?;
|
||||
fs::create_dir_all(temp_dir.path().join("src"))?;
|
||||
fs::create_dir_all(temp_dir.path().join("notes"))?;
|
||||
fs::write(
|
||||
temp_dir.path().join("src").join("session_checkpoint.rs"),
|
||||
"pub fn route_session() {}\n",
|
||||
)?;
|
||||
fs::write(
|
||||
temp_dir.path().join("notes").join("design.md"),
|
||||
"The compaction checkpoint keeps the field notebook alive.\n\
|
||||
Retrieval should honor the checkpoint before the next pass.\n",
|
||||
)?;
|
||||
|
||||
let response = retrieve_workspace(
|
||||
temp_dir.path(),
|
||||
EpiphanyRetrieveQuery {
|
||||
query: "session checkpoint".to_string(),
|
||||
limit: 5,
|
||||
path_prefixes: Vec::new(),
|
||||
},
|
||||
)?;
|
||||
|
||||
assert_eq!(
|
||||
response.results.first().map(|result| &result.kind),
|
||||
Some(&EpiphanyRetrieveResultKind::ExactFile)
|
||||
);
|
||||
assert!(
|
||||
response
|
||||
.results
|
||||
.iter()
|
||||
.any(|result| result.kind == EpiphanyRetrieveResultKind::SemanticChunk),
|
||||
"expected at least one semantic chunk result"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retrieve_workspace_finds_semantic_match_without_path_match() -> Result<()> {
|
||||
let temp_dir = TempDir::new()?;
|
||||
fs::create_dir_all(temp_dir.path().join("src"))?;
|
||||
fs::write(
|
||||
temp_dir.path().join("src").join("engine.rs"),
|
||||
"The frontier checkpoint preserves the machine map before compaction.\n\
|
||||
Keep the durable spine and discard disposable scratch.\n",
|
||||
)?;
|
||||
|
||||
let response = retrieve_workspace(
|
||||
temp_dir.path(),
|
||||
EpiphanyRetrieveQuery {
|
||||
query: "durable spine compaction checkpoint".to_string(),
|
||||
limit: 3,
|
||||
path_prefixes: Vec::new(),
|
||||
},
|
||||
)?;
|
||||
|
||||
assert!(
|
||||
response.results.iter().any(|result| {
|
||||
result.kind == EpiphanyRetrieveResultKind::SemanticChunk
|
||||
&& result.path == PathBuf::from("src").join("engine.rs")
|
||||
}),
|
||||
"expected semantic retrieval to find engine.rs"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retrieve_workspace_respects_path_prefixes() -> Result<()> {
|
||||
let temp_dir = TempDir::new()?;
|
||||
fs::create_dir_all(temp_dir.path().join("src"))?;
|
||||
fs::create_dir_all(temp_dir.path().join("docs"))?;
|
||||
fs::write(
|
||||
temp_dir.path().join("src").join("router.rs"),
|
||||
"checkpoint frontier checkpoint frontier\n",
|
||||
)?;
|
||||
fs::write(
|
||||
temp_dir.path().join("docs").join("guide.md"),
|
||||
"checkpoint frontier checkpoint frontier\n",
|
||||
)?;
|
||||
|
||||
let response = retrieve_workspace(
|
||||
temp_dir.path(),
|
||||
EpiphanyRetrieveQuery {
|
||||
query: "checkpoint frontier".to_string(),
|
||||
limit: 10,
|
||||
path_prefixes: vec![PathBuf::from("docs")],
|
||||
},
|
||||
)?;
|
||||
|
||||
assert!(
|
||||
response
|
||||
.results
|
||||
.iter()
|
||||
.all(|result| result.path.starts_with("docs")),
|
||||
"all results should stay inside the requested path prefix"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
pub use epiphany_core::EPIPHANY_RETRIEVAL_DEFAULT_LIMIT;
|
||||
pub use epiphany_core::EPIPHANY_RETRIEVAL_MAX_LIMIT;
|
||||
pub use epiphany_core::EpiphanyRetrieveQuery;
|
||||
pub use epiphany_core::EpiphanyRetrieveResponse;
|
||||
pub use epiphany_core::EpiphanyRetrieveResult;
|
||||
pub use epiphany_core::EpiphanyRetrieveResultKind;
|
||||
pub use epiphany_core::index_workspace;
|
||||
pub use epiphany_core::retrieval_state_for_workspace;
|
||||
pub use epiphany_core::retrieve_workspace;
|
||||
|
|
|
|||
311
vendor/codex/codex-rs/core/src/epiphany_rollout.rs
vendored
311
vendor/codex/codex-rs/core/src/epiphany_rollout.rs
vendored
|
|
@ -1,318 +1,9 @@
|
|||
use crate::context_manager::is_user_turn_boundary;
|
||||
use codex_protocol::protocol::EpiphanyThreadState;
|
||||
use codex_protocol::protocol::EventMsg;
|
||||
use codex_protocol::protocol::RolloutItem;
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct ActiveEpiphanyReplaySegment {
|
||||
turn_id: Option<String>,
|
||||
counts_as_user_turn: bool,
|
||||
epiphany_state: Option<EpiphanyThreadState>,
|
||||
}
|
||||
|
||||
fn turn_ids_are_compatible(active_turn_id: Option<&str>, item_turn_id: Option<&str>) -> bool {
|
||||
active_turn_id
|
||||
.is_none_or(|turn_id| item_turn_id.is_none_or(|item_turn_id| item_turn_id == turn_id))
|
||||
}
|
||||
|
||||
fn finalize_active_segment(
|
||||
active_segment: ActiveEpiphanyReplaySegment,
|
||||
epiphany_state: &mut Option<EpiphanyThreadState>,
|
||||
pending_rollback_turns: &mut usize,
|
||||
) {
|
||||
if *pending_rollback_turns > 0 {
|
||||
if active_segment.counts_as_user_turn {
|
||||
*pending_rollback_turns -= 1;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if epiphany_state.is_none() && active_segment.counts_as_user_turn {
|
||||
*epiphany_state = active_segment.epiphany_state;
|
||||
}
|
||||
}
|
||||
|
||||
pub fn latest_epiphany_state_from_rollout_items(
|
||||
rollout_items: &[RolloutItem],
|
||||
) -> Option<EpiphanyThreadState> {
|
||||
let mut epiphany_state = None;
|
||||
let mut pending_rollback_turns = 0usize;
|
||||
let mut active_segment: Option<ActiveEpiphanyReplaySegment> = None;
|
||||
|
||||
for item in rollout_items.iter().rev() {
|
||||
match item {
|
||||
RolloutItem::EventMsg(EventMsg::ThreadRolledBack(rollback)) => {
|
||||
pending_rollback_turns = pending_rollback_turns
|
||||
.saturating_add(usize::try_from(rollback.num_turns).unwrap_or(usize::MAX));
|
||||
}
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(event)) => {
|
||||
let active_segment =
|
||||
active_segment.get_or_insert_with(ActiveEpiphanyReplaySegment::default);
|
||||
if active_segment.turn_id.is_none() {
|
||||
active_segment.turn_id = Some(event.turn_id.clone());
|
||||
}
|
||||
}
|
||||
RolloutItem::EventMsg(EventMsg::TurnAborted(event)) => {
|
||||
if let Some(active_segment) = active_segment.as_mut() {
|
||||
if active_segment.turn_id.is_none()
|
||||
&& let Some(turn_id) = &event.turn_id
|
||||
{
|
||||
active_segment.turn_id = Some(turn_id.clone());
|
||||
}
|
||||
} else if let Some(turn_id) = &event.turn_id {
|
||||
active_segment = Some(ActiveEpiphanyReplaySegment {
|
||||
turn_id: Some(turn_id.clone()),
|
||||
..Default::default()
|
||||
});
|
||||
}
|
||||
}
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(_)) => {
|
||||
let active_segment =
|
||||
active_segment.get_or_insert_with(ActiveEpiphanyReplaySegment::default);
|
||||
active_segment.counts_as_user_turn = true;
|
||||
}
|
||||
RolloutItem::ResponseItem(response_item) => {
|
||||
let active_segment =
|
||||
active_segment.get_or_insert_with(ActiveEpiphanyReplaySegment::default);
|
||||
active_segment.counts_as_user_turn |= is_user_turn_boundary(response_item);
|
||||
}
|
||||
RolloutItem::EpiphanyState(item) => {
|
||||
let active_segment =
|
||||
active_segment.get_or_insert_with(ActiveEpiphanyReplaySegment::default);
|
||||
if active_segment.turn_id.is_none() {
|
||||
active_segment.turn_id = item.turn_id.clone();
|
||||
}
|
||||
if turn_ids_are_compatible(
|
||||
active_segment.turn_id.as_deref(),
|
||||
item.turn_id.as_deref(),
|
||||
) && active_segment.epiphany_state.is_none()
|
||||
{
|
||||
active_segment.epiphany_state = Some(item.state.clone());
|
||||
}
|
||||
}
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(event)) => {
|
||||
if active_segment.as_ref().is_some_and(|active_segment| {
|
||||
turn_ids_are_compatible(
|
||||
active_segment.turn_id.as_deref(),
|
||||
Some(event.turn_id.as_str()),
|
||||
)
|
||||
}) && let Some(active_segment) = active_segment.take()
|
||||
{
|
||||
finalize_active_segment(
|
||||
active_segment,
|
||||
&mut epiphany_state,
|
||||
&mut pending_rollback_turns,
|
||||
);
|
||||
}
|
||||
}
|
||||
RolloutItem::EventMsg(_)
|
||||
| RolloutItem::Compacted(_)
|
||||
| RolloutItem::TurnContext(_)
|
||||
| RolloutItem::SessionMeta(_) => {}
|
||||
}
|
||||
|
||||
if epiphany_state.is_some() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(active_segment) = active_segment.take() {
|
||||
finalize_active_segment(
|
||||
active_segment,
|
||||
&mut epiphany_state,
|
||||
&mut pending_rollback_turns,
|
||||
);
|
||||
}
|
||||
|
||||
epiphany_state
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use codex_protocol::models::ContentItem;
|
||||
use codex_protocol::models::ResponseItem;
|
||||
use codex_protocol::protocol::CompactedItem;
|
||||
use codex_protocol::protocol::EpiphanyStateItem;
|
||||
use codex_protocol::protocol::ThreadRolledBackEvent;
|
||||
use codex_protocol::protocol::TurnCompleteEvent;
|
||||
use codex_protocol::protocol::TurnStartedEvent;
|
||||
use codex_protocol::protocol::UserMessageEvent;
|
||||
|
||||
fn user_response_message(text: &str) -> ResponseItem {
|
||||
ResponseItem::Message {
|
||||
id: None,
|
||||
role: "user".to_string(),
|
||||
content: vec![ContentItem::InputText {
|
||||
text: text.to_string(),
|
||||
}],
|
||||
end_turn: None,
|
||||
phase: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn sample_epiphany_state(turn_id: &str) -> EpiphanyThreadState {
|
||||
EpiphanyThreadState {
|
||||
revision: 1,
|
||||
objective: Some(format!("Objective for {turn_id}")),
|
||||
active_subgoal_id: Some(format!("subgoal-{turn_id}")),
|
||||
last_updated_turn_id: Some(turn_id.to_string()),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn latest_epiphany_state_from_rollout_items_returns_latest_surviving_snapshot() {
|
||||
let first = sample_epiphany_state("turn-1");
|
||||
let second = sample_epiphany_state("turn-2");
|
||||
let rollout_items = vec![
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-1".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent {
|
||||
message: "first".to_string(),
|
||||
images: None,
|
||||
text_elements: Vec::new(),
|
||||
local_images: Vec::new(),
|
||||
})),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-1".to_string()),
|
||||
state: first,
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-1".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-2".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::ResponseItem(user_response_message("second")),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-2".to_string()),
|
||||
state: second.clone(),
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-2".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
];
|
||||
|
||||
assert_eq!(
|
||||
latest_epiphany_state_from_rollout_items(&rollout_items),
|
||||
Some(second)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn latest_epiphany_state_from_rollout_items_skips_rolled_back_turns() {
|
||||
let first = sample_epiphany_state("turn-1");
|
||||
let second = sample_epiphany_state("turn-2");
|
||||
let rollout_items = vec![
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-1".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent {
|
||||
message: "first".to_string(),
|
||||
images: None,
|
||||
text_elements: Vec::new(),
|
||||
local_images: Vec::new(),
|
||||
})),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-1".to_string()),
|
||||
state: first.clone(),
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-1".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-2".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent {
|
||||
message: "second".to_string(),
|
||||
images: None,
|
||||
text_elements: Vec::new(),
|
||||
local_images: Vec::new(),
|
||||
})),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-2".to_string()),
|
||||
state: second,
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-2".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::ThreadRolledBack(ThreadRolledBackEvent {
|
||||
num_turns: 1,
|
||||
})),
|
||||
];
|
||||
|
||||
assert_eq!(
|
||||
latest_epiphany_state_from_rollout_items(&rollout_items),
|
||||
Some(first)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn latest_epiphany_state_from_rollout_items_survives_compaction() {
|
||||
let state = sample_epiphany_state("turn-before-compaction");
|
||||
let rollout_items = vec![
|
||||
RolloutItem::EventMsg(EventMsg::TurnStarted(TurnStartedEvent {
|
||||
turn_id: "turn-before-compaction".to_string(),
|
||||
started_at: None,
|
||||
model_context_window: None,
|
||||
collaboration_mode_kind: Default::default(),
|
||||
})),
|
||||
RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent {
|
||||
message: "before compaction".to_string(),
|
||||
images: None,
|
||||
text_elements: Vec::new(),
|
||||
local_images: Vec::new(),
|
||||
})),
|
||||
RolloutItem::EpiphanyState(EpiphanyStateItem {
|
||||
turn_id: Some("turn-before-compaction".to_string()),
|
||||
state: state.clone(),
|
||||
}),
|
||||
RolloutItem::EventMsg(EventMsg::TurnComplete(TurnCompleteEvent {
|
||||
turn_id: "turn-before-compaction".to_string(),
|
||||
last_agent_message: None,
|
||||
completed_at: None,
|
||||
duration_ms: None,
|
||||
time_to_first_token_ms: None,
|
||||
})),
|
||||
RolloutItem::Compacted(CompactedItem {
|
||||
message: "compact".to_string(),
|
||||
replacement_history: None,
|
||||
}),
|
||||
];
|
||||
|
||||
assert_eq!(
|
||||
latest_epiphany_state_from_rollout_items(&rollout_items),
|
||||
Some(state)
|
||||
);
|
||||
}
|
||||
epiphany_core::latest_epiphany_state_from_rollout_items(rollout_items, is_user_turn_boundary)
|
||||
}
|
||||
|
|
|
|||
Reference in a new issue