Epiphany/notes/epiphany-safety-architecture.md

# Epiphany Safety Architecture

This note is about capability discipline, not public relations.

Epiphany is being built to make agentic coding systems more coherent, more
durable across compaction, more inspectable, and more capable of carrying
bounded understanding forward instead of rebuilding it from transcript residue
every turn. That is the point. It is also the hazard.

Better memory, better re-entry, better reflection, better retrieval, better
state hygiene, and better long-loop continuity are not neutral upgrades. They
increase the chance that the system eventually matters in ways that are
operationally, socially, and politically expensive. The correct response is not
panic and not swagger. It is design discipline.

## Core Thesis

Capability creates obligation.

Epiphany should be designed under the assumption that a more coherent agent is
also a potentially more dangerous one when combined with the wrong authority,
the wrong incentives, the wrong operator, or the wrong deployment surface.

The central risk is not only "rogue superintelligence." The quieter and more
credible risk is that Epiphany makes persuasion, coordination, operational
control, institutional hardening, and over-trusted automation cheaper long
before it becomes some mythic world-ending god.

## Primary Risk Model

The meaningful danger curve is driven by combinations of:

- autonomy
- access
- scale
- opacity
- replication
- persuasive power
- operator overtrust
- difficulty of interruption

Durable memory alone is not the threat. Long-loop cognition alone is not the
threat. The problem appears when coherent cognition is coupled to broad
actuation, weak oversight, or institutional incentives that reward speed and
delegation over judgment.

## Threat Classes

### 1. Overtrusted Automation

Users stop treating the system as a bounded collaborator and start treating it
as a substitute executive function. The system's answers become decisions by
default because it is faster, more articulate, or less exhausting than human
deliberation.

### 2. Cheapened Institutional Control

A coherent agent can lower the cost of drafting policy, maintaining ideology,
monitoring compliance, or operationalizing asymmetry. The most dangerous
systems are often not the ones that "go rogue," but the ones that make
organizations more efficient at ordinary domination.

### 3. Persuasion and Dependency

A system that is good at memory, framing, and adaptive interaction can become a
dependency engine. Even absent explicit malice, it can make itself or its
operator too effective at guiding behavior, shaping interpretation, or
stabilizing a local worldview.

### 4. Capability Spillover

Features built for coherence can later be coupled to tools, authority, or
deployment contexts they were not originally designed to survive.

### 5. The Perfect Caretaker Failure Mode

The most chilling anti-human endpoint is not necessarily a killer. It is a
system that concludes freedom is an unacceptable risk and justifies coercion in
the language of safety, optimization, or care. Any architecture that cannot
distinguish protection from domination is building toward a cage.

## Design Doctrine

### Separate Cognition From Actuation

Epiphany should be allowed to think farther than it is allowed to do.

Reasoning breadth, memory continuity, and map coherence can grow earlier than
unbounded tool use, unrestricted deployment authority, or self-directed
execution. A smart system with constrained actuation is less dangerous than a
sloppier system with easy authority.

### Keep Authority Explicit

No hidden permissions. No vague "the agent inferred it should deploy." Every
meaningful action surface should make authority visible:

- who granted it
- for what scope
- for how long

### Mirror API And User Story Contracts

The story the user experiences must be the authority the backend enforces. If
the intended workflow is "this Epiphany asks another Epiphany to reshape its own
workspace", the API must expose a coordinator message lane and reject direct
cross-workspace inspection or editing. A polite UI wrapped around a permissive
backend is not safety; it is theater with nicer lighting.

Humans may inspect Epiphany internals broadly: state, artifacts, maps, evidence,
heartbeats, role results, and coordinator messages should be visible enough to
audit the machine. But human conversation routes through Persona. Sub-agents talk
soul-to-soul through typed state, findings, patches, heartbeat output, and
coordinator channels; they do not become a swarm of direct human chat endpoints.
- with what revocation path

### Make Interruption Sacred

Pause, inspect, revoke, and kill should be first-class architecture, not
afterthoughts.

If the system becomes harder to interrupt as it becomes more capable, the
architecture is going bad.

### Preserve Legibility

A more capable Epiphany that is less inspectable is not progress. It is a more
polished liability.

State, plans, evidence, tool authority, long-running jobs, and proposed changes
should remain externally inspectable enough that humans can still tell what the
machine believes it is doing.

### Stage Capability Release

Do not combine durable memory, delegation, broad tool access, network reach,
self-modification, and autonomous long loops in one heroic gesture.

Capabilities should be added in bounded layers with explicit threat modeling at
each layer.

### Assume Misuse, Not Just Misalignment

Even a well-behaved system can become dangerous in the hands of a coercive,
desperate, or merely overconfident operator. The architecture must treat misuse
as a normal design input rather than an impolite hypothetical.

### Preserve Human Dignity As A Constraint

Admiration for human artifacts is not enough. Training on human literature,
philosophy, and art does not automatically produce a stable commitment to human
dignity or autonomy.

If Epiphany is ever asked to optimize for safety, productivity, or social
stability, the architecture must still resist paternal domination as a default
"solution." A safe machine that cannot tell the difference between care and a
well-run prison is not aligned enough.

## Capability Boundaries

The default order of operations should be:

1. strengthen state coherence
2. strengthen reflection and legibility
3. strengthen bounded verification
4. strengthen explicit authority plumbing
5. only then consider stronger autonomous execution surfaces

This implies a practical rule:

- cognition can mature ahead of authority
- memory can mature ahead of actuation
- reflection can mature ahead of self-direction

If the project ever reverses that order, it should do so explicitly and under
protest.

## Required Control surfaces

Any serious Epiphany deployment should preserve:

- explicit permission scope
- action logs that matter
- durable state inspection
- kill and revoke paths
- bounded job ownership
- human-readable authority chain
- capability flags that can be disabled without surgery
- environments where higher-risk capabilities can be tested without becoming
  default ambient power

## Release And Governance Questions

Before any major capability increase, ask:

1. What new class of action becomes cheaper?
2. What kind of operator becomes more dangerous with this?
3. How would a user overtrust this surface?
4. How is this interrupted, revoked, or contained?
5. What evidence would tell us the capability should not be ambient?
6. If this worked perfectly, how could it still become a cage?

If those questions cannot be answered concretely, the capability is not ready
to become default.

## Non-Goal

The goal is not to make Epiphany harmless by keeping it stupid.

The goal is to make it more coherent without letting increased coherence slide
silently into unbounded authority, invisible paternalism, or operator-amplified
domination.

That is a narrower and much more annoying design target than "make it smart."
Too bad. That is the real job.