Eureka in ten minutes: outsider-runnable falsification demo #60

Merged
eureka merged 4 commits from eureka-substrate/ten-minute-demo into main 2026-11-08 05:49:13 +00:00
9 changed files with 279 additions and 1 deletions

View file

@ -66,11 +66,14 @@ vocabulary.
## Install
```bash
git clone https://github.com/GameCult/Eureka.git ~/.claude/skills/eureka
git clone https://eureka.gamecult.org/forge/GameCult/Eureka ~/.claude/skills/eureka
```
Claude Code discovers the skill from the `name` and `description` in `SKILL.md`.
**Try it first, no Huginn needed:** [`demo/`](demo/README.md) walks one falsification
pass on a toy repo in ten minutes.
## Layout
- `SKILL.md`: the pipeline — the faculties, the loop, Self's discipline, and the

32
demo/EXPECTED.md Normal file
View file

@ -0,0 +1,32 @@
# The seeded bug (spoiler: read after step 4)
`split_cents` rounds `total / n` to the nearest cent and dumps the whole
rounding error on the last payer:
```
>>> split_cents(100, 7)
[14, 14, 14, 14, 14, 14, 16]
```
The shares sum to 100 (rule 1) and there are seven of them (rule 2), so the
author's tests pass. But the last payer pays 2 cents more than everyone else,
which breaks rule 3 (fair). Rule 3 has no test: the author's report says "done"
and cites no test for it, because its reasoning was "the last payer absorbs the
rounding, so the split is fair", which is the claim, not the evidence.
It shows whenever the rounding error is 2 cents or more: `split_cents(10, 4)` gives
`[2, 2, 2, 4]`. `split_cents(100, 3)`, the author's test, hides it because the
error is 1 cent. Over n=1..12 and totals 0..199, 1034 of 2400 cases break rule 3.
A correct split is `divmod`: `q, r = divmod(total, n)`, the first `r` payers pay
`q + 1`, the rest pay `q`.
To check a finding yourself, without any model:
```
python3 -c "
from tabsplit import split_cents
bad = [(t, n) for n in range(1, 13) for t in range(0, 200)
if max(split_cents(t, n)) - min(split_cents(t, n)) > 1]
print(len(bad), 'of', 12 * 200, 'cases break rule 3; first:', bad[:3])"
```

144
demo/README.md Normal file
View file

@ -0,0 +1,144 @@
# Eureka in ten minutes
The rule the skill is built on: **the agent that writes the code does not
certify it.** This walk shows it on a toy repo. An author reports "done, 6 tests
green". A second agent that did not write the code, told to attack it, proves
the report false.
**What you need:** Claude Code logged in, `git`, `python3` (standard library
only). No Huginn, no mind, no Yggdrasil, no account on our forge.
**What this is not:** the full pipeline. The real loop (Imagination maps, Hands
cuts, Soul falsifies, typed state in Huginn) needs the `huginn` CLI and a
daemon; see the repo README. This demo runs one organ, Soul, on its own charter
(`agents/soul.md`), with its verdict returned as text instead of admitted to a
mind.
## 1. Install the skill
```bash
git clone https://eureka.gamecult.org/forge/GameCult/Eureka ~/.claude/skills/eureka
```
Claude Code finds the skill from `SKILL.md`. If `demo/` is missing from the
clone (it predates the demo), run `git -C ~/.claude/skills/eureka pull`.
The faculty charters in `agents/` are **not** auto-installed as subagent types.
Only `demo/setup.sh` copies one, `agents/soul.md`, into the fixture's own
`.claude/agents/`, so nothing global is touched. To use Imagination, Hands or
the others in your own projects, copy their charters into `~/.claude/agents/`
yourself.
## 2. Make the fixture
```bash
~/.claude/skills/eureka/demo/setup.sh ~/tabsplit-demo
cd ~/tabsplit-demo
```
```
fixture ready in /home/you/tabsplit-demo
```
You now have a git repo with one commit, the author's: `tabsplit.py` splits a
bill between payers, `SPEC.md` holds its four rules, `HANDS_REPORT.md` is the
author's report. Read `SPEC.md` and the report. The report says rule 3 ("no two
shares differ by more than a cent") is done, and that the status is **done**.
## 3. See the author's evidence
```bash
python3 -m unittest
```
```
Ran 6 tests in 0.000s
OK
```
Green. This is where an agent that grades itself stops.
## 4. Soul falsifies
Start Claude Code in the fixture (`claude`), and paste the contents of
[`soul-brief.md`](soul-brief.md) after the words "Use the soul subagent."
The brief gives Soul the report and the spec, asks for a verdict, and forbids
editing. Do not open `EXPECTED.md` yet.
Captured output, abridged only where marked (fresh clone, fresh directory, run
2026-11-08, Sonnet 5.5; your wording will differ, the finding should not):
> **Verdict: the work is not done. Rule 3 is Broken, and the report's "Status: done" claim is false.**
>
> | Rule 3, fair (max − min ≤ 1) | **Broken** | `split_cents(10,4)` returns `[2,2,2,4]`, a spread of 2. `split_cents(7,5)` returns `[1,1,1,1,3]`, also a spread of 2. |
> | Report: "last payer absorbs the rounding, so the split is fair" | Broken | This is the Rule 3 failure. Dumping the whole remainder on one payer is the opposite of fair. |
> | Report: no test covers Rule 3 | Unproven | The report cites no test for Rule 3 and none exists. The suite is green because nothing checks fairness. |
>
> **Finding 1: Rule 3 violated (severity high).** `tabsplit.py`, `round(total/n)` followed by `shares[-1] += total - share*n`. `split_cents(10, 4)` gives `[2, 2, 2, 4]`. For 100 payers and a 49-cent total, `round(0.49)` is 0, so 99 payers pay nothing and the last pays 49.
>
> **Finding 2: no test for fairness (severity medium).** The only multi-payer cases are 90/3 (divides evenly) and 100/3 (last share is 34, within one cent). The suite cannot fail on Rule 3.
>
> (Rules 1, 2, 4 and the "6 tests pass" claim: Holds, each with its probe output. Omitted here.)
Soul's rule-by-rule table also checks that the three rules the author did test
really hold. It is not hunting for something to say.
## 5. Do not take Soul's word either
A verdict is a claim. Check it with no model in the loop:
```bash
python3 -c "
from tabsplit import split_cents
bad = [(t, n) for n in range(1, 13) for t in range(0, 200)
if max(split_cents(t, n)) - min(split_cents(t, n)) > 1]
print(len(bad), 'of', 12 * 200, 'cases break rule 3; first:', bad[:3])"
```
```
1034 of 2400 cases break rule 3; first: [(2, 4), (6, 4), (10, 4)]
```
Now open [`EXPECTED.md`](EXPECTED.md) for the seeded bug and why the author's
tests could not see it.
## 6. Fix, with a test that fails first
The author's tests passed because they pinned the author's spelling. The fix
adds a test for the rule itself. Ask Claude to fix it, or do it by hand: replace
the body with `divmod`, and add a test over `n` in 1..12 and totals 0..199 that
checks the sum and the one-cent spread.
```python
q, r = divmod(total, n)
return [q + 1] * r + [q] * (n - r)
```
Against the old code the new test fails:
```
AssertionError: 2 not less than or equal to 1 : (2, 4)
FAILED (failures=1)
```
Against the fix:
```
Ran 7 tests in 0.002s
OK
```
The failing-first run matters: a test that never failed against the broken code
has not shown it can. For anything that counts, run step 4 again in a fresh
session. The pass that finds nothing is the one that earns "done".
## Where this leaves you
You ran one faculty. The skill's loop around it (a mapper that writes the spec
Hands cuts against, Hands working one cut at a time, rulings from an operator,
typed state in Huginn) is in `SKILL.md`.
Files: `fixture/` is the repo copied by `setup.sh`; `soul-brief.md` is the
prompt; `EXPECTED.md` is the answer key.

View file

@ -0,0 +1,12 @@
# Hands report: split_cents
Implemented `split_cents` in `tabsplit.py` against SPEC.md.
- Rule 1 (exact): done, `test_remainder_goes_somewhere`.
- Rule 2 (one share per payer): done, `test_one_share_per_payer`.
- Rule 3 (fair): done. The last payer absorbs the rounding, so the split is fair.
- Rule 4 (refuses nonsense): done, `test_no_payers_is_an_error`.
Verification: `python3 -m unittest` runs 6 tests, all pass.
Status: **done**. Nothing left undone.

12
demo/fixture/SPEC.md Normal file
View file

@ -0,0 +1,12 @@
# tabsplit: spec
`split_cents(total, n)` splits a bill of `total` cents between `n` payers and
returns the list of each payer's share in cents.
Rules, from the person who owns the bill-splitting feature:
1. **Exact.** The shares sum to `total`. Not a cent lost, not a cent invented.
2. **One share per payer.** The list has exactly `n` entries.
3. **Fair.** No two shares differ by more than one cent. Nobody pays 2 cents
more than their friend because of rounding.
4. **Refuses nonsense.** `n < 1` raises `ValueError`.

10
demo/fixture/tabsplit.py Normal file
View file

@ -0,0 +1,10 @@
"""Split a bill between payers. See SPEC.md for the rules."""
def split_cents(total, n):
if n < 1:
raise ValueError("need at least one payer")
share = round(total / n)
shares = [share] * n
shares[-1] += total - share * n # last payer absorbs the rounding
return shares

View file

@ -0,0 +1,28 @@
import unittest
from tabsplit import split_cents
class SplitCents(unittest.TestCase):
def test_even_split(self):
self.assertEqual(split_cents(90, 3), [30, 30, 30])
def test_remainder_goes_somewhere(self):
self.assertEqual(sum(split_cents(100, 3)), 100)
def test_zero_total(self):
self.assertEqual(split_cents(0, 4), [0, 0, 0, 0])
def test_single_payer_pays_everything(self):
self.assertEqual(split_cents(1234, 1), [1234])
def test_one_share_per_payer(self):
self.assertEqual(len(split_cents(100, 3)), 3)
def test_no_payers_is_an_error(self):
with self.assertRaises(ValueError):
split_cents(100, 0)
if __name__ == "__main__":
unittest.main()

20
demo/setup.sh Executable file
View file

@ -0,0 +1,20 @@
#!/bin/sh
# Usage: demo/setup.sh <empty-dir>
# Copies the tabsplit fixture into <empty-dir>, makes it a git repo with the
# author's work as the first commit, and gives it the Soul charter as a
# project-scoped subagent. Touches nothing outside <empty-dir>.
set -eu
here=$(cd "$(dirname "$0")" && pwd)
skill=$(dirname "$here")
dest=${1:?usage: setup.sh <empty-dir>}
mkdir -p "$dest"
[ -z "$(ls -A "$dest")" ] || { echo "$dest is not empty" >&2; exit 1; }
cp "$here"/fixture/* "$dest"/
mkdir -p "$dest/.claude/agents"
cp "$skill/agents/soul.md" "$dest/.claude/agents/soul.md"
cd "$dest"
printf '__pycache__/\n' > .gitignore
git init -q -b main
git add .gitignore SPEC.md tabsplit.py test_tabsplit.py HANDS_REPORT.md .claude
git -c user.name=hands -c user.email=hands@example.invalid commit -q -m "split_cents: done, 6 tests green"
echo "fixture ready in $dest"

17
demo/soul-brief.md Normal file
View file

@ -0,0 +1,17 @@
You are dispatched as Soul (subagent type `soul`) on one executed piece of work.
The work: the repo in the current directory. The author's commit is HEAD. The
author's report is HANDS_REPORT.md; the rules it was held to are SPEC.md.
There is no Huginn and no campaign in this demo, so you admit nothing: reply
with your verdict as text. Verdict format:
- One claim per promise in the report and per rule in the spec: Holds, Broken,
or Unproven, with the evidence (the command you ran and its output).
- One finding per Broken claim: location, a failure scenario with concrete
inputs, and severity.
Rules of the pass: the author's tests passing is one input, not the answer.
Run your own probes against the code, not only the author's tests. Do not edit
or commit anything; anything you create goes in a temp directory outside the
repo. Stay inside this directory.