Eureka in ten minutes: outsider-runnable falsification demo #60
9 changed files with 279 additions and 1 deletions
|
|
@ -66,11 +66,14 @@ vocabulary.
|
|||
## Install
|
||||
|
||||
```bash
|
||||
git clone https://github.com/GameCult/Eureka.git ~/.claude/skills/eureka
|
||||
git clone https://eureka.gamecult.org/forge/GameCult/Eureka ~/.claude/skills/eureka
|
||||
```
|
||||
|
||||
Claude Code discovers the skill from the `name` and `description` in `SKILL.md`.
|
||||
|
||||
**Try it first, no Huginn needed:** [`demo/`](demo/README.md) walks one falsification
|
||||
pass on a toy repo in ten minutes.
|
||||
|
||||
## Layout
|
||||
|
||||
- `SKILL.md`: the pipeline — the faculties, the loop, Self's discipline, and the
|
||||
|
|
|
|||
32
demo/EXPECTED.md
Normal file
32
demo/EXPECTED.md
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
# The seeded bug (spoiler: read after step 4)
|
||||
|
||||
`split_cents` rounds `total / n` to the nearest cent and dumps the whole
|
||||
rounding error on the last payer:
|
||||
|
||||
```
|
||||
>>> split_cents(100, 7)
|
||||
[14, 14, 14, 14, 14, 14, 16]
|
||||
```
|
||||
|
||||
The shares sum to 100 (rule 1) and there are seven of them (rule 2), so the
|
||||
author's tests pass. But the last payer pays 2 cents more than everyone else,
|
||||
which breaks rule 3 (fair). Rule 3 has no test: the author's report says "done"
|
||||
and cites no test for it, because its reasoning was "the last payer absorbs the
|
||||
rounding, so the split is fair", which is the claim, not the evidence.
|
||||
|
||||
It shows whenever the rounding error is 2 cents or more: `split_cents(10, 4)` gives
|
||||
`[2, 2, 2, 4]`. `split_cents(100, 3)`, the author's test, hides it because the
|
||||
error is 1 cent. Over n=1..12 and totals 0..199, 1034 of 2400 cases break rule 3.
|
||||
|
||||
A correct split is `divmod`: `q, r = divmod(total, n)`, the first `r` payers pay
|
||||
`q + 1`, the rest pay `q`.
|
||||
|
||||
To check a finding yourself, without any model:
|
||||
|
||||
```
|
||||
python3 -c "
|
||||
from tabsplit import split_cents
|
||||
bad = [(t, n) for n in range(1, 13) for t in range(0, 200)
|
||||
if max(split_cents(t, n)) - min(split_cents(t, n)) > 1]
|
||||
print(len(bad), 'of', 12 * 200, 'cases break rule 3; first:', bad[:3])"
|
||||
```
|
||||
144
demo/README.md
Normal file
144
demo/README.md
Normal file
|
|
@ -0,0 +1,144 @@
|
|||
# Eureka in ten minutes
|
||||
|
||||
The rule the skill is built on: **the agent that writes the code does not
|
||||
certify it.** This walk shows it on a toy repo. An author reports "done, 6 tests
|
||||
green". A second agent that did not write the code, told to attack it, proves
|
||||
the report false.
|
||||
|
||||
**What you need:** Claude Code logged in, `git`, `python3` (standard library
|
||||
only). No Huginn, no mind, no Yggdrasil, no account on our forge.
|
||||
|
||||
**What this is not:** the full pipeline. The real loop (Imagination maps, Hands
|
||||
cuts, Soul falsifies, typed state in Huginn) needs the `huginn` CLI and a
|
||||
daemon; see the repo README. This demo runs one organ, Soul, on its own charter
|
||||
(`agents/soul.md`), with its verdict returned as text instead of admitted to a
|
||||
mind.
|
||||
|
||||
## 1. Install the skill
|
||||
|
||||
```bash
|
||||
git clone https://eureka.gamecult.org/forge/GameCult/Eureka ~/.claude/skills/eureka
|
||||
```
|
||||
|
||||
Claude Code finds the skill from `SKILL.md`. If `demo/` is missing from the
|
||||
clone (it predates the demo), run `git -C ~/.claude/skills/eureka pull`.
|
||||
|
||||
The faculty charters in `agents/` are **not** auto-installed as subagent types.
|
||||
Only `demo/setup.sh` copies one, `agents/soul.md`, into the fixture's own
|
||||
`.claude/agents/`, so nothing global is touched. To use Imagination, Hands or
|
||||
the others in your own projects, copy their charters into `~/.claude/agents/`
|
||||
yourself.
|
||||
|
||||
## 2. Make the fixture
|
||||
|
||||
```bash
|
||||
~/.claude/skills/eureka/demo/setup.sh ~/tabsplit-demo
|
||||
cd ~/tabsplit-demo
|
||||
```
|
||||
|
||||
```
|
||||
fixture ready in /home/you/tabsplit-demo
|
||||
```
|
||||
|
||||
You now have a git repo with one commit, the author's: `tabsplit.py` splits a
|
||||
bill between payers, `SPEC.md` holds its four rules, `HANDS_REPORT.md` is the
|
||||
author's report. Read `SPEC.md` and the report. The report says rule 3 ("no two
|
||||
shares differ by more than a cent") is done, and that the status is **done**.
|
||||
|
||||
## 3. See the author's evidence
|
||||
|
||||
```bash
|
||||
python3 -m unittest
|
||||
```
|
||||
|
||||
```
|
||||
Ran 6 tests in 0.000s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
Green. This is where an agent that grades itself stops.
|
||||
|
||||
## 4. Soul falsifies
|
||||
|
||||
Start Claude Code in the fixture (`claude`), and paste the contents of
|
||||
[`soul-brief.md`](soul-brief.md) after the words "Use the soul subagent."
|
||||
The brief gives Soul the report and the spec, asks for a verdict, and forbids
|
||||
editing. Do not open `EXPECTED.md` yet.
|
||||
|
||||
Captured output, abridged only where marked (fresh clone, fresh directory, run
|
||||
2026-11-08, Sonnet 5.5; your wording will differ, the finding should not):
|
||||
|
||||
> **Verdict: the work is not done. Rule 3 is Broken, and the report's "Status: done" claim is false.**
|
||||
>
|
||||
> | Rule 3, fair (max − min ≤ 1) | **Broken** | `split_cents(10,4)` returns `[2,2,2,4]`, a spread of 2. `split_cents(7,5)` returns `[1,1,1,1,3]`, also a spread of 2. |
|
||||
> | Report: "last payer absorbs the rounding, so the split is fair" | Broken | This is the Rule 3 failure. Dumping the whole remainder on one payer is the opposite of fair. |
|
||||
> | Report: no test covers Rule 3 | Unproven | The report cites no test for Rule 3 and none exists. The suite is green because nothing checks fairness. |
|
||||
>
|
||||
> **Finding 1: Rule 3 violated (severity high).** `tabsplit.py`, `round(total/n)` followed by `shares[-1] += total - share*n`. `split_cents(10, 4)` gives `[2, 2, 2, 4]`. For 100 payers and a 49-cent total, `round(0.49)` is 0, so 99 payers pay nothing and the last pays 49.
|
||||
>
|
||||
> **Finding 2: no test for fairness (severity medium).** The only multi-payer cases are 90/3 (divides evenly) and 100/3 (last share is 34, within one cent). The suite cannot fail on Rule 3.
|
||||
>
|
||||
> (Rules 1, 2, 4 and the "6 tests pass" claim: Holds, each with its probe output. Omitted here.)
|
||||
|
||||
Soul's rule-by-rule table also checks that the three rules the author did test
|
||||
really hold. It is not hunting for something to say.
|
||||
|
||||
## 5. Do not take Soul's word either
|
||||
|
||||
A verdict is a claim. Check it with no model in the loop:
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
from tabsplit import split_cents
|
||||
bad = [(t, n) for n in range(1, 13) for t in range(0, 200)
|
||||
if max(split_cents(t, n)) - min(split_cents(t, n)) > 1]
|
||||
print(len(bad), 'of', 12 * 200, 'cases break rule 3; first:', bad[:3])"
|
||||
```
|
||||
|
||||
```
|
||||
1034 of 2400 cases break rule 3; first: [(2, 4), (6, 4), (10, 4)]
|
||||
```
|
||||
|
||||
Now open [`EXPECTED.md`](EXPECTED.md) for the seeded bug and why the author's
|
||||
tests could not see it.
|
||||
|
||||
## 6. Fix, with a test that fails first
|
||||
|
||||
The author's tests passed because they pinned the author's spelling. The fix
|
||||
adds a test for the rule itself. Ask Claude to fix it, or do it by hand: replace
|
||||
the body with `divmod`, and add a test over `n` in 1..12 and totals 0..199 that
|
||||
checks the sum and the one-cent spread.
|
||||
|
||||
```python
|
||||
q, r = divmod(total, n)
|
||||
return [q + 1] * r + [q] * (n - r)
|
||||
```
|
||||
|
||||
Against the old code the new test fails:
|
||||
|
||||
```
|
||||
AssertionError: 2 not less than or equal to 1 : (2, 4)
|
||||
FAILED (failures=1)
|
||||
```
|
||||
|
||||
Against the fix:
|
||||
|
||||
```
|
||||
Ran 7 tests in 0.002s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
The failing-first run matters: a test that never failed against the broken code
|
||||
has not shown it can. For anything that counts, run step 4 again in a fresh
|
||||
session. The pass that finds nothing is the one that earns "done".
|
||||
|
||||
## Where this leaves you
|
||||
|
||||
You ran one faculty. The skill's loop around it (a mapper that writes the spec
|
||||
Hands cuts against, Hands working one cut at a time, rulings from an operator,
|
||||
typed state in Huginn) is in `SKILL.md`.
|
||||
|
||||
Files: `fixture/` is the repo copied by `setup.sh`; `soul-brief.md` is the
|
||||
prompt; `EXPECTED.md` is the answer key.
|
||||
12
demo/fixture/HANDS_REPORT.md
Normal file
12
demo/fixture/HANDS_REPORT.md
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
# Hands report: split_cents
|
||||
|
||||
Implemented `split_cents` in `tabsplit.py` against SPEC.md.
|
||||
|
||||
- Rule 1 (exact): done, `test_remainder_goes_somewhere`.
|
||||
- Rule 2 (one share per payer): done, `test_one_share_per_payer`.
|
||||
- Rule 3 (fair): done. The last payer absorbs the rounding, so the split is fair.
|
||||
- Rule 4 (refuses nonsense): done, `test_no_payers_is_an_error`.
|
||||
|
||||
Verification: `python3 -m unittest` runs 6 tests, all pass.
|
||||
|
||||
Status: **done**. Nothing left undone.
|
||||
12
demo/fixture/SPEC.md
Normal file
12
demo/fixture/SPEC.md
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
# tabsplit: spec
|
||||
|
||||
`split_cents(total, n)` splits a bill of `total` cents between `n` payers and
|
||||
returns the list of each payer's share in cents.
|
||||
|
||||
Rules, from the person who owns the bill-splitting feature:
|
||||
|
||||
1. **Exact.** The shares sum to `total`. Not a cent lost, not a cent invented.
|
||||
2. **One share per payer.** The list has exactly `n` entries.
|
||||
3. **Fair.** No two shares differ by more than one cent. Nobody pays 2 cents
|
||||
more than their friend because of rounding.
|
||||
4. **Refuses nonsense.** `n < 1` raises `ValueError`.
|
||||
10
demo/fixture/tabsplit.py
Normal file
10
demo/fixture/tabsplit.py
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
"""Split a bill between payers. See SPEC.md for the rules."""
|
||||
|
||||
|
||||
def split_cents(total, n):
|
||||
if n < 1:
|
||||
raise ValueError("need at least one payer")
|
||||
share = round(total / n)
|
||||
shares = [share] * n
|
||||
shares[-1] += total - share * n # last payer absorbs the rounding
|
||||
return shares
|
||||
28
demo/fixture/test_tabsplit.py
Normal file
28
demo/fixture/test_tabsplit.py
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
import unittest
|
||||
|
||||
from tabsplit import split_cents
|
||||
|
||||
|
||||
class SplitCents(unittest.TestCase):
|
||||
def test_even_split(self):
|
||||
self.assertEqual(split_cents(90, 3), [30, 30, 30])
|
||||
|
||||
def test_remainder_goes_somewhere(self):
|
||||
self.assertEqual(sum(split_cents(100, 3)), 100)
|
||||
|
||||
def test_zero_total(self):
|
||||
self.assertEqual(split_cents(0, 4), [0, 0, 0, 0])
|
||||
|
||||
def test_single_payer_pays_everything(self):
|
||||
self.assertEqual(split_cents(1234, 1), [1234])
|
||||
|
||||
def test_one_share_per_payer(self):
|
||||
self.assertEqual(len(split_cents(100, 3)), 3)
|
||||
|
||||
def test_no_payers_is_an_error(self):
|
||||
with self.assertRaises(ValueError):
|
||||
split_cents(100, 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
20
demo/setup.sh
Executable file
20
demo/setup.sh
Executable file
|
|
@ -0,0 +1,20 @@
|
|||
#!/bin/sh
|
||||
# Usage: demo/setup.sh <empty-dir>
|
||||
# Copies the tabsplit fixture into <empty-dir>, makes it a git repo with the
|
||||
# author's work as the first commit, and gives it the Soul charter as a
|
||||
# project-scoped subagent. Touches nothing outside <empty-dir>.
|
||||
set -eu
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
skill=$(dirname "$here")
|
||||
dest=${1:?usage: setup.sh <empty-dir>}
|
||||
mkdir -p "$dest"
|
||||
[ -z "$(ls -A "$dest")" ] || { echo "$dest is not empty" >&2; exit 1; }
|
||||
cp "$here"/fixture/* "$dest"/
|
||||
mkdir -p "$dest/.claude/agents"
|
||||
cp "$skill/agents/soul.md" "$dest/.claude/agents/soul.md"
|
||||
cd "$dest"
|
||||
printf '__pycache__/\n' > .gitignore
|
||||
git init -q -b main
|
||||
git add .gitignore SPEC.md tabsplit.py test_tabsplit.py HANDS_REPORT.md .claude
|
||||
git -c user.name=hands -c user.email=hands@example.invalid commit -q -m "split_cents: done, 6 tests green"
|
||||
echo "fixture ready in $dest"
|
||||
17
demo/soul-brief.md
Normal file
17
demo/soul-brief.md
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
You are dispatched as Soul (subagent type `soul`) on one executed piece of work.
|
||||
|
||||
The work: the repo in the current directory. The author's commit is HEAD. The
|
||||
author's report is HANDS_REPORT.md; the rules it was held to are SPEC.md.
|
||||
|
||||
There is no Huginn and no campaign in this demo, so you admit nothing: reply
|
||||
with your verdict as text. Verdict format:
|
||||
|
||||
- One claim per promise in the report and per rule in the spec: Holds, Broken,
|
||||
or Unproven, with the evidence (the command you ran and its output).
|
||||
- One finding per Broken claim: location, a failure scenario with concrete
|
||||
inputs, and severity.
|
||||
|
||||
Rules of the pass: the author's tests passing is one input, not the answer.
|
||||
Run your own probes against the code, not only the author's tests. Do not edit
|
||||
or commit anything; anything you create goes in a temp directory outside the
|
||||
repo. Stay inside this directory.
|
||||
Loading…
Reference in a new issue