GameCult/Idunn
Watch
9
0
Fork
You've already forked Idunn
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Routing proof waits out nginx's reload handoff and names who answered #8

Merged
eureka merged 2 commits from hands/routing-proof-handoff into main 2026-11-14 04:28:19 +00:00
Conversation 0 Commits 2 Files changed 1 +377 -10
eureka commented 2026-11-14 02:41:06 +00:00
Owner
Copy link

Cut idunn-deploy-provenance:cut_spec:cut-routing-proof-handoff.r1. Base main 7eae1c4. Not merged; PR #7 edits supervise_admitted_route in the same file and this PR does not touch that function, so expect a trivial rebase.

Cause

systemctl reload runs nginx -s reload, which only signals the master. Old workers keep accepting on the shared stream listen socket until the master has parsed the config, forked new workers and told the old ones to quit. Idunn challenged :8833 within milliseconds, got an answer from the incumbent (a presence that fails activation authentication), failed the Routing attempt, and rolled the fragment back with a second reload. The streampixels web deploy hit this (E1 web attempt 2).

My confidence in this explanation is 60%. The cut works either way: part (a) makes every failed proof name the runtime instance and activation signer that answered, so the next failure says whether it was the incumbent (race, rank 1, or nginx never switched, rank 2), the candidate itself (activation signer differs from the transaction record, rank 3) or a stranger.

Change (src/control_plane.rs only)

  • advance_routing calls prove_routed_candidate_through_handoff instead of prove_stable_route. An answer naming the preflight's incumbent instance (and not the candidate) is not-yet for 10 s, challenged every 250 ms (400 ms / 20 ms under test). One challenge per poll: the loop never installs, rolls back, charges the gate or persists. When the window ends the last error (which names the incumbent) takes the existing rollback arms, which are unchanged.
  • RouteProofAnswer context on the authentication failure in prove_stable_route_against: ids only, each cut to 96 bytes with control characters stripped, never signature bytes. It reaches the journal line and the abort's last_error through {error:#}.
  • Not-yet is decided from the unauthenticated runtime_instance_id on purpose: it can only delay, never admit. No provider-signature pre-check, no match on cultnet-rs error text.
  • Any other proof failure keeps its meaning: fails the attempt at once, rollback if allowed, retry under backoff.

Tests (stub runtime gains IncumbentFirst(n), ForeignActivation, StrangerInstance replies)

  • routing_waits_out_the_incumbent_answering_after_reload: 3 incumbent answers then honest promotes in one attempt; one reload, one Forward, four stable hits, no report.
  • routing_fails_once_when_the_incumbent_never_lets_go: fails after at least the window; install plus one rollback; report and error name incumbent and candidate; a second attempt charges one more Forward.
  • routing_fails_at_once_when_the_candidate_answers_with_a_foreign_activation and routing_fails_at_once_for_a_stranger_instance: one challenge, no wait.
  • route_proof_answer_names_ids_only: 10 KiB ids give at most 400 bytes, no signature hex.

Mutation-checked: disabling the window, matching any answerer, removing the id cut, and making the id decode return None each fail the tests that pin them.

What it gives up

  • Up to 10 s of Routing time per attempt when the incumbent keeps answering, which is a real nginx failure to switch. That costs one attempt's slice of the 120 s Routing deadline; the window leaves room for four more backoff-spaced attempts.
  • supervise_admitted_route (restore then prove) and the committing re-proof keep the reload-then-challenge shape with no wait; they self-heal on the next backoff pass. Folding connect_route_socket's refused retry into this loop is idea routing-proof-handoff.collapse-connect-retry, after PR #7.
  • The changes_route() guard on the wait has no dedicated test.
  • The scratch-nginx probe saw the old upstream answer after the reload on every reload; 20 of 20 reloads saw an old-upstream answer, longest tail 121 ms idle and 130 ms with 16 busy loops (no stress-ng here); the 10 s window has two orders of magnitude of slack over that.

🤖 Generated with Claude Code

Cut `idunn-deploy-provenance:cut_spec:cut-routing-proof-handoff.r1`. Base main 7eae1c4. Not merged; PR #7 edits `supervise_admitted_route` in the same file and this PR does not touch that function, so expect a trivial rebase. ## Cause `systemctl reload` runs `nginx -s reload`, which only signals the master. Old workers keep accepting on the shared stream listen socket until the master has parsed the config, forked new workers and told the old ones to quit. Idunn challenged :8833 within milliseconds, got an answer from the incumbent (a presence that fails activation authentication), failed the Routing attempt, and rolled the fragment back with a second reload. The streampixels web deploy hit this (E1 web attempt 2). My confidence in this explanation is 60%. The cut works either way: part (a) makes every failed proof name the runtime instance and activation signer that answered, so the next failure says whether it was the incumbent (race, rank 1, or nginx never switched, rank 2), the candidate itself (activation signer differs from the transaction record, rank 3) or a stranger. ## Change (src/control_plane.rs only) - `advance_routing` calls `prove_routed_candidate_through_handoff` instead of `prove_stable_route`. An answer naming the preflight's incumbent instance (and not the candidate) is not-yet for 10 s, challenged every 250 ms (400 ms / 20 ms under test). One challenge per poll: the loop never installs, rolls back, charges the gate or persists. When the window ends the last error (which names the incumbent) takes the existing rollback arms, which are unchanged. - `RouteProofAnswer` context on the authentication failure in `prove_stable_route_against`: ids only, each cut to 96 bytes with control characters stripped, never signature bytes. It reaches the journal line and the abort's `last_error` through `{error:#}`. - Not-yet is decided from the unauthenticated `runtime_instance_id` on purpose: it can only delay, never admit. No provider-signature pre-check, no match on cultnet-rs error text. - Any other proof failure keeps its meaning: fails the attempt at once, rollback if allowed, retry under backoff. ## Tests (stub runtime gains IncumbentFirst(n), ForeignActivation, StrangerInstance replies) - routing_waits_out_the_incumbent_answering_after_reload: 3 incumbent answers then honest promotes in one attempt; one reload, one Forward, four stable hits, no report. - routing_fails_once_when_the_incumbent_never_lets_go: fails after at least the window; install plus one rollback; report and error name incumbent and candidate; a second attempt charges one more Forward. - routing_fails_at_once_when_the_candidate_answers_with_a_foreign_activation and routing_fails_at_once_for_a_stranger_instance: one challenge, no wait. - route_proof_answer_names_ids_only: 10 KiB ids give at most 400 bytes, no signature hex. Mutation-checked: disabling the window, matching any answerer, removing the id cut, and making the id decode return None each fail the tests that pin them. ## What it gives up - Up to 10 s of Routing time per attempt when the incumbent keeps answering, which is a real nginx failure to switch. That costs one attempt's slice of the 120 s Routing deadline; the window leaves room for four more backoff-spaced attempts. - `supervise_admitted_route` (restore then prove) and the committing re-proof keep the reload-then-challenge shape with no wait; they self-heal on the next backoff pass. Folding `connect_route_socket`'s refused retry into this loop is idea routing-proof-handoff.collapse-connect-retry, after PR #7. - The `changes_route()` guard on the wait has no dedicated test. - The scratch-nginx probe saw the old upstream answer after the reload on every reload; 20 of 20 reloads saw an old-upstream answer, longest tail 121 ms idle and 130 ms with 16 busy loops (no stress-ng here); the 10 s window has two orders of magnitude of slack over that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) 
systemctl reload only signals nginx's master; old workers keep answering
the stable listener until the new config is parsed and they are told to
quit, so Idunn's first challenge could reach the incumbent and fail the
attempt, rolling the fragment back. An answer naming the preflight's
incumbent instance is now not-yet for 10 s (one challenge per 250 ms,
never admitting, never actuating). Every failed proof now carries a
RouteProofAnswer context: the runtime instance and activation signer ids
the answer claimed (unauthenticated, ids only, 96 bytes each), so the
journal line and abort last_error say who answered.

Tests drive Routing through the stub runtime with incumbent-first,
never-lets-go, foreign-activation and stranger replies.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The handoff loop reads its clock through a test-held virtual clock, so the
window and poll cadence are asserted exactly and no host load moves them; the
fails-at-once tests assert through stable_hits and the absence of sleeps.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
eureka merged commit 17edea299e into main 2026-11-14 04:28:19 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
GameCult/Idunn!8
No description provided.