GameCult/Bifrost
Watch
9
0
Fork
You've already forked Bifrost
 0
Code Issues 7 Pull requests 5 Projects Releases Packages Wiki Activity Actions

Merge pull request 'Revise the main-to-live deploy plan against 55b74275 (Libby's 11 points)' (#18) from eureka/deploy-plan-revise into heimdall/deploy-main-plan

Browse source
This commit is contained in:
Heimdall 2026-11-09 15:20:08 +00:00
commit 0eb68e498e
1 changed files with 126 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

171
docs/deploy-main-to-live.md
View file

@ -1,15 +1,16 @@
# Deploying main over the live release
Status: **plan, not executed.** Written 2026-11-09 by the heimdall Self run. Nothing here has been run. The deploy
waits on a root window, a verified database backup and a rehearsed rollback (Preconditions).
Status: **plan, not executed.** Written 2026-11-09 by the heimdall Self run; revised the same day by the eureka Self run
against main `55b74275`, from Libby's read of the first draft. Nothing here has been run. The deploy waits on a root
window, a verified database backup and a rehearsed rollback (Preconditions).
## What moves
| | Live | Target |
| --- | --- | --- |
| Release | `/srv/bifrost/app/releases/20261010043032-67072b2` | a new `<UTC>-<sha7>` directory |
| Revision | `67072b2` | `244811cb49bf65a2325792882e2d97e11115ca17` (main at writing; re-check) |
| Distance | | 141 commits; `67072b2` is an ancestor of main |
| Revision | `67072b2` | `55b74275b0d86cdb0c5db01010380a90fbc2eb69` (main at revision; re-check with `git ls-remote`) |
| Distance | | 147 commits; `67072b2` is an ancestor of main |
| Schema | up to the last pre-`AddConceptVotes` migration | + `20261010043937_AddConceptVotes`, + `20261022163159_AddOpenIddict` |
Both migrations only create tables and indexes (`Up` has no `Drop*`, `AlterColumn` or `RenameColumn`). The previous
@ -17,26 +18,51 @@ binary therefore runs against the new schema, which is what makes the rollback b
The deploy mechanism is `gamecult-ops/scripts/deploy-bifrost-alpha.sh <sha40>` and
`runbooks/bifrost-alpha-deploy.md`. There is no Idunn recipe for Bifrost; this is a manual, root-window deploy.
**Bifrost #11** (`__Host-` cookies) is not in main at writing and rides this deploy: merge it first, or the deploy is
repeated.
**Bifrost #11** (`__Host-` cookies) and #17 (CI publish workflow deleted) are in main; this page (#15) is docs only and
changes no code. #11 renames every cookie Bifrost sets (`BifrostCookies.cs`: session, OAuth correlation, antiforgery,
voter, Heimdall attempt), so **the deploy signs every member out once**, drops sign-ins in flight at the restart, and
gives each returning anonymous voter a new device cookie. Say so to the members before the window. The data-protection
key ring is unchanged, so this is the only forced sign-out; the live check below tests it.
## Configuration that changed
## Configuration
Read against the live `/srv/bifrost/env/bifrost.env` before the window, not during it. Key names only; never paste values
into a chat, a PR or a log.
Read against the live `/srv/bifrost/env/bifrost.env` before the window, not during it, by someone who can read it (mode
600, owner `bifrost`; the eureka Self run's account cannot, so **no key below is marked present or absent here**: the
column is filled in by the reader). Key names only; never paste values into a chat, a PR or a log. Name-only check:
`sudo -n cut -d= -f1 /srv/bifrost/env/bifrost.env | sort`.
- **`Heimdall__AppSecret`: new, required for sign-in.** Without it, `HeimdallOptions.IsDiscordConfigured` and
`IsPatreonConfigured` are false and Discord and Patreon sign-in stop working (GitHub sign-in is unaffected). The value is
Heimdall's app secret for slug `bifrost` (`GC_ACCESS_APP_BIFROST_SHARED_SECRET` on the Heimdall side). Bifrost sends it
as `x-heimdall-app-secret`, redacted from logs.
- **`ConceptVotes__NetworkTagSecret`**: at least 32 characters; keys the anonymous vote network tag. Check
`ConceptVoteOptions` for the other `ConceptVotes__*` keys and whether the vote surface refuses to start without it.
Outside Development and Testing, `StartupConfigurationValidator.Validate()` refuses to start (an `InvalidOperationException`
naming every failure) unless all of these hold. Environment-variable spelling uses `__` for `:`.
| Key (configuration name) | Rule | Present? |
| --- | --- | --- |
| `GitHubOAuth:ClientId`, `GitHubOAuth:ClientSecret` | both non-empty | |
| `Host:PublicBaseUrl`, `Host:ExpectedHost` | both non-empty | |
| `AllowedHosts` | non-empty and no `*` (while `Host:RequireStrictHostValidation` is true, its default) | |
| `ConnectionStrings:Bifrost` | non-empty | |
| `DataProtection:KeysPath` | non-empty | |
| **`ConceptVotes:NetworkTagSecret`** | non-empty and **at least 32 characters** (`ConceptVoteOptions.MinNetworkTagSecretLength`); a hard startup failure if missing or short (`StartupConfigurationValidator.cs:104`) | |
| `ConceptVotes:AllowedOrigins` | each entry exactly `https://host`, no path, wildcard or trailing slash | |
Conditional, only if the feature is switched on: `GitHubApp` settings when `GitHubApp:EnableWebhookSync` is true;
`Heimdall:PatronSupportIntakeSecret` while `Heimdall:EnablePatronSupportIntake` is true (its default); the Stripe keys
when `Stripe:EnableCheckout` is true. Also checked at every start, in every environment: the `ConceptVotes` prefix lengths,
permits, windows and `MaxTrackedKeys`.
New since the live release, and not read by the validator:
- **`Heimdall__AppSecret`**: required for Discord and Patreon sign-in. Without it `HeimdallOptions.IsDiscordConfigured` and
`IsPatreonConfigured` are false and those sign-ins stop working (GitHub sign-in is unaffected); startup does not fail,
so a missing key shows only at the live check. The value is Heimdall's app secret for slug `bifrost`
(`GC_ACCESS_APP_BIFROST_SHARED_SECRET` on the Heimdall side). Bifrost sends it as `x-heimdall-app-secret`, redacted
from logs.
- **`Bifrost__Oidc__*`**: optional. Absent, the OIDC issuer is not mapped and discovery answers 404. Present, it must be
complete and the key files must exist, or startup stops. Unless the window includes standing up chat sign-in, leave it
unset. Keys and files: `docs/oidc-issuer.md`.
- `DataProtection__KeysPath`: unchanged in intent. Data Protection already has a fixed application name (`Bifrost`), so a
new release directory does not log members out. Confirm the live env still sets the keys path to a directory the new
release can read.
- `DataProtection:KeysPath` keeps its intent. Data Protection has a fixed application name (`Bifrost`, `Program.cs`), so a
new release directory does not log members out; #11's cookie rename does (above). The live env points the keys path at
`/srv/bifrost/.aspnet/DataProtection-Keys` (read 2026-11-09); confirm that again in the window, and that the new release
reads it.
- `Bootstrap__ApplyMigrationsOnStartup=true` stays on, so **the first start applies both migrations**. That is the
irreversible-looking step; the backup below is for it.
@ -47,40 +73,80 @@ All must hold before step 1. A missing one means the window is not used.
1. **Root window.** Someone with root on Yggdrasil is present for the whole deploy and the 30 minutes after (installing
the unit, reading `journalctl`, restoring). The swarm brake is off for Bifrost, or the operator has said so.
2. **Verified database backup.** `pg_dump -Fc` as in the runbook section 8, taken inside the window, then verified:
`pg_restore --list <dump> | head` succeeds, and the dump restores into a scratch database
(`createdb bifrost_rehearsal; pg_restore --no-owner -d bifrost_rehearsal <dump>`), where the member count equals the
live count. Drop the scratch database afterwards. An unrestored dump is not a backup.
3. **Env and secrets backup.** The runbook's `bifrost-env-*.tar`, plus the Data Protection key ring directory if it lives
outside `/srv/bifrost/env`.
4. **Rollback rehearsal.** Before the real deploy, on the same host: unpack the old publish (or keep the live release
directory untouched, which is the plan) and prove the previous release starts against a database that already has the
two migrations: start it on a spare port against the scratch database from step 2 after applying the new migrations
to it (`dotnet ef database update` from the target revision, or one start of the target with the flag on). The
previous release must answer `/` with 200. This is the claim that the rollback is a symlink move; it is checked, not
assumed.
`pg_restore --list <dump> | head` succeeds, and the dump restores into a scratch database, where the member count
equals the live count. The `bifrost` database role has no CREATEDB, so create and drop the scratch database as the
superuser: `sudo -n -u postgres createdb -O bifrost bifrost_rehearsal`, then
`sudo -n -u postgres pg_restore --no-owner --role=bifrost -d bifrost_rehearsal <dump>`, and at the end
`sudo -n -u postgres dropdb bifrost_rehearsal`. An unrestored dump is not a backup.
3. **Env, secrets and key-ring backup.** The runbook's `bifrost-env-*.tar`, plus the live ring
**`/srv/bifrost/.aspnet/DataProtection-Keys`** (read from the host 2026-11-09: `DataProtection__KeysPath` in
`bifrost.env` names it; three key files at that reading, modes 600, owner `bifrost`, dated 2026-04-22, 2026-07-22 and
2026-10-20). It lives outside `/srv/bifrost/env` and is not in that tar. `/srv/bifrost/data-protection-keys` holds one
older key (2026-07-15) that no setting points at: it is not the ring, and is left alone. Losing the ring signs
everyone out and breaks every protected cookie and token; `cp -a` it beside the env backup and record its file list.
4. **Rollback rehearsal.** Before the real deploy, on the same host. There is no `dotnet ef` on the host and this plan
does not use it: the target applies its own migrations at start. Rehearse by starting the target once, then the
previous release, on the scratch database, from a separate env file that touches nothing live:
- a copy of `bifrost.env` at `/srv/bifrost/rehearsal/bifrost-rehearsal.env` (directory and file owned by `bifrost`,
modes 700 and 600; nothing under `/root`, which `bifrost` cannot read), with `ConnectionStrings__Bifrost` pointing at
`bifrost_rehearsal`, `DataProtection__KeysPath` at a throwaway empty directory in the same place (not the live key
ring, which the rehearsal must not write), and `ASPNETCORE_URLS` set to a port that is not 5080 (e.g.
`http://127.0.0.1:5081`). Delete the file and the directories afterwards.
- Strip from that copy every key that could act on or sign in to something real, and every key the scratch run does
not need: all `Bifrost__Oidc__*`, `Bridge__LocalBridgeToken`, `Bootstrap__AdminGitHubLogins__*`, the GitHub App,
Stripe and Heimdall intake and app-secret keys. Stripping a key is not enough: startup validation refuses to start
when a feature flag is on and its key is missing (`StartupConfigurationValidator.cs:59-62`, `:69-73`, `:75-80`), so
set `Heimdall__EnablePatronSupportIntake=false` (it defaults to true and then needs the intake secret),
`GitHubApp__EnableWebhookSync=false` and `Stripe__EnableCheckout=false` in the rehearsal env. With them off the
process starts without the secrets, and the rehearsal cannot reach Stripe, GitHub or Heimdall. The rehearsal checks that the process starts and migrates; **there is
no sign-in on 5081**, because the GitHub OAuth app and Heimdall redirect to the public host.
- start the target release from that file as `bifrost`, with the file loaded by systemd, not by a shell:
`sudo -n systemd-run --wait --pty -p User=bifrost -p EnvironmentFile=/srv/bifrost/rehearsal/bifrost-rehearsal.env
-p WorkingDirectory=<target-release> /usr/bin/dotnet <target-release>/Bifrost.Web.dll` (detached with `--unit
bifrost-rehearsal` and no `--wait`/`--pty` if the check runs from another shell; stop it with
`systemctl stop bifrost-rehearsal`). The scratch database now has both migrations. Check it listens and that
`curl -s -o /dev/null -w '%{http_code}' -H 'Host: <Host:ExpectedHost of the live env>' http://127.0.0.1:5081/readyz`
answers 200 (the app validates the host, so a bare request to `127.0.0.1` is refused), then stop it.
- start the **previous** release (`/srv/bifrost/app/releases/20261010043032-67072b2`) the same way on another spare
port (e.g. 5082) against the same, now migrated, scratch database, with the same `Host:` header; `/readyz` must answer
200. This is the claim that the
rollback is a symlink move; it is checked, not assumed.
5. **Artifact.** Published per runbook section 3: `git merge-base --is-ancestor <sha> origin/main` exits 0, publish from a
detached worktree, tar, upload. Record the tarball's SHA-256 on both ends. Build and test on Yggdrasil's verify path,
not a workstation.
6. **Env diff done.** The `Heimdall__AppSecret` line is in `bifrost.env` (mode 600, owner `bifrost`) and Heimdall holds the
same value. Nothing from the diff in "Configuration that changed" is left unresolved.
7. **Quiet.** No motion closing and no known vote in flight inside the window. Check the motions page.
6. **Env diff done.** Every row of the Configuration table is marked present by the reader, `ConceptVotes__NetworkTagSecret`
included and at least 32 characters (a count, not the value), and the `Heimdall__AppSecret` line is in `bifrost.env`
(mode 600, owner `bifrost`) with Heimdall holding the same value. Nothing in "Configuration" is left unresolved.
7. **Quiet.** No motion closing and no known vote in flight inside the window. Check the motions page. Members told that
everyone is signed out once (see "What moves").
8. **Watchdog.** `bifrost-watchdog-notice.timer` fires one minute after boot and one minute after each run
(`OnBootSec=1min`, `OnUnitActiveSec=1min`; read 2026-11-09), and the oneshot `bifrost-watchdog-notice.service`, run as
`bifrost-notice`, DMs the owner Idunn's operator incidents (`systemctl cat bifrost-watchdog-notice.service`). A deliberate restart or rollback that Idunn counts as an
incident can page the owner during the window; tell them first. Leave the timer running: it is not part of the
deploy and must not be stopped or edited.
## Steps, in order
Record the time and the output line of each in the deploy record. Stop at the first failure and go to Rollback.
1. Note the live state: `readlink -f /srv/bifrost/app/current` (expect `.../20261010043032-67072b2`),
`systemctl is-active bifrost.service`, the member count, the open motion count. This is the "before" the live check
compares to.
2. Take and verify the backups (Preconditions 2 and 3).
3. Add `Heimdall__AppSecret` (and any other new key) to `/srv/bifrost/env/bifrost.env`: `sudo -n install` the new file at
mode 600 owner `bifrost`; keep the old file as `bifrost.env.pre-<UTC>` beside it.
`systemctl is-active bifrost.service`, the member count, the open motion count, and the **log baseline**: the service
writes to files, not the journal (`StandardOutput=append:`), so `journalctl -u bifrost.service` shows only systemd's
own lines. Record `wc -l /srv/bifrost/log/bifrost.stdout.log /srv/bifrost/log/bifrost.stderr.log` (at 2026-11-09
11:55Z: 4838 and 44925 lines; stderr last written 2026-10-18, stdout 2026-11-06) so the live check counts only the
lines written after the restart. This is the "before" the live check compares to.
2. Take and verify the backups (Preconditions 2 and 3), then run the rollback rehearsal (Precondition 4).
3. Make the env complete: add `Heimdall__AppSecret` and, if the Configuration table found it absent or shorter than 32
characters, **`ConceptVotes__NetworkTagSecret`** (the target does not start without it) to
`/srv/bifrost/env/bifrost.env`: `sudo -n install` the new file at mode 600 owner `bifrost`; keep the old file as
`bifrost.env.pre-<UTC>` beside it. Re-run the name-only check after.
4. Upload the tarball; compare its SHA-256 to the build's.
5. `sudo -n bash /home/gamecultadmin/deploy-bifrost-alpha.sh <sha40>`. It unpacks into `<UTC>-<sha7>`, writes `REVISION`,
repoints `current`, restarts. The restart applies both migrations.
6. Watch the start: `journalctl -u bifrost.service -n 100 --no-pager` until the host reports listening on
`127.0.0.1:5080`; no `Failed executing DbCommand`, no `Heimdall ... refused` at start. A start that loops is a
failure: roll back, do not wait.
6. Watch the start in the real logs: `tail -n +<baseline+1> /srv/bifrost/log/bifrost.stdout.log` and the same for
`bifrost.stderr.log`, until the host reports listening on `127.0.0.1:5080`; no `Failed executing DbCommand`, no
`Bifrost startup configuration is incomplete`, no `Heimdall ... refused` at start. `journalctl -u bifrost.service -n 50
--no-pager` shows only start, stop and restart events. A start that loops is a failure: roll back, do not wait.
7. Live check (next section).
8. Record the deploy in `gamecult-ops` `inventory.md` (Bifrost alpha live state): release directory, full sha, date.
@ -89,19 +155,25 @@ Record the time and the output line of each in the deploy record. Stop at the fi
Done from outside nginx, with a real browser session, within the window. A 200 on `/` is not a check.
- `cat /srv/bifrost/app/current/REVISION` is the target sha; `systemctl is-active bifrost.service` is `active`.
- **Cookies (#11).** After the first sign-in, the browser's cookie list shows the `__Host-bifrost_*` names, each Secure with
Path `/` and no Domain, and no old `bifrost_*` name is set again. The old sessions are gone by design.
- **Login.** Sign in with GitHub; then with Discord (exercises `Heimdall__AppSecret`: the start must reach Heimdall and
return to `/auth/heimdall/callback`). A Discord or Patreon sign-in that comes back refused or loops is the new-secret
failure; an existing session surviving the restart shows Data Protection kept its keys.
failure. Data Protection having kept its keys shows as no `key ... was not found` line (name not verified) in the stderr log
after the restart; the old sessions do not survive either way (see "What moves").
- **Vote read.** The public concept-vote read endpoint answers with the existing tallies, and the tallies equal those
noted in step 1 (the schema moved; the data must not).
- **Persona picture.** Open a Persona's page and confirm its picture loads and comes from the host, not a stale cache. A broken
image with a healthy `/` is the finding this check exists for.
image with a healthy `/` is the finding this check exists for. **Unknown: which app serves that picture.** `AvatarUrl` appears at
`src/Bifrost.Web/Domain/BifrostModels.cs:261` and `src/Bifrost.Web/Program.cs:192`, so the picture is probably an
external avatar URL; unverified. Name the serving app and the page before the window, or drop the check.
- Member console, work board, motions and members pages load (runbook section 7).
- The error rate in `journalctl` over five minutes is flat against the lines noted before.
- The error rate over five minutes: the number of lines added to `bifrost.stderr.log` and `bifrost.stdout.log` since the
baseline in step 1 is small and stops growing; read what was added, not just the count.
## Rollback
Trigger: the start loops or migrates with errors, any live check fails, or the root window closes with a doubt open.
Rollback signs members out again (the old release reads the old cookie names); say so if it happens. Trigger: the start loops or migrates with errors, any live check fails, or the root window closes with a doubt open.
1. `sudo -n ln -sfn /srv/bifrost/app/releases/20261010043032-67072b2 /srv/bifrost/app/current && sudo -n chown -h
bifrost:bifrost /srv/bifrost/app/current && sudo -n systemctl restart bifrost.service`.
@ -114,5 +186,14 @@ Trigger: the start loops or migrates with errors, any live check fails, or the r
## What this plan does not do
It does not deploy, does not add an Idunn recipe, and does not decide whether Bifrost #11 merges first (it should). The
It does not deploy, does not add an Idunn recipe, and does not touch `bifrost-watchdog-notice.timer`. The
Idunn recipe is the owner of making this repeatable; until then each deploy follows this page and the runbook.
**Host facts in this plan that no one re-verified in this revision:** the key-ring file count, modes and dates (lines
~64, ~82-85), the stderr/stdout line counts and last-write dates (~135-136), the `bifrost-watchdog-notice.timer`
cadence and unit names (~122-124), and that `bifrost` cannot read `/root` (~91) were read from the host on 2026-11-09 by
an earlier pass or taken from the host's reports; this revision made no host read. The body also states these as fact
without a host check in this revision, and they are unverified too: the `bifrost` database role has no CREATEDB (~77),
there is no `dotnet ef` on the host (~87), logs go to files via `StandardOutput=append:` and the journal shows only
systemd's own lines (~134-136, 148-149), the script path `/home/gamecultadmin/deploy-bifrost-alpha.sh` (~144), and that
no Idunn recipe exists for Bifrost (~20). Treat each as unverified until checked in the window.