GameCult/Bifrost
Watch
9
0
Fork
You've already forked Bifrost
 0
Code Issues 7 Pull requests 4 Projects Releases Packages Wiki Activity Actions

docs: plan for deploying main over the live release #15

Merged
heimdall merged 10 commits from heimdall/deploy-main-plan into main 2026-11-09 15:51:44 +00:00
Conversation 1 Commits 10 Files changed 1 +220
1 changed files with 220 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

220
docs/deploy-main-to-live.md Normal file
View file

@ -0,0 +1,220 @@
# Deploying main over the live release
Status: **plan, not executed.** Written 2026-11-09 by the heimdall Self run; revised the same day by the eureka Self run
against main `55b74275`, from Libby's read of the first draft. Nothing here has been run. The deploy waits on a root
window, a verified database backup and a rehearsed rollback (Preconditions).
## What moves
| | Live | Target |
| --- | --- | --- |
| Release | `/srv/bifrost/app/releases/20261010043032-67072b2` | a new `<UTC>-<sha7>` directory |
| Revision | `67072b2` | `55b74275b0d86cdb0c5db01010380a90fbc2eb69` (main at revision; re-check with `git ls-remote`) |
| Distance | | 147 commits; `67072b2` is an ancestor of main |
| Schema | up to the last pre-`AddConceptVotes` migration | + `20261010043937_AddConceptVotes`, + `20261022163159_AddOpenIddict` |
Both migrations only create tables and indexes (`Up` has no `Drop*`, `AlterColumn` or `RenameColumn`). The previous
binary therefore runs against the new schema, which is what makes the rollback below a symlink move, not a restore.
The deploy mechanism is `gamecult-ops/scripts/deploy-bifrost-alpha.sh <sha40>` and
`runbooks/bifrost-alpha-deploy.md`. There is no Idunn recipe for Bifrost; this is a manual, root-window deploy.
**Bifrost #11** (`__Host-` cookies) and #17 (CI publish workflow deleted) are in main; this page (#15) is docs only and
changes no code. #11 renames every cookie Bifrost sets (`BifrostCookies.cs`: session, OAuth correlation, antiforgery,
voter, Heimdall attempt), so **the deploy signs every member out once**, drops sign-ins in flight at the restart, and
gives each returning anonymous voter a new device cookie. Say so to the members before the window. The data-protection
key ring is unchanged, so this is the only forced sign-out; the live check below tests it.
## Configuration
Read against the live `/srv/bifrost/env/bifrost.env` before the window, not during it, by someone who can read it (mode
600, owner `bifrost`; the eureka Self run's account cannot, so **no key below is marked present or absent here**: the
column is filled in by the reader). Key names only; never paste values into a chat, a PR or a log. Name-only check:
`sudo -n cut -d= -f1 /srv/bifrost/env/bifrost.env | sort`.
Outside Development and Testing, `StartupConfigurationValidator.Validate()` refuses to start (an `InvalidOperationException`
naming every failure) unless all of these hold. Environment-variable spelling uses `__` for `:`.
| Key (configuration name) | Rule | Present? |
| --- | --- | --- |
| `GitHubOAuth:ClientId`, `GitHubOAuth:ClientSecret` | both non-empty | |
| `Host:PublicBaseUrl`, `Host:ExpectedHost` | both non-empty | |
| `AllowedHosts` | non-empty and no `*` (while `Host:RequireStrictHostValidation` is true, its default) | |
| `ConnectionStrings:Bifrost` | non-empty | |
| `DataProtection:KeysPath` | non-empty | |
| **`ConceptVotes:NetworkTagSecret`** | non-empty and **at least 32 characters** (`ConceptVoteOptions.MinNetworkTagSecretLength`); a hard startup failure if missing or short (`StartupConfigurationValidator.cs:104`) | |
| `ConceptVotes:AllowedOrigins` | each entry exactly `https://host`, no path, wildcard or trailing slash | |
Conditional, only if the feature is switched on: `GitHubApp` settings when `GitHubApp:EnableWebhookSync` is true;
`Heimdall:PatronSupportIntakeSecret` while `Heimdall:EnablePatronSupportIntake` is true (its default); the Stripe keys
when `Stripe:EnableCheckout` is true. Two of these checks are satisfied by defaults once the one secret is present: Stripe
needs `SecretKey`, `SuccessUrl` and `CancelUrl`, and the last two default to `https://velvet.gamecult.org/...`
(`StripeOptions.cs:15,17`); Heimdall intake needs `BaseUrl`, `AppSlug` and the intake secret, and the first two default to
`https://heimdall.gamecult.org` and `bifrost` (`HeimdallOptions.cs:7,9`). Also checked at every start, in every environment: the `ConceptVotes` prefix lengths,
permits, windows and `MaxTrackedKeys`.
New since the live release, and not read by the validator:
- **`Heimdall__AppSecret`**: required for Discord and Patreon sign-in. Without it `HeimdallOptions.IsDiscordConfigured` and
`IsPatreonConfigured` are false and those sign-ins stop working (GitHub sign-in is unaffected); startup does not fail,
so a missing key shows only at the live check. The value is Heimdall's app secret for slug `bifrost`
(`GC_ACCESS_APP_BIFROST_SHARED_SECRET` on the Heimdall side). Bifrost sends it as `x-heimdall-app-secret`, redacted
from logs.
- **`Bifrost__Oidc__*`**: optional. Absent, the OIDC issuer is not mapped and discovery answers 404. Present, it must be
complete and the key files must exist, or startup stops. Unless the window includes standing up chat sign-in, leave it
unset. Keys and files: `docs/oidc-issuer.md`.
- `DataProtection:KeysPath` keeps its intent. Data Protection has a fixed application name (`Bifrost`, `Program.cs`), so a
new release directory does not log members out; #11's cookie rename does (above). The live env points the keys path at
`/srv/bifrost/.aspnet/DataProtection-Keys` (read 2026-11-09); confirm that again in the window, and that the new release
reads it.
- `Bootstrap__ApplyMigrationsOnStartup=true` stays on, so **the first start applies both migrations**. That is the
irreversible-looking step; the backup below is for it.
## Preconditions
All must hold before step 1. A missing one means the window is not used.
1. **Root window.** Someone with root on Yggdrasil is present for the whole deploy and the 30 minutes after (installing
the unit, reading `journalctl`, restoring). The swarm brake is off for Bifrost, or the operator has said so.
2. **Verified database backup.** `pg_dump -Fc` as in the runbook section 8, taken inside the window, then verified:
`pg_restore --list <dump> | head` succeeds, and the dump restores into a scratch database, where the member count
equals the live count. The `bifrost` database role has no CREATEDB, so create and drop the scratch database as the
superuser: `sudo -n -u postgres createdb -O bifrost bifrost_rehearsal`, then
`sudo -n -u postgres pg_restore --no-owner --role=bifrost -d bifrost_rehearsal <dump>`, and at the end
`sudo -n -u postgres dropdb bifrost_rehearsal`. An unrestored dump is not a backup.
3. **Env, secrets and key-ring backup.** The runbook's `bifrost-env-*.tar`, plus the live ring
**`/srv/bifrost/.aspnet/DataProtection-Keys`** (read from the host 2026-11-09: `DataProtection__KeysPath` in
`bifrost.env` names it; three key files at that reading, modes 600, owner `bifrost`, dated 2026-04-22, 2026-07-22 and
2026-10-20). It lives outside `/srv/bifrost/env` and is not in that tar. `/srv/bifrost/data-protection-keys` holds one
older key (2026-07-15) that no setting points at: it is not the ring, and is left alone. Losing the ring signs
everyone out and breaks every protected cookie and token; `cp -a` it beside the env backup and record its file list.
4. **Rollback rehearsal.** Before the real deploy, on the same host. There is no `dotnet ef` on the host and this plan
does not use it: the target applies its own migrations at start. Rehearse by starting the target once, then the
previous release, on the scratch database, from a separate env file that touches nothing live:
- a copy of `bifrost.env` at `/srv/bifrost/rehearsal/bifrost-rehearsal.env` (directory and file owned by `bifrost`,
modes 700 and 600; nothing under `/root`, which `bifrost` cannot read), with `ConnectionStrings__Bifrost` pointing at
`bifrost_rehearsal`, `DataProtection__KeysPath` at a throwaway empty directory in the same place (not the live key
ring, which the rehearsal must not write), and `ASPNETCORE_URLS` set to a port that is not 5080 (e.g.
`http://127.0.0.1:5081`). Delete the file and the directories afterwards.
- Strip from that copy every key that could act on or sign in to something real, and every key the scratch run does
not need: all `Bifrost__Oidc__*`, `Bridge__LocalBridgeToken`, `Bootstrap__AdminGitHubLogins__*`, the GitHub App,
Stripe and Heimdall intake and app-secret keys. Stripping a key is not enough: startup validation refuses to start
when a feature flag is on and its key is missing (`StartupConfigurationValidator.cs:59-62`, `:69-73`, `:75-80`), so
set `Heimdall__EnablePatronSupportIntake=false` (it defaults to true and then needs the intake secret),
`GitHubApp__EnableWebhookSync=false` and `Stripe__EnableCheckout=false` in the rehearsal env. With them off the
process starts without the secrets, and the rehearsal cannot reach Stripe, GitHub or Heimdall. The rehearsal checks that the process starts and migrates; **there is
no sign-in on 5081**, because the GitHub OAuth app and Heimdall redirect to the public host.
- start the target release from that file as `bifrost`, with the file loaded by systemd, not by a shell:
`sudo -n systemd-run --wait --pty -p User=bifrost -p EnvironmentFile=/srv/bifrost/rehearsal/bifrost-rehearsal.env
-p WorkingDirectory=<target-release> /usr/bin/dotnet <target-release>/Bifrost.Web.dll` (detached with `--unit
bifrost-rehearsal` and no `--wait`/`--pty` if the check runs from another shell; stop it with
`systemctl stop bifrost-rehearsal`). The scratch database now has both migrations. Check it listens and that
`curl -s -o /dev/null -w '%{http_code}' -H 'Host: <Host:ExpectedHost of the live env>' http://127.0.0.1:5081/readyz`
answers 200 (the app validates the host, so a bare request to `127.0.0.1` is refused), then stop it.
- start the **previous** release (`/srv/bifrost/app/releases/20261010043032-67072b2`) the same way on another spare
port (e.g. 5082) against the same, now migrated, scratch database, with the same `Host:` header; `/readyz` must answer
200. This is the claim that the
rollback is a symlink move; it is checked, not assumed.
5. **Artifact.** Published per runbook section 3: `git merge-base --is-ancestor <sha> origin/main` exits 0, publish from a
detached worktree, tar, upload. Record the tarball's SHA-256 on both ends. Build and test on Yggdrasil's verify path,
not a workstation.
6. **Env diff done.** Every row of the Configuration table is marked present by the reader, `ConceptVotes__NetworkTagSecret`
included and at least 32 characters (a count, not the value), and the `Heimdall__AppSecret` line is in `bifrost.env`
(mode 600, owner `bifrost`) with Heimdall holding the same value. Nothing in "Configuration" is left unresolved.
7. **Quiet.** No motion closing and no known vote in flight inside the window. Check the motions page. Members told that
everyone is signed out once (see "What moves").
8. **Watchdog.** `bifrost-watchdog-notice.timer` fires one minute after boot and one minute after each run
(`OnBootSec=1min`, `OnUnitActiveSec=1min`; read 2026-11-09), and the oneshot `bifrost-watchdog-notice.service`, run as
`bifrost-notice`, DMs the owner Idunn's operator incidents (`systemctl cat bifrost-watchdog-notice.service`). A deliberate restart or rollback that Idunn counts as an
incident can page the owner during the window; tell them first. Leave the timer running: it is not part of the
deploy and must not be stopped or edited.
## Steps, in order
Record the time and the output line of each in the deploy record. Stop at the first failure and go to Rollback.
1. Note the live state: `readlink -f /srv/bifrost/app/current` (expect `.../20261010043032-67072b2`),
`systemctl is-active bifrost.service`, the member count, the open motion count, and the **log baseline**: the service
writes to files, not the journal (`StandardOutput=append:`), so `journalctl -u bifrost.service` shows only systemd's
own lines. Record `wc -l /srv/bifrost/log/bifrost.stdout.log /srv/bifrost/log/bifrost.stderr.log` (at 2026-11-09
11:55Z: 4838 and 44925 lines; stderr last written 2026-10-18, stdout 2026-11-06) so the live check counts only the
lines written after the restart. This is the "before" the live check compares to.
2. Take and verify the backups (Preconditions 2 and 3), then run the rollback rehearsal (Precondition 4).
3. Make the env complete: add `Heimdall__AppSecret` and, if the Configuration table found it absent or shorter than 32
characters, **`ConceptVotes__NetworkTagSecret`** (the target does not start without it) to
`/srv/bifrost/env/bifrost.env`, after the timestamped backup, printing nothing. The secret is random, 32 or more
characters, and is never echoed, logged or pasted into a chat. For example (`openssl rand -hex 32` gives 64 hex
characters and nothing that needs quoting in a systemd `EnvironmentFile`):
`sudo -n sh -c 'umask 077; cd /srv/bifrost/env && if grep -q "^ConceptVotes__NetworkTagSecret=" bifrost.env; then echo "ConceptVotes__NetworkTagSecret already set, left unchanged"; else cp -a bifrost.env bifrost.env.pre-<UTC> && { printf "\nConceptVotes__NetworkTagSecret="; openssl rand -hex 32 | tr -d "\n"; printf "\n"; } >> bifrost.env; fi && chown bifrost bifrost.env && chmod 600 bifrost.env'`
The guard makes a re-run safe: an existing `ConceptVotes__NetworkTagSecret=` line is never overwritten or duplicated, and the
skip message prints no value. (If a duplicate key ever got in, systemd's `EnvironmentFile` takes the last one, so remove
the extra line by hand; do not leave two.) A present line shorter than 32 characters is not replaced by this command
either: fix it deliberately, with the rotation warning below in mind.
(the same, with the value from the operator's own source, for `Heimdall__AppSecret`, which is not generated here).
The file ends mode 600, owner `bifrost`, and the backup `bifrost.env.pre-<UTC>` beside it keeps the old content.
Re-run the name-only check after, and check the length by count only:
`sudo -n awk -F= '$1=="ConceptVotes__NetworkTagSecret"{print length($0)-length($1)-1}' /srv/bifrost/env/bifrost.env` is 32 or more.
**Generate it once; do not rotate it casually.** It is the key of the audit `NetworkTag` that `ConceptAppraisal` stores
per vote (`src/Bifrost.Web/Data/BifrostDbContext.cs:546`). A different secret does not touch votes, the
`(SubjectId, VoterKey)` uniqueness or sign-in, but tags already stored no longer match tags computed for the same
network, so `PurgeAsync` (`src/Bifrost.Web/Features/ConceptVotes/ConceptAppraisalService.cs:154`, equality on
`NetworkTag`) misses the old rows; only new casts and re-stanced rows carry the new tag. On this first deploy no vote row
has a tag from the new code, so nothing is orphaned; after it, treat a change as a purge-affecting event.
4. Upload the tarball; compare its SHA-256 to the build's.
5. `sudo -n bash /home/gamecultadmin/deploy-bifrost-alpha.sh <sha40>`. It unpacks into `<UTC>-<sha7>`, writes `REVISION`,
repoints `current`, restarts. The restart applies both migrations.
6. Watch the start in the real logs: `tail -n +<baseline+1> /srv/bifrost/log/bifrost.stdout.log` and the same for
`bifrost.stderr.log`, until the host reports listening on `127.0.0.1:5080`; no `Failed executing DbCommand`, no
`Bifrost startup configuration is incomplete`, no `Heimdall ... refused` at start. `journalctl -u bifrost.service -n 50
--no-pager` shows only start, stop and restart events. A start that loops is a failure: roll back, do not wait.
7. Live check (next section).
8. Record the deploy in `gamecult-ops` `inventory.md` (Bifrost alpha live state): release directory, full sha, date.
## Live check
Done from outside nginx, with a real browser session, within the window. A 200 on `/` is not a check.
- `cat /srv/bifrost/app/current/REVISION` is the target sha; `systemctl is-active bifrost.service` is `active`.
- **Cookies (#11).** After the first sign-in, the browser's cookie list shows the `__Host-bifrost_*` names, each Secure with
Path `/` and no Domain, and no old `bifrost_*` name is set again. The old sessions are gone by design.
- **Login.** Sign in with GitHub; then with Discord (exercises `Heimdall__AppSecret`: the start must reach Heimdall and
return to `/auth/heimdall/callback`). A Discord or Patreon sign-in that comes back refused or loops is the new-secret
failure. Data Protection having kept its keys shows as no `key ... was not found` line (name not verified) in the stderr log
after the restart; the old sessions do not survive either way (see "What moves").
- **Vote read.** The public concept-vote read endpoint answers with the existing tallies, and the tallies equal those
noted in step 1 (the schema moved; the data must not).
- **Persona picture.** Open a Persona's page and confirm its picture loads and comes from the host, not a stale cache. A broken
image with a healthy `/` is the finding this check exists for. **Unknown: which app serves that picture.** `AvatarUrl` appears at
`src/Bifrost.Web/Domain/BifrostModels.cs:261` and `src/Bifrost.Web/Program.cs:192`, so the picture is probably an
external avatar URL; unverified. Name the serving app and the page before the window, or drop the check.
- **Checkout page.** Load one Stripe checkout page (start a checkout and stop at the Stripe-hosted page) and confirm it
renders with the expected product and amount. Do not pay: no card is entered and no live payment is made.
- Member console, work board, motions and members pages load (runbook section 7).
- The error rate over five minutes: the number of lines added to `bifrost.stderr.log` and `bifrost.stdout.log` since the
baseline in step 1 is small and stops growing; read what was added, not just the count.
## Rollback
Rollback signs members out again (the old release reads the old cookie names); say so if it happens. Trigger: the start loops or migrates with errors, any live check fails, or the root window closes with a doubt open.
1. `sudo -n ln -sfn /srv/bifrost/app/releases/20261010043032-67072b2 /srv/bifrost/app/current && sudo -n chown -h
bifrost:bifrost /srv/bifrost/app/current && sudo -n systemctl restart bifrost.service`.
2. Restore the previous env file if step 3 changed it (`bifrost.env.pre-<UTC>`).
3. Re-run the "before" checks from step 1: revision, `active`, counts equal.
4. The two new migrations stay applied. They add tables the old release ignores, as rehearsed in Precondition 4.
5. Only if the schema or data is damaged (not merely new): the runbook section 8 restore, from the verified dump. This is
the last resort and needs the same root window.
6. Leave the failed release directory for diagnosis; do not delete it in the window.
## What this plan does not do
It does not deploy, does not add an Idunn recipe, and does not touch `bifrost-watchdog-notice.timer`. The
Idunn recipe is the owner of making this repeatable; until then each deploy follows this page and the runbook.
**Host facts in this plan that no one re-verified in this revision:** the key-ring file count, modes and dates (lines
~64, ~82-85), the stderr/stdout line counts and last-write dates (~135-136), the `bifrost-watchdog-notice.timer`
cadence and unit names (~122-124), and that `bifrost` cannot read `/root` (~91) were read from the host on 2026-11-09 by
an earlier pass or taken from the host's reports; this revision made no host read. The body also states these as fact
without a host check in this revision, and they are unverified too: the `bifrost` database role has no CREATEDB (~77),
there is no `dotnet ef` on the host (~87), logs go to files via `StandardOutput=append:` and the journal shows only
systemd's own lines (~134-136, 148-149), the script path `/home/gamecultadmin/deploy-bifrost-alpha.sh` (~144), and that
no Idunn recipe exists for Bifrost (~20). Treat each as unverified until checked in the window.