GameCult/Ratatoskr
Watch
9
0
Fork
You've already forked Ratatoskr
 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs: chat shell compression, measured state and brotli plan #120

Merged
void merged 2 commits from void/chat-shell-compression into dev 2026-11-12 18:16:05 +00:00
Conversation 0 Commits 2 Files changed 1 +108
1 changed files with 108 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

108
docs/gamecult/chat-shell-compression.md Normal file
View file

@ -0,0 +1,108 @@
# chat.gamecult.org: shell compression and caching
Follow-up `ratatoskr:follow_up:chat-shell-compression`. Measured 2026-11-12 against the live
host, which serves release `ratatoskr-nemesis-9b9b495d` (the `index-CAataSJe.js` bundle; the offline table
below also uses the `f8115609` handoff build, `index-D0yL0Dgy.js`, and says which row is which). The host config is
`nginx/chat.gamecult.org.conf` in `gamecult-ops`, owned by Eureka; nothing here is applied
to the host.
## What the host serves now
The 2026-10-31 probe (no Content-Encoding, no Cache-Control on `/assets/`) is stale: the
`nginx.conf` has `gzip on`, the site file sets level 5, `gzip_vary` and `gzip_types` for js, css, json, svg and wasm, and carries an immutable
`Cache-Control` on `/assets/` and `/preview/assets/`. Probed with `curl -H 'Accept-Encoding: X'`.
| Resource | raw | gzip (served now) | brotli (served now) |
|---|---|---|---|
| `/assets/index-CAataSJe.js` | 3,732,771 | 956,085 | none: `br` is ignored, identity returned |
| `/assets/index-BLdKqKqx.css` | 111,267 | 19,952 | none |
| `/assets/matrix_sdk_crypto_wasm_bg-DwIieW7K.wasm` | 5,573,870 | 1,878,147 | none |
| `/` (index.html, no-cache) | 3,066 | gzip | none |
Headers: hashed `/assets/` answer `Cache-Control: public, max-age=31536000, immutable`;
`/`, `index.html`, `sw.js`, `manifest.json`, `config.json` answer `no-cache`; compressed
responses carry `Vary: Accept-Encoding` and a weak ETag. `config.json` (424 B) is below
`gzip_min_length 1024` and goes uncompressed, which is correct.
So the original ask is done. What is left is brotli, which nginx 1.26.3 here cannot serve:
`/usr/sbin/nginx -V` shows `--with-http_gzip_static_module` and no brotli module.
## What brotli buys
Level-5 gzip is what the host does per request. Offline measurement on the same files
(node `zlib`, 2026-11-12):
| File | raw | gzip 5 (node) | gzip 9 | brotli 11 | brotli vs live |
|---|---|---|---|---|---|
| live main chunk `index-CAataSJe.js` | 3,732,771 | 957,199 | 945,424 | 725,167 | -24% |
| handoff main chunk `index-D0yL0Dgy.js` | 3,868,313 | 1,015,680 | 1,001,446 | 773,992 | -24% |
| crypto wasm | 5,573,870 | 1,868,282 | 1,851,108 | 1,343,595 | -28% |
| `ReactPrism-Dm2uRF4v.js` | 577,517 | 209,339 | 208,376 | 163,525 | -22% |
A cold load of the two big files falls from about 2.8 MB to about 2.1 MB on the wire; the
saving is ~0.7 MB once per hashed build per device while the HTTP cache keeps the file (immutable; the
service worker does no precaching, so eviction or a new device pays again). gzip 9 alone buys 1-2%, so
`gzip_static` without brotli is not worth a build step. Brotli is the whole prize, and it
needs one root step.
Cost: brotli q11 takes about 12 s for the wasm and 8 s for the main chunk on one core, so
the whole dist is about two minutes at build time (measured, 84 files) and zero CPU per request. Do not use q11
on the fly; that is why the plan is precompressed twins.
## Root list for Eureka (host)
1. `apt install libnginx-mod-http-brotli-filter libnginx-mod-http-brotli-static` (trixie/main, 1.0.0~rc-6,
depend on `nginx-abi-1.26.3-1`, which the installed nginx provides; apt may offer an nginx patch upgrade
alongside, say so before accepting). Do not enable the config yet.
2. Install a build with twins (`PRECOMPRESS=1`, below) the usual way
(`docs/gamecult/client-build-install.md`). Twins for files that are not there change nothing.
3. In the `chat.gamecult.org` `server` block, beside the existing gzip lines:
```nginx
brotli_static on; # serve <file>.br when the client sends br
gzip_static on; # serve <file>.gz; falls back to the on-the-fly gzip above
brotli on; # on-the-fly br for files with no twin (index.html, small files)
brotli_comp_level 5;
brotli_types text/css application/javascript text/javascript application/json application/manifest+json image/svg+xml application/wasm;
```
The `location ^~ /assets/` block and `/preview/`, `/element/` and `/mockups/` inherit these; they only add headers.
No other change: the `Cache-Control` and `Vary` lines stay as they are, and
`gzip_vary on` already makes the on-the-fly and static paths vary correctly.
4. `nginx -t && systemctl reload nginx`. Twins first, config second: with `brotli on` and no twins every
client gets on-the-fly q5 brotli of the 3.7 and 5.6 MB files, which costs CPU per request.
Rollback: remove the five lines and reload. The `.br`/`.gz` files can stay on disk, unused.
## Build side
`build-prod-dist.sh` (Void's, in `/opt/commons/handoff/ratatoskr-main-build/`) gains
`PRECOMPRESS=1`, which runs `precompress.cjs <dist>` before the manifest is written. It
writes `<file>.br` (q11) and `<file>.gz` (level 9, no mtime, deterministic for one node) next
to each js, css, wasm, json, svg, html file over 1 KiB, only when smaller than the original.
Default is off, so builds installed before the host change do not ship ~27 MB of twins that
nothing serves. The twins appear in `SHA256SUMS`; the determinism compare still applies.
Tested on `ReactPrism-Dm2uRF4v.js` (gzip roundtrip identical to the original); a full
`PRECOMPRESS=1` build has not been run because it waits on step 1.
The script is not in this repo; its placement is a standing open item of the build handoff.
## Before/after measurement
Run from any machine; `X` is the asset URL. Before = today's table above.
```sh
for ae in identity gzip br; do
printf '%s ' "$ae"; curl -s -o /dev/null -H "Accept-Encoding: $ae" -w '%{size_download} %header{content-encoding}\n' "$X"
done
curl -sI -H 'Accept-Encoding: br' "$X" | grep -iE 'content-encoding|vary|etag|cache-control'
```
Pass: with `br` the main chunk is about 774 KB and the wasm about 1.34 MB with
`Content-Encoding: br`; with `gzip` the bytes are no larger than today; with `identity`
the file is the raw size; `Vary: Accept-Encoding` and the immutable `Cache-Control` are
still present on all three; `index.html`, `sw.js` and `config.json` still `no-cache`.
The service worker's registration and an existing session must still work (open chat on
the operator's phone, hard reload once, check the network panel's "transferred" column).
Check `/preview/assets/` the same way, but `/preview/` is served from `/srv/cinny/preview`, which this build
script does not produce: it gets on-the-fly brotli, not twins, until a preview build carries them.