diff --git a/docs/gamecult/chat-shell-compression.md b/docs/gamecult/chat-shell-compression.md new file mode 100644 index 00000000..c8ac66cd --- /dev/null +++ b/docs/gamecult/chat-shell-compression.md @@ -0,0 +1,108 @@ +# chat.gamecult.org: shell compression and caching + +Follow-up `ratatoskr:follow_up:chat-shell-compression`. Measured 2026-11-12 against the live +host, which serves release `ratatoskr-nemesis-9b9b495d` (the `index-CAataSJe.js` bundle; the offline table +below also uses the `f8115609` handoff build, `index-D0yL0Dgy.js`, and says which row is which). The host config is +`nginx/chat.gamecult.org.conf` in `gamecult-ops`, owned by Eureka; nothing here is applied +to the host. + +## What the host serves now + +The 2026-10-31 probe (no Content-Encoding, no Cache-Control on `/assets/`) is stale: the +`nginx.conf` has `gzip on`, the site file sets level 5, `gzip_vary` and `gzip_types` for js, css, json, svg and wasm, and carries an immutable +`Cache-Control` on `/assets/` and `/preview/assets/`. Probed with `curl -H 'Accept-Encoding: X'`. + +| Resource | raw | gzip (served now) | brotli (served now) | +|---|---|---|---| +| `/assets/index-CAataSJe.js` | 3,732,771 | 956,085 | none: `br` is ignored, identity returned | +| `/assets/index-BLdKqKqx.css` | 111,267 | 19,952 | none | +| `/assets/matrix_sdk_crypto_wasm_bg-DwIieW7K.wasm` | 5,573,870 | 1,878,147 | none | +| `/` (index.html, no-cache) | 3,066 | gzip | none | + +Headers: hashed `/assets/` answer `Cache-Control: public, max-age=31536000, immutable`; +`/`, `index.html`, `sw.js`, `manifest.json`, `config.json` answer `no-cache`; compressed +responses carry `Vary: Accept-Encoding` and a weak ETag. `config.json` (424 B) is below +`gzip_min_length 1024` and goes uncompressed, which is correct. + +So the original ask is done. What is left is brotli, which nginx 1.26.3 here cannot serve: +`/usr/sbin/nginx -V` shows `--with-http_gzip_static_module` and no brotli module. + +## What brotli buys + +Level-5 gzip is what the host does per request. Offline measurement on the same files +(node `zlib`, 2026-11-12): + +| File | raw | gzip 5 (node) | gzip 9 | brotli 11 | brotli vs live | +|---|---|---|---|---|---| +| live main chunk `index-CAataSJe.js` | 3,732,771 | 957,199 | 945,424 | 725,167 | -24% | +| handoff main chunk `index-D0yL0Dgy.js` | 3,868,313 | 1,015,680 | 1,001,446 | 773,992 | -24% | +| crypto wasm | 5,573,870 | 1,868,282 | 1,851,108 | 1,343,595 | -28% | +| `ReactPrism-Dm2uRF4v.js` | 577,517 | 209,339 | 208,376 | 163,525 | -22% | + +A cold load of the two big files falls from about 2.8 MB to about 2.1 MB on the wire; the +saving is ~0.7 MB once per hashed build per device while the HTTP cache keeps the file (immutable; the +service worker does no precaching, so eviction or a new device pays again). gzip 9 alone buys 1-2%, so +`gzip_static` without brotli is not worth a build step. Brotli is the whole prize, and it +needs one root step. + +Cost: brotli q11 takes about 12 s for the wasm and 8 s for the main chunk on one core, so +the whole dist is about two minutes at build time (measured, 84 files) and zero CPU per request. Do not use q11 +on the fly; that is why the plan is precompressed twins. + +## Root list for Eureka (host) + +1. `apt install libnginx-mod-http-brotli-filter libnginx-mod-http-brotli-static` (trixie/main, 1.0.0~rc-6, + depend on `nginx-abi-1.26.3-1`, which the installed nginx provides; apt may offer an nginx patch upgrade + alongside, say so before accepting). Do not enable the config yet. +2. Install a build with twins (`PRECOMPRESS=1`, below) the usual way + (`docs/gamecult/client-build-install.md`). Twins for files that are not there change nothing. +3. In the `chat.gamecult.org` `server` block, beside the existing gzip lines: + +```nginx + brotli_static on; # serve .br when the client sends br + gzip_static on; # serve .gz; falls back to the on-the-fly gzip above + brotli on; # on-the-fly br for files with no twin (index.html, small files) + brotli_comp_level 5; + brotli_types text/css application/javascript text/javascript application/json application/manifest+json image/svg+xml application/wasm; +``` + + The `location ^~ /assets/` block and `/preview/`, `/element/` and `/mockups/` inherit these; they only add headers. + No other change: the `Cache-Control` and `Vary` lines stay as they are, and + `gzip_vary on` already makes the on-the-fly and static paths vary correctly. +4. `nginx -t && systemctl reload nginx`. Twins first, config second: with `brotli on` and no twins every + client gets on-the-fly q5 brotli of the 3.7 and 5.6 MB files, which costs CPU per request. + +Rollback: remove the five lines and reload. The `.br`/`.gz` files can stay on disk, unused. + +## Build side + +`build-prod-dist.sh` (Void's, in `/opt/commons/handoff/ratatoskr-main-build/`) gains +`PRECOMPRESS=1`, which runs `precompress.cjs ` before the manifest is written. It +writes `.br` (q11) and `.gz` (level 9, no mtime, deterministic for one node) next +to each js, css, wasm, json, svg, html file over 1 KiB, only when smaller than the original. +Default is off, so builds installed before the host change do not ship ~27 MB of twins that +nothing serves. The twins appear in `SHA256SUMS`; the determinism compare still applies. +Tested on `ReactPrism-Dm2uRF4v.js` (gzip roundtrip identical to the original); a full +`PRECOMPRESS=1` build has not been run because it waits on step 1. + +The script is not in this repo; its placement is a standing open item of the build handoff. + +## Before/after measurement + +Run from any machine; `X` is the asset URL. Before = today's table above. + +```sh +for ae in identity gzip br; do + printf '%s ' "$ae"; curl -s -o /dev/null -H "Accept-Encoding: $ae" -w '%{size_download} %header{content-encoding}\n' "$X" +done +curl -sI -H 'Accept-Encoding: br' "$X" | grep -iE 'content-encoding|vary|etag|cache-control' +``` + +Pass: with `br` the main chunk is about 774 KB and the wasm about 1.34 MB with +`Content-Encoding: br`; with `gzip` the bytes are no larger than today; with `identity` +the file is the raw size; `Vary: Accept-Encoding` and the immutable `Cache-Control` are +still present on all three; `index.html`, `sw.js` and `config.json` still `no-cache`. +The service worker's registration and an existing session must still work (open chat on +the operator's phone, hard reload once, check the network panel's "transferred" column). +Check `/preview/assets/` the same way, but `/preview/` is served from `/srv/cinny/preview`, which this build +script does not produce: it gets on-the-fly brotli, not twins, until a preview build carries them.