Migrate PR Preview to Clever Cloud

[tobie]

Tracking checklist for standing PR Preview back up on Clever Cloud, replacing the Heroku deployment.

Configuration details are in DEPLOYMENT.md (added in #187) — this issue is the sequence, not the reference.

What changes

	Before	After
Host	Heroku	Clever Cloud (Node.js runtime)
GitHub App	pr-preview	unchanged — same App, same App ID, same installations
S3 buckets	default + WHATWG	unchanged — same buckets, same URLs, same fronting layer
Every rotatable secret	Heroku-era	rotated

Only the host moves. The App is not re-registered and the buckets are not touched, so no repository owner has to reinstall anything and every preview URL already linked from an open PR keeps resolving. The single externally-visible change is the App's webhook URL.

One consequence of the Heroku deployment being gone rather than idle (#186): there is no rollback target. Nothing to point the webhook back at if the new host misbehaves. The mitigation is that the service is already down, so a failed deploy leaves it no worse off, and the Console's Deployments panel can redeploy an earlier commit.

Credentials

Every rotatable secret is being regenerated, not recovered. Nothing here depends on getting values back out of Heroku, which is the right call anyway given they may no longer be readable. Procedures are in DEPLOYMENT.md under Rotating the secrets and AWS S3.

Rotating now is close to free: the service is down, so there is no live traffic to break during the window where a new GITHUB_SECRET and the App settings page disagree. The same change costs rejected deliveries once it is serving.

Look up — not secrets, nothing to rotate:

• GITHUB_INTEGRATION_ID — top of the App settings page. Fixed for the life of the App; it is the one value that cannot be rotated.
• AWS_BUCKET_NAME — the bucket in the S3 console
• WHATWG_AWS_BUCKET_NAME — the hostname serving WHATWG previews, not a bucket name; visible in the preview URL of any recent whatwg PR comment

Rotate:

• GITHUB_INTEGRATION_KEY — generate a fresh private key under Private keys on the App settings page
• GITHUB_SECRET — set a new webhook secret and put the identical value in the app env; these two must match or every delivery is rejected
• GITHUB_TOKEN — issue a new personal access token from the operator's account
• AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY — new access key pair for the same IAM user
• WHATWG_AWS_ACCESS_KEY_ID / WHATWG_AWS_SECRET_ACCESS_KEY — same, if that IAM user is in an account we control. This is the only item that may not be ours to rotate unilaterally; if the user lives in a WHATWG-owned account it needs their cooperation. Worth checking early, since it gates the WHATWG path entirely.

If the WHATWG credentials turn out to be unobtainable, ALLOW_MULTIPLE_AWS_BUCKETS=no routes every repo through the default bucket and the WHATWG env vars are ignored (lib/models/pr.js:195-200). Previews still work; WHATWG's move off their own hostname to a plain S3 URL. A deliberate fallback, not a default.

Retire the old credentials — once the new host has served real traffic:

• Delete the superseded private key entry on the App settings page (generate the new one first; the App needs at least one valid key)
• Revoke the old personal access token
• Deactivate and then delete the old IAM access key pairs
• Purge Heroku-era values from wherever they are stored — they authenticate nothing after this

Provision

• Create the application in the Console, Node.js runtime
• Choose deployment source: linked GitHub repo (auto-deploys on push) or local repo (push to deploy)
• If GitHub-linked, set the deployment branch to main in the Information panel — Clever Cloud defaults to master, which this repo does not have, so nothing would deploy
• Set all environment variables, including NODE_ENV=production
• Set PORT=8080 — the platform only routes there and the code defaults to 5000
• Set CC_NODE_VERSION=22
• Verify GITHUB_INTEGRATION_KEY survived as multi-line PEM (a flattened one fails at the first webhook, not at boot)
• Scalability: min and max instances both 1, auto-scaling off — the controller holds its queue and dedup set in process memory
• Flavor pico. The app waits on remote build services, so its reduced CPU priority costs little; 256 MiB of memory is the open question. See Instance sizing and scaling in DEPLOYMENT.md.
• Confirm unzip is present on the image (the WHATWG path shells out to it)

Deploy and verify

• Deploy; confirm Express server listening on port 8080 in production mode in the Logs panel

• Add the domain; confirm TLS is live — curl -I https://<host>/ returns 404 (no GET route exists; what matters is the handshake and that the app answered)

• Smoke-test the webhook endpoint before pointing GitHub at it. Sign the payload — this is the only check that proves the GITHUB_SECRET in the app's environment is the value you think it is, and it catches a mismatch before GitHub is involved:

  BODY='{}'
  SIG=$(printf '%s' "$BODY" | openssl dgst -sha1 -hmac "$GITHUB_SECRET" | sed 's/^.*= //')
  curl -i -X POST https://<host>/github-hook \
    -H 'Content-Type: application/json' \
    -H "X-Hub-Signature: sha1=$SIG" \
    -d "$BODY"

  Expect 200 with an ISO timestamp body, e.g. 2026-09-06T20:46:59.743Z. Responses verified against the current code:

  Request	Response	Means
  Correct signature	200 + ISO timestamp	app up, NODE_ENV=production, secret matches
  Wrong signature	404	app up, secret does not match
  No X-Hub-Signature header	500	app up; the handler throws on unsigned requests (see note)
  Connection error / TLS failure	—	app not reachable

  Note that 404 is also what GET / returns, so a 404 on its own does not distinguish "wrong secret" from "wrong host" — only the signed 200 is positive evidence.

  _{The 500 is a latent defect rather than a deliberate rejection: @snyk/express-x-hub only attaches req.isXHubValid when an X-Hub-Signature header is present, so lib/app.js:16 calls an undefined function on unsigned requests and throws. Harmless — the payload is never processed — but a public URL will collect 500s and logged TypeErrors from ordinary internet scanning. Worth fixing separately; not a blocker for this migration.}

Cut over

• Point the GitHub App's Webhook URL at https://<host>/github-hook
• Verify a delivery in the App's Advanced tab — 200 with an ISO timestamp body. If the signed smoke test above passed, this confirms GitHub's copy of the secret matches the app's.
• Trigger a real PR event on a repo with a .pr-preview.json and confirm the comment updates
• Test a whatwg-owned PR separately — it is the only path using the second bucket, and the only one downloading, unzipping and diffing full HTML spec builds on the instance. A missing unzip, an undersized flavor, or bad WHATWG_AWS_* credentials surface here and nowhere else.
• Update the form action in docs/config.html from https://pr-preview.herokuapp.com/config to https://<host>/config

Clear the backlog

PR Preview has been down since #186, so open PRs have missed their previews, unfortunately this is impossible to automate afaik, so won't happen.

• Redeliver failed webhook deliveries from the App's Advanced tab where they are still retained
• For the rest, set STARTUP_QUEUE and restart, or push an empty commit to the affected PRs
• Close PR Preview is down #186 once previews are flowing again

After a week of real traffic

• Check for unexplained restarts. On pico, an OOM kill drops the whole in-memory job queue rather than just the PR that overran, so a single missing preview can be the visible edge of a lost queue. Size up if they show.