Use same build in CI for both testing and deploy to Heroku

[tsibley]

Previous commentary:

https://github.com/nextstrain/nextstrain.org/blob/3f862f95953dadb2a2ceb2bf5046a10881790a8c/.github/workflows/ci.yml#L44-L51

and more recently:

I just noticed that our nextstrain.org deploy times are now half what they used to be (8m23s vs. 16m) as a nice side effect of the Gatsby → Next.js switch. Still too long, IMO, but much more bearable. We could chop that in half again if we didn't build in CI and then build again on Heroku, instead re-using the CI build for Heroku. So this is still something I'd like to do something about one day, but it's a bit less painful now.

So how to do this?

Our options boil down to:

1. Build on Heroku, then download to GitHub Actions for testing, then deploy from Heroku
2. Build on GitHub Actions, test, and then upload slug to Heroku for deploy

Option 1 is slow—both the build itself and the downloading of the resulting slug—but means that the Heroku build process (buildpacks, stacks, etc.) remains entirely owned and managed by Heroku.

Option 2 is fast—GH Actions runners are much faster than Heroku build dynos—but it pulls aside the curtain a bit on how Heroku works and takes some of it into our own hands. Less magical maybe, but greater understanding of how things actually work (and that seems useful for deploy infra).

[tsibley]

Option 2 seems nicer to me. It gives us precise control of the build/deploy process and sets us up better for future deploy improvements (or even someday moving off Heroku). It's more involved than what we do now, but the machinery is pretty straightforward.

It starts with building the app using the Heroku Node.js buildpack (the classic one, not the Cloud Native one) and packing the result into an archive:

git clone https://github.com/nextstrain/nextstrain.org app

mkdir build
git clone https://github.com/heroku/heroku-buildpack-nodejs build/pack
mkdir build/cache # or, restore cache from previous build.
mkdir build/env   # *not* config vars/env

export STACK=heroku-20
export SOURCE_VERSION="$(git -C app describe --always)"
export SOURCE_DESCRIPTION="$(git -C app log -1 --format=%s)"

docker run \
  -u $(id -u):$(id -g) --rm --interactive --tty \
  -v "$(realpath app)":/app:rw \
  -v "$(realpath build/pack)":/build/pack:ro \
  -v "$(realpath build/cache)":/build/cache:rw \
  -v "$(realpath build/env)":/build/env:ro \
  -e STACK \
  -e SOURCE_VERSION \
  heroku/heroku:20-build bash -c '
       /build/pack/bin/detect  /app
    && /build/pack/bin/compile /app /build/cache /build/env
    && /build/pack/bin/release /app
  '

# <https://devcenter.heroku.com/articles/platform-api-deploying-slugs>
tar czvf slug.tar.gz app/

Then we create the slug on Heroku:

export SLUG_CHECKSUM="SHA256:$(sha256sum slug.tar.gz | awk '{print $1}')"

export PROCS="$(
  jq --slurp --raw-input '
      split("\n")
    | map(capture("^[[:space:]]*(?<key>[a-zA-Z0-9_-]+):?\\s+(?<value>.*)[[:space:]]*")) # <https://github.com/heroku/buildpacks-procfile/blob/df64135f/src/procfile.rs#L42-L44>
    | from_entries
  ' < app/Procfile
)"

jq --null-input '{
  "stack": env.STACK,
  "checksum": env.SLUG_CHECKSUM,
  "commit": env.SOURCE_VERSION,
  "commit_description": env.SOURCE_DESCRIPTION,
  "process_types": (env.PROCS | fromjson),
}' | tee slug-create.json

# <https://devcenter.heroku.com/articles/platform-api-reference#slug-create>
curl https://api.heroku.com/apps/nextstrain-canary/slugs \
  --data-binary @slug-create.json \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/vnd.heroku+json; version=3' \
  --fail --silent --show-error --location --netrc \
    | tee slug.json

curl "$(jq -r .blob.url slug.json)" \
  --request "$(jq -r .blob.method slug.json)" \
  --data-binary @slug.tar.gz \
  --fail --silent --show-error --location --netrc

With the slug uploaded, deploying it to an app is another API call:

# <https://devcenter.heroku.com/articles/platform-api-reference#release-create>
curl https://api.heroku.com/apps/nextstrain-canary/releases \
  --data-binary @<(jq '{slug: .id}' slug.json) \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/vnd.heroku+json; version=3' \
  --fail --silent --show-error --location --netrc \
    | tee release.json

This all gets a lot simpler too if we switch to Cloud Native buildpacks instead, which has long been on Heroku's roadmap and although support is still in preview, it's been getting recent attention.