Figure out automating deployment

[christhompson]

This has been on my wishlist for a while, and it was raised again in #443, so I thought it would be good to write up my ideas here.

Some open questions:

• How to pull in secret material (e.g., private keys)? Some possibilities:
  • Encrypt private keys (using a key in something like Cloud KMS) and include them in source; server/builder decrypts them.
    • This might make it more obvious in the repo where things exist, easier to review that at least some encrypted blob is present where it should be for a new private key.
  • Store private keys in something like Secret Manager; server/builder pulls them into the image.
    • This is effectively automating the current deployment step of copying in the "private" overlay files onto the repository.
• Should we containerize the entire server?
  • Might make it easier to integrate with actions/builders/etc.
  • We already have it for the test server.
  • Lets us use things like Cloud Build, which can access Cloud KMS or Secret Manager to pull in/decrypt secrets.

Even just starting with automated building the image would be helpful to reduce errors in the manual deploy process. Once we're comfortable with the automated builds, we could add in automatically updating the running instance to use the new build.

[christhompson]

@lgarron also suggested a couple other approaches on #443:

We could set up SSH or pull-based deploys using GitHub Actions fairly easily now, and either auto-deploy master or give project maintainers permissions to trigger a deploy.

These might be more straightforward to implement, although a little less hermetic than rebuilding a container. The server would then be pre-provisioned with a copy of the secret overlay, and integrate it when a deploy is triggered.

[lgarron]

These might be more straightforward to implement, although a little less hermetic than rebuilding a container. The server would then be pre-provisioned with a copy of the secret overlay, and integrate it when a deploy is triggered.

For what its worth, I think we've done okay with a straightforward deploy. If a malicious code change wants to leak a private key, we can’t do much except avoid landing it in the first place.

So I'd vote for a simple rsync-style deploy, or a git pull on the server triggered using SSH or an endpoint.

[christhompson]

Yeah, thinking about this more, if we can do something like a git pull on the server and a post-pull script to make sure everything is set up, that would be the easier first step. I do think that making a deploy explicitly idempotent would be a nice bonus, but automatically pulling in new commits is probably the low-hanging fruit here, so we can do that first.

There is a semi-related question for server configuration/deployment about how we continue to support certain edge cases (like supporting both TLS 1.0 and TLS 1.3 endpoints) which might be more motivating for containerizing things (to make it easier to run multiple nginx versions, for example).