PXD-2063 ⁃ implement support for canary deployments

[philloooo]

@phillis @Fauzi - I'm going to look into extending our reverse proxy to support canary deployments of new code: • setup -canary services that point at -canary deployments - configured via manifest like other deployments • configure nginx with upstream serviceName \{ default-service, canary-service } blocks weighted to send 90% of traffic to default service: https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#weights • use sticky cookie, so a tester can request a specific server - if we don't mind having multiple cookies

This only works for requests coming in to the reverse proxy, but still has some nice properties: • simple in-production canary testing - if the canary won't come up or throws errors, then back out :smile: • forces us to support running old and new versions simultaneously - a new deployment should not prevent us from going back to the previous deployment, and should be able to safely co-exist with previous deployment for a rollout period

Anyway - created a jira - we can talk about it :tada: