Serve client API from exported GCS bucket

[jonasfj]

We can point GCLB at a prefix within a GCS bucket, so ideally we should be able to export responses for everything matching:

• GET /api/packages/<package>
• GET /api/packages/<package>/advisories
• GET /api/archives/<package>-<version>.tar.gz

Export them into a GCS bucket, under prefixes:

• /latest/ (all deployments will write to this concurrently)
• /<runtimeVersion>/ (each deployment will have it's own prefix)

Then GCLB can be pointed to /latest/ for URLs matching the above routes.

If a new deployment of pub.dev starts producing garbled output in /latest/, we can update GCLB to point to /<runtimeVersion>/ from the previous deployment. That way we still have a fallback mechanism.

To do this we'll need the following:

• [ ] An abstraction around the bucket containing exported API responses.
  • Must be able to GC old /<runtimeVersion>/.
  • Must be able to GC deleted packages. (Should probably wait until they are 1 day old, to avoid race conditions between creation/deletion).
• [ ] A updater function that given a package name ensures all affected responses are updated.
  • Ideally, this can compute a hash of the contents and avoid unnecessary updates.
• [ ] A backend that runs continuously (similar to how taskBackend works) and scans for packages that have been updated, calling the updater function.
• [ ] Calls to updater function when a new version is published, retracted, moderated, deleted, etc.
• [ ] Tests for all components.
• [ ] Provisioning of new bucket.
• [ ] Mapping of traffic on staging to bucket using GCLB.
• [ ] Mapping of traffic in production to bucket using GCLB.

[sigurdm]

Does this mean that api responses can only be updated reliably after all old deployments are gone?