google
[WIP] Primitive progressive rollout implementation
Motivation
There is no available mechanism by which two clusterfuzz versions can coexist in production. This PR assumes that, during the GCE VM creation, an user data script will:
- Fetch a versions.json under gs://$DEPLOYMENT_BUCKET/versions.json, under which we define the production (gs://$DEPLOYMENT_BUCKET/linux3.zip) and candidate (gs://$DEPLOYMENT_BUCKET/linux3-candidate.zip) stages. For each stage, an integer weight and a python interpreter will be assigned. When a VM is launched, it will randomly pick one of these stages and start the correct deployment, with the assigned python version.
- The clusterfuzz docker container will be launched, during VM startup, with the CLUSTERFUZZ_STAGE environment variable, which can assume the values "prod" and "candidate". Also, the PYTHON_INTERPRETER_VERSION environment variable, which will have the format $MAJOR_VERSION.$MINOR_VERSION (ie, 3.7 or 3.11).
If these assumptions hold, we can change the docker image and start scripts so we can choose both a clusterfuzz stage and a python interpreter version at runtime, enabling us to assign weights to each version and perform production debugging without bringing the whole service down.
This also requires change to butler deploy/package/remote, in order to take into account the dual stages.
This PR belongs to this effort.
Do we also need to change the windows startup script? And Android and Mac?
Do we also need to change the windows startup script? And Android and Mac?
I would rather not touch Android and Mac. It is too much friction to implement prog rollout in an environment we do not control.
As far as windows goes, since we do have our own fleet, a change to the windows startup is on the way. Will be in the PR soon