"last_green" Bazel will result in a 404 HTTP error

[fweikert]

Example: https://buildkite.com/bazel/rules-apple-darwin/builds/1001#79e3bc6b-af8b-4a08-834e-ea686a58f848

[fweikert]

last_green works like this:

1. The public Bazel post-submit pipeline uploads the last green commit to GCS.
2. A private pipeline in the trusted org uploads Bazel binaries built at every incoming commit.
3. If using the "last_green" version is requested, CI (via Bazelisk) reads the commit hash from step 1, and downloads the matching binary created by step 2.

There is a possible race condition (commit was updated, but binary hasn't been uploaded yet), which normally isn't a problem.

However, currently we are unable to create new Windows VMs for the trusted pipeline, which means that no new binaries are uploaded. As a result, all new green commits point at a location in GCS that does not contain a binary.

Two action items:

1. Get Windows VMs back online
2. Fix the race condition by updating Bazelisk to fetch the most recent green binary that actually exists in GCS.

[jin]

This is happening again.

https://github.com/bazelbuild/bazel/issues/11242

[fweikert]

2. Fix the race condition by updating Bazelisk to fetch the most recent green binary that actually exists in GCS.

This is definitely something we should address in the near future, given that the problem comes back quite frequently.

[meteorcloudy]

The Windows issue has been fixed in https://github.com/bazelbuild/continuous-integration/issues/1845, and I just deployed the fix to the trusted VMs. Usually the publish bazel binary is way faster than the postsubmit, so this won't be a big problem in future hopefully.