Bazelisk broken with Bazel@Head on Windows

[comius]

Logs: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1915#5b5a924a-0e06-4ece-b13d-18161f44f6e2

Error message:

# test_bazel_last_green
2021/02/16 04:03:48 Using unreleased version at commit 968422dd6e6735b6066ccbe862b6bc6ab38ce4d6
2021/02/16 04:03:48 Downloading https://storage.googleapis.com/bazel-builds/artifacts/windows/968422dd6e6735b6066ccbe862b6bc6ab38ce4d6/bazel...
Bazelisk version: {STABLE_VERSION}
$TEST_TMPDIR defined: output root default is 'c:\b\cm27gswb\execroot\__main__\_tmp\ab7edd0c907fc9425929721be19c9e77' and max_idle_secs default is '15'.
Extracting Bazel installation...
FATAL: Could not find system javabase. Ensure JAVA_HOME is set, or javac is on your PATH.

[philwo]

I'm pretty sure we didn't touch our Windows images in a long time :/ @meteorcloudy Any idea?

[philwo]

The job magically turned green again on the third try: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1915#9e108f5e-8798-445c-9732-2685e48c0efc 😅

[meteorcloudy]

We've seen this kind of failure before. It's due to something wrong with the https://buildkite.com/bazel-trusted/publish-bazel-binaries pipeline.

It somehow uploaded the bazel binary without embedded JDK to gs://bazel-builds/artifacts/windows/968422dd6e6735b6066ccbe862b6bc6ab38ce4d6/bazel, which is the same as gs://bazel-builds/artifacts/macos/968422dd6e6735b6066ccbe862b6bc6ab38ce4d6/bazel_nojdk

Log: https://buildkite.com/bazel-trusted/publish-bazel-binaries/builds/9000#93aa6f46-310b-4e70-8e39-2c2973e6dc16

2021-02-15 18:43:46 INFO   Searching for artifacts: "bazel.exe" within step: "Build Bazel (:windows: (OpenJDK 8))"
2021-02-15 18:43:46 INFO   Found 1 artifacts. Starting to download to: /tmp/tmptkyogliz
2021-02-15 18:43:47 INFO   Successfully downloaded "bazel.exe" 27624407 bytes
gsutil cp /tmp/tmptkyogliz/bazel.exe gs://bazel-builds/artifacts/windows/968422dd6e6735b6066ccbe862b6bc6ab38ce4d6/bazel
Copying file:///tmp/tmptkyogliz/bazel.exe [Content-Type=application/x-msdos-program]...
/ [1 files][ 26.3 MiB/ 26.3 MiB]
Operation completed over 1 objects/26.3 MiB.
buildkite-agent artifact download bazel_nojdk.exe /tmp/tmptkyogliz --step Build Bazel (:windows: (OpenJDK 8))
2021-02-15 18:43:48 INFO   Searching for artifacts: "bazel_nojdk.exe" within step: "Build Bazel (:windows: (OpenJDK 8))"
2021-02-15 18:43:48 INFO   Found 1 artifacts. Starting to download to: /tmp/tmptkyogliz
2021-02-15 18:43:49 INFO   Successfully downloaded "bazel_nojdk.exe" 27624407 bytes
gsutil cp /tmp/tmptkyogliz/bazel_nojdk.exe gs://bazel-builds/artifacts/windows/968422dd6e6735b6066ccbe862b6bc6ab38ce4d6/bazel_nojdk

Note that both uploads have 27624407 bytes!

I reran the pipeline at the same commit, then it uploaded the correct binary, that's why your third try succeeded https://buildkite.com/bazel-trusted/publish-bazel-binaries/builds/9003#6450c2d5-0622-40fc-a1a1-8b9d0cef0fc8

2021-02-16 11:08:01 INFO   Searching for artifacts: "bazel.exe" within step: "Build Bazel (:windows: (OpenJDK 8))"
2021-02-16 11:08:01 INFO   Found 1 artifacts. Starting to download to: /tmp/tmp0tblzv5z
2021-02-16 11:08:02 INFO   Successfully downloaded "bazel.exe" 43430035 bytes
gsutil cp /tmp/tmp0tblzv5z/bazel.exe gs://bazel-builds/artifacts/windows/968422dd6e6735b6066ccbe862b6bc6ab38ce4d6/bazel
Copying file:///tmp/tmp0tblzv5z/bazel.exe [Content-Type=application/x-msdos-program]...
- [1 files][ 41.4 MiB/ 41.4 MiB]
Operation completed over 1 objects/41.4 MiB.
buildkite-agent artifact download bazel_nojdk.exe /tmp/tmp0tblzv5z --step Build Bazel (:windows: (OpenJDK 8))
2021-02-16 11:08:03 INFO   Searching for artifacts: "bazel_nojdk.exe" within step: "Build Bazel (:windows: (OpenJDK 8))"
2021-02-16 11:08:03 INFO   Found 1 artifacts. Starting to download to: /tmp/tmp0tblzv5z
2021-02-16 11:08:04 INFO   Successfully downloaded "bazel_nojdk.exe" 27624395 bytes
gsutil cp /tmp/tmp0tblzv5z/bazel_nojdk.exe gs://bazel-builds/artifacts/windows/968422dd6e6735b6066ccbe862b6bc6ab38ce4d6/bazel_nojdk
Copying file:///tmp/tmp0tblzv5z/bazel_nojdk.exe [Content-Type=application/x-msdos-program]...

[meteorcloudy]

Oh, it looks like something already went wrong in the Windows job. https://buildkite.com/bazel-trusted/publish-bazel-binaries/builds/9000#8bd697c6-5788-47cb-8cd0-aee4f39ea210

buildkite-agent artifact upload bazel.exe
2021-02-15 18:42:41 INFO   Found 1 files that match "bazel.exe"
2021-02-15 18:42:41 INFO   Creating (0-1)/1 artifacts
2021-02-15 18:42:41 INFO   Uploading artifact 6a45ee0c-cf9f-4215-9430-a16ca46e1600 bazel.exe (27624407 bytes)
buildkite-agent artifact upload bazel_nojdk.exe
2021-02-15 18:42:51 INFO   Found 1 files that match "bazel_nojdk.exe"
2021-02-15 18:42:51 INFO   Creating (0-1)/1 artifacts
2021-02-15 18:42:51 INFO   Uploading artifact 21856115-6101-402a-96ee-de433836b9bb bazel_nojdk.exe (27624407 bytes)

This is so weird..