TLS handshake timeout when listing bazel versions from GCS

[rickeylev]

I've been regularly seeing an error that looks like some issue with the CI scripts talking to GCS.

The error is a bit confusing because it looks like some problem uploading the test logs ("can't find file"), but also looks like some problem "listing bazel versions in GCS" (whatever that means).

Pressing retry on build kite almost always fixes this, so it's some sort of flake.

Agent: bk-windows-bt5g

run: https://buildkite.com/bazel/rules-python-python/builds/4779#_

bazel --output_user_root=C:/b test --flaky_test_attempts=3 --build_tests_only --local_test_jobs=8 --show_progress_rate_limit=5 --curses=yes --color=yes --terminal_columns=143 --show_timestamps --verbose_failures --jobs=30 --announce_rc --experimental_repository_cache_hardlinks --disk_cache= --experimental_build_event_json_file_path_conversion=false --build_event_json_file=C:\temp\tmpikfc45yc\test_bep.json --google_default_credentials --remote_cache=remotebuildexecution.googleapis.com --remote_instance_name=projects/bazel-untrusted/instances/default_instance --remote_timeout=60 --remote_max_connections=200 --remote_default_platform_properties=properties:{name:"cache-silo-key" value:"6a21cacbec775043b8cb5b49849575502cf8f7a8f5d7f28ce34e6c5d2982f753"} --remote_download_toplevel --test_env=LocalAppData --test_env=BAZELISK_USER_AGENT -- ...
--
  | C:\temp\tmpikfc45yc\bazelci-agent.exe artifact upload --delay=5 --mode=buildkite --build_event_json_file=C:\temp\tmpikfc45yc\test_bep.json
  | May 04 17:16:00.138 ERROR bazelci_agent::artifact::upload: The system cannot find the file specified. (os error 2)
  | May 04 17:16:00.138 ERROR bazelci_agent::artifact::upload: The system cannot find the file specified. (os error 2)
  | May 04 17:16:00.138 ERROR bazelci_agent::artifact::upload: The system cannot find the file specified. (os error 2)
  | May 04 17:16:00.138 ERROR bazelci_agent::artifact::upload: The system cannot find the file specified. (os error 2)
  | May 04 17:16:00.138 ERROR bazelci_agent::artifact::upload: The system cannot find the file specified. (os error 2)
  | Error: The system cannot find the file specified. (os error 2)
  | Exception in thread Thread-1:
  | Traceback (most recent call last):
  | File "C:\python3\lib\threading.py", line 973, in _bootstrap_inner
  | self.run()
  | File "C:\python3\lib\threading.py", line 910, in run
  | self._target(*self._args, **self._kwargs)
  | File "c:\b\bk-windows-bt5g\bazel\rules-python-python\bazelci.py", line 2424, in upload_test_logs_from_bep
  | execute_command(
  | File "c:\b\bk-windows-bt5g\bazel\rules-python-python\bazelci.py", line 2474, in execute_command
  | return subprocess.run(
  | File "C:\python3\lib\subprocess.py", line 528, in run
  | raise CalledProcessError(retcode, process.args,
  | subprocess.CalledProcessError: Command '['C:\\temp\\tmpikfc45yc\\bazelci-agent.exe', 'artifact', 'upload', '--delay=5', '--mode=buildkite', '--build_event_json_file=C:\\temp\\tmpikfc45yc\\test_bep.json']' returned non-zero exit status 1.
  | 2023/05/04 17:16:04 could not resolve the version 'latest' to an actual version number: unable to determine latest version: could not list Bazel versions in GCS bucket: could not list GCS objects at https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: could not fetch https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: Get "https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/": net/http: TLS handshake timeout
  | bazel test failed with exit code 1