Two consistent flakes - rules_python - on OSX and Windows

[chrislovecnm]

Hi all, over at rules_python, we are hitting two flakes consistently.

OS builds

I have seen the following flake 10s of times with the Mac builds.

See: https://buildkite.com/bazel/rules-python-python/builds/4826#01881ff9-bfb9-491f-a410-5ffaef79888a

(17:16:58) INFO: Reading rc options for 'build' from /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/.bazelrc:
--
  | 'build' options: --enable_runfiles
  | (17:16:59) DEBUG: [email protected]/MODULE.bazel:7:6: WARNING: The bazel_gazelle Bazel module is still highly experimental and subject to change at any time. Only use it to try out bzlmod for now.
  | (17:16:59) DEBUG: [email protected]/MODULE.bazel:8:6: WARNING: The rules_go Bazel module is still highly experimental and subject to change at any time. Only use it to try out bzlmod for now.
  | (17:16:59) DEBUG: [email protected]/MODULE.bazel:7:6: WARNING: The rules_go Bazel module is still highly experimental and subject to change at any time. Only use it to try out bzlmod for now.
  | (17:17:01) WARNING: Option 'remote_default_platform_properties' is deprecated: --remote_default_platform_properties has been deprecated in favor of --remote_default_exec_properties.
  | (17:17:01) INFO: Current date is 2023-05-15
  | (17:17:04) DEBUG: Rule 'rules_python~override~python~python3_x86_64-apple-darwin' indicated that a canonical reproducible form can be obtained by modifying arguments url = ["https://github.com/indygreg/python-build-standalone/releases/download/20230116/cpython-3.9.16+20230116-x86_64-apple-darwin-install_only.tar.gz"] and dropping ["urls"]
  | (17:17:04) DEBUG: Repository rules_python~override~python~python3_x86_64-apple-darwin instantiated at:
  | callstack not available
  | Repository rule python_repository defined at:
  | /private/var/tmp/_bazel_buildkite/7599fdbf5f3ac86c93cc4250b368bd74/external/rules_python~override/python/repositories.bzl:366:36: in <toplevel>
  | (17:17:35) INFO: Analyzed 10 targets (171 packages loaded, 15909 targets configured).
  | (17:17:35) INFO: Found 10 targets...
  | (17:17:35) WARNING: cleared convenience symlink(s) bazel-bin, bazel-testlogs because their destinations would be ambiguous
  | (17:17:37) ERROR: /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/BUILD.bazel:19:25: Middleman _middlemen/requirements_Utest-runfiles failed: missing input file '@rules_python~override~internal_deps~pypi__pip//:pip/_vendor/pyparsing/__pycache__/core.cpython-39.pyc.4317360176'
  | (17:17:37) ERROR: /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/BUILD.bazel:19:25: Middleman _middlemen/requirements_Utest-runfiles failed: missing input file '@rules_python~override~internal_deps~pypi__pip//:pip/_vendor/pyparsing/__pycache__/core.cpython-39.pyc.4358975536'
  | (17:17:37) ERROR: /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/BUILD.bazel:19:25: Middleman _middlemen/requirements.update-runfiles failed: missing input file '@rules_python~override~internal_deps~pypi__pip//:pip/_vendor/pyparsing/__pycache__/core.cpython-39.pyc.4317360176'
  | (17:17:37) ERROR: /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/BUILD.bazel:19:25: Middleman _middlemen/requirements_Utest-runfiles failed: missing input file '@rules_python~override~internal_deps~pypi__pip//:pip/_vendor/pyparsing/__pycache__/core.cpython-39.pyc.4559855664'
  | (17:17:37) ERROR: /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/BUILD.bazel:19:25: Middleman _middlemen/requirements.update-runfiles failed: missing input file '@rules_python~override~internal_deps~pypi__pip//:pip/_vendor/pyparsing/__pycache__/core.cpython-39.pyc.4358975536'
  | (17:17:37) ERROR: /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/BUILD.bazel:19:25: Middleman _middlemen/requirements.update-runfiles failed: missing input file '@rules_python~override~internal_deps~pypi__pip//:pip/_vendor/pyparsing/__pycache__/core.cpython-39.pyc.4559855664'
  | (17:17:37) ERROR: /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/BUILD.bazel:19:25: Middleman _middlemen/requirements.update-runfiles failed: 3 input file(s) do not exist
  | (17:17:37) ERROR: /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/BUILD.bazel:19:25: Middleman _middlemen/requirements_Utest-runfiles failed: 3 input file(s) do not exist
  | (17:17:38) ERROR: /Users/buildkite/builds/bk-imacpro-5/bazel/rules-python-python/examples/bzlmod_build_file_generation/BUILD.bazel:19:25 Middleman _middlemen/requirements_Utest-runfiles failed: 3 input file(s) do not exist
  | (17:17:38) INFO: Elapsed time: 39.941s, Critical Path: 2.17s
  | (17:17:38) INFO: 60 processes: 19 remote cache hit, 41 internal.
  | (17:17:38) FAILED: Build did NOT complete successfully
  | bazel build failed with exit code 1
  | 🚨 Error: The command exited with status 1
  | user command error: exit status 1

See: https://github.com/bazelbuild/rules_python/issues/1224

Next

Windows builds

I have seen the following flake 10s of times with the Windows builds.

We are getting a flake where a window host fails with a TLS certificate error. I am guessing that we have a Windows host that has TLS cert problems. If we rerun the test, it often passes.

See https://buildkite.com/bazel/rules-python-python/builds/4777#0187e7a8-8693-47b0-8dbd-21fd2264bf1d

bazel --output_user_root=C:/b --nosystem_rc --nohome_rc info
--
  | 2023/05/04 16:46:55 could not resolve the version 'latest' to an actual version number: unable to determine latest version: could not list Bazel versions in GCS bucket: could not list GCS objects at https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: could not fetch https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: Get "https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/": net/http: TLS handshake timeout

See: https://github.com/bazelbuild/rules_python/issues/1210