continuous-integration
continuous-integration copied to clipboard
Flaky Mac test log upload
Over the last few days I've noticed that the upload step after builds finish has been rather flaky. This is happening pretty often. Usuallly a retry fixes it, but sometimes it take 4-5 retries.
https://buildkite.com/bazel/rules-python-python/builds/7863#018fd0ee-0089-4499-ac64-44a7e1a0c507
A build will be successful, but then have an error like below:
Traceback (most recent call last):
File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 4461, in <module>
sys.exit(main())
^^^^^^
File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 4429, in main
execute_commands(
File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 1437, in execute_commands
_ = future.result()
^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 2646, in upload_test_logs_from_bep
execute_command(
File "/Users/buildkite/builds/bk-imacpro-19/bazel/rules-python-python/bazelci.py", line 2709, in execute_command
return subprocess.run(
^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/tmprzybvb6m/bazelci-agent', 'artifact', 'upload', '--debug', '--delay=5', '--mode=buildkite', '--build_event_json_file=/tmp/tmprzybvb6m/test_bep.json']' returned non-zero exit status 1.
/cc @coeuvre
Probably obsolete due to https://github.com/bazelbuild/continuous-integration/issues/1981