ray icon indicating copy to clipboard operation
ray copied to clipboard

[CI] Add options for release test run to run only likely failing tests

Open can-anyscale opened this issue 1 year ago • 1 comments

Why are these changes needed?

In https://docs.google.com/document/d/1JC_dHCBkmU7dZY6CaqCFI55zRQo-zNnBeDq5BUGwdBg/edit#heading=h.kbv9vy7pjqaz, I made a proposal to using a test-behavior based coverage to run only tests that are likely to fail. At the same time, we would also want to run tests more often, so bisecting and creating issues for failing test is more prompt.

This PR add flags to build-pipeline.py to be able to run only tests that are likely to fail. Also add option to run tests that belong to a certain team.

Related issue number

Issue #34242

Checks

  • [X] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • [X] I've run scripts/format.sh to lint the changes in this PR.
  • [X] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • [X] Release tests - https://buildkite.com/ray-project/release-tests-pr/builds/34595

can-anyscale avatar Apr 11 '23 18:04 can-anyscale

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

  • If you'd like to keep this open, just leave any comment, and the stale label will be removed.

stale[bot] avatar Jun 10 '23 05:06 stale[bot]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!

stale[bot] avatar Aug 12 '23 05:08 stale[bot]