[RayJob] Improve flexibility to run specified YAML test

[evalaiyc98]

Why are these changes needed?

The purpose of this PR is the same as the https://github.com/ray-project/kuberay/pull/1812#issue-2070542043 , aiming to enhance flexibility in executing sample YAML tests.

With this PR, it is possible to specify one or multiple sample YAML test files. If no files are specified, the test will run on all sample YAML tests related to RayJob.

After the parameter --yaml-files, use the filename of the YAML for testing. Here's an example:

1. [Default] Run all YAML files. python3 tests/test_sample_rayjob_yamls.py

2. Specified a single YAML file. python3 tests/test_sample_rayjob_yamls.py --yaml-files ray_v1alpha1_rayjob.yaml You can also specify multiple YAML files: python3 tests/test_sample_rayjob_yamls.py --yaml-files ray_v1alpha1_rayjob.yaml ray-job.custom-head-svc.yaml

In summary, this PR can save developers time and provide more flexible testing options.

Related issue number

https://github.com/ray-project/kuberay/pull/1812#issuecomment-1889182458

Checks

• [x] I've made sure the tests are passing.
• Testing Strategy
  • [ ] Unit tests
  • [x] Manual tests
  • [ ] This PR is not tested :(

Following is the screenshot of manual test

1. [Default] Run all YAML files.

2. Specified two YAML files.

[kevin85421]

Would you mind rebasing with the master branch? Thanks!

[kevin85421]

cc @rueian would you mind reviewing this PR?

[rueian]

Sure. The changes align with the latest test_sample_raycluster_yamls.py and look good to me.

So now, we skip untracked rayjob tests as well. I found this behavior is quite implicit and silent which may take some time to debug why a test never be executed. In follow-up PRs, I would recommend we also log a warning message, like the [SKIP] one, to notify developers there are some untracked tests skipped.