kuttl
kuttl copied to clipboard
Support for excluding subset of tests
What this PR does / why we need it:
This PR is a proposal/suggestion for allowing users to provide Test excludes as part of the TestSuite
. Its still work in progress PR and would like to understand if the community finds this useful.
In one of our project where we use kuttl
test framework for our e2e tests. We have a scenario where we have to skip some subset of tests based on some predefined conditions.
Example scenario:
We have to run all tests under tests/e2e
directory except for the below tests
1-031_validate_toolchain
1-085_validate_dynamic_plugin_installation
1-036_validate_keycloak_resource_reqs
1-038_validate_productized_images
1-051-validate_csv_permissions
1-073_validate_rhsso
1-077_validate_disable_dex_removed
1-090_validate_permissions
We couldn't find an easy way of doing that. We had to copy all the tests to a temporary location and had to remove the test sub directories (rm -rf <test_sub_dir>
) that needs to be skipped and run the tests from that temporary directory. This is not the ideal way of excluding tests. So I propose to add a new attribute in TestSuite
CR called testExcludeDirs
where user can specify the test sub directories they would like to exclude and run the remaining tests that are present in the testDirs
.
This feature could also provide an easy way to exclude tests that are broken and needs to be fixed.
Note: If there are better ways of handling test excludes, without requiring this change, please let me know and I am open to use that approach.
Did https://github.com/kudobuilder/kuttl/pull/483 meet your need @anandf ?
check again @anandf if this PR is needed or if #483 solve it
Closing, as https://github.com/kudobuilder/kuttl/pull/483 should have addressed this need. Please reopen and explain if you disagree @anandf