kuttl icon indicating copy to clipboard operation
kuttl copied to clipboard

Support for excluding subset of tests

Open anandf opened this issue 1 year ago • 2 comments

What this PR does / why we need it: This PR is a proposal/suggestion for allowing users to provide Test excludes as part of the TestSuite. Its still work in progress PR and would like to understand if the community finds this useful.

In one of our project where we use kuttl test framework for our e2e tests. We have a scenario where we have to skip some subset of tests based on some predefined conditions.

Example scenario: We have to run all tests under tests/e2e directory except for the below tests

1-031_validate_toolchain 
1-085_validate_dynamic_plugin_installation 
1-036_validate_keycloak_resource_reqs 
1-038_validate_productized_images 
1-051-validate_csv_permissions 
1-073_validate_rhsso 
1-077_validate_disable_dex_removed 
1-090_validate_permissions

We couldn't find an easy way of doing that. We had to copy all the tests to a temporary location and had to remove the test sub directories (rm -rf <test_sub_dir>) that needs to be skipped and run the tests from that temporary directory. This is not the ideal way of excluding tests. So I propose to add a new attribute in TestSuite CR called testExcludeDirs where user can specify the test sub directories they would like to exclude and run the remaining tests that are present in the testDirs.

This feature could also provide an easy way to exclude tests that are broken and needs to be fixed.

Note: If there are better ways of handling test excludes, without requiring this change, please let me know and I am open to use that approach.

anandf avatar Jul 11 '23 17:07 anandf

Did https://github.com/kudobuilder/kuttl/pull/483 meet your need @anandf ?

porridge avatar Nov 23 '23 13:11 porridge

check again @anandf if this PR is needed or if #483 solve it

kensipe avatar Jan 31 '24 16:01 kensipe

Closing, as https://github.com/kudobuilder/kuttl/pull/483 should have addressed this need. Please reopen and explain if you disagree @anandf

porridge avatar Apr 22 '24 08:04 porridge