support component-extension for openshift-tests

[kuiwang02]

support component-extension for openshift-tests so that it can get and run extenal cases except for default extension

as we known, for one component, there is external binary for it. it has one registry and then we register extension in registry. when we register the first extension, it is also named "default" besides its ID, like"production:type:xxx". from second extension, it is not named "default" as well and only has its ID.

currently when openshift-tests execute the external cases, it only support to get cases and execute them from "default" extension of the external binary. if we want openshift-tests to execute the cases only from non default extension of the external binary, we need one parameter to tell openshift-tests which extension it need to get cases so that it set "--component=xxx" when openshift-tests call info/images/list/run-test interface of the external binary.

the usage of parameter: --component-extension=tag1:id1,tag2=id2 or --component-extension=tag1:id1 --component-extension=tag2=id2 the tag is image stream tag of the component(for example, machine-api-operator), and the id is the extension id when creating extension (for example, openshift:payload:machine-api-operator).

note: for current openshift CI job, no need to any change for the steps which use openshift-tests because it suppose to get cases from default extension and we do not set parameter component-extension.

by the way, why we possible register the second extension? when we try to migrate the QE cases from openshift-tests-private repo to component repo, the case which is in default extension will be selected and executed by openshift-tests in openshift CI job. it requires the cases stable and less duration. but we have some cases which can not meet this and we can not drop them, so maybe we create the second extension and put such cases into the second extension once the openshift-tests-private repo is deprecated. and then we could create module owned prow job to use openshift-tests with parameter component-extension to execute these external cases. (if the openshift-tests-private repo is not deprecated, maybe we do not migrate such case to component repo, and do not need to create second extension and do not need this parameter)

why we do not execute such case with external binary directly in module owned prow job and have to use the openshift-tests? the run-test/run-suite of the external binary execute the case in one process, and it can only run cases in serial although the case should be ran in parallel. but openshift-tests execute the case in different process, so it could run the case in parallel.

/cc @stbenjam Hi, you are the expert of OTE. could you please help review it? thanks if you want more members to review it, please add the members. thanks

(Frankly, I am not sure if it is good and correct solution.)

[openshift-ci[bot]]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kuiwang02 Once this PR has been reviewed and has the lgtm label, please assign neisw for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[kuiwang02]

Thanks for the contribution --

Are the command line arguments you're adding intended to make it conditional which extension is used in an image? We didn't expect extensions to be used that way, if you want to control what tests are going to get run, you can use suites.

If a binary provides multiple extensions, I'd expect them to all be registered

yes. we add this command line argument so that we can set extension for selecting cases. this argument is optional. if we do not set it, it will take default extension (same behavior with before).

the binary provides multiple extensions, and we register all them. currently when openshift-tests selects cases from the external binary, it only take "default" extension (the first registered extension). while we want to select cases from other extension, so need this command line argument.

for how the openshift-tests select cases from external binary, it take ListTests with list interface without "--component" to get cases. Per List, it take "default" extension and select cases within this extension. even we set testsuite, it will testsuite's qualifiers to filter the case within default extension. (same behavior for other interface of the external binary) so, if we want to select cases from non-default extension by openshift-tests, we need this argument even we use the suites, or else it still select cases from default extension with suite's qualifiers.

hope make me clarified.

[openshift-ci[bot]]

@kuiwang02: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-metal-ipi-ovn-dualstack-local-gateway	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-metal-ipi-ovn-dualstack-local-gateway
ci/prow/e2e-gcp-ovn-etcd-scaling	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-gcp-ovn-etcd-scaling
ci/prow/e2e-metal-ipi-ovn-kube-apiserver-rollout	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-metal-ipi-ovn-kube-apiserver-rollout
ci/prow/e2e-vsphere-ovn-etcd-scaling	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-vsphere-ovn-etcd-scaling
ci/prow/4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
ci/prow/e2e-gcp-disruptive	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-gcp-disruptive
ci/prow/e2e-openstack-ovn	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-openstack-ovn
ci/prow/e2e-aws-disruptive	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-aws-disruptive
ci/prow/e2e-azure-ovn-etcd-scaling	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-azure-ovn-etcd-scaling
ci/prow/e2e-gcp-fips-serial-1of2	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-gcp-fips-serial-1of2
ci/prow/e2e-azure-ovn-upgrade	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-azure-ovn-upgrade
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-vsphere-ovn-dualstack-primaryv6
ci/prow/e2e-aws-ovn-serial-publicnet-1of2	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-aws-ovn-serial-publicnet-1of2
ci/prow/okd-e2e-gcp	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test okd-e2e-gcp
ci/prow/e2e-aws-ovn-single-node-serial	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-aws-ovn-single-node-serial
ci/prow/e2e-gcp-fips-serial-2of2	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-gcp-fips-serial-2of2
ci/prow/e2e-aws-ovn-kube-apiserver-rollout	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-aws-ovn-kube-apiserver-rollout
ci/prow/e2e-aws-ovn-etcd-scaling	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-aws-ovn-etcd-scaling
ci/prow/e2e-aws-ovn-single-node-upgrade	86b12a6cc26dbc8f7e23706212dce7352828a68f	link	false	/test e2e-aws-ovn-single-node-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-trt[bot]]

Job Failure Risk Analysis for sha: 86b12a6cc26dbc8f7e23706212dce7352828a68f

Job Name	Failure Risk
pull-ci-openshift-origin-main-4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback	MissingData
pull-ci-openshift-origin-main-e2e-aws-ovn	Medium [sig-instrumentation] Metrics should grab all metrics from kubelet /metrics/resource endpoint [Suite:openshift/conformance/parallel] [Suite:k8s] This test has passed 93.21% of 2944 runs on release 4.20 [Overall] in the last week.
pull-ci-openshift-origin-main-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (28) are below the historical average (4175): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-main-e2e-azure-ovn-etcd-scaling	Low [bz-Cloud Compute] clusteroperator/control-plane-machine-set should not change condition/Degraded This test has passed 0.00% of 1 runs on release 4.20 [Architecture:amd64 FeatureSet:default Installer:ipi JobTier:rare Network:ovn NetworkStack:ipv4 Owner:eng Platform:azure SecurityMode:default Topology:ha Upgrade:none] in the last week. Open Bugs etcd-scaling jobs failing ~60% of the time
pull-ci-openshift-origin-main-e2e-gcp-ovn-etcd-scaling	Low [bz-Cloud Compute] clusteroperator/control-plane-machine-set should not change condition/Degraded This test has passed 0.00% of 1 runs on release 4.20 [Architecture:amd64 FeatureSet:default Installer:ipi JobTier:rare Network:ovn NetworkStack:ipv4 Owner:eng Platform:gcp SecurityMode:default Topology:ha Upgrade:none] in the last week. Open Bugs etcd-scaling jobs failing ~60% of the time

[stbenjam]

Thanks for the contribution -- Are the command line arguments you're adding intended to make it conditional which extension is used in an image? We didn't expect extensions to be used that way, if you want to control what tests are going to get run, you can use suites. If a binary provides multiple extensions, I'd expect them to all be registered

yes. we add this command line argument so that we can set extension for selecting cases. this argument is optional. if we do not set it, it will take default extension (same behavior with before).

the binary provides multiple extensions, and we register all them. currently when openshift-tests selects cases from the external binary, it only take "default" extension (the first registered extension). while we want to select cases from other extension, so need this command line argument.

for how the openshift-tests select cases from external binary, it take ListTests with list interface without "--component" to get cases. Per List, it take "default" extension and select cases within this extension. even we set testsuite, it will testsuite's qualifiers to filter the case within default extension. (same behavior for other interface of the external binary) so, if we want to select cases from non-default extension by openshift-tests, we need this argument even we use the suites, or else it still select cases from default extension with suite's qualifiers.

hope make me clarified.

By default when an extension adds tests, its scope is limited to tests only from itself. To do cross-extension test selection with a suite you have to use the AddGlobalSuite helper. https://github.com/openshift/origin/pull/29888 makes the suite test selection work as designed, I'd wait for that to become available. It looks like this is a workaround for suites not being fully implemented yet.

All extensions should be registered and examined when considering the tests to run. Only suites are used for test selection, we don't want a second hidden way of filtering tests that's not obvious.

[kuiwang02]

By default when an extension adds tests, its scope is limited to tests only from itself. To do cross-extension test selection with a suite you have to use the AddGlobalSuite helper. https://github.com/openshift/origin/pull/29888 makes the suite test selection work as designed, I'd wait for that to become available. It looks like this is a workaround for suites not being fully implemented yet.

@stbenjam Thanks for the information https://github.com/openshift/origin/pull/29888. I will learn it.

It seems I do not make me cleared so it seems you misunderstand what I want to do. ( 1, mine is related to select cases from non-default extension of the one external binary. for example, olm team external binary has olmv1 extension which is also default extension. and olmv1-nondefault extension. currently openshift-tests can select case only from olmv1 extension(default). while we want openshift-tests select case from olmv1-nondefault extension sometimes. 2, yours is related to select cases from default extension of the multiple external binary with GloabalSuite. for example, olm team external binary has default extension which has parallel suite and storage team external binary has default extension which has parallel suite. openshift-tests take GlobaltestSuite parallel to get case from olm and storage team, but the case are still from default extension. it is not what I want to make. )

Here I take example to explain my requirement which is the reason why I add this command cli argument:

Suppose we have 100 cases in openshift-tests-private repo, and we have to migrate them to component repo based on OTE when openshift-tests-private repo is deprecated. 70 of 100 cases are good and meet criteria so they could be executed by current openshift CI job which take step openshift-e2e-test with openshift-tests binary. and then these case are treated as same with the cases which is in origin repo. 30 of 100 cases are not good, and can not meet criterial (due to certain reason, for example, unstable, duration etc) so they can not be executed by current openshift CI job which take step openshift-e2e-test. and we create specific step and specific prow job which belong to our component to execute these 30 cases.

So, we expect 70 cases can be executed by openshift-e2e-test and 30 cases are not. and 30 cases are executed by specific step which still use openshift-tests with this argument.

current code from origin (without https://github.com/openshift/origin/pull/29888) and openshift-tests-extension can not support this requirement if we implement 100 cases into only one extension of the external binary (the first one and default) because openshift-tests select cases only from default extension only with environment var and 30 cases is possible be selected.

so, my solution is that 1, 70 cases are added to the first extension of the external binary. default extension as well 2, 30 case are added to the second extension of the external binary. 3, add the optional argument of openshift-tests to set extension for the external binary. support more key=value so that set its extension for different team. for example, if only olm team want to use non-default extension, could only use --component-extension=<olm stream tag>=<olm non-default extension id> in their step. if both olm and storage team want to use non-default extension, could use --component-extension=<olm stream tag>=<olm non-default extension id>,<storage team tag>=<storage non-default extension id>. if we do not set it, default extension is used. 4, no any change for step openshift-e2e-test and the prow job which use openshift-e2e-test. so, these job will select case from 70 cases with environment var. 5, our component make specific step and prow job to take openshift-tests with this argument to set non-default extension to select cases from 30 cases.

by the way, even there is no my requirement, it is better to support openshift-tests to select cases from non-default extension of the external binary.

how to make my requirement if we do not add this argument? https://github.com/openshift/origin/pull/29888 could resolve my requirement if we migrate 100 cases into default extension of the external binary? thanks if it is true, could you please give me guide how to resolve it based on https://github.com/openshift/origin/pull/29888? thanks (I will also learn https://github.com/openshift/origin/pull/29888 and try to understand it.)

[kuiwang02]

HI @stbenjam after learning https://github.com/openshift/origin/pull/29888, I think it could meet my required, so I close the PR. thanks

[kuiwang02]

/close

[openshift-ci[bot]]

@kuiwang02: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.