Cumbersome experimental conformance reporting suite UX

[sunjayBhatia]

Currently in Contour we use these conformance suite options to ensure we run all tests by default, but intentionally have to exclude a few that either don't apply or don't pass. In our case, we exclude the Mesh feature and one of the extended support HTTPProxy tests.

Doing this in the experimental conformance suite currently is not as simple as that to get the full report on supported extended features out. Instead you need something like:

	cSuite, err := suite.NewExperimentalConformanceTestSuite(suite.ExperimentalConformanceOptions{
		Options: suite.Options{
                        ...
                        // Note this vs. using the EnableAllSupportedFeatures and ExemptFeatures field
			SupportedFeatures: suite.AllFeatures.Delete(suite.MeshCoreFeatures.UnsortedList()...),
			SkipTests: []string{
				tests.HTTPRouteRedirectPortAndScheme.ShortName,
			},
		},
		Implementation: conformance_v1alpha1.Implementation{
                         ...
		},
		ConformanceProfiles: sets.New(
			suite.HTTPConformanceProfileName,
			suite.TLSConformanceProfileName,
		),
	})

It would be great if using the same suite.Options setup was possible between the two versions, especially if the current one is eventually replaced by the version that can report conformance results.

Notes:

• Logic for building up the experimental suite features is here: https://github.com/kubernetes-sigs/gateway-api/blob/6d0e6ccbccc73def46bce2e52ed50ca1e480a890/conformance/utils/suite/experimental_suite.go#L114-L147
• vs. standard logic here: https://github.com/kubernetes-sigs/gateway-api/blob/6d0e6ccbccc73def46bce2e52ed50ca1e480a890/conformance/utils/suite/suite.go#L92-L105
• Seems like we can simplify the experimental suite logic a bit and add in support for the exempt features etc.

[shaneutt]

Seems like a good area for improvement, thank you for trying out the experimental conformance profile test suite and providing some feedback!

/triage accepted

I don't think we need to necessarily worry about this for GA however, so I'm not intending to add it to that milestone for now but instead as something to follow up on post-GA:

/priority important-longterm

Ultimately the experimental suite should in time (as we graduate beyond experimental) replace the current test suite so in theory this gets resolved as a bi-product of normal progression? However if there's some more immediate need or something I've missed here please do let me know.

/cc @mlavacca

[mlavacca]

Thanks @sunjayBhatia, this would look like a good improvement in usability

Ultimately the experimental suite should in time (as we graduate beyond experimental) replace the current test suite so in theory this gets resolved as a bi-product of normal progression? However if there's some more immediate need or something I've missed here please do let me know.

+1, even if we should be careful not to lose track of this one, even because when the experimental will become the new standard, most likely we'll be deleting the standard one (or deprecating it), and we should take care of adding this improvement there.

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[candita]

We seem to have lost track of this one. /remove-lifecycle rotten

[candita]

HTTPRouteRedirectPortAndScheme doesn't seem to be working.

[mlavacca]

I am not sure this is still relevant. Looking at it now, I think that the behavior is okay, given that conformance profiles have been standard for quite a while. If you enable all features and want to skip a subset of tests, you are running conformance tests in a non-standard way. Setting all features by default and then disabling them is an approach that I don't think we should encourage. You support feature X->you enable feature X.

HTTPRouteRedirectPortAndScheme doesn't seem to be working.

@candita can you provide some more context?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten