Support for a feature gate flag/marker when generating CRDs

[akutz]

Has there ever been any consideration towards the inclusion of a feature gate flag / marker combination that could be used to control the generation of CRDs from the Go code? For example:

type HelloWorldSpec struct {
}

type HelloWorldStatus struct {
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:scope=Namespace
// +kubebuilder:featuregate=MyFirstAPI
type HelloWorld struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   HelloWorldSpec   `json:"spec,omitempty"`
    Status HelloWorldStatus `json:"status,omitempty"`
}

The presence of the marker +kubebuilder:featuregate=MyFirstAPI would indicate to the CRD generation that the type should not be considered unless controller-gen was executed with the following flag:

controller-gen \
  paths=hello-world/api/... \
  crd:trivialVersions=true \
  crd:crdVersions=v1 \
  crd:preserveUnknownFields=false \
  crd:featureGate=MyFirstAPI \
  output:crd:dir=./config/crd/bases \
  output:none

if a type does not specify a kubebuilder:featuregate annotation then it is not gated, but when a type does include such a marker, then this flag could prevent CRDs from being generated with features not yet ready for consumption.

There is a real business need for this because of the way private companies, even when developing in public, may want to align on API schemas in advance of releases. However, this presents a challenge because of deadlines and the risk of a feature not being ready. Still, the API was locked months ago. The flip side is organically updating an API over the duration of a release cycle. This is also not ideal because some API schema version will be constantly changing.

What would be nice is if new features could be gated at the API level behind a feature gate. That way the development could happen in the open, but the feature's API is only included in CRDs if:

• the feature gate is enabled when generating the CRDs -- useful for testing
• the feature gate is removed -- useful ahead of release

In looking into this I did come across kubebuilder:skipversion, but it does not seem to do quite the same thing.

Anyway, I'd love to hear others' thoughts and feedback on this. Thanks in advance!

[vincepri]

Would we allow setting feature gates on fields as well? How would this show up in openAPI schema, if at all?

@nikhita was working on something similar, but for upstream Kubernetes — we could see if there are plans to support feature gated fields for CRDs as well?

[akutz]

Hi @vincepri,

I suppose I was not thinking at the field level, but I suppose, sure why not? I was really thinking more at the object level, but I suppose the field would simply not be part of the schema.

To me this has value in enabling testing pipelines for features you don't want exposed in the published CRD. Obviously this means the type code is still in the repo, and so is the feature code. But this way feature gates could be used to control both a feature's code availability at runtime and whether or not the CRDs said would be acting upon were ever part of the published artifacts in the first place.

I suppose it's not a huge distinction, but it could also, and I'm just spit-balling here, be handled different altogether. Imagine the CRDs were generated, but the feature gate marker actually indicated to the Kube API server to respond with "NotSupported" if a patch for any resource that was marked behind a feature gate was received.

Anyway, as more companies try to figure out how to align their traditional API versioning and release policies with integrated Kube APIs, figuring out how to be more flexible with the Kube APIs and feature switches/gates is going to pop up more and more I think.

[akutz]

Hi @nikhita,

I spoke with @vincepri offline. I think your KEP sounds like a better approach for addressing a similar issue. I'd like to discuss whether you think it's reasonable to add support for feature gating entire CRDs and not just their fields. If that cannot be addressed in your existing KEP, please let me know where the KEP is, so I can file an issue in that project to track a v2 of your KEP to include support for feature gating entire CRDs.

Thanks!

[akutz]

cc @karaatanassov

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[akutz]

/reopen /remove-lifecycle rotten

[k8s-ci-robot]

@akutz: Reopened this issue.

In response to this:

/reopen /remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[akutz]

/reopen /remove-lifecycle rotten

[k8s-ci-robot]

@akutz: Reopened this issue.

In response to this:

/reopen /remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[JoelSpeed]

For what it's worth, we have started using a patch in OpenShift when generating CRDs to have a distinction between fields with different feature sets (OpenShift groups feature gates into groups called feature sets).

We have multiple copies of the CRD, annotated with the desired feature set. The generator reads this annotation to identify if featuregeated fields should be included in the schema or not. OpenShift will then make sure the correct one is installed depending on which feature set the cluster is configured to use.

We haven't upstreamed this because it seemed like it was very specific to OpenShift and there are a number of questions it raises.

• How would you go backwards (ie what happens if you remove a field? - This doesn't matter in OpenShift as turning on a feature gate is a one way operation)
• How do users install their CRDs/how would they know which ones to install (In OpenShift this is operator driven, so we already have a solution)

I can see this being useful for the testing pipelines as already mentioned, but I think it does seem like the solution might need to be tailored quite specifically to each deployment system for Kubernetes, so I'm not sure how much value there is in an upstreamed solution to this

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten