api icon indicating copy to clipboard operation
api copied to clipboard
verified publisher icon openshift

OTA-1223: config/v1/types_cluster_version: Drop enum from ClusterVersion status capabilities

Open wking opened this issue 2 years ago • 10 comments

The cluster-version operator should be the only actor writing to ClusterVersion status. Trust it to pick appropriate values, instead of restricting it with an enum.

When updating into new capabilities, the current timeline is:

  1. Outgoing CVO accepts the new target, and figures out verified and implicit capabilities.
  2. Outoing CVO starts trying to write to status, but because of the new capability not being a part of the outgoing CRD's capability enum, this fails.
  3. Outgoing CVO pushes the incoming ClusterVersion CRD in runlevel 0 index 1.
  4. status sync attempts, if any, start working again.
  5. Outgoing CVO pushes the incoming CVO Deployment in runlevel 0 index 3.
  6. Deployment controller TERMs the outgoing CVO process.
  7. Outgoing CVO wraps up the manifest sycning.
  8. Outgoing CVO attempts a final status sync.
  9. Outgoing CVO releases the leader lock.
  10. Outgoing CVO exists.

With the status enum removal in this commit, ClusterVersion status syncing will remain working the whole time, reducing the risk of the outgoing CVO shutting down before it had recorded important information in status.

The enum relaxation will also help with rollbacks that drop capabilities. Those currently fail on the same status-sync issue:

$ oc -n openshift-cluster-version logs -l k8s-app=cluster-version-operator --tail 50 | grep 'ClusterVersion.config.openshift.io "version" is invalid:' | tail -n1
I1005 01:41:28.165931       1 cvo.go:600] Dropping "openshift-cluster-version/version" out of the queue &{0xc00039eae0 0xc0004142e8}: ClusterVersion.config.openshift.io "version" is invalid: [status.capabilities.knownCapabilities[0]: Unsupported value: "Build": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.knownCapabilities[3]: Unsupported value: "DeploymentConfig": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.knownCapabilities[4]: Unsupported value: "ImageRegistry": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.knownCapabilities[6]: Unsupported value: "MachineAPI": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.enabledCapabilities[0]: Unsupported value: "Build": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.enabledCapabilities[3]: Unsupported value: "DeploymentConfig": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.enabledCapabilities[4]: Unsupported value: "ImageRegistry": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.enabledCapabilities[6]: Unsupported value: "MachineAPI": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning"]

but unlike the roll-forward case, where that is a temporary issue, on rollbacks the additional capabilities never became acceptable again, and the cluster-version operator was permanently blocked from writing to ClusterVersion status. With the status enum removal in this commit, ClusterVersion status syncing will remain working, even though it will list as an enabled capability the capability which had been added during the earlier roll-forward.

wking avatar Oct 16 '23 16:10 wking

Hello @wking! Some important instructions when contributing to openshift/api: API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

openshift-ci[bot] avatar Oct 16 '23 16:10 openshift-ci[bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wking Once this PR has been reviewed and has the lgtm label, please assign jwforres for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci[bot] avatar Oct 16 '23 16:10 openshift-ci[bot]

I'm in favor of this since we should be the only ones writing to status and we've already been through at least one painful revert as a result of this validation logic(in addition to the benefits we'd get for routine upgrades that @wking outlined). But definitely want to hear from @deads2k

bparees avatar Oct 17 '23 15:10 bparees

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-merge-robot avatar Jan 15 '24 10:01 openshift-merge-robot

@wking: This pull request references OTA-1223 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

The cluster-version operator should be the only actor writing to ClusterVersion status. Trust it to pick appropriate values, instead of restricting it with an enum.

When updating into new capabilities, the current timeline is:

  1. Outgoing CVO accepts the new target, and figures out verified and implicit capabilities.
  2. Outoing CVO starts trying to write to status, but because of the new capability not being a part of the outgoing CRD's capability enum, this fails.
  3. Outgoing CVO pushes the incoming ClusterVersion CRD in runlevel 0 index 1.
  4. status sync attempts, if any, start working again.
  5. Outgoing CVO pushes the incoming CVO Deployment in runlevel 0 index 3.
  6. Deployment controller TERMs the outgoing CVO process.
  7. Outgoing CVO wraps up the manifest sycning.
  8. Outgoing CVO attempts a final status sync.
  9. Outgoing CVO releases the leader lock.
  10. Outgoing CVO exists.

With the status enum removal in this commit, ClusterVersion status syncing will remain working the whole time, reducing the risk of the outgoing CVO shutting down before it had recorded important information in status.

The enum relaxation will also help with rollbacks that drop capabilities. Those currently fail on the same status-sync issue:

$ oc -n openshift-cluster-version logs -l k8s-app=cluster-version-operator --tail 50 | grep 'ClusterVersion.config.openshift.io "version" is invalid:' | tail -n1
I1005 01:41:28.165931       1 cvo.go:600] Dropping "openshift-cluster-version/version" out of the queue &{0xc00039eae0 0xc0004142e8}: ClusterVersion.config.openshift.io "version" is invalid: [status.capabilities.knownCapabilities[0]: Unsupported value: "Build": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.knownCapabilities[3]: Unsupported value: "DeploymentConfig": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.knownCapabilities[4]: Unsupported value: "ImageRegistry": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.knownCapabilities[6]: Unsupported value: "MachineAPI": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.enabledCapabilities[0]: Unsupported value: "Build": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.enabledCapabilities[3]: Unsupported value: "DeploymentConfig": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.enabledCapabilities[4]: Unsupported value: "ImageRegistry": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning", status.capabilities.enabledCapabilities[6]: Unsupported value: "MachineAPI": supported values: "openshift-samples", "baremetal", "marketplace", "Console", "Insights", "Storage", "CSISnapshot", "NodeTuning"]

but unlike the roll-forward case, where that is a temporary issue, on rollbacks the additional capabilities never became acceptable again, and the cluster-version operator was permanently blocked from writing to ClusterVersion status. With the status enum removal in this commit, ClusterVersion status syncing will remain working, even though it will list as an enabled capability the capability which had been added during the earlier roll-forward.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Feb 27 '24 18:02 openshift-ci-robot

+1 to this from a hive (fleet manager) perspective. Scenario:

  • Hive manages a local (on the hub) CR with a field which imports the o/api schema for a field containing this enum.
  • That CRD is defined based on vendoring o/api at some version where the enum contains X,Y.
  • Hive is managing a spoke cluster at a higher OCP version where the enum contains X,Y,Z, and where a value in the spoke's ClusterVersion is Z.
  • Attempting to store the CR with the inherited field will bounce, as Z is not a valid value for the enum according to the older o/api schema registered under that CRD.

2uasimojo avatar Feb 29 '24 21:02 2uasimojo

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot avatar Jun 18 '24 01:06 openshift-bot

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

openshift-bot avatar Aug 11 '24 08:08 openshift-bot

@wking: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-serial-techpreview 75cddc1269392bcc325efc28cfaae22380ea0484 link true /test e2e-aws-serial-techpreview
ci/prow/e2e-aws-ovn 75cddc1269392bcc325efc28cfaae22380ea0484 link true /test e2e-aws-ovn
ci/prow/e2e-aws-ovn-techpreview 75cddc1269392bcc325efc28cfaae22380ea0484 link true /test e2e-aws-ovn-techpreview
ci/prow/e2e-aws-serial 75cddc1269392bcc325efc28cfaae22380ea0484 link true /test e2e-aws-serial
ci/prow/e2e-upgrade 75cddc1269392bcc325efc28cfaae22380ea0484 link true /test e2e-upgrade
ci/prow/e2e-upgrade-minor 75cddc1269392bcc325efc28cfaae22380ea0484 link true /test e2e-upgrade-minor
ci/prow/e2e-aws-ovn-hypershift 75cddc1269392bcc325efc28cfaae22380ea0484 link true /test e2e-aws-ovn-hypershift
ci/prow/minor-e2e-upgrade-minor 75cddc1269392bcc325efc28cfaae22380ea0484 link true /test minor-e2e-upgrade-minor
ci/prow/minor-images 75cddc1269392bcc325efc28cfaae22380ea0484 link true /test minor-images

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci[bot] avatar Aug 15 '24 15:08 openshift-ci[bot]

bump

2uasimojo avatar Sep 04 '24 20:09 2uasimojo

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-bot avatar Oct 05 '24 00:10 openshift-bot

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci[bot] avatar Oct 05 '24 00:10 openshift-ci[bot]