system-upgrade-controller icon indicating copy to clipboard operation
system-upgrade-controller copied to clipboard

Published 20 hours ago • verified publisher icon rancher

No generated jobs after submitting update plan on RKE2 cluster

Open kagehisa opened this issue 2 years ago • 4 comments

Version v0.10.0

Platform/Architecture RKE2 v1.24.7 on SLES 15SP4 x86_64

Describe the bug Deploying the system-upgrade-controller seems to work fine, but after submitting the update plan according to this dokumentation results in no update and no created update jobs.

kubectl -n system-upgrade get plans -o yaml returns the two submitted update plans:

click to expand 
apiVersion: v1
items:
- apiVersion: upgrade.cattle.io/v1
  kind: Plan
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"upgrade.cattle.io/v1","kind":"Plan","metadata":{"annotations":{},"labels":{"rke2-upgrade":"agent"},"name":"agent-plan","namespace":"system-upgrade"},"spec":{"concurrency":2,"cordon":true,"drain":{"force":true},"nodeSelector":{"matchExpressions":[{"key":"rke2-upgrade","operator":"Exists"},{"key":"rke2-upgrade","operator":"NotIn","values":["disabled","false"]},{"key":"node-role.kubernetes.io/control-plane","operator":"NotIn","values":["true"]}]},"prepare":{"args":["prepare","server-plan"],"image":"rancher/rke2-upgrade"},"serviceAccountName":"system-upgrade","upgrade":{"image":"rancher/rke2-upgrade"},"version":"v1.24.8-rke2r1"}}
    creationTimestamp: "2022-12-13T11:56:00Z"
    generation: 1
    labels:
      rke2-upgrade: agent
    managedFields:
    - apiVersion: upgrade.cattle.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:rke2-upgrade: {}
        f:spec:
          .: {}
          f:concurrency: {}
          f:cordon: {}
          f:drain:
            .: {}
            f:force: {}
          f:nodeSelector:
            .: {}
            f:matchExpressions: {}
          f:prepare:
            .: {}
            f:args: {}
            f:image: {}
          f:serviceAccountName: {}
          f:upgrade:
            .: {}
            f:image: {}
          f:version: {}
      manager: kubectl
      operation: Update
      time: "2022-12-13T11:56:00Z"
    - apiVersion: upgrade.cattle.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
          f:latestHash: {}
          f:latestVersion: {}
      manager: system-upgrade-controller
      operation: Update
      subresource: status
      time: "2022-12-13T11:56:00Z"
    name: agent-plan
    namespace: system-upgrade
    resourceVersion: "16986839"
    uid: 67fd9a78-50e9-41f9-a089-bac56fc22eee
  spec:
    concurrency: 2
    cordon: true
    drain:
      force: true
    nodeSelector:
      matchExpressions:
      - key: rke2-upgrade
        operator: Exists
      - key: rke2-upgrade
        operator: NotIn
        values:
        - disabled
        - "false"
      - key: node-role.kubernetes.io/control-plane
        operator: NotIn
        values:
        - "true"
    prepare:
      args:
      - prepare
      - server-plan
      image: rancher/rke2-upgrade
    serviceAccountName: system-upgrade
    upgrade:
      image: rancher/rke2-upgrade
    version: v1.24.8-rke2r1
  status:
    conditions:
    - lastUpdateTime: "2022-12-13T11:56:00Z"
      reason: Version
      status: "True"
      type: LatestResolved
    latestHash: 3a1bea586e00acd8df7303f3d37b9b69d81bf3cb108a20527a981c87
    latestVersion: v1.24.8-rke2r1
- apiVersion: upgrade.cattle.io/v1
  kind: Plan
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"upgrade.cattle.io/v1","kind":"Plan","metadata":{"annotations":{},"labels":{"rke2-upgrade":"server"},"name":"server-plan","namespace":"system-upgrade"},"spec":{"concurrency":1,"cordon":true,"nodeSelector":{"matchExpressions":[{"key":"rke2-upgrade","operator":"Exists"},{"key":"rke2-upgrade","operator":"NotIn","values":["disabled","false"]},{"key":"node-role.kubernetes.io/control-plane","operator":"In","values":["true"]}]},"serviceAccountName":"system-upgrade","upgrade":{"image":"rancher/rke2-upgrade"},"version":"v1.24.8-rke2r1"}}
    creationTimestamp: "2022-12-13T11:56:00Z"
    generation: 1
    labels:
      rke2-upgrade: server
    managedFields:
    - apiVersion: upgrade.cattle.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:rke2-upgrade: {}
        f:spec:
          .: {}
          f:concurrency: {}
          f:cordon: {}
          f:nodeSelector:
            .: {}
            f:matchExpressions: {}
          f:serviceAccountName: {}
          f:upgrade:
            .: {}
            f:image: {}
          f:version: {}
      manager: kubectl
      operation: Update
      time: "2022-12-13T11:56:00Z"
    - apiVersion: upgrade.cattle.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
          f:latestHash: {}
          f:latestVersion: {}
      manager: system-upgrade-controller
      operation: Update
      subresource: status
      time: "2022-12-13T11:56:00Z"
    name: server-plan
    namespace: system-upgrade
    resourceVersion: "16986837"
    uid: 6937a30f-d3be-4456-9783-e4132169fe1c
  spec:
    concurrency: 1
    cordon: true
    nodeSelector:
      matchExpressions:
      - key: rke2-upgrade
        operator: Exists
      - key: rke2-upgrade
        operator: NotIn
        values:
        - disabled
        - "false"
      - key: node-role.kubernetes.io/control-plane
        operator: In
        values:
        - "true"
    serviceAccountName: system-upgrade
    upgrade:
      image: rancher/rke2-upgrade
    version: v1.24.8-rke2r1
  status:
    conditions:
    - lastUpdateTime: "2022-12-13T11:56:00Z"
      reason: Version
      status: "True"
      type: LatestResolved
    latestHash: 3a1bea586e00acd8df7303f3d37b9b69d81bf3cb108a20527a981c87
    latestVersion: v1.24.8-rke2r1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

But all I get via kubectl -n system-upgrade get jobs -o yaml is an empty list:

apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I deployed the update-controller with debug output enabled, the logs are of no real help to me. Maybe somebody can see something in them:

click to open 
11:55:18.725648       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2022-12-13T11:55:19Z" level=info msg="Applying CRD plans.upgrade.cattle.io" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:55:19Z" level=debug msg="DesiredSet - Patch apiextensions.k8s.io/v1, Kind=CustomResourceDefinition /plans.upgrade.cattle.io for  plans.upgrade.cattle.io -- [PATCH:{\"metadata\":{},\"spec\":{\"preserveUnknownFields\":false}}, ORIGINAL:{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"objectset.rio.cattle.io/applied\":\"H4sIAAAAAAAA/+xZUW/bNhD+KwOfNS9umrQwsIciabCiXWA0afYw5OFEnmTWFMkdKTVG4P8+ULJkJ7ElznWHDfAjyePdx+/u6M/iIwMr75CcNJpNwgAfPOowdKP5WzeS5pdqzBI2l1qwCbsonTfFZ3SmJI6XmEktfdibsAI9CPDAJo8MtDYewoILQ5N+Re4d+hFJM+LgvcLgWQaXLNm5br5ppJ/zar4bWvLTR6nFrz24+p1rKJBNmFWg3ai0OYHAtVHUdmeBBx9smTAFKareM8/AzdiE4QnPXqWvT8/O34xPT96enL06Hacnb9ITfg7pq/HZ+esUx2eCB6cDGJcJcxZ5CJqTKS2bsG0HqaEGIw4ec0MyjP5sTdl9l+OpgsCbVSWBauOyhDmp81IBraZCXEvokCr8oufafNNXEpVwbJKBcpgwx40NwK9bkgRLWNUUW4j9yECIOk2gpiS1R7owqixWi1+d0VPwga1ROGB3dFlAjqzj5cNq6Bd1NOdJ6pwtky0e+Ay0RrXee9FNROxeQV/vvusmnu2+70zq5nF8hkXdGMaifjf9cHd6001ZMhbJyyY5bSafzra4J49Ml0pBqpBNPJX4EjfjRvOSCDVfBPvVemA3R2oMSIRu79ZSYxQ2CRUEUm8L8xSPQIUePxkO6nLV8jtwbfqWLhi8rySvW3MrgMxQaKZtSzkBxymSNKIn4MZBZa4N4SVgYfQNehcH082l/QOkvzJ0WR/zVhZoSr+dTL9eHAK07Kaau6FubSPwBhVyb2iY9gI8n71/CF3n2rtVeizcy4qZ4yKqWsIe2BH8hXEFqsSnUQf37DAAIljs4KR3R9Kw8Km7aDfvkE0G9kXWAtkGzRJaIBzOFFB+UJ5C1xYFaHFYp6irKzJFTx1xozOZ/w72M2bDx24uvaiya1IW05I7EpHJh6hQDjmh/y/B36foUVd97R6NvG7heMu2Ovp564rkIy6iiI69nf61esqCcomCvqmXI4DVjhsVMWi9DRitJO1VNEButAepka5jyROyki7yJ6DFs+dxmmb8n5bJYTq5Ua9RHRjEMPa1vUDnpQYfW43RRH1Xkv/5T35TFge54Oz+vTaUufBXR3J8x7kptY9uL29U0FgDgg2zLMCIcfhDtN0a5Q1yo0WfwNjQv7G/J/vw3f41Paqto9o6qq2j2jqqraPaOqqt71dbVXRH1SX4VykJxbNvxS9rzYMvt5AG1qpFcHbg32fdfPXpy5cC528JtKstb2VkNsK2L1ZA8+UvakuBzsXWGSG4yAJaczos3+qJHyLDFHh0/rf63SKKvmAef2tHdn79ziBaH84bqglfDcu0vSqbr+gtccvl8v4JkcA5Wo/iun0NWT16bD54PC+wcIAmJIq77vkizC6XfwcAAP//MtmtGkEbAAA\",\"objectset.rio.cattle.io/id\":\"\",\"objectset.rio.cattle.io/owner-gvk\":\"apiextensions.k8s.io/v1, Kind=CustomResourceDefinition\",\"objectset.rio.cattle.io/owner-name\":\"plans.upgrade.cattle.io\",\"objectset.rio.cattle.io/owner-namespace\":\"\"},\"labels\":{\"objectset.rio.cattle.io/hash\":\"e0cf2b43567130805231b07b0c6ab21564be15dc\"},\"name\":\"plans.upgrade.cattle.io\"},\"spec\":{\"group\":\"upgrade.cattle.io\",\"names\":{\"categories\":[\"upgrade\"],\"kind\":\"Plan\",\"plural\":\"plans\",\"singular\":\"plan\"},\"preserveUnknownFields\":false,\"scope\":\"Namespaced\",\"versions\":[{\"additionalPrinterColumns\":[{\"jsonPath\":\".spec.upgrade.image\",\"name\":\"Image\",\"type\":\"string\"},{\"jsonPath\":\".spec.channel\",\"name\":\"Channel\",\"type\":\"string\"},{\"jsonPath\":\".spec.version\",\"name\":\"Version\",\"type\":\"string\"}],\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"properties\":{\"spec\":{\"properties\":{\"channel\":{\"nullable\":true,\"type\":\"string\"},\"concurrency\":{\"type\":\"integer\"},\"cordon\":{\"type\":\"boolean\"},\"drain\":{\"nullable\":true,\"properties\":{\"deleteLocalData\":{\"nullable\":true,\"type\":\"boolean\"},\"disableEviction\":{\"type\":\"boolean\"},\"force\":{\"type\":\"boolean\"},\"gracePeriod\":{\"nullable\":true,\"type\":\"integer\"},\"ignoreDaemonSets\":{\"nullable\":true,\"type\":\"boolean\"},\"skipWaitForDeleteTimeout\":{\"type\":\"integer\"},\"timeout\":{\"nullable\":true,\"type\":\"integer\"}},\"type\":\"object\"},\"nodeSelector\":{\"nullable\":true,\"properties\":{\"matchExpressions\":{\"items\":{\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"operator\":{\"nullable\":true,\"type\":\"string\"},\"values\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"matchLabels\":{\"additionalProperties\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"object\"}},\"type\":\"object\"},\"prepare\":{\"nullable\":true,\"properties\":{\"args\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"command\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"envFrom\":{\"items\":{\"properties\":{\"configMapRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"prefix\":{\"nullable\":true,\"type\":\"string\"},\"secretRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"envs\":{\"items\":{\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"value\":{\"nullable\":true,\"type\":\"string\"},\"valueFrom\":{\"nullable\":true,\"properties\":{\"configMapKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"fieldRef\":{\"nullable\":true,\"properties\":{\"apiVersion\":{\"nullable\":true,\"type\":\"string\"},\"fieldPath\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"resourceFieldRef\":{\"nullable\":true,\"properties\":{\"containerName\":{\"nullable\":true,\"type\":\"string\"},\"divisor\":{\"nullable\":true,\"type\":\"string\"},\"resource\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"secretKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"image\":{\"nullable\":true,\"type\":\"string\"},\"volumes\":{\"items\":{\"properties\":{\"destination\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"source\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"}},\"type\":\"object\"},\"secrets\":{\"items\":{\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"path\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"serviceAccountName\":{\"nullable\":true,\"type\":\"string\"},\"tolerations\":{\"items\":{\"properties\":{\"effect\":{\"nullable\":true,\"type\":\"string\"},\"key\":{\"nullable\":true,\"type\":\"string\"},\"operator\":{\"nullable\":true,\"type\":\"string\"},\"tolerationSeconds\":{\"nullable\":true,\"type\":\"integer\"},\"value\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"upgrade\":{\"nullable\":true,\"properties\":{\"args\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"command\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"envFrom\":{\"items\":{\"properties\":{\"configMapRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"prefix\":{\"nullable\":true,\"type\":\"string\"},\"secretRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"envs\":{\"items\":{\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"value\":{\"nullable\":true,\"type\":\"string\"},\"valueFrom\":{\"nullable\":true,\"properties\":{\"configMapKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"fieldRef\":{\"nullable\":true,\"properties\":{\"apiVersion\":{\"nullable\":true,\"type\":\"string\"},\"fieldPath\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"resourceFieldRef\":{\"nullable\":true,\"properties\":{\"containerName\":{\"nullable\":true,\"type\":\"string\"},\"divisor\":{\"nullable\":true,\"type\":\"string\"},\"resource\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"secretKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"image\":{\"nullable\":true,\"type\":\"string\"},\"volumes\":{\"items\":{\"properties\":{\"destination\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"source\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"}},\"type\":\"object\"},\"version\":{\"nullable\":true,\"type\":\"string\"}},\"required\":[\"upgrade\"],\"type\":\"object\"},\"status\":{\"properties\":{\"applying\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"conditions\":{\"items\":{\"properties\":{\"lastTransitionTime\":{\"nullable\":true,\"type\":\"string\"},\"lastUpdateTime\":{\"nullable\":true,\"type\":\"string\"},\"message\":{\"nullable\":true,\"type\":\"string\"},\"reason\":{\"nullable\":true,\"type\":\"string\"},\"status\":{\"nullable\":true,\"type\":\"string\"},\"type\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"latestHash\":{\"nullable\":true,\"type\":\"string\"},\"latestVersion\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true,\"subresources\":{\"status\":{}}}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":null,\"storedVersions\":null}}, MODIFIED:{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"objectset.rio.cattle.io/applied\":\"H4sIAAAAAAAA/+xZW2/bNhT+KwOfNS+XJi0E7KFIGqxoFwRNmz0MeTgij2TWFMkdUmqMwP99oGTJTmJLnOsOG+BHkufy8TsXH4uPDKy8Q3LSaJaGBT541GHpJrM3biLNL/UxS9hMasFSdlE5b8pP6ExFHC8xl1r6oJuwEj0I8MDSRwZaGw/hwIWlyb4i9w79hKSZcPBeYbAsg0mWbD033zTSz0U92w4t+emD1OLXAVzDxjWUyFJmFWg3qWxBIHAlFKXuLPBggy0Sxgmba3+WJToPpWWprpRKmIIM1SAZU3BTljI84vlJ9ur07Pz18enRm6Ozk9Pj7Oh1dsTPITs5Pjt/leHxmeDB2wj4RcKcRR6cFmQqy1K26YbNHYIQB4+FIRlWf3ai7L4P/o2CQKhVFYHq/LKEOamLSgEtt4JfS+iQavyiZ9p801cSlXAszUE5TJjjxgbg1x17giWsbrMw+H5kIEQTP1A3JLVHujCqKpeHX53RN+ADW5Nwwf7qsoQCWc/L++XSzxtvzpPUBVskGyzwKWiNaqV70W9EaC+hr7Tv+o1n2ve9SFNVjk+xbCrGWNRvb97fnd72W5aMRfKyDU4Xyae7He70kYU8g0whSz1V+BI340bzigg1nwf55Xlgt0BqBUiENtCfZcYobAMqCKTe5OYpHoEKPX40HNTlshdswbVuW7og8K6WvKnZjQByQ6HKNh0VBBxvkKQRAw7XLioLbQgvAUujb9G7OJhuJu0fIP2VocvmmqHITeU3k+lXh2OAFv1W2xua0jYCb1Eh94bGaS/B8+m7h1B1rmu60mPpXmbMDOdR2RJ0YIvzF8I1qAqfeh3V2SIARDDfwsmgRtKy8LFvtOs9ZJ2BXZF1QDZBs4QWCMcjBVTsladQtWUJWuzXKOr6ikw5kEfc6FwWv4P9hPn4tdumF5V2bchiSnJLIHL5EOXKISf0/yX4uyQ96nqo3KORNyUcL9llxzBvfZJ8wHkU0bHd6V/LpzxMLlHQ1wfpCGCN4XaKGJXeBIyWs+5VNEButAepka5jyROyli7yJ6DDs+N12mL8n6bJfiq5nV6jKjAMwzhU9gKdlxp8bDZGE/VdQf7nP/ltWuylwdnda20scuGvjuT4lnNTaR9dXt6oMGONDGyY5wFGjMEfMtutUN4iN1oMDRhr82/s78kufHd/TQ/T1mHaOkxbh2nrMG0dpq3DtPX901YdXVFNCv5VSULx7Fvxy1zz4KsNpIG1ah6M7fn3WbdffYbipcD5zwTaye5bfZTvoPbFCmi//EWplOhcbJ4RgotMoBWn4+Nbs/FDxjAFHp3/rXm3iKIviMd37cjKb94ZRGfDeUMN4ctllXWtsv2K3hG3WCzunxAJnKP1KK6715Dlo8f6g8fzBGufdoJLFHf980XYXSz+DgAA//+zr/NTWhsAAA\",\"objectset.rio.cattle.io/id\":\"\",\"objectset.rio.cattle.io/owner-gvk\":\"apiextensions.k8s.io/v1, Kind=CustomResourceDefinition\",\"objectset.rio.cattle.io/owner-name\":\"plans.upgrade.cattle.io\",\"objectset.rio.cattle.io/owner-namespace\":\"\"},\"creationTimestamp\":null,\"labels\":{\"objectset.rio.cattle.io/hash\":\"e0cf2b43567130805231b07b0c6ab21564be15dc\"},\"name\":\"plans.upgrade.cattle.io\"},\"spec\":{\"group\":\"upgrade.cattle.io\",\"names\":{\"categories\":[\"upgrade\"],\"kind\":\"Plan\",\"plural\":\"plans\",\"singular\":\"plan\"},\"preserveUnknownFields\":false,\"scope\":\"Namespaced\",\"versions\":[{\"additionalPrinterColumns\":[{\"jsonPath\":\".spec.upgrade.image\",\"name\":\"Image\",\"type\":\"string\"},{\"jsonPath\":\".spec.channel\",\"name\":\"Channel\",\"type\":\"string\"},{\"jsonPath\":\".spec.version\",\"name\":\"Version\",\"type\":\"string\"}],\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"properties\":{\"spec\":{\"properties\":{\"channel\":{\"nullable\":true,\"type\":\"string\"},\"concurrency\":{\"type\":\"integer\"},\"cordon\":{\"type\":\"boolean\"},\"drain\":{\"nullable\":true,\"properties\":{\"deleteLocalData\":{\"nullable\":true,\"type\":\"boolean\"},\"disableEviction\":{\"type\":\"boolean\"},\"force\":{\"type\":\"boolean\"},\"gracePeriod\":{\"nullable\":true,\"type\":\"integer\"},\"ignoreDaemonSets\":{\"nullable\":true,\"type\":\"boolean\"},\"skipWaitForDeleteTimeout\":{\"type\":\"integer\"},\"timeout\":{\"nullable\":true,\"type\":\"integer\"}},\"type\":\"object\"},\"nodeSelector\":{\"nullable\":true,\"properties\":{\"matchExpressions\":{\"items\":{\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"operator\":{\"nullable\":true,\"type\":\"string\"},\"values\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"matchLabels\":{\"additionalProperties\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"object\"}},\"type\":\"object\"},\"prepare\":{\"nullable\":true,\"properties\":{\"args\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"command\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"envFrom\":{\"items\":{\"properties\":{\"configMapRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"prefix\":{\"nullable\":true,\"type\":\"string\"},\"secretRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"envs\":{\"items\":{\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"value\":{\"nullable\":true,\"type\":\"string\"},\"valueFrom\":{\"nullable\":true,\"properties\":{\"configMapKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"fieldRef\":{\"nullable\":true,\"properties\":{\"apiVersion\":{\"nullable\":true,\"type\":\"string\"},\"fieldPath\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"resourceFieldRef\":{\"nullable\":true,\"properties\":{\"containerName\":{\"nullable\":true,\"type\":\"string\"},\"divisor\":{\"nullable\":true,\"type\":\"string\"},\"resource\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"secretKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"image\":{\"nullable\":true,\"type\":\"string\"},\"volumes\":{\"items\":{\"properties\":{\"destination\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"source\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"}},\"type\":\"object\"},\"secrets\":{\"items\":{\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"path\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"serviceAccountName\":{\"nullable\":true,\"type\":\"string\"},\"tolerations\":{\"items\":{\"properties\":{\"effect\":{\"nullable\":true,\"type\":\"string\"},\"key\":{\"nullable\":true,\"type\":\"string\"},\"operator\":{\"nullable\":true,\"type\":\"string\"},\"tolerationSeconds\":{\"nullable\":true,\"type\":\"integer\"},\"value\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"upgrade\":{\"nullable\":true,\"properties\":{\"args\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"command\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"envFrom\":{\"items\":{\"properties\":{\"configMapRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"prefix\":{\"nullable\":true,\"type\":\"string\"},\"secretRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"envs\":{\"items\":{\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"value\":{\"nullable\":true,\"type\":\"string\"},\"valueFrom\":{\"nullable\":true,\"properties\":{\"configMapKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"fieldRef\":{\"nullable\":true,\"properties\":{\"apiVersion\":{\"nullable\":true,\"type\":\"string\"},\"fieldPath\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"resourceFieldRef\":{\"nullable\":true,\"properties\":{\"containerName\":{\"nullable\":true,\"type\":\"string\"},\"divisor\":{\"nullable\":true,\"type\":\"string\"},\"resource\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"secretKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"image\":{\"nullable\":true,\"type\":\"string\"},\"volumes\":{\"items\":{\"properties\":{\"destination\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"source\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"}},\"type\":\"object\"},\"version\":{\"nullable\":true,\"type\":\"string\"}},\"required\":[\"upgrade\"],\"type\":\"object\"},\"status\":{\"properties\":{\"applying\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"conditions\":{\"items\":{\"properties\":{\"lastTransitionTime\":{\"nullable\":true,\"type\":\"string\"},\"lastUpdateTime\":{\"nullable\":true,\"type\":\"string\"},\"message\":{\"nullable\":true,\"type\":\"string\"},\"reason\":{\"nullable\":true,\"type\":\"string\"},\"status\":{\"nullable\":true,\"type\":\"string\"},\"type\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"latestHash\":{\"nullable\":true,\"type\":\"string\"},\"latestVersion\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true,\"subresources\":{\"status\":{}}}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":null,\"storedVersions\":null}}, CURRENT:{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"apiextensions.k8s.io/v1\\\",\\\"kind\\\":\\\"CustomResourceDefinition\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"name\\\":\\\"plans.upgrade.cattle.io\\\"},\\\"spec\\\":{\\\"group\\\":\\\"upgrade.cattle.io\\\",\\\"names\\\":{\\\"categories\\\":[\\\"upgrade\\\"],\\\"kind\\\":\\\"Plan\\\",\\\"plural\\\":\\\"plans\\\",\\\"singular\\\":\\\"plan\\\"},\\\"preserveUnknownFields\\\":false,\\\"scope\\\":\\\"Namespaced\\\",\\\"versions\\\":[{\\\"additionalPrinterColumns\\\":[{\\\"jsonPath\\\":\\\".spec.upgrade.image\\\",\\\"name\\\":\\\"Image\\\",\\\"type\\\":\\\"string\\\"},{\\\"jsonPath\\\":\\\".spec.channel\\\",\\\"name\\\":\\\"Channel\\\",\\\"type\\\":\\\"string\\\"},{\\\"jsonPath\\\":\\\".spec.version\\\",\\\"name\\\":\\\"Version\\\",\\\"type\\\":\\\"string\\\"}],\\\"name\\\":\\\"v1\\\",\\\"schema\\\":{\\\"openAPIV3Schema\\\":{\\\"properties\\\":{\\\"spec\\\":{\\\"properties\\\":{\\\"channel\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"concurrency\\\":{\\\"type\\\":\\\"integer\\\"},\\\"cordon\\\":{\\\"type\\\":\\\"boolean\\\"},\\\"drain\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"deleteLocalData\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"},\\\"disableEviction\\\":{\\\"type\\\":\\\"boolean\\\"},\\\"force\\\":{\\\"type\\\":\\\"boolean\\\"},\\\"gracePeriod\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"integer\\\"},\\\"ignoreDaemonSets\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"},\\\"skipWaitForDeleteTimeout\\\":{\\\"type\\\":\\\"integer\\\"},\\\"timeout\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"integer\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nodeSelector\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"matchExpressions\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"key\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"operator\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"values\\\":{\\\"items\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"matchLabels\\\":{\\\"additionalProperties\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"object\\\"}},\\\"type\\\":\\\"object\\\"},\\\"prepare\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"args\\\":{\\\"items\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"command\\\":{\\\"items\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"envFrom\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"configMapRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"optional\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"}},\\\"type\\\":\\\"object\\\"},\\\"prefix\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"secretRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"optional\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"}},\\\"type\\\":\\\"object\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"envs\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"value\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"valueFrom\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"configMapKeyRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"key\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"optional\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"}},\\\"type\\\":\\\"object\\\"},\\\"fieldRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"apiVersion\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"fieldPath\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"},\\\"resourceFieldRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"containerName\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"divisor\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"resource\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"},\\\"secretKeyRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"key\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"optional\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"}},\\\"type\\\":\\\"object\\\"}},\\\"type\\\":\\\"object\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"image\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"volumes\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"destination\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"source\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"}},\\\"type\\\":\\\"object\\\"},\\\"secrets\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"path\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"serviceAccountName\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"tolerations\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"effect\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"key\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"operator\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"tolerationSeconds\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"integer\\\"},\\\"value\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"upgrade\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"args\\\":{\\\"items\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"command\\\":{\\\"items\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"envFrom\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"configMapRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"optional\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"}},\\\"type\\\":\\\"object\\\"},\\\"prefix\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"secretRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"optional\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"}},\\\"type\\\":\\\"object\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"envs\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"value\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"valueFrom\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"configMapKeyRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"key\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"optional\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"}},\\\"type\\\":\\\"object\\\"},\\\"fieldRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"apiVersion\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"fieldPath\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"},\\\"resourceFieldRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"containerName\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"divisor\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"resource\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"},\\\"secretKeyRef\\\":{\\\"nullable\\\":true,\\\"properties\\\":{\\\"key\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"optional\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"boolean\\\"}},\\\"type\\\":\\\"object\\\"}},\\\"type\\\":\\\"object\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"image\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"volumes\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"destination\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"name\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"source\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"}},\\\"type\\\":\\\"object\\\"},\\\"version\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"required\\\":[\\\"upgrade\\\"],\\\"type\\\":\\\"object\\\"},\\\"status\\\":{\\\"properties\\\":{\\\"applying\\\":{\\\"items\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"conditions\\\":{\\\"items\\\":{\\\"properties\\\":{\\\"lastTransitionTime\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"lastUpdateTime\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"message\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"reason\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"status\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"type\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"},\\\"nullable\\\":true,\\\"type\\\":\\\"array\\\"},\\\"latestHash\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"},\\\"latestVersion\\\":{\\\"nullable\\\":true,\\\"type\\\":\\\"string\\\"}},\\\"type\\\":\\\"object\\\"}},\\\"type\\\":\\\"object\\\"}},\\\"served\\\":true,\\\"storage\\\":true,\\\"subresources\\\":{\\\"status\\\":{}}}]}}\\n\",\"objectset.rio.cattle.io/applied\":\"H4sIAAAAAAAA/+xZW2/bNhT+KwOfNS+XJi0E7KFIGqxoFwRNmz0MeTgij2TWFMkdUmqMwP99oGTJTmJLnOsOG+BHkufy8TsXH4uPDKy8Q3LSaJaGBT541GHpJrM3biLNL/UxS9hMasFSdlE5b8pP6ExFHC8xl1r6oJuwEj0I8MDSRwZaGw/hwIWlyb4i9w79hKSZcPBeYbAsg0mWbD033zTSz0U92w4t+emD1OLXAVzDxjWUyFJmFWg3qWxBIHAlFKXuLPBggy0Sxgmba3+WJToPpWWprpRKmIIM1SAZU3BTljI84vlJ9ur07Pz18enRm6Ozk9Pj7Oh1dsTPITs5Pjt/leHxmeDB2wj4RcKcRR6cFmQqy1K26YbNHYIQB4+FIRlWf3ai7L4P/o2CQKhVFYHq/LKEOamLSgEtt4JfS+iQavyiZ9p801cSlXAszUE5TJjjxgbg1x17giWsbrMw+H5kIEQTP1A3JLVHujCqKpeHX53RN+ADW5Nwwf7qsoQCWc/L++XSzxtvzpPUBVskGyzwKWiNaqV70W9EaC+hr7Tv+o1n2ve9SFNVjk+xbCrGWNRvb97fnd72W5aMRfKyDU4Xyae7He70kYU8g0whSz1V+BI340bzigg1nwf55Xlgt0BqBUiENtCfZcYobAMqCKTe5OYpHoEKPX40HNTlshdswbVuW7og8K6WvKnZjQByQ6HKNh0VBBxvkKQRAw7XLioLbQgvAUujb9G7OJhuJu0fIP2VocvmmqHITeU3k+lXh2OAFv1W2xua0jYCb1Eh94bGaS/B8+m7h1B1rmu60mPpXmbMDOdR2RJ0YIvzF8I1qAqfeh3V2SIARDDfwsmgRtKy8LFvtOs9ZJ2BXZF1QDZBs4QWCMcjBVTsladQtWUJWuzXKOr6ikw5kEfc6FwWv4P9hPn4tdumF5V2bchiSnJLIHL5EOXKISf0/yX4uyQ96nqo3KORNyUcL9llxzBvfZJ8wHkU0bHd6V/LpzxMLlHQ1wfpCGCN4XaKGJXeBIyWs+5VNEButAepka5jyROyli7yJ6DDs+N12mL8n6bJfiq5nV6jKjAMwzhU9gKdlxp8bDZGE/VdQf7nP/ltWuylwdnda20scuGvjuT4lnNTaR9dXt6oMGONDGyY5wFGjMEfMtutUN4iN1oMDRhr82/s78kufHd/TQ/T1mHaOkxbh2nrMG0dpq3DtPX901YdXVFNCv5VSULx7Fvxy1zz4KsNpIG1ah6M7fn3WbdffYbipcD5zwTaye5bfZTvoPbFCmi//EWplOhcbJ4RgotMoBWn4+Nbs/FDxjAFHp3/rXm3iKIviMd37cjKb94ZRGfDeUMN4ctllXWtsv2K3hG3WCzunxAJnKP1KK6715Dlo8f6g8fzBGufdoJLFHf980XYXSz+DgAA//+zr/NTWhsAAA\",\"objectset.rio.cattle.io/id\":\"\",\"objectset.rio.cattle.io/owner-gvk\":\"apiextensions.k8s.io/v1, Kind=CustomResourceDefinition\",\"objectset.rio.cattle.io/owner-name\":\"plans.upgrade.cattle.io\",\"objectset.rio.cattle.io/owner-namespace\":\"\"},\"creationTimestamp\":\"2022-12-13T07:24:19Z\",\"generation\":1,\"labels\":{\"objectset.rio.cattle.io/hash\":\"e0cf2b43567130805231b07b0c6ab21564be15dc\"},\"managedFields\":[{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}}},\"f:spec\":{\"f:conversion\":{\".\":{},\"f:strategy\":{}},\"f:group\":{},\"f:names\":{\"f:categories\":{},\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:scope\":{},\"f:versions\":{}}},\"manager\":\"kubectl\",\"operation\":\"Update\",\"time\":\"2022-12-13T07:24:19Z\"},{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:acceptedNames\":{\"f:categories\":{},\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:conditions\":{\"k:{\\\"type\\\":\\\"Established\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NamesAccepted\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}}}},\"manager\":\"kube-apiserver\",\"operation\":\"Update\",\"subresource\":\"status\",\"time\":\"2022-12-13T07:24:20Z\"},{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:objectset.rio.cattle.io/applied\":{},\"f:objectset.rio.cattle.io/id\":{},\"f:objectset.rio.cattle.io/owner-gvk\":{},\"f:objectset.rio.cattle.io/owner-name\":{},\"f:objectset.rio.cattle.io/owner-namespace\":{}},\"f:labels\":{\".\":{},\"f:objectset.rio.cattle.io/hash\":{}}}},\"manager\":\"system-upgrade-controller\",\"operation\":\"Update\",\"time\":\"2022-12-13T08:13:04Z\"}],\"name\":\"plans.upgrade.cattle.io\",\"resourceVersion\":\"16904021\",\"uid\":\"86fe60fb-c64a-4798-bb1c-f23e49c8c882\"},\"spec\":{\"conversion\":{\"strategy\":\"None\"},\"group\":\"upgrade.cattle.io\",\"names\":{\"categories\":[\"upgrade\"],\"kind\":\"Plan\",\"listKind\":\"PlanList\",\"plural\":\"plans\",\"singular\":\"plan\"},\"scope\":\"Namespaced\",\"versions\":[{\"additionalPrinterColumns\":[{\"jsonPath\":\".spec.upgrade.image\",\"name\":\"Image\",\"type\":\"string\"},{\"jsonPath\":\".spec.channel\",\"name\":\"Channel\",\"type\":\"string\"},{\"jsonPath\":\".spec.version\",\"name\":\"Version\",\"type\":\"string\"}],\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"properties\":{\"spec\":{\"properties\":{\"channel\":{\"nullable\":true,\"type\":\"string\"},\"concurrency\":{\"type\":\"integer\"},\"cordon\":{\"type\":\"boolean\"},\"drain\":{\"nullable\":true,\"properties\":{\"deleteLocalData\":{\"nullable\":true,\"type\":\"boolean\"},\"disableEviction\":{\"type\":\"boolean\"},\"force\":{\"type\":\"boolean\"},\"gracePeriod\":{\"nullable\":true,\"type\":\"integer\"},\"ignoreDaemonSets\":{\"nullable\":true,\"type\":\"boolean\"},\"skipWaitForDeleteTimeout\":{\"type\":\"integer\"},\"timeout\":{\"nullable\":true,\"type\":\"integer\"}},\"type\":\"object\"},\"nodeSelector\":{\"nullable\":true,\"properties\":{\"matchExpressions\":{\"items\":{\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"operator\":{\"nullable\":true,\"type\":\"string\"},\"values\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"matchLabels\":{\"additionalProperties\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"object\"}},\"type\":\"object\"},\"prepare\":{\"nullable\":true,\"properties\":{\"args\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"command\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"envFrom\":{\"items\":{\"properties\":{\"configMapRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"prefix\":{\"nullable\":true,\"type\":\"string\"},\"secretRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"envs\":{\"items\":{\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"value\":{\"nullable\":true,\"type\":\"string\"},\"valueFrom\":{\"nullable\":true,\"properties\":{\"configMapKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"fieldRef\":{\"nullable\":true,\"properties\":{\"apiVersion\":{\"nullable\":true,\"type\":\"string\"},\"fieldPath\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"resourceFieldRef\":{\"nullable\":true,\"properties\":{\"containerName\":{\"nullable\":true,\"type\":\"string\"},\"divisor\":{\"nullable\":true,\"type\":\"string\"},\"resource\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"secretKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"image\":{\"nullable\":true,\"type\":\"string\"},\"volumes\":{\"items\":{\"properties\":{\"destination\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"source\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"}},\"type\":\"object\"},\"secrets\":{\"items\":{\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"path\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"serviceAccountName\":{\"nullable\":true,\"type\":\"string\"},\"tolerations\":{\"items\":{\"properties\":{\"effect\":{\"nullable\":true,\"type\":\"string\"},\"key\":{\"nullable\":true,\"type\":\"string\"},\"operator\":{\"nullable\":true,\"type\":\"string\"},\"tolerationSeconds\":{\"nullable\":true,\"type\":\"integer\"},\"value\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"upgrade\":{\"nullable\":true,\"properties\":{\"args\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"command\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"envFrom\":{\"items\":{\"properties\":{\"configMapRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"prefix\":{\"nullable\":true,\"type\":\"string\"},\"secretRef\":{\"nullable\":true,\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"envs\":{\"items\":{\"properties\":{\"name\":{\"nullable\":true,\"type\":\"string\"},\"value\":{\"nullable\":true,\"type\":\"string\"},\"valueFrom\":{\"nullable\":true,\"properties\":{\"configMapKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"},\"fieldRef\":{\"nullable\":true,\"properties\":{\"apiVersion\":{\"nullable\":true,\"type\":\"string\"},\"fieldPath\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"resourceFieldRef\":{\"nullable\":true,\"properties\":{\"containerName\":{\"nullable\":true,\"type\":\"string\"},\"divisor\":{\"nullable\":true,\"type\":\"string\"},\"resource\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"secretKeyRef\":{\"nullable\":true,\"properties\":{\"key\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"optional\":{\"nullable\":true,\"type\":\"boolean\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"image\":{\"nullable\":true,\"type\":\"string\"},\"volumes\":{\"items\":{\"properties\":{\"destination\":{\"nullable\":true,\"type\":\"string\"},\"name\":{\"nullable\":true,\"type\":\"string\"},\"source\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"}},\"type\":\"object\"},\"version\":{\"nullable\":true,\"type\":\"string\"}},\"required\":[\"upgrade\"],\"type\":\"object\"},\"status\":{\"properties\":{\"applying\":{\"items\":{\"nullable\":true,\"type\":\"string\"},\"nullable\":true,\"type\":\"array\"},\"conditions\":{\"items\":{\"properties\":{\"lastTransitionTime\":{\"nullable\":true,\"type\":\"string\"},\"lastUpdateTime\":{\"nullable\":true,\"type\":\"string\"},\"message\":{\"nullable\":true,\"type\":\"string\"},\"reason\":{\"nullable\":true,\"type\":\"string\"},\"status\":{\"nullable\":true,\"type\":\"string\"},\"type\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"},\"nullable\":true,\"type\":\"array\"},\"latestHash\":{\"nullable\":true,\"type\":\"string\"},\"latestVersion\":{\"nullable\":true,\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true,\"subresources\":{\"status\":{}}}]},\"status\":{\"acceptedNames\":{\"categories\":[\"upgrade\"],\"kind\":\"Plan\",\"listKind\":\"PlanList\",\"plural\":\"plans\",\"singular\":\"plan\"},\"conditions\":[{\"lastTransitionTime\":\"2022-12-13T07:24:20Z\",\"message\":\"no conflicts found\",\"reason\":\"NoConflicts\",\"status\":\"True\",\"type\":\"NamesAccepted\"},{\"lastTransitionTime\":\"2022-12-13T07:24:20Z\",\"message\":\"the initial names have been accepted\",\"reason\":\"InitialNamesAccepted\",\"status\":\"True\",\"type\":\"Established\"}],\"storedVersions\":[\"v1\"]}}]" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:55:19Z" level=debug msg="DesiredSet - Updated apiextensions.k8s.io/v1, Kind=CustomResourceDefinition /plans.upgrade.cattle.io for  plans.upgrade.cattle.io -- application/merge-patch+json {\"metadata\":{},\"spec\":{\"preserveUnknownFields\":false}}" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:55:19Z" level=info msg="Starting /v1, Kind=Node controller" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:55:19Z" level=info msg="Starting /v1, Kind=Secret controller" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:55:19Z" level=info msg="Starting batch/v1, Kind=Job controller" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:55:19Z" level=info msg="Starting upgrade.cattle.io/v1, Kind=Plan controller" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:56:00Z" level=debug msg="PLAN STATUS HANDLER: plan=system-upgrade/server-plan@16986836, status={Conditions:[] LatestVersion: LatestHash: Applying:[]}" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:56:00Z" level=debug msg="PLAN GENERATING HANDLER: plan=system-upgrade/server-plan@16986837, status={Conditions:[{Type:LatestResolved Status:True LastUpdateTime:2022-12-13T11:56:00Z LastTransitionTime: Reason:Version Message:}] LatestVersion:v1.24.8-rke2r1 LatestHash:3a1bea586e00acd8df7303f3d37b9b69d81bf3cb108a20527a981c87 Applying:[]}" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:56:00Z" level=debug msg="PLAN STATUS HANDLER: plan=system-upgrade/agent-plan@16986838, status={Conditions:[] LatestVersion: LatestHash: Applying:[]}" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:56:00Z" level=debug msg="PLAN STATUS HANDLER: plan=system-upgrade/server-plan@16986837, status={Conditions:[{Type:LatestResolved Status:True LastUpdateTime:2022-12-13T11:56:00Z LastTransitionTime: Reason:Version Message:}] LatestVersion:v1.24.8-rke2r1 LatestHash:3a1bea586e00acd8df7303f3d37b9b69d81bf3cb108a20527a981c87 Applying:[]}" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:56:00Z" level=debug msg="PLAN GENERATING HANDLER: plan=system-upgrade/server-plan@16986837, status={Conditions:[{Type:LatestResolved Status:True LastUpdateTime:2022-12-13T11:56:00Z LastTransitionTime: Reason:Version Message:}] LatestVersion:v1.24.8-rke2r1 LatestHash:3a1bea586e00acd8df7303f3d37b9b69d81bf3cb108a20527a981c87 Applying:[]}" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:56:00Z" level=debug msg="PLAN GENERATING HANDLER: plan=system-upgrade/agent-plan@16986839, status={Conditions:[{Type:LatestResolved Status:True LastUpdateTime:2022-12-13T11:56:00Z LastTransitionTime: Reason:Version Message:}] LatestVersion:v1.24.8-rke2r1 LatestHash:3a1bea586e00acd8df7303f3d37b9b69d81bf3cb108a20527a981c87 Applying:[]}" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:56:00Z" level=debug msg="PLAN STATUS HANDLER: plan=system-upgrade/agent-plan@16986839, status={Conditions:[{Type:LatestResolved Status:True LastUpdateTime:2022-12-13T11:56:00Z LastTransitionTime: Reason:Version Message:}] LatestVersion:v1.24.8-rke2r1 LatestHash:3a1bea586e00acd8df7303f3d37b9b69d81bf3cb108a20527a981c87 Applying:[]}" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"
time="2022-12-13T11:56:00Z" level=debug msg="PLAN GENERATING HANDLER: plan=system-upgrade/agent-plan@16986839, status={Conditions:[{Type:LatestResolved Status:True LastUpdateTime:2022-12-13T11:56:00Z LastTransitionTime: Reason:Version Message:}] LatestVersion:v1.24.8-rke2r1 LatestHash:3a1bea586e00acd8df7303f3d37b9b69d81bf3cb108a20527a981c87 Applying:[]}" func="github.com/sirupsen/logrus.(*Entry).Logf" file="/root/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314"

The controller reads the plan and seems to somehow accept it, but there is no job created and no update performed.

To Reproduce

  • Use cluster based on RKE2 stable channel
  • execute: kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/download/v0.10.0/system-upgrade-controller.yaml
  • wait for deployment to be finished
  • submit the following plan with kubectl apply -n system-upgrade -f update-plan.yaml
click to open update plan 
# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
  labels:
    rke2-upgrade: server
spec:
  concurrency: 1
  nodeSelector:
    matchExpressions:
       - {key: rke2-upgrade, operator: Exists}
       - {key: rke2-upgrade, operator: Not, values: ["disabled", "false"]}
       # When using k8s version 1.19 or older, swap control-plane with master
       - {key: node-role.kubernetes.io/control-plane, operator: In, values: ["true"]}
  serviceAccountName: system-upgrade
  cordon: true
#  drain:
#    force: true
  upgrade:
    image: rancher/rke2-upgrade
  version: v1.24.7-rke2r1
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan
  namespace: system-upgrade
  labels:
    rke2-upgrade: agent
spec:
  concurrency: 2
  nodeSelector:
    matchExpressions:
      - {key: rke2-upgrade, operator: Exists}
      - {key: rke2-upgrade, operator: Not, values: ["disabled", "false"]}
      # When using k8s version 1.19 or older, swap control-plane with master
      - {key: node-role.kubernetes.io/control-plane, operator: Not, values: ["true"]}
  prepare:
    args:
    - prepare
    - server-plan
    image: rancher/rke2-upgrade
  serviceAccountName: system-upgrade
  cordon: true
  drain:
    force: true
  upgrade:
    image: rancher/rke2-upgrade
  version: v1.24.8-rke2r1
- check submitted plan with `kubectl -n system-upgrade get plans -o yaml` (see output in the beginning of the issue) - check if jobs are created with `kubectl -n system-upgrade get jobs -o yaml`

Expected behavior A list containing at least one planned update job or updated RKE2 nodes to the version specified in the plan.

Actual behavior Plan seems to be accepted, but no jobs are being planned. No update to the desired K8s version.

Additional context We are behind a corporate proxy, but containerd and RKE2 have been configured accordingly. There is no indicator to a connection issue in the logs either.

kagehisa avatar Dec 13 '22 13:12 kagehisa

Ok there seems to be an issue with the upgrade-controller and our proxy environment. When I exchange the line

version: v1.24.8-rke2r1

with

channel: https://update.rke2.io/v1-release/channels/stable

in my update plan I get TCP I/O timeout messages when trying the rke2 channel url in the update-controller log. Why it doesn't happen when I specify a version directly is another topic, but I would assume our company proxy is not used by the update-controller.

Is there a way to tell the update-controller to use a proxy or if need be a local registry? I don't think I'm the only one with a corporate proxy setup.

kagehisa avatar Dec 20 '22 09:12 kagehisa

I have the exact same issue. I do not have a proxy. Simply following the docs and when I do kubectl -n system-upgrade get plans -o yaml I see that the controller picked up the stable release channel :

kubectl -n system-upgrade get plans -o yaml                
apiVersion: v1
items:
- apiVersion: upgrade.cattle.io/v1
  kind: Plan
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"upgrade.cattle.io/v1","kind":"Plan","metadata":{"annotations":{},"labels":{"rke2-upgrade":"agent"},"name":"agent-plan","namespace":"system-upgrade"},"spec":{"channel":"https://update.rke2.io/v1-release/channels/stable","concurrency":2,"cordon":true,"drain":{"force":true},"nodeSelector":{"matchExpressions":[{"key":"rke2-upgrade","operator":"Exists"},{"key":"rke2-upgrade","operator":"NotIn","values":["disabled","false"]},{"key":"node-role.kubernetes.io/control-plane","operator":"NotIn","values":["true"]}]},"prepare":{"args":["prepare","server-plan"],"image":"rancher/rke2-upgrade"},"serviceAccountName":"system-upgrade","upgrade":{"image":"rancher/rke2-upgrade"}}}
    creationTimestamp: "2023-01-02T07:54:53Z"
    generation: 1
    labels:
      rke2-upgrade: agent
    name: agent-plan
    namespace: system-upgrade
    resourceVersion: "27147316"
    uid: 7c7f4be3-ffb7-4c04-9b33-9552e009dc93
  spec:
    channel: https://update.rke2.io/v1-release/channels/stable
    concurrency: 2
    cordon: true
    drain:
      force: true
    nodeSelector:
      matchExpressions:
      - key: rke2-upgrade
        operator: Exists
      - key: rke2-upgrade
        operator: NotIn
        values:
        - disabled
        - "false"
      - key: node-role.kubernetes.io/control-plane
        operator: NotIn
        values:
        - "true"
    prepare:
      args:
      - prepare
      - server-plan
      image: rancher/rke2-upgrade
    serviceAccountName: system-upgrade
    upgrade:
      image: rancher/rke2-upgrade
  status:
    conditions:
    - lastUpdateTime: "2023-01-02T07:54:53Z"
      reason: Channel
      status: "True"
      type: LatestResolved
    latestHash: d150703718b5fed708cf45d28c50cf45d9f403b7fca3b579b34bb544
    latestVersion: v1.24.9-rke2r1
- apiVersion: upgrade.cattle.io/v1
  kind: Plan
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"upgrade.cattle.io/v1","kind":"Plan","metadata":{"annotations":{},"labels":{"rke2-upgrade":"server"},"name":"server-plan","namespace":"system-upgrade"},"spec":{"channel":"https://update.rke2.io/v1-release/channels/stable","concurrency":1,"cordon":true,"nodeSelector":{"matchExpressions":[{"key":"rke2-upgrade","operator":"Exists"},{"key":"rke2-upgrade","operator":"NotIn","values":["disabled","false"]},{"key":"node-role.kubernetes.io/control-plane","operator":"In","values":["true"]}]},"serviceAccountName":"system-upgrade","upgrade":{"image":"rancher/rke2-upgrade"}}}
    creationTimestamp: "2023-01-02T07:48:09Z"
    generation: 1
    labels:
      rke2-upgrade: server
    name: server-plan
    namespace: system-upgrade
    resourceVersion: "27145514"
    uid: bca68c1a-7e81-486a-8773-75ae40dd1e14
  spec:
    channel: https://update.rke2.io/v1-release/channels/stable
    concurrency: 1
    cordon: true
    nodeSelector:
      matchExpressions:
      - key: rke2-upgrade
        operator: Exists
      - key: rke2-upgrade
        operator: NotIn
        values:
        - disabled
        - "false"
      - key: node-role.kubernetes.io/control-plane
        operator: In
        values:
        - "true"
    serviceAccountName: system-upgrade
    upgrade:
      image: rancher/rke2-upgrade
  status:
    conditions:
    - lastUpdateTime: "2023-01-02T07:48:10Z"
      reason: Channel
      status: "True"
      type: LatestResolved
    latestHash: d150703718b5fed708cf45d28c50cf45d9f403b7fca3b579b34bb544
    latestVersion: v1.24.9-rke2r1

but no jobs :

kubectl -n system-upgrade get jobs -o yaml                 
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""

current version is

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7+rke2r1", GitCommit:"e6f35974b08862a23e7f4aad8e5d7f7f2de26c15", GitTreeState:"clean", BuildDate:"2022-10-13T18:58:02Z", GoVersion:"go1.18.7b7", Compiler:"gc", Platform:"linux/amd64"}

Maybe I am missing something ? Do I need to run an rke2 upgrade... command or something like that? I just followed the docs at https://docs.rke2.io/upgrade/automated_upgrade. I am running on a Single-Node cluster for testing

bernardgut avatar Jan 02 '23 08:01 bernardgut

I had this same problem as well. I noticed however that my nodes didn't have the rke2-upgrade labels mentioned in the plan's nodeSelector section. I removed the matchExpressions which mentioned them, then it started working for me. I'm not sure where that rke2-upgrade label is supposed to come form, but I'm sure you could add the label to your nodes manually also if desired.

devthane avatar Feb 02 '23 06:02 devthane

Thanks, that was the issue! Well for me part of it. If I use the version field it works with the correct labels. But since I am behind a corporate proxy I can not use the channel variant for an update since there seems to be no support for configuring a proxy for the update controller.

Nevertheless thanks a lot for your answer!

kagehisa avatar Feb 02 '23 12:02 kagehisa