kapp
kapp copied to clipboard
Published
20 hours ago •
carvel-dev
carvel-dev
Previous deployments' `status` is send and rejected by admission webhook
What steps did you take:
I tried to use kapp to upgrade a Tanzu Kubernetes Cluster object to use a new API version for a cluster: run.tanzu.vmware.com/v1alpha2.
Attached:
kubectl get tkcs/tanzu-work -o=yaml
apiVersion: run.tanzu.vmware.com/v1alpha2
kind: TanzuKubernetesCluster
metadata:
annotations:
kapp.k14s.io/identity: v1;ns-tanzu/run.tanzu.vmware.com/TanzuKubernetesCluster/tanzu-work;run.tanzu.vmware.com/v1alpha1
kapp.k14s.io/original: '{"apiVersion":"run.tanzu.vmware.com/v1alpha1","kind":"TanzuKubernetesCluster","metadata":{"annotations":{},"labels":{"kapp.k14s.io/app":"1648223162845653016","kapp.k14s.io/association":"v1.22ed984d1417c7e185b435c027150cc5","run.tanzu.vmware.com/tkr":"v1.21.6---vmware.1-tkg.1.b3d708a"},"name":"tanzu-work","namespace":"ns-tanzu"},"spec":{"distribution":{"fullVersion":"v1.21.6---vmware.1-tkg.1.b3d708a"},"settings":{"network":{"cni":{"name":"antrea"},"pods":{"cidrBlocks":["192.168.0.0/16"]},"serviceDomain":"cluster.local","services":{"cidrBlocks":["10.96.0.0/12"]}},"storage":{"defaultClass":"k8s-performance"}},"topology":{"controlPlane":{"class":"best-effort-small","count":1,"storageClass":"k8s-performance"},"workers":{"class":"best-effort-large","count":3,"storageClass":"k8s-performance"}}}}'
kapp.k14s.io/original-diff-md5: 22b91a69e85241fd7a17de8ea3878f13
creationTimestamp: "2022-04-07T20:36:51Z"
finalizers:
- tanzukubernetescluster.run.tanzu.vmware.com
generation: 6
labels:
kapp.k14s.io/app: "1648223162845653016"
kapp.k14s.io/association: v1.22ed984d1417c7e185b435c027150cc5
run.tanzu.vmware.com/tkr: v1.21.6---vmware.1-tkg.1.b3d708a
name: tanzu-work
namespace: ns-tanzu
resourceVersion: "40954920"
selfLink: /apis/run.tanzu.vmware.com/v1alpha2/namespaces/ns-tanzu/tanzukubernetesclusters/tanzu-work
uid: 80d96f3f-1602-4c34-a31e-b1c1deed7f56
spec:
distribution:
fullVersion: v1.21.6+vmware.1-tkg.1.b3d708a
version: ""
settings:
network:
cni:
name: antrea
pods:
cidrBlocks:
- 192.168.0.0/16
serviceDomain: cluster.local
services:
cidrBlocks:
- 10.96.0.0/12
storage:
defaultClass: k8s-performance
topology:
controlPlane:
replicas: 1
storageClass: k8s-performance
tkr:
reference:
name: v1.21.6---vmware.1-tkg.1.b3d708a
vmClass: best-effort-small
nodePools:
- name: workers
replicas: 3
storageClass: k8s-performance
tkr:
reference:
name: v1.21.6---vmware.1-tkg.1.b3d708a
vmClass: best-effort-large
status:
addons:
- conditions:
- lastTransitionTime: "2022-04-07T20:38:48Z"
status: "True"
type: Provisioned
name: CoreDNS
type: DNS
version: v1.8.0_vmware.9
- conditions:
- lastTransitionTime: "2022-04-07T20:38:49Z"
status: "True"
type: Provisioned
name: kube-proxy
type: Proxy
version: 1.21.6+vmware.1
- conditions:
- lastTransitionTime: "2022-04-07T20:38:33Z"
status: "True"
type: Provisioned
name: defaultpsp
type: PSP
version: v1.21.6+vmware.1-tkg.1.b3d708a
- conditions:
- lastTransitionTime: "2022-04-07T20:38:45Z"
status: "True"
type: Provisioned
name: antrea
type: CNI
version: v0.13.5_vmware.4
- conditions:
- lastTransitionTime: "2022-04-07T20:38:37Z"
status: "True"
type: Provisioned
name: pvcsi
type: CSI
version: v2.3.0-d154d1c
- conditions:
- lastTransitionTime: "2022-04-07T20:38:36Z"
status: "True"
type: Provisioned
name: vmware-guest-cluster
type: CPI
version: v1.21.0_vmware.1
- conditions:
- lastTransitionTime: "2022-04-07T20:38:38Z"
status: "True"
type: Provisioned
name: authsvc
type: AuthService
version: 0.1-71-g64e1c73
- conditions:
- lastTransitionTime: "2022-04-07T20:38:39Z"
status: "True"
type: Provisioned
name: metrics-server
type: MetricsServer
version: v0.4.0+vmware.1
apiEndpoints:
- host: 10.116.200.223
port: 6443
conditions:
- lastTransitionTime: "2022-05-05T14:37:14Z"
status: "True"
type: Ready
- lastTransitionTime: "2022-04-07T20:38:56Z"
status: "True"
type: AddonsReady
- lastTransitionTime: "2022-04-07T20:38:56Z"
status: "True"
type: ControlPlaneReady
- lastTransitionTime: "2022-05-05T14:37:14Z"
status: "True"
type: NodePoolsReady
- lastTransitionTime: "2022-05-05T14:39:48Z"
message: 1/1 Control Plane Node(s) healthy. 3/3 Worker Node(s) healthy
status: "True"
type: NodesHealthy
- lastTransitionTime: "2022-04-07T20:38:33Z"
status: "True"
type: ProviderServiceAccountsReady
- lastTransitionTime: "2022-04-07T20:38:33Z"
status: "True"
type: RoleBindingSynced
- lastTransitionTime: "2022-04-07T20:38:38Z"
status: "True"
type: ServiceDiscoveryReady
- lastTransitionTime: "2022-04-07T20:38:36Z"
status: "True"
type: StorageClassSynced
- lastTransitionTime: "2022-04-07T20:38:35Z"
status: "True"
type: TanzuKubernetesReleaseCompatible
- lastTransitionTime: "2022-03-18T20:05:07Z"
reason: NoUpdates
status: "False"
type: UpdatesAvailable
phase: running
totalWorkerReplicas: 3
The state of the cluster I want to reach
apiVersion: run.tanzu.vmware.com/v1alpha2
kind: TanzuKubernetesCluster
metadata:
name: tanzu-work
namespace: ns-tanzu
spec:
topology:
controlPlane:
replicas: 1
vmClass: best-effort-small
storageClass: k8s-performance
tkr:
reference:
name: v1.21.6---vmware.1-tkg.1.b3d708a
nodePools:
- name: primary
replicas: 3
vmClass: best-effort-large
storageClass: k8s-performance
volumes:
- name: containerd
mountPath: /var/lib/containerd/
capacity:
storage: 10Gi
tkr:
reference:
name: v1.21.6---vmware.1-tkg.1.b3d708a
settings:
network:
cni:
name: antrea
pods:
cidrBlocks:
- 192.168.0.0/16
serviceDomain: cluster.local
services:
cidrBlocks:
- 10.96.0.0/12
storage:
defaultClass: k8s-performance
kapp apply -a "" --diff-context=-1 --diff-changes --diff-run -f upgrade.yaml
Target cluster '<redacted>'
@@ update tanzukubernetescluster/tanzu-work (run.tanzu.vmware.com/v1alpha2) namespace: ns-tanzu @@
0 - apiVersion: run.tanzu.vmware.com/v1alpha1
0 + apiVersion: run.tanzu.vmware.com/v1alpha2
...
1, 1 kind: TanzuKubernetesCluster
2, 2 metadata:
3 - annotations:
4 - cluster.x-k8s.io/conversion-data: '{"apiVersion":"run.tanzu.vmware.com/v1alpha2","kind":"TanzuKubernetesCluster","spec":{"distribution":{"fullVersion":"v1.21.6+vmware.1-tkg.1.b3d708a","version":""},"settings":{"network":{"cni":{"name":"antrea"},"pods":{"cidrBlocks":["192.168.0.0/16"]},"serviceDomain":"cluster.local","services":{"cidrBlocks":["10.96.0.0/12"]}},"storage":{"defaultClass":"k8s-performance"}},"topology":{"controlPlane":{"replicas":1,"storageClass":"k8s-performance","tkr":{"reference":{"name":"v1.21.6---vmware.1-tkg.1.b3d708a"}},"vmClass":"best-effort-small"},"nodePools":[{"name":"workers","replicas":3,"storageClass":"k8s-performance","tkr":{"reference":{"name":"v1.21.6---vmware.1-tkg.1.b3d708a"}},"vmClass":"best-effort-large"}]}},"status":{"addons":[{"conditions":[{"lastTransitionTime":"2022-04-07T20:38:48Z","status":"True","type":"Provisioned"}],"name":"CoreDNS","type":"DNS","version":"v1.8.0_vmware.9"},{"conditions":[{"lastTransitionTime":"2022-04-07T20:38:49Z","status":"True","type":"Provisioned"}],"name":"kube-proxy","type":"Proxy","version":"1.21.6+vmware.1"},{"conditions":[{"lastTransitionTime":"2022-04-07T20:38:33Z","status":"True","type":"Provisioned"}],"name":"defaultpsp","type":"PSP","version":"v1.21.6+vmware.1-tkg.1.b3d708a"},{"conditions":[{"lastTransitionTime":"2022-04-07T20:38:45Z","status":"True","type":"Provisioned"}],"name":"antrea","type":"CNI","version":"v0.13.5_vmware.4"},{"conditions":[{"lastTransitionTime":"2022-04-07T20:38:37Z","status":"True","type":"Provisioned"}],"name":"pvcsi","type":"CSI","version":"v2.3.0-d154d1c"},{"conditions":[{"lastTransitionTime":"2022-04-07T20:38:36Z","status":"True","type":"Provisioned"}],"name":"vmware-guest-cluster","type":"CPI","version":"v1.21.0_vmware.1"},{"conditions":[{"lastTransitionTime":"2022-04-07T20:38:38Z","status":"True","type":"Provisioned"}],"name":"authsvc","type":"AuthService","version":"0.1-71-g64e1c73"},{"conditions":[{"lastTransitionTime":"2022-04-07T20:38:39Z","status":"True","type":"Provisioned"}],"name":"metrics-server","type":"MetricsServer","version":"v0.4.0+vmware.1"}],"apiEndpoints":[{"host":"10.116.200.223","port":6443}],"conditions":[{"lastTransitionTime":"2022-05-05T14:37:14Z","status":"True","type":"Ready"},{"lastTransitionTime":"2022-04-07T20:38:56Z","status":"True","type":"AddonsReady"},{"lastTransitionTime":"2022-04-07T20:38:56Z","status":"True","type":"ControlPlaneReady"},{"lastTransitionTime":"2022-05-05T14:37:14Z","status":"True","type":"NodePoolsReady"},{"lastTransitionTime":"2022-05-05T14:39:48Z","message":"1/1
5 - Control Plane Node(s) healthy. 3/3 Worker Node(s) healthy","status":"True","type":"NodesHealthy"},{"lastTransitionTime":"2022-04-07T20:38:33Z","status":"True","type":"ProviderServiceAccountsReady"},{"lastTransitionTime":"2022-04-07T20:38:33Z","status":"True","type":"RoleBindingSynced"},{"lastTransitionTime":"2022-04-07T20:38:38Z","status":"True","type":"ServiceDiscoveryReady"},{"lastTransitionTime":"2022-04-07T20:38:36Z","status":"True","type":"StorageClassSynced"},{"lastTransitionTime":"2022-04-07T20:38:35Z","status":"True","type":"TanzuKubernetesReleaseCompatible"},{"lastTransitionTime":"2022-03-18T20:05:07Z","reason":"NoUpdates","status":"False","type":"UpdatesAvailable"}],"phase":"running","totalWorkerReplicas":3}}'
6, 3 creationTimestamp: "2022-04-07T20:36:51Z"
7, 4 finalizers:
8, 5 - tanzukubernetescluster.run.tanzu.vmware.com
9, 6 generation: 6
10, 7 labels:
11, 8 kapp.k14s.io/app: "1648223162845653016"
12, 9 kapp.k14s.io/association: v1.22ed984d1417c7e185b435c027150cc5
13 - run.tanzu.vmware.com/tkr: v1.21.6---vmware.1-tkg.1.b3d708a
14, 10 managedFields:
15, 11 - apiVersion: run.tanzu.vmware.com/v1alpha1
16, 12 fieldsType: FieldsV1
17, 13 fieldsV1:
18, 14 f:metadata:
19, 15 f:annotations:
20, 16 .: {}
21, 17 f:kapp.k14s.io/identity: {}
22, 18 f:kapp.k14s.io/original: {}
23, 19 f:kapp.k14s.io/original-diff-md5: {}
24, 20 f:labels:
25, 21 .: {}
26, 22 f:kapp.k14s.io/app: {}
27, 23 f:kapp.k14s.io/association: {}
28, 24 f:run.tanzu.vmware.com/tkr: {}
29, 25 f:spec:
30, 26 .: {}
31, 27 f:distribution:
32, 28 .: {}
33, 29 f:fullVersion: {}
34, 30 f:settings:
35, 31 .: {}
36, 32 f:network:
37, 33 .: {}
38, 34 f:cni:
39, 35 .: {}
40, 36 f:name: {}
41, 37 f:pods:
42, 38 .: {}
43, 39 f:cidrBlocks: {}
44, 40 f:serviceDomain: {}
45, 41 f:services:
46, 42 .: {}
47, 43 f:cidrBlocks: {}
48, 44 f:storage:
49, 45 .: {}
50, 46 f:defaultClass: {}
51, 47 f:topology:
52, 48 .: {}
53, 49 f:controlPlane:
54, 50 .: {}
55, 51 f:class: {}
56, 52 f:count: {}
57, 53 f:storageClass: {}
58, 54 f:workers:
59, 55 .: {}
60, 56 f:class: {}
61, 57 f:count: {}
62, 58 f:storageClass: {}
63, 59 manager: kapp
64, 60 operation: Update
65, 61 time: "2022-05-05T14:35:22Z"
66, 62 - apiVersion: run.tanzu.vmware.com/v1alpha2
67, 63 fieldsType: FieldsV1
68, 64 fieldsV1:
69, 65 f:metadata:
70, 66 f:finalizers:
71, 67 .: {}
72, 68 v:"tanzukubernetescluster.run.tanzu.vmware.com": {}
73, 69 f:status:
74, 70 f:apiEndpoints: {}
75, 71 f:conditions: {}
76, 72 f:phase: {}
77, 73 f:totalWorkerReplicas: {}
78, 74 manager: manager
79, 75 operation: Update
80, 76 time: "2022-05-05T14:35:23Z"
81, 77 name: tanzu-work
82, 78 namespace: ns-tanzu
83, 79 resourceVersion: "40954920"
84, 80 selfLink: /apis/run.tanzu.vmware.com/v1alpha1/namespaces/ns-tanzu/tanzukubernetesclusters/tanzu-work
85, 81 uid: 80d96f3f-1602-4c34-a31e-b1c1deed7f56
86, 82 spec:
87 - distribution:
88 - fullVersion: 1.21.6+vmware.1-tkg.1.b3d708a
89 - version: ""
90, 83 settings:
91, 84 network:
92, 85 cni:
93, 86 name: antrea
94, 87 pods:
95, 88 cidrBlocks:
96, 89 - 192.168.0.0/16
97, 90 serviceDomain: cluster.local
98, 91 services:
99, 92 cidrBlocks:
100, 93 - 10.96.0.0/12
101, 94 storage:
102, 95 defaultClass: k8s-performance
103, 96 topology:
104, 97 controlPlane:
105 - class: best-effort-small
106 - count: 1
98 + replicas: 1
107, 99 storageClass: k8s-performance
108 - workers:
109 - class: best-effort-large
110 - count: 3
100 + tkr:
101 + reference:
102 + name: v1.21.6---vmware.1-tkg.1.b3d708a
103 + vmClass: best-effort-small
104 + nodePools:
105 + - name: primary
106 + replicas: 3
111,107 storageClass: k8s-performance
108 + tkr:
109 + reference:
110 + name: v1.21.6---vmware.1-tkg.1.b3d708a
111 + vmClass: best-effort-large
112 + volumes:
113 + - capacity:
114 + storage: 10Gi
115 + mountPath: /var/lib/containerd/
116 + name: containerd
112,117 status:
113,118 addons:
114,119 authsvc:
115,120 conditions:
116,121 - lastTransitionTime: "2022-04-07T20:38:38Z"
117,122 status: "True"
118,123 type: AuthServiceProvisioned
119,124 name: authsvc
120,125 status: ""
121,126 version: 0.1-71-g64e1c73
122,127 cloudprovider:
123,128 conditions:
124,129 - lastTransitionTime: "2022-04-07T20:38:36Z"
125,130 status: "True"
126,131 type: CPIProvisioned
127,132 name: vmware-guest-cluster
128,133 status: ""
129,134 version: v1.21.0_vmware.1
130,135 cni:
131,136 conditions:
132,137 - lastTransitionTime: "2022-04-07T20:38:45Z"
133,138 status: "True"
134,139 type: CNIProvisioned
135,140 name: antrea
136,141 status: ""
137,142 version: v0.13.5_vmware.4
138,143 csi:
139,144 conditions:
140,145 - lastTransitionTime: "2022-04-07T20:38:37Z"
141,146 status: "True"
142,147 type: CSIProvisioned
143,148 name: pvcsi
144,149 status: ""
145,150 version: v2.3.0-d154d1c
146,151 dns:
147,152 conditions:
148,153 - lastTransitionTime: "2022-04-07T20:38:48Z"
149,154 status: "True"
150,155 type: CoreDNSProvisioned
151,156 name: CoreDNS
152,157 status: ""
153,158 version: v1.8.0_vmware.9
154,159 metrics-server:
155,160 conditions:
156,161 - lastTransitionTime: "2022-04-07T20:38:39Z"
157,162 status: "True"
158,163 type: MetricsServerProvisioned
159,164 name: metrics-server
160,165 status: ""
161,166 version: v0.4.0+vmware.1
162,167 proxy:
163,168 conditions:
164,169 - lastTransitionTime: "2022-04-07T20:38:49Z"
165,170 status: "True"
166,171 type: KubeProxyProvisioned
167,172 name: kube-proxy
168,173 status: ""
169,174 version: 1.21.6+vmware.1
170,175 psp:
171,176 conditions:
172,177 - lastTransitionTime: "2022-04-07T20:38:33Z"
173,178 status: "True"
174,179 type: PSPProvisioned
175,180 name: defaultpsp
176,181 status: ""
177,182 version: v1.21.6+vmware.1-tkg.1.b3d708a
178,183 clusterApiStatus:
179,184 apiEndpoints:
180,185 - host: 10.116.200.223
181,186 port: 6443
182,187 conditions:
183,188 - lastTransitionTime: "2022-05-05T14:37:14Z"
184,189 status: "True"
185,190 type: Ready
186,191 - lastTransitionTime: "2022-04-07T20:38:56Z"
187,192 status: "True"
188,193 type: AddonsReady
189,194 - lastTransitionTime: "2022-04-07T20:38:56Z"
190,195 status: "True"
191,196 type: ControlPlaneReady
192,197 - lastTransitionTime: "2022-05-05T14:37:14Z"
193,198 status: "True"
194,199 type: NodePoolsReady
195,200 - lastTransitionTime: "2022-05-05T14:39:48Z"
196,201 message: 1/1 Control Plane Node(s) healthy. 3/3 Worker Node(s) healthy
197,202 status: "True"
198,203 type: NodesHealthy
199,204 - lastTransitionTime: "2022-04-07T20:38:33Z"
200,205 status: "True"
201,206 type: ProviderServiceAccountsReady
202,207 - lastTransitionTime: "2022-04-07T20:38:33Z"
203,208 status: "True"
204,209 type: RoleBindingSynced
205,210 - lastTransitionTime: "2022-04-07T20:38:38Z"
206,211 status: "True"
207,212 type: ServiceDiscoveryReady
208,213 - lastTransitionTime: "2022-04-07T20:38:36Z"
209,214 status: "True"
210,215 type: StorageClassSynced
211,216 - lastTransitionTime: "2022-04-07T20:38:35Z"
212,217 status: "True"
213,218 type: TanzuKubernetesReleaseCompatible
214,219 - lastTransitionTime: "2022-03-18T20:05:07Z"
215,220 reason: NoUpdates
216,221 status: "False"
217,222 type: UpdatesAvailable
218,223 phase: running
219,224
Changes
Namespace Name Kind Age Op Op st. Wait to Rs Ri
ns-tanzu tanzu-work TanzuKubernetesCluster 28d update - reconcile ok -
Op: 0 create, 0 delete, 1 update, 0 noop, 0 exists
Wait to: 1 reconcile, 0 delete, 0 noop
Succeeded
What happened:
kapp tried to apply the update, but this got rejected by the admission webhook and threw the following error:
kapp: Error: Applying update tanzukubernetescluster/tanzu-work ([run.tanzu.vmware.com/v1alpha2](http://run.tanzu.vmware.com/v1alpha2)) namespace: ns-tanzu:
Updating resource tanzukubernetescluster/tanzu-work ([run.tanzu.vmware.com/v1alpha2](http://run.tanzu.vmware.com/v1alpha2)) namespace: ns-tanzu:
API server says: admission webhook "[default.mutating.tanzukubernetescluster.run.tanzu.vmware.com](http://default.mutating.tanzukubernetescluster.run.tanzu.vmware.com/)" denied the request: v1alpha2.TanzuKubernetesCluster.Status: v1alpha2.TanzuKubernetesClusterStatus.Addons: []v1alpha2.AddonStatus: decode slice: expect [ or n, but found {, error found in #10 byte of ...|"addons":{},"conditi|..., bigger context ...|d/","name":"containerd"}]}]}},"status":{"addons":{},"conditions":[{"lastTransitionTime":"2022-05-05T|... (reason: )
What did you expect:
The deployment to go through, since I don't define nor manage the status anywhere myself, nor did I ever manipulate it.
Anything else you would like to add:
This new version turns out to have changed it's status' structure (see extra additions) but kapp preserves the one from the previous API version, causing the webhook to deny it.
Did my best to trace this in the source:
Definition of v1alpha1 status Definition of v1alpha2 status.
I would never have found this if it wasn't for the help I received on Slack: https://kubernetes.slack.com/archives/CH8KCCKA5/p1651838286810069
Environment:
- kapp version (use
kapp --version): 0.46 - OS (e.g. from
/etc/os-release): Ubuntu 20 - WSL2 - Kubernetes version (use
kubectl version): 1.21
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
Thank you so much for creating the issue @daneov. We will move this to our prioritised backlog.
Acceptance criteria:
After going through all the slack discussion found that copying status from existing to new resource(current default behaviour of kapp) is not needed. Hence as a solution for this issue the three things going to do are:
- Add functionality to apply
removerebase rule to existing resource. Currently kapp does not have support of this. - Add
removerebase rule to removestatusfrom both existing and new resource. This will be the default behaviour of kapp. - Add an annotation to disable the default behaviour (mentioned in 2nd point) so that if user want to add some rebase rule to
statusthey can and disable the default behaviour.
