kapp
kapp copied to clipboard
carvel-dev
kapp deploy mutates created/updated resources twice causing conflicts
What steps did you take:
Create a deployment yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 0
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
In another terminal watch deployments
kubectl get deployment -w -oyaml
What happened:
After the deployment is applied kapp immediately updates it - you can see generation: 2.
Looking at my k8s service API logs it causes Conflicts on creating and updating resources
This is causing to fail deploying at times
kapp: Error: Applying update deployment/activator (apps/v1) namespace: 24f0bc17-06dc-424b-a9db-8d4e744d7790:
Failed to update due to resource conflict (approved diff no longer matches):
Updating resource deployment/activator (apps/v1) namespace: 24f0bc17-06dc-424b-a9db-8d4e744d7790:
API server says:
Operation cannot be fulfilled on deployments.apps "activator": the object has been modified; please apply your changes to the latest version and try again (reason: Conflict)
What did you expect:
kapp doesn't mutate resources after they are created/updated
Anything else you would like to add: [Additional information that will assist in solving the issue.]
Environment:
- kapp version (use
kapp --version): v0.46.0 - OS (e.g. from
/etc/os-release): Mac OS X/Linux - Kubernetes version (use
kubectl version) - K8s 1.21/1.22
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
I think the additional update is the one happening to add/update the kapp.k14s.io/original annotation which can be disabled by using the kapp.k14s.io/disable-original annotation.
@dprotaso Would you be able to share some details on the specific scenario that leads to the conflict?
Relavant slack discussion: https://kubernetes.slack.com/archives/CH8KCCKA5/p1649116160660209
Background
From slack thread
Originally I was encountering the conflict because my deployment has an HPA adjusting it's spec.replicas
kapp: Error: Applying update deployment/activator (apps/v1) namespace: 24f0bc17-06dc-424b-a9db-8d4e744d7790:
Failed to update due to resource conflict (approved diff no longer matches):
Updating resource deployment/activator (apps/v1) namespace: 24f0bc17-06dc-424b-a9db-8d4e744d7790:
API server says:
Operation cannot be fulfilled on deployments.apps "activator": the object has been modified; please apply your changes to the latest version and try again (reason: Conflict)
But I wanted kapp to ignore the property so I created a rebase rule where I remove it.
apiVersion: kapp.k14s.io/v1alpha1
kind: Config
rebaseRules:
- path: [spec, replicas]
type: remove
resourceMatchers:
- kindNamespaceNameMatcher:
kind: Deployment
name: activator
namespace: knative-serving
This rebaseRule still leads to the above conflict - (copy:[existing,new] does work as expected).
From the slack thread I understand why kapp does the extra update and I understand that external modification will result in kapp to error out and reporting a conflict.
Issue
But I'm seeing the first attempt to update that annotation is always a conflict (for which kapp retries). I'm seeing this for deployments that don't have an HPA or some external entity making modifications to it.
Here are the audit entries:
Create Audit Log
{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "[email protected]"
},
"authorizationInfo": [
{
"granted": true,
"permission": "io.k8s.apps.v1.deployments.create",
"resource": "apps/v1/namespaces/0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/deployments/controller"
}
],
"methodName": "io.k8s.apps.v1.deployments.create",
"request": {
"@type": "apps.k8s.io/v1.Deployment",
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"annotations": {
"kapp.k14s.io/identity": "v1;0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/apps/Deployment/controller;apps/v1"
},
"creationTimestamp": null,
"labels": {
"app.kubernetes.io/component": "controller",
"app.kubernetes.io/name": "knative-serving",
"app.kubernetes.io/version": "1.3.0",
"kapp.k14s.io/app": "1649125589828112425",
"kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
"serving.knative.dev/release": "v1.3.0"
},
"name": "controller",
"namespace": "0c19d0cf-3301-40b9-ba33-c8718b8eaaa8"
},
"spec": {
"progressDeadlineSeconds": 600,
"replicas": 1,
"revisionHistoryLimit": 10,
"selector": {
"matchLabels": {
"app": "controller",
"kapp.k14s.io/app": "1649125589828112425"
}
},
"strategy": {
"rollingUpdate": {
"maxSurge": "25%",
"maxUnavailable": "25%"
},
"type": "RollingUpdate"
},
"template": {
"metadata": {
"annotations": {
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
},
"creationTimestamp": null,
"labels": {
"app": "controller",
"app.kubernetes.io/component": "controller",
"app.kubernetes.io/name": "knative-serving",
"app.kubernetes.io/version": "1.3.0",
"kapp.k14s.io/app": "1649125589828112425",
"kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
"serving.knative.dev/release": "v1.3.0"
}
},
"spec": {
"affinity": {
"podAntiAffinity": {
"preferredDuringSchedulingIgnoredDuringExecution": [
{
"podAffinityTerm": {
"labelSelector": {
"matchLabels": {
"app": "controller"
}
},
"topologyKey": "kubernetes.io/hostname"
},
"weight": 100
}
]
}
},
"containers": [
{
"env": [
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "SYSTEM_NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "CONFIG_LOGGING_NAME",
"value": "config-logging"
},
{
"name": "CONFIG_OBSERVABILITY_NAME",
"value": "config-observability"
},
{
"name": "METRICS_DOMAIN",
"value": "knative.dev/internal/serving"
}
],
"image": "gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:712ac88a1ec328ad366a0245882061de01bdf631cb21bac2b621af8e20e7082d",
"imagePullPolicy": "IfNotPresent",
"name": "controller",
"ports": [
{
"containerPort": 9090,
"name": "metrics",
"protocol": "TCP"
},
{
"containerPort": 8008,
"name": "profiling",
"protocol": "TCP"
}
],
"resources": {
"limits": {
"cpu": "1",
"memory": "1000Mi"
},
"requests": {
"cpu": "100m",
"memory": "100Mi"
}
},
"securityContext": {
"allowPrivilegeEscalation": false,
"capabilities": {
"drop": [
"all"
]
},
"readOnlyRootFilesystem": true,
"runAsNonRoot": true
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File"
}
],
"dnsPolicy": "ClusterFirst",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"serviceAccount": "controller",
"serviceAccountName": "controller",
"terminationGracePeriodSeconds": 30
}
}
},
"status": {}
},
"requestMetadata": {
"callerIp": "35.232.127.123",
"callerSuppliedUserAgent": "kapp/v0.0.0 (linux/amd64) kubernetes/$Format"
},
"resourceName": "apps/v1/namespaces/0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/deployments/controller",
"response": {
"@type": "apps.k8s.io/v1.Deployment",
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"annotations": {
"kapp.k14s.io/identity": "v1;0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/apps/Deployment/controller;apps/v1"
},
"creationTimestamp": "2022-04-05T02:27:23Z",
"generation": 1,
"labels": {
"app.kubernetes.io/component": "controller",
"app.kubernetes.io/name": "knative-serving",
"app.kubernetes.io/version": "1.3.0",
"kapp.k14s.io/app": "1649125589828112425",
"kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
"serving.knative.dev/release": "v1.3.0"
},
"managedFields": [
{
"apiVersion": "apps/v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:kapp.k14s.io/identity": {}
},
"f:labels": {
".": {},
"f:app.kubernetes.io/component": {},
"f:app.kubernetes.io/name": {},
"f:app.kubernetes.io/version": {},
"f:kapp.k14s.io/app": {},
"f:kapp.k14s.io/association": {},
"f:serving.knative.dev/release": {}
}
},
"f:spec": {
"f:progressDeadlineSeconds": {},
"f:replicas": {},
"f:revisionHistoryLimit": {},
"f:selector": {},
"f:strategy": {
"f:rollingUpdate": {
".": {},
"f:maxSurge": {},
"f:maxUnavailable": {}
},
"f:type": {}
},
"f:template": {
"f:metadata": {
"f:annotations": {
".": {},
"f:cluster-autoscaler.kubernetes.io/safe-to-evict": {}
},
"f:labels": {
".": {},
"f:app": {},
"f:app.kubernetes.io/component": {},
"f:app.kubernetes.io/name": {},
"f:app.kubernetes.io/version": {},
"f:kapp.k14s.io/app": {},
"f:kapp.k14s.io/association": {},
"f:serving.knative.dev/release": {}
}
},
"f:spec": {
"f:affinity": {
".": {},
"f:podAntiAffinity": {
".": {},
"f:preferredDuringSchedulingIgnoredDuringExecution": {}
}
},
"f:containers": {
"k:{\"name\":\"controller\"}": {
".": {},
"f:env": {
".": {},
"k:{\"name\":\"CONFIG_LOGGING_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"METRICS_DOMAIN\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"SYSTEM_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:imagePullPolicy": {},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:resources": {
".": {},
"f:limits": {
".": {},
"f:cpu": {},
"f:memory": {}
},
"f:requests": {
".": {},
"f:cpu": {},
"f:memory": {}
}
},
"f:securityContext": {
".": {},
"f:allowPrivilegeEscalation": {},
"f:capabilities": {
".": {},
"f:drop": {}
},
"f:readOnlyRootFilesystem": {},
"f:runAsNonRoot": {}
},
"f:terminationMessagePath": {},
"f:terminationMessagePolicy": {}
}
},
"f:dnsPolicy": {},
"f:restartPolicy": {},
"f:schedulerName": {},
"f:securityContext": {},
"f:serviceAccount": {},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {}
}
}
}
},
"manager": "kapp",
"operation": "Update",
"time": "2022-04-05T02:27:23Z"
}
],
"name": "controller",
"namespace": "0c19d0cf-3301-40b9-ba33-c8718b8eaaa8",
"resourceVersion": "23836",
"uid": "fee6f8bc-26df-4204-b5e9-52e8b67e88a9"
},
"spec": {
"progressDeadlineSeconds": 600,
"replicas": 1,
"revisionHistoryLimit": 10,
"selector": {
"matchLabels": {
"app": "controller",
"kapp.k14s.io/app": "1649125589828112425"
}
},
"strategy": {
"rollingUpdate": {
"maxSurge": "25%",
"maxUnavailable": "25%"
},
"type": "RollingUpdate"
},
"template": {
"metadata": {
"annotations": {
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
},
"creationTimestamp": null,
"labels": {
"app": "controller",
"app.kubernetes.io/component": "controller",
"app.kubernetes.io/name": "knative-serving",
"app.kubernetes.io/version": "1.3.0",
"kapp.k14s.io/app": "1649125589828112425",
"kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
"serving.knative.dev/release": "v1.3.0"
}
},
"spec": {
"affinity": {
"podAntiAffinity": {
"preferredDuringSchedulingIgnoredDuringExecution": [
{
"podAffinityTerm": {
"labelSelector": {
"matchLabels": {
"app": "controller"
}
},
"topologyKey": "kubernetes.io/hostname"
},
"weight": 100
}
]
}
},
"containers": [
{
"env": [
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "SYSTEM_NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "CONFIG_LOGGING_NAME",
"value": "config-logging"
},
{
"name": "CONFIG_OBSERVABILITY_NAME",
"value": "config-observability"
},
{
"name": "METRICS_DOMAIN",
"value": "knative.dev/internal/serving"
}
],
"image": "gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:712ac88a1ec328ad366a0245882061de01bdf631cb21bac2b621af8e20e7082d",
"imagePullPolicy": "IfNotPresent",
"name": "controller",
"ports": [
{
"containerPort": 9090,
"name": "metrics",
"protocol": "TCP"
},
{
"containerPort": 8008,
"name": "profiling",
"protocol": "TCP"
}
],
"resources": {
"limits": {
"cpu": "1",
"memory": "1000Mi"
},
"requests": {
"cpu": "100m",
"memory": "100Mi"
}
},
"securityContext": {
"allowPrivilegeEscalation": false,
"capabilities": {
"drop": [
"all"
]
},
"readOnlyRootFilesystem": true,
"runAsNonRoot": true
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File"
}
],
"dnsPolicy": "ClusterFirst",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"serviceAccount": "controller",
"serviceAccountName": "controller",
"terminationGracePeriodSeconds": 30
}
}
},
"status": {}
},
"serviceName": "k8s.io",
"status": {
"code": 0
}
},
"insertId": "0c8c2851-5686-45bb-8129-41e454115b2a",
"resource": {
"type": "k8s_cluster",
"labels": {
"cluster_name": "e2e-cls-pnwjdyda",
"project_id": "knative-boskos-85",
"location": "us-east1"
}
},
"timestamp": "2022-04-05T02:27:23.680396Z",
"labels": {
"authorization.k8s.io/reason": "access granted by IAM permissions.",
"authorization.k8s.io/decision": "allow"
},
"logName": "projects/knative-boskos-85/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "0c8c2851-5686-45bb-8129-41e454115b2a",
"producer": "k8s.io",
"first": true,
"last": true
},
"receiveTimestamp": "2022-04-05T02:27:24.036307301Z"
}
Update Conflict
{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "[email protected]"
},
"authorizationInfo": [
{
"granted": true,
"permission": "io.k8s.apps.v1.deployments.update",
"resource": "apps/v1/namespaces/0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/deployments/controller"
}
],
"methodName": "io.k8s.apps.v1.deployments.update",
"request": {
"@type": "apps.k8s.io/v1.Deployment",
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"annotations": {
"kapp.k14s.io/identity": "v1;0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/apps/Deployment/controller;apps/v1",
"kapp.k14s.io/original": "{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"controller\",\"app.kubernetes.io/name\":\"knative-serving\",\"app.kubernetes.io/version\":\"1.3.0\",\"kapp.k14s.io/app\":\"1649125589828112425\",\"kapp.k14s.io/association\":\"v1.6d64db857f7da29e4c8063c4a89bd73e\",\"serving.knative.dev/release\":\"v1.3.0\"},\"name\":\"controller\",\"namespace\":\"0c19d0cf-3301-40b9-ba33-c8718b8eaaa8\"},\"spec\":{\"selector\":{\"matchLabels\":{\"app\":\"controller\",\"kapp.k14s.io/app\":\"1649125589828112425\"}},\"template\":{\"metadata\":{\"annotations\":{\"cluster-autoscaler.kubernetes.io/safe-to-evict\":\"true\"},\"labels\":{\"app\":\"controller\",\"app.kubernetes.io/component\":\"controller\",\"app.kubernetes.io/name\":\"knative-serving\",\"app.kubernetes.io/version\":\"1.3.0\",\"kapp.k14s.io/app\":\"1649125589828112425\",\"kapp.k14s.io/association\":\"v1.6d64db857f7da29e4c8063c4a89bd73e\",\"serving.knative.dev/release\":\"v1.3.0\"}},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchLabels\":{\"app\":\"controller\"}},\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"containers\":[{\"env\":[{\"name\":\"POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"SYSTEM_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"CONFIG_LOGGING_NAME\",\"value\":\"config-logging\"},{\"name\":\"CONFIG_OBSERVABILITY_NAME\",\"value\":\"config-observability\"},{\"name\":\"METRICS_DOMAIN\",\"value\":\"knative.dev/internal/serving\"}],\"image\":\"gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:712ac88a1ec328ad366a0245882061de01bdf631cb21bac2b621af8e20e7082d\",\"name\":\"controller\",\"ports\":[{\"containerPort\":9090,\"name\":\"metrics\"},{\"containerPort\":8008,\"name\":\"profiling\"}],\"resources\":{\"limits\":{\"cpu\":\"1000m\",\"memory\":\"1000Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"100Mi\"}},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"capabilities\":{\"drop\":[\"all\"]},\"readOnlyRootFilesystem\":true,\"runAsNonRoot\":true}}],\"serviceAccountName\":\"controller\"}}}}",
"kapp.k14s.io/original-diff-md5": "779caecc17d30b78a99080e6330527c2"
},
"creationTimestamp": "2022-04-05T02:27:23Z",
"generation": 1,
"labels": {
"app.kubernetes.io/component": "controller",
"app.kubernetes.io/name": "knative-serving",
"app.kubernetes.io/version": "1.3.0",
"kapp.k14s.io/app": "1649125589828112425",
"kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
"serving.knative.dev/release": "v1.3.0"
},
"managedFields": [
{
"apiVersion": "apps/v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:kapp.k14s.io/identity": {}
},
"f:labels": {
".": {},
"f:app.kubernetes.io/component": {},
"f:app.kubernetes.io/name": {},
"f:app.kubernetes.io/version": {},
"f:kapp.k14s.io/app": {},
"f:kapp.k14s.io/association": {},
"f:serving.knative.dev/release": {}
}
},
"f:spec": {
"f:progressDeadlineSeconds": {},
"f:replicas": {},
"f:revisionHistoryLimit": {},
"f:selector": {},
"f:strategy": {
"f:rollingUpdate": {
".": {},
"f:maxSurge": {},
"f:maxUnavailable": {}
},
"f:type": {}
},
"f:template": {
"f:metadata": {
"f:annotations": {
".": {},
"f:cluster-autoscaler.kubernetes.io/safe-to-evict": {}
},
"f:labels": {
".": {},
"f:app": {},
"f:app.kubernetes.io/component": {},
"f:app.kubernetes.io/name": {},
"f:app.kubernetes.io/version": {},
"f:kapp.k14s.io/app": {},
"f:kapp.k14s.io/association": {},
"f:serving.knative.dev/release": {}
}
},
"f:spec": {
"f:affinity": {
".": {},
"f:podAntiAffinity": {
".": {},
"f:preferredDuringSchedulingIgnoredDuringExecution": {}
}
},
"f:containers": {
"k:{\"name\":\"controller\"}": {
".": {},
"f:env": {
".": {},
"k:{\"name\":\"CONFIG_LOGGING_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"METRICS_DOMAIN\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"SYSTEM_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:imagePullPolicy": {},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:resources": {
".": {},
"f:limits": {
".": {},
"f:cpu": {},
"f:memory": {}
},
"f:requests": {
".": {},
"f:cpu": {},
"f:memory": {}
}
},
"f:securityContext": {
".": {},
"f:allowPrivilegeEscalation": {},
"f:capabilities": {
".": {},
"f:drop": {}
},
"f:readOnlyRootFilesystem": {},
"f:runAsNonRoot": {}
},
"f:terminationMessagePath": {},
"f:terminationMessagePolicy": {}
}
},
"f:dnsPolicy": {},
"f:restartPolicy": {},
"f:schedulerName": {},
"f:securityContext": {},
"f:serviceAccount": {},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {}
}
}
}
},
"manager": "kapp",
"operation": "Update",
"time": "2022-04-05T02:27:23Z"
}
],
"name": "controller",
"namespace": "0c19d0cf-3301-40b9-ba33-c8718b8eaaa8",
"resourceVersion": "23836",
"uid": "fee6f8bc-26df-4204-b5e9-52e8b67e88a9"
},
"spec": {
"progressDeadlineSeconds": 600,
"replicas": 1,
"revisionHistoryLimit": 10,
"selector": {
"matchLabels": {
"app": "controller",
"kapp.k14s.io/app": "1649125589828112425"
}
},
"strategy": {
"rollingUpdate": {
"maxSurge": "25%",
"maxUnavailable": "25%"
},
"type": "RollingUpdate"
},
"template": {
"metadata": {
"annotations": {
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
},
"creationTimestamp": null,
"labels": {
"app": "controller",
"app.kubernetes.io/component": "controller",
"app.kubernetes.io/name": "knative-serving",
"app.kubernetes.io/version": "1.3.0",
"kapp.k14s.io/app": "1649125589828112425",
"kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
"serving.knative.dev/release": "v1.3.0"
}
},
"spec": {
"affinity": {
"podAntiAffinity": {
"preferredDuringSchedulingIgnoredDuringExecution": [
{
"podAffinityTerm": {
"labelSelector": {
"matchLabels": {
"app": "controller"
}
},
"topologyKey": "kubernetes.io/hostname"
},
"weight": 100
}
]
}
},
"containers": [
{
"env": [
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "SYSTEM_NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "CONFIG_LOGGING_NAME",
"value": "config-logging"
},
{
"name": "CONFIG_OBSERVABILITY_NAME",
"value": "config-observability"
},
{
"name": "METRICS_DOMAIN",
"value": "knative.dev/internal/serving"
}
],
"image": "gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:712ac88a1ec328ad366a0245882061de01bdf631cb21bac2b621af8e20e7082d",
"imagePullPolicy": "IfNotPresent",
"name": "controller",
"ports": [
{
"containerPort": 9090,
"name": "metrics",
"protocol": "TCP"
},
{
"containerPort": 8008,
"name": "profiling",
"protocol": "TCP"
}
],
"resources": {
"limits": {
"cpu": "1",
"memory": "1000Mi"
},
"requests": {
"cpu": "100m",
"memory": "100Mi"
}
},
"securityContext": {
"allowPrivilegeEscalation": false,
"capabilities": {
"drop": [
"all"
]
},
"readOnlyRootFilesystem": true,
"runAsNonRoot": true
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File"
}
],
"dnsPolicy": "ClusterFirst",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"serviceAccount": "controller",
"serviceAccountName": "controller",
"terminationGracePeriodSeconds": 30
}
}
},
"status": {}
},
"requestMetadata": {
"callerIp": "35.232.127.123",
"callerSuppliedUserAgent": "kapp/v0.0.0 (linux/amd64) kubernetes/$Format"
},
"resourceName": "apps/v1/namespaces/0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/deployments/controller",
"response": {
"@type": "core.k8s.io/v1.Status",
"apiVersion": "v1",
"code": 409,
"details": {
"group": "apps",
"kind": "deployments",
"name": "controller"
},
"kind": "Status",
"message": "Operation cannot be fulfilled on deployments.apps \"controller\": the object has been modified; please apply your changes to the latest version and try again",
"metadata": {},
"reason": "Conflict",
"status": "Failure"
},
"serviceName": "k8s.io",
"status": {
"code": 10,
"message": "Conflict"
}
},
"insertId": "62e16bb8-0394-446d-8b06-2775203e1c9a",
"resource": {
"type": "k8s_cluster",
"labels": {
"location": "us-east1",
"project_id": "knative-boskos-85",
"cluster_name": "e2e-cls-pnwjdyda"
}
},
"timestamp": "2022-04-05T02:27:23.814137Z",
"labels": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "access granted by IAM permissions."
},
"logName": "projects/knative-boskos-85/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "62e16bb8-0394-446d-8b06-2775203e1c9a",
"producer": "k8s.io",
"first": true,
"last": true
},
"receiveTimestamp": "2022-04-05T02:27:24.603465412Z"
}
But I'm seeing the first attempt to update that annotation is always a conflict (for which kapp retries)
i think that's expected because deployments are annotated with revision annotation by deployments controller.
Reasons for the conflict
- k8s controller adds status conditions
- I see something adds
deployment.kubernetes.io/revisionannotation
I guess should adding the annotation kapp.k14s.io/original be a PATCH instead of an update
I guess should adding the annotation kapp.k14s.io/original be a PATCH instead of an update
@dprotaso Can you please through some light on this. I was trying to understand how PATCH would be of help here but haven't figured it out yet 😅
See: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/
Essentially you can apply a patch to an existing object and you shouldn't receive the API conflicts.
Thanks for sharing this. I will try out a few scenarios and update here.
I hit the same issue when using kapp-controller and PackageInstall to deploy calico tigera-operator helm chart which creates a CR and deploys a controller reconciling this CR. kapp-controller and tigera-operator controller reconciles the same CR with different content, then causes updating conflict and makes the CR updated for many times. Details on https://kubernetes.slack.com/archives/CH8KCCKA5/p1686843485979029
