camel-k
camel-k copied to clipboard
apache
Bad revision when running jvm and native builds with knative
What happened?
I'm not sure whether this is a Camel K or Knative issue. This is what happens:
I created an Integration with the following source code:
apiVersion: camel.apache.org/v1
kind: Integration
metadata:
name: test
spec:
sources:
- name: main.groovy
content: |-
from('rest://GET:/test')
.to("direct:start")
from("direct:start")
.to("log:info")
traits:
container:
requestCPU: "200m"
requestMemory: 400Mi
limitMemory: 400Mi
quarkus:
buildMode:
- jvm
- native
builder:
properties:
- "quarkus.native.native-image-xmx=13g"
tasksRequestCPU:
- "quarkus-native:8000m"
tasksRequestMemory:
- "quarkus-native:16Gi"
tasksLimitMemory:
- "quarkus-native:16Gi"
nodeSelector:
karpenter: "true"
affinity:
enabled: true
nodeAffinityLabels:
- "karpenter=false"
knative-service:
minScale: 1
Right after it is applied, the operator starts both jvm and native builds. As soon as the jvm build finishes, I have the integration running with the jvm image. And there's one Knative revision and one pod:
k get revision
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON ACTUAL REPLICAS DESIRED REPLICAS
test-00001 test 1 True 1 1
k get pod
NAME READY STATUS RESTARTS AGE
camel-k-kit-cmin9roc3t7tsqciksu0-builder 0/1 Init:1/3 0 4m
camel-k-operator-6cbc656bbd-wpzk2 1/1 Running 0 4m36s
test-00001-deployment-b84f65c69-fjllp 2/2 Running 0 3m
After a few minutes, when the native build is ready, Camel K tries to replace the running one with the native version, but something goes wrong. I have 2 revisions/pods running:
k get revision
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON ACTUAL REPLICAS DESIRED REPLICAS
test-00002 test 2 Unknown Deploying 0
test-00003 test 3 True 1 1
k get pod
NAME READY STATUS RESTARTS AGE
camel-k-kit-cmin9roc3t7tsqciksu0-builder 0/1 Completed 0 7m27s
camel-k-operator-6cbc656bbd-wpzk2 1/1 Running 0 8m3s
test-00002-deployment-7d9846f9b5-wmb5c 0/2 Error 4 (52s ago) 97s
test-00003-deployment-df97fb85d-7v78t 2/2 Running 0 97s
test-0003 replaced test-0001 correctly and uses the native build image. however, test-0002 was created with the native image, but the startup command is the jvm one:
k logs test-00002-deployment-7d9846f9b5-wmb5c
Defaulted container "integration" out of: integration, queue-proxy
exec java -Xmx210M -cp ./resources:/etc/camel/application.properties:/etc/camel/conf.d/_resources:/etc/camel/resources:/etc/camel/sources/main.groovy:dependencies/app/camel-k-integration-2.2.0.jar:dependencies/lib/boot/io.github.crac.org-crac-0.1.3.jar:dependencies/lib/boot/io.quarkus.quarkus-bootstrap-runner-3.2.9.Final.jar:dependencies/lib/boot/io.quarkus.quarkus-development-mode-spi-3.2.9.Final.jar:dependencies/lib/boot/io.quarkus.quarkus-vertx-latebound-mdc-provider-3.2.9.Final.jar:dependencies/lib/boot/io.smallrye.common.smallrye-common-io-2.1.0.jar:dependencies/lib/boot/jakarta.json.jakarta.json-api-2.1.2.jar:dependencies/lib/boot/org.eclipse.parsson.parsson-1.1.2.jar:dependencies/lib/boot/org.graalvm.sdk.graal-sdk-23.0.1.jar:dependencies/lib/boot/org.jboss.logging.jboss-logging-3.5.1.Final.jar:dependencies/lib/boot/org.jboss.logmanager.jboss-logmanager-embedded-1.1.1.jar:dependencies/lib/boot/org.wildfly.common.wildfly-common-1.5.4.Final-format-001.jar:dependencies/lib/main/com.aayushatharva.brotli4j.brotli4j-1.12.0.jar:dependencies/lib/main/com.aayushatharva.brotli4j.native-linux-x86_64-1.12.0.jar:dependencies/lib/main/com.aayushatharva.brotli4j.service-1.12.0.jar:dependencies/lib/main/com.fasterxml.jackson.core.jackson-annotations-2.15.2.jar:dependencies/lib/main/com.fasterxml.jackson.core.jackson-core-2.15.2.jar:dependencies/lib/main/com.fasterxml.jackson.core.jackson-databind-2.15.2.jar:dependencies/lib/main/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.15.2.jar:dependencies/lib/main/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.15.2.jar:dependencies/lib/main/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.15.2.jar:dependencies/lib/main/com.fasterxml.jackson.module.jackson-module-parameter-names-2.15.2.jar:dependencies/lib/main/com.github.mifmif.generex-1.0.2.jar:dependencies/lib/main/dk.brics.automaton.automaton-1.11-8.jar:dependencies/lib/main/io.fabric8.kubernetes-client-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-client-api-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-httpclient-vertx-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-admissionregistration-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-apiextensions-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-apps-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-autoscaling-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-batch-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-certificates-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-common-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-coordination-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-core-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-discovery-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-events-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-extensions-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-flowcontrol-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-gatewayapi-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-metrics-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-networking-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-node-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-policy-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-rbac-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-resource-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-scheduling-6.7.2.jar:dependencies/lib/main/io.fabric8.kubernetes-model-storageclass-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-client-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-client-api-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-clusterautoscaling-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-config-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-console-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-hive-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-installer-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-machine-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-machineconfig-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-miscellaneous-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-monitoring-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-operator-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-operatorhub-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-storageversionmigrator-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-tuned-6.7.2.jar:dependencies/lib/main/io.fabric8.openshift-model-whereabouts-6.7.2.jar:dependencies/lib/main/io.fabric8.zjsonpatch-0.3.0.jar:dependencies/lib/main/io.netty.netty-buffer-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-codec-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-codec-dns-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-codec-haproxy-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-codec-http-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-codec-http2-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-codec-socks-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-common-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-handler-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-handler-proxy-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-resolver-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-resolver-dns-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-transport-4.1.100.Final.jar:dependencies/lib/main/io.netty.netty-transport-native-unix-common-4.1.100.Final.jar:dependencies/lib/main/io.quarkiverse.groovy.quarkus-groovy-3.2.2.jar:dependencies/lib/main/io.quarkus.arc.arc-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-arc-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-core-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-credentials-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-fs-util-0.0.9.jar:dependencies/lib/main/io.quarkus.quarkus-jackson-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-jsonp-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-kubernetes-client-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-kubernetes-client-internal-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-logging-json-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-mutiny-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-netty-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-security-runtime-spi-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-smallrye-context-propagation-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-vertx-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-vertx-http-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.quarkus-vertx-http-dev-console-runtime-spi-3.2.9.Final.jar:dependencies/lib/main/io.quarkus.security.quarkus-security-2.0.2.Final.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-annotation-2.1.0.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-classloader-2.1.0.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-constraint-2.1.0.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-expression-2.1.0.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-function-2.1.0.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-os-2.1.0.jar:dependencies/lib/main/io.smallrye.common.smallrye-common-vertx-context-2.1.0.jar:dependencies/lib/main/io.smallrye.config.smallrye-config-3.3.2.jar:dependencies/lib/main/io.smallrye.config.smallrye-config-common-3.3.2.jar:dependencies/lib/main/io.smallrye.config.smallrye-config-core-3.3.2.jar:dependencies/lib/main/io.smallrye.config.smallrye-config-source-yaml-3.3.2.jar:dependencies/lib/main/io.smallrye.reactive.mutiny-2.3.1.jar:dependencies/lib/main/io.smallrye.reactive.mutiny-smallrye-context-propagation-2.3.1.jar:dependencies/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-auth-common-3.7.2.jar:dependencies/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-bridge-common-3.7.2.jar:dependencies/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-core-3.7.2.jar:dependencies/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-runtime-3.7.2.jar:dependencies/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-uri-template-3.7.2.jar:dependencies/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-web-3.7.2.jar:dependencies/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-web-common-3.7.2.jar:dependencies/lib/main/io.smallrye.reactive.vertx-mutiny-generator-3.7.2.jar:dependencies/lib/main/io.smallrye.smallrye-context-propagation-2.1.0.jar:dependencies/lib/main/io.smallrye.smallrye-context-propagation-api-2.1.0.jar:dependencies/lib/main/io.smallrye.smallrye-context-propagation-storage-2.1.0.jar:dependencies/lib/main/io.smallrye.smallrye-fault-tolerance-vertx-6.2.6.jar:dependencies/lib/main/io.vertx.vertx-auth-common-4.4.6.jar:dependencies/lib/main/io.vertx.vertx-bridge-common-4.4.6.jar:dependencies/lib/main/io.vertx.vertx-codegen-4.4.6.jar:dependencies/lib/main/io.vertx.vertx-core-4.4.6.jar:dependencies/lib/main/io.vertx.vertx-uri-template-4.4.6.jar:dependencies/lib/main/io.vertx.vertx-web-4.4.6.jar:dependencies/lib/main/io.vertx.vertx-web-client-4.4.6.jar:dependencies/lib/main/io.vertx.vertx-web-common-4.4.6.jar:dependencies/lib/main/jakarta.activation.jakarta.activation-api-2.1.2.jar:dependencies/lib/main/jakarta.annotation.jakarta.annotation-api-2.1.1.jar:dependencies/lib/main/jakarta.el.jakarta.el-api-5.0.1.jar:dependencies/lib/main/jakarta.enterprise.jakarta.enterprise.cdi-api-4.0.1.jar:dependencies/lib/main/jakarta.enterprise.jakarta.enterprise.lang-model-4.0.1.jar:dependencies/lib/main/jakarta.inject.jakarta.inject-api-2.0.1.jar:dependencies/lib/main/jakarta.interceptor.jakarta.interceptor-api-2.1.0.jar:dependencies/lib/main/jakarta.transaction.jakarta.transaction-api-2.0.1.jar:dependencies/lib/main/jakarta.xml.bind.jakarta.xml.bind-api-4.0.0.jar:dependencies/lib/main/org.apache.camel.camel-api-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-attachments-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-base-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-base-engine-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-bean-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-cloud-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-cloudevents-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-cluster-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-componentdsl-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-core-catalog-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-core-engine-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-core-languages-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-core-model-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-core-processor-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-core-reifier-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-direct-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-dsl-support-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-endpointdsl-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-endpointdsl-support-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-groovy-dsl-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-groovy-dsl-common-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-knative-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-knative-api-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-knative-http-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-kubernetes-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-log-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-main-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-management-api-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-microprofile-config-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-platform-http-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-platform-http-vertx-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-rest-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-support-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-tooling-model-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-util-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-util-json-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-vertx-common-4.0.2.jar:dependencies/lib/main/org.apache.camel.camel-xml-jaxp-util-4.0.2.jar:dependencies/lib/main/org.apache.camel.k.camel-k-core-3.2.3.jar:dependencies/lib/main/org.apache.camel.k.camel-k-core-api-3.2.3.jar:dependencies/lib/main/org.apache.camel.k.camel-k-core-support-3.2.3.jar:dependencies/lib/main/org.apache.camel.k.camel-k-knative-impl-3.2.3.jar:dependencies/lib/main/org.apache.camel.k.camel-k-runtime-3.2.3.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-attachments-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-bean-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-cloudevents-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-core-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-core-cloud-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-direct-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-groovy-dsl-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-knative-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-kubernetes-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-log-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-platform-http-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-rest-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-support-commons-logging-3.2.2.jar:dependencies/lib/main/org.apache.camel.quarkus.camel-quarkus-support-dsl-3.2.2.jar:dependencies/lib/main/org.apache.commons.commons-compress-1.24.0.jar:dependencies/lib/main/org.apache.groovy.groovy-4.0.13.jar:dependencies/lib/main/org.eclipse.angus.angus-activation-2.0.1.jar:dependencies/lib/main/org.eclipse.microprofile.config.microprofile-config-api-3.0.3.jar:dependencies/lib/main/org.eclipse.microprofile.context-propagation.microprofile-context-propagation-api-1.3.jar:dependencies/lib/main/org.jboss.logging.commons-logging-jboss-logging-1.0.0.Final.jar:dependencies/lib/main/org.jboss.logging.jboss-logging-annotations-2.2.1.Final.jar:dependencies/lib/main/org.jboss.slf4j.slf4j-jboss-logmanager-2.0.0.Final.jar:dependencies/lib/main/org.jboss.threads.jboss-threads-3.5.0.Final.jar:dependencies/lib/main/org.slf4j.slf4j-api-2.0.6.jar:dependencies/lib/main/org.snakeyaml.snakeyaml-engine-2.6.jar:dependencies/lib/main/org.yaml.snakeyaml-2.0.jar:dependencies/quarkus-app-dependencies.txt:dependencies/quarkus-run.jar:dependencies/quarkus/generated-bytecode.jar:dependencies/quarkus/quarkus-application.dat:dependencies/quarkus/transformed-bytecode.jar io.quarkus.bootstrap.runner.QuarkusEntryPoint
Error: Could not find or load main class io.quarkus.bootstrap.runner.QuarkusEntryPoint
Caused by: java.lang.ClassNotFoundException: io.quarkus.bootstrap.runner.QuarkusEntryPoint
The Integration status is Error, and describing it shows the test-00002 ClassNotFoundException error.
After about 10 minutes, the test-0002 pod disappeared and the integration is running.
So I see 3 possible issues here:
- there was a pod with the correct image, but wrong command
- this pod affected the integration status
- It took too long for it to fix.
Steps to reproduce
Install Knative 1.12.3 with contour network layer, Camel K 2.2.0 Apply the integration described above, adjusting the Traits for your environment. Wait for the native build to finish. Check the number of pods and their statuses.
Relevant log output
No response
Camel K version
2.2.0, knative serving v1.12.3 with contour enabled
Thanks for reporting. I have the vague feeling this is kind of unsupported scenario. When we create the native build, what we do is creating a "native" Deployment which attempts to replace the "jvm" deployment. However, in the Knative scenario, we do not control directly a Deployment CR, but a KnativeService which is the one in charge to manage Deployment lifecycle. In order to fix it we need to make sure that the native build replaces a Deployment, a KnativeService or a CronJob, which are the application types we control.
Please, check the KnativeService CR object in order to verify if a new one is created or in general how it evolves during this use case.
I inspected the kservices and this is what I found out:
- While only the jvm build is ready, the ksvc uses the jvm image and the container command is echo exec java -Xmx210M -cp...
- When the native build finishes, the ksvc is updated to the native image and the the native command switches to - ./camel-k-integration-2.2.0-runner as expected. At this time I have test-00003 pod working as expected, but test-00002 is there with the wrong command.
- When test-00002 disappears, nothing changes on the ksvc.
Thanks for the details. Then, it seems that the operator behavior is correct as it applies the changes to the KSVC custom resource as it is expected. The only additional thing it is worth to check is the ownership of those pods. Can you please verify they are correctly bound to the ksvc/revision hierarchy? if that's the case, I think it could be more a knative bug. You may even do a test removing the Camel K operator from the scene to double check that. Ie, you can create a ksvc with a given application and manually updates the KSVC and see if it reproduces the same behavior.
More information:
I inspected the kubernetes events after the native image is built and found this:
3m53s Normal ReasonRelatedObjectChanged integration/test Build "kit-cmj7n4tmc86c7nh4hhv0", created by Integration "test", changed phase to "Succeeded"
3m53s Normal ReasonRelatedObjectChanged integration/test Integration Kit "kit-cmj7n4tmc86c7nh4hhv0", created by Integration "test", changed phase to "Ready"
116s Normal IntegrationConditionChanged integration/test (combined from similar events): Condition "Ready" is "False" for Integration test: back-off 1m20s restarting failed container=integration pod=test-00002-deployment-55767d5469-rwr92_sensedia(e9698721-4e13-470e-bbed-ed35ce2f3aae)
39s Normal IntegrationConditionChanged integration/test (combined from similar events): Condition "Ready" is "True" for Integration test
3m52s Normal Created configuration/test Created Revision "test-00002"
3m52s Warning InternalError service/test failed to reconcile Configuration: Operation cannot be fulfilled on configurations.serving.knative.dev "test": the object has been modified; please apply your changes to the latest version and try again
3m52s Normal Created configuration/test Created Revision "test-00003"
3m48s Normal LatestReadyUpdate configuration/test LatestReadyRevisionName updated to "test-00003"
3m48s Warning InternalError ingress/test Operation cannot be fulfilled on httpproxies.projectcontour.io "test-contour-internal-test.sensedia.svc": the object has been modified; please apply your changes to the latest version and try again
test-0003 is created after a failed attempt to reconcile the Configuration object.
With that info I inspected the knative serving controller logs and got:
{"severity":"INFO","timestamp":"2024-01-16T13:15:45.601340739Z","logger":"controller","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"13b19c56-6863-4914-b3ae-59f9d8f4bf37","knative.dev/key":"sensedia/test-00002","duration":"385.188151ms"}
{"severity":"WARNING","timestamp":"2024-01-16T13:15:45.620704334Z","logger":"controller","caller":"logging/warning_handler.go:32","message":"API Warning: Kubernetes default value is insecure, Knative may default this to secure in a future release: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation, spec.template.spec.containers[0].securityContext.capabilities, spec.template.spec.containers[0].securityContext.runAsNonRoot, spec.template.spec.containers[0].securityContext.seccompProfile","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g"}
{"severity":"WARNING","timestamp":"2024-01-16T13:15:45.621195713Z","logger":"controller","caller":"reconciler/reconcile_common.go:87","message":"A reconciler observed a new generation without updating the resource status","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.service.Reconciler","knative.dev/kind":"serving.knative.dev.Service","knative.dev/traceid":"926d8d6c-6a3e-481e-948a-292de000e740","knative.dev/key":"sensedia/test"}
{"severity":"INFO","timestamp":"2024-01-16T13:15:45.637014072Z","logger":"controller","caller":"revision/reconcile_resources.go:56","message":"Created deployment \"test-00002-deployment\"","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"567a37f0-6e1f-4421-8eef-aae905d36bee","knative.dev/key":"sensedia/test-00002","knative.dev/deployment":"test-00002-deployment"}
{"severity":"INFO","timestamp":"2024-01-16T13:15:45.637715554Z","logger":"controller","caller":"route/route.go:179","message":"Route successfully synced","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.route.Reconciler","knative.dev/kind":"serving.knative.dev.Route","knative.dev/traceid":"cb4d9fed-c77b-474f-9436-193eaacfc433","knative.dev/key":"sensedia/test"}
{"severity":"INFO","timestamp":"2024-01-16T13:15:45.637812876Z","logger":"controller","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.route.Reconciler","knative.dev/kind":"serving.knative.dev.Route","knative.dev/traceid":"cb4d9fed-c77b-474f-9436-193eaacfc433","knative.dev/key":"sensedia/test","duration":"53.875255ms"}
{"severity":"ERROR","timestamp":"2024-01-16T13:15:45.652289959Z","logger":"controller","caller":"service/reconciler.go:303","message":"Returned an error","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.service.Reconciler","knative.dev/kind":"serving.knative.dev.Service","knative.dev/traceid":"926d8d6c-6a3e-481e-948a-292de000e740","knative.dev/key":"sensedia/test","targetMethod":"ReconcileKind","error":"failed to reconcile Configuration: Operation cannot be fulfilled on configurations.serving.knative.dev \"test\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/pkg/client/injection/reconciler/serving/v1/service.(*reconcilerImpl).Reconcile\n\tknative.dev/serving/pkg/client/injection/reconciler/serving/v1/service/reconciler.go:303\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2024-01-16T13:15:45.65235102Z","logger":"controller","caller":"controller/controller.go:566","message":"Reconcile error","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.service.Reconciler","knative.dev/kind":"serving.knative.dev.Service","knative.dev/traceid":"926d8d6c-6a3e-481e-948a-292de000e740","knative.dev/key":"sensedia/test","duration":"192.246443ms","error":"failed to reconcile Configuration: Operation cannot be fulfilled on configurations.serving.knative.dev \"test\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/[email protected]/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:491"}
{"severity":"INFO","timestamp":"2024-01-16T13:15:45.653651244Z","logger":"controller","caller":"service/service.go:201","message":"Reconciling configuration diff (-desired, +observed):\n v1.ConfigurationSpec{\n \tTemplate: v1.RevisionTemplateSpec{\n \t\tObjectMeta: {Labels: {\"camel.apache.org/integration\": \"test\"}, Annotations: {\"autoscaling.knative.dev/minScale\": \"1\"}},\n \t\tSpec: v1.RevisionSpec{\n \t\t\tPodSpec: v1.PodSpec{\n- \t\t\t\tVolumes: nil,\n+ \t\t\t\tVolumes: []v1.Volume{\n+ \t\t\t\t\t{\n+ \t\t\t\t\t\tName: \"i-source-000\",\n+ \t\t\t\t\t\tVolumeSource: v1.VolumeSource{ConfigMap: s\"&ConfigMapVolumeSource{LocalObje\"...},\n+ \t\t\t\t\t},\n+ \t\t\t\t\t{\n+ \t\t\t\t\t\tName: \"application-properties\",\n+ \t\t\t\t\t\tVolumeSource: v1.VolumeSource{ConfigMap: s\"&ConfigMapVolumeSource{LocalObje\"...},\n+ \t\t\t\t\t},\n+ \t\t\t\t},\n \t\t\t\tInitContainers: nil,\n \t\t\t\tContainers: []v1.Container{\n \t\t\t\t\t{\n \t\t\t\t\t\tName: \"integration\",\n \t\t\t\t\t\tImage: \"gcr.io/registry-ipaas-testing/camel-k-kit-cmj80ni4opqktlej1b70@s\"...,\n \t\t\t\t\t\tCommand: []string{\n- \t\t\t\t\t\t\t\"./camel-k-integration-2.2.0-runner\",\n+ \t\t\t\t\t\t\t\"/bin/sh\",\n+ \t\t\t\t\t\t\t\"-c\",\n \t\t\t\t\t\t},\n- \t\t\t\t\t\tArgs: nil,\n+ \t\t\t\t\t\tArgs: []string{\n+ \t\t\t\t\t\t\t\"echo exec java -Xmx210M -cp ./resources:/etc/camel/application.properties:/etc/camel/conf.d/_resources:/etc/camel/resources:/etc\"...,\n+ \t\t\t\t\t\t},\n \t\t\t\t\t\tWorkingDir: \"/deployments\",\n \t\t\t\t\t\tPorts: nil,\n \t\t\t\t\t\t... // 2 identical fields\n \t\t\t\t\t\tResources: {Limits: {s\"memory\": {i: {...}, Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"200m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\t\t\t\tResizePolicy: nil,\n- \t\t\t\t\t\tVolumeMounts: nil,\n+ \t\t\t\t\t\tVolumeMounts: []v1.VolumeMount{\n+ \t\t\t\t\t\t\t{\n+ \t\t\t\t\t\t\t\tName: \"i-source-000\",\n+ \t\t\t\t\t\t\t\tReadOnly: true,\n+ \t\t\t\t\t\t\t\tMountPath: \"/etc/camel/sources/main.groovy\",\n+ \t\t\t\t\t\t\t\tSubPath: \"main.groovy\",\n+ \t\t\t\t\t\t\t},\n+ \t\t\t\t\t\t\t{\n+ \t\t\t\t\t\t\t\tName: \"application-properties\",\n+ \t\t\t\t\t\t\t\tReadOnly: true,\n+ \t\t\t\t\t\t\t\tMountPath: \"/etc/camel/application.properties\",\n+ \t\t\t\t\t\t\t\tSubPath: \"application.properties\",\n+ \t\t\t\t\t\t\t},\n+ \t\t\t\t\t\t},\n \t\t\t\t\t\tVolumeDevices: nil,\n \t\t\t\t\t\tLivenessProbe: nil,\n \t\t\t\t\t\t... // 10 identical fields\n \t\t\t\t\t},\n \t\t\t\t},\n \t\t\t\tEphemeralContainers: nil,\n \t\t\t\tRestartPolicy: \"\",\n \t\t\t\t... // 34 identical fields\n \t\t\t},\n \t\t\tContainerConcurrency: &0,\n \t\t\tTimeoutSeconds: &300,\n \t\t\t... // 2 identical fields\n \t\t},\n \t},\n }\n","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.service.Reconciler","knative.dev/kind":"serving.knative.dev.Service","knative.dev/traceid":"6eaabd4b-f1f9-41bc-ac38-3ba2998e9f8f","knative.dev/key":"sensedia/test"}
{"severity":"WARNING","timestamp":"2024-01-16T13:15:45.653732437Z","logger":"controller","caller":"service/service.go:226","message":"Service-delegated Configuration \"test\" diff found. Clobbering.","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.service.Reconciler","knative.dev/kind":"serving.knative.dev.Service","knative.dev/traceid":"6eaabd4b-f1f9-41bc-ac38-3ba2998e9f8f","knative.dev/key":"sensedia/test"}
{"severity":"INFO","timestamp":"2024-01-16T13:15:45.654236465Z","logger":"controller.event-broadcaster","caller":"record/event.go:298","message":"Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"sensedia\", Name:\"test\", UID:\"2ee04f54-55e4-49ca-9ace-a9fd4299fe67\", APIVersion:\"serving.knative.dev/v1\", ResourceVersion:\"17694935\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' failed to reconcile Configuration: Operation cannot be fulfilled on configurations.serving.knative.dev \"test\": the object has been modified; please apply your changes to the latest version and try again","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g"}
{"severity":"INFO","timestamp":"2024-01-16T13:15:45.671505301Z","logger":"controller","caller":"revision/reconcile_resources.go:144","message":"Created image cache \"test-00002-cache-integration\"","commit":"f1bd929","knative.dev/pod":"controller-6456689d7b-bcb6g","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"567a37f0-6e1f-4421-8eef-aae905d36bee","knative.dev/key":"sensedia/test-00002"}
I'll try and reproduce the error without using the operator and let you know how it goes.
I could not reproduce the error working directly with Knative. These are the steps what I followed:
- Exported the yaml files for the config maps created by camel k: test-application-properties and test-source-000
- Exported the yaml file for the ksvc while the jvm pod was running
- Exported the yaml file for the ksvc when the Integration was in the error state
Then I deleted the Integration and recreated the config maps.
For next step I applied the first ksvc yaml file and the service started successfully. Finally, I updated the ksvc by using the second yaml file. A new knative revision was created and the previous one was deleted. The pod ran successfully
During more experimentation, I ended up with a scenario that might have the same root cause:
I created an Integration and provided less resources than it needs in the Container trait (only 20MB for a jvm integration):
apiVersion: camel.apache.org/v1
kind: Integration
metadata:
name: test
spec:
sources:
- name: main.groovy
content: |-
from('rest://GET:/test')
.to("direct:start")
from("direct:start")
.to("https://httpbin.org/delay/2?bridgeEndpoint=true")
.to("log:info")
traits:
container:
requestCPU: "200m"
requestMemory: 20Mi
limitMemory: 20Mi
quarkus:
buildMode:
- jvm
affinity:
enabled: true
nodeAffinityLabels:
- "karpenter=false"
knative-service:
minScale: 1
After applying it, I see the pod in a CrashLookBackoff due to OOM error and the Integration is in error status:
k get pod
NAME READY STATUS RESTARTS AGE
camel-k-operator-6cbc656bbd-bmmwp 1/1 Running 0 156m
test-00001-deployment-8456cb79f9-9pxbl 0/2 CrashLoopBackOff 1 (14s ago) 16s
k get it
NAME PHASE RUNTIME PROVIDER RUNTIME VERSION KIT REPLICAS
test Error quarkus 3.2.3 kit-cml7drk14nurfir6ingg 1
Then I fixed the amount of memory in the Container trait and updated the Integration. Now I see 2 pods for my Integration:
k get pod
NAME READY STATUS RESTARTS AGE
camel-k-operator-6cbc656bbd-bmmwp 1/1 Running 0 158m
test-00001-deployment-8456cb79f9-9pxbl 0/2 CrashLoopBackOff 4 (55s ago) 2m25s
test-00002-deployment-6ffff56fb5-r9wqc 1/2 Running 0 7s
where the second one is running correctly. But the Integration is still in Error status:
NAME PHASE RUNTIME PROVIDER RUNTIME VERSION KIT REPLICAS
test Error quarkus 3.2.3 kit-cml7drk14nurfir6ingg 2
As it shows 2 replicas, I assume it's considering both pods for determining that it is failing. At this point, the ksvc state is:
k get ksvc
NAME URL LATESTCREATED LATESTREADY READY REASON
test http://test.sensedia.poc-luis.sensedia-eng.com test-00002 test-00002 True
And there are 2 knative revisions:
k get revision
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON ACTUAL REPLICAS DESIRED REPLICAS
test-00001 test 1 False ExitCode137 0
test-00002 test 2 True 1 1
After around ~10 minutes, the old pod disappeated and the Integration moved to a success state. And the revision list shows me:
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON ACTUAL REPLICAS DESIRED REPLICAS
test-00001 test 1 False ProgressDeadlineExceeded 0
test-00002 test 2 True 1 1
We have worked to several enhancement in version 2.3, I think this may have been fixed during these developments. Would you mind testing this scenario again with latest nightly to confirm if it's gone or still there? We're about to release version 2.3, so, if it's cleared in the nightly, we can close this.