camel-k icon indicating copy to clipboard operation
camel-k copied to clipboard

Fix flaky health trait test

Open rinaldodev opened this issue 1 year ago • 4 comments
trafficstars

What happened?

While running this test, I noticed the Integration's Ready condition is true for a while before becoming false, which in turns makes this assertion fail sometimes, making the test flaky.

- firstTruthyTime: "2023-11-20T13:02:00Z"
  lastTransitionTime: "2023-11-20T13:02:00Z"
  lastUpdateTime: "2023-11-20T13:02:00Z"
  message: 1/1 ready replicas
  reason: DeploymentReady
  status: "True"
  type: Ready

See comment https://github.com/apache/camel-k/issues/4922#issuecomment-1858276507

Steps to reproduce

Run this e2e test a few times until it fails.

Relevant log output

apiVersion: camel.apache.org/v1
kind: Integration
metadata:
  annotations:
    camel.apache.org/operator.id: test-305d00bf-726e-4415-9567-2573502fa0bf
  creationTimestamp: "2023-11-20T12:49:24Z"
  generation: 1
  name: never-readypnryt
  namespace: test-305d00bf-726e-4415-9567-2573502fa0bf
  resourceVersion: "19430"
  uid: aa15f5f4-1343-4d20-b61c-48cc8f187298
spec:
  sources:
  - content: |
      /*
       * Licensed to the Apache Software Foundation (ASF) under one or more
       * contributor license agreements.  See the NOTICE file distributed with
       * this work for additional information regarding copyright ownership.
       * The ASF licenses this file to You under the Apache License, Version 2.0
       * (the "License"); you may not use this file except in compliance with
       * the License.  You may obtain a copy of the License at
       *
       *      http://www.apache.org/licenses/LICENSE-2.0
       *
       * Unless required by applicable law or agreed to in writing, software
       * distributed under the License is distributed on an "AS IS" BASIS,
       * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       * See the License for the specific language governing permissions and
       * limitations under the License.
       */

      import org.apache.camel.builder.RouteBuilder;

      public class NeverReady extends RouteBuilder {
          @Override
          public void configure() throws Exception {
              from("timer:tick").id("never-ready")
                  .to("controlbus:route?routeId=never-ready&action=stop&async=true")
                  .setHeader("m").constant("string!")
                  .setBody().simple("Magic${header.m}")
                  .log("${body}");
          }
      }
    name: NeverReady.java
  traits:
    health:
      enabled: true
      livenessProbeEnabled: false
      livenessScheme: HTTP
      readinessProbeEnabled: true
      readinessScheme: HTTP
      startupProbeEnabled: false
      startupScheme: HTTP
status:
  conditions:
  - firstTruthyTime: "2023-11-20T12:49:24Z"
    lastTransitionTime: "2023-11-20T12:49:24Z"
    lastUpdateTime: "2023-11-20T12:49:24Z"
    message: test-305d00bf-726e-4415-9567-2573502fa0bf/test-305d00bf-726e-4415-9567-2573502fa0bf
    reason: IntegrationPlatformAvailable
    status: "True"
    type: IntegrationPlatformAvailable
  - firstTruthyTime: "2023-11-20T12:49:24Z"
    lastTransitionTime: "2023-11-20T12:49:24Z"
    lastUpdateTime: "2023-11-20T12:49:24Z"
    message: explicitly disabled by the user
    reason: masterTraitConfiguration
    status: "True"
    type: TraitInfo
  - firstTruthyTime: "2023-11-20T13:01:56Z"
    lastTransitionTime: "2023-11-20T13:01:56Z"
    lastUpdateTime: "2023-11-20T13:01:56Z"
    message: kit-cldlbl2hqaes73e5b8j0
    reason: IntegrationKitAvailable
    status: "True"
    type: IntegrationKitAvailable
  - firstTruthyTime: "2023-11-20T13:01:56Z"
    lastTransitionTime: "2023-11-20T13:01:56Z"
    lastUpdateTime: "2023-11-20T13:01:56Z"
    message: deployment name is never-readypnryt
    reason: DeploymentAvailable
    status: "True"
    type: DeploymentAvailable
  - firstTruthyTime: "2023-11-20T13:02:00Z"
    lastTransitionTime: "2023-11-20T13:02:00Z"
    lastUpdateTime: "2023-11-20T13:02:00Z"
    message: 1/1 ready replicas
    reason: DeploymentReady
    status: "True"
    type: Ready
  dependencies:
  - camel:controlbus
  - camel:timer
  - mvn:org.apache.camel.k:camel-k-runtime
  - mvn:org.apache.camel.quarkus:camel-quarkus-java-joor-dsl
  - mvn:org.apache.camel.quarkus:camel-quarkus-microprofile-health
  digest: vZi_hPRbPLDXMUuj2SPoFJLW2OPdK7PcY7M79Rh_eGB8
  image: 10.98.173.76/test-305d00bf-726e-4415-9567-2573502fa0bf/camel-k-kit-cldlbl2hqaes73e5b8j0@sha256:15df1691ea5f08295324eb828228c69335bfe56ef4c544645e7813ad829539e5
  integrationKit:
    name: kit-cldlbl2hqaes73e5b8j0
    namespace: test-305d00bf-726e-4415-9567-2573502fa0bf
  lastInitTimestamp: "2023-11-20T12:49:24Z"
  observedGeneration: 1
  phase: Running
  platform: test-305d00bf-726e-4415-9567-2573502fa0bf
  profile: Kubernetes
  replicas: 1
  runtimeProvider: quarkus
  runtimeVersion: 3.2.0
  selector: camel.apache.org/integration=never-readypnryt
  version: 2.2.0-SNAPSHOT

Camel K version

2.1.0

rinaldodev avatar Nov 20 '23 16:11 rinaldodev

Would someone please update the labels on this issue?

rinaldodev avatar Nov 20 '23 16:11 rinaldodev

@rinaldodev thanks for reporting. What labels do you think it is required? Please also let me know if you are going to work on this so I can assign to you.

squakez avatar Nov 21 '23 08:11 squakez

@squakez I believe that one is fine. Yes, you can assign it to me, thanks.

rinaldodev avatar Nov 21 '23 17:11 rinaldodev

To bring more clarity to this issue: this appears to be caused by the camel-routes health check reporting as Ready before the routes had some time to load. See https://issues.apache.org/jira/browse/CAMEL-20242.

rinaldodev avatar Dec 15 '23 17:12 rinaldodev

@rinaldodev are you planning to work on this for next 2.4 release? Should I add it to the milestone?

squakez avatar Apr 10 '24 07:04 squakez

Some more analysis on this flaky test can be found in #5345 which is a duplicate of this one

christophd avatar Apr 10 '24 08:04 christophd

Fixed with #5346

christophd avatar Apr 24 '24 17:04 christophd