Meridio icon indicating copy to clipboard operation
Meridio copied to clipboard

Published 20 hours ago • verified publisher icon Nordix

E2E tests instability

Open LionelJouin opened this issue 1 year ago • 2 comments

Describe the bug Tests are failing randomly around 19% of the time.

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/activity/

meridio-e2e-kind-meridio

meridio-e2e-kind-tapa

meridio-e2e-kind-nsm

meridio-e2e-kind-ip-family

meridio-e2e-kind-kubernetes

https://github.com/Nordix/Meridio/blob/master/docs/test.md

To Reproduce /

Expected behavior /

Logs

This report is from job run 24 to 172 (149 runs).

invalid character 'R' looking for beginning of value

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/71/pipeline/22/

[38;5;13m•! [PANICKED] [190.727 seconds][0m
Scaling
[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:30[0m
  With one trench containing a stream with 2 VIP addresses and 4 target pods running
  [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:32[0m
    when scaling targets up by 1
    [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:148[0m
      [38;5;13m[1m[It] should receive the traffic correctly[0m
      [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:152[0m

  [38;5;243mBegin Captured GinkgoWriter Output >>[0m
    [1mSTEP:[0m Checking if all targets have receive ipv4 traffic with no traffic interruption (no lost connection) [38;5;243m11/01/22 16:09:02.33[0m
    [1mSTEP:[0m Checking if all targets have receive ipv6 traffic with no traffic interruption (no lost connection) [38;5;243m11/01/22 16:09:03.53[0m
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;13mTest Panicked[0m
  [38;5;13mIn [1m[It][0m[38;5;13m at: [1m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/utils/trafficgenerator.go:96[0m

  [38;5;13minvalid character 'R' looking for beginning of value[0m

  [38;5;13mFull Stack Trace[0m
    github.com/nordix/meridio/test/e2e/utils.(*MConnect).AnalyzeTraffic(0x15fa73d?, {0xc0000fb000, 0x6c9, 0x1000})
    	/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/utils/trafficgenerator.go:96 +0x1a6
    github.com/nordix/meridio/test/e2e/utils.(*TrafficGeneratorHost).SendTraffic(0xc0004ddf68?, {0x1821bd0, 0xc000133968}, {0x7fff08b74cb3?, 0x8}, {0x7fff08b74c82, 0x3}, {0xc000494450, 0xe}, {0x15f7463, ...})
    	/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/utils/trafficgenerator.go:45 +0x277
    github.com/nordix/meridio/test/e2e_test.glob..func6.1.5.2()
    	/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:161 +0x3cd

3 Targets are receiving traffic instead of 4 (most frequent) - https://github.com/Nordix/Meridio/issues/234

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/160/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/154/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/148/pipeline/22 https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/131/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/114/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/99/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/84/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/75/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/68/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/56/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/53/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/41/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/31/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/28/pipeline/22/

[38;5;9m• [FAILED] [29.806 seconds][0m
Target
[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/68/test/e2e/target_test.go:30[0m
  With one trench containing a stream with 2 VIP addresses and 4 target pods running
  [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/68/test/e2e/target_test.go:32[0m
    when a target is closing a stream
    [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/68/test/e2e/target_test.go:51[0m
      [38;5;9m[1m[It] should receive traffic anymore[0m
      [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/68/test/e2e/target_test.go:81[0m

  [38;5;243mBegin Captured GinkgoWriter Output >>[0m
    [1mSTEP:[0m Checking the target has not receive ipv4 traffic [38;5;243m11/01/22 15:40:36.855[0m
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;9mExpected
      <int>: 4
  to equal
      <int>: 3[0m

unable to upgrade connection: container not found ("example-target")

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/2760/pipeline/22

[38;5;9m• [FAILED] [0.050 seconds][0m
MultiTrenches
[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:30[0m
  With two trenches containing both a stream with 2 VIP addresses and 4 target pods running
  [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:32[0m
    [38;5;9m[1mwhen a target disconnects from a trench and connect to another one [BeforeEach][0m
    [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:109[0m
      should receive the traffic on the other trench
      [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:141[0m

  [38;5;9mUnexpected error:
      <*fmt.wrapError | 0xc0002bd4e0>: {
          msg: "unable to upgrade connection: container not found (\"example-target\"); ",
          err: <*errors.errorString | 0xc00050efc0>{
              s: "unable to upgrade connection: container not found (\"example-target\")",
          },
      }
      unable to upgrade connection: container not found ("example-target"); 
  occurred[0m
  [38;5;9mIn [1m[BeforeEach][0m[38;5;9m at: [1m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:111[0m

command terminated with exit code 137

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/165/pipeline/22 https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/81/pipeline/22/ https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/43/pipeline/22/

[38;5;9m• [FAILED] [29.309 seconds][0m
MultiTrenches
[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:30[0m
  With two trenches containing both a stream with 2 VIP addresses and 4 target pods running
  [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:32[0m
    [38;5;9m[1mwhen a target disconnects from a trench and connect to another one [BeforeEach][0m
    [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:109[0m
      should receive the traffic on the other trench
      [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:141[0m

  [38;5;9mUnexpected error:
      <*fmt.wrapError | 0xc000548160>: {
          msg: "command terminated with exit code 137; ",
          err: <exec.CodeExitError>{
              Err: <*errors.errorString | 0xc0003ba310>{
                  s: "command terminated with exit code 137",
              },
              Code: 137,
          },
      }
      command terminated with exit code 137; 
  occurred[0m
  [38;5;9mIn [1m[BeforeEach][0m[38;5;9m at: [1m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:113[0m

pod not found

[38;5;9m• [FAILED] [0.025 seconds][0m
MultiTrenches
[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:30[0m
  With two trenches containing both a stream with 2 VIP addresses and 4 target pods running
  [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:32[0m
    [38;5;9m[1mwhen a target disconnects from a trench and connect to another one [BeforeEach][0m
    [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:109[0m
      should receive the traffic on the other trench
      [38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:141[0m

  [38;5;9mUnexpected error:
      <*fmt.wrapError | 0xc0004f8980>: {
          msg: "pods \"target-a-878b7cb64-dxw9x\" not found; ",
          err: <*errors.StatusError | 0xc0004f2be0>{
              ErrStatus: {
                  TypeMeta: {Kind: "", APIVersion: ""},
                  ListMeta: {
                      SelfLink: "",
                      ResourceVersion: "",
                      Continue: "",
                      RemainingItemCount: nil,
                  },
                  Status: "Failure",
                  Message: "pods \"target-a-878b7cb64-dxw9x\" not found",
                  Reason: "NotFound",
                  Details: {
                      Name: "target-a-878b7cb64-dxw9x",
                      Group: "",
                      Kind: "pods",
                      UID: "",
                      Causes: nil,
                      RetryAfterSeconds: 0,
                  },
                  Code: 404,
              },
          },
      }
      pods "target-a-878b7cb64-dxw9x" not found; 
  occurred[0m
  [38;5;9mIn [1m[BeforeEach][0m[38;5;9m at: [1m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:111[0m

Timeout

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/364/pipeline/22/

Internal error occurred: error executing command in container: failed to exec in container: failed to create exec

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/2548/pipeline/22

• [FAILED] [394.636 seconds]
MultiTrenches
/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:30
  MT-Switch
  /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:110
    when Disconnect a target from target-a-deployment-name from trench-a and connect it to trench-b [BeforeEach]
    /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:112
      (Traffic) is received by the targets
      /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:144

  Unexpected error:
      <*fmt.wrapError | 0xc0001ee0c0>: {
          msg: "Internal error occurred: error executing command in container: failed to exec in container: failed to create exec \"fe45234377476bf7e74dabbe876cf7c7db8ca19dcd5723744357955ecad1c054\": task b63acb78a9f04481223ad020bf9fa7fba8297d01265934b930906360c91af8a3 not found: not found; ",
          err: <*errors.errorString | 0xc0003ae200>{
              s: "Internal error occurred: error executing command in container: failed to exec in container: failed to create exec \"fe45234377476bf7e74dabbe876cf7c7db8ca19dcd5723744357955ecad1c054\": task b63acb78a9f04481223ad020bf9fa7fba8297d01265934b930906360c91af8a3 not found: not found",
          },
      }
      Internal error occurred: error executing command in container: failed to exec in container: failed to create exec "fe45234377476bf7e74dabbe876cf7c7db8ca19dcd5723744357955ecad1c054": task b63acb78a9f04481223ad020bf9fa7fba8297d01265934b930906360c91af8a3 not found: not found; 
  occurred
  In [BeforeEach] at: /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:119

Interrupted by Timeout

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/2936/pipeline

command terminated with exit code 1; FAILED mapSharedData: tshm-stream-a-i

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/3154

• [FAILED] [451.770 seconds]
TAPA
/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:30
  close-open
  /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:49
    when Close stream-a-I in one of the target from target-a-deployment-name and re-open it [BeforeEach]
    /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:55
      (Traffic) is received by the targets
      /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:118

  Unexpected error:
      <*fmt.wrapError | 0xc000424160>: {
          msg: "command terminated with exit code 1; FAILED mapSharedData: tshm-stream-a-i\n",
          err: <exec.CodeExitError>{
              Err: <*errors.errorString | 0xc0005820c0>{
                  s: "command terminated with exit code 1",
              },
              Code: 1,
          },
      }
      command terminated with exit code 1; FAILED mapSharedData: tshm-stream-a-i
      
  occurred
  In [BeforeEach] at: /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:76

2 tests fail

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/164/pipeline/22/ (3 instead of 4 x6 + pod not found) https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/145/pipeline/22/ (3 instead of 4 x6 + pod not found) https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/129/pipeline/22/ (3 instead of 4 x6 + container not found) https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/124/pipeline/22/ (3 instead of 4 x6 + pod not found) https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/118/pipeline/22/ (code 137 + container not found) https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/96/pipeline/22/ (3 instead of 4 + container not found) https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/89/pipeline/22/ (code 137 + container not found) https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/85/pipeline/22/ (3 instead of 4 + container not found) https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/78/pipeline/22/ (3 instead of 4 + container not found) https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/64/pipeline/22/ (3 instead of 4 + container not found)

LionelJouin avatar Nov 01 '22 16:11 LionelJouin