CI: cilium health-probe failing in v1.13.4 and v1.14.2

[Rammurthy5]

CI failure

it goes on forever. had to manually interrupt to stop.

[=] Test [health]
.⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...
⚠️  cilium-health probe failed: "command terminated with exit code 1", stdout: {"" '\x00' '\x00'}, retrying...

clusters (running clustermesh in eu-west-1, eu-west-2) runs v1.13.4, their sysdump below.

cilium-cli: v0.15.8 compiled with go1.21.0 on darwin/arm64
cilium image (default): v1.14.1
cilium image (stable): v1.14.3
cilium image (running): 1.13.4

cilium-sysdump-20231103-111558.zip cilium-sysdump-20231103-111512.zip

clusters (running clustermesh in eu-west-1, eu-west-2) runs v1.14.2, their sysdump below.

cilium-cli: v0.15.8 compiled with go1.21.0 on darwin/arm64
cilium image (default): v1.14.1
cilium image (stable): v1.14.3
cilium image (running): 1.14.2

cilium-sysdump-20231103-111613.zip cilium-sysdump-20231103-111447.zip

another test failure which is related to pod to external connectivity (1.1.1.1 at 443, 1.0.0.1 at 443)

  [.] Action [to-cidr-external-knp/pod-to-cidr/external-1111-0: cilium-test/client-84bfddc76b-4wq42 (10.0.1.186) -> external-1111 (1.1.1.1:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 --retry 3 --retry-all-errors --retry-delay 3 https://1.1.1.1:443" failed: command terminated with exit code 28
  ℹ️  curl output:
  
  📄 No flows recorded for peer cilium-test/client-84bfddc76b-4wq42 during action external-1111-0
  📄 No flows recorded for peer external-1111 during action external-1111-0

📄 No flows recorded for peer external-1001 during action external-1001-0

  [.] Action [all-ingress-deny-knp/pod-to-cidr/external-1001-1: cilium-test/client2-764b565764-l62ss (10.0.1.175) -> external-1001 (1.0.0.1:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 --retry 3 --retry-all-errors --retry-delay 3 https://1.0.0.1:443" failed: command terminated with exit code 28
  ℹ️  curl output:
  
  📄 No flows recorded for peer cilium-test/client2-764b565764-l62ss during action external-1001-1
  📄 No flows recorded for peer external-1001 during action external-1001-1

[larssb]

Any news on this I see the same issue on Cilium v1.15.3 and cilium-cli v0.16.3. Cilium is running on a K3s v1.28.7+k3s1 cluster.

Thank you.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[github-actions[bot]]

This issue has not seen any activity since it was marked stale. Closing.