cilium icon indicating copy to clipboard operation
cilium copied to clipboard
Published 3 months ago • verified publisher icon cilium

CI: Conformance E2E - allow-all-except-world/pod-to-pod/curl-ipv4-5 - command terminated with exit code 28

Open giorio94 opened this issue 2 years ago • 1 comments

CI failure

Hit on https://github.com/cilium/cilium/pull/29896 Link: https://github.com/cilium/cilium/actions/runs/7258862251/job/19774908559 Sysdump: cilium-sysdump-11-20231219-075521.zip

[=] Test [allow-all-except-world] [5/69]
...........
  ℹ️  📜 Applying CiliumNetworkPolicy 'allow-all-except-world' to namespace 'cilium-test'..
  [-] Scenario [allow-all-except-world/pod-to-pod]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-0: cilium-test/client-75bff5f5b9-j4hcq (10.244.3.250) -> cilium-test/echo-other-node-5c6df4fb89-t78fw (10.244.1.77:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv6-0: cilium-test/client-75bff5f5b9-j4hcq (fd00:10:244:3::4c6d) -> cilium-test/echo-other-node-5c6df4fb89-t78fw (fd00:10:244:1::58f0:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-1: cilium-test/client-75bff5f5b9-j4hcq (10.244.3.250) -> cilium-test/echo-same-node-55fdb9f64c-7sljb (10.244.3.69:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv6-1: cilium-test/client-75bff5f5b9-j4hcq (fd00:10:244:3::4c6d) -> cilium-test/echo-same-node-55fdb9f64c-7sljb (fd00:10:244:3::510f:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-2: cilium-test/client2-88575dbb7-q9d4j (10.244.3.194) -> cilium-test/echo-other-node-5c6df4fb89-t78fw (10.244.1.77:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv6-2: cilium-test/client2-88575dbb7-q9d4j (fd00:10:244:3::df3a) -> cilium-test/echo-other-node-5c6df4fb89-t78fw (fd00:10:244:1::58f0:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-3: cilium-test/client2-88575dbb7-q9d4j (10.244.3.194) -> cilium-test/echo-same-node-55fdb9f64c-7sljb (10.244.3.69:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv6-3: cilium-test/client2-88575dbb7-q9d4j (fd00:10:244:3::df3a) -> cilium-test/echo-same-node-55fdb9f64c-7sljb (fd00:10:244:3::510f:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-4: cilium-test/client3-79bfc664b-5vccj (10.244.1.79) -> cilium-test/echo-other-node-5c6df4fb89-t78fw (10.244.1.77:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv6-4: cilium-test/client3-79bfc664b-5vccj (fd00:10:244:1::d663) -> cilium-test/echo-other-node-5c6df4fb89-t78fw (fd00:10:244:1::58f0:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-5: cilium-test/client3-79bfc664b-5vccj (10.244.1.79) -> cilium-test/echo-same-node-55fdb9f64c-7sljb (10.244.3.69:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://10.244.3.69:8080/" failed: error with exec request (pod=cilium-test/client3-79bfc664b-5vccj, container=client3): command terminated with exit code 28

Relevant hubble flows:

10.244.1.79:41946 172.18.0.2:4000 

Dec 19 07:55:09.395: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) from-endpoint FORWARDED (TCP Flags: SYN)
Dec 19 07:55:09.395: 10.244.1.79:41946 (ID:41368) -> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: SYN)
Dec 19 07:55:09.395: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: SYN)
Dec 19 07:55:09.396: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (host) from-overlay FORWARDED (TCP Flags: SYN)
Dec 19 07:55:09.396: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-overlay FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:09.396: 172.18.0.2:4000 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-overlay FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:09.396: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:09.396: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) from-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.396: 10.244.1.79:41946 (ID:41368) -> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.396: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.397: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) from-endpoint FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.397: 10.244.1.79:41946 (ID:41368) -> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.397: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.397: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (host) from-overlay FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.397: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.397: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (host) from-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.398: 172.18.0.2:4000 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.398: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.399: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.399: 172.18.0.2:4000 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.399: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.399: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) from-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.399: 10.244.1.79:41946 (ID:41368) -> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.399: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.399: 172.18.0.2:4000 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-overlay FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.399: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.399: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) from-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.399: 10.244.1.79:41946 (ID:41368) -> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (host) from-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (host) from-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) from-endpoint FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) -> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (host) from-overlay FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.404: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-overlay FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.404: 172.18.0.2:4000 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-overlay FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.404: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.404: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) from-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.404: 10.244.1.79:41946 (ID:41368) -> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.404: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.404: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (host) from-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:18.741: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-overlay FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:18.742: 172.18.0.2:4000 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-overlay FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:18.742: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:18.742: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) from-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:18.742: 10.244.1.79:41946 (ID:41368) -> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:18.742: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:18.742: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (host) from-overlay FORWARDED (TCP Flags: ACK)
Dec 19 07:55:18.742: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-overlay FORWARDED (TCP Flags: RST)
Dec 19 07:55:18.743: 172.18.0.2:4000 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-overlay FORWARDED (TCP Flags: RST)
Dec 19 07:55:18.743: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-endpoint FORWARDED (TCP Flags: RST)
Dec 19 07:55:19.749: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-overlay FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:19.749: 172.18.0.2:4000 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-overlay FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:19.749: 10.244.1.79:41946 (ID:41368) <- 172.18.0.2:4000 (ID:59980) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:19.749: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) from-endpoint FORWARDED (TCP Flags: RST)
Dec 19 07:55:19.749: 10.244.1.79:41946 (ID:41368) -> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: RST)
Dec 19 07:55:19.749: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (remote-node) to-overlay FORWARDED (TCP Flags: RST)
Dec 19 07:55:19.750: 10.244.1.79:41946 (ID:41368) <> 172.18.0.2:4000 (host) from-overlay FORWARDED (TCP Flags: RST)
10.244.1.79:41946 10.244.3.69:8080 

Dec 19 07:55:09.396: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: SYN)
Dec 19 07:55:09.396: 10.244.3.69:8080 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-endpoint FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:09.396: 10.244.1.79:41946 (ID:41368) <- 10.244.3.69:8080 (ID:59980) to-network FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:09.397: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.397: 10.244.3.69:8080 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.397: 10.244.1.79:41946 (ID:41368) <- 10.244.3.69:8080 (ID:59980) to-network FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.397: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.399: 10.244.3.69:8080 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-endpoint FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.399: 10.244.1.79:41946 (ID:41368) <- 10.244.3.69:8080 (ID:59980) to-network FORWARDED (TCP Flags: ACK, PSH)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:09.400: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.403: 10.244.3.69:8080 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-endpoint FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.404: 10.244.1.79:41946 (ID:41368) <- 10.244.3.69:8080 (ID:59980) to-network FORWARDED (TCP Flags: ACK, FIN)
Dec 19 07:55:09.404: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:18.741: 10.244.1.79:41946 (ID:41368) <> 10.244.3.69:8080 (ID:59980) from-endpoint FORWARDED (TCP Flags: SYN)
Dec 19 07:55:18.741: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) policy-verdict:L3-Only EGRESS ALLOWED (TCP Flags: SYN)
Dec 19 07:55:18.741: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-overlay FORWARDED (TCP Flags: SYN)
Dec 19 07:55:18.741: 10.244.1.79:41946 (ID:41368) <> 10.244.3.69:8080 (ID:59980) to-overlay FORWARDED (TCP Flags: SYN)
Dec 19 07:55:18.741: 10.244.1.79:41946 (ID:41368) <> 10.244.3.69:8080 (ID:59980) from-overlay FORWARDED (TCP Flags: SYN)
Dec 19 07:55:18.741: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) policy-verdict:L3-Only INGRESS ALLOWED (TCP Flags: SYN)
Dec 19 07:55:18.741: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: SYN)
Dec 19 07:55:18.741: 10.244.3.69:8080 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-endpoint FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:18.741: 10.244.1.79:41946 (ID:41368) <- 10.244.3.69:8080 (ID:59980) to-network FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:18.742: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: ACK)
Dec 19 07:55:18.742: 10.244.3.69:8080 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-endpoint FORWARDED (TCP Flags: RST)
Dec 19 07:55:18.742: 10.244.1.79:41946 (ID:41368) <- 10.244.3.69:8080 (ID:59980) to-network FORWARDED (TCP Flags: RST)
Dec 19 07:55:19.748: 10.244.1.79:41946 (ID:41368) <> 10.244.3.69:8080 (ID:59980) from-endpoint FORWARDED (TCP Flags: SYN)
Dec 19 07:55:19.748: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-overlay FORWARDED (TCP Flags: SYN)
Dec 19 07:55:19.748: 10.244.1.79:41946 (ID:41368) <> 10.244.3.69:8080 (ID:59980) to-overlay FORWARDED (TCP Flags: SYN)
Dec 19 07:55:19.749: 10.244.1.79:41946 (ID:41368) <> 10.244.3.69:8080 (ID:59980) from-overlay FORWARDED (TCP Flags: SYN)
Dec 19 07:55:19.749: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) policy-verdict:L3-Only INGRESS ALLOWED (TCP Flags: SYN)
Dec 19 07:55:19.749: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: SYN)
Dec 19 07:55:19.749: 10.244.3.69:8080 (ID:59980) <> 10.244.1.79:41946 (ID:41368) from-endpoint FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:19.749: 10.244.1.79:41946 (ID:41368) <- 10.244.3.69:8080 (ID:59980) to-network FORWARDED (TCP Flags: SYN, ACK)
Dec 19 07:55:19.750: 10.244.1.79:41946 (ID:41368) -> 10.244.3.69:8080 (ID:59980) to-endpoint FORWARDED (TCP Flags: RST)

giorio94 avatar Dec 19 '23 09:12 giorio94

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

github-actions[bot] avatar Feb 18 '24 01:02 github-actions[bot]

This issue has not seen any activity since it was marked stale. Closing.

github-actions[bot] avatar Mar 04 '24 01:03 github-actions[bot]