kuma
kuma copied to clipboard
Published
20 hours ago •
kumahq
kumahq
Slow DP startup time in OpenShift
What happened?
We have some trouble with the startup of our dataplanes. There are always 5 seconds between the request and the response from the control plane with the corresponding bootstrap configuration. As an environment we use RedHat OpenShift as Runtime. So we don't use the init container, but CNI mode. Here's a sample of our sidecar logs:
2022-04-28T07:27:19.104Z INFO dataplane trying to fetch bootstrap configuration from the Control Plane
2022-04-28T07:27:24.135Z INFO kuma-dp.run received bootstrap configuration {"adminPort": 9901}
We managed to retrieve some tracing-information about the http-connection. After a quick look at the logs, it becomes clear that the DNS resolution takes the most significant part of the time.
2022-04-28T08:53:08.017Z INFO kuma-dp.run generating bootstrap configuration
2022-04-28T08:53:08.017Z INFO dataplane trying to fetch bootstrap configuration from the Control Plane
2022-04-28T08:53:08.018Z DEBUG "dns":{"start":"","end":"","host":"","address":null,"error":null},"dial":{"start":"","end":""},"connection":{"time":""},"wrote_all_request_header":{"time":""},"wrote_all_request":{"time":""},"first_received_response_byte":{"time":""}}
2022-04-28T08:53:08.018Z DEBUG dataplane Trace build error:
2022-04-28T08:53:08.018Z DEBUG dataplane === START post request
2022-04-28T08:53:08.018Z DEBUG :: dns start
2022-04-28T08:53:08.018Z INFO dataplane 2022-04-28 08:53:08.018480715 +0000 UTC
2022-04-28T08:53:13.022Z DEBUG :: dns end
2022-04-28T08:53:13.022Z INFO dataplane 2022-04-28 08:53:13.022189486 +0000 UTC
2022-04-28T08:53:13.022Z DEBUG :: dial start
2022-04-28T08:53:13.022Z INFO dataplane 2022-04-28 08:53:13.022275425 +0000 UTC
2022-04-28T08:53:13.025Z DEBUG :: dial end
2022-04-28T08:53:13.025Z INFO dataplane 2022-04-28 08:53:13.025074862 +0000 UTC
2022-04-28T08:53:13.032Z DEBUG :: conn time
2022-04-28T08:53:13.032Z INFO dataplane 2022-04-28 08:53:13.032413881 +0000 UTC
2022-04-28T08:53:13.032Z DEBUG :: wrote all request headers
2022-04-28T08:53:13.032Z INFO dataplane 2022-04-28 08:53:13.032563455 +0000 UTC
2022-04-28T08:53:13.032Z DEBUG :: wrote all request
2022-04-28T08:53:13.032Z INFO dataplane 2022-04-28 08:53:13.032646403 +0000 UTC
2022-04-28T08:53:13.037Z DEBUG :: first received response byte
2022-04-28T08:53:13.037Z INFO dataplane 2022-04-28 08:53:13.037235309 +0000 UTC
2022-04-28T08:53:13.037Z DEBUG dataplane === END post request
We have found that this time can be shortened if you include a DNS config with the deployment. It is therefore probably DNS timeouts, which are set to 5 seconds by default. What we included to deployment is the following:
dnsConfig:
options:
- name: single-request-reopen
- name: timeout
value: '1'
We have already discussed this issue in Slack (with Charly Molter), but now we have decided to open an issue after all. Thanks to you all!
Could you trace the DNS requests maybe? Also is this with ipv4, v6 or dual stack?
The error only appears in IPv4 ONLY mode. IPv6 is completely deactivated in our OpenShift cluster. I don't really know how to trace the DNS queries to Kubernetes internal CoreDNS Server in detail. Do you have any hints for that?
I will also check #2000, maybe it's a similar situation as there.
I was thinking of using tcpdump. Looking at this issue and #2000 I have a strong feeling it might be the same and an engineer is working on the fix this week.
In the dump you'll a first AAAA which times out and then a A that succeeds very quickly.
I just tried the fix of #2000, but it's not working in our case. I continue to observe the 5 second timeout during sidecar launch. I will try to make a network recording as soon as possible in order to be able to provide further information.
Took a while but here's the tcp dump. As you can see, no IPv6 is active:
No. Time Protocol Length Info
27 2.248240 DNS 118 Standard query 0x9ebf A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
28 2.248853 DNS 222 Standard query response 0x9ebf No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
29 2.249118 DNS 114 Standard query 0x4787 A kuma-control-plane.fha-kuma-cp.svc.cluster.local
30 2.249583 DNS 189 Standard query response 0x4787 A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
61 7.249544 DNS 118 Standard query 0x38c4 A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
62 7.250281 DNS 222 Standard query response 0x38c4 No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
63 7.250371 DNS 114 Standard query 0x203a A kuma-control-plane.fha-kuma-cp.svc.cluster.local
64 7.250687 DNS 189 Standard query response 0x203a A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
97 12.250355 DNS 118 Standard query 0xee3d A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
98 12.250822 DNS 222 Standard query response 0xee3d No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
99 12.250962 DNS 114 Standard query 0x35f5 A kuma-control-plane.fha-kuma-cp.svc.cluster.local
100 12.251243 DNS 189 Standard query response 0x35f5 A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
127 17.252332 DNS 118 Standard query 0x2795 A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
128 17.252790 DNS 222 Standard query response 0x2795 No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
129 17.252937 DNS 114 Standard query 0xc968 A kuma-control-plane.fha-kuma-cp.svc.cluster.local
130 17.253199 DNS 189 Standard query response 0xc968 A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
159 22.253419 DNS 118 Standard query 0xf88c A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
160 22.254029 DNS 222 Standard query response 0xf88c No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
161 22.254131 DNS 114 Standard query 0x4cae A kuma-control-plane.fha-kuma-cp.svc.cluster.local
162 22.254508 DNS 189 Standard query response 0x4cae A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
191 27.254300 DNS 118 Standard query 0xc6e8 A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
192 27.254798 DNS 222 Standard query response 0xc6e8 No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
193 27.254929 DNS 114 Standard query 0x1f6c A kuma-control-plane.fha-kuma-cp.svc.cluster.local
194 27.255179 DNS 189 Standard query response 0x1f6c A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
223 32.256109 DNS 118 Standard query 0x985a A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
224 32.256668 DNS 222 Standard query response 0x985a No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
225 32.256757 DNS 114 Standard query 0x8529 A kuma-control-plane.fha-kuma-cp.svc.cluster.local
226 32.257277 DNS 189 Standard query response 0x8529 A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
253 37.258214 DNS 118 Standard query 0x2b60 A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
254 37.258804 DNS 222 Standard query response 0x2b60 No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
255 37.258936 DNS 114 Standard query 0xf5ec A kuma-control-plane.fha-kuma-cp.svc.cluster.local
256 37.259333 DNS 189 Standard query response 0xf5ec A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
289 42.261209 DNS 118 Standard query 0xd657 A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
290 42.261595 DNS 222 Standard query response 0xd657 No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
291 42.261741 DNS 114 Standard query 0xcceb A kuma-control-plane.fha-kuma-cp.svc.cluster.local
292 42.261955 DNS 189 Standard query response 0xcceb A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
306 44.780749 DNS 117 Standard query 0x16a8 A example_example-api_svc_8080.mesh.fha.svc.cluster.local
307 44.781291 DNS 221 Standard query response 0x16a8 No such name A example_example-api_svc_8080.mesh.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
308 44.781403 DNS 221 Standard query response 0x16a8 No such name A example_example-api_svc_8080.mesh.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
322 44.782320 DNS 113 Standard query 0x32f8 A example_example-api_svc_8080.mesh.svc.cluster.local
323 44.782537 DNS 217 Standard query response 0x32f8 No such name A example_example-api_svc_8080.mesh.svc.cluster.local SOA ns.dns.cluster.local OPT
324 44.782603 DNS 217 Standard query response 0x32f8 No such name A example_example-api_svc_8080.mesh.svc.cluster.local SOA ns.dns.cluster.local OPT
338 44.783286 DNS 109 Standard query 0x86d6 A example_example-api_svc_8080.mesh.cluster.local
339 44.783444 DNS 213 Standard query response 0x86d6 No such name A example_example-api_svc_8080.mesh.cluster.local SOA ns.dns.cluster.local OPT
340 44.783506 DNS 213 Standard query response 0x86d6 No such name A example_example-api_svc_8080.mesh.cluster.local SOA ns.dns.cluster.local OPT
354 44.784055 DNS 109 Standard query 0xf274 A example_example-api_svc_8080.mesh.ocp.example.com
355 44.784981 DNS 200 Standard query response 0xf274 No such name A example_example-api_svc_8080.mesh.ocp.example.com SOA dns-1.example.com OPT
356 44.785038 DNS 200 Standard query response 0xf274 No such name A example_example-api_svc_8080.mesh.ocp.example.com SOA dns-1.example.com OPT
370 44.785586 DNS 95 Standard query 0xbfce A example_example-api_svc_8080.mesh
397 46.787634 DNS 95 Standard query 0xbfce A example_example-api_svc_8080.mesh
398 46.787851 DNS 65 Standard query 0xa5fc NS <Root>
399 46.788835 DNS 76 Standard query response 0xa5fc Server failure NS <Root> OPT
400 47.264134 DNS 118 Standard query 0xb0f3 A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local
401 47.264417 DNS 222 Standard query response 0xb0f3 No such name A kuma-control-plane.fha-kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
402 47.264489 DNS 114 Standard query 0x6bea A kuma-control-plane.fha-kuma-cp.svc.cluster.local
403 47.264612 DNS 189 Standard query response 0x6bea A kuma-control-plane.fha-kuma-cp.svc.cluster.local A 172.30.9.205 OPT
Do you also have the app logs with the DNS for this one so we can correlate both? Could you share the /etc/resolvconf too? Maybe we do something wrong...
Hi @subnetmarco, here are some further insights for the latest version 1.7.0:
CONTROLPLANE
see https://gist.github.com/d4kine/4906034d89acda7702722ad1c3d91032 (stored as gist because of length)
DATAPLANE
ksniff-9z8bv kuma-sidecar 2022-07-01T11:43:38.754Z INFO dataplane trying to fetch bootstrap configuration from the Control Plane
ksniff-9z8bv kuma-sidecar 2022-07-01T11:43:43.794Z INFO kuma-dp.run received bootstrap configuration {"adminPort": 9901}
ksniff-9z8bv kuma-sidecar 2022-07-01T11:43:43.795Z INFO kuma-dp.run starting Kuma DP {"version": "1.7.0-amd64"}
ksniff-9z8bv kuma-sidecar 2022-07-01T11:43:43.796Z INFO accesslogs-server starting Access Log Server {"address": "unix:///tmp/kuma-al-ksniff-9z8bv.fha-default.sock"}
ksniff-9z8bv kuma-sidecar 2022-07-01T11:43:43.796Z INFO kuma-dp.run.dns-server configuration saved to a file {"file": "/tmp/kuma-dp-1894034830/Corefile"}
ksniff-9z8bv kuma-sidecar 2022-07-01T11:43:43.796Z INFO kuma-dp.run.dns-server starting DNS Server (coredns) {"args": ["-conf", "/tmp/kuma-dp-1894034830/Corefile", "-quiet"]}
ksniff-9z8bv kuma-sidecar 2022-07-01T11:43:43.797Z INFO metrics-hijacker starting Metrics Hijacker Server {"socketPath": "unix:///tmp/kuma-mh-ksniff-9z8bv.fha-default.sock"}
ksniff-9z8bv kuma-sidecar 2022-07-01T11:43:43.797Z INFO kuma-dp.run.envoy bootstrap configuration saved to a file {"file": "/tmp/kuma-dp-1894034830/bootstrap.yaml"}
ksniff-9z8bv kuma-sidecar 2022-07-01T11:43:43.797Z INFO kuma-dp.run.envoy starting Envoy {"path": "/usr/bin/envoy", "arguments": ["--config-path", "/tmp/kuma-dp-1894034830/bootstrap.yaml", "--drain-time-s", "30", "--disable-hot-restart", "--log-level", "info", "--concurrency", "2"]}
ksniff-9z8bv kuma-sidecar [2022-07-01 11:43:43.831][468717][info][main] [source/server/server.cc:390] initializing epoch 0 (base id=0, hot restart version=disabled)
ksniff-9z8bv kuma-sidecar [2022-07-01 11:43:43.831][468717][info][main] [source/server/server.cc:392] statically linked extensions:
TCP-DUMP
No. Time Source Destination Protocol Length Info
1 2022-07-01 11:43:41.856598 10.131.0.1 10.131.1.253 TCP 80 43478 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667974107 TSecr=0 WS=128
2 2022-07-01 11:43:41.856598 10.131.0.1 10.131.1.253 TCP 80 43476 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667974107 TSecr=0 WS=128
3 2022-07-01 11:43:41.856634 10.131.1.253 10.131.0.1 TCP 80 9901 → 43478 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758309361 TSecr=667974107 WS=128
4 2022-07-01 11:43:41.856635 10.131.1.253 10.131.0.1 TCP 80 9901 → 43476 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758309361 TSecr=667974107 WS=128
5 2022-07-01 11:43:41.856671 10.131.0.1 10.131.1.253 TCP 72 43476 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667974107 TSecr=1758309361
6 2022-07-01 11:43:41.856673 10.131.0.1 10.131.1.253 TCP 72 43478 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667974107 TSecr=1758309361
7 2022-07-01 11:43:41.856857 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
8 2022-07-01 11:43:41.856867 10.131.1.253 10.131.0.1 TCP 72 9901 → 43476 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758309361 TSecr=667974107
9 2022-07-01 11:43:41.856900 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
10 2022-07-01 11:43:41.856915 10.131.1.253 10.131.0.1 TCP 72 9901 → 43478 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758309361 TSecr=667974107
11 2022-07-01 11:43:41.857537 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
12 2022-07-01 11:43:41.857564 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
13 2022-07-01 11:43:41.857772 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
14 2022-07-01 11:43:41.857782 127.0.0.1 127.0.0.1 TCP 72 59532 → 9901 [ACK] Seq=238 Ack=225 Win=1648 Len=0 TSval=2680267144 TSecr=2680267144
15 2022-07-01 11:43:41.857793 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
16 2022-07-01 11:43:41.857798 127.0.0.1 127.0.0.1 TCP 72 60492 → 9901 [ACK] Seq=238 Ack=225 Win=1665 Len=0 TSval=2680267144 TSecr=2680267144
17 2022-07-01 11:43:41.857991 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
18 2022-07-01 11:43:41.858014 10.131.0.1 10.131.1.253 TCP 72 43476 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667974109 TSecr=1758309363
19 2022-07-01 11:43:41.858034 10.131.1.253 10.131.0.1 TCP 72 9901 → 43476 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758309363 TSecr=667974109
20 2022-07-01 11:43:41.858056 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
21 2022-07-01 11:43:41.858072 10.131.0.1 10.131.1.253 TCP 72 43478 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667974109 TSecr=1758309363
22 2022-07-01 11:43:41.858083 10.131.1.253 10.131.0.1 TCP 72 9901 → 43478 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758309363 TSecr=667974109
23 2022-07-01 11:43:41.858198 10.131.0.1 10.131.1.253 TCP 72 43478 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667974109 TSecr=1758309363
24 2022-07-01 11:43:41.858205 10.131.1.253 10.131.0.1 TCP 72 9901 → 43478 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758309363 TSecr=667974109
25 2022-07-01 11:43:41.858280 10.131.0.1 10.131.1.253 TCP 72 43476 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667974109 TSecr=1758309363
26 2022-07-01 11:43:41.858285 10.131.1.253 10.131.0.1 TCP 72 9901 → 43476 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758309363 TSecr=667974109
27 2022-07-01 11:43:42.011320 172.30.224.221 10.131.1.253 TLSv1.2 1627 Application Data
28 2022-07-01 11:43:42.011513 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=1 Ack=1556 Win=1388 Len=0 TSval=844428562 TSecr=1116316566
29 2022-07-01 11:43:42.026435 10.131.1.253 172.30.224.221 TLSv1.2 404 Application Data
30 2022-07-01 11:43:42.026901 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
31 2022-07-01 11:43:42.026940 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
32 2022-07-01 11:43:42.027683 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=333 Ack=1640 Win=1393 Len=0 TSval=844428578 TSecr=1116316582
33 2022-07-01 11:43:43.833163 10.131.1.253 172.30.0.10 DNS 114 Standard query 0x21c5 A kuma-control-plane.kuma-cp.fha.svc.cluster.local
34 2022-07-01 11:43:43.833452 172.30.0.10 10.131.1.253 DNS 218 Standard query response 0x21c5 No such name A kuma-control-plane.kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
35 2022-07-01 11:43:43.833512 10.131.1.253 172.30.0.10 DNS 110 Standard query 0xa24d A kuma-control-plane.kuma-cp.svc.cluster.local
36 2022-07-01 11:43:43.833623 172.30.0.10 10.131.1.253 DNS 181 Standard query response 0xa24d A kuma-control-plane.kuma-cp.svc.cluster.local A 172.30.224.221 OPT
37 2022-07-01 11:43:45.265417 01:01:01:01:01:01 ARP 48 Who has 10.131.1.253? Tell 10.131.1.249
38 2022-07-01 11:43:45.265423 01:01:01:01:01:01 ARP 48 10.131.1.253 is at 01:01:01:01:01:01
39 2022-07-01 11:43:46.856146 10.131.0.1 10.131.1.253 TCP 80 43564 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667979107 TSecr=0 WS=128
40 2022-07-01 11:43:46.856183 10.131.1.253 10.131.0.1 TCP 80 9901 → 43564 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758314361 TSecr=667979107 WS=128
41 2022-07-01 11:43:46.856222 10.131.0.1 10.131.1.253 TCP 72 43564 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667979107 TSecr=1758314361
42 2022-07-01 11:43:46.856403 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
43 2022-07-01 11:43:46.856413 10.131.1.253 10.131.0.1 TCP 72 9901 → 43564 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758314361 TSecr=667979107
44 2022-07-01 11:43:46.856925 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
45 2022-07-01 11:43:46.857222 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
46 2022-07-01 11:43:46.857242 127.0.0.1 127.0.0.1 TCP 72 60492 → 9901 [ACK] Seq=475 Ack=449 Win=1673 Len=0 TSval=2680272144 TSecr=2680272144
47 2022-07-01 11:43:46.857432 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
48 2022-07-01 11:43:46.857467 10.131.0.1 10.131.1.253 TCP 72 43564 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667979108 TSecr=1758314362
49 2022-07-01 11:43:46.857499 10.131.1.253 10.131.0.1 TCP 72 9901 → 43564 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758314362 TSecr=667979108
50 2022-07-01 11:43:46.857684 10.131.0.1 10.131.1.253 TCP 72 43564 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667979108 TSecr=1758314362
51 2022-07-01 11:43:46.857696 10.131.1.253 10.131.0.1 TCP 72 9901 → 43564 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758314362 TSecr=667979108
52 2022-07-01 11:43:46.858123 10.131.0.1 10.131.1.253 TCP 80 43566 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667979109 TSecr=0 WS=128
53 2022-07-01 11:43:46.858151 10.131.1.253 10.131.0.1 TCP 80 9901 → 43566 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758314363 TSecr=667979109 WS=128
54 2022-07-01 11:43:46.858178 10.131.0.1 10.131.1.253 TCP 72 43566 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667979109 TSecr=1758314363
55 2022-07-01 11:43:46.858386 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
56 2022-07-01 11:43:46.858396 10.131.1.253 10.131.0.1 TCP 72 9901 → 43566 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758314363 TSecr=667979109
57 2022-07-01 11:43:46.858792 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
58 2022-07-01 11:43:46.858991 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
59 2022-07-01 11:43:46.859007 127.0.0.1 127.0.0.1 TCP 72 59706 → 9901 [ACK] Seq=238 Ack=225 Win=1640 Len=0 TSval=2680272146 TSecr=2680272145
60 2022-07-01 11:43:46.859154 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
61 2022-07-01 11:43:46.859183 10.131.0.1 10.131.1.253 TCP 72 43566 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667979110 TSecr=1758314364
62 2022-07-01 11:43:46.859210 10.131.1.253 10.131.0.1 TCP 72 9901 → 43566 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758314364 TSecr=667979110
63 2022-07-01 11:43:46.859370 10.131.0.1 10.131.1.253 TCP 72 43566 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667979110 TSecr=1758314364
64 2022-07-01 11:43:46.859382 10.131.1.253 10.131.0.1 TCP 72 9901 → 43566 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758314364 TSecr=667979110
65 2022-07-01 11:43:47.017530 172.30.224.221 10.131.1.253 TLSv1.2 4206 Application Data
66 2022-07-01 11:43:47.017653 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=333 Ack=5774 Win=1381 Len=0 TSval=844433568 TSecr=1116321572
67 2022-07-01 11:43:47.017754 172.30.224.221 10.131.1.253 TLSv1.2 1290 Application Data
68 2022-07-01 11:43:47.024696 10.131.1.253 172.30.224.221 TLSv1.2 366 Application Data
69 2022-07-01 11:43:47.025065 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
70 2022-07-01 11:43:47.025089 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
71 2022-07-01 11:43:47.025219 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=627 Ack=7076 Win=1393 Len=0 TSval=844433576 TSecr=1116321580
72 2022-07-01 11:43:48.835788 10.131.1.253 172.30.0.10 DNS 114 Standard query 0xbd7c A kuma-control-plane.kuma-cp.fha.svc.cluster.local
73 2022-07-01 11:43:48.836236 172.30.0.10 10.131.1.253 DNS 218 Standard query response 0xbd7c No such name A kuma-control-plane.kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
74 2022-07-01 11:43:48.836313 10.131.1.253 172.30.0.10 DNS 110 Standard query 0x6cce A kuma-control-plane.kuma-cp.svc.cluster.local
75 2022-07-01 11:43:48.836559 172.30.0.10 10.131.1.253 DNS 181 Standard query response 0x6cce A kuma-control-plane.kuma-cp.svc.cluster.local A 172.30.224.221 OPT
76 2022-07-01 11:43:51.856410 10.131.0.1 10.131.1.253 TCP 80 43648 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667984107 TSecr=0 WS=128
77 2022-07-01 11:43:51.856450 10.131.1.253 10.131.0.1 TCP 80 9901 → 43648 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758319361 TSecr=667984107 WS=128
78 2022-07-01 11:43:51.856494 10.131.0.1 10.131.1.253 TCP 72 43648 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667984107 TSecr=1758319361
79 2022-07-01 11:43:51.856735 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
80 2022-07-01 11:43:51.856751 10.131.1.253 10.131.0.1 TCP 72 9901 → 43648 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758319361 TSecr=667984107
81 2022-07-01 11:43:51.857034 10.131.0.1 10.131.1.253 TCP 80 43650 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667984107 TSecr=0 WS=128
82 2022-07-01 11:43:51.857065 10.131.1.253 10.131.0.1 TCP 80 9901 → 43650 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758319362 TSecr=667984107 WS=128
83 2022-07-01 11:43:51.857113 10.131.0.1 10.131.1.253 TCP 72 43650 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667984108 TSecr=1758319362
84 2022-07-01 11:43:51.857339 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
85 2022-07-01 11:43:51.857357 10.131.1.253 10.131.0.1 TCP 72 9901 → 43650 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758319362 TSecr=667984108
86 2022-07-01 11:43:51.857806 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
87 2022-07-01 11:43:51.857856 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
88 2022-07-01 11:43:51.858430 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
89 2022-07-01 11:43:51.858465 127.0.0.1 127.0.0.1 TCP 72 59706 → 9901 [ACK] Seq=475 Ack=449 Win=1648 Len=0 TSval=2680277145 TSecr=2680277145
90 2022-07-01 11:43:51.858511 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
91 2022-07-01 11:43:51.858524 127.0.0.1 127.0.0.1 TCP 72 60788 → 9901 [ACK] Seq=238 Ack=225 Win=1632 Len=0 TSval=2680277145 TSecr=2680277145
92 2022-07-01 11:43:51.858742 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
93 2022-07-01 11:43:51.858789 10.131.0.1 10.131.1.253 TCP 72 43648 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667984109 TSecr=1758319363
94 2022-07-01 11:43:51.858832 10.131.1.253 10.131.0.1 TCP 72 9901 → 43648 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758319363 TSecr=667984109
95 2022-07-01 11:43:51.858894 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
96 2022-07-01 11:43:51.858913 10.131.0.1 10.131.1.253 TCP 72 43650 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667984109 TSecr=1758319363
97 2022-07-01 11:43:51.858931 10.131.1.253 10.131.0.1 TCP 72 9901 → 43650 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758319363 TSecr=667984109
98 2022-07-01 11:43:51.858967 10.131.0.1 10.131.1.253 TCP 72 43648 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667984109 TSecr=1758319363
99 2022-07-01 11:43:51.859016 10.131.1.253 10.131.0.1 TCP 72 9901 → 43648 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758319364 TSecr=667984109
100 2022-07-01 11:43:51.859126 10.131.0.1 10.131.1.253 TCP 72 43650 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667984110 TSecr=1758319363
101 2022-07-01 11:43:51.859138 10.131.1.253 10.131.0.1 TCP 72 9901 → 43650 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758319364 TSecr=667984110
102 2022-07-01 11:43:52.024988 172.30.224.221 10.131.1.253 TLSv1.2 2196 Application Data
103 2022-07-01 11:43:52.025091 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=627 Ack=9200 Win=1388 Len=0 TSval=844438576 TSecr=1116326580
104 2022-07-01 11:43:52.050356 10.131.1.253 172.30.224.221 TLSv1.2 273 Application Data
105 2022-07-01 11:43:52.050771 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
106 2022-07-01 11:43:52.050823 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
107 2022-07-01 11:43:52.050839 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=828 Ack=9284 Win=1393 Len=0 TSval=844438601 TSecr=1116326606
108 2022-07-01 11:43:53.837223 10.131.1.253 172.30.0.10 DNS 114 Standard query 0x8fdb A kuma-control-plane.kuma-cp.fha.svc.cluster.local
109 2022-07-01 11:43:53.837554 172.30.0.10 10.131.1.253 DNS 218 Standard query response 0x8fdb No such name A kuma-control-plane.kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
110 2022-07-01 11:43:53.837627 10.131.1.253 172.30.0.10 DNS 110 Standard query 0xe2e1 A kuma-control-plane.kuma-cp.svc.cluster.local
111 2022-07-01 11:43:53.837766 172.30.0.10 10.131.1.253 DNS 181 Standard query response 0xe2e1 A kuma-control-plane.kuma-cp.svc.cluster.local A 172.30.224.221 OPT
112 2022-07-01 11:43:55.506001 10.131.1.253 172.30.224.221 TCP 72 35924 → 5678 [ACK] Seq=1 Ack=1 Win=389 Len=0 TSval=844442057 TSecr=3370369251
113 2022-07-01 11:43:55.506363 172.30.224.221 10.131.1.253 TCP 72 [TCP ACKed unseen segment] 5678 → 35924 [ACK] Seq=0 Ack=2 Win=277 Len=0 TSval=3370384612 TSecr=844426696
114 2022-07-01 11:43:55.506371 10.131.1.253 172.30.224.221 TCP 72 [TCP ACKed unseen segment] [TCP Previous segment not captured] 35924 → 5678 [ACK] Seq=2 Ack=1 Win=389 Len=0 TSval=844442057 TSecr=3370369251
115 2022-07-01 11:43:55.506399 172.30.224.221 10.131.1.253 TCP 72 [TCP ACKed unseen segment] [TCP Previous segment not captured] 5678 → 35924 [ACK] Seq=1 Ack=2 Win=277 Len=0 TSval=3370384612 TSecr=844426696
116 2022-07-01 11:43:56.856100 10.131.0.1 10.131.1.253 TCP 80 43742 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667989107 TSecr=0 WS=128
117 2022-07-01 11:43:56.856129 10.131.1.253 10.131.0.1 TCP 80 9901 → 43742 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758324361 TSecr=667989107 WS=128
118 2022-07-01 11:43:56.856151 10.131.0.1 10.131.1.253 TCP 72 43742 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667989107 TSecr=1758324361
119 2022-07-01 11:43:56.856263 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
120 2022-07-01 11:43:56.856268 10.131.1.253 10.131.0.1 TCP 72 9901 → 43742 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758324361 TSecr=667989107
121 2022-07-01 11:43:56.856366 10.131.0.1 10.131.1.253 TCP 80 43744 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667989107 TSecr=0 WS=128
122 2022-07-01 11:43:56.856375 10.131.1.253 10.131.0.1 TCP 80 9901 → 43744 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758324361 TSecr=667989107 WS=128
123 2022-07-01 11:43:56.856387 10.131.0.1 10.131.1.253 TCP 72 43744 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667989107 TSecr=1758324361
124 2022-07-01 11:43:56.856439 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
125 2022-07-01 11:43:56.856442 10.131.1.253 10.131.0.1 TCP 72 9901 → 43744 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758324361 TSecr=667989107
126 2022-07-01 11:43:56.857658 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
127 2022-07-01 11:43:56.857687 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
128 2022-07-01 11:43:56.858003 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
129 2022-07-01 11:43:56.858021 127.0.0.1 127.0.0.1 TCP 72 60788 → 9901 [ACK] Seq=475 Ack=449 Win=1640 Len=0 TSval=2680282145 TSecr=2680282145
130 2022-07-01 11:43:56.858042 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
131 2022-07-01 11:43:56.858052 127.0.0.1 127.0.0.1 TCP 72 59706 → 9901 [ACK] Seq=712 Ack=673 Win=1657 Len=0 TSval=2680282145 TSecr=2680282145
132 2022-07-01 11:43:56.858304 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
133 2022-07-01 11:43:56.858347 10.131.0.1 10.131.1.253 TCP 72 43742 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667989109 TSecr=1758324363
134 2022-07-01 11:43:56.858379 10.131.1.253 10.131.0.1 TCP 72 9901 → 43742 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758324363 TSecr=667989109
135 2022-07-01 11:43:56.858416 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
136 2022-07-01 11:43:56.858434 10.131.0.1 10.131.1.253 TCP 72 43744 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667989109 TSecr=1758324363
137 2022-07-01 11:43:56.858449 10.131.1.253 10.131.0.1 TCP 72 9901 → 43744 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758324363 TSecr=667989109
138 2022-07-01 11:43:56.858455 10.131.0.1 10.131.1.253 TCP 72 43742 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667989109 TSecr=1758324363
139 2022-07-01 11:43:56.858466 10.131.1.253 10.131.0.1 TCP 72 9901 → 43742 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758324363 TSecr=667989109
140 2022-07-01 11:43:56.858536 10.131.0.1 10.131.1.253 TCP 72 43744 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667989109 TSecr=1758324363
141 2022-07-01 11:43:56.858540 10.131.1.253 10.131.0.1 TCP 72 9901 → 43744 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758324363 TSecr=667989109
142 2022-07-01 11:43:57.030730 172.30.224.221 10.131.1.253 TLSv1.2 4206 Application Data
143 2022-07-01 11:43:57.030757 172.30.224.221 10.131.1.253 TLSv1.2 1290 Application Data
144 2022-07-01 11:43:57.030844 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=828 Ack=14636 Win=1373 Len=0 TSval=844443581 TSecr=1116331586
145 2022-07-01 11:43:57.032744 172.30.224.221 10.131.1.253 TLSv1.2 867 Application Data
146 2022-07-01 11:43:57.038715 10.131.1.253 172.30.224.221 TLSv1.2 366 Application Data
147 2022-07-01 11:43:57.039464 10.131.1.253 172.30.224.221 TLSv1.2 426 Application Data
148 2022-07-01 11:43:57.044991 10.131.1.253 172.30.224.221 TCP 426 [TCP Retransmission] 35942 → 5678 [PSH, ACK] Seq=1122 Ack=15431 Win=1393 Len=354 TSval=844443596 TSecr=1116331588
149 2022-07-01 11:43:57.045134 172.30.224.221 10.131.1.253 TCP 84 5678 → 35942 [ACK] Seq=15431 Ack=1476 Win=1401 Len=0 TSval=1116331600 TSecr=844443589 SLE=1122 SRE=1476
150 2022-07-01 11:43:57.052191 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
151 2022-07-01 11:43:57.052193 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
152 2022-07-01 11:43:57.052253 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=1476 Ack=15515 Win=1393 Len=0 TSval=844443603 TSecr=1116331607
153 2022-07-01 11:43:58.838789 10.131.1.253 172.30.0.10 DNS 114 Standard query 0x7240 A kuma-control-plane.kuma-cp.fha.svc.cluster.local
154 2022-07-01 11:43:58.839423 172.30.0.10 10.131.1.253 DNS 218 Standard query response 0x7240 No such name A kuma-control-plane.kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
155 2022-07-01 11:43:58.839507 10.131.1.253 172.30.0.10 DNS 110 Standard query 0x477a A kuma-control-plane.kuma-cp.svc.cluster.local
156 2022-07-01 11:43:58.839901 172.30.0.10 10.131.1.253 DNS 181 Standard query response 0x477a A kuma-control-plane.kuma-cp.svc.cluster.local A 172.30.224.221 OPT
157 2022-07-01 11:44:01.857060 10.131.0.1 10.131.1.253 TCP 80 43826 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667994107 TSecr=0 WS=128
158 2022-07-01 11:44:01.857098 10.131.1.253 10.131.0.1 TCP 80 9901 → 43826 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758329362 TSecr=667994107 WS=128
159 2022-07-01 11:44:01.857132 10.131.0.1 10.131.1.253 TCP 72 43826 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667994108 TSecr=1758329362
160 2022-07-01 11:44:01.857314 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
161 2022-07-01 11:44:01.857324 10.131.1.253 10.131.0.1 TCP 72 9901 → 43826 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758329362 TSecr=667994108
162 2022-07-01 11:44:01.857402 10.131.0.1 10.131.1.253 TCP 80 43828 → 9901 [SYN] Seq=0 Win=28200 Len=0 MSS=1410 SACK_PERM=1 TSval=667994108 TSecr=0 WS=128
163 2022-07-01 11:44:01.857440 10.131.1.253 10.131.0.1 TCP 80 9901 → 43828 [SYN, ACK] Seq=0 Ack=1 Win=27960 Len=0 MSS=1410 SACK_PERM=1 TSval=1758329362 TSecr=667994108 WS=128
164 2022-07-01 11:44:01.857484 10.131.0.1 10.131.1.253 TCP 72 43828 → 9901 [ACK] Seq=1 Ack=1 Win=28288 Len=0 TSval=667994108 TSecr=1758329362
165 2022-07-01 11:44:01.857875 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
166 2022-07-01 11:44:01.858167 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
167 2022-07-01 11:44:01.858183 127.0.0.1 127.0.0.1 TCP 72 59706 → 9901 [ACK] Seq=949 Ack=897 Win=1665 Len=0 TSval=2680287145 TSecr=2680287145
168 2022-07-01 11:44:01.858362 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
169 2022-07-01 11:44:01.858393 10.131.0.1 10.131.1.253 TCP 72 43826 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667994109 TSecr=1758329363
170 2022-07-01 11:44:01.858419 10.131.1.253 10.131.0.1 TCP 72 9901 → 43826 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758329363 TSecr=667994109
171 2022-07-01 11:44:01.858609 10.131.0.1 10.131.1.253 TCP 72 43826 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667994109 TSecr=1758329363
172 2022-07-01 11:44:01.858621 10.131.1.253 10.131.0.1 TCP 72 9901 → 43826 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758329363 TSecr=667994109
173 2022-07-01 11:44:01.858672 10.131.0.1 10.131.1.253 HTTP 181 GET /ready HTTP/1.1
174 2022-07-01 11:44:01.858702 10.131.1.253 10.131.0.1 TCP 72 9901 → 43828 [ACK] Seq=1 Ack=110 Win=28032 Len=0 TSval=1758329363 TSecr=667994109
175 2022-07-01 11:44:01.859043 127.0.0.1 127.0.0.1 HTTP 309 GET /ready HTTP/1.1
176 2022-07-01 11:44:01.859214 127.0.0.1 127.0.0.1 HTTP 296 HTTP/1.1 200 OK (text/plain)
177 2022-07-01 11:44:01.859363 10.131.1.253 10.131.0.1 HTTP 349 HTTP/1.1 200 OK (text/plain)
178 2022-07-01 11:44:01.859404 10.131.0.1 10.131.1.253 TCP 72 43828 → 9901 [ACK] Seq=110 Ack=278 Win=29312 Len=0 TSval=667994110 TSecr=1758329364
179 2022-07-01 11:44:01.859431 10.131.1.253 10.131.0.1 TCP 72 9901 → 43828 [FIN, ACK] Seq=278 Ack=110 Win=28032 Len=0 TSval=1758329364 TSecr=667994110
180 2022-07-01 11:44:01.859584 10.131.0.1 10.131.1.253 TCP 72 43828 → 9901 [FIN, ACK] Seq=110 Ack=279 Win=29312 Len=0 TSval=667994110 TSecr=1758329364
181 2022-07-01 11:44:01.859595 10.131.1.253 10.131.0.1 TCP 72 9901 → 43828 [ACK] Seq=279 Ack=111 Win=28032 Len=0 TSval=1758329364 TSecr=667994110
182 2022-07-01 11:44:01.900010 127.0.0.1 127.0.0.1 TCP 72 59706 → 9901 [ACK] Seq=1186 Ack=1121 Win=1673 Len=0 TSval=2680287187 TSecr=2680287146
183 2022-07-01 11:44:02.055175 172.30.224.221 10.131.1.253 TLSv1.2 2196 Application Data
184 2022-07-01 11:44:02.055279 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=1476 Ack=17639 Win=1388 Len=0 TSval=844448606 TSecr=1116336610
185 2022-07-01 11:44:02.067329 10.131.1.253 172.30.224.221 TLSv1.2 263 Application Data
186 2022-07-01 11:44:02.067798 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
187 2022-07-01 11:44:02.067862 172.30.224.221 10.131.1.253 TLSv1.2 114 Application Data
188 2022-07-01 11:44:02.067892 10.131.1.253 172.30.224.221 TCP 72 35942 → 5678 [ACK] Seq=1667 Ack=17723 Win=1393 Len=0 TSval=844448618 TSecr=1116336623
Additionally here is the resolv.conf:
search fha.svc.cluster.local svc.cluster.local cluster.local ocp.example.com
nameserver 172.30.0.10
options ndots:5
Sorry for the late reply..
@FabianHardt looking at the tcpdump there are no more weird wrong host:
153 2022-07-01 11:43:58.838789 10.131.1.253 172.30.0.10 DNS 114 Standard query 0x7240 A kuma-control-plane.kuma-cp.fha.svc.cluster.local
154 2022-07-01 11:43:58.839423 172.30.0.10 10.131.1.253 DNS 218 Standard query response 0x7240 No such name A kuma-control-plane.kuma-cp.fha.svc.cluster.local SOA ns.dns.cluster.local OPT
155 2022-07-01 11:43:58.839507 10.131.1.253 172.30.0.10 DNS 110 Standard query 0x477a A kuma-control-plane.kuma-cp.svc.cluster.local
156 2022-07-01 11:43:58.839901 172.30.0.10 10.131.1.253 DNS 181 Standard query response 0x477a A kuma-control-plane.kuma-cp.svc.cluster.local A 172.30.224.221 OPT
Looks like DNS resolves as expected now no? However, there's still 5s delay which I don't make sense of still.
If not can you show the dump of service and endpoint(slice) for the kuma-control-plane k8s service?
Actually just noticed the tcpdump doesn't cover the 5s pause we observe in the DP startup logs.
Indeed, we've also noticed the missing timestamp and tried to fetch again as much information as possible. Sadly we can't gather more information with ksniff at a sooner timestamp (2 seconds delay) as you can see in the attached file below.
Could you maybe delay the start of the dp by a few seconds with: https://kuma.io/docs/dev/explore/dpp-on-kubernetes/#custom-container-configuration ?
With the ContainerPatch CRD activated, the config will be fetched immediately while kuma-dp start... We assume, that the control-plane has enough time to gather information for the new dataplane and prepare everything in advance. Is this scenario possible? If we reduce the initial delay by a 1 second sleep timer, everything will start up fine. Seems like there is no DNS-issue involved in this case and may rely on the CNI.
2022-07-06T08:34:46.149Z waiting 5 seconds...
2022-07-06T08:39:46.150Z INFO Skipping reading config from file
2022-07-06T08:39:46.150Z INFO kuma-dp.run effective configuration {"config": "{\"controlPlane\":{\"caCert\":\"-----BEGIN CERTIFICATE-----\\nMIIDDzCCAfegAwIBAgIQF2QGMNhWYNDk4GSereetUTANBgkqhkiG9w0BAQsFADAS\\nMRAwDgYDVQQDEwdrdW1hLWNhMB4XDTIyMDcwMTA5MDUyMloXDTMyMDYyODA5MDUy\\nMlowEjEQMA4GA1UEAxMHa3VtYS1jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC\\nAQoCggEBALpcT2PGkJ2FtqbdQ0JX61vIQ6lUTIlG8t6/g1MDZ65q6krOwzEnMnXv\\nfpA1sSWDanV862xp0Rk2Rlc8xddvNgVJ8Cw1xnLnnnXfk7pr+lezDAO6WfG3AUWH\\nPZ5tttsXLY92UcOXW3PycKNfxj8qSvALsaQovfYEjcAw6BBf6AtpZiwpxirAFGDF\\nENaJJoXaa5KwxLWVuv54PbsX5d+7edsLIikTvy1Swmb1/zuZ1MG/yoKclyo3EEuy\\nY9zkUfW/kC5eUw97vACdEU+GDit3IsgHPFziYrutrVfy7ByPCHdh0Po9OSPQu17j\\neeWHrHMgRBChi/UoyphLWF0igoQYIfcCAwEAAaNhMF8wDgYDVR0PAQH/BAQDAgKk\\nMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAPBgNVHRMBAf8EBTADAQH/\\nMB0GA1UdDgQWBBRHkLsrbOOw4YEyk7G/6qE0yotupDANBgkqhkiG9w0BAQsFAAOC\\nAQEAo5Wty+gAtiJSjQmd8A+LQumyO6UTplzQjlVJjMJuLwztNzhK8VV4YBv2ilZF\\nj7bBhxwQuimz9BuYD8dClmeVAbRVP0CRZ4HBGTJK4eccOzL86EyjsAnRgFm53xuE\\njWYKCoPRRO...
2022-07-06T08:39:46.150Z INFO kuma-dp.run generated configurations will be stored in a temporary directory {"dir": "/tmp/kuma-dp-684685739"}
2022-07-06T08:39:46.190Z INFO kuma-dp.run Envoy version incompatible {"expected": "~1.21.1", "current": "1.22.1"}
2022-07-06T08:39:46.190Z INFO kuma-dp.run fetched Envoy version {"version": {"Build":"ae27fb5280d30e1400b7e9c9cbd448bfcd4ad9f5/1.22.1/Modified/RELEASE/BoringSSL","Version":"1.22.1","KumaDpCompatible":false}}
2022-07-06T08:39:46.191Z INFO kuma-dp.run generating bootstrap configuration
2022-07-06T08:39:46.191Z INFO dataplane trying to fetch bootstrap configuration from the Control Plane
2022-07-06T08:39:46.208Z INFO kuma-dp.run received bootstrap configuration {"adminPort": 9901}
2022-07-06T08:39:46.210Z INFO kuma-dp.run starting Kuma DP {"version": "1.7.0-amd64"}
The bootstrap is fairly static while it's not impossible I'd be surprised if CP being slow in the problem here.
Are you saying this is something you are not observing with the CNI but it happens with init-containers? If yes could you maybe compare the log between init-container and sidecar start (without the containerPatch). My gut feeling is that you are hitting: #2483
Unfortunately, since we use OpenShift, we cannot rely on the init container because iptables are not supported here. Therefore, we only have the option to work with CNI.
But it is conceivable that the delay is caused by the unlocking of the network, as described in #2483. Unfortunately, I have no idea how to further debug this behavior.
Thx @FabianHardt that's already a bunch of info here. I'm going to loop back with the team to see what they think.
Triage: Can you also send logs of CoreDNS in the cluster when the bootstrap request is executed? You would have to change the logging settings in CoreDNS ConfigMap.
@FabianHardt Can you also send logs of CoreDNS in the cluster when the bootstrap request is executed? You would have to change the logging settings in CoreDNS ConfigMap. I think this piece of information might be really helpful.
Sorry for the late reply. Here are the logs of our CoreDNS. From my point of view, requests that should not be here but should already be resolved in the sidecar (240.X.X.X) also end up here.
[ERROR] plugin/errors: 2 . NS: read udp 10.130.3.74:36475->X.X.X.140:53: i/o timeout
[INFO] 10.130.3.72:42790 - 48683 "PTR IN 35.0.0.240.in-addr.arpa. udp 52 false 512" - - 0 6.002862281s
[ERROR] plugin/errors: 2 35.0.0.240.in-addr.arpa. PTR: read udp 10.130.3.74:47187->X.X.X.140:53: i/o timeout
[INFO] 10.130.3.76:52688 - 14939 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000268847s
[INFO] 10.130.3.72:57793 - 37322 "A IN kuma-control-plane.kuma-cp.namespace.svc.cluster.local. udp 89 false 512" NXDOMAIN qr,aa,rd 171 0.000301503s
[INFO] 10.130.3.72:33505 - 61148 "NS IN . udp 28 false 512" - - 0 6.002759062s
[ERROR] plugin/errors: 2 . NS: read udp 10.130.3.74:42962->X.X.X.140:53: i/o timeout
[INFO] 10.130.3.72:54148 - 48683 "PTR IN 35.0.0.240.in-addr.arpa. udp 52 false 512" - - 0 6.003738909s
[ERROR] plugin/errors: 2 35.0.0.240.in-addr.arpa. PTR: read udp 10.130.3.74:52709->X.X.X.140:53: i/o timeout
[INFO] 10.130.2.2:47795 - 15518 "AAAA IN kubernetes.default.svc.openshift-logging.svc.cluster.local. udp 87 false 512" NXDOMAIN qr,aa,rd 169 0.000173907s
[INFO] 10.130.2.2:47795 - 53666 "A IN kubernetes.default.svc.openshift-logging.svc.cluster.local. udp 87 false 512" NXDOMAIN qr,aa,rd 169 0.00024682s
[INFO] 10.130.2.2:60697 - 414 "AAAA IN kubernetes.default.svc.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 151 0.000238133s
[INFO] 10.130.2.2:60697 - 58256 "A IN kubernetes.default.svc.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 151 0.000296786s
[INFO] 10.130.2.2:43702 - 46097 "AAAA IN kubernetes.default.svc.cluster.local. udp 65 false 512" NOERROR qr,aa,rd 147 0.000146928s
[INFO] 10.130.2.2:40350 - 6893 "A IN kubernetes.default.svc.openshift-logging.svc.cluster.local. udp 87 false 512" NXDOMAIN qr,aa,rd 169 0.000165731s
[INFO] 10.130.2.2:40350 - 46574 "AAAA IN kubernetes.default.svc.openshift-logging.svc.cluster.local. udp 87 false 512" NXDOMAIN qr,aa,rd 169 0.00029576s
[INFO] 10.130.2.2:45318 - 34730 "AAAA IN kubernetes.default.svc.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 151 0.000051976s
[INFO] 10.130.2.2:45318 - 25513 "A IN kubernetes.default.svc.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 151 0.000100435s
[INFO] 10.130.2.2:39101 - 58281 "AAAA IN kubernetes.default.svc.cluster.local. udp 65 false 512" NOERROR qr,aa,rd 147 0.000155806s
[INFO] 10.130.2.46:37044 - 59384 "A IN external-service.example.com.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000199466s
[INFO] 10.130.2.46:37044 - 23958 "A IN external-service.example.com.cluster.local. udp 73 false 512" NXDOMAIN qr,aa,rd 155 0.000208816s
[INFO] 10.130.2.46:37044 - 46368 "A IN external-service.example.com.ocp4.example.com. udp 74 false 512" NXDOMAIN qr,aa,rd,ra 153 0.000118227s
[INFO] 10.130.3.76:33595 - 12401 "A IN external-service-02.example.com.fha.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 156 0.000316282s
[INFO] 10.130.3.76:33595 - 59374 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000133761s
[INFO] 10.130.3.76:33595 - 28592 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000135724s
[INFO] 10.130.3.76:33595 - 35844 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000102311s
[INFO] 10.130.3.72:57742 - 48683 "PTR IN 35.0.0.240.in-addr.arpa. udp 52 false 512" - - 0 6.002205805s
[ERROR] plugin/errors: 2 35.0.0.240.in-addr.arpa. PTR: read udp 10.130.3.74:50306->X.X.X.140:53: i/o timeout
[INFO] 10.130.3.77:47857 - 19099 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000249699s
[INFO] 10.130.3.77:36313 - 25451 "AAAA IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000466381s
[INFO] 10.130.3.77:58480 - 7244 "AAAA IN kuma-control-plane.kuma-cp.svc.cluster.local. udp 73 false 512" NOERROR qr,aa,rd 155 0.000327229s
[INFO] 10.130.3.77:39598 - 52265 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000150599s
[INFO] 10.130.3.72:34826 - 16981 "A IN external-service-02.example.com.namespace.svc.cluster.local. udp 86 false 512" NXDOMAIN qr,aa,rd 168 0.000259705s
[INFO] 10.130.3.72:34826 - 15186 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000232759s
[INFO] 10.130.3.72:34826 - 56443 "A IN external-service-03.example.com.namespace.svc.cluster.local. udp 79 false 512" NXDOMAIN qr,aa,rd 161 0.000147418s
[INFO] 10.130.2.46:46673 - 18861 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000196792s
[INFO] 10.130.2.46:46673 - 37170 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.00019888s
[INFO] 10.130.3.72:34826 - 16887 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000087904s
[INFO] 10.130.3.72:34826 - 9907 "A IN external-service-03.example.com.svc.cluster.local. udp 63 false 512" NXDOMAIN qr,aa,rd 145 0.000219039s
[INFO] 10.130.3.72:34826 - 24443 "A IN external-service-03.example.com.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 141 0.000159942s
[INFO] 10.130.2.46:46673 - 61763 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000990348s
[INFO] 10.130.3.72:34826 - 41891 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.00085846s
[INFO] 10.130.3.72:34826 - 25571 "A IN external-service-03.example.com.ocp4.example.com. udp 60 false 512" NXDOMAIN qr,aa,rd,ra 139 0.001095668s
[INFO] 10.130.2.1:19019 - 56077 "A IN image-registry.openshift-image-registry.svc.cluster.local. udp 98 false 512" NXDOMAIN qr,aa,rd 168 0.000366758s
[INFO] 10.130.2.1:41136 - 7509 "AAAA IN image-registry.openshift-image-registry.svc.cluster.local. udp 98 false 512" NXDOMAIN qr,aa,rd 168 0.000322953s
[INFO] 10.130.3.76:60970 - 13757 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000319469s
[INFO] 10.130.3.72:51682 - 1175 "A IN kuma-control-plane.kuma-cp.namespace.svc.cluster.local. udp 89 false 512" NXDOMAIN qr,aa,rd 171 0.00027552s
[INFO] 10.130.3.77:57001 - 44381 "A IN external-service-02.example.com.fha.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 156 0.000128329s
[INFO] 10.130.3.77:57001 - 27643 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000146443s
[INFO] 10.130.3.77:57001 - 38836 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000130605s
[INFO] 10.130.3.77:57001 - 57666 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000138133s
[INFO] 10.130.2.46:39478 - 64319 "A IN external-service.example.com.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000316928s
[INFO] 10.130.2.46:39478 - 48995 "A IN external-service.example.com.cluster.local. udp 73 false 512" NXDOMAIN qr,aa,rd 155 0.000234891s
[INFO] 10.130.2.46:39478 - 42133 "A IN external-service.example.com.ocp4.example.com. udp 74 false 512" NXDOMAIN qr,aa,rd,ra 153 0.000726482s
[INFO] 10.130.3.76:33837 - 38302 "A IN external-service-02.example.com.fha.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 156 0.000229284s
[INFO] 10.130.3.76:33837 - 37011 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000124648s
[INFO] 10.130.3.76:33837 - 39764 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000124031s
[INFO] 10.130.3.76:33837 - 57793 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.00011737s
[INFO] 10.130.3.77:40194 - 52208 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000177413s
[INFO] 10.130.3.72:38330 - 35227 "A IN external-service-03.example.com.namespace.svc.cluster.local. udp 79 false 512" NXDOMAIN qr,aa,rd 161 0.000191764s
[INFO] 10.130.3.72:38330 - 8640 "A IN external-service-02.example.com.namespace.svc.cluster.local. udp 86 false 512" NXDOMAIN qr,aa,rd 168 0.000211687s
[INFO] 10.130.3.72:38330 - 26828 "A IN external-service-03.example.com.svc.cluster.local. udp 63 false 512" NXDOMAIN qr,aa,rd 145 0.000221747s
[INFO] 10.130.3.72:38330 - 28597 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000459159s
[INFO] 10.130.3.72:38330 - 22368 "A IN external-service-03.example.com.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 141 0.000163027s
[INFO] 10.130.2.46:50357 - 50311 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000154366s
[INFO] 10.130.3.72:38330 - 25322 "A IN external-service-03.example.com.ocp4.example.com. udp 60 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000048032s
[INFO] 10.130.3.72:38330 - 31619 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000121857s
[INFO] 10.130.2.46:50357 - 63904 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000124118s
[INFO] 10.130.3.72:38330 - 33198 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000040963s
[INFO] 10.130.2.46:50357 - 28845 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.00008349s
[INFO] 10.130.3.76:60325 - 57472 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000307396s
[INFO] 10.130.3.72:36898 - 30993 "A IN kuma-control-plane.kuma-cp.namespace.svc.cluster.local. udp 89 false 512" NXDOMAIN qr,aa,rd 171 0.000243007s
[INFO] 10.130.3.77:56922 - 35180 "A IN external-service-02.example.com.fha.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 156 0.0001685s
[INFO] 10.130.3.77:56922 - 19693 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000098216s
[INFO] 10.130.3.77:56922 - 62655 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000112116s
[INFO] 10.130.3.77:56922 - 38021 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000056611s
[INFO] 10.130.2.46:56301 - 65494 "A IN external-service.example.com.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000246028s
[INFO] 10.130.2.46:56301 - 47600 "A IN external-service.example.com.cluster.local. udp 73 false 512" NXDOMAIN qr,aa,rd 155 0.000196472s
[INFO] 10.130.2.46:56301 - 42272 "A IN external-service.example.com.ocp4.example.com. udp 74 false 512" NXDOMAIN qr,aa,rd,ra 153 0.00014659s
[INFO] 10.130.3.76:40236 - 16786 "A IN external-service-02.example.com.fha.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 156 0.000226333s
[INFO] 10.130.3.76:40236 - 43227 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.00013453s
[INFO] 10.130.3.76:40236 - 59687 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.00011447s
[INFO] 10.130.3.76:40236 - 14745 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000077262s
[INFO] 10.130.3.77:57801 - 55982 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000181806s
[INFO] 10.130.3.72:39884 - 34109 "A IN external-service-03.example.com.namespace.svc.cluster.local. udp 79 false 512" NXDOMAIN qr,aa,rd 161 0.000235914s
[INFO] 10.130.3.72:39884 - 38987 "A IN external-service-02.example.com.namespace.svc.cluster.local. udp 86 false 512" NXDOMAIN qr,aa,rd 168 0.00030137s
[INFO] 10.130.3.72:39884 - 16561 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000106398s
[INFO] 10.130.2.46:52075 - 21295 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000128253s
[INFO] 10.130.3.72:39884 - 17195 "A IN external-service-03.example.com.svc.cluster.local. udp 63 false 512" NXDOMAIN qr,aa,rd 145 0.000153456s
[INFO] 10.130.2.46:52075 - 17007 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000098192s
[INFO] 10.130.3.72:39884 - 25579 "A IN external-service-03.example.com.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 141 0.000140177s
[INFO] 10.130.3.72:39884 - 50086 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000157678s
[INFO] 10.130.3.72:39884 - 10318 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000115921s
[INFO] 10.130.2.46:52075 - 50729 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.0001084s
[INFO] 10.130.3.72:39884 - 8091 "A IN external-service-03.example.com.ocp4.example.com. udp 60 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000112938s
[INFO] 10.130.3.76:54088 - 26462 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000270177s
[INFO] 10.130.3.72:52873 - 21830 "A IN kuma-control-plane.kuma-cp.namespace.svc.cluster.local. udp 89 false 512" NXDOMAIN qr,aa,rd 171 0.000201889s
[INFO] 10.130.3.77:55599 - 44344 "A IN external-service-02.example.com.fha.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 156 0.000170829s
[INFO] 10.130.3.77:55599 - 61504 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000121079s
[INFO] 10.130.3.77:55599 - 63730 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000131459s
[INFO] 10.130.3.77:55599 - 44535 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000149484s
[INFO] 10.130.2.46:34004 - 34729 "A IN external-service.example.com.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000243629s
[INFO] 10.130.2.46:34004 - 9870 "A IN external-service.example.com.cluster.local. udp 73 false 512" NXDOMAIN qr,aa,rd 155 0.000206191s
[INFO] 10.130.2.46:34004 - 12611 "A IN external-service.example.com.ocp4.example.com. udp 74 false 512" NXDOMAIN qr,aa,rd,ra 153 0.000151782s
[INFO] 10.130.3.76:33208 - 49350 "A IN external-service-02.example.com.fha.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 156 0.000252533s
[INFO] 10.130.3.76:33208 - 44955 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000129001s
[INFO] 10.130.3.76:33208 - 25039 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000092515s
[INFO] 10.130.3.76:33208 - 30916 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000078515s
[INFO] 10.130.3.77:60313 - 19509 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.00011466s
[INFO] 10.130.3.72:56297 - 53364 "A IN external-service-02.example.com.namespace.svc.cluster.local. udp 86 false 512" NXDOMAIN qr,aa,rd 168 0.00024792s
[INFO] 10.130.3.72:56297 - 56305 "A IN external-service-03.example.com.namespace.svc.cluster.local. udp 79 false 512" NXDOMAIN qr,aa,rd 161 0.000202954s
[INFO] 10.130.3.72:56297 - 36452 "A IN external-service-03.example.com.svc.cluster.local. udp 63 false 512" NXDOMAIN qr,aa,rd 145 0.000138089s
[INFO] 10.130.3.72:56297 - 50493 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000268574s
[INFO] 10.130.3.72:56297 - 16373 "A IN external-service-03.example.com.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 141 0.000158668s
[INFO] 10.130.3.72:56297 - 48878 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.00014953s
[INFO] 10.130.3.72:56297 - 27517 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000090033s
[INFO] 10.130.2.46:47286 - 40443 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000091908s
[INFO] 10.130.3.72:56297 - 24779 "A IN external-service-03.example.com.ocp4.example.com. udp 60 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000116778s
[INFO] 10.130.2.46:47286 - 63430 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000096234s
[INFO] 10.130.2.46:47286 - 34481 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000070922s
[INFO] 10.130.3.76:45473 - 40930 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000193792s
[INFO] 10.130.3.72:39392 - 49475 "A IN kuma-control-plane.kuma-cp.namespace.svc.cluster.local. udp 89 false 512" NXDOMAIN qr,aa,rd 171 0.000277633s
[INFO] 10.130.3.77:45040 - 19636 "A IN external-service-02.example.com.fha.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 156 0.000102158s
[INFO] 10.130.3.77:45040 - 37802 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000118284s
[INFO] 10.130.3.77:45040 - 15379 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000112555s
[INFO] 10.130.3.77:45040 - 57732 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000082536s
[INFO] 10.130.3.72:33773 - 18857 "A IN kerberos-01.example.com.namespace.svc.cluster.local. udp 84 false 512" NXDOMAIN qr,aa,rd 166 0.000230774s
[INFO] 10.130.3.72:33773 - 16749 "A IN kerberos-01.example.com.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 150 0.000208102s
[INFO] 10.130.3.72:33773 - 54321 "A IN kerberos-01.example.com.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 146 0.000169656s
[INFO] 10.130.3.72:33773 - 21093 "A IN kerberos-01.example.com.ocp4.example.com. udp 65 false 512" NXDOMAIN qr,aa,rd,ra 144 0.001420811s
[INFO] 10.130.2.46:52620 - 12734 "A IN external-service.example.com.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000223538s
[INFO] 10.130.2.46:52620 - 34500 "A IN external-service.example.com.cluster.local. udp 73 false 512" NXDOMAIN qr,aa,rd 155 0.000179544s
[INFO] 10.130.2.46:52620 - 13486 "A IN external-service.example.com.ocp4.example.com. udp 74 false 512" NXDOMAIN qr,aa,rd,ra 153 0.000085793s
[INFO] 10.130.3.76:60732 - 33362 "A IN external-service-02.example.com.fha.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 156 0.000227489s
[INFO] 10.130.3.76:60732 - 10653 "A IN external-service-02.example.com.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 152 0.000114139s
[INFO] 10.130.3.76:60732 - 9369 "A IN external-service-02.example.com.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 148 0.000106565s
[INFO] 10.130.3.76:60732 - 39080 "A IN external-service-02.example.com.ocp4.example.com. udp 67 false 512" NXDOMAIN qr,aa,rd,ra 146 0.000114938s
[INFO] 10.130.3.77:47601 - 53460 "A IN kuma-control-plane.kuma-cp.fha.svc.cluster.local. udp 77 false 512" NXDOMAIN qr,aa,rd 159 0.000161145s
Triage: let's try to reproduce this on our side Things to check. Logs of bootstrap xds handler to see if CP processes the request for 5s or is it on dp side
@FabianHardt I'm wondering if maybe https://github.com/gliderlabs/docker-alpine/issues/539 is at play.
There's been a new implementation of our CNI could you please try to repro with this version? https://kuma.io/docs/2.0.x/networking/cni/#kuma-cni-v2
@FabianHardt any updates here? Have you had a chance to check out the CNI plugin?
Unfortunately we haven't gotten to that yet. We would like to test this as part of the Kuma update to version 2.0. But that won't happen until the new year. Sorry, but I hope we can be patient for a few more days?
Unfortunately we haven't gotten to that yet. We would like to test this as part of the Kuma update to version 2.0. But that won't happen until the new year. Sorry, but I hope we can be patient for a few more days?
@FabianHardt sure, take your time with this
Hello, everyone, here is some feedback about this issue. I updated Kuma to the new version 2.0.1 just over a week ago and have been monitoring our environment ever since. It actually looks good now! The delays and waiting times when starting and especially when resolving DNS are no longer visible. The update of the CNI really seems to have brought an improvement here.
The wait was worth it. Thanks for your support and ideas! From my point of view, the issue can be closed.