calico
calico copied to clipboard
Published
1 month ago
•
projectcalico
projectcalico
Calico's performance is not as expected and seems unstable on my environment.
Expected Behavior
Under normal circumstances, the network performance loss of Calico should be within 10%, right?
Current Behavior
Calico's performance is not as expected and seems unstable. How can I further analyze the cause of network degradation and solve this problem?
The network loss from testing on nodes to pods is over 50%！！！
Possible Solution
Have I missed any important system kernel parameters or Calico configurations?
Steps to Reproduce (for bugs)
Context
[root@master-1 ~]# kubectl get pods -o wide | grep iperf
iperf3-0 1/1 Running 0 63m 10.233.97.252 worker-3 <none> <none>
iperf3-1 1/1 Running 0 63m 10.233.36.107 master-0 <none> <none>
iperf3-2 1/1 Running 0 63m 10.233.226.112 worker-1 <none> <none>
iperf3-3 1/1 Running 0 63m 10.233.36.108 master-0 <none> <none>
[root@master-1 ~]# calicoctl get workloadEndpoint -o wide | grep iperf
worker--3-k8s-iperf3--0-eth0 iperf3-0 worker-3 10.233.97.252/32 caliee510e5ea76 kns.default,ksa.default.default
master--0-k8s-iperf3--1-eth0 iperf3-1 master-0 10.233.36.107/32 cali9d99c50de22 kns.default,ksa.default.default
worker--1-k8s-iperf3--2-eth0 iperf3-2 worker-1 10.233.226.112/32 cali9431011ac1f kns.default,ksa.default.default
master--0-k8s-iperf3--3-eth0 iperf3-3 master-0 10.233.36.108/32 cali78a252f89c2 kns.default,ksa.default.default
1、node to node：
# master-0（10.83.3.48） --> worker-1（10.83.3.52）
[root@master-0 ~]# iperf3 -c worker-1 -i 1 -t 15
Connecting to host worker-1, port 5201
[ 5] local 10.83.3.48 port 47406 connected to 10.83.3.52 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.10 GBytes 9.43 Gbits/sec 75 1.43 MBytes
[ 5] 1.00-2.00 sec 1.09 GBytes 9.41 Gbits/sec 23 1.43 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.44 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.44 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.45 MBytes
[ 5] 5.00-6.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.47 MBytes
[ 5] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.49 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.53 MBytes
[ 5] 8.00-9.00 sec 1.09 GBytes 9.36 Gbits/sec 308 1.14 MBytes
[ 5] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.37 MBytes
[ 5] 10.00-11.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.39 MBytes
[ 5] 11.00-12.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.40 MBytes
[ 5] 12.00-13.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.41 MBytes
[ 5] 13.00-14.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.41 MBytes
[ 5] 14.00-15.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.66 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 16.4 GBytes 9.41 Gbits/sec 406 sender
[ 5] 0.00-15.04 sec 16.4 GBytes 9.38 Gbits/sec receiver
iperf Done.
2、pod to node:
# iperf3-1 --> worker-1
[root@master-0 ~]# kubectl exec -ti iperf3-1 -- bash
root@iperf3-1:/# iperf3 -c 10.83.3.52 -i 1 -t 15
Connecting to host 10.83.3.52, port 5201
[ 5] local 10.233.36.107 port 44216 connected to 10.83.3.52 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 632 MBytes 5.30 Gbits/sec 777 682 KBytes
[ 5] 1.00-2.00 sec 556 MBytes 4.67 Gbits/sec 32 724 KBytes
[ 5] 2.00-3.00 sec 544 MBytes 4.56 Gbits/sec 35 1.01 MBytes
[ 5] 3.00-4.00 sec 522 MBytes 4.38 Gbits/sec 44 1.02 MBytes
[ 5] 4.00-5.00 sec 465 MBytes 3.90 Gbits/sec 7 1.02 MBytes
[ 5] 5.00-6.00 sec 461 MBytes 3.87 Gbits/sec 63 984 KBytes
[ 5] 6.00-7.00 sec 496 MBytes 4.16 Gbits/sec 11 954 KBytes
[ 5] 7.00-8.00 sec 532 MBytes 4.47 Gbits/sec 10 974 KBytes
[ 5] 8.00-9.00 sec 491 MBytes 4.12 Gbits/sec 7 1010 KBytes
[ 5] 9.00-10.00 sec 522 MBytes 4.38 Gbits/sec 14 1.02 MBytes
[ 5] 10.00-11.00 sec 531 MBytes 4.46 Gbits/sec 8 1.08 MBytes
[ 5] 11.00-12.00 sec 415 MBytes 3.48 Gbits/sec 3 1.06 MBytes
[ 5] 12.00-13.00 sec 564 MBytes 4.73 Gbits/sec 4 830 KBytes
[ 5] 13.00-14.00 sec 492 MBytes 4.13 Gbits/sec 71 847 KBytes
[ 5] 14.00-15.00 sec 505 MBytes 4.24 Gbits/sec 1 930 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 7.55 GBytes 4.32 Gbits/sec 1087 sender
[ 5] 0.00-15.04 sec 7.55 GBytes 4.31 Gbits/sec receiver
iperf Done.
root@iperf3-1:/# iperf3 -c 10.83.3.52 -i 1 -t 15
Connecting to host 10.83.3.52, port 5201
[ 5] local 10.233.36.107 port 46434 connected to 10.83.3.52 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 980 MBytes 8.22 Gbits/sec 1529 839 KBytes
[ 5] 1.00-2.00 sec 605 MBytes 5.08 Gbits/sec 261 888 KBytes
[ 5] 2.00-3.00 sec 572 MBytes 4.80 Gbits/sec 17 1.12 MBytes
[ 5] 3.00-4.00 sec 658 MBytes 5.52 Gbits/sec 183 1.02 MBytes
[ 5] 4.00-5.00 sec 609 MBytes 5.11 Gbits/sec 81 813 KBytes
[ 5] 5.00-6.00 sec 661 MBytes 5.55 Gbits/sec 28 990 KBytes
[ 5] 6.00-7.00 sec 569 MBytes 4.77 Gbits/sec 14 1.04 MBytes
[ 5] 7.00-8.00 sec 509 MBytes 4.27 Gbits/sec 12 1.06 MBytes
[ 5] 8.00-9.00 sec 991 MBytes 8.31 Gbits/sec 2 1.39 MBytes
[ 5] 9.00-10.00 sec 831 MBytes 6.97 Gbits/sec 196 1.09 MBytes
[ 5] 10.00-11.00 sec 846 MBytes 7.10 Gbits/sec 60 1022 KBytes
[ 5] 11.00-12.00 sec 586 MBytes 4.92 Gbits/sec 30 1.07 MBytes
[ 5] 12.00-13.00 sec 568 MBytes 4.76 Gbits/sec 711 945 KBytes
[ 5] 13.00-14.00 sec 726 MBytes 6.09 Gbits/sec 8 1.10 MBytes
[ 5] 14.00-15.00 sec 562 MBytes 4.72 Gbits/sec 4 858 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 10.0 GBytes 5.75 Gbits/sec 3136 sender
[ 5] 0.00-15.04 sec 10.0 GBytes 5.73 Gbits/sec receiver
iperf Done.
root@iperf3-1:/# iperf3 -c 10.83.3.52 -i 1 -t 15
Connecting to host 10.83.3.52, port 5201
[ 5] local 10.233.36.107 port 47750 connected to 10.83.3.52 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 636 MBytes 5.33 Gbits/sec 958 796 KBytes
[ 5] 1.00-2.00 sec 585 MBytes 4.91 Gbits/sec 18 1.05 MBytes
[ 5] 2.00-3.00 sec 509 MBytes 4.27 Gbits/sec 7 1.04 MBytes
[ 5] 3.00-4.00 sec 544 MBytes 4.56 Gbits/sec 23 1.04 MBytes
[ 5] 4.00-5.00 sec 450 MBytes 3.77 Gbits/sec 16 1.03 MBytes
[ 5] 5.00-6.00 sec 521 MBytes 4.37 Gbits/sec 3 1.07 MBytes
[ 5] 6.00-7.00 sec 471 MBytes 3.95 Gbits/sec 7 1.03 MBytes
[ 5] 7.00-8.00 sec 586 MBytes 4.92 Gbits/sec 13 796 KBytes
[ 5] 8.00-9.00 sec 555 MBytes 4.66 Gbits/sec 6 888 KBytes
[ 5] 9.00-10.00 sec 601 MBytes 5.04 Gbits/sec 8 1010 KBytes
[ 5] 10.00-11.00 sec 506 MBytes 4.25 Gbits/sec 35 1024 KBytes
[ 5] 11.00-12.00 sec 571 MBytes 4.79 Gbits/sec 12 1.05 MBytes
[ 5] 12.00-13.00 sec 662 MBytes 5.56 Gbits/sec 19 1.10 MBytes
[ 5] 13.00-14.00 sec 480 MBytes 4.03 Gbits/sec 4 803 KBytes
[ 5] 14.00-15.00 sec 509 MBytes 4.27 Gbits/sec 2 871 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 8.00 GBytes 4.58 Gbits/sec 1131 sender
[ 5] 0.00-15.04 sec 7.99 GBytes 4.57 Gbits/sec receiver
iperf Done.
root@iperf3-1:/# iperf3 -c 10.83.3.52 -i 1 -t 15
Connecting to host 10.83.3.52, port 5201
[ 5] local 10.233.36.107 port 47384 connected to 10.83.3.52 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 801 MBytes 6.72 Gbits/sec 1954 665 KBytes
[ 5] 1.00-2.00 sec 820 MBytes 6.88 Gbits/sec 153 836 KBytes
[ 5] 2.00-3.00 sec 888 MBytes 7.45 Gbits/sec 571 1.13 MBytes
[ 5] 3.00-4.00 sec 815 MBytes 6.84 Gbits/sec 309 911 KBytes
[ 5] 4.00-5.00 sec 581 MBytes 4.88 Gbits/sec 5 981 KBytes
[ 5] 5.00-6.00 sec 599 MBytes 5.02 Gbits/sec 13 1.06 MBytes
[ 5] 6.00-7.00 sec 605 MBytes 5.08 Gbits/sec 34 865 KBytes
[ 5] 7.00-8.00 sec 772 MBytes 6.48 Gbits/sec 7 1.10 MBytes
[ 5] 8.00-9.00 sec 978 MBytes 8.20 Gbits/sec 178 1.00 MBytes
[ 5] 9.00-10.00 sec 605 MBytes 5.08 Gbits/sec 25 1.11 MBytes
[ 5] 10.00-11.00 sec 542 MBytes 4.55 Gbits/sec 15 837 KBytes
[ 5] 11.00-12.00 sec 556 MBytes 4.67 Gbits/sec 17 926 KBytes
[ 5] 12.00-13.00 sec 562 MBytes 4.72 Gbits/sec 69 994 KBytes
[ 5] 13.00-14.00 sec 532 MBytes 4.47 Gbits/sec 420 1.07 MBytes
[ 5] 14.00-15.00 sec 519 MBytes 4.35 Gbits/sec 6 1.10 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 9.94 GBytes 5.69 Gbits/sec 3776 sender
[ 5] 0.00-15.04 sec 9.93 GBytes 5.67 Gbits/sec receiver
iperf Done.
3、node to pod：
# worker-1（10.83.3.52） --> iperf3-1（10.233.36.107）
[root@worker-1 ~]# iperf3 -c 10.233.36.107 -i 1 -t 15
Connecting to host 10.233.36.107, port 5201
[ 5] local 10.83.3.52 port 60422 connected to 10.233.36.107 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.09 GBytes 9.37 Gbits/sec 14 1.24 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.38 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 12 1.40 MBytes
[ 5] 3.00-4.00 sec 1.04 GBytes 8.94 Gbits/sec 1438 1.02 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.38 MBytes
[ 5] 5.00-6.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.39 MBytes
[ 5] 6.00-7.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.64 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.90 MBytes
[ 5] 8.00-9.00 sec 1.09 GBytes 9.33 Gbits/sec 0 1.90 MBytes
[ 5] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.90 MBytes
[ 5] 10.00-11.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.90 MBytes
[ 5] 11.00-12.00 sec 1.09 GBytes 9.41 Gbits/sec 43 1.90 MBytes
[ 5] 12.00-13.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.90 MBytes
[ 5] 13.00-14.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.90 MBytes
[ 5] 14.00-15.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.90 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 16.4 GBytes 9.37 Gbits/sec 1507 sender
[ 5] 0.00-15.04 sec 16.4 GBytes 9.35 Gbits/sec receiver
iperf Done.
4、pod to pod：
# iperf3-1（10.233.36.107） --> iperf3-2（10.233.226.112）
[root@master-0 ~]# kubectl exec -ti iperf3-1 -- bash
root@iperf3-1:/# iperf3 -c 10.233.226.112 -i 1 -t 15
Connecting to host 10.233.226.112, port 5201
[ 5] local 10.233.36.107 port 53670 connected to 10.233.226.112 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 867 MBytes 7.27 Gbits/sec 3080 587 KBytes
[ 5] 1.00-2.00 sec 825 MBytes 6.92 Gbits/sec 1864 551 KBytes
[ 5] 2.00-3.00 sec 654 MBytes 5.48 Gbits/sec 1865 448 KBytes
[ 5] 3.00-4.00 sec 680 MBytes 5.70 Gbits/sec 2474 416 KBytes
[ 5] 4.00-5.00 sec 684 MBytes 5.74 Gbits/sec 2649 431 KBytes
[ 5] 5.00-6.00 sec 556 MBytes 4.67 Gbits/sec 2480 238 KBytes
[ 5] 6.00-7.00 sec 685 MBytes 5.75 Gbits/sec 2410 229 KBytes
[ 5] 7.00-8.00 sec 740 MBytes 6.21 Gbits/sec 2563 580 KBytes
[ 5] 8.00-9.00 sec 638 MBytes 5.35 Gbits/sec 1373 211 KBytes
[ 5] 9.00-10.00 sec 652 MBytes 5.47 Gbits/sec 1347 547 KBytes
[ 5] 10.00-11.00 sec 761 MBytes 6.39 Gbits/sec 1816 631 KBytes
[ 5] 11.00-12.00 sec 761 MBytes 6.39 Gbits/sec 2100 472 KBytes
[ 5] 12.00-13.00 sec 706 MBytes 5.92 Gbits/sec 2034 609 KBytes
[ 5] 13.00-14.00 sec 710 MBytes 5.96 Gbits/sec 3791 329 KBytes
[ 5] 14.00-15.00 sec 749 MBytes 6.28 Gbits/sec 684 658 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 10.4 GBytes 5.97 Gbits/sec 32530 sender
[ 5] 0.00-15.04 sec 10.4 GBytes 5.95 Gbits/sec receiver
iperf Done.
root@iperf3-1:/# iperf3 -c 10.233.226.112 -i 1 -t 15
Connecting to host 10.233.226.112, port 5201
[ 5] local 10.233.36.107 port 35270 connected to 10.233.226.112 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 709 MBytes 5.95 Gbits/sec 2567 509 KBytes
[ 5] 1.00-2.00 sec 639 MBytes 5.36 Gbits/sec 44 918 KBytes
[ 5] 2.00-3.00 sec 582 MBytes 4.89 Gbits/sec 54 602 KBytes
[ 5] 3.00-4.00 sec 620 MBytes 5.20 Gbits/sec 79 416 KBytes
[ 5] 4.00-5.00 sec 675 MBytes 5.66 Gbits/sec 46 776 KBytes
[ 5] 5.00-6.00 sec 591 MBytes 4.96 Gbits/sec 33 445 KBytes
[ 5] 6.00-7.00 sec 528 MBytes 4.42 Gbits/sec 24 619 KBytes
[ 5] 7.00-8.00 sec 570 MBytes 4.78 Gbits/sec 95 448 KBytes
[ 5] 8.00-9.00 sec 619 MBytes 5.19 Gbits/sec 14 612 KBytes
[ 5] 9.00-10.00 sec 639 MBytes 5.36 Gbits/sec 42 536 KBytes
[ 5] 10.00-11.00 sec 585 MBytes 4.91 Gbits/sec 66 560 KBytes
[ 5] 11.00-12.00 sec 569 MBytes 4.77 Gbits/sec 54 560 KBytes
[ 5] 12.00-13.00 sec 636 MBytes 5.34 Gbits/sec 23 611 KBytes
[ 5] 13.00-14.00 sec 538 MBytes 4.51 Gbits/sec 29 700 KBytes
[ 5] 14.00-15.00 sec 631 MBytes 5.30 Gbits/sec 143 491 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 8.92 GBytes 5.11 Gbits/sec 3313 sender
[ 5] 0.00-15.04 sec 8.91 GBytes 5.09 Gbits/sec receiver
iperf Done.
Your Environment
- Calico version
[root@master-0 ~]# calicoctl version
Client Version: v3.27.2
Git commit: 402c0b381
Cluster Version: v3.27.2
Cluster Type: typha,kdd,k8s,operator,bgp,kubeadm
- Orchestrator version (e.g. kubernetes, mesos, rkt):
[root@master-0 ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.12", GitCommit:"12031002905c0410706974560cbdf2dad9278919", GitTreeState:"clean", BuildDate:"2024-03-15T02:15:31Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.12", GitCommit:"12031002905c0410706974560cbdf2dad9278919", GitTreeState:"clean", BuildDate:"2024-03-15T02:06:14Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}
- Operating System and version:
[root@master-0 ~]# uname -a
Linux master-0 4.19.90-52.33.v2207.ky10.x86_64 #1 SMP Fri Dec 22 17:04:59 CST 2023 x86_64 x86_64 x86_64 GNU/Linux
- Testing process and results:
[root@worker-0 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: p1p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bond state UP group default qlen 1000
link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
3: em1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 9c:c2:c4:5f:0f:aa brd ff:ff:ff:ff:ff:ff
4: p1p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bond state UP group default qlen 1000
link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
5: p5p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bond state UP group default qlen 1000
link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
6: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 9c:c2:c4:5f:0f:ab brd ff:ff:ff:ff:ff:ff
7: p5p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master nm-bond state UP group default qlen 1000
link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
8: nm-bond: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 9c:c2:c4:55:f6:4a brd ff:ff:ff:ff:ff:ff
inet 10.83.3.51/24 brd 10.83.3.255 scope global noprefixroute nm-bond
valid_lft forever preferred_lft forever
[root@worker-0 ~]# ethtool nm-bond
Settings for nm-bond:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 40000Mb/s
Duplex: Full
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Link detected: yes
[root@worker-0 ~]# ethtool -k nm-bond
Features for nm-bond:
rx-checksumming: off [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
[root@master-1 ~]# kubectl get felixconfigurations.projectcalico.org default -o yaml
apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
creationTimestamp: "2024-04-24T12:35:04Z"
generation: 1
name: default
resourceVersion: "3943999"
uid: cbd24723-21c4-4afd-81cd-5d8b4eb8a184
spec:
bpfConnectTimeLoadBalancing: TCP
bpfHostNetworkedNATWithoutCTLB: Enabled
bpfLogLevel: ""
floatingIPs: Disabled
healthPort: 9099
logSeverityScreen: Info
mtuIfacePattern: nm-bond
reportingInterval: 0s
vxlanVNI: 4096
[root@master-1 ~]# kubectl get installations.operator.tigera.io default -o yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
annotations:
meta.helm.sh/release-name: calico
meta.helm.sh/release-namespace: tigera-operator
creationTimestamp: "2024-04-24T12:31:14Z"
finalizers:
- tigera.io/operator-cleanup
generation: 7
labels:
app.kubernetes.io/managed-by: Helm
operator.dameng.com/instance: calico
operator.dameng.com/set: calico
name: default
resourceVersion: "4025413"
uid: 44ed66f9-f993-4a42-a23e-b28564ecac4d
spec:
calicoNetwork:
bgp: Enabled
hostPorts: Enabled
ipPools:
- blockSize: 26
cidr: 10.233.0.0/16
disableBGPExport: false
encapsulation: None
natOutgoing: Enabled
nodeSelector: all()
linuxDataplane: Iptables
multiInterfaceMode: None
nodeAddressAutodetectionV4:
kubernetes: NodeInternalIP
windowsDataplane: Disabled
cni:
ipam:
type: Calico
type: Calico
controlPlaneReplicas: 2
flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
imagePullSecrets: []
kubeletVolumePluginPath: /var/lib/kubelet
kubernetesProvider: ""
logging:
cni:
logFileMaxAgeDays: 30
logFileMaxCount: 10
logFileMaxSize: 100Mi
logSeverity: Info
nodeUpdateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
nonPrivileged: Disabled
variant: Calico
status:
calicoVersion: v3.27.2
computed:
calicoNetwork:
bgp: Enabled
hostPorts: Enabled
ipPools:
- blockSize: 26
cidr: 10.233.0.0/16
disableBGPExport: false
encapsulation: None
natOutgoing: Enabled
nodeSelector: all()
linuxDataplane: Iptables
multiInterfaceMode: None
nodeAddressAutodetectionV4:
kubernetes: NodeInternalIP
windowsDataplane: Disabled
cni:
ipam:
type: Calico
type: Calico
controlPlaneReplicas: 2
flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
kubeletVolumePluginPath: /var/lib/kubelet
logging:
cni:
logFileMaxAgeDays: 30
logFileMaxCount: 10
logFileMaxSize: 100Mi
logSeverity: Info
nodeUpdateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
nonPrivileged: Disabled
variant: Calico
conditions:
- lastTransitionTime: "2024-04-25T05:14:57Z"
message: All Objects Available
observedGeneration: 7
reason: AllObjectsAvailable
status: "False"
type: Degraded
- lastTransitionTime: "2024-04-25T05:14:57Z"
message: All objects available
observedGeneration: 7
reason: AllObjectsAvailable
status: "True"
type: Ready
- lastTransitionTime: "2024-04-25T05:14:57Z"
message: All Objects Available
observedGeneration: 7
reason: AllObjectsAvailable
status: "False"
type: Progressing
mtu: 1500
variant: Calico
[root@master-1 ~]# cat /etc/cni/net.d/10-calico.conflist | jq
{
"name": "k8s-pod-network",
"cniVersion": "0.3.1",
"plugins": [
{
"container_settings": {
"allow_ip_forwarding": false
},
"datastore_type": "kubernetes",
"ipam": {
"assign_ipv4": "true",
"assign_ipv6": "false",
"type": "calico-ipam"
},
"kubernetes": {
"k8s_api_root": "https://10.96.0.1:443",
"kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
},
"log_file_max_age": 30,
"log_file_max_count": 10,
"log_file_max_size": 100,
"log_file_path": "/var/log/calico/cni/cni.log",
"log_level": "Info",
"mtu": 0,
"nodename_file_optional": false,
"policy": {
"type": "k8s"
},
"type": "calico"
},
{
"capabilities": {
"bandwidth": true
},
"type": "bandwidth"
},
{
"capabilities": {
"portMappings": true
},
"snat": true,
"type": "portmap"
}
]
}
[root@master-0 ~]# ip link show cali9d99c50de22
76833: cali9d99c50de22@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-e677d036-a8cf-3323-ce92-c0682de0a022
[root@master-0 ~]# ethtool -k cali9d99c50de22
Features for cali9d99c50de22:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: off [fixed]
tx-sctp-segmentation: on
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: on
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
Do you use vxlan? Are the nodes in different subnets? I suppose there is offloading turned off on vxlan.calico. It is turned off by default due to a kernel bug in older kernels, but we are turning it on in 3.28 again. You can set "ChecksumOffloadBroken=true" in the FelixConfiguration's featureDetectOverride field. You would need to restart the nodes. You can also manually turn it on using ethtool. Let us know if it helped.
@tomastigera Yes, all nodes are connected to the same switch and VXLAN is not being used. I have set the encapsulation to None. I found in subsequent testing that the total bandwidth can reach the expected value when using concurrency parameters(-P 10), but it is still only about half of the physical bandwidth in single-threaded scenarios.
I also tried using ethtool to disable rx-checksumming and tx-checksumming, but I didn't see any significant change.
I also tried starting two iperf3 containers on the same node to test the same target simultaneously. I expected the results of the two iperf3 containers to add up to the physical bandwidth. However, in fact, the test results of each iperf3 were lower. I can't think of where the problem might be. I thought that in the case of "encapsulation: None", calico only needs to maintain the local routing table and veth pair, and there should not be such a large difference with the physical network.
Sorry for not getting back to you. Did you make any progress? I think it is expected that you can only use half the phys bandwidth with a bond in most modes/setups since your flow would only fo through one phys device. And you get full bw with multiple flows. Calico doesn't really have much to do with it.
In response to the needs of this environment, I replaced other CSI plugin.