cloudpods icon indicating copy to clipboard operation
cloudpods copied to clipboard

Published 20 hours ago • verified publisher icon yunionio

[求助/Help] lbagent部署后工作不正常

Open shengxianli opened this issue 1 year ago • 4 comments

负载均衡节点 安装完成后，好像有点问题 安装没有报错 RUNNING HANDLER [utils/config-network-manager : Reload NetworkManager] ********* changed: [10.1.18.62] changed: [10.1.18.63]

RUNNING HANDLER [utils/config-network-manager : Remove immutable flag on /etc/resolv.conf] *** changed: [10.1.18.62] changed: [10.1.18.63]

PLAY RECAP ********************************************************************* 10.1.18.62 : ok=109 changed=49 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0 10.1.18.63 : ok=105 changed=47 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0

[root@lv01 ~]# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Tue 2023-11-28 02:04:21 EST; 1h 30min ago Docs: https://kubernetes.io/docs/ Main PID: 636 (kubelet) Tasks: 21 Memory: 121.5M CGroup: /system.slice/kubelet.service └─636 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=syste...

Nov 28 03:33:49 lv01 kubelet[636]: E1128 03:33:49.661455 636 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Nov 28 03:33:59 lv01 kubelet[636]: E1128 03:33:59.670968 636 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Nov 28 03:34:09 lv01 kubelet[636]: E1128 03:34:09.680287 636 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Nov 28 03:34:19 lv01 kubelet[636]: E1128 03:34:19.532714 636 summary_sys_containers.go:82] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Nov 28 03:34:19 lv01 kubelet[636]: E1128 03:34:19.686270 636 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Nov 28 03:34:29 lv01 kubelet[636]: E1128 03:34:29.695212 636 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Nov 28 03:34:39 lv01 kubelet[636]: E1128 03:34:39.703855 636 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Nov 28 03:34:49 lv01 kubelet[636]: E1128 03:34:49.712971 636 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Nov 28 03:34:59 lv01 kubelet[636]: E1128 03:34:59.721227 636 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Nov 28 03:35:09 lv01 kubelet[636]: E1128 03:35:09.731517 636 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats fo... Hint: Some lines were ellipsized, use -l to show in full. [root@lv01 ~]# echo $KUBECONFIG

[root@lv01 ~]# minikube start -bash: minikube: command not found [root@lv01 ~]# kubectl get pod The connection to the server localhost:8080 was refused - did you specify the right host or port?

image image image image

lishengxian@lishengxiandeMacBook-Pro ~ % ping 10.1.18.61 PING 10.1.18.61 (10.1.18.61): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 R

shengxianli avatar Nov 28 '23 08:11 shengxianli

@shengxianli 请排查该节点对应的lbagent pod的日志是否正常

swordqiu avatar Nov 29 '23 01:11 swordqiu

@swordqiu 您看下是这个吗 lb.log

shengxianli avatar Nov 29 '23 03:11 shengxianli

@shengxianli 看起来是下面这个错误导致的。

[error 2023-11-29 03:33:41 ovnutils.configBridgeMtu.func1(ovnutils.go:39)] configuring mtu fail: ovs-vsctl ovs-vsctl: no row "brvpc" in table Interface : exit status 1, retry...

执行 ip addr 看下现在系统有哪些网卡？

zexi avatar Dec 05 '23 04:12 zexi

@zexi [root@IT-Cloudpods ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000 link/ether 00:16:3e:16:e3:c9 brd ff:ff:ff:ff:ff:ff 3: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether ce:29:00:8f:52:93 brd ff:ff:ff:ff:ff:ff 4: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default link/ether 36:c4:a4:83:08:17 brd ff:ff:ff:ff:ff:ff inet 10.110.223.11/32 brd 10.110.223.11 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.98.239.155/32 brd 10.98.239.155 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.106.190.107/32 brd 10.106.190.107 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.103.201.250/32 brd 10.103.201.250 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.106.60.217/32 brd 10.106.60.217 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.107.243.122/32 brd 10.107.243.122 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.105.138.136/32 brd 10.105.138.136 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.105.61.249/32 brd 10.105.61.249 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.107.8.7/32 brd 10.107.8.7 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.107.86.127/32 brd 10.107.86.127 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.102.179.57/32 brd 10.102.179.57 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.98.215.7/32 brd 10.98.215.7 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.96.0.10/32 brd 10.96.0.10 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.111.118.37/32 brd 10.111.118.37 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.106.88.171/32 brd 10.106.88.171 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.101.27.215/32 brd 10.101.27.215 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.110.34.181/32 brd 10.110.34.181 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.98.155.135/32 brd 10.98.155.135 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.96.171.198/32 brd 10.96.171.198 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.108.102.220/32 brd 10.108.102.220 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.108.148.221/32 brd 10.108.148.221 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.111.57.135/32 brd 10.111.57.135 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.107.21.76/32 brd 10.107.21.76 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.104.71.233/32 brd 10.104.71.233 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.108.245.54/32 brd 10.108.245.54 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.96.0.1/32 brd 10.96.0.1 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.100.139.127/32 brd 10.100.139.127 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.108.100.97/32 brd 10.108.100.97 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.111.188.235/32 brd 10.111.188.235 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.106.48.212/32 brd 10.106.48.212 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.109.140.84/32 brd 10.109.140.84 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.102.199.95/32 brd 10.102.199.95 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.101.89.254/32 brd 10.101.89.254 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.110.213.150/32 brd 10.110.213.150 scope global kube-ipvs0 valid_lft forever preferred_lft forever 5: cali9d6d8e357ba@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0 6: calia9c2dbbc162@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1 7: cali5d346c22ca9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2 8: cali492acb2ec70@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 3 9: cali9223ae2954c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 4 10: cali93f70375fde@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 5 11: calia0e1dda614f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 6 12: cali6bf0c86e1fe@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 7 13: cali7dd554cf202@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 8 14: cali604184d05a9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 9 15: cali2c0e26dd9c1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 10 16: cali3fac22157c9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 11 17: cali34fa9fbbf1f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 12 18: calif85ab853f32@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 13 19: cali516e6b52f94@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 14 20: calibc4c227b359@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 15 21: cali01ce9155b14@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 16 22: cali008753f9c13@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 17 23: cali5dc20f0ff99@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 18 24: cali4f0d7a97c64@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 19 25: cali00bf9b079b1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 20 26: cali1369e847161@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 21 27: cali8d5c24effbd@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 22 28: cali545c4651d46@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 23 29: cali9fc18ed1fb2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 24 30: caliea1eed65843@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 25 31: cali5ad1dfb71ec@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 26 32: cali227c3add973@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 27 33: calib25ded1e444@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 28 34: calic642eb67988@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 29 35: cali13c9a5e94b8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 30 36: calic41bd1eb9bb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 31 37: cali7c90a89311c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 32 38: calicd0be229351@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 33 39: cali78fc7573972@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 34 40: calid255bd89c3c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 35 41: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 5e:0d:28:fd:cb:58 brd ff:ff:ff:ff:ff:ff 42: cali86718d900c4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 36 43: breip: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether ee:ee:ee:ee:ee:f0 brd ff:ff:ff:ff:ff:ff inet 100.64.128.3/17 brd 100.64.255.255 scope global breip valid_lft forever preferred_lft forever 44: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 00:16:3e:16:e3:c9 brd ff:ff:ff:ff:ff:ff inet 10.0.254.17/24 brd 10.0.254.255 scope global br0 valid_lft forever preferred_lft forever inet 10.20.30.1/24 scope global br0 valid_lft forever preferred_lft forever 45: brtap: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether ae:ab:de:cf:bf:49 brd ff:ff:ff:ff:ff:ff 46: brvpc: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 9a:a5:6e:31:98:48 brd ff:ff:ff:ff:ff:ff 47: calid49af0bbfe7@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 37 48: calia6715cd5a5a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 38 49: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 inet 10.40.101.192/32 brd 10.40.101.192 scope global tunl0 valid_lft forever preferred_lft forever 53: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 link/ether aa:87:21:2a:04:a8 brd ff:ff:ff:ff:ff:ff

shengxianli avatar Dec 05 '23 14:12 shengxianli