cloudstack
cloudstack copied to clipboard
Published
20 hours ago •
apache
apache
The Cloudstack Multi VLAN, But one of the Virtual Routers can not work
ISSUE TYPE
- Bug Report
CLOUDSTACK VERSION
- CloudStack:4.15.2.0
- ceph version:15.2.16
- cloudstack agent os version: Oracle Linux Server 8.3
The cloudstack env is using the Ceph RBD and local storage as cloudstack cluster primary storage.
The cloudstack cluster TEST work node is eno49 and eno50 to bond0.
- bond0 ---> bond0.1935 ---> cloudbr0
-------> VLAN 1115
- bond0 ---> cloudbr1 -
-------> VLAN 1118
The details config:
- ifcfg-eno49
TYPE=Ethernet
BOOTPROTO=none
NAME=bond0-slave
MASTER=bond0
SLAVE=yes
DEVICE=eno49
ONBOOT=yes
HOTPLUG=no
DELAY=0
STP=no
- ifcfg-eno50
TYPE=Ethernet
BOOTPROTO=none
NAME=bond0-slave
MASTER=bond0
SLAVE=yes
DEVICE=eno50
ONBOOT=yes
- ifcfg-bond0
DEVICE=bond0
NAME=bond0
TYPE=Bond
BONDING_MASTER=yes
ONBOOT=yes
HOTPLUG=no
BOOTPROTO=none
BRIDGE=cloudbr1
BONDING_OPTS="mode=1 miimon=100"
- ifcfg-bond.1935
DEVICE=bond0.1935
NAME=bond0.1935
ONBOOT=yes
HOTPLUG=no
BOOTPROTO=none
VLAN=yes
BRIDGE=cloudbr0
- ifcfg-cloudbr0
DEVICE=cloudbr0
NAME=cloudbr0
TYPE=Bridge
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPADDR=10.26.246.11
GATEWAY=10.26.246.254
NETMASK=255.255.255.0
DELAY=5
STP=yes
- ifcfg-cloudbr1
DEVICE=cloudbr1
NAME=cloudbr1
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=none
IPV6INIT=no
IPV6_AUTOCONF=no
DELAY=5
STP=yes
The network command:
# nmcli d
DEVICE TYPE STATE CONNECTION
cloudbr0 bridge connected cloudbr0
cloud0 bridge connected (externally) cloud0
brbond0-1115 bridge connected (externally) brbond0-1115
brbond0-1118 bridge connected (externally) brbond0-1118
cloudbr1 bridge connected cloudbr1
bond0 bond connected bond0
eno49 ethernet connected bond0-slave
eno50 ethernet connected bond0-slave
vnet0 tun connected (externally) vnet0
vnet10 tun connected (externally) vnet10
vnet11 tun connected (externally) vnet11
vnet12 tun connected (externally) vnet12
vnet16 tun connected (externally) vnet16
vnet17 tun connected (externally) vnet17
vnet18 tun connected (externally) vnet18
vnet2 tun connected (externally) vnet2
vnet21 tun connected (externally) vnet21
vnet22 tun connected (externally) vnet22
vnet23 tun connected (externally) vnet23
vnet4 tun connected (externally) vnet4
bond0.1115 vlan connected (externally) bond0.1115
bond0.1118 vlan connected (externally) bond0.1118
bond0.1935 vlan connected bond0.1935
lo loopback unmanaged --
# nmcli c
NAME UUID TYPE DEVICE
cloudbr0 d4b789ba-7321-548d-dabd-5c4150da0266 bridge cloudbr0
cloud0 4d28dc6a-dcd4-456d-b6d7-65907a6b2963 bridge cloud0
brbond0-1115 ee9a06a6-c856-478a-9d68-ff7125c45ae6 bridge brbond0-1115
brbond0-1118 d8548bd2-9a02-4f5d-9f19-669e7b46e483 bridge brbond0-1118
cloudbr1 6df8d4a7-7ee1-528a-dcf9-20d35734f675 bridge cloudbr1
bond0 ad33d8b0-1f7b-cab9-9447-ba07f855b143 bond bond0
bond0.1115 58e5fbc3-3571-4c3b-a82f-6e8722fb9b46 vlan bond0.1115
bond0.1118 614180b3-92e1-46e3-8693-e74289807831 vlan bond0.1118
bond0.1935 79f54c00-8b72-f4d7-9ae6-74ea2b65af0a vlan bond0.1935
bond0-slave 31fc52de-447b-46f6-33cf-1d4dbbcddef5 ethernet eno49
bond0-slave 12e7087a-5ced-5405-d350-7726d3570bae ethernet eno50
vnet0 3b8c138f-a27d-4aee-b6c6-b80e307e652d tun vnet0
vnet10 631f7272-a06c-4521-b03e-f4b13643ac3c tun vnet10
vnet11 c0e6aebd-53d3-4214-a6ed-12c224136ba5 tun vnet11
vnet12 061633df-d815-4b91-9373-1098c231e4ec tun vnet12
vnet16 aeee6eba-0da0-49e0-a10d-39a15f0a5d70 tun vnet16
vnet17 13a77f1c-3235-414d-90c9-3be38aadbbfb tun vnet17
vnet18 245fafbc-f80e-46dd-8c92-30daac5989de tun vnet18
vnet2 832d0870-7325-487a-b2c8-10420bddec92 tun vnet2
vnet21 169f8145-7d96-4e38-afcc-6b4398b11513 tun vnet21
vnet22 26b7676b-ec89-4d18-b944-39d4d7ba9244 tun vnet22
vnet23 5ee8c080-21b2-4894-8d08-034609ddcb4b tun vnet23
vnet4 d4108da5-276e-4b37-82d4-ef7411b5d893 tun vnet4
The work node (10.26.246.11) can ping gateway: 10.26.18.254 ( VLAN 1118 ) and 10.26.15.254 ( VLAN 1115 ).
# ping 10.26.18.254 (VLAN 1118)
PING 10.26.18.254 (10.26.18.254) 56(84) bytes of data.
64 bytes from 10.26.18.254: icmp_seq=1 ttl=253 time=0.613 ms
64 bytes from 10.26.18.254: icmp_seq=2 ttl=253 time=0.633 ms
^C
--- 10.26.18.254 ping statistics ---
# ping 10.26.15.254 (VLAN 1115)
PING 10.26.15.254 (10.26.15.254) 56(84) bytes of data.
64 bytes from 10.26.15.254: icmp_seq=1 ttl=253 time=99.10 ms
64 bytes from 10.26.15.254: icmp_seq=2 ttl=253 time=0.627 ms
64 bytes from 10.26.15.254: icmp_seq=3 ttl=253 time=86.5 ms
^C
--- 10.26.15.254 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 23ms
rtt min/avg/max/mdev = 0.627/62.375/99.968/44.006 ms
For system vm (VLAN 1115 and VLAN 1118 ) can ping gateway: 10.26.18.254 ( VLAN 1118 ) and 10.26.15.254 ( VLAN 1115 ).
root@v-25921-VM:~# ping 10.26.15.254
PING 10.26.15.254 (10.26.15.254): 56 data bytes
64 bytes from 10.26.15.254: icmp_seq=0 ttl=255 time=1.070 ms
64 bytes from 10.26.15.254: icmp_seq=1 ttl=255 time=1.358 ms
^C--- 10.26.15.254 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.070/1.214/1.358/0.144 ms
root@v-25921-VM:~# ping 10.26.18.254
PING 10.26.18.254 (10.26.18.254): 56 data bytes
64 bytes from 10.26.18.254: icmp_seq=0 ttl=255 time=115.542 ms
64 bytes from 10.26.18.254: icmp_seq=1 ttl=255 time=1.187 ms
64 bytes from 10.26.18.254: icmp_seq=2 ttl=255 time=105.334 ms
64 bytes from 10.26.18.254: icmp_seq=3 ttl=255 time=1.069 ms
^C--- 10.26.18.254 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.069/55.783/115.542/54.774 ms
root@s-25928-VM:~# ping 10.26.15.254
PING 10.26.15.254 (10.26.15.254): 56 data bytes
64 bytes from 10.26.15.254: icmp_seq=0 ttl=255 time=7.469 ms
64 bytes from 10.26.15.254: icmp_seq=1 ttl=255 time=0.652 ms
^C--- 10.26.15.254 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.652/4.061/7.469/3.409 ms
root@s-25928-VM:~# ping 10.26.18.254
PING 10.26.18.254 (10.26.18.254): 56 data bytes
64 bytes from 10.26.18.254: icmp_seq=0 ttl=255 time=0.750 ms
64 bytes from 10.26.18.254: icmp_seq=1 ttl=255 time=0.683 ms
^C--- 10.26.18.254 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.683/0.717/0.750/0.034 ms
For Virtual route (VLAN 1115) can ping gateway: 10.26.18.254 ( VLAN 1118 ) and 10.26.15.254 ( VLAN 1115 )
root@r-25932-VM:~# ping 10.26.18.254
PING 10.26.18.254 (10.26.18.254): 56 data bytes
64 bytes from 10.26.18.254: icmp_seq=0 ttl=255 time=0.666 ms
64 bytes from 10.26.18.254: icmp_seq=1 ttl=255 time=0.621 ms
^C--- 10.26.18.254 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.621/0.643/0.666/0.023 ms
root@r-25932-VM:~# ping 10.26.15.254
PING 10.26.15.254 (10.26.15.254): 56 data bytes
64 bytes from 10.26.15.254: icmp_seq=0 ttl=255 time=0.663 ms
64 bytes from 10.26.15.254: icmp_seq=1 ttl=255 time=0.709 ms
64 bytes from 10.26.15.254: icmp_seq=2 ttl=255 time=27.521 ms
^C--- 10.26.15.254 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.663/9.631/27.521/12.650 ms
But, For Virtual route (VLAN 1118) can not ping gateway: 10.26.18.254 ( VLAN 1118 ) and 10.26.15.254 ( VLAN 1115 ). The VLAN 1115 works fine, I can create VM in this VLAN, But the VLAN 1118 can not work. Could you please give some advices?
root@r-25938-VM:~/health_checks# ping 10.26.18.254
PING 10.26.18.254 (10.26.18.254): 56 data bytes
92 bytes from r-25938-VM (10.26.18.31): Destination Host Unreachable
92 bytes from r-25938-VM (10.26.18.31): Destination Host Unreachable
92 bytes from r-25938-VM (10.26.18.31): Destination Host Unreachable
92 bytes from r-25938-VM (10.26.18.31): Destination Host Unreachable
^C--- 10.26.18.254 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
root@r-25938-VM:~/health_checks# ping 10.26.15.254
PING 10.26.15.254 (10.26.15.254): 56 data bytes
92 bytes from r-25938-VM (10.26.18.31): Destination Host Unreachable
92 bytes from r-25938-VM (10.26.18.31): Destination Host Unreachable
92 bytes from r-25938-VM (10.26.18.31): Destination Host Unreachable
92 bytes from r-25938-VM (10.26.18.31): Destination Host Unreachable
^C--- 10.26.15.254 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
root@r-25938-VM:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.26.18.31 netmask 255.255.255.0 broadcast 10.26.18.255
ether 1e:00:aa:00:00:65 txqueuelen 1000 (Ethernet)
RX packets 3280 bytes 327616 (319.9 KiB)
RX errors 0 dropped 2844 overruns 0 frame 0
TX packets 11223 bytes 481384 (470.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.165.209 netmask 255.255.0.0 broadcast 169.254.255.255
ether 0e:00:a9:fe:a5:d1 txqueuelen 1000 (Ethernet)
RX packets 4087 bytes 637250 (622.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3965 bytes 706746 (690.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 9417 bytes 935849 (913.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9417 bytes 935849 (913.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@r-25938-VM:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.26.18.254 0.0.0.0 UG 0 0 0 eth0
10.26.18.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
