Meridio
Meridio copied to clipboard
Published
20 hours ago •
Nordix
Nordix
Some targets are not receiving traffic after scaling
Describe the bug The issue is similar to this one: https://github.com/Nordix/Meridio/issues/55 After scaling the targets, for instance from 4 to 5, some targets are not receiving traffic for x seconds while they are correctly configured in nfqlb with correct IP rules and IP routes. It's random, most of the time it works correctly, sometimes only 1 is receiving traffic, sometimes 2.
I tried with ctraffic and mconnect, the result is the same.
To Reproduce
- Scale out targets
- Wait x (like 20) seconds
- Send traffic
Expected behavior All targets should receive the traffic.
Context
- Kernel: 5.15
- Kubernetes: v1.23.4
- Spire: v1.2.2
- Network Service Mesh: v1.3.1
- Meridio: v0.5.1
Running on Kind, I haven't tried any other environment, but it happened to the internal CI team.
Logs
time=2022-06-03T11:08:45+02:00 level=info msg=load-balancer-trench-a-666f6bc986-99kpp: <nil> - Shm: tshm-stream-a
Fw: own=0
Maglev: M=9973, N=100
Lookup: 30 68 68 57 16 16 16 75 30 16 16 16 75 16 16 75 30 16 16 16 75 16 30 75 68...
Active: 17(16) 31(30) 58(57) 69(68) 76(75)
time=2022-06-03T11:08:45+02:00 level=info msg=load-balancer-trench-a-666f6bc986-99kpp: <nil> - 0: from all lookup local
93: from all fwmark 0x11 lookup 17
94: from all fwmark 0x3a lookup 58
95: from all fwmark 0x1f lookup 31
98: from all fwmark 0x45 lookup 69
99: from all fwmark 0x4c lookup 76
100: from 20.0.0.1 lookup 4096
100: from 40.0.0.0/24 lookup 4096
32766: from all lookup main
32767: from all lookup default
time=2022-06-03T11:08:45+02:00 level=info msg=load-balancer-trench-a-666f6bc986-99kpp: <nil> - default via 169.254.100.150 dev ext-vlan.100 table 4096 proto bird metric 32
default via 172.16.0.50 dev load-balan-861a table 17
default via 172.16.0.36 dev load-balan-861a table 31
default via 172.16.1.44 dev load-balan-5f4e table 58
default via 172.16.0.6 dev load-balan-861a table 69
default via 172.16.1.6 dev load-balan-5f4e table 76
default via 10.244.2.1 dev eth0
10.244.2.0/24 via 10.244.2.1 dev eth0 src 10.244.2.5
10.244.2.1 dev eth0 scope link src 10.244.2.5
169.254.100.0/24 dev ext-vlan.100 proto kernel scope link src 169.254.100.2
172.16.0.0/24 dev load-balan-861a proto kernel scope link src 172.16.0.2
172.16.1.0/24 dev load-balan-5f4e proto kernel scope link src 172.16.1.4
local 10.244.2.5 dev eth0 table local proto kernel scope host src 10.244.2.5
broadcast 10.244.2.255 dev eth0 table local proto kernel scope link src 10.244.2.5
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 169.254.100.2 dev ext-vlan.100 table local proto kernel scope host src 169.254.100.2
broadcast 169.254.100.255 dev ext-vlan.100 table local proto kernel scope link src 169.254.100.2
local 172.16.0.2 dev load-balan-861a table local proto kernel scope host src 172.16.0.2
broadcast 172.16.0.255 dev load-balan-861a table local proto kernel scope link src 172.16.0.2
local 172.16.1.4 dev load-balan-5f4e table local proto kernel scope host src 172.16.1.4
broadcast 172.16.1.255 dev load-balan-5f4e table local proto kernel scope link src 172.16.1.4
default via 100:100::150 dev ext-vlan.100 table 4096 proto bird metric 32 pref medium
default via fd00::32 dev load-balan-861a table 17 metric 1024 pref medium
default via fd00::24 dev load-balan-861a table 31 metric 1024 pref medium
default via fd00:0:0:1::2c dev load-balan-5f4e table 58 metric 1024 pref medium
default via fd00::6 dev load-balan-861a table 69 metric 1024 pref medium
default via fd00:0:0:1::6 dev load-balan-5f4e table 76 metric 1024 pref medium
100:100::/64 dev ext-vlan.100 proto kernel metric 256 pref medium
fd00::/64 dev load-balan-861a proto kernel metric 256 pref medium
fd00:0:0:1::/64 dev load-balan-5f4e proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev ext-vlan.100 proto kernel metric 256 pref medium
fe80::/64 dev load-balan-861a proto kernel metric 256 pref medium
fe80::/64 dev load-balan-5f4e proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast 100:100:: dev ext-vlan.100 table local proto kernel metric 0 pref medium
local 100:100::2 dev ext-vlan.100 table local proto kernel metric 0 pref medium
anycast fd00:: dev load-balan-861a table local proto kernel metric 0 pref medium
local fd00::2 dev load-balan-861a table local proto kernel metric 0 pref medium
anycast fd00:0:0:1:: dev load-balan-5f4e table local proto kernel metric 0 pref medium
local fd00:0:0:1::4 dev load-balan-5f4e table local proto kernel metric 0 pref medium
anycast fe80:: dev eth0 table local proto kernel metric 0 pref medium
anycast fe80:: dev ext-vlan.100 table local proto kernel metric 0 pref medium
anycast fe80:: dev load-balan-5f4e table local proto kernel metric 0 pref medium
anycast fe80:: dev load-balan-861a table local proto kernel metric 0 pref medium
local fe80::42:acff:fe12:3 dev ext-vlan.100 table local proto kernel metric 0 pref medium
local fe80::fe:59ff:febf:9523 dev load-balan-861a table local proto kernel metric 0 pref medium
local fe80::fe:78ff:fef0:9892 dev load-balan-5f4e table local proto kernel metric 0 pref medium
local fe80::54f9:19ff:fe30:e6d1 dev eth0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev ext-vlan.100 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev load-balan-861a table local proto kernel metric 256 pref medium
multicast ff00::/8 dev load-balan-5f4e table local proto kernel metric 256 pref medium
time=2022-06-03T11:08:45+02:00 level=info msg=load-balancer-trench-a-666f6bc986-qjd4m: <nil> - Shm: tshm-stream-a
Fw: own=0
Maglev: M=9973, N=100
Lookup: 30 68 68 57 16 16 16 75 30 16 16 16 75 16 16 75 30 16 16 16 75 16 30 75 68...
Active: 17(16) 31(30) 58(57) 69(68) 76(75)
time=2022-06-03T11:08:45+02:00 level=info msg=load-balancer-trench-a-666f6bc986-qjd4m: <nil> - 0: from all lookup local
93: from all fwmark 0x11 lookup 17
94: from all fwmark 0x3a lookup 58
95: from all fwmark 0x1f lookup 31
98: from all fwmark 0x45 lookup 69
99: from all fwmark 0x4c lookup 76
100: from 20.0.0.1 lookup 4096
100: from 40.0.0.0/24 lookup 4096
32766: from all lookup main
32767: from all lookup default
time=2022-06-03T11:08:45+02:00 level=info msg=load-balancer-trench-a-666f6bc986-qjd4m: <nil> - default via 169.254.100.150 dev ext-vlan.100 table 4096 proto bird metric 32
default via 172.16.0.50 dev load-balan-5a6d table 17
default via 172.16.0.36 dev load-balan-5a6d table 31
default via 172.16.1.44 dev load-balan-3a3f table 58
default via 172.16.0.6 dev load-balan-5a6d table 69
default via 172.16.1.6 dev load-balan-3a3f table 76
default via 10.244.1.1 dev eth0
10.244.1.0/24 via 10.244.1.1 dev eth0 src 10.244.1.7
10.244.1.1 dev eth0 scope link src 10.244.1.7
169.254.100.0/24 dev ext-vlan.100 proto kernel scope link src 169.254.100.1
172.16.0.0/24 dev load-balan-5a6d proto kernel scope link src 172.16.0.4
172.16.1.0/24 dev load-balan-3a3f proto kernel scope link src 172.16.1.2
local 10.244.1.7 dev eth0 table local proto kernel scope host src 10.244.1.7
broadcast 10.244.1.255 dev eth0 table local proto kernel scope link src 10.244.1.7
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 169.254.100.1 dev ext-vlan.100 table local proto kernel scope host src 169.254.100.1
broadcast 169.254.100.255 dev ext-vlan.100 table local proto kernel scope link src 169.254.100.1
local 172.16.0.4 dev load-balan-5a6d table local proto kernel scope host src 172.16.0.4
broadcast 172.16.0.255 dev load-balan-5a6d table local proto kernel scope link src 172.16.0.4
local 172.16.1.2 dev load-balan-3a3f table local proto kernel scope host src 172.16.1.2
broadcast 172.16.1.255 dev load-balan-3a3f table local proto kernel scope link src 172.16.1.2
default via 100:100::150 dev ext-vlan.100 table 4096 proto bird metric 32 pref medium
default via fd00::32 dev load-balan-5a6d table 17 metric 1024 pref medium
default via fd00::24 dev load-balan-5a6d table 31 metric 1024 pref medium
default via fd00:0:0:1::2c dev load-balan-3a3f table 58 metric 1024 pref medium
default via fd00::6 dev load-balan-5a6d table 69 metric 1024 pref medium
default via fd00:0:0:1::6 dev load-balan-3a3f table 76 metric 1024 pref medium
100:100::/64 dev ext-vlan.100 proto kernel metric 256 pref medium
fd00::/64 dev load-balan-5a6d proto kernel metric 256 pref medium
fd00:0:0:1::/64 dev load-balan-3a3f proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev ext-vlan.100 proto kernel metric 256 pref medium
fe80::/64 dev load-balan-5a6d proto kernel metric 256 pref medium
fe80::/64 dev load-balan-3a3f proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast 100:100:: dev ext-vlan.100 table local proto kernel metric 0 pref medium
local 100:100::1 dev ext-vlan.100 table local proto kernel metric 0 pref medium
anycast fd00:: dev load-balan-5a6d table local proto kernel metric 0 pref medium
local fd00::4 dev load-balan-5a6d table local proto kernel metric 0 pref medium
anycast fd00:0:0:1:: dev load-balan-3a3f table local proto kernel metric 0 pref medium
local fd00:0:0:1::2 dev load-balan-3a3f table local proto kernel metric 0 pref medium
anycast fe80:: dev eth0 table local proto kernel metric 0 pref medium
anycast fe80:: dev ext-vlan.100 table local proto kernel metric 0 pref medium
anycast fe80:: dev load-balan-5a6d table local proto kernel metric 0 pref medium
anycast fe80:: dev load-balan-3a3f table local proto kernel metric 0 pref medium
local fe80::42:acff:fe12:5 dev ext-vlan.100 table local proto kernel metric 0 pref medium
local fe80::fe:4ff:fe84:48cb dev load-balan-5a6d table local proto kernel metric 0 pref medium
local fe80::fe:54ff:fe43:6876 dev load-balan-3a3f table local proto kernel metric 0 pref medium
local fe80::9ccc:93ff:fedb:e700 dev eth0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev ext-vlan.100 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev load-balan-5a6d table local proto kernel metric 256 pref medium
multicast ff00::/8 dev load-balan-3a3f table local proto kernel metric 256 pref medium
STEP: Checking if all targets have receive traffic with no traffic interruption (no lost connection) 06/03/22 11:08:55.486
------------------------------
• [FAILED] [32.998 seconds]
Scaling
/home/lionelj/Workspaces/Meridio/test/e2e/scaling_test.go:32
When trench is with 2 VIP addresses (20.0.0.1:5000, [2000::1]:5000) and 4 target pods running ctraffic
/home/lionelj/Workspaces/Meridio/test/e2e/scaling_test.go:34
when scaling targets up to 5
/home/lionelj/Workspaces/Meridio/test/e2e/scaling_test.go:107
[It] should receive the traffic correctly
/home/lionelj/Workspaces/Meridio/test/e2e/scaling_test.go:111
Begin Captured GinkgoWriter Output >>
STEP: Waiting for the new targets to be registered 06/03/22 11:08:25.084
STEP: Checking if all targets have receive traffic with no traffic interruption (no lost connection) 06/03/22 11:08:55.486
<< End Captured GinkgoWriter Output
Expected
<int>: 3
to equal
<int>: 5
In [It] at: /home/lionelj/Workspaces/Meridio/test/e2e/scaling_test.go:157
