Kernel routes are not (re-)programmed on link flap when using ECMP and RFC5549

[toelke]

Description

If an interface flaps (even one not used by frr), the routes previously configured into the kernel are lost; I can only reproduce this using ECMP.

I have attached a Vagrantfile for reproduction of this issue: vagrant.tgz. vagrant up will create two ubuntu machines; they will do unnumbered BGP via two interfaces eth1 and eth2. They announce 10.0.0.1/32 and 10.0.0.2/32 respectively.

vagrant@ubuntu2204:~$ ip r s
default via 192.168.121.1 dev eth0 proto dhcp src 192.168.121.90 metric 100
10.0.0.2 nhid 22 proto bgp metric 20
        nexthop via inet6 fe80::5054:ff:fe60:ffa5 dev eth2 weight 1
        nexthop via inet6 fe80::5054:ff:fee8:d628 dev eth1 weight 1
192.168.50.0/24 dev eth1 proto kernel scope link src 192.168.50.11
192.168.51.0/24 dev eth2 proto kernel scope link src 192.168.51.11
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.90 metric 100
192.168.121.1 dev eth0 proto dhcp scope link src 192.168.121.90 metric 100

The route to 10.0.0.2 is configured correctly as ECMP over two unnumbered interfaces.

When we then flap eth0 (which has no bearing on the BGP peering) by doing sudo ip link set down eth0, the link is brought up again by systemd immedieatly, but the route learned via BGP is missing:

agrant@ubuntu2204:~$ ip r s
default via 192.168.121.1 dev eth0 proto dhcp src 192.168.121.90 metric 100
192.168.50.0/24 dev eth1 proto kernel scope link src 192.168.50.11
192.168.51.0/24 dev eth2 proto kernel scope link src 192.168.51.11
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.90 metric 100
192.168.121.1 dev eth0 proto dhcp scope link src 192.168.121.90 metric 100

FRR still knows about it, though:

node-1# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/100] via 192.168.121.1, eth0, src 192.168.121.90, 00:01:24
C>* 10.0.0.1/32 is directly connected, lo, 00:02:47
B>* 10.0.0.2/32 [20/0] via fe80::5054:ff:fe60:ffa5, eth2, weight 1, 00:02:39
  *                    via fe80::5054:ff:fee8:d628, eth1, weight 1, 00:02:39
C>* 192.168.50.0/24 is directly connected, eth1, 00:02:47
C>* 192.168.51.0/24 is directly connected, eth2, 00:02:47
C>* 192.168.121.0/24 [0/100] is directly connected, eth0, 00:01:24
K>* 192.168.121.1/32 [0/100] is directly connected, eth0, 00:01:24

This issue is related to at least

• https://github.com/FRRouting/frr/issues/7299
• https://bugs.launchpad.net/ubuntu/+source/frr/+bug/1968550
• https://github.com/FRRouting/frr/issues/10884

Version

FRRouting 8.1 (node-1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--localstatedir=/var/run/frr' '--sbindir=/usr/lib/frr' '--sysconfdir=/etc/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'

How to reproduce

See above:

• vagrant up (I tested with the libvirt driver, needs ansible to be installed)
• vagrant ssh node-1
• sudo ip r s, verify that the route to 10.0.0.2 is there
• sudo ip link set down eth0, verify that the route to 10.0.0.2 is missing
• sudo vtysh->show ip route, verify that the route to 10.0.0.2 is there

Expected behavior

I expect kernel-routes to be reprogrammed or kept when a link flaps.

Actual behavior

The routes in the kernel go missing.

Additional context

No response

Checklist

• [X] I have searched the open issues for this bug.
• [X] I have not included sensitive information in this report.

[ne-vlezay80]

Confirmed:

~ # vtysh

Hello, this is FRRouting (version 9.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

test2# show ipv6 route
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>r 307:9435:953c:777c::/128 [200/0] via fe80::216:ff:fe00:1601, eth0, weight 1, 00:00:22
B>r 2002:1::1/128 [200/0] via fe80::216:ff:fe00:1601, eth0, weight 1, 00:00:22
B>r 2002:1::2/128 [200/0] via fe80::216:3eff:feab:f4bb, eth0, weight 1, 00:00:22
C * fe80::/64 is directly connected, eth0, 00:00:12
C * fe80::/64 is directly connected, vxlan10, 00:00:28
C>* fe80::/64 is directly connected, bridge0, 00:00:28
test2# exit
~ # tail /var/log/messages 
Mar  7 16:27:07 test2 daemon.err zebra[4570]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Network is down, type=RTM_NEWNEXTHOP(104), seq=40, pid=3051428375
Mar  7 16:27:07 test2 daemon.err zebra[4570]: [HSYZM-HV7HF] Extended Error: Nexthop id does not exist
Mar  7 16:27:07 test2 daemon.err zebra[4570]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=41, pid=3051428375
Mar  7 16:27:07 test2 daemon.err zebra[4570]: [HSYZM-HV7HF] Extended Error: Nexthop id does not exist
Mar  7 16:27:07 test2 daemon.err zebra[4570]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=42, pid=3051428375
Mar  7 16:27:07 test2 daemon.err zebra[4570]: [X5XE1-RS0SW][EC 4043309074] Failed to install Nexthop (135[fe80::216:3eff:feab:f4bb if 8 vrfid 0]) into the kernel
Mar  7 16:27:07 test2 daemon.warn zebra[4570]: [VYKYC-709DP] default(0:254):2002:1::2/128: Route install failed
Mar  7 16:27:07 test2 daemon.err zebra[4570]: [X5XE1-RS0SW][EC 4043309074] Failed to install Nexthop (134[fe80::216:ff:fe00:1601 if 8 vrfid 0]) into the kernel
Mar  7 16:27:07 test2 daemon.warn zebra[4570]: [VYKYC-709DP] default(0:254):307:9435:953c:777c::/128: Route install failed
Mar  7 16:27:07 test2 daemon.warn zebra[4570]: [VYKYC-709DP] default(0:254):2002:1::1/128: Route install failed

The bug is related to, if install route is finished fail, is not install attepting, because the flap interface is not link-local address.

Configuring timers from the neighbor or peer group fixed the problem.

neighbor INTERNAL timers 4 6

BUG FOUND: Frrouting is not detecting link state [if running on containers].

link=$(lxc info  test2|grep Host|awk -F: '{print $2}'); (ip link set dev $link down && ip link set dev $link up)

bird is going state linkdown from bgp sessions, frrouting is wait keep alive timers.

Normal: bgp session is stopping if link going NOCARRIER state.

[donaldsharp]

I believe this will be fixed in master and the 10.0 release in a couple of weeks.

[toelke]

I can reproduce the issue with frr 10.0. I have attached an updated vagrant/ansible tarball. vagrant.tgz

[toelke]

This is related to systemd-networkd: On a system without that, I cannot reproduce this issue. Even with ManageForeignRoutes=no and KeepConfiguration=yes this happens. Is there a way that FRR could see the interfaces bouncing and re-install the routes?

[toelke]

It turns out that systemd-networkd know about that issue and have a fix merged but not released: https://github.com/systemd/systemd/pull/30433

Of note:

networkd manages an interface fully or not at all. if you don't want the interface managed by networkd, then do not define a matching .network interface file. (https://github.com/systemd/systemd/issues/29034#issuecomment-1702878543)

It may thus be a bad idea to use systemd-networkd on hosts that should receive routes via a routing protocol.