healthcheck problem on IPv6 --dnamic-ip-setup on linux

[ghost]

I noticed an interesting behaviour regarding the healthcheck in combination with IPv6 and --dynamic-ip-setup. This is a combination of linux behaviour and the way that the healthcheck configures IPs. It was observed on exabgp 4.2.11 running in a freshly built Alpine Linux container.

It seems that the linux kernel does not use or support the label option the for ipv6 the same way that it does for ipv4. Right now the healtcheck --dynamic-ip-setup routine will create ips in the system and bind them to the loopback interface. It uses iproute2 "label" option to mark them.

https://github.com/Exa-Networks/exabgp/blob/52ad73c53a5f1396dbf8d34c627f21db6906a737/lib/exabgp/application/healthcheck.py#L239

https://github.com/Exa-Networks/exabgp/blob/52ad73c53a5f1396dbf8d34c627f21db6906a737/lib/exabgp/application/healthcheck.py#L243

It will only create an ip if a label is set for the healthcheck:

https://github.com/Exa-Networks/exabgp/blob/52ad73c53a5f1396dbf8d34c627f21db6906a737/lib/exabgp/application/healthcheck.py#L242 following.

This works fine for ipv4, but does not work for ipv6. it seems that the label option is used differently.

If we add an ipv6 prefix with the command line (icluding the label option) used by the healhcheck, it will no complain and create the ip on the loopback interface. Unfortunately the label will not be treated the same way as the ipv4 labe. e.g. it will not show in the output requriered by the loopback ips function. https://github.com/Exa-Networks/exabgp/blob/52ad73c53a5f1396dbf8d34c627f21db6906a737/lib/exabgp/application/healthcheck.py#L187

ip a a 2000::1 dev lo label lo:v6 ip -o a s dev lo | grep 2000::1 1: lo inet6 2000::1/128 scope global \ valid_lft forever preferred_lft forever

vs.

ip a a 1.1.1.1 dev lo label lo:v4 ip -o a s dev lo | grep 1.1.1.1 1: lo inet 1.1.1.1/32 scope global lo:v4\ valid_lft forever preferred_lft forever

This leads to the follwoing behaviour:

• once the healthcheck is succesfull, it will setup the ipv6 ip and the ip will show up in the system, unfortunately the loopback_ips function will never list it, as it tries to list all loopbacks with the given label. And as the label is not there the list is empty.

python3 -m exabgp healthcheck -d --cmd "nc -z -w2 ::1 53" --label v6 --no-syslog --withdraw-on-down --ip 2000::1/128 --dynamic-ip-setup .... DEBUG[healthcheck] Checking command 'nc -z -w2 ::1 53' DEBUG[healthcheck] Command was executed successfully 0 b'' INFO[healthcheck] service up, restoring loopback ips DEBUG[healthcheck] Retrieve loopback IP addresses DEBUG[healthcheck] Loopback addresses: [] DEBUG[healthcheck] Setup loopback IP address 2000::1/128 ..... DEBUG[healthcheck] Retrieve loopback IP addresses DEBUG[healthcheck] Loopback addresses: [] DEBUG[healthcheck] Setup loopback IP address 2000::1/128 .....

• once the healtcheck is no longer succesfull the loopback_ips function will still not report the loopback, hence it will never be removed

WARNING[healthcheck] Check command was unsuccessful: 1 DEBUG[healthcheck] Transition to DOWN INFO[healthcheck] service down, deleting loopback ips DEBUG[healthcheck] Retrieve loopback IP addresses DEBUG[healthcheck] Loopback addresses: [] .... INFO[healthcheck] service down, deleting loopback ips DEBUG[healthcheck] Retrieve loopback IP addresses DEBUG[healthcheck] Loopback addresses: [] ....

Withdrawel of the routes from BGP works as expected, but the ip stays active in the OS.

Quick hack i tried to far:

• set label_only to false for the loopback_ips call in remove_ips https://github.com/Exa-Networks/exabgp/blob/52ad73c53a5f1396dbf8d34c627f21db6906a737/lib/exabgp/application/healthcheck.py#L255 If we do not provide the --label option for a healthcheck, this would succesfully remove the ip as intended, unfortunately the ip would not get created in the first place as the setup_ips function only does so if a label is present: https://github.com/Exa-Networks/exabgp/blob/52ad73c53a5f1396dbf8d34c627f21db6906a737/lib/exabgp/application/healthcheck.py#L242

From my perspective there is not an easy fix to that problem right away, a rewrite of the ipv6 --dynamic-ip-setup logic would probably be needed.

The iproute2/kernel behaviour was tested on kernel 4.19.0-10-amd64 (debian 10 stock kernel) ans also verified on kernel 5.7.14

[thomas-mangin]

Thank you for this comprehensive bug report, much appreciated. @vincentbernat may have an opinion on how it could be handled. If not, I will look into how we can fix this issue.

[awlx]

As I was also interested in this issue. I verified with pyroute2 that the kernel indeed misses the IFA_LABEL for ipv6 addresses.

See this dump (I dumped with pyroute2 to make sure not only iproute2 is missing this flag):

Dump v6 addresses: (::3 and fe80::d8:cf98:3f86:3d09 should have a custom label)
 
({'family': 10, 'prefixlen': 128, 'flags': 128, 'scope': 0, 'index': 1, 'attrs': [('IFA_ADDRESS', '::3'), ('IFA_CACHEINFO', {'ifa_preferred': 4294967295, 'ifa_valid': 4294967295, 'cstamp': 249918929, 'tstamp': 249918929}), ('IFA_FLAGS', 128)], 'header': {'length': 72, 'type': 20, 'flags': 2, 'sequence_number': 255, 'pid': 18426, 'error': None}, 'event': 'RTM_NEWADDR'}, {'family': 10, 'prefixlen': 128, 'flags': 128, 'scope': 0, 'index': 1, 'attrs': [('IFA_ADDRESS', '2001:608:a01:ffff::30'), ('IFA_CACHEINFO', {'ifa_preferred': 4294967295, 'ifa_valid': 4294967295, 'cstamp': 423, 'tstamp': 423}), ('IFA_FLAGS', 128)], 'header': {'length': 72, 'type': 20, 'flags': 2, 'sequence_number': 255, 'pid': 18426, 'error': None}, 'event': 'RTM_NEWADDR'}, {'family': 10, 'prefixlen': 128, 'flags': 128, 'scope': 253, 'index': 1, 'attrs': [('IFA_ADDRESS', 'fe80::d8:cf98:3f86:3d09'), ('IFA_CACHEINFO', {'ifa_preferred': 4294967295, 'ifa_valid': 4294967295, 'cstamp': 249987507, 'tstamp': 249987507}), ('IFA_FLAGS', 128)], 'header': {'length': 72, 'type': 20, 'flags': 2, 'sequence_number': 255, 'pid': 18426, 'error': None}, 'event': 'RTM_NEWADDR'}, {'family': 10, 'prefixlen': 128, 'flags': 128, 'scope': 254, 'index': 1, 'attrs': [('IFA_ADDRESS', '::1'), ('IFA_CACHEINFO', {'ifa_preferred': 4294967295, 'ifa_valid': 4294967295, 'cstamp': 273, 'tstamp': 273}), ('IFA_FLAGS', 128)], 'header': {'length': 72, 'type': 20, 'flags': 2, 'sequence_number': 255, 'pid': 18426, 'error': None}, 'event': 'RTM_NEWADDR'})
 
Dump v4 addresses (you see IFA_LABEL for all addresses):
({'family': 2, 'prefixlen': 8, 'flags': 128, 'scope': 254, 'index': 1, 'attrs': [('IFA_ADDRESS', '127.0.0.1'), ('IFA_LOCAL', '127.0.0.1'), ('IFA_LABEL', 'lo'), ('IFA_FLAGS', 128), ('IFA_CACHEINFO', {'ifa_preferred': 4294967295, 'ifa_valid': 4294967295, 'cstamp': 273, 'tstamp': 273})], 'header': {'length': 76, 'type': 20, 'flags': 2, 'sequence_number': 256, 'pid': 18426, 'error': None}, 'event': 'RTM_NEWADDR'}, {'family': 2, 'prefixlen': 32, 'flags': 128, 'scope': 0, 'index': 1, 'attrs': [('IFA_ADDRESS', '10.80.255.30'), ('IFA_LOCAL', '10.80.255.30'), ('IFA_LABEL', 'lo'), ('IFA_FLAGS', 128), ('IFA_CACHEINFO', {'ifa_preferred': 4294967295, 'ifa_valid': 4294967295, 'cstamp': 423, 'tstamp': 423})], 'header': {'length': 76, 'type': 20, 'flags': 2, 'sequence_number': 256, 'pid': 18426, 'error': None}, 'event': 'RTM_NEWADDR'}, {'family': 2, 'prefixlen': 32, 'flags': 128, 'scope': 0, 'index': 1, 'attrs': [('IFA_ADDRESS', '9.9.9.9'), ('IFA_LOCAL', '9.9.9.9'), ('IFA_LABEL', 'lo:foo'), ('IFA_FLAGS', 128), ('IFA_CACHEINFO', {'ifa_preferred': 4294967295, 'ifa_valid': 4294967295, 'cstamp': 249889350, 'tstamp': 249889350})], 'header': {'length': 80, 'type': 20, 'flags': 2, 'sequence_number': 256, 'pid': 18426, 'error': None}, 'event': 'RTM_NEWADDR'})

[vincentbernat]

Looking at the kernel source code, IFLA_LABEL is only handled for IPv4. I am a bit surprised as I was pretty sure I did use that in the past successfully. I don't see anything else that could be used instead of a label for IPv6. This could be added to Linux, as routes have such an attribute (protocol), but it won't solve the immediate issue.

So, we need a special handling for IPv6 or a fatal errors that label are not compatible with IPv6 (I think this would be safer).

[theister-xan]

Its not rocket sience, but i "solved" it in our case by explicitly setting --up-execute and --down-execute per healthchek to e.g.

python3 -m exabgp healthcheck -d --cmd "nc -z -w2 ::1 53" --label v6 --no-syslog --withdraw-on-down --up-execute "ip a a 2000::1/128 dev lo" --down-execute "ip a d 2000::1/128 dev lo";