Bonding issues in 3227.2.0 / systemd 250: Cannot enslave bond to itself

[defo89]

Description

This is just a friendly warning that we observed changes in bonding configuration and order of operation in Flatcar 3227.2.0 that comes with systemd 250.

Impact

Server with LACP bond configuration would not come up online after upgrade from 3139.2.3 to 3227.2.0 (or fresh 3227.2.0 deployment).

Environment and steps to reproduce

1. Set-up: Our old configuration which was always working is:

Physical network interfaces:

server3 /etc/systemd/network # cat PCI5-P0.network
[Match]
MACAddress=XY:ZZ:AD:D9:7F:68
[Network]
LLDP=true
EmitLLDP=true
DHCP=no
Bond=bond2

server3 /etc/systemd/network # cat PCI5-P1.network
[Match]
MACAddress=XY:ZZ:AD:D9:7F:69
[Network]
LLDP=true
EmitLLDP=true
DHCP=no
Bond=bond2

Bond config:

server3 /etc/systemd/network # cat bond2.netdev
[NetDev]
Name=bond2
Kind=bond
MTUBytes=8950
[Bond]
Mode=802.3ad
MIIMonitorSec=1s
TransmitHashPolicy=layer3+4
LACPTransmitRate=fast
UpDelaySec=3s
DownDelaySec=3s
MinLinks=1

server3 /etc/systemd/network # cat bond2.network
[Match]
Name=bond2
[Network]
DHCP=no
Address=1.2.3.4/29
Gateway=1.2.3.1
Domains=some.corp

2. Error:

Bond will not come up. Error Failed to set master interface: Cannot enslave bond to itself. Operation not permitted

server3 /etc/systemd/network # networkctl status bond2
● 31: bond2
             Link File: /usr/lib/systemd/network/99-default.link
          Network File: /etc/systemd/network/PCI2-P1.network
                  Type: bond
                 State: off (failed)
          Online state: offline
                Driver: bonding
      Hardware Address: XY:ZZ:AD:D9:7F:68
                   MTU: 8950 (min: 68, max: 65535)
                 QDisc: noqueue
                  Mode: 802.3ad
                Miimon: 10s
               Updelay: 30s
             Downdelay: 30s
  Queue Length (Tx/Rx): 16/16
      Auto negotiation: no
                 Speed: n/a
     Activation Policy: up
   Required For Online: yes

Jul 28 07:19:30 server3 systemd-networkd[4162042]: bond2: Specified MAC address has the multicast bit set, clearing the bit.
Jul 28 07:19:31 server3 systemd-networkd[4162042]: bond2: Link UP
Jul 28 07:19:31 server3 systemd-networkd[4162042]: bond2: Gained carrier
Jul 28 07:19:31 server3 systemd-networkd[4162042]: bond2: Lost carrier
Jul 28 07:19:31 server3 systemd-networkd[4162042]: bond2: Gained carrier
Jul 28 07:19:31 server3 systemd-networkd[4162042]: bond2: Reconfiguring with /etc/systemd/network/PCI5-P0.network.
Jul 28 07:19:31 server3 systemd-networkd[4162042]: bond2: Link DOWN
Jul 28 07:19:31 server3 systemd-networkd[4162042]: bond2: Lost carrier
Jul 28 07:19:31 server3 systemd-networkd[4162042]: bond2: Failed to set master interface: Cannot enslave bond to itself. Operation not permitted
Jul 28 07:19:31 server3 systemd-networkd[4162042]: bond2: Failed

Expected behavior

Due to some order of operation change, might be this one, bond tries to assign itself, hence the error message.

Additional information

Following workaround with link is working for us, bond comes up successfully. In our setup we rely on Netbox to configure correct interfaces (matched by MAC) into a bond.

Interfaces matched by MAC in links:

server3 /etc/systemd/network # cat 50-PCI5-P0.link
[Match]
MACAddress=XY:ZZ:AD:D9:7F:68
[Link]
Name=PCI5-P0

server3 /etc/systemd/network # cat 50-PCI5-P1.link
[Match]
MACAddress=XY:ZZ:AD:D9:7F:69
[Link]
Name=PCI5-P1

Physical network interfaces:

server3 /etc/systemd/network # cat PCI5-P0.network
[Match]
Name=PCI5-P0
[Network]
LLDP=true
EmitLLDP=true
DHCP=no
Bond=bond2

server3 /etc/systemd/network # cat PCI5-P1.network
[Match]
Name=PCI5-P1
[Network]
LLDP=true
EmitLLDP=true
DHCP=no
Bond=bond2

Bond config is left unchanged.

[pothos]

It looks like your problem is that when the bond device gets the same MAC address as one of the bonded interfaces, the bonded match rule for the boned interface applies. I think the proper solution would be to add Type=!bond to the [Match] section of PCI5-P0.network and PCI5-P1.network.

[pothos]

I'll close but feel free to comment if anything is needed.