Ubuntu (Jammy, kernel 5.15 ) Disabling none trackable interface stops traffic thru host till keepalived restart

[3hedgehogs]

Describe the issue

I see problems with test servers on Ubuntu Jammy and keepalived. As title says after none trackable interface temporary disabled/interrupted (example: cross cable connected host reboot) traffic could not go thru Ubuntu server till keepalive server restart.

Keepalived v2.2.7 (01/16,2022) from last snap image Also tested with : keepalived/jammy,now 1:2.2.4-0.2build1 amd64

[pqarmitage]

@3hedgehogs Can you please provide you keepalived configuration and a more detailed description of what you are doing and what the problem is.

[3hedgehogs]

Here is may network and keepalived.conf files

We expect traffic goes from vrrp_int_1,vrrp_int_2 to vrrp_ext_1,vrrp_ext_2 , they are vmac interfaces on vlan126 (internal VLAN) and vlan127 (external VLAN).

So if any "local" interface, as example eno2 is down, than traffic thru vrrp_int_1,vrrp_int_2 to/from vrrp_ext_1,vrrp_ext_2 is not possible.

We have exactly such configuration on Bionic and Focal servers for years in stable state. And I currently see such behavior only in Ubuntu Jammy. Could not find changes in network system for Jammy in Google at least.

linkbeat_use_polling
use_pid_dir

global_defs {
   router_id VIPS_0102

   dynamic_interfaces

   process_names

   vrrp_no_swap
   vrrp_priority  -5
   vrrp_rt_priority 15
   max_auto_priority 21
   min_auto_priority_delay 100000
   vrrp_rlimit_rttime  8000000
   vrrp_startup_delay 35.5
   vrrp_higher_prio_send_advert yes

   vrrp_garp_master_delay     0
   vrrp_garp_master_repeat    1
   vrrp_garp_master_refresh   0
   vrrp_garp_master_refresh_repeat  1
   vrrp_garp_lower_prio_repeat      1
   vrrp_garp_interval     0.501
   vrrp_gna_interval   0.500001


   vrrp_netlink_cmd_rcv_bufs  851968
   vrrp_netlink_cmd_rcv_bufs_force yes
   vrrp_netlink_monitor_rcv_bufs 851968
   vrrp_netlink_monitor_rcv_bufs_force yes

   process_monitor_rcv_bufs 500000
   process_monitor_rcv_bufs_force yes

   enable_script_security

   disable_local_igmp

   enable_dbus
}

linkbeat_interfaces {
   dummy0
   eno1
   ebo2
   enp8s0
   enp8s0d1
   vlan126
   vlan127
}

vrrp_sync_group VI_01 {
  group {
      VI_EXT_01
      VI_INT_01
  }

  notify /opt/mng/keepalived-scripts/notify.sh
}
vrrp_sync_group VI_02 {
  group {
      VI_EXT_02
      VI_INT_02
  }

  notify /opt/mng/keepalived-scripts/notify.sh
}


!!! Statics rules
include /var/cache/mng/keepalived/iprules.inc
# static_rules {
#  from 0.0.0.0/0 fwmark 0x17968 lookup 9 preference 1
# ...

vrrp_instance VI_EXT_01 {
  interface                 vlan127
  state                     MASTER
  virtual_router_id         174
  priority                  151
  advert_int                 10
  garp_master_delay           5
  preempt_delay               0

  use_vmac vrrp_ext_1
  vmac_xmit_base

  authentication {
      auth_type PASS
      auth_pass x
  }

  unicast_src_ip 169.254.202.1
  unicast_peer {
        169.254.202.2
  }

  track_interface {
      enp8s0
      enp8s0d1
  }
  # virtual_ipaddress
  include /var/cache/mng/keepalived/external_first_master.inc
# virtual_ipaddress {
#  YY.YY.YY.YY/21 dev vrrp_ext_1
# }

  # virtual_ipaddress_excluded
  include /var/cache/mng/keepalived/external_master.inc

  # virtual_routes
  include /var/cache/mng/keepalived/virtual_routes_master.inc
}

vrrp_instance VI_INT_01 {
  interface                 vlan126
  state                     MASTER
  virtual_router_id         190
  priority                  151
  advert_int                  5
  garp_master_delay           5
  preempt_delay               0

  use_vmac vrrp_int_1
  vmac_xmit_base

  authentication {
      auth_type PASS
      auth_pass x
  }

  unicast_src_ip 169.254.201.1
  unicast_peer {
        169.254.201.2
  }

  track_interface {
      eno1
      dummy0
  }
  # virtual_ipaddress
  include /var/cache/mng/keepalived/internal_master.inc
}



!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!  BACKUP instances:

vrrp_instance VI_EXT_02 {
  interface                 vlan127
  state                     BACKUP
  virtual_router_id         175
  priority                  101
  advert_int                 10
  garp_master_delay           5
  preempt_delay               0

  use_vmac vrrp_ext_2
  vmac_xmit_base

  authentication {
      auth_type PASS
      auth_pass x
  }

  unicast_src_ip 169.254.202.1
  unicast_peer {
        169.254.202.2
  }

  track_interface {
      enp8s0
      enp8s0d1
  }
  # virtual_ipaddress
  include /var/cache/mng/keepalived/external_first_backup.inc

  # virtual_ipaddress_excluded
  include /var/cache/mng/keepalived/external_backup.inc

  # virtual_routes
  include /var/cache/mng/keepalived/virtual_routes_backup.inc
}

vrrp_instance VI_INT_02 {
  interface                 vlan126
  state                     BACKUP
  virtual_router_id         191
  priority                  101
  advert_int                  5
  garp_master_delay           5
  preempt_delay               0

  use_vmac vrrp_int_2
  vmac_xmit_base

  authentication {
      auth_type PASS
      auth_pass x
  }

  unicast_src_ip 169.254.201.1
  unicast_peer {
        169.254.201.2
  }

  track_interface {
      eno1
      dummy0
  }

  # virtual_ipaddress
  include /var/cache/mng/keepalived/internal_backup.inc
}

network:
  version: 2
  renderer: networkd

  ethernets:
    eno1:
      addresses:
        - A.A.A.A/20
      routes:
        - to: default
          via: B.B.B.1
      nameservers:
        search:
          - a.a.com
          - ab.ab.de
        addresses:
          - C.C.C.1
          - C.C.C.2
      routes:
        - to: D.D.D.0/23
          scope: link
    eno2:
      addresses:
        - 10.0.0.1/30
    enp8s0:
      dhcp4: no
    enp8s0d1:
      dhcp4: no

  bonds:
    bond0:
      interfaces:
        - enp8s0
        - enp8s0d1
      parameters:
        mode: 802.3ad
        mii-monitor-interval: 100
        up-delay: 200
        down-delay: 200
        lacp-rate: fast

  vlans:
    vlan126:
      accept-ra: no
      id: 126
      link: bond0
      addresses:
        - 169.254.201.1/24:
           lifetime: 0
           label: "internal"
    vlan127:
      accept-ra: no
      id: 127
      link: bond0
      addresses:
        - 169.254.202.1/24:
           lifetime: 0
           label: "extermal"

[pqarmitage]

@3hedgehogs

keepalived is not responsible for forwarding network traffic, it just sets network configuration. To discover why traffic is not being forwarded you will need to see what is different in the network configuration between when traffic is being forwarded and when it is not being forwarded. It could be that keepalived is not setting configuration correctly, but we will need to see what is wrong in the configuration before we can investigate this further. We will also need to see the /var/cache/mng/keepalived/*.inc files to understand what is happening.

It is possible that when eno2 goes down, routes are deleted, and since no vrrp interface tracks eno2 the routes will not be recreated when eno2 comes back up.

Some comments on your configuration:

1. Why are you using linkbeat to monitor interfaces?
2. In linkbeat_interfaces, you specify ebo2 which does not appear to exist. Should this be eno2?
3. Why do you specify eno2/ebo2 in linkbeat_interfaces when the interface is not tracked?
4. Why are you using VMACs with unicast? The purpose of using VMACs is so that multicast adverts are sent with a 00:00:5e:00:0[01]:VRID MAC address.

[pqarmitage]

Closing due to no response for over 3 weeks.