Bonding SR-IOV interfaces fails

[howels]

Followed the README step by step. Normal SR-IOV Pods with a single interface and an IP set on the SR-IOV interface work fine, but trying to create a bond on top of SRIOV VFs fails every time.

I am expecting to see a Pod started with the bonded interface present but Pods all sit in ContainerCreating and do not proceed. I am able to replace the definition shown below with a basic SRIOV network and everything springs to life. Likewise all MACVLAN, Canal, Hostnetwork and other network types seem to work also.

OS is SLES12SP4 on Intel X710 hardware. Kubernetes is RKE2 version v1.19.8+rke2r1

I see that the first interface is being added but it never proceeds further:

 Normal   AddedInterface          6s                      multus             Add net1 [] from kube-system/sriov-pf1
  Normal   AddedInterface          5s                      multus             Add net1 [] from kube-system/sriov-pf1
  Normal   AddedInterface          5s                      multus             Add eth0 [10.42.1.130/32]
  Normal   AddedInterface          4s                      multus             Add eth0 [10.42.1.131/32]
  Normal   AddedInterface          4s                      multus             Add net1 [] from kube-system/sriov-pf1
  Normal   AddedInterface          3s                      multus             Add net1 [] from kube-system/sriov-pf1
  Normal   AddedInterface          3s                      multus             Add eth0 [10.42.1.132/32]
  Normal   AddedInterface          2s                      multus             Add eth0 [10.42.1.133/32]
  Normal   AddedInterface          2s                      multus             Add net1 [] from kube-system/sriov-pf1
  Normal   AddedInterface          1s                      multus             Add eth0 [10.42.1.134/32]
  Normal   AddedInterface          1s                      multus             Add net1 [] from kube-system/sriov-pf1
  Normal   AddedInterface          0s                      multus             Add eth0 [10.42.1.135/32]
  Normal   AddedInterface          0s                      multus             Add net1 [] from kube-system/sriov-pf1

Annotations used:

      annotations:
        k8s.v1.cni.cncf.io/networks: '[
          {"name": "sriov-pf1",
          "interface": "net1"},
          {"name": "sriov-pf2",
          "interface": "net2"},
          {"name": "bond-net1",
          "interface": "bond0"}
          ]'

sriovdp-config (note the intel_sriov_netdevice pool which is working just fine):

apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
  namespace: kube-system
data:
  config.json: |
    {
        "resourceList": [{
                "resourceName": "intel_sriov_PF_1",
                "selectors": {
                    "vendors": ["8086"],
                    "devices": ["154c", "10ed"],
                    "drivers": ["i40evf", "iavf", "ixgbevf"],
                    "pfNames": ["pslave-0#1-10"]
                }
            },
            {
                "resourceName": "intel_sriov_PF_2",
                "selectors": {
                    "vendors": ["8086"],
                    "devices": ["154c", "10ed"],
                    "drivers": ["i40evf", "iavf", "ixgbevf"],
                    "pfNames": ["pslave-1#1-10"]
                }
            },
            {
                "resourceName": "intel_sriov_netdevice",
                "selectors": {
                    "vendors": ["8086"],
                    "devices": ["154c", "10ed"],
                    "drivers": ["i40evf", "iavf", "ixgbevf"]
                }
            },
            {
                "resourceName": "intel_sriov_dpdk",
                "selectors": {
                    "vendors": ["8086"],
                    "devices": ["154c", "10ed"],
                    "drivers": ["vfio-pci"],
                    "pfNames": ["enp0s0f0","enp2s2f1"]
                }
            },
            {
                "resourceName": "mlnx_sriov_rdma",
                "selectors": {
                    "vendors": ["15b3"],
                    "devices": ["1018"],
                    "drivers": ["mlx5_ib"],
                    "isRdma": true
                }
            }
        ]
    }

First SRIOV NetworkAttachmentDefinition:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-pf1
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_PF_1
spec:
  config: '{
  "type": "sriov",
  "name": "sriov-network",
  "spoofchk": "off"
}'

Second SRIOV NetworkAttachmentDefinition:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-pf2
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_PF_2
spec:
  config: '{
  "type": "sriov",
  "name": "sriov-network",
  "spoofchk": "off"
}'

Bond NetworkAttachmentDefinition:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: bond-net1
spec:
  config: '{
    "type": "bond",
    "cniVersion": "0.3.1",
    "name": "bond-net1",
    "ifname": "bond0",
    "mode": "802.3ad",
    "xmit_hash_policy": "layer3+4",
    "failOverMac": 1,
    "linksInContainer": true,
    "miimon": "100",
    "links": [
      {"name": "net1"},
      {"name": "net2"}
    ],
    "ipam": {
      "type": "host-local",
      "subnet": "192.168.3.0/24",
      "rangeStart": "192.168.3.200",
      "rangeEnd": "192.168.3.216",
      "routes": [{
        "dst": "0.0.0.0/0"
      }]
    }
  }'

If I swap all this for a basic SRIOV network config with IPs directly on the VFs then it works fine:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
 name: sriov-net1
 annotations:
   k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_netdevice
spec:
 config: '{
 "type": "sriov",
 "cniVersion": "0.3.1",
 "name": "sriov-network",
 "ipam": {
   "type": "host-local",
   "subnet": "192.168.1.0/24",
   "rangeStart": "192.168.1.100",
   "rangeEnd": "192.168.1.116",
   "routes": [{
     "dst": "0.0.0.0/0"
   }]
 }
}'

[killianmuldoon]

@howels can you show the CNI error? Either from the Pod description or from the Multus logs? A pod stuck in container creating usually means the CNI isn't able to create the interface at all and returns an error.

Also - where did you get the bond config?
"mode": "802.3ad", "xmit_hash_policy": "layer3+4",

Aren't supported in the main branch right now. 802.3ad will be accepted and set, but xmit_hash_policy isn't part of the config.

c/f https://github.com/intel/bond-cni/pull/17

[howels]

I found that this works, but only when I moved to SLES 15 and removed the default network statement ( v1.multus-cni.io/default-network: bond-net1). Otherwise I would get a message saying that my slave interfaces could not be found.

As for the bond config, this was just the normal way that we configure bonds so I re-used that for my SR-IOV bonds. What are your defaults for xmit_hash_policy?

[killianmuldoon]

The default setting can be per-OS I think, but we expect it to be 0, which is hashing based on the MAC. The real impact of setting this depends on the network set up - but we can definitely move to include it from #17 if it meets your use case.

v1.multus-cni.io/default-network: bond-net1

I'll try to track down why this is interfering with the bond network. I think this is a new setting for Multus, and it may be an issue because of the order in which it's ran compared to the bonding command.