Update multus to v4.1.0 and clarify cilium compatibility

[ThisIsQasim]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Updates multus version to v4.1.0 and updates docs to clarify cilium config

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Update multus to v4.1.0

[k8s-ci-robot]

Hi @ThisIsQasim. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[yankay]

/ok-to-test

[yankay]

HI @ThisIsQasim

It's better to squash the commit.

[ThisIsQasim]

Hi @yankay, sure I have squashed the commits

[ThisIsQasim]

Wondering if I should add tests for multus with cilium?

[cyclinder]

Wondering if I should add tests for multus with cilium?

You can try to see if this is easy to add, or you can manually test it.

[ThisIsQasim]

Wondering if I should add tests for multus with cilium?

You can try to see if this is easy to add, or you can manually test it.

I have a cluster with cilium where I manually tested this but I can try adding automated tests.

[ThisIsQasim]

So seems like #10934 introduced a bug where if multus was enabled the template would throw an error about multus_manifest_2.results attribute not existing. I have added a fix for that as well.

fatal: [node1]: FAILED! => {"msg": "'ansible.vars.hostvars.HostVarsVars object' has no attribute 'multus_manifest_2.results'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'multus_manifest_2.results'"}

Should I create a separate issue for that and reference it here?

I have also added a priorityclass for the pods as mentioned in #11304 cc: @yankay @cyclinder

[yankay]

Thanks @ThisIsQasim /approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cyclinder, ThisIsQasim, yankay

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [yankay]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment