Change calico_feature_detect_override default value to ChecksumOffloadBroken=true

[yankay]

What type of PR is this?

/kind bug

What this PR does / why we need it:

When installed in the Rocky9, there are some bugs with calico and kernel. The network is broken. The kubespray's default configuration needs to provision a healthy cluster.

So I want to recommand to the set the defaul value calico_feature_detect_override: "ChecksumOffloadBroken=true". And if the user wants to use VXLAN offload, he can change the configuration.

The solution is also used by:

Rancher: https://github.com/rancher/rke2-charts/blob/main-source/packages/rke2-calico/generated-changes/patch/values.yaml.patch#L55 Azure: https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/templates/addons/calico/patches/calico-node.yaml

So, I suggest setting the default value to the ChecksumOffloadBroken=true, to disable the VXLAN offload by default, so the calico can be worked in most cases.

Which issue(s) this PR fixes:

Fixes https://github.com/kubernetes-sigs/kubespray/issues/8992

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Disalbe the Calico VXLAN hardware Offload by default with setting `ChecksumOffloadBroken=true`, to work around the issue: https://github.com/projectcalico/calico/issues/4727

[yankay]

HI @cristicalin, would you please review it :-) ?

[yankay]

ref to: https://github.com/projectcalico/calico/issues/4727

[oomichi]

Hi @yankay

Thank you for doing this. After reading the related links, I have a question about this change.

The original issue depends on the combination of Linux kernel version and HW. In addition, if setting ChecksumOffloadBroken=true on non-problematic environments, that could make performance impact. Is it correct?

Kubespray supports a lot of Linux distributions, so I want to make sure this point before setting it as the default.

[yankay]

HI @oomichi

On the kernel < 5.7, the VXLAN offload default is off. So the performance does not change. On the kernel >= 5.7(Ubuntu 22.04, RockyLinux 9), The VXLAN offload default is on. So if the performance would be bad on the non-problematic environments, and the network would be broken on vSphere and Azure.

There are some performance data from: https://randomsecurity.dev/posts/vxlan-offload/. The VXLan offload increases the performance.

So I think, the default 'ChecksumOffloadBroken=true' to let the calico can be worked in most environment. If the user wants to tunning , he can set the 'ChecksumOffloadBroken=false' to enable the VXLAN offload (Requires kernel > 5.7 and HW support).

And it's a calico's bug, if calico solves the problem, the kubespray can change back.

[floryut]

HI @oomichi

On the kernel < 5.7, the VXLAN offload default is off. So the performance does not change. On the kernel >= 5.7(Ubuntu 22.04, RockyLinux 9), The VXLAN offload default is on. So if the performance would be bad on the non-problematic environments, and the network would be broken on vSphere and Azure.

There are some performance data from: https://randomsecurity.dev/posts/vxlan-offload/. The VXLan offload increases the performance.

So I think, the default 'ChecksumOffloadBroken=true' to let the calico can be worked in most environment. If the user wants to tunning , he can set the 'ChecksumOffloadBroken=false' to enable the VXLAN offload (Requires kernel > 5.7 and HW support).

And it's a calico's bug, if calico solves the problem, the kubespray can change back.

That's a good point, if we go this road a big warning in the release note needs to be there.

[yankay]

Thanks @floryut , The calico docs have been updated, and add release note comments on the PR.

Would you please review it ?

[floryut]

Thanks @floryut , The calico docs have been updated, and add release note comments on the PR.

Would you please review it ?

Not sure we should rush this onto 2.20, WDYT @oomichi @cristicalin ?

[cristicalin]

This has the potential to cause some head scratching on folks not affected by the bug so I would actually just document this in the docs and mention situations where it needs to be applied. If we can apply some detection logic it would be better but having kubespray figure all corner cases out of the fox seems like a recipe for complexity and code rot over time.

[yankay]

This has the potential to cause some head scratching on folks not affected by the bug so I would actually just document this in the docs and mention situations where it needs to be applied. If we can apply some detection logic it would be better but having kubespray figure all corner cases out of the fox seems like a recipe for complexity and code rot over time.

HI @cristicalin , thank you for the comments.

[oomichi]

I hesitate to set the value which could cause performance side-effect on some environments as the default.

It would be easier to find the solution if some features(in this case, calico) don’t work, because the error messages would be output and be helpful. However it is hard to find the solution if some performance issues happen on such environments.

One more thing is the default IPIP of calico doesn’t work on Azure and we need to specify VXLAN as https://projectcalico.docs.tigera.io/reference/public-cloud/azure So I feel it is fine to keep ChecksumOffloadBroken=true on the document at this time (at least for Kubespray v2.20).

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: yankay Once this PR has been reviewed and has the lgtm label, please assign luckysb for approval by writing /assign @luckysb in a comment. For more information see:The Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment