Unable to run non-privileged mode on arm64 nodes (AWS graviton) on AWS EKS

[LucasMouraoFerreira]

Was unable to run calico in non-privileged mode on arm64 nodes (AWS graviton) on AWS EKS (Using calico only for Network Policies). Privileged mode works fine on arm64 nodes tho, and non-privileged mode works fine on amd64 nodes.

Expected Behavior

Calico nodes should work fine in non-privileged mode on arm64, just as it works on amd64.

Current Behavior

Calico nodes don't get into ready state, They keep receiving the following kernel permission error.

2022-07-28 14:13:35.569 [INFO][2043] felix/ipsets.go 312: Retrying after an ipsets update failure... family="inet"
2022-07-28 14:13:35.570 [ERROR][2043] felix/ipsets.go 574: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.11: Kernel error received: Operation not permitted\n"
2022-07-28 14:13:35.570 [WARNING][2043] felix/ipsets.go 322: Failed to resync with dataplane error=exit status 1 family="inet"

Steps to Reproduce (for bugs)

1. Spin an EKS 1.22 cluster with arm64 AWS Graviton nodes.
2. Deploy calico v3.23.3 via helm with the following values.

installation:
  enabled: true
  kubernetesProvider: EKS
  cni:
    type: AmazonVPC
  nonPrivileged: Enabled

Context

Privileged containers are avoided as much as possible in our organization's environment, for security reasons. And they must be very well justified.

Am I missing something, or there is still a lack of support for arm64 in non-privileged mode?

Your Environment

• Calico version: 3.23.3
• Orchestrator version: EKS 1.22
• Operating System and version: Amazon Linux 2 - ami-0dff8c6d4d1dc8ff0 (k8s: 1.22.9, docker: 20.10.13-2.amzn2, containerd: 1.4.13-3.amzn2)
• Instance types: c6g.large, t4g.medium

Thank you for your time!!

[caseydavenport]

@mgleung could you take a look at this?

[mgleung]

Oops, looks like the non-privileged changes were only made for amd64. We will need to port over a lot of the permission changes in the image (https://github.com/projectcalico/calico/blob/master/node/Dockerfile.amd64#L180-L193) as well as a few other lines (setting the SUID bit) in order to make this work for arm64.

[mgleung]

I believe that https://github.com/projectcalico/calico/pull/6637 should fix this up.

[caseydavenport]

Guessing we still need this for other architectures though @mgleung ?

[mgleung]

Guessing we still need this for other architectures though

Yup, we'll need to eventually get similar changes for the other architectures but this should handle arm64.