azure-cli-extensions
azure-cli-extensions copied to clipboard
Azure
Azure Arc connectedk8s support for arm64
Extension name (the extension in question)
connectedk8s
Description of issue (in as much detail as possible)
Configuration:
- Platform: Raspberry Pi 4
- OS: Ubuntu Server 20.10
- K8s: v1.19.4
I'm trying to enroll my Raspberry Pi 4 K8s cluster in Azure Arc:
> az connectedk8s connect --name <name> --resource-group <resource group> --location eastus
The enrollment times out and all pods report FailedScheduling:
> for pod in $(kubectl get pods -n azure-arc | grep 'Pending' | cut -d' ' -f1); do kubectl describe pod $pod -n azure-arc | grep FailedScheduling; done
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Digging into the pod manifest, they are requesting amd64 nodes:
nodeSelector:
kubernetes.io/arch: amd64
kubernetes.io/os: linux
I'm running on arm64, and nodes a labeled as such (b/c I'm on Raspberry Pi):
labels:
beta.kubernetes.io/arch: arm64
beta.kubernetes.io/os: linux
kubernetes.io/arch: arm64
kubernetes.io/hostname: rpi4-02
kubernetes.io/os: linux
Is the amd64 node selector an oversight, or intentional? Does Azure Arc support arm64?
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @akashkeshari.
Issue Details
Extension name (the extension in question)
connectedk8s
Description of issue (in as much detail as possible)
Configuration:
- Platform: Raspberry Pi 4
- OS: Ubuntu Server 20.10
- K8s: v1.19.4
I'm trying to enroll my Raspberry Pi 4 K8s cluster in Azure Arc:
> az connectedk8s connect --name <name> --resource-group <resource group> --location eastus
The enrollment times out and all pods report FailedScheduling:
> for pod in $(kubectl get pods -n azure-arc | grep 'Pending' | cut -d' ' -f1); do kubectl describe pod $pod -n azure-arc | grep FailedScheduling; done
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Warning FailedScheduling 22m default-scheduler 0/5 nodes are available: 5 node(s) didn't match node selector.
Digging into the pod manifest, they are requesting amd64 nodes:
nodeSelector:
kubernetes.io/arch: amd64
kubernetes.io/os: linux
I'm running on arm64, and nodes a labeled as such (b/c I'm on Raspberry Pi):
labels:
beta.kubernetes.io/arch: arm64
beta.kubernetes.io/os: linux
kubernetes.io/arch: arm64
kubernetes.io/hostname: rpi4-02
kubernetes.io/os: linux
Is the amd64 node selector an oversight, or intentional? Does Azure Arc support arm64?
| Author: | mikebranstein |
|---|---|
| Assignees: | - |
| Labels: |
|
| Milestone: | - |
Seeing exactly the same thing there, also on K8s 1.19.4 and Ubuntu 20.04, with a cluster of Pi 4s. The recent Azure Friday video seems to imply that ARM is supported. Perhaps we can manually edit the chart before deployment for now?
That's what I'm thinking @PostlMC. The issue appears to be an assumption that amd64 is the preferred node type.
I'm not sure if we'll be able to modify the deployment before it has been applied to the cluster, but we may be able to edit it after it's been applied. That should allow the scheduler to select an appropriate node.
Azure Arc enabled K8s doesn't have support for arm64 yet. We currently have this in our backlog and will need to work/collaborate on making the following ready for arm64 - Arc agents, flux and its dependencies. The transitive dependencies may take more time for arm64 readiness. To help us prioritise this in our backlog, it'd be great if we could discuss the scenarios (GitOps, Azure Monitor, Azure Policy,...) that are of immediate priority to you and your adoption timelines (for POC, production workloads,...). @mikebranstein, @PostlMC please reach out to me on [email protected]. Happy to gather requirements and adjust our roadmap and priorities to help you out.
@shashankbarsin , how was it shown to be working in the Azure Friday video? I've seen it as well.
Anyway, I've got it working by building a mixed-arch (arm64 and amd64) k3s (3 RPI 4 and 1 Rock Pi X).
However, I'm struggling to get az appservice kube complete the provisioning. I can see the pods running on amd64 node. However, they fail on the arm64 nodes. Will dig further in the logs and open another issue.
I'm also very interested in ARC support for arm64. This would enable me to test Arc at a small scale so I can better understand its usefulness for clients.
Would be great to integrate arm64 machines. $customer uses 100s of Raspberrys and would love to use it.
Trying multipass/k3d/azure arc on my M1 Mac but to no avail. Any idea if the support for arm64 is coming soon? :)
With my az cli installed in the default location, on an aarch64 Linux (arm64) I noticed it was possible to install the connectedk8s extension, until it tried to download and install an amd64 binary for helm and resulted in an exec format error.
$ cd ~/.azure/helm/v3.6.3/linux-amd64
$ wget https://get.helm.sh/helm-v3.6.3-linux-arm64.tar.gz
$ tar xvf helm-v3.6.3-linux-arm64.tar.gz
$ mv linux-arm64/helm helm
My expectations were pretty low that this would work, given the kind cluster behind this node is also arm64, and indeed the images behind Arc are apparently not multiarch either.
Node-Selectors: kubernetes.io/arch=amd64
kubernetes.io/os=linux
---- ------ ---- ---- -------
Warning FailedScheduling 6s (x4 over 3m52s) default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector.
With this announcement, I think we should prioritize adding arm64 support for the connectedk8s extension!
https://azure.microsoft.com/en-us/updates/public-preview-arm64based-azure-vms-can-deliver-up-to-50-better-priceperformance/
Anyway, once I installed the correct Helm architecture binary, I was able to see this error message I should have seen, but was blocked by the previously referenced error:
$ az connectedk8s connect --name flux-kind-aarch64 --resource-group aks-kingdon --location eastus
This operation might take a while...
Please ensure that this Kubernetes cluster have any nodes with OS 'linux' and architecture 'amd64', for scheduling the Arc-Agents onto and connecting to Azure. Learn more at https://aka.ms/ArcK8sSupportedOSArchitecture
Please check if the azure-arc namespace was deployed and run 'kubectl get pods -n azure-arc' to check if all the pods are in running state. A possible cause for pods stuck in pending state could be insufficient resources on the kubernetes cluster to onboard to arc.
Unable to install helm release: Error: timed out waiting for the condition
It would be great to fix all of this as part of the Arm64 preview support 🎉 👍
I'm simply trying to do local development using my M1 based Mac Studio and having this same problem. I will upvote this. My target device class is Raspberry Pi 4's as well and other similar devices clustered together in 3's and 5's.
Similar. Since moving to M1 and Volterra I have no Intel boxes in my homelab anymore. Would love to tinker with Arc to build out PoCs but am now blocked due to architecture support. If there's opensource stuff I can help with please let me know. I'm also currently getting arm64-support working for Apache Camel-K.
Also awaiting this! Seeing price / perf ratio we can't do without ARM nodes
Azure Arc-enabled Kubernetes now supports ARM64 nodes: https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/system-requirements#cluster-requirements
err... @omkark95 , the cluster extensions still aren't supported on arm64: https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/system-requirements#cluster-requirements
@mohsinonxrm that's correct, currently we support feature such as cluster connect and viewing Kubernetes resources in the Azure portal. Work item to support cluster extension on ARM64 is in our backlog. To help us prioritise this in our backlog, it'd be great if we could discuss the scenarios (GitOps, Azure Monitor, Azure Policy, KeyVault,...) that are of immediate priority to you and your adoption timelines. Happy to gather requirements and adjust our roadmap and priorities to help you out.
Arm64 is required to evaluate this feature and also for home lab use while using Azure DevOps
