Default Environment does not support arm64

[blackliner]

Problem

The goal was to follow these guidelines to create a cluster with both adm64 VMs and a few arm64 Raspberry Pi 4s.

• https://www.sidero.dev/v0.6/getting-started/
• https://www.sidero.dev/v0.6/guides/rpi4-as-servers/

What happens

The Raspberry won't boot from UEFI to talos, because it receives the default environment, which will point it to a amd64 based vmlinuz and initramfs.xz.

What should happen

The default environment should support both/all archictectures. It would also be very helpful to document how to make one environment work with multiple platforms, if this is possible at all. Apparently it is already supported in the implicit "enlisting" environment, which does not exist as a kubernetes resource of type environment.

Logs

✔️ During enlisting, this is what the sidero manager logs (see the agent-arm64 environment):

manager 2023/10/22 09:03:45 HTTP GET /ipxe?uuid=00c03114-0000-0000-0000-e45f01158ec6&mac=e4-5f-01-15-8e-c6&domain=local.berchtold.live&hostname=&serial=0000E45F01158EC6&arch=arm64 192.168.1.167:59225                                                                                       
manager 2023/10/22 09:03:45 Using "agent-arm64" environment                                                                                    
manager 2023/10/22 09:03:45 HTTP GET /env/agent-arm64/vmlinuz 192.168.1.167:59225                                                              
manager 2023/10/22 09:03:46 HTTP GET /env/agent-arm64/initramfs.xz 192.168.1.167:59225

❌ During provisioning, it looks like the raspi is asked to boot from the default environment:

manager 2023/10/22 09:00:51 HTTP GET /ipxe?uuid=00c03114-0000-0000-0000-e45f01158ec6&mac=e4-5f-01-15-8e-c6&domain=local.berchtold.live&hostname=&serial=0000E45F01158EC6&arch=arm64 192.168.1.167:59225                                                                                       
manager 2023/10/22 09:00:51 Using "default" environment for "00c03114-0000-0000-0000-e45f01158ec6"                                             
manager 2023/10/22 09:00:51 HTTP GET /env/default/vmlinuz 192.168.1.167:59225                                                                  
manager 2023/10/22 09:00:51 HTTP GET /env/default/initramfs.xz 192.168.1.167:59225

Manifest

The default environment looks like it supports amd64 only out of the box:

apiVersion: metal.sidero.dev/v1alpha2
kind: Environment
metadata:
  name: default
spec:
  initrd:
    url: https://github.com/siderolabs/talos/releases/download/v1.6.0-alpha.0/initramfs-amd64.xz

Workaround

Manually modify the default environment and update any amd64 string with arm64. But this only helps to boot the Raspberries, and will break amd64-based machines.

Slack

Slack 🧵 (will be gone in 90 days 😞 ): https://taloscommunity.slack.com/archives/CK8H5DDDM/p1697966721306259

[smira]

Environment is picked up based on the Server/SeverClass, so it should be straightforward to use that to target RPis. Sidero Metal has enough features for it.

[linuxmaniac]

yes, it's just a matter of defining a a proper arm64 Environment and Server/ServerClass that references it. Maybe you could provide a PR that improves the documentation since it seems that is not clear enough regarding to arm64 support

[blackliner]

The most confusing part was that at an earlier stage, when the server is analyzed for the first time, some internal logic decides for the correct image. But then when the serverclass logic takes over, it is amd64. Yes docu would help, especially for the Raspi guides, they should definitely mention this (otherwise the guide is broken as of now)

[michael-robbins]

Late reply, but if you're after some raspberry pi specific server classes/environments I've got some here: https://github.com/dalmura/infrastructure/blob/main/sites/archived/indigo-sidero/sidero/serverclasses.yaml