Adding extensions during install, doesn't activate the extension until we upgrade the node

[jbmorgado]

Bug Report

Description

Adding an extension to during the installation phase like:

machine:
  install:
    extensions:
      - image: ghcr.io/siderolabs/iscsi-tools:v0.1.1 # System extension image.

Doesn't pull the image into the node.

The only way to have the node actually pulling the extension is to issue:

talosctl -e <VIP IP> -n <NODE IP> upgrade --image=ghcr.io/siderolabs/installer:v1.1.1

After the node is upgraded, the extension is finally working on the node.

Logs

Environment

• Talos version: Client: Tag: v1.1.1 SHA: 40a050c6 Built: Go version: go1.18.4 OS/Arch: linux/amd64 Server: NODE: 192.168.1.212 Tag: v1.1.1 SHA: 40a050c6 Built: Go version: go1.18.4 OS/Arch: linux/amd64 Enabled: RBAC

- Kubernetes version:

Client Version: v1.24.3 Kustomize Version: v4.5.4 Server Version: v1.24.2

- Platform:
VSphere 7

[frezbo]

This is expected behavior, extensions are only enabled on upgrades or during install from network

[jbmorgado]

This is expected behavior, extensions are only enabled on upgrades or during install from network

Can you clarify what do you mean by "during install from network"?

[smira]

Extensions are activated during install, it could be that your node was booted from disk image skipping install?

[smira]

Image can be built with system extensions pre-installed via https://www.talos.dev/v1.1/talos-guides/configuration/system-extensions/#building-a-talos-image-with-system-extensions

[jbmorgado]

I see, thank you. This is a bug with the documentation then, maybe? As it states that during install, on boot, these system extensions will be overlaid on top on the root filesystem:

"During the initial install (e.g. when PXE booting or booting from an ISO), Talos will pull down container images for system extensions, validate them, and include them into the Talos initramfs image. System extensions will be activated on boot and overlaid on top of the Talos root filesystem."

https://www.talos.dev/v1.1/talos-guides/configuration/system-extensions/#configuration

[smira]

Yes, probably it's not clear, but when booting from a disk image install phase is skipped, as Talos is already "installed" in the disk image. That's why I referred to the section about creating an image with system extension pre-installed.

[btrepp]

This is probably the case with the container images as-well?. E.g what maybe used from talosctl cluster create.

At the moment I also don't have a workaround for myself on this (on macOS).

1. Upgrade states that the cluster create doesn't support upgrades, fair enough, it's pulling an image
2. Trying to use the qemu provisioner gives qemu provisioner is not supported on this platform. https://github.com/siderolabs/talos/blob/96aa9638f724d81ba8ef64db0ed7032694e7da5d/pkg/provision/providers/qemu_other.go#L17. I don't know much about go, but I imagine it's hitting this for macOS.
3. The makefiles seem to only cover backing in extensions to the image based ones. The env var in the documentation is only used on the make image-% and sbc commands.

I was hoping to use this as a way of developing extensions, I think probably 2) getting qemu to work, would be the most promising path, qemu does work on this machine.

[smira]

System extensions are not supported in container mode, and the container mode is pretty limited.

On OS X, you can still use native solutions to run VMs, and use generic Talos guides on how to run Talos.

talosctl cluster create relies on CNI plugins which do not work on OS X.

[frezbo]

I think this can be closed now