talos icon indicating copy to clipboard operation
talos copied to clipboard
verified publisher icon siderolabs

Rasperry Pi CM4 - Allow edits to the /boot/config.txt file

Open danmanners opened this issue 3 years ago • 23 comments

Feature Request

I want to be able to enable USB devices on my CM4 on a DeskPi Super6c board as well as the Waveshare CM4-IO-Base-B board.

Description

Currently running Talos OS on Raspberry Pi CM4 units on the DeskPi Super6c (referenced here: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/425), but there does not seem to be any way with TalosOS to prepend the way that Raspberry Pi OS or anything else handles the /boot/config.txt file to enable USB ports, for example.

Happy to test and do whatever might be required to help out here!

danmanners avatar Jun 03 '22 20:06 danmanners

Confirmed that trying to create the file doesn't work, as you can't create files outside of /var:

Config Modification

machine:
  files:
    - permissions: 0o744 # The file's permissions in octal.
      path: /boot/config.txt # The path of the file.
      op: create
      content: |
        dtoverlay=dwc2,dr_mode=host

Error:

image

Next Step

I'm going to write a quick extension that can be installed to enable USB 2.0 on the CM4 units 🙂

PR hopefully coming in soon!

danmanners avatar Jul 04 '22 16:07 danmanners

In fact this file is in the /boot partition, so there are two ways to modify it:

  • mount the image and perform changes
  • modify RPi platform to accept some setting to modify the file (but there's no interface for it afaik)

Talos doesn't have /boot mounted while running, so it can't modify that.

smira avatar Jul 04 '22 17:07 smira

I believe a second option is more suitable and build a custom image using imager. I thought of having a single run extension service which would read a file on the host created by machinefFiles and doing all the necessary work, but since it's modifying files in /boot an improper config/run could render the system unable to boot.

frezbo avatar Jul 04 '22 17:07 frezbo

Ha, yes I learned yesterday that modifying the /boot/config.txt file will absolutely brick the node 😉

I'll get that fixed, and may end up submitting a patch for the extensions repo. I'll link things here if everything works as expected.

danmanners avatar Jul 05 '22 14:07 danmanners

Hey all, where did this get left off?

My interest in doing this is tweaking the PoE hat fan speeds to make the curve less aggressive. Given that I try to follow closely along on the latest release of Talos, I'm currently "suffering" 😆 with the default fan curve but that's obviously not optimal.

An extension service that performs some kind of sanity check of the config.txt and then copies it onto the boot partition seems like a kludge, but is IMO acceptable presuming that the changes will automatically get picked up.

DWSR avatar Dec 31 '22 14:12 DWSR

@danmanners the Pi firmware has support for specific tags; maybe we can do something like this:

[cm4]
dtoverlay=...

xvzf avatar Jan 15 '23 14:01 xvzf

@danmanners the Pi firmware has support for specific tags; maybe we can do something like this:

[cm4]
dtoverlay=...

Could definitely be a good option. I've still got one or two hosts where having USB enabled would be great, but it's less of a priority now for me.

danmanners avatar Jan 16 '23 15:01 danmanners

So the fix of enabling u-boot for the CM4 should be in the next release (it's merged in pkg). Same as NVME support; I'll likely need that as well, so will have a look in the coming weeks

xvzf avatar Jan 16 '23 16:01 xvzf

I would love to pick this up, what is the current status? I'm running into this issue as well on RPI4.

mount the image and perform changes

This is the approach I've taken for a while, but now that I just upgraded Talos, it wipes the disk, and therefore resets the modification. In theory, using --preserve on upgrade should work? I did not test this.

But, after performing my Talos upgrade and hearing all the fans spin up as my fan settings got reset kind of pushed me to now help find a proper solution :D

@smira

modify RPi platform to accept some setting to modify the file (but there's no interface for it afaik)

Can you give me some pointers on how to get this done please?

OGKevin avatar Mar 12 '23 10:03 OGKevin

I don't have any great idea, and I don't know how to handle that during upgrades.

https://github.com/siderolabs/talos/blob/9948a646d20f4ba80916a263ed7bca3e5ca2f0ad/internal/app/machined/pkg/runtime/v1alpha1/board/rpi_generic/rpi_generic.go#L44

This is the place the file is written. When creating an image, I could imagine one could pass something e.g. via the environment variable.

But when the upgrade happens, it runs on the RPi itself, and it's not trivial to pass anything to that process at the moment.

Thinking about the node itself, I would probably have something in the machine.install section which will provide an override for /boot/config.txt.

smira avatar Mar 13 '23 11:03 smira

Alright, I'll slowly have a look.

I just quickly looked at machine.install and saw https://www.talos.dev/v1.3/talos-guides/configuration/system-extensions/

Would extensions be an option?

I'm not sure on the whole install flow, but depending on how extensions work, would it be possible to write an extension that would modify /boot/config.txt during installation?

And then users can build and push their own extension with custom /boot/config.txt 🤔

If extension is not an option, I'll go through the code that handles machine.install to try and understand how all that works and see how I can modify it for custom /boot/config.txt

OGKevin avatar Mar 15 '23 07:03 OGKevin

yes, an extension could do it, mount the boot partition and edit the file, but that would mean only once talos starts the extension service the file would be edited, meaning another reboot would be needed for the pi to load the changed config.txt

frezbo avatar Mar 15 '23 08:03 frezbo

I take 2 reboots over manually extracting the disk and updating this file any day :D. I'll start to look into this next week after my trip, just to manage expectations.

OGKevin avatar Mar 15 '23 16:03 OGKevin

I could use some pointers again.

how do I make my extension run in privileged mode so I can mount the partition? Is the spec of https://github.com/siderolabs/extensions/blob/main/power/nut-client/nut-client.yaml documented somewhere? I expect I need to pass something here under containers section. 🤔

name: rpi-boot-config-loader
container:
  entrypoint: ./generic-pi
  args: []
  security:
    maskedPaths: []
    readonlyPaths: []
    writeableSysfs: true
  mounts:
    - source: /dev
      destination: /host/dev
      type: bind
      options:
        - rshared
        - rbind
        - rw
    - source: /boot
      destination: /boot
      type: bind
      options:
        - rshared
        - rbind
        - rw
depends: []
restart: never

This manifests seems to not be the one I need.

192.168.1.179: time="2023-04-01T17:47:44Z" level=debug msg="listing dir /host/dev/sda"
192.168.1.179: time="2023-04-01T17:47:44Z" level=debug msg="listing dir /host/dev/sda1"
192.168.1.179: time="2023-04-01T17:47:44Z" level=debug msg="listing dir /host/dev/sda2"
192.168.1.179: time="2023-04-01T17:47:44Z" level=debug msg="listing dir /host/dev/sda3"
192.168.1.179: time="2023-04-01T17:47:44Z" level=debug msg="listing dir /host/dev/sda4"
192.168.1.179: time="2023-04-01T17:47:44Z" level=debug msg="listing dir /host/dev/sda5"
192.168.1.179: time="2023-04-01T17:47:44Z" level=debug msg="listing dir /host/dev/sda6"
192.168.1.179: time="2023-04-01T17:42:30Z" level=debug msg="mounting /host/dev/sda3 to /boot"
192.168.1.179: time="2023-04-01T17:42:30Z" level=error msg="failed to mount boot partition: not a directory" error="failed to mount boot partition: not a directory"

Manually creating the /boot dir in the container fails due to / being mounted in RO.

The container can see the partitions under /host/dev. But I'm struggling with the mounting part. According to SO the container needs to be in privileges mode 🤔

Following snippet is the mounting code:

	if err := unix.Mount("/host/dev/sda3", "/boot", "", unix.MS_BIND, ""); err != nil {
		err = errors.Wrapf(err, "failed to mount boot partition")
		log.WithError(err).Fatal(err.Error())
	}

OGKevin avatar Apr 01 '23 17:04 OGKevin

how do I make my extension run in privileged mode so I can mount the partition?

the extension services runs as privileged by default

The spec is mentioned here https://www.talos.dev/v1.3/advanced/extension-services/

you don't need the /dev mounted, it should be already there

 mounts:
    - source: /dev
      destination: /host/dev
      type: bind
      options:
        - rshared
        - rbind
        - rw

so in code use as:

if err := unix.Mount("/dev/sda3", "/boot", "", unix.MS_BIND, ""); err != nil {
		err = errors.Wrapf(err, "failed to mount boot partition")
		log.WithError(err).Fatal(err.Error())
	}

frezbo avatar Apr 01 '23 18:04 frezbo

Hmm it's not there.

/dev is mounted as tmpfs when not specifying it as mount in the manifest.

⬢ [Docker] ❯ talosctl -n 192.168.1.179 -e 192.168.1.179 logs ext-rpi-boot-config-loader
192.168.1.179: time="2023-04-01T21:41:14Z" level=info msg=running
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition overlay mounted at /"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition proc mounted at /proc"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition tmpfs mounted at /dev"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition devpts mounted at /dev/pts"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition shm mounted at /dev/shm"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition mqueue mounted at /dev/mqueue"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition sysfs mounted at /sys"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition tmpfs mounted at /run"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition tmpfs mounted at /etc/hosts"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition tmpfs mounted at /etc/resolv.conf"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition overlay mounted at /boot"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition proc mounted at /proc/bus"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition proc mounted at /proc/fs"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition proc mounted at /proc/irq"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition proc mounted at /proc/sys"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition proc mounted at /proc/sysrq-trigger"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition tmpfs mounted at /proc/keys"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition tmpfs mounted at /proc/timer_list"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition tmpfs mounted at /sys/firmware"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="found parition tmpfs mounted at /proc/scsi"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/fd"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/full"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/mqueue"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/null"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/ptmx"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/pts"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/random"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/shm"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/stderr"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/stdin"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/stdout"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/tty"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/urandom"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="listing dir /dev/zero"
192.168.1.179: time="2023-04-01T21:41:14Z" level=debug msg="mounting /dev/sda3 to /boot"
192.168.1.179: time="2023-04-01T21:41:14Z" level=fatal msg="failed to mount boot partition: no such file or directory" error="failed to mount boot partition: no such file or directory"

OGKevin avatar Apr 01 '23 21:04 OGKevin

https://github.com/moby/moby/issues/16160 also suggests to mount /dev:/dev.

Then I run in the same issue as above: msg="failed to mount boot partition: not a directory" error="failed to mount boot partition: not a directory" will try and figure out what this means 😩

OGKevin avatar Apr 01 '23 22:04 OGKevin

Ok, I've managed to mount it using https://pkg.go.dev/github.com/u-root/u-root/pkg/mount/block#BlockDev. At last, progress.

OGKevin avatar Apr 02 '23 08:04 OGKevin

semi working poc: https://github.com/OGKevin/talos-ext-rpi-generic/pull/1

Next part of the puzzle is to discover why the newly written config is not persisted after reboot. Specifically, https://github.com/OGKevin/talos-ext-rpi-generic/blob/d265d5e75927759fcea2dfd304ce0156b02af6c7/main.go#L88-L93 always prints 2 versions, after reboot I expect them to be the same, but this is not the case yet.

OGKevin avatar Apr 02 '23 20:04 OGKevin

Ok, I've a working poc! 🎊.

Next I have to clean up the code and use a boot-config that the user places on the node via machine config instead of my poc hardcoded file. I think this approach is more sustainable.

Afterwards ill make the appropriate PR's.

OGKevin avatar Apr 09 '23 12:04 OGKevin

@smira @frezbo when you have a min, can you have a look at the linked MR so we can get this merged and closed 🙏🏾. Do you also know of a way to test this end to end? I find it quite annoying having to test this on a real PI.

OGKevin avatar Jul 18 '23 15:07 OGKevin

@smira @frezbo when you have a min, can you have a look at the linked MR so we can get this merged and closed 🙏🏾. Do you also know of a way to test this end to end? I find it quite annoying having to test this on a real PI.

will take a look and get it merged before 1.5 release

/boot exists on a normal Talos install so you could create a cluster with qemu and test this. https://github.com/siderolabs/talos/blob/main/hack/test/e2e.sh

frezbo avatar Jul 18 '23 16:07 frezbo

So, re: https://github.com/siderolabs/extensions/pull/145#issuecomment-1650080156

what is now the way forward? @smira @frezbo

OGKevin avatar Feb 17 '24 12:02 OGKevin

Been diving into the overlays functionality in the 1.7.0 alpha. Have successfully been able to build an image with a custom config.txt like this:

docker run --rm -t -v ./_out:/out -v /dev:/dev --privileged ghcr.io/siderolabs/imager:v1.7.0-alpha.1 rpi_generic \
--arch arm64 \
--overlay-image ghcr.io/siderolabs/sbc-raspberrypi:v0.1.0-alpha.1 \
--overlay-name rpi_generic \
--overlay-option configTxt="$(cat config-talos.txt)" \
--extra-kernel-arg=-console --extra-kernel-arg=console=serial0,115200 --extra-kernel-arg=console=tty1 --extra-kernel-arg=consoleblank=0

cfraz89 avatar Apr 01 '24 23:04 cfraz89

Been diving into the overlays functionality in the 1.7.0 alpha. Have successfully been able to build an image with a custom config.txt like this:

docker run --rm -t -v ./_out:/out -v /dev:/dev --privileged ghcr.io/siderolabs/imager:v1.7.0-alpha.1 rpi_generic \
--arch arm64 \
--overlay-image ghcr.io/siderolabs/sbc-raspberrypi:v0.1.0-alpha.1 \
--overlay-name rpi_generic \
--overlay-option configTxt="$(cat config-talos.txt)" \
--extra-kernel-arg=-console --extra-kernel-arg=console=serial0,115200 --extra-kernel-arg=console=tty1 --extra-kernel-arg=consoleblank=0

You can pass in [email protected] which you read the specified option from a file and also use [email protected] which would read the full options from a yaml doc.

I would recommend using configTxtAppend unless you wish to overwrite the full config.txt

https://github.com/siderolabs/sbc-raspberrypi/?tab=readme-ov-file#overlay-options

frezbo avatar Apr 02 '24 04:04 frezbo

I'm closing this since this is now supported.

frezbo avatar Apr 02 '24 04:04 frezbo