distrobuilder icon indicating copy to clipboard operation
distrobuilder copied to clipboard
verified publisher icon lxc

OpenWRT VM images

Open tregubovav-dev opened this issue 1 year ago • 23 comments

Linux containers Image server distributes OpenWRT containers-only images, which is suitable for multiple use cases. However, containers are unsuitable for some cases, especially when specific devices and kernel modules are required. The VM is a good solution as OpenWRT resources requirements are low. For example, a customer needs internet access via a mobile network solution running in the Incus environment in ARM-bases SBC. So OpenWRT container could not be used for him as container can't handle kernel modules and devices for mobile modem. I was able to build an OpenWRT VM (aarch64) image for him using the LXC/Incus distrobuilder with some caveats:

  • Unable to prepare EFI bootloader with the distrobuilder (Issue 880). The distrobuilder uses grub-setup packages to build the EFI bootloader in other Linux VMS. However, OpenWRT does not provide such a package, but the EFI partition content could be taken directly from the combined image.
  • Unable to include the incus-agent to the build, making it impossible to access the VM console from the Incus client.

What needs to be done to build OpenWRT VM image:

  • Distrobuilder needs to be able to build an EFI partition and upload the following files from the EFI partition of generic-ext4-combined.img :
    • /efi/boot/bootaa64.efi
    • /efi/openwrt/bootaa64.efi
    • /efi/openwrt/grub.cfg
    • /efi/openwrt/vmlinux
  • Distrobuilder must be able to substitute PARTUUIDs in the /efi/openwrt/grub.cfg file
  • Distrobuilder must install incus-agent into VM. OpenWRT uses a "procd" based init system.

One more caveats is OpenWRT VM unable to give access to boot console. The command "incus start owrt-vm --console" displays the error "Error: Failed to swap console ring buffer with socket: Chardev user does not support chardev hotswap" instead of console access.

tregubovav-dev avatar Dec 30 '24 07:12 tregubovav-dev

I was tried to lunch an incus-agent in the OpenWRT VM and it fails:

incus-agent --verbose --debug
INFO   [2024-12-31T03:07:19Z] Starting
INFO   [2024-12-31T03:07:19Z] Loading vsock module
INFO   [2024-12-31T03:07:19Z] Stopped
Error: Unable to load the vsock kernel module: Failed to run: modprobe -b vsock: exit status 255 (modprobe: unrecognized option: b)

This issue displays at least 2 problems:

  1. Most of OpenWRT commands have reduced list of options.
  2. Most of modules must be installed via package manager
  3. OpenWRT armsr/armv8 (aarch64) kernel is not built with CONFIG_VHOST_VSOCK flag and there is no package which includes vsock module. I looked to the incus-agent and find that it can load several kernel modules. How can I know which modules are required and which are optional? I'm going to check whether these module available and ask to add support for missing ones.

tregubovav-dev avatar Dec 31 '24 03:12 tregubovav-dev

The only hard dependency for the agent is vsock. Our LoadModule calls will usually detect an already loaded module and just move on, so if you set things up to load the vsock module prior to starting the agent, things should just work.

stgraber avatar Dec 31 '24 15:12 stgraber

Thank you Stephane for the confirmation!

I looked through the code and see how does it check already loaded modules. The biggest problem now is that the none of vhost_vsock or vsock module exists in OpwnWRT (nor with bundled kernel, now as a package). I need to see whether it possible build this kernel module and then make request for include it in the snapshot.

Happy New Year!

tregubovav-dev avatar Jan 01 '25 02:01 tregubovav-dev

Looking through the qemu-agent I see it can use three communication methods between guest and host. The default method is a serial, which does not require loading any additional modules. Do you think the incus-agent functionality can be extended with this method ?

tregubovav-dev avatar Jan 03 '25 20:01 tregubovav-dev

It's not something we're interested in attending, no.

The incus-agent is designed for high throughput parallel network transfers, it exposes a full REST API over HTTPS to the host system. Trying to carry all of that onto a single serial connection wouldn't be pleasant.

stgraber avatar Jan 03 '25 20:01 stgraber

Understood.

tregubovav-dev avatar Jan 03 '25 21:01 tregubovav-dev

One more question, Stephane.

Is it possible to use custom local rootfs tar file or directory structure to build image using the distrobuilder? I tried to see the available options, but did not find the solution. The distrobuilder downloads rootfs image every time

tregubovav-dev avatar Jan 05 '25 21:01 tregubovav-dev

One more question, Stephane.

Is it possible to use custom local rootfs tar file or directory structure to build image using the distrobuilder? I tried to see the available options, but did not find the solution. The distrobuilder downloads rootfs image every time

pack-incus command resolved the question.

tregubovav-dev avatar Jan 06 '25 00:01 tregubovav-dev

I'm very interested in an OpenWRT image for Incus VMs as well. I've been using the image for Incus containers and it's been amazing. I'm now in need to install some kernel modules and to set some kernel settings that I'm not able to set on the host. Using a VM would solve this issue. I know there are ways to take a raw image and import it into Incus but having an image available on the image server would be fantastic. Not sure I can be of much help but I wanted to let you know there's more interest in this.

chilicheech avatar Apr 10 '25 04:04 chilicheech

I'm glad to hear that I'm not alone who like running OpenWRT VM under incus or lxd. I would say, the distrobuilder can't solve this problem now until we have OpenWRT build fully supported incus-agent and lxd-agent.

Technically any OpenWRT official build can run as incus or lxd VM, however, incus-agent or lxd-agent is not able to communicate with the host service (incusd or 'lxd). Тor official OpenWRT armv8, nor x86_64builds currently contains full set of kernel modules required to runningincus-agent or 'lxd-agent correctly. So, the snapshot with customized kernel is required to run fully managed OpenWRT VM under incus or lxd. Maintaining OpenWRT with customized kernel require the separate kmod repository maintenance as well.

There are several solutions possible:

  • To request OpenWRT developers to include required changes into x86_64 and armv8 kernel configs. In
  • To ask @stgraber to whether it possible to build customized OpenWRT builds and keep kmod repository at Linuc containers Image Server
  • To build incus/lxd specific OpenWRT build somewhere else and maintain kmod repo separately.

P.S. I prepared steps required for building such steps, and I'm planning to publish them soon.

tregubovav-dev avatar Apr 10 '25 18:04 tregubovav-dev

What are the missing modules at this point? If it's just a few non-controversial ones, hopefully we can get OpenWRT upstream to build them. They should be fine not being built into the kernel, so they wouldn't increase the memory size of any other OpenWRT system, just those that have the modules installed and loaded.

stgraber avatar Apr 10 '25 19:04 stgraber

I will post the config diffs with short explanations later today.

tregubovav-dev avatar Apr 10 '25 19:04 tregubovav-dev

The list of required kernel modules was discussed at Which 9p/other kernel modules are required for incus-agent? earlier. I did experiments with the custom OpenWRT build a month ago and had good progress with building fully managed by incus x86_64 and armv8 OpenWRT 24.10.0 images.

The one of the major challenges was to decide whether it possible to make kmod packages for missed modules without rebuilding the kernel. Unfortunately, this is not possible by couple reasons:

  1. Official builds includes different set of 9p*, vmw_vsock-*, etc. per different architectures. And these modules linked statically to the kernel.
  2. Adding missed modules require changing kernel config. Changing kernel config rebuilds kernel hash and invalidates all kmod packages built for official releases and/or snapshots.
Kernel configuration (optional)
Note that make kernel_menuconfig modifies the Kernel configuration templates of the build tree and clearing the build_dir will not revert them. :!: Also you won't be able to install kernel packages from the official repositories when you make changes here.

So, until the OpenWRT development approve such kernel change change - we unable to use official builds :(

However, it's possible to maintain custom snapshot builds if we can host kernel modules (kmod) repo for such builds.

Below are high-level steps to build custom OpenWRT build: assumes that OpenWRT cloned from official repo, checked out to openwrt-24.10 and complete all prerequisites. More details are here: Build system usage

  • Apply Diff 24.10.0.modules.diff.txt This diff updates kernel configs
  • Use make menuconfig or apply Using official build config. (I would prefer not to .gzip images, so I'm updating Target Images option).
  • Extract files.tar.gz.b64.txt into ./files/ directory. (Use cat files.tar.gz.b64 | base64 -d | tar -xz to extract it). This is a service to install and run incus-agent in the VM.
  • Build the build
  • Convert ./bin/targets<target>/<subtartget>/openwrt-24.10-snapshot-xxxxx-xxxx-xxxx-xxxx-ext4-combined-efi.img to qcow2 format.
  • Prepare image for import: Prepare Image metadata tarball
  • Import image using incus image import <metadata_tarball_path> <rootfs_tarball_path> command.
  • Launch VM in the incus. You should be able connect to the OpenWRT VM console in short time after it starts.

Couple notes:

  • opkg package manager needs to be configure to load kmods from the custom repository.
  • packages kmod-button-hotplug and kmod-input-core needs to be installed to support graceful shutdown operations on aarch64 architecture.

tregubovav-dev avatar Apr 11 '25 05:04 tregubovav-dev

@tregubovav-dev I am interested in OpenWRT vm images as well (in my case, x86-64 only). I really appreciate your notes. I was wondering if there have been additional discussion or progress (upstream?) since your last post you are willing to share. Thanks!

maertsen avatar Jul 14 '25 06:07 maertsen

Hello @maertsen.

Unfortunately, there is no much interest to this topic nor from the OpenWRT community, not from the Linux Container community.

As I wrote before the simplest way to resolve this issue is to enforce OpenWRT maintainers making small changes above. Otherwise someone needs support custom OpenWRT builds and kernel module repository for it.

tregubovav-dev avatar Jul 14 '25 17:07 tregubovav-dev

Thanks. Did you open a PR against the relevant repo for the OpenWRT kernel and/or for an incus-agent package? I saw you asking questions on the OpenWRT forum, but in my experience, that's a user-centric not developer-centric place.

I'm currently trying to understand the missing bits working from distrobuilder itself (using --vm, as opposed to converting the combined-efi images for direct import).

maertsen avatar Jul 14 '25 18:07 maertsen

Thanks. Did you open a PR against the relevant repo for the OpenWRT kernel and/or for an incus-agent package? I saw you asking questions on the OpenWRT forum, but in my experience, that's a user-centric not developer-centric place.

OpenWRT community does not accept PRs without preliminary discussion at the forum, as I know. There is a huge PRs backlog there. The PR must have a support from multiple contributors to be accepted and merged into the snapshot at least.

I'm currently trying to understand the missing bits working from distrobuilder itself (using --vm, as opposed to converting the combined-efi images for direct import).

If we could imagine that the OpenWRT accept the change and start producing builds with fixes - the distroubuilder does need only to integrate scripts to run the incus-agent at startup.

tregubovav-dev avatar Jul 14 '25 23:07 tregubovav-dev

There is some progress with the OpenWRT vm image for incus here: https://forum.openwrt.org/t/vsock-ko-kernel-module-needed-aarch64-x86-64-targets/220316/34

tregubovav-dev avatar Jul 21 '25 03:07 tregubovav-dev

Ah, that's great to see some movement over there, hopefully we can get a suitable kernel eventually!

stgraber avatar Jul 21 '25 09:07 stgraber

We need have tests it and provide feedback for the OpenWRT community and then make some pressure to have it merged to product.

tregubovav-dev avatar Jul 21 '25 16:07 tregubovav-dev

Folks,

If some one interested in to use OpenWRT VM fully compatible to run in Linux Containers environment (both incus and lxd). please participate in the testing and provide your feedback at the OpenWRT Forum

Repost from the OpenWRT forum thread:

Hello Folks. I have finally build openwrt snapshot images (from a main branch) and updated images for 24.10.2.

Images

Kernel Modules and system packages

(for images above)

Could you please test openwrt-snapshot-snapshot.aarch64.tar.xz or openwrt-snapshot-snapshot.x86_64.tar.xz first? I'm going to publish my PR right after that.

tregubovav-dev avatar Jul 23 '25 05:07 tregubovav-dev

Hello Stephane (@stgraber)!

I would appreciate it if you cold have chance to do short test with openwrt-snapshot-* images in your test lab and provide short feedback. I'm going to push PR to the openwrt repo, but I need more independent test confirmations.

tregubovav-dev avatar Jul 25 '25 18:07 tregubovav-dev

Hello, thank you for this effort.

TCB13 avatar Sep 01 '25 08:09 TCB13