gluon
gluon copied to clipboard
freifunk-gluon
LXC virtualiziation for embedded devices
Hi,
more powerful embedded devices like the Turris Omnia have enough resources to run Gluon in a LXC-container but not enough resources and hardware to run VMWare or VirtualBox.
I suggest to implement support to run Gluon in LXC-containers and offer target-specific containers via the Linux Containers - Image server (e.g. ARMv7L on Turris Omnia).
We are only a downstream project of OpenWrt, imo container support would need to happen upstream.
OpenWRT already supports LXC (just install luci-app-lxc and dependencies for testing). The missing link are LXC-compliant container-images of Gluon.
The container-image needs a config-file and a rootfs-directory:
container directory
|- config
|- rootfs/
Example config-file of Ubuntu LXC-container with the name "smarthome" on Turris Omnia (/srv/lxc/smarthome/config):
# Distribution configuration
lxc.arch = armv7l
# Container specific configuration
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /srv/lxc/smarthome/rootfs
lxc.utsname = smarthome
# Network configuration
lxc.network.type = veth
lxc.network.link = br-lan
lxc.network.flags = up
lxc.network.name = eth0
lxc.network.script.up = /usr/share/lxc/hooks/tx-off
lxc.network.hwaddr = xx:xx:xx:xx:xx:xx
This creates a virtual "eth0" in the LXC-container bound to "br-lan"-bridge on the host.
A LXC-template downloads/copies the files of the rootfs.
The host kernel would need to provide batman-adv support (and I'm not sure if all relevant debugfs interfaces have been replaced by Netlink yet - this is a prerequisite for using batman-adv with containers). Generally, I'm not convinced creating such dependencies between a Gluon container and its host is a good idea - at the very least, LXC would never be an officially supported target of Gluon with batman-adv.
We might look into containers when our support for alternative routing protocols like Babel has matured a bit, but for now, this is still considered experimental.
the babel build does not require specific kernel modules apart from tun when run with fastd.
For wireguard the same limitation applies, the modules must be provided by the host. @renne if you are willing to commit some resources to solving this, be my guest. However I have to say I question the usefulness of the approach.
On gateways the story may be different. Even then you are bound to using the same network stack in order to get anywhere meaningful in terms of performance.
Actually, since batman-adv v2018.2 the debugfs part in batman-adv is even disabled by default:
batman-adv: Disable CONFIG_BATMAN_ADV_DEBUGFS by default
All tools which were known to the batman-adv development team are
supporting the batman-adv netlink interface since a while. Also debugfs is
not supported for batman-adv interfaces in any non-default netns. Thus
disabling CONFIG_BATMAN_ADV_DEBUGFS by default should not cause problems on
most systems. It is still possible to enable it in case it is still
required in a specific setup.
Signed-off-by: Sven Eckelmann
https://git.open-mesh.org/batman-adv.git/commit/6f542489678248cc99131f2856158bd2c3989368
So you might actually be quite good to go now for a try to run Gluon in a container.
@renne: If you could build the Gluon x86 target, copy its rootfs, craft a matching LXC config yourself and if you could let us know whether that works for you already or what does not work yet, that would be very interesting.
I don't know what requirements OpenWrt would have to be able to run within an LX container. You might need to try with a very recent Linux kernel or with an OpenWrt (Linux kernel) on the host itself.
I would also love to run Gluon via LXC (LXD? systemd-nspawn?). For instance, i have a Pi 3 here which runs LXD but which is not capable of full virtualization. There a Gluon container would be pretty nice, too.
I created a LXC-container-config on the Turris Omnia and copied the rootfs of gluon-ffffm-v2.4-stable-0102-raspberry-pi-2.img. The first problem I ran into is ubus:
root@freifunk:/# /etc/init.d/network reload
Failed to connect to ubus
How is ubus integrated in Gluon? First config:
# Distribution configuration
lxc.arch = armv7l
# Container specific configuration
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /srv/lxc/freifunk/rootfs
lxc.utsname = freifunk
# Network configuration
lxc.network.type = veth
lxc.network.link = br-lan
lxc.network.name = eth0
lxc.network.hwaddr = 92:e3:2d:2c:8d:41
lxc.network.flags = up
lxc.network.script.up = /usr/share/lxc/hooks/tx-off
lxc.network.type = vlan
lxc.network.link = eth0
lxc.network.vlan.id = 4
lxc.network.name = eth1
lxc.network.hwaddr = 92:e3:2d:2c:8d:42
lxc.network.flags = up
lxc.network.script.up = /usr/share/lxc/hooks/tx-off
@renne:
How is ubus integrated in Gluon?
In the same way as it is integrated in OpenWrt. I think the ubusd should be started by procd very early in the boot process.
Could you check whether a) ubusd (and procd?) is running? And if not, whether you can spot some error message with the "logread" command or on a serial console?
@CodeFetch: I found your comment very inappropriate... especially as it's disrespecting the work renne has (started) to put into this. Or at least that the tone of your comment could (even if unintentional) come across that way.
@T-X Actually I think there should not be put any effort into supporting this. It is not my intention to discredit rennes work. There are just so many possible pitfalls with using Gluon in LXC containers, that we might shoot ourselves in the foot with that. Gluon depends on all kind of (possibly patched) kernel modules and the LXC host needs to support them. Ordinarily we do not encourage running Gluon on non-dedicated hardware. With supporting LXC containers we actually do that and I think that is a bad idea.
@CodeFetch: There has been a lot of work going into upstreaming kernel changes. The only issue I'm currently aware of is concerning the gluon-ebtables-limit-arp package, that it would not perform that well (and the according kernel patch to change that was just rejected, as noted in #1113 - we will probably need to keep that patch until we or OpenWrt switch to ebtables-netfilter).
There are currently no batman-adv or bridge related patches I'm aware which would be missing in an upstream kernel.
Concerning non-dedicated hardware: This is already common. Many communities use x86 Gluon images in VMs on servers and boxes they want to have accessible within the mesh network. And that's really neat because they will be autoupdated. Unfortunately, virtualization is not really performing well on ARM yet...
Another option I keep thinking about, especially after the admin gateway discussions @Adorfer had organized here at the 35c3 yesterday (thanks!): Maybe it could be interesting to actually use a Gluon image on gateways. At least for the mesh routing and VPN parts as a start. That would only need relatively minor adjustments in Gluon like firewall settings and batman-adv gw-mode which could be achieved via some new package and site.conf/mk options. And people could then remove the VPN and mesh routing parts from their Ansible/Puppet/Chef/... And it would hopefully be less work to update routing protocols or VPNs thanks to the autoupdater and #1555 again then. And then all the internet / bird / dhcp / ... fun stuff would be added outside of the Gluon image (as a start).
Unfortunately, most/many gateways are running within VMs. And some nested virtualization was added in some recent kernel version only. Performance wise it would be great if for such an approach the Gluon image could be integrated as a container instead of nested virtualization or the necessity of two VMs next to each other.
Sorry, this got longer than I had intended. @CodeFetch, if there is more to discuss regarding the general fitness of LXC then let's best do it here at the 35c3 in person :-).
@renne : Great idea :-)
Running in mininet-wifi [1] is also an interesting option for automated integration testing. The ability to start gluon by firing up a process in a namespace (using lxc or not) is interesting, too. It is clear, that gluon has to rely / use the host kernel than ... but actually, this is a plus some situations.
[1] https://github.com/intrig-unicamp/mininet-wifi
We have the infrastructure to use KVMs for this. Everything else means more effort and we already have a lack of contributors. If you want to simulate wireless links (which should be the wireless driver developer's job; and Qualcomm has a real hardware testbed with a Github repo somewhere for this), there is a kernel API for simulations as far as I know.
but actually, this is a plus some situations.
In which case is this an improvement? Don't understand me wrong I am not against the idea of LXC containers in general. I just don't think there is anyone willing to implement and maintain this. Whether a "Freifunk-Verein" pays 20€ (KVM-capable machine) or 10€ (non KVM-capable) for a server per month does not make a difference as the non KVM-capable machine has likely a bad performance anyway. And LXC machines from ordinary providers would use a kernel which they couldn't use without much pain... The only use-case I see for supporting LXC-containers would be that someone out of 10,000 would want to run a Gluon-node in an LXC on a router, because he or she "doesn't want to run a dedicated Freifunk router". There is no off-the-shelf router which supports LXC "apps" in their "app store" where communities could provide this "app" and tell people: "Just install the Freifunk app in your Fritz!Store and you have Freifunk". Find me 10 people who ~~want~~ need this feature and I would change my mind.
We have the infrastructure to use KVMs for this. Everything else means more effort and we already have a lack of contributors.
Please reconsider. It's neither inviting nor motivating to reject new ideas, especially, if they're not driven by existing contributors. It's ok, to consider maintainability - but understand it as an architectural requirement (e.g. code should not be tightly coupled with existing scripts, should be easy to remove).
If you want to simulate wireless links (which should be the wireless driver developer's job; and > Qualcomm has a real hardware testbed with a Github repo somewhere for this), there is a kernel API for simulations as far as I know.
Testbeds and simulations go hand in hand. The kernel API is actually scripted (Python) by mininet-wifi (mac80211_hwsim).
but actually, this is a plus some situations.
In which case is this an improvement? Don't understand me wrong I am not against the idea of LXC containers in general.
Performance / footprint is a plus. It's easier to fire hundreds of processes than to maintain the same amount of isolated virtual machines.
I just don't think there is anyone willing to implement and maintain this. Whether a "Freifunk-Verein" pays 20€ (KVM-capable machine) or 10€ (non KVM-capable) for a server per month does not make a difference as the non KVM-capable machine has likely a bad performance anyway.
That's not the use-case. It's about hacking and research (benchmarking hundreds of nodes) esp. in conjunction with SDN.
I guess, its easier to have this discussion on IRC, though. However, I'll sponsor some Club Mate for @renne if LXC is up and running :1st_place_medal:
Greetz, yanosz
(e.g. code should not be tightly coupled with existing scripts, should be easy to remove).
It frustrates people more if we remove a feature, because it is unmaintained and breaks at some point.
Testbeds and simulations go hand in hand. The kernel API is actually scripted (Python) by mininet-wifi (mac80211_hwsim).
That part is something @lemoer might find useful, but as it needs to run on nodes, it might make sense to port it to C.
Performance / footprint is a plus. It's easier to fire hundreds of processes than to maintain the same amount of isolated virtual machines.
I don't believe that the performance impact is so big.
That's not the use-case. It's about hacking and research (benchmarking hundreds of nodes) esp. in conjunction with SDN.
We have real-world problems and many important projects in Gluon that are stuck at the moment where a solution is already known, but not implemented. Gluon is a collaborative, some say meritocratic, but quite anarchistic project. Thus if someone wants to do this, feel free to do it, but don't expect many people to help. They are busy. That's all what I wanted to say:
If you want to implement this, please do it in a way that does not need maintenance or please maintain it. Be aware that this project has very few maintainers while this is a quite complex project and that there are stuck features already that need support with a very widespread need.
It's the same when you do kernel development. In most cases you are a kernel janitor for years. Some are for their whole life. Look at how many patches get rejected, because there will likely be nobody who does the maintenance. Gluon does not reject this. What you read here is just an opinion of someone who does not even have any authority here.
@christf Everyone who does virtualization in a way which would work with LXC Gluon - including every Freifunk community. In LXC you can only use the kernel/kernel modules provided by the host machine. Thus my advice is: Just use KVM which some people already did with Gluon and don't do anything you can potentially shoot yourself in the foot or waste time on. KVM's performance is sometimes even better than LXC's. The only scenarios I can think of where one would use LXC is on an underpowered board whose architecture can't do proper virtualization or on a $1 virtual machine from a budget-hoster where you have the problem that you can't influence the kernel which will likely not have the features needed for Gluon.
A small update, as this came up in the meetup today:
OpenWrt 19.07 and newer now seems to work as a container in LXC/LXD, LXD even provides prebuild container images you can just install:
- https://us.lxd.images.canonical.com/
There is an OpenWrt Wiki entry here for configuring an LXC container:
- https://openwrt.org/docs/guide-user/virtualization/lxc
And here is a longer article on how to build+configure an LXD container (which also mentions a procd bug which was fixed in OpenWrt 19.07):
- https://github.com/cvmiller/openwrt-lxd
I'm not sure if OpenWrt on LXC/LXD now also works with unprivileged containers or if a privileged one is needed.
Regarding kernel on host, I think as long as it's recent and has a relatively recent batman-adv (same or newer as in Gluon's/OpenWrt) it should work. There are two patches which are not upstream yet, but should only result in missing features and not breakage (needs to be tested/verified):
- https://github.com/freifunk-gluon/gluon/blob/v2021.1.x/patches/packages/routing/0003-batman-adv-Introduce-no-noflood-mark.patch
- https://github.com/freifunk-gluon/gluon/blob/v2021.1.x/patches/openwrt/0015-kernel-bridge-Implement-MLD-Querier-wake-up-calls-Android-bug-workaround.patch
batman-adv would probably need a privileged container for now as the following patch was merged upstream but it is not part of a release yet (should be in the upcoming batman-adv v2021.5 / Linux 5.17). After this one batman-adv itself works fine in an unprivileged container and an unprivileged container can create and fully maintain batman-adv interfaces on its own:
- https://git.open-mesh.org/batman-adv.git/commit/055fa41b73ca8dae1c1ed41777e32a8f02e80c82
Sysupgrade might not work yet, as @NeoRaider mentioned in #2355. The LXC/LXD tutorials I found were using and unpacking the rootfs tarball, not the OpenWrt combined image.
Generally it might be easier to use and maintain LXD instead of plain LXC, as LXD has a portable image concept. So maybe upstream OpenWrt could be taught to produce LXD images on its own (which is basically just a rootfs tarball and a YAML config file combined in one tarball, as far as I understand).
My opinion is that Gluon should not be used in containers. The kernel is an integral part of Gluon, and we occasionally need to include additional patches to make things work (and that doesn't even consider all the patches that are part of OpenWrt). Besides batman-adv, we also use many netfilter modules, and optionally L2TP. While most desktop distributions will provide these by default, the complex (and changing) dependencies between Gluon and the kernel would be an additional source of error.
procd has some support for running OpenWrt in LXC (the mentioned luci-app-lxc is for running LXC on OpenWrt, not for running OpenWrt in LXC though), but I'm pretty sure that sysupgrades are not working. Supporting sysupgrade in a container would require adding a completely separate code path, as (as far as I know) all current sysupgrade implementations work on MTD/UBI/Block Device level, replacing whole filesystems, while a container would need to replace the individual files of its rootfs.
With the addition of scripts like contrib/run_qemu.sh, using qemu for development has become very easy, so I'd generally recommend to use that instead of containers.