OpenDroneMap
Alpine Linux base image
We have someone willing and able to test migrating the base images for WebODM to Alpine Linux, thus enforcing musl lbc, no systemd userland thus elimitating most CVEs, etc. etc.. Before diving into the project wanted to check interest on the WebODM side and if there's interest, logistics to look out for from go.
We'd start with lower complexity images, e.g. redis. And if all images were migrated, we'd have to address the ODM base image as well. But, best to start downstream of ODM and where most of the images are built.
If someone wants to contribute an Alpine image, that's great, we can keep it alongside the Ubuntu one (so long as there's an active contributor willing to keep it up to date). I personally wouldn't like to replace the Ubuntu base.
Aside, it seems widely accepted that musl binaries can be significantly slower than those compiled with glibc, I'm not sure why even consider the tradeoff of image size for speed, especially in programs like ODM/WebODM.
Not sure anyone sees real world penalties with Alpine, probably due to the combination of all the factors of the OS, not just musl/glibc. I know Brett switched over to Alpine as a daily driver on ancient hardware without incident, but that's anecdata.
In his A/B testing against XUbuntu today, some things were ahead and some behind with little difference overall. It will depend on the application, so as we port, we'll test so that folks know if there are penalties or improvements or no measurable difference.
Even though I dislike Ubuntu today (they have chosen certain libraries or tools that only they use, such as netplan, which is a total failure in some cases), it is better to replace it with Debian. I think there are more important things to do before switching to Alpine, such as the Docker container as a non-root user for security reasons, which will help improve compatibility with Singularity/Apptainer.
We have already begun working on ensuring the WebODM docker images are more broadly OCI-compliant and will be able to work both in rootful (current) and rootless methods and across other runtimes like podman/podman desktop, so that is happening independent of this effort.
As for security, switching to the Alpine base image greatly improves our security posture and CVE exposure by having more current depends, and enforcing SSP/PIE and other hardening methods across the entire toolchain. This, in combination with MUSL libc, a busybox userland and no systemd, prevents the vast majority of CVEs (mostly memory corruption attacks) from applying, as I have noted in the past five years of dailying Alpine on my machines.
Docker container as a non-root user for security reasons, which will help improve compatibility with Singularity/Apptainer
Full agree we have a few high priority things to work on, from 24.04 upgrades on ODM in the meantime to rootless containerization, to looking at dependency needs across the stack. My hope is that work on Alpine, regardless of where it lands in testing, helps us identify some dependency upgrades in the Ubuntu stack as well.
This project is almost complete. We've got a couple more generations of CPUs to test on, but we're seeing both image size and performance improvements, in addition to the memory safety that comes "for free" with Alpine. We'll share specific details shortly once testing is complete, likely in the next couple weeks.
I assumed performance would be a wash, but pleasantly surprised to see improvements at least for the images tested so far. More soon.
I am seeing about 18% increase in performance thus far on two machines sampled.
We are adding more machines of different performance levels to see how that performance delta scales, but given about 18% is holding between an Intel Celeron N3450 with 4GB RAM and a FrameWork Intel Core i5-1135G7 with 64GB RAM, I expect it to hold.
I'm not an expert or anything, but I've heard that the default musl allocator is quite slow, and so see people usually using mimalloc instead. supposedly it should perform well otherwise.
here are some notes about it:
- https://nickb.dev/blog/default-musl-allocator-considered-harmful-to-performance/
- https://news.ycombinator.com/item?id=23081071
- https://lobste.rs/s/78syo9/default_musl_allocator_considered
Yes, there is a lot people say about Alpine being slow, but our testing (outlined for a few machines above) illustrates that the common knowledge isn't terribly accurate in many cases.
That matches my experience dailying Alpine for the past five years on various machines.
exactly, wonderful!
I just want to imply, that maybe changing the allocator to mimalloc on musl boosts those performance improvements even further.
what do you think?