RedBearAK
(Enh) Installing on immutables (or in containers in general)
FYI: @bronson
This is a thread to keep track of attempts to get Toshy installed and working reliably on "immutable" Linux distros, and/or in a container environment on top of some distro that might not otherwise be immediately compatible with the installer script.
Container tools that should probably be supported:
toolbox(Silverblue/uBlue, Fedora-based immutables)distrobox(many distros)- Others?
Goals
- No layering of packages (not compatible with clean "rebase" operations)
- Install/use any native packages necessary from containers
- Survive both reboots and rebases (if possible)
General situation
-
Immutables still have an editable
/home/username(where most of Toshy's components are already being installed), and an editable/etc(where the Toshyudevrules file anduinputmodprobe conf file is placed). So, theoretically the installation of native packages that are not inherent to the base image is the main issue, which will probably require creating and working in a container image to avoid any package layering on the base system image. -
All dependencies Toshy needs should be prompted for install, regardless of whether they are expected to be in any base image.
-
It may or may not be possible to standardize on using the same container image on different immutable base distros.
-
Detection of being on an immutable distro may be difficult, depending on how the distro info reflects the nature of the distro. A new CLI argument might be necessary to prompt the attempt to do an "immutable-compatible" install procedure.
-
Given the different options for container tools that might be used, the script might need to ask the user for input on the available choices that are detected on the system. With rational defaults, if possible.
Approach
The installer script already uses the binaries installed inside the Python virtual environment to run Python and the pip package installer process inside the venv without "activating" the venv. The activation script in the venv would open a new shell and interrupt the installer script process until that sub-shell was exited.
A similar approach should work for both installing native packages and running commands inside a container, using the appropriate commands for the container tool that is chosen, such as toolbox run. The container can be used from the outside like a tool, rather than treated as a shell that must be entered to run commands inside the container environment.
All input on the most correct way to support immutables and/or containers is welcome.
The experimental nature of immutables may require some odd workarounds for a while. There is one already in the Toshy installer, that pulls group info from another file on the system to prevent failure of adding the user to the input group:
if cnfg.DISTRO_NAME == 'silverblue-experimental':
# https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/
# Special command to make Fedora Silverblue/uBlue work, or usermod will fail:
# grep -E '^input:' /usr/lib/group | sudo tee -a /etc/group
command = "grep -E '^input:' /usr/lib/group | sudo tee -a /etc/group >/dev/null"
subprocess.run(command, shell=True, check=True)
Note that for now the distro ID of "silverblue-experimental" will need to be given as a distro override option to the installer in order to use the related code blocks in the installer. There are some preliminary sections to deal with Silverblue, but they are incomplete.
Now that I have more experience using immutable distros, I have a better idea of how I'd like Toshy to work with them.
First, I don't think using toolbox or distrobox are useful. They're too unreliable. They're great for spinning up an environment to do one thing, but they're too fragile to rely on for system stuff. I want Toshy to keep working for years without needing attention, and that's just no distrobox or toolbox. Their guests require a lot of maintenance and they both still spontaneously break now and then. That's not much of an issue if it's something you spin up a couple times a week to do some exotic builds, but that's a big issue if you want your system to rely on it 24/7)
As for your other goals, 100% agree:
- No layering. Layering was already seen as an antipattern when you wrote the above, and now bootc is getting rid of it entirely.
- Survive Reboots. For sure!
- Survive Rebases. welllll that's a big ask. Some rebases change out the whole world.
Suggestion
So, how should Toshy make itself available to immutable distros? I think Flatpak. I think that would be the easiest way to distribute Toshy's services to the widest number of distributions and wild custom one-off images.
Admittedly, a Toshy Flatpak would be pretty exotic... definitely a lot more involved to create than a typical flatpak. But this would come with some upsides: it takes care of building and dependencies, so the same package would work in lots of different places. It takes care of discoverability and distribution. And of course Flatpak is easy to use (well, once Gnome fixes its software app to not constantly force you to wait on a spinner, so needless and tedious ... we'll get there one day).
Flatpak nicely solves all the issues you note in the "General Situation" section above.
Alternatives
Why not just build in ~/ as Toshy currently does? Well, most immutables very explicitly omit build dependencies. Last time I looked into Toshy-on-Immutables, my plan was to script building Toshy in a distrobox guest, then install its artifacts on the host. I no longer think that this would be a good use of time. Building would be super time consuming, very fragile, and fairly custom for each distro.
Another option, very much in the spirit of immutables, would be to simply add Toshy as a variant during the image build process. bluefin-latest-toshy-nvidia. This would be reasonably easy to get working for one image once but, to get any sort of coverage among all the immutables, this would be a full-time and very tedious job. No bueno.
Admittedly
Flatpaking Toshy would be a massive architectural change. It would jettison some very mature software that you've written (there's a lot of knowledge in those build scripts), and would probably require some exotic Flatpak workarounds. Some install steps would need to be moved to runtime and be able to convey the necessary fixes to the user ("Please install this Gnome extension.") It wouldn't merely be packaging existing software; it would require a fair bit of new code to be written.
Personally, I think it would be worth it, but I may change my mind once I start messing with Flatpak... we'll see.
[^1]: Bluefin, still really impressed with it
@bronson
I think a container would theoretically work, except for the mapping of permissions that would have to be done so that processes running inside the container can see and interact with things like the event devices and uinput. But yes, they seem fragile and need maintenance. Which I notice almost nobody ever talks about; updating containers. They're always just creating and destroying them.
Flatpak in a sense is a similar idea, where you're partially isolating the software from the host OS so it's easier to make it portable across OSes. But, I've had many conversations with the LLMs about the possibility of making a Toshy Flatpak, and the conclusion has been pretty universal that various parts of what Toshy needs (I show them the setup script) would have to be external to some kind of central Flatpak app. In other words, probably a fairly pointless endeavor. The Flatpak itself probably could not take the necessary steps internally due to sandboxing and permission issues. Not without a lot of manual intervention in expanding its privileges.
Anyone and everyone is of course welcome to take the project and try to melt it down into a different form entirely, that might be more portable (hopefully under a different name if the modifications are drastic). I always just wanted something that would install and work on most of the standard non-immutable popular desktop distros that people who also use macOS might be likely to use for "desktop" tasks. And I've covered that better than I had initially hoped was possible.
Layering was already seen as an antipattern when you wrote the above, and now bootc is getting rid of it entirely.
If that's true then there probably won't be an easy way to use Toshy on immutables when that happens. I wasn't able to live in an immutable for other reasons (tried Aurora for a while), so I'm not highly motivated to find a resolution to that myself. Though I am curious what the intended replacement is for what people were accomplishing with package layering.
No matter the outcome, I appreciate you taking the time to lay out your viewpoints on the situation. That's always a good thing to do. 👍🏽
I'll be away for a couple of weeks, so I'll be slow to respond to anything for a while.
Which I notice almost nobody ever talks about; updating containers. They're always just creating and destroying them.
Right! I think that's because they're expected to be fragile. They're cattle: delete and recreate if anything goes wrong. (and it will go wrong...)
Yeah, there are a number of things in Toshy's install script that would need to be moved to runtime, which would require fundamental changes to the code. So, writing new code. Code that would be useless to Toshy as it exists today. Yeah, not great. I don't see a way that I like better though.
I am curious what the intended replacement is [for layering]
In theory, sysexts: images that you can compose on top of other images. They're still quite a long way off though. Layering will be available for a long time.
As things mature, hopefully there will be less need for system-level extensions at all. For example, I used to layer kernel mods to make things work on my Framework... those are now upstream. As images get better, layering will decrease. Hopefully.
As things mature, hopefully there will be less need for system-level extensions at all. For example, I used to layer kernel mods to make things work on my Framework... those are now upstream. As images get better, layering will decrease. Hopefully.
To me this is a Zeno's Paradox kind of thing. You'll keep continually decreasing the number of people who need to do something like layering to support some commonly used hardware or software, but you can't include all possible packages in the base image, so there will always be someone... Lather, rinse, repeat.
But I could be completely wrong about that. Eventually.
Naw, I think you're right. Image-based OSes are great for many people, but a poor fit for people who want deep customization and aren't interested in building their own images. And that's fine -- they'll continue to use distros and go deep and show the world how things could be even better. Porque no los dos.
The great thing about Zeno's paradox is that, after enough time, you get close enough. :)
update: I've found that I personally need Mac keybindings in Zed and VS Code most of all. I'm going to experiment with fixing that first and, if I can get Vim keys on Ctrl and all other keys on Cmd/Alt in a pleasing manner (like the Mac has OUT OF THE BOX), then I'll loop back and look at this.
Until there's a more clear path forward, I vote to close for now.