nix-installers icon indicating copy to clipboard operation
nix-installers copied to clipboard

fix: make nix work out of the box on rpm-ostree

Open yajo opened this issue 3 years ago • 27 comments

The main difference is that restorecon is executed against /var/nix instead of against /nix, because that's the path that actually exists at that point.

To make it work, SELinux policies are duplicated for the /var/nix prefix.

From there onwards, no more recursive restorecon calls are needed.

Also fixed a bug in the start conditions for the socket.

@moduon MT-83


I know you closed #5 but I hope you can reconsider it now that, with this patch, all works out of the box.

All you need to do is build it, and then rpm-ostree install --reboot ./nix.rpm. Then, after reboot: systemctl enable --now nix-daemon.

yajo avatar Apr 25 '22 09:04 yajo

Booting still warns with:

abr 25 11:32:54 yajolap systemd[1]: nix-daemon.socket: Failed to create listening socket (/nix/var/nix/daemon-socket/socket): Permission denied
abr 25 11:32:54 yajolap systemd[1]: nix-daemon.socket: Failed to listen on sockets: Permission denied
abr 25 11:32:54 yajolap systemd[1]: nix-daemon.socket: Failed with result 'resources'.
abr 25 11:32:54 yajolap systemd[1]: Failed to listen on Nix Daemon Socket.
abr 25 11:32:54 yajolap systemd[1]: Starting Nix Daemon...
abr 25 11:32:54 yajolap systemd[1]: Started Nix Daemon.

However it seems to work as expected.

Also, since Silverblue is so special, it's not really working after boot. You still need to do:

grep -E '^nixbld:' /usr/lib/group | sudo tee -a /etc/group
grep -E '^nixbld.*:' /usr/lib/passwd | sudo tee -a /etc/passwd
systemctl reboot

After installing and rebooting.

yajo avatar Apr 25 '22 10:04 yajo

Hmm it still fails weird when I rpm-ostree update.

yajo avatar Apr 25 '22 11:04 yajo

I know you closed https://github.com/nix-community/nix-installers/pull/5 but I hope you can reconsider it now that, with this patch, all works out of the box.

I closed it because I didn't feel motivated to push it through. Of course it's more than welcome for others to pick up the mantle :)

adisbladis avatar Apr 25 '22 15:04 adisbladis

Thanks! I really love Fedora Silverblue and nix, and both share similar mindsets, but not being able to mix both was very sad.

I just hope I can get this to a point where I don't sacrifice experience for normal Fedora users. Having a couple extra systemd units shouldn't be a big problem as long as they just work IMHO.

The point of having to add nixbld group and users manually is relatively normal in rpm-ostree world. It's actually documented. We would just drop a note in the readme and it should be enough.

Now the biggest problem I have with current code is that it logs this after each boot:

abr 26 07:11:44 yajolap systemd[1]: sockets.target: Found ordering cycle on nix-daemon.socket/start
abr 26 07:11:44 yajolap systemd[1]: sockets.target: Found dependency on nix.mount/start
abr 26 07:11:44 yajolap systemd[1]: sockets.target: Found dependency on nix-setup.service/start
abr 26 07:11:44 yajolap systemd[1]: sockets.target: Found dependency on basic.target/start
abr 26 07:11:44 yajolap systemd[1]: sockets.target: Found dependency on sockets.target/start
abr 26 07:11:44 yajolap systemd[1]: sockets.target: Job nix-daemon.socket/start deleted to break ordering cycle starting with sockets.target/start

For some weird reason, it thinks that nix-setup.service depends on basic.target, which itself depends on sockets.target and creates a cycle. The error itself produces some weird logs at boot and shutdown, and has some side effects such as not being able to log in an Xorg session. Obviously not acceptable.

My current workaround is to systemctl disable nix-daemon.socket nix-daemon.service and then systemctl start nix-daemon manually after each boot. But that's not very elegant.

I've tried several ideas to fix the loop, but none of them worked. Do you have any suggestion?

yajo avatar Apr 26 '22 06:04 yajo

Finally it seems I found how to deal with all corner cases and unit loops. I added to the README specific rpm-ostree instructions. I'm using it right now on my dev PC and works fine. Also tested in a bare-bones Fedora Silverblue VM I've set up. Just follow the instructions and all will go fine.

yajo avatar May 02 '22 14:05 yajo

On Silverblue 36, after rpm-ostree install --reboot ./nix-multi-user.rpm (and rebooting), nix --version failed, because nix wasn't even in $PATH. From what little I gathered from the boot logs, no nix-related services were started either. I ran systemctl enable --now nix-daemon once and after

# Verify all works
nix --version
# Create nixbld group and users
grep -E '^nixbld:' /usr/lib/group | sudo tee -a /etc/group
grep -E '^nixbld' /usr/lib/passwd | sudo tee -a /etc/passwd
sudo groupmod nixbld -aU $(grep -oE '^nixbld[[:digit:]]+' /usr/lib/passwd | tr '\n' ,)
# Reboot again
systemctl reboot

everything worked as expected. I've run rpm-ostree upgrade multiple times since and I haven't run into any issues. Thank you for your work, it's much nicer than running nix in a toolbox.

blackberryshortcake avatar May 26 '22 20:05 blackberryshortcake

Thanks for your feedback. I rebased and updated the rpm-ostree-specific instructions on the readme to make them more straightforward.

yajo avatar May 27 '22 06:05 yajo

Are there any plans to merge this? I am currently using my rather janky script to get Nix running on my machines, and would like to migrate to this.

jcdickinson avatar Aug 02 '22 00:08 jcdickinson

I've been using this for 2 months on my dev Silverblue machine and it works like a charm. It's up to the maintainer to decide if he wants to merge. On my side AFAIK it's all done.

yajo avatar Aug 02 '22 09:08 yajo

@yajo: I am attempting to bake this into an OSTree. I have found that this is needed in nix-setup.

+ mkdir -p /var/nix/var/nix/daemon-socket/
+ touch /var/nix/var/nix/daemon-socket/.keep

  # Set up root channels ()

Without that, the daemon-socket dir doesn't exist, which fails the prerequisite for the systemd socket.

jcdickinson avatar Aug 08 '22 21:08 jcdickinson

Would you mind to open a PR against my branch?

yajo avatar Aug 22 '22 11:08 yajo

@yajo: https://github.com/moduon/nix-installers/pull/1

jcdickinson avatar Aug 22 '22 14:08 jcdickinson

Hi folks. This has been working without any hassle for some months. Do you think you could merge it?

FWIW I built it locally, so you can test it fast: nix-multi-user.rpm.zip

yajo avatar Oct 27 '22 12:10 yajo

I rebased and rebuilt to include #14 and other changes. Updated zip: nix-multi-user.rpm.zip

yajo avatar Nov 04 '22 16:11 yajo

https://gitlab.com/ahayzen/silverblue-nix check this out

dnkmmr69420 avatar Nov 23 '22 17:11 dnkmmr69420

@dankmemermusic

We need to disable SELinux otherwise the nix socket cannot be created.

This repo does not have that issue, it creates the required SELinux policies.

jcdickinson avatar Nov 23 '22 21:11 jcdickinson

What's the current status of this being merged? If it needs more testing, I can install it locally and see if I can find any issues.

orowith2os avatar Nov 25 '22 16:11 orowith2os

Hm, found a weird issue.

If I run my rpm-ostree config without this installer, it works just fine. However, the moment I install it, my terminal apps don't launch, and Firefox's theme resets. Uninstalling this installer fixes all of that. It seems to happen regardless of whether or not the systemd service is going, too.

orowith2os avatar Nov 26 '22 04:11 orowith2os

I've been using this package for more than 6 months without an issue.

@orowith2os have you followed instructions in the readme? Those that are added in this PR

yajo avatar Nov 28 '22 08:11 yajo

I've been using this package for more than 6 months without an issue.

@orowith2os have you followed instructions in the readme? Those that are added in this PR

I did them step-by-step, and still have the issue.

My Firefox theme goes back to light, Nautilus and any terminal apps don't open, and some other (GNOME) apps don't work, like Disks. Are there any logs I should upload?

orowith2os avatar Nov 28 '22 17:11 orowith2os

Could you make sure your nix profile is totally empty? I'm thinking that maybe some nix apps are conflicting with usual fedora apps.

Otherwise, you could post your logs for all *nix* systemd units. And the same for any other failed units you find when booting with the nix package installed.

yajo avatar Nov 29 '22 09:11 yajo

@yajo I haven't installed anything via Nix, only enabled it, so I don't see why anything would conflict.

On the systemd unit topic, any commands I should use to grab the logs?

orowith2os avatar Dec 02 '22 02:12 orowith2os

okay this is a lil weird

I thought I could get into a terminal using vscode's terminal emulator, but something's weird with dbus, I think, and it gives me "the name is not activatable"

orowith2os avatar Dec 02 '22 02:12 orowith2os

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/should-the-default-nix-store-location-be-below-usr/32222/13

nixos-discourse avatar Aug 31 '23 09:08 nixos-discourse

I now installed nix with the rpm package, but when I run nix in my shell I get command not found.

The only places where I have found a nix executable are:

/home/<username>/cargo/registry/index/index.crates.io-6f17d22bba15001f/.cache/3/n/nix
/nix/store/11glc2yk4jmycvk42q6kikpdcpbnbf6w-nix-2.17.1/bin/nix
/nix/store/11glc2yk4jmycvk42q6kikpdcpbnbf6w-nix-2.17.1/share/bash-completion/completions/nix

should I just add the bin in the store to path? 😬

maribox avatar Mar 11 '24 05:03 maribox

Not even after rebooting?

yajo avatar Mar 11 '24 08:03 yajo

I found out that it works when I run bash but in my standard shell zsh it does not work

maribox avatar Mar 11 '24 19:03 maribox

I'm no longer using rpm-ostree systems, and https://github.com/DeterminateSystems/nix-installer supports those, so I have no longer interest in this feature. Closing. You can continue in #22 if you wish.

yajo avatar Aug 28 '24 09:08 yajo