Relocatable store objects

[Ericson2314]

Is your feature request related to a problem? Please describe.

Windows

In https://github.com/NixOS/nix/issues/9389 I propose N:\x\store which is the same length, allowing some janky sedding. But it is still not the same thing. I don't think it is not possible to have the same store dir on Windows and Unix, which is a big impediment to having e.g. a single CI system managing remote build on both platforms.

Unprivilaged users without fancy sandboxing

Even on Unix /nix/store is a impediment on systems that do not have good sandboxing support for regular users. Many people have dreamed of instead being able to dump a store in a home directory and simplify the installation process greatly. Of course, without sandboxing builds are sketchy, but that may be an acceptable tradeoff for those wanting to just try out Nix lightly for the first time.

Content addressing build products

We support self-references for content addressed build products, but this solution doesn't always work. For example, imagine if we file paths need to be sorted keys in an on-disk map. When we rewrite hashes, we can break the alphabetical sorting, and it may be impossible to choose a place in the order for our self reference that doesn't have this problem: for every slot chosen, "the content addresses modulo self references" would give us a hash that moves the reference elsewhere in the lexicographical order!

The proper solution to this problem is domain-specific ways to do self references, like $ORIGIN in ELF RUNPATHs, or simply relative paths. But if we do that for self references, we can consider doing that for all store paths.

Describe the solution you'd like

A store object that isn't allowed to refer to the store directory at all. References are still tracked, by looking for their hashes as is done today, but /nix/store/ (or whatever the store dir is) cannot appear before any of those hashes. We will call these relocatable store objects.

Additionally the property much be transitive: relocatable store objects must only refer to other relocatable store objects, that way entire closures are relocatable if the root object(s) is (are) relocatable.

Normally the store object is used as part of the computation of the store path, but for this that would defeat the purpose, so we will not include it, and just use the other information. I think it would also be nice to not support this for input-adressing --- easy to just drop support for this "legacy" way of doing things in this case.

For Nix itself this should be quite easy. The big question is whether packages in the wild (i.e. in Nixpkgs) can be made to cope without absolute store paths. There's only one way to figure out! :)

Describe alternatives you've considered

While only transitively relatable store objects are really relocatable, it might be useful to still also assert in a derivation that its outputs are "locally" relatable. (I.e. "I only refer to my references with relative store paths, but they sadly my use absolute store paths"). Such store objects as those asserted derivation outputs are sadly non-reloctable, but the assertion it still useful when fixing packages in nixpkgs, so progress can be made "out of order" (referring packages before referenced packages) without accidental regressions.

Then again, this sort of check could also just be done "in user space" with a script at the end of the derivation, rather than by Nix itself.

Additional context

Add any other context or screenshots about the feature request here.

Priorities

Add :+1: to issues you find important.

[bjornfor]

For Nix itself this should be quite easy. The big question is whether packages in the wild (i.e. in Nixpkgs) can be made to cope without absolute store paths.

What about the program interpreter in ELF exectuable files, which has to be an absolute path? That's the first thing that comes to mind (as a blocker).

[Ericson2314]

Patch the kernel? :D

In general, on the Nixpkgs side, this would be a process of creating workaround patches and negotiating with upstream what to do about it.

[edolstra]

I cannot realistically see this happening since it's so far outside of how Nix and in particular Nixpkgs are designed. References to the store directory are everywhere, and there is no reliable way to prevent them (grepping for the store directory wouldn't work). So we would have to trust the user that a store path doesn't contain such references.

[Ericson2314]

I think it should be quite easy in Nix itself thanks to you introducing the StorePath type? We just need to store the store dir or lack thereof in ValidPathInfo and voila we can represent this stuff. For reference scanning we already scan for hashes not store path. I think we should just check that that before every found reference there isn't the store directory.

For Nixpkgs, I don't know how are it would be. I think the only way to figure out is to try. It's pretty hard, but so is squashing bugs relating to the limitations of rewriting self-references. Package by package, I would rather spend the time making it relocatable than ensuring absolute self reference rewriting works. (And it's a good thing that many packages already care about relocatability!)

A lot of people that want to distributed relocatable stuff created with Nix would also benefit from this.

[edolstra]

Yes, the Nix changes wouldn't be too hard. But in Nixpkgs, every dependency is expressed as an absolute path. Without the /nix/store, scripts wouldn't even know where to look for dependencies in the filesystem (presumably we would need to pass an environment variable to pass the store directory). Relative paths would sometimes work, but they require knowing the path is relative to. And dynamically-linked ELF executables wouldn't work at all, due to the ELF interpreter field.

BTW I don't see the problem with self-references. Seems to me that they should work exactly the same as they do today?

[Ericson2314]

BTW I don't see the problem with self-references. Seems to me that they should work exactly the same as they do today?

Supposed we have this file which is supposed to be in lexicographical order

/nix/store/bbbbbbbbbbbbb-foo
/nix/store/xxxxxxxxxxxxx-out

and the second one is the temporary path. Then we do are usual hash modulo and rewrite, and get:

/nix/store/bbbbbbbbbbbbb-foo
/nix/store/aaaaaaaaaaaaa-out

uh oh, they are no longer in order!

now, we try it the other way:

/nix/store/xxxxxxxxxxxxx-out
/nix/store/bbbbbbbbbbbbb-foo

but then after rewriting we get:

/nix/store/zzzzzzzzzzzzz-out
/nix/store/bbbbbbbbbbbbb-foo

still out of order!

[ikcii]

I am relatively new to Nix but I've been looking into just how portable you can make Nix without root perms and I have found the following potential approaches:

1. The regular /nix dir way, requires root privileges and/or access to the /nix dir where the Nix store sits

2. Bind mount, also requires root perms for at minimum the initial setup. While we're here, I'll bring up that hard linking might also work in some capacity. It will also usually require root or might not be allowed on some systems such as Android, also from what I'm aware it doesn't work if the source and target dirs aren't within the same filesystem or something like that

3. Proot (fake chroot without the need for root perms), this method allows for things such as faking mounts without the root permissions but is extremely slow for a lot of operations because a wrapper needs to capture them instead of letting the kernel handle them. This approach is used by the nix-on-droid project and it leads to EXTREMELY long rebuilds so it should be avoided if possible, I bet it doesn't help much if any more complex software is ran inside of it

4. User Namespaces (allows for binding dirs to different paths out of other things, which lets you make a program ran inside of such namespace think a /nix dir exists even though it actually points to some dir inside of your home dir) this approach can be implemented by anyone in a single line of bash using bwrap as far as I'm aware, and it is also used by the nix-user-chroot project. It's a relatively good way of handling this problem, however on some systems User Namespaces are disabled or restricted. For example Android, iirc it has this feature disabled in the kernel itself. We'd have to beg Google to enable it which is unlikely to ever happen. On top of that individual distros or system configurations can disable unprivileged namespaces, or limit them to a single "layer" which could limit break things such as using bwrap inside of the namespace, which itself is used by Flatpak and Steam out of others. I don't remember for sure but changes are nix shell itself too? Correct me if I'm wrong here

5. Specifying an alternate nix store directory by hand to some dir inside of the user's home dir (using env vars + Nix's ~/.config). I didn't test this approach thoroughly, however it appears that this is actually completely possible due to how versatile Nix's build system is (then again, it might turn out that this approach doesn't work at all, I was on a weaker laptop when testing things with podman and I didn't finish the initial compile, though it did appear to be functioning properly at a glance). It does pose a big problem though: Changing the absolute store path changes the hash of every compiled package. This means that you have to larp as a Gentoo user and recompile the entire world set or whatever it's called by hand, rendering the entire nix binaries cache useless. On top of that, since the home dir's absolute path changes depending on your username, only users with identical usernames would be able to reuse the compiled binaries. This extremely reduces the viability of a binary of an official/global nix binary cache for most regular Nix users.

6. Alternate nix store dir inside of user's home dir or similar, but on Termux specifically. Termux is an Android app (the earlier mentioned nix-on-droid project is a fork of it) that combines a terminal emulator and a non-FHS-compliant Linux-like environment where Termux's own binaries and libraries are stored inside of that app's data dir, while reusing the Android's native Linux kernel and libraries whenever possible for the most part. Crucially, Termux is by it's nature single-user (with a randomly generated username for the purposes of software compatibility) and as I've already mentioned it isn't FHS-compliant. As such, the home dir as well as all other dirs belonging to Termux have an absolute path (/data/data/com.termux/files which contains the home/ home dir and some other dirs such as usr/) and this path does NOT change based on username/uid/etc. This makes compiling an official/global binary cache for a lot of aarch64-linux nixpkgs potentially completely viable for the purposes of running Nix on Android at native level of performance. I have a relatively powerful PC and I used to run Gentoo on it in the past. For testing purposes I might try to make this work for myself in near future as a test, and if it works out I might even turn this into a proper full on FOSS project by myself if things go smoothly.

7. The final relatively promising solution is specifying an alternate nix store dir in /var/tmp. The /var/tmp dir is usually accessible for reading and writing by unprivileged users, and unlike /tmp it's usually meant for longer term temporary storage: Its contents are basically never stored in RAM only, and again unlike /tmp they aren't often wiped upon reboot. Using something like /var/tmp/nix for Nix store would provide a universal absolute path that does not change depending on username/uid/etc while at the same time not requiring root permissions to create/manage it. This means if we compiled a new official binary cache for that path, you could finally install Nix on a non-nix system with zero need for any privileged permissions, with zero extra workarounds and a potential for an actually useful public/global binary cache for that purpose. But sadly it's still not 100% drawback-free at the end of the day: 1) If another user creates a /var/tmp/nix dir before you do on a given system, bye bye reusable binary cache. If you trust that user and they give you read and write perms, that nix store can be shared between multiple users, but if either of the users gets in any way compromised, we now have an amazing an attack vector to infect all other users using the same unprivileged nix store, which is far from ideal. There is a potential workaround to these issues on systemd distros: PrivateTmp setting for services. I have no clue if this is at all viable, I'm not too familiar with systemd in general, but from what I've found, on most systemd distros an unprivileged user can create an unprivileged user-specific daemon and enable PrivateTmp for it. This causes the system to expose a /var/tmp and /tmp subdirectory protected from other unprivileged users to the individual user's daemon, and from that daemon's perspective it's operating on the absolute /var/tmp and /tmp dirs rather than the actual subdirs. Lastly, at the end of the day /var/tmp still has tmp in the name. This means that from what I'm aware, distros can sometimes delete files stored there according to various different conditions. Workarounds could be potentially developed to avoid that problem, such as storing the nix store in the home dir then hard linking it to /var/tmp if they're in the same filesystem, creating a fallback environment for repairing the main one if it gets partially or fully deleted, running a user daemon that periodically "touches" all the files so that they aren't deleted due to being too old or whatever else helps with this issue. Again, I have no clue if this would be sufficient, but it potentially provides us with an absolute path that can be used to compile a nixpkgs cache against, shouldn't require any extra levels of elevated privilege to use and if we're lucky it won't collide with other users either possibly maybe

[ikcii]

I looked into method 7 a bit more, and there is a non-zero chance that a PrivateTmp approach might work. If you're on a systemd distro without root perms, you could potentially create a daemon out of a tmux session or similar with PrivateTmp turned on, then put it's socket in a dir accessible by you from outside the PrivateTmp sandbox, such as in your XDG_RUNTIME_DIR, and after user login enter that daemon's sandbox and use it as your main shell effectively. It should theoretically work with most software, since only /tmp and /var/tmp would be isolated. Apparently you can also specify some custom binds to an extent for that user daemon, which could fix potential issues with xorg-related sockets and such, and possibly some tmp-based lockfiles as well