Nix feature doesn't cache /nix

[dsyer]

Environment setup after container restarts can take a really long time if all of nixos.org has to be downloaded every time. I guess you might want the option to clear the cache or switch it off, but IMO it would be better if /nix was a volume mount by default.

[Chuxel]

@dsyer Just to clarify, do you mean rebuild? Container start shouldn't download nix each time.

[dsyer]

I'm not sure I understand the difference. What I do often is SHIFT-CTRL-P "Remote Containers: Rebuild container". Or shutdown a workspace and then open it up again the next day.

[Chuxel]

@dsyer If you just open the container without rebuilding, everything will still be there. There are two cases where it won't:

1. "Rebuild container"
2. docker system prune -a while the dev container is shut down. That would delete it like any other container.

The goal is that you can always "reset" to zero with a rebuild. So, in those two cases, you've got a fresh environment. A volume can be used to save off /nix as you mention, but other state in the container would be gone.

Note that docker system prune would also blow away any local Docker cache which would slow down the rebuild process.

[domenkozar]

Would it be useful if the image came with pkgs.cachix installed so that pushing/pulling Nix packages would be trivial?

[dsyer]

I'm not sure how that would help, but I'm ignorant, so please help me understand. What is being cached in pkgs.cachix? Either it's all of nixos.org and the image would be unmanageable, or it gets refreshed when you install packages (just like /nix) and would be lost when the container is rebuilt. My understanding is it's useful for managing a fleet of nixes all sharing one cache. If that's right then it doesn't seem like there would be much benefit in having it in all the containers?

[domenkozar]

If you use something like cachix watch-store mycache it will upload all newly built entries to https://cachix.org, and then running nix build as part of updateContentCommand, it will make sure your cache entries pre-warm the container on startup.

[shikanime]

Over the past year, I have been utilizing volumes for caching on my system with minimal issues, except for nix-collect-garbage that obviously will wipe more data than necessary. A notable distinction between having a mounted /nix volume (or should I just mount /nix/store, never really gave a deep thought) and utilizing cachix lies in caching locality.

In my container, I use vscode dotfiles with home-manager along with numerous utilities, each consuming a few gigabytes. The need to redownload everything for every container has resulted in a remarkably large inconvenience. While having to add a mounts in devcontainer.json is fine, it could be better.

@Chuxel, I have been considering initiating a discussion about the possibility of mounting well-known cache directories, such as /home/${USER}/.cache/pip. This differs from the runtime libraries directory, such as /usr/local/lib/python3.11/site-packages for Python, which in contrast could cause potential troubles. This issue has become particularly problematic for data scientists dealing with multi-gigabyte dependencies such as PyTorch at around 3GB.