Flake copying performance regressed on macOS in 2.19

[cigrainger]

Describe the bug

I'm on nix-darwin and copying to the store can take several minutes. This first appears on 2.19. 2.18 is fast (a few seconds). E.g. nix run nixpkgs#scc takes >5 minutes.

Steps To Reproduce

Set nix.package to any version > 2.18 and try nix run nixpkgs#scc or nix flake update.

Expected behavior

The speed should be consistent after 2.18.

nix-env --version output

nix-env (Nix) 2.19.6

Additional context

I'm on nix-darwin.

Priorities

Add :+1: to issues you find important.

[emilazy]

It isn‘t 5 minutes bad for me, but I definitely also noticed a regression in flake copying performance on my Intel MacBook Pro after switching from 2.18 to 2.23 to pick up the Darwin sandbox fix. That‘s already one of the most painful parts of flakes, especially when working on Nixpkgs, so it‘s very unfortunate.

[Montmorency]

Also noticing this regression in flake performance and /nix/store interactions. [Current config is here] (https://github.com/Montmorency/nix-darwin).

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-07-17-nix-team-meeting-minutes-162/49255/1

[Mic92]

I cannot reproduce this:

$ nix --version   
nix (Nix) 2.23.3
$ time nix eval --store $HOME/test-store nixpkgs#nix.version
nix eval --store $HOME/test-store nixpkgs#nix.version  1.86s user 7.07s system 50% cpu 17.615 total 

[nix-shell:~/git/nixpkgs]$ sudo rm -rf $HOME/test-store/   
[nix-shell:~/git/nixpkgs]$ nix --version
nix (Nix) 2.18.5                       
[nix-shell:~/git/nixpkgs]$ time nix eval --store $HOME/test-store nixpkgs#nix.version        
"2.18.5"
real    0m22.181s
user    0m2.058s
sys     0m17.071s

In this case the old version was even slower, but my system is also more under load running some nix-builds at the same time.

[roberth]

We might be holding libgit2 wrong. I'm looking into packfile generation to reduce the vast amount of small I/O.

[worldofgeese]

Just another anecdote: with Nix 2.33.3 I'm seeing hours-long copy operations. I'd like to provide more substantive logs but the most I get is copying '/Users/$USER/.local/share/flake-config/' to the store.

[roberth]

2.33.3

Either geese can travel through time, or this has a typo. Which version did you run? Did it complete, or did you interrupt Nix?

Possibly related: https://github.com/NixOS/nix/issues/10630#issuecomment-2289997497

If you could get stack traces with a debugger, that may help narrow down the cause.

[thomasjm]

Hi, I was wondering what the status of this is. I saw that #11330 landed in Sept 2024, so it's presumably in Nix after 2.25 or so? I can't seem to find a mention in the release notes though.

I'm using Nix 2.31.3 and still finding long delays (like several minutes) while I see messages like this:

copying '/nix/store/dzwwydggypqg4p8r6if5zdaqyzhsxjkd-source' to the store

This particular path is a Nixpkgs checkout that's one of my flake inputs.

I think #14041 improves the situation in 2.32 but it's a little hard to tell.