Find ways to reduce RAM usage of nix-index

[suhr]

summary

To compute the mapping from the nixpkgs attribute name (ex: nixpkgs.hello) to the store path (ex: /nix/store/13208adkjlqwe...-hello), nix-index needs to evaluate nixpkgs. The current implementation uses nix-env -qaP --out-path --xml, which has high RAM usage since it needs to evaluate all derivations in nixpkgs.

Some possible ways to improve this within nix-index itself:

• offer an option to turn off the store path -> attribute mapping (resulting in a less helpful index, but one that can be built more easily)
• try to gather the mapping from the hydra job names (need to find a way to do this without causing too much load on the Hydra API)

---

original issue

It runs an instance of nix-env which eats all the RAM available in my system.

[bennofs]

This issue cannot easily be fixed in nix-index itself. The evaluation of the Hydra jobset is known to consume a lot of RAM.

[anka-213]

How much RAM does it use currently?

Also, how much time does it usually take and what size can I expect the final index to have? It might be nice to add some estimates of this to the readme.

[tilpner]

@anka-213

00:00:01 + querying available packages
...
00:21:40 + wrote index of 28,717,754 bytes
nixpkgs-index.service: Succeeded.
nixpkgs-index.service: Consumed 3min 43.981s CPU time, received 212.2M IP traffic, sent 32.3M IP traffic.

[milahu]

How much RAM does it use currently?

at least 1.3 GB

in ksysguard it looked more like 3 GB with 8 GB of RAM and no swap, i ran into out-of-memory (other apps took around 3 GB)

mprof (simple)

nix-shell -p python38Packages.memory_profiler nix-index \
--run 'time mprof run nix-index'

nix-shell -p gnuplot --run gnuplot <<< '
set title "mprof nix-index"; set xlabel "Minutes"; set ylabel "MB RAM"; 
set term png; set output "mprof.png"; 
set key autotitle columnhead; unset key; # ignore first line
first=0; offset=0; shift(x)=(offset=(first==0)?x:offset,first=1,x-offset); # relative time
plot "mprofile_20210420003255.dat" using (shift($3/60)):2 w lp;'

valgrind massif (fancy)

recompile nix-index/default.nix with

buildType = "debug";

and run

valgrind --tool=massif nix-index
ls massif.out.*
massif-visualizer massif.out.*

that 800 MB baseline looks wasteful

tempfiles

on the first run, nix-index stores 800 MB of tempfiles in /run/user/1000/file_listing.json* until /run/user/1000 is 100% full nix-index runs for 20 minutes, with lots of

Error: fetching the file listing for store path '/nix/store/w2qimbfmzl4p9xw3ahx7vqm5kz8imca4-flat-remix-icon-theme-20200116' failed
Caused by: response to GET 'http://cache.nixos.org/w2qimbfmzl4p9xw3ahx7vqm5kz8imca4.ls' failed to parse
Caused by: trailing characters at line 1 column 10146950

probably caused by the no space left on tmpfs

on the second run, nix-index stores 3.5 GB of tempfiles in /tmp/file_listing.json*

all these tempfiles are not deleted on finish

[turion]

Any ideas to improve this?

[bennofs]

The tempfiles are definitely a bug. They should only be kept if there is an error (to aid debugging), but perhaps there should be a limit on the number of errors to store (maybe even as a flag, which could default to 0)

[tobiasBora]

I also hit this issue on raspberry pi: I can't even compile the system (quite basic, just xfce and a few basic packages) as it runs out of memory. I use flake via nixos-rebuild switch --flake . to install the system, any idea if a fix for this issue would also help flake? Or should I open another issue?

[milahu]

on raspberry pi: I can't even compile the system ... as it runs out of memory

cross compile

[tobiasBora]

Yeah, well for now I can create a 2G swapfile and it seems to be good enough… (1G works sometimes… but not always.) The problem with cross-compilation is that it won't be cached. Also, people also recommended setting the raspberry pi as a remote builder (compilation does not take RAM apparently), or using binfmt to fake an Aarch64 system on a x86_64 system… but I don't really like this idea as it means that I need another computer, that it's harder to work in team (binfmt must be enabled on all systems… and the raspberry pi may not be accessible because of firewalls).

In any case I guess it's cool to solve this problem, burning GB of RAM just to evaluate nixpkgs is not great, and limits its applications on low-end devices/old devices (yeah I have some old computers at home that have very little RAM but that work great for simple servers). Also, I remember having issues with nix being really slow on another computer (like maybe one minute just to do a no-op nixos-rebuild switch) without any SSD: I think that the reason for that is that it tries to evaluate the whole nixpkgs repository and therefore needs to access the disk a lot… It would be much better to only evaluate what's needed.

[milahu]

were getting offtopic, this is nix-index, not nix

burning GB of RAM just to evaluate nixpkgs

please verify. i guess that building needs more ram

nix-index needs much ram, because it fetches a million files from cache.nixos.org

some more ideas: disable parallel building disable parallel fetching? log memory usage of all processes during nix-build

one problem for low-memory devices is that xz -d needs about 5x more memory than gzip -d and cache.nixos.org does not serve nar.gz files

curl https://cache.nixos.org/$(readlink -f $(which gimp) | cut -c12-43).narinfo
StorePath: /nix/store/sqvlmp2dkrvyisi11dq6w6wdlbhyikfj-gimp-2.10.32
URL: nar/1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.xz
Compression: xz
FileHash: sha256:1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j
FileSize: 19313388
NarHash: sha256:14gpflz0d8ng4zpyx9sm3g4z2p7a4zscdmrp3dph5l6ijkdyvmiv
NarSize: 114638192

curl -I https://cache.nixos.org/nar/1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.xz
HTTP/2 200

curl -I https://cache.nixos.org/nar/1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.gz
HTTP/2 404

cd $(mktemp -d)

nix-shell -p time

wget https://cache.nixos.org/nar/1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.xz

$(which time) -v xz -d -k *.xz 2>&1 | grep "Maximum resident set size"
        Maximum resident set size (kbytes): 10592

gzip -k *.nar

rm *.nar

$(which time) -v gzip -d -k *.gz 2>&1 | grep "Maximum resident set size"
        Maximum resident set size (kbytes): 1836

du -sh *
110M    1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar
32M     1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.gz
19M     1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.xz

[socherbyc]

Just as a note: I have 16GB RAM and I'm running out of memory too :/