rules_nixpkgs icon indicating copy to clipboard operation
rules_nixpkgs copied to clipboard

Add copy_many function to address performance issue

Open benradf opened this issue 3 years ago • 1 comments

This is an attempt to improve the MacOS performance issue described in #256 by adding a cp_many function to util.bzl. Instead of writing each file with repository_ctx.file and invoking chmod to fix the permissions, cp_many uses tar from an external shell process.

My hope is this will be more efficient than spawning a new chmod process for every file. However I do not have a Mac to test on. On Linux I attempted to reproduce the slowness by including 4096 random 16K files in nix_file_deps. For the original code that took 7m59s to run, and with this change it took 7m27s. So not much of an improvement, but maybe the speed up is greater on MacOS?

benradf avatar Aug 11 '22 11:08 benradf

Thanks for the fix @benradf ! Just tested on my Mac and it seems to have around the same amount of improvement as my stat solution (also improves the build from 5 mins to 1 min).

lamcw avatar Aug 12 '22 01:08 lamcw

Thanks for checking this on Mac @lamcw. It's good to hear the performance improvement there is greater than on Linux.

Though since it's about the same as your stat solution I think I'll park this PR for now. Seems to better to avoid shelling out to tar if it doesn't provide a significant improvement.

benradf avatar Aug 12 '22 11:08 benradf

Going with #259 as a solution instead.

benradf avatar Aug 12 '22 12:08 benradf