crate2nix
crate2nix copied to clipboard
Prefetch mutliple crates in parallel and/or asynchronously
When running crate2nix generate
currently the tool appears to be prefetching the dependencies sequentially and struggles to saturate the bandwidth. Parallelizing (or making it asynchronous scatter-gather) would speed up this tool significantly.
Since our current efforts go suspended and there was significant discussion about this:
It is important for me, not to increase the already sizable number of dependencies of crate2nix
for a feature that I don't think is essential for a lot of users:
- We shouldn't need any prefetches for crates from crates-io since the checksum is already taken from the log file.
- Furthermore, after #82 there shouldn't be merge conflicts from checked in
crate-hashes.json
files anymore. That should remove any reason to not check them in.
In effect, a typical user will only have to prefetch depdencies sparingly. If that assumption is incorrect, let me know!
I would still accept well-coded/tested PR to add parallel fetching if it does not add additional crate dependencies to crate2nix
. The tools in std
should be sufficient.