crate2nix icon indicating copy to clipboard operation
crate2nix copied to clipboard

Prefetch mutliple crates in parallel and/or asynchronously

Open nagisa opened this issue 4 years ago • 1 comments

When running crate2nix generate currently the tool appears to be prefetching the dependencies sequentially and struggles to saturate the bandwidth. Parallelizing (or making it asynchronous scatter-gather) would speed up this tool significantly.

nagisa avatar Jan 21 '20 21:01 nagisa

Since our current efforts go suspended and there was significant discussion about this:

It is important for me, not to increase the already sizable number of dependencies of crate2nix for a feature that I don't think is essential for a lot of users:

  • We shouldn't need any prefetches for crates from crates-io since the checksum is already taken from the log file.
  • Furthermore, after #82 there shouldn't be merge conflicts from checked in crate-hashes.json files anymore. That should remove any reason to not check them in.

In effect, a typical user will only have to prefetch depdencies sparingly. If that assumption is incorrect, let me know!

I would still accept well-coded/tested PR to add parallel fetching if it does not add additional crate dependencies to crate2nix. The tools in std should be sufficient.

kolloch avatar Feb 12 '20 17:02 kolloch