astral-sh
Feature request: Control parallelism of downloads, sdist builds, and installs
Not significantly high priority but I would like to be able to control how many parallel downloads, sdist builds, and installs uv runs, three main reasons:
- I've had a couple of recent errors in WSL2 when running uv that the device is not ready yet (I will capture it if I see it again and make a separate issue) and it appeared to be from uv running too much against the filesystem at once
- I may wish uv to use less resources and don't mind if it takes longer to run, e.g. on a low power device with other things running and building the Python environment isn't latency sensitive
- I would like to be able to do more specific benchmarks compared to pip, and it's difficult to measure in a full install where uv is actually more efficient, or where it is doing things in parallel
There is an open PR (https://github.com/pypa/pip/pull/12388) on pip side that will expose the number of downloads at once via "--parallel_downloads"
I think we should support this. I'd prefer environment variables to command line flags though.
Does --parallel-downloads set the number of threads used to download in parallel, or the maximum number of inflight concurrent downloads? We run most downloads on the main thread so we could probably only expose the latter.
Does
--parallel-downloadsset the number of threads used to download in parallel, or the maximum number of inflight concurrent downloads? We run most downloads on the main thread so we could probably only expose the latter.
The pip PR adds the feature to use multiple threads, and the feature controls the number of threads.
But for my use case at least I was wanting to set one download at a time, I wasn't too concerned with whether things are technically parallel or concurrent.
https://github.com/astral-sh/uv/pull/3493 gives you a way to manage network and build concurrency limits. We might still want a way to limit installs, and limit the size of the blocking threadpool if you want to limit fs operations. Maybe those could be combined.
It doesn't completely fulfill your 3rd point of benchmarking against pip. However, serializing all uv operations would be quite difficult if not impossible and I don't think something worth doing. I think the best we can do is expose concurrency limits on specific operations.
That makes sense, however, installing time is easy to seperate out from the other numbers when benchmarking, and if needs be I can always test installing one package at a time with no dependencies.
I was more referring to other fs operations, like reading from the cache, that could still occur concurrently during downloads. However, during a cold download you should be fine. I'll look into adding an install concurrency limit in a separate PR.
Resolved by https://github.com/astral-sh/uv/pull/3493 and https://github.com/astral-sh/uv/pull/3646.