pip
pip copied to clipboard
pypa
Introduce resumable downloads with --resume-retries
Resolves #4796
Introduced the --resume-retries option in order to allow resuming incomplete downloads incase of dropped or timed out connections.
This option uses the values specified for --retries and --timeout for each resume attempt, since they are passed in the session.
Used 0 as the default in order to keep backwards compatibility.
This PR is based on https://github.com/pypa/pip/pull/11180
The downloader will make new requests and attempt to resume downloading using a
Rangeheader. If the initial response includes anETag(preferred) orDateheader, the downloader will ask the server to resume downloading only when it is safe (i.e., the file hasn't changed since the initial request) using anIf-Rangeheader.If the server responds with a
200(e.g. if the server doesn't support partial content or can't check if the file has changed), the downloader will restart the download (i.e. start from the very first byte); if the server responds with a206 Partial Content, the downloader will resume the download from the partially downloaded file.
I'm guessing the CI fails because of the new linter rules introduced in https://github.com/pypa/pip/commit/102d8187a1f5a4cd5de7a549fd8a9af34e89a54f
I'm guessing the CI fails because of the new linter rules introduced in 102d818
This is CI fix, failing until it's merged: https://github.com/pypa/pip/pull/12964
Hey @notatallshaw 👋
Is there anything that I can do to move this one forward?
Is there anything that I can do to move this one forward?
A pip maintainer needs to take up the task of reviewing it, as we're all volunteers it's a matter of finding time.
I think my main concern would be the behavior when interacting with index servers that behave badly, e.g. give the wrong content length (usually 0). Your description looks good to me, but I haven't had time to look over the code yet.
A pip maintainer needs to take up the task of reviewing it, as we're all volunteers it's a matter of finding time.
Yeah, I know how it goes, so no worries!
If you need any clarifications or would like me to make changes, I'd be happy to help!
I've had an initial cursory glace at this PR and it appears to be sufficiently high quality.
I've also run the functionality locally (select a large wheel to download and then disconnect my WiFi midway through the download) and it has a good UX.
My main concern, although this is a ship that has probably sailed, is it would be nice for pip not to have to directly handle HTTP intricacies and leave that to a separate library.
I can’t promise a full review or other maintainers will agree, but I am adding it to the 25.1 milestone for it to be tracked.
The PR looks good, although I’m not a http expert so I can’t comment on details like status and header handling. Like @notatallshaw I wish we could leave this sort of detail to a 3rd party library, but that would be a major refactoring. Add this PR (along with cert handling, parallel downloads, etc) to the list of reasons we should consider such a refactoring, but in the meantime I’m in favour of adding this.
There isn’t an “approve with conditions” button, but I approve this change on the basis that someone who understands http should check the header and status handling.
I'll tack this onto my to-do list. Not sure if I can call myself a HTTP expert, but I've done a fair bit of webdev as a hobby so I'm decently familiar with HTTP statuses and header handling.
Sorry for taking so long to review. Large PRs like these are appreciated since they do often implement major improvements, but they're also tedious to review and pretty daunting. Not really a good excuse, but that's how it feels. Thanks @notatallshaw for the initial pass and confirming this is worth the look.
Awesome! Thank you for all your efforts!
Don’t worry about it, I know how it feels! Let me know if you need anything ✌️
Hopefully this gets added soon, downloading GBs of stuff over slow internet and then having to restart from the beginning is not an experience i would recommend
@Ibrahima-prog I hear ya! This is on my radar to review. I haven't gotten around to it yet. And truthfully, I probably won't find the time until at least next Thursday. This will make it into the pip 25.1 release.
Does this do rsync-style checksums? That would increase reliability.
@thk686 I'm very late to the party, but could you elaborate on how checksums come into play? AFAIK, indices don't serve the checksums of their distributions so there is no way pip could double check the download wasn't corrupted unless the checksums were given by the user. This PR uses conditional range requests (via the If-Range HTTP request header) which will avoid the issue of the file being changed on the server in-between requests.
I was thinking of over-the-wire corruption. In some cases it can be beneficial to checksum on both ends.
On Wed, Mar 12, 2025 at 6:05 PM Richard Si @.***> wrote:
Does this do rsync-style checksums https://pypi.org/project/pyrsync/? That would increase reliability.
@thk686 https://github.com/thk686 I'm very late to the party, but could you elaborate on how checksums come into play? AFAIK, indices don't serve the checksums of their distributions so there is no way pip could double check the download wasn't corrupted unless the checksums were given by the user. This PR uses conditional range requests (via the If-Range HTTP request header) which will avoid the issue of the file being changed on the server in-between requests.
— Reply to this email directly, view it on GitHub https://github.com/pypa/pip/pull/12991#issuecomment-2719315225, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEQXSKWTOCZKZHHELLCWID2UC4SHAVCNFSM6AAAAABPMRUVG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZGMYTKMRSGU . You are receiving this because you were mentioned.Message ID: @.***> [image: ichard26]ichard26 left a comment (pypa/pip#12991) https://github.com/pypa/pip/pull/12991#issuecomment-2719315225
Does this do rsync-style checksums https://pypi.org/project/pyrsync/? That would increase reliability.
@thk686 https://github.com/thk686 I'm very late to the party, but could you elaborate on how checksums come into play? AFAIK, indices don't serve the checksums of their distributions so there is no way pip could double check the download wasn't corrupted unless the checksums were given by the user. This PR uses conditional range requests (via the If-Range HTTP request header) which will avoid the issue of the file being changed on the server in-between requests.
— Reply to this email directly, view it on GitHub https://github.com/pypa/pip/pull/12991#issuecomment-2719315225, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEQXSKWTOCZKZHHELLCWID2UC4SHAVCNFSM6AAAAABPMRUVG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZGMYTKMRSGU . You are receiving this because you were mentioned.Message ID: @.***>
-- Timothy H. Keitt www keittlab org
@ichard26
I'd prefer if pip's behaviour emulated that of a browser where it caches the incomplete download so the download can be resumed at some later point, but I realize that would be significantly increase the complexity of the implementation (and it'd also introduce some friction as the download would have to be manually restarted).
There has been some discussion around this in the past^1 and I’d pretty much prefer it. However, I think that it’s out of scope for this first step in implementing resumable downloads, considering the amount of work needed.
Would be possible to default to allowing a few (1-3) resume attempts? That way, if the download fails halfway through, the download will be given another shot. It may not be enough if the connection is so unstable that it requires a ton of resumes, but for one-off failures, it would still be a major improvement. As long as the messaging is clear, I don't think automatic resumes would be that annoying to the user.1 I consider resumes as the preferred option and opting out of resumption to be an exceptional (but still important to support!) case.
I initially set the default --resume-retries to 0 for backward compatibility and to get the discussion going. I agree that a low default (e.g., 2–5 attempts) would provide a better UX, but I'd also be cautious about changing pip’s default install behavior.
We have two options:
- Set a low default right away, as you suggested.
- Release it as-is, monitor and fix any issues that arise, and consider making it the default in a future version.
Just so everyone is on the same page, I plan on re-reviewing this PR sometime this week. I'm working on prototyping some code style changes which I'll share soon. Beyond that, I'd like to review the other parts of the resuming UX. After that, I should be happy enough with this to merge it and let any other suggestions be handled at a later date.
Oh right, here are some screenshots of the new messaging:
(It says "Resuming download of" here. I'd made this adjustment earlier, but I reverted the change in my last commit)
The urllib3 request retrying warnings are very noisy here, but this is probably the less common scenario for retrying. I'm disconnecting the Internet entirely to trigger the retry logic. In practice, I'd imagine the download would fail, but the request/connection can be easily restored (so urllib3 won't complain). Either way, this is a pre-existing issue and should be dealt with separately.
@ichard26 WHat's the status on this? It seems close to ready, but there are still some design questions being asked. Is release 25.1 (in a couple of weeks) still a realistic target here, or should we defer it? This is a big enough feature that I'd like to see it go in, but conversely, I really don't want it as a last-minute merge a few days before the release.
What's the status on this? It seems close to ready, but there are still some design questions being asked.
@pfmoore we've decided that this feature should be opt-in upon release, only to be enabled by default in pip 25.2 or later once we've gotten some feedback. Thus, I'm not too worried about the exact implementation details. Those can change in a future release if needed. I'm also pretty happy with the code as-is.
The sticking point I have is that I'm still not sure of the UI of resumable downloads. --resume-retries is a weird flag.[^1] As someone who understands the implementation, it makes sense, but it's likely to be rather obtuse for users. How many resumes should I allow? How does it work differently than --retries? Part of me wants to suggest that we reuse the --retries flag to enable resumable downloads[^2] to keep the UI simpler. OTOH, all of the networking flags (except for --proxy and --timeout) are already "advanced" features so maybe it's fine to expose these fine knobs to the users. If that's the case, I'm also not really happy with the current name. See https://github.com/pypa/pip/pull/12991#pullrequestreview-2727795819 for more.
While the rest of the feature can be reworked in future releases, it's likely not feasible to rename a flag once released.
Any thoughts @pfmoore @notatallshaw? I'm rather torn and can't make up my mind.
[^1]: Does any other tool that accesses the network have a similar flag?
[^2]: Although once automatic resuming is the default, reusing the --retries default of 5 does seem a bit high... Also to handle the opt-in phase, we'd need to switch to using --use-feature=resume-downloads (and set an reasonable internal limit [10?] since it wouldn't be configurable).
My current feeling is that it's better to stick with a simpler UI for pip 25.1. If we get complaints, then we can consider giving the users more control later (à la --resume-retries or whatever name we end up choosing) before it's enabled by default. It's harder to remove/restrict the UI after the fact (as packaging standardization has shown time after time).
Proposal:
- The
--resume-retriesflag is removed - To opt-in into resumable/restartable downloads, one must pass
--use-feature=retry-downloads[^1]. This has the benefit that--use-featureis explicitly meant for experimental features - The resume/restart limit is hard-coded to some value (10? - given it's opt-in, it's fine to err on the higher end)
- Once it's made the default, the current proposal would be to link it to the
--retriesflag, unless feedback received after the pip 25.1 release indicates a separate flag is necessary.
[^1]: I'm using the word "retry" over "resume" as restarting will be done if range requests aren't supported.
Any thoughts @pfmoore @notatallshaw?
I'm going to keep out of the design discussions. I have a bunch of other things on my plate, and not much spare time, so I don't want to add anything else.
I will say, though, that I don't like the idea of changing the UI once it becomes default. That's not (in my mind) what the --use-feature flag is for - it should be an opt-in to using a new feature that's complete, and won't change, to give people a chance to try it out before it's made the default[^1].
I'm going to say that I'd rather not have it in 25.1 unless it's complete (so assuming it's gated behind --use-feature, the only change needed to make it the default is to remove the need for --use-feature).
[^1]: I'll note that as described, this is a weird feature to use --use-feature on. Because the default is no retries, --use-feature=retry-downloads on its own is a no-op. So why have --use-feature at all? But as I said, I don't want to get sucked into design discussions, so I'm not really looking for an explanation here, just pointing out the oddity.
@ichard26 I'm not a fan of this proposal for a couple of reasons:
Firstly, I really dislike changing how features are enabled between pip versions, it makes guides outdated quickly and it more difficult to write scripts against pip.
Secondly, I mildly dislike overloading flag with multiple meanings, particularly to simplify reading the help at the cost of user control. What if user is working against an index they need to disable resumable retries but enable regular retries , will that be possible under the new scheme?
In terms of having nuanced options, I think most users should be served well by good defaults, and users who really need something other than the default will learn the names and meanings of those additional options.
Thank you all for your input and efforts!
There have been various discussions about the naming convention in the past (#11180, #4796).
In terms of having nuanced options, I think most users should be served well by good defaults, and users who really need something other than the default will learn the names and meanings of those additional options.
I fully agree, and I suggest keeping the existing implementation of --resume-retries.
Merging the behavior with --retries requires a deeper technical and UX discussion, so keeping them separate for now ensures clarity without limiting future improvements.
@gmargaritis I didn't have time to follow up on this like I wanted today, so I'll just say that I have no more blocking concerns. I've dropped all design questions having thought about it more. I do think this could benefit from some documentation, but I can handle that later. Right now, what I would appreciate is that you take a look at the three commits I pushed and confirm that I didn't do anything you object to. You can see my review for an explanation of the changes.
Once I have time—ideally tomorrow—I'll follow up properly and clear this for merge. Thanks for your tenacity on this PR!
@ichard26 I've reviewed the code once more and it looks great! Thank you for implementing these improvements.
If I can suggest one thing, it would be to update the help text for the--retries and--resume-retries options to eliminate any potential confusion.
Specifically, I recommend changing the help text for --retries from:
--retries <retries> Maximum number of retries each connection should attempt (default 5 times).
to:
--retries <retries> Maximum attempts to establish a new connection (default: 5).
This change clarifies that --retries applies to new connection attempts, instead of retries within an already established connection.
Similarly, I suggest updating the text for --resume-retries to:
--resume-retries <resume_retries> Maximum attempts to resume an incomplete download (default: 0).
I believe this will help our users better understand the purpose of these options and how they differ. Overall, I feel confident that this PR is in great shape and will be a valuable improvement.
Let me know if you need anything, I'd be happy to help!
Aargh, of course it's Black—the very tool I used to maintain—that's causing CI to be red :sweat_smile:
I just pushed a commit to fix the formatting. Now, this should be good to land.
Awesome to hear, @ichard26! Thank you all for your efforts, this is going to be a real improvement for everyone!
@notatallshaw your comment here suggested that you still have reservations about this. Is that still the case? If so, how do we resolve this? We have a week, maybe 2 at the absolute most, to get this ready to merge if it's to go into 25.1.
@ichard26 unless @notatallshaw still objects, can you do the honours and merge this? I think that if anyone else had any reservations, they would have raised them by now.