grab
grab copied to clipboard
cavaliergopher
Support multiple connections / multipart download
grab works very well and I see that you are already making use of Accept-Ranges response to support resuming of downloads, but do you have any supports or plans to support multipart downloading? (i.e. allowing us to download a file in chunks using several connections).
Scenario: I have a large file to download (several GB) and I find that when I use other go packages that support multipart I am able to download the file in a fraction of the time when compared to grab's single connection.
I am also looking to use a library for downloading files and multiple connection seems to be the only major piece that is missing for my needs
I am also looking to use a library for downloading files and multiple connection seems to be the only major piece that is missing for my needs
As there was not a reply here, I went looking and adopted another project “GetParty” which has been fantastic:
https://github.com/vbauerster/getparty
It supports multi-part chunked downloading and is amazing at retrying the parts if they fail. Absolutely perfect for my needs and any issues I raise are resolved within 24 hours as the maintainer is very active.
I think that if Grab is still being maintained (not sure) it would be great for them to adopt the same approach and/or work with GetParty to create the best overall combination of the two projects. One go download library to rule them all!
I'm using Grab pretty deeply integrated into my current solution, with the callback hooks, and now I also want to support Range chunk downloading. I was looking at the current implementation to see if there is a decent hook in to apply it in my own code. The only spot I see is a custom HTTPClient interface to intercept and handle the http Request. But it doesn't seem like it plays well with the rest of the design of the state machine. GetParty, mentioned in the earlier comment, doesn't appear to be a library; only a CLI. I've had a play with this option, which is both a CLI and library: https://github.com/pgollangi/fastget I do need this support, so I may end up trying to implement the fastget logic into Grab and see where I get. If it works out, I can submit a PR.
I've got a PR up with support for parallel "Range" requests, if anyone wants to have a look? https://github.com/cavaliergopher/grab/pull/102