wget2 icon indicating copy to clipboard operation
wget2 copied to clipboard

Download a single file from different sources?

Open WolfgangDpunkt opened this issue 3 years ago • 3 comments

Maybe I missed it until now, but has wget2 the ability to load the same file from multiple mirror servers?

For example, a large iso file exists on multiple servers/mirrors and wget2 downloads this file in parallel to increase speed.

Ohne Titel

Thanks for a hint or the possibly to include this idea as a feature request.

Multithreaded download of single files (option --chunk-size)

As far as I understand a file would be loaded with "--chunk-size" only from one server, with several connections, but it would be even faster if it would be possible from several servers.

Thank you and happy new year!

WolfgangDpunkt avatar Jan 04 '22 14:01 WolfgangDpunkt

What you want is Metalink. It allows you to specify several mirrors as well as checksums for every part/chunk. Wget2 supports it.

I currently don't know how user-friendly the tooling is. But maybe you give it a try.

Normally, the server provides a metalink file, wget2 recognizes it, parses it and downloads the resources inside (single file or several files).

You can build your own metalink files and feed them into wget2 via --input-file file.xml --force-metalink. Though I am currently not sure how wget2 reacts when there is no checksum (which you likely don't know in advance).

rockdaboot avatar Jan 05 '22 14:01 rockdaboot

Thank you for the explanation.

I had seen Metalink as an option, but I had understood it that way that wget2 just uses it to select the fastest one from a list of mirror servers and then does the download from that one only. If using the Metalink feature allows actually downloading chunks 1-5 from server1 and chunks 6-10 from server2 to be loaded simultaneously and then merged, that would be exactly what I have in mind. I will try it. :-)

WolfgangDpunkt avatar Jan 05 '22 14:01 WolfgangDpunkt

Don't be surprised if that isn't implemented yet (I currently can't test nor check the code). But it shouldn't be too hard to implement it. Possibly you run into issues as this seems to be a rare use case

rockdaboot avatar Jan 05 '22 19:01 rockdaboot