Add Concurrent Downloads of Multiple Packages

[robmathers]

I'd like to see pkg have the ability to use parallel downloads to better take advantage of fast connections. With a connection capable of sustaining 5+ MB/s, it's rarely used to its full capacity when upgrading packages, which are typically many small downloads, rather than large ones (especially if set to a quarterly upgrade cycle).

By offering an option for multiple concurrent downloads, bandwidth could be used fully and upgrades completed sooner.

I'd suggest a default value of 2 concurrent downloads, but my 150Mbit connection could easily sustain 5 or 10 or more.

[v1ne]

Please do! Debian added this years ago and it helps a lot with small packages. (tuning FreeBSD so that the TCP ramp-up is faster would help, too. ;) )

[grahamperrin]

Also:

• FreeBSD bug 119196 – fetch(1): parallel download from multiple servers by fetch
• #1961

[suchipi]

+1 on this; installing large metapackages like xorg can take a long time since downloads are all serial. Many web servers can send data faster with multiple connections open.

[Crest]

Does pkg (1.20+) try to take advantage of HTTP pipelining when fetching multiple packages to avoid having to establish and ramp up a connection per packet? How would using multiple concurrent downloads (by default without trickery like running pkg fetch from xargs) impact the official package mirrors (e.g. more concurrent connections)?

[michael-o]

Does pkg (1.20+) try to take advantage of HTTP pipelining when fetching multiple packages to avoid having to establish and ramp up a connection per packet? How would using multiple concurrent downloads (by default without trickery like running pkg fetch from xargs) impact the official package mirrors (e.g. more concurrent connections)?

HTTP pipelining is dead with http/2.

[Crest]

I "solved" the problem for my usecase with a Varnish caching server, because it's flexible enough to override caching hostile official package mirror HTTP headers and still validate each cache hit using an HTTP HEAD requests to compare via ETag. Finally pkg fetch saturates a 10Gb/s LAN connection and I don't have to feel guilty for hammering the CDN with lots of repeated downloads while testing my automations.