Jonas Haag
Jonas Haag
> Even the uncompressed is much faster: I'm not sure you can compare this. Does curl report the speed relative to the compressed or uncompressed file size?
> why don't you use .bz2? I guess we just rely on whatever compression curl negotiates with the server.
For me it's roughly the same _duration_ of download with curl and Mamba: ``` conda-forge/noarch 9.9MB @ 3.7MB/s 3.0s ``` ``` time curl https://conda.anaconda.org/conda-forge/noarch/repodata.json >/dev/null % Total % Received %...
For `osx-64` curl reports the same download speed and it takes 6 s to download (via curl). With `bz2` it takes a little over 1 s via curl.
I guess it would be good to change to the `bz2` variant if it's available for a channel. cc @wolfv tl;dr: `.json.bz2` download seems to be 5x faster than HTTP...
> How do I find download_threads? Couldn't find anything in docs and info etc. Yeah unfortunately not everything is documented yet. You can put it into your `.condarc`: ``` local_repodata_ttl:...
> it shows you download speed of compressed yet downloads at speed of uncompressed Yeah that seems to be the key point here. Mamba and curl report different numbers. But...
Yeah we might want to consider increasing that number. It isn't actually the number of _threads_ (xref https://github.com/mamba-org/mamba/pull/1963) and the curl default is 50 if you use `curl --parallel`.
``` $ hyperfine -M 5 'curl https://conda.anaconda.org/conda-forge/noarch/repodata.json.bz2 > /dev/null' Benchmark 1: curl https://conda.anaconda.org/conda-forge/noarch/repodata.json.bz2 > /dev/null Time (mean ± σ): 742.2 ms ± 226.6 ms [User: 108.3 ms, System: 26.6 ms]...
The `bz2` file is 10% smaller than the gzip compressed download, so that doesn't explain the 5x slowdown. Also bandwidth doesn't seem to be limited: ``` Download of uncompressed json...