erigon icon indicating copy to clipboard operation
erigon copied to clipboard

[wip][discuss] downloader: increase default net chunk size

Open AskAlexSharov opened this issue 1 year ago • 1 comments

// DefaultNetworkChunkSize - how much data request per 1 network call to peer.
// `256 * 1024` is home-routers-friendly - otherwise wifi is not usable by other devices (but I know it only for p2p traffic, not for http) 
// `2 * 1024 * 1024` is webseed-friendly (it's implementation in lib is not feature-complete yet, but doable, and increase of net chunk improving speed)
// default: 16Kb

my experiments with file which exists only on webseed - shows:

  • it creasing net chunk size helps with speed (2x increasing of value -> 2x increasing of speed. speed stop grow when DefaultNetworkChunkSize > DefaultPieceSize, I see speed 8mb/s)
  • amount of webseeds (buckets with same file) also increasing downloading speed (8mb/s from each bucket)

alternatively:

  • can leave defaults and make it parametrizable
  • can increase default --torrent.download.slots to download more files in parallel

@awskii @mh0lt what do you think?

AskAlexSharov avatar Mar 26 '24 06:03 AskAlexSharov

I think for such questions we should decide first the other thing: what is our expected default environment?

In my opinion, most of our users will start erigon on cloud/remote hosts with good bandwidth and enough disk/ram space, so we could use defaults as we run on server. Most effect will be noticed on larger nets which currently stalls downloader. Larger nets are always on clouds (why on earth you want to have 4tb snaps locally).

On the other hand, casual users never read docs and never want to customize defaults. We could have "command to start on home machine" in docs with all needed flags for home stations (and mark help messages for those parameters which are "better to change if start at home" to ease further updates of docs e.g. with --help | grep "homeUse".

to sum up: lets use defaults better for server hosting and webseeds (do less requests)

awskii avatar Apr 03 '24 14:04 awskii