Snapshot download performance

[LesnyRumcajs]

Issue summary

Downloading a calibnet snapshot via forest-tool snapshot fetch --calibnet takes 1m30s on a DO VPS (s-4vcpu-16gb-amd) wget https://forest-archive.chainsafe.dev/latest/calibnet/ takes 29s and aria2c -x5 … takes 19s. These numbers are from v0.15.2, on v0.16.0 I got even worse numbers.

This is causing timeouts in the sync check (the timeout is set to two hours, downloading a snapshot takes more than that now).

It was not the case in the past, so we must check what is the culprit for this regression and fix it.

Other information and links

From sync-check failure today:

2023-11-14T13:20:10.091474Z  INFO retry: forest::progress: Loading 65490591744 (elapsed time: 2h 3m 32s)
2023-11-14T13:20:15.091484Z  INFO retry: forest::progress: Loading 65532768256 (elapsed time: 2h 3m 37s)
2023-11-14T13:20:20.283048Z  INFO retry: forest::progress: Loading 65577553920 (elapsed time: 2h 3m 43s)
2023-11-14T13:20:21.594140Z  INFO forest_filecoin::daemon::db_util: Imported snapshot in: 7425s, heaviest tipset epoch: 3387023

[LesnyRumcajs]

I checked on some old versions (I believe downto 0.13.0) and it was slow there as well. I'm leaning towards resurrecting aria2c, at least optionally. Streaming zealots may get angry, though.