simple-optimism-node icon indicating copy to clipboard operation
simple-optimism-node copied to clipboard

perf(utils): utilize axel for faster, multi-threaded downloads (#92)

Open imtipi opened this issue 2 years ago • 4 comments

imtipi avatar Jul 17 '23 21:07 imtipi

currently, when I download snapshot from Cloudflare, I have an unstable connection and can't download it at full speed, and I use axel can mitigate the issue. download progress command changed to du -b because axel will generate a .st file for continue download the file

imtipi avatar Jul 17 '23 21:07 imtipi

axel looks good, but very verbosibe.

[ 24%]  .......... .......... .......... .......... ..........  [48311.5KB/s]
[ 24%]  .......... .......... .......... .......... ..........  [48312.6KB/s]
[ 24%]  .......... .......... .......... .......... ..........  [48313.8KB/s]
[ 24%]  .......... .......... .......... .......... ..........  [48313.2KB/s]

While only this should report about progress.

while true ; do sleep 60; echo...

mkostrikin avatar Jul 20 '23 07:07 mkostrikin

axel looks good, but very verbosibe.

sometimes I found the downloading file has incorrect file size, like download 10secs but the file size has already been 200G in the disk, I can't add -q to disable axel status output until I can figure out this issue

imtipi avatar Jul 20 '23 08:07 imtipi

Merged another fix that uses aria2c. Can you see if that works well? If it's good enough then I'll use that, otherwise will explore using a multi-threaded downloader.

smartcontracts avatar Nov 02 '23 23:11 smartcontracts

close due to already switch to snap sync and arid2c

imtipi avatar May 29 '24 20:05 imtipi