Improve experience of `download-spec-tests`

[ensi321]

Problem description

As the size of spec test files increases in every release (from ~670MB for v1.4.0 to ~780MB for v1.5.0-alpha.3), it takes 20+ minutes for me to download all files.

Currently downloading spec test provides minimal log output on the download progress. Only at the beginning of download, and when the download is finished, which could be 10-20 minutes apart. It would be great to provide more info (eg. a progress bar, download time remaining) during download such that users/devs know what to expect. Here is a sample log of the current behaviour:

NC@Chainsafe beacon-node % yarn download-spec-tests
yarn run v1.22.22
$ node --loader=ts-node/esm test/spec/downloadTests.ts
outputDir = /Users/NC/Documents/lodestar/packages/beacon-node/spec-tests
Downloading new version v1.4.0-beta.6
Cleaning existing version v1.5.0-alpha.3 at /Users/NC/Documents/lodestar/packages/beacon-node/spec-tests
outputDir = /Users/NC/Documents/lodestar/packages/beacon-node/spec-tests-bls
version v0.1.1 already downloaded
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/general.tar.gz - 15889101 bytes
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/mainnet.tar.gz - 421709376 bytes
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz - 239776755 bytes
Downloaded  https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/general.tar.gz
Download attempt 2 for https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/mainnet.tar.gz failed: aborted
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/mainnet.tar.gz - 421709376 bytes
Downloaded  https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/mainnet.tar.gz
Download attempt 2 for https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz failed: aborted
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz - 239776755 bytes
Download attempt 3 for https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz failed: aborted
Downloading https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz - 239776755 bytes
Downloaded https://github.com/ethereum/consensus-spec-tests/releases/download/v1.4.0-beta.6/minimal.tar.gz - 239776755 bytes

Moreover, partially downloaded files are deleted when reattempting to download spec test. This is an issue when having slow and unstable internet, a user may already have downloaded general and minimal spec tests, but internet drops when downloading mainnet. When he runs yarn download-spec-tests again, general and minimal are deleted and everything is being re-downloaded.

Solution description

• Provide more helpful log output on the progress of spec test download
• Only download missing spec tests

Additional context

No response

[wemeetagain]

definitely agree, I've run into the same problem

[mihirrchauhan]

Hey! I'm planning to work on this issue and would like to be assigned to it. I propose using the cli-progress package to display download progress. Do you think this is the right approach? Please assign me this issue so I can get started!

[ensi321]

Hey! I'm planning to work on this issue and would like to be assigned to it. I propose using the cli-progress package to display download progress. Do you think this is the right approach? Please assign me this issue so I can get started!

Yea I think that's a good idea! Thanks for picking this up

[hara-desu]

@ensi321 Hello! As I understand this issue isn't closed yet because the solution in the PR of @mihirrchauhan prints a new progress bar every time there's a progress update. Can I take this issue and try to work on it? I'm doing it as a part of Women in Ethereum Protocol studies.

[nflaig]

@ensi321 Hello! As I understand this issue isn't closed yet because the solution in the PR of @mihirrchauhan prints a new progress bar every time there's a progress update. Can I take this issue and try to work on it? I'm doing it as a part of Women in Ethereum Protocol studies.

Thanks for picking this up, I have assigned you to the issue and will close the open PR, it became stale but you could continue on some of the work that was already done there.