stacks-blockchain-docker
stacks-blockchain-docker copied to clipboard
stacks-network
feat: skip downloading files if they already exist
Description
During the seed process, something might go wrong when you insert things in the DB so you want to retry the process. Currently you will always redownload the required files even tho they are on your system. To speed up things we could download these files only if they don't exist. Wdyt of this approach @wileyj ?
Type of Change
- Other
Does this introduce a breaking change?
I don't think so
Are documentation updates required?
No
rather than not download the file, i think it would be better to check the checksum of local vs remote before downloading. edit: typed "delete" when i meant "download"
@wileyj updated the pr, now, it check the local checksum to decide if a download is required
@wileyj updated the pr, now, it check the local checksum to decide if a download is required
the change looks like it would work, but i have a question about how this may be used. consider the last steps of the script: https://github.com/pradel/stacks-blockchain-docker/blob/feat/skip-download-file-if-already-exist/scripts/seed-chainstate.sh#L253-L265
is the intent to check the checksum if an archive file was downloaded separately, and then removed at end of the script execution?
i'm on the fence here, but i'm tempted to say we should keep the downloaded files (it's painful to redownload if you need them again).
pending your thoughts on that idea, this looks good to merge though - thanks!
I tested the code on a new server to confirm things are working as intended. Checking the checksum files also helps to see if there is a new version available. In case the seed failed for some reason and you need to restart the process.
i'm on the fence here, but i'm tempted to say we should keep the downloaded files (it's painful to redownload if you need them again).
It's indeed painful (I actually had a lot of "connection reset by peers" errors while downloading the files, maybe we could add some options to curl like automatic retry and some other flags that could help here?) , but once the seed step is done you don't really need these files anymore right?