ChainSafe
Stream CAR and CAR.zst files directly to the block store
Issue summary
Forest prefers to download snapshots with aria2c for better download speeds in non-western countries. Unfortunately, using aria2c means we cannot stream data directly into our block store. Files have to be downloaded separately and then imported into the block store. If we use a CDN (either DO CDN or CloudFlare), we won't need to use aria2c anymore.
- [x] Upload new snapshots to CloudFlare.
- [x] Update snapshot URLs to point to CloudFlare.
- [x] Remove support for
aria2c. - [ ] Check hash digests when streaming CAR and CAR.zst files.
- [ ] Run the GC to clear unused data from the block store if anything goes wrong.
Other information and links
Download speeds from China:
| Source | Single connection speed | Multi connection speed (aria2c -x5) |
|---|---|---|
| NonCDN | 27 KiB/s | 230 KiB/s |
| DO CDN | 6.5 MiB/s | 19 MiB/s |
| CloudFlare | 7.8 MiB/s | 11MiB/s |
Note: ForestCAR.zst files will soon be used directly as block stores. This means we won't have to copy the data into parityDB.
ForestCAR.zst should not be streamed into ParityDB. See: https://github.com/ChainSafe/forest/issues/3334
It is no longer on the roadmap, as we'll nearly always be using .forest.car.zst files.