tonic icon indicating copy to clipboard operation
tonic copied to clipboard

Provide peer-to-peer links to increase download speed and resilience

Open Jegp opened this issue 2 years ago • 2 comments

Thanks for a great library! We're starting to rely on it quite a bit at my group.

I can see some datasets, such as the NMNIST, relies on a Dropbox link. This may make complete sense, but it may also be beneficial to use decentralized peer-to-peer services using, for instance, IPFS. That should make the datasets more resilient to link expiration, service outage, and download bandwidth.

This could also be coupled with #111 to decrease the download size and increase unpacking/processing speed.

Jegp avatar Jul 26 '21 19:07 Jegp

thank you for this suggestion. I like IPFS as a technology. Something to keep in mind: Some dataset owners might not like us mirroring their dataset so we have to get permissions.

biphasic avatar Jul 28 '21 08:07 biphasic

Certainly, we shouldn't upload anything we're not allowed to :-)

Jegp avatar Jul 28 '21 20:07 Jegp