TkTech
S3 BitTorrent downloads
"Any object in Amazon S3 that can be read anonymously can also be downloaded via BitTorrent."
https://docs.aws.amazon.com/AmazonS3/latest/dev/S3TorrentRetrieve.html
"Retrieving a .torrent file for any publicly available object is easy. Simply add a "?torrent" query string parameter at the end of the REST GET request for the object."
It'd be nice if there's an INI option to expose a torrent download link. This will help CKAN instances not be penalized for success when publishing large popular datasets - in terms of availability and bandwidth overages.
Any ideas on how to display it? There's no real way in the UI to currently handle multiple download links for a single resource, and the templates used to list the resources are practically always customized by other sites, so any custom template we provided would either conflict or not be used at all.
Perhaps, using something like a DownloadTorrent resource extras flag to indicate that a resource should be available as a torrent link? If its not present, its just a normal download, if its TorrentOnly, its only available as a Torrent link, and if its TorrentAlso, its available as both a normal download and a torrent link?
Implementing this in a reference template should help guide sites that want to take advantage of this torrents. And if they don't, it will still behave as before...
BTW, this idea was borne out by one user whose datasets are very popular with academia. Oftentimes, universities have very big, practically "free" pipes, and giving this option allows them to have de-facto peering with them.
This also helps if there's another government shutdown similar to 2013 - https://opendata.stackexchange.com/questions/1199/is-data-gov-down-due-to-the-government-shutdown-or-is-it-down-for-good
which inspired the data mirror work you did with UC3 last year - http://uc3.cdlib.org/2017/02/09/government-data-at-risk/
cc @philipashlock @JJediny