Independently verify downloaded datasets

[ghost]

Some datasets may contain hashes/checksums of the included files, which makes it easy to verify the integrity of the downloaded files. For datasets which don't, it's useful to download them again somewhere else, and compare the ipfs hashes.

This implies that downloaders sign and publish the ipfs hashes of the stuff they've finished downloading.

[mguentner]

You could use git-annex for that: http://git-annex.branchable.com/ It's already tested for large archives and could be used if the IPFS import needs to be repeated in the future. This could also serve as a basis for the initial download processing.