dh-core icon indicating copy to clipboard operation
dh-core copied to clipboard

datasets : verify downloads with hashes, multithread large downloads

Open stites opened this issue 6 years ago • 2 comments

The datasets downloader could use the above improvements: verifying downloads with hashes, and multithreading large downloads. I've written a version of the first feature in the Setup.ht for a personal moby/dictd replacement, so it might look something like the below:

https://gist.github.com/stites/82acb2036d1654b0ef0c34ec4443579b

stites avatar Jan 02 '19 05:01 stites

The tensorflow mnist downloader also does this and provides an example (recalling from an ancient dangling PR i submitted lol)

https://github.com/tensorflow/haskell/blob/8e1d85b5e5bd56d54ff6d463c8581c57ab5526d9/tensorflow-mnist-input-data/Setup.hs

austinvhuang avatar Jan 03 '19 05:01 austinvhuang

Definitely more official (and better probably better community-mojo) if we use your tf work, @austinvhuang. Removing my Setup.hs -- it's pretty much the same thing.

stites avatar Jan 03 '19 21:01 stites