Parallelize HDF5 → Zarr conversion

Open ryan-williams opened this issue 7 years ago • 0 comments

@tomwhite's old prototype for traversing HDF5 files (via NetCDF) using Spark is still in the singlecell module

It would be good to add parallelization options to the convert CLI; most likely a Spark-specific code-path will be necessary for that, though in general it would be great to plumb parallelization through cats.Traverse / using Cats' Parallel. Futures and cloud-functions might be workable there.

Oct 17 '18 14:10 ryan-williams