cucim icon indicating copy to clipboard operation
cucim copied to clipboard

[FEA] Support Cloud-native filesystem (S3, GCS)

Open gigony opened this issue 3 years ago • 0 comments

Is your feature request related to a problem? Please describe.

It would be great if cuCIM's input can be a pointer(URL) of remote storage.

Describe the solution you'd like

For remote/network file system using RDMA, cuFile (GDS) API can be used. For Cloud storage file system, we can leverage existing libraries in C++ or Python (https://fsspec.github.io/kerchunk/, https://github.com/fsspec/filesystem_spec ).

Additional context

  • rasterio implements virtual file system in C++ : https://rasterio.readthedocs.io/en/latest/topics/vsi.html
  • tifffile supports Zarr and filesystem_spec: https://github.com/cgohlke/tifffile/issues/56
  • tiffslide that uses tifffile supports filesystem_spec : https://github.com/bayer-science-for-a-better-life/tiffslide#access-files-in-the-cloud
  • https://github.com/rapidsai/cucim/issues/94
  • https://github.com/rapidsai/cucim/issues/221

gigony avatar Mar 31 '22 15:03 gigony