z5 icon indicating copy to clipboard operation
z5 copied to clipboard

Implement GoogleCloud storage backend

Open constantinpape opened this issue 4 years ago • 3 comments

The new C++ API allows to implement other backends than the filesystem. There is a mock-up implementation for this already. In order to implement actually implement it, the functionality in the gcs namespace needs to be implemented using the google cloud sdk. For reference, see the implementation in namespace filesystem.

The steps involved should be analog to #136. I would prefer to have the aws implementation first, but if someone wants to take a shot at implementing the gcs binding this would be very welcome too.

constantinpape avatar Aug 23 '19 12:08 constantinpape

Trying to build with GCS

/usr/local/google/home/vharron/dev/sca/z5/include/z5/gcs/dataset.hxx:101:25: error: cannot declare field 'z5::gcs::Dataset<signed char>::handle_' to be of abstract type 'z5::gcs::handle::Dataset'
  101 |         handle::Dataset handle_;
      |                         ^~~~~~~

I'm trying to figure out what should be happening, so I'm trying to understand what handles do. I'm not sure what the purpose of handles are. It would be great to have an explanation comment around include/z5/handle.hxx:12

vharron avatar Apr 26 '21 23:04 vharron

I'm trying to figure out what should be happening, so I'm trying to understand what handles do. I'm not sure what the purpose of handles are. It would be great to have an explanation comment around include/z5/handle.hxx:12

That's a good point, I will try to document this part of the C++ API better.

Please note that neither S3 nor GCS are fully supported yet; the API is defined but many of the functions are not implemented yet. Any help on this would be highly appreciated!

constantinpape avatar Apr 27 '21 09:04 constantinpape

@vharron I have fixed the build issues with GCS now and added some explanation on the handles, see https://github.com/constantinpape/z5/blob/master/include/z5/handle.hxx#L14.

constantinpape avatar Apr 30 '21 19:04 constantinpape