scalableminds
Allow uploading ZARR files
One of the following would probably make sense to have for better zarr support:
- Support arbitrary zarr files (probably should be converted with wkcuber so that a mag hierarchy and meta data exist)
- Support zarr files which already follow our format
I’d say it makes sense to run a conversion job for zarr uploads to do re-chunking, sharding, etc.
We could also skip that in case there is already a datasource-properties.json (assuming that the user used the libs to create the zarr dataset already with optimal parameters). In this case, the backend also does not have to infer anything, but can just put the dataset on disk as it comes.
@normanrz @philippotto Do you think the existence of a datasource-properties.json (maybe together with the format-identifying zarr.json) is a good enough heuristic here? My guess is that we would always want to do re-chunking for zarr2 because it does not support sharding?
We don't have a rechunking job yet. So, maybe just ingest the zarr as is and write a datasource-properties.json?
Fair enough. @frcroth I guess a good spot for this would be postProcessUploadedDataSource in UploadService.scala.
It would be nice if you could reuse some of the Explorer code to create the json from the files. I’m not sure how to do that in the datastore. Maybe you can figure that out. The Explorer classes will probably need to be moved. Also have a look at https://github.com/scalableminds/webknossos/pull/7389 for recent changes of the FileSystemDataVault.
The frontend should also be adapted to not set needsConversion=true in this case. You can find a heuristic in the frontend (I think it checks for wkw files being present?).
Please let us know if you need further information!
You can find a heuristic in the frontend (I think it checks for wkw files being present?).
Yes, if the uploaded files contain a WKW file (or a ZIP which contains a WKW file), it is assumed that no conversion is needed. This is implemented in this method:
https://github.com/scalableminds/webknossos/blob/9cecaabfa06e05f9dd2cd5ee1992117a852f742c/frontend/javascripts/admin/dataset/dataset_upload_view.tsx#L475-L475