tensorstore icon indicating copy to clipboard operation
tensorstore copied to clipboard

write multiple chunks into one single file

Open myron0330 opened this issue 4 months ago • 3 comments

in my case, I want to save array into chunks, but I don't want them saved as split files 0.0, 1.0, 2.0 .... Is there any way to write multiple chunks into a file named f"{version}.data", and write meta info into f"{version}.meta", which can be more easily to manage data/meta objects.

myron0330 avatar Aug 13 '25 05:08 myron0330

There isn't a format supported by tensorstore that works exactly like that, but the OCDBT format (https://google.github.io/tensorstore/kvstore/ocdbt/index.html) does something similar and might suit your needs.

jbms avatar Aug 13 '25 16:08 jbms

Not to spam but Just linking to #57 here for future reference.

MosGeo avatar Aug 22 '25 20:08 MosGeo

Isn't that what zarr v3 sharding and the OCDBT format for as @jbms suggested?

tasansal avatar Sep 04 '25 14:09 tasansal