Shard format with CloudVolume?

[manoaman]

Hello,

I'm tackling with chunking high resolution 3D image and I don't have much experience yet on designing the chunks in this scale. And I was hoping if anyone can share your experience with high res images.

I have a 3D volumetric image (7370 x 10131 x 3900), and 16bit. First, I would like to accomplish chunking XY plane in either 128x128x1 or 256x256x1. And at later stages, I plan to rechunk in 128x128x64 or 256x256x32, and use shard format to reduce the number of chunk size.

The problem is, the network storage which I'm using only allows me to create a little bet over 2 million chunks in a folder. However, the initial upload with CloudVolume is expected to generate this much.

(7370/128)x(10131/128)x(3900/1) = 17,773,152 chunks. (7370/256)x(10131/256)x(3900/1) = 4,443,288 chunks.

Does CloudVolume support shard format at an initial upload? How do I get around with this issue?

(My apology on the redundant post on the Igneous issue board.) https://github.com/seung-lab/cloud-volume/issues/505#issuecomment-1179402935

Thank you so much for your help. -m

[william-silversmith]

Hi m,

I would recommend increasing the chunk size to 1024x1024x1 which is still a reasonable size and would result in a much more manageable number of chunks. I'm investigating providing a way to do sharded initial uploads, but that may be a few weeks or months away from availability.

Will

[manoaman]

Hi Will,

I'm investigating providing a way to do sharded initial uploads, but that may be a few weeks or months away from availability.

Thank you, much appreciated @william-silversmith !!! It's definitely a great software but there are so much to understand!!! 😁

[william-silversmith]

The shard upload procedure is documented here: https://github.com/seung-lab/cloud-volume/wiki/Creating-a-Sharded-Image-from-Scratch