cloud-volume
cloud-volume copied to clipboard
seung-lab
Provide Fortran Order Flag
We can decrease memory requirements a lot by not performing the transposes necessary to convert from fortran to C order. Ran Lu for instance wants to work with Fortran order anyway so it's just wasted work. Provide a flag, e.g. order='C' or order='F' like numpy
why do we close this? Ran do not need it anymore? I am using C order in chunkflow, and was always doing transposing before and after inference. The transpose of affinitymap increased the memory overhead a lot. I would like to make the order flag work. I can work on this and create a PR.
This was closed because all precomputed volumes are in F order. We've been trying to not upload unviewable volumes.
Could you describe your use case?
On Wed, Jan 23, 2019, 4:42 PM Jingpeng Wu <[email protected] wrote:
Reopened #68 https://github.com/seung-lab/cloud-volume/issues/68.
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/seung-lab/cloud-volume/issues/68#event-2092919278, or mute the thread https://github.com/notifications/unsubscribe-auth/ACZoSZ0qI0YL7WKlGsgibTZ2J-n5sNpJks5vGNdjgaJpZM4SCp5v .
we can use C order if you just rename the xyz to zyx? My packages are all based on C order numpy array, so I have to transpose before or after using cloudvolume.
the volume should be viewable if we use C order with c z y x indexing. The memory layout will be exactly the same with neuroglancer precomputed and the indexing style will be consistent with numpy.
I have PR to work on this: https://github.com/seung-lab/cloud-volume/pull/192
I wrote fastremap, which offers in-place transposition to avoid the memory overhead for 2D, 3D, and 4D rectangular arrays.
https://github.com/seung-lab/fastremap
I was initially worried that doing that forces users to use in-place transposition, which can take longer than out of place, but a user could also just not set the flag and do out-of-place transposition outside of CloudVolume. It does also create an additional dependency, but supporting graphene will probably require fastremap anyway.
Actually, while it's clear what to do for downloads, uploading is more tricky. If we applied an IPT to the upload, it would change the input the user feeds to CloudVolume. We could apply it twice, but that would be slow, and there would be a gap where the data was changing which could be weird in a multithreaded environment (e.g. for shared memory).
I think I'm going to close this and just mention fastremap in the documentation for CloudVolume.
Added some documentation.
https://github.com/seung-lab/cloud-volume/wiki/Advanced-Topic:-C-vs-Fortran-Order
Adding this flag will make my downstream code much more cleaner, I am working on it now.