webknossos
webknossos copied to clipboard
Out of heap space during volume annotation merging
Volume annotation merging is implemented in pure scala and holds a SegmentInteger object for each voxel of all output buckets in memory. No wonder we got OOMs when merging large volume annotations.
Should be optimized
- Streaming if possible, write out output buckets directly to db? But when merging more than two input annotations, how to figure out if a bucket is final?
- Avoid the SegmentIntegers entirely, go for primitive arrays, compare #3810
- Might even be enough to do this only for the big map, and use SegmentIntegers intermediately
- Possible alternative: extract parts of volume merging (of the whole?) to C++ via JNI
Also see previous considerations on segment integers: https://www.notion.so/scalableminds/Design-Doc-Typed-Arrays-in-Backend-e4432cf17eef483281eee8eb9623fbaf