webknossos icon indicating copy to clipboard operation
webknossos copied to clipboard

Out of heap space during volume annotation merging

Open fm3 opened this issue 1 week ago • 1 comments

Volume annotation merging is implemented in pure scala and holds a SegmentInteger object for each voxel of all output buckets in memory. No wonder we got OOMs when merging large volume annotations.

Should be optimized

  • Streaming if possible, write out output buckets directly to db? But when merging more than two input annotations, how to figure out if a bucket is final?
  • Avoid the SegmentIntegers entirely, go for primitive arrays, compare #3810
    • Might even be enough to do this only for the big map, and use SegmentIntegers intermediately
    • Possible alternative: extract parts of volume merging (of the whole?) to C++ via JNI

Also see previous considerations on segment integers: https://www.notion.so/scalableminds/Design-Doc-Typed-Arrays-in-Backend-e4432cf17eef483281eee8eb9623fbaf

fm3 avatar Dec 21 '25 18:12 fm3