improve memory consumption while image processing

[cyBerta]

Using the current iOS app 1.22.1 with core 1.60.0 an image of 9.5 mb sent by deltachat consumes more than 250 MB RAM. This leads to crashes of the share extension app as described here https://github.com/deltachat/deltachat-ios/issues/1330.

I wonder if the image processing can be memory optimized. Moreover it would help if there was an API to define a fixed memory limit an image processing task can allocate and if it exceeds to raise an error, the UI can catch. This way we could avoid the situation that iOS throws an EXC_RESOURCE RESOURCE_TYPE_MEMORY exception, which can't be handled by try/catch.

[r10s]

background figured out at https://github.com/deltachat/deltachat-ios/issues/1330#issuecomment-919018488: the image in question has about 120 MB raw data - so the mentioned 250 MB needed for recoding seems reasonable.

however, as recoding fails we do not really know how if the 250 MB would be sufficient - would be interested to know the memory consumption for an image with say, 16mb raw data (2000px * 2000px) - if that is 32mb then, i'd say, core cannot do much upon that.

[cyBerta]

There are ways to load only tiles of an image, scale them and write them back to the memory before loading the next chunk and probably other even more efficient ways I'm not just aware of. Fixing this issue would give a UX boost for sharing larger images on iOS and generally it would help to not being killed under memory pressure.

[link2xt]

I have profiled memory consumption of the test suite. When you run cargo test, it outputs something like

Running unittests src/lib.rs (target/debug/deps/deltachat-c710e567f9c3c143)

in the first few lines. This way I find the binary name (target/debug/deps/deltachat-c710e567f9c3c143) and run

$ heaptrack target/debug/deps/deltachat-c710e567f9c3c143

At the end of testing, heaptrack outputs:

heaptrack stats:
        allocations:            20903794
        leaked allocations:     2735
        temporary allocations:  521322
Heaptrack finished! Now run the following to investigate the data:

  heaptrack --analyze ".../deltachat-core-rust/heaptrack.deltachat-c710e567f9c3c143.420312.zst"

heaptrack_gui detected, automatically opening the file...

JPEG-related functions in image recoding are near the top of the memory peak flamegraph:

[r10s]

thanks for the profiling - and sorry for coming back so late.

i am currently on trying to find a solution for https://github.com/deltachat/deltachat-ios/issues/1330 - the core issue seems to be that iOS extensions must not use more than 120 mb in no means

JPEG-related functions in image recoding are near the top of the memory peak flamegraph:

tbh, i have no good idea what that should tell me :) which image size does the benchmark try to decode? a typical panorama may be sth as 16382 x 3690 pixels (tested on iphone7, iphone13) - so 180mb raw data assuming 3 bytes/pixel - i assume that is fully loaded to memory before trying to recode to sth. as 1280 x 288 would be? can you confirm the 3 bytes/pixel?

unless the recoding in core is achieved smarter, my idea is to not pass such large images to core from the share extension but show a message to the user, saying, you can do this in the app (which allows using up to 2000mb peak). alternatively, we could try to pre-scale the image using iOS, that seems to work better (eg. the thumbnails of the huge panoramas is displayed)

there is probably not much that can be improved in core about that and we could close this issue then (which is also no bug then)