Large stickers lose their transparency when sent

[Amzd]

In iOS 17+ you can select a subject in a photo and drag it out as a sticker. When copying or dragging these they are not resized so if the image was massive, the sticker will be massive.

When pasted into delta chat, we can natively detect that it has transparent corners and send it as sticker. Now when I send a sticker that is too big, core resizes it and it loses transparency and gains a black background. Preferably the sticker would be resized by core without losing transparency. If that is out of scope for core we can resize natively in iOS.

• Operating System: iOS 17
• Delta Chat Version: master
• Expected behavior: Stickers are resized without transparency loss
• Actual behavior: Stickers gain black background
• Steps to reproduce the problem: Drag (or copy) a subject from a large image to Delta Chat
• Screenshots:

• Logs:

[link2xt]

First thing I want to know is if this is still sent as a sticker viewtype or gets converted to image by heuristics. For some history regarding sticker heuristics:

• https://github.com/deltachat/deltachat-core-rust/issues/4611 (screenshots are sent as stickers, original issue mostly happened on iOS IIRC)
• https://github.com/deltachat/deltachat-core-rust/pull/4619 (added heuristics to convert stickers to non-stickers)
• https://github.com/deltachat/deltachat-core-rust/issues/4739 (request to make it possible to force sticker viewtype, used by desktop sticker picker to force the sticker viewtype)
• https://github.com/deltachat/deltachat-core-rust/pull/4819 (PR to add the ability to force stickers) If UI is absolutely sure that it is a sticker, it can set viewtype to sticker and force sticker viewtype, then heuristics is not used. Core heuristics already checks for transparent corners an I think it does not look at the image size or dimensions.

If it is sent as a sticker, then the only bug is that it converts to JPEG instead of resizing the image as PNG which can preserve transparency.

[Amzd]

It is sent as view type DC_MSG_STICKER on iOS

[r10s]

then the only bug is that it converts to JPEG instead of resizing the image as PNG which can preserve transparency.

if goal is to have small byte sizes, this is not a bug. JPG is often at least a magnitude smaller than (even optimized) PNG, see https://github.com/deltachat/deltachat-core-rust/issues/3956#issuecomment-1382725717 and https://github.com/deltachat/deltachat-core-rust/pull/4037#issuecomment-1458878959

ftr, the arguably too large stickers resulting from iOS-auto-generated-drag-from-gallery are scaled down on iOS at https://github.com/deltachat/deltachat-ios/pull/2535 - so, well, maybe we should just stop changing stickers in core.

so, tbh, not sure, what do here. the iOS issue seems to be fixed. and tweaking things again might result in new bugs, might totally be we did not consider all cornercases, at least i cannot say i have a good overview 🤷‍♂️

[adbenitez]

just had the same problem, sending an sticker with transparency, the image got converted to jpeg, I really think stickers should just be sent as is if they are not jpeg, if some compression or down-scaling can be done to PNG images that should be keeping the PNG/transparency not downgrading to JPEG, specially in this case where the image is being sent as sticker

[Amzd]

Also this is not just an iOS issue, you can also get this on macOS if you right click > copy subject from the Photos app there.

[iequidoo]

There's the code in BlobObject::check_or_recode_to_size():

            if *vt == Viewtype::Sticker && exif.is_none() {
                return Ok(name);
            }

which returns before any recoding is done. So apparently the problem is stickers containing Exif. Core always tries to remove Exif for safety, but this is done together with image recoding. Maybe we should keep Exif and not recode a sticker if we detect transparency? NB: We try to detect transparency by checking the image corners.

[adbenitez]

yes in the case of stickers with transparency it doesn't make sense to break them

[iequidoo]

yes in the case of stickers with transparency it doesn't make sense to break them

Still it's interesting where Exif come from. Maybe UIs should remove it if they preprocess the sticker anyway? ... The file from the bug report contains Exif with info like "Color Space", i.e. nothing sensitive, so it's apparently normal for stickers and we should allow this.

[iequidoo]

I tried to reproduce this with adding Exif with Gimp to our test logo image (see above), but for some reason exif crate doesn't detect Exif added by Gimp. With the rabbit image it's reproducible. If somebody has a more lightweight image for adding a test, would be great. I'll try other tools meanwhile. The Exif detection problem is one more bug unfortunately. ... exiftool also creates images with Exif unrecognizable by our code. ... The same with GIF, i can't find a way to add recognizable Exif to formats with transparency. Going to fix and test this with the rabbit image, but it can't be added to the repo.

[iequidoo]

For Exif detection problem in PNGs i created an upstream issue: https://github.com/kamadak/exif-rs/issues/52