AUX images support

[bigcat88]

Divided #83 topic into two separate.

Briefly: I can easy add AUX image support whenI will have any non iPhone image, with different auxiliary_type (e.g. non hdrgainmap)

Or when I get information from here: strukturag/libheif/issues/926

Currently we can not determine image mode, and I need to know, can all aux images be decoded as grayscales ones or not. Cause if not, then algorithm will be a little bit weird, need first to try to decode as Greyscale and if decodes fails, then decode it as RGB/RGBA...

[johncf]

Hello. I'm trying to get the HDR gain map aux image, and I see aux image support is yet to be implemented...

with different auxiliary_type (e.g. non hdrgainmap)

Why non-hdrgainmap? Is HDR gain map not planned to be supported, or is it somehow already supported? (I couldn't find any details about this in this repo.)

[bigcat88]

No, at the moment AUX images are not supported (I don't want to write a conversion from 16/12/10 bit images to 8 bit for them) - I'm waiting for Pillow to support HDR images, they seem to be working on it.

As soon as Pillow starts supporting RGB(a) 10/12/16 bit, pillow_heif will immediately be refactored and support for AUX and other things will appear.

[johncf]

I'm waiting for Pillow to support HDR images, they seem to be working on it.

Are you saying that once Pillow supports HDR images, pillow_heif will be refactored to read HDR gain map and combine the SDR and gain maps into a single 10/12/16 bit HDR representation?

If that's what you meant, I wasn't referring to that. I simply wanted to read the HDR gain map as is, independently from the SDR image. From what I can see, libheif already provides a way to extract all AUX images. Would it be within pillow_heif's scope/interest to expose that feature?

[bigcat88]

Would it be within pillow_heif's scope/interest to expose that feature?

definitely yes, but I myself don't want to add reading of only part of AUX images (for example, add reading of gain map which are 8-bit and their reading is relatively easy to add)

I myself will only have time for the final refactoring, when Pillow will support this:

https://github.com/python-pillow/Pillow/issues/1888

---

if you want and you have time, I will gladly accept and test the PR for this, even for supporting only part of the AUX image variants.

[johncf]

Great! I do have some time and started digging in https://github.com/johncf/pillow_heif/tree/aux-images

For now, I made a change that exposes all aux images through aux_image_list property in C (and .info["aux_images"] in Python) so that I can look at them properly. Here, I'm making an assumption that all AUX images are greyscale, which is likely wrong.

1. Do you think it's good (or possible) to make a general implementation that exposes all (non-alpha, non-depth) aux images like this at the moment? (If heif_image_handle_get_preferred_decoding_colorspace works correctly on an aux image handle, then the above false-assumption could be fixed.)
2. If it's not a good idea to expose all aux images as of now (due to various limitations), do you have a preference on how HDR gain map should be exposed? What I'm interested in is the gain map captured by an iPhone, which has aux_type as urn:com:apple:photo:2020:aux:hdrgainmap. Would this be an acceptable path for this: .info["apple2020_hdrgainmap"]?
3. Do you know whether a single HEIF file can have more than 1 aux images having the same aux_type? (this can be accessed through himage.info["aux_images"][0].info["aux_image_type"]) If they're always unique, then .info["aux_images"] can be a dictionary from aux_type to image.

[bigcat88]

1. You can make a list in info["aux"] - all "AUX" images (not all need to be supported at once, for example, to start we can support only 8-bit)
2. Yes, why not. Alternative can be .info["hdrgainmap"]["apple2020"] or .info["aux"]["hdrgainmap"] and set "apple2020" as some property for easy filtering for example.
3. AFAIK, yes, they can be. There is also "subtype" from libheif sources. If anything, you can always ask in the libheif repository, they know a lot there.

[johncf]

I made a PR (#297) with a slightly different approach that doesn't give any special treatment to Apple's way of doing things.

Out of topic: If you like that approach of doing things (i.e., expose only a list of IDs from the HeifImage, and provide a method to get the aux image, given an ID), would it be a good idea to do the same with thumbnails or depth images? (If so, please create an issue and tag me.)

[bigcat88]

Thank you for making PRs, I will review them tomorrow or a day after and ping you of course