image Palette PNG support(Willing to implement)

I work for the weather network and we are looking to potentially use this library for processing our map images. This library looks like it would suit our needs, but there is a big issue we would need to address first.

I would like to be able to use palette pngs from this library due to their small size. We are hoping to use this library so that we don't have to do image processing in python anymore, but a blocking issue for us is that we require support for palette pngs.

My specific use case for this functionality is that I would like to be able to load in a palette png, modify or crop it and then output the cropped png file.

Draft

I'm unsure at the moment where I would place the information of a palette for an image, or if a palette should even be a concept exposed to the user at the high level. Should palettes essentially be hidden and palette images be loaded into an rgb or rgba image, and then given an option to write the file as a palette with automatic palette generation? I think this could be the simpler way of doing this, otherwise with each pixel placement to the image buffer we would have to determine what palette color to use when placing a pixel.

Should there be an enum added to Dynamic Image for dealing with palettes, or should the image just be an rgb or rgba?

If we can answer these questions, I would be happy to attempt to add this support and get feedback from others regarding its feasibility.

Feb 28 '23 15:02 dbrenot-pelmorex

Adding onto this actually, I think it would be good to represent a indexed image at the highest level, but then be able to convert it to an rgba image in order to easily edit it as rgba, and then convert it back to a indexed image, allowing for specifying a palette or automatically generating the best palette using octree depending on the pixels in the image.

Feb 28 '23 16:02 dbrenot-pelmorex

After testing this out I have found the following path forward:

Add a method to the ImageDecoder trait called palette that returns an option with the palette(this matches the behavior of icc)
Add a similar method for transparency, since in indexed images often the transparency info is stored in the transparency section as opposed to with the data or the palette
Use this palette when decoding from png to populate a new field in ImageBuffer called palette which will store the palette for the image. This can be modified by the end user and will be exposed all the way up to the DynamicImage
The from_raw signature will stay the same for compatibility, as I really don't want to break any API's if I don't have to
At least 2 new variants for palette images will be added: ImagePalette8 and ImagePaletteA8. Others can also be added for 1, 2 and 4 bit but i'll have to see how they would practically work since u8 is the smallest data type and this may very well mean checking the values before write or on setting the pixel value
The best way to interact with palette images is to first convert them to ImageRgb8 or ImageRgba and edit them in that state, and then to turn it back into a ImagePalette8 or ImagePaletteA8 image. This would automatically use an OcTree algorithm to find the best color palette for the new image. If the user would prefer to use their own algorithm, they will be able to do so, and the octree algorithm will sit in the imageops colorops section available to be called.

Right now i'm only aiming to support PNG files, but this doesn't exclude paletted bitmap pictures, which i believe i've seen other users complain about not being able to use.

Please let me know if these changes would work and I will update this issue and provide a PR when I have a working solution.

Feb 28 '23 19:02 dbrenot-pelmorex

Add a method to the ImageDecoder trait called palette that returns an option with the palette(this matches the behavior of icc)

That sounds like a decent match in terms of API, definitely pursue that further. Although, minor difference, it may need to take an output buffer to be agnostic of the color representation within the palette. Doing it for transparency / mask (note those are different concepts for instance in TIFF) as a separate method would also work for me, 4 methods for metadata is not getting too close to a threshold where something more generic may be helpful, imo.

At least 2 new variants for palette images will be added: ImagePalette8 and ImagePaletteA8.

Is it necessary to modify DynamicImage? In particular, none of the IO interfaces are written in terms of that type. It wouldn't be too much of a stretch that more complex cases that aren't 'simple rgb matrices' would be stored in a separate buffer type. Modifying ImageBuffer is slightly unexpected but might work? I guess we'll find out in the PR based on the number of changes to provide a proper type argument for its generic P: Pixel. I'm afraid this will run into problems trying to provide a struct ColorIndex(u32) with the implementation but it certainly doesn't hurt to try. Anyways, if that fails then there's the fallback to full untyped Canvas floating around as a PR. This extension would be all-the-more reason to switch the central bag-of-color-bits.

The best way to interact with palette images is to first convert them to ImageRgb8 or ImageRgba and edit them in that state, and then to turn it back into a ImagePalette8 or ImagePaletteA8 image.

Full agree, an approach using explicit conversion methods as opposed to trait would also open the door to provide multiple algorithms for the transformation between the buffer types. Which could be helpful if there are any huge difference in required data, precomputation, hardware usage, … and could let the user choose. I don't really worry about the implementation detail too much, but could you maybe motivate what exact OcTree you'd be considering? As a neighbor-distance in the RGB color space? What about consider another metric such as distance in a Lab space? What about other dithering algorithms?

Right now i'm only aiming to support PNG files, but this doesn't exclude paletted bitmap pictures, which i believe i've seen other users complain about not being able to use.

Sounds good to me. What do you think about making this an incremental work instead of a big PR, split somewhat according to the following list:

Adding a palette extension method to trait ImageDecoder and bindings for png.
ImageBuffer<Indexed> experiments, or familiarizing with other buffer representations.
Rigging up the buffer representation, whether that is ImageBuffer or something else, to read the palette during decoding.
Providing some method of conversion to make the buffer interact with the RGB-pixel-matrix DynamicImage.
Reviewing other color types and implicit conversions like in jpeg, and expanding the concepts for storing the palette to concepts for storing ICC within the buffer for delayed conversion, as well as trying planar images, etc. Okay, you don't have to do that PR yourself :joy:

Feb 28 '23 20:02 197g

Across the image-rs org we already have a bunch of support for handling paletted images. There's our color_quant crate for converting an RGBA image to indexed color. Also, this crate loads PNGs with the EXPAND transformation which causes the decoder to:

Expand paletted images to RGB; expand grayscale images of less than 8-bit depth to 8-bit depth; and expand tRNS chunks to alpha channels.

And while the main image crate doesn't give a lot of options for how PNGs are encoded, the underlying png crate exposes a set_palette method which should enable you to encode paletted PNGs.

In summary, we already transparently convert paletted PNGs to RGB when loading them and have the code for doing the reverse transformation but don't do so automatically when saving.

Feb 28 '23 21:02 fintelia

Across the image-rs org we already have a bunch of support for handling paletted images. There's our color_quant crate for converting an RGBA image to indexed color. Also, this crate loads PNGs with the EXPAND transformation which causes the decoder to:

Expand paletted images to RGB; expand grayscale images of less than 8-bit depth to 8-bit depth; and expand tRNS chunks to alpha channels.

And while the main image crate doesn't give a lot of options for how PNGs are encoded, the underlying png crate exposes a set_palette method which should enable you to encode paletted PNGs.

In summary, we already transparently convert paletted PNGs to RGB when loading them and have the code for doing the reverse transformation but don't do so automatically when saving.

So then if there was just a way to provide a palette or generate one when saving the image then the rest of the support is there?

Mar 01 '23 15:03 dbrenot-pelmorex

I'm not sure if the usecase actually fits what is there. In particular, if I understand properly, it requires us not implicitly converting the palette and keeping the palette available. Also, with regards to color_quant, do note that this is very specific to 8-bit-rgb. This is only one instance of the potential mechanisms to consider for creating a palette, it's not meant to be generic and is not in practice.

Mar 01 '23 15:03 197g

I'm not sure if the usecase actually fits what is there. In particular, if I understand properly, it requires us not implicitly converting the palette and keeping the palette available. Also, with regards to color_quant, do not that this is very specific to 8-bit-rgb. This is only one instance of the potential mechanisms to consider for creating a palette, it's not meant to be generic and is not in practice.

The more I try to reason about how to represent the palette while being able to modify the image in standard ways, the more I find myself agreeing. It may very well be that the best way to achieve what I want is simply with better tooling with the encoding and decoding. I'm a bit worn out on this at the moment, but next I will look at just trying to get the png encoder and decoder to support importing and exporting palette pngs more easily and I think that will suit my usecase just fine

Mar 01 '23 16:03 dbrenot-pelmorex