Improve img2img color correction by performing a luminosity blend

[ThereforeGames]

Hi,

This PR attempts to solve two longstanding issues with the img2img color correction feature:

• As per #484, "color correction" often produces images that look degraded or posterized
• Inpainted regions often exhibit a slight "color drift" which is particularly noticeable in human subjects, e.g. painting out the head and shoulders may result in a person who has inconsistent skin tone

We can fix (or at least greatly improve) these issues by performing a luminosity blend after the initial histogram match.

When I say "luminosity blend" I am referring to the blending mode function that is present in PhotoShop and other photo editing software. Here's how it works:

Keeps the luminance of the active layer, and blends it with hue and saturation (the color) of the composite view of the layers below. This results in the colors of the underlying layers being blended with the active layer, and replacing them.

Source: https://photoblogstop.com/photoshop/photoshop-blend-modes-explained

In this case, we are blending the color-corrected image with the original output.

Here are some quick examples I threw together:

The differences are subtle, but let me call your attention to a few areas (you may want to zoom in):

• Row 1, Column 3: You can see the posterization effect in Dafoe's shirt - it feels like loss of detail.
• Row 2, Column 3: The current implementation of color correct tends to produce really stark contrast between light and shadow. I'm actually partial to the dramatic look this time... but it's a bit unnatural and not consistent with the source material.
• Row 3, Column 3: There's a dark halo behind robo-Walter's head.

Here's an example featuring subjects of different skin tones - you shouldn't need a magnifying glass for this one:

My modified processing.py requires two new packages, blendmodes and aenum, which I added to requirements.txt.

[atagen]

Could I suggest you use OKLAB rather than plain LAB (which is not designed for image processing and has some inconsistencies of its own)? Might as well fix it for good..

https://bottosson.github.io/posts/oklab/

edit: also, I noticed the openCV routine you call into has a boolean (in C++) for whether or not to linearise (ie. whether or not you are feeding it sRGB). Can't tell at a glance what the python binding's default is but that's something you might want to be wary of - blending in gamma space is a well known source of issues in image processing.

[wywywywy]

Nice work. This may help fix the LDSR upscaler colour issue as well https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5636

[catboxanon]

I'll try writing up a more detailed bug report and maybe look into making a PR if I can, but I wanted to mention briefly here it seems like this feature is broken if it's enabled with the built-in LoRA extension enabled (even with no LoRA model entered in the prompts), and if any upscaler for img2img is used other than a very basic one, like Lanczos. If something like an ESRGAN or SwinIR model is used, it requires the built-in LoRA extension to be disabled, otherwise it will throw an error during the upscaling process.

ESRGAN

RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[1, 4, 192, 192] to have 3 channels, but got 4 channels instead

SwinIR

RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 1

I haven't thoroughly tested with all upscaler types but I imagine it's going to give similar errors.