[RFC] SIXEL dequantization

[saitoha]

SIXEL images generated by libsixel can exhibit characteristic grainy or banded patterns from error diffusion, especially when using palettes of 64 colors or fewer by suppressing or masking these artifacts, we can improve the balance between visual quality and file size.

Initially I looked at ImageMagick's -selective-dither (which blurs only low-contrast regions), but then I tried the filter from kornelski/undither and implemented its smart idea as sixel2png -d k_undither option, now pushed to the develop branch

core idea of the algorithm:

step 1. mix the reference pixel color p1 with a neighbor p2 to form an average avg, and take the distance between p1 and avg as distance. step 2. search for the palette color pk (excluding p1 and p2) that is closest to avg, and take that distance as min_diff. step 3. weight the blend according to the ratio min_diff / distance, and apply that weight when mixing p1 with surrounding colors.

many thanks to @kornelski for the excellent idea.

In -d k_undither, I slightly customized how p1 and p2 are mixed in step 1 and how the weighting is applied in step 3.

test

original image:

I compared magick -selective-dither, kornelski/undither, and sixel2png -d k_undither on SIXEL images quantized to 256, 128, 64, 32, 16, 8, and 4 colors.

For terminal emulators, we should discuss how to expose this -- either as an DEC-style private mode (default off), or as a control-sequence parameter that selects "the N-th dequantization method". Another option is to declare "the N-th dequantization method" as a decoder hint via the X-th parameter of SIXEL's DCS.

[saitoha]

The undither filter may be most valuable as a first stage before CNN filter like waifu2x, improving the accuracy of subsequent processing plain sixel images - especially with small palettes - don’t benefit much from waifu2x denoise alone, but applying undither first yields a measurable reduction in artifacts.