Lua draw alpha blending is weird

[warmCabin]

Here are a few example screenshots using Mega Man 2 sprites. In each image, the left sprite shows the true colors, the middle sprite is FCEUX's attempt at blending it with the background, and the right sprite is a true alpha blend.

• Bubble Lead
• Quick Boomerang

Notice that the gray Bubble Lead color gets blended down to a perfect gray of #808080, instead of something with a bit of blue in it. The soft pink of the Quick Boomerang did not get blended at all.

Not sure if it has to do with the alpha blending, the 8 bit color palette quantizing, or both. Are all 256 colors always taken up by the NES colors and 8 levels of deemphasis?

I'm also not sure how the 256 color palette interacts with the 64 "unvarying colors."

[bbbradsmith]

1. I believe what it's supposed to do is match the intended blended colour to one already in the palette. I don't think it can do actual RGB blending directly.
2. I think when palettes were updated to be bigger than 256 colours, the lua blend result mapping code was left behind only checking the first 256 colours which might leave it getting poorer results than before, especially since the colours have been rearranged. It should search the whole palette table not just 256 colours. (IIRC the "unvarying" colours are there to provide nice bright colours suitable for lua.)
3. It appears to greatly favour matching green over red, and especially over blue. I've seen perceptual justification for that kind of thing but in my experience you better match if you do something like this instead:

test_score =
    abs(r - tr) *  66 +
    abs(g - tg) * 129 +
    abs(b - tb) * 25;

Might do better as:

test+score =
    (r-tr)*(r-tr) +
    (g-tg)*(g-tg) +
    (b-tb)*(b-tb);

i.e. 1. don't let a good green match dominate other colours, and 2. use squared distance instead of abs to favour matching all colours closely instead of the linear trade of error you get with abs.

[warmCabin]

1. For as long as the main window uses a 512-color palette, that seems to be the case. Maybe we could add something about that to the "on colors" section of the documentation.

2. https://github.com/TASVideos/fceux/blob/master/src/lua-engine.cpp#L3611

for (test = 0; test < 0xff; test++)

Yep! I wonder what kind of effect simply changing 0xff to 0x1ff would have on things.

I'm still kind of thrown off by the fact that all of them should be taken up by the 64 main colors * 8 tint modes but somehow aren't. Check out this scene in Woodman's stage with a regular palette and a weirdo sepia palette.

3. Are those ratios based on the biology of the human eye? Maybe I'll open up a PR and see what happens.

[zeromus]

hey @warmCabin will you review the situation if I poke around at the code?