BUG Alpha blending on hard edge textures

[ryotoko]

Tracking this separately, as it's different than the fog problem.

As you can see in this image, the outline of the tree is clipping tris with other alpha functions: the tree's shadow, and other trees. Strangely, the tree on the right is not being clipped, but the tree on the left is.

Edit: The problem is that the 3DS does not have a fragment shader or a programmable pixel shader, which is where true depth and alpha tests usually operate. The 3DS implements these at the last stage of the vertex shading pipeline and simulates them, and a depth and alpha test can only be applied once per frame. (This is one way the code might be cleaned up slightly, since Gericom added a lot of depth and alpha test calls apparently assuming the 3DS had fragment shading capabilities when it doesn't.)

As it stands, the texture filter is filtering the textures, which interpolates not only the color values but alpha values as well, and the result is a semi-transparency on the edge. You can cull all alpha values below 255 but you will lose the smoothed edge effect (and part of the texture). Any alpha values not culled will be the testing point of the depth test, which is next in the pipe, and considers the remaining alpha values as solid and culls what's behind them.

The reason that it only happens sometimes has to do with the draw order. The depth test only culls textures that fail the depth test. If the tree was properly drawn in front of a tree that it should be in front of, it is actually not culled and then proper blending occurs, which can be seen by disabling the depth mask.

As changing the depth test and alpha test values will adversely affect something else, the solution here needs to be fairly creative. I've come up with the following:

• Disable the depth mask and create a render queue that first sorts the Z value of the closest vertex, for alpha textures, before sending them to the pipe, drawing them in their proper order and eliminating the need for the depth mask
• Create a custom linear filter function for alpha textures that sets a hard gradient for alpha levels, setting all above a certain line to 255 and all below to 0, eliminating the semi-transparency. Depending on where filtering happens, this could be difficult or computationally expensive
• Create "HD" textures for the problem textures (pretty much just trees) that are pre-filtered and have no semi-transparency, and flag them so the renderer knows not to filter them again

[mkst]

Does this happen on Gericom's version or is this a regression?

[ryotoko]

Gericom's is also affected