OpenGL integration lookup textures confusion

[cguentherTUChemnitz]

Hi there,

i am integrating this great SMAA in my own OpenGL test framework. I am a bit confused if i handle correctly the lookup textures.

First i had to apply this patch to get the edge detection itself working: https://github.com/pushrax/smaa/commit/cd8f88cd1548a2966eb1c4bfe0677427fa885140

Also i noticed that OpenGL and DirectX have different coordinate system locations in terms of uv-coordinates. So i inverted the Y-axes of the fullscreen triangle texture coordinates to get a buttom up rendering. But then my confusion starts. I use bmp files dumped by the provided python scripts. After reading them in i mirror them, so the Y-coordinate is flipped. After that step i get reasonable results but still different ones compared to the DX10 demo.

So i have no idea what causes the differences in my edge, weight and result pass. I adjusted the images to fit the lower left corner. So direkt comparison should work.

The edges seems to be shifted a little bit, but only some of them.

The weights seem to have flipped red and green parts as well as different intensity. This is the point where i am mostly confused. Is this flipping of red and green parts a result of failed usage of the lookup textures or is it a result of the very small differences from the first pass?

The reults itself looks reasonable and quite good, but on some parts still different. Everything seems thicker in the OpenGL results. Good visable at the wire for example.

So it seems to be related to my questions asked at the old opengl port project: https://github.com/scrawl/smaa-opengl/issues/3

[clusty]

I never got it working properly in openGL. According to the author, the OpenGL version has been broken since some time.

[cguentherTUChemnitz]

What exactly is promoted as broken? I linked the (in my eyes) only necessary patch. The results itself look quite good. Maybe the differences are caused by some pixel/subpixel offsett of the image plane, Hardware and API differences (DX10 did run GTX Titan win 7, OpenGL did run on Quadro K5000 Linux Arch).

So i would like to close the ticket if somone could confirm that the swapped red and green colors in the weighting pass are still reasonable for the image. It is hard for me to validate the weighting colors for reasonability.

[nidefawl]

I can't tell if the swapped red/green are due to a bug or not. But I get the same behavior with my OpenGL implementation. DX edges OpenGL edges DX weights OpenGL weights DX Output OpengGL Output

[iryoku]

I'd love to take a look into this soon. If somebody with troubles in OpenGL could provide me with a minimal OpenGL SMAA demo, that would be help me out to match the DX10 output and the OpenGL one.

I'd like to more officially support OpenGL in future releases.

[turol]

@iryoku I have one at https://github.com/turol/smaaDemo . It uses @mbechard's modified shader from #5 and as far as I can tell renders identically to the DX version.

[iryoku]

That is going to be helpful, thanks!

[mbechard]

Looking at my code I've realized the .dds loading library was automatically doing a vertical flip on the texture to keep up/down correct for the image in GL. So that explains why @turol also needed to do the flip. As a gut-check I tried removing that texture flip and removed the shader modifications. The result is actually much much better than I previously had, however comparing the pixels they are not 100% the same. I would expect the pixels to match exactly. Comparing to the sample DX implementation, the pixels that are different are closer to the flipped GL solution than the unflipped GL solution. FYI.

[RobertBeckebans]

I implemented SMAA into my Doom 3 BFG fork which also uses OpenGL. I also thought that I had to use mbechard's modified shader to get rid of my wrong blend weight calculation pass output but I was wrong. I can confirm that it was only necessary to y-flip the area and search textures.

I only implemented SMAA 1x so far but it is quite fun to experiment with.

If someone is interested in my implementation then here it is: https://github.com/RobertBeckebans/RBDOOM-3-BFG/compare/HDR...RobertBeckebans:SMAA

[biller23]

@RobertBeckebans It seems to me you're using the textures as they are, without y-flip, Am I correct?

[RobertBeckebans]

@biller23 Oh sorry you are right. I forgot that my main problem was that my edges and weights RGBA rendertargets had nearest filtering instead of linear filtering. The weights image is a little bit darker than in the DX10 demo but it seems to work.

[hadadzhi]

Greetings! I'm having a similar-looking problem with DX11: my edges texture is identical to the DX10 demo, but my blending weights texture is darker (no shifting though), resulting in a slightly blurred final image. I checked the loading of the precomputed textures, everything seems fine. What may be causing this?

[iryoku]

Perhaps alpha blending is enabled in the blending weight calculation pass?

[hadadzhi]

@iryoku Thanks for a quick reply! Great suggestion, but in my case blending is disabled across the board (I'm integrating 1x mode for now). Also tried disabling stenciling: no difference. Checked that targets are cleared to 0, including alpha. Maybe there's a special requirement for DX11 that I'm missing?

[iryoku]

I brought the images to photoshop, I think it is a gamma display issue. On the demo the display render target performs gamma correction, if you don't do that in your setup, that might explain the issue. It should yield the same results in practice, as this is just a render target visualization difference. I'll add this to the documentation.

[hadadzhi]

@iryoku Thanks for looking into this! The thing is, other outputs (edges, depth) look identical in my demo and the DX10 one. Also, there is visible blurring of the final result in my case(most visible on the hose in the picture), which is consistent with lower blending weights. May this be an issue with how I compile the shaders or setup defines? I don't use the Effects framework, as the demo does, I compile the shaders with fxc, as vs_4_1 and ps_4_1. (And HLSL version is defined prior to including SMAA.hlsl)

[iryoku]

The edges should look the same because they are either 0 or 1, so they are not affected by gamma.

Judging at the differences, I believe you are doing the blending in gamma space (blending in gamma space gives "thicker" AA). You have to setup the textures as SRGB for the final pass to get the same results as the demo.

[hadadzhi]

@iryoku Turns out, I did in fact mix up the sRGB and non-sRGB views in my demo, and my render target was not sRGB. After a quick fix everything seems to work as it should. Thank you for your time and patience, and for developing a great antialiasing technique!