OpenGL: wireframe issues in Picross 3D

[windwakr]

The wireframe rendering in the OpenGL renderer has some differences with hardware in this game(and probably others that use it?).

https://i.imgur.com/AviL83A.jpg https://i.imgur.com/tEZXjSy.png

https://i.imgur.com/7QRKg5p.jpg https://i.imgur.com/9jMSzu0.png

I don't really know my 3D terminology, but there are some edges facing the camera that don't render on the DS. Also, there is something going on at the edge of the screen where the wireframe gets clipped(?).

[windwakr]

I've done some testing, and I've found the unrendered edges are from the polygon_attr for the wireframe polygons being set for back surface rendering. If you modify the ROM* to set polygon_attr for the wireframe polygons to front surface rendering instead, then the real hardware will render all the edges.

*US ROM: 0x16598 | 10 00 9D E5 -> 02 00 A0 E3

[rogerman]

@windwakr, thanks for looking into this further -- that really helps us out.

This issue will have to be fixed in two stages:

1. The OpenGL renderer doesn't do the same polygon clipping that SoftRasterizer does (OpenGL clips fragments only, but not the polygons). The first fix would be to port SoftRasterizer's polygon clipping over to OpenGL, allowing wireframe polygons to render correctly at the screen edges. This seems to be a relatively easy fix.

2. The OpenGL renderer draws wireframes by converting the polygons to line primitives instead of using glPolygonMode(). Unfortunately, OpenGL does not respect backface culling with line primitives.

So why aren't we using glPolygonMode()? Normally, using glPolygonMode() isn't a problem if we're trying to draw triangles as wireframes, but NDS quads need to be converted into two triangles because quad primitives are not available in OpenGL 3.2. Drawing an NDS quad as wireframe using glPolygonMode() will draw an extra line between the two triangles.

Unfortunately, Picross 3D shows that we need to respect the backface culling of wireframe polygons, for which OpenGL only respects backface culling with glPolygonMode(). Therefore, the fix is to go back to using glPolygonMode(), but also figure out a way to prevent drawing of the extra line for NDS quads. This seems to be a much more difficult fix.

[zeromus]

can't we just not produce line primitives for back-facing quads? Why are we sending backface-culled polys to opengl at all?

[rogerman]

That's the problem -- we're not sending backface-culled polygons, but Picross 3D needs them to be culled.

Right now, we're using line primitives in lieu of glPolygonMode(), but OpenGL doesn't backface cull line primitives.

[zeromus]

We'd cull them by not sending them. If we're drawing lines instead of sending them, then we should cull them by not drawing lines. I don't understand you at all.

[rogerman]

After banging my head against this one and trying a lot of very esoteric methods of trying to fix this, I'm gonna have to backburner this issue for now.

AFAIK, this bug is merely cosmetic and does not cause any gamebreaking issues. I'd would rather prioritize on fixing some truly gamebreaking bugs in the OpenGL renderer before attempting to tackle this issue again.