godot icon indicating copy to clipboard operation
godot copied to clipboard

Add stencil support to spatial materials

Open apples opened this issue 1 year ago • 43 comments

Adds stencil buffer support to spatial materials.

Linked Issues

Supersedes https://github.com/godotengine/godot/pull/78542

Implements the proposed design from https://github.com/godotengine/godot-proposals/issues/7174

Currently integrates https://github.com/godotengine/godot/pull/73527

Testing with the sample project https://github.com/apples/godot-stencil-demo

Differences from the design

  • Added compare operator ALWAYS. The ALWAYS operator is needed for some effects.
  • Added outline and xray presets ahead of schedule because it was easy and made testing easier.
    • The way that next_pass materials are handled is a bit strange right now, but I tried to make it as non-destructive as possible.

Currently implemented

  • Added stencil_mode to shaders, which works very similarly to render_mode.
    • read - enables stencil comparisons.
    • write - enables stencil writes on depth pass.
    • write_depth_fail - enables stencil writes on depth fail.
    • compare_(never|less|equal|less_or_equal|greater|not_equal|greater_or_equal|always) - sets comparison operator.
    • (integer) - sets the reference value.
  • BaseMaterial3D stencil properties.
    • stencil_mode - choose between disabled, custom, or presets.
    • stencil_flags - set read/write/write_depth_fail flags.
    • stencil_compare - set stencil comparison operator.
    • stencil_reference - set stencil reference value.
    • stencil_effect_color - used by outline and xray presets.
    • stencil_outline_thickness - used by outline preset.
  • BaseMaterial3D stencil presets.
    • STENCIL_MODE_OUTLINE - adds a next pass which uses the grow property to create an outline.
    • STENCIL_MODE_XRAY - adds a next pass which uses depth_test_disabled to draw a silhouette of the object behind other geometry.

Supported Renderers

  • [x] Forward+
  • [x] Mobile
  • [ ] Compatibility

Depth Prepass

Stencil effects work well when rendered with a second opaque pass immediately following the current opaque pass. However, with depth prepass enabled, the depth information from any stencil materials is not available for screen-space effects. For instance, with SSAO enabled, opaque stencil materials have strong visual artifacts. Due to this, there might not be a good way to support opaque stencil effects when depth prepass is enabled.

GLES3

More work is needed for GLES3 support. The depth buffer attachment for the scene FBO must be changed to a depth-stencil attachment. Making this change without breaking existing projects will be difficult.

  • See https://github.com/godotengine/godot/commit/f2ec38d26d69943c1e53828f8df88a78f187d106 for an experimental implementation, and find concerns in this comment: https://github.com/godotengine/godot/pull/80710#issuecomment-1694398430.

Sorting Problems

Currently the biggest annoyance with sorting, is that next_pass materials aren't necessarily rendered immediately after their parent. Users have to rely entirely on render_priority to get correct sorting.

Individual stencil effects which have material passes in both the opaque and the alpha passes will have more difficulty being correctly sorted. This might make effects such as portals and impossible geometry more challenging to implement.

apples avatar Aug 17 '23 10:08 apples