Add parallax mapping to bevy PBR

[nicopap]

trafficstars

Objective

Add a parallax mapping shader to bevy. Please note that this is a 3d technique, NOT a 2d sidescroller feature.

Solution

• Add fields to StandardMaterial
• update the pbr shader
• Add an example taking advantage of parallax mapping

A pre-existing implementation exists at: https://github.com/nicopap/bevy_mod_paramap/

The implementation is derived from: https://web.archive.org/web/20150419215321/http://sunandblackcat.com/tipFullView.php?l=eng&topicid=28

Further discussion on literature is found in the bevy_mod_paramap README.

TODO

• [x] Remove the "minimum 2 layers" limitation, would be great to set it to 0 to dynamically disable parallaxing
• [x] Define properly what is the "parallax_depth" field unit. Is it bevy world units? relative to texture size? relative to screen size? I actually don't know :grimacing:
• [x] Add recommendation to blur height maps for better visuals
• [x] rename height_map to depth_map for more explicity
• [ ] License of diffuse texture :grimacing: (from this demo with a Wiphala pattern added on top)

Limitations

• This is a standard Parallax mapping implementation, without hull extraction, so it will look weird on bent/non-planar surfaces
• The depth of the pixel does not reflect its visual position, resulting in artifacts for depth-dependent features such as fog or SSAO
• For the same reason, the the geometry silhouette will always be the one of the actual geometry, not the parallaxed version, resulting in awkward looks on intersecting parallaxed surfaces.
• GLTF does not define a height map texture, so somehow the user will always need to work around this limitation, though an extension is in the works

Future work

• It's possible to update the depth in the depth buffer to follow the parallaxed texture. This would enable interop with depth-based visual effects, it also allows discarding pixels of materials when computed depth is higher than the one in depth buffer
• Using hull extraction would make the shader work on non-linear surfaces
• GLTF extension to allow defining height maps. Or a workaround implemented through a blender plugin to the GLTF exporter that uses the extras field to add height map.
• noise based sampling, to limit the pancake artifacts.
• Add distance fading, to disable parallaxing (relatively expensive) on distant objects
• Simcity (2013) Cone-based approach (also described in GPU gems) (also this) would optimize greatly, it replaces the initial O(n) linear search with a binary search O(log₂(n)). At the cost of some pre-processing and adding a channel to the depth map texture encoding the "breadth" of each pixel. This is blocked on implementation of asset preprocessing.
• Self-shadowing of parallax-mapped surfaces by modifying the shadow map

https://user-images.githubusercontent.com/26321040/223563792-dffcc6ab-70e8-4ff9-90d1-b36c338695ad.mp4

---

Changelog

• Add a depth_map field to the StandardMaterial, it is a greyscale image where white represents bottom and black the top. If depth_map is set, bevy's pbr shader will use it to do parallax mapping to give an increased feel of depth to the material. This is similar to a displacement map, but with infinite precision at fairly low cost.
• The fields parallax_algorithm, parallax_depth and max_parallax_layer_count allow finer grained control over the behavior of the parallax shader.
• Add the parallax_mapping example to show off the effect.

[nicopap]

Demo available at https://nicopap.github.io/bevy_mod_paramap/

[nicopap]

A few limitations to take into considerations:

• The height map is inverted compared to what usually people would expect
• Self Shadowing is not implemented, which results in awkward looks with shadow-enabled light sources. Not sure how hard it is to implement it.
• The earth example looks awkward at the poles, not sure why and if it is something that will generally show up

[schrottkatze]

The height map is inverted compared to what usually people would expect

Godot has the height map inverted too, but solves the confusion by simply calling it a depth map.

Self Shadowing is not implemented, which results in awkward looks with shadow-enabled light sources. Not sure how hard it is to implement it.

Godot doesn't seem to have that either, they suggest just using normal maps for that.

The earth example looks awkward at the poles, not sure why and if it is something that will generally show up

From what the parallax mapping that I've seen in other places so far, that seems to be a general limitation of parallax mapping if used on spheres.

I'd love if this gets some more attention since parallax mapping generally looks amazing if it's used correctly!

[JMS55]

Haven't looked at the PR beyond quickly skimming the code, but I'd like more documentation from the user's perspective. I'm not familiar with parallax mapping - it's some kind of normal map like technique? The StandardMaterial fields should have some documentation explaining what parallax mapping is, briefly how it differs from normal maps, and when/why you would want to use it :)

[superdump]

For those following along, this stalled a bit as I wanted to understand from where the -B was coming in the transformation of the view vector to camera space. And, I wanted to properly understand the details of the algorithm.

As I dug into it, I understood that the -B was because our bitangent calculation follows our right-handed y-up convention in tangent space, which means the bitangent points up. However, in texture coordinates (UVs) the V direction goes down. I modified this to use B but then flip the tangent space view vector's y component when calculating the delta_uv.

Also, I felt it would be more intuitive for understanding the algorithm to have the tangent space view vector pointing into the surface and step along it through the depth layers. By passing -Vt into the parallax_uv function and adjusting the calculation of delta_uv to divide by view_steepness (which is abs(Vt.z) instead of the signed Vt.z, the rest of the code could then be modified to step forward (as in + delta_uv is forwards) along the tangent-space view vector that points from the camera to the fragment.

And finally, because the texture we use has a value of 0 meaning no depth below the geometric surface, and 1 meaning maximum depth, I felt it appropriate to rename height to depth everywhere in the code. Also renamed parallax_depth to parallax_depth_scale to make it clearer that this parameter scales the depth effect.

PR here: https://github.com/nicopap/bevy/pull/1

[superdump]

Some documentation errors: https://github.com/bevyengine/bevy/actions/runs/4658001849/jobs/8243172945?pr=5928#step:6:1700

[robtfm]

@superdump regarding the non-srgb texture, perhaps we could add a loader that looks for a .linear.jpg extension and treats it as non-srgb?

[superdump]

@robtfm I feel like requiring jpeg is better than using more bytes in the repo, but it's not a particularly strong opinion. Good idea about '.linear.jpeg' and I guess '.linear.png', though it's a hack and I hope the asset rework will include metadata files that will make this no-longer a problem. :)

[mockersf]

@robtfm I feel like requiring jpeg is better than using more bytes in the repo, but it's not a particularly strong opinion.

Not requiring a feature would be great... and worth the extra 100KB for me. Most of the size increase comes from the first one, we could make it smaller if that's still an issue for you

Here are pngs that aren't a lot bigger:

[superdump]

Also the examples failed on windows saying something about the lip falling to complete after 1009 iterations. When testing locally I did notice a little stuttering but I was doing other stuff on my laptop so I didn’t think anything of it. Though I did wonder if it was related to tracing rays at glancing angles.

[nicopap]

@mockersf I have the clean base assets I can generate better pngs. Note that when I run cargo run --example parallax_mapping, cargo just refuses to run the example and prints out in the terminal the following:

error: target `parallax_mapping` in package `bevy` requires the features: `jpeg`
Consider enabling them by passing, e.g., `--features="jpeg"`

So in terms of user experience, this is fine IMO, as the solution is directly given to the user. And in any case, teaching them to use the proper bevy features is better in an environment where failure to do so results in the program not running rather than assets mysteriously not loading.

But you are probably worrying about compilation time of examples right? How much of a worry is this?

@superdump Ahah! I happen to only have a very weak Windows machine with a very not modern GPU on hand, so it's going to be hard for me to test out. It seems possible to trigger nagga compilation for Dx12 on linux though? Just for checking? How would I go about that?

[mockersf]

Note that when I run cargo run --example parallax_mapping, cargo just refuses to run the example and prints out in the terminal the following:

error: target `parallax_mapping` in package `bevy` requires the features: `jpeg`
Consider enabling them by passing, e.g., `--features="jpeg"`

So in terms of user experience, this is fine IMO, as the solution is directly given to the user. And in any case, teaching them to use the proper bevy features is better in an environment where failure to do so results in the program not running rather than assets mysteriously not loading.

But you are probably worrying about compilation time of examples right? How much of a worry is this?

Yup it's mostly fine... but from a user point of view, it means they can't simply try one example after the other, they will have recompilation when switching between examples with different required features.

From a CI point of view, it means we'll have to add special case when building examples for tests in CI, or for building the website example page. Or for https://rparrett.github.io/prototype_bevy_example_runner/

Unless really necessary, not requiring a feature for any example is simpler.

[nicopap]

You make a compelling case. I'll swap the assets for pngs.

[nicopap]

Todo when/if this gets merged: Open a tracking issue on possible parallax mapping improvements, ordered by complexity, from easiest to most complex.

1. Distance fading
2. offset limiting method (inaccurate, fast and predictable)
3. Depth buffer update (SSAO, proper shadow maps, proper silhouette)
4. Depth map generation from normal map
5. glTF extras non-standard parallax mapping extension.
6. cone mapping method & cone map generation
7. Quadratic surface vertex attribute generation & usage (parallax mapping on curved surfaces)