Implement normal mapping

[markschlosseratbentley]

Look at this Three.js tutorial about normal mapping: https://sbcode.net/threejs/normalmap/

Could we do something similar in the iTwin.js renderer? This could be valuable when rendering terrain, etc. if the source data has a normal map for a mostly-flat surface.

[pmconne]

Yes, we can do normal mapping. It would require WebGL 2 because max texture units, but that's no longer a problem except for iOS users who refuse to update to 15. IIRC the DGN converter already preserves normal maps (and bump, glow, etc maps) from MicroStation when converting materials.

[markschlosseratbentley]

Appears to be high priority.

[pmconne]

(for one user).

[markschlosseratbentley]

@DStradley will look at relevant QV code. @MarcNeely will help code it up in iTwin.

[markschlosseratbentley]

@PeterJBassett We are looking at adding support for normal maps -- could you provide us any information or documentation on how the UVs are stored in the data passed in from FutureOn subsea models?

cc @MarcNeely @pmconne

[markschlosseratbentley]

@PeterJBassett We are looking at adding support for normal maps -- could you provide us any information or documentation on how the UVs are stored in the data passed in from FutureOn subsea models?

cc @MarcNeely @pmconne

Hi @PeterJBassett Just checking in again -- we would like to implement normal mapping for FutureOn subsea models -- do you know what process you are following to create the UV coordinates from a FutureOn normal map?

[PeterJBassett]

Hello @MarcNeely - I apologise, I forgot to reply. Here is some information from a colleague.

So first the UV wrapping should be set to ClampToEdgeWrapping. Filter to Linear.

We use tangent space normal maps. Each pixel in the normal map is actually a normal encoded, where R = X, G = Y and B = Z. The encoding follow the usual :

normal = texture2D( normalMap, vUv ).xyz * 2.0 - 1.0;

The code for genarting the UVs are quite simple, and I think it should be part of the exporter ?

const vertices = new Float32Array(this.resource._heightmapWidth * this.resource._heightmapHeight * 3)
const normals = new Float32Array(this.resource._heightmapWidth * this.resource._heightmapHeight * 3)
const uvs = new Float32Array(this.resource._heightmapWidth * this.resource._heightmapHeight * 2)

let offset = 0
let offset2 = 0
let offset3 = 0

for (let iy = 0; iy < gridY1; iy++) {
  const y = iy * segmentHeight - heightHalf
  for (let ix = 0; ix < gridX1; ix++) {
    const x = ix * segmentWidth - widthHalf

    vertices[offset3] = x
    vertices[offset3 + 1] = y
    vertices[offset3 + 2] = heights[offset]

    normals[offset3] = 0
    normals[offset3 + 1] = 0
    normals[offset3 + 2] = 1

    uvs[offset2] = ix / gridX
    uvs[offset2 + 1] = 1 - iy / gridY

    offset3 += 3
    offset2 += 2
    offset++
  }
}

[markschlosseratbentley]

@MarcNeely @DStradley @pmconne See above reply about UVs + normal maps.

[markschlosseratbentley]

@PeterJBassett

Thanks for the information.

A few more questions--

In the FutureOn example data we have, there is a pattern texture depicting the seabed being drawn in addition to the normal map texture. Is it correct that this is displayed in addition to the normal map?

We also need to know where the UV coordinates for the pattern texture originate. Do you use the same UV coordinates as the normal map texture?

If they are not the same UV coordinates, can they be derived from them using a scaling factor or some kind of transformation?

What is the wrapping mode for the pattern texture?

cc @MarcNeely @DStradley @pmconne

[PeterJBassett]

@MarcNeely @DStradley @pmconne Here is some further info from my colleague.

So, this will get a bit hairy.

There is actually two way of rendering the user can chose from : Simple color ( in this case, we just use the color defined in “seabedColor”. In this case we just display the seabed with the normal map on top. We usually scale a bit the normals so that it looks a bit better.

We use a texture, in this case the textures can be found inside the “public” assets path of the software. The “texture key” is defined in “seaBedTextureName”. We default to “muddyDiffuse” if this value is not set

  const filenamesToLoad = {
    rocksDiffuse: 'RockySeabed_dark_01_1024x1024.jpg',
    rocksLightDiffuse: 'RockySeabed_light_01_1024x1024.jpg',
    rocks2Diffuse: 'rocks2.jpg',
    sandsDiffuse: 'SandySeabed_dark_01_1024x1024.jpg',
    sandsLightDiffuse: 'SandySeabed_light_01_1024x1024.jpg',
    muddyDiffuse: 'MuddySeabed_dark_01_1024x1024.jpg',
    muddyLightDiffuse: 'MuddySeabed_light_01_1024x1024.jpg',
    desertDiffuse: 'DesertSand_01_1024x1024.jpg',
  }
  for (const key in filenamesToLoad) {
    const filename = filenamesToLoad[key]
    const texture = textureLoader.load('/assets/textures/seabed/' + filename, (texture) => {
      threeVisualizer.requestRender()
    })
    texture.wrapS = RepeatWrapping
    texture.wrapT = RepeatWrapping
    texture.encoding = sRGBEncoding
    this.seabedTextures[key] = texture
  }

So for example, using my local dev : if “ https://futureon-designer.lvh.me/assets/textures/seabed/RockySeabed_dark_01_1024x1024.jpg

If we use the texture, the shader is a bit particular as we try to tile it depending on the depth so that it does not look to bad. So basically we generate the UVs dynamically :

{
  shader.vertexShader =
    `
      //attribute vec4 homogeneousPosition;

      uniform vec2 uvOffsetCustom;
      //uniform mat4 modelViewProjectionMatrix;
      uniform float orthographicFakeDistance;

      varying vec2 vUvCustom;
      varying float vDepth;

    `
    +
    insertStringAfterSubstring('<fog_vertex>', shader.vertexShader, `
      vec2 modelPosition = (modelMatrix * vec4(position, 0.0)).xy; // 0.0 is used to ignore translation
      vUvCustom = uvOffsetCustom + modelPosition;
      vec4 viewPosition = modelViewMatrix * vec4(position, 1.0);

      if (isOrthographic) {
        vDepth = orthographicFakeDistance;
      } else {
        vDepth = -viewPosition.z;
      }

      //#if (HAS_PRECALCULATED_HOMOGENEOUS_POSITION == 1)
      //  gl_Position = homogeneousPosition;
      //#else

      //gl_Position = modelViewProjectionMatrix * vec4(position, 1.0);
      // WEIRD! The modelViewProjectionMatrix had more artifacts than in-shader "projectionMatrix * viewPosition"; flashing artifacts for the infinite seabed,
      // and sometimes weird clipping or something making the quad concave. Bug story for future reference: To fix infinite seabed flashing, I wrote code to update
      // the position attribute to be the frustum-to-infinite-seabed intersection. Still had flashing. So I did the homogeneous position calculation on CPU and
      // passed that directly in instead. That fixed it. Later, I looked at the water, and noticed that even though I don't do any special frustum intersection or
      // such (I just set a 1x1 quad's scale to be camera.far * 2), it didn't have any flashing artifacts. I tried doing "projectionMatrix * viewPosition" in
      // the shader instead of passing it as a uniform for the seabed material, to match the water material, and sure enough, that was what caused the flashing
      // for some reason. Not sure why, but perhaps the uniforms have less precision than the in-shader calculations here? Or perhaps a large number of uniforms
      // just messes things up without throwing an error? (Edit: I checked and .capabilities says we have 1024 uniforms available, so weird if it would mess up this way.)
      // Take note of this for the future. However, the frustum intersection still makes UVs precise on the seabed, so I will keep that. However, the homogeneous
      // position attribute doesn't seem like it's needed, hence I commented it out.

      gl_Position = projectionMatrix * viewPosition;

      //#endif

    `)
}

// Extend fragment shader
{
  shader.fragmentShader =
    `
      varying vec2 vUvCustom;
      varying float vDepth;
    `
    +
    replaceSubstring('#include <map_fragment>', shader.fragmentShader, `
      float logDepth = log2(vDepth);
      float repetitions = 3.0;
      vec4 texelColor = mix(
        texture2D(map, vUvCustom / clamp(pow(2.0, floor(logDepth)), float(MIN_TILE_SIZE), float(MAX_TILE_SIZE)) * repetitions),
        texture2D(map, vUvCustom / clamp(pow(2.0, floor(logDepth) + 1.0), float(MIN_TILE_SIZE), float(MAX_TILE_SIZE)) * repetitions),
        fract(logDepth)
      );
      diffuseColor *= texelColor;

    `)
}

[markschlosseratbentley]

Hi @PeterJBassett --

The texture mapping mode you describe above appears to be feasible to implement in iTwin.js as well as regular normal mapping.

Do you know the origin of this texture mapping technique and its name? Is this something you would like to see implemented in iTwin.js? (In addition to normal mapping).

cc @MarcNeely @DStradley @pmconne

[PeterJBassett]

Hello @markschlosseratbentley @MarcNeely @DStradley @pmconne Apologise for the delay replying. There is no standard name for the texture mapping technique.

This is the reply from my colleague: If you mean the way we keep the seabed detailed as we zoom in and out, it's a shader part i wrote, found in TerrainMaterial.js. Not sure if it has a name. It basically shaderized your old LOD mesh method, but with fading between the two closest levels, to avoid popping as you zoom.

[markschlosseratbentley]

In addition to the frontend code, we need normal maps transferred from the backend. We also need scale.

[markschlosseratbentley]

Still in progress.

[pmconne]

@MarcNeely please let me know what I can help with.

[markschlosseratbentley]

@PeterJBassett FYI work on implementing the normal mapping effect in iTwin is finished up. Separate work continues for the special texture mapping mode discussed earlier.

[PeterJBassett]

Hello @markschlosseratbentley Thanks for your update, apologies for not replying sooner. Can you provide any more info on how to use the normal mapping feature, eg. how to input the normal map image into iTwin.

[pmconne]

@PeterJBassett supply a normal map when creating a RenderMaterialElement.