Incorrect normal maps rendered using AOV integrator with `normalmap` BSDF

[stymbhrdwj]

Summary

I am trying to render normal maps after applying a normal texture on the object using normalmap BSDF with the sh_normal variable of the AOV integrator. Below is the desired normal map

However, I am getting this normal map

System configuration

System information:

OS: Ubuntu 20.04.6 LTS CPU: Intel(R) Core(TM) i9-14900K GPU: NVIDIA RTX A5000 NVIDIA GeForce RTX 4090 Python: 3.11.8 (main, Feb 26 2024, 21:39:34) [GCC 11.2.0] NVidia driver: 535.183.01 CUDA: 12.2.91 LLVM: 12.0.0

Dr.Jit: 0.4.6 Mitsuba: 3.5.2 Is custom build? True Compiled with: Clang 10.0.0 Variants: scalar_rgb scalar_spectral cuda_rgb cuda_ad_rgb cuda_mono cuda_ad_mono llvm_rgb llvm_ad_rgb llvm_mono llvm_ad_mono cuda_rgb_polarized cuda_ad_rgb_polarized cuda_mono_polarized cuda_ad_mono_polarized

[ashisht96]

I have been facing the same issue for a while. I'm not sure if it's the issue with the AOV integrator or the NormalMap BSDF. Under the same setting, the roughness map is rendered correctly according to the given roughness texture. However, the normal map is still flat (incorrect) with no texture mapped to it.

[rtabbara]

Hi @stymbhrdwj ,

Could you please provide a copy of the scene file/assets that you used so that we can attempt to reproduce the issue you're encountering?

[stymbhrdwj]

reproduce_aov_normalmap.zip

I've added the scene file and assets in the above zip file. Run the provided python script to render the normal.

[stymbhrdwj]

I modified the file aov.cpp to add a custom variable tex_normal to the AOV integrator which fetches normals from the BSDF as follows,

case Type::TexturedNormal: {
      Normal3f n(0.f);
      if (dr::any_or<true>(si.is_valid()))
      {
          Mask valid = active && si.is_valid();
          BSDFPtr m_bsdf = si.bsdf(ray);
  
          Frame3f frame = m_bsdf->frame(si, valid);
          n = dr::normalize(si.to_world(frame.n));
      }
  
      *aovs++ = n.x();
      *aovs++ = n.y();
      *aovs++ = n.z();
  }
  break;

with appropriate changes to bsdf.h and bsdf.cpp to make this work. I am able to compile with scalar_rgb and cuda_rgb variants. It works as desired for the scalar_rgb variant, giving me this normal map. For the cuda_rgb variant, I'm getting the following error

Caught a critical exception: drjit::detail::collect_indices(): encountered an uninitialized function argument while recording a virtual function call!

I'm sharing my changes to src folder and include folder in this patch.zip file. I was working on the v3.5.2 branch.

[rtabbara]

Hi @stymbhrdwj,

As you've discovered the issue was that the shading frame perturbations are computed internally within the normalmap BSDF itself and so you need some mechanism to fetch them to write into the AOV buffer.

We're focusing on putting out a new Mitsuba release at the moment, but we'll definitely revisit this to add support for users not developing with a custom build.

[stymbhrdwj]

@rtabbara Thanks for looking into this. For now, I would greatly appreciate any help in resolving the error encountered when running the code on cuda_rgb variant. Any intuitions as to what could be causing this?

[rtabbara]

Hi @stymbhrdwj,

I had a look at this and the issue is in your implementation of the "default" vectorized function call frame in bsdf.cpp

MI_VARIANT Frame<Float> BSDF<Float, Spectrum>::frame(
    const SurfaceInteraction3f & /*si*/, Mask /* active */) const {
    return Frame3f();
}

Unfortunately, when using JIT variants, the default constructor Frame is not going to initialize the internal JIT variables which is why you're getting the encountered an uninitialized function argument while recording a virtual function call! error. I think changing it to something like

MI_VARIANT Frame<Float> BSDF<Float, Spectrum>::frame(
    const SurfaceInteraction3f & /*si*/, Mask /* active */) const {
    return dr::zeros<Frame<Float>>(dr::width(si));
}

should hopefully work. Indeed, if I make the change I get

[stymbhrdwj]

@rtabbara It works! Thanks a ton.

[imanolooo]

Hi,

Is this (applying the proposed patch in aov.cpp, bsdf.cpp,and bsdf.h files) the unique way to be able to have normal map information in the denoiser?

Best,

[ngottrx]

Hi,

I would like to ask whether this issue is solved with the latest release? Thank you!

[rtabbara]

Hi @ngottrx ,

No, not yet. But I'll be working on a fix for this today.

[ngottrx]

@rtabbara Thank you very much! I am waiting for this. Could you inform whenever it is available!

Bests

[minhthao1628]

Hi @rtabbara , I am expecting this feature

[wjakob]

@minhthao1628 You can ask nicely, but this is not reasonable way to get for help in an open source project. You can check out the commits referenced and compile your own version of Mitsuba if you are in such a hurry.

[ngottrx]

Thank the teams so much! It worked as expected!

@wjakob @rtabbara