Implementation of the `pathreparam` integrator with OptiX 7

[Speierers]

Continuation of PR #44.

[abhinavvs]

Does this mean that "pathreparam" plug-in still is not fully released yet (not even on OptiX 6)?

Any rough idea on when this plug-in will be available (and fully integrated with Mitsuba2)? I am really looking forward to this plug-in as I need this feature for my research project.

Thanks!

[Speierers]

The previous implementation based on "Optix 6" wasn't release, although it is publicly available here: https://github.com/loubetg/mitsuba2

[realWDC]

Great works!

Quick question: How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp?

I have seen following comments in this PR; Any hints on how to proceed?

 // TODO: make si differentiable w.r.t. shape parameters if necessary

Thanks!

[Speierers]

Great works!

Quick question: How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp?

I have seen following comments in this PR; Any hints on how to proceed?

 // TODO: make si differentiable w.r.t. shape parameters if necessary

Thanks!

This is not supported at the moment. For this to work, Shape::m_to_world would need to become a differentiable type (e.g. Transform<Point<CUDAArray<float>, 4>> which involves a lot of changes accross the different shape plugins. This will definitively be part of a future PR, but I won't be able to give any ETA at the moment.

[priyamvad]

Hello, any movement on this PR? I'd love to test out the ability to derive a displacement map to match the reference render, something it sounds like this feature should help with?

[Speierers]

Hello, any movement on this PR? I'd love to test out the ability to derive a displacement map to match the reference render, something it sounds like this feature should help with?

Hi @priyamvad, We are currently busy working on a major refactoring of the Mitsuba 2 + Enoki codebase.

You should be able to use this branch as it is for you experiments. Have a look at the heightfield optimization tutorial to get started.

[RiverIntheSky]

Great works! Quick question: How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp? I have seen following comments in this PR; Any hints on how to proceed?

 // TODO: make si differentiable w.r.t. shape parameters if necessary

Thanks!

This is not supported at the moment. For this to work, Shape::m_to_world would need to become a differentiable type (e.g. Transform<Point<CUDAArray<float>, 4>> which involves a lot of changes accross the different shape plugins. This will definitively be part of a future PR, but I won't be able to give any ETA at the moment.

Excuse me, is there any update on differentiable to_world? Is this pathreparam branch necessary for this goal compared to master branch? If I'm going to proceed in this direction myself, where should I make changes? Currently I can't pass Transform4f to optix::Transform4f

[Speierers]

Great works! Quick question: How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp? I have seen following comments in this PR; Any hints on how to proceed?

 // TODO: make si differentiable w.r.t. shape parameters if necessary

Thanks!

This is not supported at the moment. For this to work, Shape::m_to_world would need to become a differentiable type (e.g. Transform<Point<CUDAArray<float>, 4>> which involves a lot of changes accross the different shape plugins. This will definitively be part of a future PR, but I won't be able to give any ETA at the moment.

Excuse me, is there any update on differentiable to_world? Is this pathreparam branch necessary for this goal compared to master branch? If I'm going to proceed in this direction myself, where should I make changes? Currently I can't pass Transform4f to optix::Transform4f

We are busy working on a major refactoring and haven't made any progress on this unfortunately.

Differentiable to_world will for sure introduce discontinuities in the rendering equation, therefore using pathreparam will be necessary to account for those.

You will need to make changes to the shape class and the shape plugins so that there traverse() and parameters_changed() method expose and handle changes in m_to_world.

To pass the transform to optix, you will need to evaluate it and reduce it to a scalar value (e.g. using enoki::hsum which won't do much as every entry of that matrix is a single value).

Good luck!

[RiverIntheSky]

Great works! Quick question: How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp? I have seen following comments in this PR; Any hints on how to proceed?

 // TODO: make si differentiable w.r.t. shape parameters if necessary

Thanks!

This is not supported at the moment. For this to work, Shape::m_to_world would need to become a differentiable type (e.g. Transform<Point<CUDAArray<float>, 4>> which involves a lot of changes accross the different shape plugins. This will definitively be part of a future PR, but I won't be able to give any ETA at the moment.

Excuse me, is there any update on differentiable to_world? Is this pathreparam branch necessary for this goal compared to master branch? If I'm going to proceed in this direction myself, where should I make changes? Currently I can't pass Transform4f to optix::Transform4f

We are busy working on a major refactoring and haven't made any progress on this unfortunately.

Differentiable to_world will for sure introduce discontinuities in the rendering equation, therefore using pathreparam will be necessary to account for those.

You will need to make changes to the shape class and the shape plugins so that there traverse() and parameters_changed() method expose and handle changes in m_to_world.

To pass the transform to optix, you will need to evaluate it and reduce it to a scalar value (e.g. using enoki::hsum which won't do much as every entry of that matrix is a single value).

Good luck!

I am not sure if I understand enoki::hsum correctly. As I read in Enoki Reference, value_t<Array> hsum(Array value), so hsum(Float m_radius) should return value_t<Float>, which I tested is float (my compiler says 'value_t<enoki::DiffArray<CUDAArray<float>>>' (aka 'float'). But as I tried to assign it to float it failed:

error: no viable conversion from 'enoki::DiffArray<CUDAArray<float>>' to 'float'
        float radius = hsum(m_radius);
              ^        ~~~~~~~~~~~~~~

Why is this? It looks like hsum(Float m_radius) still returns Float.

[Speierers]

Hi @RiverIntheSky ,

Could you please open a seperate issue for this? Could you try something like hsum(detach(m_radius)) maybe?

[lo-zed]

Dear developers, First of all thank you for your great tool and all the work done. If I understand correctly differentiation of weights for blended BSDFs as well as mesh positions depend on this PR? Or was it already included as part of another refactoring mentioned above? Do you have an idea if work on this PR will start again in the near future? Thank you very much for your time.

[Speierers]

Hi @idiot-z ,

This PR will stay as it is for now. The refactored codebase will be very different, requiring a complete re-write of this plugin to fully leverage the various new features of the system. I can't say for now whether it will included or not "natively" in the upcoming release.

[lo-zed]

Hi @Speierers Thank you for your prompt response. Just to be sure, is differentiation of weights for blended bsdf supposed to be functional in the current release (fe2a1bd)? Because there is a star next to the corresponding parameter, however I get very strange results.

[Speierers]

Hi @idiot-z , Please open a seperate issue for this as it is completely unrelated to this PR ;)

[mehrab2603]

Is there any plan of merging this with master? I need some features from master (projector emitter) that were implemented after this. Tried merging myself but got so many conflicts that I don't know how to fix.

[Speierers]

@mehrab2603 No we are not going to merge this with master as we are about to release our refactored/redesigned version of the codebase which will render this PR completely out of sync.

In the future, we are planning on implementing integrators that can deal with discontinuities. But this won't be part of the upcoming release unfortunately.