[🐛 bug report] Python custom integrator: type mismatch in load_dict + runtime error when rendering

[tomasiser]

Summary

When using a custom integrator in Python based on docs/examples/03_direct_integrator/direct_integrator.py, I noticed 2 issues:

1. The integrator is not properly registered in load_dict, only in load_file.
2. Calling scene.integrator().render causes a runtime error.

System configuration

• Platform: Linux
• Compiler: gcc
• Python version: 3.6.9
• Mitsuba 2 version: 2fb0c9634e696887fae3921a60818a3a503c892e
• Compiled variants: "scalar_rgb", "scalar_spectral", "packet_rgb", "packet_spectral", "packet_mono"

Steps to reproduce

Python script (based on docs/examples/03_direct_integrator/direct_integrator.py)

import os
import enoki as ek
import mitsuba

# Set the desired mitsuba variant
mitsuba.set_variant('packet_rgb')

from mitsuba.core import Float, Vector3f, Thread, xml
from mitsuba.core.xml import load_file
from mitsuba.render import (BSDF, BSDFContext, BSDFFlags,
                            DirectionSample3f, Emitter, ImageBlock,
                            SamplingIntegrator, has_flag,
                            register_integrator)


def mis_weight(pdf_a, pdf_b):
    pdf_a *= pdf_a
    pdf_b *= pdf_b
    return ek.select(pdf_a > 0.0, pdf_a / (pdf_a + pdf_b), Float(0.0))


def integrator_sample(scene, sampler, rays, medium, active=True):
    si = scene.ray_intersect(rays)
    active = si.is_valid() & active

    # Visible emitters
    emitter_vis = si.emitter(scene, active)
    result = ek.select(active, Emitter.eval_vec(emitter_vis, si, active), Vector3f(0.0))

    ctx = BSDFContext()
    bsdf = si.bsdf(rays)

    # Emitter sampling
    sample_emitter = active & has_flag(BSDF.flags_vec(bsdf), BSDFFlags.Smooth)
    ds, emitter_val = scene.sample_emitter_direction(si, sampler.next_2d(sample_emitter), True, sample_emitter)
    active_e = sample_emitter & ek.neq(ds.pdf, 0.0)
    wo = si.to_local(ds.d)
    bsdf_val = BSDF.eval_vec(bsdf, ctx, si, wo, active_e)
    bsdf_pdf = BSDF.pdf_vec(bsdf, ctx, si, wo, active_e)
    mis = ek.select(ds.delta, Float(1), mis_weight(ds.pdf, bsdf_pdf))
    result += ek.select(active_e, emitter_val * bsdf_val * mis, Vector3f(0))

    # BSDF sampling
    active_b = active
    bs, bsdf_val = BSDF.sample_vec(bsdf, ctx, si, sampler.next_1d(active), sampler.next_2d(active), active_b)
    si_bsdf = scene.ray_intersect(si.spawn_ray(si.to_world(bs.wo)), active_b)
    emitter = si_bsdf.emitter(scene, active_b)
    active_b &= ek.neq(emitter, 0)
    emitter_val = Emitter.eval_vec(emitter, si_bsdf, active_b)
    delta = has_flag(bs.sampled_type, BSDFFlags.Delta)
    ds = DirectionSample3f(si_bsdf, si)
    ds.object = emitter
    emitter_pdf = ek.select(delta, Float(0), scene.pdf_emitter_direction(si, ds, active_b))
    result += ek.select(active_b, bsdf_val * emitter_val * mis_weight(bs.pdf, emitter_pdf), Vector3f(0))
    return result, si.is_valid(), ek.select(si.is_valid(), si.t, Float(0.0))


class MyDirectIntegrator(SamplingIntegrator):
    def __init__(self, props):
        SamplingIntegrator.__init__(self, props)

    def sample(self, scene, sampler, ray, medium, active):
        result, is_valid, depth = integrator_sample(scene, sampler, ray, medium, active)
        return result, is_valid, [depth]

    def aov_names(self):
        return ["depth.Y"]

    def to_string(self):
        return "MyDirectIntegrator[]"


# Register our integrator such that the XML file loader can instantiate it when loading a scene
register_integrator("mydirectintegrator", lambda props: MyDirectIntegrator(props))

#load_dict({"type": "mydirectintegrator"})

# Load an XML file which specifies "mydirectintegrator" as the scene's integrator
filename = 'test.xml'
Thread.thread().file_resolver().append(os.path.dirname(filename))
scene = load_file(filename)

sensor = scene.sensors()[0]
scene.integrator().render(scene, sensor)

film = scene.sensors()[0].film()
film.set_destination_file('my-direct-integrator.exr')
film.develop()

Scene file

<scene version='2.0.0'>
    <integrator type='mydirectintegrator'>
    </integrator>

    <sensor type="perspective">
        <float name="near_clip" value="1"/>
        <float name="far_clip" value="1000"/>

        <transform name="to_world">
            <lookat target="0.5, 0.0, 1.5"
                    origin="0.0, -12.0, 1.5"
                    up    ="0.0, 0.0, 1.0"/>
        </transform>

        <film type="hdrfilm">
            <rfilter type="box"/>
            <integer name="width" value="1024"/>
            <integer name="height" value="768"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="4"/>
        </sampler>
    </sensor>

    <shape type="ply">
        <string name="filename"
                value="teapot.ply"/>

        <bsdf type="diffuse">
            <rgb name="reflectance" value="0.9 0.9 0.0"/>
        </bsdf>

        <transform name="to_world">
            <rotate x="0.0" y="1.0" z="1.0" angle="15"/>
            <rotate x="0.0" y="0.0" z="1.0" angle="-15"/>
        </transform>
    </shape>

    <emitter type="point">
        <point name="position" x="2" y="-6.0" z="4.5"/>
        <rgb name="intensity" value="10.0"/>
    </emitter>

    <emitter type="point">
        <point name="position" x="-3" y="-3.0" z="-0.5"/>
        <rgb name="intensity" value="1.0"/>
    </emitter>
</scene>

Runtime error

Running the script with this scene ends with the following runtime error:

2020-08-17 08:44:11 INFO main [xml.cpp:1221] Loading XML file "test.xml" ..
2020-08-17 08:44:11 INFO main [xml.cpp:1222] Using variant "packet_rgb"
2020-08-17 08:44:11 INFO main [xml.cpp:355] "test.xml": in-memory version upgrade (v2.0.0 -> v2.2.1) ..
2020-08-17 08:44:11 INFO main [ShapeKDTree] Building a SAH kd-tree (2256 primitives) ..
2020-08-17 08:44:11 INFO main [ShapeKDTree] Finished. (191 KiB of storage, took 26ms)
2020-08-17 08:44:11 INFO main [SamplingIntegrator] Starting render job (1024x768, 4 samples, 48 threads)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-3d54293d3f66> in <module>
     82 
     83 sensor = scene.sensors()[0]
---> 84 scene.integrator().render(scene, sensor)
     85 
     86 film = scene.sensors()[0].film()

RuntimeError: make_tuple(): unable to convert arguments to Python object (compile in debug mode for details)

load_dict not working

Furthermore, when I try to load the integrator directly via load_dict, I get the following error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-7-bd642e5188e6> in <module>
     74 register_integrator("mydirectintegrator", lambda props: MyDirectIntegrator(props))
     75 
---> 76 load_dict({"type": "mydirectintegrator"})
     77 
     78 # Load an XML file which specifies "mydirectintegrator" as the scene's integrator

RuntimeError: ​[PluginManager] Type mismatch when loading plugin "mydirectintegrator": Expected an instance of type "mydirectintegrator" (variant "packet_rgb"), got an instance of type "Integrator" (variant "packet_rgb")

[Speierers]

Hi @tomasiser ,

The integrator is not properly registered in load_dict, only in load_file.

Indeed this isn't supported at the moment, I will add this to my todo list.

Regarding the Runtime error, could you compile in debug mode for details?

[tomasiser]

Indeed this isn't supported at the moment, I will add this to my todo list.

Wonderful, thank you!

Regarding the Runtime error, could you compile in debug mode for details?

This is the message in debug:

2020-08-18 13:17:36 INFO main [SamplingIntegrator] Starting render job (1024x768, 4 samples, 48 threads)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-3d54293d3f66> in <module>
     82 
     83 sensor = scene.sensors()[0]
---> 84 scene.integrator().render(scene, sensor)
     85 
     86 film = scene.sensors()[0].film()

RuntimeError: make_tuple(): unable to convert argument of type 'mitsuba::RayDifferential<mitsuba::Point<enoki::Packet<float, 8ul>, 3ul>, mitsuba::Color<enoki::Packet<float, 8ul>, 3ul> >' to Python object

I have no idea where that comes from, as scene.sensors()[0] is a PerspectiveCamera:

[Speierers]

Ok this is a well know issue that will be fixed in the refactoring we are working on ATM. The problem is that for packet_* modes, types like Ray are only exposed to Python in there "dynamic" form. In your case Python is looking for the "packet" form of the ray class, which is bound.

Could you use the gpu_* modes instead for the custom python integrator?

[tomasiser]

Is there a quick fix we could do in our own branch to fix the packet_ modes? Unfortunately, the GPU modes are quite slow compared to CPU packet modes on our hardware.

[Speierers]

You could try to edit the ray_v.cpp file to fix this. See how the macro MTS_PY_IMPORT_TYPES_DYNAMIC() defines "dynamic" types while what you want here are "packet" types instead. Just replace it with MTS_PY_IMPORT_TYPES(). This might work but it will break bindings for the GPU modes for sure.

At least that's how I would start ;)

Let me know if this helped.