Using different light sources for each view in a batch

[ashwath98]

❓ Questions on how to use PyTorch3D

I saw in a previous issue that Pytorch3D supports 1 lighting source for a batch, Is it possible to have a different light source for each view in a batch? I noticed in the Point Light Source Object , location can be a torch tensor of size (N,3) , I assumed it was for this purpose

In the rendered_textured_meshes.ipynb NB I have edited the Batch Rendering Section to the Following

---

batch_size = 2
meshes = mesh.extend(batch_size)

azim = torch.linspace(0, 180, batch_size)


R, T = look_at_view_transform(dist=2.7, elev=0,azim=azim)

cameras = OpenGLPerspectiveCameras(device=device, R=R, T=T)

lights.location = torch.tensor([[0.0, 0.0, 1.0],[0,1,0]], device=device)

images = renderer(meshes, cameras=cameras, lights=lights)

---

but I get the following error : RuntimeError: The size of tensor a (2) must match the size of tensor b (1048576) at non-singleton dimension 0

with the following stack trace

---

RuntimeError Traceback (most recent call last) in 1 # We can pass arbirary keyword arguments to the rasterizer/shader via the renderer 2 # so the renderer does not need to be reinitialized if any of the settings change. ----> 3 images = renderer(meshes, cameras=cameras, lights=lights)

~/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs) 720 result = self._slow_forward(*input, **kwargs) 721 else: --> 722 result = self.forward(*input, **kwargs) 723 for hook in itertools.chain( 724 _global_forward_hooks.values(),

~/Desktop/pytorch3d/pytorch3d/renderer/mesh/renderer.py in forward(self, meshes_world, **kwargs) 47 """ 48 fragments = self.rasterizer(meshes_world, **kwargs) ---> 49 images = self.shader(fragments, meshes_world, **kwargs) 50 51 return images

~/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs) 720 result = self._slow_forward(*input, **kwargs) 721 else: --> 722 result = self.forward(*input, **kwargs) 723 for hook in itertools.chain( 724 _global_forward_hooks.values(),

~/Desktop/pytorch3d/pytorch3d/renderer/mesh/shader.py in forward(self, fragments, meshes, **kwargs) 110 materials = kwargs.get("materials", self.materials) 111 blend_params = kwargs.get("blend_params", self.blend_params) --> 112 colors = phong_shading( 113 meshes=meshes, 114 fragments=fragments,

~/Desktop/pytorch3d/pytorch3d/renderer/mesh/shading.py in phong_shading(meshes, fragments, lights, cameras, materials, texels) 78 fragments.pix_to_face, fragments.bary_coords, faces_normals 79 ) ---> 80 ambient, diffuse, specular = _apply_lighting( 81 pixel_coords, pixel_normals, lights, cameras, materials 82 )

~/Desktop/pytorch3d/pytorch3d/renderer/mesh/shading.py in _apply_lighting(points, normals, lights, cameras, materials) 26 specular_color: same shape as the input points 27 """ ---> 28 light_diffuse = lights.diffuse(normals=normals, points=points) 29 light_specular = lights.specular( 30 normals=normals,

~/Desktop/pytorch3d/pytorch3d/renderer/lighting.py in diffuse(self, normals, points) 263 direction = self.location - points 264 # pyre-fixme[16]: PointLights has no attribute diffuse_color. --> 265 return diffuse(normals=normals, color=self.diffuse_color, direction=direction) 266 267 def specular(self, normals, points, camera_position, shininess) -> torch.Tensor:

~/Desktop/pytorch3d/pytorch3d/renderer/lighting.py in diffuse(normals, color, direction) 65 normals = F.normalize(normals, p=2, dim=-1, eps=1e-6) 66 direction = F.normalize(direction, p=2, dim=-1, eps=1e-6) ---> 67 angle = F.relu(torch.sum(normals * direction, dim=-1)) 68 return color * angle[..., None] 69

RuntimeError: The size of tensor a (2) must match the size of tensor b (1048576) at non-singleton dimension 0

[liuzhihui2046]

This is a bug, you can modify direction= self.location-points in lighting.py. And colors = (ambient + diffuse) * texels + specular in shading.py to make the dimensions consistent

[simofoti]

Hi, I faced the same issue and I think that the bug can be fixed, as @liuzhihui2046 suggested, by changing the lines below:

in lighting.py

- direction = self.location - points
+ direction = self.location[:, None, None, None, : ] - points

in shading.py

- colors = (ambient + diffuse) * texels + specular
+ colors = (ambient[:, None, None, None, :] + diffuse) * texels + specular

[nikhilaravi]

@simofoti I will add a fix for this issue.

[simofoti]

Oh nice :) thanks @nikhilaravi!

[kjchalup]

This was fixed a year ago, closing the issue.