Why my rendered object is totally white and black ?

[wwdok]

❓ Questions on how to use PyTorch3D

I am learning pytorch3d started by rendering my custom object(a human face), this is the notebook and used assets: render_demo2.ipynb.zip

import torch
import numpy as np
import matplotlib.pyplot as plt
from skimage.io import imread
from utils import *
from scipy.io import loadmat
from pytorch3d.io import load_obj
from pytorch3d.structures import Meshes
from pytorch3d.renderer import (
    look_at_view_transform,
    FoVPerspectiveCameras, 
    FoVOrthographicCameras, 
    Materials, 
    PointLights,
    RasterizationSettings,
    blending,
    MeshRenderer, 
    MeshRasterizer,  
    SoftPhongShader,
    TexturesVertex,
    TexturesAtlas,
    PointsRenderer,
    PointsRasterizationSettings,
    PointsRasterizer
)

if torch.cuda.is_available():
    device = torch.device("cuda:0")
    torch.cuda.set_device(device)
else:
    device = torch.device("cpu")
print("==>> device: ", device)
image_size = 256
camera_dist = 10
elevation = 0
azim_angle = 0
focal = 1015
obj_filename="../BFM09_mesh_35709.obj"
# Get vertices, faces, and auxiliary information:
verts, faces, aux = load_obj(
    obj_filename,
    device=device,
    load_textures=True)

# prepare the texture
# face_color = torch.tensor([[[100,100,100]]],dtype=torch.float32, device=device).repeat((1, 35709, 1))  # torch.Size([1, 35709, 3])
BFM09_face_color = loadmat(r"../face_color.mat")
face_color = torch.tensor(BFM09_face_color['meantex'].reshape(1,35709,3),dtype=torch.float32, device=device)
print(face_color[0,0,:])
face_color_tv = TexturesVertex(face_color)

# Created Meshes object
face_mesh = Meshes(
    verts=[verts],
    faces=[faces.verts_idx],
    textures=face_color_tv)
# Initialize the camera with camera distance, elevation, and azimuth angle
R, T = look_at_view_transform(dist = camera_dist, elev = elevation, azim = azim_angle) 
cameras = FoVPerspectiveCameras(device=device, R=R, T=T,znear=0.01, zfar=50,
                                fov=2*np.arctan(image_size//2/focal)*180./np.pi)

lights = PointLights(device=device, location=[[0.0, 0.0, 1e5]],
                        ambient_color=[[1, 1, 1]],
                        specular_color=[[0., 0., 0.]], diffuse_color=[[0., 0., 0.]])

# Here we set the output image
raster_settings = RasterizationSettings(
    image_size = image_size, 
    blur_radius = 0.0, 
    faces_per_pixel = 1, 
)

blend_params = blending.BlendParams(background_color=[0, 0, 0])

# Create a mesh renderer by composing a rasterizer and a shader
renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=cameras,
        raster_settings=raster_settings
    ),
    shader=SoftPhongShader(
        device=device,
        cameras=cameras,
        lights=lights,
        blend_params=blend_params
    )
)
# Render Meshes object
image = renderer(face_mesh)
# Plot rendered image
plt.figure(figsize=(10, 10))
plt.imshow(image[0, ..., :3].cpu().numpy())
plt.grid("off");
plt.axis("off");

BFM09_mesh_35709.obj.zip face_color.mat.zip This is the obj viewed by MeshLab: This is the rendered image: I am suspecting this issue might caused by the loaded obj, the BFM09_mesh_35709.obj has following format:

v -0.572390 0.429657 0.804101 182.875046 135.040024 107.140038

f 1 2 131

I do not know if pytorch3d support this format well. I thought pytorch3d can not recognize the vertex color in the last three values of each v line, so i use face_color.mat to additionally add texture.

[bottler]

No we don't support those vertex colors in obj. But we could. It's weird no one has requested it before and now two mentions (here and #1265) in one day.

What is facecolor.mat? Is it a color per vertex (like the obj) or per face?

[wwdok]

Haha, this might be just a coincidence ~ facecolor.mat is a matlab file, it stores the color of 35709 vertices, a face has 35709 vertices. The reason why i use facecolor.mat is because as far as i know the common way of instantiating Meshes() is:

mesh = Meshes(
    verts=[verts],
    faces=[faces.verts_idx],
    textures=TexturesAtlas(atlas=[atlas]),)

but my loaded obj's atlas is None, so i change it to this way:

BFM09_face_color = loadmat(r"../face_color.mat")
face_color = torch.tensor(BFM09_face_color['meantex'].reshape(1,35709,3),dtype=torch.float32, device=device)
print(face_color[0,0,:])

face_color_tv = TexturesVertex(face_color)

# Created Meshes object
face_mesh = Meshes(
    verts=[verts],
    faces=[faces.verts_idx],
    textures=face_color_tv)

I am not sure if this is a correct way to instantiate Meshes(). If not, will it result in rendered image only have white and black ?