Is it possible to export to an mtl file?

[PsychedelicPalimpsest]

I know from #1 how to export as a ply file, however I have had no luck at exporting any textures. I saw that in shap_e/examples/example_data/cactus/ is an obj file and a mtl file, is this something that could be generated from shap-e?

[PsychedelicPalimpsest]

Update, If anybody else needs this, see PR #20

And if you want a ply file to display color in blender use this tutorial

[eliphatfs]

I have a basic alternative solution using the trimesh library. Also, if you have this same concern as me: vertices generated by marching cubes are very dense, and vertex color is enough for the color resolution.

@torch.no_grad()
def decode_render(
    xm: Union[Transmitter, VectorDecoder],
    latent: torch.Tensor,
    cameras: DifferentiableCameraBatch,
    rendering_mode: str = "stf",
):
    decoded = xm.renderer.render_views(
        AttrDict(cameras=cameras),
        params=(xm.encoder if isinstance(xm, Transmitter) else xm).bottleneck_to_params(
            latent[None]
        ),
        options=AttrDict(rendering_mode=rendering_mode, render_with_direction=False),
    )
    arr = decoded.channels.clamp(0, 255).to(torch.uint8)[0].cpu().numpy()
    return decoded, [Image.fromarray(x) for x in arr]


render_mode = 'stf' # you can change this to 'stf' for mesh rendering
size = 2  # this is the size of the renders; higher values take longer to render.

cameras = create_pan_cameras(size, device)
for i, latent in enumerate(latents):
    decoded, images = decode_render(xm, latent, cameras, rendering_mode=render_mode)
    assert len(decoded['meshes']) == 1
    mesh = decoded['meshes'][0]
    trimesh.exchange.export.export_mesh(trimesh.Trimesh(
        mesh.vertices.cpu().numpy(),
        mesh.faces.cpu().numpy(),
        vertex_colors=mesh.vertex_colors.cpu().numpy()
    ), "barrel-gen.glb")

[dennisushi]

I used meshlab to generate a texture.png from the vertex colors. The export to .obj takes care of the rest.

Ref: https://github.com/facebookresearch/pifuhd/issues/8#issuecomment-645173846

[vangoghvapor]

I have a basic alternative solution using the trimesh library. Also, if you have this same concern as me: vertices generated by marching cubes are very dense, and vertex color is enough for the color resolution.

@torch.no_grad()
def decode_render(
    xm: Union[Transmitter, VectorDecoder],
    latent: torch.Tensor,
    cameras: DifferentiableCameraBatch,
    rendering_mode: str = "stf",
):
    decoded = xm.renderer.render_views(
        AttrDict(cameras=cameras),
        params=(xm.encoder if isinstance(xm, Transmitter) else xm).bottleneck_to_params(
            latent[None]
        ),
        options=AttrDict(rendering_mode=rendering_mode, render_with_direction=False),
    )
    arr = decoded.channels.clamp(0, 255).to(torch.uint8)[0].cpu().numpy()
    return decoded, [Image.fromarray(x) for x in arr]


render_mode = 'stf' # you can change this to 'stf' for mesh rendering
size = 2  # this is the size of the renders; higher values take longer to render.

cameras = create_pan_cameras(size, device)
for i, latent in enumerate(latents):
    decoded, images = decode_render(xm, latent, cameras, rendering_mode=render_mode)
    assert len(decoded['meshes']) == 1
    mesh = decoded['meshes'][0]
    trimesh.exchange.export.export_mesh(trimesh.Trimesh(
        mesh.vertices.cpu().numpy(),
        mesh.faces.cpu().numpy(),
        vertex_colors=mesh.vertex_colors.cpu().numpy()
    ), "barrel-gen.glb")

what is 'trimesh.exchange'? it doesn't seem to exist in my library

[vangoghvapor]

couple follow up questions, how could this be implemented using other render modes (the others throw errors)? how could this be implemented with pytorch3d (texture seems to be implemented differently vs a trimesh)? can this be added as an example in the project?

[InTimmyDate]

I have a basic alternative solution using the trimesh library. Also, if you have this same concern as me: vertices generated by marching cubes are very dense, and vertex color is enough for the color resolution.

@torch.no_grad()
def decode_render(
    xm: Union[Transmitter, VectorDecoder],
    latent: torch.Tensor,
    cameras: DifferentiableCameraBatch,
    rendering_mode: str = "stf",
):
    decoded = xm.renderer.render_views(
        AttrDict(cameras=cameras),
        params=(xm.encoder if isinstance(xm, Transmitter) else xm).bottleneck_to_params(
            latent[None]
        ),
        options=AttrDict(rendering_mode=rendering_mode, render_with_direction=False),
    )
    arr = decoded.channels.clamp(0, 255).to(torch.uint8)[0].cpu().numpy()
    return decoded, [Image.fromarray(x) for x in arr]


render_mode = 'stf' # you can change this to 'stf' for mesh rendering
size = 2  # this is the size of the renders; higher values take longer to render.

cameras = create_pan_cameras(size, device)
for i, latent in enumerate(latents):
    decoded, images = decode_render(xm, latent, cameras, rendering_mode=render_mode)
    assert len(decoded['meshes']) == 1
    mesh = decoded['meshes'][0]
    trimesh.exchange.export.export_mesh(trimesh.Trimesh(
        mesh.vertices.cpu().numpy(),
        mesh.faces.cpu().numpy(),
        vertex_colors=mesh.vertex_colors.cpu().numpy()
    ), "barrel-gen.glb")

Where do I get Transmitter, VectorDecoder, and DifferentiableCameraBatch?

[lemorage]

I have a basic alternative solution using the trimesh library. Also, if you have this same concern as me: vertices generated by marching cubes are very dense, and vertex color is enough for the color resolution.

@torch.no_grad()
def decode_render(
    xm: Union[Transmitter, VectorDecoder],
    latent: torch.Tensor,
    cameras: DifferentiableCameraBatch,
    rendering_mode: str = "stf",
):
    decoded = xm.renderer.render_views(
        AttrDict(cameras=cameras),
        params=(xm.encoder if isinstance(xm, Transmitter) else xm).bottleneck_to_params(
            latent[None]
        ),
        options=AttrDict(rendering_mode=rendering_mode, render_with_direction=False),
    )
    arr = decoded.channels.clamp(0, 255).to(torch.uint8)[0].cpu().numpy()
    return decoded, [Image.fromarray(x) for x in arr]


render_mode = 'stf' # you can change this to 'stf' for mesh rendering
size = 2  # this is the size of the renders; higher values take longer to render.

cameras = create_pan_cameras(size, device)
for i, latent in enumerate(latents):
    decoded, images = decode_render(xm, latent, cameras, rendering_mode=render_mode)
    assert len(decoded['meshes']) == 1
    mesh = decoded['meshes'][0]
    trimesh.exchange.export.export_mesh(trimesh.Trimesh(
        mesh.vertices.cpu().numpy(),
        mesh.faces.cpu().numpy(),
        vertex_colors=mesh.vertex_colors.cpu().numpy()
    ), "barrel-gen.glb")

Where do I get Transmitter, VectorDecoder, and DifferentiableCameraBatch?

I am thinking of the same question, which package do i need to import in my code to make them run?