kaolin Is it possible to render gaussian splatting and mesh simultaneously?

Hi.

I want to render gaussian splatting and mesh, pointcloud, etc. in the scene at the same time with one renderer. The picture in "https://kaolin.readthedocs.io/en/latest/notes/simplicits.html" seems to suggest that this is possible.

However, the examples given in the tutorial seem to only allow rendering of one mesh or gaussian splatting, such as "kal.render.easy_render.render_mesh" or "kaolin.render.camera.kaolin_camera_to_gsplats".

Is it possible to render gaussian splatting and mesh at the same time with kaolin's render? I wonder if it is possible with kaolin's render alone without implementing an additional depth-based z-buffer method.

May 14 '25 05:05 SangHunHan92

Hi there!

Check out this notebook from 3dgrut: https://github.com/nv-tlabs/3dgrut/blob/main/threedgrut_playground/headless.ipynb

3dgrut is a variant of 3D radiative Gaussians that renders with ray tracing (the representation is identical, but rendering is different as opposed to Gaussian Splatting's rasterization). The codebase includes an engine that can render gaussians and meshes simultaneously. The notebook here shows you how to load the a pretrained gaussian scene with the 3dgrut engine, and add meshes to it.

kaolin's visualizer then hooks on top of that engine to provide interactive camera control in Jupyter, with basic gui controls - from there you can customize the viewer as you wish.

Note: while you can load a 3dgs scene, it's recommended to optimize it with the 3dgrut trainer for optimal quality.

Let me know if you need further clarifications!

May 14 '25 06:05 orperel

Thanks! It looks great to use ray tracing to render both the gaussians and the mesh!

However, I'm facing some issues using 3dgrut:

It seems that there is no code in 3dgrut to load the gaussians already trained by Gaussian Splatting's rasterization. 3dgrut's github only contains the code to train the gaussians from the image dataset to train 3dgrut's style gaussian splatting and render them. I want to render the pretrained gaussians with some code modifications, but some of the variable names are different. (https://github.com/nv-tlabs/3dgrut/blob/43947a728a7d171984ee810ebde653288562ae9e/threedgrut/model/model.py#L388) Here is the link to the pretrained gaussian models provided at PhysGaussian. Can I render these directly with 3dgrut? or should I re-train them? (https://drive.google.com/drive/folders/1EMUOJbyJ2QdeUz8GpPrLEyN4LBvCO3Nx)
Looking at Engine3DGRUT class, it takes paths of gs_object and mesh_folder and renders them. Of course, it's not hard to modify them a little bit and use them. However, I would like a simpler way to take as input the parameters of the Gaussian (pos, rot, SH, ...) and the mesh (vertices, faces, colors) and render them directly. Does 3dgrut support this? I want to use 3dgrut only as a renderer.

May 14 '25 12:05 SangHunHan92

If not, is there a way to render mesh and gaussian splats simultaneously in kaolin?

May 14 '25 12:05 SangHunHan92

It seems that there is no code in 3dgrut to load the gaussians already trained by Gaussian Splatting's rasterization.

There is:

# Ply exported from Inria's trainer
gs_object = "<YOUR_ROOT>/7f666990-9/point_cloud/iteration_30000/point_cloud.ply"

# Folder containing available mesh assets in obj, glb, gltf format. Code supports usd but this flow is untested.
mesh_assets_folder = "./assets"

# Default config to use for the gaussian object if the saved model doesn't include it
default_config = "apps/colmap_3dgrt.yaml"

from threedgrut_playground.engine import Engine3DGRUT

engine = Engine3DGRUT(
    gs_object=gs_object,
    mesh_assets_folder=mesh_assets_folder,
    default_config=default_config
)

Different trainers and renderers may have different notions of the order of gaussians per pixel / ray. That's why I recommended you at least fine tune the scene with 3dgrt - otherwise it will load but quality may look degraded (i.e. Inria's implementation exhibits the so called popping effect, 3dgrt doesn't - hence the difference).

I would like a simpler way to take as input the parameters of the Gaussian (pos, rot, SH, ...) and the mesh (vertices, faces, colors) and render them directly.

Custom Gaussian fields:

# After loading the engine:
gaussians_3dgrut = engine.scene_mog

# Fetch your fields from PhysGaussians
positions, scale, rotation, ... = torch.load(...)

# Override fields
gaussians_3dgrut.positions = torch.nn.Parameter(positions)
gaussians_3dgrut.scale = torch.nn.Parameter(scale)
gaussians_3dgrut.rotation = torch.nn.Parameter(rotation)
gaussians_3dgrut.density = torch.nn.Parameter(density)
gaussians_3dgrut.features_albedo = torch.nn.Parameter(features_albedo)
gaussians_3dgrut.features_specular = torch.nn.Parameter(features_specular)

# Double check everything is ok
gaussians_3dgrut.validate_fields()

# Update engine about changes
engine.rebuild_bvh(gaussians_3dgrut)

For your mesh, I recommend you just export an obj with your fields and place it under the assets folder.

Otherwise its a matter of registering a procedural constructor, something similar to this (I haven't tested, but off the top of my head this should work):

import kaolin
import torch
from threedgrut_playground.engine import OptixPrimitiveTypes

def create_procedural_mesh(vertices, faces, face_uvs, device):
    mesh = kaolin.rep.SurfaceMesh(vertices=vertices, faces=faces, face_uvs=face_uvs)
    mesh.vertex_tangents = torch.zeros([len(mesh.vertices), 3], dtype=torch.bool)
    mesh.material_assignments = torch.zeros([len(mesh.faces)], device=device)
    return mesh.to(device)

def create_your_mesh(device):
    """ Creates a procedurally generated mesh. """
    MS = 1.0
    MZ = 2.5
    v0 = [-MS, -MS, MZ]
    v1 = [-MS, +MS, MZ]
    v2 = [+MS, -MS, MZ]
    v3 = [+MS, +MS, MZ]
    faces = torch.tensor([[0, 1, 2], [2, 1, 3]])
    vertex_uvs = torch.tensor([[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]])
    mesh = create_procedural_mesh(
        vertices=torch.tensor([v0, v1, v2, v3]),
        faces=faces,
        face_uvs=vertex_uvs[faces].contiguous(), # (F, 3, 2)
        device=device
    )
    return mesh

# Register your new construction func
engine.primitives.PROCEDURAL_SHAPES['Your Shape Name'] = create_your_mesh

# Use it to add a new instance
engine.primitives.add_primitive(
    geometry_type='Your Shape Name',
    primitive_type=OptixPrimitiveTypes.DIFFUSE,
    device='cuda'
)

# Let engine know it should refresh internal structures due to changes
engine.invalidate_materials_on_gpu()
engine.primitives.rebuild_bvh_if_needed(True, True)

May 14 '25 13:05 orperel

Thanks for the reply! With your help, I was able to write a simple toy example. The result is an image that renders both Gaussian splats and meshes at the same time.

I have a few additional questions.

When initial declaring Engine3DGRUT, do I "must" pass the paths to gs_object and mesh_assets to initialize it? Engine3DGRUT's __init__ is supposed to receive the paths for gs and mesh folders, so it seems inevitable to initialize it by passing the temporary paths.
I'm curious about how to add colors to the mesh. In the code below, the "PLEASE check here !!" part is where I added for the mesh colors. However, even after adding the colors, only the blue plane mesh is still rendered. What is the correct solution?

import numpy as np
import torch
import kaolin
import torch
from kaolin.render.camera import Camera
import math
from plyfile import PlyData, PlyElement
from threedgrut_playground.engine import Engine3DGRUT, OptixPrimitiveTypes
import torchvision.transforms.functional as F
from matplotlib import pyplot as plt

# 0. make gs parameters, ignore this function.
@torch.no_grad()
def init_from_ply(self, mogt_path:str, init_model=True):
    plydata = PlyData.read(mogt_path)

    mogt_pos = np.stack((np.asarray(plydata.elements[0]["x"]),
                    np.asarray(plydata.elements[0]["y"]),
                    np.asarray(plydata.elements[0]["z"])),  axis=1)
    mogt_densities = np.asarray(plydata.elements[0]["opacity"])[..., np.newaxis]

    num_gaussians = mogt_pos.shape[0]
    mogt_albedo = np.zeros((num_gaussians, 3))
    mogt_albedo[:, 0] = np.asarray(plydata.elements[0]["f_dc_0"])
    mogt_albedo[:, 1] = np.asarray(plydata.elements[0]["f_dc_1"])
    mogt_albedo[:, 2] = np.asarray(plydata.elements[0]["f_dc_2"])

    extra_f_names = [p.name for p in plydata.elements[0].properties if p.name.startswith("f_rest_")]
    extra_f_names = sorted(extra_f_names, key = lambda x: int(x.split('_')[-1]))
    num_speculars = (self.max_n_features + 1) ** 2 - 1
    assert len(extra_f_names)==3*num_speculars
    mogt_specular = np.zeros((num_gaussians, len(extra_f_names)))
    for idx, attr_name in enumerate(extra_f_names):
        mogt_specular[:, idx] = np.asarray(plydata.elements[0][attr_name])
    mogt_specular = mogt_specular.reshape((num_gaussians,3,num_speculars))
    mogt_specular = mogt_specular.transpose(0, 2, 1).reshape((num_gaussians,num_speculars*3))

    scale_names = [p.name for p in plydata.elements[0].properties if p.name.startswith("scale_")]
    scale_names = sorted(scale_names, key = lambda x: int(x.split('_')[-1]))
    mogt_scales = np.zeros((num_gaussians, len(scale_names)))
    for idx, attr_name in enumerate(scale_names):
        mogt_scales[:, idx] = np.asarray(plydata.elements[0][attr_name])

    rot_names = [p.name for p in plydata.elements[0].properties if p.name.startswith("rot")]
    rot_names = sorted(rot_names, key = lambda x: int(x.split('_')[-1]))
    mogt_rotation = np.zeros((num_gaussians, len(rot_names)))
    for idx, attr_name in enumerate(rot_names):
        mogt_rotation[:, idx] = np.asarray(plydata.elements[0][attr_name])

    self.positions = torch.nn.Parameter(torch.tensor(mogt_pos, dtype=self.positions.dtype,device=self.device))
    self.features_albedo = torch.nn.Parameter(torch.tensor(mogt_albedo, dtype=self.features_albedo.dtype,device=self.device))
    self.features_specular = torch.nn.Parameter(torch.tensor(mogt_specular,dtype=self.features_specular.dtype,device=self.device))
    self.density = torch.nn.Parameter(torch.tensor(mogt_densities,dtype=self.density.dtype,device=self.device))
    self.scale = torch.nn.Parameter(torch.tensor(mogt_scales,dtype=self.scale.dtype,device=self.device))
    self.rotation = torch.nn.Parameter(torch.tensor(mogt_rotation,dtype=self.rotation.dtype,device=self.device))

    self.n_active_features = self.max_n_features
    
# Ply exported from Inria's trainer
gs_object = "./models/ficus_whitebg-trained/point_cloud/iteration_60000/point_cloud.ply"

# Folder containing available mesh assets in obj, glb, gltf format. Code supports usd but this flow is untested.
mesh_assets_folder = "./threedgrut_playground/assets_"

# Default config to use for the gaussian object if the saved model doesn't include it
default_config = "apps/colmap_3dgrt.yaml"

# 1. Temp engine
engine = Engine3DGRUT(
    gs_object=gs_object,
    mesh_assets_folder=mesh_assets_folder,
    default_config=default_config
)

# 1.1 Remove initial glass sphere from scene
for mesh_name in list(engine.primitives.objects.keys()):
    engine.primitives.remove_primitive(mesh_name)

# 2. Load Gaussian, MixtureOfGaussians
gaussians_3dgrut = engine.scene_mog
gaussians_3dgrut.init_from_ply(gs_object, init_model=False) # temp code
gaussians_3dgrut.validate_fields()

engine.rebuild_bvh(gaussians_3dgrut)

# 3. Load mesh
def create_procedural_mesh(vertices, faces, face_uvs, vertex_colors, device):
    mesh = kaolin.rep.SurfaceMesh(vertices=vertices, faces=faces, face_uvs=face_uvs, 
                                  vertex_colors=vertex_colors)  # PLEASE check here !! 
                                # vertex_features=vertex_colors)
    mesh.vertex_tangents = torch.zeros([len(mesh.vertices), 3], dtype=torch.bool)
    mesh.material_assignments = torch.zeros([len(mesh.faces)], device=device)
    return mesh.to(device)

def create_your_mesh(device):
    """ Creates a procedurally generated mesh. """
    MS = 1.0
    MZ = 2.5
    v0 = [-MS, -MS, MZ]
    v1 = [-MS, +MS, MZ]
    v2 = [+MS, -MS, MZ]
    v3 = [+MS, +MS, MZ]
    c0 = [1.0, 0.0, 1.0] * 255                # PLEASE check here !!
    c1 = [0.0, 1.0, 1.0] * 255 
    c2 = [1.0, 0.0, 0.0] * 255 
    c3 = [0.0, 0.0, 1.0] * 255 
    faces = torch.tensor([[0, 1, 2], [2, 1, 3]])
    vertex_uvs = torch.tensor([[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]])
    mesh = create_procedural_mesh(
        vertices=torch.tensor([v0, v1, v2, v3]),
        faces=faces,
        face_uvs=vertex_uvs[faces].contiguous(),
        vertex_colors=torch.tensor([c0, c1, c2, c3]), # PLEASE check here !!        
        device=device
    )
    return mesh

engine.primitives.PROCEDURAL_SHAPES['testmesh'] = create_your_mesh

engine.primitives.add_primitive(
    geometry_type='testmesh',
    primitive_type=OptixPrimitiveTypes.DIFFUSE,
    device='cuda'
)

engine.invalidate_materials_on_gpu()
engine.primitives.rebuild_bvh_if_needed(True, True)

# 4. Camera setting
engine.camera_type = 'Pinhole'
engine.camera_fov = 60.0
engine.use_spp = True
engine.antialiasing_mode = '8x MSAA'

view_matrix = torch.tensor([[-0.45889947,  0.5104062,  -0.7272533,   0.        ],
                            [-0.32795742,  0.66341954, 0.67254806,  0.        ],
                            [ 0.8257468  , 0.54713947, -0.13705136, -3.0033283 ],
                            [ 0.          ,0.         , 0.        ,  1.        ]])
camera = Camera.from_args(
                view_matrix=view_matrix,
                fov=math.pi * 45 / 180,
                width=1920, height=1080,
            )

# 5. Render a full quality frame
framebuffer = engine.render(camera)
rgba_buffer = torch.cat([framebuffer['rgb'], framebuffer['opacity']], dim=-1)

# 5.1 Display
chw_buffer = rgba_buffer[0].permute(2, 0, 1)
img = F.to_pil_image(chw_buffer)
plt.imshow(img)
plt.axis('off')
plt.show()

May 15 '25 12:05 SangHunHan92

When initial declaring Engine3DGRUT, do I "must" pass the paths to gs_object and mesh_assets to initialize it? Engine3DGRUT's init is supposed to receive the paths for gs and mesh folders, so it seems inevitable to initialize it by passing the temporary paths.

Correct. The playground engine is meant to visualize trained scenes after all. You could load a dummy gs object if you want but the engine sets the scale of the scene when the first gs is loaded, so a slightly better workaround is postpone the creation of the engine until you have a Gaussian mixture object.

I'm curious about how to add colors to the mesh. In the code below, the "PLEASE check here !!" part is where I added for the mesh colors. However, even after adding the colors, only the blue plane mesh is still rendered. What is the correct solution?

Diffused meshes use materials. What you want to do is either add a new material and set it to your mesh, or edit an existing material.

This is how you add a new procedural material: https://github.com/nv-tlabs/3dgrut/blob/main/threedgrut_playground/engine.py#L345

For editing an existing material, you could pick the material from engine.primitives.registered_materials and change the diffuse_factor. Similar to here,

Don't forget to call engine.invalidate_materials_on_gpu() by the end!

May 18 '25 08:05 orperel

Thanks to you, I was able to apply a material to my mesh.

Now I would like to ask a final question. In the code above, I set MS = 100.0, MZ = 250.0 to make the mesh size very large, but the rendering result still shows the mesh with the same size. It seems that the code internally adjusts the mesh size when rendering. Looking at the code, it seems that mesh_autoscale_func is responsible for this.

However, I don't know how to adjust mesh_autoscale_func in Engine3DGRUT and Primitives. Is it possible to disable mesh_autoscale_func in add_primitive, or prevent mesh_autoscale_func from being used when engine=Engine3DGRUT is first declared?

May 20 '25 14:05 SangHunHan92

Stale issue, please reopen if still relevant

Jul 19 '25 21:07 github-actions[bot]