Render depth map

[aiyb1314]

Thank you for your outstanding work, does the code implement an interface for rendering depth maps?

[inspirelt]

Hi, two ways can generate the depth map:

1. Override the color with distance from xyz to camera_center and do rasterization:

xyz_dist = torch.norm(xyz - viewpoint_camera.camera_center, dim=-1, keepdims=True)
color = xyz_dist.repeat([1, 3])

2. Modify the CUDA rasterization according to https://github.com/graphdeco-inria/diff-gaussian-rasterization/pull/5/files

[ingra14m]

Hi @aiyb1314 , I have implemented a Differential Gaussian Rasterization with forward depth pass diff-gaussian-rasterization-extentions modified to Scaffold-GS. This is an independent repo that has the same contents with my another repo folked from official diff-gaussian-rasterization.

The only change you need to make is to include depth in the return values of gaussian_renderer/__init__.py:

rendered_image, radii, depth = rasterizer(
        means3D=xyz,
        means2D=screenspace_points,
        shs=None,
        colors_precomp=color,
        opacities=opacity,
        scales=scaling,
        rotations=rot,
        cov3D_precomp=None)

I did not include depth backward pass in that branch. It just outputs the depth. Feel free to use it.

[zsz-pro]

Hi, two ways can generate the depth map:

1. Override the color with distance from xyz to camera_center and do rasterization:

xyz_dist = torch.norm(xyz - viewpoint_camera.camera_center, dim=-1, keepdims=True)
color = xyz_dist.repeat([1, 3])

2. Modify the CUDA rasterization according to https://github.com/graphdeco-inria/diff-gaussian-rasterization/pull/5/files

I tried both methods mentioned above. The middle image corresponds to method 1, and the right image corresponds to method 2. These two are quite similar. However, the image on the left is obtained from a monocular depth model, and compared to that, neither of these methods produces correct depth maps. Here, I used cv2.COLORMAP_JET for the color space of depth mapping. Could you give me some guidance？ Thanks

[inspirelt]

'blue' denotes 'far' in the right two images, but denotes 'near' in the left image. Are there some misaligned rules?

[zsz-pro]

'blue' denotes 'far' in the right two images, but denotes 'near' in the left image. Are there some misaligned rules?

Thanks, I got it. The depth estimated by midas（the pretrained model I use) is the inverse depth maps

[fanzz1208]

I have modify the CUDA rasterization according to https://github.com/graphdeco-inria/diff-gaussian-rasterization/pull/5/files but error:

Training progress: 0%| | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last): File "train.py", line 533, in training(lp.extract(args), op.extract(args), pp.extract(args), dataset, args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, wandb, logger) File "train.py", line 134, in training voxel_visible_mask = prefilter_voxel(viewpoint_cam, gaussians, pipe,background) File "/workspace/sca-gs-depth/gaussian_renderer/init.py", line 236, in prefilter_voxel radii_pure = rasterizer.visible_filter(means3D = means3D, File "/opt/conda/envs/sca_gs_depth/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1695, in getattr raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'") AttributeError: 'GaussianRasterizer' object has no attribute 'visible_filter'

[ingra14m]

Hi @fanzz1208, Scaffold-GS employs a customized rasterization pipeline that adds visible_filter to per-filter the visible Gaussians. It can reduce the number of Gaussians to inference MLPs. This pipeline (https://github.com/graphdeco-inria/diff-gaussian-rasterization/pull/5/files) did not include pre-filter. You can use this modified Gaussian rasterization to achieve your goal.

[fanzz1208]

Hi @fanzz1208, Scaffold-GS employs a customized rasterization pipeline that adds visible_filter to per-filter the visible Gaussians. It can reduce the number of Gaussians to inference MLPs. This pipeline (https://github.com/graphdeco-inria/diff-gaussian-rasterization/pull/5/files) did not include pre-filter. You can use this modified Gaussian rasterization to achieve your goal.

Thank you for your reply！
I use this modified Gaussian rasterization to achieve my goal.
I uninstalled the old version of diff_gaussian_rasterization ：pip uninstall diff_gaussian_rasterization then, : git clone https://github.com/ingra14m/diff-gaussian-rasterization-extentions.git and installed this diff_gaussian_rasterization: pip install submodules/diff-gaussian-rasterization-extentions but still encountered an error.

Traceback (most recent call last):
 File "train.py", line 533, in <module>
   training(lp.extract(args), op.extract(args), pp.extract(args), dataset,  args.test_iterations, args.save_iterations, args.checkpoint_itera
tions, args.start_checkpoint, args.debug_from, wandb, logger)
 File "train.py", line 134, in training
   voxel_visible_mask = prefilter_voxel(viewpoint_cam, gaussians, pipe,background)
 File "/workspace/sca-gs-depth/gaussian_renderer/__init__.py", line 236, in prefilter_voxel
   radii_pure = rasterizer.visible_filter(means3D = means3D,
 File "/opt/conda/envs/sca_gs_depth/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
   raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'GaussianRasterizer' object has no attribute 'visible_filter'
Training progress:   0%|                                                                                            | 0/30000 [00:00<?, ?it/s]

I check ./diff-gaussian-rasterization-extentions/diff_gaussian_rasterization/__init__.py there is no 'visible_filter' in GaussianRasterizer

[ingra14m]

Hi, I'm not sure if you cloned the right branch. You are expected to clone the filter-depth branch for depth visualization and visible_filter.

[DuHao55]

I followed @ingra14m and it has solved the problem, but remember to use git clone https://github.com/ingra14m/diff-gaussian-rasterization-extentions.git --recursive， or you will get another error https://github.com/graphdeco-inria/gaussian-splatting/issues/993#issue-2542272601