RealSense Support

[hhcxx2006]

Hi, I wonder how to use my own RealSense data in this project. I use your realsense dataloader in datasets/gradslam_datasets/realsense.py. There are total 1777 frames, and each frame resolution is 1280*720. The config file is like below. And I initialize the camera pose with P = torch.tensor([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]]).float(), the num is equal the number of frames

After I execute `python3 scripts/splatam.py configs/realsense/splatam.py`, the speed is super slow than Replica dataset. And after a while, it shows an error that cuda is out of memory. However, I use Tesla V100 which has 32G memory. Is the memory enough? And I want it to execute faster like Replica dataset. What can I do? Thank you! Here is the configs/realsense/splatam.py file [splatam.zip](https://github.com/spla-tam/SplaTAM/files/13611133/splatam.zip)

[hhcxx2006]

And even I changed the picture resolution from 1280 * 720 to 640 * 360. The error is also occured. CUDA is out of memory.

[hhcxx2006]

What configs should I use for RealSense D435i? Could you please provide a config file?

[Nik-V9]

Hi, the error could be due to the following reasons:

1. Could you instead use torch.eye(4).float() for the dummy pose? The GradSLAM dataloader converts all the poses relative to the first frame, where the first frame is identity. We use that identity pose to initialize our first frame pose. This might be causing potential bugs in that step.

2. Is the depth scaling & intrinsics correct?

3. If it's a Tracking Failure:

How many iterations of tracking and mapping are you using? Also, what's the frame rate of the capture, and how much is the sensor moving between each frame?

In the current version, SplaTAM's number of tracking iterations depends on the camera motion between the frames.

An easy way to debug this would be to set the following config to True and see if your tracking is converging to the correct pose: https://github.com/spla-tam/SplaTAM/blob/df59ee2759f4d6c43bbd4641859976b587b2e88b/configs/iphone/splatam.py#L64

You can also set the num_frames to a shorter number (10-50) to see if things are working. https://github.com/spla-tam/SplaTAM/blob/df59ee2759f4d6c43bbd4641859976b587b2e88b/configs/iphone/splatam.py#L59

[hhcxx2006]

Hi, the depth scaling & intrinsics are correct, I am sure. The intrinsics in above picture is not the newest.

I use 40 iterations of tracking and 60 iterations of mapping, just the same as the config file of Replica.

The frame rate is 10 FPS, maybe the frame rate is too low. I will use 30 FPS to have another test. Thank you!

By the way, I wonder if there is a way to visualize the whole process of reconstruction? The words in the terminal are not quite intuitive. Can I visualize the whole process for RealSense data?

[Nik-V9]

I recommend not using the Replica Config file as your base config. The iPhone config file is more reasonable in terms of learning rate and other parameters.

Unfortunately, we don't currently have a way to look at the reconstruction while SplaTAM is running. That's in the work. We generally use the visualize_tracking_loss config to debug out the tracking. Once tracking works, the rest of the pipeline typically works seamlessly.

[hhcxx2006]

Okay! Thank you for your help! I get it. I will have another try. Thank you!

[hhcxx2006]

Hi，I have collected data many times. During the collection process, I move the camera slowly and walk slowly. I use 30 fps to collect the data, but finally the code still shows cuda is out of memory. The 32G memory is used up. What should I do? What's wrong with this?The resolution is 1280 * 720. If num_frames set to a shorter number (10-50), it can run sucessfully, but I need to reconstruct the whole scene. I need the whole data. Thank you!

[hhcxx2006]

If I provide external camera pose and do not want to use your tracking method, what should I do?

[hhcxx2006]

This is tracking loss visualization. Is it correct? How to judge the loss is correct?How to debug tracking?

[libaiwanovo]

Thank you for providing the code and for your question. I am using a Realsense D435i camera. Could you please explain how to create a dataset? Also, what modifications are needed for the author's code? I am encountering the following error.Looking forward to your reply.

[ALeHitachi]

Also interested in this! Starting to try with Realsense D455

[Nik-V9]

Hi @hhcxx2006, it would be great if you could share the data so that we can take a look (if things aren't still working).

Thanks for your interest in a RealSense demo. We will potentially consider this for V2.

[Zhirui86]

I'm using the config file for replica to test datasets collected by realsense d435.It turns out that the mapping quality is not good enough,especially the edges and corners cannot be reconstructed.(The intrinsics and depth scale are set accurately).Can I get some suggestions on selecting configuration parameters?

[jeezrick]

If I provide external camera pose and do not want to use your tracking method, what should I do?

Just set use_gt_pose=True, as `configs/replica/splatam.py':

    tracking=dict(
        use_gt_poses=False, # Use GT Poses for Tracking
        forward_prop=True, # Forward Propagate Poses
        num_iters=tracking_iters,
        use_sil_for_loss=True,
        sil_thres=0.99,
        use_l1=True,
        ignore_outlier_depth_loss=False,
        loss_weights=dict(
            im=0.5,
            depth=1.0,
        ),
        lrs=dict(
            means3D=0.0,
            rgb_colors=0.0,
            unnorm_rotations=0.0,
            logit_opacities=0.0,
            log_scales=0.0,
            cam_unnorm_rots=0.0004,
            cam_trans=0.002,
        ),
    ),

But in my experience, even if I use gt_poses, the mapping result isn't good enough. I come up with some reasons for this :

• Tracking trajectory isn't complicated enough. I mean maybe good mapping result need up and down trajectory like replica dataset.
• Gaussians prune method. Maybe the authors finetuned those methods to accommodate those datasets they evaluate at.
• As @Nik-V9 mentioned, learning rate issue.

[Carloslius]

Hello, have you solved this problem? Even if I set the parameters very low, my computer will still run out of GPU memory.

[novellaalvina]

Hi, I'm trying to run splatam on a realsense D415c camera. It works but I've been having this issue where there is a black hole at the middle if I use the pose calculation from the algorithm you provided, which is: datasets/gradslam_datasets/realsense.py

for posefile in posefiles:
    c2w = torch.from_numpy(np.load(posefile)).float()
    _R = c2w[:3, :3]
    _t = c2w[:3, 3]
    _pose = P @ c2w @ P.T # transforming the pose to the appropriate coordinate system
    poses.append(_pose)

Right now, I'm using iphone config and here is what I've tried so far:

1. try recording from a fixed spot (from one angle)
2. try recording while also moving
3. try recording not with a black or white background
4. use dummy pose = torch.eye(4) as suggested in the post above
5. set use_gt_pose = True in the config file as also suggested in the post above

Results:

1. if use_gt_pose = False as default with non-blackorwhite background: Tracking result can only be seen using visualize_tracking_loss=True in the config file, because at iteration 30ish, it freezes the computer. It has some little black holes at the beginning but after several keyframe iteration it shifts and eventually the results kind of look like attempt 4

before

after

2. if use_gt_pose = False with white background: Tracking result can only be seen using visualize_tracking_loss=True in the config file, because at iteration 30ish, it freezes the computer.

3. if use_gt_pose = True: no freezes:

   Attempt 2: fixed spot - dummy pose

   Attempt 4: fixed spot - c2w transform pose

   Attempt 5: moving - dummy pose

Do you have any suggestions on what I should do to refine the results?