microsoft
Question about data-refinement in V2
Thank you for your greet work!
- I would like to ask if data-refinement has plans to open source it, and when it will be open sourced.
- I noticed that the paper used a lot of auto-driving datasets for training. For the sparse lidar depth, is there any completion or other processing during the training process?
Looking for your reply!
Hi, thanks for your interest!
We wished to open-source the data refinement code if possible. However, the pipeline depends on a model trained on synthetic data. In our experiments, this model was based on an older version of the codebase, which is no longer compatible with the current version. As a result, the pipeline cannot run properly even if we release the script as-is. To make the script usable, we would need to retrain the synthetic model, which is currently a low priority for us. We hope for your understanding and appreciate your patience.
The processing of sparse LiDAR depth follows the same pipeline (filtering and completion) applied to SfM datasets, producing dense depth maps that remain consistent with the filtered sparse input.
Thanks for your reply!
I still have a question. As far as I know, the percentage of lidar point cloud data in depth maps is usually less than 5%. In this case, how reliable is the completed depth map? Especially, since lidar has a limited field of view, there is usually no true value for higher areas. Have you dealt with this issue specifically?
could one not use the final model with your refinement pipeline? I understand you needed a synthetic-only model initially to produce sharp boundaries, but your final model should have inherited this trait also due to exactly this data refinement pipeline, right? For finetuning efforts, the refinement scripts would be very valuable and appreciated, even if we can only use them with the final model.
@EasternJournalist Could you explain more about the data refinement pipeline implementation details? I reproduced this pipeline based on the description in your paper using logarithmicspace Poisson completion, however the completed depth quality is poor. Could you give me some tips? Thank you very much.
@CongliangLi @jackchinor Here are some details that might be helpful:
-
Depth filtering in the first stage is critical. It's important to remove misalignments in the ground-truth depth as thoroughly as possible. Otherwise, these artifacts will directly lead to incorrect completions in the corresponding regions during the second-stage Poisson reconstruction.
-
We recommend solving the Poisson equation in a multi-resolution iterative manner. Start from a low resolution (e.g., 64×64), and use the result as initialization for the next higher resolution. This greatly speeds up convergence and improves results, especially when there is large missing area. Directly solving Poisson at high resolution tends to be slow and produces suboptimal results.
could one not use the final model with your refinement pipeline? I understand you needed a synthetic-only model initially to produce sharp boundaries, but your final model should have inherited this trait also due to exactly this data refinement pipeline, right? For finetuning efforts, the refinement scripts would be very valuable and appreciated, even if we can only use them with the final model.
@atonderski You're right. I think you've convinced me. We'll clean up and release the refinement scripts soon ahead of the synthetic-only model.
That said, please keep in mind that according to our ablation results, the final model still shows slightly reduced boundary sharpness compared to the synthetic-only model. This is a minor effect but worth noting in case you're targeting experiments where boundary precision is critical.
We also finalizing training code. And, you know, it’s a busy time with paper rebuttals. Thanks for bearing with us🤡!
@EasternJournalist Thanks for your great work! I would like to ask when will the refinement scripts be open sourced. Looking forward to your reply!
Hi @jackchinor,
I was able to successfully reproduce the Mismatch Filtering script in paper. I used MoGe v2 as the G_{syn} and filtered the sparse LiDAR depth from the NuScenes dataset, and it successfully filtered the outlier depth points, as shown in the image below.
However, I am having trouble implementing the Geometry Completion part. Would it be possible for me to reference your code for that section?
Thank you for your time and help.
Hi @jackchinor, I was able to successfully reproduce the Mismatch Filtering script in paper. I used MoGe v2 as the G_{syn} and filtered the sparse LiDAR depth from the NuScenes dataset, and it successfully filtered the outlier depth points, as shown in the image below.
However, I am having trouble implementing the Geometry Completion part. Would it be possible for me to reference your code for that section? Thank you for your time and help.
[email protected], my email
Hi @jackchinor , would you mind sharing the implementation of depth data refinement? It would help a lot for many other people who interested in this work. Thanks a lot for your time and great help!
Hi @jackchinor , would you mind sharing the implementation of depth data refinement? It would help a lot for many other people who interested in this work. Thanks a lot for your time and great help!
+1
Hi @jackchinor, I was able to successfully reproduce the Mismatch Filtering script in paper. I used MoGe v2 as the G_{syn} and filtered the sparse LiDAR depth from the NuScenes dataset, and it successfully filtered the outlier depth points, as shown in the image below.
However, I am having trouble implementing the Geometry Completion part. Would it be possible for me to reference your code for that section? Thank you for your time and help.
Hi, would you mind sharing the implementation of mismatch fitering script?
