microsoft
Many thanks for this great repo
Hello,
I just want to say thanks for this great repo! The ERP (equirectangular images) looks great in point cloud reprojection. Can't believe my eyes!
A question - if I have a set of images - how do I align their point cloud scale in some way? I notice that each generated depth image from MoGe seems to scale each point cloud differently. If I take a set of ERP images in the same room, the room dimensions have different sizes.
Here is the same room, side by side (both are downsampled to avoid large point clouds btw),
Hi! I'm glad to hear it was helpful.
Regarding the scale consistency across multiple inferences, a simple solution is to use MoGe-2, which supports metric-scale prediction. It should provide consistent scales, especially for indoor scenes.
Currently, the infer_panorama script hasn’t been updated since the release of MoGe-2 and still uses the MoGe-1 model. I’ll update it soon. In the meantime, if you're able to, you could try modifying the script to use MoGe-2.
Hello,
I did actually change the panorama script use the v2 model, but the issue was the same unfortunately. I am also interefencing a single image at a time. Or do you mean I can bundle several images together in one go to provide a better scaled depth?
Although the metric itself is good, the scaling can be quite different per image that is depth estimated.
Ah, I just noticed that the current script for merging panorama depth maps doesn't account for metric scale. It ignores input metric scales. The merged panorama depth needs to be aligned with the scales of the individual split-view depth maps to produce a correct metric panorama depth. The script needs delicate updates to support it.
That said, you're right—some inconsistency in metric scale across views is expected, since each view is inferred independently. While metric prediction helps reduce this issue, it can't eliminate the issue entirely. To further improve consistency, optimizations like bundle adjustment may be necessary. Alternatively, you might consider trying VGGT (link), which is designed for processing multi-view images jointly.
Hi authors
Thank you so much for the great publication. I am waiting for the MoGe-2 paper release!
@abrahamezzeddine could you share a example how you have done this?
Could you please elaborate what you mean?
How I projected the point cloud or how I changed to use MoGe v2?
how you projected the point cloud from equirectangular images
There is a script already in moge/scripts/infer_panorama.py
Ah, I just noticed that the current script for merging panorama depth maps doesn't account for metric scale. It ignores input metric scales. The merged panorama depth needs to be aligned with the scales of the individual split-view depth maps to produce a correct metric panorama depth. The script needs delicate updates to support it.
That said, you're right—some inconsistency in metric scale across views is expected, since each view is inferred independently. While metric prediction helps reduce this issue, it can't eliminate the issue entirely. To further improve consistency, optimizations like bundle adjustment may be necessary. Alternatively, you might consider trying VGGT (link), which is designed for processing multi-view images jointly.
Is there any plans to update this script to account for the individual image scales before merging into panorama?
@EasternJournalist EDIT: I just tried to check out the different scales, and since we have overlaps during panorama splits, it should be relatively easy to calculate the scales between the successive overlaps when depth has been interferenced... I noticed the following (selecting an arbitrary split view)
[Scale] Using tile 2 as reference (most forward-facing camera) [Scale] Individual tile scales: [Scale] Tile 0: 1.1207 [Scale] Tile 1: 1.4123 [Scale] Tile 2: 1.0000 (REF) [Scale] Tile 3: 0.9531 [Scale] Tile 4: 1.1035 [Scale] Tile 5: 1.1085 [Scale] Tile 6: 1.1041 [Scale] Tile 7: 1.0889 [Scale] Tile 8: 1.1013 [Scale] Tile 9: 0.9935 [Scale] Tile 10: 1.1494 [Scale] Tile 11: 0.9868 [Scale] 0.953 – 1.412
There seems to be quite the discrepancy which we must account for to handle a global scale. Also managed to implement normal into 360 via MoGe-2 successfully;
I also tried to adjust the edge removal threshold but seems to have no effect. Have you got it to work?
Hi @abrahamezzeddine, Thanks for the helpful info! Could you please share the modified code you used for MoGe-2 in infer_panorama.py, especially the part with tile scale alignment? Would be super useful — thanks!
Hi @abrahamezzeddine, Thanks for the helpful info! Could you please share the modified code you used for MoGe-2 in infer_panorama.py, especially the part with tile scale alignment? Would be super useful — thanks!
Hello,
I still have not managed to fix the scale, because the scale is always relative to its own estimation. That is, it’s not easy to determine what is the correct ”reference”-scale for each interference.
Once I get it to work, I can share it. But as of now, it’s not consistent. I just wanted to see given a arbitrary tile (1 of the 12 split images per panorama) how much the overlapping scales differentiate between them.
Hi @abrahamezzeddine,
Would you mind sharing the code you used to merge normals into the panorama using MoGe v2? It would be really helpful.
Thank you in advance!
