Using VGGT on MAD Dataset with Transparent Background – Why So Many White Points?

[sIHURs]

We used VGGT to reconstruct the MAD dataset. During image processing, we removed the white background and made it transparent. The results from training look like the image below: the points on top are visible, but there are a lot of white points elsewhere. Does anyone know why this happens? How does VGGT handle transparent backgrounds?

Would appreciate it if anyone could share your experience or discuss this!

[barwojcik]

The RGBA images are blended onto the white background in load_and_preprocess_images: https://github.com/facebookresearch/vggt/blob/main/vggt/utils/load_fn.py#L142

You can modify this function to return alpha channel as mask. Then use these masks to mask out the output point maps/unprojected depth.

[sIHURs]

The RGBA images are blended onto the white background in load_and_preprocess_images: https://github.com/facebookresearch/vggt/blob/main/vggt/utils/load_fn.py#L142

You can modify this function to return alpha channel as mask. Then use these masks to mask out the output point maps/unprojected depth.

Thanks for the suggestion.

I used the demo_colmap code after that, which only predicts the camera poses and depth maps, with those compute the point maps, and the reconstruction results turned out quite good actually.

The one I showed above was directly using the point maps predicted by VGGT, and it seems those point maps had some reconstruction errors, not just the color issue I think.