Wrong depth mask in the dataset

[zzhuolun]

Hi, Many, if not all, depth masks are wrong, as shown in the image. I only checked the car and orange category, but I believe in other categories depth masks are also wrong. I also found that the depth masks in CO3D v1 are better.

[davnov134]

Hi, can you share the code you used to create these images? The depth mask files have to be loaded with our custom loading functions.

[zzhuolun]

from omegaconf import DictConfig
from pytorch3d.implicitron.dataset.json_index_dataset_map_provider_v2 import (
    JsonIndexDatasetMapProviderV2,
)
from pytorch3d.implicitron.tools.config import expand_args_fields
import matplotlib.pyplot as plt

expand_args_fields(JsonIndexDatasetMapProviderV2)
dataset_map = JsonIndexDatasetMapProviderV2(
    category="car",
    subset_name="fewview_test",
    test_on_train=False,
    only_test_set=False,
    load_eval_batches=True,
    dataset_root=DATASET_ROOT,
    dataset_JsonIndexDataset_args=DictConfig(
        {
            "remove_empty_masks": False,
            "load_point_clouds": True,
            "dataset_root": DATASET_ROOT,
        }
    ),
).get_dataset_map()
train_dataset = dataset_map["train"]
N = 5
fig, ax = plt.subplots(nrows=4, ncols=N)
for i in range(N):
    data = train_dataset[i * 200]
    ax[0][i].imshow(data.image_rgb.permute(1, 2, 0).numpy())
    ax[1][i].imshow(data.fg_probability.permute(1, 2, 0).numpy())
    ax[2][i].imshow(data.depth_map.permute(1, 2, 0).numpy())
    ax[3][i].imshow(data.depth_mask.permute(1, 2, 0).numpy())
    for j in range(4):
        ax[j][i].axes.xaxis.set_visible(False)
        ax[j][i].axes.yaxis.set_visible(False)
plt.show()

Hi, thanks for the reply! I used the JsonIndexDataset to load the co3d dataset and the above code produces an images like this:

[davnov134]

Hi, thanks a lot for spotting this. Indeed something is wrong with the depth masks. Please bear with us while we issue a fix. Should be done by end of this week.

[Jangmin-Lee]

Hello, I want to know that progress of this issue. Is it resolved or working in progress?

[davnov134]

Hi apologies for the delay here, the a new version with correct masks is currently uploaded to the mirror. We are running last checks and should be able to release by beginning of the next week.

[yashkant]

I also found that the depth masks in CO3D v1 are better.

Hi! I was wondering how I could find few sequences with correct/better depth masks (in the meanwhile)? I am currently facing this issue of incorrect depth masks within laptop and apple categories.

Also, is there a way to patch the holes in the depth masks (0 values)? I made sure to use the correct 16-bit depth masks loading functions. My current output looks like below.

Appreciate your help!

[davnov134]

Hi, unfortunately, the depth masks are not compatible across datasets. Apologies again for the delays, re-uploading the whole dataset is taking longer than expected ...

[davnov134]

Hi, sorry for a late fix. We have now fixed the depth masks. Please re-download the dataset (you might notice that links.json and the sha256 json have changed).