No mask directory in Mobile-stage dataset

[raven38]

I have downloaded the Mobile-Stage dataset, but I was unable to find the masks directory after extracting the data. The masks directory seems to be essential data for rendering using the pre-trained models of the Mobile-Stage dataset, and I was unable to perform rendering. Could you tell me how to obtain the mask data?

[dendenxu]

Hi, the provided dance3_libx265.tar.gz should contain 3 folders: optimized, videos_libx265 and videos_masks_libx265. The mask data is contained in videos_masks_libx265. The processing steps are listed here in the readme: https://github.com/zju3dv/4K4D?tab=readme-ov-file#rendering-with-minimal-dataset-only-encoded-videos

Specifically, the command to extract the mask images from mask videos is the last line of the codeblock linked above:

python scripts/realtime4dv/extract_images.py --data_root data/mobile_stage/dance3 --vcodec none --hwaccel none --videos_dir videos_masks_libx265 --images_dir masks_libx265 --single_channel

[raven38]

Thank you for your reply. Indeed, I followed your instructions and was able to obtain the mask. However, I could only get the first 300 frames in videos_masks_libx265, while dance3/images seems to contain 1600 frames. Can I get masks for all frames?

[dendenxu]

Hi, glad to hear that the mask extraction works!

We only uploaded the encoded video of the first 300 frames as the minimal dataset.

For downloading the full dataset, you can refer to this section in the readme.

[raven38]

Thanks! I downloaded the full dataset and extracted it. The dataset contains four directories: images, bkgd, surfs, optimized. And I couldn't find the directory for masks. Which directory contains the masks?

[dendenxu]

Ah sorry for the confusion, looks like I didn't include the mask directory when uploading.

Will upload the extracted masks separately and get back to you with a link.

In the meantime time, if you feel like extracting them yourself, we used the RobustVideoMatting method for the Mobile-Stage dataset. We detailed the segmentation process in this documentation.

[raven38]

Thank you.

[dendenxu]

Hi @raven38, I updated the dataset to include our extracted mask. You can find the updated tar.gz and a separated mask only tar.gz following the original mobile stage download link: https://docs.google.com/forms/d/e/1FAIpQLSeEbjuTV7w0lfryl-9FPX1VteuPGkbqjDvxXebY02Tm6BMejQ/viewform?usp=sf_link

Correction: we used background matting v2 instead of RVM as said in the previous comment