HOPE-Video in BOP format

[nv-nguyen]

Hi Stephen @swtyree,

For BOP'24, we would like to add more testing instances for HOPE dataset. One simple approach is to include instances from HOPE-Video, which could be useful for multi-view or tracking settings in the future.

Do you have HOPE-Video in BOP format so that we can proceed with this? (I downloaded and found that HOPE-Video is not in BOP format for now.)

cc @thodan, @TontonTremblay

Thanks, Nguyen

[swtyree]

Hi @nv-nguyen, I do have some annotations that another researcher generated. See https://github.com/swtyree/hope-dataset/issues/2#issuecomment-1183971040 (and https://github.com/swtyree/hope-dataset/issues/2#issuecomment-1924249689 where the masks have been improved).

However, the quality of HOPE-Video is much lower than HOPE-Image, both in terms of image quality/resolution and annotation accuracy. If this is important, we might want to discuss capturing a few more scenes and annotating them using the HANDAL toolkit.

[AnasIbrahim]

@swtyree if you are interested in an extension for hope, I can help.

I collected a few scenes with HOPE objects in the BOP format a few months ago. The collection process I use is mostly automated with a VICON motion capture, manual annotations need to be done once per object in a scene using an annotation tool I wrote before. I wrote this for a collection of a dataset for mobile robots but I was testing with HOPE objects at the beginning, so I have these 2 samples. You can also find in the "vis" folder the projection of the masks on the objects generated with the BOP toolkit (the mask quality is pretty good).

You can download the samples here: https://drive.google.com/file/d/1NOVEc7f4jI4QWOw_BA-qYYA30TdwFLUU/view?usp=sharing

Here is a wide preview of what the 2 scenes look like:

[nv-nguyen]

Hi @AnasIbrahim,

Thank you so much for sharing! We would like to use them as an extension as an extension for HOPE, but we found the annotation quality is not great as HOPE-Image. Here is the visualisation with bop-toolkit.

Could you please improve them (for example, by using HANDAL's pipeline) ?

Thanks, Nguyen

[AnasIbrahim]

Hi @nv-nguyen,

It is possible to improve the quality. This was just a scene for development that I never intended to clean up totally.

There are things that I already did for this other dataset that I wrote this collection process:

1- reduce the frequency of the VICON motion capture. When I collected the samples it was set at 200 HZ. Which is absolutely not needed and reduces the accuracy. Also, I was blocking some of the motion capture while collecting. 1- the manual annotation: we did it as fast as possible and had no revision. You see some errors are persisting because of that. 2- produce a concatenated depth from all depth images and use it during annotation. This would fill the holes in the depth images. These concatenated depth images would only be used for the data generation and wouldn't be provided in the data. 3- run a colored ICP on the individual scenes with aggressive constraints (1 mm constraints): this is only possible because the Zivid2 camera we use has a super good quality, if a depth point is there it is exactly true.

You can find some of these scripts on my fork of the BOP toolkit. I didn't merge these in the original BOP repo because some of these changes are not that relevant.

if you want those 2 scenes then I can do just a manual annotation revision and run the ColoredICP on the individual scenes.

If you need more scenes that is also possible. How many scenes (assume 150 frames/scene)? what should the background be? I only have one set of the HOPE objects so no object can be repeated.

[nv-nguyen]

Hi @AnasIbrahim,

Thank you for the information! We truly appreciate your efforts to improve and share the dataset.

Our top priority is to improve the annotation quality of the two scenes shared above. You can use this script in the BOP toolkit to verify the quality.

Additionally, if you could capture a few more scenes, it would be highly appreciated. Currently, there are 1314 instances in total for the two scenes you provided. We aim to have 2500-3000 instances, so capturing at least 2-3 more scenes (or even more) would be great. Since we always use this script to subsample and create a test list, having more instances will contribute to increased diversity in terms of object poses and visible fractions.

Thank you once again, and please keep us posted about your next steps. Nguyen