How to convert BOP Format to a format that is acceptable by PVNET (custom data format)?

[monajalal]

trafficstars

Describe the issue

I made a dataset of my own object of interest using similar script as main_lm_upright.py. However, it doens't create poses in same format as needed for pvnet.

I have these:

(clean-pvnet) mona@ada:~/BlenderProc/examples/datasets/bop_challenge/output_data/bop_data/lm/train_pbr/000000$ ls
total 2.4M
drwxrwxr-x 3 mona mona 4.0K Jan  9 13:13 ..
drwxrwxr-x 6 mona mona 4.0K Jan  9 13:14 .
drwxrwxr-x 2 mona mona  12K Jan  9 13:29 rgb
drwxrwxr-x 2 mona mona  12K Jan  9 13:29 depth
drwxrwxr-x 2 mona mona  12K Jan  9 13:29 mask
drwxrwxr-x 2 mona mona  12K Jan  9 13:29 mask_visib
-rw-rw-r-- 1 mona mona  80K Jan  9 13:29 scene_gt.json
-rw-rw-r-- 1 mona mona 111K Jan  9 13:29 scene_camera.json
-rw-rw-r-- 1 mona mona  51K Jan  9 13:29 scene_gt_info.json
-rw-rw-r-- 1 mona mona 2.1M Jan  9 13:29 scene_gt_coco.json

and I need to have something like:

$ ls
total 11M
drwxrwxrwx 5 mona mona 4.0K Jan  9 09:07 .
drwxrwxrwx 2 mona mona 1.1M Jan  9 09:48 rgb
drwxrwxrwx 2 mona mona 1.1M Jan  9 09:49 mask
drwxrwxrwx 2 mona mona 1.4M Jan  9 09:49 pose
drwxrwxrwx 7 mona mona 4.0K Jan  9 13:40 ..
-rwxrwxrwx 1 mona mona   94 Jan  9 09:07 camera.txt
-rwxrwxrwx 1 mona mona 6.8M Jan  9 09:07 model.ply
-rwxrwxrwx 1 mona mona   20 Jan  9 09:07 diameter.txt

where:

$ ls pose/  | head -10
total 167M
drwxrwxrwx 5 mona mona 4.0K Jan  9 09:07 ..
drwxrwxrwx 2 mona mona 1.4M Jan  9 09:49 .
-rwxrwxrwx 1 mona mona  224 Jan  9 09:07 pose9720.npy
-rwxrwxrwx 1 mona mona  224 Jan  9 09:07 pose9712.npy
-rwxrwxrwx 1 mona mona  224 Jan  9 09:07 pose9497.npy
-rwxrwxrwx 1 mona mona  224 Jan  9 09:07 pose8845.npy
-rwxrwxrwx 1 mona mona  224 Jan  9 09:07 pose883.npy
-rwxrwxrwx 1 mona mona  224 Jan  9 09:07 pose8755.npy
-rwxrwxrwx 1 mona mona  224 Jan  9 09:07 pose8699.npy

Could you please guide how I can achieve it?

Currently, with what I have created using BlenderProc, I have :

(blenderproc) mona@ada:~/BlenderProc/examples/datasets/bop_challenge/output_data/bop_data/lm/train_pbr/000000$ cat scene_gt.json 

"269": [{"cam_R_m2c": [0.8593733906745911, 0.03154050186276436, 0.5103749632835388, 0.1305074691772461, -0.9785698652267456, -0.15927515923976898, 0.49441400170326233, 0.20348457992076874, -0.8450731635093689], "cam_t_m2c": [-27.049041748046875, 8.441537857055664, 3606.619140625], "obj_id": 1}]

and

(blenderproc) mona@ada:~/BlenderProc/examples/datasets/bop_challenge/output_data/bop_data/lm/train_pbr/000000$ cat scene_gt_info.json 


  "269": [{"bbox_obj": [234, 121, 160, 218], "bbox_visib": [234, 121, 160, 217], "px_count_all": 23369, "px_count_valid": 23369, "px_count_visib": 22977, "visib_fract": 0.9832256408061962}]

I don't see any pose information at all.

Below is the format that is required for custom dataset training by clean-pvnet repo: https://github.com/zju3dv/clean-pvnet

Minimal code example

No response

Files required to run the code

No response

Expected behavior

Be able to pass the data created by main_lm_upright.py that is in BOP format to clean-pvnet for custom dataset training.

BlenderProc version

commit f34774b9d835b6dada02d7ab79ab33c542f87cf3

[cornerfarmer]

Hey @monajalal,

in the bop format, the object poses are stored in scene_gt.json, see https://github.com/thodan/bop_toolkit/blob/master/docs/bop_datasets_format.md#ground-truth-annotations. They are stored in the camera frame. So if you need them in the world frame, you would need to transform them first.

Alternatively, you could also store the object poses separately to the bop writer in a custom file, similar to how its done here: https://github.com/DLR-RM/BlenderProc/blob/main/examples/basics/light_sampling/main.py#L43