How to create /saved_detections/ycbv_posecnn.pkl for own dataset

[LucZed]

Hello, I want to train and run cosypose on my own custom dataset. Till now following scripts work fine:

• run_detector_training.py
• run_detection_eval

Now i want to train the the "run_pose_training.py" script. For now, the whole script finish without exceptions. To come to the Problem: In the /local_data/experiments directory the last run showing up, but the "errors_(ds_name).txt is empty. I traced my problem back to the function "train_pose.py "-> "make_eval_bundle". There i use "load_posecnn_results" to get detections from the "yxbv_posecnn.pkl".

• First: This file has like 15k gt-data for the YCBV-dataset
• Second: There is a mismatch between this file and my own dataset (kind of obvious)
• Third: For me it looks like, that it is necessary later in (train_pose.py --- run_eval) -> (runner_utils.py --- run_pred_eval) -> (multiview_predictions.py --- get_predictions), that both, my data and the .pkl has to match, so the detection list in line 120 wont be empty.

There are already closed issues like #65 or #57 but these got closed with no answer.

How do i create my own ycbv_posecnn.pkl, which match my dataset? Is there an other, easier way to get eval results?

If you need more details about my issue, i will provide more information

Thanks :)

[LucZed]

Hello again, I still were not able to create my own .pkl file which is like the "ycbv_posecnn.pkl" but i found a workaround. (For now i don't know what impact my workaround have in the eval results)

Preparation:

• Recreate the internal data format for the detection, which got read in from the .pkl file.
• Below i copied the "detection"-format from: "cosypose -> scripts -> run_cosypose_eval.py -> load_posecnn_results()"

data = tc.PandasTensorCollection(
        infos=pd.DataFrame(infos),
        poses=torch.as_tensor(np.stack(poses)).float(),
        bboxes=torch.as_tensor(np.stack(bboxes)).float(),
).cpu()

• The final format should look like this:

PandasTensorCollection(
    poses: torch.Size([15435, 4, 4]) torch.float32 cpu,
    bboxes: torch.Size([15435, 4]) torch.float32 cpu,
----------------------------------------
    infos:
       scene_id  view_id  score       label
0            48        1    1.0  obj_000001
1            48        1   20.0  obj_000020
2            48        1   14.0  obj_000014
3            48        1   19.0  obj_000019
4            48        1    6.0  obj_000006
...         ...      ...    ...         ...
15430        59     1857   15.0  obj_000015
15431        59     1857    6.0  obj_000006
15432        59     1857    9.0  obj_000009
15433        59     1857    4.0  obj_000004
15434        59     1857   18.0  obj_000018

[15435 rows x 4 columns]
)

Create detection from BOP's scene_gt_info / scene_gt

• This was just a test to know if my workaround would work:

def own_detection_from_gt():
    import os
    import cosypose.utils.tensor_collection as tc
    import pandas as pd
    path_data_dir = "PATH/TO/REPO/cosypose/local_data/bop_datasets/DATASET_NAME/"
    path_scene_dir = os.path.join(path_data_dir, "train")
    scene_names = os.listdir(path_scene_dir)
    infos, poses, bboxes = [], [], []
    for scene_id, scene_name in enumerate(scene_names):
        path_scene_gt_info = os.path.join(path_scene_dir, scene_name, "scene_gt_info.json")
        path_scene_gt = os.path.join(path_scene_dir, scene_name, "scene_gt.json")
        with open(path_scene_gt_info, "r") as f:
            json_data_gt_info = json.load(f)
        with open(path_scene_gt, "r") as f:
            json_data_gt = json.load(f)
        img_names_rgb = os.listdir(os.path.join(path_scene_dir, scene_name, "rgb"))
        for img_id, img_name in enumerate(img_names_rgb):
            list_bbox = json_data_gt_info[f"{img_id}"][0]["bbox_obj"]
            xmin = list_bbox[0]
            ymin = list_bbox[1]
            xmax = list_bbox[0] + list_bbox[2]
            ymax = list_bbox[1] + list_bbox[3]
            list_bbox = [xmin, ymin, xmax, ymax]
            list_rot  = json_data_gt[f"{img_id}"][0]["cam_R_m2c"]
            list_loc  = json_data_gt[f"{img_id}"][0]["cam_t_m2c"]
            row0 = [list_rot[0], list_rot[1], list_rot[2], list_loc[0]] 
            row1 = [list_rot[3], list_rot[4], list_rot[5], list_loc[1]]
            row2 = [list_rot[6], list_rot[7], list_rot[8], list_loc[2]]
            row3 = [0, 0, 0, 1]
            rot_loc_mat = [row0, row1, row2, row3]
            infos.append(dict(
                    scene_id=scene_id,
                    view_id=img_id,
                    score=1,
                    label="obj_000001",
                ))
            poses.append(rot_loc_mat)
            bboxes.append(list_bbox)
    data = tc.PandasTensorCollection(
        infos=pd.DataFrame(infos),
        poses=torch.as_tensor(np.stack(poses)).float(),
        bboxes=torch.as_tensor(np.stack(bboxes)).float(),
    ).cpu()
    return data

• Since i have only one object, the label is always "obj_000001"
• Because of the GT-Data the score is always "1"

Other Changes

Changes in "run_cosypose_eval.py":

• Line 160: Add something like this for your own Dataset:

elif 'OWN_DS_NAME' in ds_name:
        compute_add = True
        visib_gt_min = -1
        targets_filename = None
        n_top = 1
        spheres_overlap_check = False

• Line 218: Add something like this for your own Dataset: (check "match_threshold")

if 'OWN_DS_NAME' in ds_name:
            meters[f'{error_type}_ntop=1_matching=CLASS'] = PoseErrorMeter(
                error_type=error_type, consider_all_predictions=False,
                match_threshold=np.inf, 
                report_error_stats=False, report_error_AUC=True, **base_kwargs

• Line 300: Add something like this for your own Dataset:

elif 'OWN_DS_NAME' in args.config:
        object_set = 'OWN_DS_NAME'
        refiner_run_id = 'OWN_DS_NAME--12345'
        n_coarse_iterations = 0
        n_refiner_iterations = 2

• Line 310: Add something like this for your own Dataset:

elif args.config == 'OWN_DS_NAME':
        ds_name = 'OWN_DATASET_NAME'

• Line 372: Add something like this for your own Dataset:

elif "OWN_DS_NAME" in ds_name:
        OWN_DS_detections = own_detection_from_gt()
        pred_kwargs = {
            'OWN_DS_NAME_GT': dict(
                detections=OWN_DS_detections,
                use_detections_TCO=OWN_DS_detections,
                **base_pred_kwargs
            ),
        }

• Line 405: Add something like this for your own Dataset:

elif "OWN_DS_NAME" in ds_name:
        det_key = 'OWN_DS_NAME_GT'
        all_predictions['OWN_DS_NAME_GT'] = OWN_DS_NAME_gt
        predictions_to_evaluate.add('OWN_DS_NAME_GT')

• Line 462: Add something like this for your own Dataset:

elif 'OWN_DS_NAME' in ds_name:
        metrics_to_print.update({
            f'posecnn/ADD(-S)_ntop=1_matching=CLASS/AUC/objects/mean': f'PoseCNN/AUC of ADD(-S)',

            f'{det_key}/refiner/iteration={n_refiner_iterations}/ADD(-S)_ntop=1_matching=CLASS/AUC/objects/mean': f'Singleview/AUC of ADD(-S)',
            f'{det_key}/refiner/iteration={n_refiner_iterations}/ADD-S_ntop=1_matching=CLASS/AUC/objects/mean': f'Singleview/AUC of ADD-S',

            f'{det_key}/ba_output+all_cand/ADD(-S)_ntop=1_matching=CLASS/AUC/objects/mean': f'Multiview (n={args.n_views})/AUC of ADD(-S)',
            f'{det_key}/ba_output+all_cand/ADD-S_ntop=1_matching=CLASS/AUC/objects/mean': f'Multiview (n={args.n_views})/AUC of ADD-S',
        })

Additional Information

• There are more checks for the DS_NAME. Just copy everything that is necessary form ycbv or tless.
• I just tried to make everything work. If i made any mistakes or my code makes no sense, feel free to give me a hint :)
• If you get NaN Results in the summary.txt / full_summary.txt, maybe the threshold is to strict or your training is not good enough.
• in the full_summary.txt you will find the section "OWN_DS_NAME/refiner/iteration=2" (or something similar) which should be the score for the checkpoint you chose to evaluate. (But I am not 100% sure)

[tensarflow]

Hi there, thanks for sharing your results! How did you create your dataset in the first place? This is the first time I come across PKL files and I have no clue :)

[LucZed]

I used:

• Blenderproc: https://github.com/DLR-RM/BlenderProc
• BopToolkit: https://github.com/thodan/bop_toolkit

I wrote my own rendering pipeline based on blenderproc and used my own 3D-Model (.obj). In blenderproc there is a method called bproc.writer.write_bop(), which save rendered data in bop format.

To get the necessary scene_gt_info.json for own Dataset (BOP-Format) i used BopToolkit. I used bop_toolkit/scripts/calc_gt_info.py. I made several changes to the code. But most of the time you just have to add "if"-statements which contain dataset specific variables.

if you dont have a .ply 3D-Model than you have to convert it to that format using Meshlab.

till now i dont know how to create that .pkl file. i still use my workarround. If you want to open and create Pickel (.pkl) Files. Use Pandas. pd.read_pickle()

if you need more information i try my best to give proper answers.

There could be better ways to do it. If you find one, let me know :)

[tensarflow]

Thanks a lot for the guidance! I will have a look at those and get back to you.