Why cann't I get AP/APH=1 when submitting ground truth to validation leaderboard?

[peisun1115]

Several people have asked this. So I am just creating this issue as a FAQ.

Possible causes:

1. You did not use 1.2 ground truth. Note that if you train with 1.0 and eval on 1.2. It should be almost the same. But 1.0 does not have this populated. So you cannot do possible cause 2 below.

2. This is very likely because you did not filter boxes with 0 points as we do not classify boxes with 0 points as ground truth.

I have uploaded the ground truth as a serialized proto of open_dataset.Objects in gs://waymo_open_dataset_v_1_2_0/validation/ground_truth_objects/gt.bin . You can directly feed it to create_submission tool to create a submission. You can also compare it with your submission if you still see any problem.

You should see something like this if you directly submit that. https://waymo.com/open/challenges/entry/?timestamp=1585853480160909&challenge=DETECTION_3D&[email protected]&submissionType=VALIDATION

[xbjxh]

Can you be so kind to provide a gt.bin example for 2D detection submission as well? I have made my own to submit to the website but it returned "No submission information". I have no idea about what happened.

[peisun1115]

did you run the create_submission tool? did you point the submission_file to a valid file?

[xbjxh]

Yes, I run the C++ create_submission tool. It creates 4 files under the output dir. Then I use those two codes in the Quick Start to generate a .tar.gz file and then I submit it to the website.

tar cvf /tmp/my_model.tar /tmp/my_model/
gzip /tmp/my_model.tar

Am I doing anything wrong?

[peisun1115]

make sure --submission_file is passed properly
https://github.com/waymo-research/waymo-open-dataset/blob/master/waymo_open_dataset/metrics/tools/create_submission.cc#L22

This usually happens when this file is missing. The tool does not fail if the file is missing for now... that should be fixed.

[xbjxh]

Yes, I provide the submission.txtpb. By the way, for 2D detection, the box.center_z, box.height and box.heading should be set to 0 or I should delete them in create_prediction_file_example.py?

[ChipsSpectre]

Hello, I also would like to enquire which attributes of these should be used for 2D detection: My guess would be to only provide values for the center_x, center_y, length, and width attributes.

[peisun1115]

@xbjxh make sure the file exists. I've seen that many times. All were caused by the file passed is missing. Try to use absolute path.

I checked this commit to fail if the required info is missing (https://github.com/waymo-research/waymo-open-dataset/commit/fc98afd0ed221e6549a27c28d5ffefb696ba6e3e).

@ChipsSpectre yeah, you are right. but it is ok if you populate other stuff. they won't be used even if you populate.

[pyy799]

Can you upload the ground truth file to github? I can't get access to gs://waymo_open_dataset_v_1_2_0/validation/ground_truth_objects/gt.bin

[peisun1115]

@pyy799 The file is too big to be uploaded here.

To access, make sure:

1. You are registered in waymo/open and have access to gs://waymo_open_dataset_v_1_2_0 (try gsutil ls)
2. Make sure your gsutil is setup. Follow https://cloud.google.com/storage/docs/gsutil_install

[bilalsattar]

While trying to exclude boxes with zero points, i am getting this error AttributeError: 'Label' object has no attribute 'num_lidar_points_in_box'

I am using v1.2

[peisun1115]

@bilalsattar You can read a little more about protobuf here. For a label object a, you can do something like a.num_lidar_points_in_box

[bilalsattar]

        for obj in frame.laser_labels:
            if obj.num_lidar_points_in_box == 0:
                continue

this is my code. I don't see any problem here.

[peisun1115]

Did you install one of the 1.2 packages? https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md#use-pre-compiled-pippip3-packages

If yes, this should not happen.

[harish-kamath]

Can you be so kind to provide a gt.bin example for 2D detection submission as well? I have made my own to submit to the website but it returned "No submission information". I have no idea about what happened.

Did you ever get this? The validation ground truth still seems to be in 3D for us.

[peisun1115]

The validation ground truth for 2d is in 2d. We just store it with the same proto as 3d. we only populate center_x (along image width), center_y (along image height), length (along image width), width (along image height).

https://github.com/waymo-research/waymo-open-dataset/blob/master/waymo_open_dataset/label.proto#L23-L46

[yinjunbo]

Hello, how to evaluate on the train set? Is there a gt.bin file for the train set? @peisun1115

[SM1991CODES]

Hello,

Following your comments, if I use the gt.bin file that you have uploaded, can I use it directly with create_submission even if my predictions don't cover the entire val set? i.e., I am predicting only on a subset of the val set. Will I still get correct results if I use your gt.bin file to create a submission? Will it automatically just consider frames for which there is a prediction?

Best Regards Sambit

[ly3106]

Hello, how to evaluate on the train set? Is there a gt.bin file for the train set? @peisun1115

During my exploration across all versions of WOD, I found that gt.bin is only available in the validation set of the WOD v1.2.0 archived_files.