[3D Camera-Only Task] Some camera labels seem to have their laser labels missing

[joonjeon]

Hi!

Provided that I correctly understood from the prototxt definitions of Waymo (Perception) Dataset,

each of the 2D object labels within a camera image has its corresponding 3D object label in laser_label.camera_synced_box, and the corresponding 3D label can be accessed by label.association.laser_object_id

Currently, I am trying to

1. parse 2D labels,
2. stack the parsed 2D labels into a Python dictionary with having label.association.laser_object_id values as key values,
3. parse 3D labels,
4. and match the parsed 2D and 3D labels together by label.association.laser_object_id of 2D label and laser_label.id of 3D label.

However, the following error occurs:

KeyError: '-2jGGMAxCaV1lmjDPO8JkQ'

Note that this is just one example, and such error seems to occur for many data frames in all TFrecord segments. The following is the output of 2D and 3D labels in one TFrecord segment of Waymo Dataset:

box {
  center_x: 1059.1763068501098
  center_y: 754.4342242125274
  width: 83.84345448135525
  length: 125.21265079380953
}
type: TYPE_VEHICLE
id: "39d7516c-b50e-44a5-9ee6-9dc2a96374bb"

box {
  center_x: 1461.49926
  center_y: 696.01218
  width: 16.421339999999987
  length: 48.632429999999886
}
type: TYPE_VEHICLE
id: "622187f4-be5a-4764-9f3b-3c9f97f74de8"
detection_difficulty_level: LEVEL_2
tracking_difficulty_level: LEVEL_2

box {
  center_x: 78.94874999999999
  center_y: 667.59063
  width: 29.053139999999985
  length: 123.16004999999998
}
type: TYPE_VEHICLE
id: "67df5b0e-a677-420a-bd3b-75c6c8748af1"
detection_difficulty_level: LEVEL_2
tracking_difficulty_level: LEVEL_2

box {
  center_x: 1100.86137
  center_y: 648.6428647457055
  width: 12.631799999999998
  length: 16.4213400000001
}
type: TYPE_VEHICLE
id: "6f9d31b6-a3f1-43f4-ab41-5c9ca689d888"
detection_difficulty_level: LEVEL_2
tracking_difficulty_level: LEVEL_2

box {
  center_x: 73.89603000000001
  center_y: 728.22327
  width: 86.52783
  length: 138.31821000000002
}
type: TYPE_VEHICLE
id: "735a792d-6beb-4ff1-845b-51a7d3c06f11"
detection_difficulty_level: LEVEL_2
tracking_difficulty_level: LEVEL_2

box {
  center_x: 1610.5545
  center_y: 778.75047
  width: 137.05503
  length: 63.159000000000106
}
type: TYPE_PEDESTRIAN
id: "7bcdff4d-738a-496f-b2be-6a019b3531e1"
detection_difficulty_level: LEVEL_2
tracking_difficulty_level: LEVEL_2
association {
  laser_object_id: "6Mk2xkcGhYmh6q2LSevCSQ"
}

box {
  center_x: 1080.9663363124541
  center_y: 662.8536947375092
  width: 13.894980000000032
  length: 21.47406000000001
}
type: TYPE_VEHICLE
id: "884505c9-d6b7-45d8-8572-72957313405c"
detection_difficulty_level: LEVEL_2
tracking_difficulty_level: LEVEL_2

...

box {
  center_x: 424.42848
  center_y: 687.80151
  width: 38.526989999999955
  length: 66.94853999999998
}
type: TYPE_VEHICLE
id: "f45d4f67-899d-4c6e-a88d-6b2bbf44cd57"
detection_difficulty_level: LEVEL_2
tracking_difficulty_level: LEVEL_2

box {
  center_x: 26.54423948311137
  center_y: -3.162608929167618
  center_z: 3.1754199828649234
  width: 0.9682625827413484
  length: 0.08732838926254353
  height: 0.9699999999999989
  heading: -3.1101825391674414
}
metadata {
  speed_x: 0.013751475001666728
  speed_y: 0.0104217199743539
  accel_x: -0.0222165197309216
  accel_y: -0.01236308683741193
  speed_z: -0.0023066128603219226
  accel_z: -0.0059212111272465805
}
type: TYPE_SIGN
id: "-2jGGMAxCaV1lmjDPO8JkQ"
num_lidar_points_in_box: 116
most_visible_camera_name: "FRONT"
camera_synced_box {
  center_x: 26.57634695090939
  center_y: -3.1633730436426446
  center_z: 3.1749770370023733
  width: 0.9682625827413484
  length: 0.08732838926254353
  height: 0.9699999999999989
  heading: -3.1101825391674414
}

As can be seen above, some camera labels seem to have their association.laser_object_id values missing (including the ones abridged in "..." marks), thereby causing the error.

Is there something wrong when parsing the 2D/3D labels and matching them correctly? Or is this an issue with Waymo toolkit or dataset?

Thanks in advance.

[hfslyc]

Hi,

Thanks for reporting this. We are currently investigating this issue. However, please note that the association between camera labels and lidar labels are only provided for Pedestrian currently (for supporting the Human Keypoint labels).

For 3D Camera-Only Detection challenge, please use frame.projected_lidar_labels for correspondence between 2D/3D for all classes. The projected_lidar_label.id is in the format of LidarId_CameraName, for example: ABCED321_FRONT_LEFT. Note that the projected lidar boxes won't be as tight as camera labels.

Best, Wayne, on behalf of the Waymo Open Dataset team