Temporal Misalignment lack of results["cam_sweeps"]

[daixinguang]

trafficstars

How is the results['cam_sweeps'] in nuscenes Dataset generated?

[kkkcx]

Thank you very much for addressing the question. To generate the results["cam_sweeps"], you will need to make the necessary modifications during the preprocessing of the nuscenes raw data. Specifically, you should add the corresponding processing code in the 3D_Corruptions_AD/TransFusion/tools/data_converter/nuscenes_converter.py file. We have already updated the nuscenes_converter.py file, and you can refer to line 177 and line 230 to preprocess the nuscenes data accordingly. This should help resolve your issue!

info = {
            'lidar_path': lidar_path,
            'token': sample['token'],
            'sweeps': [],
            'sweeps_back': [],
            'cams': dict(),
            'cam_sweeps': [],
            'lidar2ego_translation': cs_record['translation'],
            'lidar2ego_rotation': cs_record['rotation'],
            'ego2global_translation': pose_record['translation'],
            'ego2global_rotation': pose_record['rotation'],
            'timestamp': sample['timestamp'],
        }

 cam_sweep = []
        
        for cam in camera_types:
            # print(cam)
            camk = []
            while len(camk) < max_sweeps:
                cam_rec = nusc.get('sample_data', sample['data']['%s'%cam])
                if not cam_rec['prev'] == '':
                    cam_re = nusc.get('sample_data', cam_rec['prev'])
                    camk.append(cam_re)
                else:
                    break
            cam_sweep.append(camk)
        info['cam_sweeps'] = cam_sweep

[daixinguang]

OK! I know what to do. thanks.

[daixinguang]

The code you provided has a problem, you camk[0]==camk[1] ==cam_rec ['prev'] and the cam_rec not update!!!

second, The paper describes it this way:

Temporal misalignment happens when the data is stuck or delayed for a sensor. We keep the input of one modality the same as that at the previous timestamp to simulate temporal misalignment between the two modalities.

Your code replaces the current frame of both Lidar and Camera with the previous $n$ frames, but the paper means to only replace Camera or Lidar.

[daixinguang]

@kkkcx @cuge1995