Problem in parsing waymo dataset

[KaiLong1]

I attempted to download the Waymo dataset from the official website, but encountered the following bug while parsing the data.

Traceback (most recent call last): File "/home/longk/anaconda3/envs/detzero/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/longk/anaconda3/envs/detzero/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_preprocess.py", line 241, in create_waymo_infos( File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_preprocess.py", line 70, in create_waymo_infos waymo_infos_train = get_infos_worker( File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_preprocess.py", line 36, in get_infos_worker sequence_infos = list(tqdm(executor.map(process_single_sequence, sample_sequence_file_list), File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tqdm/std.py", line 1182, in iter for obj in iterable: File "/home/longk/anaconda3/envs/detzero/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator yield fs.pop().result() File "/home/longk/anaconda3/envs/detzero/lib/python3.8/concurrent/futures/_base.py", line 444, in result return self.__get_result() File "/home/longk/anaconda3/envs/detzero/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/home/longk/anaconda3/envs/detzero/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py", line 206, in process_single_sequence_and_save process_single_tfrecord_multiprocessing( File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py", line 250, in process_single_tfrecord_multiprocessing for cnt, data in enumerate(dataset): File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 761, in next return self._next_internal() File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 744, in _next_internal ret = gen_dataset_ops.iterator_get_next( File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2728, in iterator_get_next _ops.raise_from_not_ok_status(e, name) File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 6941, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from tensorflow.python.framework.errors_impl.NotFoundError: /mnt/disk8Tone/longkai/DetZero/data/waymo/raw_data/segment-1005081002024129653_5313_150_5333_150_with_camera_labels_with_camera_labels.tfrecord; No such file or directory [Op:IteratorGetNext]

I removed '_with_camera_labels.tfrecord' from the line sequence_file_tfrecord = sequence_file[:-9] + '_with_camera_labels.tfrecord'

[YuRui-Learning]

hello，i also meet this problem，and fix by index modify in /root/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py sequence_file_tfrecord = sequence_file[:-28] + '_with_camera_labels.tfrecord' but i find prepare data is chucked ，do u meet this problem？like 0%| | 0/24 [00:00<?, ?it/s]

[KaiLong1]

I can parse it here, but there is an issue with the parsed data. Visualization reveals that the point cloud is messy.

And the parsing speed is very slow, far slower than the parsing data of OpenPCDET.

[KaiLong1]

I have reanalyzed some waymo data, but it is still this error. But I have no problem parsing with OpenPCDET. I haven't made any changes to the code, why did this error occur?

[L00qqq]

你好，我也遇到这个问题，通过修改 /root/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py 中的索引来修复， sequence_file_tfrecord = sequence_file[:-28] + '_with_camera_labels.tfrecord' 但是我发现准备数据被卡住了，你遇到这个问题吗？比如 0%| | 0/24 [00:00<?, ?it/s]

hello，i also meet this problem，and fix by index modify in /root/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py sequence_file_tfrecord = sequence_file[:-28] + '_with_camera_labels.tfrecord' but i find prepare data is chucked ，do u meet this problem？like 0%| | 0/24 [00:00<?, ?it/s]

Hello, I also encountered the same problem, have you solved it?

[Furchtlos001]

@KaiLong1 I also encountered the same problem, have you solved it?

[superkoma]

I have encountered this issue before, could you please provide the version of your installed waymo-open-dataset-tf, such as 2-6-0 or 2-0-0?

[zw-92]

我以前遇到过这个问题，您能否提供您安装的版本，例如 或 ？waymo-open-dataset-tf``2-6-0``2-0-0

Hello, I used waymo-open-dataset-tf-2-5-0 1.4.1, but I found that there was a duplicate part of sequence_file_tfrecord = sequence_file[:-9] + '_with_camera_labels.tfrecord' in the waymo_utils.py, so I changed it to sequence_file_tfrecord= sequence_file[:-9] + '.tfrecord', but it gets stuck again in execution, log: CPU frequency: 3187200000 Hz 0%| | 0/48 [00:00<?, ?it/s], and I limited the number of processes.