[Bug] tools/create_data.py crashes with nuScenes after 50min of processing, 'nuscenes_infos_train.pkl' missing

[Huxwell]

Prerequisite

• [X] I have searched Issues and Discussions but cannot get the expected help.
• [X] I have read the FAQ documentation but cannot get the expected help.
• [ ] The bug has not been fixed in the latest version (dev) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmdetection3d

Environment

UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. warnings.warn( sys.platform: linux Python: 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0] CUDA available: True GPU 0,1,2,3: NVIDIA A100 80GB PCIe CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.7, V11.7.99 GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 PyTorch: 1.13.1+cu117 PyTorch compiling details: PyTorch built with:

• GCC 9.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.7
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
• CuDNN 8.5
• Magma 2.6.1
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.14.1+cu117 OpenCV: 4.7.0 MMCV: 1.7.0 MMCV Compiler: GCC 9.3 MMCV CUDA Compiler: 11.7 MMDetection: 2.28.2 MMSegmentation: 0.30.0 MMDetection3D: 1.0.0rc6+47285b3 spconv2.0: False

Reproduces the problem - code sample

https://github.com/open-mmlab/mmdetection3d/blob/master/tools/create_data.py

Reproduces the problem - command or script

python tools/create_data.py nuscenes --root-path /home/ubuntu/data/nuscenes-mini/ --out-dir /home/ubuntu/data/nuscenes-mini-mmdetection3d --extra-tag nuscenes

same for:

python tools/create_data.py nuscenes --root-path /home/ubuntu/data/nuscenes-mini/ --out-dir /home/ubuntu/data/nuscenes-full-mmdetection3d --extra-tag nuscenes

Contents of nuscenes-mini/ and nuscenes-full looks the same:

• samples/
• sweeps/
• v1.0-trainval/ (for nuscenes mini this is renamed from v1.0-mini/ )
• maps/

Reproduces the problem - error message

Loading NuScenes tables for version v1.0-trainval...
23 category,
8 attribute,
4 visibility,
911 instance,
12 sensor,
120 calibrated_sensor,
31206 ego_pose,
8 log,
10 scene,
404 sample,
31206 sample_data,
18538 sample_annotation,
4 map,
Done loading in 0.408 seconds.
======
Reverse indexing ...
Done reverse indexing in 0.1 seconds.
======
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 162/162, 6.2 task/s, elapsed: 26s, ETA:     0s
Create GT Database of NuScenesDataset
Traceback (most recent call last):
  File "/home/ubuntu/data/mmdetection3d/mmdet-venv/lib/python3.8/site-packages/mmcv/utils/registry.py", line 69, in build_from_cfg
    return obj_cls(**args)
  File "/home/ubuntu/data/mmdetection3d/mmdet3d/datasets/nuscenes_dataset.py", line 131, in __init__
    super().__init__(
  File "/home/ubuntu/data/mmdetection3d/mmdet3d/datasets/custom_3d.py", line 88, in __init__
    self.data_infos = self.load_annotations(open(local_path, 'rb'))
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/data/nuscenes-mini-mmdetection3d/nuscenes_infos_train.pkl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tools/create_data.py", line 250, in <module>
    nuscenes_data_prep(
  File "tools/create_data.py", line 89, in nuscenes_data_prep
    create_groundtruth_database(dataset_name, root_path, info_prefix,
  File "/home/ubuntu/data/mmdetection3d/tools/data_converter/create_gt_database.py", line 219, in create_groundtruth_database
    dataset = build_dataset(dataset_cfg)
  File "/home/ubuntu/data/mmdetection3d/mmdet3d/datasets/builder.py", line 44, in build_dataset
    dataset = build_from_cfg(cfg, DATASETS, default_args)
  File "/home/ubuntu/data/mmdetection3d/mmdet-venv/lib/python3.8/site-packages/mmcv/utils/registry.py", line 72, in build_from_cfg
    raise type(e)(f'{obj_cls.__name__}: {e}')
FileNotFoundError: NuScenesDataset: [Errno 2] No such file or directory: '/home/ubuntu/data/nuscenes-mini-mmdetection3d/nuscenes_infos_train.pkl'

Additional information

1. Expected result: successfully running nuScenes submission of mmdetection3D as demo visualization on nuScenes data and successfully reproducing eval scores, I presume this is the correct model: https://github.com/open-mmlab/mmdetection3d/tree/master/configs/free_anchor
2. For this purpose I assume preparing the dataset to mmdetection3d format is required, but it crushes for me (both with full and mini versions of nuScenes)
3. The script processes for over 50minutes before it crashes. If my setup (paths or smth) is wrong, you might like to consider checking if correct dirs exist etc in the beginning of the script.

[Huxwell]

For the mini version of nuScenes :

• copying test data from full nuScenes
• adding "--version v1.0-mini"
• trick of having the same input and output dir (or manually copying files from input to output after crash)

Seems to help (the script finalizes).

[hbc2020]

Hi, When I ran the command "python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes", I met the same problem, no *.pkl files had been generated, could you please tell me how did you solve this problem? Thank you very much!

[hankeceli]

Hi, When I ran the command "python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes", I met the same problem, no *.pkl files had been generated, could you please tell me how did you solve this problem? Thank you very much!

You have the option to manually download the .pkl files made available by the author. https://github.com/open-mmlab/mmdetection3d/issues/2454#issuecomment-1517743436

[anithselva]

For the mini version of nuScenes :

• copying test data from full nuScenes
• adding "--version v1.0-mini"
• trick of having the same input and output dir (or manually copying files from input to output after crash)

Seems to help (the script finalizes).

I tried this with no luck. Has there been any update or fix for this issue?