perception: camera_obstacle module run failed

Open hjlhhh-eng opened this issue 6 months ago • 1 comments

I have connected CARLA 0.9.14 with Apollo 8.0 using a bridge(https://github.com/guardstrikelab/carla_apollo_bridge/blob/master/docs/GettingStarted.md). I would like to disable the obstacle and traffic light information automatically provided by the bridge, so that Apollo’s perception module can obtain this information on its own.

Currently, the traffic light module has been successfully launched. However, when I try to start the obstacle detection module, an error occurs.

System information

OS Platform and Distribution (Linux Ubuntu 20.04):
Apollo installed from (source ):
Apollo version (8.0):
**carla bridge0.9.14 with apollo8.0
**NVIDIA 3060

According to the visual perception practice steps in the Apollo Open Platform documentation, the visual obstacle detection module was launched (with transform and image_depress already enabled) using the following command: mainboard -d modules/perception/production/dag/dag_streaming_perception_camera.dag. After running for a period of time, the module failed to start, and the error message was as follows:

WARNING: Logging before InitGoogleLogging() is written to STDERR I0615 13:56:48.300129 10053 module_argument.cc:81] []command: mainboard -d modules/perception/production/dag/dag_streaming_perception_camera.dag I0615 13:56:48.300311 10053 global_data.cc:153] []host ip: 10.97.154.128 I0615 13:56:48.302367 10053 module_argument.cc:57] []binary_name_ is mainboard, process_group_ is mainboard_default, has 1 dag conf I0615 13:56:48.302381 10053 module_argument.cc:60] []dag_conf: modules/perception/production/dag/dag_streaming_perception_camera.dag terminate called after throwing an instance of 'std::out_of_range' what(): map::at Aborted (core dumped)

The mainboard output is as follows:

Log file created at: 2025/06/15 13:56:48 Running on machine: in-dev-docker Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg W0615 13:56:48.302500 10053 scheduler_factory.cc:63] Scheduler conf named /apollo/cyber/conf/mainboard_default.conf not found, use default. I0615 13:56:48.304147 10055 processor.cc:42] processor_tid: 10055 I0615 13:56:48.304266 10056 processor.cc:42] processor_tid: 10056 I0615 13:56:48.304445 10057 processor.cc:42] processor_tid: 10057 I0615 13:56:48.304507 10058 processor.cc:42] processor_tid: 10058 I0615 13:56:48.304553 10059 processor.cc:42] processor_tid: 10059 I0615 13:56:48.304596 10060 processor.cc:42] processor_tid: 10060 I0615 13:56:48.304647 10061 processor.cc:42] processor_tid: 10061 I0615 13:56:48.304682 10062 processor.cc:42] processor_tid: 10062 I0615 13:56:48.304703 10063 processor.cc:42] processor_tid: 10063 I0615 13:56:48.305292 10064 processor.cc:42] processor_tid: 10064 I0615 13:56:48.305344 10065 processor.cc:42] processor_tid: 10065 I0615 13:56:48.305382 10066 processor.cc:42] processor_tid: 10066 I0615 13:56:48.305421 10067 processor.cc:42] processor_tid: 10067 I0615 13:56:48.305459 10068 processor.cc:42] processor_tid: 10068 I0615 13:56:48.305490 10069 processor.cc:42] processor_tid: 10069 I0615 13:56:48.305524 10070 processor.cc:42] processor_tid: 10070 I0615 13:56:48.305538 10053 init.cc:113] Register exit handle succ. I0615 13:56:48.306088 10053 module_controller.cc:65] Start initialize dag: /apollo/modules/perception/production/dag/dag_streaming_perception_camera.dag I0615 13:56:48.306131 10053 class_loader.cc:37] Begin LoadLibrary: /apollo/bazel-bin/modules/perception/onboard/component/libperception_component_camera.so W0615 13:56:49.978575 10053 environment.h:33] Environment variable [MODULE_PATH] not set, fallback to ....... I0615 13:56:58.582691 10060 transform_wrapper.cc:183] Get sensor2novatel extrinsics successfully. I0615 13:56:58.582840 10060 transform_wrapper.cc:211] Get pose timestamp: 1.74997e+09, pose: 0-0.00975926 0000.0697532 00000.997517 0000-189.018 000-0.999952 -0.000680771 0-0.00973548 00000159.202 -5.47954e-18 000-0.997564 0000.0697565 0000001.8735 000000000000 000000000000 000000000000 000000000001 I0615 13:56:58.583016 10060 data_provider.cc:153] Fill in GPU mode ... I0615 13:56:58.583062 10060 data_provider.cc:237] Done! (1)

I’m not sure what caused the issue. Sometimes the process stops during the inference stage, and the error message is as follows:

WARNING: Logging before InitGoogleLogging() is written to STDERR I0615 14:12:53.816545 10634 module_argument.cc:81] []command: mainboard -d modules/perception/production/dag/dag_streaming_perception_camera.dag I0615 14:12:53.816953 10634 global_data.cc:153] []host ip: 10.97.154.128 I0615 14:12:53.819531 10634 module_argument.cc:57] []binary_name_ is mainboard, process_group_ is mainboard_default, has 1 dag conf I0615 14:12:53.819547 10634 module_argument.cc:60] []dag_conf: modules/perception/production/dag/dag_streaming_perception_camera.dag terminate called after throwing an instance of 'c10::Error' what(): CUDA error: invalid argument Exception raised from launch_vectorized_kernel at /apollo/pytorch.git/aten/src/ATen/native/cuda/CUDALoops.cuh:119 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x6b (0x7f578470124b in /usr/local/libtorch_gpu/lib/libc10.so) frame #1: void at::native::gpu_kernel_impl<at::native::CompareEqFunctor >(at::TensorIterator&, at::native::CompareEqFunctor const&) + 0xab2 (0x7f5743fea7e2 in /usr/local/libtorch_gpu/lib/libtorch_cuda.so) frame #2: void at::native::gpu_kernel<at::native::CompareEqFunctor >(at::TensorIterator&, at::native::CompareEqFunctor const&) + 0x153 (0x7f5743fec313 in /usr/local/libtorch_gpu/lib/libtorch_cuda.so) frame #3: void at::native::gpu_kernel_with_scalars<at::native::CompareEqFunctor >(at::TensorIterator&, at::native::CompareEqFunctor const&) + 0x74 (0x7f5743fec4e4 in /usr/local/libtorch_gpu/lib/libtorch_cuda.so) frame #4: at::native::eq_kernel_cuda(at::TensorIterator&) + 0x104 (0x7f5743fbe8e4 in /usr/local/libtorch_gpu/lib/libtorch_cuda.so) frame #5: + 0x9e32fc (0x7f577da3c2fc in /usr/local/libtorch_gpu/lib/libtorch_cpu.so) frame #6: + 0x2c0b450 (0x7f5745679450 in /usr/local/libtorch_gpu/lib/libtorch_cuda.so) frame #7: at::Tensor& c10::Dispatcher::callWithDispatchKey<at::Tensor&, at::Tensor&, at::Tensor const&, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor& (at::Tensor&, at::Tensor const&, at::Tensor const&)> const&, c10::DispatchKey, at::Tensor&, at::Tensor const&, at::Tensor const&) const + 0xaf (0x7f577e24940f in /usr/local/libtorch_gpu/lib/libtorch_cpu.so) frame #8: + 0x9de8c8 (0x7f577da378c8 in /usr/local/libtorch_gpu/lib/libtorch_cpu.so) frame #9: at::native::eq(at::Tensor const&, at::Tensor const&) + 0x24 (0x7f577da299f4 in /usr/local/libtorch_gpu/lib/libtorch_cpu.so) ........ frame #47: + 0xc2f935 (0x7f58c02c3935 in /apollo/.cache/bazel/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/k8-fastbuild/bin/cyber/mainboard/../../_solib_local/_U_S_Scyber_Ccyber_Ucore___Ucyber/libcyber_core.so)

Aborted (core dumped)

mainboard outputs are:

Log file created at: 2025/06/15 14:12:53 Running on machine: in-dev-docker Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg W0615 14:12:53.819646 10634 scheduler_factory.cc:63] Scheduler conf named /apollo/cyber/conf/mainboard_default.conf not found, use default. I0615 14:12:53.820003 10636 processor.cc:42] processor_tid: 10636 I0615 14:12:53.820055 10637 processor.cc:42] processor_tid: 10637 I0615 14:12:53.820092 10638 processor.cc:42] processor_tid: 10638 I0615 14:12:53.820124 10639 processor.cc:42] processor_tid: 10639 I0615 14:12:53.820158 10640 processor.cc:42] processor_tid: 10640 I0615 14:12:53.820185 10641 processor.cc:42] processor_tid: 10641 I0615 14:12:53.820215 10642 processor.cc:42] processor_tid: 10642 I0615 14:12:53.820246 10643 processor.cc:42] processor_tid: 10643 I0615 14:12:53.820279 10644 processor.cc:42] processor_tid: 10644 I0615 14:12:53.820302 10645 processor.cc:42] processor_tid: 10645 I0615 14:12:53.820330 10646 processor.cc:42] processor_tid: 10646 I0615 14:12:53.820354 10647 processor.cc:42] processor_tid: 10647 I0615 14:12:53.820384 10648 processor.cc:42] processor_tid: 10648 I0615 14:12:53.820406 10649 processor.cc:42] processor_tid: 10649 I0615 14:12:53.820431 10650 processor.cc:42] processor_tid: 10650 I0615 14:12:53.820457 10651 processor.cc:42] processor_tid: 10651 I0615 14:12:53.820467 10634 init.cc:113] Register exit handle succ. I0615 14:12:53.821892 10634 module_controller.cc:65] Start initialize dag: /apollo/modules/perception/production/dag/dag_streaming_perception_camera.dag I0615 14:12:53.823156 10634 class_loader.cc:37] Begin LoadLibrary: /apollo/bazel-bin/modules/perception/onboard/component/libperception_component_camera.so W0615 14:12:56.248301 10634 environment.h:33] Environment variable [MODULE_PATH] not set, fallback to I0615 14:12:56.265229 10634 sensor_manager.cc:97] Init sensor_manager success. ................ I0615 14:13:23.063452 10647 data_provider.cc:153] Fill in GPU mode ... I0615 14:13:23.063494 10647 data_provider.cc:237] Done! (1) I0615 14:13:23.063535 10647 pipeline.cc:115] Stage: OMT_OBSTACLE_TRACKER Cost: 5.48363e-06 I0615 14:13:23.063980 10647 smoke_obstacle_detector.cc:439] Camera k matrix input to obstacle postprocessor: 0.000863533, 0, -0.828991 0, 0.000863533, -0.466308 0, 0, 1 I0615 14:13:23.063992 10647 smoke_obstacle_detector.cc:448] Start: 0.452ms I0615 14:13:23.063997 10647 data_provider.cc:292] GetImage ... I0615 14:13:23.064010 10647 data_provider.cc:319] cropping ... I0615 14:13:23.064023 10647 data_provider.cc:322] Done! I0615 14:13:23.064025 10647 smoke_obstacle_detector.cc:456] GetImageBlob: 0.033ms I0615 14:13:23.106339 10647 smoke_obstacle_detector.cc:461] Camera type: front_6mm I0615 14:13:23.106377 10647 libtorch_obstacle_detector.cc:111] BEFORE FORWARD I0615 14:13:23.106388 10647 libtorch_obstacle_detector.cc:115] 0x55f91f009390 I0615 14:13:23.106395 10647 libtorch_obstacle_detector.cc:116] 0x55f91f009f90 I0615 14:13:23.106396 10647 libtorch_obstacle_detector.cc:117] 0x55f91f009700 I0615 14:13:23.107365 10647 libtorch_obstacle_detector.cc:141] OK I0615 14:13:23.107642 10647 libtorch_obstacle_detector.cc:146] Start to do inference

I’m not sure what caused the problem. Could it be a GPU memory issue, or is there a data format problem during the inference stage? I would really appreciate your help. Thank you!

Jun 15 '25 06:06 hjlhhh-eng

May I ask if users who use a bridge to bridge between carla0.9.14 and Apollo 8.0 experience the aforementioned error when activating the obstacle detection module in the perception module.

Jun 23 '25 07:06 hjlhhh-eng