autoware.universe
autoware.universe copied to clipboard
autowarefoundation
Image messages have latency or dropped
Checklist
- [X] I've read the contribution guidelines.
- [X] I've searched other issues and no duplicate issues were found.
- [X] I'm convinced that this is not my fault but a bug.
Description
We are using the CycloneDDS configuration provided by @takam5f2 in this discussion with tuned DDS advised by the ROS2 page. Without running Autoware.Universe, we are getting 20 fps with 3 cameras without any problem. But if we run the entire Autoware.Universe pipeline, image messages drop to ~3,4 FPS. When we reduce the resolution of the images to around 600x500 and fix the fps to 15, there is no frame loss, but the latency in the topic is around ~70,80 ms. Is there anyone have any advice or solution?
CycloneDDS supports shared memory with ROS2 Humble. is there any plan to convert autoware messages to a fixed size for using zero-copy DDS operation with shared memory.
Expected behavior
Image message topics must arrive with minimal latency.
Actual behavior
Image messages are being dropped or coming with latency
Steps to reproduce
- Run Autoware with a 2D object detection pipeline.
Versions
No response
Possible causes
No response
Additional context
No response
Have you made a test with Multiple high channel Lidars e.g. 1 x 128 channel + 5 x 16 channel lidar without the cameras? Is it possible that you might have the same problem or do you think this is only related to cameras?
Can you try to run image related nodes as Composable Nodes ?
For visualizing, I think rviz2 is not good for camera images.
And I am not sure if rqt works as a Composable node. If it doesn't work as one, you might need to write your own image visualizing node for it.
Possible relevant discussions:
- https://github.com/orgs/autowarefoundation/discussions/2617
- https://github.com/orgs/autowarefoundation/discussions/2621
- https://github.com/orgs/autowarefoundation/discussions/2555
Also, as @xmfcx mentioned, if we put all image processing nodes in a single component, we should be able to reduce the image latency.
It seems all present messages were recorded instead of just the sensor messages. I will extract the sensor messages using this tool https://github.com/Kettenhoax/ros2bag_tools and upload the bag file.
The shared bag files are/will be available in autoware documentation through this pr: https://github.com/autowarefoundation/autoware-documentation/pull/200
Note: the bag file only contains velodyne_packets so needs conversion to pointcloud.
Sensor configuration: https://autowarefoundation.github.io/autoware-documentation/main/datasets/#leo-drive-isuzu-sensor-data Front lidar is VLP16
@xmfcx could you share the link to sensor description repository for your vehicle?
https://github.com/vautonomous/isuzu_sensor_description @xmfcx also add this to the documentation.
From the discussion from LeoDrive's update in ASWG, the latency is likely to be caused by CPU usage. We might not be able to fix this within Bus ODD timeline.
Using Iceoryx zero copy improved the message transportation, but using Iceoryx as AWF projects needs further discussion.
In the relevant PRs, I passed the same pointcloud_container to Velodyne nodes and missed the occupancy grid map package. When working with sample_sensor_kit and logging_simulator.launch.xml/autoware.lauch.xml, it decreased network traffic by around -%30 and also CPU utilization decreased.
I will test it with a real Velodyne devices, making sure multiple Velodyne drivers work without any problem in the same container with a different namespace.
Once
- https://github.com/autowarefoundation/autoware.universe/pull/2114
- https://github.com/autowarefoundation/sample_sensor_kit_launch/pull/48
are merged, we can close this issue.
Once
* [feat(tier4_localization_launch): pass pc container to localization #2114](https://github.com/autowarefoundation/autoware.universe/pull/2114) * [feat(sample_sensor_kit_launch): pass container to velodyne nodes sample_sensor_kit_launch#48](https://github.com/autowarefoundation/sample_sensor_kit_launch/pull/48)are merged, we can close this issue.
PRs are ongoing, milestone updated.
