Periodic latency observed as number of subscirbers increases

[jwrl7]

I have a system that runs ROS2 Galactic using CycloneDDS on a Jetson Xavier. Recently we converted a non ROS2 program that runs on the same xavier to use native cyclonedds to talk to the ROS2 ecosystem. (which is awesome)

What we have observed, is that as the number of subscribers increases to the topic published by the non ROS2 program using cycloneDDS, increases latency and experience high spikes that seems to be roughly periodic.

How we are measuring this, is creating a timer before the all the cyclone writers and then after the all the cyclone writers.

What settings or configurations can we observer to get a better understanding of what is going on? What settings or configurations can we tune to improve the performance of the native cyclonedds writers in the non ROS2 program.

Thanks!

[thijsmie]

Hi @jwrl7, I must apologise, this issue seems to have slipped everyone's attention, you should have gotten a reply much sooner. I can share some vague suspicions about what is going on, but in the meanwhile you might also have moved on from this...

• Data has to go from the publishing side to the subscribing side somehow. What does your network situation look like? Is this within a machine, or over the network? Could the increased latency that is periodic correlate with socket buffers overflowing? Or caches inside Cyclone? Both of these options are configurable, the configuration guide goes into details.
• What does your data look like? For example with pointclouds you might spend all your time memcopying, especially with multiple subscribers since the deserialisation step is repeated for every single one. Here is a similar issue.

You can run ddsperf or i11eperf to do some benchmarking and see the effects of configuration changes.

[thijsmie]

No more feedback was forthcoming. Please open a new issue if there are new insights.