Add computation of size contribution to info verb

[nicolaloi]

Addresses #1601 feature request (Show size contribution of each topic with ros2 bag info). This is a test draft.

Output of ros2 bag info test_bag -v --size-contribution:

Files:             test_bag.mcap
Bag size:          2.6 GiB
Storage id:        mcap
ROS Distro:        unknown
Duration:          39.541s
Start:             Jun 19 2024 23:40:38.345 (1718833238.345)
End:               Jun 19 2024 23:41:17.886 (1718833277.886)
Messages:          2310
Topic information: Topic: /parameter_events | Type: rcl_interfaces/msg/ParameterEvent | Count: 0 | Serialization Format: cdr
                   Topic: /velodyne_points | Type: sensor_msgs/msg/PointCloud2 | Count: 765 | Size Contribution: 557.0 MiB | Serialization Format: cdr
                   Topic: /events/read_split | Type: rosbag2_interfaces/msg/ReadSplitEvent | Count: 0 | Serialization Format: cdr
                   Topic: /image_raw_1 | Type: sensor_msgs/msg/Image | Count: 763 | Size Contribution: 1.0 GiB | Serialization Format: cdr
                   Topic: /image_raw_2 | Type: sensor_msgs/msg/Image | Count: 763 | Size Contribution: 1.0 GiB | Serialization Format: cdr
                   Topic: /events/write_split | Type: rosbag2_interfaces/msg/WriteSplitEvent | Count: 0 | Serialization Format: cdr
                   Topic: /rosout | Type: rcl_interfaces/msg/Log | Count: 19 | Size Contribution: 4.9 KiB | Serialization Format: cdr
Service:           0
Service information:

The total size contribution of each topic is computed by sequentially reading every message in the rosbag. As mentioned in #1601, this approach can be slow.

In a first test A (previous output), the computation of the size contribution took ~520ms for a bag with 2310 messages, 2.6GiB large, 39.5s long, with two sensor_msgs/msg/Image topics with 763 messages each, and one sensor_msgs/msg/PointCloud2 topic with 765 messages.

In a second test B, the size computation took ~78ms for a bag with 14577 messages, 5.2MiB large, 74.4s long, with one sensor_msgs/msg/Imu topic with 14559 messages.

So to me, it seems the computation time is not simply dependent on the number of messages, but on their types and individual sizes too. As a test, I was curious about skipping the message serialization step inside the reader. I tried a brute modification of the mcap_storage interface to directly access the messageView.message.dataSize variable while skipping the message serialization step. However, this only resulted in a 15-30% improvement in computation time.

As a test, removing the actual size computation code and just reading the rosbag in an empty while loop doesn't really change the timing (only real difference was in test B, 78ms->68ms).

In rosbag2_py/_info.cpp, the new function compute_topics_size_contribution can be combined with the existing read_service_info to avoid reading the rosbag twice.

@MichaelOrlov looking forward to your feedback and suggested changes.

[nicolaloi]

Addressed review comments, added size contribution for services, and updated tests and design doc.

[MichaelOrlov]

@ros-pull-request-builder retest this please

[nicolaloi]

I will edit the file with the style divergence error. I have missed it since locally the tests no longer gave me any errors.

[MichaelOrlov]

@nicolaloi The Rpr job failed because need to make rebase. Please rebase your branch on top of latest rolling branch.

[nicolaloi]

@MichaelOrlov I have rebased the branch and I have updated the new size contribution test I have added to match the latest changes in rolling, but now the Rpr test is failing due to an unrelated rcpputils::fs issue, as in the latest rolling merged PR #1740 (log L32773)

[MichaelOrlov]

@nicolaloi No worries. Those test failures are unrelated to the changes from this PR and test failing because the CI job from GitHub actions uses binaries for the other dependent packages from the latest release on rolling.
The test failure will go away when we will release a new version of the rcpputils package and do rolling synch.

Anyway, I will run CI jobs on the ROS 2 build farm, which compiles all packages from sources. If everything is correct, all tests should pass.

[MichaelOrlov]

Pulls: ros2/rosbag2#1726 Gist: https://gist.githubusercontent.com/MichaelOrlov/5179a9d2b5bc8685c059758e5d9397de/raw/af0dc6b9cdc1954a535c28a5eeb3e5f43c031848/ros2.repos BUILD args: --packages-above-and-dependencies ros2bag rosbag2_cpp rosbag2_py rosbag2_tests TEST args: --packages-above ros2bag rosbag2_cpp rosbag2_py rosbag2_tests ROS Distro: rolling Job: ci_launcher ci_launcher ran: https://ci.ros2.org/job/ci_launcher/14281

• Linux
• Linux-aarch64
• Linux-rhel
• Windows

[MichaelOrlov]

https://github.com/Mergifyio backport jazzy

[mergify[bot]]

backport jazzy

✅ Backports have been created

• #1872 Add computation of size contribution to info verb (backport #1726) has been created for branch jazzy

[jdcast]

Will this be backported to Humble?

[MichaelOrlov]

It is difficult to backport since the code base between Jazzy and Humble is significantly different.
The major difference is the feature for recording and playback services. I have tried to create an automatic backport to humble, but there are a bunch of conflicts and missing files in delta. Please see comment https://github.com/ros2/rosbag2/pull/1899#issuecomment-2603463783. I personally don't have the capacity and resources to address all those merge conflicts and make all adjustments.

If someone will be able to pick up the draft PR https://github.com/ros2/rosbag2/pull/1899 and finalize it by adopting it to humble keeping API/ABI compatibility will be nice.

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-kilted-kaiju-release/43902/1