rclpy icon indicating copy to clipboard operation
rclpy copied to clipboard

Add benchmark tests

Open sloretz opened this issue 3 years ago • 6 comments

This PR adds benchmarks that measure how quickly different types of entities are handled. This would be useful for testing performance improvements.

The benchmark tests are skipped unless AMENT_RUN_PERFORMANCE_TESTS is set, which is the same behavior as as ament_add_google_benchmark_test.

WIP because this needs a new rosdep key python3-pytest-benchmark for https://pypi.org/project/pytest-benchmark/ Requires https://github.com/ros/rosdistro/pull/33918

sloretz avatar Jul 19 '22 23:07 sloretz

Do we have any plan to integrate this kind of benchmark in CI/CD so that it can detect the performance degression?

There are benchmark jobs run for each release on build.ros2.org. Here's the one for Rolling https://build.ros2.org/view/Rci/job/Rci__benchmark_ubuntu_jammy_amd64/ . It looks like it runs nightly, but I think it wouldn't show an issue until the package got released, and I don't think there's an automatic comparison with past jobs.

sloretz avatar Jul 25 '22 18:07 sloretz

@ros-pull-request-builder retest this please

sloretz avatar Jul 25 '22 22:07 sloretz

CI (build: --packages-up-to rclpy test: --packages-select rclpy)

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status

And one with the benchmark tests enabled (build: -DAMENT_RUN_PERFORMANCE_TESTS=ON --packages-up-to rclpy test: --packages-select rclpy)

  • Linux Build Status

sloretz avatar Jul 25 '22 22:07 sloretz

Benchmark tests one more time with https://github.com/ros2/ci/pull/674

  • Linux: Build Status

sloretz avatar Jul 27 '22 18:07 sloretz

CI LGTM!

sloretz avatar Jul 27 '22 20:07 sloretz

CI re-run

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status

Benchmark job: Build Status

sloretz avatar Oct 14 '22 01:10 sloretz