rai icon indicating copy to clipboard operation
rai copied to clipboard

benchmark execution issues

Open jmatejcz opened this issue 5 months ago • 0 comments

Describe the bug Investigate issues that sometimes occurs when running benchmarks:

  • Model (qwen3:14b for example) sometimes gets stuck on thinking till timeout
  • When running long manipulation benchmark with large models that take a lot of RAM, the python process after couple hours, gets killed because of RAM usage, which suggests that there is some memory leakage.
  • When a large model takes up a lot of VRAM, there is no space for openset models and program thrors CUDA OOM, should we adress that issue?

To Reproduce

  1. Run manipulation benchmark with qwen3:14b, no thinking
  2. Run manipulation benchmark with large model like 32GB on CPU
  3. Run manipulation benchmark with large model like 32GB on GPU

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Platform

  • OS: Ubuntu 22.04
  • ROS 2 Humble

Version Release number or commit hash.

Additional context Add any other context about the problem here.

jmatejcz avatar Jul 25 '25 09:07 jmatejcz