rai
rai copied to clipboard
benchmark execution issues
Describe the bug Investigate issues that sometimes occurs when running benchmarks:
- Model (qwen3:14b for example) sometimes gets stuck on thinking till timeout
- When running long manipulation benchmark with large models that take a lot of RAM, the python process after couple hours, gets killed because of RAM usage, which suggests that there is some memory leakage.
- When a large model takes up a lot of VRAM, there is no space for openset models and program thrors CUDA OOM, should we adress that issue?
To Reproduce
- Run manipulation benchmark with qwen3:14b, no thinking
- Run manipulation benchmark with large model like 32GB on CPU
- Run manipulation benchmark with large model like 32GB on GPU
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Platform
- OS: Ubuntu 22.04
- ROS 2 Humble
Version Release number or commit hash.
Additional context Add any other context about the problem here.