isaac-sim
[Question] Tiled Camera Keeps Accumulating Frames in VRAM
I'm using the latest IsaacLab version, alongside IsaacSim 4.5 on a system 16GB of VRAM. I'm currently working on training an RL agent to fly in a cluttered environments using cameras using the OmniDrones library (and its port to IsaacSim 4.5). For the camera, I'm using TiledCamera with distance_to_image_plane, and a resolution of 80x60. As the sim runs, the VRAM usage keeps increasing, it doesn't hold steady or decrease at any point. I added my environment code below in the env file, I'm not sure if there's a clear cache sort of function that I missed, or should I re-initialize the cameras or what. I also added the WandB output to show what's going on. I don't think this is a result of the number of parallel environments I have, as I keep encountering the same issue regardless of the high/low env count, VRAM usage starts out reasonable, at like 9GB, then climbs to 16GB where it crashes. The sim collects enough experience to run through several PPO iterations as well, so I know that the issue does not stem from encountering the PPO model for the first time, the issue is with the camera itself. Finally, I just want to point out that in the environment file I provided, the camera information is not being processed by any DL model, currently I'm just collecting it without any processing, so the VRAM issues don't stem from any image processing either.
Should something change in the pre_sim_step/post_sim_step/reset_idx functions? I'm stuck here. Thanks for any help that you can provide.
The OmniDrones library link if it proves useful: https://github.com/btx0424/OmniDrones
Thank you for posting this. How many GPUs are you using? Could you share details about your platform? Thanks.
Hi, I'm using a single desktop version RTX 4080 with 16GB VRAM, a Ryzen7 5800x3D alongside 32GB of system RAM. The GPU is running "Driver Version: 560.35.05", "CUDA Version: 12.6". The conda environment has torchrl-nightly, tensordict-nightly, and PyTorch 2.7.0, though I don't think the library versions are the cause of the issue.
Thanks! Do you see a similar increase in GPU memory allocated when running tutorials, such as the carpool camera environment?
DId you solve this problem, i encounter a same problem when using camera for RL @tkzemrhn
Thank you for following up. Please try the recommended driver for Isaac Sim, 535.129.03. If you still see this issue, please open a new issue as a bug report. I will close this question for now. Thanks.