Potential memory leak on Maniskill3?

[IrvingF7]

Hi!

Thanks for open-sourcing this awesome project.

Recently, I switched to the maniskill3 branch, and I noticed that I have been getting OOM issues if I switch between too many envs.

My workflow is roughly as follows:

I make env A, do some parallel testing, make sure to close and delete this env by calling env.close() and del env, and make env B, rinse and repeat. All in a single process.

But I noticed that the VRAM does not drop when an env is closed.

I double-checked the time, and the time at which VRAM increases is indeed the time a new env is made.

The specific error message I got is this

RuntimeError: Unable to create GPU parallelized camera group. If the error is about being unable to create a buffer, you are likely using too many Cameras. Either use less cameras (via less parallel envs) and/or reduce the size of the cameras. Another common cause is using a memory intensive shader, you can try using the 'minimal' shader which optimizes for GPU memory but disables some advanced functionalities. Another option is to avoid rendering with the rgb_array mode / using the human render cameras as they can be more memory intensive as they typically have higher resolutions for the purposes of visualization.

I came across this issue on Maniskill's main repo, but it seems like the API to manually clear the assets is no longer in the codebase anymore